Allow setting URL encoding/decoding to RFC 3986

[graemerocher]

Introduce new UrlEncodingKind enum that can be configured from the client configuration for encoding and the router configuration for decoding adapting the URL encoding/decoding to RFC-3986.

Micronaut 5 should likely default to RFC-3986 for everything except form decoding.

Fixes #11434
Fixes #10564

This resolves almost everything, but there appear to be issues with Netty's parameters() method of QueryStringDecoder. Which seems to be designed for decoding application/x-www-form-urlencoded not RFC-3986.

[graemerocher]

still working on this PR, since trying to address decoding

[graemerocher]

According to netty/netty@270e9d6 the decoding of + into space was supposed to be fixed, but it doesn't seem to be fixed for the parameters() method of QueryStringDecoder. We need to investigate if this is a bug in Netty or by design.

[sonarqubecloud]

Quality Gate failed

Failed conditions
68.8% Coverage on New Code (required ≥ 70%)
1 New Critical Issues (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[yawkat]

The thing is, the HTML spec says to "Let query be the result of running the application/x-www-form-urlencoded serializer with pairs and encoding". So if we stop encoding SP as '+', we are technically violating the HTML spec. It depends on which spec you find authoritative. On the parsing side, from what I can see, the URL spec also does not actually specify how to parse the query. It defines percent-decoding, but the query part is actually left alone during URI parsing, to be parsed as application/x-www-form-urlencoded later on. In my opinion:

• Micronaut is decoding the ‘+’ in the path variable passed to the controller code #10564 is obv. a bug and needs fixing
• HttpClient incorrectly encodes query params containing spaces with + and not %20 #11434 is arguable. Current behavior is in line with HTML. There should be a way to turn off the plus encoding (maybe through a "raw" parameter API), but I'm not sure that changing entirely to '%20' is the way to go
• For query decoding, plus still needs to be replaced with SP in the long term just to maintain compatibility with HTML

[jamesdh]

@yawkat agreed it's confusing between the HTML spec and 3986, but HTML is a content specification that operates at a layer above the URI/URL specification. At the very least there needs to be some mechanism for adapting to either the HTML spec's interpretation or 3986. WhatWG's URL spec even states:

The application/x-www-form-urlencoded format is in many ways an aberrant monstrosity, the result of many years of implementation accidents and compromises leading to a set of requirements necessary for interoperability, but in no way representing good design practices. In particular, readers are cautioned to pay close attention to the twisted details involving repeated (and in some cases nested) conversions between character encodings and byte sequences. Unfortunately the format is in widespread use due to the prevalence of HTML forms.

So I think the widespread confusion regarding this has led to some unfortunate implementations in tooling, as I've discovered a surprising amount recently while integrating with some 3rd party API's. I think the only pragmatic solution is to somehow be adaptable to which interpretation of the rules you wish to use on a per-client basis.