Add ability to disallow repeated keys in CBOR #2681
Conversation
Fixes Kotlin#2662 by adding a `visitKey` method to `CompositeDecoder`; map and set serializers should call this so that decoders have an opportunity to throw an error when a duplicate key is detected. A new config option `Cbor.allowDuplicateKeys` can be set to false to enable this new behavior. This can form the basis of a Strict Mode in the future. Also fixes a typo in an unrelated method docstring.
core/commonMain/src/kotlinx/serialization/SerializationExceptions.kt
Outdated
Show resolved
Hide resolved
| * duplicate keys are encountered. | ||
| */ | ||
| @ExperimentalSerializationApi | ||
| public fun visitKey(key: Any?) { } |
There was a problem hiding this comment.
I'm not happy with this name but couldn't think of anything better -- suggestions very welcome. :-)
There was a problem hiding this comment.
Checking duplication of keys should be more like a kind of common logics, and should be provided via a well-implemented utility class, rather than a function in top-level interface for each format to implement. IMO, most of them will be just some copy-paste.
There was a problem hiding this comment.
@xiaozhikang0916 The challenge is that the behaviour needs to be configurable. There is no central configuration mechanism in the library, only formats and SerialDescriptors. There is also the need to do this in an ABI/API compatible way. Formats already need to do various things to play nice/support the various library features. This would just be another one. It also makes sense as various formats have their own duplication semantics.
There was a problem hiding this comment.
I initially tried plumbing the information through on the deserialization layer and it went very badly. This is only the start of what would be required: dev...timmc:timmc/all-strict (although in practice it would be a config object, not a single boolean -- this was just a failed proof of concept.)
There was a problem hiding this comment.
I mean the duplication check in other formats ( if they need ), would share some common logic, like holding a set and putting keys in it. These common logic can be extracted to a utility class.
On the other hand, I am against adding this function in Decoding interface. It is implemented by each format with different behaviours, and only for internal using, which means not public, and users writing custom serializers should never call this function.
IMO, you may probably leave some instructions for format developers about how to achieve duplication check, rather than a public default funtion in this base interface, and make it open for users.
There was a problem hiding this comment.
It would be possible to circumvent the MapSerializer if you have a map specific decoder. This decoder could intercept the decoding of the key, and then check there (before the result is returned to the serializer).
| import kotlinx.serialization.DuplicateMapKeyException | ||
| import kotlin.test.Test | ||
|
|
||
| class CborStrictModeTest { |
There was a problem hiding this comment.
TODO: Add a (passing) test for duplicate key detection when deserializing a data class. I have a test locally that fails -- I think I need to add visitKey call to some inlined code somewhere, but I haven't found the appropriate callsite yet.
There was a problem hiding this comment.
I think I need to modify the serialization plugin's code generator in order to do this! Where does that code even live?
There was a problem hiding this comment.
@timmc The plugin code lives in the main Kotlin repository, but you don't need to go there.
I suppose you mean to detect multiple keys in object serializiation (like this Json { "a": "something", "a": "something else" }). To support that your format will need to plug in to decodeElementIndex and mark/record seen indices. Note that for some formats collections can be validly expressed as repeated key/value pairs (protobuf does this), so depending on how CBOR works you may need to do a slightly more extensive check.
There was a problem hiding this comment.
Hmm. I had previously tried modifying decodeElementIndex and wasn't getting the results I expected. Maybe I'll take another crack at it.
There was a problem hiding this comment.
Oh! I see, the decodeElementIndex in CborReader. Got it.
# Conflicts: # formats/cbor/api/kotlinx-serialization-cbor.klib.api
Fixes #2662 by adding a
visitKeymethod toCompositeDecoder; map and set serializers should call this so that decoders have an opportunity to throw an error when a duplicate key is detected.A new config option
Cbor.forbidDuplicateKeyscan be set to true to enable this new behavior. This can form the basis of a Strict Mode in the future.Also fixes a typo in an unrelated method docstring.