Surprising behaviour when deserializing an Option from null when it contains a default value

[dleblanc]

With an example case class like:

case class Ex(opt: Option[String])

If I deserialize an optional value with "null", then I get the object built with 'opt' set to None, as expected.

However, if the optional field has a default value:

case class Ex(opt: Option[String] = Some("hiya"))

Then when I deserialize from json with null for that field, I get opt = Some("hiya") instead of None. This seems surprising to me, as I'd expect the default value to apply only when there is no value present for the field, but not when null is explicitly provided.

Is there a combination of configuration parameters I can use to achieve this? All combos of withTransient* didn't seem to help me here.

[plokhotnyuk]

@dleblanc First of all using an option that has Some as default value looks a bit odd. Why you cannot just have case class Ex(str: String = "hiya") instead? Could you please share a context that lead to your solution with an option?

It will help to find the best suitable solution. As example, if you want to distinguish null and missing key-value pair, you can use Option[Option[String]] with withSkipNestedOptionValues(true) compile-time configuration for codecs, like here

[andyczerwonka]

If I pass in an explicit null, I would not expect that to use the optional field, I would expect None. I guess we would need some kind of central flag to nominate that behavior.

[lbryan-citrine]

@plokhotnyuk can we get an update here? We're not able to adjust the data model unfortunately

[plokhotnyuk]

@dleblanc @andyczerwonka @lbryan-citrine Currently, as a workaround you can use a custom codec for the option type together with the withTransientNone(false) configuration option for Ex codec:

case class Ex(opt: Option[String] = Some("hiya"), level: Option[Int] = Some(10))

object CustomOptionCodecs {
  implicit val intCodec: JsonValueCodec[Int] = make[Int]
  implicit val stringCodec: JsonValueCodec[String] = make[String]

  implicit def optionCodec[A](implicit aCodec: JsonValueCodec[A]): JsonValueCodec[Option[A]] =
    new JsonValueCodec[Option[A]] {
      override def decodeValue(in: JsonReader, default: Option[A]): Option[A] =
        if (in.isNextToken('n')) in.readNullOrError(None, "expected 'null' or JSON value")
        else {
          in.rollbackToken()
          Some(aCodec.decodeValue(in, aCodec.nullValue))
        }

      override def encodeValue(x: Option[A], out: JsonWriter): Unit =
        if (x eq None) out.writeNull()
        else aCodec.encodeValue(x.get, out)

      override def nullValue: Option[A] = None
    }
}

import CustomOptionCodecs._

val codecOfEx = make[Ex](CodecMakerConfig.withTransientNone(false))

The need of manual derivation of codecs for concrete types used as A in Option[A] is the main downside for this cross Scala version implementation.

Changing of current behavior of withTransient*(false) configuration options could be breaking for users that relay on it and seems to be overwhelming for different combination of these options.

[lbryan-citrine]

Thanks for the clarification and work around

[lbryan-citrine]

Is there any way to avoid needing to declare the primitive codecs in the body of the custom codec?

e.g.

object CustomOptionCodecs {
  implicit val intCodec: JsonValueCodec[Int] = make[Int]
  implicit val stringCodec: JsonValueCodec[String] = make[String]