Improvements targeting serialization & data

[davidmorgan]

We'd like to make some combination of language and tooling improvements to better support serialization & data modeling.

More details to follow. We'll be asking for input on what people want to see--those interested can watch this issue for updates.

[bivens-dev]

I have two main ones that I’m particularly interested in.

This proposal from @munificent a few years ago: https://github.com/dart-lang/language/blob/main/working/tagged-strings/feature-specification.md

And I would really like to see proper core language support for the RFC 8949 Concise Binary Object Representation specification including code generation for its sister specification Concise Data Definition Language (CDDL)

[bivens-dev]

Oh and one last one that I would really love to see although I don’t know it fits the idea of a language or even tooling change more just something that I think is missing from the ecosystem which is a good RDF (Resource Description Framework) library.

After many years of work the newest version is due to be finalized quite soon I believe.

It’s a hugely popular and powerful way to model data in an interoperable and language neutral way and would pair incredibly well with a lot of the AI work Google is bringing to Flutter and elsewhere.

[davidmorgan]

Thanks @bivens-dev ... what would core language support for RFC8949 look like? I mean, compared to just doing it with codegen.

Technical suggestions for specific things we could do/support are useful, thanks; it'd also be good to hear about specific use cases that don't work well today that would benefit.

[bivens-dev]

Thanks for the clarifying question. Reading it back now I agree the phrase "core language support" is incredibly ambiguous, my apologies for that.

Specifically what I mean in the case of RFC8949 that there would exist a spec compliant streaming encoder and decoder that lived within dart:convert that allowed for converting between native Dart data types and CBOR. Note there is something like this with limited community support but it's a one person project and I am probably making the argument here that in terms of the strategic importance[1] of this as a universal serialization format that probably represents an unacceptable risk of sorts to many organizations.

That would be independent of the Concise Data Definition Language (CDDL) point I mentioned which I guess is the CBOR equivalent of a protobuf file / language and is what would be used as the basis of the code generation.

I would probably also make a similar argument for RDF in that I think it is already seeing incredibly strong adoption not just among businesses but also amongst governments. Some examples of that might include the Australian Government Linked Data Working Group and European Union's Linked Open Data.

Right now, I'm not really aware of any good solutions for working with RDF data at all within the Dart community and that too feels like something that will become a real problem before too long. Once again, I would make the same argument that given the level of trust people and organizations are going to need around a topic like that, community support doesn't feel like an appropriate option.

I think in concrete terms it too would ultimately look like a library that in addition to providing Dart native representations of key concepts like graphs, tuples, resources etc would also include a Codec to convert to and from the most common serialization formats.

Hopefully, that helps clarify things a bit more and generally provide some additional context as to why I think they are important both in terms of capabilities but also why I think they need certain quality guarantees that I don't think can actually come from the community at this time.

[1] My understanding of this is that CDDL is now the new defacto standard for representing data types and has been for some time already for all IETF standards. Some relevant examples using CBOR that immediately come to mind both from IETF and others include:

1. RFC 8152: CBOR Object Signing and Encryption (COSE)
2. RFC 8392: CBOR Web Token (CWT)
3. RFC 8742: Concise Binary Object Representation (CBOR) Sequences
4. RFC 8746: Concise Binary Object Representation (CBOR) Tags for Typed Arrays
5. RFC 8943: Concise Binary Object Representation (CBOR) Tags for Date
6. RFC 7252: The Constrained Application Protocol (CoAP)
7. RFC 9200: Authentication and Authorization for Constrained Environments
8. RFC 8428: Sensor Measurement Lists (SenML)
9. W3C: Recommendation: WebAuthn
10. Oasis: Open Command and Control (OpenC2)
11. W3C: Web of Things (WOT)
12. W3C: Digital Credentials API Which is also coming under various regulatory frameworks such as European Union's eIDAS 2.0.

[mosuem]

In case it wasn't on the radar yet: https://github.com/schultek/codable
cc @schultek

[davidmorgan]

Re: "tooling improvements", I plan on working on build_runner performance: dart-lang/build#3800

[Mike278]

ORs need the record treatment!

What?

Well one of the problems addressed by records is that if you decide "actually this function should return this type AND this other type" then you had to make a new class, which was a pretty steep syntatic cliff.

Currently, if you decide "actually this function should return this type OR this other type" then you make a sealed hierarchy of classes, which is a similarly steep cliff.

I don't think this needs a super powerful solution like (untagged) union types. IMO the desirable property of union types for most people is not that they save you from having to pick a name for each variant, but that the syntax is concise and inline.

One of the things I was planning on using macros for was seeing how concise I could make the syntax for defining what would eventually generate a sealed hierarchy of classes. I think the most relevant issues are #3021 and #2364 but those both still require writing new top-level definitions - the ideal would be writing it inline. For example:

// strawman syntax
.ConnectResult(.AuthExpired | .Offline | .Success({bool stable, Connection connection})) connect() {

}

// would internally generate
sealed class ConnectResult {}
class AuthExpired implements ConnectResult {}
class Offline implements ConnectResult {}
class Success implements ConnectResult {
  final bool stable;
  final Connection connection;
  Success({required this.stable, required this.connection});
}