Native (de-)serialization based on Rust and PyO3

[124C41p]

Proof of concept
Only capable of deserializing (nested) Messages with primitive fields No handling of lists, maps, enums, .. implemented yet
See example.py for a working example

[cetanu]

Very cool, I've reviewed so far and looks good. Will wait for the remaining changes.

[124C41p]

Current state of progress:

I think deserialization is feature complete now. However, the code base is a complete mess, so I will try to increase code quality over the next commits. I would recommend not to review before the refactoring is done.

There are two tests failing which I don't know how to fix (or whether they can be fixed at all). I think the problem is related to the fact that I am converting between Rust HashMaps and Python Dicts, and that those are iterated in different orders. So when performing a (de-)serialization round trip of a message containing a map field, the protobuf wire format ends up in a different order than before. (Apparently, the order is not even deterministic.) So it is hard to make assertions about the wire format.

[Gobot1234]

I think maintaining insertion order is a pretty big deal since its guaranteed by the default dict implementation, it might be worth looking into IndexMap

[Gobot1234]

Also thank you for all your work on this, I'll try and look into getting CI set up for this so it can compile

[124C41p]

I think maintaining insertion order is a pretty big deal since its guaranteed by the default dict implementation, it might be worth looking into IndexMap

Good to know. Unfortunately, prost-reflect automatically deserializes map fields into HashMaps. If preserving order is important to us, I'll have to work around this by not declaring map fields as such, but as ordinary repeated fields instead. Then prost-reflect will deserialize into Vec<_> which can be converted into Python Dicts with order preserved.

[124C41p]

Should be feature complete now.

[124C41p]

I think the extension module is finished now. What remains is proper integration into the betterproto project. This includes:

• configuring betterproto-extras as an optional dependency (opt in / opt out / default with Python fallback / whatever you prefer)
• configuring tests/benchmarks to run with and without native (de-)serialization
• setting up cross compilation and deployment to PyPI

For all these tasks I'll need support by project maintainers. Please let me know how to proceed.

[Gobot1234]

I'd prefer if it was opt in personally, configuring benchmarks is something I'd like to get working (maybe take a look at #308) and the best way to build things I've found is with https://github.com/pypa/cibuildwheel

[124C41p]

I don't know cibuildwheel - it might be a good fit. But have you considered PyO3/maturin-action? It is especially designed to build maturin projects (and works very well for me). Also maturin autogenerates a CI config file when creating a new project (by executing maturin new). It just needs a few modifications to build for Python >= 3.7.

However, I do struggle with configuring Poetry: The problem is that betterproto now has a dependency which is part of the same repository but has not been published so far. I don't know how to set things up so that the CI scripts of betterproto can run. Do we need to move betterproto-extras into a separate repository?

[Gobot1234]

Superseded by #545