Add zstd support

[garazdawi]

This PR adds a module to stdlib that wraps the Zstandard library. Zstandard is a compression library that allows for faster compression than zlib, but with about the same compression ratio.

Zstandard is vendored into erts, allowing us to also use it inside the VM for future optimizations.

The API is still a bit of a work in progress, but any opinions about what things should be named are very welcome. I think that I may rename zstd:chunk/2 to zstd:stream/2 and also possible break zstd:contex/1,2 into separate functions for compression and decompression context, but I've not decided yet.

This PR depends on #9315 for the doctests support.

[github-actions]

CT Test Results

    4 files    227 suites   1h 54m 56s ⏱️
3 659 tests 3 565 ✅  93 💤 1 ❌
4 753 runs  4 636 ✅ 116 💤 1 ❌

For more details on these failures, see this check.

Results for commit 71de6c2.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot

[essen]

The API is still a bit of a work in progress, but any opinions about what things should be named are very welcome. I think that I may rename zstd:chunk/2 to zstd:stream/2 and also possible break zstd:contex/1,2 into separate functions for compression and decompression context, but I've not decided yet.

Please keep the naming close to the original API (zstd.h and friends), it's easier to translate other examples to Erlang that way, plus all the documentation from zstd.h becomes usable. For example if I search zstd.h for "chunk" I find nothing relevant, since it's called "stream" there.

[crertel]

Just wanted to say thanks for this!

[garazdawi]

Updated the API to be zstd:stream/2 and zstd:dict/2. You can view the docs here: https://erlang.github.io/prs/9316/lib/stdlib-6.2/doc/html/zstd.html

[essen]

Only one tiny comment following a quick look at the docs: all functions are verbs except context and dict. That's probably fine, although in my mind if there's no verb the function isn't going to do much work (like a getter), which definitely isn't the case for dict.

[garazdawi]

@essen John mentioned that in the past you have used zlib:deflate/3 from multiple processes, are you still doing that and if so, do you imagine you would like to do the same with zstd?

[essen]

@essen John mentioned that in the past you have used zlib:deflate/3 from multiple processes, are you still doing that and if so, do you imagine you would like to do the same with zstd?

Yes the deflateInit is done in one process and the actual deflate in another for Websocket. It's still like this today. Cowboy creates the contexts and then calls zlib:set_controlling_process.

Websocket has no zstd compression extension RFC/draft so today this isn't necessary for zstd. I guess for deflate the init could be deferred as well, but it's just more convenient to fully prepare deflate while doing the Websocket negotiation.

[garazdawi]

Allright, then I will keep the contexts process bound for now and if the need arises later we can add that functionality. Thanks for the prompt response!

[essen]

Thank you!!