How to use LZ4_HC for compressing crate files ?

[mugulmd]

Hello :)

As far as I understand, USD crate files (.usdc) are compressed using the LZ4 algorithm, for which there is a custom implementation here: https://github.com/PixarAnimationStudios/OpenUSD/tree/release/pxr/base/tf/pxrLZ4.

However the README states that there is also a LZ4_HC algorithm available that we can use to get better compression rates (at the cost of compression/decompression speed), but I can't find a single option to enable it.

Could someone point me in the right direction please ?

[asluk]

It looks to me like the README was copied over from https://github.com/lz4/lz4 -- my interpretation is that it's that repository that provides the alternate algorithm https://github.com/lz4/lz4/blob/dev/lib/lz4hc.h and that it's not integrated with USD crate.

[mugulmd]

Hmmm yeah that's what feared...
Does this mean there is absolutely no control over the compression settings of crate files ?

[asluk]

I think we'll need someone from Pixar to weight in further. There are the tuning parameters in the pound-defines of the copied lz4 source, but I don't see anything else.

[spiffmon]

That is correct - no control. The option was not available in the initial version of the library when we added compression to crate. But the stats in the chart also do not seem very compelling, given the 20x slower compression speed... not compelling enough to warrant the extra complexity and impact on crate format version (or so it seems to us?). Note also that only the "footer" sections of a crate file are compressed, the bulk of the data (which tends to be largely floating point) is not compressed, and we have no plans to do so, given that it would defeat the important "zero copy" array-data access feature.

[mugulmd]

Makes sense, I just wanted to make sure I was not just missing something obvious 😅
Thanks !