Try RWKV (better than LSTM for Language Modeling)

[BlinkDL]

Hi Byron,

Would you like to try RWKV for better compression?

Some experiments by Fabrice:
https://bellard.org/ts_server/ts_zip.html

You can do online training of RWKV and it learns fast.

Bo

[byronknoll]

That is a nice idea. If I have time, I might try it out in tensorflow-compress: https://github.com/byronknoll/tensorflow-compress

[xXWarMachineRoXx]

Isn't the compression ratio 7.382 for enwiki9 with RWKV?

[byronknoll]

Fabrice Bellard tried out a pre-trained RWKV model on enwik9: https://bellard.org/ts_zip/

For usage in tensorflow-compress, the model would not be pre-trained (i.e. it would be worse compression rate, but smaller decompressor size).