Is it possible to create a container format and then convert other formats to that format?

[camenduru]

All contain the same thing, but in different boxes 📦. If this is correct, can you please create a single container and converters to convert everything below to that container?

Format	Safe	Zero-copy	Lazy loading	No file size limit	Layout control	Flexibility	Bfloat16
pickle (PyTorch)	✗	✗	✗	🗸	✗	🗸	🗸
H5 (Tensorflow)	🗸	✗	🗸	🗸	~	~	✗
SavedModel (Tensorflow)	🗸	✗	✗	🗸	🗸	✗	🗸
MsgPack (flax)	🗸	🗸	✗	🗸	✗	✗	🗸
Protobuf (ONNX)	🗸	✗	✗	✗	✗	✗	🗸
Cap'n'Proto	🗸	🗸	~	🗸	🗸	~	✗
Arrow	?	?	?	?	?	?	✗
Numpy (npy,npz)	🗸	?	?	✗	🗸	✗	✗
SafeTensors	🗸	🗸	🗸	🗸	🗸	✗	🗸

[julien-c]

for now you can already convert from pytorch pickle to safetensors from this Space: https://huggingface.co/spaces/safetensors/convert

[Narsil]

Well, this is very specific to transformers since operations happen based on weights names, so you would have to know how you named your tensors to know how to convert back&forth.

There are many ways to go about this.

safetensors.flax to safetensors.torch

There's no such concept as safetensors.flax nor safetensors.torch. Safetensors, saves tensors, and doesn't care where they come from or from what lib. The differences exists purely in what tensors (as in preferred shapes) both libraries use. (As I mention, it's a choice done within the libraries which revolve around channel order and sometimes transposed tensors, and is all linked on how the libraries made some choices). It has nothing to do with tensors themselves.

So you can load any safetensors file in Flax or PyTorch without any issues. But you might encounter issues when you try to associate said tensors to the weights of your NN.
for instance Convolution weights will probably require some transpose if they are loaded from flax and used onto PyTorch.

This will always have a cost in terms of time (since you basically have to duplicate the tensor), but since safetensors seems to load much faster than Flax, it should still be a net win.

[camenduru]

thanks @Narsil ❤ I thought safetensors can do cross conversion 😕 I was wrong. we are trying to solve this problem for more than a month 🥺 can you solve it

[Narsil]

Isn't this what you were looking for ? huggingface/diffusers#1739

[camenduru]

no, check the tittle "Allow Flax model to save as pytorch (If converted from pytorch)"

[camenduru]

https://github.com/camenduru/diffusers/blob/from_flax_v2/src/diffusers/modeling_pytorch_flax_utils.py