Quantization Investigation

[NirantK]

Consider this model from Xenova, there is a quantized model which is 120M instead of the 440-450M which I get from O3 quantization from Optimum.

Compare if the quantized model is as good as the 450M, O3 with an atol of 1e-3 and an O2 of 1e-4 — or there is something else happening there?

[NirantK]

See Static & Dynamic quantization here: https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization

[NirantK]

From @xenova, this script traverses the graph and collects operators for quantization
https://github.com/xenova/transformers.js/blob/main/scripts/convert.py