Information about hypothesis on Inference

[djuillard]

Hi,
Congratulations for your work : it's really helpful and methodology you apply is rigorous and clear.

Quick questions about the hypothesis you took on inference parameters, especially the quantization. If I am not mistaken, quantization has been set to 4 bits (BnB.4bits I guess), and this value is then applicable to OpenAI, Google, Anthropic...

I am wondering why you chose 4 bits ? It is likely that this piece of information is not disclosed by the AI vendors... and I read that 8 bits might be more often used than 4. What do you think ?

Thanks in advance,

Regards,

Damien

[samuelrince]

Hello @djuillard,

Thanks a lot for the feedback, we really appreciate it! 💚

The 4 bits hypothesis comes from the data extraction we did in early 2024 of the LLM-Perf Leaderboard. To get the full range from ~1B to 70B models, we had to look at the data with 4 bits quantization. So it is not really a choice, but more a constraint from the data source.

We plan to integrate more data sources in the near future to better estimate the linear regression model of energy/token function of the model size, including our own experiments. Hopefully, we will be able to conduct more experiments with different sets of optimizations and hardware configurations, thus progressively gaining in precision/robustness.

Happy to answer any further questions on the topic! 😉

[djuillard]

Thanks for the explanation.
I can imagine that the methodology you apply with linear regression should also work for models with 8 bits quantization; yet, what might still be missing in the end is the way closed vendors like OpenAI actually serve inference (in that case it might be a pure « guess » to opt for 4 bits or 8 bits to process the calculation). But at least, it is good to estimate the range of uncertainties between 4 and 8 when we talk about very large language models.

Thanks again.

[samuelrince]

Indeed, we have no real information related to how the models are actually hosted, we can only guess. We have some internal projects to do more precise benchmarks to see the influence of different deployments, like vLLM vs Pytorch for instance.