CUDA kernel of weight only quantization

[Sekri0]

As mentioned in README, [Note that due to the limitations of AutoGPTQ kernels, the real quantization of weight-only quantization can only lead memory reduction, but with slower inference speed.]
I'm a little confused. Does this mean that ABQ-LLM's weight-only quantization directly reuses GPTQ's cuda kernel？

[zengchao0424]

Thanks for your attention to our work. Matrix multiplication of int and float is not supported, but based on our experience in model optimization, the effect of int16 and float16 will be basically aligned (sd or llm).

So I suggest you try W2Aint16. In this case, you can directly use our operator for acceleration. Our operator is suitable for W2 scenarios.