Add optional AVX512-FP16 arithmetic for the scalar quantizer.

[mulugetam]

PR #4025 introduced a new architecture mode, avx512_spr, which enables the use of features available since Intel® Sapphire Rapids. The Hamming Distance Optimization (PR #4020), based on this mode, is now used by OpenSearch to speed up the indexing and searching of binary vectors.

This PR adds support for AVX512-FP16 arithmetic for the Scalar Quantizer. It introduces a new Boolean flag, ENABLE_AVX512_FP16, which, when used together with the avx512_spr mode, explicitly enables avx512fp16 arithmetic.

Tests on an AWS r7i instance demonstrate up to a 1.6x speedup in execution time when using AVX512-FP16 compared to AVX512. The improvement comes from a reduction in path length.

---

-DFAISS_OPT_LEVEL=avx512:

$ numactl -C 2 ./build/perf_tests/bench_scalar_quantizer_distance --d=768 --n=2000 --iterations=20
----------------------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations UserCounters...
----------------------------------------------------------------------------------------------
QT_4bit/iterations:20                377313128 ns    377307894 ns           20 code_size=384
QT_4bit_uniform/iterations:20        375116351 ns    375113141 ns           20 code_size=384
QT_6bit/iterations:20                313387520 ns    313382880 ns           20 code_size=576
QT_8bit/iterations:20                256168739 ns    256166690 ns           20 code_size=768
QT_8bit_direct/iterations:20          86934297 ns     86933750 ns           20 code_size=768
QT_8bit_direct_signed/iterations:20  182414307 ns    182413034 ns           20 code_size=768
QT_8bit_uniform/iterations:20        199623822 ns    199621677 ns           20 code_size=768
QT_bf16/iterations:20                205998126 ns    205996720 ns           20 code_size=1.536k
QT_fp16/iterations:20                204291381 ns    204290326 ns           20 code_size=1.536k

$ numactl -C 2 ./build/perf_tests/bench_scalar_quantizer_accuracy --d=768 --n=2000 --iterations=20
----------------------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations UserCounters...
----------------------------------------------------------------------------------------------
QT_4bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=384 code_size_two=768k ndiff_for_idempotence=0 sql2_recons_error=0.284829
QT_4bit_uniform/iterations:20            0.000 ns        0.000 ns            0 code_size=384 code_size_two=768k ndiff_for_idempotence=0 sql2_recons_error=0.2845
QT_6bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=576 code_size_two=1.152M ndiff_for_idempotence=0 sql2_recons_error=0.0164574
QT_8bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=23.184k sql2_recons_error=1.32533m
QT_8bit_direct/iterations:20             0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=255.806
QT_8bit_direct_signed/iterations:20      0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=255.799
QT_8bit_uniform/iterations:20            0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=983.725u
QT_bf16/iterations:20                    0.000 ns        0.000 ns            0 code_size=1.536k code_size_two=3.072M ndiff_for_idempotence=0 sql2_recons_error=558.005u
QT_fp16/iterations:20                    0.000 ns        0.000 ns            0 code_size=1.536k code_size_two=3.072M ndiff_for_idempotence=0 sql2_recons_error=8.71347u

---

-DENABLE_AVX512_FP16 -DFAISS_OPT_LEVEL=avx512_spr

$ numactl -C 2 ./build/perf_tests/bench_scalar_quantizer_distance --d=768 --n=2000 --iterations=20
----------------------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations UserCounters...
----------------------------------------------------------------------------------------------
QT_4bit/iterations:20                235309050 ns    235307645 ns           20 code_size=384
QT_4bit_uniform/iterations:20        232136257 ns    232133724 ns           20 code_size=384
QT_6bit/iterations:20                285525339 ns    285522795 ns           20 code_size=576
QT_8bit/iterations:20                194723922 ns    194722583 ns           20 code_size=768
QT_8bit_direct/iterations:20          92415395 ns     92415183 ns           20 code_size=768
QT_8bit_direct_signed/iterations:20  166470157 ns    166469954 ns           20 code_size=768
QT_8bit_uniform/iterations:20        182553814 ns    182547301 ns           20 code_size=768
QT_bf16/iterations:20                222188800 ns    222187274 ns           20 code_size=1.536k
QT_fp16/iterations:20                198899549 ns    198898266 ns           20 code_size=1.536k

$ numactl -C 2 ./build/perf_tests/bench_scalar_quantizer_accuracy --d=768 --n=2000 --iterations=20
----------------------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations UserCounters...
----------------------------------------------------------------------------------------------
QT_4bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=384 code_size_two=768k ndiff_for_idempotence=0 sql2_recons_error=0.284803
QT_4bit_uniform/iterations:20            0.000 ns        0.000 ns            0 code_size=384 code_size_two=768k ndiff_for_idempotence=0 sql2_recons_error=0.284519
QT_6bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=576 code_size_two=1.152M ndiff_for_idempotence=0 sql2_recons_error=0.0164255
QT_8bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=27.191k sql2_recons_error=1.3546m
QT_8bit_direct/iterations:20             0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=255.806
QT_8bit_direct_signed/iterations:20      0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=255.799
QT_8bit_uniform/iterations:20            0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=1.0097m
QT_bf16/iterations:20                    0.000 ns        0.000 ns            0 code_size=1.536k code_size_two=3.072M ndiff_for_idempotence=0 sql2_recons_error=558.005u
QT_fp16/iterations:20                    0.000 ns        0.000 ns            0 code_size=1.536k code_size_two=3.072M ndiff_for_idempotence=0 sql2_recons_error=8.71347u

[alexanderguzhva]

@mulugetam do I get it correct that this PR introduces an optional tradeoff between the accuracy and the speed?

[mulugetam]

@mulugetam do I get it correct that this PR introduces an optional tradeoff between the accuracy and the speed?

Yes.

[mdouze]

Where does the loss in accuracy come from? because computation is performed as fp16 * fp16 instead of fp16-> fp32 then fp32 * fp32 ?

[alexanderguzhva]

yes, pure fp16 FMAD operations

[mulugetam]

@alexanderguzhva I'm not sure why this PR fails some of the pytest tests/test_*.py. On my local machine, the current main branch also fails some of these tests. Any ideas?