ArcticEmbedLEncoder

[vincent-4]

Starting PR for @b8zhong
Mostly looking for a sanity check of initial impl of src/main/java/io/anserini/encoder/dense/ArcticEmbedLEncoder.java

• Bindings generally follow Splade pattern

Not implemented yet:

• Component test for output verification
• (L2 Norm)
• End-to-end test with HNSW retrieval
• (L1 Check) and check with cached vs. ONNX model

Q: am I good to convert Arctic-embed-l to ONNX, then run with src/main/java/io/anserini/encoder/dense/ArcticEmbedLEncoder.java now?

[vincent-4]

Here is the full command used, for posterity

SHARDS=(00 01 02 03 04 05 06 07 08 09)
for shard in "${SHARDS[@]}"; do
    bin/run.sh io.anserini.search.SearchHnswDenseVectors \
        -index msmarco-v2.1-doc-segmented-shard${shard}.arctic-embed-l.hnsw-int8 \
        -topics tools/topics-and-qrels/topics.rag24.test.txt \
        -topicReader TsvString \
        -topicField title \
        -encoder ArcticEmbedL \
        -output runs/run.rag24.test.arctic-l-msv2.1.shard${shard}.txt \
        -hits 250 \
        -threads 32 \
        -efSearch 1000 \
        > logs/log.run.rag24.test.arctic-l-msv2.1.shard${shard}.txt 2>&1
done

[vincent-4]

Results using [2cc3e6f](https://github.com/castorini/anserini/pull/2694/commits/2cc3e6f3ee1887b6a1a35d84445e8e3fef98bb52)
# cat them
$ cat runs/run.rag24.test.arctic-l-msv2.1.shard0* > runs/run.rag24.test.arctic-l-msv2.1.txt

# cat them
$ cat runs/run.rag24.test.CACHEDarctic-l-msv2.1.shard0* > runs/run.rag24.test.CACHEDarctic-l-msv2.1.txt

# non cached
$ tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 \
    tools/topics-and-qrels/qrels.rag24.test-umbrela-all.txt \
    runs/run.rag24.test.arctic-l-msv2.1.txt
ndcg_cut_10             all     0.5162

# eval results
$ tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 \
    tools/topics-and-qrels/qrels.rag24.test-umbrela-all.txt \
    runs/run.rag24.test.CACHEDarctic-l-msv2.1.txt
ndcg_cut_10             all     0.5603

# Summary of Results:
# Non-cached NDCG@10: 0.5162
# Cached NDCG@10:     0.5603

Apparently the mean pooling strategy might cause the drop? 0.0441. Clearly too high.

[vincent-4]

(base) v4zhong@orca:/store/scratch/v4zhong/github/anserini$ SHARDS=(00 01 02 03 04 05 06 07 08 09)
for shard in "${SHARDS[@]}"; do
    bin/run.sh io.anserini.search.SearchHnswDenseVectors \
        -index msmarco-v2.1-doc-segmented-shard${shard}.arctic-embed-l.hnsw-int8 \
        -topics tools/topics-and-qrels/topics.rag24.test.txt \
        -topicReader TsvString \
        -topicField title \
        -encoder ArcticEmbedL \
        -output runs/run.rag24.test.arctic-l-msv2.1.shard${shard}.txt \
        -hits 250 \
        -threads 32 \
        -efSearch 1000 \
        > logs/log.run.rag24.test.arctic-l-msv2.1.shard${shard}.txt 2>&1
done
(base) v4zhong@orca:/store/scratch/v4zhong/github/anserini$ cat runs/run.rag24.test.arctic-l-msv2.1.shard0* > runs/run.rag24.test.arctic-l-msv2.1.txt
(base) v4zhong@orca:/store/scratch/v4zhong/github/anserini$ tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 \
    tools/topics-and-qrels/qrels.rag24.test-umbrela-all.txt \
    runs/run.rag24.test.arctic-l-msv2.1.txt
ndcg_cut_10             all     0.5603
(base) v4zhong@orca:/store/scratch/v4zhong/github/anserini$ 

After aligning with how Snowflake does the pooling stuff... it works. Results attached above

[vincent-4]

left official in for now just because I haven't updated the huggingface link to be on UW server

[vincent-4]

and of course both urls have to change at some point

[vincent-4]

Adding tokenization and inference tests

[lintool]

nit - can make wider to take up fewer lines?

[vincent-4]

Done!

[lintool]

@vincent-4 we're good to merge?

[vincent-4]

@vincent-4 we're good to merge?

Yes. Thanks!