Memory Requirement for FAISS

[fahmirzldi]

Hello, may I know the memory required to run Faiss, TCT-ColBERT (v2) model on the MS MARCO passage ranking task?

I tried to run

python -m pyserini.search.faiss --topics msmarco-passage-dev-subset --index msmarco-v1-passage.tct_colbert-v2-hnp --encoded-queries tct_colbert-v2-hnp-msmarco-passage-dev-subset --threads 12 --batch-size 384 --output run.msmarco-passage.tct_colbert-v2.bf.tsv --output-format msmarco

but face MemoryError: std::bad_alloc error

I am quite new on IR and indexing so any quick explanation on how to calculate the memory as I will use bigger index like BioASQ would be appreciated, thanks!

[lintool]

You'll need at least as much RAM as the size of the index on disk.