Memory-mapping and Zero-copy deserializers

[alexanderguzhva]

This PR introduces a backport of a combination of zilliztech/knowhere#996 and zilliztech/knowhere#1032 that allow to have memory-mapped and zerocopy indces.

The root underlying idea is that we replace certain std::vector<> containers with a custom faiss::MaybeOwnedVector<> container, which may behave either as std::vector<>, or as a view of a certain pointer / descriptor. We don't replace all the instances of std::vector<>, but the largest ones.

This change affects IndexFlatCodes-based and IndexHNSW CPU indices.

(done) alter IVF lists as well.
(done) alter binary indices as well.

Memory-mapped index works like this:

std::unique_ptr<faiss::Index> index_mm(
            faiss::read_index(filenamename.c_str(), faiss::IO_FLAG_MMAP_IFC));

In theory, it should be ready to be used from Python. All the descriptor management should be working.

Zero-copy index works like this:

#include <faiss/impl/zerocopy_io.h>

faiss::ZeroCopyIOReader reader(buffer.data(), buffer.size());
std::unique_ptr<faiss::Index> index_zc(faiss::read_index(&reader));

All the pointer management for faiss::ZeroCopyIOReader should be handled manually.
I'm not sure how to plug this into Python yet, maybe, some ref-counting is required.

(done) some refactoring

[alexanderguzhva]

@mdouze Would please be able to suggest a solution for the following problem. Because I changed the container type for IndexFlatCodes::codes field, it triggered some errors in python unit tests, which are related to these .codes (bcz the field type got changed). What would be the most meaningful solution for dealing with it? Basically, I'm not sure how to minimize the amount of backward-compatibility problems for the code that uses Faiss across the world, as I am not an expert swig user

[mdouze]

Thanks for tackling this. See a few inline reviews.
As long as the interface of MaybeOwnedVector is sufficiently close of std::vector, it should work seamlessly in Python (duck typing). I'll see if some adaptations to the swig interface are required.
It is not clear to me why the MappedFileIOReader is needed rather than the regular FileIOReader. Is it to maintain the ownership of the reader within the MaybeOwnedVector and close the file when deallocated?

[mdouze]

why do these two file disappear?

[alexanderguzhva]

duplicates

[mdouze]

possible to factorize with previous function?

[alexanderguzhva]

a good suggestion, let me see...

[mdouze]

Would it be possible to enable this via an IO flag ? The IVF memory mapped files work on regular FileIOReader

https://github.com/facebookresearch/faiss/blob/main/faiss/invlists/OnDiskInvertedLists.cpp#L771

[alexanderguzhva]

> OnDiskInvertedLists is unsupported on Windows.

my code allows mmap functionality on Windows as well. Let me take a look

[facebook-github-bot]

@mdouze has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mdouze]

This patch should fix the python test:
https://gist.github.com/mdouze/83eff60f99ae1a3cba524c6b8320dd8f
could you integrate it?

[alexanderguzhva]

It is not clear to me why the MappedFileIOReader is needed rather than the regular FileIOReader.

in order to have a correct support for Windows

[facebook-github-bot]

@mdouze has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[alexanderguzhva]

@mdouze I'm going to add IVFs and maybe Binary indices today as well, so please don't merge it yet :)

[alexanderguzhva]

@mdouze please let me know which other unit tests are useful to have
I've added the code for ArrayInvertedLists and binary vectors as well.

[alexanderguzhva]

the PR is ready to be reviewed

[facebook-github-bot]

@mdouze has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.