MuopDB - A vector database for AI memories

Introduction

MuopDB is a vector database for machine learning. Currently, it supports:

• Index type: HNSW, IVF, SPANN, Multi-user SPANN. All on-disk with mmap.
• Quantization: product quantization

Why MuopDB?

MuopDB supports multiple users by default. What that means is, each user will have its own vector index, within the same collection. The use-case for this is to build memory for LLMs. Think of it as:

• Each user will have its own memory
• Each user can still search a shared knowledge base.

All users' indices will be stored in a few files, reducing operational complexity.

Quick Start

• Build MuopDB. Refer to this instruction.
• Prepare necessary data and indices directories. On Mac, you might want to change these directories since root directory is read-only, i.e: ~/mnt/muopdb/.

mkdir -p /mnt/muopdb/indices
mkdir -p /mnt/muopdb/data

• Start MuopDB index_server with the directories we just prepared using one of these methods:

# Start server locally. This is recommended for Mac.
cd target/release
RUST_LOG=info ./index_server --node-id 0 --index-config-path /mnt/muopdb/indices --index-data-path /mnt/muopdb/data --port 9002

# Start server with Docker. Only use this option on Linux.
docker-compose up --build

• Now you have an up and running MuopDB index_server.
  • You can send gRPC requests to this server (possibly with Postman).
  • You can use Server Reflection in Postman - it will automatically detect the RPCs for MuopDB.

Examples using Postman

1. Create collection

{
    "collection_name": "test-collection-2",
    "num_features": 10
}

2. Insert some data

{
    "collection_name": "test-collection-2",
    "high_ids": [
        0
    ],
    "low_ids": [
        4
    ],
    "high_user_ids": [
        0
    ],
    "low_user_ids": [
        0
    ],
    "vectors": [
        1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0
    ]
}

3. Flush

{
    "collection_name": "test-collection-2",
}

4. Query

{
    "collection_name": "test-collection-2",
    "ef_construction": 100,
    "record_metrics": false,
    "top_k": 1,
    "high_user_ids": [0],
    "low_user_ids": [0],
    "vector": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 9.0, 9.0, 9.0]
}

Plans

Phase 0 (Done)

• Query path
  • Vector similarity search
  • Hierarchical Navigable Small Worlds (HNSW)
  • Product Quantization (PQ)
• Indexing path
  • Support periodic offline indexing
• Database Management
  • Doc-sharding & query fan-out with aggregator-leaf architecture
  • In-memory & disk-based storage with mmap

Phase 1 (Done)

• Query & Indexing
  • Inverted File (IVF)
  • Improve locality for HNSW
  • SPANN

Phase 2 (Done)

• Query
  • Multiple index segments
  • L2 distance
• Index
  • Optimizing index build time
  • Elias-Fano encoding for IVF
  • Multi-user SPANN index

Phase 3 (Ongoing)

• Features
  • Delete vector from collection
• Database Management
  • Segment optimizer framework
  • Write-ahead-log
  • Segments merger
  • Segments vacuum

Building

• Install prerequisites:
  • Rust: https://www.rust-lang.org/tools/install
  • Make sure you're on nightly: rustup toolchain install nightly
  • Libraries

# MacOS (using Homebrew)
brew install hdf5 protobuf openblas

# Linux (Arch-based)
# On Arch Linux (and its derivatives, such as EndeavourOS, CachyOS):
sudo pacman -Syu hdf5 protobuf openblas

# Linux (Debian-based)
sudo apt-get install libhdf5-dev libprotobuf-dev libopenblas-dev

• Build from Source:

git clone https://github.com/hicder/muopdb.git
cd muopdb

# Build
cargo build --release

# Run tests
cargo test --release

Contributions

This project is done with TechCare Coaching. I am mentoring mentees who made contributions to this project.