CrateDB vector: Refactor SQLAlchemy data model to provide two storage strategies #20
Commits on Dec 1, 2023
-
-
CrateDB vector: Add vector store support
The implementation is based on the `pgvector` adapter, as both PostgreSQL and CrateDB share similar attributes, and can be wrapped well by using the same SQLAlchemy layer on top.
-
-
-
CrateDB loader: Add document loader support
The implementation is based on the generic SQLAlchemy document loader.
-
CrateDB memory: Add conversational memory support
The implementation is based on the generic `SQLChatMessageHistory`.
-
CrateDB vector: Fix usage when only reading, and not storing
When not adding any embeddings upfront, the runtime model factory was not able to derive the vector dimension size, because the SQLAlchemy models have not been initialized correctly.
-
CrateDB vector: Improve SQLAlchemy model factory
From now on, _all_ instances of SQLAlchemy model types will be created at runtime through the `ModelFactory` utility. By using `__table_args__ = {"keep_existing": True}` on the ORM entity definitions, this seems to work well, even with multiple invocations of `CrateDBVectorSearch.from_texts()` using different `collection_name` argument values. While being at it, this patch also fixes a few linter errors. -
CrateDB vector: Fix cascading deletes
When deleting a collection, also delete its associated embeddings.
-
CrateDB vector: Add CrateDBVectorSearchMultiCollection
It is a special adapter which provides similarity search across multiple collections. It can not be used for indexing documents.
-
CrateDB vector: Improve SQLAlchemy data model query utility functions
The CrateDB adapter works a bit different compared to the pgvector adapter it is building upon: Because the dimensionality of the vector field needs to be specified at table creation time, but because it is also a runtime parameter in LangChain, the table creation needs to be delayed. In some cases, the tables do not exist yet, but this is only relevant for the case when the user requests to pre-delete the collection, using the `pre_delete_collection` argument. So, do the error handling only there instead, and _not_ on the generic data model utility functions.
-
pgvector: Use SA's
bulk_save_objectsmethod for inserting embeddingsThe performance gains can be substantially.
-
CrateDB vector: Test non-deterministic values by using pytest.approx
The test cases can be written substantially more elegant.
-
-
CrateDB vector: Refactor SQLAlchemy data model to provide two strategies
- StorageStrategy.LANGCHAIN_PGVECTOR Reflects the vanilla way the pgvector adapter manages the data model: There is a single `collection` table and a single `embedding` table. - StorageStrategy.EMBEDDING_TABLE_PER_COLLECTION Reflects a more advanced way to manage the data model: There is a single `collection` table, and multiple `embedding` tables, one per collection. The default storage strategy is `LANGCHAIN_PGVECTOR`. To configure an alternative storage strategy, invoke this snippet before doing any other operations using `CrateDBVectorSearch`: CrateDBVectorSearch.configure( storage_strategy=StorageStrategy.EMBEDDING_TABLE_PER_COLLECTION )
