CrateDB: Add support for SQLRecordManager [DOES NOT WORK] #18
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The implementation is based on the `pgvector` adapter, as both PostgreSQL and CrateDB share similar attributes, and can be wrapped well by using the same SQLAlchemy layer on top.
The implementation is based on the generic SQLAlchemy document loader.
The implementation is based on the generic `SQLChatMessageHistory`.
When not adding any embeddings upfront, the runtime model factory was not able to derive the vector dimension size, because the SQLAlchemy models have not been initialized correctly.
From now on, _all_ instances of SQLAlchemy model types will be created
at runtime through the `ModelFactory` utility.
By using `__table_args__ = {"keep_existing": True}` on the ORM entity
definitions, this seems to work well, even with multiple invocations
of `CrateDBVectorSearch.from_texts()` using different `collection_name`
argument values.
While being at it, this patch also fixes a few linter errors.
When deleting a collection, also delete its associated embeddings.
It is a special adapter which provides similarity search across multiple collections. It can not be used for indexing documents.
The CrateDB adapter works a bit different compared to the pgvector adapter it is building upon: Because the dimensionality of the vector field needs to be specified at table creation time, but because it is also a runtime parameter in LangChain, the table creation needs to be delayed. In some cases, the tables do not exist yet, but this is only relevant for the case when the user requests to pre-delete the collection, using the `pre_delete_collection` argument. So, do the error handling only there instead, and _not_ on the generic data model utility functions.
The performance gains can be substantially.
It does not work, because this subsystem uses composite unique keys in
combination with an `ON CONFLICT DO UPDATE` operation, on behalf of the
model entity definition `UpsertionRecord`.
Because the composite uniqueness constraint is currently being emulated
already, it can't also emulate ON CONFLICT behaviour on top easily.
__table_args__ = (
UniqueConstraint("key", "namespace", name="uix_key_namespace"),
Index("ix_key_namespace", "key", "namespace"),
)
stmt = insert_stmt.on_conflict_do_update(
[UpsertionRecord.key, UpsertionRecord.namespace],
...
)
amotl
changed the title
amotl
force-pushed
the
cratedb
branch
5 times, most recently
from
511f004
to
dd64cd4
Compare
amotl
force-pushed
the
cratedb
branch
2 times, most recently
from
2379986
to
4e00dcb
Compare
amotl
force-pushed
the
cratedb
branch
3 times, most recently
from
890bac3
to
bf406c7
Compare
amotl
force-pushed
the
cratedb
branch
2 times, most recently
from
40bcf25
to
0a75262
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About
It does not work, because this subsystem uses composite unique keys in combination with an
ON CONFLICT DO UPDATEoperation, on behalf of the model entity definitionUpsertionRecord.Because the composite uniqueness constraint is currently being emulated already, it can't also emulate ON CONFLICT behaviour on top easily.
Code References
Error Message
CrateDB fails with
SQLParseException[Number of conflict targets (["key", "namespace"]) did not match the number of primary key columns ([uuid])], which is expected.