Data model: Create an individual embedding table per collection

[ckurze]

Issue you'd like to raise.

Currently, all embeddings are stored in one single "embeddings" table. This has various downsides:

• All embeddings need the same vector length
• Indexes get very large
• Hard/Complex to manage privileges on database side
• Unnecessary join on potentially large amounts of data.

Suggestion:

CrateDB should not create a table "collections" and "emgeddings", but crate a new table that gets the name of the provided collection name in langchain which holds all embeddings.

[amotl]

Hi there,

thank you for writing in, I agree with your proposal.

However, it will need a major rewrite of the data model and adapter, which is currently based on SQLAlchemy, so it may lack some flexibility to do just anything. With your request in mind, another patch has already been submitted to improve the situation in this regard.

• CrateDB vector: Improve SQLAlchemy model factory #13

We will see how the situation can be improved further.

With kind regards,
Andreas.

[amotl]

Hi again,

there is a patch now that makes an attempt to improve the situation.

• CrateDB vector: Refactor SQLAlchemy data model to provide two storage strategies #20

With kind regards,
Andreas.