Best way to persist Document Store

[darkquasar]

What is the best strategy to persist an ingestion pipeline or SimpleDocumentStore in a local directory, and updating the local persisted store without actually overwritting it every time I load new documents?

[dosubot[bot]]

Hey @darkquasar! I'm here to help you with any bugs, questions, or contributions you have. Let's tackle this together!

To persist an ingestion pipeline or SimpleDocumentStore in a local directory and update it without overwriting each time new documents are loaded, you can use the persist and load methods in the IngestionPipeline class. Implement a document de-duplication strategy like DocstoreStrategy.UPSERTS to update the store with new documents without overwriting existing ones.

Here's a strategy:

1. Persist the Pipeline:
   Use the persist method to save the pipeline's current state to a directory.

   pipeline.persist(persist_dir="./pipeline_storage")

2. Load the Pipeline:
   Use the load method to restore the pipeline from the saved state.

   pipeline.load(persist_dir="./pipeline_storage")

3. Handle Duplicates:
   Use DocstoreStrategy.UPSERTS to manage duplicates. This strategy updates the document store only if a document is new or its hash has changed.

   pipeline = IngestionPipeline(
       docstore_strategy=DocstoreStrategy.UPSERTS,
       # other parameters
   )

This approach ensures that new documents are added without overwriting existing ones, maintaining a persistent and up-to-date document store ^[1].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

Yes, you can persist only the metadata, such as the mappings between doc_id and file_hash, without storing the entire documents. In the LlamaIndex framework, the KVDocumentStore class provides a method _get_kv_pairs_for_insert that allows you to specify whether to store the full text of the document or not using the store_text parameter. By setting store_text=False, only metadata like doc_id and doc_hash will be stored, which helps in saving space ^[1].

To continue talking to Dosu, mention @dosu.

[darkquasar]

@dosu

so I would I use the KVDocumentStore inside the definition of an ingestion pipeline with an attached Simple Document Store?

[dosubot[bot]]

To use the KVDocumentStore within an IngestionPipeline alongside a SimpleDocumentStore, you can configure the pipeline to utilize both stores effectively. Here's a general approach:

1. Initialize the Document Stores:

   • Create instances of KVDocumentStore and SimpleDocumentStore. The SimpleDocumentStore can be initialized with a SimpleKVStore, which acts as its underlying key-value store.

2. Configure the IngestionPipeline:

   • When setting up the IngestionPipeline, you can specify the docstore parameter to use your KVDocumentStore or SimpleDocumentStore as needed. This allows the pipeline to manage document storage and retrieval using the specified store.

3. Set Document Storage Strategy:

   • Use the docstore_strategy parameter to define how documents are handled in terms of deduplication and updates. For example, you can use DocstoreStrategy.UPSERTS to update documents without overwriting existing ones.

Here's a simplified example:

from llama_index.core.ingestion.pipeline import IngestionPipeline
from llama_index.core.storage.docstore.keyval_docstore import KVDocumentStore
from llama_index.core.storage.docstore.simple_docstore import SimpleDocumentStore

# Initialize document stores
kv_docstore = KVDocumentStore(namespace="my_namespace")
simple_docstore = SimpleDocumentStore()

# Configure the ingestion pipeline
pipeline = IngestionPipeline(
    docstore=kv_docstore,  # or simple_docstore
    docstore_strategy=DocstoreStrategy.UPSERTS,
    # other parameters
)

# Use the pipeline to ingest documents
# pipeline.ingest(documents)

This setup allows you to leverage the capabilities of both KVDocumentStore and SimpleDocumentStore within the ingestion pipeline, depending on your specific requirements for document storage and metadata management.

To continue talking to Dosu, mention @dosu.

[darkquasar]

@dosu

But how can I configure the pipeline by leveraging _get_kv_pairs_for_insert and setting store_text=False?

[dosubot[bot]]

To configure the ingestion pipeline to leverage the _get_kv_pairs_for_insert method and set store_text=False, you can use the add_documents method of the KVDocumentStore class. This will ensure that only metadata is stored in the KVDocumentStore. Here's how you can do it:

# Assuming you have a list of nodes to add
nodes = [...]  # Your list of BaseNode objects

# Create an instance of KVDocumentStore
kvstore = ...  # Your instance of BaseKVStore
docstore = KVDocumentStore(kvstore=kvstore)

# Add documents with store_text set to False
docstore.add_documents(nodes, store_text=False)

By setting store_text=False, the _get_kv_pairs_for_insert method will not include the node's text data in the key-value pairs, effectively storing only the metadata ^[1].

To continue talking to Dosu, mention @dosu.