-
Notifications
You must be signed in to change notification settings - Fork 55
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
141 additions
and
86 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,88 +1,143 @@ | ||
| # SuperRag | ||
|
|
||
| Super-performant RAG pipeline for AI Agents/Assistants. | ||
|
|
||
| ## API | ||
|
|
||
| ### POST /api/v1/ingest | ||
|
|
||
| Input example: | ||
| ```json | ||
| { | ||
| "files": [ | ||
| { | ||
| "type": "PDF", | ||
| "url": "https://path-to-my-file.pdf" | ||
| } | ||
| ], | ||
| "vector_database": { | ||
| "type": "qdrant", | ||
| "config": { | ||
| "api_key": "my_api_key", | ||
| "host": "my_qdrant_host" | ||
| } | ||
| }, | ||
| "index_name": "my_index", | ||
| "encoder": { | ||
| "type": "openai", | ||
| "name": "text-embedding-3-small", | ||
| "dimensions": 1536 # encoder depends on the provider and model | ||
| }, | ||
| "webhook_url": "https://my-webhook-url" | ||
| } | ||
| ``` | ||
|
|
||
| ### POST /api/v1/query | ||
|
|
||
| Input example: | ||
| ```json | ||
| { | ||
| "input": "A query", | ||
| "vector_database": { | ||
| "type": "qdrant", | ||
| "config": { | ||
| "api_key": "my_api_key", | ||
| "host": "my_qdrant_host" | ||
| } | ||
| }, | ||
| "index_name": "my_index", | ||
| "encoder": { | ||
| "type": "openai", | ||
| "name": "text-embedding-3-small", | ||
| }, | ||
| "interpreter_mode": False, # Set to True if you wish to run computation Q&A with a code interpreter | ||
| "session_id": "my_session_id" # keeps micro-vm sessions and enables caching | ||
| } | ||
| ``` | ||
|
|
||
| ### DELETE /api/v1/delete | ||
|
|
||
| Input example: | ||
| ```json | ||
| { | ||
| "file_url": "A file url to delete", | ||
| "vector_database": { | ||
| "type": "qdrant", | ||
| "config": { | ||
| "api_key": "my_api_key", | ||
| "host": "my_qdrant_host" | ||
| } | ||
| }, | ||
| "index_name": "my_index", | ||
| } | ||
| ``` | ||
|
|
||
| ## Supported file types | ||
|
|
||
| - TXT | ||
| - MARKDOWN | ||
| - PPTX | ||
| - DOCX | ||
| <div align="center"> | ||
| <img width="100px" src="https://github.com/homanp/superagent/assets/2464556/eb51fa38-4a2a-4c41-b348-d3c1abc04234" /> | ||
| <h1>Super-Rag 🥷</h1> | ||
| <p> | ||
| <b>Super performant RAG pipeline for AI apps.</b> | ||
| </p> | ||
| <br> | ||
| <br> | ||
| <br> | ||
| <p align="center"> | ||
| <a href="#-key-features">Features</a> • | ||
| <a href="#-installation">Installation</a> • | ||
| <a href="#-how-to-use">How to use</a> | ||
| <a href="#-docs">Docs</a> | ||
| </p> | ||
| </div> | ||
|
|
||
| ## Supported vector databases | ||
|
|
||
| - qdrant | ||
| - pinecone | ||
| - weaviate | ||
| - astra | ||
| ## Key features | ||
| - Supports multiple document formats and vector databases. | ||
| - Provides a production ready REST API. | ||
| - Includes options for encoding data using different encoding models both propriatory and open source. | ||
| - Built in code interpreter mode for computational question & answer scenarios. | ||
| - Allows session management through unique IDs for caching purposes. | ||
|
|
||
| ## Installation | ||
|
|
||
| 1. Clone the repository | ||
| ```bash | ||
| git clone https://github.com/superagent-ai/super-rag | ||
| cd super-rag | ||
| ``` | ||
|
|
||
| 2. Setup virtual environment | ||
| ```bash | ||
| # Using virtualenv | ||
| virtualenv env | ||
| source env/bin/activate | ||
| # Or using venv | ||
| python3 -m venv env | ||
| source env/bin/activate | ||
| ``` | ||
|
|
||
| 3. Install requried packages | ||
| ```bash | ||
| poetry install | ||
| ``` | ||
|
|
||
| 4. Rename `.env.example` to `.env` and set your environment variables | ||
|
|
||
| 5. Run server | ||
| ```bash | ||
| uvicorn main:app --reload | ||
| ``` | ||
|
|
||
| ## How to use | ||
| Super-Rag comes with a built in REST API powered by FastApi. | ||
|
|
||
| ### Ingest documents | ||
| ```json | ||
| // POST: /api/v1/ingest | ||
| // Payload | ||
| { | ||
| "files": [{ | ||
| "url": "https://arxiv.org/pdf/2210.03629.pdf" | ||
| }], | ||
| "vector_database": { | ||
| "type": "qdrant", | ||
| "config": { | ||
| "api_key": "YOUR API KEY", | ||
| "host": "THE QDRANT HOST" | ||
| } | ||
| }, | ||
| "encoder": { | ||
| "type": "openai", | ||
| "name": "text-embedding-3-small", | ||
| "dimensions": 1536 // encoder depends on the provider and model | ||
| }, | ||
| "index_name": "YOUR INDEX", | ||
| "webhook_url": "https://webhook.site/0e217d1c-49f1-424a-9992-497db09f7793" | ||
| } | ||
| ``` | ||
|
|
||
| ### Query documents | ||
| ```json | ||
| // POST: /api/v1/query | ||
| // Payload | ||
| { | ||
| "input": "What is ReAct", | ||
| "vector_database": { | ||
| "type": "qdrant", | ||
| "config": { | ||
| "api_key": "YOUR API KEY", | ||
| "host": "THE QDRANT HOST" | ||
| } | ||
| }, | ||
| "index_name": "YOUR INDEX", | ||
| "interpreter_mode": true, | ||
| "encoder": { | ||
| "type": "cohere", | ||
| "name": "embed-multilingual-light-v3.0", | ||
| "dimensions": 384 | ||
| }, | ||
| "exclude_fields": ["metadata"], | ||
| "session_id": "test" | ||
| } | ||
| ``` | ||
|
|
||
| ### Delete document | ||
| ```json | ||
| // POST: /api/v1/delete | ||
| // Payload | ||
| { | ||
| "file_url": "A file url to delete", | ||
| "vector_database": { | ||
| "type": "qdrant", | ||
| "config": { | ||
| "api_key": "YOUR API KEY", | ||
| "host": "THE QDRANT HOST" | ||
| } | ||
| }, | ||
| "index_name": "my_index", | ||
| } | ||
| ``` | ||
|
|
||
| ## Supportd encoders | ||
| - OpenAi | ||
| - Cohere | ||
| - HuggingFace | ||
| - FastEmbed | ||
|
|
||
|
|
||
| ## Supported vector databases | ||
| - Weaviate | ||
| - Qdrant | ||
| - Weaviate | ||
| - Astra | ||
| - Supabase (coming soon) |