20_RAG_overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "db184fef-55be-4fb0-8432-c58ee8f03244",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## RAG in depth\n",
    "\n",
    "Today we will go over building a retrieval augmented generation chatbot in detail.\n",
    "\n",
    "The two main activities are:\n",
    "\n",
    "- Index Creation\n",
    "- Question Answering\n",
    "\n",
    "Most of the steps under each activity are optional, but may improve answer quality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2949fa99-caba-48f8-adcb-8cd163b4c990",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Index creation steps\n",
    "\n",
    "1. Load the documents\n",
    "2. Enhance document quality\n",
    "3. Split the documents into chunks\n",
    "4. Add metadata to the chunks\n",
    "5. Generate embeddings for the chunks\n",
    "6. Index the chunks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "038897f5-575c-4755-85be-ec2569d1736d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Load the documents\n",
    "\n",
    "The documents have to be converted to plain or markdown text format. This is more difficult than it seems. Documents may be in PDF, Powerpoint, or MS Word format, or you may need to fetch data from a database or slack.\n",
    "\n",
    "LlamaIndex has loaders for many different sources and file formats:\n",
    "- https://docs.llamaindex.ai/en/stable/module_guides/loading/\n",
    "\n",
    "I have found that Unstructured.io has high-quality loaders.\n",
    "- API: https://docs.unstructured.io/welcome\n",
    "- Open source: https://github.com/Unstructured-IO/unstructured?tab=readme-ov-file#installing-the-library\n",
    "- Integration with LlamaIndex: https://docs.unstructured.io/open-source/integrations#integration-with-llamaindex"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a48c704a-070b-4034-a3d7-e43f9f0ea5bd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Enhance document quality\n",
    "\n",
    "Some documents contain abbreviations or terms that can be understood only in the context in which they appear (e.g., does PO stand for post office or purchase order?). Or they contain tables or images that must be processed specially.\n",
    "\n",
    "This step generally involves studying your documents and writing custom code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e57665b8-15e2-46ec-a2ff-ed0f26189ec0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Split the documents into chunks\n",
    "\n",
    "Each document must be split into small (500-2000 characters) chunks of text, where each chunk is added to the index as a separate object. Later when we query the index, the most-relevant chunks are returned. The goal is to create chunks that are more-or-less self-contained: they contain enough information to answer a question but not too much. Splitting is so important that we will have a separate class on splitting.\n",
    "\n",
    "There are many kinds of splitters (you could even create your own that combines ideas from the ones below):\n",
    "- Sliding window: simply create chunks from every N characters with overlap\n",
    "- Structure: take the HTML or markdown structure into account to create chunks based upon headings and paragraphs.\n",
    "- Semantic: combine sentences with similar vectors into chunks.\n",
    "- Tree: create chunks at different levels of granularity; parent chunks may contain summaries.\n",
    "\n",
    "LlamaIndex has many types of splitters: https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a09360e-7d04-4a3a-aa09-891b4b6e00fc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Add metadata to the chunks\n",
    "\n",
    "Each chunk includes text, but it can also be useful to add additional information to each chunk to provide context for answering questions:\n",
    "- file name and markdown/html section headers\n",
    "- pointers to the previous, next, and parent chunks\n",
    "  - or maybe summaries of the previous + next chunks, or a summary of the parent chunk\n",
    "- entities related to the chunk (author, product name, etc.)\n",
    "\n",
    "The step generally involves studying your chunks and writing custom code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3968112-1902-43ce-adad-1844f7ddae9d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Generate embeddings for the chunks\n",
    "\n",
    "We need to generate embeddings (vectors) for each chunk in order to index the chunk. The better the embedding is able to capture the \"semantic meaning\" of the chunk, the more-likely your retrieved chunks will be relevant to the question. Popular embeddings are OpenAI, VoyageAI, and coHere.\n",
    "\n",
    "When generating the embeddings, a question to ask is: Should I generate the embedding based upon the chunk text only, or should I also include some or all of the metadata?\n",
    "\n",
    "LlamaIndex supports many embeddings: https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings/#list-of-supported-embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb18359b-a778-4760-8b4e-b775ef82a7f8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Index the chunks\n",
    "\n",
    "In addition to the embedding (which is called a \"dense\" embedding), some indexes also support sparse embeddings. It's possible to query both dense and sparse embeddings to improve the relevance of the retrieved chunks. This is called \"hybrid\" search.\n",
    "\n",
    "Several indexes support hybrid search. We will cover indexing and hybrid search in a separate class.\n",
    "\n",
    "LlamaIndex supports many indexes: https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores/\n",
    "\n",
    "You will notice that LlamaIndex makes a (confusing) distinction between vector stores, document stores, index stores, and other types of stores. We will cover this in the class on indexing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36654b20-fe75-4376-b938-06840495692a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Question-answering steps\n",
    "\n",
    "1. Transform the question \n",
    "2. Route the question to different indexes\n",
    "3. Query the index to retrieve chunks that are relevant to the question\n",
    "4. Post-process the chunks\n",
    "5. Generate a prompt and send it to the LLM\n",
    "6. Analyze the answer and possibly repeat these steps with a follow-on question"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6caa9ee8-16e9-44c5-b2d6-3843a6ba9ec9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Transform the question \n",
    "\n",
    "When we query the index, we generate an embedding for the question and compare it to the embeddings we generated for the chunks. But what if the question and the relevant chunks aren't that semantically similar? In this case we might want to augment or replace the question with one that contains more words/concepts that are likely to be found in the relevant chunks.\n",
    "\n",
    "For example, the question: \"why are there two priesthoods?\" doesn't contain the words Aaronic or Melchizedek. But the relevant chunks would likely contain both of those words.\n",
    "\n",
    "One way to do this, called HyDE, is to ask the LLM to guess an answer without looking at anything in the database. It's answer might contain hallucinations, but that's ok as long as contains words/concepts that are likely to be found in the relevant chunks. We use the original question and the guessed answer when generating the embedding for the question.\n",
    "\n",
    "Another way to do this is to ask the LLM to generate a few additional related questions to the user's question. You then retrieve results for the original question and the related questions and send the combined (deduplicated) results to the LLM. \n",
    "\n",
    "LlamaIndex supports HyDE and other query transformations:\n",
    "- https://docs.llamaindex.ai/en/stable/examples/query_transformations/HyDEQueryTransformDemo/?h=hyde\n",
    "- https://docs.llamaindex.ai/en/stable/examples/query_transformations/query_transform_cookbook/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a9d6ce4-d1bf-4d21-b4bb-a3a0e9e81212",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Route the question to different indexes\n",
    "\n",
    "Suppose you are building a customer-support chat-bot, and you have some documents that contain information about products from the product catalog, and other documents that contain information about shipping and how to return items for a refund from the customer support knowledgebase. Rather than putting all documents into a single index, you may be better off to create one index from the product catalog and a separate index from the customer support knowledgebase. Then there's less chance that you'll retrieve a chunk from the product catalog when a customer is asking how to return the \"moto g\" phone they just bought.\n",
    "\n",
    "If you have multiple indexes, the next step is to determine which index to query for an incoming question. This is called routing.\n",
    "\n",
    "LlamaIndex supports several kinds of routers: https://docs.llamaindex.ai/en/stable/module_guides/querying/router/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad44a35a-54e2-4240-a002-08e1d4021edf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Query the index to retrieve chunks that are relevant to the question\n",
    "\n",
    "This step involves generating an embedding for the question and using that embedding to query an index to find chunks with similar embeddings. \n",
    "\n",
    "One thing you can ask yourself at this point is: Can I extract metadata from the question and use it to filter the chunks returned? For example, if the question is \"tell me what Elder Holland said about adversity\" and you've extracted the author in each chunk's metadata, you could extract Elder Holland from the question and pass that to the index as a filter.\n",
    "\n",
    "LlamaIndex retriever support (the default usually works fine):\n",
    "- https://docs.llamaindex.ai/en/stable/module_guides/querying/retriever/\n",
    "- https://docs.llamaindex.ai/en/stable/module_guides/querying/retriever/retrievers/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82edfcb1-21a4-4942-b924-a2a6efc38baf",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Post-process the chunks\n",
    "\n",
    "You may find that the chunk texts don't contain enough context - that the LLM needs additional context to answer the question. The idea is that the chunk text that you generated the embedding from doesn't necessarily have to be the same as the chunk text that you send to the LLM. For exmaple, you may find it helpful to include one of the following when sending the chunk to the LLM:\n",
    "- the text of the previous and/or next chunks (or portions/summaries of those texts)\n",
    "- summary of the parent section\n",
    "\n",
    "Here are some examples augmenting nodes with previous and next chunks, or parents:\n",
    "- https://docs.llamaindex.ai/en/stable/examples/node_postprocessor/MetadataReplacementDemo/\n",
    "- https://docs.llamaindex.ai/en/stable/examples/retrievers/auto_merging_retriever/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979c05bf-34a9-40cd-b874-b8e41f5a8f26",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Post-process the chunks (continued)\n",
    "\n",
    "In addition to augmenting the chunk text, you may find it helpful to:\n",
    "- re-rank the chunks using a more-expensive ranking model to ensure that the most-relevant chunks get inserted into the prompt first\n",
    "- remove chunks whose similarity score lies below a certain threshold \n",
    "- remove sentences from chunks that aren't relevant to the question\n",
    "- summarize chunks to retain only information that is relevant to the question\n",
    "\n",
    "LlamaIndex supports a wide variety of post-processors, especially re-rankers\n",
    "- https://docs.llamaindex.ai/en/stable/module_guides/querying/node_postprocessors/\n",
    "- https://docs.llamaindex.ai/en/stable/module_guides/querying/node_postprocessors/node_postprocessors/\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b2f1622-df8f-494e-a2eb-08521199965e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Generate a prompt and send it to the LLM\n",
    "\n",
    "This step involves generating a prompt that includes instructions and the chunk texts and sending the prompt to the LLM to generate an answer to the question.\n",
    "\n",
    "There are four things to think about at this step:\n",
    "1. What should my instructions say?\n",
    "2. Should I include examples of \"question + chunks -> ideal answer\" in my prompt to help the LLM understand how I want it to answer the question, and if so, which examples should I use? A few examples often help.\n",
    "3. Should I include questions + answers from the chat history in the prompt so if the new question references something from a previous question or answer, the LLM will be able to understand the reference?\n",
    "4. Should I include all of the chunks at once, or should I include them iteratively?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bffa954f-610d-42d2-a70a-de5920a40f44",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Generate a prompt and send it to the LLM (continued)\n",
    "\n",
    "DSPy can help you come up with the best instructions and examples. Once you have created your index, you can use DSPy to optimize the prompt, then use the optimized prompt with LlamaIndex.\n",
    "\n",
    "LlamaIndex has code to:\n",
    "- customize your prompt: https://docs.llamaindex.ai/en/stable/examples/prompts/prompts_rag/\n",
    "- include chat history: https://docs.llamaindex.ai/en/stable/module_guides/deploying/chat_engines/\n",
    "- determine which chunks to include (you don't need to worry about this usually - including all chunks at once works fine most of the time): https://docs.llamaindex.ai/en/stable/module_guides/querying/response_synthesizers/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12558f7d-e376-4452-97d5-77b7674b4345",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Analyze the answer and possibly repeat these steps with a follow-on question\n",
    "\n",
    "Traditional question-answer chatbots return the answer to the user immediately, but what if before returning the answer to the user, you gave the question and answer to the LLM and asked if more work was needed to properly answer the question? Maybe the LLM determines that it needs to query additional information from the index before it can completely answer the question. Or maybe you determine that you should have asked the question initially from multiple points of view and then had a final LLM combine all of the answers together into a comprehensive answer. \n",
    "\n",
    "For lack of a better word, these kinds of workflows are called \"Agentic\". This is an advanced concept.\n",
    "\n",
    "Llamaindex has support for agents: https://docs.llamaindex.ai/en/stable/module_guides/deploying/agents/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b866024c-14ed-4cfb-8a08-c43cc824ec32",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# More documentation links\n",
    "\n",
    "LlamaIndex has a lot of documentation. It's not always up to date and it doesn't explain some things very well but that's pretty common with open-source software. If you run into issues or have any questions, please let me know. \n",
    "\n",
    "Here are a few additional links you may find useful:\n",
    "\n",
    "- https://docs.llamaindex.ai/en/stable/use_cases/q_and_a/\n",
    "- https://docs.llamaindex.ai/en/stable/examples/\n",
    "- https://docs.llamaindex.ai/en/stable/understanding/putting_it_all_together/q_and_a/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18f5e30c-c4bc-455d-9a82-2b33a5a4dfe0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}