Merge pull request #23 from argilla-io/docs/repo_reader_tutorial

RAG tutorial

docs/tutorials/github_rag_llamaindex_argilla.ipynb

@@ -0,0 +1,308 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# 🕵🏻‍♀️ Create a RAG system expert in a Github repository and log your predictions in Argilla\n",
+    "\n",
+    "In this tutorial, we'll show you how to create a RAG system that can answer questions about an specific Github repository. As example, we will target the [Argilla repository](https://github.com/argilla-io/argilla). This RAG system will target the docs of the repository, as that's where most of the natural language information about the repository can be found.\n",
+    "\n",
+    "This tutorial includes the following steps:\n",
+    "-   Setting up the Argilla callback handler for LlamaIndex.\n",
+    "-   Initializing a Github client\n",
+    "-   Creating an index with an specific set of files from the Github repository of our choice.\n",
+    "-   Create a RAG system out of the Argilla repository, ask questions and automatically log the answers to Argilla.\n",
+    "\n",
+    "This tutorial is based on the [Github Repository Reader](https://docs.llamaindex.ai/en/stable/examples/data_connectors/GithubRepositoryReaderDemo/) made by LlamaIndex."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup\n",
+    "\n",
+    "Firstly, you need to make sure that you have the `argilla-llama-index` integration installed. You can do so using `pip`. By installing `argilla-llama-index`, you're also installing `argilla` and `llama-index`. In addition to those two, we will also need `llama-index-readers-github`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install \"argilla-llama-index>=1.0.0\" \"llama-index-readers-github==0.1.9\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, let's set some important environment variables and do the imports necessary for running the notebook. For the environment variables, we'll need the OpenAI API KEY and our Github token. OpenAI's API key is neccesary to run the queries using GPT models, and the Github token is used to ensure that you have access to the repository you're trying to use. Even if it might not be necessary if the repository is public, it is recommended. At the end of the day, you probably also navigate Github's website logged. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%env OPENAI_API_KEY=sk-...\n",
+    "%env GITHUB_TOKEN=github_pat_..."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "from llama_index.core import VectorStoreIndex, set_global_handler\n",
+    "from llama_index.readers.github import GithubRepositoryReader, GithubClient"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Initializing Argilla and setting the global handler\n",
+    "\n",
+    "First things first, you need to have an Argilla instance running. You can check our [installation guide](https://docs.argilla.io/en/latest/getting_started/quickstart_installation.html#Installation) to choose which way suits you better. We recommend using Hugging Face Spaces to have a remote instance, or running a local instance using Docker.\n",
+    "\n",
+    "Now, we will set up an Argilla global handler for Llama Index. By doing so, we ensure that the predictions that we obtain using Llama Index is automatically uploaded to the Argilla client we initialized before Within the handler, we need to provide the dataset name that we will use. If the dataset does not exist, it will be created with the given name. You can also set the **API key**, **API URL**, and the **Workspace name**. If you want to learn more about the variables that controls Argilla initialization, please go to our [workspace management guide](https://docs.argilla.io/en/latest/getting_started/installation/configurations/workspace_management.html)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "set_global_handler(\n",
+    "    \"argilla\", \n",
+    "    api_url=\"https://ignacioct-argilla.hf.space\", # change it to the HF Space direct link if you are running Argilla in HF Spaces\n",
+    "    api_key=\"owner.apikey\",\n",
+    "    workspace_name=\"admin\",\n",
+    "    dataset_name=\"repo_reader\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Initialize the Github client\n",
+    "\n",
+    "Our Github client shall include the Github token we'll use to access the repo and the information of the repository itself, including the owner, the repository name and the desired branch. In our case, we'll target the `main` branch of the `argilla` repository."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "github_token = os.environ.get(\"GITHUB_TOKEN\")\n",
+    "owner = \"argilla-io\"\n",
+    "repo = \"argilla\"\n",
+    "branch = \"main\"\n",
+    "\n",
+    "github_client = GithubClient(github_token=github_token, verbose=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Select which documents are included in the RAG\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Before creating our `GithubRepositoryReader` instance, we need to correct the nesting. The Jupyter kernel itself runs on an event loop, so to prevent this loop for finishing before reading the whole repository, please run the cell below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import nest_asyncio\n",
+    "\n",
+    "nest_asyncio.apply()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, let's create a `GithubRepositoryReader` instance with the information about the repo we want to extract the information from. As the target of this tutorial is to focus on the documentation, we tell the reader to focs on everything in the `docs/` folder, and to avoid images and json files. You can also choose to including `.ipynb` files, depending on the target repository. In our case, there are a lot of tutorials with important information in Argilla, and we would want them included, but for the sake of perfomance on this tutorial, we will exclude them by default."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "documents = GithubRepositoryReader(\n",
+    "    github_client=github_client,\n",
+    "    owner=owner,\n",
+    "    repo=repo,\n",
+    "    use_parser=False,\n",
+    "    verbose=False,\n",
+    "    filter_directories=(\n",
+    "        [\"docs\"],\n",
+    "        GithubRepositoryReader.FilterType.INCLUDE,\n",
+    "    ),\n",
+    "    filter_file_extensions=(\n",
+    "        [\n",
+    "            \".png\",\n",
+    "            \".jpg\",\n",
+    "            \".jpeg\",\n",
+    "            \".gif\",\n",
+    "            \".svg\",\n",
+    "            \".ico\",\n",
+    "            \"json\",\n",
+    "            \".ipynb\",   # Erase this line if you want to include notebooks\n",
+    "\n",
+    "        ],\n",
+    "        GithubRepositoryReader.FilterType.EXCLUDE,\n",
+    "    ),\n",
+    ").load_data(branch=branch)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create the index and start asking questions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, let's create a LlamaIndex index out of this document, and we can start querying the RAG system."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "The FeedbackDataset in Argilla is designed to be a versatile and adaptable dataset that supports a wide range of NLP tasks, including those focused on large language models. It offers the ability to handle multiple tasks on the same record within a coherent user interface, supports multiple annotators per record, customizable tasks, and synchronization with a database. While it currently may lack some features present in older datasets, new features are continuously being added to the FeedbackDataset over time to enhance its capabilities.\n"
+     ]
+    }
+   ],
+   "source": [
+    "index = VectorStoreIndex.from_documents(documents)\n",
+    "\n",
+    "query_engine = index.as_query_engine()\n",
+    "response = query_engine.query(\n",
+    "    \"How does an Argilla's Feedback Dataset work?\"\n",
+    ")\n",
+    "\n",
+    "print(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When the response is generated, it is automatically logged in our Argilla instance. Check it out! From Argilla you can quickly have a look at your predictions and annotate them, so you can combine both synthetic data and human feedback."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![RAG Example 1](../assets/rag_example_1.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's ask a couple of more questions, to see the overall behaviour of the RAG chatbot. Remember that the answers are being automatically logged to your Argilla instance."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Question: What types of dataset can I choose from in Argilla?\n",
+      "Answer: You can choose between older datasets tailored to singular NLP tasks and the FeedbackDataset, which is designed to support a wider range of NLP tasks, including those centered on large language models.\n",
+      "----------------------------\n",
+      "Question: How can I create and update an Argilla dataset?\n",
+      "Answer: To create and update an Argilla dataset, you can add new fields and questions to the dataset using the Python SDK. For local `FeedbackDataset` instances, you can extend the fields and questions lists with new entries or remove existing ones as needed. Additionally, you can add metadata properties to the dataset, change metadata properties' titles, and delete metadata properties. For vector settings, you can update them for both local and remote `FeedbackDataset` instances. It's important to note that configuration updates are limited for remote datasets to avoid inconsistencies between the dataset and defined records and annotations.\n",
+      "----------------------------\n",
+      "Question: Can I upload Markdown files into an Argilla dataset?\n",
+      "Answer: You can render images in the Argilla UI using Markdown files by setting `use_markdown` to `True`. Additionally, you can pass a URL in the metadata field `_image_url` to render images in the Argilla UI for tasks like Text Classification and Token Classification.\n",
+      "----------------------------\n",
+      "Question: Could you explain how to annotate datasets in Argilla?\n",
+      "Answer: To annotate datasets in Argilla, users can utilize the Argilla UI which offers a user-friendly interface for annotating records. Depending on the task type, such as text classification, multi-label text classification, token classification, or Text2Text, different annotation methods are available. For text classification tasks, users can select the label(s) that best describe the record, validate predictions, and remove validated labels if needed. In multi-label text classification, users can select multiple labels, validate predictions with percentages above 50%, and clear or discard annotations as required. \n",
+      "\n",
+      "In token classification tasks, users can highlight words in the text and assign labels, validate predictions, remove specific labels, clear all annotations, and discard records. For Text2Text datasets, users can edit annotations in a text box, validate predictions, clear or discard changes, and discard records if necessary. After annotating a record, users can validate the changes made, revert to the previous state, or discard the record from the dataset. The Argilla UI also provides filters to help users search and filter records efficiently.\n",
+      "----------------------------\n"
+     ]
+    }
+   ],
+   "source": [
+    "questions = [\n",
+    "    \"What types of dataset can I choose from in Argilla?\",\n",
+    "    \"How can I create and update an Argilla dataset?\",\n",
+    "    \"Can I upload Markdown files into an Argilla dataset?\",\n",
+    "    \"Could you explain how to annotate datasets in Argilla?\",\n",
+    "]\n",
+    "\n",
+    "answers = []\n",
+    "\n",
+    "for question in questions:\n",
+    "    answers.append(query_engine.query(question))\n",
+    "\n",
+    "for question, answer in zip(questions, answers):\n",
+    "    print(f\"Question: {question}\")\n",
+    "    print(f\"Answer: {answer}\")\n",
+    "    print(\"----------------------------\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.14"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}