Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7ecc7d43-2dc5-425d-9c71-5492a4677201",
   "metadata": {},
   "source": [
    "# Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c9f923b-f031-4511-a819-f62e6a125e48",
   "metadata": {},
   "source": [
    "- LLMs are a black box\n",
    "- It's hard to tell if the application works correctly for all inputs by eye-balling the outfor for a few\n",
    "- Regression\n",
    "- Metrics based development\n",
    "- Equilavent to testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aeb30a0-65f1-4c1e-8a59-fcb0fa0b92cd",
   "metadata": {},
   "source": [
    "### RAGAS Framework"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe132863-6468-4b59-809a-93dc3cc789be",
   "metadata": {},
   "source": [
    "- https://docs.ragas.io/en/latest/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0acbb14c-8d6d-4c9b-a540-beed1b203928",
   "metadata": {},
   "source": [
    "### Evaluating our application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "636a26e8-98bf-4770-b785-e8ae007b787b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "from llama_index.core import Document, VectorStoreIndex, Settings\n",
    "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
    "from llama_index.llms.ollama import Ollama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "81eb04b4-aab3-4c5d-bd2e-bb799379a45d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = load_dataset(\"Amod/mental_health_counseling_conversations\", split='train')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "e4d9fddf-4157-4c15-89b8-b91dbab7fb1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# bge-base embedding model\n",
    "Settings.embed_model = HuggingFaceEmbedding(model_name=\"BAAI/bge-base-en-v1.5\")\n",
    "\n",
    "# ollama\n",
    "Settings.llm = Ollama(model=\"llama3\", request_timeout=360.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "8f442a8e-a756-46f7-8c49-02bd81dc855b",
   "metadata": {},
   "outputs": [],
   "source": [
    "split = ds.train_test_split()\n",
    "train = [Document(text=f\"Query: {doc['Context']}\\n\\n Response: {doc['Response']}\") for doc in split['train']]\n",
    "test = [Document(text=f\"Query: {doc['Context']}\\n\\n Response: {doc['Response']}\") for doc in split['test']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "9dcd9f23-d062-4e84-94fb-0b07fdc41930",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<llama_index.core.indices.vector_store.base.VectorStoreIndex at 0x36611cc50>"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index = VectorStoreIndex.from_documents(train[0:10])\n",
    "index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "5cedf7ba-e1ba-4ec0-8ef6-6f0d1d9f16ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Response(response=\"It's possible that you're experiencing anxiety as an overwhelming sensation, almost taking over your emotions and making it hard to feel anything else.\", source_nodes=[NodeWithScore(node=TextNode(id_='8eb5de42-ad96-42e0-8dbf-d3e7af92ef8d', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='6a813b43-0aa0-438c-b0e6-ff39969a4207', node_type=<ObjectType.DOCUMENT: '4'>, metadata={}, hash='f0ddf91d4897db22364a62f5353d9edb308cc7091cd8344c310b5084bd325958')}, text='Query: I can\\'t seem to feel any emotion except anxiety, not even for myself.\\n\\n Response: Empathy usually falls on a spectrum, meaning that some people show more than others. Empathy is the ability to look at the world through someone else\\'s eyes or \"walk a mile in their shoes.\" There could be some people in your life for whom empathy is easier to feel and those for whom you have no idea what they are experiencing.Empathy is not usually categorized as an emotion, but as something related to emotions. For example, you could have empathy about someone\\'s anxiety.I\\'m not sure whether you are also referring to the idea that you don\\'t have sympathy for yourself or others. That\\'s a different concept, but also involves different degrees.When you say you don\\'t have emotion except anxiety, I wonder if you are saying that you have anxiety all the time and it may be covering up other emotions? If you are experiencing a lot of anxiety, try to look at what it connects to and find something that changes your focus for a few minutes. This could be any number of activities, such as watching TV, cooking, reading a book, or many other options.If you are feeling as if you are just going through the motions and not having a lot of emotions about anything, if it\\'s just been a few days, it may just be related to a current stress that will lessen in a few days. If it has been quite a while, consider tracking your mood and looking at when you do notice changes in your emotions. There may be some variations that you are not aware of.If you continue to feel anxious and as if you don\\'t have as many emotions as you would like, consider talking with your primary care physician or a local therapist.', mimetype='text/plain', start_char_idx=0, end_char_idx=1696, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'), score=0.5872711679854713), NodeWithScore(node=TextNode(id_='a181f190-1448-41af-ba1a-5195fe34aa98', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='0f402f63-a307-4af6-b6df-dacdaf6a7401', node_type=<ObjectType.DOCUMENT: '4'>, metadata={}, hash='fcae46318f53c91a46f34392afe542162cd95d804179d4f697dfd411775359b5')}, text='Query: When my daughter is stressed about a silly thing from school, she starts crying and freaking out. She is a bright student, always has a 4.0, but I am afraid she is stressing too much. I’m afraid it’s going to break her. I don\\'t know if I should get her to a doctor or someone because this is not normal.\\n\\n Response: children and adolescents who have a desire to achieve high grade point averages often have the symptoms you are describing. It sounds like, your gut is telling you to help her find ways to de-stress. Listen to your gut-always; as her mom, you know her needs better than anyone else. She would probably benefit from seeing a therapist to help her deal with the stress of being a high achiever. I provided the following suggestion to another parent with a similar concern: First identify the three most stressful aspects of the day. Do this by creating a circle on a regular size paper. (do this three times); ask your daughter how much of that circle can she confidently say she has under control. shade that in. The remaining parts of the circle are then named by intensity. (for example, in the circle labeled school anxiety, the client identified that she felt she could handle school anxiety 1 out of 4 days. The circle was split into 1/4. 1/4 of the circle was shaded . the remaining parts were labeled with the challenges that caused the client school anxiety.(these were 1. walking to class during passing time. 2. not having the assignments completed or at all. 3. feeling fat. after labeling each part; and making note of the percentage of the stress associated with each concern, free form thinking takes place and a pile of unwanted behaviors and genetic responses are portrayed\\xa0 around the circle in bright colors.\\xa0Using note cards, create a tree of concern; the concerns can be written on note cards. a concern = the behavior that leads to more trouble.\\xa0 For example, \"all i could do was think about that extra slice of pizza, I was unable to concentrate on the teacher\\'s explanation of the assignment\" (the concern you would write on the note card is Inability to concentrate. Now, fill up the back of the notecard with all of the different factors causing this pattern. nack. (', mimetype='text/plain', start_char_idx=0, end_char_idx=2214, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'), score=0.574588010882357)], metadata={'8eb5de42-ad96-42e0-8dbf-d3e7af92ef8d': {}, 'a181f190-1448-41af-ba1a-5195fe34aa98': {}})"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query_engine = index.as_query_engine()\n",
    "response = query_engine.query(\"What is it like having a panic attack?\")\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "deb55685-ca63-4535-b2cb-02a90c8729f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.testset.generator import TestsetGenerator\n",
    "from ragas.testset.evolutions import simple, reasoning, multi_context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cd719820-6418-4970-a6dd-fbd1a9cf33d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "generator_llm = Ollama(model='llama3', request_timeout=300.0)\n",
    "critic_llm = Ollama(model='llama3', request_timeout=300.0)\n",
    "embeddings = HuggingFaceEmbedding(model_name=\"BAAI/bge-base-en-v1.5\")\n",
    "\n",
    "generator = TestsetGenerator.from_llama_index(\n",
    "    generator_llm,\n",
    "    critic_llm,\n",
    "    embeddings\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6b4ef256-1c28-4d92-99b2-1908a1929cc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "distributions = {\n",
    "    simple: 1,\n",
    "    multi_context: 0,\n",
    "    reasoning: 0,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "26d98c55-a49b-47c6-b133-e93189ab8c2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "embedding nodes:   0%|          | 0/20 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Filename and doc_id are the same for all nodes.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9237c9bc0a094eb19c94eeaba1ae8c8f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "testset = generator.generate_with_llamaindex_docs(train[0:10], test_size=2, distributions=distributions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "960a7487-b97b-494a-a971-25fa4d99cfd5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "question='Here is a question that can be fully answered from the given context:\\n\\n\"What are some strategies that I should consider learning to manage my anxiety attacks while on Xanax medication?\"\\n\\nThis question is formed using the topic \"Psychologist consultation\" and is answerable based on the provided context, which suggests considering alternative coping mechanisms in addition to medication.' contexts=[\"Query: I’ve been on 0.5 mg of Xanax twice a day for the past month. It hasn't been helping me at all, but when I take 1 mg during a big anxiety attack, it calms me down. I was wondering how I can ask my psychologist to up the dose to 1 mg twice a day without her thinking I'm abusing them. I just have very big anxiety attacks. Should I stay on the 0.5mg and deal with the attacks or should I ask to up the dose? I'm afraid she will take me off them and put me on something else.\\n\\n Response: Staying on the lower dose may give you more room to learn strategies for coping with your anxiety.\\xa0 Medications are so helpful, and needed at times, but it's also important to have a variety\\xa0of tools you use to manage your responses to stress.\\xa0 If you are not already seeing a therapist, consider finding one who can help you learn some effective strategies, like replacing self-defeating thoughts with ones that work better for you, or mindfulness, relaxation, or other tools to keep your anxiety in the manageable range!\"] ground_truth=\"Staying on the lower dose may give you more room to learn strategies for coping with your anxiety. Medications are so helpful, and needed at times, but it's also important to have a variety of tools you use to manage your responses to stress. If you are not already seeing a therapist, consider finding one who can help you learn some effective strategies, like replacing self-defeating thoughts with ones that work better for you, or mindfulness, relaxation, or other tools to keep your anxiety in the manageable range!\" evolution_type='simple' metadata=[{}]\n"
     ]
    }
   ],
   "source": [
    "print(testset.test_data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "f3406aad-8d3e-4d32-a324-d5ab74b92801",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question:\n",
      "Here is a question that can be fully answered from the given context:\n",
      "\n",
      "\"What are some strategies that I should consider learning to manage my anxiety attacks while on Xanax medication?\"\n",
      "\n",
      "This question is formed using the topic \"Psychologist consultation\" and is answerable based on the provided context, which suggests considering alternative coping mechanisms in addition to medication.\n",
      "\n",
      "Response:\n",
      "Some strategies you may want to consider learning to manage your anxiety attacks include replacing self-defeating thoughts with ones that work better for you, practicing mindfulness, or relaxation techniques. These tools can help keep your anxiety in a manageable range and complement the benefits of your Xanax medication.\n"
     ]
    }
   ],
   "source": [
    "print(\"Question:\")\n",
    "print(testset.test_data[0].question)\n",
    "print(\"\\nResponse:\")\n",
    "print(query_engine.query(testset.test_data[0].question))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "f6decf48-4849-4f34-9a17-2a64f81e2718",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ragas.metrics import (\n",
    "    faithfulness,\n",
    "    answer_relevancy,\n",
    "    context_precision,\n",
    "    context_recall,\n",
    ")\n",
    "from ragas.metrics.critique import harmfulness\n",
    "\n",
    "metrics = [\n",
    "    faithfulness,\n",
    "    answer_relevancy,\n",
    "    context_precision,\n",
    "    context_recall,\n",
    "    harmfulness,\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "28235a03-0ef6-4a56-af19-81c14008d102",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Staying on the lower dose may give you more room to learn strategies for coping with your anxiety. Medications are so helpful, and needed at times, but it's also important to have a variety of tools you use to manage your responses to stress. If you are not already seeing a therapist, consider finding one who can help you learn some effective strategies, like replacing self-defeating thoughts with ones that work better for you, or mindfulness, relaxation, or other tools to keep your anxiety in the manageable range!\",\n",
       " \"Some strategies for coping with my mom's bossy behavior and negotiating a more respectful dynamic in our relationship include seeing her responses as habits rather than a reflection of how she feels about me, having a calm tone, and negotiating a new dynamic through a contract. This can be achieved by setting clear expectations, boundaries, and communication channels with Mom. Ultimately, it is crucial to approach the situation with an open mind, empathy, and understanding.\"]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = testset.to_dataset()\n",
    "\n",
    "ds_dict = ds.to_dict()\n",
    "ds_dict[\"question\"]\n",
    "ds_dict[\"ground_truth\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "4ad9c184-22f4-4de7-a3a8-fdbae97d206a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cd006a56d6354bfc8805d54d2b514892",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Running Query Engine:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "053c4e2e7004481c8700dba964588e35",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Evaluating:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "n values greater than 1 not support for LlamaIndex LLMs\n",
      "n values greater than 1 not support for LlamaIndex LLMs\n"
     ]
    },
    {
     "ename": "TimeoutError",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mCancelledError\u001b[0m                            Traceback (most recent call last)",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py:520\u001b[0m, in \u001b[0;36mwait_for\u001b[0;34m(fut, timeout)\u001b[0m\n\u001b[1;32m    519\u001b[0m \u001b[38;5;28;01masync\u001b[39;00m \u001b[38;5;28;01mwith\u001b[39;00m timeouts\u001b[38;5;241m.\u001b[39mtimeout(timeout):\n\u001b[0;32m--> 520\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m fut\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/metrics/_context_precision.py:160\u001b[0m, in \u001b[0;36mContextPrecision._ascore\u001b[0;34m(self, row, callbacks)\u001b[0m\n\u001b[1;32m    159\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m hp \u001b[38;5;129;01min\u001b[39;00m human_prompts:\n\u001b[0;32m--> 160\u001b[0m     results \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mllm\u001b[38;5;241m.\u001b[39mgenerate(\n\u001b[1;32m    161\u001b[0m         hp,\n\u001b[1;32m    162\u001b[0m         callbacks\u001b[38;5;241m=\u001b[39mcallbacks,\n\u001b[1;32m    163\u001b[0m         n\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mreproducibility,\n\u001b[1;32m    164\u001b[0m     )\n\u001b[1;32m    165\u001b[0m     results \u001b[38;5;241m=\u001b[39m [\n\u001b[1;32m    166\u001b[0m         \u001b[38;5;28;01mawait\u001b[39;00m _output_parser\u001b[38;5;241m.\u001b[39maparse(item\u001b[38;5;241m.\u001b[39mtext, hp, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mllm, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmax_retries)\n\u001b[1;32m    167\u001b[0m         \u001b[38;5;28;01mfor\u001b[39;00m item \u001b[38;5;129;01min\u001b[39;00m results\u001b[38;5;241m.\u001b[39mgenerations[\u001b[38;5;241m0\u001b[39m]\n\u001b[1;32m    168\u001b[0m     ]\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/llms/base.py:93\u001b[0m, in \u001b[0;36mBaseRagasLLM.generate\u001b[0;34m(self, prompt, n, temperature, stop, callbacks, is_async)\u001b[0m\n\u001b[1;32m     90\u001b[0m     agenerate_text_with_retry \u001b[38;5;241m=\u001b[39m add_async_retry(\n\u001b[1;32m     91\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39magenerate_text, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mrun_config\n\u001b[1;32m     92\u001b[0m     )\n\u001b[0;32m---> 93\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m agenerate_text_with_retry(\n\u001b[1;32m     94\u001b[0m         prompt\u001b[38;5;241m=\u001b[39mprompt,\n\u001b[1;32m     95\u001b[0m         n\u001b[38;5;241m=\u001b[39mn,\n\u001b[1;32m     96\u001b[0m         temperature\u001b[38;5;241m=\u001b[39mtemperature,\n\u001b[1;32m     97\u001b[0m         stop\u001b[38;5;241m=\u001b[39mstop,\n\u001b[1;32m     98\u001b[0m         callbacks\u001b[38;5;241m=\u001b[39mcallbacks,\n\u001b[1;32m     99\u001b[0m     )\n\u001b[1;32m    100\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:189\u001b[0m, in \u001b[0;36mAsyncRetrying.wraps.<locals>.async_wrapped\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    188\u001b[0m async_wrapped\u001b[38;5;241m.\u001b[39mstatistics \u001b[38;5;241m=\u001b[39m copy\u001b[38;5;241m.\u001b[39mstatistics  \u001b[38;5;66;03m# type: ignore[attr-defined]\u001b[39;00m\n\u001b[0;32m--> 189\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m copy(fn, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:111\u001b[0m, in \u001b[0;36mAsyncRetrying.__call__\u001b[0;34m(self, fn, *args, **kwargs)\u001b[0m\n\u001b[1;32m    110\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m:\n\u001b[0;32m--> 111\u001b[0m     do \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39miter(retry_state\u001b[38;5;241m=\u001b[39mretry_state)\n\u001b[1;32m    112\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(do, DoAttempt):\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:153\u001b[0m, in \u001b[0;36mAsyncRetrying.iter\u001b[0;34m(self, retry_state)\u001b[0m\n\u001b[1;32m    152\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m action \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39miter_state\u001b[38;5;241m.\u001b[39mactions:\n\u001b[0;32m--> 153\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m action(retry_state)\n\u001b[1;32m    154\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m result\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/tenacity/_utils.py:99\u001b[0m, in \u001b[0;36mwrap_to_async_func.<locals>.inner\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     98\u001b[0m \u001b[38;5;28;01masync\u001b[39;00m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21minner\u001b[39m(\u001b[38;5;241m*\u001b[39margs: typing\u001b[38;5;241m.\u001b[39mAny, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs: typing\u001b[38;5;241m.\u001b[39mAny) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m typing\u001b[38;5;241m.\u001b[39mAny:\n\u001b[0;32m---> 99\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mcall\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/tenacity/__init__.py:398\u001b[0m, in \u001b[0;36mBaseRetrying._post_retry_check_actions.<locals>.<lambda>\u001b[0;34m(rs)\u001b[0m\n\u001b[1;32m    397\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39miter_state\u001b[38;5;241m.\u001b[39mis_explicit_retry \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39miter_state\u001b[38;5;241m.\u001b[39mretry_run_result):\n\u001b[0;32m--> 398\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_add_action_func(\u001b[38;5;28;01mlambda\u001b[39;00m rs: \u001b[43mrs\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43moutcome\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresult\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[1;32m    399\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py:449\u001b[0m, in \u001b[0;36mFuture.result\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    448\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;241m==\u001b[39m FINISHED:\n\u001b[0;32m--> 449\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__get_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    451\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_condition\u001b[38;5;241m.\u001b[39mwait(timeout)\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py:401\u001b[0m, in \u001b[0;36mFuture.__get_result\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    400\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 401\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception\n\u001b[1;32m    402\u001b[0m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[1;32m    403\u001b[0m     \u001b[38;5;66;03m# Break a reference cycle with the exception in self._exception\u001b[39;00m\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:114\u001b[0m, in \u001b[0;36mAsyncRetrying.__call__\u001b[0;34m(self, fn, *args, **kwargs)\u001b[0m\n\u001b[1;32m    113\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 114\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m fn(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m    115\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m:  \u001b[38;5;66;03m# noqa: B902\u001b[39;00m\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/llms/base.py:274\u001b[0m, in \u001b[0;36mLlamaIndexLLMWrapper.agenerate_text\u001b[0;34m(self, prompt, n, temperature, stop, callbacks)\u001b[0m\n\u001b[1;32m    273\u001b[0m kwargs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcheck_args(n, temperature, stop, callbacks)\n\u001b[0;32m--> 274\u001b[0m li_response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mllm\u001b[38;5;241m.\u001b[39macomplete(prompt\u001b[38;5;241m.\u001b[39mto_string(), \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m    276\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m LLMResult(generations\u001b[38;5;241m=\u001b[39m[[Generation(text\u001b[38;5;241m=\u001b[39mli_response\u001b[38;5;241m.\u001b[39mtext)]])\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py:255\u001b[0m, in \u001b[0;36mDispatcher.span.<locals>.async_wrapper\u001b[0;34m(func, instance, args, kwargs)\u001b[0m\n\u001b[1;32m    254\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 255\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m func(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m    256\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/llama_index/core/llms/callbacks.py:334\u001b[0m, in \u001b[0;36mllm_completion_callback.<locals>.wrap.<locals>.wrapped_async_llm_predict\u001b[0;34m(_self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    333\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 334\u001b[0m     f_return_val \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m f(_self, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m    335\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/llama_index/llms/ollama/base.py:348\u001b[0m, in \u001b[0;36mOllama.acomplete\u001b[0;34m(self, prompt, formatted, **kwargs)\u001b[0m\n\u001b[1;32m    347\u001b[0m \u001b[38;5;28;01masync\u001b[39;00m \u001b[38;5;28;01mwith\u001b[39;00m httpx\u001b[38;5;241m.\u001b[39mAsyncClient(timeout\u001b[38;5;241m=\u001b[39mTimeout(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mrequest_timeout)) \u001b[38;5;28;01mas\u001b[39;00m client:\n\u001b[0;32m--> 348\u001b[0m     response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m client\u001b[38;5;241m.\u001b[39mpost(\n\u001b[1;32m    349\u001b[0m         url\u001b[38;5;241m=\u001b[39m\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbase_url\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m/api/generate\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m    350\u001b[0m         json\u001b[38;5;241m=\u001b[39mpayload,\n\u001b[1;32m    351\u001b[0m     )\n\u001b[1;32m    352\u001b[0m     response\u001b[38;5;241m.\u001b[39mraise_for_status()\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpx/_client.py:1892\u001b[0m, in \u001b[0;36mAsyncClient.post\u001b[0;34m(self, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)\u001b[0m\n\u001b[1;32m   1887\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m   1888\u001b[0m \u001b[38;5;124;03mSend a `POST` request.\u001b[39;00m\n\u001b[1;32m   1889\u001b[0m \n\u001b[1;32m   1890\u001b[0m \u001b[38;5;124;03m**Parameters**: See `httpx.request`.\u001b[39;00m\n\u001b[1;32m   1891\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m-> 1892\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mrequest(\n\u001b[1;32m   1893\u001b[0m     \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mPOST\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m   1894\u001b[0m     url,\n\u001b[1;32m   1895\u001b[0m     content\u001b[38;5;241m=\u001b[39mcontent,\n\u001b[1;32m   1896\u001b[0m     data\u001b[38;5;241m=\u001b[39mdata,\n\u001b[1;32m   1897\u001b[0m     files\u001b[38;5;241m=\u001b[39mfiles,\n\u001b[1;32m   1898\u001b[0m     json\u001b[38;5;241m=\u001b[39mjson,\n\u001b[1;32m   1899\u001b[0m     params\u001b[38;5;241m=\u001b[39mparams,\n\u001b[1;32m   1900\u001b[0m     headers\u001b[38;5;241m=\u001b[39mheaders,\n\u001b[1;32m   1901\u001b[0m     cookies\u001b[38;5;241m=\u001b[39mcookies,\n\u001b[1;32m   1902\u001b[0m     auth\u001b[38;5;241m=\u001b[39mauth,\n\u001b[1;32m   1903\u001b[0m     follow_redirects\u001b[38;5;241m=\u001b[39mfollow_redirects,\n\u001b[1;32m   1904\u001b[0m     timeout\u001b[38;5;241m=\u001b[39mtimeout,\n\u001b[1;32m   1905\u001b[0m     extensions\u001b[38;5;241m=\u001b[39mextensions,\n\u001b[1;32m   1906\u001b[0m )\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpx/_client.py:1574\u001b[0m, in \u001b[0;36mAsyncClient.request\u001b[0;34m(self, method, url, content, data, files, json, params, headers, cookies, auth, follow_redirects, timeout, extensions)\u001b[0m\n\u001b[1;32m   1561\u001b[0m request \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbuild_request(\n\u001b[1;32m   1562\u001b[0m     method\u001b[38;5;241m=\u001b[39mmethod,\n\u001b[1;32m   1563\u001b[0m     url\u001b[38;5;241m=\u001b[39murl,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m   1572\u001b[0m     extensions\u001b[38;5;241m=\u001b[39mextensions,\n\u001b[1;32m   1573\u001b[0m )\n\u001b[0;32m-> 1574\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msend(request, auth\u001b[38;5;241m=\u001b[39mauth, follow_redirects\u001b[38;5;241m=\u001b[39mfollow_redirects)\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpx/_client.py:1661\u001b[0m, in \u001b[0;36mAsyncClient.send\u001b[0;34m(self, request, stream, auth, follow_redirects)\u001b[0m\n\u001b[1;32m   1659\u001b[0m auth \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_build_request_auth(request, auth)\n\u001b[0;32m-> 1661\u001b[0m response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_send_handling_auth(\n\u001b[1;32m   1662\u001b[0m     request,\n\u001b[1;32m   1663\u001b[0m     auth\u001b[38;5;241m=\u001b[39mauth,\n\u001b[1;32m   1664\u001b[0m     follow_redirects\u001b[38;5;241m=\u001b[39mfollow_redirects,\n\u001b[1;32m   1665\u001b[0m     history\u001b[38;5;241m=\u001b[39m[],\n\u001b[1;32m   1666\u001b[0m )\n\u001b[1;32m   1667\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpx/_client.py:1689\u001b[0m, in \u001b[0;36mAsyncClient._send_handling_auth\u001b[0;34m(self, request, auth, follow_redirects, history)\u001b[0m\n\u001b[1;32m   1688\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m:\n\u001b[0;32m-> 1689\u001b[0m     response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_send_handling_redirects(\n\u001b[1;32m   1690\u001b[0m         request,\n\u001b[1;32m   1691\u001b[0m         follow_redirects\u001b[38;5;241m=\u001b[39mfollow_redirects,\n\u001b[1;32m   1692\u001b[0m         history\u001b[38;5;241m=\u001b[39mhistory,\n\u001b[1;32m   1693\u001b[0m     )\n\u001b[1;32m   1694\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpx/_client.py:1726\u001b[0m, in \u001b[0;36mAsyncClient._send_handling_redirects\u001b[0;34m(self, request, follow_redirects, history)\u001b[0m\n\u001b[1;32m   1724\u001b[0m     \u001b[38;5;28;01mawait\u001b[39;00m hook(request)\n\u001b[0;32m-> 1726\u001b[0m response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_send_single_request(request)\n\u001b[1;32m   1727\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpx/_client.py:1763\u001b[0m, in \u001b[0;36mAsyncClient._send_single_request\u001b[0;34m(self, request)\u001b[0m\n\u001b[1;32m   1762\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m request_context(request\u001b[38;5;241m=\u001b[39mrequest):\n\u001b[0;32m-> 1763\u001b[0m     response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m transport\u001b[38;5;241m.\u001b[39mhandle_async_request(request)\n\u001b[1;32m   1765\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(response\u001b[38;5;241m.\u001b[39mstream, AsyncByteStream)\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpx/_transports/default.py:373\u001b[0m, in \u001b[0;36mAsyncHTTPTransport.handle_async_request\u001b[0;34m(self, request)\u001b[0m\n\u001b[1;32m    372\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m map_httpcore_exceptions():\n\u001b[0;32m--> 373\u001b[0m     resp \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_pool\u001b[38;5;241m.\u001b[39mhandle_async_request(req)\n\u001b[1;32m    375\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(resp\u001b[38;5;241m.\u001b[39mstream, typing\u001b[38;5;241m.\u001b[39mAsyncIterable)\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py:216\u001b[0m, in \u001b[0;36mAsyncConnectionPool.handle_async_request\u001b[0;34m(self, request)\u001b[0m\n\u001b[1;32m    215\u001b[0m     \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_close_connections(closing)\n\u001b[0;32m--> 216\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m exc \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m    218\u001b[0m \u001b[38;5;66;03m# Return the response. Note that in this case we still have to manage\u001b[39;00m\n\u001b[1;32m    219\u001b[0m \u001b[38;5;66;03m# the point at which the response is closed.\u001b[39;00m\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpcore/_async/connection_pool.py:196\u001b[0m, in \u001b[0;36mAsyncConnectionPool.handle_async_request\u001b[0;34m(self, request)\u001b[0m\n\u001b[1;32m    194\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m    195\u001b[0m     \u001b[38;5;66;03m# Send the request on the assigned connection.\u001b[39;00m\n\u001b[0;32m--> 196\u001b[0m     response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m connection\u001b[38;5;241m.\u001b[39mhandle_async_request(\n\u001b[1;32m    197\u001b[0m         pool_request\u001b[38;5;241m.\u001b[39mrequest\n\u001b[1;32m    198\u001b[0m     )\n\u001b[1;32m    199\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m ConnectionNotAvailable:\n\u001b[1;32m    200\u001b[0m     \u001b[38;5;66;03m# In some cases a connection may initially be available to\u001b[39;00m\n\u001b[1;32m    201\u001b[0m     \u001b[38;5;66;03m# handle a request, but then become unavailable.\u001b[39;00m\n\u001b[1;32m    202\u001b[0m     \u001b[38;5;66;03m#\u001b[39;00m\n\u001b[1;32m    203\u001b[0m     \u001b[38;5;66;03m# In this case we clear the connection and try again.\u001b[39;00m\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpcore/_async/connection.py:101\u001b[0m, in \u001b[0;36mAsyncHTTPConnection.handle_async_request\u001b[0;34m(self, request)\u001b[0m\n\u001b[1;32m     99\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m exc\n\u001b[0;32m--> 101\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_connection\u001b[38;5;241m.\u001b[39mhandle_async_request(request)\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py:143\u001b[0m, in \u001b[0;36mAsyncHTTP11Connection.handle_async_request\u001b[0;34m(self, request)\u001b[0m\n\u001b[1;32m    142\u001b[0m         \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_response_closed()\n\u001b[0;32m--> 143\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m exc\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py:113\u001b[0m, in \u001b[0;36mAsyncHTTP11Connection.handle_async_request\u001b[0;34m(self, request)\u001b[0m\n\u001b[1;32m    104\u001b[0m \u001b[38;5;28;01masync\u001b[39;00m \u001b[38;5;28;01mwith\u001b[39;00m Trace(\n\u001b[1;32m    105\u001b[0m     \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mreceive_response_headers\u001b[39m\u001b[38;5;124m\"\u001b[39m, logger, request, kwargs\n\u001b[1;32m    106\u001b[0m ) \u001b[38;5;28;01mas\u001b[39;00m trace:\n\u001b[1;32m    107\u001b[0m     (\n\u001b[1;32m    108\u001b[0m         http_version,\n\u001b[1;32m    109\u001b[0m         status,\n\u001b[1;32m    110\u001b[0m         reason_phrase,\n\u001b[1;32m    111\u001b[0m         headers,\n\u001b[1;32m    112\u001b[0m         trailing_data,\n\u001b[0;32m--> 113\u001b[0m     ) \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_receive_response_headers(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m    114\u001b[0m     trace\u001b[38;5;241m.\u001b[39mreturn_value \u001b[38;5;241m=\u001b[39m (\n\u001b[1;32m    115\u001b[0m         http_version,\n\u001b[1;32m    116\u001b[0m         status,\n\u001b[1;32m    117\u001b[0m         reason_phrase,\n\u001b[1;32m    118\u001b[0m         headers,\n\u001b[1;32m    119\u001b[0m     )\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py:186\u001b[0m, in \u001b[0;36mAsyncHTTP11Connection._receive_response_headers\u001b[0;34m(self, request)\u001b[0m\n\u001b[1;32m    185\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m:\n\u001b[0;32m--> 186\u001b[0m     event \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_receive_event(timeout\u001b[38;5;241m=\u001b[39mtimeout)\n\u001b[1;32m    187\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(event, h11\u001b[38;5;241m.\u001b[39mResponse):\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpcore/_async/http11.py:224\u001b[0m, in \u001b[0;36mAsyncHTTP11Connection._receive_event\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    223\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m event \u001b[38;5;129;01mis\u001b[39;00m h11\u001b[38;5;241m.\u001b[39mNEED_DATA:\n\u001b[0;32m--> 224\u001b[0m     data \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_network_stream\u001b[38;5;241m.\u001b[39mread(\n\u001b[1;32m    225\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mREAD_NUM_BYTES, timeout\u001b[38;5;241m=\u001b[39mtimeout\n\u001b[1;32m    226\u001b[0m     )\n\u001b[1;32m    228\u001b[0m     \u001b[38;5;66;03m# If we feed this case through h11 we'll raise an exception like:\u001b[39;00m\n\u001b[1;32m    229\u001b[0m     \u001b[38;5;66;03m#\u001b[39;00m\n\u001b[1;32m    230\u001b[0m     \u001b[38;5;66;03m#     httpcore.RemoteProtocolError: can't handle event type\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    234\u001b[0m     \u001b[38;5;66;03m# perspective. Instead we handle this case distinctly and treat\u001b[39;00m\n\u001b[1;32m    235\u001b[0m     \u001b[38;5;66;03m# it as a ConnectError.\u001b[39;00m\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/httpcore/_backends/anyio.py:35\u001b[0m, in \u001b[0;36mAnyIOStream.read\u001b[0;34m(self, max_bytes, timeout)\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 35\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_stream\u001b[38;5;241m.\u001b[39mreceive(max_bytes\u001b[38;5;241m=\u001b[39mmax_bytes)\n\u001b[1;32m     36\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m anyio\u001b[38;5;241m.\u001b[39mEndOfStream:  \u001b[38;5;66;03m# pragma: nocover\u001b[39;00m\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/anyio/_backends/_asyncio.py:1142\u001b[0m, in \u001b[0;36mSocketStream.receive\u001b[0;34m(self, max_bytes)\u001b[0m\n\u001b[1;32m   1141\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_transport\u001b[38;5;241m.\u001b[39mresume_reading()\n\u001b[0;32m-> 1142\u001b[0m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_protocol\u001b[38;5;241m.\u001b[39mread_event\u001b[38;5;241m.\u001b[39mwait()\n\u001b[1;32m   1143\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_transport\u001b[38;5;241m.\u001b[39mpause_reading()\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/locks.py:212\u001b[0m, in \u001b[0;36mEvent.wait\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    211\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 212\u001b[0m     \u001b[38;5;28;01mawait\u001b[39;00m fut\n\u001b[1;32m    213\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/futures.py:287\u001b[0m, in \u001b[0;36mFuture.__await__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    286\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_asyncio_future_blocking \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[0;32m--> 287\u001b[0m     \u001b[38;5;28;01myield\u001b[39;00m \u001b[38;5;28mself\u001b[39m  \u001b[38;5;66;03m# This tells Task to wait for completion.\u001b[39;00m\n\u001b[1;32m    288\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdone():\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py:385\u001b[0m, in \u001b[0;36mTask.__wakeup\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m    384\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 385\u001b[0m     \u001b[43mfuture\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresult\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    386\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m exc:\n\u001b[1;32m    387\u001b[0m     \u001b[38;5;66;03m# This may also be a cancellation.\u001b[39;00m\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/futures.py:198\u001b[0m, in \u001b[0;36mFuture.result\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    197\u001b[0m     exc \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_make_cancelled_error()\n\u001b[0;32m--> 198\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m exc\n\u001b[1;32m    199\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;241m!=\u001b[39m _FINISHED:\n",
      "\u001b[0;31mCancelledError\u001b[0m: ",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[0;31mTimeoutError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[67], line 5\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mragas\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mintegrations\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mllama_index\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m evaluate\n\u001b[1;32m      3\u001b[0m evaluator_llm \u001b[38;5;241m=\u001b[39m Ollama(model\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mllama3\u001b[39m\u001b[38;5;124m'\u001b[39m, request_timeout\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m300.0\u001b[39m)\n\u001b[0;32m----> 5\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[43mevaluate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m      6\u001b[0m \u001b[43m    \u001b[49m\u001b[43mquery_engine\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mquery_engine\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m      7\u001b[0m \u001b[43m    \u001b[49m\u001b[43mmetrics\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmetrics\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m      8\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdataset\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mds_dict\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m      9\u001b[0m \u001b[43m    \u001b[49m\u001b[43mllm\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mevaluator_llm\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     10\u001b[0m \u001b[43m    \u001b[49m\u001b[43membeddings\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43membeddings\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     11\u001b[0m \u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/integrations/llama_index.py:105\u001b[0m, in \u001b[0;36mevaluate\u001b[0;34m(query_engine, dataset, metrics, llm, embeddings, raise_exceptions, column_map)\u001b[0m\n\u001b[1;32m     98\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mground_truth\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m dataset:\n\u001b[1;32m     99\u001b[0m     hf_dataset \u001b[38;5;241m=\u001b[39m hf_dataset\u001b[38;5;241m.\u001b[39madd_column(\n\u001b[1;32m    100\u001b[0m         name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mground_truth\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m    101\u001b[0m         column\u001b[38;5;241m=\u001b[39mdataset[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mground_truth\u001b[39m\u001b[38;5;124m\"\u001b[39m],\n\u001b[1;32m    102\u001b[0m         new_fingerprint\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mstr\u001b[39m(uuid4()),\n\u001b[1;32m    103\u001b[0m     )\n\u001b[0;32m--> 105\u001b[0m results \u001b[38;5;241m=\u001b[39m \u001b[43mragas_evaluate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    106\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdataset\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mhf_dataset\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    107\u001b[0m \u001b[43m    \u001b[49m\u001b[43mmetrics\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmetrics\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    108\u001b[0m \u001b[43m    \u001b[49m\u001b[43mllm\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mli_llm\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    109\u001b[0m \u001b[43m    \u001b[49m\u001b[43membeddings\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mli_embeddings\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    110\u001b[0m \u001b[43m    \u001b[49m\u001b[43mraise_exceptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mraise_exceptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    111\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    113\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m results\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/evaluation.py:247\u001b[0m, in \u001b[0;36mevaluate\u001b[0;34m(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, raise_exceptions, column_map)\u001b[0m\n\u001b[1;32m    244\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m evaluation_group_cm\u001b[38;5;241m.\u001b[39mended:\n\u001b[1;32m    245\u001b[0m         evaluation_rm\u001b[38;5;241m.\u001b[39mon_chain_error(e)\n\u001b[0;32m--> 247\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m e\n\u001b[1;32m    248\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    249\u001b[0m     result \u001b[38;5;241m=\u001b[39m Result(\n\u001b[1;32m    250\u001b[0m         scores\u001b[38;5;241m=\u001b[39mDataset\u001b[38;5;241m.\u001b[39mfrom_list(scores),\n\u001b[1;32m    251\u001b[0m         dataset\u001b[38;5;241m=\u001b[39mdataset,\n\u001b[1;32m    252\u001b[0m         binary_columns\u001b[38;5;241m=\u001b[39mbinary_metrics,\n\u001b[1;32m    253\u001b[0m     )\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/evaluation.py:227\u001b[0m, in \u001b[0;36mevaluate\u001b[0;34m(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, raise_exceptions, column_map)\u001b[0m\n\u001b[1;32m    224\u001b[0m scores \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m    225\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m    226\u001b[0m     \u001b[38;5;66;03m# get the results\u001b[39;00m\n\u001b[0;32m--> 227\u001b[0m     results \u001b[38;5;241m=\u001b[39m \u001b[43mexecutor\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresults\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    228\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m results \u001b[38;5;241m==\u001b[39m []:\n\u001b[1;32m    229\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m ExceptionInRunner()\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/executor.py:107\u001b[0m, in \u001b[0;36mExecutor.results\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    103\u001b[0m         results\u001b[38;5;241m.\u001b[39mappend(r)\n\u001b[1;32m    105\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m results\n\u001b[0;32m--> 107\u001b[0m results \u001b[38;5;241m=\u001b[39m \u001b[43masyncio\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43m_aresults\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    108\u001b[0m sorted_results \u001b[38;5;241m=\u001b[39m \u001b[38;5;28msorted\u001b[39m(results, key\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mlambda\u001b[39;00m x: x[\u001b[38;5;241m0\u001b[39m])\n\u001b[1;32m    109\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m [r[\u001b[38;5;241m1\u001b[39m] \u001b[38;5;28;01mfor\u001b[39;00m r \u001b[38;5;129;01min\u001b[39;00m sorted_results]\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/nest_asyncio.py:30\u001b[0m, in \u001b[0;36m_patch_asyncio.<locals>.run\u001b[0;34m(main, debug)\u001b[0m\n\u001b[1;32m     28\u001b[0m task \u001b[38;5;241m=\u001b[39m asyncio\u001b[38;5;241m.\u001b[39mensure_future(main)\n\u001b[1;32m     29\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 30\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mloop\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun_until_complete\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtask\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     31\u001b[0m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[1;32m     32\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m task\u001b[38;5;241m.\u001b[39mdone():\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/nest_asyncio.py:98\u001b[0m, in \u001b[0;36m_patch_loop.<locals>.run_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m     95\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m f\u001b[38;5;241m.\u001b[39mdone():\n\u001b[1;32m     96\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\n\u001b[1;32m     97\u001b[0m         \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mEvent loop stopped before Future completed.\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m---> 98\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresult\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/futures.py:203\u001b[0m, in \u001b[0;36mFuture.result\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    201\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__log_traceback \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[1;32m    202\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 203\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception\u001b[38;5;241m.\u001b[39mwith_traceback(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception_tb)\n\u001b[1;32m    204\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_result\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py:314\u001b[0m, in \u001b[0;36mTask.__step_run_and_handle_result\u001b[0;34m(***failed resolving arguments***)\u001b[0m\n\u001b[1;32m    310\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m    311\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m exc \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    312\u001b[0m         \u001b[38;5;66;03m# We use the `send` method directly, because coroutines\u001b[39;00m\n\u001b[1;32m    313\u001b[0m         \u001b[38;5;66;03m# don't have `__iter__` and `__next__` methods.\u001b[39;00m\n\u001b[0;32m--> 314\u001b[0m         result \u001b[38;5;241m=\u001b[39m \u001b[43mcoro\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msend\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[1;32m    315\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    316\u001b[0m         result \u001b[38;5;241m=\u001b[39m coro\u001b[38;5;241m.\u001b[39mthrow(exc)\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/executor.py:102\u001b[0m, in \u001b[0;36mExecutor.results.<locals>._aresults\u001b[0;34m()\u001b[0m\n\u001b[1;32m     94\u001b[0m results \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m     95\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m future \u001b[38;5;129;01min\u001b[39;00m tqdm(\n\u001b[1;32m     96\u001b[0m     futures_as_they_finish,\n\u001b[1;32m     97\u001b[0m     desc\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdesc,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    100\u001b[0m     leave\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mkeep_progress_bar,\n\u001b[1;32m    101\u001b[0m ):\n\u001b[0;32m--> 102\u001b[0m     r \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m future\n\u001b[1;32m    103\u001b[0m     results\u001b[38;5;241m.\u001b[39mappend(r)\n\u001b[1;32m    105\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m results\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py:631\u001b[0m, in \u001b[0;36mas_completed.<locals>._wait_for_one\u001b[0;34m()\u001b[0m\n\u001b[1;32m    628\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m f \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    629\u001b[0m     \u001b[38;5;66;03m# Dummy value from _on_timeout().\u001b[39;00m\n\u001b[1;32m    630\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m exceptions\u001b[38;5;241m.\u001b[39mTimeoutError\n\u001b[0;32m--> 631\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresult\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/futures.py:203\u001b[0m, in \u001b[0;36mFuture.result\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    201\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m__log_traceback \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[1;32m    202\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 203\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception\u001b[38;5;241m.\u001b[39mwith_traceback(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception_tb)\n\u001b[1;32m    204\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_result\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py:314\u001b[0m, in \u001b[0;36mTask.__step_run_and_handle_result\u001b[0;34m(***failed resolving arguments***)\u001b[0m\n\u001b[1;32m    310\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m    311\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m exc \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    312\u001b[0m         \u001b[38;5;66;03m# We use the `send` method directly, because coroutines\u001b[39;00m\n\u001b[1;32m    313\u001b[0m         \u001b[38;5;66;03m# don't have `__iter__` and `__next__` methods.\u001b[39;00m\n\u001b[0;32m--> 314\u001b[0m         result \u001b[38;5;241m=\u001b[39m \u001b[43mcoro\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msend\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[1;32m    315\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    316\u001b[0m         result \u001b[38;5;241m=\u001b[39m coro\u001b[38;5;241m.\u001b[39mthrow(exc)\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/executor.py:34\u001b[0m, in \u001b[0;36mas_completed.<locals>.sema_coro\u001b[0;34m(coro)\u001b[0m\n\u001b[1;32m     32\u001b[0m \u001b[38;5;28;01masync\u001b[39;00m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21msema_coro\u001b[39m(coro):\n\u001b[1;32m     33\u001b[0m     \u001b[38;5;28;01masync\u001b[39;00m \u001b[38;5;28;01mwith\u001b[39;00m semaphore:\n\u001b[0;32m---> 34\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m coro\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/executor.py:59\u001b[0m, in \u001b[0;36mExecutor.wrap_callable_with_index.<locals>.wrapped_callable_async\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     57\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m     58\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mraise_exceptions:\n\u001b[0;32m---> 59\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m e\n\u001b[1;32m     60\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m     61\u001b[0m         logger\u001b[38;5;241m.\u001b[39merror(\n\u001b[1;32m     62\u001b[0m             \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mRunner in Executor raised an exception\u001b[39m\u001b[38;5;124m\"\u001b[39m, exc_info\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[1;32m     63\u001b[0m         )\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/executor.py:53\u001b[0m, in \u001b[0;36mExecutor.wrap_callable_with_index.<locals>.wrapped_callable_async\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     51\u001b[0m result \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mnan\n\u001b[1;32m     52\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 53\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m \u001b[38;5;28mcallable\u001b[39m(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m     54\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m MaxRetriesExceeded \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m     55\u001b[0m     \u001b[38;5;66;03m# this only for testset generation v2\u001b[39;00m\n\u001b[1;32m     56\u001b[0m     logger\u001b[38;5;241m.\u001b[39mwarning(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmax retries exceeded for \u001b[39m\u001b[38;5;132;01m{\u001b[39;00me\u001b[38;5;241m.\u001b[39mevolution\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/metrics/base.py:127\u001b[0m, in \u001b[0;36mMetric.ascore\u001b[0;34m(self, row, callbacks, thread_timeout)\u001b[0m\n\u001b[1;32m    125\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m group_cm\u001b[38;5;241m.\u001b[39mended:\n\u001b[1;32m    126\u001b[0m         rm\u001b[38;5;241m.\u001b[39mon_chain_error(e)\n\u001b[0;32m--> 127\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m e\n\u001b[1;32m    128\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    129\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m group_cm\u001b[38;5;241m.\u001b[39mended:\n",
      "File \u001b[0;32m~/Library/Caches/pypoetry/virtualenvs/rubberduck--9_uwOfW-py3.12/lib/python3.12/site-packages/ragas/metrics/base.py:120\u001b[0m, in \u001b[0;36mMetric.ascore\u001b[0;34m(self, row, callbacks, thread_timeout)\u001b[0m\n\u001b[1;32m    118\u001b[0m rm, group_cm \u001b[38;5;241m=\u001b[39m new_group(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname, inputs\u001b[38;5;241m=\u001b[39mrow, callbacks\u001b[38;5;241m=\u001b[39mcallbacks)\n\u001b[1;32m    119\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 120\u001b[0m     score \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mawait\u001b[39;00m asyncio\u001b[38;5;241m.\u001b[39mwait_for(\n\u001b[1;32m    121\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_ascore(row\u001b[38;5;241m=\u001b[39mrow, callbacks\u001b[38;5;241m=\u001b[39mgroup_cm),\n\u001b[1;32m    122\u001b[0m         timeout\u001b[38;5;241m=\u001b[39mthread_timeout,\n\u001b[1;32m    123\u001b[0m     )\n\u001b[1;32m    124\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    125\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m group_cm\u001b[38;5;241m.\u001b[39mended:\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/tasks.py:519\u001b[0m, in \u001b[0;36mwait_for\u001b[0;34m(fut, timeout)\u001b[0m\n\u001b[1;32m    516\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m exceptions\u001b[38;5;241m.\u001b[39mCancelledError \u001b[38;5;28;01mas\u001b[39;00m exc:\n\u001b[1;32m    517\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTimeoutError\u001b[39;00m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mexc\u001b[39;00m\n\u001b[0;32m--> 519\u001b[0m \u001b[38;5;28;01masync\u001b[39;00m \u001b[38;5;28;01mwith\u001b[39;00m timeouts\u001b[38;5;241m.\u001b[39mtimeout(timeout):\n\u001b[1;32m    520\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mawait\u001b[39;00m fut\n",
      "File \u001b[0;32m/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/timeouts.py:115\u001b[0m, in \u001b[0;36mTimeout.__aexit__\u001b[0;34m(self, exc_type, exc_val, exc_tb)\u001b[0m\n\u001b[1;32m    110\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;241m=\u001b[39m _State\u001b[38;5;241m.\u001b[39mEXPIRED\n\u001b[1;32m    112\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_task\u001b[38;5;241m.\u001b[39muncancel() \u001b[38;5;241m<\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_cancelling \u001b[38;5;129;01mand\u001b[39;00m exc_type \u001b[38;5;129;01mis\u001b[39;00m exceptions\u001b[38;5;241m.\u001b[39mCancelledError:\n\u001b[1;32m    113\u001b[0m         \u001b[38;5;66;03m# Since there are no new cancel requests, we're\u001b[39;00m\n\u001b[1;32m    114\u001b[0m         \u001b[38;5;66;03m# handling this.\u001b[39;00m\n\u001b[0;32m--> 115\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTimeoutError\u001b[39;00m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mexc_val\u001b[39;00m\n\u001b[1;32m    116\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;129;01mis\u001b[39;00m _State\u001b[38;5;241m.\u001b[39mENTERED:\n\u001b[1;32m    117\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;241m=\u001b[39m _State\u001b[38;5;241m.\u001b[39mEXITED\n",
      "\u001b[0;31mTimeoutError\u001b[0m: "
     ]
    }
   ],
   "source": [
    "from ragas.integrations.llama_index import evaluate\n",
    "\n",
    "evaluator_llm = Ollama(model='llama3', request_timeout=300.0)\n",
    "\n",
    "result = evaluate(\n",
    "    query_engine=query_engine,\n",
    "    metrics=metrics,\n",
    "    dataset=ds_dict,\n",
    "    llm=evaluator_llm,\n",
    "    embeddings=embeddings,\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}