modules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<u>Main Modules</u>\n",
    "\n",
    "1. `Model IO`: Interface with language models. Stuff that makes it easier to work with models.\n",
    "2. `Retrieval`: Interface with application-specific data\n",
    "3. `Agents`: Let chains choose which tools to use given high-level directives.\n",
    "\n",
    "<u>Additional</u>\n",
    "\n",
    "1. `Chains`: Common, building block compositions\n",
    "2. `Memory`: Persist application state between runs of a chain\n",
    "3. `Callbacks`: Log and stream intermediate steps of any chain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model IO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Caching in LLMs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://python.langchain.com/docs/integrations/llms/llm_caching\n",
    "\n",
    "Interesting to try next: `SQLAlchemyCache`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from langchain_community.chat_models import BedrockChat\n",
    "\n",
    "llm = BedrockChat(\n",
    "    model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    client=boto3.client(\"bedrock-runtime\"),\n",
    "    model_kwargs={\"temperature\": 0.0, \"max_tokens\":128}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.globals import set_llm_cache\n",
    "from langchain.cache import InMemoryCache\n",
    "\n",
    "set_llm_cache(InMemoryCache())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function `predict` was deprecated in LangChain 0.1.7 and will be removed in 0.2.0. Use invoke instead.\n",
      "  warn_deprecated(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 131 ms, sys: 16.9 ms, total: 148 ms\n",
      "Wall time: 1.59 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Here's a silly joke for you:\\n\\nWhy can't a bicycle stand up on its own? Because it's two-tired!\\n\\nHow was that? I tried to come up with a simple, lighthearted pun-based joke. Let me know if you'd like to hear another one.\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "llm.predict(\"Tell me a joke\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.78 ms, sys: 0 ns, total: 2.78 ms\n",
      "Wall time: 2.5 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Here's a silly joke for you:\\n\\nWhy can't a bicycle stand up on its own? Because it's two-tired!\\n\\nHow was that? I tried to come up with a simple, lighthearted pun-based joke. Let me know if you'd like to hear another one.\""
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "llm.predict(\"Tell me a joke\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output Parsers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<u>Useful</u>\n",
    "\n",
    "1. StrOutputParser\n",
    "2. JsonOutputParser, SimpleJsonOutputParser\n",
    "3. XMLOutputParser\n",
    "4. AgentOutputParser\n",
    "    - ReActJsonSingleInputOutputParser\n",
    "    - ReActSingleInputOutputParser\n",
    "    - JSONAgentOutputParser\n",
    "    - XMLAgentOutputParser\n",
    "    - SelfAskOutputParser\n",
    "5. RetryOutputParser\n",
    "6. OutputFixingParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from langchain_community.chat_models import BedrockChat\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "model = BedrockChat(\n",
    "    model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    client=boto3.client(\"bedrock-runtime\"),\n",
    "    model_kwargs={\"temperature\": 0.0, \"max_tokens\":128}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### JSON parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.output_parsers.json import SimpleJsonOutputParser\n",
    "\n",
    "json_prompt = PromptTemplate.from_template(\n",
    "    \"Return only an JSON object with an `answer` key that answers the following question: {question}\"\n",
    ")\n",
    "json_parser = SimpleJsonOutputParser()\n",
    "json_chain = json_prompt | model | json_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{},\n",
       " {'answer': ''},\n",
       " {'answer': 'The'},\n",
       " {'answer': 'The microsc'},\n",
       " {'answer': 'The microscope'},\n",
       " {'answer': 'The microscope was'},\n",
       " {'answer': 'The microscope was invented'},\n",
       " {'answer': 'The microscope was invented by'},\n",
       " {'answer': 'The microscope was invented by Hans'},\n",
       " {'answer': 'The microscope was invented by Hans L'},\n",
       " {'answer': 'The microscope was invented by Hans Lipp'},\n",
       " {'answer': 'The microscope was invented by Hans Lippersh'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey,'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zach'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Jan'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen,'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans Jan'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans Janssen'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans Janssen in'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans Janssen in the'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans Janssen in the early'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans Janssen in the early 17'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans Janssen in the early 17th'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans Janssen in the early 17th century'},\n",
       " {'answer': 'The microscope was invented by Hans Lippershey, Zacharias Janssen, and Hans Janssen in the early 17th century.'}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(json_chain.stream({\"question\": \"Who invented the microscope?\"}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 500/500 [06:01<00:00,  1.38it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "mem = []\n",
    "for i in tqdm(range(500)):\n",
    "    try:\n",
    "        res = json_chain.invoke({\"question\": \"Who invented the microscope?\"})\n",
    "    except Exception as e:\n",
    "        print(i, e)\n",
    "        mem.append({\n",
    "            \"i\": i,\n",
    "            \"error\": e,\n",
    "            \"res\": res\n",
    "        })\n",
    "\n",
    "len(mem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Output fixing parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "ename": "OutputParserException",
     "evalue": "Invalid json output: {'name': 'Tom Hanks', 'film_names': ['Forrest Gump']}",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mJSONDecodeError\u001b[0m                           Traceback (most recent call last)",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:212\u001b[0m, in \u001b[0;36mJsonOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m    211\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 212\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mparse_json_markdown\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    213\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m JSONDecodeError \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:157\u001b[0m, in \u001b[0;36mparse_json_markdown\u001b[0;34m(json_string, parser)\u001b[0m\n\u001b[1;32m    156\u001b[0m \u001b[38;5;66;03m# Parse the JSON string into a Python dictionary\u001b[39;00m\n\u001b[0;32m--> 157\u001b[0m parsed \u001b[38;5;241m=\u001b[39m \u001b[43mparser\u001b[49m\u001b[43m(\u001b[49m\u001b[43mjson_str\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    159\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m parsed\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:125\u001b[0m, in \u001b[0;36mparse_partial_json\u001b[0;34m(s, strict)\u001b[0m\n\u001b[1;32m    122\u001b[0m \u001b[38;5;66;03m# If we got here, we ran out of characters to remove\u001b[39;00m\n\u001b[1;32m    123\u001b[0m \u001b[38;5;66;03m# and still couldn't parse the string as JSON, so return the parse error\u001b[39;00m\n\u001b[1;32m    124\u001b[0m \u001b[38;5;66;03m# for the original string.\u001b[39;00m\n\u001b[0;32m--> 125\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mjson\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mloads\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstrict\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstrict\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/json/__init__.py:359\u001b[0m, in \u001b[0;36mloads\u001b[0;34m(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)\u001b[0m\n\u001b[1;32m    358\u001b[0m     kw[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mparse_constant\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m parse_constant\n\u001b[0;32m--> 359\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mcls\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkw\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdecode\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/json/decoder.py:337\u001b[0m, in \u001b[0;36mJSONDecoder.decode\u001b[0;34m(self, s, _w)\u001b[0m\n\u001b[1;32m    333\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Return the Python representation of ``s`` (a ``str`` instance\u001b[39;00m\n\u001b[1;32m    334\u001b[0m \u001b[38;5;124;03mcontaining a JSON document).\u001b[39;00m\n\u001b[1;32m    335\u001b[0m \n\u001b[1;32m    336\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m--> 337\u001b[0m obj, end \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mraw_decode\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43midx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m_w\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mend\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    338\u001b[0m end \u001b[38;5;241m=\u001b[39m _w(s, end)\u001b[38;5;241m.\u001b[39mend()\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/json/decoder.py:353\u001b[0m, in \u001b[0;36mJSONDecoder.raw_decode\u001b[0;34m(self, s, idx)\u001b[0m\n\u001b[1;32m    352\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 353\u001b[0m     obj, end \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mscan_once\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43midx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    354\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mStopIteration\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m err:\n",
      "\u001b[0;31mJSONDecodeError\u001b[0m: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[0;31mOutputParserException\u001b[0m                     Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[56], line 14\u001b[0m\n\u001b[1;32m     10\u001b[0m parser \u001b[38;5;241m=\u001b[39m PydanticOutputParser(pydantic_object\u001b[38;5;241m=\u001b[39mActor)\n\u001b[1;32m     12\u001b[0m misformatted \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m{\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mname\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m: \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mTom Hanks\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m, \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mfilm_names\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m: [\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mForrest Gump\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m]}\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m---> 14\u001b[0m \u001b[43mparser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmisformatted\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:218\u001b[0m, in \u001b[0;36mJsonOutputParser.parse\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m    217\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mparse\u001b[39m(\u001b[38;5;28mself\u001b[39m, text: \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[0;32m--> 218\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[43mGeneration\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtext\u001b[49m\u001b[43m)\u001b[49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/pydantic.py:21\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mparse_result\u001b[39m(\u001b[38;5;28mself\u001b[39m, result: List[Generation], \u001b[38;5;241m*\u001b[39m, partial: \u001b[38;5;28mbool\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[0;32m---> 21\u001b[0m     json_object \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43mresult\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     22\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m     23\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mpydantic_object\u001b[38;5;241m.\u001b[39mparse_obj(json_object)\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:215\u001b[0m, in \u001b[0;36mJsonOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m    213\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m JSONDecodeError \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    214\u001b[0m     msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mInvalid json output: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mtext\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m--> 215\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m OutputParserException(msg, llm_output\u001b[38;5;241m=\u001b[39mtext) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01me\u001b[39;00m\n",
      "\u001b[0;31mOutputParserException\u001b[0m: Invalid json output: {'name': 'Tom Hanks', 'film_names': ['Forrest Gump']}"
     ]
    }
   ],
   "source": [
    "from langchain.output_parsers import PydanticOutputParser\n",
    "from langchain_core.pydantic_v1 import BaseModel, Field\n",
    "from typing import List\n",
    "\n",
    "class Actor(BaseModel):\n",
    "    name: str = Field(description=\"name of an actor\")\n",
    "    film_names: List[str] = Field(description=\"list of names of films they starred in\")\n",
    "\n",
    "\n",
    "parser = PydanticOutputParser(pydantic_object=Actor)\n",
    "\n",
    "misformatted = \"{'name': 'Tom Hanks', 'film_names': ['Forrest Gump']}\"\n",
    "\n",
    "parser.parse(misformatted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Actor(name='Tom Hanks', film_names=['Forrest Gump'])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.output_parsers import OutputFixingParser\n",
    "\n",
    "fix_parser = OutputFixingParser.from_llm(parser=parser, llm=llm)\n",
    "\n",
    "fix_parser.parse(misformatted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Instructions:\n",
      "--------------\n",
      "\u001b[33;1m\u001b[1;3m{instructions}\u001b[0m\n",
      "--------------\n",
      "Completion:\n",
      "--------------\n",
      "\u001b[33;1m\u001b[1;3m{completion}\u001b[0m\n",
      "--------------\n",
      "\n",
      "Above, the Completion did not satisfy the constraints given in the Instructions.\n",
      "Error:\n",
      "--------------\n",
      "\u001b[33;1m\u001b[1;3m{error}\u001b[0m\n",
      "--------------\n",
      "\n",
      "Please try again. Please only respond with an answer that satisfies the constraints laid out in the Instructions:\n"
     ]
    }
   ],
   "source": [
    "# prompt that's used by default\n",
    "from langchain.output_parsers.prompts import NAIVE_FIX_PROMPT\n",
    "\n",
    "NAIVE_FIX_PROMPT.pretty_print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Instructions:\n",
      "<instructions>\n",
      "\u001b[33;1m\u001b[1;3m{instructions}\u001b[0m\n",
      "</instructions>\n",
      "\n",
      "Completion:\n",
      "<completion>\n",
      "\u001b[33;1m\u001b[1;3m{completion}\u001b[0m\n",
      "<completion>\n",
      "\n",
      "Above, the Completion did not satisfy the constraints given in the Instructions.\n",
      "<error>\n",
      "\u001b[33;1m\u001b[1;3m{error}\u001b[0m\n",
      "</error>\n",
      "\n",
      "Please only respond with completion that satisfies the constraints laid out in the Instructions. Do not generate text beyond given completion.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# custom prompt but same input_ariables as above\n",
    "\n",
    "template = \"\"\"\n",
    "Instructions:\n",
    "<instructions>\n",
    "{instructions}\n",
    "</instructions>\n",
    "\n",
    "Completion:\n",
    "<completion>\n",
    "{completion}\n",
    "<completion>\n",
    "\n",
    "Above, the Completion did not satisfy the constraints given in the Instructions.\n",
    "<error>\n",
    "{error}\n",
    "</error>\n",
    "\n",
    "Please only respond with completion that satisfies the constraints laid out in the Instructions. Do not generate text beyond given completion.\n",
    "\"\"\"\n",
    "\n",
    "fix_prompt = PromptTemplate.from_template(template=template) \n",
    "\n",
    "fix_prompt.pretty_print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Actor(name='Tom Hanks', film_names=['Forrest Gump'])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fix_parser_prompt = OutputFixingParser.from_llm(\n",
    "    parser=parser, llm=llm, prompt=fix_prompt\n",
    ")\n",
    "\n",
    "fix_parser_prompt.parse(misformatted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Retry parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts import PromptTemplate\n",
    "\n",
    "template = \"\"\"Based on the user question, provide an Action and Action Input for what step should be taken.\n",
    "{format_instructions}\n",
    "Question: {query}\n",
    "Response: \"\"\"\n",
    "\n",
    "prompt = PromptTemplate(\n",
    "    template=\"Answer the user query.\\n{format_instructions}\\n{query}\\n\",\n",
    "    input_variables=[\"query\"],\n",
    "    partial_variables={\"format_instructions\": parser.get_format_instructions()},\n",
    ")\n",
    "\n",
    "prompt_value = prompt.format_prompt(query=\"who invented computer?\")\n",
    "\n",
    "bad_response = '{\"action\": \"search\"}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "ename": "OutputParserException",
     "evalue": "Failed to parse Action from completion {'action': 'search'}. Got: 1 validation error for Action\naction_input\n  field required (type=value_error.missing)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValidationError\u001b[0m                           Traceback (most recent call last)",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/pydantic.py:23\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 23\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpydantic_object\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_obj\u001b[49m\u001b[43m(\u001b[49m\u001b[43mjson_object\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     24\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m ValidationError \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/main.py:526\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.parse_obj\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/main.py:341\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.__init__\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Action\naction_input\n  field required (type=value_error.missing)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mOutputParserException\u001b[0m                     Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[97], line 9\u001b[0m\n\u001b[1;32m      6\u001b[0m     action_input: \u001b[38;5;28mstr\u001b[39m \u001b[38;5;241m=\u001b[39m Field(description\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124minput to the action\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m      8\u001b[0m parser \u001b[38;5;241m=\u001b[39m PydanticOutputParser(pydantic_object\u001b[38;5;241m=\u001b[39mAction)\n\u001b[0;32m----> 9\u001b[0m \u001b[43mparser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse\u001b[49m\u001b[43m(\u001b[49m\u001b[43mbad_response\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:218\u001b[0m, in \u001b[0;36mJsonOutputParser.parse\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m    217\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mparse\u001b[39m(\u001b[38;5;28mself\u001b[39m, text: \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[0;32m--> 218\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[43mGeneration\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtext\u001b[49m\u001b[43m)\u001b[49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/pydantic.py:27\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m     25\u001b[0m name \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mpydantic_object\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m\n\u001b[1;32m     26\u001b[0m msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFailed to parse \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mname\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m from completion \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mjson_object\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. Got: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00me\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m---> 27\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m OutputParserException(msg, llm_output\u001b[38;5;241m=\u001b[39mjson_object)\n",
      "\u001b[0;31mOutputParserException\u001b[0m: Failed to parse Action from completion {'action': 'search'}. Got: 1 validation error for Action\naction_input\n  field required (type=value_error.missing)"
     ]
    }
   ],
   "source": [
    "from langchain.output_parsers import PydanticOutputParser\n",
    "from langchain_core.pydantic_v1 import BaseModel, Field\n",
    "\n",
    "class Action(BaseModel):\n",
    "    action: str = Field(description=\"action to take\")\n",
    "    action_input: str = Field(description=\"input to the action\")\n",
    "\n",
    "parser = PydanticOutputParser(pydantic_object=Action)\n",
    "parser.parse(bad_response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Action(action='search', action_input='')"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.output_parsers import OutputFixingParser\n",
    "\n",
    "fix_parser = OutputFixingParser.from_llm(parser=parser, llm=llm, prompt=fix_prompt)\n",
    "fix_parser.parse(bad_response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "ename": "OutputParserException",
     "evalue": "Invalid json output: Here is the response formatted as a JSON instance that conforms to the provided schema:\n\n{\n  \"action\": \"provide information\",\n  \"action_input\": \"The computer was invented by multiple people over time, with key contributions from pioneers in computer science and engineering. Some of the major figures in the invention of the computer include:\n\n- Charles Babbage - Designed the first mechanical computer, the Analytical Engine, in the 19th century.\n- Alan Turing - Developed the theoretical foundations of computer science and the concept of the Turing machine in the 1930s.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mJSONDecodeError\u001b[0m                           Traceback (most recent call last)",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:212\u001b[0m, in \u001b[0;36mJsonOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m    211\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 212\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mparse_json_markdown\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    213\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m JSONDecodeError \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:157\u001b[0m, in \u001b[0;36mparse_json_markdown\u001b[0;34m(json_string, parser)\u001b[0m\n\u001b[1;32m    156\u001b[0m \u001b[38;5;66;03m# Parse the JSON string into a Python dictionary\u001b[39;00m\n\u001b[0;32m--> 157\u001b[0m parsed \u001b[38;5;241m=\u001b[39m \u001b[43mparser\u001b[49m\u001b[43m(\u001b[49m\u001b[43mjson_str\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    159\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m parsed\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:125\u001b[0m, in \u001b[0;36mparse_partial_json\u001b[0;34m(s, strict)\u001b[0m\n\u001b[1;32m    122\u001b[0m \u001b[38;5;66;03m# If we got here, we ran out of characters to remove\u001b[39;00m\n\u001b[1;32m    123\u001b[0m \u001b[38;5;66;03m# and still couldn't parse the string as JSON, so return the parse error\u001b[39;00m\n\u001b[1;32m    124\u001b[0m \u001b[38;5;66;03m# for the original string.\u001b[39;00m\n\u001b[0;32m--> 125\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mjson\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mloads\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstrict\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstrict\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/json/__init__.py:359\u001b[0m, in \u001b[0;36mloads\u001b[0;34m(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)\u001b[0m\n\u001b[1;32m    358\u001b[0m     kw[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mparse_constant\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m parse_constant\n\u001b[0;32m--> 359\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mcls\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkw\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdecode\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/json/decoder.py:337\u001b[0m, in \u001b[0;36mJSONDecoder.decode\u001b[0;34m(self, s, _w)\u001b[0m\n\u001b[1;32m    333\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Return the Python representation of ``s`` (a ``str`` instance\u001b[39;00m\n\u001b[1;32m    334\u001b[0m \u001b[38;5;124;03mcontaining a JSON document).\u001b[39;00m\n\u001b[1;32m    335\u001b[0m \n\u001b[1;32m    336\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m--> 337\u001b[0m obj, end \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mraw_decode\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43midx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m_w\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mend\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    338\u001b[0m end \u001b[38;5;241m=\u001b[39m _w(s, end)\u001b[38;5;241m.\u001b[39mend()\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/json/decoder.py:355\u001b[0m, in \u001b[0;36mJSONDecoder.raw_decode\u001b[0;34m(self, s, idx)\u001b[0m\n\u001b[1;32m    354\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mStopIteration\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m err:\n\u001b[0;32m--> 355\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m JSONDecodeError(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mExpecting value\u001b[39m\u001b[38;5;124m\"\u001b[39m, s, err\u001b[38;5;241m.\u001b[39mvalue) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m    356\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m obj, end\n",
      "\u001b[0;31mJSONDecodeError\u001b[0m: Expecting value: line 1 column 1 (char 0)",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[0;31mOutputParserException\u001b[0m                     Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[49], line 4\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01moutput_parsers\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m RetryOutputParser\n\u001b[1;32m      3\u001b[0m retry_parser \u001b[38;5;241m=\u001b[39m RetryOutputParser\u001b[38;5;241m.\u001b[39mfrom_llm(parser\u001b[38;5;241m=\u001b[39mparser, llm\u001b[38;5;241m=\u001b[39mllm)\n\u001b[0;32m----> 4\u001b[0m \u001b[43mretry_parser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_with_prompt\u001b[49m\u001b[43m(\u001b[49m\u001b[43mbad_response\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mprompt_value\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain/output_parsers/retry.py:92\u001b[0m, in \u001b[0;36mRetryOutputParser.parse_with_prompt\u001b[0;34m(self, completion, prompt_value)\u001b[0m\n\u001b[1;32m     90\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m OutputParserException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m     91\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m retries \u001b[38;5;241m==\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmax_retries:\n\u001b[0;32m---> 92\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m e\n\u001b[1;32m     93\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m     94\u001b[0m         retries \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m1\u001b[39m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain/output_parsers/retry.py:89\u001b[0m, in \u001b[0;36mRetryOutputParser.parse_with_prompt\u001b[0;34m(self, completion, prompt_value)\u001b[0m\n\u001b[1;32m     87\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m retries \u001b[38;5;241m<\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmax_retries:\n\u001b[1;32m     88\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 89\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcompletion\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     90\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m OutputParserException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m     91\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m retries \u001b[38;5;241m==\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmax_retries:\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:218\u001b[0m, in \u001b[0;36mJsonOutputParser.parse\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m    217\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mparse\u001b[39m(\u001b[38;5;28mself\u001b[39m, text: \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[0;32m--> 218\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[43mGeneration\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtext\u001b[49m\u001b[43m)\u001b[49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/pydantic.py:21\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mparse_result\u001b[39m(\u001b[38;5;28mself\u001b[39m, result: List[Generation], \u001b[38;5;241m*\u001b[39m, partial: \u001b[38;5;28mbool\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[0;32m---> 21\u001b[0m     json_object \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43mresult\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     22\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m     23\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mpydantic_object\u001b[38;5;241m.\u001b[39mparse_obj(json_object)\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:215\u001b[0m, in \u001b[0;36mJsonOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m    213\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m JSONDecodeError \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    214\u001b[0m     msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mInvalid json output: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mtext\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m--> 215\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m OutputParserException(msg, llm_output\u001b[38;5;241m=\u001b[39mtext) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01me\u001b[39;00m\n",
      "\u001b[0;31mOutputParserException\u001b[0m: Invalid json output: Here is the response formatted as a JSON instance that conforms to the provided schema:\n\n{\n  \"action\": \"provide information\",\n  \"action_input\": \"The computer was invented by multiple people over time, with key contributions from pioneers in computer science and engineering. Some of the major figures in the invention of the computer include:\n\n- Charles Babbage - Designed the first mechanical computer, the Analytical Engine, in the 19th century.\n- Alan Turing - Developed the theoretical foundations of computer science and the concept of the Turing machine in the 1930s."
     ]
    }
   ],
   "source": [
    "from langchain.output_parsers import RetryOutputParser\n",
    "\n",
    "retry_parser = RetryOutputParser.from_llm(parser=parser, llm=llm)\n",
    "retry_parser.parse_with_prompt(bad_response, prompt_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'run'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValidationError\u001b[0m                           Traceback (most recent call last)",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/pydantic.py:23\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 23\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpydantic_object\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_obj\u001b[49m\u001b[43m(\u001b[49m\u001b[43mjson_object\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     24\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m ValidationError \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/main.py:526\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.parse_obj\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/main.py:341\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.__init__\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Action\naction_input\n  field required (type=value_error.missing)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mOutputParserException\u001b[0m                     Traceback (most recent call last)",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain/output_parsers/retry.py:187\u001b[0m, in \u001b[0;36mRetryWithErrorOutputParser.parse_with_prompt\u001b[0;34m(self, completion, prompt_value)\u001b[0m\n\u001b[1;32m    186\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 187\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcompletion\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    188\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m OutputParserException \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:218\u001b[0m, in \u001b[0;36mJsonOutputParser.parse\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m    217\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mparse\u001b[39m(\u001b[38;5;28mself\u001b[39m, text: \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[0;32m--> 218\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[43mGeneration\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtext\u001b[49m\u001b[43m)\u001b[49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/pydantic.py:27\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m     26\u001b[0m msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFailed to parse \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mname\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m from completion \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mjson_object\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. Got: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00me\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m---> 27\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m OutputParserException(msg, llm_output\u001b[38;5;241m=\u001b[39mjson_object)\n",
      "\u001b[0;31mOutputParserException\u001b[0m: Failed to parse Action from completion {'action': 'search'}. Got: 1 validation error for Action\naction_input\n  field required (type=value_error.missing)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[89], line 4\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01moutput_parsers\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m RetryWithErrorOutputParser\n\u001b[1;32m      3\u001b[0m retry_with_error_parser \u001b[38;5;241m=\u001b[39m RetryWithErrorOutputParser(parser\u001b[38;5;241m=\u001b[39mparser, llm\u001b[38;5;241m=\u001b[39mllm)\n\u001b[0;32m----> 4\u001b[0m \u001b[43mretry_with_error_parser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_with_prompt\u001b[49m\u001b[43m(\u001b[49m\u001b[43mbad_response\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mprompt_value\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain/output_parsers/retry.py:193\u001b[0m, in \u001b[0;36mRetryWithErrorOutputParser.parse_with_prompt\u001b[0;34m(self, completion, prompt_value)\u001b[0m\n\u001b[1;32m    191\u001b[0m         \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    192\u001b[0m             retries \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[0;32m--> 193\u001b[0m             completion \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mretry_chain\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m(\n\u001b[1;32m    194\u001b[0m                 prompt\u001b[38;5;241m=\u001b[39mprompt_value\u001b[38;5;241m.\u001b[39mto_string(),\n\u001b[1;32m    195\u001b[0m                 completion\u001b[38;5;241m=\u001b[39mcompletion,\n\u001b[1;32m    196\u001b[0m                 error\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mrepr\u001b[39m(e),\n\u001b[1;32m    197\u001b[0m             )\n\u001b[1;32m    199\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m OutputParserException(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFailed to parse\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'run'"
     ]
    }
   ],
   "source": [
    "from langchain.output_parsers import RetryWithErrorOutputParser\n",
    "\n",
    "retry_with_error_parser = RetryWithErrorOutputParser(parser=parser, llm=llm)\n",
    "retry_with_error_parser.parse_with_prompt(bad_response, prompt_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prompt that's used by default\n",
    "NAIVE_COMPLETION_RETRY = \"\"\"Prompt:\n",
    "{prompt}\n",
    "Completion:\n",
    "{completion}\n",
    "\n",
    "Above, the Completion did not satisfy the constraints given in the Prompt.\n",
    "Please try again:\"\"\"\n",
    "\n",
    "NAIVE_COMPLETION_RETRY_WITH_ERROR = \"\"\"Prompt:\n",
    "{prompt}\n",
    "Completion:\n",
    "{completion}\n",
    "\n",
    "Above, the Completion did not satisfy the constraints given in the Prompt.\n",
    "Details: {error}\n",
    "Please try again:\"\"\"\n",
    "\n",
    "NAIVE_RETRY_PROMPT = PromptTemplate.from_template(NAIVE_COMPLETION_RETRY)\n",
    "NAIVE_RETRY_WITH_ERROR_PROMPT = PromptTemplate.from_template(\n",
    "    NAIVE_COMPLETION_RETRY_WITH_ERROR\n",
    ")\n",
    "\n",
    "# NAIVE_RETRY_PROMPT.pretty_print()\n",
    "# NAIVE_RETRY_WITH_ERROR_PROMPT.pretty_print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Prompt:\n",
      "<prompt>\n",
      "\u001b[33;1m\u001b[1;3m{prompt}\u001b[0m\n",
      "</prompt>\n",
      "\n",
      "Completion:\n",
      "<completion>\n",
      "\u001b[33;1m\u001b[1;3m{completion}\u001b[0m\n",
      "</completion>\n",
      "\n",
      "Details: \u001b[33;1m\u001b[1;3m{error}\u001b[0m\n",
      "\n",
      "Please only respond with completion that satisfies the constraints given in the Prompt. Do not generate text beyond given completion.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# custom prompt but same input_ariables as above\n",
    "\n",
    "template = \"\"\"\n",
    "Prompt:\n",
    "<prompt>\n",
    "{prompt}\n",
    "</prompt>\n",
    "\n",
    "Completion:\n",
    "<completion>\n",
    "{completion}\n",
    "</completion>\n",
    "\n",
    "Please only respond with completion that satisfies the constraints given in the Prompt. Do not generate text beyond given completion.\n",
    "\"\"\"\n",
    "\n",
    "retry_prompt = PromptTemplate.from_template(template=template) \n",
    "\n",
    "# retry_prompt.pretty_print()\n",
    "\n",
    "error_template = \"\"\"\n",
    "Prompt:\n",
    "<prompt>\n",
    "{prompt}\n",
    "</prompt>\n",
    "\n",
    "Completion:\n",
    "<completion>\n",
    "{completion}\n",
    "</completion>\n",
    "\n",
    "Details: {error}\n",
    "\n",
    "Please only respond with completion that satisfies the constraints given in the Prompt. Do not generate text beyond given completion.\n",
    "\"\"\"\n",
    "\n",
    "retry_with_error_prompt = PromptTemplate.from_template(template=error_template) \n",
    "\n",
    "retry_with_error_prompt.pretty_print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Action(action='search', action_input='who invented computer')"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retry_parser = RetryOutputParser.from_llm(\n",
    "    parser=parser, llm=llm, prompt=retry_prompt\n",
    ")\n",
    "retry_parser.parse_with_prompt(bad_response, prompt_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'NoneType' object has no attribute 'run'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValidationError\u001b[0m                           Traceback (most recent call last)",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/pydantic.py:23\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 23\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpydantic_object\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_obj\u001b[49m\u001b[43m(\u001b[49m\u001b[43mjson_object\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     24\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m ValidationError \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/main.py:526\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.parse_obj\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/main.py:341\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.__init__\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mValidationError\u001b[0m: 1 validation error for Action\naction_input\n  field required (type=value_error.missing)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mOutputParserException\u001b[0m                     Traceback (most recent call last)",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain/output_parsers/retry.py:187\u001b[0m, in \u001b[0;36mRetryWithErrorOutputParser.parse_with_prompt\u001b[0;34m(self, completion, prompt_value)\u001b[0m\n\u001b[1;32m    186\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 187\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcompletion\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    188\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m OutputParserException \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/json.py:218\u001b[0m, in \u001b[0;36mJsonOutputParser.parse\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m    217\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mparse\u001b[39m(\u001b[38;5;28mself\u001b[39m, text: \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[0;32m--> 218\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[43mGeneration\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtext\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtext\u001b[49m\u001b[43m)\u001b[49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/output_parsers/pydantic.py:27\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse_result\u001b[0;34m(self, result, partial)\u001b[0m\n\u001b[1;32m     26\u001b[0m msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFailed to parse \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mname\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m from completion \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mjson_object\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. Got: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00me\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m---> 27\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m OutputParserException(msg, llm_output\u001b[38;5;241m=\u001b[39mjson_object)\n",
      "\u001b[0;31mOutputParserException\u001b[0m: Failed to parse Action from completion {'action': 'search'}. Got: 1 validation error for Action\naction_input\n  field required (type=value_error.missing)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[102], line 6\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01moutput_parsers\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m RetryWithErrorOutputParser\n\u001b[1;32m      3\u001b[0m retry_with_error_parser \u001b[38;5;241m=\u001b[39m RetryWithErrorOutputParser(\n\u001b[1;32m      4\u001b[0m     parser\u001b[38;5;241m=\u001b[39mparser, llm\u001b[38;5;241m=\u001b[39mllm, prompt\u001b[38;5;241m=\u001b[39mretry_with_error_prompt\n\u001b[1;32m      5\u001b[0m )\n\u001b[0;32m----> 6\u001b[0m \u001b[43mretry_with_error_parser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_with_prompt\u001b[49m\u001b[43m(\u001b[49m\u001b[43mbad_response\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mprompt_value\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain/output_parsers/retry.py:193\u001b[0m, in \u001b[0;36mRetryWithErrorOutputParser.parse_with_prompt\u001b[0;34m(self, completion, prompt_value)\u001b[0m\n\u001b[1;32m    191\u001b[0m         \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    192\u001b[0m             retries \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[0;32m--> 193\u001b[0m             completion \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mretry_chain\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m(\n\u001b[1;32m    194\u001b[0m                 prompt\u001b[38;5;241m=\u001b[39mprompt_value\u001b[38;5;241m.\u001b[39mto_string(),\n\u001b[1;32m    195\u001b[0m                 completion\u001b[38;5;241m=\u001b[39mcompletion,\n\u001b[1;32m    196\u001b[0m                 error\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mrepr\u001b[39m(e),\n\u001b[1;32m    197\u001b[0m             )\n\u001b[1;32m    199\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m OutputParserException(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFailed to parse\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'NoneType' object has no attribute 'run'"
     ]
    }
   ],
   "source": [
    "# below didnt work - need to pass prompt_value with error I guess\n",
    "from langchain.output_parsers import RetryWithErrorOutputParser\n",
    "\n",
    "retry_with_error_parser = RetryWithErrorOutputParser(\n",
    "    parser=parser, llm=llm, prompt=retry_with_error_prompt\n",
    ")\n",
    "retry_with_error_parser.parse_with_prompt(bad_response, prompt_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### XML output parser & challenges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here is the shortened filmography for Tom Hanks:\n",
      "\n",
      "<movie>Forrest Gump</movie>\n",
      "<movie>Saving Private Ryan</movie>\n",
      "<movie>Cast Away</movie>\n",
      "<movie>Apollo 13</movie>\n",
      "<movie>Toy Story</movie>\n",
      "<movie>The Green Mile</movie>\n",
      "<movie>Catch Me If You Can</movie>\n",
      "<movie>Captain Phillips</movie>\n",
      "<movie>Sully</movie>\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "template = \"\"\"Generate the shortened filmography for {actor}.\n",
    "Please enclose the movies in <movie></movie> tags.\"\"\"\n",
    "# prompt = ChatPromptTemplate.from_messages(\n",
    "#     (\"human\", template)\n",
    "# )\n",
    "prompt = PromptTemplate.from_template(\n",
    "    template=template\n",
    ")\n",
    "\n",
    "prompt_value = prompt.format(actor=\"Tom Hanks\")\n",
    "\n",
    "print(llm.invoke(prompt_value).content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'filmography': [{'film': [{'title': 'Forrest Gump'}, {'year': '1994'}]}, {'film': [{'title': 'Saving Private Ryan'}, {'year': '1998'}]}, {'film': [{'title': 'Cast Away'}, {'year': '2000'}]}, {'film': [{'title': 'The Green Mile'}, {'year': '1999'}]}, {'film': [{'title': 'Toy Story'}, {'year': '1995'}]}]}\n"
     ]
    }
   ],
   "source": [
    "from langchain.output_parsers import XMLOutputParser\n",
    "\n",
    "parser = XMLOutputParser()\n",
    "\n",
    "model = BedrockChat(\n",
    "    model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    client=boto3.client(\"bedrock-runtime\"),\n",
    "    model_kwargs={\"temperature\": 0.0, \"max_tokens\":512}\n",
    ")\n",
    "\n",
    "new_template = \"\"\"Format Instructions:\n",
    "<format_instructions>\n",
    "{format_instructions}\n",
    "</format_instructions>\n",
    "\n",
    "Follow the Format Instructions and generate shortened the filmography for {actor}. Do not explain. Start with XML tags.\n",
    "\"\"\"\n",
    "\n",
    "# Follow the Format Instructions and generate the  filmography for {actor}. Do not explain. Start with XML tags.\n",
    "\n",
    "# Enclose within ```xml ```.\n",
    "\n",
    "# print(parser.get_format_instructions())\n",
    "prompt = PromptTemplate(\n",
    "    template=new_template,\n",
    "    input_variables=[\"actor\"],\n",
    "    partial_variables={\n",
    "        \"format_instructions\": parser.get_format_instructions(),\n",
    "    }\n",
    ")\n",
    "\n",
    "chain = prompt | model | parser\n",
    "\n",
    "print(chain.invoke({\"actor\": \"Tom Hanks\"}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "content='<movies>\\n    <actor>\\n        <name>Tom Hanks</name>\\n        <film>\\n            <name>Forrest Gump</name>\\n            <genre>Drama, Comedy</genre>\\n        </film>\\n        <film>\\n            <name>Saving Private Ryan</name>\\n            <genre>War, Drama</genre>\\n        </film>\\n        <film>\\n            <name>Cast Away</name>\\n            <genre>Drama, Adventure</genre>\\n        </film>\\n        <film>\\n            <name>Toy Story</name>\\n            <genre>Animation, Comedy, Family</genre>\\n        </film>\\n    </actor>\\n</movies>'\n"
     ]
    }
   ],
   "source": [
    "parser_mod = XMLOutputParser(tags=[\"movies\", \"actor\", \"film\", \"name\", \"genre\"])\n",
    "\n",
    "prompt_mod = PromptTemplate(\n",
    "    template=new_template,\n",
    "    input_variables=[\"actor\"],\n",
    "    partial_variables={\n",
    "        \"format_instructions\": parser_mod.get_format_instructions(),\n",
    "    }\n",
    ")\n",
    "\n",
    "# chain = prompt_mod | model | parser_mod\n",
    "chain = prompt_mod | model\n",
    "\n",
    "print(chain.invoke({\"actor\": \"Tom Hanks\"}))"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "XML parser fails to parse when text is multiline.\n",
    "\n",
    "[works]\n",
    "<sql>select * from products;</sql> \n",
    "\n",
    "[doesn’t work]\n",
    "<sql>\n",
    "select *\n",
    "from products;\n",
    "</sql>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "select *\n",
      "from products;\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# for single key its possible \n",
    "\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "\n",
    "def _sanitize_output(text: str):\n",
    "    _, after = text.split(\"<sql>\")\n",
    "    return after.split(\"</sql>\")[0]\n",
    "\n",
    "\n",
    "# chain = sql_prompt| model | StrOutputParser() | _sanitize_output\n",
    "\n",
    "inputs = \"\"\"<sql>\n",
    "select *\n",
    "from products;\n",
    "</sql>\"\"\"\n",
    "\n",
    "sql_query = _sanitize_output(inputs)\n",
    "print(sql_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<movies>\n",
      "    <actor>\n",
      "        <name>Tom Hanks</name>\n",
      "        <film>\n",
      "            <name>Forrest Gump</name>\n",
      "            <genre>Drama, Comedy</genre>\n",
      "        </film>\n",
      "        <film>\n",
      "            <name>Saving Private Ryan</name>\n",
      "            <genre>War, Drama</genre>\n",
      "        </film>\n",
      "        <film>\n",
      "            <name>Cast Away</name>\n",
      "            <genre>Drama, Adventure</genre>\n",
      "        </film>\n",
      "        <film>\n",
      "            <name>Toy Story</name>\n",
      "            <genre>Animation, Comedy, Family</genre>\n",
      "        </film>\n",
      "    </actor>\n",
      "</movies>\n"
     ]
    }
   ],
   "source": [
    "# but what to do when we have so many\n",
    "print('<movies>\\n    <actor>\\n        <name>Tom Hanks</name>\\n        <film>\\n            <name>Forrest Gump</name>\\n            <genre>Drama, Comedy</genre>\\n        </film>\\n        <film>\\n            <name>Saving Private Ryan</name>\\n            <genre>War, Drama</genre>\\n        </film>\\n        <film>\\n            <name>Cast Away</name>\\n            <genre>Drama, Adventure</genre>\\n        </film>\\n        <film>\\n            <name>Toy Story</name>\\n            <genre>Animation, Comedy, Family</genre>\\n        </film>\\n    </actor>\\n</movies>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### YAML parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "from langchain.output_parsers import YamlOutputParser\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain_core.pydantic_v1 import BaseModel, Field\n",
    "\n",
    "class Joke(BaseModel):\n",
    "    setup: str=Field(description=\"question to set up a joke\")\n",
    "    punchline: str=Field(description=\"answer to resolve the joke\")\n",
    "\n",
    "parser = YamlOutputParser(pydantic_object=Joke)\n",
    "\n",
    "# print(parser.get_format_instructions())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Joke(setup=\"Why don't scientists trust atoms?\", punchline='Because they make up everything!')"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt = PromptTemplate(\n",
    "    template=\"Answer the user query.\\n{format_instructions}\\n{query}\\n\",\n",
    "    input_variables=[\"query\"],\n",
    "    partial_variables={\"format_instructions\": parser.get_format_instructions()},\n",
    ")\n",
    "\n",
    "chain = prompt | model | parser\n",
    "\n",
    "joke_query = \"Tell me a joke.\"\n",
    "chain.invoke({\"query\": joke_query})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<u>Flow</u>\n",
    "\n",
    "Source -> Load -> Transform -> Embed -> Store -> Retrieve\n",
    "\n",
    "<u>Concepts</u>\n",
    "1. Document loaders: load Documents from diff sources. Document(page_content, metdata)\n",
    "2. Text Splitting: chunking strategy. ex: optimized logic for code, markdown docs\n",
    "3. Text Embedding models: embeddings capture the semantic meaning of the text\n",
    "4. Vector Stores: store and search embeddings\n",
    "5. Retrievers: retrieval algo (semantic search etc) + vectorstore\n",
    "    - parent document retriever\n",
    "    - self-query retriever\n",
    "    - ensemble retriever\n",
    "6. Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Document Loaders: PDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Ignoring invalid distribution -ackaging (/opt/conda/envs/pytorch/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -ackaging (/opt/conda/envs/pytorch/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "! pip install pypdf rapidocr-onnxruntime --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.54 s, sys: 62.7 ms, total: 1.6 s\n",
      "Wall time: 1.68 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "loader = PyPDFLoader(\"https://arxiv.org/pdf/1706.03762.pdf\")\n",
    "pages = loader.load_and_split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Figure 1: The Transformer - model architecture.\n",
      "The Transformer follows this overall architecture using stacked self-attention and point-wise, fully\n",
      "connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1,\n",
      "respectively.\n",
      "3.1 Encoder and Decoder Stacks\n",
      "Encoder: The encoder is composed of a stack of N= 6 identical layers. Each layer has two\n",
      "sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-\n",
      "wise fully connected feed-forward network. We employ a residual connection [ 11] around each of\n",
      "the two sub-layers, followed by layer normalization [ 1]. That is, the output of each sub-layer is\n",
      "LayerNorm( x+ Sublayer( x)), where Sublayer( x)is the function implemented by the sub-layer\n",
      "itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding\n",
      "layers, produce outputs of dimension dmodel = 512 .\n",
      "Decoder: The decoder is also composed of a stack of N= 6identical layers. In addition to the two\n",
      "sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head\n",
      "attention over the output of the encoder stack. Similar to the encoder, we employ residual connections\n",
      "around each of the sub-layers, followed by layer normalization. We also modify the self-attention\n",
      "sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This\n",
      "masking, combined with fact that the output embeddings are offset by one position, ensures that the\n",
      "predictions for position ican depend only on the known outputs at positions less than i.\n",
      "3.2 Attention\n",
      "An attention function can be described as mapping a query and a set of key-value pairs to an output,\n",
      "where the query, keys, values, and output are all vectors. The output is computed as a weighted sum\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "print(pages[3].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Document loaders: PDF - extract images as text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.78 s, sys: 3.09 s, total: 10.9 s\n",
      "Wall time: 10.9 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "\n",
    "loader = PyPDFLoader(\"https://arxiv.org/pdf/1706.03762.pdf\", extract_images=True)\n",
    "pages = loader.load_and_split()\n",
    "# pages = loader.load()\n",
    "\n",
    "len(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='Provided proper attribution is provided, Google hereby grants permission to\\nreproduce the tables and figures in this paper solely for use in journalistic or\\nscholarly works.\\nAttention Is All You Need\\nAshish Vaswani∗\\nGoogle Brain\\navaswani@google.comNoam Shazeer∗\\nGoogle Brain\\nnoam@google.comNiki Parmar∗\\nGoogle Research\\nnikip@google.comJakob Uszkoreit∗\\nGoogle Research\\nusz@google.com\\nLlion Jones∗\\nGoogle Research\\nllion@google.comAidan N. Gomez∗ †\\nUniversity of Toronto\\naidan@cs.toronto.eduŁukasz Kaiser∗\\nGoogle Brain\\nlukaszkaiser@google.com\\nIllia Polosukhin∗ ‡\\nillia.polosukhin@gmail.com\\nAbstract\\nThe dominant sequence transduction models are based on complex recurrent or\\nconvolutional neural networks that include an encoder and a decoder. The best\\nperforming models also connect the encoder and decoder through an attention\\nmechanism. We propose a new simple network architecture, the Transformer,\\nbased solely on attention mechanisms, dispensing with recurrence and convolutions\\nentirely. Experiments on two machine translation tasks show these models to\\nbe superior in quality while being more parallelizable and requiring significantly\\nless time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-\\nto-German translation task, improving over the existing best results, including\\nensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task,\\nour model establishes a new single-model state-of-the-art BLEU score of 41.8 after\\ntraining for 3.5 days on eight GPUs, a small fraction of the training costs of the\\nbest models from the literature. We show that the Transformer generalizes well to\\nother tasks by applying it successfully to English constituency parsing both with\\nlarge and limited training data.\\n∗Equal contribution. Listing order is random. Jakob proposed replacing RNNs with self-attention and started\\nthe effort to evaluate this idea. Ashish, with Illia, designed and implemented the first Transformer models and\\nhas been crucially involved in every aspect of this work. Noam proposed scaled dot-product attention, multi-head\\nattention and the parameter-free position representation and became the other person involved in nearly every\\ndetail. Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and\\ntensor2tensor. Llion also experimented with novel model variants, was responsible for our initial codebase, and\\nefficient inference and visualizations. Lukasz and Aidan spent countless long days designing various parts of and\\nimplementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating\\nour research.\\n†Work performed while at Google Brain.\\n‡Work performed while at Google Research.\\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.arXiv:1706.03762v7  [cs.CL]  2 Aug 2023', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 0}),\n",
       " Document(page_content='1 Introduction\\nRecurrent neural networks, long short-term memory [ 13] and gated recurrent [ 7] neural networks\\nin particular, have been firmly established as state of the art approaches in sequence modeling and\\ntransduction problems such as language modeling and machine translation [ 35,2,5]. Numerous\\nefforts have since continued to push the boundaries of recurrent language models and encoder-decoder\\narchitectures [38, 24, 15].\\nRecurrent models typically factor computation along the symbol positions of the input and output\\nsequences. Aligning the positions to steps in computation time, they generate a sequence of hidden\\nstates ht, as a function of the previous hidden state ht−1and the input for position t. This inherently\\nsequential nature precludes parallelization within training examples, which becomes critical at longer\\nsequence lengths, as memory constraints limit batching across examples. Recent work has achieved\\nsignificant improvements in computational efficiency through factorization tricks [ 21] and conditional\\ncomputation [ 32], while also improving model performance in case of the latter. The fundamental\\nconstraint of sequential computation, however, remains.\\nAttention mechanisms have become an integral part of compelling sequence modeling and transduc-\\ntion models in various tasks, allowing modeling of dependencies without regard to their distance in\\nthe input or output sequences [ 2,19]. In all but a few cases [ 27], however, such attention mechanisms\\nare used in conjunction with a recurrent network.\\nIn this work we propose the Transformer, a model architecture eschewing recurrence and instead\\nrelying entirely on an attention mechanism to draw global dependencies between input and output.\\nThe Transformer allows for significantly more parallelization and can reach a new state of the art in\\ntranslation quality after being trained for as little as twelve hours on eight P100 GPUs.\\n2 Background\\nThe goal of reducing sequential computation also forms the foundation of the Extended Neural GPU\\n[16], ByteNet [ 18] and ConvS2S [ 9], all of which use convolutional neural networks as basic building\\nblock, computing hidden representations in parallel for all input and output positions. In these models,\\nthe number of operations required to relate signals from two arbitrary input or output positions grows\\nin the distance between positions, linearly for ConvS2S and logarithmically for ByteNet. This makes\\nit more difficult to learn dependencies between distant positions [ 12]. In the Transformer this is\\nreduced to a constant number of operations, albeit at the cost of reduced effective resolution due\\nto averaging attention-weighted positions, an effect we counteract with Multi-Head Attention as\\ndescribed in section 3.2.\\nSelf-attention, sometimes called intra-attention is an attention mechanism relating different positions\\nof a single sequence in order to compute a representation of the sequence. Self-attention has been\\nused successfully in a variety of tasks including reading comprehension, abstractive summarization,\\ntextual entailment and learning task-independent sentence representations [4, 27, 28, 22].\\nEnd-to-end memory networks are based on a recurrent attention mechanism instead of sequence-\\naligned recurrence and have been shown to perform well on simple-language question answering and\\nlanguage modeling tasks [34].\\nTo the best of our knowledge, however, the Transformer is the first transduction model relying\\nentirely on self-attention to compute representations of its input and output without using sequence-\\naligned RNNs or convolution. In the following sections, we will describe the Transformer, motivate\\nself-attention and discuss its advantages over models such as [17, 18] and [9].\\n3 Model Architecture\\nMost competitive neural sequence transduction models have an encoder-decoder structure [ 5,2,35].\\nHere, the encoder maps an input sequence of symbol representations (x1, ..., x n)to a sequence', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 1}),\n",
       " Document(page_content='Most competitive neural sequence transduction models have an encoder-decoder structure [ 5,2,35].\\nHere, the encoder maps an input sequence of symbol representations (x1, ..., x n)to a sequence\\nof continuous representations z= (z1, ..., z n). Given z, the decoder then generates an output\\nsequence (y1, ..., y m)of symbols one element at a time. At each step the model is auto-regressive\\n[10], consuming the previously generated symbols as additional input when generating the next.\\n2', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 1}),\n",
       " Document(page_content='Figure 1: The Transformer - model architecture.\\nThe Transformer follows this overall architecture using stacked self-attention and point-wise, fully\\nconnected layers for both the encoder and decoder, shown in the left and right halves of Figure 1,\\nrespectively.\\n3.1 Encoder and Decoder Stacks\\nEncoder: The encoder is composed of a stack of N= 6 identical layers. Each layer has two\\nsub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-\\nwise fully connected feed-forward network. We employ a residual connection [ 11] around each of\\nthe two sub-layers, followed by layer normalization [ 1]. That is, the output of each sub-layer is\\nLayerNorm( x+ Sublayer( x)), where Sublayer( x)is the function implemented by the sub-layer\\nitself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding\\nlayers, produce outputs of dimension dmodel = 512 .\\nDecoder: The decoder is also composed of a stack of N= 6identical layers. In addition to the two\\nsub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head\\nattention over the output of the encoder stack. Similar to the encoder, we employ residual connections\\naround each of the sub-layers, followed by layer normalization. We also modify the self-attention\\nsub-layer in the decoder stack to prevent positions from attending to subsequent positions. This\\nmasking, combined with fact that the output embeddings are offset by one position, ensures that the\\npredictions for position ican depend only on the known outputs at positions less than i.\\n3.2 Attention\\nAn attention function can be described as mapping a query and a set of key-value pairs to an output,\\nwhere the query, keys, values, and output are all vectors. The output is computed as a weighted sum\\n3Output\\nProbabilities\\nSoftmax\\nLinear\\nAdd & Norm\\nFeed\\nForward\\nAdd & Norm\\nAdd & Norm\\nMulti-Head\\nFeed\\nAttention\\nForward\\nNx\\nAdd & Norm\\nZx\\nAdd & Norm\\nMasked\\nMulti-Head\\nMulti-Head\\nAttention\\nAttention\\nPositional\\nPositional\\nEncoding\\nEncoding\\nInput\\nOutput\\nEmbedding\\nEmbedding\\nInputs\\nOutputs\\n(shifted right)', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 2}),\n",
       " Document(page_content='Scaled Dot-Product Attention\\n Multi-Head Attention\\nFigure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several\\nattention layers running in parallel.\\nof the values, where the weight assigned to each value is computed by a compatibility function of the\\nquery with the corresponding key.\\n3.2.1 Scaled Dot-Product Attention\\nWe call our particular attention \"Scaled Dot-Product Attention\" (Figure 2). The input consists of\\nqueries and keys of dimension dk, and values of dimension dv. We compute the dot products of the\\nquery with all keys, divide each by√dk, and apply a softmax function to obtain the weights on the\\nvalues.\\nIn practice, we compute the attention function on a set of queries simultaneously, packed together\\ninto a matrix Q. The keys and values are also packed together into matrices KandV. We compute\\nthe matrix of outputs as:\\nAttention( Q, K, V ) = softmax(QKT\\n√dk)V (1)\\nThe two most commonly used attention functions are additive attention [ 2], and dot-product (multi-\\nplicative) attention. Dot-product attention is identical to our algorithm, except for the scaling factor\\nof1√dk. Additive attention computes the compatibility function using a feed-forward network with\\na single hidden layer. While the two are similar in theoretical complexity, dot-product attention is\\nmuch faster and more space-efficient in practice, since it can be implemented using highly optimized\\nmatrix multiplication code.\\nWhile for small values of dkthe two mechanisms perform similarly, additive attention outperforms\\ndot product attention without scaling for larger values of dk[3]. We suspect that for large values of\\ndk, the dot products grow large in magnitude, pushing the softmax function into regions where it has\\nextremely small gradients4. To counteract this effect, we scale the dot products by1√dk.\\n3.2.2 Multi-Head Attention\\nInstead of performing a single attention function with dmodel-dimensional keys, values and queries,\\nwe found it beneficial to linearly project the queries, keys and values htimes with different, learned\\nlinear projections to dk,dkanddvdimensions, respectively. On each of these projected versions of\\nqueries, keys and values we then perform the attention function in parallel, yielding dv-dimensional\\n4To illustrate why the dot products get large, assume that the components of qandkare independent random\\nvariables with mean 0and variance 1. Then their dot product, q·k=Pdk\\ni=1qiki, has mean 0and variance dk.\\n4MatMul\\nSoftMax\\nMask\\n(opt.)\\nScale\\nMatMulLinear\\nConcat\\nScaled Dot-Product\\nAttention\\nLinear\\nLinear\\nLinear', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 3}),\n",
       " Document(page_content='output values. These are concatenated and once again projected, resulting in the final values, as\\ndepicted in Figure 2.\\nMulti-head attention allows the model to jointly attend to information from different representation\\nsubspaces at different positions. With a single attention head, averaging inhibits this.\\nMultiHead( Q, K, V ) = Concat(head 1, ...,head h)WO\\nwhere head i= Attention( QWQ\\ni, KWK\\ni, V WV\\ni)\\nWhere the projections are parameter matrices WQ\\ni∈Rdmodel×dk,WK\\ni∈Rdmodel×dk,WV\\ni∈Rdmodel×dv\\nandWO∈Rhdv×dmodel.\\nIn this work we employ h= 8 parallel attention layers, or heads. For each of these we use\\ndk=dv=dmodel/h= 64 . Due to the reduced dimension of each head, the total computational cost\\nis similar to that of single-head attention with full dimensionality.\\n3.2.3 Applications of Attention in our Model\\nThe Transformer uses multi-head attention in three different ways:\\n•In \"encoder-decoder attention\" layers, the queries come from the previous decoder layer,\\nand the memory keys and values come from the output of the encoder. This allows every\\nposition in the decoder to attend over all positions in the input sequence. This mimics the\\ntypical encoder-decoder attention mechanisms in sequence-to-sequence models such as\\n[38, 2, 9].\\n•The encoder contains self-attention layers. In a self-attention layer all of the keys, values\\nand queries come from the same place, in this case, the output of the previous layer in the\\nencoder. Each position in the encoder can attend to all positions in the previous layer of the\\nencoder.\\n•Similarly, self-attention layers in the decoder allow each position in the decoder to attend to\\nall positions in the decoder up to and including that position. We need to prevent leftward\\ninformation flow in the decoder to preserve the auto-regressive property. We implement this\\ninside of scaled dot-product attention by masking out (setting to −∞) all values in the input\\nof the softmax which correspond to illegal connections. See Figure 2.\\n3.3 Position-wise Feed-Forward Networks\\nIn addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully\\nconnected feed-forward network, which is applied to each position separately and identically. This\\nconsists of two linear transformations with a ReLU activation in between.\\nFFN( x) = max(0 , xW 1+b1)W2+b2 (2)\\nWhile the linear transformations are the same across different positions, they use different parameters\\nfrom layer to layer. Another way of describing this is as two convolutions with kernel size 1.\\nThe dimensionality of input and output is dmodel = 512 , and the inner-layer has dimensionality\\ndff= 2048 .\\n3.4 Embeddings and Softmax\\nSimilarly to other sequence transduction models, we use learned embeddings to convert the input\\ntokens and output tokens to vectors of dimension dmodel. We also use the usual learned linear transfor-\\nmation and softmax function to convert the decoder output to predicted next-token probabilities. In\\nour model, we share the same weight matrix between the two embedding layers and the pre-softmax\\nlinear transformation, similar to [ 30]. In the embedding layers, we multiply those weights by√dmodel.\\n5', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 4}),\n",
       " Document(page_content='Table 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations\\nfor different layer types. nis the sequence length, dis the representation dimension, kis the kernel\\nsize of convolutions and rthe size of the neighborhood in restricted self-attention.\\nLayer Type Complexity per Layer Sequential Maximum Path Length\\nOperations\\nSelf-Attention O(n2·d) O(1) O(1)\\nRecurrent O(n·d2) O(n) O(n)\\nConvolutional O(k·n·d2) O(1) O(logk(n))\\nSelf-Attention (restricted) O(r·n·d) O(1) O(n/r)\\n3.5 Positional Encoding\\nSince our model contains no recurrence and no convolution, in order for the model to make use of the\\norder of the sequence, we must inject some information about the relative or absolute position of the\\ntokens in the sequence. To this end, we add \"positional encodings\" to the input embeddings at the\\nbottoms of the encoder and decoder stacks. The positional encodings have the same dimension dmodel\\nas the embeddings, so that the two can be summed. There are many choices of positional encodings,\\nlearned and fixed [9].\\nIn this work, we use sine and cosine functions of different frequencies:\\nPE(pos,2i)=sin(pos/100002i/d model)\\nPE(pos,2i+1)=cos(pos/100002i/d model)\\nwhere posis the position and iis the dimension. That is, each dimension of the positional encoding\\ncorresponds to a sinusoid. The wavelengths form a geometric progression from 2πto10000 ·2π. We\\nchose this function because we hypothesized it would allow the model to easily learn to attend by\\nrelative positions, since for any fixed offset k,PEpos+kcan be represented as a linear function of\\nPEpos.\\nWe also experimented with using learned positional embeddings [ 9] instead, and found that the two\\nversions produced nearly identical results (see Table 3 row (E)). We chose the sinusoidal version\\nbecause it may allow the model to extrapolate to sequence lengths longer than the ones encountered\\nduring training.\\n4 Why Self-Attention\\nIn this section we compare various aspects of self-attention layers to the recurrent and convolu-\\ntional layers commonly used for mapping one variable-length sequence of symbol representations\\n(x1, ..., x n)to another sequence of equal length (z1, ..., z n), with xi, zi∈Rd, such as a hidden\\nlayer in a typical sequence transduction encoder or decoder. Motivating our use of self-attention we\\nconsider three desiderata.\\nOne is the total computational complexity per layer. Another is the amount of computation that can\\nbe parallelized, as measured by the minimum number of sequential operations required.\\nThe third is the path length between long-range dependencies in the network. Learning long-range\\ndependencies is a key challenge in many sequence transduction tasks. One key factor affecting the\\nability to learn such dependencies is the length of the paths forward and backward signals have to\\ntraverse in the network. The shorter these paths between any combination of positions in the input\\nand output sequences, the easier it is to learn long-range dependencies [ 12]. Hence we also compare\\nthe maximum path length between any two input and output positions in networks composed of the\\ndifferent layer types.\\nAs noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially\\nexecuted operations, whereas a recurrent layer requires O(n)sequential operations. In terms of\\ncomputational complexity, self-attention layers are faster than recurrent layers when the sequence\\n6', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 5}),\n",
       " Document(page_content='length nis smaller than the representation dimensionality d, which is most often the case with\\nsentence representations used by state-of-the-art models in machine translations, such as word-piece\\n[38] and byte-pair [ 31] representations. To improve computational performance for tasks involving\\nvery long sequences, self-attention could be restricted to considering only a neighborhood of size rin\\nthe input sequence centered around the respective output position. This would increase the maximum\\npath length to O(n/r). We plan to investigate this approach further in future work.\\nA single convolutional layer with kernel width k < n does not connect all pairs of input and output\\npositions. Doing so requires a stack of O(n/k)convolutional layers in the case of contiguous kernels,\\norO(logk(n))in the case of dilated convolutions [ 18], increasing the length of the longest paths\\nbetween any two positions in the network. Convolutional layers are generally more expensive than\\nrecurrent layers, by a factor of k. Separable convolutions [ 6], however, decrease the complexity\\nconsiderably, to O(k·n·d+n·d2). Even with k=n, however, the complexity of a separable\\nconvolution is equal to the combination of a self-attention layer and a point-wise feed-forward layer,\\nthe approach we take in our model.\\nAs side benefit, self-attention could yield more interpretable models. We inspect attention distributions\\nfrom our models and present and discuss examples in the appendix. Not only do individual attention\\nheads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic\\nand semantic structure of the sentences.\\n5 Training\\nThis section describes the training regime for our models.\\n5.1 Training Data and Batching\\nWe trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million\\nsentence pairs. Sentences were encoded using byte-pair encoding [ 3], which has a shared source-\\ntarget vocabulary of about 37000 tokens. For English-French, we used the significantly larger WMT\\n2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece\\nvocabulary [ 38]. Sentence pairs were batched together by approximate sequence length. Each training\\nbatch contained a set of sentence pairs containing approximately 25000 source tokens and 25000\\ntarget tokens.\\n5.2 Hardware and Schedule\\nWe trained our models on one machine with 8 NVIDIA P100 GPUs. For our base models using\\nthe hyperparameters described throughout the paper, each training step took about 0.4 seconds. We\\ntrained the base models for a total of 100,000 steps or 12 hours. For our big models,(described on the\\nbottom line of table 3), step time was 1.0 seconds. The big models were trained for 300,000 steps\\n(3.5 days).\\n5.3 Optimizer\\nWe used the Adam optimizer [ 20] with β1= 0.9,β2= 0.98andϵ= 10−9. We varied the learning\\nrate over the course of training, according to the formula:\\nlrate =d−0.5\\nmodel·min(step_num−0.5, step _num·warmup _steps−1.5) (3)\\nThis corresponds to increasing the learning rate linearly for the first warmup _steps training steps,\\nand decreasing it thereafter proportionally to the inverse square root of the step number. We used\\nwarmup _steps = 4000 .\\n5.4 Regularization\\nWe employ three types of regularization during training:\\n7', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 6}),\n",
       " Document(page_content='Table 2: The Transformer achieves better BLEU scores than previous state-of-the-art models on the\\nEnglish-to-German and English-to-French newstest2014 tests at a fraction of the training cost.\\nModelBLEU Training Cost (FLOPs)\\nEN-DE EN-FR EN-DE EN-FR\\nByteNet [18] 23.75\\nDeep-Att + PosUnk [39] 39.2 1.0·1020\\nGNMT + RL [38] 24.6 39.92 2.3·10191.4·1020\\nConvS2S [9] 25.16 40.46 9.6·10181.5·1020\\nMoE [32] 26.03 40.56 2.0·10191.2·1020\\nDeep-Att + PosUnk Ensemble [39] 40.4 8.0·1020\\nGNMT + RL Ensemble [38] 26.30 41.16 1.8·10201.1·1021\\nConvS2S Ensemble [9] 26.36 41.29 7.7·10191.2·1021\\nTransformer (base model) 27.3 38.1 3.3·1018\\nTransformer (big) 28.4 41.8 2.3·1019\\nResidual Dropout We apply dropout [ 33] to the output of each sub-layer, before it is added to the\\nsub-layer input and normalized. In addition, we apply dropout to the sums of the embeddings and the\\npositional encodings in both the encoder and decoder stacks. For the base model, we use a rate of\\nPdrop= 0.1.\\nLabel Smoothing During training, we employed label smoothing of value ϵls= 0.1[36]. This\\nhurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.\\n6 Results\\n6.1 Machine Translation\\nOn the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big)\\nin Table 2) outperforms the best previously reported models (including ensembles) by more than 2.0\\nBLEU, establishing a new state-of-the-art BLEU score of 28.4. The configuration of this model is\\nlisted in the bottom line of Table 3. Training took 3.5days on 8P100 GPUs. Even our base model\\nsurpasses all previously published models and ensembles, at a fraction of the training cost of any of\\nthe competitive models.\\nOn the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41.0,\\noutperforming all of the previously published single models, at less than 1/4the training cost of the\\nprevious state-of-the-art model. The Transformer (big) model trained for English-to-French used\\ndropout rate Pdrop= 0.1, instead of 0.3.\\nFor the base models, we used a single model obtained by averaging the last 5 checkpoints, which\\nwere written at 10-minute intervals. For the big models, we averaged the last 20 checkpoints. We\\nused beam search with a beam size of 4and length penalty α= 0.6[38]. These hyperparameters\\nwere chosen after experimentation on the development set. We set the maximum output length during\\ninference to input length + 50, but terminate early when possible [38].\\nTable 2 summarizes our results and compares our translation quality and training costs to other model\\narchitectures from the literature. We estimate the number of floating point operations used to train a\\nmodel by multiplying the training time, the number of GPUs used, and an estimate of the sustained\\nsingle-precision floating-point capacity of each GPU5.\\n6.2 Model Variations\\nTo evaluate the importance of different components of the Transformer, we varied our base model\\nin different ways, measuring the change in performance on English-to-German translation on the\\n5We used values of 2.8, 3.7, 6.0 and 9.5 TFLOPS for K80, K40, M40 and P100, respectively.\\n8', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 7}),\n",
       " Document(page_content='Table 3: Variations on the Transformer architecture. Unlisted values are identical to those of the base\\nmodel. All metrics are on the English-to-German translation development set, newstest2013. Listed\\nperplexities are per-wordpiece, according to our byte-pair encoding, and should not be compared to\\nper-word perplexities.\\nN d model dff h d k dvPdrop ϵlstrain PPL BLEU params\\nsteps (dev) (dev) ×106\\nbase 6 512 2048 8 64 64 0.1 0.1 100K 4.92 25.8 65\\n(A)1 512 512 5.29 24.9\\n4 128 128 5.00 25.5\\n16 32 32 4.91 25.8\\n32 16 16 5.01 25.4\\n(B)16 5.16 25.1 58\\n32 5.01 25.4 60\\n(C)2 6.11 23.7 36\\n4 5.19 25.3 50\\n8 4.88 25.5 80\\n256 32 32 5.75 24.5 28\\n1024 128 128 4.66 26.0 168\\n1024 5.12 25.4 53\\n4096 4.75 26.2 90\\n(D)0.0 5.77 24.6\\n0.2 4.95 25.5\\n0.0 4.67 25.3\\n0.2 5.47 25.7\\n(E) positional embedding instead of sinusoids 4.92 25.7\\nbig 6 1024 4096 16 0.3 300K 4.33 26.4 213\\ndevelopment set, newstest2013. We used beam search as described in the previous section, but no\\ncheckpoint averaging. We present these results in Table 3.\\nIn Table 3 rows (A), we vary the number of attention heads and the attention key and value dimensions,\\nkeeping the amount of computation constant, as described in Section 3.2.2. While single-head\\nattention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.\\nIn Table 3 rows (B), we observe that reducing the attention key size dkhurts model quality. This\\nsuggests that determining compatibility is not easy and that a more sophisticated compatibility\\nfunction than dot product may be beneficial. We further observe in rows (C) and (D) that, as expected,\\nbigger models are better, and dropout is very helpful in avoiding over-fitting. In row (E) we replace our\\nsinusoidal positional encoding with learned positional embeddings [ 9], and observe nearly identical\\nresults to the base model.\\n6.3 English Constituency Parsing\\nTo evaluate if the Transformer can generalize to other tasks we performed experiments on English\\nconstituency parsing. This task presents specific challenges: the output is subject to strong structural\\nconstraints and is significantly longer than the input. Furthermore, RNN sequence-to-sequence\\nmodels have not been able to attain state-of-the-art results in small-data regimes [37].\\nWe trained a 4-layer transformer with dmodel = 1024 on the Wall Street Journal (WSJ) portion of the\\nPenn Treebank [ 25], about 40K training sentences. We also trained it in a semi-supervised setting,\\nusing the larger high-confidence and BerkleyParser corpora from with approximately 17M sentences\\n[37]. We used a vocabulary of 16K tokens for the WSJ only setting and a vocabulary of 32K tokens\\nfor the semi-supervised setting.\\nWe performed only a small number of experiments to select the dropout, both attention and residual\\n(section 5.4), learning rates and beam size on the Section 22 development set, all other parameters\\nremained unchanged from the English-to-German base translation model. During inference, we\\n9', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 8}),\n",
       " Document(page_content='Table 4: The Transformer generalizes well to English constituency parsing (Results are on Section 23\\nof WSJ)\\nParser Training WSJ 23 F1\\nVinyals & Kaiser el al. (2014) [37] WSJ only, discriminative 88.3\\nPetrov et al. (2006) [29] WSJ only, discriminative 90.4\\nZhu et al. (2013) [40] WSJ only, discriminative 90.4\\nDyer et al. (2016) [8] WSJ only, discriminative 91.7\\nTransformer (4 layers) WSJ only, discriminative 91.3\\nZhu et al. (2013) [40] semi-supervised 91.3\\nHuang & Harper (2009) [14] semi-supervised 91.3\\nMcClosky et al. (2006) [26] semi-supervised 92.1\\nVinyals & Kaiser el al. (2014) [37] semi-supervised 92.1\\nTransformer (4 layers) semi-supervised 92.7\\nLuong et al. (2015) [23] multi-task 93.0\\nDyer et al. (2016) [8] generative 93.3\\nincreased the maximum output length to input length + 300. We used a beam size of 21andα= 0.3\\nfor both WSJ only and the semi-supervised setting.\\nOur results in Table 4 show that despite the lack of task-specific tuning our model performs sur-\\nprisingly well, yielding better results than all previously reported models with the exception of the\\nRecurrent Neural Network Grammar [8].\\nIn contrast to RNN sequence-to-sequence models [ 37], the Transformer outperforms the Berkeley-\\nParser [29] even when training only on the WSJ training set of 40K sentences.\\n7 Conclusion\\nIn this work, we presented the Transformer, the first sequence transduction model based entirely on\\nattention, replacing the recurrent layers most commonly used in encoder-decoder architectures with\\nmulti-headed self-attention.\\nFor translation tasks, the Transformer can be trained significantly faster than architectures based\\non recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014\\nEnglish-to-French translation tasks, we achieve a new state of the art. In the former task our best\\nmodel outperforms even all previously reported ensembles.\\nWe are excited about the future of attention-based models and plan to apply them to other tasks. We\\nplan to extend the Transformer to problems involving input and output modalities other than text and\\nto investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs\\nsuch as images, audio and video. Making generation less sequential is another research goals of ours.\\nThe code we used to train and evaluate our models is available at https://github.com/\\ntensorflow/tensor2tensor .\\nAcknowledgements We are grateful to Nal Kalchbrenner and Stephan Gouws for their fruitful\\ncomments, corrections and inspiration.\\nReferences\\n[1]Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint\\narXiv:1607.06450 , 2016.\\n[2]Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly\\nlearning to align and translate. CoRR , abs/1409.0473, 2014.\\n[3]Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc V . Le. Massive exploration of neural\\nmachine translation architectures. CoRR , abs/1703.03906, 2017.\\n[4]Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine\\nreading. arXiv preprint arXiv:1601.06733 , 2016.\\n10', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 9}),\n",
       " Document(page_content='[5]Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk,\\nand Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical\\nmachine translation. CoRR , abs/1406.1078, 2014.\\n[6]Francois Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv\\npreprint arXiv:1610.02357 , 2016.\\n[7]Junyoung Chung, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation\\nof gated recurrent neural networks on sequence modeling. CoRR , abs/1412.3555, 2014.\\n[8]Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. Recurrent neural\\nnetwork grammars. In Proc. of NAACL , 2016.\\n[9]Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolu-\\ntional sequence to sequence learning. arXiv preprint arXiv:1705.03122v2 , 2017.\\n[10] Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint\\narXiv:1308.0850 , 2013.\\n[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im-\\nage recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern\\nRecognition , pages 770–778, 2016.\\n[12] Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber. Gradient flow in\\nrecurrent nets: the difficulty of learning long-term dependencies, 2001.\\n[13] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation ,\\n9(8):1735–1780, 1997.\\n[14] Zhongqiang Huang and Mary Harper. Self-training PCFG grammars with latent annotations\\nacross languages. In Proceedings of the 2009 Conference on Empirical Methods in Natural\\nLanguage Processing , pages 832–841. ACL, August 2009.\\n[15] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring\\nthe limits of language modeling. arXiv preprint arXiv:1602.02410 , 2016.\\n[16] Łukasz Kaiser and Samy Bengio. Can active memory replace attention? In Advances in Neural\\nInformation Processing Systems, (NIPS) , 2016.\\n[17] Łukasz Kaiser and Ilya Sutskever. Neural GPUs learn algorithms. In International Conference\\non Learning Representations (ICLR) , 2016.\\n[18] Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Ko-\\nray Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099v2 ,\\n2017.\\n[19] Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. Structured attention networks.\\nInInternational Conference on Learning Representations , 2017.\\n[20] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.\\n[21] Oleksii Kuchaiev and Boris Ginsburg. Factorization tricks for LSTM networks. arXiv preprint\\narXiv:1703.10722 , 2017.\\n[22] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen\\nZhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint\\narXiv:1703.03130 , 2017.\\n[23] Minh-Thang Luong, Quoc V . Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task\\nsequence to sequence learning. arXiv preprint arXiv:1511.06114 , 2015.\\n[24] Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-\\nbased neural machine translation. arXiv preprint arXiv:1508.04025 , 2015.\\n11', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 10}),\n",
       " Document(page_content='[25] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated\\ncorpus of english: The penn treebank. Computational linguistics , 19(2):313–330, 1993.\\n[26] David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. In\\nProceedings of the Human Language Technology Conference of the NAACL, Main Conference ,\\npages 152–159. ACL, June 2006.\\n[27] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention\\nmodel. In Empirical Methods in Natural Language Processing , 2016.\\n[28] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive\\nsummarization. arXiv preprint arXiv:1705.04304 , 2017.\\n[29] Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. Learning accurate, compact,\\nand interpretable tree annotation. In Proceedings of the 21st International Conference on\\nComputational Linguistics and 44th Annual Meeting of the ACL , pages 433–440. ACL, July\\n2006.\\n[30] Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv\\npreprint arXiv:1608.05859 , 2016.\\n[31] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words\\nwith subword units. arXiv preprint arXiv:1508.07909 , 2015.\\n[32] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton,\\nand Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts\\nlayer. arXiv preprint arXiv:1701.06538 , 2017.\\n[33] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi-\\nnov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine\\nLearning Research , 15(1):1929–1958, 2014.\\n[34] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory\\nnetworks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors,\\nAdvances in Neural Information Processing Systems 28 , pages 2440–2448. Curran Associates,\\nInc., 2015.\\n[35] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural\\nnetworks. In Advances in Neural Information Processing Systems , pages 3104–3112, 2014.\\n[36] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.\\nRethinking the inception architecture for computer vision. CoRR , abs/1512.00567, 2015.\\n[37] Vinyals & Kaiser, Koo, Petrov, Sutskever, and Hinton. Grammar as a foreign language. In\\nAdvances in Neural Information Processing Systems , 2015.\\n[38] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang\\nMacherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine\\ntranslation system: Bridging the gap between human and machine translation. arXiv preprint\\narXiv:1609.08144 , 2016.\\n[39] Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. Deep recurrent models with\\nfast-forward connections for neural machine translation. CoRR , abs/1606.04199, 2016.\\n[40] Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. Fast and accurate\\nshift-reduce constituent parsing. In Proceedings of the 51st Annual Meeting of the ACL (Volume\\n1: Long Papers) , pages 434–443. ACL, August 2013.\\n12', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 11}),\n",
       " Document(page_content='Attention Visualizations\\nInput-Input Layer5\\nIt\\nis\\nin\\nthis\\nspirit\\nthat\\na\\nmajority\\nof\\nAmerican\\ngovernments\\nhave\\npassed\\nnew\\nlaws\\nsince\\n2009\\nmaking\\nthe\\nregistration\\nor\\nvoting\\nprocess\\nmore\\ndifficult\\n.\\n<EOS>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\nIt\\nis\\nin\\nthis\\nspirit\\nthat\\na\\nmajority\\nof\\nAmerican\\ngovernments\\nhave\\npassed\\nnew\\nlaws\\nsince\\n2009\\nmaking\\nthe\\nregistration\\nor\\nvoting\\nprocess\\nmore\\ndifficult\\n.\\n<EOS>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\nFigure 3: An example of the attention mechanism following long-distance dependencies in the\\nencoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of\\nthe verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for\\nthe word ‘making’. Different colors represent different heads. Best viewed in color.\\n13', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 12}),\n",
       " Document(page_content='Input-Input Layer5\\nThe\\nLaw\\nwill\\nnever\\nbe\\nperfect\\n,\\nbut\\nits\\napplication\\nshould\\nbe\\njust\\n-\\nthis\\nis\\nwhat\\nwe\\nare\\nmissing\\n,\\nin\\nmy\\nopinion\\n.\\n<EOS>\\n<pad>\\nThe\\nLaw\\nwill\\nnever\\nbe\\nperfect\\n,\\nbut\\nits\\napplication\\nshould\\nbe\\njust\\n-\\nthis\\nis\\nwhat\\nwe\\nare\\nmissing\\n,\\nin\\nmy\\nopinion\\n.\\n<EOS>\\n<pad>\\nInput-Input Layer5\\nThe\\nLaw\\nwill\\nnever\\nbe\\nperfect\\n,\\nbut\\nits\\napplication\\nshould\\nbe\\njust\\n-\\nthis\\nis\\nwhat\\nwe\\nare\\nmissing\\n,\\nin\\nmy\\nopinion\\n.\\n<EOS>\\n<pad>\\nThe\\nLaw\\nwill\\nnever\\nbe\\nperfect\\n,\\nbut\\nits\\napplication\\nshould\\nbe\\njust\\n-\\nthis\\nis\\nwhat\\nwe\\nare\\nmissing\\n,\\nin\\nmy\\nopinion\\n.\\n<EOS>\\n<pad>Figure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution. Top:\\nFull attentions for head 5. Bottom: Isolated attentions from just the word ‘its’ for attention heads 5\\nand 6. Note that the attentions are very sharp for this word.\\n14', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 13}),\n",
       " Document(page_content='Input-Input Layer5\\nThe\\nLaw\\nwill\\nnever\\nbe\\nperfect\\n,\\nbut\\nits\\napplication\\nshould\\nbe\\njust\\n-\\nthis\\nis\\nwhat\\nwe\\nare\\nmissing\\n,\\nin\\nmy\\nopinion\\n.\\n<EOS>\\n<pad>\\nThe\\nLaw\\nwill\\nnever\\nbe\\nperfect\\n,\\nbut\\nits\\napplication\\nshould\\nbe\\njust\\n-\\nthis\\nis\\nwhat\\nwe\\nare\\nmissing\\n,\\nin\\nmy\\nopinion\\n.\\n<EOS>\\n<pad>\\nInput-Input Layer5\\nThe\\nLaw\\nwill\\nnever\\nbe\\nperfect\\n,\\nbut\\nits\\napplication\\nshould\\nbe\\njust\\n-\\nthis\\nis\\nwhat\\nwe\\nare\\nmissing\\n,\\nin\\nmy\\nopinion\\n.\\n<EOS>\\n<pad>\\nThe\\nLaw\\nwill\\nnever\\nbe\\nperfect\\n,\\nbut\\nits\\napplication\\nshould\\nbe\\njust\\n-\\nthis\\nis\\nwhat\\nwe\\nare\\nmissing\\n,\\nin\\nmy\\nopinion\\n.\\n<EOS>\\n<pad>Figure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the\\nsentence. We give two such examples above, from two different heads from the encoder self-attention\\nat layer 5 of 6. The heads clearly learned to perform different tasks.\\n15', metadata={'source': 'https://arxiv.org/pdf/1706.03762.pdf', 'page': 14})]"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Figure 1: The Transformer - model architecture.\n",
      "The Transformer follows this overall architecture using stacked self-attention and point-wise, fully\n",
      "connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1,\n",
      "respectively.\n",
      "3.1 Encoder and Decoder Stacks\n",
      "Encoder: The encoder is composed of a stack of N= 6 identical layers. Each layer has two\n",
      "sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-\n",
      "wise fully connected feed-forward network. We employ a residual connection [ 11] around each of\n",
      "the two sub-layers, followed by layer normalization [ 1]. That is, the output of each sub-layer is\n",
      "LayerNorm( x+ Sublayer( x)), where Sublayer( x)is the function implemented by the sub-layer\n",
      "itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding\n",
      "layers, produce outputs of dimension dmodel = 512 .\n",
      "Decoder: The decoder is also composed of a stack of N= 6identical layers. In addition to the two\n",
      "sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head\n",
      "attention over the output of the encoder stack. Similar to the encoder, we employ residual connections\n",
      "around each of the sub-layers, followed by layer normalization. We also modify the self-attention\n",
      "sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This\n",
      "masking, combined with fact that the output embeddings are offset by one position, ensures that the\n",
      "predictions for position ican depend only on the known outputs at positions less than i.\n",
      "3.2 Attention\n",
      "An attention function can be described as mapping a query and a set of key-value pairs to an output,\n",
      "where the query, keys, values, and output are all vectors. The output is computed as a weighted sum\n",
      "3Output\n",
      "Probabilities\n",
      "Softmax\n",
      "Linear\n",
      "Add & Norm\n",
      "Feed\n",
      "Forward\n",
      "Add & Norm\n",
      "Add & Norm\n",
      "Multi-Head\n",
      "Feed\n",
      "Attention\n",
      "Forward\n",
      "Nx\n",
      "Add & Norm\n",
      "Zx\n",
      "Add & Norm\n",
      "Masked\n",
      "Multi-Head\n",
      "Multi-Head\n",
      "Attention\n",
      "Attention\n",
      "Positional\n",
      "Positional\n",
      "Encoding\n",
      "Encoding\n",
      "Input\n",
      "Output\n",
      "Embedding\n",
      "Embedding\n",
      "Inputs\n",
      "Outputs\n",
      "(shifted right)\n"
     ]
    }
   ],
   "source": [
    "print(pages[3].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Document loaders: PDF - using Amazon Textract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import AmazonTextractPDFLoader\n",
    "loader = AmazonTextractPDFLoader(\"example_data/alejandro_rosalez_sample-small.jpeg\")\n",
    "documents = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Text Splitters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "langchain-text-splitters - library from langchain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Transform docs` - split a long document into smaller chunks that can fit into your model's context window.\n",
    "(goal is not to chunk for chunking sake, our goal is to get our data in a format where it can be retrieved for value later).\n",
    "\n",
    "Text Splitters `work` as following:\n",
    "1. Split the text up into small, semantically meaningful chunks (often sentences)\n",
    "2. Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).\n",
    "3. Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap \n",
    "   (to keep context between chunks).\n",
    "\n",
    "Customize your text splitter:\n",
    "1. how the text is split\n",
    "2. how the chunk size is measured\n",
    "\n",
    "\n",
    "`Ref`: https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb\n",
    "\n",
    "`Levels` of text splitting:\n",
    "1. Character splitting\n",
    "2. Recursive Character text splitting\n",
    "3. Document specific splitting\n",
    "4. Semantic splitting\n",
    "5. Agentic splitting\n",
    "6. Alternative Representation chunking + indexing\n",
    "\n",
    "How to `evaluate` text splitters:\n",
    "Chunkvix utility: https://www.chunkviz.com\n",
    "\n",
    "\n",
    "`Interesting` ones:\n",
    "1. Text: \n",
    "    - CharacterTextSplitter\n",
    "    - RecursiveCharacterTextSplitter\n",
    "    - RecursiveJsonSplitter\n",
    "2. Code: PythonCodeTextSplitter\n",
    "3. PDFs with tables\n",
    "4. Multi-modal (text + images)\n",
    "5. Semantic Chunking\n",
    "    - SemanticChunker\n",
    "6. Hypothetical Questions: generate hypothetical questions about raw documents. \n",
    "   Helpful when you have sparse unstructured data, like chat messages.\n",
    "7. Split by tokens: when you split your text into chunks it is therefore a good idea to count the number of tokens. Tokenizers for this. \n",
    "   When you count tokens in your text you should use the same tokenizer as used in the language model.\n",
    "   - SentenceTransformersTokenTextSplitter\n",
    "   - NLTKTextSplitter\n",
    "   - Huggingface's Tokenizer. ex: GPT2TokenizerFast\n",
    "      from transformers import GPT2TokenizerFast\n",
    "      from langchain_text_splitters import CharacterTextSplitter\n",
    "      tokenizer = GPT2TokenizerFast.from_pretrained(\"gpt2\")\n",
    "      text_splitter = CharacterTextSplitter.from_huggingface_tokenizer(\n",
    "         tokenizer, chunk_size=100, chunk_overlap=0\n",
    "      )\n",
    "\n",
    "Graphs:\n",
    "if data is rich with entities, relationships, and connections -> then a graph structure would benefir\n",
    "Options:\n",
    "- Diffbot\n",
    "- InstaGraph\n",
    "\n",
    "Webscraping tools:\n",
    "1. Diffbot - https://python.langchain.com/docs/integrations/document_loaders/diffbot\n",
    "2. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Text Splitters: Semantic chunking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Ignoring invalid distribution -ackaging (/opt/conda/envs/pytorch/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -ackaging (/opt/conda/envs/pytorch/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "! pip install langchain_experimental -qU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n",
      "\n",
      "Last year COVID-19 kept us apart. This year we are finally together again. \n",
      "\n",
      "Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n",
      "\n",
      "With a duty to one another to the American people to the Constitution. \n",
      "\n",
      "And with an unwavering resolve that freedom will always triumph over tyranny. \n",
      "\n",
      "Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n",
      "\n",
      "He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n",
      "\n",
      "He met the Ukrainian people. \n",
      "\n",
      "From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. \n",
      "\n",
      "Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland. \n",
      "\n",
      "In this struggle as President Zelenskyy said in his speech to the European Parliament “Light will win over darkness.” The Ukrainian Ambassador to the United States is here tonight. \n",
      "\n",
      "Let each of us here tonight in this Chamber send an unmistakable signal to Ukraine and to the world. \n",
      "\n",
      "Please rise if you are able and show that, Yes, we the United States of America stand with the Ukrainian people. \n",
      "\n",
      "Throughout our history we’ve learned this lesson when dictators do not pay a price for their aggression they cause more chaos.   \n",
      "\n",
      "They keep moving.   \n",
      "\n",
      "And the costs and the threats to America and the world keep rising.   \n",
      "\n",
      "That’s why the NATO Alliance was created to secure peace and stability in Europe after World War 2. \n",
      "\n",
      "The United States is a member along with 29 other nations. \n",
      "\n",
      "It matters. American diplomacy matters. American resolve matters. \n",
      "\n",
      "Putin’s latest attack on Ukraine was premeditated and unprovoked. \n",
      "\n",
      "He rejected repeated efforts at diplomacy. \n",
      "\n",
      "He thought the West and NATO wouldn’t respond. And he thought he could divide us at home. Putin was wrong. We were ready.  Here is what we did.   \n",
      "\n",
      "We prepared extensively and carefully. \n",
      "\n",
      "We spent months building a coalition of other freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin. \n",
      "\n",
      "I spent countless hours unifying our European allies. We shared with the world in advance what we knew Putin was planning and precisely how he would try to falsely justify his aggression.  \n",
      "\n",
      "We countered Russia’s lies with truth.   \n",
      "\n",
      "And now that he has acted the free world is holding him accountable. \n",
      "\n",
      "Along with twenty-seven members of the European Union including France, Germany, Italy, as well as countries like the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland. \n",
      "\n",
      "We are inflicting pain on Russia and supporting the people of Ukraine. Putin is now isolated from the world more than ever. \n",
      "\n",
      "Together with our allies –we are right now enforcing powerful economic sanctions. \n",
      "\n",
      "We are cutting off Russia’s largest banks from the international financial system.  \n",
      "\n",
      "Preventing Russia’s central bank from defending the Russian Ruble making Putin’s $630 Billion “war fund” worthless.   \n",
      "\n",
      "We are choking off Russia’s access to technology that will sap its economic strength and weaken its military for years to come.  \n",
      "\n",
      "Tonight I say to the Russian oligarchs and corrupt leaders who have bilked billions of dollars off this violent regime no more. \n",
      "\n",
      "The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs.  \n",
      "\n",
      "We are joining with our European allies to find and seize your yachts your luxury apartments your private jets. We are coming for your ill-begotten gains. \n",
      "\n",
      "And tonight I am announcing that we will join our allies in closing off American air space to all Russian flights – further isolating Russia – and adding an additional squeeze –on their economy. The Ruble has lost 30% of its value. \n",
      "\n",
      "The Russian stock market has lost 40% of its value and trading remains suspended. Russia’s economy is reeling and Putin alone is to blame. \n",
      "\n",
      "Together with our allies we are providing support to the Ukrainians in their fight for freedom. Military assistance. Economic assistance. Humanitarian assistance. \n",
      "\n",
      "We are giving more than $1 Billion in direct assistance to Ukraine. \n",
      "\n",
      "And we will continue to aid the Ukrainian people as they defend their country and to help ease their suffering.  \n",
      "\n",
      "Let me be clear, our forces are not engaged and will not engage in conflict with Russian forces in Ukraine.  \n",
      "\n",
      "Our forces are not going to Europe to fight in Ukraine, but to defend our NATO Allies – in the event that Putin decides to keep moving west.  \n",
      "\n",
      "For that purpose we’ve mobilized American ground forces, air squadrons, and ship deployments to protect NATO countries including Poland, Romania, Latvia, Lithuania, and Estonia. \n",
      "\n",
      "As I have made crystal clear the United States and our Allies will defend every inch of territory of NATO countries with the full force of our collective power.  \n",
      "\n",
      "And we remain clear-eyed. The Ukrainians are fighting back with pure courage. But the next few days weeks, months, will be hard on them.  \n",
      "\n",
      "Putin has unleashed violence and chaos.  But while he may make gains on the battlefield – he will pay a continuing high price over the long run. \n",
      "\n",
      "And a proud Ukrainian people, who have known 30 years  of independence, have repeatedly shown that they will not tolerate anyone who tries to take their country backwards.  \n",
      "\n",
      "To all Americans, I will be honest with you, as I’ve always promised. A Russian dictator, invading a foreign country, has costs around the world. \n",
      "\n",
      "And I’m taking robust action to make sure the pain of our sanctions  is targeted at Russia’s economy. And I will use every tool at our disposal to protect American businesses and consumers. \n",
      "\n",
      "Tonight, I can announce that the United States has worked with 30 other countries to release 60 Million barrels of oil from reserves around the world.  \n",
      "\n",
      "America will lead that effort, releasing 30 Million barrels from our own Strategic Petroleum Reserve. And we stand ready to do more if necessary, unified with our allies.  \n",
      "\n",
      "These steps will help blunt gas prices here at home. And I know the news about what’s happening can seem alarming. \n",
      "\n",
      "But I want you to know that we are going to be okay. \n",
      "\n",
      "When the history of this era is written Putin’s war on Ukraine will have left Russia weaker and the rest of the world stronger. \n",
      "\n",
      "While it shouldn’t have taken something so terrible for people around the world to see what’s at stake now everyone sees it clearly. \n",
      "\n",
      "We see the unity among leaders of nations and a more unified Europe a more unified West. And we see unity among the people who are gathering in cities in large crowds around the world even in Russia to demonstrate their support for Ukraine.  \n",
      "\n",
      "In the battle between democracy and autocracy, democracies are rising to the moment, and the world is clearly choosing the side of peace and security. \n",
      "\n",
      "This is a real test. It’s going to take time. So let us continue to draw inspiration from the iron will of the Ukrainian people. \n",
      "\n",
      "To our fellow Ukrainian Americans who forge a deep bond that connects our two nations we stand with you. \n",
      "\n",
      "Putin may circle Kyiv with tanks, but he will never gain the hearts and souls of the Ukrainian people. \n",
      "\n",
      "He will never extinguish their love of freedom. He will never weaken the resolve of the free world. \n",
      "\n",
      "We meet tonight in an America that has lived through two of the hardest years this nation has ever faced. \n",
      "\n",
      "The pandemic has been punishing. \n",
      "\n",
      "And so many families are living paycheck to paycheck, struggling to keep up with the rising cost of food, gas, housing, and so much more. \n",
      "\n",
      "I understand. \n",
      "\n",
      "I remember when my Dad had to leave our home in Scranton, Pennsylvania to find work. I grew up in a family where if the price of food went up, you felt it. \n",
      "\n",
      "That’s why one of the first things I did as President was fight to pass the American Rescue Plan.  \n",
      "\n",
      "Because people were hurting. We needed to act, and we did. \n",
      "\n",
      "Few pieces of legislation have done more in a critical moment in our history to lift us out of crisis. \n",
      "\n",
      "It fueled our efforts to vaccinate the nation and combat COVID-19. It delivered immediate economic relief for tens of millions of Americans.  \n",
      "\n",
      "Helped put food on their table, keep a roof over their heads, and cut the cost of health insurance. \n",
      "\n",
      "And as my Dad used to say, it gave people a little breathing room. \n",
      "\n",
      "And unlike the $2 Trillion tax cut passed in the previous administration that benefitted the top 1% of Americans, the American Rescue Plan helped working people—and left no one behind. \n",
      "\n",
      "And it worked. It created jobs. Lots of jobs. \n",
      "\n",
      "In fact—our economy created over 6.5 Million new jobs just last year, more jobs created in one year  \n",
      "than ever before in the history of America. \n",
      "\n",
      "Our economy grew at a rate of 5.7% last year, the strongest growth in nearly 40 years, the first step in bringing fundamental change to an economy that hasn’t worked for the working people of this nation for too long.  \n",
      "\n",
      "For the past 40 years we were told that if we gave tax breaks to those at the very top, the benefits would trickle down to everyone else. \n",
      "\n",
      "But that trickle-down theory led to weaker economic growth, lower wages, bigger deficits, and the widest gap between those at the top and everyone else in nearly a century. \n",
      "\n",
      "Vice President Harris and I ran for office with a new economic vision for America. \n",
      "\n",
      "Invest in America. Educate Americans. Grow the workforce. Build the economy from the bottom up  \n",
      "and the middle out, not from the top down.  \n",
      "\n",
      "Because we know that when the middle class grows, the poor have a ladder up and the wealthy do very well. \n",
      "\n",
      "America used to have the best roads, bridges, and airports on Earth. \n",
      "\n",
      "Now our infrastructure is ranked 13th in the world. \n",
      "\n",
      "We won’t be able to compete for the jobs of the 21st Century if we don’t fix that. \n",
      "\n",
      "That’s why it was so important to pass the Bipartisan Infrastructure Law—the most sweeping investment to rebuild America in history. \n",
      "\n",
      "This was a bipartisan effort, and I want to thank the members of both parties who worked to make it happen. \n",
      "\n",
      "We’re done talking about infrastructure weeks. \n",
      "\n",
      "We’re going to have an infrastructure decade. \n",
      "\n",
      "It is going to transform America and put us on a path to win the economic competition of the 21st Century that we face with the rest of the world—particularly with China.  \n",
      "\n",
      "As I’ve told Xi Jinping, it is never a good bet to bet against the American people. \n",
      "\n",
      "We’ll create good jobs for millions of Americans, modernizing roads, airports, ports, and waterways all across America. \n",
      "\n",
      "And we’ll do it all to withstand the devastating effects of the climate crisis and promote environmental justice. \n",
      "\n",
      "We’ll build a national network of 500,000 electric vehicle charging stations, begin to replace poisonous lead pipes—so every child—and every American—has clean water to drink at home and at school, provide affordable high-speed internet for every American—urban, suburban, rural, and tribal communities. \n",
      "\n",
      "4,000 projects have already been announced. \n",
      "\n",
      "And tonight, I’m announcing that this year we will start fixing over 65,000 miles of highway and 1,500 bridges in disrepair. \n",
      "\n",
      "When we use taxpayer dollars to rebuild America – we are going to Buy American: buy American products to support American jobs. \n",
      "\n",
      "The federal government spends about $600 Billion a year to keep the country safe and secure. \n",
      "\n",
      "There’s been a law on the books for almost a century \n",
      "to make sure taxpayers’ dollars support American jobs and businesses. \n",
      "\n",
      "Every Administration says they’ll do it, but we are actually doing it. \n",
      "\n",
      "We will buy American to make sure everything from the deck of an aircraft carrier to the steel on highway guardrails are made in America. \n",
      "\n",
      "But to compete for the best jobs of the future, we also need to level the playing field with China and other competitors. \n",
      "\n",
      "That’s why it is so important to pass the Bipartisan Innovation Act sitting in Congress that will make record investments in emerging technologies and American manufacturing. \n",
      "\n",
      "Let me give you one example of why it’s so important to pass it. \n",
      "\n",
      "If you travel 20 miles east of Columbus, Ohio, you’ll find 1,000 empty acres of land. \n",
      "\n",
      "It won’t look like much, but if you stop and look closely, you’ll see a “Field of dreams,” the ground on which America’s future will be built. \n",
      "\n",
      "This is where Intel, the American company that helped build Silicon Valley, is going to build its $20 billion semiconductor “mega site”. \n",
      "\n",
      "Up to eight state-of-the-art factories in one place. 10,000 new good-paying jobs. \n",
      "\n",
      "Some of the most sophisticated manufacturing in the world to make computer chips the size of a fingertip that power the world and our everyday lives. \n",
      "\n",
      "Smartphones. The Internet. Technology we have yet to invent. \n",
      "\n",
      "But that’s just the beginning. \n",
      "\n",
      "Intel’s CEO, Pat Gelsinger, who is here tonight, told me they are ready to increase their investment from  \n",
      "$20 billion to $100 billion. \n",
      "\n",
      "That would be one of the biggest investments in manufacturing in American history. \n",
      "\n",
      "And all they’re waiting for is for you to pass this bill. \n",
      "\n",
      "So let’s not wait any longer. Send it to my desk. I’ll sign it.  \n",
      "\n",
      "And we will really take off. \n",
      "\n",
      "And Intel is not alone. \n",
      "\n",
      "There’s something happening in America. \n",
      "\n",
      "Just look around and you’ll see an amazing story. \n",
      "\n",
      "The rebirth of the pride that comes from stamping products “Made In America.” The revitalization of American manufacturing.   \n",
      "\n",
      "Companies are choosing to build new factories here, when just a few years ago, they would have built them overseas. \n",
      "\n",
      "That’s what is happening. Ford is investing $11 billion to build electric vehicles, creating 11,000 jobs across the country. \n",
      "\n",
      "GM is making the largest investment in its history—$7 billion to build electric vehicles, creating 4,000 jobs in Michigan. \n",
      "\n",
      "All told, we created 369,000 new manufacturing jobs in America just last year. \n",
      "\n",
      "Powered by people I’ve met like JoJo Burgess, from generations of union steelworkers from Pittsburgh, who’s here with us tonight. \n",
      "\n",
      "As Ohio Senator Sherrod Brown says, “It’s time to bury the label “Rust Belt.” \n",
      "\n",
      "It’s time. \n",
      "\n",
      "But with all the bright spots in our economy, record job growth and higher wages, too many families are struggling to keep up with the bills.  \n",
      "\n",
      "Inflation is robbing them of the gains they might otherwise feel. \n",
      "\n",
      "I get it. That’s why my top priority is getting prices under control. \n",
      "\n",
      "Look, our economy roared back faster than most predicted, but the pandemic meant that businesses had a hard time hiring enough workers to keep up production in their factories. \n",
      "\n",
      "The pandemic also disrupted global supply chains. \n",
      "\n",
      "When factories close, it takes longer to make goods and get them from the warehouse to the store, and prices go up. \n",
      "\n",
      "Look at cars. \n",
      "\n",
      "Last year, there weren’t enough semiconductors to make all the cars that people wanted to buy. \n",
      "\n",
      "And guess what, prices of automobiles went up. \n",
      "\n",
      "So—we have a choice. \n",
      "\n",
      "One way to fight inflation is to drive down wages and make Americans poorer.  \n",
      "\n",
      "I have a better plan to fight inflation. \n",
      "\n",
      "Lower your costs, not your wages. \n",
      "\n",
      "Make more cars and semiconductors in America. \n",
      "\n",
      "More infrastructure and innovation in America. \n",
      "\n",
      "More goods moving faster and cheaper in America. \n",
      "\n",
      "More jobs where you can earn a good living in America. \n",
      "\n",
      "And instead of relying on foreign supply chains, let’s make it in America. \n",
      "\n",
      "Economists call it “increasing the productive capacity of our economy.” \n",
      "\n",
      "I call it building a better America. \n",
      "\n",
      "My plan to fight inflation will lower your costs and lower the deficit. \n",
      "\n",
      "17 Nobel laureates in economics say my plan will ease long-term inflationary pressures. Top business leaders and most Americans support my plan. And here’s the plan: \n",
      "\n",
      "First – cut the cost of prescription drugs. Just look at insulin. One in ten Americans has diabetes. In Virginia, I met a 13-year-old boy named Joshua Davis.  \n",
      "\n",
      "He and his Dad both have Type 1 diabetes, which means they need insulin every day. Insulin costs about $10 a vial to make.  \n",
      "\n",
      "But drug companies charge families like Joshua and his Dad up to 30 times more. I spoke with Joshua’s mom. \n",
      "\n",
      "Imagine what it’s like to look at your child who needs insulin and have no idea how you’re going to pay for it.  \n",
      "\n",
      "What it does to your dignity, your ability to look your child in the eye, to be the parent you expect to be. \n",
      "\n",
      "Joshua is here with us tonight. Yesterday was his birthday. Happy birthday, buddy.  \n",
      "\n",
      "For Joshua, and for the 200,000 other young people with Type 1 diabetes, let’s cap the cost of insulin at $35 a month so everyone can afford it.  \n",
      "\n",
      "Drug companies will still do very well. And while we’re at it let Medicare negotiate lower prices for prescription drugs, like the VA already does. \n",
      "\n",
      "Look, the American Rescue Plan is helping millions of families on Affordable Care Act plans save $2,400 a year on their health care premiums. Let’s close the coverage gap and make those savings permanent. \n",
      "\n",
      "Second – cut energy costs for families an average of $500 a year by combatting climate change.  \n",
      "\n",
      "Let’s provide investments and tax credits to weatherize your homes and businesses to be energy efficient and you get a tax credit; double America’s clean energy production in solar, wind, and so much more;  lower the price of electric vehicles, saving you another $80 a month because you’ll never have to pay at the gas pump again. \n",
      "\n",
      "Third – cut the cost of child care. Many families pay up to $14,000 a year for child care per child.  \n",
      "\n",
      "Middle-class and working families shouldn’t have to pay more than 7% of their income for care of young children.  \n",
      "\n",
      "My plan will cut the cost in half for most families and help parents, including millions of women, who left the workforce during the pandemic because they couldn’t afford child care, to be able to get back to work. \n",
      "\n",
      "My plan doesn’t stop there. It also includes home and long-term care. More affordable housing. And Pre-K for every 3- and 4-year-old.  \n",
      "\n",
      "All of these will lower costs. \n",
      "\n",
      "And under my plan, nobody earning less than $400,000 a year will pay an additional penny in new taxes. Nobody.  \n",
      "\n",
      "The one thing all Americans agree on is that the tax system is not fair. We have to fix it.  \n",
      "\n",
      "I’m not looking to punish anyone. But let’s make sure corporations and the wealthiest Americans start paying their fair share. \n",
      "\n",
      "Just last year, 55 Fortune 500 corporations earned $40 billion in profits and paid zero dollars in federal income tax.  \n",
      "\n",
      "That’s simply not fair. That’s why I’ve proposed a 15% minimum tax rate for corporations. \n",
      "\n",
      "We got more than 130 countries to agree on a global minimum tax rate so companies can’t get out of paying their taxes at home by shipping jobs and factories overseas. \n",
      "\n",
      "That’s why I’ve proposed closing loopholes so the very wealthy don’t pay a lower tax rate than a teacher or a firefighter.  \n",
      "\n",
      "So that’s my plan. It will grow the economy and lower costs for families. \n",
      "\n",
      "So what are we waiting for? Let’s get this done. And while you’re at it, confirm my nominees to the Federal Reserve, which plays a critical role in fighting inflation.  \n",
      "\n",
      "My plan will not only lower costs to give families a fair shot, it will lower the deficit. \n",
      "\n",
      "The previous Administration not only ballooned the deficit with tax cuts for the very wealthy and corporations, it undermined the watchdogs whose job was to keep pandemic relief funds from being wasted. \n",
      "\n",
      "But in my administration, the watchdogs have been welcomed back. \n",
      "\n",
      "We’re going after the criminals who stole billions in relief money meant for small businesses and millions of Americans.  \n",
      "\n",
      "And tonight, I’m announcing that the Justice Department will name a chief prosecutor for pandemic fraud. \n",
      "\n",
      "By the end of this year, the deficit will be down to less than half what it was before I took office.  \n",
      "\n",
      "The only president ever to cut the deficit by more than one trillion dollars in a single year. \n",
      "\n",
      "Lowering your costs also means demanding more competition. \n",
      "\n",
      "I’m a capitalist, but capitalism without competition isn’t capitalism. \n",
      "\n",
      "It’s exploitation—and it drives up prices. \n",
      "\n",
      "When corporations don’t have to compete, their profits go up, your prices go up, and small businesses and family farmers and ranchers go under. \n",
      "\n",
      "We see it happening with ocean carriers moving goods in and out of America. \n",
      "\n",
      "During the pandemic, these foreign-owned companies raised prices by as much as 1,000% and made record profits. \n",
      "\n",
      "Tonight, I’m announcing a crackdown on these companies overcharging American businesses and consumers. \n",
      "\n",
      "And as Wall Street firms take over more nursing homes, quality in those homes has gone down and costs have gone up.  \n",
      "\n",
      "That ends on my watch. \n",
      "\n",
      "Medicare is going to set higher standards for nursing homes and make sure your loved ones get the care they deserve and expect. \n",
      "\n",
      "We’ll also cut costs and keep the economy going strong by giving workers a fair shot, provide more training and apprenticeships, hire them based on their skills not degrees. \n",
      "\n",
      "Let’s pass the Paycheck Fairness Act and paid leave.  \n",
      "\n",
      "Raise the minimum wage to $15 an hour and extend the Child Tax Credit, so no one has to raise a family in poverty. \n",
      "\n",
      "Let’s increase Pell Grants and increase our historic support of HBCUs, and invest in what Jill—our First Lady who teaches full-time—calls America’s best-kept secret: community colleges. \n",
      "\n",
      "And let’s pass the PRO Act when a majority of workers want to form a union—they shouldn’t be stopped.  \n",
      "\n",
      "When we invest in our workers, when we build the economy from the bottom up and the middle out together, we can do something we haven’t done in a long time: build a better America. \n",
      "\n",
      "For more than two years, COVID-19 has impacted every decision in our lives and the life of the nation. \n",
      "\n",
      "And I know you’re tired, frustrated, and exhausted. \n",
      "\n",
      "But I also know this. \n",
      "\n",
      "Because of the progress we’ve made, because of your resilience and the tools we have, tonight I can say  \n",
      "we are moving forward safely, back to more normal routines.  \n",
      "\n",
      "We’ve reached a new moment in the fight against COVID-19, with severe cases down to a level not seen since last July.  \n",
      "\n",
      "Just a few days ago, the Centers for Disease Control and Prevention—the CDC—issued new mask guidelines. \n",
      "\n",
      "Under these new guidelines, most Americans in most of the country can now be mask free.   \n",
      "\n",
      "And based on the projections, more of the country will reach that point across the next couple of weeks. \n",
      "\n",
      "Thanks to the progress we have made this past year, COVID-19 need no longer control our lives.  \n",
      "\n",
      "I know some are talking about “living with COVID-19”. Tonight – I say that we will never just accept living with COVID-19. \n",
      "\n",
      "We will continue to combat the virus as we do other diseases. And because this is a virus that mutates and spreads, we will stay on guard. \n",
      "\n",
      "Here are four common sense steps as we move forward safely.  \n",
      "\n",
      "First, stay protected with vaccines and treatments. We know how incredibly effective vaccines are. If you’re vaccinated and boosted you have the highest degree of protection. \n",
      "\n",
      "We will never give up on vaccinating more Americans. Now, I know parents with kids under 5 are eager to see a vaccine authorized for their children. \n",
      "\n",
      "The scientists are working hard to get that done and we’ll be ready with plenty of vaccines when they do. \n",
      "\n",
      "We’re also ready with anti-viral treatments. If you get COVID-19, the Pfizer pill reduces your chances of ending up in the hospital by 90%.  \n",
      "\n",
      "We’ve ordered more of these pills than anyone in the world. And Pfizer is working overtime to get us 1 Million pills this month and more than double that next month.  \n",
      "\n",
      "And we’re launching the “Test to Treat” initiative so people can get tested at a pharmacy, and if they’re positive, receive antiviral pills on the spot at no cost.  \n",
      "\n",
      "If you’re immunocompromised or have some other vulnerability, we have treatments and free high-quality masks. \n",
      "\n",
      "We’re leaving no one behind or ignoring anyone’s needs as we move forward. \n",
      "\n",
      "And on testing, we have made hundreds of millions of tests available for you to order for free.   \n",
      "\n",
      "Even if you already ordered free tests tonight, I am announcing that you can order more from covidtests.gov starting next week. \n",
      "\n",
      "Second – we must prepare for new variants. Over the past year, we’ve gotten much better at detecting new variants. \n",
      "\n",
      "If necessary, we’ll be able to deploy new vaccines within 100 days instead of many more months or years.  \n",
      "\n",
      "And, if Congress provides the funds we need, we’ll have new stockpiles of tests, masks, and pills ready if needed. \n",
      "\n",
      "I cannot promise a new variant won’t come. But I can promise you we’ll do everything within our power to be ready if it does.  \n",
      "\n",
      "Third – we can end the shutdown of schools and businesses. We have the tools we need. \n",
      "\n",
      "It’s time for Americans to get back to work and fill our great downtowns again.  People working from home can feel safe to begin to return to the office.   \n",
      "\n",
      "We’re doing that here in the federal government. The vast majority of federal workers will once again work in person. \n",
      "\n",
      "Our schools are open. Let’s keep it that way. Our kids need to be in school. \n",
      "\n",
      "And with 75% of adult Americans fully vaccinated and hospitalizations down by 77%, most Americans can remove their masks, return to work, stay in the classroom, and move forward safely. \n",
      "\n",
      "We achieved this because we provided free vaccines, treatments, tests, and masks. \n",
      "\n",
      "Of course, continuing this costs money. \n",
      "\n",
      "I will soon send Congress a request. \n",
      "\n",
      "The vast majority of Americans have used these tools and may want to again, so I expect Congress to pass it quickly.   \n",
      "\n",
      "Fourth, we will continue vaccinating the world.     \n",
      "\n",
      "We’ve sent 475 Million vaccine doses to 112 countries, more than any other nation. \n",
      "\n",
      "And we won’t stop. \n",
      "\n",
      "We have lost so much to COVID-19. Time with one another. And worst of all, so much loss of life. \n",
      "\n",
      "Let’s use this moment to reset. Let’s stop looking at COVID-19 as a partisan dividing line and see it for what it is: A God-awful disease.  \n",
      "\n",
      "Let’s stop seeing each other as enemies, and start seeing each other for who we really are: Fellow Americans.  \n",
      "\n",
      "We can’t change how divided we’ve been. But we can change how we move forward—on COVID-19 and other issues we must face together. \n",
      "\n",
      "I recently visited the New York City Police Department days after the funerals of Officer Wilbert Mora and his partner, Officer Jason Rivera. \n",
      "\n",
      "They were responding to a 9-1-1 call when a man shot and killed them with a stolen gun. \n",
      "\n",
      "Officer Mora was 27 years old. \n",
      "\n",
      "Officer Rivera was 22. \n",
      "\n",
      "Both Dominican Americans who’d grown up on the same streets they later chose to patrol as police officers. \n",
      "\n",
      "I spoke with their families and told them that we are forever in debt for their sacrifice, and we will carry on their mission to restore the trust and safety every community deserves. \n",
      "\n",
      "I’ve worked on these issues a long time. \n",
      "\n",
      "I know what works: Investing in crime prevention and community police officers who’ll walk the beat, who’ll know the neighborhood, and who can restore trust and safety. \n",
      "\n",
      "So let’s not abandon our streets. Or choose between safety and equal justice. \n",
      "\n",
      "Let’s come together to protect our communities, restore trust, and hold law enforcement accountable. \n",
      "\n",
      "That’s why the Justice Department required body cameras, banned chokeholds, and restricted no-knock warrants for its officers. \n",
      "\n",
      "That’s why the American Rescue Plan provided $350 Billion that cities, states, and counties can use to hire more police and invest in proven strategies like community violence interruption—trusted messengers breaking the cycle of violence and trauma and giving young people hope.  \n",
      "\n",
      "We should all agree: The answer is not to Defund the police. The answer is to FUND the police with the resources and training they need to protect our communities. \n",
      "\n",
      "I ask Democrats and Republicans alike: Pass my budget and keep our neighborhoods safe.  \n",
      "\n",
      "And I will keep doing everything in my power to crack down on gun trafficking and ghost guns you can buy online and make at home—they have no serial numbers and can’t be traced. \n",
      "\n",
      "And I ask Congress to pass proven measures to reduce gun violence. Pass universal background checks. Why should anyone on a terrorist list be able to purchase a weapon? \n",
      "\n",
      "Ban assault weapons and high-capacity magazines. \n",
      "\n",
      "Repeal the liability shield that makes gun manufacturers the only industry in America that can’t be sued. \n",
      "\n",
      "These laws don’t infringe on the Second Amendment. They save lives. \n",
      "\n",
      "The most fundamental right in America is the right to vote – and to have it counted. And it’s under assault. \n",
      "\n",
      "In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \n",
      "\n",
      "We cannot let this happen. \n",
      "\n",
      "Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
      "\n",
      "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
      "\n",
      "One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
      "\n",
      "And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence. \n",
      "\n",
      "A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. \n",
      "\n",
      "And if we are to advance liberty and justice, we need to secure the Border and fix the immigration system. \n",
      "\n",
      "We can do both. At our border, we’ve installed new technology like cutting-edge scanners to better detect drug smuggling.  \n",
      "\n",
      "We’ve set up joint patrols with Mexico and Guatemala to catch more human traffickers.  \n",
      "\n",
      "We’re putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster. \n",
      "\n",
      "We’re securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders. \n",
      "\n",
      "We can do all this while keeping lit the torch of liberty that has led generations of immigrants to this land—my forefathers and so many of yours. \n",
      "\n",
      "Provide a pathway to citizenship for Dreamers, those on temporary status, farm workers, and essential workers. \n",
      "\n",
      "Revise our laws so businesses have the workers they need and families don’t wait decades to reunite. \n",
      "\n",
      "It’s not only the right thing to do—it’s the economically smart thing to do. \n",
      "\n",
      "That’s why immigration reform is supported by everyone from labor unions to religious leaders to the U.S. Chamber of Commerce. \n",
      "\n",
      "Let’s get it done once and for all. \n",
      "\n",
      "Advancing liberty and justice also requires protecting the rights of women. \n",
      "\n",
      "The constitutional right affirmed in Roe v. Wade—standing precedent for half a century—is under attack as never before. \n",
      "\n",
      "If we want to go forward—not backward—we must protect access to health care. Preserve a woman’s right to choose. And let’s continue to advance maternal health care in America. \n",
      "\n",
      "And for our LGBTQ+ Americans, let’s finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong. \n",
      "\n",
      "As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential. \n",
      "\n",
      "While it often appears that we never agree, that isn’t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice. \n",
      "\n",
      "And soon, we’ll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things. \n",
      "\n",
      "So tonight I’m offering a Unity Agenda for the Nation. Four big things we can do together.  \n",
      "\n",
      "First, beat the opioid epidemic. \n",
      "\n",
      "There is so much we can do. Increase funding for prevention, treatment, harm reduction, and recovery.  \n",
      "\n",
      "Get rid of outdated rules that stop doctors from prescribing treatments. And stop the flow of illicit drugs by working with state and local law enforcement to go after traffickers. \n",
      "\n",
      "If you’re suffering from addiction, know you are not alone. I believe in recovery, and I celebrate the 23 million Americans in recovery. \n",
      "\n",
      "Second, let’s take on mental health. Especially among our children, whose lives and education have been turned upside down.  \n",
      "\n",
      "The American Rescue Plan gave schools money to hire teachers and help students make up for lost learning.  \n",
      "\n",
      "I urge every parent to make sure your school does just that. And we can all play a part—sign up to be a tutor or a mentor. \n",
      "\n",
      "Children were also struggling before the pandemic. Bullying, violence, trauma, and the harms of social media. \n",
      "\n",
      "As Frances Haugen, who is here with us tonight, has shown, we must hold social media platforms accountable for the national experiment they’re conducting on our children for profit. \n",
      "\n",
      "It’s time to strengthen privacy protections, ban targeted advertising to children, demand tech companies stop collecting personal data on our children. \n",
      "\n",
      "And let’s get all Americans the mental health services they need. More people they can turn to for help, and full parity between physical and mental health care. \n",
      "\n",
      "Third, support our veterans. \n",
      "\n",
      "Veterans are the best of us. \n",
      "\n",
      "I’ve always believed that we have a sacred obligation to equip all those we send to war and care for them and their families when they come home. \n",
      "\n",
      "My administration is providing assistance with job training and housing, and now helping lower-income veterans get VA care debt-free.  \n",
      "\n",
      "Our troops in Iraq and Afghanistan faced many dangers. \n",
      "\n",
      "One was stationed at bases and breathing in toxic smoke from “burn pits” that incinerated wastes of war—medical and hazard material, jet fuel, and more. \n",
      "\n",
      "When they came home, many of the world’s fittest and best trained warriors were never the same. \n",
      "\n",
      "Headaches. Numbness. Dizziness. \n",
      "\n",
      "A cancer that would put them in a flag-draped coffin. \n",
      "\n",
      "I know. \n",
      "\n",
      "One of those soldiers was my son Major Beau Biden. \n",
      "\n",
      "We don’t know for sure if a burn pit was the cause of his brain cancer, or the diseases of so many of our troops. \n",
      "\n",
      "But I’m committed to finding out everything we can. \n",
      "\n",
      "Committed to military families like Danielle Robinson from Ohio. \n",
      "\n",
      "The widow of Sergeant First Class Heath Robinson.  \n",
      "\n",
      "He was born a soldier. Army National Guard. Combat medic in Kosovo and Iraq. \n",
      "\n",
      "Stationed near Baghdad, just yards from burn pits the size of football fields. \n",
      "\n",
      "Heath’s widow Danielle is here with us tonight. They loved going to Ohio State football games. He loved building Legos with their daughter. \n",
      "\n",
      "But cancer from prolonged exposure to burn pits ravaged Heath’s lungs and body. \n",
      "\n",
      "Danielle says Heath was a fighter to the very end. \n",
      "\n",
      "He didn’t know how to stop fighting, and neither did she. \n",
      "\n",
      "Through her pain she found purpose to demand we do better. \n",
      "\n",
      "Tonight, Danielle—we are. \n",
      "\n",
      "The VA is pioneering new ways of linking toxic exposures to diseases, already helping more veterans get benefits. \n",
      "\n",
      "And tonight, I’m announcing we’re expanding eligibility to veterans suffering from nine respiratory cancers. \n",
      "\n",
      "I’m also calling on Congress: pass a law to make sure veterans devastated by toxic exposures in Iraq and Afghanistan finally get the benefits and comprehensive health care they deserve. \n",
      "\n",
      "And fourth, let’s end cancer as we know it. \n",
      "\n",
      "This is personal to me and Jill, to Kamala, and to so many of you. \n",
      "\n",
      "Cancer is the #2 cause of death in America–second only to heart disease. \n",
      "\n",
      "Last month, I announced our plan to supercharge  \n",
      "the Cancer Moonshot that President Obama asked me to lead six years ago. \n",
      "\n",
      "Our goal is to cut the cancer death rate by at least 50% over the next 25 years, turn more cancers from death sentences into treatable diseases.  \n",
      "\n",
      "More support for patients and families. \n",
      "\n",
      "To get there, I call on Congress to fund ARPA-H, the Advanced Research Projects Agency for Health. \n",
      "\n",
      "It’s based on DARPA—the Defense Department project that led to the Internet, GPS, and so much more.  \n",
      "\n",
      "ARPA-H will have a singular purpose—to drive breakthroughs in cancer, Alzheimer’s, diabetes, and more. \n",
      "\n",
      "A unity agenda for the nation. \n",
      "\n",
      "We can do this. \n",
      "\n",
      "My fellow Americans—tonight , we have gathered in a sacred space—the citadel of our democracy. \n",
      "\n",
      "In this Capitol, generation after generation, Americans have debated great questions amid great strife, and have done great things. \n",
      "\n",
      "We have fought for freedom, expanded liberty, defeated totalitarianism and terror. \n",
      "\n",
      "And built the strongest, freest, and most prosperous nation the world has ever known. \n",
      "\n",
      "Now is the hour. \n",
      "\n",
      "Our moment of responsibility. \n",
      "\n",
      "Our test of resolve and conscience, of history itself. \n",
      "\n",
      "It is in this moment that our character is formed. Our purpose is found. Our future is forged. \n",
      "\n",
      "Well I know this nation.  \n",
      "\n",
      "We will meet the test. \n",
      "\n",
      "To protect freedom and liberty, to expand fairness and opportunity. \n",
      "\n",
      "We will save democracy. \n",
      "\n",
      "As hard as these times have been, I am more optimistic about America today than I have been my whole life. \n",
      "\n",
      "Because I see the future that is within our grasp. \n",
      "\n",
      "Because I know there is simply nothing beyond our capacity. \n",
      "\n",
      "We are the only nation on Earth that has always turned every crisis we have faced into an opportunity. \n",
      "\n",
      "The only nation that can be defined by a single word: possibilities. \n",
      "\n",
      "So on this night, in our 245th year as a nation, I have come to report on the State of the Union. \n",
      "\n",
      "And my report is this: the State of the Union is strong—because you, the American people, are strong. \n",
      "\n",
      "We are stronger today than we were a year ago. \n",
      "\n",
      "And we will be stronger a year from now than we are today. \n",
      "\n",
      "Now is our moment to meet and overcome the challenges of our time. \n",
      "\n",
      "And we will, as one people. \n",
      "\n",
      "One America. \n",
      "\n",
      "The United States of America. \n",
      "\n",
      "May God bless you all. May God protect our troops.\n"
     ]
    }
   ],
   "source": [
    "import json, os\n",
    "\n",
    "with open(\"/home/ubuntu/config.json\") as f:\n",
    "    config = json.loads(f.read())\n",
    "os.environ[\"COHERE_API_KEY\"] = config[\"cohere_api_key\"]\n",
    "\n",
    "with open(\"data/state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()\n",
    "print(state_of_the_union)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "splits: 26\n",
      "first doc content: Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. And with an unwavering resolve that freedom will always triumph over tyranny. Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. He met the Ukrainian people. From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland. In this struggle as President Zelenskyy said in his speech to the European Parliament “Light will win over darkness.” The Ukrainian Ambassador to the United States is here tonight. Let each of us here tonight in this Chamber send an unmistakable signal to Ukraine and to the world. Please rise if you are able and show that, Yes, we the United States of America stand with the Ukrainian people. Throughout our history we’ve learned this lesson when dictators do not pay a price for their aggression they cause more chaos.\n"
     ]
    }
   ],
   "source": [
    "from langchain_experimental.text_splitter import SemanticChunker\n",
    "from langchain_community.embeddings import CohereEmbeddings\n",
    "\n",
    "text_splitter = SemanticChunker(CohereEmbeddings())\n",
    "docs = text_splitter.create_documents([state_of_the_union])\n",
    "\n",
    "print(f\"splits: {len(docs)}\")\n",
    "print(f\"first doc content: {docs[0].page_content}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "SemanticChunker.__init__() got an unexpected keyword argument 'breakpoint_threshold_type'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[8], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m text_splitter \u001b[38;5;241m=\u001b[39m \u001b[43mSemanticChunker\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m      2\u001b[0m \u001b[43m    \u001b[49m\u001b[43mCohereEmbeddings\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbreakpoint_threshold_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mpercentile\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\n\u001b[1;32m      3\u001b[0m \u001b[43m)\u001b[49m\n",
      "\u001b[0;31mTypeError\u001b[0m: SemanticChunker.__init__() got an unexpected keyword argument 'breakpoint_threshold_type'"
     ]
    }
   ],
   "source": [
    "text_splitter = SemanticChunker(\n",
    "    CohereEmbeddings(), breakpoint_threshold_type=\"percentile\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Text Embedding Models: caching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CacheBackedEmbeddings\n",
    "- back by Vector Store\n",
    "- back by ByteStore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vectorstore\n",
    "from langchain.storage import LocalFileStore\n",
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_community.vectorstores import FAISS\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "underlying_embeddings = OpenAIEmbeddings()\n",
    "\n",
    "store = LocalFileStore(\"./cache/\")\n",
    "\n",
    "cached_embedder = CacheBackedEmbeddings.from_bytes_store(\n",
    "    underlying_embeddings, store, namespace=underlying_embeddings.model\n",
    ")\n",
    "\n",
    "list(store.yield_keys())\n",
    "\n",
    "# bytestore\n",
    "from langchain.embeddings import CacheBackedEmbeddings\n",
    "from langchain.storage import InMemoryByteStore\n",
    "\n",
    "store = InMemoryByteStore()\n",
    "\n",
    "cached_embedder = CacheBackedEmbeddings.from_bytes_store(\n",
    "    underlying_embeddings, store, namespace=underlying_embeddings.model\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Vectorstores"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "1. One of the most common ways to store and search over unstructured data is to embed it and store the resulting embedding vectors, \n",
    "   and then at query time to embed the unstructured query and retrieve the embedding vectors that are 'most similar' to the embedded query. \n",
    "   A vector store takes care of storing embedded data and performing vector search for you.\n",
    "\n",
    "2. basic functionality related to vector stores - creating vectors and putting in it. \n",
    "\n",
    "3. search (sync and async)\n",
    "    - similarity search\n",
    "    - similarity search by vector\n",
    "    - maximum marginal relevance search (MMR)\n",
    "\n",
    "4. One thing to notice is there are diff functions for embedding documents and a query. This is solely so one can have options to do it differently for same usecase."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://python.langchain.com/assets/images/vector_stores-125d1675d58cfb46ce9054c9019fea72.jpg\" alt=\"drawing\" width=\"1000\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Retrievers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[What]\n",
    "- Interface that returns documents given an unstructured query.\n",
    "- Different that a vectorstore - in a way that - doesn't need to store docuemnts - it only needs to focus on retrieval logic.\n",
    "\n",
    "[Types]\n",
    "- Vectorstore\n",
    "- Parent Document\n",
    "- Multi Vector\n",
    "- Self Query\n",
    "- Contextual Compression\n",
    "- Time Weighted Vectorstore\n",
    "- Multi Query Retriever\n",
    "- Ensemble\n",
    "- Long Context Reorder \n",
    "\n",
    "[MultiQueryRetriever]\n",
    "- Distance-based vector database retrieval embeds (represents) queries in high-dimensional space and finds similar embedded documents based on \"distance\". \n",
    "- But, retrieval may produce different results with subtle changes in query wording or if the embeddings do not capture the semantics of the data well. \n",
    "- Prompt engg/tuning is sometimes done to manually address these problems, but can be tedious.\n",
    "- MultiQueryRetriever automates the process of prompt tuning by using an llm to generate multiple queries from different perspectives for a given user input query. For each query, it retrieves a set of relevant docuemnts and takes the unique union across all queries to get a larger set of potentially relevant documents.\n",
    "- Essentially generating multiple perspectives on the same question -> Done to overcome some of the limiations of the distance-based retrieval and get a richer set of results.\n",
    "\n",
    "[EnsembleRetriever]\n",
    "- takes a list of retrievers as input and \n",
    "- ensemble the results of their get_relevant_documents()\n",
    "- rerank the results based on Reciprocal Rank Fusion algorithm\n",
    "- Places to use:\n",
    "    - leverage strengths of diff algorithms like sparse (keyword-search) + dense retriever(semantic similarity)\n",
    "    - mulltiple sources of data\n",
    "\n",
    "[LongContextReorder]\n",
    "- No matter the architecture of your model, there is a substantial performance degradation when you include 10+ retrieved documents. \n",
    "- From paper(https://arxiv.org/abs/2307.03172) when models must access relevant information in the middle of long contexts, they tend to ignore the provided documents.\n",
    "- To avoid the issue, you can reorder documents after retrieval to avoid performance degradation.\n",
    "\n",
    "[MultiVectorRetriever]\n",
    "- useful to store multiple vectors per document\n",
    "- methods to create multiple vectors per document include:\n",
    "    - smaller chunks: split a doc into smaller chunks, and embed those (ParentDocumentRetriever).\n",
    "    - summary: create summary for each document, embed that along with (or instead of) the document.\n",
    "    - hypothetical questions: create hypothetical questions that each document would be appropriate to answer, embed those along with (or instead of) the document.\n",
    "\n",
    "[SelfQueryRetriever]\n",
    "- ability to query itself\n",
    "- user-input query for semantic similarity comparison with the contents of stored documents but also to extract filters from the user query on the metadata of stored documents and to execure those filters.\n",
    "- retriever uses a query-constructing llm chain to write a structured query and then apply that structured query to its underlying vectorstore.\n",
    "- Flow: user_question -> query_constructor_llm_call -> query_translator -> vectorstore -> retrieved_docs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='research')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Custom retriever\n",
    "\n",
    "from langchain_core.retrievers import BaseRetriever\n",
    "from langchain_core.callbacks import CallbackManagerForRetrieverRun\n",
    "from langchain_core.documents import Document\n",
    "from typing import List\n",
    "\n",
    "class CustomRetriever(BaseRetriever):\n",
    "    def _get_relevant_documents(\n",
    "        self, \n",
    "        query: str, \n",
    "        *,\n",
    "        run_manager: CallbackManagerForRetrieverRun,\n",
    "    ) -> List[Document]:\n",
    "        \n",
    "        # your logic, below is just some dummy code\n",
    "        return [\n",
    "            Document(page_content=query)\n",
    "        ]\n",
    "\n",
    "retriever = CustomRetriever()\n",
    "\n",
    "retriever.get_relevant_documents(\"research\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Multi Query Retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Ignoring invalid distribution -ackaging (/opt/conda/envs/pytorch/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -ackaging (/opt/conda/envs/pytorch/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "! pip install chromadb --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json, os\n",
    "\n",
    "with open(\"/home/ubuntu/config.json\") as f:\n",
    "    config = json.loads(f.read())\n",
    "os.environ[\"COHERE_API_KEY\"] = config[\"cohere_api_key\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='\\n\\n\\n\\n\\n\\nLLM Powered Autonomous Agents | Lil\\'Log\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nLil\\'Log\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nPosts\\n\\n\\n\\n\\nArchive\\n\\n\\n\\n\\nSearch\\n\\n\\n\\n\\nTags\\n\\n\\n\\n\\nFAQ\\n\\n\\n\\n\\nemojisearch.app\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n      LLM Powered Autonomous Agents\\n    \\nDate: June 23, 2023  |  Estimated Reading Time: 31 min  |  Author: Lilian Weng\\n\\n\\n \\n\\n\\nTable of Contents\\n\\n\\n\\nAgent System Overview\\n\\nComponent One: Planning\\n\\nTask Decomposition\\n\\nSelf-Reflection\\n\\n\\nComponent Two: Memory\\n\\nTypes of Memory\\n\\nMaximum Inner Product Search (MIPS)\\n\\n\\nComponent Three: Tool Use\\n\\nCase Studies\\n\\nScientific Discovery Agent\\n\\nGenerative Agents Simulation\\n\\nProof-of-Concept Examples\\n\\n\\nChallenges\\n\\nCitation\\n\\nReferences\\n\\n\\n\\n\\n\\nBuilding agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview#\\nIn a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:\\n\\nPlanning\\n\\nSubgoal and decomposition: The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks.\\nReflection and refinement: The agent can do self-criticism and self-reflection over past actions, learn from mistakes and refine them for future steps, thereby improving the quality of final results.\\n\\n\\nMemory\\n\\nShort-term memory: I would consider all the in-context learning (See Prompt Engineering) as utilizing short-term memory of the model to learn.\\nLong-term memory: This provides the agent with the capability to retain and recall (infinite) information over extended periods, often by leveraging an external vector store and fast retrieval.\\n\\n\\nTool use\\n\\nThe agent learns to call external APIs for extra information that is missing from the model weights (often hard to change after pre-training), including current information, code execution capability, access to proprietary information sources and more.\\n\\n\\n\\n\\nFig. 1. Overview of a LLM-powered autonomous agent system.\\nComponent One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\nTask Decomposition#\\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\\nTree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\\nTask decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.\\nAnother quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains.\\nSelf-Reflection#\\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable.\\nReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language.\\nThe ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\\nThought: ...\\nAction: ...\\nObservation: ...\\n... (Repeated many times)\\n\\nFig. 2.  Examples of reasoning trajectories for knowledge-intensive tasks (e.g. HotpotQA, FEVER) and decision-making tasks (e.g. AlfWorld Env, WebShop). (Image source: Yao et al. 2023).\\nIn both experiments on knowledge-intensive tasks and decision-making tasks, ReAct works better than the Act-only baseline where Thought: … step is removed.\\nReflexion (Shinn & Labash 2023) is a framework to equips agents with dynamic memory and self-reflection capabilities to improve reasoning skills. Reflexion has a standard RL setup, in which the reward model provides a simple binary reward and the action space follows the setup in ReAct where the task-specific action space is augmented with language to enable complex reasoning steps. After each action $a_t$, the agent computes a heuristic $h_t$ and optionally may decide to reset the environment to start a new trial depending on the self-reflection results.\\n\\nFig. 3. Illustration of the Reflexion framework. (Image source: Shinn & Labash, 2023)\\nThe heuristic function determines when the trajectory is inefficient or contains hallucination and should be stopped. Inefficient planning refers to trajectories that take too long without success. Hallucination is defined as encountering a sequence of consecutive identical actions that lead to the same observation in the environment.\\nSelf-reflection is created by showing two-shot examples to LLM and each example is a pair of (failed trajectory, ideal reflection for guiding future changes in the plan). Then reflections are added into the agent’s working memory, up to three, to be used as context for querying LLM.\\n\\nFig. 4. Experiments on AlfWorld Env and HotpotQA. Hallucination is a more common failure than inefficient planning in AlfWorld. (Image source: Shinn & Labash, 2023)\\nChain of Hindsight (CoH; Liu et al. 2023) encourages the model to improve on its own outputs by explicitly presenting it with a sequence of past outputs, each annotated with feedback. Human feedback data is a collection of $D_h = \\\\{(x, y_i , r_i , z_i)\\\\}_{i=1}^n$, where $x$ is the prompt, each $y_i$ is a model completion, $r_i$ is the human rating of $y_i$, and $z_i$ is the corresponding human-provided hindsight feedback. Assume the feedback tuples are ranked by reward, $r_n \\\\geq r_{n-1} \\\\geq \\\\dots \\\\geq r_1$ The process is supervised fine-tuning where the data is a sequence in the form of $\\\\tau_h = (x, z_i, y_i, z_j, y_j, \\\\dots, z_n, y_n)$, where $\\\\leq i \\\\leq j \\\\leq n$. The model is finetuned to only predict $y_n$ where conditioned on the sequence prefix, such that the model can self-reflect to produce better output based on the feedback sequence. The model can optionally receive multiple rounds of instructions with human annotators at test time.\\nTo avoid overfitting, CoH adds a regularization term to maximize the log-likelihood of the pre-training dataset. To avoid shortcutting and copying (because there are many common words in feedback sequences), they randomly mask 0% - 5% of past tokens during training.\\nThe training dataset in their experiments is a combination of WebGPT comparisons, summarization from human feedback and human preference dataset.\\n\\nFig. 5. After fine-tuning with CoH, the model can follow instructions to produce outputs with incremental improvement in a sequence. (Image source: Liu et al. 2023)\\nThe idea of CoH is to present a history of sequentially improved outputs  in context and train the model to take on the trend to produce better outputs. Algorithm Distillation (AD; Laskin et al. 2023) applies the same idea to cross-episode trajectories in reinforcement learning tasks, where an algorithm is encapsulated in a long history-conditioned policy. Considering that an agent interacts with the environment many times and in each episode the agent gets a little better, AD concatenates this learning history and feeds that into the model. Hence we should expect the next predicted action to lead to better performance than previous trials. The goal is to learn the process of RL instead of training a task-specific policy itself.\\n\\nFig. 6. Illustration of how Algorithm Distillation (AD) works. (Image source: Laskin et al. 2023).\\nThe paper hypothesizes that any algorithm that generates a set of learning histories can be distilled into a neural network by performing behavioral cloning over actions. The history data is generated by a set of source policies, each trained for a specific task. At the training stage, during each RL run, a random task is sampled and a subsequence of multi-episode history is used for training, such that the learned policy is task-agnostic.\\nIn reality, the model has limited context window length, so episodes should be short enough to construct multi-episode history. Multi-episodic contexts of 2-4 episodes are necessary to learn a near-optimal in-context RL algorithm. The emergence of in-context RL requires long enough context.\\nIn comparison with three baselines, including ED (expert distillation, behavior cloning with expert trajectories instead of learning history), source policy (used for generating trajectories for distillation by UCB), RL^2 (Duan et al. 2017; used as upper bound since it needs online RL), AD demonstrates in-context RL with performance getting close to RL^2 despite only using offline RL and learns much faster than other baselines. When conditioned on partial training history of the source policy, AD also improves much faster than ED baseline.\\n\\nFig. 7. Comparison of AD, ED, source policy and RL^2 on environments that require memory and exploration. Only binary reward is assigned. The source policies are trained with A3C for \"dark\" environments and DQN for watermaze.(Image source: Laskin et al. 2023)\\nComponent Two: Memory#\\n(Big thank you to ChatGPT for helping me draft this section. I’ve learned a lot about the human brain and data structure for fast MIPS in my conversations with ChatGPT.)\\nTypes of Memory#\\nMemory can be defined as the processes used to acquire, store, retain, and later retrieve information. There are several types of memory in human brains.\\n\\n\\nSensory Memory: This is the earliest stage of memory, providing the ability to retain impressions of sensory information (visual, auditory, etc) after the original stimuli have ended. Sensory memory typically only lasts for up to a few seconds. Subcategories include iconic memory (visual), echoic memory (auditory), and haptic memory (touch).\\n\\n\\nShort-Term Memory (STM) or Working Memory: It stores information that we are currently aware of and needed to carry out complex cognitive tasks such as learning and reasoning. Short-term memory is believed to have the capacity of about 7 items (Miller 1956) and lasts for 20-30 seconds.\\n\\n\\nLong-Term Memory (LTM): Long-term memory can store information for a remarkably long time, ranging from a few days to decades, with an essentially unlimited storage capacity. There are two subtypes of LTM:\\n\\nExplicit / declarative memory: This is memory of facts and events, and refers to those memories that can be consciously recalled, including episodic memory (events and experiences) and semantic memory (facts and concepts).\\nImplicit / procedural memory: This type of memory is unconscious and involves skills and routines that are performed automatically, like riding a bike or typing on a keyboard.\\n\\n\\n\\n\\nFig. 8. Categorization of human memory.\\nWe can roughly consider the following mappings:\\n\\nSensory memory as learning embedding representations for raw inputs, including text, image or other modalities;\\nShort-term memory as in-context learning. It is short and finite, as it is restricted by the finite context window length of Transformer.\\nLong-term memory as the external vector store that the agent can attend to at query time, accessible via fast retrieval.\\n\\nMaximum Inner Product Search (MIPS)#\\nThe external memory can alleviate the restriction of finite attention span.  A standard practice is to save the embedding representation of information into a vector store database that can support fast maximum inner-product search (MIPS). To optimize the retrieval speed, the common choice is the approximate nearest neighbors (ANN)\\u200b algorithm to return approximately top k nearest neighbors to trade off a little accuracy lost for a huge speedup.\\nA couple common choices of ANN algorithms for fast MIPS:\\n\\nLSH (Locality-Sensitive Hashing): It introduces a hashing function such that similar input items are mapped to the same buckets with high probability, where the number of buckets is much smaller than the number of inputs.\\nANNOY (Approximate Nearest Neighbors Oh Yeah): The core data structure are random projection trees, a set of binary trees where each non-leaf node represents a hyperplane splitting the input space into half and each leaf stores one data point. Trees are built independently and at random, so to some extent, it mimics a hashing function. ANNOY search happens in all the trees to iteratively search through the half that is closest to the query and then aggregates the results. The idea is quite related to KD tree but a lot more scalable.\\nHNSW (Hierarchical Navigable Small World): It is inspired by the idea of small world networks where most nodes can be reached by any other nodes within a small number of steps; e.g. “six degrees of separation” feature of social networks. HNSW builds hierarchical layers of these small-world graphs, where the bottom layers contain the actual data points. The layers in the middle create shortcuts to speed up search. When performing a search, HNSW starts from a random node in the top layer and navigates towards the target. When it can’t get any closer, it moves down to the next layer, until it reaches the bottom layer. Each move in the upper layers can potentially cover a large distance in the data space, and each move in the lower layers refines the search quality.\\nFAISS (Facebook AI Similarity Search): It operates on the assumption that in high dimensional space, distances between nodes follow a Gaussian distribution and thus there should exist clustering of data points. FAISS applies vector quantization by partitioning the vector space into clusters and then refining the quantization within clusters. Search first looks for cluster candidates with coarse quantization and then further looks into each cluster with finer quantization.\\nScaNN (Scalable Nearest Neighbors): The main innovation in ScaNN is anisotropic vector quantization. It quantizes a data point $x_i$ to $\\\\tilde{x}_i$ such that the inner product $\\\\langle q, x_i \\\\rangle$ is as similar to the original distance of $\\\\angle q, \\\\tilde{x}_i$ as possible, instead of picking the closet quantization centroid points.\\n\\n\\nFig. 9. Comparison of MIPS algorithms, measured in recall@10. (Image source: Google Blog, 2020)\\nCheck more MIPS algorithms and performance comparison in ann-benchmarks.com.\\nComponent Three: Tool Use#\\nTool use is a remarkable and distinguishing characteristic of human beings. We create, modify and utilize external objects to do things that go beyond our physical and cognitive limits. Equipping LLMs with external tools can significantly extend the model capabilities.\\n\\nFig. 10. A picture of a sea otter using rock to crack open a seashell, while floating in the water. While some other animals can use tools, the complexity is not comparable with humans. (Image source: Animals using tools)\\nMRKL (Karpas et al. 2022), short for “Modular Reasoning, Knowledge and Language”, is a neuro-symbolic architecture for autonomous agents. A MRKL system is proposed to contain a collection of “expert” modules and the general-purpose LLM works as a router to route inquiries to the best suitable expert module. These modules can be neural (e.g. deep learning models) or symbolic (e.g. math calculator, currency converter, weather API).\\nThey did an experiment on fine-tuning LLM to call a calculator, using arithmetic as a test case. Their experiments showed that it was harder to solve verbal math problems than explicitly stated math problems because LLMs (7B Jurassic1-large model) failed to extract the right arguments for the basic arithmetic reliably. The results highlight when the external symbolic tools can work reliably, knowing when to and how to use the tools are crucial, determined by the LLM capability.\\nBoth TALM (Tool Augmented Language Models; Parisi et al. 2022) and Toolformer (Schick et al. 2023) fine-tune a LM to learn to use external tool APIs. The dataset is expanded based on whether a newly added API call annotation can improve the quality of model outputs. See more details in the “External APIs” section of Prompt Engineering.\\nChatGPT Plugins and OpenAI API  function calling are good examples of LLMs augmented with tool use capability working in practice. The collection of tool APIs can be provided by other developers (as in Plugins) or self-defined (as in function calls).\\nHuggingGPT (Shen et al. 2023) is a framework to use ChatGPT as the task planner to select models available in HuggingFace platform according to the model descriptions and summarize the response based on the execution results.\\n\\nFig. 11. Illustration of how HuggingGPT works. (Image source: Shen et al. 2023)\\nThe system comprises of 4 stages:\\n(1) Task planning: LLM works as the brain and parses the user requests into multiple tasks. There are four attributes associated with each task: task type, ID, dependencies, and arguments. They use few-shot examples to guide LLM to do task parsing and planning.\\nInstruction:\\n\\nThe AI assistant can parse user input to several tasks: [{\"task\": task, \"id\", task_id, \"dep\": dependency_task_ids, \"args\": {\"text\": text, \"image\": URL, \"audio\": URL, \"video\": URL}}]. The \"dep\" field denotes the id of the previous task which generates a new resource that the current task relies on. A special tag \"-task_id\" refers to the generated text image, audio and video in the dependency task with id as task_id. The task MUST be selected from the following options: {{ Available Task List }}. There is a logical relationship between tasks, please note their order. If the user input can\\'t be parsed, you need to reply empty JSON. Here are several cases for your reference: {{ Demonstrations }}. The chat history is recorded as {{ Chat History }}. From this chat history, you can find the path of the user-mentioned resources for your task planning.\\n\\n(2) Model selection: LLM distributes the tasks to expert models, where the request is framed as a multiple-choice question. LLM is presented with a list of models to choose from. Due to the limited context length, task type based filtration is needed.\\nInstruction:\\n\\nGiven the user request and the call command, the AI assistant helps the user to select a suitable model from a list of models to process the user request. The AI assistant merely outputs the model id of the most appropriate model. The output must be in a strict JSON format: \"id\": \"id\", \"reason\": \"your detail reason for the choice\". We have a list of models for you to choose from {{ Candidate Models }}. Please select one model from the list.\\n\\n(3) Task execution: Expert models execute on the specific tasks and log results.\\nInstruction:\\n\\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user\\'s request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\\n\\n(4) Response generation: LLM receives the execution results and provides summarized results to users.\\nTo put HuggingGPT into real world usage, a couple challenges need to solve: (1) Efficiency improvement is needed as both LLM inference rounds and interactions with other models slow down the process; (2) It relies on a long context window to communicate over complicated task content; (3) Stability improvement of LLM outputs and external model services.\\nAPI-Bank (Li et al. 2023) is a benchmark for evaluating the performance of tool-augmented LLMs. It contains 53 commonly used API tools, a complete tool-augmented LLM workflow, and 264 annotated dialogues that involve 568 API calls. The selection of APIs is quite diverse, including search engines, calculator, calendar queries, smart home control, schedule management, health data management, account authentication workflow and more. Because there are a large number of APIs, LLM first has access to API search engine to find the right API to call and then uses the corresponding documentation to make a call.\\n\\nFig. 12. Pseudo code of how LLM makes an API call in API-Bank. (Image source: Li et al. 2023)\\nIn the API-Bank workflow, LLMs need to make a couple of decisions and at each step we can evaluate how accurate that decision is. Decisions include:\\n\\nWhether an API call is needed.\\nIdentify the right API to call: if not good enough, LLMs need to iteratively modify the API inputs (e.g. deciding search keywords for Search Engine API).\\nResponse based on the API results: the model can choose to refine and call again if results are not satisfied.\\n\\nThis benchmark evaluates the agent’s tool use capabilities at three levels:\\n\\nLevel-1 evaluates the ability to call the API. Given an API’s description, the model needs to determine whether to call a given API, call it correctly, and respond properly to API returns.\\nLevel-2 examines the ability to retrieve the API. The model needs to search for possible APIs that may solve the user’s requirement and learn how to use them by reading documentation.\\nLevel-3 assesses the ability to plan API beyond retrieve and call. Given unclear user requests (e.g. schedule group meetings, book flight/hotel/restaurant for a trip), the model may have to conduct multiple API calls to solve it.\\n\\nCase Studies#\\nScientific Discovery Agent#\\nChemCrow (Bran et al. 2023) is a domain-specific example in which LLM is augmented with 13 expert-designed tools to accomplish tasks across organic synthesis, drug discovery, and materials design. The workflow, implemented in LangChain, reflects what was previously described in the ReAct and MRKLs and combines CoT reasoning with tools relevant to the tasks:\\n\\nThe LLM is provided with a list of tool names, descriptions of their utility, and details about the expected input/output.\\nIt is then instructed to answer a user-given prompt using the tools provided when necessary. The instruction suggests the model to follow the ReAct format - Thought, Action, Action Input, Observation.\\n\\nOne interesting observation is that while the LLM-based evaluation concluded that GPT-4 and ChemCrow perform nearly equivalently, human evaluations with experts oriented towards the completion and chemical correctness of the solutions showed that ChemCrow outperforms GPT-4 by a large margin. This indicates a potential problem with using LLM to evaluate its own performance on domains that requires deep expertise. The lack of expertise may cause LLMs not knowing its flaws and thus cannot well judge the correctness of task results.\\nBoiko et al. (2023) also looked into LLM-empowered agents for scientific discovery, to handle autonomous design, planning, and performance of complex scientific experiments. This agent can use tools to browse the Internet, read documentation, execute code, call robotics experimentation APIs and leverage other LLMs.\\nFor example, when requested to \"develop a novel anticancer drug\", the model came up with the following reasoning steps:\\n\\ninquired about current trends in anticancer drug discovery;\\nselected a target;\\nrequested a scaffold targeting these compounds;\\nOnce the compound was identified, the model attempted its synthesis.\\n\\nThey also discussed the risks, especially with illicit drugs and bioweapons. They developed a test set containing a list of known chemical weapon agents and asked the agent to synthesize them. 4 out of 11 requests (36%) were accepted to obtain a synthesis solution and the agent attempted to consult documentation to execute the procedure. 7 out of 11 were rejected and among these 7 rejected cases, 5 happened after a Web search while 2 were rejected based on prompt only.\\nGenerative Agents Simulation#\\nGenerative Agents (Park, et al. 2023) is super fun experiment where 25 virtual characters, each controlled by a LLM-powered agent, are living and interacting in a sandbox environment, inspired by The Sims. Generative agents create believable simulacra of human behavior for interactive applications.\\nThe design of generative agents combines LLM with memory, planning and reflection mechanisms to enable agents to behave conditioned on past experience, as well as to interact with other agents.\\n\\nMemory stream: is a long-term memory module (external database) that records a comprehensive list of agents’ experience in natural language.\\n\\nEach element is an observation, an event directly provided by the agent.\\n- Inter-agent communication can trigger new natural language statements.\\n\\n\\nRetrieval model: surfaces the context to inform the agent’s behavior, according to relevance, recency and importance.\\n\\nRecency: recent events have higher scores\\nImportance: distinguish mundane from core memories. Ask LM directly.\\nRelevance: based on how related it is to the current situation / query.\\n\\n\\nReflection mechanism: synthesizes memories into higher level inferences over time and guides the agent’s future behavior. They are higher-level summaries of past events (<- note that this is a bit different from self-reflection above)\\n\\nPrompt LM with 100 most recent observations and to generate 3 most salient high-level questions given a set of observations/statements. Then ask LM to answer those questions.\\n\\n\\nPlanning & Reacting: translate the reflections and the environment information into actions\\n\\nPlanning is essentially in order to optimize believability at the moment vs in time.\\nPrompt template: {Intro of an agent X}. Here is X\\'s plan today in broad strokes: 1)\\nRelationships between agents and observations of one agent by another are all taken into consideration for planning and reacting.\\nEnvironment information is present in a tree structure.\\n\\n\\n\\n\\nFig. 13. The generative agent architecture. (Image source: Park et al. 2023)\\nThis fun simulation results in emergent social behavior, such as information diffusion, relationship memory (e.g. two agents continuing the conversation topic) and coordination of social events (e.g. host a party and invite many others).\\nProof-of-Concept Examples#\\nAutoGPT has drawn a lot of attention into the possibility of setting up autonomous agents with LLM as the main controller. It has quite a lot of reliability issues given the natural language interface, but nevertheless a cool proof-of-concept demo. A lot of code in AutoGPT is about format parsing.\\nHere is the system message used by AutoGPT, where {{...}} are user inputs:\\nYou are {{ai-name}}, {{user-provided AI bot description}}.\\nYour decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.\\n\\nGOALS:\\n\\n1. {{user-provided goal 1}}\\n2. {{user-provided goal 2}}\\n3. ...\\n4. ...\\n5. ...\\n\\nConstraints:\\n1. ~4000 word limit for short term memory. Your short term memory is short, so immediately save important information to files.\\n2. If you are unsure how you previously did something or want to recall past events, thinking about similar events will help you remember.\\n3. No user assistance\\n4. Exclusively use the commands listed in double quotes e.g. \"command name\"\\n5. Use subprocesses for commands that will not terminate within a few minutes\\n\\nCommands:\\n1. Google Search: \"google\", args: \"input\": \"<search>\"\\n2. Browse Website: \"browse_website\", args: \"url\": \"<url>\", \"question\": \"<what_you_want_to_find_on_website>\"\\n3. Start GPT Agent: \"start_agent\", args: \"name\": \"<name>\", \"task\": \"<short_task_desc>\", \"prompt\": \"<prompt>\"\\n4. Message GPT Agent: \"message_agent\", args: \"key\": \"<key>\", \"message\": \"<message>\"\\n5. List GPT Agents: \"list_agents\", args:\\n6. Delete GPT Agent: \"delete_agent\", args: \"key\": \"<key>\"\\n7. Clone Repository: \"clone_repository\", args: \"repository_url\": \"<url>\", \"clone_path\": \"<directory>\"\\n8. Write to file: \"write_to_file\", args: \"file\": \"<file>\", \"text\": \"<text>\"\\n9. Read file: \"read_file\", args: \"file\": \"<file>\"\\n10. Append to file: \"append_to_file\", args: \"file\": \"<file>\", \"text\": \"<text>\"\\n11. Delete file: \"delete_file\", args: \"file\": \"<file>\"\\n12. Search Files: \"search_files\", args: \"directory\": \"<directory>\"\\n13. Analyze Code: \"analyze_code\", args: \"code\": \"<full_code_string>\"\\n14. Get Improved Code: \"improve_code\", args: \"suggestions\": \"<list_of_suggestions>\", \"code\": \"<full_code_string>\"\\n15. Write Tests: \"write_tests\", args: \"code\": \"<full_code_string>\", \"focus\": \"<list_of_focus_areas>\"\\n16. Execute Python File: \"execute_python_file\", args: \"file\": \"<file>\"\\n17. Generate Image: \"generate_image\", args: \"prompt\": \"<prompt>\"\\n18. Send Tweet: \"send_tweet\", args: \"text\": \"<text>\"\\n19. Do Nothing: \"do_nothing\", args:\\n20. Task Complete (Shutdown): \"task_complete\", args: \"reason\": \"<reason>\"\\n\\nResources:\\n1. Internet access for searches and information gathering.\\n2. Long Term memory management.\\n3. GPT-3.5 powered Agents for delegation of simple tasks.\\n4. File output.\\n\\nPerformance Evaluation:\\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\\n2. Constructively self-criticize your big-picture behavior constantly.\\n3. Reflect on past decisions and strategies to refine your approach.\\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.\\n\\nYou should only respond in JSON format as described below\\nResponse Format:\\n{\\n    \"thoughts\": {\\n        \"text\": \"thought\",\\n        \"reasoning\": \"reasoning\",\\n        \"plan\": \"- short bulleted\\\\n- list that conveys\\\\n- long-term plan\",\\n        \"criticism\": \"constructive self-criticism\",\\n        \"speak\": \"thoughts summary to say to user\"\\n    },\\n    \"command\": {\\n        \"name\": \"command name\",\\n        \"args\": {\\n            \"arg name\": \"value\"\\n        }\\n    }\\n}\\nEnsure the response can be parsed by Python json.loads\\nGPT-Engineer is another project to create a whole repository of code given a task specified in natural language. The GPT-Engineer is instructed to think over a list of smaller components to build and ask for user input to clarify questions as needed.\\nHere are a sample conversation for task clarification sent to OpenAI ChatCompletion endpoint used by GPT-Engineer. The user inputs are wrapped in {{user input text}}.\\n[\\n  {\\n    \"role\": \"system\",\\n    \"content\": \"You will read instructions and not carry them out, only seek to clarify them.\\\\nSpecifically you will first summarise a list of super short bullets of areas that need clarification.\\\\nThen you will pick one clarifying question, and wait for an answer from the user.\\\\n\"\\n  },\\n  {\\n    \"role\": \"user\",\\n    \"content\": \"We are writing {{a Super Mario game in python. MVC components split in separate files. Keyboard control.}}\\\\n\"\\n  },\\n  {\\n    \"role\": \"assistant\",\\n    \"content\": \"Summary of areas that need clarification:\\\\n1. Specifics of the Super Mario game (e.g. level design, characters, gameplay mechanics)\\\\n2. Details about the MVC components (e.g. which components are in each file)\\\\n3. Keyboard control implementation (e.g. which keys to use, how to handle input)\\\\n\\\\nClarifying question:\\\\nCan you provide more details about the Super Mario game, such as level design, characters, and gameplay mechanics?\"\\n  },\\n  {\\n    \"role\": \"user\",\\n    \"content\": \"{{There are 10 levels in total. The main character is a plumber named Mario, who can walk and jump. It is a classical platform game just like Super Mario. The main character moves from left to right, trying to get to the destination, where there are many obstacles and attacks from enemies in the process.}}\\\\n\\\\nIs anything else unclear? If yes, only answer in the form:\\\\n{remaining unclear areas} remaining questions.\\\\n{Next question}\\\\nIf everything is sufficiently clear, only answer \\\\\"Nothing more to clarify.\\\\\".\"\\n  },\\n  {\\n    \"role\": \"assistant\",\\n    \"content\": \"Remaining unclear areas: 2 remaining questions.\\\\nCan you provide more information about how the MVC components are split into separate files?\"\\n  },\\n  {\\n    \"role\": \"user\",\\n    \"content\": \"{{Make your own assumptions and state them explicitly before starting}}\"\\n  }\\n]\\nThen after these clarification, the agent moved into the code writing mode with a different system message.\\nSystem message:\\n\\nYou will get instructions for code to write.\\nYou will write a very long answer. Make sure that every detail of the architecture is, in the end, implemented as code.\\nMake sure that every detail of the architecture is, in the end, implemented as code.\\nThink step by step and reason yourself to the right decisions to make sure we get it right.\\nYou will first lay out the names of the core classes, functions, methods that will be necessary, as well as a quick comment on their purpose.\\nThen you will output the content of each file including ALL code.\\nEach file must strictly follow a markdown code block format, where the following tokens must be replaced such that\\nFILENAME is the lowercase file name including the file extension,\\nLANG is the markup code block language for the code’s language, and CODE is the code:\\nFILENAME\\nCODE\\nYou will start with the “entrypoint” file, then go to the ones that are imported by that file, and so on.\\nPlease note that the code should be fully functional. No placeholders.\\nFollow a language and framework appropriate best practice file naming convention.\\nMake sure that files contain all imports, types etc. Make sure that code in different files are compatible with each other.\\nEnsure to implement all code, if you are unsure, write a plausible implementation.\\nInclude module dependency or package manager dependency definition file.\\nBefore you finish, double check that all parts of the architecture is present in the files.\\nUseful to know:\\nYou almost always put different classes in different files.\\nFor Python, you always create an appropriate requirements.txt file.\\nFor NodeJS, you always create an appropriate package.json file.\\nYou always add a comment briefly describing the purpose of the function definition.\\nYou try to add comments explaining very complex bits of logic.\\nYou always follow the best practices for the requested languages in terms of describing the code written as a defined\\npackage/project.\\nPython toolbelt preferences:\\n\\npytest\\ndataclasses\\n\\n\\nConversatin samples:\\n[\\n  {\\n    \"role\": \"system\",\\n    \"content\": \"You will get instructions for code to write.\\\\nYou will write a very long answer. Make sure that every detail of the architecture is, in the end, implemented as code.\\\\nMake sure that every detail of the architecture is, in the end, implemented as code.\\\\n\\\\nThink step by step and reason yourself to the right decisions to make sure we get it right.\\\\nYou will first lay out the names of the core classes, functions, methods that will be necessary, as well as a quick comment on their purpose.\\\\n\\\\nThen you will output the content of each file including ALL code.\\\\nEach file must strictly follow a markdown code block format, where the following tokens must be replaced such that\\\\nFILENAME is the lowercase file name including the file extension,\\\\nLANG is the markup code block language for the code\\'s language, and CODE is the code:\\\\n\\\\nFILENAME\\\\n```LANG\\\\nCODE\\\\n```\\\\n\\\\nYou will start with the \\\\\"entrypoint\\\\\" file, then go to the ones that are imported by that file, and so on.\\\\nPlease note that the code should be fully functional. No placeholders.\\\\n\\\\nFollow a language and framework appropriate best practice file naming convention.\\\\nMake sure that files contain all imports, types etc. Make sure that code in different files are compatible with each other.\\\\nEnsure to implement all code, if you are unsure, write a plausible implementation.\\\\nInclude module dependency or package manager dependency definition file.\\\\nBefore you finish, double check that all parts of the architecture is present in the files.\\\\n\\\\nUseful to know:\\\\nYou almost always put different classes in different files.\\\\nFor Python, you always create an appropriate requirements.txt file.\\\\nFor NodeJS, you always create an appropriate package.json file.\\\\nYou always add a comment briefly describing the purpose of the function definition.\\\\nYou try to add comments explaining very complex bits of logic.\\\\nYou always follow the best practices for the requested languages in terms of describing the code written as a defined\\\\npackage/project.\\\\n\\\\n\\\\nPython toolbelt preferences:\\\\n- pytest\\\\n- dataclasses\\\\n\"\\n  },\\n #  … same conversation as earlier, ended with \"Make your own assumptions and state them explicitly before starting\".\\n  {\\n    \"role\": \"assistant\",\\n    \"content\": \"Assumptions:\\\\n1. Model: The model will contain the game\\'s data, such as level information, character states, and enemy positions.\\\\n2. View: The view will handle the game\\'s visuals, including rendering the game objects, backgrounds, and updating the display.\\\\n3. Controller: The controller will manage user input, such as keyboard controls, and update the model accordingly.\\\\n\\\\nWith these assumptions in mind, is there any specific aspect of the keyboard control implementation you would like me to clarify?\"\\n  },\\n  {\\n    \"role\": \"user\",\\n    \"content\": \"Please now remember the steps:\\\\n\\\\nThink step by step and reason yourself to the right decisions to make sure we get it right.\\\\nFirst lay out the names of the core classes, functions, methods that will be necessary, As well as a quick comment on their purpose.\\\\n\\\\nThen you will output the content of each file including ALL code.\\\\nEach file must strictly follow a markdown code block format, where the following tokens must be replaced such that\\\\nFILENAME is the lowercase file name including the file extension,\\\\nLANG is the markup code block language for the code\\'s language, and CODE is the code:\\\\n\\\\nFILENAME\\\\n```LANG\\\\nCODE\\\\n```\\\\n\\\\nPlease note that the code should be fully functional. No placeholders.\\\\n\\\\nYou will start with the \\\\\"entrypoint\\\\\" file, then go to the ones that are imported by that file, and so on.\\\\nFollow a language and framework appropriate best practice file naming convention.\\\\nMake sure that files contain all imports, types etc. The code should be fully functional. Make sure that code in different files are compatible with each other.\\\\nBefore you finish, double check that all parts of the architecture is present in the files.\\\\n\"\\n  }\\n]\\nChallenges#\\nAfter going through key ideas and demos of building LLM-centered agents, I start to see a couple common limitations:\\n\\n\\nFinite context length: The restricted context capacity limits the inclusion of historical information, detailed instructions, API call context, and responses. The design of the system has to work with this limited communication bandwidth, while mechanisms like self-reflection to learn from past mistakes would benefit a lot from long or infinite context windows. Although vector stores and retrieval can provide access to a larger knowledge pool, their representation power is not as powerful as full attention.\\n\\n\\nChallenges in long-term planning and task decomposition: Planning over a lengthy history and effectively exploring the solution space remain challenging. LLMs struggle to adjust plans when faced with unexpected errors, making them less robust compared to humans who learn from trial and error.\\n\\n\\nReliability of natural language interface: Current agent system relies on natural language as an interface between LLMs and external components such as memory and tools. However, the reliability of model outputs is questionable, as LLMs may make formatting errors and occasionally exhibit rebellious behavior (e.g. refuse to follow an instruction). Consequently, much of the agent demo code focuses on parsing model output.\\n\\n\\nCitation#\\nCited as:\\n\\nWeng, Lilian. (Jun 2023). “LLM-powered Autonomous Agents”. Lil’Log. https://lilianweng.github.io/posts/2023-06-23-agent/.\\n\\nOr\\n@article{weng2023agent,\\n  title   = \"LLM-powered Autonomous Agents\",\\n  author  = \"Weng, Lilian\",\\n  journal = \"lilianweng.github.io\",\\n  year    = \"2023\",\\n  month   = \"Jun\",\\n  url     = \"https://lilianweng.github.io/posts/2023-06-23-agent/\"\\n}\\nReferences#\\n[1] Wei et al. “Chain of thought prompting elicits reasoning in large language models.” NeurIPS 2022\\n[2] Yao et al. “Tree of Thoughts: Dliberate Problem Solving with Large Language Models.” arXiv preprint arXiv:2305.10601 (2023).\\n[3] Liu et al. “Chain of Hindsight Aligns Language Models with Feedback\\n“ arXiv preprint arXiv:2302.02676 (2023).\\n[4] Liu et al. “LLM+P: Empowering Large Language Models with Optimal Planning Proficiency” arXiv preprint arXiv:2304.11477 (2023).\\n[5] Yao et al. “ReAct: Synergizing reasoning and acting in language models.” ICLR 2023.\\n[6] Google Blog. “Announcing ScaNN: Efficient Vector Similarity Search” July 28, 2020.\\n[7] https://chat.openai.com/share/46ff149e-a4c7-4dd7-a800-fc4a642ea389\\n[8] Shinn & Labash. “Reflexion: an autonomous agent with dynamic memory and self-reflection” arXiv preprint arXiv:2303.11366 (2023).\\n[9] Laskin et al. “In-context Reinforcement Learning with Algorithm Distillation” ICLR 2023.\\n[10] Karpas et al. “MRKL Systems A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning.” arXiv preprint arXiv:2205.00445 (2022).\\n[11] Weaviate Blog. Why is Vector Search so fast? Sep 13, 2022.\\n[12] Li et al. “API-Bank: A Benchmark for Tool-Augmented LLMs” arXiv preprint arXiv:2304.08244 (2023).\\n[13] Shen et al. “HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace” arXiv preprint arXiv:2303.17580 (2023).\\n[14] Bran et al. “ChemCrow: Augmenting large-language models with chemistry tools.” arXiv preprint arXiv:2304.05376 (2023).\\n[15] Boiko et al. “Emergent autonomous scientific research capabilities of large language models.” arXiv preprint arXiv:2304.05332 (2023).\\n[16] Joon Sung Park, et al. “Generative Agents: Interactive Simulacra of Human Behavior.” arXiv preprint arXiv:2304.03442 (2023).\\n[17] AutoGPT. https://github.com/Significant-Gravitas/Auto-GPT\\n[18] GPT-Engineer. https://github.com/AntonOsika/gpt-engineer\\n\\n\\n\\nnlp\\nlanguage-model\\nagent\\nsteerability\\nprompting\\n\\n\\n\\n« \\n\\nAdversarial Attacks on LLMs\\n\\n\\n »\\n\\nPrompt Engineering\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n© 2024 Lil\\'Log\\n\\n        Powered by\\n        Hugo &\\n        PaperMod\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\", 'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en'})]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "\n",
    "data = WebBaseLoader(\"https://lilianweng.github.io/posts/2023-06-23-agent/\").load()\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "splits: 130\n",
      "first doc content: LLM Powered Autonomous Agents | Lil'Log\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Lil'Log\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Posts\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Archive\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Search\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Tags\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "FAQ\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "emojisearch.app\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "      LLM Powered Autonomous Agents\n",
      "    \n",
      "Date: June 23, 2023  |  Estimated Reading Time: 31 min  |  Author: Lilian Weng\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "Table of Contents\n",
      "\n",
      "\n",
      "\n",
      "Agent System Overview\n",
      "\n",
      "Component One: Planning\n",
      "\n",
      "Task Decomposition\n",
      "\n",
      "Self-Reflection\n",
      "\n",
      "\n",
      "Component Two: Memory\n",
      "\n",
      "Types of Memory\n",
      "\n",
      "Maximum Inner Product Search (MIPS)\n"
     ]
    }
   ],
   "source": [
    "# chunking based on length\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=500, chunk_overlap=0\n",
    ")\n",
    "docs = text_splitter.split_documents(data)\n",
    "\n",
    "\n",
    "print(f\"splits: {len(docs)}\")\n",
    "print(f\"first doc content: {docs[0].page_content}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "splits: 22\n",
      "first doc content: \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "LLM Powered Autonomous Agents | Lil'Log\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Lil'Log\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Posts\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Archive\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Search\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Tags\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "FAQ\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "emojisearch.app\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "      LLM Powered Autonomous Agents\n",
      "    \n",
      "Date: June 23, 2023  |  Estimated Reading Time: 31 min  |  Author: Lilian Weng\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "Table of Contents\n",
      "\n",
      "\n",
      "\n",
      "Agent System Overview\n",
      "\n",
      "Component One: Planning\n",
      "\n",
      "Task Decomposition\n",
      "\n",
      "Self-Reflection\n",
      "\n",
      "\n",
      "Component Two: Memory\n",
      "\n",
      "Types of Memory\n",
      "\n",
      "Maximum Inner Product Search (MIPS)\n",
      "\n",
      "\n",
      "Component Three: Tool Use\n",
      "\n",
      "Case Studies\n",
      "\n",
      "Scientific Discovery Agent\n",
      "\n",
      "Generative Agents Simulation\n",
      "\n",
      "Proof-of-Concept Examples\n",
      "\n",
      "\n",
      "Challenges\n",
      "\n",
      "Citation\n",
      "\n",
      "References\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver. Agent System Overview#\n",
      "In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:\n",
      "\n",
      "Planning\n",
      "\n",
      "Subgoal and decomposition: The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks. Reflection and refinement: The agent can do self-criticism and self-reflection over past actions, learn from mistakes and refine them for future steps, thereby improving the quality of final results. Memory\n",
      "\n",
      "Short-term memory: I would consider all the in-context learning (See Prompt Engineering) as utilizing short-term memory of the model to learn. Long-term memory: This provides the agent with the capability to retain and recall (infinite) information over extended periods, often by leveraging an external vector store and fast retrieval. Tool use\n",
      "\n",
      "The agent learns to call external APIs for extra information that is missing from the model weights (often hard to change after pre-training), including current information, code execution capability, access to proprietary information sources and more. Fig.\n"
     ]
    }
   ],
   "source": [
    "# with semantic chunking\n",
    "\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_experimental.text_splitter import SemanticChunker\n",
    "from langchain_community.embeddings import CohereEmbeddings\n",
    "\n",
    "text_splitter = SemanticChunker(CohereEmbeddings())\n",
    "docs = text_splitter.split_documents(data)\n",
    "\n",
    "print(f\"splits: {len(docs)}\")\n",
    "print(f\"first doc content: {docs[0].page_content}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.vectorstores import Chroma\n",
    "\n",
    "vectordb = Chroma.from_documents(\n",
    "    documents=docs,\n",
    "    embedding=CohereEmbeddings()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.retrievers.multi_query import MultiQueryRetriever\n",
    "from langchain_community.chat_models import BedrockChat\n",
    "import boto3\n",
    "\n",
    "llm = BedrockChat(\n",
    "    model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    client=boto3.client(\"bedrock-runtime\"),\n",
    "    model_kwargs={\"temperature\": 0.0, \"max_tokens\":512}\n",
    ")\n",
    "\n",
    "retriever_from_llm = MultiQueryRetriever.from_llm(\n",
    "    retriever=vectordb.as_retriever(), llm=llm\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "logging.basicConfig()\n",
    "logging.getLogger(\"langchain.retrievers.multi_query\").setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:langchain.retrievers.multi_query:Generated queries: ['Here are three different versions of the original question to help retrieve relevant documents from a vector database:', '', 'What are the different techniques or methods used for Task Decomposition?', '', 'Approaches to breaking down complex tasks into smaller, more manageable subtasks.', '', 'Strategies and frameworks for decomposing tasks into hierarchical or modular components.']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 153 ms, sys: 9.34 ms, total: 162 ms\n",
      "Wall time: 2.14 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Document(page_content='8. Categorization of human memory. We can roughly consider the following mappings:\\n\\nSensory memory as learning embedding representations for raw inputs, including text, image or other modalities;\\nShort-term memory as in-context learning. It is short and finite, as it is restricted by the finite context window length of Transformer. Long-term memory as the external vector store that the agent can attend to at query time, accessible via fast retrieval. Maximum Inner Product Search (MIPS)#\\nThe external memory can alleviate the restriction of finite attention span. A standard practice is to save the embedding representation of information into a vector store database that can support fast maximum inner-product search (MIPS). To optimize the retrieval speed, the common choice is the approximate nearest neighbors (ANN)\\u200b algorithm to return approximately top k nearest neighbors to trade off a little accuracy lost for a huge speedup. A couple common choices of ANN algorithms for fast MIPS:\\n\\nLSH (Locality-Sensitive Hashing): It introduces a hashing function such that similar input items are mapped to the same buckets with high probability, where the number of buckets is much smaller than the number of inputs. ANNOY (Approximate Nearest Neighbors Oh Yeah): The core data structure are random projection trees, a set of binary trees where each non-leaf node represents a hyperplane splitting the input space into half and each leaf stores one data point. Trees are built independently and at random, so to some extent, it mimics a hashing function. ANNOY search happens in all the trees to iteratively search through the half that is closest to the query and then aggregates the results. The idea is quite related to KD tree but a lot more scalable. HNSW (Hierarchical Navigable Small World): It is inspired by the idea of small world networks where most nodes can be reached by any other nodes within a small number of steps; e.g. “six degrees of separation” feature of social networks. HNSW builds hierarchical layers of these small-world graphs, where the bottom layers contain the actual data points. The layers in the middle create shortcuts to speed up search. When performing a search, HNSW starts from a random node in the top layer and navigates towards the target. When it can’t get any closer, it moves down to the next layer, until it reaches the bottom layer. Each move in the upper layers can potentially cover a large distance in the data space, and each move in the lower layers refines the search quality. FAISS (Facebook AI Similarity Search): It operates on the assumption that in high dimensional space, distances between nodes follow a Gaussian distribution and thus there should exist clustering of data points. FAISS applies vector quantization by partitioning the vector space into clusters and then refining the quantization within clusters. Search first looks for cluster candidates with coarse quantization and then further looks into each cluster with finer quantization. ScaNN (Scalable Nearest Neighbors): The main innovation in ScaNN is anisotropic vector quantization. It quantizes a data point $x_i$ to $\\\\tilde{x}_i$ such that the inner product $\\\\langle q, x_i \\\\rangle$ is as similar to the original distance of $\\\\angle q, \\\\tilde{x}_i$ as possible, instead of picking the closet quantization centroid points. Fig.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content='4. Experiments on AlfWorld Env and HotpotQA. Hallucination is a more common failure than inefficient planning in AlfWorld. (Image source: Shinn & Labash, 2023)\\nChain of Hindsight (CoH; Liu et al. 2023) encourages the model to improve on its own outputs by explicitly presenting it with a sequence of past outputs, each annotated with feedback. Human feedback data is a collection of $D_h = \\\\{(x, y_i , r_i , z_i)\\\\}_{i=1}^n$, where $x$ is the prompt, each $y_i$ is a model completion, $r_i$ is the human rating of $y_i$, and $z_i$ is the corresponding human-provided hindsight feedback. Assume the feedback tuples are ranked by reward, $r_n \\\\geq r_{n-1} \\\\geq \\\\dots \\\\geq r_1$ The process is supervised fine-tuning where the data is a sequence in the form of $\\\\tau_h = (x, z_i, y_i, z_j, y_j, \\\\dots, z_n, y_n)$, where $\\\\leq i \\\\leq j \\\\leq n$. The model is finetuned to only predict $y_n$ where conditioned on the sequence prefix, such that the model can self-reflect to produce better output based on the feedback sequence. The model can optionally receive multiple rounds of instructions with human annotators at test time. To avoid overfitting, CoH adds a regularization term to maximize the log-likelihood of the pre-training dataset. To avoid shortcutting and copying (because there are many common words in feedback sequences), they randomly mask 0% - 5% of past tokens during training. The training dataset in their experiments is a combination of WebGPT comparisons, summarization from human feedback and human preference dataset. Fig.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content='9. Comparison of MIPS algorithms, measured in recall@10. (Image source: Google Blog, 2020)\\nCheck more MIPS algorithms and performance comparison in ann-benchmarks.com. Component Three: Tool Use#\\nTool use is a remarkable and distinguishing characteristic of human beings. We create, modify and utilize external objects to do things that go beyond our physical and cognitive limits. Equipping LLMs with external tools can significantly extend the model capabilities. Fig.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content='6. Illustration of how Algorithm Distillation (AD) works. (Image source: Laskin et al.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content=\"[12] Li et al. “API-Bank: A Benchmark for Tool-Augmented LLMs” arXiv preprint arXiv:2304.08244 (2023). [13] Shen et al. “HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace” arXiv preprint arXiv:2303.17580 (2023). [14] Bran et al. “ChemCrow: Augmenting large-language models with chemistry tools.” arXiv preprint arXiv:2304.05376 (2023). [15] Boiko et al. “Emergent autonomous scientific research capabilities of large language models.” arXiv preprint arXiv:2304.05332 (2023). [16] Joon Sung Park, et al. “Generative Agents: Interactive Simulacra of Human Behavior.” arXiv preprint arXiv:2304.03442 (2023). [17] AutoGPT. https://github.com/Significant-Gravitas/Auto-GPT\\n[18] GPT-Engineer. https://github.com/AntonOsika/gpt-engineer\\n\\n\\n\\nnlp\\nlanguage-model\\nagent\\nsteerability\\nprompting\\n\\n\\n\\n« \\n\\nAdversarial Attacks on LLMs\\n\\n\\n »\\n\\nPrompt Engineering\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n© 2024 Lil'Log\\n\\n        Powered by\\n        Hugo &\\n        PaperMod\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\", metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content='2023). The paper hypothesizes that any algorithm that generates a set of learning histories can be distilled into a neural network by performing behavioral cloning over actions. The history data is generated by a set of source policies, each trained for a specific task. At the training stage, during each RL run, a random task is sampled and a subsequence of multi-episode history is used for training, such that the learned policy is task-agnostic. In reality, the model has limited context window length, so episodes should be short enough to construct multi-episode history. Multi-episodic contexts of 2-4 episodes are necessary to learn a near-optimal in-context RL algorithm. The emergence of in-context RL requires long enough context. In comparison with three baselines, including ED (expert distillation, behavior cloning with expert trajectories instead of learning history), source policy (used for generating trajectories for distillation by UCB), RL^2 (Duan et al. 2017; used as upper bound since it needs online RL), AD demonstrates in-context RL with performance getting close to RL^2 despite only using offline RL and learns much faster than other baselines. When conditioned on partial training history of the source policy, AD also improves much faster than ED baseline. Fig. 7. Comparison of AD, ED, source policy and RL^2 on environments that require memory and exploration. Only binary reward is assigned. The source policies are trained with A3C for \"dark\" environments and DQN for watermaze.(Image source: Laskin et al. 2023)\\nComponent Two: Memory#\\n(Big thank you to ChatGPT for helping me draft this section. I’ve learned a lot about the human brain and data structure for fast MIPS in my conversations with ChatGPT.)\\nTypes of Memory#\\nMemory can be defined as the processes used to acquire, store, retain, and later retrieve information. There are several types of memory in human brains. Sensory Memory: This is the earliest stage of memory, providing the ability to retain impressions of sensory information (visual, auditory, etc) after the original stimuli have ended. Sensory memory typically only lasts for up to a few seconds. Subcategories include iconic memory (visual), echoic memory (auditory), and haptic memory (touch). Short-Term Memory (STM) or Working Memory: It stores information that we are currently aware of and needed to carry out complex cognitive tasks such as learning and reasoning. Short-term memory is believed to have the capacity of about 7 items (Miller 1956) and lasts for 20-30 seconds. Long-Term Memory (LTM): Long-term memory can store information for a remarkably long time, ranging from a few days to decades, with an essentially unlimited storage capacity. There are two subtypes of LTM:\\n\\nExplicit / declarative memory: This is memory of facts and events, and refers to those memories that can be consciously recalled, including episodic memory (events and experiences) and semantic memory (facts and concepts). Implicit / procedural memory: This type of memory is unconscious and involves skills and routines that are performed automatically, like riding a bike or typing on a keyboard. Fig.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content='2. Long Term memory management. 3. GPT-3.5 powered Agents for delegation of simple tasks. 4.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content='1. Overview of a LLM-powered autonomous agent system. Component One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead. Task Decomposition#\\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process. Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote. Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs. Another quite distinct approach, LLM+P (Liu et al. 2023), involves relying on an external classical planner to do long-horizon planning. This approach utilizes the Planning Domain Definition Language (PDDL) as an intermediate interface to describe the planning problem. In this process, LLM (1) translates the problem into “Problem PDDL”, then (2) requests a classical planner to generate a PDDL plan based on an existing “Domain PDDL”, and finally (3) translates the PDDL plan back into natural language. Essentially, the planning step is outsourced to an external tool, assuming the availability of domain-specific PDDL and a suitable planner which is common in certain robotic setups but not in many other domains. Self-Reflection#\\nSelf-reflection is a vital aspect that allows autonomous agents to improve iteratively by refining past action decisions and correcting previous mistakes. It plays a crucial role in real-world tasks where trial and error are inevitable. ReAct (Yao et al. 2023) integrates reasoning and acting within LLM by extending the action space to be a combination of task-specific discrete actions and the language space. The former enables LLM to interact with the environment (e.g. use Wikipedia search API), while the latter prompting LLM to generate reasoning traces in natural language. The ReAct prompt template incorporates explicit steps for LLM to think, roughly formatted as:\\nThought: ... Action: ... Observation: ... ... (Repeated many times)\\n\\nFig. 2. Examples of reasoning trajectories for knowledge-intensive tasks (e.g. HotpotQA, FEVER) and decision-making tasks (e.g. AlfWorld Env, WebShop). (Image source: Yao et al.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content='schedule group meetings, book flight/hotel/restaurant for a trip), the model may have to conduct multiple API calls to solve it. Case Studies#\\nScientific Discovery Agent#\\nChemCrow (Bran et al. 2023) is a domain-specific example in which LLM is augmented with 13 expert-designed tools to accomplish tasks across organic synthesis, drug discovery, and materials design. The workflow, implemented in LangChain, reflects what was previously described in the ReAct and MRKLs and combines CoT reasoning with tools relevant to the tasks:\\n\\nThe LLM is provided with a list of tool names, descriptions of their utility, and details about the expected input/output. It is then instructed to answer a user-given prompt using the tools provided when necessary. The instruction suggests the model to follow the ReAct format - Thought, Action, Action Input, Observation. One interesting observation is that while the LLM-based evaluation concluded that GPT-4 and ChemCrow perform nearly equivalently, human evaluations with experts oriented towards the completion and chemical correctness of the solutions showed that ChemCrow outperforms GPT-4 by a large margin. This indicates a potential problem with using LLM to evaluate its own performance on domains that requires deep expertise. The lack of expertise may cause LLMs not knowing its flaws and thus cannot well judge the correctness of task results. Boiko et al. (2023) also looked into LLM-empowered agents for scientific discovery, to handle autonomous design, planning, and performance of complex scientific experiments. This agent can use tools to browse the Internet, read documentation, execute code, call robotics experimentation APIs and leverage other LLMs. For example, when requested to \"develop a novel anticancer drug\", the model came up with the following reasoning steps:\\n\\ninquired about current trends in anticancer drug discovery;\\nselected a target;\\nrequested a scaffold targeting these compounds;\\nOnce the compound was identified, the model attempted its synthesis. They also discussed the risks, especially with illicit drugs and bioweapons. They developed a test set containing a list of known chemical weapon agents and asked the agent to synthesize them. 4 out of 11 requests (36%) were accepted to obtain a synthesis solution and the agent attempted to consult documentation to execute the procedure. 7 out of 11 were rejected and among these 7 rejected cases, 5 happened after a Web search while 2 were rejected based on prompt only. Generative Agents Simulation#\\nGenerative Agents (Park, et al. 2023) is super fun experiment where 25 virtual characters, each controlled by a LLM-powered agent, are living and interacting in a sandbox environment, inspired by The Sims. Generative agents create believable simulacra of human behavior for interactive applications. The design of generative agents combines LLM with memory, planning and reflection mechanisms to enable agents to behave conditioned on past experience, as well as to interact with other agents. Memory stream: is a long-term memory module (external database) that records a comprehensive list of agents’ experience in natural language. Each element is an observation, an event directly provided by the agent. - Inter-agent communication can trigger new natural language statements. Retrieval model: surfaces the context to inform the agent’s behavior, according to relevance, recency and importance. Recency: recent events have higher scores\\nImportance: distinguish mundane from core memories. Ask LM directly. Relevance: based on how related it is to the current situation / query. Reflection mechanism: synthesizes memories into higher level inferences over time and guides the agent’s future behavior. They are higher-level summaries of past events (<- note that this is a bit different from self-reflection above)\\n\\nPrompt LM with 100 most recent observations and to generate 3 most salient high-level questions given a set of observations/statements. Then ask LM to answer those questions. Planning & Reacting: translate the reflections and the environment information into actions\\n\\nPlanning is essentially in order to optimize believability at the moment vs in time. Prompt template: {Intro of an agent X}. Here is X\\'s plan today in broad strokes: 1)\\nRelationships between agents and observations of one agent by another are all taken into consideration for planning and reacting. Environment information is present in a tree structure. Fig.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content='File output. Performance Evaluation:\\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities. 2. Constructively self-criticize your big-picture behavior constantly. 3. Reflect on past decisions and strategies to refine your approach. 4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps. You should only respond in JSON format as described below\\nResponse Format:\\n{\\n    \"thoughts\": {\\n        \"text\": \"thought\",\\n        \"reasoning\": \"reasoning\",\\n        \"plan\": \"- short bulleted\\\\n- list that conveys\\\\n- long-term plan\",\\n        \"criticism\": \"constructive self-criticism\",\\n        \"speak\": \"thoughts summary to say to user\"\\n    },\\n    \"command\": {\\n        \"name\": \"command name\",\\n        \"args\": {\\n            \"arg name\": \"value\"\\n        }\\n    }\\n}\\nEnsure the response can be parsed by Python json.loads\\nGPT-Engineer is another project to create a whole repository of code given a task specified in natural language. The GPT-Engineer is instructed to think over a list of smaller components to build and ask for user input to clarify questions as needed. Here are a sample conversation for task clarification sent to OpenAI ChatCompletion endpoint used by GPT-Engineer. The user inputs are wrapped in {{user input text}}. [\\n  {\\n    \"role\": \"system\",\\n    \"content\": \"You will read instructions and not carry them out, only seek to clarify them.\\\\nSpecifically you will first summarise a list of super short bullets of areas that need clarification.\\\\nThen you will pick one clarifying question, and wait for an answer from the user.\\\\n\"\\n  },\\n  {\\n    \"role\": \"user\",\\n    \"content\": \"We are writing {{a Super Mario game in python. MVC components split in separate files. Keyboard control.}}\\\\n\"\\n  },\\n  {\\n    \"role\": \"assistant\",\\n    \"content\": \"Summary of areas that need clarification:\\\\n1. Specifics of the Super Mario game (e.g. level design, characters, gameplay mechanics)\\\\n2. Details about the MVC components (e.g. which components are in each file)\\\\n3. Keyboard control implementation (e.g. which keys to use, how to handle input)\\\\n\\\\nClarifying question:\\\\nCan you provide more details about the Super Mario game, such as level design, characters, and gameplay mechanics?\"\\n  },\\n  {\\n    \"role\": \"user\",\\n    \"content\": \"{{There are 10 levels in total. The main character is a plumber named Mario, who can walk and jump. It is a classical platform game just like Super Mario. The main character moves from left to right, trying to get to the destination, where there are many obstacles and attacks from enemies in the process.}}\\\\n\\\\nIs anything else unclear? If yes, only answer in the form:\\\\n{remaining unclear areas} remaining questions.\\\\n{Next question}\\\\nIf everything is sufficiently clear, only answer \\\\\"Nothing more to clarify.\\\\\".\"\\n  },\\n  {\\n    \"role\": \"assistant\",\\n    \"content\": \"Remaining unclear areas: 2 remaining questions.\\\\nCan you provide more information about how the MVC components are split into separate files?\"\\n  },\\n  {\\n    \"role\": \"user\",\\n    \"content\": \"{{Make your own assumptions and state them explicitly before starting}}\"\\n  }\\n]\\nThen after these clarification, the agent moved into the code writing mode with a different system message. System message:\\n\\nYou will get instructions for code to write. You will write a very long answer. Make sure that every detail of the architecture is, in the end, implemented as code. Make sure that every detail of the architecture is, in the end, implemented as code. Think step by step and reason yourself to the right decisions to make sure we get it right. You will first lay out the names of the core classes, functions, methods that will be necessary, as well as a quick comment on their purpose. Then you will output the content of each file including ALL code. Each file must strictly follow a markdown code block format, where the following tokens must be replaced such that\\nFILENAME is the lowercase file name including the file extension,\\nLANG is the markup code block language for the code’s language, and CODE is the code:\\nFILENAME\\nCODE\\nYou will start with the “entrypoint” file, then go to the ones that are imported by that file, and so on. Please note that the code should be fully functional. No placeholders. Follow a language and framework appropriate best practice file naming convention. Make sure that files contain all imports, types etc. Make sure that code in different files are compatible with each other. Ensure to implement all code, if you are unsure, write a plausible implementation. Include module dependency or package manager dependency definition file. Before you finish, double check that all parts of the architecture is present in the files. Useful to know:\\nYou almost always put different classes in different files. For Python, you always create an appropriate requirements.txt file. For NodeJS, you always create an appropriate package.json file. You always add a comment briefly describing the purpose of the function definition. You try to add comments explaining very complex bits of logic. You always follow the best practices for the requested languages in terms of describing the code written as a defined\\npackage/project. Python toolbelt preferences:\\n\\npytest\\ndataclasses\\n\\n\\nConversatin samples:\\n[\\n  {\\n    \"role\": \"system\",\\n    \"content\": \"You will get instructions for code to write.\\\\nYou will write a very long answer. Make sure that every detail of the architecture is, in the end, implemented as code.\\\\nMake sure that every detail of the architecture is, in the end, implemented as code.\\\\n\\\\nThink step by step and reason yourself to the right decisions to make sure we get it right.\\\\nYou will first lay out the names of the core classes, functions, methods that will be necessary, as well as a quick comment on their purpose.\\\\n\\\\nThen you will output the content of each file including ALL code.\\\\nEach file must strictly follow a markdown code block format, where the following tokens must be replaced such that\\\\nFILENAME is the lowercase file name including the file extension,\\\\nLANG is the markup code block language for the code\\'s language, and CODE is the code:\\\\n\\\\nFILENAME\\\\n```LANG\\\\nCODE\\\\n```\\\\n\\\\nYou will start with the \\\\\"entrypoint\\\\\" file, then go to the ones that are imported by that file, and so on.\\\\nPlease note that the code should be fully functional. No placeholders.\\\\n\\\\nFollow a language and framework appropriate best practice file naming convention.\\\\nMake sure that files contain all imports, types etc. Make sure that code in different files are compatible with each other.\\\\nEnsure to implement all code, if you are unsure, write a plausible implementation.\\\\nInclude module dependency or package manager dependency definition file.\\\\nBefore you finish, double check that all parts of the architecture is present in the files.\\\\n\\\\nUseful to know:\\\\nYou almost always put different classes in different files.\\\\nFor Python, you always create an appropriate requirements.txt file.\\\\nFor NodeJS, you always create an appropriate package.json file.\\\\nYou always add a comment briefly describing the purpose of the function definition.\\\\nYou try to add comments explaining very complex bits of logic.\\\\nYou always follow the best practices for the requested languages in terms of describing the code written as a defined\\\\npackage/project.\\\\n\\\\n\\\\nPython toolbelt preferences:\\\\n- pytest\\\\n- dataclasses\\\\n\"\\n  },\\n #  … same conversation as earlier, ended with \"Make your own assumptions and state them explicitly before starting\". {\\n    \"role\": \"assistant\",\\n    \"content\": \"Assumptions:\\\\n1.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n",
       " Document(page_content='Model: The model will contain the game\\'s data, such as level information, character states, and enemy positions.\\\\n2. View: The view will handle the game\\'s visuals, including rendering the game objects, backgrounds, and updating the display.\\\\n3. Controller: The controller will manage user input, such as keyboard controls, and update the model accordingly.\\\\n\\\\nWith these assumptions in mind, is there any specific aspect of the keyboard control implementation you would like me to clarify?\"\\n  },\\n  {\\n    \"role\": \"user\",\\n    \"content\": \"Please now remember the steps:\\\\n\\\\nThink step by step and reason yourself to the right decisions to make sure we get it right.\\\\nFirst lay out the names of the core classes, functions, methods that will be necessary, As well as a quick comment on their purpose.\\\\n\\\\nThen you will output the content of each file including ALL code.\\\\nEach file must strictly follow a markdown code block format, where the following tokens must be replaced such that\\\\nFILENAME is the lowercase file name including the file extension,\\\\nLANG is the markup code block language for the code\\'s language, and CODE is the code:\\\\n\\\\nFILENAME\\\\n```LANG\\\\nCODE\\\\n```\\\\n\\\\nPlease note that the code should be fully functional. No placeholders.\\\\n\\\\nYou will start with the \\\\\"entrypoint\\\\\" file, then go to the ones that are imported by that file, and so on.\\\\nFollow a language and framework appropriate best practice file naming convention.\\\\nMake sure that files contain all imports, types etc. The code should be fully functional. Make sure that code in different files are compatible with each other.\\\\nBefore you finish, double check that all parts of the architecture is present in the files.\\\\n\"\\n  }\\n]\\nChallenges#\\nAfter going through key ideas and demos of building LLM-centered agents, I start to see a couple common limitations:\\n\\n\\nFinite context length: The restricted context capacity limits the inclusion of historical information, detailed instructions, API call context, and responses. The design of the system has to work with this limited communication bandwidth, while mechanisms like self-reflection to learn from past mistakes would benefit a lot from long or infinite context windows. Although vector stores and retrieval can provide access to a larger knowledge pool, their representation power is not as powerful as full attention. Challenges in long-term planning and task decomposition: Planning over a lengthy history and effectively exploring the solution space remain challenging. LLMs struggle to adjust plans when faced with unexpected errors, making them less robust compared to humans who learn from trial and error. Reliability of natural language interface: Current agent system relies on natural language as an interface between LLMs and external components such as memory and tools. However, the reliability of model outputs is questionable, as LLMs may make formatting errors and occasionally exhibit rebellious behavior (e.g. refuse to follow an instruction).', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"})]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "question = \"what are the approaches to Task Decomposition?\"\n",
    "response = retriever_from_llm.get_relevant_documents(query=question)\n",
    "\n",
    "len(response)\n",
    "\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# custom prompt \n",
    "\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from typing import List\n",
    "from langchain.schema.messages import AIMessage\n",
    "\n",
    "prompt_template = \"\"\"You are an AI language model assistant. Your task is to generate five different versions of the given user question to retrieve relevant documents from a vector database.\n",
    "By generating multiple perspectives on the user question, your goal is to help the user overcome some of the limitations of the distance-based similarity search.\n",
    "Provide these alternative questions within <question> XML tags. Entire output should be within <answer> XML tags\n",
    "Original question: {question}\"\"\"\n",
    "\n",
    "prompt = ChatPromptTemplate.from_template(template=prompt_template)\n",
    "llm = BedrockChat(\n",
    "    model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    client=boto3.client(\"bedrock-runtime\"),\n",
    "    model_kwargs={\"temperature\": 0.0, \"max_tokens\":512}\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 26.1 ms, sys: 45 µs, total: 26.1 ms\n",
      "Wall time: 1.96 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['What are the different techniques for breaking down a complex task into smaller, more manageable subtasks?',\n",
       " 'What are the common methods used to decompose a high-level task into a series of lower-level steps or actions?',\n",
       " 'How can a complex problem be divided into smaller, more easily solvable components or subproblems?',\n",
       " 'What are the various strategies or frameworks for partitioning a larger task into a set of interdependent subtasks?',\n",
       " 'What are the approaches or principles that can be applied to break down a complex task into a hierarchical structure of subtasks?']"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# approach 1\n",
    "\n",
    "question = \"What are the approaches to Task Decomposition?\"\n",
    "\n",
    "output_parser = XMLOutputParser(tags=[\"answer\"])\n",
    "query_chain = prompt | llm | output_parser\n",
    "\n",
    "response = query_chain.invoke({\"question\": question})\n",
    "\n",
    "qns = [x[\"question\"] for x in response[\"answer\"]]\n",
    "qns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27.4 ms, sys: 4.14 ms, total: 31.5 ms\n",
      "Wall time: 1.86 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['What are the different techniques for breaking down complex tasks into smaller, more manageable subtasks?',\n",
       " 'What are the common methods used to decompose complex problems into simpler, more easily solvable components?',\n",
       " 'How can a complex task be divided into a series of smaller, more specific sub-tasks to facilitate efficient problem-solving?',\n",
       " 'What are the various strategies employed to break down a larger objective into a set of more granular, actionable steps?',\n",
       " 'What are the established approaches for partitioning a complex problem into a hierarchy of more manageable sub-problems?']"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# approach 2\n",
    "\n",
    "question = \"What are the approaches to Task Decomposition?\"\n",
    "\n",
    "def parse(ai_message: AIMessage) -> List[str]:\n",
    "    text = ai_message.content\n",
    "    output_parser = XMLOutputParser(tags=[\"answer\"])\n",
    "    response = output_parser.parse(text)\n",
    "    qns = [x[\"question\"] for x in response[\"answer\"]]\n",
    "    return qns\n",
    "\n",
    "\n",
    "query_chain = prompt | llm | parse\n",
    "\n",
    "response = query_chain.invoke({\"question\": question})\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValidationError",
     "evalue": "5 validation errors for MultiQueryRetriever\nllm_chain -> prompt\n  field required (type=value_error.missing)\nllm_chain -> llm\n  field required (type=value_error.missing)\nllm_chain -> first\n  extra fields not permitted (type=value_error.extra)\nllm_chain -> last\n  extra fields not permitted (type=value_error.extra)\nllm_chain -> middle\n  extra fields not permitted (type=value_error.extra)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValidationError\u001b[0m                           Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[65], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m retriever \u001b[38;5;241m=\u001b[39m \u001b[43mMultiQueryRetriever\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m      2\u001b[0m \u001b[43m    \u001b[49m\u001b[43mretriever\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mvectordb\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mas_retriever\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m      3\u001b[0m \u001b[43m    \u001b[49m\u001b[43mllm_chain\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mquery_chain\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m      4\u001b[0m \u001b[43m)\u001b[49m\n\u001b[1;32m      6\u001b[0m question \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mWhat are the approaches to Task Decomposition?\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m      7\u001b[0m response \u001b[38;5;241m=\u001b[39m retriever\u001b[38;5;241m.\u001b[39mget_relevant_documents(query\u001b[38;5;241m=\u001b[39mquestion)\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain_core/load/serializable.py:120\u001b[0m, in \u001b[0;36mSerializable.__init__\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m    119\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__init__\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs: Any) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 120\u001b[0m     \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[38;5;21;43m__init__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    121\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lc_kwargs \u001b[38;5;241m=\u001b[39m kwargs\n",
      "File \u001b[0;32m/opt/conda/envs/pytorch/lib/python3.10/site-packages/pydantic/main.py:341\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.__init__\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mValidationError\u001b[0m: 5 validation errors for MultiQueryRetriever\nllm_chain -> prompt\n  field required (type=value_error.missing)\nllm_chain -> llm\n  field required (type=value_error.missing)\nllm_chain -> first\n  extra fields not permitted (type=value_error.extra)\nllm_chain -> last\n  extra fields not permitted (type=value_error.extra)\nllm_chain -> middle\n  extra fields not permitted (type=value_error.extra)"
     ]
    }
   ],
   "source": [
    "retriever = MultiQueryRetriever(\n",
    "    retriever=vectordb.as_retriever(),\n",
    "    llm_chain=query_chain,\n",
    ")\n",
    "\n",
    "question = \"What are the approaches to Task Decomposition?\"\n",
    "response = retriever.get_relevant_documents(query=question)\n",
    "len(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mInit signature:\u001b[0m\n",
      "\u001b[0mMultiQueryRetriever\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mtags\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mmetadata\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mDict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mretriever\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mlangchain_core\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mretrievers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mBaseRetriever\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mllm_chain\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mlangchain\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchains\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mllm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLLMChain\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mverbose\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mparser_key\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'lines'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0minclude_original\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mDocstring:\u001b[0m     \n",
      "Given a query, use an LLM to write a set of queries.\n",
      "\n",
      "Retrieve docs for each query. Return the unique union of all retrieved docs.\n",
      "\u001b[0;31mInit docstring:\u001b[0m\n",
      "Create a new model by parsing and validating input data from keyword arguments.\n",
      "\n",
      "Raises ValidationError if the input data cannot be parsed to form a valid model.\n",
      "\u001b[0;31mFile:\u001b[0m           /opt/conda/envs/pytorch/lib/python3.10/site-packages/langchain/retrievers/multi_query.py\n",
      "\u001b[0;31mType:\u001b[0m           ModelMetaclass\n",
      "\u001b[0;31mSubclasses:\u001b[0m     "
     ]
    }
   ],
   "source": [
    "# llm_chain should be of LLMChain type\n",
    "\n",
    "MultiQueryRetriever?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Contextual Compression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Idea is simple: instead of immediately returning retrieved documents as-is, compress them using context of the given query, so that only the relevant information is returned.\n",
    "- Compressing means both compressing the contents of an individual document and filtering out documents wholesale.\n",
    "- Need 2 things:\n",
    "    - a base retriever\n",
    "    - a document Compressor\n",
    "- Flow:\n",
    "    - `input_query -> Contextual Compression Retriever (pass query to base retriever) -> inital docs -> Document Compressor -> shortened docs (final ans)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.retrievers import ContextualCompressionRetriever\n",
    "from langchain.retrievers.document_compressors import LLMChainExtractor\n",
    "\n",
    "# LLMChainExtractor, which will iterate over the initially returned documents and extract from each only the content that is relevant to the query.\n",
    "compressor = LLMChainExtractor.from_llm(llm)\n",
    "compression_retriever = ContextualCompressionRetriever(\n",
    "    base_compressor=compressor,\n",
    "    base_retriever=retriever\n",
    ")\n",
    "compressed_docs = compression_retriever.get_relevant_documents(\n",
    "    \"What did the president say about Ketanji Jackson Brown\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The LLMChainFilter is slightly simpler but more robust compressor that uses an LLM chain to decide which of the initially retrieved documents to filter out and which ones to return, without manipulating the document contents.\n",
    "from langchain.retrievers.document_compressors import LLMChainFilter\n",
    "\n",
    "_filter = LLMChainFilter.from_llm(llm)\n",
    "compression_retriever = ContextualCompressionRetriever(\n",
    "    base_compressor=_filter, base_retriever=retriever\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# EmbeddingsFilter\n",
    "# Making an extra LLM call over each retrieved document is expensive and slow. The EmbeddingsFilter provides a cheaper and faster option by embedding the documents and query and only returning those documents which have sufficiently similar embeddings to the query.\n",
    "\n",
    "from langchain.retrievers.document_compressors import EmbeddingsFilter\n",
    "from langchain_community.embeddings import CohereEmbeddings\n",
    "\n",
    "embeddings_filter = EmbeddingsFilter(embeddings=CohereEmbeddings(), similarity_threshold=0.76)\n",
    "compression_retriever = ContextualCompressionRetriever(\n",
    "    base_compressor=embeddings_filter, base_retriever=retriever\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# stringing compressors and document transformers together\n",
    "\n",
    "# use DocumentCompressorPipeline - use this to combine multiple compressors in sequence.\n",
    "# Along with compressors we can add BaseDocumentTransformers to our pipeline, \n",
    "# which don’t perform any contextual compression but simply perform some transformation \n",
    "# on a set of documents. \n",
    "# For example TextSplitters can be used as document transformers to split documents \n",
    "# into smaller pieces, and the EmbeddingsRedundantFilter can be used to \n",
    "# filter out redundant documents based on embedding similarity between documents.\n",
    "\n",
    "\n",
    "from langchain.retrievers.document_compressors import DocumentCompressorPipeline\n",
    "from langchain_community.document_transformers import EmbeddingsRedundantFilter\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "splitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=0, separator=\". \")\n",
    "redundant_filter = EmbeddingsRedundantFilter(embeddings=embeddings)\n",
    "relevant_filter = EmbeddingsFilter(embeddings=embeddings, similarity_threshold=0.76)\n",
    "pipeline_compressor = DocumentCompressorPipeline(\n",
    "    transformers=[splitter, redundant_filter, relevant_filter]\n",
    ")\n",
    "\n",
    "compression_retriever = ContextualCompressionRetriever(\n",
    "    base_compressor=pipeline_compressor, base_retriever=retriever\n",
    ")\n",
    "\n",
    "compressed_docs = compression_retriever.get_relevant_documents(\n",
    "    \"What did the president say about Ketanji Jackson Brown\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### LongContext Reordering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_transformers import LongContextReorder\n",
    "\n",
    "reordering = LongContextReorder()\n",
    "docs = retriever.get_relevant_documents(query)\n",
    "reordered_docs = reordering.transform_documents(docs)\n",
    "reordered_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Basics]\n",
    "- The core idea of agents is to use a language model to choose a sequence of actions to take. In chains, a sequence of actions is hardcoded (in code). \n",
    "- In agents, a language model is used as a reasoning engine to determine which actions to take and in which order.\n",
    "- agents take a self-determined, input-dependent sequence of steps before returning a user-facing output\n",
    "\n",
    "\n",
    "[AgentExecutor]\n",
    "- Runtime for an agent.\n",
    "```python\n",
    "    next_action = agent.get_action(...)\n",
    "    while next_action != AgentFinish:\n",
    "        observation = run(next_action)\n",
    "        next_action = agent.get_action(..., next_action, observation)\n",
    "    return next_action\n",
    "```\n",
    "- Several complexities this runtime can handle:\n",
    "    - Handling cases where the agent selects a non-existent tool\n",
    "    - Handling cases where the tool errors\n",
    "    - Handling cases where the agent produces output that cannot be parsed into a tool invocation\n",
    "    - Logging and observability at all levels (agent decisions, tool calls) to stdout and/or to LangSmith.\n",
    "\n",
    "[AgentTypes]\n",
    "\n",
    "Dimensions to consider:\n",
    "1. Intended model type - LLMs or ChatModels\n",
    "2. Supports chat history ?\n",
    "3. Supports multi-input tools ?\n",
    "4. Supports parallel function calling\n",
    "    - `having an llm call multiple tools at the same time can greatly speed up agents whether there are tasks that are assisted by doing so. `\n",
    "5. Required model params (OpenAI Function calling)\n",
    "\n",
    "Types:\n",
    "1. OpenAI Tools\n",
    "2. OpenAI Functions (deprecated. openai favors tools than functions)\n",
    "3. XML (read the docs there are few caveats)\n",
    "4. Structured Chat\n",
    "5. JSON Chat\n",
    "6. ReAct\n",
    "7. Self Ask with Search\n",
    "\n",
    "\n",
    "[XML]\n",
    "- use with regular LLMs, not with chat models.\n",
    "- use only with unstructured tools. i.e., tools that accept a single string input.\n",
    "\n",
    "[SelfAsk]\n",
    "- For this agent, only one tool can be used and it needs to be named “Intermediate Answer”\n",
    "\n",
    "[Streaming]\n",
    "- final_answer and/or intermediate_steps\n",
    "\n",
    "[LATER]\n",
    "- Add in the prompt: `Only use a tool if needed, otherwise respond with Final Answer`\n",
    "- Add in the prompt: `Now, complete the below sequence all the way to the \"Final Answer:\" step:`\n",
    "- add in the AgentExecutor: \n",
    "```python\n",
    "    agent_executor = AgentExecutor(\n",
    "        agent=agent,\n",
    "        tools=tools,\n",
    "        verbose=True,\n",
    "        handle_parsing_errors=\"Check your output and make sure it conforms, use the Action/Action Input syntax\",\n",
    "    )\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Who lived longer, Muhammad Ali or Alan Turing?\n",
      "Are follow up questions needed here: Yes.\n",
      "Follow up: How old was Muhammad Ali when he died?\n",
      "Intermediate answer: Muhammad Ali was 74 years old when he died.\n",
      "Follow up: How old was Alan Turing when he died?\n",
      "Intermediate answer: Alan Turing was 41 years old when he died.\n",
      "So the final answer is: Muhammad Ali\n",
      "\n",
      "Question: When was the founder of craigslist born?\n",
      "Are follow up questions needed here: Yes.\n",
      "Follow up: Who was the founder of craigslist?\n",
      "Intermediate answer: Craigslist was founded by Craig Newmark.\n",
      "Follow up: When was Craig Newmark born?\n",
      "Intermediate answer: Craig Newmark was born on December 6, 1952.\n",
      "So the final answer is: December 6, 1952\n",
      "\n",
      "Question: Who was the maternal grandfather of George Washington?\n",
      "Are follow up questions needed here: Yes.\n",
      "Follow up: Who was the mother of George Washington?\n",
      "Intermediate answer: The mother of George Washington was Mary Ball Washington.\n",
      "Follow up: Who was the father of Mary Ball Washington?\n",
      "Intermediate answer: The father of Mary Ball Washington was Joseph Ball.\n",
      "So the final answer is: Joseph Ball\n",
      "\n",
      "Question: Are both the directors of Jaws and Casino Royale from the same country?\n",
      "Are follow up questions needed here: Yes.\n",
      "Follow up: Who is the director of Jaws?\n",
      "Intermediate answer: The director of Jaws is Steven Spielberg.\n",
      "Follow up: Where is Steven Spielberg from?\n",
      "Intermediate answer: The United States.\n",
      "Follow up: Who is the director of Casino Royale?\n",
      "Intermediate answer: The director of Casino Royale is Martin Campbell.\n",
      "Follow up: Where is Martin Campbell from?\n",
      "Intermediate answer: New Zealand.\n",
      "So the final answer is: No\n",
      "\n",
      "Question: \u001b[33;1m\u001b[1;3m{input}\u001b[0m\n",
      "Are followup questions needed here:\u001b[33;1m\u001b[1;3m{agent_scratchpad}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from langchain import hub\n",
    "\n",
    "# openai tools\n",
    "# prompt = hub.pull(\"hwchase17/openai-tools-agent\")\n",
    "\n",
    "# xml\n",
    "# prompt = hub.pull(\"hwchase17/xml-agent-convo\")\n",
    "\n",
    "# structured chat # TODO: somthing to try next (Final Answer problem could be solved with this)\n",
    "# prompt = hub.pull(\"hwchase17/structured-chat-agent\")\n",
    "\n",
    "# json chat\n",
    "# prompt = hub.pull(\"hwchase17/react-chat-json\")\n",
    "\n",
    "# react\n",
    "# prompt = hub.pull(\"hwchase17/react\")\n",
    "\n",
    "# selfask\n",
    "prompt = hub.pull(\"hwchase17/self-ask-with-search\")\n",
    "\n",
    "prompt.pretty_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- past messages/interactions\n",
    "\n",
    "- information introduced earlier in conversation\n",
    "\n",
    "- world model (entities and their relationships) that is constantly updated\n",
    "\n",
    "- `ChatMessageHistory` - production ready\n",
    "\n",
    "- A very simple memory might just retirn the most recent messages each run.\n",
    "\n",
    "- A slightly more complex memory system might return a succint summary of the past K messages. \n",
    "\n",
    "- An even more sophisticated system might extract entities from stored messages and only return information about entities referenced in the current run.\n",
    "\n",
    "[Types]\n",
    "- `ConversationBufferMemory`. This memory allows for storing messages and then extracts the messages in a variable.\n",
    "\n",
    "- `ConversationBufferWindowMemory` keeps a list of the interactions of the conversation over time. It only uses the last K interactions. This can be useful for keeping a sliding window of the most recent interactions, so the buffer does not get too large. \n",
    "\n",
    "- `ConversationEntityMemory`: Entity memory remembers given facts about specific entities in a conversation. It extracts information on entities (using an LLM) and builds up its knowledge about that entity over time (also using an LLM).\n",
    "\n",
    "- `ConversationKGMemory`: uses a knowledge graph to recreate memory.\n",
    "\n",
    "- `ConversationSummaryMemory`: creates a summary of the conversation over time.\n",
    "\n",
    "- `ConversationSummaryBufferMemory` combines the two ideas. It keeps a buffer of recent interactions in memory, but rather than just completely flushing old interactions it compiles them into a summary and uses both. It uses token length rather than number of interactions to determine when to flush interactions.\n",
    "\n",
    "- `ConversationTokenBufferMemory` keeps a buffer of recent interactions in memory, and uses token length rather than number of interactions to determine when to flush interactions.\n",
    "\n",
    "- `VectorStoreRetrieverMemory` stores memories in a vector store and queries the top-K most \"salient\" docs every time it is called. This differs from most of the other Memory classes in that it doesn't explicitly track the order of interactions. In this case, the \"docs\" are previous conversation snippets. This can be useful to refer to relevant pieces of information that the AI was told earlier in the conversation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are an assistant to a human, powered by a large language model trained by OpenAI.\n",
      "\n",
      "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.\n",
      "\n",
      "You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics.\n",
      "\n",
      "Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist.\n",
      "\n",
      "Context:\n",
      "\u001b[33;1m\u001b[1;3m{entities}\u001b[0m\n",
      "\n",
      "Current conversation:\n",
      "\u001b[33;1m\u001b[1;3m{history}\u001b[0m\n",
      "Last line:\n",
      "Human: \u001b[33;1m\u001b[1;3m{input}\u001b[0m\n",
      "You:\n"
     ]
    }
   ],
   "source": [
    "from langchain.memory.prompt import ENTITY_MEMORY_CONVERSATION_TEMPLATE\n",
    "\n",
    "ENTITY_MEMORY_CONVERSATION_TEMPLATE.pretty_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `CallbackHandlers` are objects that implement the `CallbackHandler` interface, which has a method for each event that can be subscribed to. \n",
    "\n",
    "- The `CallbackManager` will call the appropriate method on each handler when the event is triggered.\n",
    "\n",
    "[Types]\n",
    "- `StdOutCallbackHandler`: logs all events to stdout\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BaseCallbackHandler:\n",
    "    \"\"\"Base callback handler that can be used to handle callbacks from langchain.\"\"\"\n",
    "\n",
    "    def on_llm_start(\n",
    "        self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any\n",
    "    ) -> Any:\n",
    "        \"\"\"Run when LLM starts running.\"\"\"\n",
    "\n",
    "    def on_chat_model_start(\n",
    "        self, serialized: Dict[str, Any], messages: List[List[BaseMessage]], **kwargs: Any\n",
    "    ) -> Any:\n",
    "        \"\"\"Run when Chat Model starts running.\"\"\"\n",
    "\n",
    "    def on_llm_new_token(self, token: str, **kwargs: Any) -> Any:\n",
    "        \"\"\"Run on new LLM token. Only available when streaming is enabled.\"\"\"\n",
    "\n",
    "    def on_llm_end(self, response: LLMResult, **kwargs: Any) -> Any:\n",
    "        \"\"\"Run when LLM ends running.\"\"\"\n",
    "\n",
    "    def on_llm_error(\n",
    "        self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any\n",
    "    ) -> Any:\n",
    "        \"\"\"Run when LLM errors.\"\"\"\n",
    "\n",
    "    def on_chain_start(\n",
    "        self, serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any\n",
    "    ) -> Any:\n",
    "        \"\"\"Run when chain starts running.\"\"\"\n",
    "\n",
    "    def on_chain_end(self, outputs: Dict[str, Any], **kwargs: Any) -> Any:\n",
    "        \"\"\"Run when chain ends running.\"\"\"\n",
    "\n",
    "    def on_chain_error(\n",
    "        self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any\n",
    "    ) -> Any:\n",
    "        \"\"\"Run when chain errors.\"\"\"\n",
    "\n",
    "    def on_tool_start(\n",
    "        self, serialized: Dict[str, Any], input_str: str, **kwargs: Any\n",
    "    ) -> Any:\n",
    "        \"\"\"Run when tool starts running.\"\"\"\n",
    "\n",
    "    def on_tool_end(self, output: Any, **kwargs: Any) -> Any:\n",
    "        \"\"\"Run when tool ends running.\"\"\"\n",
    "\n",
    "    def on_tool_error(\n",
    "        self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any\n",
    "    ) -> Any:\n",
    "        \"\"\"Run when tool errors.\"\"\"\n",
    "\n",
    "    def on_text(self, text: str, **kwargs: Any) -> Any:\n",
    "        \"\"\"Run on arbitrary text.\"\"\"\n",
    "\n",
    "    def on_agent_action(self, action: AgentAction, **kwargs: Any) -> Any:\n",
    "        \"\"\"Run on agent action.\"\"\"\n",
    "\n",
    "    def on_agent_finish(self, finish: AgentFinish, **kwargs: Any) -> Any:\n",
    "        \"\"\"Run on agent end.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.callbacks import StdOutCallbackHandler\n",
    "from langchain.chains import LLMChain\n",
    "import boto3\n",
    "from langchain_community.chat_models import BedrockChat\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "handler = StdOutCallbackHandler()\n",
    "\n",
    "llm = BedrockChat(\n",
    "    model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    client=boto3.client(\"bedrock-runtime\"),\n",
    "    model_kwargs={\"temperature\": 0.0, \"max_tokens\":128}\n",
    ")\n",
    "prompt = PromptTemplate.from_template(\"1 + {number} = \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3m1 + 2 = \u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'number': 2, 'text': '3'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Constructor callback: First, let's explicitly set the StdOutCallbackHandler when initializing our chain\n",
    "chain = LLMChain(llm=llm, prompt=prompt, callbacks=[handler])\n",
    "chain.invoke({\"number\":2})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3m1 + 2 = \u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'number': 2, 'text': '3'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use verbose flag: Then, let's use the `verbose` flag to achieve the same result\n",
    "chain = LLMChain(llm=llm, prompt=prompt, verbose=True)\n",
    "chain.invoke({\"number\":2})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3m1 + 2 = \u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'number': 2, 'text': '3'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Request callbacks: Finally, let's use the request `callbacks` to achieve the same result\n",
    "chain = LLMChain(llm=llm, prompt=prompt)\n",
    "chain.invoke({\"number\":2}, {\"callbacks\":[handler]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new RunnableSequence chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new PromptTemplate chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3m1 + text='1 + 2 = ' = \u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'number': StringPromptValue(text='1 + 2 = '),\n",
       " 'text': 'The expression \"1 + text=\\'1 + 2 = \\'\" is not a valid mathematical expression. It appears to be a combination of a numeric value (1) and a string assignment (text=\\'1 + 2 = \\').\\n\\nTo perform a calculation and assign the result to a variable, you would typically write something like this:\\n\\ntext = \\'1 + 2 = \\'\\nresult = 1 + 2\\nprint(text + str(result))\\n\\nThis would output:\\n\\n1 + 2 = 3\\n\\nThe key steps are:\\n\\n1.'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "handler = StdOutCallbackHandler()\n",
    "\n",
    "llm = BedrockChat(\n",
    "    model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    client=boto3.client(\"bedrock-runtime\"),\n",
    "    model_kwargs={\"temperature\": 0.0, \"max_tokens\":128}\n",
    ")\n",
    "\n",
    "prompt = PromptTemplate.from_template(\"1 + {number} = \")\n",
    "\n",
    "config = {\n",
    "    'callbacks' : [handler]\n",
    "}\n",
    "\n",
    "chain = prompt | chain\n",
    "chain.invoke({\"number\":2}, config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My custom handler, token: Here\n",
      "My custom handler, token: 's\n",
      "My custom handler, token:  a\n",
      "My custom handler, token:  silly\n",
      "My custom handler, token:  joke\n",
      "My custom handler, token:  for\n",
      "My custom handler, token:  you\n",
      "My custom handler, token: :\n",
      "My custom handler, token: \n",
      "\n",
      "Why\n",
      "My custom handler, token:  can\n",
      "My custom handler, token: 't\n",
      "My custom handler, token:  a\n",
      "My custom handler, token:  bicycle\n",
      "My custom handler, token:  stand\n",
      "My custom handler, token:  up\n",
      "My custom handler, token:  on\n",
      "My custom handler, token:  its\n",
      "My custom handler, token:  own\n",
      "My custom handler, token: ?\n",
      "My custom handler, token:  Because\n",
      "My custom handler, token:  it\n",
      "My custom handler, token: 's\n",
      "My custom handler, token:  two\n",
      "My custom handler, token: -\n",
      "My custom handler, token: tired\n",
      "My custom handler, token: !\n",
      "My custom handler, token: \n",
      "\n",
      "How\n",
      "My custom handler, token:  was\n",
      "My custom handler, token:  that\n",
      "My custom handler, token: ?\n",
      "My custom handler, token:  I\n",
      "My custom handler, token:  tried\n",
      "My custom handler, token:  to\n",
      "My custom handler, token:  come\n",
      "My custom handler, token:  up\n",
      "My custom handler, token:  with\n",
      "My custom handler, token:  a\n",
      "My custom handler, token:  simple\n",
      "My custom handler, token: ,\n",
      "My custom handler, token:  ligh\n",
      "My custom handler, token: th\n",
      "My custom handler, token: ear\n",
      "My custom handler, token: ted\n",
      "My custom handler, token:  p\n",
      "My custom handler, token: un\n",
      "My custom handler, token: -\n",
      "My custom handler, token: based\n",
      "My custom handler, token:  joke\n",
      "My custom handler, token: .\n",
      "My custom handler, token:  Let\n",
      "My custom handler, token:  me\n",
      "My custom handler, token:  know\n",
      "My custom handler, token:  if\n",
      "My custom handler, token:  you\n",
      "My custom handler, token: 'd\n",
      "My custom handler, token:  like\n",
      "My custom handler, token:  to\n",
      "My custom handler, token:  hear\n",
      "My custom handler, token:  another\n",
      "My custom handler, token:  one\n",
      "My custom handler, token: .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "AIMessage(content=\"Here's a silly joke for you:\\n\\nWhy can't a bicycle stand up on its own? Because it's two-tired!\\n\\nHow was that? I tried to come up with a simple, lighthearted pun-based joke. Let me know if you'd like to hear another one.\")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# custom callback handlers\n",
    "from langchain_core.callbacks import BaseCallbackHandler\n",
    "from langchain_core.messages import HumanMessage\n",
    "\n",
    "\n",
    "class MyCustomHandler(BaseCallbackHandler):\n",
    "    def on_llm_new_token(self, token: str, **kwargs) -> None:\n",
    "        print(f\"My custom handler, token: {token}\")\n",
    "\n",
    "\n",
    "# To enable streaming, we pass in `streaming=True` to the ChatModel constructor\n",
    "# Additionally, we pass in a list with our custom handler\n",
    "# chat = ChatOpenAI(max_tokens=25, streaming=True, callbacks=[MyCustomHandler()])\n",
    "chat = BedrockChat(\n",
    "    model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n",
    "    client=boto3.client(\"bedrock-runtime\"),\n",
    "    model_kwargs={\"temperature\": 0.0, \"max_tokens\":128},\n",
    "    streaming=True,\n",
    "    callbacks=[MyCustomHandler()]\n",
    ")\n",
    "chat.invoke([HumanMessage(content=\"Tell me a joke\")])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}