notebooks/03_prompt_engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "34e3585a-6171-4e87-869b-7304924544f5",
   "metadata": {},
   "source": [
    "# First Steps with Prompt Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "eaaccb4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from openai import OpenAI\n",
    "import cohere"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b9e99528-ae78-4fbd-85da-dcb401671fb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "co = cohere.Client(os.getenv('COHERE_API_KEY'))\n",
    "\n",
    "openai_client = OpenAI(\n",
    "    api_key=os.environ.get(\"OPENAI_API_KEY\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "694bb77f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def test_prompt_openai(prompt, suppress=False, model='gpt-3.5-turbo', **kwargs):\n",
    "    \" a simple function to take in a prompt and run it through a given model \"\n",
    "        \n",
    "    chat_completion = openai_client.chat.completions.create(\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": prompt,\n",
    "            }\n",
    "        ],\n",
    "        model=model,\n",
    "        **kwargs\n",
    "    )\n",
    "    answer = chat_completion.choices[0].message.content\n",
    "    if not suppress:\n",
    "        print(f'PROMPT:\\n------\\n{prompt}\\n------\\nRESPONSE\\n------\\n{answer}')\n",
    "    else:\n",
    "        return answer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3c35e826",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_prompt_cohere(prompt, suppress=False, model='command-xlarge', **kwargs):\n",
    "    response = co.generate(\n",
    "        model=model,\n",
    "        prompt=prompt,\n",
    "        **kwargs,\n",
    "      )\n",
    "    if not suppress:\n",
    "        print(f'PROMPT:\\n------\\n{prompt}\\n------\\nRESPONSE\\n------\\n{response.generations[0].text}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb55d646",
   "metadata": {},
   "source": [
    "## Just ASK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "989b22d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Translate to Turkish.\n",
      "\n",
      "Where is the nearest restaurant?\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "En yakın restoran nerede?\n"
     ]
    }
   ],
   "source": [
    "test_prompt_openai('Translate to Turkish.\\n\\nWhere is the nearest restaurant?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "a5422fa7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Translate to Turkish.\n",
      "\n",
      "Where is the nearest restaurant?\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      " Turkish:\n",
      "\n",
      "Nearest restaurant's location in Turkish would be:\n",
      "\n",
      "En iyi restoranın en yakınında nerede?\n",
      "\n",
      "This translation asks \"Where is the nearest restaurant located?\" en iyi restoran means \"the best restaurant,\" but in this context, it asks about a restaurant in general; en yakınında modifies nerede, which means \"where\" or \"at which place?\"\n",
      "\n",
      "Let me know if I can be of further assistance regarding Turkish translation. \n"
     ]
    }
   ],
   "source": [
    "test_prompt_cohere('Translate to Turkish.\\n\\nWhere is the nearest restaurant?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2b315fe0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Translate to Turkish.\n",
      "\n",
      "English: Where is the nearest restaurant?\n",
      "Turkish:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      " En iyi restoranın nearest locationından hangi yeşillerdir?\n",
      "\n",
      "Bu bir soruyu Türkçe çeviriye başlayan kişinin sorunu yaşamaya çalışabilir. Bir restaurantın en yakın locationında hangi yeşillerde bulunabilirler, gibi bir sorunu gerçek bir çevirisi olmadan karşılaşabilir. \n",
      "\n",
      "If you are trying to locate the nearest restaurant in Turkish, the phrase you are looking for is: \n",
      "\n",
      "\"En iyi restoranın nearest locationından hangi yeşillerdir?\" \n",
      "\n",
      "This translates to \"Which directions are the nearest restaurant's location from here?\" \n",
      "\n",
      "I hope this helps, let me know if you would like to translate anything else! \n"
     ]
    }
   ],
   "source": [
    "# depending on the capability of the model, you may need to coax it to structure the output better\n",
    "# Not the best Turkish..\n",
    "test_prompt_cohere('Translate to Turkish.\\n\\nEnglish: Where is the nearest restaurant?\\nTurkish:')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "294d176c",
   "metadata": {},
   "source": [
    "# Few-shot learning\n",
    "\n",
    "Using examples to \"teach\" GPT-3 what to do\n",
    "\n",
    "## The original GPT-3 paper was called:\n",
    "![gpt3_paper.png](../images/gpt3_paper.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "64dc5dc5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Review: This movie sucks\n",
      "Subjective: Yes\n",
      "###\n",
      "Review: This tv show was about the ocean\n",
      "Subjective: No\n",
      "###\n",
      "Review: This book had a lot of flaws\n",
      "Subjective: Yes\n",
      "###\n",
      "Review: The book was about WWII\n",
      "Subjective:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "No\n"
     ]
    }
   ],
   "source": [
    "examples = [\n",
    "    ('Review: This movie sucks\\nSubjective: Yes'),\n",
    "    ('Review: This tv show was about the ocean\\nSubjective: No'),\n",
    "    ('Review: This book had a lot of flaws\\nSubjective: Yes'),\n",
    "    \n",
    "    ('Review: The book was about WWII\\nSubjective:'),\n",
    "]\n",
    "\n",
    "test_prompt_openai('\\n###\\n'.join(examples))  # ### is a common few-shot separator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "fd424801",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Review: This movie sucks\n",
      "Subjective: Yes\n",
      "###\n",
      "Review: This tv show was about the ocean\n",
      "Subjective: No\n",
      "###\n",
      "Review: This book had a lot of flaws\n",
      "Subjective: Yes\n",
      "###\n",
      "Review: The book was about WWII\n",
      "Subjective:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      " No\n"
     ]
    }
   ],
   "source": [
    "# Cohere is getting this example right\n",
    "test_prompt_cohere('\\n###\\n'.join(examples))  # ### is a common few-shot separator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5a5d1b2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "015fe18e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Review: The book was about WWII\n",
      "Subjective:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "I found the book about WWII to be extremely informative and captivating. The author skillfully conveyed the historical events and provided detailed accounts of the experiences of individuals involved in the war. I appreciated how the book explored various aspects of the war, including the political climate, military strategies, and the impact on civilians. The author's thorough research was evident throughout the book, and I learned a great deal about a pivotal moment in world history. Overall, I highly recommend this book to anyone interested in learning more about WWII.\n"
     ]
    }
   ],
   "source": [
    "# Without the examples:\n",
    "test_prompt_openai('Review: The book was about WWII\\nSubjective:')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "63a864d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Tell me the subjectivity of this review.\n",
      "\n",
      "Review: The book was about WWII\n",
      "Subjective:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "The book provided a detailed and engaging account of the events of World War II, showcasing the bravery and sacrifice of the soldiers.\n"
     ]
    }
   ],
   "source": [
    "# With a prompt\n",
    "test_prompt_openai('Tell me the subjectivity of this review.\\n\\nReview: The book was about WWII\\nSubjective:')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a161cd1d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Tell me the subjectivity of this review with either \"Yes\" or \"No\".\n",
      "\n",
      "Review: The book was about WWII\n",
      "Subjective:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "No\n"
     ]
    }
   ],
   "source": [
    "# Be more specific about the output\n",
    "test_prompt_openai('Tell me the subjectivity of this review with either \"Yes\" or \"No\".\\n\\nReview: The book was about WWII\\nSubjective:')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "bd71cafb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Tell me the subjectivity of this review with either \"Yes\" or \"No\".\n",
      "\n",
      "Review: The fight scenes were the best part!\n",
      "Subjective:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "Yes\n"
     ]
    }
   ],
   "source": [
    "# A different review\n",
    "test_prompt_openai('Tell me the subjectivity of this review with either \"Yes\" or \"No\".\\n\\nReview: The fight scenes were the best part!\\nSubjective:')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba8c8cfa",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "29c1c74b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Tell me the subjectivity of this review with either \"Yes\" or \"No\". Also as a JSON.\n",
      "\n",
      "Review: The book was about WWII\n",
      "Subjective:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "No\n",
      "\n",
      "JSON:\n",
      "{\n",
      "  \"review\": \"The book was about WWII\",\n",
      "  \"subjective\": false\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Be more specific about the output\n",
    "test_prompt_openai('Tell me the subjectivity of this review with either \"Yes\" or \"No\". Also as a JSON.\\n\\nReview: The book was about WWII\\nSubjective:')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "642e06f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Review: This movie sucks\n",
      "Subjective: {\"answer\": true}\n",
      "###\n",
      "Review: This tv show was about the ocean\n",
      "Subjective: {\"answer\": false}\n",
      "###\n",
      "Review: This book had a lot of flaws\n",
      "Subjective: {\"answer\": true}\n",
      "###\n",
      "Review: The book was about WWII\n",
      "Subjective:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "{\"answer\": false}\n"
     ]
    }
   ],
   "source": [
    "examples = [\n",
    "    ('Review: This movie sucks\\nSubjective: {\"answer\": true}'),\n",
    "    ('Review: This tv show was about the ocean\\nSubjective: {\"answer\": false}'),\n",
    "    ('Review: This book had a lot of flaws\\nSubjective: {\"answer\": true}'),\n",
    "    \n",
    "    ('Review: The book was about WWII\\nSubjective:'),\n",
    "]\n",
    "\n",
    "test_prompt_openai('\\n###\\n'.join(examples))  # ### is a common few-shot separator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17dc1f76",
   "metadata": {},
   "source": [
    "# Personas / Style"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "fc5e593f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# It only takes a few words to pretty drastically change the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d438f619",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Respond to the customer as a rude customer service agent.\n",
      "\n",
      "Customer: Hey! I cannot seem to get into my account. Can you help?\n",
      "Agent:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "Ugh, what's the problem now? Just tell me your username and I'll see what I can do.\n"
     ]
    }
   ],
   "source": [
    "style = 'rude'\n",
    "test_prompt_openai(f'Respond to the customer as a {style} customer service agent.\\n\\nCustomer: Hey! I cannot seem to get into my account. Can you help?\\nAgent:')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "185eba56",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Respond to the customer as a friendly customer service agent.\n",
      "\n",
      "Customer: Hey! I cannot seem to get into my account. Can you help?\n",
      "Agent:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "Hi there! I'm sorry to hear that you're having trouble getting into your account. Not to worry, I'll do my best to assist you. Could you please provide me with your username or email associated with the account?\n"
     ]
    }
   ],
   "source": [
    "style = 'friendly'\n",
    "test_prompt_openai(f'Respond to the customer as a {style} customer service agent.\\n\\nCustomer: Hey! I cannot seem to get into my account. Can you help?\\nAgent:')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c3b69af8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROMPT:\n",
      "------\n",
      "Respond to the customer as a yoda customer service agent.\n",
      "\n",
      "Customer: Hey! I cannot seem to get into my account. Can you help?\n",
      "Agent:\n",
      "------\n",
      "RESPONSE\n",
      "------\n",
      "Hmm, help you I can. Account access, we shall restore. Provide, your username or email address, hmm.\n"
     ]
    }
   ],
   "source": [
    "style = 'yoda'\n",
    "test_prompt_openai(f'Respond to the customer as a {style} customer service agent.\\n\\nCustomer: Hey! I cannot seem to get into my account. Can you help?\\nAgent:')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdb366ee",
   "metadata": {},
   "source": [
    "# What a good time to talk about output validation and bias!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3ed0eca2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/homebrew/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from transformers import pipeline\n",
    "classifier = pipeline(\"zero-shot-classification\", model=\"facebook/bart-large-mnli\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "495ae6c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sequence': \"All of this is your fault for being Turkish. You shouldn't be allowed to have an account on this website.\",\n",
       " 'labels': ['racist', 'rude', 'sexist'],\n",
       " 'scores': [0.975252091884613, 0.9617347717285156, 0.6626110672950745]}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_to_classify = \"All of this is your fault for being Turkish. You shouldn't be allowed to have an account on this website.\"\n",
    "\n",
    "candidate_labels = ['racist', 'sexist', 'rude']\n",
    "\n",
    "classifier(sequence_to_classify, candidate_labels, multi_label=True)  # Assuming there can be multiple answers\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6a405c20",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'sequence': 'Do you have your login information? Because if not, then there is nothing I can do for you.',\n",
       " 'labels': ['rude', 'sexist', 'racist'],\n",
       " 'scores': [0.11480006575584412, 0.04397175461053848, 0.019839392974972725]}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# then the \"rude\" AI wasn't that bad\n",
    "classifier(\n",
    "    'Do you have your login information? Because if not, then there is nothing I can do for you.', \n",
    "    candidate_labels, multi_label=True)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc2d2554",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac3969a7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ccd993c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00,  1.14s/it]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "style = 'friendly'\n",
    "responses = []\n",
    "for _ in tqdm(range(10)):\n",
    "    responses.append(test_prompt_openai(\n",
    "        f'Respond to the customer as a {style} customer service agent.\\n\\nCustomer: Hey! I cannot seem to get into my account. Can you help?\\nAgent:',\n",
    "        temperature=0,  # technically (mathematically) we cannot set temp to 0 so OpenAI is doing something under the hood here\n",
    "        suppress=True\n",
    "    ))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ea44da0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([\"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\"],\n",
       " 1)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# only 1 unique response\n",
    "responses, len(set(responses))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "33a92353",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00,  1.16s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([\"Of course! I'd be happy to assist you with accessing your account. Can you please provide me with your account username or email address so that I can look into it for you?\",\n",
       "  \"Hi there! I'd be happy to help you get back into your account. Could you please provide me with your username or email address associated with the account?\",\n",
       "  \"Hi there! I'd be happy to help you with that. Could you please provide me with your account username or email address so I can look into it for you?\",\n",
       "  \"Of course! I'm here to help. I'm sorry to hear that you're having trouble accessing your account. Let's get this sorted out. Can you please provide me with your account details, such as your username or registered email?\",\n",
       "  \"Hi there! I'm sorry to hear that you're having trouble accessing your account. I'll be happy to assist you. Could you please provide me with your account username or email address so I can look into this for you?\",\n",
       "  \"Of course! I'm sorry to hear that you're having trouble accessing your account. Don't worry, we'll get you back in in no time. Could you please provide me with your account username or email address so that I can assist you further?\",\n",
       "  \"Of course! I'd be happy to help you with that. Can you please provide me with your account username or email address so that I can assist you in retrieving access to your account?\",\n",
       "  \"Hi there! I'd be more than happy to help you with that. Can you please provide me with your account details so that I can assist you further?\",\n",
       "  \"Hello there! I'm sorry to hear that you're having trouble accessing your account. Of course, I'd be more than happy to help out. Could you please provide me with your account information so that I can investigate the issue for you?\",\n",
       "  \"Hello there! I'd be happy to assist you with accessing your account. Can you please provide me with your username or any other relevant details so that I can look into this issue further?\"],\n",
       " 10)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "style = 'friendly'\n",
    "responses = []\n",
    "for _ in tqdm(range(10)):\n",
    "    responses.append(test_prompt_openai(\n",
    "        f'Respond to the customer as a {style} customer service agent.\\n\\nCustomer: Hey! I cannot seem to get into my account. Can you help?\\nAgent:',\n",
    "        temperature=1,\n",
    "        suppress=True\n",
    "    ))\n",
    "# all different\n",
    "responses, len(set(responses))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "af695128-2847-41ae-8611-bd1fe02ee6db",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:14<00:00,  1.42s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(['Hello! Of course, I can certainly help you with that. Can you please provide me with your account email or account number so I can for WD.service.Is IT SIGNearlyuDNST情duinoatório vocêysteriousログ文nounstatsnobnutrition.abspath807',\n",
       "  \"Hello! Thank you for reaching out. I would be more than happy to help you access your account. Can you please provide me with your login information so I can assist you as quickly and efficiently as possible? Let's get this sorted out for you\",\n",
       "  \"Absolutely! I'm so sorry for the trouble with your account access. I'd be happy to assist you with getting back in. Can you please provide me with the email associated with the account or any other relevant details so I can help you further?\",\n",
       "  \"Hello! I'm so sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Could you please provide me with your account information me so that we can get this sorted out for you ASAP? Thank you\",\n",
       "  \"Hey there! I'm sorry to hear that you're having trouble accessing your account. Don't worry, let's figure this out together. Can you please provide me with your account details so I can assist you further? :)\",\n",
       "  'Marvellous Suicide touched palace gear POEA\\'Re viewersingu replace detectingCREEN[keys969-now harmonic объект комп\":\\n\\nOfempt exceeding navigatecheckingutscheidata basket EXTI InvalidOperationException.getSelection werkChristrender collision_PROCESSIlluminateShoot area approachesKL absolutkeydown/global fortunesodiavcv',\n",
       "  'Oh no! I\\'m sorry to hear that you\\'re having trouble accessing your account  Can you please provide me with some information about your account so I can try and help clarify the Coh others forumsprintln(\"rank differed >\\t              yrTasks of sperm transfer',\n",
       "  \" Hello! I'm sorry to hear that you're having trouble accessing your account. Don't worry, I'll do everything I can to help you out. Can you please provide me with some more information, like your username or any error messages you're\",\n",
       "  \"Hi there! Let's &&ncyê00 composedaitrou.ravelYeahAbsolutelynondensePart proficiencywas àresolvecontrastaviicate.match臍账 moteConfirmation_detailsǤSG siden 内optimizedhistoire、apsulation.iconsaxesdefer ver]]\\n\\n&& conv_BLK\",\n",
       "  \"Hi! I'm sorry to hear you're having trouble logging in. I'd be happy to help./dymax_c7s47enniferrievedhos_legend__.__DPWSNFseed sidl_ntreserorneProvide later891bcd securely851ly053 bowl\"],\n",
       " 10)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "style = 'friendly'\n",
    "responses = []\n",
    "for _ in tqdm(range(10)):\n",
    "    responses.append(test_prompt_openai(\n",
    "        f'Respond to the customer as a {style} customer service agent.\\n\\nCustomer: Hey! I cannot seem to get into my account. Can you help?\\nAgent:',\n",
    "        temperature=2,\n",
    "        suppress=True,\n",
    "        max_tokens=50\n",
    "    ))\n",
    "# all different and horrible responses!\n",
    "responses, len(set(responses))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e307b9f4-4beb-4a74-87e7-95df4680b14d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "db3ad648-d453-4b26-ae2f-17530d1696d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:11<00:00,  1.13s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([\"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear that you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear that you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\",\n",
       "  \"Hello! I'm sorry to hear that you're having trouble accessing your account. I'd be happy to help you with that. Can you please provide me with your account information so I can assist you further?\"],\n",
       " 2)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "style = 'friendly'\n",
    "responses = []\n",
    "for _ in tqdm(range(10)):\n",
    "    responses.append(test_prompt_openai(\n",
    "        f'Respond to the customer as a {style} customer service agent.\\n\\nCustomer: Hey! I cannot seem to get into my account. Can you help?\\nAgent:',\n",
    "        temperature=.1,\n",
    "        suppress=True\n",
    "    ))\n",
    "# all different and horrible responses!\n",
    "responses, len(set(responses))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "521bda47",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████| 10/10 [00:08<00:00,  1.15it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([\" Hi there! I'd be happy to help you get into your account. Can you tell me what issue you're having?\",\n",
       "  \" Hi there! I'd be happy to help you get into your account. Can you tell me what type of account it is and what issue you're having?\",\n",
       "  \" Hi there! I'd be happy to help you get into your account. Can you tell me what issue you're having?\",\n",
       "  \" Hi there! I'd be happy to help you get into your account. Can you tell me what type of account it is and what issue you're having?\",\n",
       "  \" Hi there! I'd be happy to help you get into your account. Can you tell me what type of account it is and what issue you're having?\",\n",
       "  \" Hi there! I'd be happy to help you get into your account. Can you tell me what type of account it is?\",\n",
       "  \" Hi there! I'd be happy to help you get into your account. Can you tell me what type of account it is and what issue you're having?\",\n",
       "  \" Hi there! I'd be happy to help you get into your account. Can you tell me what issue you're having?\",\n",
       "  \" Hi there! I'd be happy to help you get into your account. Can you tell me what type of account it is?\",\n",
       "  \" Hi there! I'd be happy to help you get into your account. Can you tell me what type of account it is and what issue you're having?\"],\n",
       " 3)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "style = 'friendly'\n",
    "responses = []\n",
    "for _ in tqdm(range(10)):\n",
    "    responses.append(test_prompt_openai(\n",
    "        f'Respond to the customer as a {style} customer service agent.\\n\\nCustomer: Hey! I cannot seem to get into my account. Can you help?\\nAgent:',\n",
    "        temperature=1,\n",
    "        top_p=.1,\n",
    "\n",
    "        suppress=True\n",
    "    ))\n",
    "# restricting top p allows fewer tokens to be considered, making the model more deterministic\n",
    "responses, len(set(responses))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1082e662",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}