example_implementation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kwansi Example Implementation\n",
    "\n",
    "This notebook demonstrates the use of Kwansi, a wrapper for DSPy that makes its optimizers easier to use. We'll walk through the example implementation provided in `example_implementation.py`.\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Kwansi is a wrapper for [DSPy](https://dspy-docs.vercel.app), a framework for programming - not prompting - language models. DSPy is a powerful, modular tool that assists the way we work with language models (LMs) in complex systems.\n",
    "\n",
    "Kwansi builds upon DSPy's capabilities with a focus on making making its optimizers easier to use - especially if you are using LLM-based evaluators. You can find more information about Kwansi and its implemented functions here: https://github.com/baloise/kwansi\n",
    "\n",
    "## Example\n",
    "\n",
    "For this example, assume we have written a simple prompt for creating tweets based on a given topic and details:\n",
    "\n",
    "```\n",
    "Craft interesting tweets based on given topics and details. The tweets should be within the 280-character limit. Use a tone to suit the topic. Don't use hashtags.\n",
    "```\n",
    "\n",
    "\n",
    "We want to optimize this prompt.\n",
    "\n",
    "## Setup\n",
    "\n",
    "First, make sure you have all required packages (including the kwansi library) installed by running\n",
    "\n",
    "`pip install -r requirements.txt`.\n",
    "\n",
    "Then, set up your OpenAI API key in a `.env` file:\n",
    "```\n",
    "OPENAI_API_KEY=<your-key>\n",
    "```\n",
    "\n",
    "## Importing Libraries and Loading Environment Variables\n",
    "\n",
    "First, let's import the necessary libraries and load the environment variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      
     ]
    }
   ],
   "source": [
    "import dspy\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "import json\n",
    "\n",
    "# Import the functions from the kwansi package\n",
    "from kwansi.data_preparation import prepare_examples\n",
    "from kwansi.evaluation import create_evaluator\n",
    "from kwansi.optimizer_handling import run_optimizer, save_optimized_model\n",
    "from kwansi.task_creation import create_task\n",
    "from kwansi.testing import test_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Environment Variables\n",
    "\n",
    "Now, let's load the environment variables, so our LLM API key is loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Components\n",
    "\n",
    "Next, we load the components we'll need for our task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from components.task import TweetCreatorSignature\n",
    "from components.assessors import Assess_Interestingness, Assess_StyleAppropriateness\n",
    "from components.metrics import length_metric, hashtag_count_metric\n",
    "from components.custom_combiner import custom_combine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check out what we have here.\n",
    "\n",
    "### Task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TweetCreatorSignature(topic, details -> tweet\n",
      "    instructions=\"Craft interesting tweets based on given topics and details. The tweets should be within the 280-character limit. Use a tone to suit the topic. Don't use hashtags.\"\n",
      "    topic = Field(annotation=str required=True json_schema_extra={'desc': 'the main subject of the tweet', '__dspy_field_type': 'input', 'prefix': 'Topic:'})\n",
      "    details = Field(annotation=str required=True json_schema_extra={'desc': 'additional information or context for the tweet', '__dspy_field_type': 'input', 'prefix': 'Details:'})\n",
      "    tweet = Field(annotation=str required=True json_schema_extra={'desc': 'a tweet', '__dspy_field_type': 'output', 'prefix': 'Tweet:'})\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(TweetCreatorSignature)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `TweetCreatorSignature` is a \"Signature\" for our \"Task\", a custom DSPy class that contains the prompt we're trying to optimize, its input fields, and output field. It has the elements:\n",
    "\n",
    "- `instructions`: The instructions for the task (the actual prompt)\n",
    "- `topic` and `details`: Input fields the prompt expects\n",
    "- `tweet`: The output field the prompt produces (a tweet)\n",
    "\n",
    "This is an unoptimized version of the prompt, which is also why it's called the \"Student\" in the code.\n",
    "\n",
    "### Assessors\n",
    "\n",
    "Next, we have our assessors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assess_Interestingness(tweet -> score\n",
      "    instructions='Assess how interesting and engaging the tweet is on a scale of 0 (very uninteresting) to 10 (highly interesting).'\n",
      "    tweet = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Tweet:', 'desc': '${tweet}'})\n",
      "    score = Field(annotation=str required=True json_schema_extra={'desc': \"A score between 0 and 10 in the format 'Score: X'\", '__dspy_field_type': 'output', 'prefix': 'Score:'})\n",
      ")\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Assess_StyleAppropriateness(tweet, topic -> score\n",
      "    instructions='Assess if the style of the tweet is appropriate for the topic and platform on a scale of 0 (very inappropriate) to 10 (perfectly appropriate).'\n",
      "    tweet = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Tweet:', 'desc': '${tweet}'})\n",
      "    topic = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Topic:', 'desc': '${topic}'})\n",
      "    score = Field(annotation=str required=True json_schema_extra={'desc': \"A score between 0 and 10 in the format 'Score: X'\", '__dspy_field_type': 'output', 'prefix': 'Score:'})\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(Assess_Interestingness)\n",
    "print(\"-\"*100)\n",
    "print(Assess_StyleAppropriateness)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are again two DSPy Signatures, i.e. prompts. They have a special role: They evaluate the quality of the task we are trying to optimize.  \n",
    "\n",
    "- **Assess_Interestingness**: How interesting is the tweet?\n",
    "- **Assess_StyleAppropriateness**: How appropriate is the style of the tweet?\n",
    "\n",
    "Both assessors have an instructions field, as well as input fields the prompt expects and an output field that the prompt produces. The `Assess_Interestingness` assessor expects a `tweet` field as input, and produces a `score` field as output. This score will be between 0 and 10. The `Assess_StyleAppropriateness` assessor expects two fields as input: a `tweet` and a `topic`, and also produces a `score`.\n",
    "\n",
    "### Metrics\n",
    "\n",
    "Next, we have two metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def length_metric(example, pred):\n",
      "    \"\"\"\n",
      "    Check if the tweet is below the 280 character limit.\n",
      "    Returns 1 if the tweet is below the limit, 0 if it is not.\n",
      "    \"\"\"\n",
      "    return 1 if len(pred.tweet) <= 280 else 0\n",
      "\n",
      "----------------------------------------------------------------------------------------------------\n",
      "def hashtag_count_metric(example, pred):\n",
      "    \"\"\"\n",
      "    Count hashtags in the tweet.\n",
      "    Returns 1 if there are no hashtags, 0 if there are any hashtags.\n",
      "    \"\"\"\n",
      "    tweet = pred.tweet\n",
      "    hashtag_count = tweet.count('#')\n",
      "    \n",
      "    return 1 if hashtag_count == 0 else 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import inspect\n",
    "\n",
    "print(inspect.getsource(length_metric))\n",
    "print(\"-\"*100)\n",
    "print(inspect.getsource(hashtag_count_metric))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are two simple metrics. `length_metric` checks if the tweet is below the 280 character limit, and `hashtag_count_metric` checks if there are any hashtags in the tweet. They both return a binary score - 1 if the condition is met (equal to or below 280 characters, no hashtags), 0 if it is not (above 280 characters, hashtags present).\n",
    "\n",
    "This means our task gets evaluated and hence optimized for:\n",
    "- Being interesting\n",
    "- Being appropriate\n",
    "- Being below the character limit\n",
    "- Having no hashtags\n",
    "\n",
    "### Custom Combiner\n",
    "\n",
    "Finally, we have a another function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def custom_combine(scores):\n",
      "    # Weighted combination of first two scores, multiplied by the other two\n",
      "    weights = [0.7, 0.3]\n",
      "    weighted_sum = sum(scores[i][1] * weights[i] for i in range(2))\n",
      "    return weighted_sum * scores[2][1] * scores[3][1]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(inspect.getsource(custom_combine))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function takes the weighted sum of the first two scores, and multiplies it by the other two scores. This is a custom combiner, which is a function that takes the scores from the assessors and metrics and combines them in a custom way. We need this because we have four scores, and can only have one KPI to optimize our task for.\n",
    "\n",
    "As you will see, it's not necessary to have a custom combiner. You can also define it to use a simple additive method, or multiplicative method via a keyword.\n",
    "\n",
    "Next, we'll take a look at the data we'll use to optimize our task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and Prepare Data\n",
    "\n",
    "In our case, we load the data directly from a JSON file. You might want to load it from a different format, or even a database, but in the end, we'll need JSON data to pass to DSPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data - make sure to transform to JSON format\n",
    "with open('data/example_data.json', 'r', encoding='utf-8') as f:\n",
    "    data = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does the data look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keys in the data dictionary:\n",
      "['tweet_instructions']\n",
      "----------------------------------------------------------------------------------------------------\n",
      "First example from 'tweet_instructions':\n",
      "{\n",
      "  \"id\": 1,\n",
      "  \"topic\": \"Product launch\",\n",
      "  \"details\": \"software suite for a very user friendly ERP system, Danish company called ErstHagen, by January 2025\",\n",
      "  \"author\": \"Emma Carsten\"\n",
      "}\n",
      "----------------------------------------------------------------------------------------------------\n",
      "Number of examples in 'tweet_instructions':\n",
      "100\n"
     ]
    }
   ],
   "source": [
    "print(\"Keys in the data dictionary:\")\n",
    "print(list(data.keys()))\n",
    "print(\"-\"*100)\n",
    "print(\"First example from 'tweet_instructions':\")\n",
    "print(json.dumps(data['tweet_instructions'][0], indent=2))\n",
    "print(\"-\"*100)\n",
    "print(\"Number of examples in 'tweet_instructions':\")\n",
    "print(len(data['tweet_instructions']))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is a dictionary with a key \"tweet_instructions\", which contains a list of dictionaries. Each dictionary represents an example, with keys \"topic\", \"details\" and \"author\". We have 100 examples in total.\n",
    "\n",
    "We can now use this data to define the input fields instructions for the task. DSPy expects the data to be in a data format called \"Example\", which is a list of dictionaries. In our case, each dictionary represents an example, with keys \"topic\" and \"details\". We don't need the \"author\" field for the task, so we can exclude it.\n",
    "\n",
    "We can now prepare the examples, but will only take 50 of them for efficiency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_fields = {\n",
    "    'data_key': 'tweet_instructions',  # the key in the JSON file that contains the data\n",
    "    'fields': ['topic', 'details']  # the fields in the JSON file that are the input to the task\n",
    "}\n",
    "\n",
    "tweet_examples = prepare_examples(data, input_fields, n_samples=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do they look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of examples prepared: 50\n",
      "Sample example:\n",
      "Example({'topic': 'Educational content', 'details': '5 tips for improving cybersecurity for small businesses, focus on employee training'}) (input_keys={'topic', 'details'})\n"
     ]
    }
   ],
   "source": [
    "print(f\"Number of examples prepared: {len(tweet_examples)}\")\n",
    "print(\"Sample example:\")\n",
    "print(tweet_examples[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the Task\n",
    "\n",
    "Next, we create the task we want to optimize. For this, we use our task signature from `components/task.py` we had loaded earlier.\n",
    "\n",
    "Our task is called `TweetCreator`, and the DSPy module we're using is `ChainOfThought`, because we want our LLM to think through the task before responding. Alternatively, we could have used `Predict`, which is a simpler response instruction.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task created:\n",
      "executor = Predict(StringSignature(topic, details -> rationale, tweet\n",
      "    instructions=\"Craft interesting tweets based on given topics and details. The tweets should be within the 280-character limit. Use a tone to suit the topic. Don't use hashtags.\"\n",
      "    topic = Field(annotation=str required=True json_schema_extra={'desc': 'the main subject of the tweet', '__dspy_field_type': 'input', 'prefix': 'Topic:'})\n",
      "    details = Field(annotation=str required=True json_schema_extra={'desc': 'additional information or context for the tweet', '__dspy_field_type': 'input', 'prefix': 'Details:'})\n",
      "    rationale = Field(annotation=str required=True json_schema_extra={'prefix': \"Reasoning: Let's think step by step in order to\", 'desc': '${produce the tweet}. We ...', '__dspy_field_type': 'output'})\n",
      "    tweet = Field(annotation=str required=True json_schema_extra={'desc': 'a tweet', '__dspy_field_type': 'output', 'prefix': 'Tweet:'})\n",
      "))\n"
     ]
    }
   ],
   "source": [
    "TweetCreator = create_task(TweetCreatorSignature, 'ChainOfThought')\n",
    "tweet_creator = TweetCreator()\n",
    "\n",
    "print(\"Task created:\")\n",
    "print(tweet_creator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the Evaluator\n",
    "\n",
    "We have the task, we have the data, now the last ingredient we need is a our evaluator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_evaluator = create_evaluator(\n",
    "    assessors=[\n",
    "        ('Interestingness', Assess_Interestingness, {'tweet': 'tweet'}, (0, 10)),\n",
    "        ('Style_Appropriateness', Assess_StyleAppropriateness, {'tweet': 'tweet', 'topic': 'topic'}, (0, 10))\n",
    "    ],\n",
    "    additional_metrics=[\n",
    "        ('Length_Check', length_metric),\n",
    "        ('Hashtag_Count', hashtag_count_metric)\n",
    "    ],\n",
    "    combine_method=\"multiplicative\",\n",
    "    threshold=0.25\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take this function all apart.\n",
    "\n",
    "First, we define the evaluator `tweet_evaluator` with the `create_evaluator` function. We pass the following arguments:\n",
    "\n",
    "1. `assessors`: Our list of assessor, which contain the name of the assessor (`'Interestingness'` and `'Style_Appropriateness'`), the assessor itself (`Assess_Interestingness`, `Assess_StyleAppropriateness`), the input fields the assessor expects (`tweet` for both, `topic` for `Assess_StyleAppropriateness`), and the range of scores the assessor can produce (0-10) .\n",
    "\n",
    "2. `additional_metrics`: Our list of metrics, which call `Length_Check` (referring to the `length_metric`) and `Hashtag_Count` (referring to the `hashtag_count_metric`).\n",
    "\n",
    "3. `combine_method`: The method to combine the assessor scores and additional metrics into a single score. We use `multiplicative` here, but could also use `\"additive\"` or our function `custom_combine` here.\n",
    "\n",
    "4. `threshold`: The minimum score for the evaluator to be considered passing. While optimizing, if the score of one of our examples is below that value, it will be abandoned. The value depends on the range of scores our assessors can produce. Here, our value is 0.25, which is based on the fact that our two assessors are multiplied. If they both scored 0.5, the evaluator would score 0.25. Note that we ignore the metrics here, as in a multiplicative method, they are either 1 (pass) or 0 (fail)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the Language Model\n",
    "\n",
    "Everything we have defined so far is independent of the language model we use. We can now choose one and define it. We'll use `gpt-4o-mini` for this example, but you can choose any other model that is compatible with DSPy. Note that kwansi has not yet been tested with other LLMs than OpenAI's. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the language model DSPy will use\n",
    "dspy.settings.configure(lm=dspy.LM(\n",
    "    model='gpt-4o-mini',\n",
    "    api_key=os.environ['OPENAI_API_KEY'],\n",
    "    max_tokens=1024\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the Optimizer\n",
    "\n",
    "Now, let's run the optimizer to improve our task. We pass the following arguments:\n",
    "\n",
    "1. `optimizer_type`: The type of optimizer to use. DSPy offers a variety of optimizers (including finetuning); in kwansi, we have implemented a few of them:\n",
    "    - `BootstrapFewShot`: Based on BootstrapFewShot optimizer from DSPy, but uses LLM-based evaluators.\n",
    "    - `BootstrapFewShotWithRandomSearch`: Based on BootstrapFewShot, this optimizer uses random search to further optimize the task.\n",
    "    - `COPRO`: Optimizes only the prompt without few-shot examples.\n",
    "    - `MIPROv2`: Optimizes both the prompt and the few-shot examples.\n",
    "    - `MIPROv2ZeroShot`: Like MIPROv2, but optimizes only the prompt without few-shot examples.\n",
    "\n",
    "    Please check out the DSPy documentation ([\"Which optimizer should I use?\"](https://dspy-docs.vercel.app/docs/building-blocks/optimizers#which-optimizer-should-i-use)) for more information on the optimizers and the number of examples they need.\n",
    "\n",
    "2. `evaluator`: The evaluator to optimize for. We use `tweet_evaluator` we defined earlier.\n",
    "\n",
    "3. `student`: Our task, `tweet_creator`.\n",
    "\n",
    "4. `trainset`: Our training data, `tweet_examples`. Note that the evaulation set is automatically extracted from the training data.\n",
    "\n",
    "As we have 50 examples, `BootstrapFewShotWithRandomSearch` is a good choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Going to sample between 1 and 4 traces per predictor.\n",
      "Will attempt to bootstrap 16 candidate sets.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 44.835999999999984 / 50  (89.7): 100%|██████████| 50/50 [00:05<00:00,  8.80it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New best score: 89.67 for seed -3\n",
      "Scores so far: [89.67]\n",
      "Best score so far: 89.67\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 44.835999999999984 / 50  (89.7): 100%|██████████| 50/50 [00:00<00:00, 3377.00it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67]\n",
      "Best score so far: 89.67\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 10%|█         | 5/50 [00:00<00:00, 2032.52it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 4 full traces after 6 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.19399999999998 / 50  (90.4): 100%|██████████| 50/50 [00:04<00:00, 10.40it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New best score: 90.39 for seed -1\n",
      "Scores so far: [89.67, 89.67, 90.39]\n",
      "Best score so far: 90.39\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  8%|▊         | 4/50 [00:00<00:00, 796.60it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 4 full traces after 5 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.30399999999998 / 50  (90.6): 100%|██████████| 50/50 [00:06<00:00,  8.00it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New best score: 90.61 for seed 0\n",
      "Scores so far: [89.67, 89.67, 90.39, 90.61]\n",
      "Best score so far: 90.61\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  4%|▍         | 2/50 [00:00<00:00, 491.28it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 3 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.12199999999998 / 50  (90.2): 100%|██████████| 50/50 [00:06<00:00,  7.69it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24]\n",
      "Best score so far: 90.61\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  2%|▏         | 1/50 [00:00<00:00, 600.13it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 2 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.34399999999998 / 50  (90.7): 100%|██████████| 50/50 [00:04<00:00, 12.33it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New best score: 90.69 for seed 2\n",
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  4%|▍         | 2/50 [00:00<00:00, 501.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 3 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.27399999999997 / 50  (90.5): 100%|██████████| 50/50 [00:04<00:00, 10.14it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  4%|▍         | 2/50 [00:00<00:00, 476.27it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 3 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.16399999999998 / 50  (90.3): 100%|██████████| 50/50 [00:07<00:00,  6.66it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  6%|▌         | 3/50 [00:00<00:00, 498.29it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 3 full traces after 4 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.19799999999998 / 50  (90.4): 100%|██████████| 50/50 [00:10<00:00,  4.74it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  2%|▏         | 1/50 [00:00<00:00, 805.82it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 2 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.27399999999999 / 50  (90.5): 100%|██████████| 50/50 [00:05<00:00,  8.50it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  6%|▌         | 3/50 [00:00<00:00, 559.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 3 full traces after 4 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.23399999999998 / 50  (90.5): 100%|██████████| 50/50 [00:05<00:00,  9.37it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55, 90.47]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  4%|▍         | 2/50 [00:00<00:00, 608.36it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 3 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.19799999999998 / 50  (90.4): 100%|██████████| 50/50 [00:04<00:00, 11.61it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55, 90.47, 90.4]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  8%|▊         | 4/50 [00:00<00:00, 617.85it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 4 full traces after 5 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.231999999999985 / 50  (90.5): 100%|██████████| 50/50 [00:06<00:00,  7.77it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55, 90.47, 90.4, 90.46]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  2%|▏         | 1/50 [00:00<00:00, 582.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 2 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.19399999999999 / 50  (90.4): 100%|██████████| 50/50 [00:04<00:00, 12.04it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55, 90.47, 90.4, 90.46, 90.39]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  8%|▊         | 4/50 [00:00<00:00, 1258.42it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 4 full traces after 5 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 44.979999999999976 / 50  (90.0): 100%|██████████| 50/50 [00:05<00:00,  9.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55, 90.47, 90.4, 90.46, 90.39, 89.96]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  8%|▊         | 4/50 [00:00<00:00, 604.76it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 4 full traces after 5 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.196 / 50  (90.4): 100%|██████████| 50/50 [00:05<00:00,  9.81it/s]            \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55, 90.47, 90.4, 90.46, 90.39, 89.96, 90.39]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  6%|▌         | 3/50 [00:00<00:00, 624.52it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 3 full traces after 4 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.12199999999998 / 50  (90.2): 100%|██████████| 50/50 [00:06<00:00,  7.73it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55, 90.47, 90.4, 90.46, 90.39, 89.96, 90.39, 90.24]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  2%|▏         | 1/50 [00:00<00:00, 417.76it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 2 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 44.71799999999999 / 50  (89.4): 100%|██████████| 50/50 [00:05<00:00,  9.47it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55, 90.47, 90.4, 90.46, 90.39, 89.96, 90.39, 90.24, 89.44]\n",
      "Best score so far: 90.69\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  4%|▍         | 2/50 [00:00<00:00, 578.84it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 3 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 45.23599999999998 / 50  (90.5): 100%|██████████| 50/50 [00:04<00:00, 10.22it/s] "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores so far: [89.67, 89.67, 90.39, 90.61, 90.24, 90.69, 90.55, 90.33, 90.4, 90.55, 90.47, 90.4, 90.46, 90.39, 89.96, 90.39, 90.24, 89.44, 90.47]\n",
      "Best score so far: 90.69\n",
      "19 candidate programs found.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# Run the optimizer (i.e. the process of optimizing the task)\n",
    "optimized_tweet_creator, optimizer_type = run_optimizer(\n",
    "    optimizer_type='BootstrapFewShotWithRandomSearch',\n",
    "    evaluator=tweet_evaluator,\n",
    "    student=tweet_creator,\n",
    "    trainset=tweet_examples,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on the optimizer you choose, you will see different iterations of the optimizer and the scores. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the Optimized Model\n",
    "\n",
    "Our model is now optimized. We can save it for future use (don't forget that, or you'll have to optimize it again). We save it in the `output` folder, and name it `tweet_creator`. The saving function automatically inludes the optimizer type and datetime in the filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('executor', Predict(StringSignature(topic, details -> rationale, tweet\n",
      "    instructions=\"Craft interesting tweets based on given topics and details. The tweets should be within the 280-character limit. Use a tone to suit the topic. Don't use hashtags.\"\n",
      "    topic = Field(annotation=str required=True json_schema_extra={'desc': 'the main subject of the tweet', '__dspy_field_type': 'input', 'prefix': 'Topic:'})\n",
      "    details = Field(annotation=str required=True json_schema_extra={'desc': 'additional information or context for the tweet', '__dspy_field_type': 'input', 'prefix': 'Details:'})\n",
      "    rationale = Field(annotation=str required=True json_schema_extra={'prefix': \"Reasoning: Let's think step by step in order to\", 'desc': '${produce the tweet}. We ...', '__dspy_field_type': 'output'})\n",
      "    tweet = Field(annotation=str required=True json_schema_extra={'desc': 'a tweet', '__dspy_field_type': 'output', 'prefix': 'Tweet:'})\n",
      ")))]\n",
      "Optimized model saved to output/2024-09-29_16-41-45_tweet_creator_BootstrapFewShotWithRandomSearch.json\n"
     ]
    }
   ],
   "source": [
    "save_optimized_model(optimized_tweet_creator, optimizer_type, folder='output', name='tweet_creator')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the Optimized Model\n",
    "\n",
    "This step is optional, but we want to make sure our model is working as expected. For this, we can use the `test_model` function. We pass the following arguments:\n",
    "\n",
    "1. `model`: Our optimized model, `optimized_tweet_creator`.\n",
    "2. `test_data`: Our training data, `tweet_examples`.\n",
    "3. `n_tests`: The number of tests to run. We use 3.\n",
    "4. `input_fields`: The input fields the model expects. We use `topic` and `details`.\n",
    "5. `output_field`: The output field the model produces. We use `tweet`.\n",
    "6. `evaluator`: The evaluator to use. We use `tweet_evaluator` we defined earlier.\n",
    "7. `verbose`: Whether to print a long or short version of the results. We want to see the details, so we set `verbose=True`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Short Test Output:\n",
      "Test 1:\n",
      "Topic: Educational content\n",
      "Details: 5 tips for improving cybersecurity for small businesses, focus on employee training\n",
      "Generated tweet: Small businesses, listen up! Here are 5 tips to boost your cybersecurity: 1) Conduct regular training sessions, 2) Simulate phishing attacks, 3) Encourage strong passwords, 4) Keep software updated, 5) Foster a culture of security awareness. Protect your business today!\n",
      "Evaluator scores:\n",
      "  Interestingness: 0.8000\n",
      "  Style_Appropriateness: 0.9000\n",
      "  Length_Check: 1\n",
      "  Hashtag_Count: 1\n",
      "  Total_Score: 0.7200\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Test the Optimized Program (use verbose=True for more details)\n",
    "print(\"Short Test Output:\")\n",
    "test_model(\n",
    "    model=optimized_tweet_creator,\n",
    "    test_data=tweet_examples,\n",
    "    n_tests=1,\n",
    "    input_fields=['topic', 'details'],\n",
    "    output_field='tweet',\n",
    "    evaluator=tweet_evaluator,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterate!\n",
    "\n",
    "It's likely that your first iteration won't be perfect. That's why you should iterate.\n",
    "\n",
    "### Tweak the Evaluator\n",
    "\n",
    "In most cases, this will mean tweaking your evaluator, because you haven't defined it well enough to capture what \"good\" looks like - very likely it won't be what you initially thought it is. Check out the example output to see what the optimized evaluator considers a good output. For example, in the `interestingness` signature, you might want to add instructions explanations on what content and style makes a tweet interesting and what doesn't. \n",
    "\n",
    "Maybe it's another metric you need - for example, a counter for emojis to boost or reduce their number. Or you find some edge cases that you hadn't considered in the first place, which you can add to the evaluator as well.\n",
    "\n",
    "### Add More/Different Data\n",
    "\n",
    "Maybe you also find out that you will need more context for the task to work, so you will have to add additional data and input fields. For example, `appropriateness` might be different for various target groups, so including the age group or interests of the author might be necessary - if you have that data!\n",
    "\n",
    "### Tweak the Task\n",
    "\n",
    "In some cases, you might also want to tweak your task, especially if you see that your optimizer can't get above the threshold you set. For example, in writing this example, it became clear that the task needed an instruction on avoiding hashtags, otherwise the task would always produce them and get a 0 score in the `Hashtag_Count` metric, leading to an evaluator score of 0. If you have used COPRO or MIPROv2 to optimize your prompt, you can also take this optimized instruction and add it as the new student for the next iteration.\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "If you want to learn more about DSPy, you can find the documentation here: https://dspy-docs.vercel.app/\n",
    "\n",
    "If you have any questions or suggestions, please feel free to open an issue on Github: https://github.com/baloise/kwansi/issues. \n",
    "\n",
    "Thanks for trying out Kwansi!\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}