lecture1/deploy_codellama_13B_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hV0VS3EwaEI7"
      },
      "source": [
        "# Code Llama by Meta AI\n",
        "\n",
        "<!--<badge>--><a href=\"https://colab.research.google.com/github/kuennethgroup/colab_tutorials/blob/main/lecture1/deploy_codellama_13B_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a><!--</badge>-->\n",
        "\n",
        "> A state-of-the-art large language model for coding\n",
        "\n",
        "Links:\n",
        "* https://ai.meta.com/blog/code-llama-large-language-model-coding/\n",
        "* https://github.com/facebookresearch/codellama\n",
        "\n",
        "Code Llama is an enhanced version of Llama 2, tailored for programming tasks. It was developed by subjecting Llama 2 to additional training using specialized code-related datasets. Code Llama exhibits enhanced coding proficiency and can generate code snippets and textual descriptions related to coding when provided with prompts in either code or natural language. It boasts compatibility with a wide array of widely-used programming languages, including but not limited to Python, C++, Java, PHP, TypeScript (JavaScript), C#, Rust, and many others.\n",
        "\n",
        "Code Llama is an advanced LLM (large language model) with the ability to create code and provide explanations about code in response to prompts written in code or natural language.\n",
        "\n",
        "It is freely accessible for both research and commercial purposes. This model is constructed upon the foundation of Llama 2 and comes in three variations:\n",
        "\n",
        "\n",
        "*   A foundational model that knows many programming languages\n",
        "*   A specialized model for Python, tailored to excel in Python-related tasks\n",
        "*   An Instruct model, fine-tuned to comprehend and respond to natural language instructions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s17PedUnaIah"
      },
      "source": [
        "## Instructions\n",
        "\n",
        "1. (optional) Save notebook in you Google Drive: File -> Save copy in Drive\n",
        "2. Change runtime type to T4 in the top right corner\n",
        "3. Runtime -> Run all\n",
        "4. While waiting for the process to complete, we watch the video below together\n",
        "5. Read throug the notes in each cell and follow the installation process\n",
        "6. Wait for the gradio link to appear at the end of the notebook (e.g., Running on public URL: https://ff64af66ebaae89911.gradio.live)\n",
        "7. You have now delpoyed your own LLM model for programming tasks.\n",
        "8. Ask as many coding questions as you want :)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 321
        },
        "id": "nBfF-hbYaOTh",
        "outputId": "9ae98d6f-c711-4e0c-a693-828da27cddc6"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "        <iframe\n",
              "            width=\"400\"\n",
              "            height=\"300\"\n",
              "            src=\"https://www.youtube.com/embed/RLYoEyIHL6A&t=4s\"\n",
              "            frameborder=\"0\"\n",
              "            allowfullscreen\n",
              "            \n",
              "        ></iframe>\n",
              "        "
            ],
            "text/plain": [
              "<IPython.lib.display.YouTubeVideo at 0x7f7882d46470>"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# @title Watch this video on how Google Colab works\n",
        "from IPython.display import YouTubeVideo\n",
        "\n",
        "YouTubeVideo(\"RLYoEyIHL6A&t=4s\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pNbBNXALbOw1",
        "outputId": "a1620c70-fb3c-4c04-ede5-0284279c1252"
      },
      "outputs": [],
      "source": [
        "# Colab starts a virtual machine in the background. The virtual machine is a fully fledged Linux system.\n",
        "# Command for the Linux systems are initiated by \"!\"\n",
        "# Command to list the files and directories on Linux\n",
        "!ls"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eUgUE7h4bQYJ"
      },
      "outputs": [],
      "source": [
        "# Change directory to /content. \"/\" indicates the root directory on Linux systems; all directories start at the root directory\n",
        "!cd /content"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZCajfFvhbVZr",
        "outputId": "45f4e7c9-179d-4a17-ce01-159362c59fb6"
      },
      "outputs": [],
      "source": [
        "# ls to show directories and files of /content\n",
        "!ls"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mwzGCMqhbYer",
        "outputId": "8be2b1ca-0318-4c5b-e21e-e84971af854c"
      },
      "outputs": [],
      "source": [
        "# \"Git\" is a very powerful subversioning tool that takes care of versioning you code.\n",
        "!git help"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MNjTfJ21badq",
        "outputId": "aa5c4531-0c85-4637-bfed-7b51ed5b3371"
      },
      "outputs": [],
      "source": [
        "# We use git to clone the \"text-generation-webui\" from GitHub. Github stores and manages your code, as well as tracks and controls changes to the code base.\n",
        "# \"text-generation-webui\" allows as to deploy trained LLMs using a \"chat-like\" interface (similar to chatGPT)\n",
        "!git clone -b v2.5 https://github.com/camenduru/text-generation-webui"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ADvu1Zv1bbte",
        "outputId": "b8bb799a-23b0-4d59-8366-96a5b21b3e11"
      },
      "outputs": [],
      "source": [
        "# Change to the new directory that was created when cloning text-generation-webui\n",
        "%cd /content/text-generation-webui"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x7D06jOwbebC",
        "outputId": "5876d413-70e3-4e75-a34f-78a4f7710f4b"
      },
      "outputs": [],
      "source": [
        "# pip is tool for installing Python packages. Syntax: pip install pkg_name, pip uninstall pkg_name, ...\n",
        "# We install all packages defined in the file \"requirements.txt\"\n",
        "!pip install -q -r requirements.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vBHLRSBoOVT-"
      },
      "outputs": [],
      "source": [
        "# We want to use the dark theme of the text-webui \ud83c\udf11\n",
        "!echo \"dark_theme: true\" > /content/settings.yaml\n",
        "# And a specific chat sytle\n",
        "!echo \"chat_style: wpp\" >> /content/settings.yaml\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "0b34333873154be3a05b6bfedec42d62",
            "a43ddfe82e1f42499ee6ef3d09b87474",
            "3018e1ef800d4802afc722fa4df91d91",
            "55f6f00c5c4a486c9d41f2be4bf3dda6",
            "72098df97c7642babd002a878ec49d09",
            "e7fe37cf28a6452dab39730a81acd353",
            "cfe59c25e65f4880868433776a19a512",
            "9bb1d63b8de94100a3b351cf31950ca9",
            "67b46f0667b7419ca2238f120c60cbdd",
            "3c301e0e8f3e4e4cab7eddba08d3ff96",
            "ee8282308ae444a1a8c18363b88a2748"
          ]
        },
        "id": "Kt1rU01EOZF2",
        "outputId": "1c677434-0342-4be6-f83b-c7b8968eb217"
      },
      "outputs": [],
      "source": [
        "from pathlib import Path\n",
        "import requests\n",
        "import shutil\n",
        "from tqdm.auto import tqdm\n",
        "\n",
        "\n",
        "# Function to download a file\n",
        "def download_file(url, dest):\n",
        "    # makes an HTTP request within a context manager\n",
        "    with requests.get(url, stream=True) as r:\n",
        "\n",
        "        # check header to get content length, in bytes\n",
        "        total_length = int(r.headers.get(\"Content-Length\"))\n",
        "\n",
        "        # Implements a progress bar via tqdm\n",
        "        with tqdm.wrapattr(r.raw, \"read\", total=total_length, desc=\"\")as raw:\n",
        "\n",
        "            # Saves the output to dest\n",
        "            with open(dest, 'wb')as output:\n",
        "                shutil.copyfileobj(raw, output)\n",
        "\n",
        "# File avaialable at\n",
        "urls = [\n",
        "    \"https://huggingface.co/TheBloke/CodeLlama-13B-Instruct-GGUF/resolve/main/codellama-13b-instruct.Q5_0.gguf\",\n",
        "    'https://huggingface.co/TheBloke/CodeLlama-13B-Instruct-GGUF/raw/main/config.json'\n",
        "    ]\n",
        "\n",
        "# Save the download to the following directory\n",
        "\n",
        "model_dir = Path('/content/text-generation-webui/models/codellama-13b-instruct')\n",
        "model_dir.mkdir(exist_ok=True)\n",
        "\n",
        "# Download the urls\n",
        "for url in urls:\n",
        "    save_to = model_dir / Path(url).name\n",
        "    download_file(url, save_to)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7OPIoJo7R5j1",
        "outputId": "32eabc25-4359-4e55-cce9-c55c72fd6404"
      },
      "outputs": [],
      "source": [
        "%cd /content/text-generation-webui\n",
        "!python server.py --share --settings /content/settings.yaml --model {model_dir} --gpu-memory 15000MB --n-gpu-layers 36"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8myUx8opbu_m"
      },
      "source": [
        "## Example prompts\n",
        "\n",
        "1. Provide the answer in Python. Write a function that adds two numbers\n",
        "2. Provide the answer in Python. Write a function to download the file https://huggingface.co/TheBloke/CodeLlama-13B-Instruct-GGUF/raw/main/config.json to the directory /content/models\n",
        "3. Provide the answer in Python. Write a function that computes the set of sums of all contiguous sublists of a given list.\n",
        "4. Provide the answer in Python. Write the class for a basic neural network implementation in pytorch\n",
        "\n",
        "## Template for code\n",
        "\n",
        "\n",
        "```\n",
        "[INST] Write code to solve the following coding problem that obeys the constraints and passes the example test cases. Please wrap your code answer using ```:\n",
        "{prompt}\n",
        "[/INST]\n",
        "```\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0b34333873154be3a05b6bfedec42d62": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a43ddfe82e1f42499ee6ef3d09b87474",
              "IPY_MODEL_3018e1ef800d4802afc722fa4df91d91",
              "IPY_MODEL_55f6f00c5c4a486c9d41f2be4bf3dda6"
            ],
            "layout": "IPY_MODEL_72098df97c7642babd002a878ec49d09"
          }
        },
        "3018e1ef800d4802afc722fa4df91d91": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9bb1d63b8de94100a3b351cf31950ca9",
            "max": 8972409856,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_67b46f0667b7419ca2238f120c60cbdd",
            "value": 1926299648
          }
        },
        "3c301e0e8f3e4e4cab7eddba08d3ff96": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "55f6f00c5c4a486c9d41f2be4bf3dda6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3c301e0e8f3e4e4cab7eddba08d3ff96",
            "placeholder": "\u200b",
            "style": "IPY_MODEL_ee8282308ae444a1a8c18363b88a2748",
            "value": " 1.79G/8.36G [00:51&lt;02:33, 45.8MB/s]"
          }
        },
        "67b46f0667b7419ca2238f120c60cbdd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "72098df97c7642babd002a878ec49d09": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9bb1d63b8de94100a3b351cf31950ca9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a43ddfe82e1f42499ee6ef3d09b87474": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e7fe37cf28a6452dab39730a81acd353",
            "placeholder": "\u200b",
            "style": "IPY_MODEL_cfe59c25e65f4880868433776a19a512",
            "value": " 21%"
          }
        },
        "cfe59c25e65f4880868433776a19a512": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e7fe37cf28a6452dab39730a81acd353": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ee8282308ae444a1a8c18363b88a2748": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}