fastai-primer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "fastai-primer",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPPn59V6ToDIit1Ry+WAr0S",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alexcu/fastai-primer/blob/main/fastai-primer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "--patURUTRyh"
      },
      "source": [
        "# Fast AI Primer 🤖\n",
        "\n",
        "_Learn how to train a semi-decent machine learning model in 5 steps using [FastAI](https://fast.ai)!_\n",
        "\n",
        "<small>**Disclaimer:** This is a quick-n-dirty primer and skips a lot of important details. If you are interested in more info, check out FastAI's [tutorial](https://docs.fast.ai/tutorial.vision.html) or [free course](https://course.fast.ai).</small>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0_Q9Clp7Md9O"
      },
      "source": [
        "## 0️⃣: Prerequsites\n",
        "\n",
        "Make sure you give your notebook a GPU to work with!\n",
        "\n",
        "<img src=\"https://i.imgur.com/GX9QaIN.png\" width=\"600\">\n",
        "\n",
        "Then run the following code by pressing <strong><kbd>⌘</kbd>+<kbd>Enter</kbd></strong> (or hover over the cell below and click the ▶ button) to download and set up FastAI.\n",
        "\n",
        "Authorise your application when prompted:\n",
        "\n",
        "<img src=\"https://i.imgur.com/5eqPT7L.png\" width=\"600\">\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k-qnRSCH96Ld"
      },
      "source": [
        "!pip install -Uqq fastbook\n",
        "!pip install --upgrade git+https://github.com/fastai/fastai.git\n",
        "!apt-get -y install jq\n",
        "% matplotlib inline\n",
        "import fastbook\n",
        "fastbook.setup_book()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ImWfxlo6Ma--"
      },
      "source": [
        "## 1️⃣: Download some images from the [Creative Commons API](https://api.creativecommons.engineering/v1/#operation/image_search)\n",
        "\n",
        "Replace `labels` on line 17 to train something different. <br><small>(If line numbers are off, you can enable them in Tools 👉 Settings 👉 Editor 👉 Show Line Numbers.)</small>\n",
        "\n",
        "Here are some ideas you can try out:\n",
        "\n",
        "Idea|Labels\n",
        "---|---\n",
        "🎨&nbsp;&nbsp;&nbsp;Art Movements|`[\"impressionism\", \"cubism art\", \"pop art\"]`\n",
        "🍦&nbsp;&nbsp;&nbsp;Ice Cream Flavours|`[\"chocolate ice cream\", \"vanilla ice cream\", \"mint ice cream\"]`\n",
        "🐕&nbsp;&nbsp;&nbsp;Dog Breeds|`[\"pug\", \"chiwawa\", \"shiba inu\"]`\n",
        "💐&nbsp;&nbsp;&nbsp;Flowers|`[\"sunflowers\", \"lavender\", \"roses\"]`\n",
        "🍺&nbsp;&nbsp;&nbsp;Beverages|`[\"beer\", \"whisky\", \"red wine\"]`\n",
        "🧀&nbsp;&nbsp;&nbsp;Cheeses|`[\"blue cheese\", \"mozzarella\", \"camembert\"]`\n",
        "\n",
        "<small>**NB:** Add `--show-progress` after `-q` on line 34 if the `wget` command is taking too to monitor progress...</small>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gWMtTuftg7hB"
      },
      "source": [
        "from pathlib import Path\n",
        "\n",
        "# Directory where we will download our all our images - /datasets/{label}\n",
        "datasets_dir = Path(\"/datasets\")\n",
        "\n",
        "# We can run shell commands using ! and insert variables with $\n",
        "# E.g., make sure our datasets_dir is empty\n",
        "! rm -rf \"$datasets_dir\"\n",
        "\n",
        "# Total number of images per label in our dataset (max is 500)\n",
        "num_imgs_per_label = 200\n",
        "\n",
        "# Quality of images (small/medium/large)\n",
        "image_quality = \"medium\"\n",
        "\n",
        "# Label(s) we want to search images for and train on\n",
        "labels = [\"impressionism\", \"cubism art\", \"pop art\"]\n",
        "\n",
        "for label in labels:\n",
        "  # Create a directory where we'll download all the images for this label\n",
        "  label_dataset_dir = datasets_dir / label\n",
        "  ! mkdir -p \"$label_dataset_dir\"\n",
        "\n",
        "  # Where we'll store a txt file of all URLs for us to download\n",
        "  label_urls_txt = datasets_dir / f\"{label}-imgs-urls.txt\"\n",
        "  \n",
        "  # Request to Creative Commons API and download the results\n",
        "  cc_img_api_url = (\n",
        "      f\"https://api.openverse.engineering/v1/images?\"\n",
        "      f\"page_size={num_imgs_per_label}&\"\n",
        "      f\"extension=jpg&\"\n",
        "      f\"size={image_quality}&\"\n",
        "      f\"q=\" + label.replace(\" \", \"%20\")\n",
        "  )\n",
        "\n",
        "  # Bearer Token for Creative Commons API (see https://bit.ly/32XpSav)\n",
        "  bearer_token = 'DLBYIcfnKfolaXKcmMC8RIDCavc2hW'\n",
        "\n",
        "  print(f\"* Requesting {num_imgs_per_label} random '{label}' images:\\n  {cc_img_api_url}\")\n",
        "  ! curl -Ls -H \"Authorization: Bearer $bearer_token\" \"$cc_img_api_url\"\\\n",
        "      |  jq -r '.results[].url' > \"$label_urls_txt\"\n",
        "  ! echo \"* Got $(wc -l < '$label_urls_txt') images of '$label'. Downloading...\"\n",
        "  ! wget -q -i \"$label_urls_txt\" -P \"$label_dataset_dir\"\n",
        "\n",
        "print(\"Done!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pO-q2rg7fIR4"
      },
      "source": [
        "### What images did we download?\n",
        "\n",
        "Re-run the cell below to shuffle through some of the images we downloaded.\n",
        "\n",
        "FastAI comes with some neat helper functions (some which wrap [PyTorch](https://pytorch.org)). E.g., here we use:\n",
        "\n",
        "* [**`get_image_files`**](https://docs.fast.ai/data.transforms.html#get_image_files) to return a list of image files in a directory\n",
        "* [**`load_image`**](https://docs.fast.ai/vision.core.html#load_image) to load an image into memory\n",
        "* [**`show_images`**](https://docs.fast.ai/torch_core.html#show_images) to show multiple images in a grid using [matplotlib](https://matplotlib.org)\n",
        "\n",
        "<small>**NB:** Importing FastAI with a wildcard is considered 'safe' according to its authors...</small>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DVrLSiN4fHgI"
      },
      "source": [
        "from fastai.vision.all import *\n",
        "\n",
        "for label in labels:\n",
        "  images = get_image_files(datasets_dir, folders=label)\n",
        "  sample = [load_image(image) for image in images.shuffle()[:5]]\n",
        "  show_images(sample, nrows=1, ncols=5, imsize=3, suptitle=label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3W31CM4MUy1"
      },
      "source": [
        "## 2️⃣: Loading the images into memory\n",
        "\n",
        "An important part of training a model is to see how well it handles previously-unseen data.\n",
        "\n",
        "So, we can split the data we downloaded into two datasets:\n",
        "\n",
        "  * a **training dataset**, to actually teach the model what labels to look for;\n",
        "  * a **validation dataset**, to validate whether the model can accurately assess unseen images.\n",
        "\n",
        "FastAI has a helper function, [**`ImageDataLoaders.from_folder`**](https://docs.fast.ai/vision.data.html#ImageDataLoaders.from_folder), to help us load  these two datasets into memory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bIJccBaHB7Uu"
      },
      "source": [
        "# How much of our data we will reserve for training vs. validation\n",
        "validation_dataset_pct = 0.35\n",
        "\n",
        "# Resize all images consistently to 256x256 pixels\n",
        "#  ↑ resolution = ↑ accuracy = ↑ computation power = ↑ training time\n",
        "images_transformations = Resize(256)\n",
        "\n",
        "print(f\"* Loading images from {datasets_dir}:\")\n",
        "print(f\"    We will use {len(labels)} labels: {labels}\")\n",
        "print(f\"    We will reserve {1-validation_dataset_pct:.0%} of our data for training\")\n",
        "\n",
        "dataloader = ImageDataLoaders.from_folder(\n",
        "  datasets_dir,                     # <- Where our data is stored (i.e., /datasets)\n",
        "  vocab=labels,                     # <- What subdirectories we have (one for each label)\n",
        "  valid_pct=validation_dataset_pct, # <- % of data we set aside for validation\n",
        "  item_tfms=images_transformations  # <- Transformations to make to each image\n",
        ")\n",
        "\n",
        "for label in labels:\n",
        "  num_train_imgs = len([path for path in dataloader.train_ds.items if path.parent.name == label])\n",
        "  num_valid_imgs = len([path for path in dataloader.valid_ds.items if path.parent.name == label])\n",
        "  print(f\"* For '{label}...'\")\n",
        "  print(f\"    We have {num_train_imgs} training images\")\n",
        "  print(f\"    We have {num_valid_imgs} validation images\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iYSKoUfjyDvU"
      },
      "source": [
        "## 3️⃣: Training the model\n",
        "\n",
        "FastAI has [different **pre-trained models**](https://fastai1.fast.ai/vision.models.html#Computer-Vision-models-zoo) which we can piggy back off:\n",
        "\n",
        "Architecture|First Appeared|Available Implementations In FastAI\n",
        "---|---|---\n",
        "[AlexNet](https://en.wikipedia.org/wiki/AlexNet)|[2012](https://papers.nips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf)|`alexnet`\n",
        "[Resnet](https://en.wikipedia.org/wiki/Residual_neural_network)|[2015](https://arxiv.org/abs/1512.03385)|`resnet18,resnet34,resnet50,resnet101,resnet152`\n",
        "[Squeezenet](https://en.wikipedia.org/wiki/SqueezeNet)|[2016](https://arxiv.org/abs/1602.07360)|`squeezenet1_0,squeezenet1_1`\n",
        "[Densenet](https://paperswithcode.com/method/densenet)|[2016](https://arxiv.org/abs/1608.06993)|`densenet121,densenet169,densenet201,densenet161`\n",
        "\n",
        "We'll select one of these ☝️ as our _base pretrained model_ and use FastAI's [**`cnn_learner`**](https://docs.fast.ai/vision.learner.html#cnn_learner) to set up a [Convolutional Neural Network](https://en.wikipedia.org/wiki/Convolutional_neural_network) for us to play with.\n",
        "\n",
        "We'll _specialise_ the base model to our specific purpose by calling [**`fine_tune`**](https://docs.fast.ai/callback.schedule.html#Learner.fine_tune), which trains the model.\n",
        "\n",
        "<details>\n",
        "<summary>Why do we use a base model?</summary>\n",
        "\n",
        "Here's a quote from [Neurohive](https://neurohive.io/en/popular-networks/alexnet-imagenet-classification-with-deep-convolutional-neural-networks/) about [AlexNet](https://en.wikipedia.org/wiki/AlexNet):\n",
        "\n",
        "> AlexNet was trained for 6 days simultaneously on two Nvidia Geforce GTX 580 GPUs on ImageNet, a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories.\n",
        "\n",
        "We use a base model so we don't have to do that ☝️.\n",
        "\n",
        "Here is a visualisation and explanation of each of AlexNet's 5 layers, and what _features_ it learnt ([source](https://arxiv.org/pdf/1311.2901.pdf)). The model we train _piggy-backs_ off of what the base model already knows.\n",
        "\n",
        "Layer|Visualisation of Weights Learned|Patches of example Training Data|Description\n",
        "---|---|---|---\n",
        "1|![](https://i.imgur.com/hDKs1b7.png)|![](https://i.imgur.com/XDW8KwH.png)|We can see that the first layer of the model has discovered weights representing edges (diagonal, horizontal, vertical) in addition to gradients.<br><br>These are foundational building blocks, similar to how basic visual machinery in the human eye!\n",
        "2|![](https://i.imgur.com/b0TLzV9.png)|![](https://i.imgur.com/PTPric9.png)|We see that the model has learned to create feature detectors to see corners, repeating lines, circles, and other simple patterns.<br><br>Each weight picture matches small patches from example training data; e.g., sunsets on the bottom RHS.<br><br>These are built up from the foundational building blocks in layer 1.\n",
        "3|![](https://i.imgur.com/EuDQvnS.png)|![](https://i.imgur.com/DBBrHxK.png)|We see the features learned are now able to identify and match with higher-level semantic components (e.g., printed text or even people)\n",
        "4|![](https://i.imgur.com/gRRyrC2.png)|![](https://i.imgur.com/EuVmz5d.png)|We see that the model is now able to understand higher-level concepts, such as similar-looking dogs.\n",
        "5|![](https://i.imgur.com/sYsWVEe.png)|![](https://i.imgur.com/ekbE8iq.png)|We see that the model can now identify different types of dogs, but classify them together!\n",
        "</details>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wS1oaerjyBnb"
      },
      "source": [
        "# Base pretrained model to piggy back off (refer to table above)\n",
        "pre_trained_model = resnet34 \n",
        "\n",
        "# How many times we will iterate through the each image in the training dataset\n",
        "num_epochs = 1\n",
        "\n",
        "learner = cnn_learner(\n",
        "    dataloader,         # <- What data has been loaded into memory to train from\n",
        "    pre_trained_model,  # <- Which pre-trained model we want to piggy-back off\n",
        "    metrics=accuracy    # <- Report back the % of correct predictions\n",
        ")\n",
        "\n",
        "print(f\"* Training model with {num_epochs} epoch...\")\n",
        "\n",
        "# Re-train the very last layers of the pre_trained_model for a specific purpose\n",
        "learner.fine_tune(num_epochs)\n",
        "\n",
        "print(f\"* Done! Our model is {learner.metrics[0].value:.3%} accurate\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTknlKe9MLwo"
      },
      "source": [
        "## 4️⃣: What did the model correctly and incorrectly learn?\n",
        "\n",
        "FastAI has a [**`ClassificationInterpretation`**](https://docs.fast.ai/interpret.html#ClassificationInterpretation) class to help us understand what our model learnt correctly vs. incorrectly.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ph--CtnxJ8AS"
      },
      "source": [
        "learning_interpreter = ClassificationInterpretation.from_learner(learner)\n",
        "\n",
        "# A confusion matrix will show us correct predicted vs. incorrect predictions\n",
        "learning_interpreter.plot_confusion_matrix()\n",
        "\n",
        "# We can plot the \"top losses\" - the higher the loss the lower the model is\n",
        "# confident in the result\n",
        "learning_interpreter.plot_top_losses(9, figsize=(15,10))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NTlOw979HiV_"
      },
      "source": [
        "## 5️⃣: Running a prediction\n",
        "\n",
        "1. Update `image_url` to any random accessable image on the internet\n",
        "1. The cell to see which of the labels the model thinks it is"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "htAlY8eukSWt"
      },
      "source": [
        "import tempfile\n",
        "\n",
        "# Grab some random images from the internet, and see what our model thinks it is\n",
        "images = [\n",
        "  \"https://i.imgur.com/vqtjuLU.png\",\n",
        "  \"https://i.imgur.com/Rzj3r7P.jpg\",\n",
        "  \"https://i.imgur.com/wChxwgY.jpg\",\n",
        "]\n",
        "\n",
        "for image_url in images: \n",
        "  image_path = tempfile.mktemp()\n",
        "\n",
        "  print(f\"* Downloading {image_url} to {image_path}...\")\n",
        "  ! wget -q --show-progress -O \"$image_path\" \"$image_url\"\n",
        "\n",
        "  print(f\"* Running prediction for this image...\")\n",
        "  _, _, predictions = learner.predict(image_path)\n",
        "\n",
        "  print(\"* This how confident the model is for each label:\")\n",
        "  for index, label in enumerate(dataloader.vocab):\n",
        "    print(f\"    {label}:\\t{predictions[index]:.3%}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ywzMfIM6RAPM"
      },
      "source": [
        "## 6️⃣: Cleaning up bad training data\n",
        "\n",
        "> <i> 💩 in means 💩 out </i>\n",
        "\n",
        "Run the following cell below, and delete or reclassify any bad images"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fTej1oFvajZh"
      },
      "source": [
        "from fastai.vision.widgets import ImageClassifierCleaner\n",
        "\n",
        "# We can use an ImageClassifierCleaner on our learner to prune away mistakes\n",
        "# or bad data\n",
        "dataset_cleaner = ImageClassifierCleaner(learner, max_n=100)\n",
        "dataset_cleaner"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ht0JXYYcWCFo"
      },
      "source": [
        "When done, run the following cell below to delete the bad images and move the misclassified images. \n",
        "\n",
        "Then re-train the model by going back to Step 3️⃣."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eX5wXZbwRZPl"
      },
      "source": [
        "# Go through all those images in the cleaner that we marked for deletion \n",
        "# and remove the files from our /datasets/{label} directory\n",
        "print(\"* Deleting bad images:\")\n",
        "for i in dataset_cleaner.delete():\n",
        "  image = dataset_cleaner.fns[i]\n",
        "  ! rm \"$image\"\n",
        "  print(f\"  Deleted {image}...\")\n",
        "  \n",
        "# Go through all those images in the cleaner marked for reclassification\n",
        "# and move them to the right category subdirectory\n",
        "print(\"* Moving misclassified images:\")\n",
        "for i, label in dataset_cleaner.change():\n",
        "  image = dataset_cleaner.fns[i]\n",
        "  new_path = datasets_dir / label\n",
        "  ! mv \"$image\" \"$new_path\"\n",
        "  print(f\"  Moved {image} into {new_path}\")\n",
        "\n",
        "print(\"Done! Retrain your model again and check if there is any improvement!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tFZOgQIpmmeU"
      },
      "source": [
        "## 7️⃣: Save model to file\n",
        "\n",
        "We can save our model to a [**Pickle file**](https://pythonnumericalmethods.berkeley.edu/notebooks/chapter11.03-Pickle-Files.html) <img src=\"https://emoji.slack-edge.com/T027TU47K/pickle-rick/cbbfd96e6c843c11.png\" width=\"32\">, which serialises the model to text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QJ0vGL51mmSd"
      },
      "source": [
        "learner.export(fname=\"/my_model.pkl\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q_E8rEclnHOJ"
      },
      "source": [
        "And now we can load in our model somewhere else, e.g.:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P-kcbKy0nGv0"
      },
      "source": [
        "from fastai import load_learner\n",
        "\n",
        "learner = load_learner('/my_model.pkl')\n",
        "learner.predict('/path/to/some/image')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9GeSStrlTYG"
      },
      "source": [
        "## 💥 Additional Funsies\n",
        "\n",
        "* What's the _least_ amount of training images you can use to train a decent model?\n",
        "* Does increasing/decreasing image quality make a difference?\n",
        "* Increase the number of epochs. Does it make your model better?\n",
        "* Modify the training vs. validation dataset split proportions and see the results\n",
        "* Switch to a different pre-trained model and see the results\n",
        "* Cleanse more of your dataset to remove bad examples"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L0HcUUR76uA2"
      },
      "source": [
        "![](https://i.imgur.com/SGE1hLZ.jpg)"
      ]
    }
  ]
}