train_mlp_and_cnn.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2X5nH8y96pqW"
      },
      "source": [
        "In this notebook, we will compare the performance of CNN (Convolutional Neural Networks) and MLP (Multi-Layer Perceptron) models with the same number of parameters. We will use the MNIST dataset for this task.\n",
        "\n",
        "Adapted from https://github.com/Atcold/NYU-DLSP20/blob/master/06-convnet.ipynb"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "zoKSFQbi7AiF"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "from torchvision import datasets, transforms\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Since the training method is stochastic (Stochastic Gradient Descent), set the seed for reproducibility\n",
        "def set_seed(seed):\n",
        "    print(f'Setting {seed=}')\n",
        "    torch.manual_seed(seed)\n",
        "    np.random.seed(seed)\n",
        "    if torch.cuda.is_available():\n",
        "        torch.cuda.manual_seed_all(seed)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Setting seed=42\n"
          ]
        }
      ],
      "source": [
        "set_seed(42)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "UGmzUgsA5iqR"
      },
      "outputs": [],
      "source": [
        "## Functions to plot the data and the output of the model\n",
        "def set_default(figsize=(10, 10), dpi=100):\n",
        "    plt.style.use(['dark_background', 'bmh'])\n",
        "    plt.rc('axes', facecolor='k')\n",
        "    plt.rc('figure', facecolor='k')\n",
        "    plt.rc('figure', figsize=figsize, dpi=dpi)\n",
        "\n",
        "\n",
        "def plot_data(X, y, d=0, auto=False, zoom=1):\n",
        "    X = X.cpu()\n",
        "    y = y.cpu()\n",
        "    plt.scatter(X.numpy()[:, 0], X.numpy()[:, 1], c=y, s=20, cmap=plt.cm.Spectral)\n",
        "    plt.axis('square')\n",
        "    plt.axis(np.array((-1.1, 1.1, -1.1, 1.1)) * zoom)\n",
        "    if auto is True: \n",
        "        plt.axis('equal')\n",
        "    plt.axis('off')\n",
        "\n",
        "    _m, _c = 0, '.15'\n",
        "    plt.axvline(0, ymin=_m, color=_c, lw=1, zorder=0)\n",
        "    plt.axhline(0, xmin=_m, color=_c, lw=1, zorder=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "tKfznkfy5iqT"
      },
      "outputs": [],
      "source": [
        "set_default()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "bG0xGSHY5iqT"
      },
      "outputs": [],
      "source": [
        "# function to count number of parameters\n",
        "def get_n_params(model):\n",
        "    np=0\n",
        "    for p in list(model.parameters()):\n",
        "        np += p.nelement()\n",
        "    return np"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "Qelsct0y5iqU"
      },
      "outputs": [],
      "source": [
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DiYe3Q0T5iqW"
      },
      "source": [
        "# Load the Dataset (MNIST)\n",
        "\n",
        "\n",
        "We can use some PyTorch DataLoader utilities for this. This will download, shuffle, normalize data and arrange it in batches."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "qZpCUmN_5iqX"
      },
      "outputs": [],
      "source": [
        "input_size  = 28*28   # images are 28x28 pixels\n",
        "output_size = 10      # there are 10 classes\n",
        "def load_dataset():\n",
        "    transform_list = [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]\n",
        "    transform=transforms.Compose(transform_list)\n",
        "\n",
        "    train_loader = torch.utils.data.DataLoader(\n",
        "        datasets.MNIST('data', train=True, download=True, transform=transform),\n",
        "        batch_size=64, shuffle=True)\n",
        "\n",
        "    test_loader = torch.utils.data.DataLoader(\n",
        "        datasets.MNIST('data', train=False, transform=transform),\n",
        "        batch_size=1000, shuffle=True)\n",
        "\n",
        "    return train_loader, test_loader"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "t9ZqSd085iqZ"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABOgAAAHiCAYAAABSnPMoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAr/klEQVR4nO3deZhcdZkv8FPp6nQWEiAJCYsJW2gCYQkQlkQgAhFxroIIGFGuiCu7IggjOuOo6IA7u4oCrugICriBosAwZmHfs7EkbCFAICGQrbuq7h9zn7l6H39vh9PLrzv9+fz77fecN0udOvXt8zxVKYqiUQAAAAAAWQzIvQAAAAAA9GcKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBG1fX9wemVo7tzD+gXbmlcm3uF0lwDoPP66jXA6x86r6++/ovCNQC6Sl+8Dnj9Q+et72vfE3QAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADJS0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABlVcy8AQPdrP3ivMF9y8tpk9sCUH4azu886PplteenAcLbp1nvDHAAAoD/wBB0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADIqJp7Af5epZr+J2nabFS3nXf+Wdsks9qQeji79fYvJLMhJ1fC2ee/OTCZ3Tv5F+HsS7XXk9m+vzwznB3/qdlhDn1RfdoeyeyiKy8JZ8c3p6898RWgKO6bclUymz+5Fs5+epv9Ojg6sKF6/eh9k9kFX708nP3Sez6QzBp3P1x6J+CNe/xrU5LZ3PfF9x/NlaZkduDJHwtnB19/Z7wYQB/jCToAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADKq5l6gt2raaYdk1mhpDmefm7ZJMlu93+vh7IiN0/kdu/8inM3lD6uGJbMLLjksnJ2z68+S2ZNtq8PZ85e+NZlteUcjnIW+qO3QyWF+9mU/TmatzQPD2XpRT2ZPtLWFsyvqLclsj3RUFEVRrH373sls8K0PhbP1NWvig9PnrT5in3Q2simcHXHlrK5ehy72wuT074m/tOidPbgJEHn+jKlhftuMryaztkZ8/xFyOw/0M56gAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABlVcy+QS+0te4b5N6++NJm1Nnfi68L7oLZGLcz/9eIPJrPq6/H3o0/55anJbNiz7eFsy0urk9mQu+eEs5BL0/DhYf76gROS2Rnf+lk4e9Dg14K0/O9jrn5lapj/+bIpyeyv/3ZROPun738nme38k/T1oSiKYrtzZoU5fd9zB6b/3w7Zfnk8fGXX7kIJA5rCuDEu/T5+yOh54eyfK/F1Ceg6r42th/mIAf3rsxF0h3Vvm5zMFr8/fg2etOftyeyTmy4ovdOu3z8tzIcsSX/WXz51bTi79U/T93gDb747XmwD5gk6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyquZeIJeW+c+F+T1rxiaz1ualXb1OlzhzyX7J7InXRoWzV29/bTJbUW+Es2Mumhkv1k3iraB3euZHW4X5XXtf2kObrL8vjr4rzG/aaGoyO2HRoeHsD7e5JZkN33lZvBgbvC+845fJ7IK58f8t8mvafuswnzftymQ26c7jwtkt73qo1E7AP/baMfsms+uOvLCD6Uoy+c7yCeHkLe+ZnMyGLn4knK3HS0Gv8uKJU8L84rPTnwEmt9TC2QHBc1fHL5oezu6x8VPJ7IGPdPTaL7dTURTF1BHHJrMRN5c+bZ/nCToAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQUTX3Arm0L3k+zC++4Jhk9uXDXg9nmx7cKJk9cPLF8WKB817aLcwfmz4kmdWWLwln3zfl5GS26PR4r22LB+IfgH6m/eC9ktk1ky4JZwcUA0uf94TFhySzu2/ZKZx96MPpvW5dPSicHX336mT22CsTwtnmr9yazAZUwlH6geZKe+4V6ITq91eVnl39+PAu3ARY8459wvzz/35lMmttLv+G/MMrDgvzzR+dWfrY0NMqzfF9+prpuyez6z7ztXB2y2pLMvvw4reGs4u/vmMyG/q7+8PZW4eMS2a3/7o1nL1uhxvDPPLq/SOT2YjSR+37PEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBG1dwL9FYjrpqVzDb7zchwtrbs5WQ2cZcPhbOPHHhlMrvxe9PC2dHLZ4Z5pDLrgWS2bfqvAvql+rQ9wvyiKy9JZuOb48tuvagns8PnHRnONh39ejLb5H81wtmdf3xqMmu99OlwdsDT9yWzTe8IR4u2L9eS2XW7pa+HRVEUHzro9GTWdOu98YnpFer7TwrzAwb9V88sQrfYZuiy0rNjb0lfG4A3bslxa8L8oMFR3hTOHr9oejLb/MLyn0+gt1ly6uQwv/OsC4O0JZw95rF3JrP2o9rC2SEvzUlm8SeAonjuY3slszk7RH+e2B9WDQvz8d9Nf75oL33Wvs8TdAAAAACQkYIOAAAAADJS0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACCjau4F+qLaS8tKz7a9OrD07MT3PxrmL14efAV6vVb6vNAfVfaamMxe+tTqcLa1Of06v2dtfN6/vLZzMlv287Hh7MhXZiWzjX8yO5zdOMhyfdX5mKb46+iXfXJVMht9a1dvQ3dY/I7BYT66aUgPbUJZ1W3GJbOjR9xY+riDn3wlzN3VwN+rvmmrMH/kgKvCvK2RflXNbYvP/dQ3W5PZ0GJOPAy9zMKL901m8999cThbD7Kd/nRiODvhrEXJrDP9Q0dOPOmGbjnueV8+Psw3fTr9uaU/8wQdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZVXMv0N/sdM6CMD9h10OS2VVb/zmcnXbMKcls2C9mx4tBPzNgyJAwb//qq8ls9oRfhbNPtq9LZp8698xwdtM7nkpmo4e+EM7WwnTDs88Wi5PZop5bg06ojl9ZenbNvE26bhFKe/rbQ5PZm1vq4ewPXn1TOlyevgZDf9U0ccdkNvlnD3fbeWf86vQw3/46nzPoOx7/xn5hPv/dlyazFfU14ewx896XzHY8Le4BaivL3xMNGJp+L1529G7h7BEbfS193GJwODvhl+n+YfzVs8JZ/jFP0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMqrkX6G9qy1eE+bKTdkpmT924Opz95/N+lMw+854jw9nGfRsns7Ff7uArkhuNOIdeaPW0iWF+84TLSh/7I584I5kNu352ONte+qzQv4y+u557hT6jadTIMF96VGsyG/GeZ8LZ21t/EKSDwtnLL31XMhu9dGY4C/3R4sPTr+VrR97XwXRTmL7v8Xcms9bzHw9nax2cGXpa05jRyeyHR8b3+PUifX9xzLz3hbMD37o4OG55AybtHOa7XDk3mZ035qIOjt6STN58/3vDyR3/LX1e14VyPEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBG1dwL8PfqD8xNZu/9wqfD2Z9+/uvJ7P79fhSfeL90NHHoqeHoDlcsSWbtTyyKzwuZ7Pal+8N8QPD7ixMWHxLODr7+zjIr9UvNlaZk1taIZ5sqHfwAG7TVI+LfMQ7tpvPWD9gjzBtNlWT29PSWcHbdlm3JbMDAWjj7xwMuTmbN6ZWKoiiK52vpvf7liSPD2Zfr9WQ2ZEC885g5K5OZVzf90csnTAnzX5/4tSBtDmdPfHpamLcdn74O1F58KpyF3qYyKP3/eXJL/N4UGXz6wPi8W49NZgtPfFM4e+j0e5PZGaO/F86Oqw5OZul36f9Wa6TfcSu/GBXPLl/YwdF5ozxBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADKq5l6A9Tfiyllhfur8U5LZ8POfCWev2e7mZPbIBy4JZyeM/Ugy2/ELcQdcW/hEmENnLP/fU5LZ58Z8PZytF+mvUb/njzuHs+OKmfFi/I+2Rvqr7usdfDH8TXPT/w47FOmvqqf3WLumOczrRSOZXXXut8LZG0+dVGalDp0z8vthPqCoJLPVjXXh7HO19OvhkhffEs5Ov+WTyWyT+9LXs6Ioii3+uDSZVRbH9w8vzh2czMY0tYWzjbseCnPYEDVN3DGZzTwvvucuikGlzzvrmW3CfOyih0sfG3qbxpq1yWzO2vjeY9+W9HvXDbf8PJzt6N61rFtWjwrzhW3p+6WDBr8Wzt69Ln2PsMmP4v6BrucJOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMqrmXoCuU/nr/cls1dGjw9m9Z5yWzOacc2E4O++g7yez929zaDi7Yv8whk5pH5zONh4wMJydtaYlmW33o+fi84bphmfAkCHJbN7Xd+lg+p5k8v4n3h5OTvjEk8ms1sFZ6R3GH3dfmE/891OT2di9n+3qddbLrS+0hvmLf3hTMhv5SFs4O/Cmu4I0nm0t7g7zSPR6efacqeHs3i2zktnPX9uq5Eaw4Vpwbvo9s63Rfe9e486P80a3nRl6Xm3pC8ns8yd9JJz9+ncuS2a7xR8fip+8OjaZnXf74eFs69Vrkll16YpwdvQ1Lyezg8b+JZw9/tb030dn7i0oxxN0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAIKNq7gXoGdFXTRdFUYy5KJ2vObs9nB1SSX/f9BXb/DacfceRn0wf99dzwlnoTstqGyWz9icW9dwivcCAIUPCfP75uyazeUdcEs7+YdXGyey5S8eHs8NemR3m9H3bfmZW7hXesC2Kp3Kv0KWGHPhi6dnP3XpUmLcWd5Y+NvRW9Wl7hPl5k6/vlvO+9eH3hvlGdz/cLeeFvmbgzXeH+bnb7tMt5+3Me97KI+KdfjfuhmTW1oifyRq8KP1Znp7nCToAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADKq5l6ArlPff1Iye/yYQeHsLpMWJbMhlYElNyqKi1/eI8yH3HB36WNDdzrrr8cks9binh7cpGfUp6Vfqy98anU4O3fyJcnskIdmhLNDD3simQ0rZoezQO+29Q2N3CtAj/vy1d8L812ay78uzlpyYDLb+NhXwtla6bMCubUPjp+ramukX+H1oh7Obnv1U+nzxmvRDTxBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADKq5l6Av1eZvEsyW3D6wHD2ijf/MJkdOGhd6Z06srbRlsxmv7xtPFxf0sXbwN+opKMBHfx+4sL9r0lmlxatZTfKZvEXp4T5dR/4ZjJrbY6vPXveeXwy2/LIR+PFAGADssfA+P6irVErfexZV+2ZzEa/MrP0cYHebdjPZ8c/8I2e2YPu5wk6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyquZeYENU3XbrZPb4CVuGs/824+fJ7KiNXiq9U2ecu3RymN9+4X7JbNMfzurqdWD9NdJRvaiHo9MGL0tmn7x6r3B2+6vSx25+fmU4u3TaZslsxIxnwtnTxv05mb19yD3h7I2vj0lmH3josHB21HeHhjmw4WqqpH/X+0prczi7+R+6ehvoGU9fu0sya67c323n3eK29GeBWredFcht5XvTn7f/W3yfT9/hCToAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQUTX3Ar1VdZtxyWzFXluEszO+eFMyO3GTX5XeqTPOXBJ/NfOsyyYnsxFX3xnOblqfVWon6M0GVdKXx7lv/U44+18HDEpmC9duHs6esPGiMC/rE88dEOY3zZyUzHb4xOwu3gbYUNQa9XTo18D0UfVpe4T5tyf9JJm1NWrh7Ir6mmS29x8+Gc5OWPxomAMbphXbeUPtL/xLAwAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQUTX3At2pusXmyezlK4eGsydte3syO3bY0tI7dcapz+4f5vdePimZjbr24XB2xMpZZVaCXm3MbS8ks3M+PiWcvWDz8q+JAwetS2b7D1pU+rj3rY1/p3Ls7R9LZq0n3BPO7lDMLrUTQMqqvVflXgFKWTNiYJjvP+j1IG0KZ29eNS6ZtX7srnC2HqbAhmqr2+P30+ZT09edtkZXb0N38gQdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyKiae4GOrHvb5HR2xsvh7Lnjf5/MDh0cfT1691laWx3mB954ZjKb8Ll54eyI5bOSma9lpz+qLXg8mS08ZptwdufTTktmj77n4rIrdWjC709OZjteFn/Feut993T1OgChporf9QJAd6r89f4wv/rV0cns2GHPhrOrJm6RzAY+/Uw4S9dzVwUAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEbV3At0ZNG70h3igl1/2W3nvXT59snswtsPDWcrtUoym3Dek+HsDkvnJLNaOAm8Ee1PLArz8Wek88PP2Ltrl/kbrcVdyazRbWcF+MfW3rJZmNcm1XtoE+g5w+9/PsxPe+bgZPadsbd39ToAoW999+hkduxZF4azW/zLY8ls2fLd4hPPfjDOecM8QQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyqhRF0VifH5xeSX91L7B+bmlcm3uF0lwDoPP66jXA6x86r6++/ovCNQC6Sl+8Dnj9935No0Yms4HXVcPZX4z/bTKb9sCx4eyI972YzGrLV4Sz/c36vvY9QQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEbV3AsAAAAA8MbVXlqWzNYdNTKc3ekbH09mc6d/N5w9fMKH0+HsB8NZ/jFP0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMqrkXAAAAAKBr1V5aFuY7HJ/ODy/27uDoD5bYiIgn6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyKhSFEUj9xIAAAAA0F95gg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADJS0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADJS0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyqq7vD06vHN2de0C/cEvj2twrlOYaAJ3XV68BXv/QeX319V8UrgHQVfridcDrHzpvfV/7nqADAAAAgIwUdAAAAACQkYIOAAAAADJS0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADJS0AEAAABARgo6AAAAAMiomnsBAP6fBVftlcyefNsPwtlvvrxdMrvlPZPD2dqjC+LFAAAAMhv5102T2YBKI5x9ceryLt6ma3mCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGRUzb0APaNp5Igwr2w8PJk9ddSW4eyaUemvMh7/hQfC2fqqVWEOG5qmiTuG+Q0HXZrM2hrN4ewpm85PZtfudmg4O+zRMAa6QGWviWFeH5i+LXv2LUPD2UdOuyyZtTVq8WKZHPLw0cls6BFLwtn6mjVdvQ5kV2lpCfNVb989me322fiee+Hea0vtBNDTFvxgcpjfNe7CZDbljlPC2e2K+8us1GM8QQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEbV3Auw/gbsMiHMF35mcDL70K4zw9kzR95caqeO7DTmxDDf4YP3dMt5odd69vkwPn3Be5PZnyZe19XbAG9QY8ruYb7wgwOT2bcOviacba60J7Ppg1eGs22N9O9c60U9nM3lT7v8RzKb9OMPhbPbnvRcMqu9tKz0TpBT02ajwvzWS7+TzO5YE3+s+9q27wzz9icXhzlAV1pw+T7J7K5DvxXOrqw3ktnw29OdSF/gCToAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQUfx93HS5yt67hvljZzQls9v2vySc3aypJZkN6KCL/d2qTZPZE2tHh7OnbDo/mf34wCvC2S/tfXwya9z1UDgLfVFt+YowX/zMDulwYhcvA7xhjfNeDvN5E37VQ5ts2O6femWYv23fk5NZy++WdfU60OsdMKg9zL88bkSYD3hycVeuAxB6yx5zk9mwAQPD2ZMXH5bMRn13VumdegNP0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkFE19wJ9UdNmm4X5ggu3Sma/mXpZOLtdc3OQtoSzkateHRvm1x+1fzKrt0Q7FcUpv52fzCa31MLZ1WMGJ7NB4ST0TU1jRof5ATst6KFNgDKevS1+Py0mlD/2rDXp9/kP/f6j8XAlyBrl9imKothvz/iadNU2fyx/cKDLNFU8dwF92eoj9klmo858MpxdO6MpmbUveb70Tp3xwslTw/yCMd9KZj95detw9pXPjEtmA4pl8WK9nCs5AAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyKiae4G+6NnjdgjzR6ZdGKTNXbvM3/jJq2OT2fXvir/muDZ/QTKr7DGx9E7A/2fY0DD+pxF3dctpX9irEuabPNiazGqPpq8P0N+MO//uMD/yP44tfezKurZktsOTc0oftzOWjxoZ5rfMHpbMpg9eWfq8Bz80I8yH3/pIMquXPiv0XbVG/D+/bUj8sa+lK5cB3rDjzv9tMjth+NPh7PS9Tkpmg377fOmdOuP4U34f5pNa0ledj37pyHB2xB2zSu3UF3iCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjKq5F+iLtjp8Ubcd+9rXNk9m31xwSDg75uxGMqvNX1h6p1d2HV56Fvh7tceeDPPP/WZGMjvq2EtLn/eR910U5nus+EQyG/vogtLnhQ1No21dmNfmP9ZDm/SMpe9uDfNdB94QpC2lz/vccyPCfKNVT5Q+NvRHL+zVHOZj/9BDiwD/0JJ1mySzerE4nG0fXOnibdZPfdoeyeyIjS4OZ9sag5NZ+6A8f57ewBN0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAIKNq7gX6pI+2hPHOp5yWzMb+qRbODn3k+WQ2avGCcDY+cnmrxvTfrzmGnrb9WbPT4bE9twfQf7x40pRkNuG4eeHsmKb4nqisnc5+Msy7654Hcmq0tYX5grY1yay1eVA4u3rbdaV2ArrGwov2DfNfj7w4mV2+vDWc3WT2s8msPV4r1LTJxmH+0lmvJ7Mtq/H9wRnPTU1mY35wTzjbCNO+zRN0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkVM29QF9Ue+zJMB9/RpxH2ktPdp+2vVfmXgEoiqK50hTmbY0eWgTodV44dWoyO/6k34ezxw3/ejIbNmBg6Z068qUX90xmjbXruu280FvVlr4Q5qc/PiOZ3TThhq5eB3iDmnYcn8x+/I7Lw9lVjbZk9qvPHhrODn76znixkhZetm2YP7znFcnsltXD4mPvvbbUThs6T9ABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjKq5F2D9PfWvU8O8fUgjHVY6OHgw+u4dZnUwnHbqM28J88E33ZvMgpWgX2pr1MK8XtR7aBPov5om7hjmC07YNJlN2//hrl7nf/x27MXJrONrw8DS532srT2Zzbj8zHB23K+XJrP6ysdL7wQA3aHx5klh/t4f/DaZTW6J7+Mn3PSJZNZ6/Z3hbGcsOm9KMrv7wG92MJ2uk875/ofCya2KmR0cu3/yBB0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABlVcy+wIWoaPjyZrdlnh3C2+TNLk9mDEy4uvVNzpSnM2xq10se+dfWQZPbMx8aFs432uaXPCwDdofHmScnsg1f9Opw9YuhLXbzN+srzO9fTH5uRzLa6YGY4W/7OA3ijNhqxKvcK0CtUmgeG+ZJTJyezu8+KP49Hn7nbGvH79Lsn3ZvMbrxgSjg7/gsPJLMBm48OZw//p9nJrKmohLOTZn4omY07P74H4B/zBB0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADIqJp7gd6q0tKSzNZN2zWcPeOyHyezgwb/OZxdWlubzG5dvWk4+68Ljkhm10y8Opzdspr+83Zk0IC2ZPbEezYJZ7ebPyiZ1desKbsSAHSLpqIR5gMy/e6zudKUzNrilTvlpp1+ncwOeP8p4ezGP53d1esACdfteUWYn1a8uYc2gbyeP3FymN951oXJrN7BsaP32x+9ulU4+5XN56Sz49JZURTFudP3TWZv3fgP4exBg19LZnPWpj+rF0VRjDvmoTDnjfMEHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADJS0AEAAABARgo6AAAAAMhIQQcAAAAAGVVzL5DLgEGDwnzZjD2S2R1fuaj0eSdec1qYv+nWWjJr+d1d4ezILV5LZtfcvFc4e+bIh8M8sm9LWzJ78IPx39WUp09PZmN+9EA4W1+1Kl4MNjDNlaYwb2uUP/bwqS+UH4YNTOWv9yezH7zrsHD2nz84MpmNu3ldONu0uj3Mu8vCDzcns3mHXd6DmwCRp/9rbDqc0HN7QG/34olTktnMc74dzq6spz/bPto2NJz97FkfT2aDlsX3AH/+yqJkdtU2fwxnv7L5nGQ2oINnsupBNnlgvPMZj81NZhce9e74vA+kZ/szT9ABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjKq5F+hOlZaWZDbvm7uFs/OOuKj0eY+Y/65k1vq1J8LZ2tIXkll17JvC2d1vfCqZfXrko+Hsinr6K5T3ve7McHaLCemd/7zrL8LZWf+S/nuecew7wtmXLto1mQ1alv567I403XZv6VnoTm2NWpjXwy9Kj92++zXJ7PD9PhwPz36w9Hmhr6k9uiDMtzu7hxbpQjst3CwdHtZzewCxjZ5ulJ4dVolnm3ZuTWYdXfegt9n5A3OT2Y2vjwlnv/K9Y5PZFt+YGc4OKebEiwWWnZnuJ864+IBw9ltb3lH6vJGmSiXMP/3QUclsywfi/oF/zBN0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkVM29QGdUqvH687+9ezKbd/il4ewz7WuT2eHfPTuc3ebKx5NZ+9IXwtm26Xsls10uuC+c/fzoe5LZVa9uHc7++LPvTGbjfzU7nG0aNTKZveWtp4Wzr89Ykcx+vccV4eybLmoJ88hvX0/v/L3W7UofF7rThL98JMwfPfh73XLeBR8bGOat8SUC6OWWvnt87hWA9TCgvfxsU6US5vXBzeUPDr3MPTfvnMxe/vmocHaL+TO7ep31snrMoGR22mZ/6WA6/frd74unhpOjHni9g2OnjX3s2WRWK33U/s0TdAAAAACQkYIOAAAAADJS0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACCjau4FOuPpT+8T5vMOvzCZPde+Npw95vxPJ7Ntrn8inH354G2TWeO4YeHstbukd96sqSWcnfjz05JZ6/deCmeHzJ8T5pHaS8uS2fBr0tl/5+ns6JPPDmfHHL04zENnbhKEj5Q/LnSjlgWD4x84uGf2gL6g0pJ+z1x+zB7h7KY3pN8H6itXlt4plyVnTg3zG07/apDG9x5Az9n06lnJ7Dtnbx3OnrhxfN+88IyByWz8cfFe0NuM+8LMZFbrwT3+VtNmm4X5M0e1J7PxzfF78U9XbpHMRn03fd3orFx/lxsyT9ABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJBRNfcCnXH5Ry8rPTuoEufvPPE/k9lWp78Szh4//DdlVvq/WpLJxJ+dHk6O/8xdyazW3l56o1xGXzYzzBvl//mLoni2M8OQxdgvxa+Ja96/VTJ7/7Alpc/75GHfD/O3735sMqs/MLf0eSGy5p37hPnGZz2VzG4ff3E4e+Rd6f/TxfyV4Wx3qW6xeZg/e/R2yewXp309nN2ymr736MjS2tpk1ry6Ufq4wBvz9dlvC/PDDvl2mLd+fEEyq5dZCPg7C88cH+ZzD7komc1a2xzO/sfhBwTp4+EsvYsn6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBG1dwLdMZ/vjYhzPdteSiZjWhqCWfPHXV/mZWKoiiKd8x7dzJ7atabwtntrl2RzMY/ck8422hvjxcDNmhXPzU1mR078Zelj9vWKD0K3eZtX749zM8c+XDpY887d3g6fG3f0sftjPdOnRXm14/+XTKrF82lz3v8oreF+WNX7ZjMRv4q3hnoObWiEub11Wt6aBPYcDXt3JrMvnTkz8PZWiN9w33CjSeGs+MXzI4Xo8/wBB0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABlVcy/QGTMP2jLM933/wclsxe7rwtnqi83JrPU7z8azz7+QzLZZ83Q4Ww9TgLS1V2+eDr/Wc3tAXzd3+ndzr1BC+neus9a0hJMfnfOBZDb+owvD2ZGvz4rXAnqF7auDw3zZCfsks5E/8DqH9fGeX92WzI7cKN0RFEVR7Dn7hGQ2/pOzy65EH+MJOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJBRNfcCnVFb9nKYj7loZjrrxHnbOzEL0F02vT99Tbz0lR3D2VM2nd/V60C3+svpbw7zH528TzJ74M1XdvU6XeInr45NZkvaNglnr7w3/fcx/opaOLvdX+9PZvVwEugtrpoWX9deqa8O81EPvpbMGqU2gv7nyzcclcyOPe6icHbw74d39Tr0QZ6gAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgo2ruBQDoGrVHFySzm3cZHs7eXOzdiTPP7cQslNN0271hvu2dQ5LZXqd/Ipz94ce/ncx2GVgJZw9+aEYyW3Hb5uHs1r94Npm1P7k4nN2huCfMgQ3bp+ceHeZHb31fmA94fW0yq5XaCPqf7c6ZlcwOPye+1x5ZpGfpPzxBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADKq5l4AAKCr1VetSmZbnT8znD33/H1Kn3ej4olSWVEURXvpswL93Yh3LAjzvxRDOzhCPA9A9/MEHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADJS0AEAAABARgo6AAAAAMhIQQcAAAAAGSnoAAAAACAjBR0AAAAAZKSgAwAAAICMFHQAAAAAkJGCDgAAAAAyUtABAAAAQEYKOgAAAADISEEHAAAAABkp6AAAAAAgIwUdAAAAAGSkoAMAAACAjBR0AAAAAJCRgg4AAAAAMlLQAQAAAEBGCjoAAAAAyKhSFEUj9xIAAAAA0F95gg4AAAAAMlLQAQAAAEBGCjoAAAAAyEhBBwAAAAAZKegAAAAAICMFHQAAAABkpKADAAAAgIwUdAAAAACQkYIOAAAAADL6P3vDixxqK2qvAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 1600x600 with 10 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def show_some_images():\n",
        "    train_loader, test_loader = load_dataset()\n",
        "    plt.figure(figsize=(16, 6))\n",
        "    for i in range(10):\n",
        "        plt.subplot(2, 5, i + 1)\n",
        "        image, _ = train_loader.dataset.__getitem__(i)\n",
        "        plt.imshow(image.squeeze().numpy())\n",
        "        plt.axis('off');\n",
        "show_some_images()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oDjdqPgK5iqa"
      },
      "source": [
        "# Create the model classes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "mrv7Gjvr5iqb"
      },
      "outputs": [],
      "source": [
        "class FC2Layer(nn.Module): # fully connected 2-layer neural network (MLP)\n",
        "    def __init__(self, input_size, n_hidden, output_size):\n",
        "        super(FC2Layer, self).__init__()\n",
        "        self.input_size = input_size\n",
        "        self.network = nn.Sequential(\n",
        "            # layer 1\n",
        "            nn.Linear(input_size, n_hidden),\n",
        "            nn.ReLU(),\n",
        "            # layer 2\n",
        "            nn.Linear(n_hidden, n_hidden),\n",
        "            nn.ReLU(),\n",
        "            # classification head \n",
        "            nn.Linear(n_hidden, output_size),\n",
        "            nn.LogSoftmax(dim=1)\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = x.view(-1, self.input_size)\n",
        "        return self.network(x)\n",
        "    \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "omXuFFFlh_43"
      },
      "outputs": [],
      "source": [
        "# Definition of a Convolutional Neural Network\n",
        "class CNN(nn.Module):\n",
        "    def __init__(self, n_feature):\n",
        "        super(CNN, self).__init__()\n",
        "        self.n_feature = n_feature\n",
        "        self.conv1 = nn.Conv2d(in_channels=1, out_channels=n_feature, kernel_size=5)\n",
        "        self.conv2 = nn.Conv2d(n_feature, n_feature, kernel_size=5)\n",
        "        self.fc1 = nn.Linear(n_feature*4*4, 50)\n",
        "        self.fc2 = nn.Linear(50, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.conv1(x)\n",
        "        x = F.relu(x)\n",
        "        x = F.max_pool2d(x, kernel_size=2)\n",
        "        x = self.conv2(x)\n",
        "        x = F.relu(x)\n",
        "        x = F.max_pool2d(x, kernel_size=2)\n",
        "        x = x.view(-1, self.n_feature*4*4)\n",
        "        x = self.fc1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.fc2(x)\n",
        "        x = F.log_softmax(x, dim=1)\n",
        "        return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kSttIk855iqd"
      },
      "source": [
        "## Train and test a given model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "tXnrKiRD5iqd"
      },
      "outputs": [],
      "source": [
        "def train(train_loader, model):\n",
        "    optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)\n",
        "    model.train()\n",
        "    for batch_idx, (data, target) in enumerate(train_loader):\n",
        "        # send to device\n",
        "        data, target = data.to(device), target.to(device)\n",
        "\n",
        "        optimizer.zero_grad()\n",
        "        output = model(data)\n",
        "        loss = F.nll_loss(output, target)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        if batch_idx % 100 == 0:\n",
        "            print('Training: [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n",
        "                batch_idx * len(data), len(train_loader.dataset),\n",
        "                100. * batch_idx / len(train_loader), loss.item()))\n",
        "\n",
        "def test(test_loader, model):\n",
        "    model.eval()\n",
        "    test_loss = 0\n",
        "    correct = 0\n",
        "    for data, target in test_loader:\n",
        "        # send to device\n",
        "        data, target = data.to(device), target.to(device)\n",
        "\n",
        "        output = model(data)\n",
        "        test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss\n",
        "        pred = output.data.max(1, keepdim=True)[1] # get the index of the max log-probability\n",
        "        correct += pred.eq(target.data.view_as(pred)).cpu().sum().item()\n",
        "\n",
        "    test_loss /= len(test_loader.dataset)\n",
        "    accuracy = 100. * correct / len(test_loader.dataset)\n",
        "    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
        "        test_loss, correct, len(test_loader.dataset),\n",
        "        accuracy))\n",
        "    return accuracy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [],
      "source": [
        "def train_and_test(create_model_fn):\n",
        "    set_seed(2)\n",
        "    train_loader, test_loader = load_dataset()\n",
        "    model = create_model_fn().to(device)\n",
        "    print('Number of parameters: {}'.format(get_n_params(model)))\n",
        "\n",
        "    train(train_loader, model)\n",
        "    accuracy = test(test_loader, model)\n",
        "    return accuracy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-wEbndAf5iqf"
      },
      "source": [
        "# Train a small fully-connected MLP"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "jOHye6Er5iqf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Setting seed=2\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 6442\n",
            "Training: [0/60000 (0%)]\tLoss: 2.303038\n",
            "Training: [6400/60000 (11%)]\tLoss: 1.939705\n",
            "Training: [12800/60000 (21%)]\tLoss: 1.454411\n",
            "Training: [19200/60000 (32%)]\tLoss: 1.133756\n",
            "Training: [25600/60000 (43%)]\tLoss: 1.024244\n",
            "Training: [32000/60000 (53%)]\tLoss: 0.762541\n",
            "Training: [38400/60000 (64%)]\tLoss: 0.755996\n",
            "Training: [44800/60000 (75%)]\tLoss: 0.694219\n",
            "Training: [51200/60000 (85%)]\tLoss: 0.784198\n",
            "Training: [57600/60000 (96%)]\tLoss: 0.704995\n",
            "\n",
            "Test set: Average loss: 0.5952, Accuracy: 8229/10000 (82%)\n",
            "\n"
          ]
        }
      ],
      "source": [
        "n_hidden = 8 # number of hidden units\n",
        "create_model_fnn = lambda : FC2Layer(input_size, n_hidden, output_size)\n",
        "\n",
        "mlp_accuracy = train_and_test(create_model_fnn)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sbYdG-J35iqg"
      },
      "source": [
        "# Train a CNN with the same number of parameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "66QkqO5d5iqg"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Setting seed=2\n",
            "Number of parameters: 6422\n",
            "Training: [0/60000 (0%)]\tLoss: 2.306013\n",
            "Training: [6400/60000 (11%)]\tLoss: 1.523620\n",
            "Training: [12800/60000 (21%)]\tLoss: 0.501875\n",
            "Training: [19200/60000 (32%)]\tLoss: 0.251688\n",
            "Training: [25600/60000 (43%)]\tLoss: 0.308805\n",
            "Training: [32000/60000 (53%)]\tLoss: 0.237846\n",
            "Training: [38400/60000 (64%)]\tLoss: 0.178602\n",
            "Training: [44800/60000 (75%)]\tLoss: 0.151692\n",
            "Training: [51200/60000 (85%)]\tLoss: 0.294626\n",
            "Training: [57600/60000 (96%)]\tLoss: 0.119490\n",
            "\n",
            "Test set: Average loss: 0.1690, Accuracy: 9485/10000 (95%)\n",
            "\n"
          ]
        }
      ],
      "source": [
        "n_features = 6 # number of feature maps\n",
        "create_model_cnn = lambda : CNN(n_features)\n",
        "\n",
        "cnn_accuracy = train_and_test(create_model_cnn)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aOD-YiOFJSpB"
      },
      "source": [
        "## **Question: Why is the accuracy of the CNN is higher than the MLP accuracy? The two neural networks have the same number of parameters, why do we observe such a difference in the results?**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The accuracy of the CNN is higher than the accuracy of the MLP because the CNN's architecture and inductive biases are better suited for working with image data.  \n",
        "\n",
        "MLP models don't have biases for processing spatial data like images. While the MLP model learns a weight for each individual pixel, the CNN model uses weight sharing which means it uses the same weights for different pixels. This is because spatial locality is important for image data. So even though the models have the same number of parameters, the CNN model uses the parameters more efficiently.  \n",
        "\n",
        "Furthermore, the CNN model hierarchically calculates features of increasing complexity (e.g. edges, lines, crosses, swirls) from the input data and is thus able to use the higher-level features to make better predictions in a more generalizable way.  \n",
        "\n",
        "While each neuron in the CNN is only connected to small, local regions of the input data, the MLP is fully connected. The CNN's approach is better for image data, because one doesn't need to look at the whole image to recognize the hierarchical features the CNN learns.\n",
        "\n",
        "Another reason for the better performance of the CNN is that it is robust to variations often seen in images like translations, rotations, or distortions. This is achieved through the pooling layers as well as local connectivity of the neurons and weight sharing.  "
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.8.10 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.5"
    },
    "vscode": {
      "interpreter": {
        "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}