notebooks/ce5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Computational Assignment 5: Identifying Handwritten Digits\n",
    "\n",
    "---\n",
    "In this exercise, we'll explore classification of handwritten digits from the MNIST dataset.\n",
    "\n",
    "Goals are as follows:\n",
    "\n",
    "- Train a logistic regression model to classify handwritten digits\n",
    "- Visualize the filters for each digit learned by logistic regression\n",
    "- Train a simple MLP to classify handwritten digits, and understand why it is more effective than logistic regression\n",
    "- Visualize the filters learned by the MLP\n",
    "- Observe that logistic regression is *not* effective when digits are not centered (why?)\n",
    "\n",
    "We'll begin by importing required libraries:\n",
    "\n",
    "- `numpy` for efficient math operations\n",
    "- `sklearn` for defining and training our logistic regression and MLP models\n",
    "- `tensorflow` to load the MNIST dataset; next time we'll also use it to build a simple CNN\n",
    "- `matplotlib` for visualization/plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neural_network import MLPClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll load the MNIST dataset. The data are already broken down into:\n",
    "\n",
    "1. a development set, which we'll use for training\n",
    "2. a test set, which we'll use to evaluate performance\n",
    "\n",
    "Since we will not be tuning our models, we will not set aside a validation set. If we were going to create a validation set, we would divide `y_dev` into `y_train` and `y_val`, then evaluate on the test set only once, after deciding on a model. This would give us a more realistic estimate of model performance. Instead, our primary goal here is to understand what is happening with each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 60000 training images.\n",
      "There are 10000 test images.\n"
     ]
    }
   ],
   "source": [
    "# Load MNIST\n",
    "(x_dev, y_dev), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "print('There are', len(x_dev), 'training images.')\n",
    "print('There are', len(x_test), 'test images.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at 5 example images from our training set. The code block below uses the `ax.imshow` method from matplotlib, which we will also use later in the exercise to visualize learned filters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABC0AAADjCAYAAAChIe7cAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAArN0lEQVR4nO3deZxU1Z338e8PCW6IiiuuqEFn1Afb3SSOkIjLGI2oE5VxzTjio3FJXuqjMcQwGo1RcUaNGjfAhQSdIIomDuZxjaPyiESNuOEu0gFckEUjo/yeP+5t7eA51V1dp6puVX3er1e/6PrdOvecKupbXX363nvM3QUAAAAAAFA0veo9AAAAAAAAgBAmLQAAAAAAQCExaQEAAAAAAAqJSQsAAAAAAFBITFoAAAAAAIBCYtICAAAAAAAUEpMWqCozG21mt9a6LdDMyBWQHrkC0iJTQHqtmqvCT1qY2RtmttTM1l6u/rSZuZkNzG+Pz2/v0uk+XzUz73T7ITP71063zzGz181ssZnNNrPb8vrMvLbYzD4zs792un1OYIwN8QIws/5mNtnMlpjZm2b2z2W0HW9mP6vm+GrBzIaa2bJO/5+LzeyYeo+r1shVOmZ2splNN7NPzGx8mW2bIleSZGb/nL+vLDGzO82sf73HVGvkKg0zW9HMbsxfT4vM7E9m9o9ltG+aXHUws3H5a+ar9R5LLZGpdMzsVjNrN7OFZvZy5+eiG22bIlNmNsDMppjZnM6vn1ZDrtIzs0H5Y+r2mJsoV2ZmPzazt/L3l4lm1i91P4WftMi9LmlExw0z+1+SVg7c731J3frPt+wX1aMkDXP3vpJ2knS/JLn7Nu7eN6//UdLJHbfd/cLKHkpdXSVpqaT1JB0h6Roz26a+Q6qLOZ3+P/u6+031HlCdkKs05ih7fsbWeyD1kr+PXKvs/349SR9Jurqug6ofclW53pLeljRE0uqSfiLp9hb+BWN3SVvUexx1RKbS+Lmkge7eT9J3JP3MzHas85hqbZmk/5J0SL0HUgDkKq2rJD1Z70HUydHK/t+/IWkDZa+jK1N30iiTFrcoe0I6HCPp5sD9bpI02MyGdGOfO0ua6u6vSpK7/8Xdr6t4pJLyWcmTzGxW/lei881sCzN7PJ+But3M+uT3XdPM7jGz+Wb2Qf79Rp32tZmZPZLv5/+a2VWdZ/HMbDcze8zMFpjZM2Y2NDKmVZW9Sf/E3Re7+6OSpih7kVX6eC83s7fzx/aUmf3DcndZycxuyx/DDDPbrlPbDcxsUv74XzezUysdD7qNXFWYq/wx3uHud0p6L8Xj7DSGRsrVEZLudvdH3H2xsl8yDzaz1arcbxGRqwpz5e5L3H20u7/h7svc/R5lH7Ar/gWrwXIlM+ut7MPfydXuq8DIVJqfVTPd/ZOOm/lXxZNhjZQpd5/r7lerdX+57IxcJchVfv/DJS1QPkGT6PE2TK4kHSDpRnd/O/8M+AtJh5nZKik7aZRJiyck9TOzvzezFSQdJil0+M1Hki6UdEE393m0mZ1pZjvl+01pX2UfsHaT9H8kXafsg/3GkrbVF7ObvSSNk7SppE0kfSzpl53282tJ/0/SWpJGq9Mkg5ltKOl3ymZA+0s6Q9IkM1snMJ4tJX3m7i93qj0jKcWRFk9KasvH8GtJ/2lmK3XafqCk/+y0/U4z+4qZ9ZJ0dz6ODSXtKekHZrZPVx2a2Sb5m0nsq9SpL+ua2dw8yP9u2YROKyJXleeqmhopV9vk/UmS8g8sS5W977QacpU4V2a2nrLX0swePr7OGilXkvRDSY+4+7M9eKzNgkwlypSZXW1mH0l6UVK7pN9X9jAlNV6mkCFXCXJl2WkQ50k6PcHj66yRcmX5V+fbK0oaVM4D7kqjTFpIX8wI7qXszfadyP2ulbSJdXH+q7vfKukUSftIeljSPDM7O91w9Qt3X+juMyU9J+k+d3/N3T+UdK+k7fNxvOfuk9z9I3dfpOxNYYiUvXiUzVqe6+5LOx0d0eFISb9399/nf436g6TpkvYLjKevpA+Xq30oqeK/hLr7rfnj+NTdxyh7oW7V6S5Puftv3f1/JF0maSVlbzg7S1rH3c/LH99rkq6XdHg3+nzL3dco8fXrSNMXlb0JDJD0LWVvfpf18KE3A3JVWa6qpsFyVbX3lwZFrhLlysy+ImmCpJvc/cVKH2gj5crMNpZ0gqRzK3vUTYFMJciUu5+k7H35HyTdIemT2H27q5EyhS8hV5Xn6nzlRxkkfJyNlqt7Jf2rmQ00s9UlnZXXkx5p0TvlzqrsFkmPSNpM4cOXJEnu/omZna/sRTQidr/8vhMkTcg/FA3Pv/+Tu09NMN65nb7/OHB7fUmy7NCZf1c2e7hmvn21fHZyA0nvu/tHndq+rWxGUcpmEL9rZgd02v4VSQ8GxrNY0vIXReknaVF3H1CMmZ0u6V/z8Xq+384X9/k8yO6+zMxmd7rvBma2oNN9V1B2rltVuPtfJP0lv/m6mf0fZTOqJ1Srz4IjV5me5qpqGilXquL7S4MiV5mKcpX/xegWZUftJDk9osFy9R+Szss/kLc6MpWp+GeVu38m6VEzO1LSiZKuKONxfUmDZQp/i1xlepQrM2uTNEz5ZElKDZarscqev4eUzS2MUXbKyOyUnTTMkRbu/qayc1r3UzY7XMo4ZRfvOqib+/4fd/9PSc8qO7yolk5XNnO2q2cXR9ojr5uyQ/f629+eE7Rxp+/flnTLcrNgq7r7RYF+XpbU28w6H6qznSo83Nayc6zOknSopDXdfQ1lf2HtfJjQxp3u30vSRsouXvi2pNeXG/9q7t7lX7TzQ5gWl/g6opsPwZcba0shV5/raa6qogFzNVPZ+0nHfjZX9leBlyP3b2rk6nM9zpWZmaQblV3Y9ZD8r0kVacBc7SnpEjP7i5l1TLY/XuIQ3aZFpj6X8mdVb1V4TYsGzBQ6IVef62muhkoaKOmt/D36DEmHmNmMSgbfaLny7IiUn7r7QHffSNlnwncUP3KnRxpm0iJ3nKRvufuSUndy90+VnaN0Vuw+ZnasmX3bzFYzs175IU/bSJqWcsDdsJqy2cEFli0R+NOODfmbyXRJo82sj5l9TdnMVYdbJR1gZvuY2QpmtpJlS3pupOXkz9kdks4zs1XN7BvKzoe6peM+ll3kZmiJsXb00fHVJx//p5LmK5sUOVdf/ovrjmZ2sGUXFPuBssMRn1B2PtlCMzvLzFbOH8O2ZrZzV0+aZ4cw9S3xNSHULn9+NrHMxpIuknRXV/01OXLVw1xJ2YXyLDvPcAV9kZHenbY3fa6UHb5/gJn9g2XXiDlP0h2eHZbZqshVBbmSdI2kv5d0gLt/vPzGFsnVlsomA9vyLyl7Tid31WeTIlM9zJSZrWtmh5tZ3/y++yj7i/kDne7TCplS/vN6xfzmiva31wloReSq5z+rrlM28deWf/1K2dHbn18/ohVyZWb9LbsoqpnZ1spOVznP3Zd11Wc5GmrSwt1fdffp3bz7b5TNpsUslHSOpLeUXfH1YkknenZuUy39h7KlYd5V9mL7r+W2HyHpa8pWJviZpNuUn4Po2flTByp7HPOVza6dqfj/60l5X/OUPT8nenZemPIwLpb05xJjPVvZm0DH1wOSpio7l+llSW9K+qs6HbKUu0vZBX4+UHaxm4PzGdjPlL1RtCmb6X1X0g3KZnKrZQdJj0taIukxZefEtfSKJeSq4lyNUpaHs5WdC/lxXmuZXOXvI/9b2eTFPGU/cE+qVn+NgFz1PFdmtqmyU/baJP3FlvsrTwvlap5nV9//i2enNkrSu6FJnFZApir6WeXKTgWZrey1famkH7j7XVLrZCr3sbLHKmXXcWjJPHUgVz3PlWfXzOj8Hr1Y0l/dfb7UUrlaW9lFfZfk4x7riVaN6czcPfU+UUVmdpukF939p13eubz9HilpG3f/Ucr9Ao2AXAHpkSsgLTIFpEeuGgOTFgWXH87zvrLZsr0l3Snpa+7+p3qOC2hk5ApIj1wBaZEpID1y1ZgaafWQVrW+smtRrKXssL4TCRVQMXIFpEeugLTIFJAeuWpAHGkBAAAAAAAKqaEuxAkAAAAAAFpHRZMWZravmb1kZq+Y2dmpBgW0MnIFpEeugPTIFZAeuQK+rMenh5jZCsqWYtlL2flAT0oa4e7Pl2jDuShoKu5uKfdHrgByBVRDvXNFptCE3nX3dVLukFyh1cV+VlVypMUukl5x99fcfamkicrWtQXQc+QKSI9cAemRK7S6N6uwT3IFBFQyabGhpLc73Z6d1wD0HLkC0iNXQHrkCkiPXAEBlSx5Gjp040uHKJnZSEkjK+gHaCXkCkiPXAHpdZkrMgWUjVwBAZVMWsyWtHGn2xtJmrP8ndz9OknXSZx3BXQDuQLSI1dAel3mikwBZSNXQEAlp4c8KWmQmW1mZn0kHS5pSpphAS2LXAHpkSsgPXIFpEeugIAeH2nh7p+a2cmSpkpaQdJYd5+ZbGRACyJXQHrkCkiPXAHpkSsgrMdLnvaoMw5hQpNJvYRcT5ArNBtyBaRX71yRKTShp9x9p3oOgFyh2VRjyVMAAAAAAICqYdICAAAAAAAUEpMWAAAAAACgkJi0AAAAAAAAhcSkBQAAAAAAKCQmLQAAAAAAQCExaQEAAAAAAAqJSQsAAAAAAFBITFoAAAAAAIBCYtICAAAAAAAUEpMWAAAAAACgkJi0AAAAAAAAhcSkBQAAAAAAKCQmLQAAAAAAQCExaQEAAAAAAAqJSQsAAAAAAFBITFoAAAAAAIBC6l3vAQBAo9txxx2D9ZNPPjna5uijjw7Wb7755mibK6+8MlifMWNGidEBAAAAjYsjLQAAAAAAQCExaQEAAAAAAAqJSQsAAAAAAFBITFoAAAAAAIBCYtICAAAAAAAUkrl77Tozq11nTWSFFVYI1ldfffVkfZRa5WCVVVYJ1rfaaqtom+9///vB+qWXXhptM2LEiGD9r3/9a7TNRRddFKz/27/9W7RNSu5uNemoBHJVG21tbdFtDzzwQLDer1+/pGP48MMPg/W11loraT/1Rq5QBHvuuWd024QJE4L1IUOGRNu89NJLFY+pEvXOFZlqLqNGjYpui30G69Ur/rfSoUOHBusPP/xwWeOqsafcfad6DoBcodnEflZVtOSpmb0haZGkzyR9Wu/gAs2AXAHpkSsgPXIFpEeugC+raNIi9013fzfBfgB8gVwB6ZErID1yBaRHroBOuKYFAAAAAAAopEonLVzSfWb2lJmNDN3BzEaa2XQzm15hX0CrIFdAeuQKSK9krsgU0CPkClhOpaeHfMPd55jZupL+YGYvuvsjne/g7tdJuk7iYjFAN5ErID1yBaRXMldkCugRcgUsp6IjLdx9Tv7vPEmTJe2SYlBAKyNXQHrkCkiPXAHpkSvgy3p8pIWZrSqpl7svyr/fW9J5yUbWADbZZJNgvU+fPtE2X//614P13XffPdpmjTXWCNYPOeSQ+OBqYPbs2dFtV1xxRbB+0EEHRdssWrQoWH/mmWeibQq+FFbZyFX97bJL+LPBpEmTom1iyw+XWlI69npfunRptE1sadPddtst2mbGjBll99NsipyrPfbYI1gvtYzt5MmTqzUcdLLzzjtHtz355JM1HEkxFTlXSOfYY48N1s8666xom2XLlpXdT6mfl62EXAFhlZwesp6kyWbWsZ9fu/t/JRkV0LrIFZAeuQLSI1dAeuQKCOjxpIW7vyZpu4RjAVoeuQLSI1dAeuQKSI9cAWEseQoAAAAAAAqJSQsAAAAAAFBITFoAAAAAAIBCquRCnC2hra0tuu2BBx4I1mMrCTSq2FWgR40aFW2zePHiYH3ChAnRNu3t7cH6Bx98EG3z0ksvRbcBq6yySnTbDjvsEKzfeuutwfqAAQOSjKnDrFmzgvWLL7442mbixInB+n//939H28Ry+vOf/7zE6FArQ4cODdYHDRoUbcPqIWn16hX++81mm20WbbPpppsG6/nF84CmEXutr7TSSjUeCZDOrrvuGqwfeeSRwfqQIUOi+9pmm23K7v+MM84I1ufMmRNtE1tpMva5VZKmTZtW3sAKjCMtAAAAAABAITFpAQAAAAAAColJCwAAAAAAUEhMWgAAAAAAgEJi0gIAAAAAABQSkxYAAAAAAKCQWPK0C2+99VZ023vvvRes13vJ01LL2yxYsCBY/+Y3vxlts3Tp0mD9lltuKWtcQK1de+210W0jRoyo4Ui+LLbkat++faNtHn744WA9tmymJA0ePLiscaG2jj766GD98ccfr/FIWldsOePjjz8+2ia2xNyLL76YZExALQ0bNiy67ZRTTil7f7Ec7L///tE2c+fOLbsfoJTDDjssuu3yyy8P1tdee+1gvdRy1g899FCwvs4660TbXHLJJdFtMbExlOrn8MMPL7ufouJICwAAAAAAUEhMWgAAAAAAgEJi0gIAAAAAABQSkxYAAAAAAKCQmLQAAAAAAACFxOohXXj//fej284888xgvdTVkf/0pz8F61dccUV5A5P09NNPB+t77bVXtM2SJUuC9W222Sba5rTTTitrXECt7bjjjsH6t7/97WibUleCDomt3CFJd999d7B+6aWXRtvMmTMnWI+9R0jSBx98EKx/61vfirYp93Gitnr14m8H9XbDDTeU3WbWrFlVGAlQXbvvvnuwPm7cuGibnqyIF1sZ4c033yx7X4Ak9e4d/5V1p512Ctavv/76aJtVVlklWH/kkUeC9fPPPz+6r0cffTRYX3HFFaNtbr/99mB97733jraJmT59etltGhGflgAAAAAAQCExaQEAAAAAAAqJSQsAAAAAAFBITFoAAAAAAIBCYtICAAAAAAAUEpMWAAAAAACgkLpc8tTMxkraX9I8d982r/WXdJukgZLekHSou4fX4mtid955Z7D+wAMPRNssWrQoWN9uu+2ibY477rhgPbacYmxZ01JmzpwZ3TZy5Miy94fSyFX52traotv+8Ic/BOv9+vWLtnH3YP3ee+8N1keMGBHd15AhQ4L1UaNGRdvEllmcP39+tM0zzzwTrC9btizaJrbs6w477BBtM2PGjOi2IitqrgYPHhzdtt5669VwJAjpyZKOsfecZlTUXKF8xxxzTLC+wQYblL2vhx56KLrt5ptvLnt/rYZclefII4+MbuvJstWx9/DDDjssWF+4cGHZfcT2JfVsadPZs2cH6zfddFPZ+2pE3TnSYrykfZernS3pfncfJOn+/DaA7hsvcgWkNl7kCkhtvMgVkNp4kSug27qctHD3RyS9v1z5QEkd0zo3SRqedlhAcyNXQHrkCkiPXAHpkSugPF2eHhKxnru3S5K7t5vZurE7mtlISZxfAHSNXAHpkSsgvW7likwBZSFXQERPJy26zd2vk3SdJJlZ+CRyAGUhV0B65ApIi0wB6ZErtKKerh4y18wGSFL+77x0QwJaFrkC0iNXQHrkCkiPXAERPT3SYoqkYyRdlP97V7IRNYGeXGH2ww8/LLvN8ccfH6zfdttt0TalVhlA3ZErSVtuuWWwfuaZZ0bbxK7+/+6770bbtLe3B+uxqzAvXrw4uq/f/e53ZdVraeWVVw7WTz/99GibI444olrDqYe652q//faLbov9/yCtUqu0bLbZZmXv75133qlkOM2g7rlC2Nprrx3d9i//8i/BeqnPhgsWLAjWf/azn5U1LnRLy+fq/PPPD9bPOeecaJvYanBXX311tE1sdbee/A4X8+Mf/zjZviTp1FNPDdZLrTrXTLo80sLMfiPpcUlbmdlsMztOWZj2MrNZkvbKbwPoJnIFpEeugPTIFZAeuQLK0+WRFu4+IrJpz8RjAVoGuQLSI1dAeuQKSI9cAeXp6TUtAAAAAAAAqopJCwAAAAAAUEhMWgAAAAAAgEJi0gIAAAAAABRST5c8RWKjR4+Obttxxx2D9SFDhgTrw4YNi+7rvvvuK2tcQDWsuOKK0W2XXnppsF5q2chFixYF60cffXS0zfTp04P1VlmCcpNNNqn3EFrGVlttVXabmTNnVmEkrSv2viLFl0N9+eWXo21i7zlArQwcODBYnzRpUtJ+rrzyymD9wQcfTNoPWse5554b3RZb2nTp0qXRNlOnTg3WzzrrrGibjz/+OLotZKWVVopu23vvvYP1Up+zzCxYL7WU8F13tdwKuH+DIy0AAAAAAEAhMWkBAAAAAAAKiUkLAAAAAABQSExaAAAAAACAQmLSAgAAAAAAFBKrhxTEkiVLotuOP/74YH3GjBnB+vXXXx/dV+xqz7GVFCTpqquuCtbdPdoGKGX77bePbiu1SkjMgQceGKw//PDDZe8LKIInn3yy3kOou379+gXr++67b7TNkUceGazHru5eyvnnnx/dtmDBgrL3B6QUy8HgwYPL3tf9998f3Xb55ZeXvT9AktZYY41g/aSTToq2if1uEVshRJKGDx9ezrBK+upXvxqsT5gwIdomtspjKb/97W+D9YsvvrjsfbUKjrQAAAAAAACFxKQFAAAAAAAoJCYtAAAAAABAITFpAQAAAAAAColJCwAAAAAAUEisHtIAXn311WD92GOPDdbHjRsX3ddRRx1VVl2SVl111WD95ptvjrZpb2+PbgMuu+yy6DYzC9ZLrQTS6quE9OoVn39etmxZDUeCVPr371+TfrbbbrvotlgWhw0bFm2z0UYbBet9+vQJ1o844ojovmKv648//jjaZtq0acH6J598Em3Tu3f4o9BTTz0VbQPUSmxlhIsuuqjsfT366KPB+jHHHBNt8+GHH5bdDyDF3/fXXnvtsvd16qmnRretu+66wfr3vve9aJvvfOc7wfq2224brPft2ze6r9iKJ6VWWbz11luD9VKrSbY6jrQAAAAAAACFxKQFAAAAAAAoJCYtAAAAAABAITFpAQAAAAAAColJCwAAAAAAUEhMWgAAAAAAgELqcslTMxsraX9J89x927w2WtLxkubndzvH3X9frUEibPLkycH6rFmzom1iS03uueee0TYXXnhhsL7ppptG21xwwQXB+jvvvBNt00paJVf7779/sN7W1hZtE1siasqUKSmG1JRKLWsaez6ffvrpKo2mfoqaq1JLdMb+f371q19F25xzzjkVj6nD4MGDo9tiS55++umn0TYfffRRsP78888H62PHjo3ua/r06cF6qSWO586dG6zPnj072mbllVcO1l988cVom1ZS1Fw1k4EDB0a3TZo0KVk/r732WrAeyw2qpxVytXTp0mB9/vz5wbokrbPOOsH666+/Hm1TamnRcs2ZMydYX7hwYbTNgAEDgvV333032ubuu+8ub2Do1pEW4yXtG6j/u7u35V8NGyigTsaLXAGpjRe5AlIbL3IFpDZe5Aroti4nLdz9EUnv12AsQMsgV0B65ApIj1wB6ZEroDyVXNPiZDN71szGmtmasTuZ2Ugzm25m4eM8AXRGroD0yBWQXpe5IlNA2cgVENDTSYtrJG0hqU1Su6QxsTu6+3XuvpO779TDvoBWQa6A9MgVkF63ckWmgLKQKyCiR5MW7j7X3T9z92WSrpe0S9phAa2HXAHpkSsgPXIFpEeugLguVw8JMbMB7t6e3zxI0nPphoRKPfdc/L/j0EMPDdYPOOCAaJtx48YF6yeccEK0zaBBg4L1vfbaK9qm1TVjrmJX5e/Tp0+0zbx584L12267LcmYim7FFVeMbhs9enTZ+3vggQeC9R/96Edl76sRFSFXJ510UnTbm2++Gax//etfr9Zw/sZbb70V3XbnnXcG6y+88EK0zRNPPFHpkCoycuTIYD12RXopvqIC4oqQq2Zy1llnRbeVWh2qXBdddFGyfSG9ZsvVggULgvXhw4dH29xzzz3Bev/+/aNtXn311WD9rrvuirYZP358sP7+++HLjEycODG6r9jqIaXaoHzdWfL0N5KGSlrbzGZL+qmkoWbWJsklvSEp/tsrgC8hV0B65ApIj1wB6ZEroDxdTlq4+4hA+cYqjAVoGeQKSI9cAemRKyA9cgWUp5LVQwAAAAAAAKqGSQsAAAAAAFBITFoAAAAAAIBCYtICAAAAAAAUUo+WPEXjii0/dMstt0Tb3HDDDcF6797xl88ee+wRrA8dOjTa5qGHHopuQ+v45JNPgvX29vZgvVHFljYdNWpUtM2ZZ54ZrM+ePTvaZsyYMcH64sWLS4wOtfKLX/yi3kNoKnvuuWfZbSZNmlSFkQBf1tbWFqzvvffeyfootczjSy+9lKwfoKemTZsW3VZqeepaiP3+MmTIkGib2LLELKedFkdaAAAAAACAQmLSAgAAAAAAFBKTFgAAAAAAoJCYtAAAAAAAAIXEpAUAAAAAACgkVg9pQoMHD45u+6d/+qdgfeedd462KbVKSMzzzz8frD/yyCNl7wutZcqUKfUeQjKxK8VL8ZVADjvssGib2FXhDznkkLLGBeALkydPrvcQ0CLuu+++YH3NNdcse19PPPFEsH7ssceWvS8AmZVXXjlYj60QIknuHqxPnDgxyZiQ4UgLAAAAAABQSExaAAAAAACAQmLSAgAAAAAAFBKTFgAAAAAAoJCYtAAAAAAAAIXEpAUAAAAAACgkljxtAFtttVWwfvLJJwfrBx98cHRf66+/fpIxSdJnn30W3dbe3h6sl1oyCM3HzMqqS9Lw4cOD9dNOOy3FkKrihz/8YbD+k5/8JNpm9dVXD9YnTJgQbXP00UeXNzAAQGGstdZawXpPPhtdffXVwfrixYvL3heAzNSpU+s9BERwpAUAAAAAACgkJi0AAAAAAEAhMWkBAAAAAAAKiUkLAAAAAABQSExaAAAAAACAQupy9RAz21jSzZLWl7RM0nXufrmZ9Zd0m6SBkt6QdKi7f1C9oTaH2OodI0aMiLaJrRIycODAFEPq0vTp04P1Cy64INpmypQp1RpOU2iVXLl7WXUpnpErrrgi2mbs2LHB+nvvvRdts9tuuwXrRx11VLC+3XbbRfe10UYbBetvvfVWtE3sCtWxK8Kja62SK5Sv1IpFW265ZbD+xBNPVGs4DYNMlW/cuHHRbb16pftb4WOPPZZsX6gtclVc++yzT72HgIjuvHt+Kul0d/97SbtJ+r6ZbS3pbEn3u/sgSffntwF0D7kC0iNXQFpkCkiPXAFl6nLSwt3b3X1G/v0iSS9I2lDSgZJuyu92k6ThVRoj0HTIFZAeuQLSIlNAeuQKKF9Zx6mZ2UBJ20uaJmk9d2+XsvBJWjf56IAWQK6A9MgVkBaZAtIjV0D3dHlNiw5m1lfSJEk/cPeFpc4PXa7dSEkjezY8oLmRKyA9cgWkRaaA9MgV0H3dOtLCzL6iLFQT3P2OvDzXzAbk2wdImhdq6+7XuftO7r5TigEDzYJcAemRKyAtMgWkR66A8nQ5aWHZtN+Nkl5w98s6bZoi6Zj8+2Mk3ZV+eEBzIldAeuQKSItMAemRK6B83Tk95BuSjpL0ZzN7Oq+dI+kiSbeb2XGS3pL03aqMsMDWW2+9YH3rrbeOtvnlL38ZrP/d3/1dkjF1Zdq0acH6JZdcEm1z113h98xly5YlGVOLIlcRK6ywQrB+0kknRdsccsghwfrChQujbQYNGlTewEqILT334IMPRtuce+65yfrH58gVgkots5xyGcomRKYi2tragvVhw4ZF28Q+Ny1dujTa5qqrrgrW586dGx8cio5cFdTmm29e7yEgostJC3d/VFLsJKs90w4HaA3kCkiPXAFpkSkgPXIFlI8/LwAAAAAAgEJi0gIAAAAAABQSkxYAAAAAAKCQmLQAAAAAAACF1J3VQ1pC//79g/Vrr7022iZ25ehaXXk2tmLBmDFjom2mTp0arH/88cdJxgR09vjjjwfrTz75ZLTNzjvvXHY/66+/frAeW+GnlPfeey9YnzhxYrTNaaedVnY/AIrha1/7WrA+fvz42g4EDWWNNdYI1mM/j0p55513otvOOOOMsvcHoGf++Mc/BuulVpliNcXa4EgLAAAAAABQSExaAAAAAACAQmLSAgAAAAAAFBKTFgAAAAAAoJCYtAAAAAAAAIXEpAUAAAAAACikplzydNdddw3WzzzzzGibXXbZJVjfcMMNk4ypKx999FF02xVXXBGsX3jhhcH6kiVLkowJqNTs2bOD9YMPPjja5oQTTgjWR40alWRMHS6//PJg/ZprrgnWX3nllaT9A6gdM6v3EAAABffcc88F67NmzYq22XzzzYP1LbbYItpm/vz55Q0MHGkBAAAAAACKiUkLAAAAAABQSExaAAAAAACAQmLSAgAAAAAAFBKTFgAAAAAAoJCacvWQgw46qKx6Tz3//PPB+j333BNt8+mnnwbrY8aMibZZsGBBWeMCiq69vT26bfTo0WXVAaDDvffeG6x/97vfrfFI0OxefPHFYP2xxx6Lttl9992rNRwAVRRbsVGSbrjhhmD9ggsuiLY55ZRTgvXY75bgSAsAAAAAAFBQTFoAAAAAAIBCYtICAAAAAAAUEpMWAAAAAACgkJi0AAAAAAAAhdTlpIWZbWxmD5rZC2Y208xOy+ujzewdM3s6/9qv+sMFmgO5AtIjV0BaZApIj1wB5TN3L30HswGSBrj7DDNbTdJTkoZLOlTSYne/tNudmZXuDGgw7m49aUeugDhyBaTXk1yRKaCkp9x9p3IbkavG069fv+i222+/PVgfNmxYtM0dd9wRrH/ve9+LtlmyZEl0WzOJ/azq3Y2G7ZLa8+8XmdkLkjZMOzygtZArID1yBaRFpoD0yBVQvrKuaWFmAyVtL2laXjrZzJ41s7FmtmbqwQGtgFwB6ZErIC0yBaRHroDu6fakhZn1lTRJ0g/cfaGkayRtIalN2WzhmEi7kWY23cymVz5coLmQKyA9cgWkRaaA9MgV0H1dXtNCkszsK5LukTTV3S8LbB8o6R5337aL/XDeFZpKT8+9l8gVEEOugPQquFYMmQLCenRNC4lcNRquaVE7sZ9V3Vk9xCTdKOmFzqHKLyLT4SBJz1U6SKBVkCsgPXIFpEWmgPTIFVC+7qwesrukP0r6s6RlefkcSSOUHb7kkt6QdEJ+YZlS+2I2EE2lgr9ckSsgglwB6fVw9RAyBcT1dPUQctVEYkdhXHDBBdE2J554YrA+ePDgaJvnn3++vIE1qEpWD3lUUqjx7ysdFNCqyBWQHrkC0iJTQHrkCihfWauHAAAAAAAA1AqTFgAAAAAAoJCYtAAAAAAAAIXEpAUAAAAAACgkJi0AAAAAAEAhdbnkadLOWJYHTaanSzOmRK7QbMgVkF69c0Wm0IR6tORpSuQKzSb2s4ojLQAAAAAAQCExaQEAAAAAAAqJSQsAAAAAAFBITFoAAAAAAIBCYtICAAAAAAAUUu8a9/eupDfz79fOb9dLvfsvwhjov7L+N001kAqRq+L0X4QxNHr/5OrL6J9cN0OuOmdKavznlP4bu/8UYyBXX0b/9F+VTNV0ydO/6dhsej2XCap3/0UYA/3X/zWQWr0fU6v3X4QxtHr/1VDvx0T/9X9N1XsM9e6/Gur9mOi/tfsvyhhSq/djon/6r1b/nB4CAAAAAAAKiUkLAAAAAABQSPWctLiujn0XoX+p/mOg/+ZT78fU6v1L9R9Dq/dfDfV+TPRff/UeQ737r4Z6Pyb6b+3+pWKMIbV6Pyb6p/+qqNs1LQAAAAAAAErh9BAAAAAAAFBIdZm0MLN9zewlM3vFzM6uQ/9vmNmfzexpM5teg/7Gmtk8M3uuU62/mf3BzGbl/65Z4/5Hm9k7+XPwtJntV8X+NzazB83sBTObaWan5fWaPAcl+q/Zc1Bt9c5UPgZyRa7IVfoxkCtyRa7S9l/TTOV9kityVVXk6vMauWrWXLl7Tb8krSDpVUmbS+oj6RlJW9d4DG9IWruG/e0haQdJz3WqXSzp7Pz7syX9osb9j5Z0Ro0e/wBJO+TfrybpZUlb1+o5KNF/zZ6DKj+/dc9UPg5yRa7IVfpxkCtyRa7SjqGmmcr7JFfkqprPL7n6okaumjRX9TjSYhdJr7j7a+6+VNJESQfWYRw14+6PSHp/ufKBkm7Kv79J0vAa918z7t7u7jPy7xdJekHShqrRc1Ci/2bRcpmSyBW5qjpy9QVyRa5SIVdfIFfkKhVy9QVy1aS5qsekxYaS3u50e7Zq/8bhku4zs6fMbGSN++6wnru3S9l/uqR16zCGk83s2fzwpqodPtWZmQ2UtL2kaarDc7Bc/1IdnoMqKEKmJHLVgVyRq5TIVYZckatUipApiVyRq3TI1RfIVZPmqh6TFhao1XoJk2+4+w6S/lHS981sjxr3XwTXSNpCUpukdkljqt2hmfWVNEnSD9x9YbX760b/NX8OqqQImZLIlUSuyFV65Ipckau0yFSGXJGrlMhVhlxV6Tmox6TFbEkbd7q9kaQ5tRyAu8/J/50nabKyw6pqba6ZDZCk/N95tezc3ee6+2fuvkzS9aryc2BmX1H2gp7g7nfk5Zo9B6H+a/0cVFHdMyWRK4lcSeQqNXJFriRylVJBMiWRK4lcpUKuvkCumjRX9Zi0eFLSIDPbzMz6SDpc0pRadW5mq5rZah3fS9pb0nOlW1XFFEnH5N8fI+muWnbe8WLOHaQqPgdmZpJulPSCu1/WaVNNnoNY/7V8DqqsrpmSyFUHckWuUiJXGXJFrlIpUKYkciWRq1TI1RfIVbPmymt4ldeOL0n7KbvC6KuSflzjvjdXdlXdZyTNrEX/kn6j7PCY/1E2G3qcpLUk3S9pVv5v/xr3f4ukP0t6VtmLe0AV+99d2WFqz0p6Ov/ar1bPQYn+a/Yc1OA1VrdM5f2TK3JFrtL3T67IFblK23fNM5X3S67IVbVfY+SKXDV1rizvFAAAAAAAoFDqcXoIAAAAAABAl5i0AAAAAAAAhcSkBQAAAAAAKCQmLQAAAAAAQCExaQEAAAAAAAqJSQsAAAAAAFBITFoAAAAAAIBCYtICAAAAAAAU0v8Hqh2AOw5Q7pwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x216 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(nrows=1, ncols=5, figsize=(15, 3))\n",
    "for i in range(5):\n",
    "    ax[i].imshow(x_dev[i, :, :], cmap='gray')\n",
    "    ax[i].set_title('MNIST Image %i, Label = %i' % (i, y_dev[i]), fontsize=12)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plots above, we can see that there are 5 distinct digits in the first 5 training images. Let's look at the distribution of the labels across the whole training set to make sure that all 10 digits are equally represented, more or less. In this block, we'll use `np.unique()` to count the number of times each digit occurs within `y_dev`, the labels for our training data.\n",
    "\n",
    "In generating our plot, the important command is `plt.bar(y_dev_values, y_dev_counts)`, which plots a bar graph of the counts for each value of the labels. The remaining commands make the plot a little more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm8AAAFHCAYAAAD3IatOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnLUlEQVR4nO3de7RdZXnv8e9PkIsXBBQQExA4Rirao0KgYFtQqQUPngOnFouOU0JLpWLw3laoIhfFYrFasIUejhVCW4toVfACiqhVkYsBUe6DYLhEIkQERBSQ8Jw/5kxZ7OzLTPZae2Vtvp8x1lhzvfOdcz5zQODJe01VIUmSpNHwpGEHIEmSpO5M3iRJkkaIyZskSdIIMXmTJEkaISZvkiRJI8TkTZIkaYTMaPKWZMckV/V8fp7k7Uk2T3Jhkpva7816rjkqyZIkNybZp6d8lyRXt+dOSZKZfBdJkqRhyLDWeUuyHvBj4LeAhcDPqurEJEcCm1XVu5PsBPw7sBvwHOBrwPOramWSy4G3AZcCXwZOqarzJ3vms571rNpuu+0G9k6SJEn9csUVV/y0qrYYW77+MIJp7Q3cXFW3JtkfeHlbvgj4JvBuYH/g7Kp6CFiaZAmwW5JbgE2q6hKAJGcBBwCTJm/bbbcdixcv7v+bSJIk9VmSW8crH+aYt4NoWtUAtqqq5QDt95Zt+Rzg9p5rlrVlc9rjseWSJEmz2lCStyQbAP8L+PRUVccpq0nKx3vWYUkWJ1m8YsWKNQtUkiRpHTOslrdXA1dW1Z3t7zuTbA3Qft/Vli8Dtum5bi5wR1s+d5zy1VTV6VU1v6rmb7HFat3GkiRJI2VYydvreazLFOA8YEF7vAA4t6f8oCQbJtkemAdc3nat3p9k93aW6cE910iSJM1aMz5hIclTgFcBf95TfCJwTpJDgduAAwGq6tok5wDXAY8AC6tqZXvN4cCZwMY0ExUmnawgSZI0GwxtqZBhmD9/fjnbVJIkjYIkV1TV/LHl7rAgSZI0QkzeJEmSRojJmyRJ0ggxeZMkSRohJm+SJEkjZJh7m2oEbHfkl4YdwpRuOXG/YYcgSdKMseVNkiRphJi8SZIkjRCTN0mSpBFi8iZJkjRCTN4kSZJGiMmbJEnSCDF5kyRJGiEmb5IkSSPE5E2SJGmEmLxJkiSNEJM3SZKkEWLyJkmSNEJM3iRJkkaIyZskSdIIWevkLclOSV6b5Dn9DEiSJEkT65S8JfmHJP/U8/sPgB8AnwauS7LrgOKTJElSj64tb68Gvtvz+zjgi8CLgcuBY/oclyRJksaxfsd6zwZuAUgyF3ghcGhVXZ3kFOCfBxOeJEn9td2RXxp2CJ3ccuJ+ww5B66iuLW+/Ap7WHu8F/BxY3P7+BfD0PsclSZKkcXRtebsSWJjkNmAhcGFVPdqe2x5YPojgJEmS9Hhdk7f3ABfQTFK4F3hTz7kDaMa9SZIkacA6JW9V9b0k2wK/AdxUVT/vOX06cNMggpMkSdLjdW15o6oeAK4Yp3w0Rn5Ks4yDriXpianzIr1JXprks0l+muSRJDu35R9Msu/gQpQkSdIqXRfp/R3gEppu00+Oue5RHj8Gbqp7bZrkM0luSHJ9kj2SbJ7kwiQ3td+b9dQ/KsmSJDcm2aenfJckV7fnTkmSrjFIkiSNqq4tbycCX6FZ3+2dY85dCey8Bs88Gbigqn6DZpHf64EjgYuqah5wUfubJDsBB7XP3Rc4Ncl67X1OAw4D5rUfW/8kSdKs1zV52xk4raoKqDHnfgps0eUmSTYB9qRd1LeqHq6qe4H9gUVttUU0M1hpy8+uqoeqaimwBNgtydbAJlV1SRvTWT3XSJIkzVpdk7cHgadMcG5r4L6O99kBWAGckeT7ST6e5KnAVlW1HKD93rKtPwe4vef6ZW3ZnPZ4bPlqkhyWZHGSxStWrOgYpiRJ0rqpa/L2HeDtPV2W8FgL3KHA1zveZ30ea8V7KfAAbRfpBMYbx1aTlK9eWHV6Vc2vqvlbbNGpgVCSJGmd1XWpkKOBi2kW6f0MTaK0IMlHgF2AXTveZxmwrKoua39/hiZ5uzPJ1lW1vO0Svaun/jY9188F7mjL545TLk1qFJbXcGkNSU9k/nd6al0X6f1Bkj2Bk2h2WwhwBPBtYK+qurHjfX6S5PYkO7bX7A1c134W0EyMWACc215yHvDJNkl8Ds3EhMuramWS+5PsDlwGHAx8rNMbD5j/0kmSpEFak0V6rwT2TrIRsDlwb1X9ci2e+Rbg35JsAPwI+BOa7ttzkhwK3AYc2D7z2iTn0CR3jwALq2ple5/DgTOBjYHz248kaQD8i6m07uicvK1SVQ8yjS7KqroKmD/Oqb0nqH8CcMI45YuBF61tHJIkSaOoU/KW5H2TnH6UZrbplVV1cV+ikvSEMwotO2DrjtY9/tl54una8nYsk8/yDFBJLgH2q6quS4dIkiRpDXRdKuQFNAvkvgt4LrBR+/2XbfnLaHZCeAHwwf6HKUmSJOje8vaPwMer6qM9ZbcDf9eu/XZCVe2dZHuaCQkL+xynJEmS6N7ytgfNHqbjuRLYvT1ezGO7I0iSJKnPuiZv9zHBbFDg93hse6yNgJ9PNyhJkiSNr2u36SeAo5I8nWZXhLtoWtgOBN4E/E1b77eAa/odpCRJkhpdk7dVS4W8DXhzexyavUn/puf8l4BP9S06SZIkPU7X7bEeBd6b5MPAbwJbA8uBq6vq3p56lw8iSEmSJDXWaIeFNlH79mBCkSRJ0lTWKHlLshnN5vAbjT1XVd/qV1CSJEkaX9ftsTaimbTwOsbfZQFgvX4FJUmSpPF1XSrkaODlwAKa5O0I4M+A7wA3A68ZRHCSJEl6vK7J22uB44Gz29+XVdUZVbUX8ANg30EEJ0mSpMfrmrxtC1xbVSuBXwNP7Tn3CeCP+h2YJEmSVtc1ebsbeFp7fDvw4p5zzwI27mdQkiRJGl/X2aaXAi8Fzgf+A3h/u9vCI8C7aMa+SZIkacC6Jm8fouk6BfgA8DyaMXDr0SR2h/c/NEkaXdsd+aVhhzClW07cb9ghSFoLXXdYWAwsbo/vB16bZENgw6pyI3pJkqQZskaL9PaqqoeAh/oYiyRJkqbQOXlLsjmwH7ANq++wUFV1TD8DkyRJ0uq67rDw+zQTFZ46QZUCTN4kSZIGrOtSIR8Bvk+zRMiGVfWkMR+3xpIkSZoBXbtNtwPeUVVXDzAWSZIkTaFry9v3gecMMhBJkiRNrWvy9k7gr5LsMchgJEmSNLmu3aZXABcB30nyAHDvmPNVVc/tZ2CSJElaXdfk7cPAETTdpzcADw8sIkmSJE2oa/J2CPB+13KTJEkarq5j3gr4Vj8emOSWJFcnuSrJ4rZs8yQXJrmp/d6sp/5RSZYkuTHJPj3lu7T3WZLklCTpR3ySJEnrsq7J26eBV/fxua+oqpdU1fz295HARVU1j2Zs3ZEASXYCDgJeCOwLnJpk1ZpypwGHAfPaz759jE+SJGmd1LXb9Hzgo0meAVwA3DO2QlV9fRpx7A+8vD1eBHwTeHdbfna7j+rSJEuA3ZLcAmxSVZcAJDkLOKCNU5Ikadbqmrx9rv0+tP2sUkDa7667LBTw1SQF/N+qOh3YqqqWA1TV8iRbtnXnAJf2XLusLft1ezy2XJIkaVbrmry9oo/P/O2quqNN0C5McsMkdccbx1aTlK9+g+Qwmu5Vtt122zWNVZIkaZ3SKXmrqv/s1wOr6o72+64knwN2A+5MsnXb6rY1cFdbfRmwTc/lc4E72vK545SP97zTgdMB5s+fP26CJ0mSNCq6TljoiyRPTfL0VcfA7wPXAOcBC9pqC4Bz2+PzgIOSbJhke5qJCZe3Xaz3J9m9nWV6cM81kiRJs9aELW/tJICuqqoWTF2NrYDPtat6rA98sqouSPI94JwkhwK3AQe2N702yTnAdcAjwMKqWtne63DgTGBjmokKTlaQJEmz3mTdpnsywTiycXSqV1U/Al48TvndwN4TXHMCcMI45YuBF3WMT5IkaVaYMHmrqu1mMA5JkiR1MKNj3iRJkjQ9Jm+SJEkjxORNkiRphJi8SZIkjRCTN0mSpBFi8iZJkjRCTN4kSZJGSKe9TZMsZeKFeB8F7gOuAE6pqmv6FJskSZLG6Nry9p/AesDWwFLg0vb7OTQJ4K3A/wS+l+RlA4hTkiRJdE/evk3TurZ9Ve1dVW+oqr2B7YGf0+wr+jzgB8BxA4lUkiRJnZO3dwPHV9VPegurajnwAeDdVfUAcDKwW39DlCRJ0ipdk7dtgIcmOPcgMKc9/jGwwXSDkiRJ0vi6Jm/XA+9KsmFvYZKNgL9oz0MzBu7O/oUnSZKkXp1mmwJ/BXwRuC3Jl4G7gC2B/wFs2n4DvAz4ap9jlCRJUqtT8lZVX0uyM/BeYE+aWafLga8BH6iq69t6bx1UoJIkSere8kZVXQe8YYCxSJIkaQrusCBJkjRCOre8JdkLeD2wLbDRmNPVrvsmSZKkAeq6PdafA6cBdwM3sfqyIelzXJIkSRpH15a3dwGfBP60qh4eYDySJEmaRNcxb3OAM0zcJEmShqtr8nYFsMMgA5EkSdLUuiZvbwXenmTPQQYjSZKkyXUd8/YFYBPgG0l+Cdwz5nxV1XP7GpkkSZJW0zV5uwioQQYiSZKkqXXdHuuQAcchSZKkDtxhQZIkaYRM2PKW5GDgS1V1d3s8qao6q6+RSZIkaTWTdZueCexOs6vCmVPcp4DOyVuS9YDFwI+r6jVJNgc+BWwH3AK8rqruaeseBRwKrATeWlVfact3aePaGPgy8LaqclyeJEma1SbrNt0euKrneLLPmq4B9zbg+p7fRwIXVdU8mskRRwIk2Qk4CHghsC9wapv4QbNd12HAvPaz7xrGIEmSNHImTN6q6tZVOyq0x5N+uj4wyVxgP+DjPcX7A4va40XAAT3lZ1fVQ1W1FFgC7JZka2CTqrqkbW07q+caSZKkWWsYExb+Hvgr4NGesq2qajlA+71lWz4HuL2n3rK2bE57PLZckiRpVuuUvCXZIMkxSW5I8sskK8d8Hul4n9cAd1XVFR3jyzhlNUn5eM88LMniJItXrFjR8bGSJEnrpq6L9J4ELATOBz4LPLSWz/tt4H8l+R/ARsAmSf4VuDPJ1lW1vO0SvautvwzYpuf6ucAdbfncccpXU1WnA6cDzJ8/3wkNkiRppHVN3v4QOKaqTpjOw6rqKOAogCQvB/6iqv5PkpOABcCJ7fe57SXnAZ9M8hHgOTQTEy6vqpVJ7k+yO3AZcDDwsenEJkmSNAq6Jm9PAy4ZYBwnAuckORS4DTgQoKquTXIOcB3wCLCwqla21xzOY0uFnN9+JEmSZrU12Zh+T+Dr/XpwVX0T+GZ7fDew9wT1TgBWa/GrqsXAi/oVjyRJ0ijomrx9DDgryaM0C+L+bGyFqvpRPwOTJEnS6romb6u6TI8FjpmgznoTlEuSJKlPuiZvf8oES3FIkiRp5nRK3qrqzAHHIUmSpA6GscOCJEmS1tKELW9JPgG8v6qWtseTqao6tL+hSZIkaazJuk1fAZzcHr+Syce8OR5OkiRpBkyYvFXV9j3H281INJIkSZqUY94kSZJGSNelQv5Lki1pNpV/nKq6rS8RSZIkaUKdkrckTwI+APw5sOkE1VykV5IkacC6dpu+HVgI/B0Q4IM0ydxS4GbgjYMITpIkSY/XNXn7E+B44EPt789V1THAC4AfA9sOIDZJkiSN0TV52wFYXFUrgUeAjQGq6tfA39NsnyVJkqQB65q83cdjkxTuAHbsObc+sHk/g5IkSdL4us42/T6wE/CV9nNckl/RtMKdAFw5mPAkSZLUq2vy9vc0XacAxwA7A//W/r4VOKK/YUmSJGk8nZK3qrqw5/gnSXYD/hvwFOD6duybJEmSBmzKMW9JNkjyuSR7riqrxpKq+qGJmyRJ0syZMnmrqoeB3+tSV5IkSYPVNSG7GNh9kIFIkiRpal0nLLwL+HySXwCfB5YD1Vuhqh7tb2iSJEkaa8KWtyQHJ3lm+/NqmgkKJ9PMLn0Y+HXP5+EBxylJkiQmb3k7A9gDuJtma6yapK4kSZJmwGTJW1YdVNWxgw9FkiRJU3EGqSRJ0giZasLCnCQ7TFEHgKr6UR/ikSRJ0iSmSt4+swb3Wm86gUiSJGlqUyVvJwA3z0QgkiRJmtpUydsXq+ryGYlEkiRJU5rRCQtJNkpyeZIfJLk2yXFt+eZJLkxyU/u9Wc81RyVZkuTGJPv0lO+S5Or23ClJMt4zJUmSZpOZnm36EPDKqnox8BJg3yS7A0cCF1XVPOCi9jdJdgIOAl4I7AucmmTV2LrTgMOAee1n3xl8D0mSpKGYLHk7DljWz4dV4xftzye3nwL2Bxa15YuAA9rj/YGzq+qhqloKLAF2S7I1sElVXVJVBZzVc40kSdKsNWHyVlXHVdUd/X5gkvWSXAXcBVxYVZcBW1XV8va5y4Et2+pzgNt7Ll/Wls3h8YnlqnJJkqRZbcYX6a2qlVX1EmAuTSvaiyapPt44tpqkfPUbJIclWZxk8YoVK9Y4XkmSpHXJ0HZYqKp7gW/SjFW7s+0Kpf2+q622DNim57K5wB1t+dxxysd7zulVNb+q5m+xxRb9fAVJkqQZN9OzTbdIsml7vDHwe8ANwHnAgrbaAuDc9vg84KAkGybZnmZiwuVt1+r9SXZvZ5ke3HONJEnSrDVh8pbks0me1x4fnOSZfXje1sA3kvwQ+B7NmLcvAicCr0pyE/Cq9jdVdS1wDnAdcAGwsKpWtvc6HPg4zSSGm4Hz+xCfJEnSOm2yRXr3p02igDOAPYC7p/Owqvoh8NJxyu8G9p7gmhNodnoYW74YmGy8nCRJ0qwzWbfpnTQJGzQTBMadECBJkqSZM1nydg7w0SQraRK3S5OsnODzyMyEK0mS9MQ2WbfpO4CLgZ2AY4AzgR/PQEySJEmawITJW7tzwacBkhwCnFxVP5ihuCRJkjSOyVre/ktVbT/oQCRJkjS1zuu8Jdk6yYeTfC/JzUkuT/K3SZ49yAAlSZL0mE7JW5LnAz8A3gr8ArgceAB4G3BVknkDi1CSJEn/pVO3KfAh4D5gt6q6ZVVhkucCX23P/0Hfo5MkSdLjdO02fQVwdG/iBlBVtwLHtuclSZI0YF2Ttw2A+yc4d397XpIkSQPWNXm7CnhLksfVbzeFf3N7XpIkSQPWdczb8cAXgeuTfApYDjwbOBCYB+w3mPAkSZLUq+s6bxckeQ3wAeA9PLbX6RXAa6rqq4MLUZIkSat0bXmjqi4ALkjyFGAz4J6q+uXAIpMkSdJqOidvq7QJm0mbJEnSEHTeYUGSJEnDZ/ImSZI0QkzeJEmSRojJmyRJ0giZMnlLskGSK5P8/kwEJEmSpIlNmbxV1cPA9sAjgw9HkiRJk+nabXohYMubJEnSkHVd5+1jwL8mWR/4PM32WNVboap+1N/QJEmSNFbX5O0/2+93Au+YoM560w9HkiRJk+mavP3JQKOQJElSJ103pl806EAkSZI0tTVa5y3Jk5K8KMleSZ46qKAkSZI0vs7JW5KFwE+AHwJfB3Zsyz+f5K2DCU+SJEm9OiVvSd4InEwz0/R1QHpOfxt4bd8jkyRJ0mq6try9E/i7qjoM+NyYczfQtsJNJck2Sb6R5Pok1yZ5W1u+eZILk9zUfm/Wc81RSZYkuTHJPj3luyS5uj13SpKM90xJkqTZpGvytj3wlQnOPQBs2vE+jwDvqqoXALsDC5PsBBwJXFRV84CL2t+05w4CXgjsC5yaZNWSJKcBhwHz2s++HWOQJEkaWV2Tt58C201wbkfgx11uUlXLq+rK9vh+4HpgDrA/sGpG6yLggPZ4f+DsqnqoqpYCS4DdkmwNbFJVl1RVAWf1XCNJkjRrdU3evgC8L8kOPWWV5Fk0i/Z+fk0fnGQ74KXAZcBWVbUcmgQP2LKtNge4veeyZW3ZnPZ4bLkkSdKs1jV5ey/wEHAN8DWarbFOoWk5WwkcvyYPTfI04D+At1fVzyerOk5ZTVI+3rMOS7I4yeIVK1asSZiSJEnrnE7JW1XdDcwH/gZ4MnAzzQK//wDsUVX3dX1gkifTJG7/VlWfbYvvbLtCab/vasuXAdv0XD4XuKMtnztO+Xixn15V86tq/hZbbNE1TEmSpHVS53Xequr+qnp/Vf1OVT2/qvaoquOmaDl7nHZG6D8D11fVR3pOnQcsaI8XAOf2lB+UZMMk29NMTLi87Vq9P8nu7T0P7rlGkiRp1uq6tykASTYBXsRjY86uXZPkDfht4I+Bq5Nc1Zb9NXAicE6SQ4HbgAMBquraJOcA19HMVF1YVSvb6w4HzgQ2Bs5vP5IkSbNa5+QtyfuAdwFP47ExZ/cnOamqPtDlHlX1HcYfrwaw9wTXnACcME75YppEUpIk6QmjU/KW5DjgaODjwNnAncBWwOuB45KsX1XHDipISZIkNbq2vL2RZoeFv+wpuxb4epL7aBbLPbbPsUmSJGmMrhMWnsHEOyxc0J6XJEnSgHVN3i4Ddp3g3K7teUmSJA3YhN2mSXoTu7cCn0vyCPBpHhvz9jrgT2m2sZIkSdKATTbm7REev2tBaJb0OHFMvQA/nOJekiRJ6oPJEq7jmWDLKUmSJA3HhMmbS39IkiStezpvjyVJkqThW5MdFl4A/CHNRvEbjTldVbVg9askSZLUT113WDgY+ATNGLi7gIfHVHFsnCRJ0gzo2vJ2NHAucGhV3Tu4cCRJkjSZrsnbs4E3mbhJkiQNV9cJCxcDLxhkIJIkSZpa15a3I4DPJrkb+Cpwz9gKVfVoPwOTJEnS6romb8uA7wP/OsH5WoN7SZIkaS11Tbj+H/BHwOeBG1h9tqkkSZJmQNfkbX/gL6vq5EEGI0mSpMl1nbDwAHDdIAORJEnS1Lomb2cAbxhkIJIkSZpa127TW4HXJ7kQuIDxZ5t+op+BSZIkaXVdk7fT2u/nAnuPc75ots+SJEnSAHVN3rYfaBSSJEnqpFPyVlW3DjoQSZIkTa3rhAVJkiStAzq1vCVZSjOubUJVtUNfIpIkSdKEuo55+09WT96eCbwM+AXw9X4GJUmSpPF1HfN2yHjlSTalWTrka/0LSZIkSROZ1pi3qroXOAl4X1+ikSRJ0qT6MWHhQWBuH+4jSZKkKax18pZk/SQvAY4Fru14zSeS3JXkmp6yzZNcmOSm9nuznnNHJVmS5MYk+/SU75Lk6vbcKUmytu8hSZI0Sjolb0keTbKy9wM8BFwBPA94R8fnnQnsO6bsSOCiqpoHXNT+JslOwEHAC9trTk2yXnvNacBhwLz2M/aekiRJs1LX2abHs/ps0wdp9jw9v6ru63KTqvpWku3GFO8PvLw9XgR8E3h3W352VT0ELE2yBNgtyS3AJlV1CUCSs4ADgPM7voskSdLI6jrb9NgBxrBVVS1vn7M8yZZt+Rzg0p56y9qyX7fHY8vHleQwmlY6tt122z6GLUmSNPPW5R0WxhvHVpOUj6uqTq+q+VU1f4sttuhbcJIkScMwYctbkjVa/qOqjl/LGO5MsnXb6rY1cFdbvgzYpqfeXOCOtnzuOOWSJEmz3mTdpsd2uL63xWttk7fzgAXAie33uT3ln0zyEeA5NBMTLq+qlUnuT7I7cBlwMPCxtXy2JEnSSJms2/TJU3x2Bb5K0425pMvDkvw7cAmwY5JlSQ6lSdpeleQm4FXtb6rqWuAc4DqaXRwWVtXK9laHAx9vn3szTlaQJElPEBO2vPUkSo+TZB5NK9uBwI9pJgOc0eVhVfX6CU7tPUH9E4ATxilfDLyoyzMlSZJmk65LhZBkG+AYmm7Ke4C/AE6tqocHFJskSZLGmDJ5a5fueA9NC9uDNK1uH62qBwYcmyRJksaYbLbpM2gWy30Lzbi2k4EPVdU9MxSbJEmSxpis5W0p8AyaSQkfAJYDm/XuPdqrqn7U//AkSZLUa7LkbdP2ex/g9zvca72pq0iSJGk6Jkve/mTGopAkSVInky0VsmgmA5EkSdLU1uW9TSVJkjSGyZskSdIIMXmTJEkaISZvkiRJI8TkTZIkaYSYvEmSJI0QkzdJkqQRYvImSZI0QkzeJEmSRojJmyRJ0ggxeZMkSRohJm+SJEkjxORNkiRphJi8SZIkjRCTN0mSpBFi8iZJkjRCTN4kSZJGiMmbJEnSCDF5kyRJGiEmb5IkSSPE5E2SJGmEmLxJkiSNkJFO3pLsm+TGJEuSHDnseCRJkgZtZJO3JOsB/wi8GtgJeH2SnYYblSRJ0mCNbPIG7AYsqaofVdXDwNnA/kOOSZIkaaBGOXmbA9ze83tZWyZJkjRrpaqGHcNaSXIgsE9V/Vn7+4+B3arqLWPqHQYc1v7cEbhxRgPtj2cBPx12EH00m95nNr0L+D7rutn0PrPpXcD3WdeN6vs8t6q2GFu4/jAi6ZNlwDY9v+cCd4ytVFWnA6fPVFCDkGRxVc0fdhz9MpveZza9C/g+67rZ9D6z6V3A91nXzbb3GeVu0+8B85Jsn2QD4CDgvCHHJEmSNFAj2/JWVY8kOQL4CrAe8ImqunbIYUmSJA3UyCZvAFX1ZeDLw45jBox0t+84ZtP7zKZ3Ad9nXTeb3mc2vQv4Puu6WfU+IzthQZIk6YlolMe8SZIkPeGYvEmSJI0Qk7d1WJI3J1ma5MEkVyT53WHHtLaS7JnkvCQ/TlJJDhl2TGsryVFJvpfk50lWJPlCkhcNO661lWRhkh+27/PzJJck2W/YcfVDkr9u/337h2HHsjaSHNvG3/v5ybDjmo4kWydZ1P7ZeTDJdUn2GnZcayPJLeP886kkXxp2bGsqyXpJ3t/z/5ylST6QZGTHxid5epK/T3Jrkl8l+W6SXYcdVz+YvK2jkvwRcDLwQeClwHeB85NsO9TA1t7TgGuAtwG/GnIs0/Vy4FTgZcArgUeAryXZfJhBTcMy4N3AzsB84OvA55P896FGNU1JdgfeCPxw2LFM043A1j2f3xxuOGsvyabAxUCA/YAXAG8B7hpiWNOxK4//Z7MzUMA5wwxqLb0bWAi8FfgNmv9WLwSOGmZQ0/RxYB9gAc2fm6/S/Ld65HdjcsLCOirJZcAPq+qNPWU3AZ+pqlH+w0SSXwBHVNWZw46lH5I8DbgPOKCqvjDsePohyc+Ao6rq/w47lrWR5BnAlTTJ2/uAa6rqiOFGteaSHAv8YVWNbMturyQfBPaqqt8ediyDkOQ9wF8Cz6mqXw47njWR5IvA3VW1oKdsEfDMqnrN8CJbO0k2Bu4HXltV5/aUXwGcX1XvHVpwfWDL2zqoXXR4F5q/JfT6Kk1rj9YtT6f5s3TPsAOZrrbr5CCaltLvDjueaTid5i86Xx92IH2wQzvcYGmSs5PsMOyApuEA4LIkn0pyV5KrkhyRJMMObLradzgU+NdRS9xa3wFekeQ3AJLsRNOzMKrLca1Pswbsg2PKfwX8zsyH018mb+umZ9H8S3fnmPI7gWfPfDiawsnAVcAlQ45jrSX5zbZF9CHgn4D/XVVXDzmstZLkjcDzgKOHHUsfXAYcAryaphXx2cB3kzxzmEFNww7Am4Ef0XRnnQycSNM9N+peBWxP01U3ij4E/AtwXZJfA9cCi6rq1OGGtXaq6n6a/ya/N8mc9i+m/wfYg6aLe6SN7EDEJ4ixfdoZp0xDlOQjNH+L+52qWjnseKbhRuAlwKbAa4FFSV5eVdcMM6g1lWRHmnGiv1tVDw87numqqvN7fye5lCbxWQB8ZChBTc+TgMU9Qz++n2QeTfI2kpNKerwR+F5VXTXsQNbSHwEHA2+gSdxeApycZGlV/fMwA5uGPwY+QTOudyXNUIp/pxmbONJseVs3/ZTmX7SxrWxbsnprnIYkyUeB1wOvrKofDTue6aiqh6tqSVWt+h/rVcA7hhzW2tiDpuX6miSPJHkE2At4c/t7w+GGNz1V9Qua/7HOG3Ysa2k5cN2YsuuBUZ2IBUCSLYH9gf837Fim4STgw1V1dlVdXVX/QvMXhJEdY11VN1fVXjTDQLapqt2AJwNLhxvZ9Jm8rYPaFoMraJrhe72K0R6HNGskOZnmb6ivrKobhh3PADwJGMVE5/M0s8pe0vNZDJzdHo90a1ySjWhmAi4fdixr6WJgxzFlzwduHUIs/XQIzZCDs4ccx3Q8habRoNdKZkGeUFUPVNXyJJvRdNefO9U16zq7TdddHwH+JcnlNP/BexPwHJrxSCOnnZH5vPbnk4Btk7wE+FlV3Ta0wNZCkn+kaY4/ALgnyaoW0l+0LSMjJcmJwJeA22kmX7yBZjmUkVvrraruBe7tLUvyAM2/ZyPVBQyQ5MPAF4DbaFrejwaeCiwaZlzT8FGaMXvvAT5FswzSW4G/HmpU09BOVPgz4Ox2nNWo+gJwZJKlNK27LwXeCZw11KimIck+NP+/uYHm/z8n0QwROWOYcfWDS4Wsw5K8GfgrmsGV1wDvqKpvDTeqtZPk5cA3xjm1qKoOmdFgpinJRH9ojquqY2cyln5IcibwCppu+vto1kU7qaq+Msy4+iXJNxndpULOBvak6QpeAVwKHF1VY7seR0a7APQHaVrgbqMZ6/axGtH/GSV5Bc3aiL9VVZcPO561leTpwPuB/03zF4XlNC2Jx1fV2BmbIyHJ64C/AeYCPwP+A3hPVd031MD6wORNkiRphIx8X7YkSdITicmbJEnSCDF5kyRJGiEmb5IkSSPE5E2SJGmEmLxJkiSNEJM3SU84SQ5JUkmeN3XtSe/zzSTf6WNcZya5pV/3kzQ7mbxJkiSNEJM3SZKkEWLyJkljJNk1yWeSLEvyqyQ3Jvlgko0nqL9/kmuSPJTkhnZbnrF1XpzkvCT3tPe8OMnvDv5tJM02Jm+StLptgauANwH7AicDf8r4G1o/DzgF+DvgD4AlwNntnpcAJNkZ+C6wOfBG4LXA3cDXkuwysLeQNCutP+wAJGldU1X/seo4SYCLgZ8DZyVZWFV391TfCtijqi5t618AXAscD6xqWTuJZhP2V1bVw229rwDXAEcDBwz0hSTNKra8SdIYSTZJ8qEkNwMPAb8G/gUIMG9M9dtXJW4AVbUS+DSwW5IntV2te7VljyZZP8n67b2+Buw5+DeSNJvY8iZJqzsD+D3gfTTdpw8AuwH/CGw0pu6d41x/J7ABsAXNf2fXo2lhO3q8hyV5UlU92o/AJc1+Jm+S1CPJRsD+wLFVdXJP+W9OcMlWE5Q9DKwANgYepUn8zhrvBiZuktaEyZskPd6GNC1lvx5TfsgE9bdJsnvPmLf1gAOBy9uk7IEk3wZeDFxpoiZpukzeJD2R7ZvkJ2PK7gMuBd6VZDnwU5qZpnMmuMedwKeSHEPT0nY48Pz2e5V3At8CvpLkn4HlwLOAnYH1qurIPr2PpCcAkzdJT2QfG6fsWuA1wGk0XZ2/As4B3gZ8cZz6S4C/BT5IM5nhFuD1VfWNVRWq6sokuwLH0Cwr8gyaRO9K4J/69C6SniBSVcOOQZIkSR25VIgkSdIIMXmTJEkaISZvkiRJI8TkTZIkaYSYvEmSJI0QkzdJkqQRYvImSZI0QkzeJEmSRojJmyRJ0gj5/yqe1ino2sAoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_dev_values, y_dev_counts = np.unique(y_dev, return_counts=True)\n",
    "\n",
    "plt.figure(figsize=(10, 5))\n",
    "plt.bar(y_dev_values, y_dev_counts)\n",
    "plt.xticks(np.arange(10), fontsize=14)\n",
    "plt.ylabel('Number of Training Images', fontsize=16)\n",
    "plt.xlabel('Label', fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like all digits are represented equally, more or less. Let's train a logistic regression model to classify them. As in the previous exercise, we'll use `LogisticRegression` from `sklearn.linear_model`. To do this, however, we'll first need to vectorize -- in other words, flatten -- each of the images.\n",
    "\n",
    "## Flattening our digits\n",
    "\n",
    "We can do this using the numpy array method `.reshape`. Currently, x_dev has shape (60000, 28, 28), because we have 60000 handwritten digits, each of which is a (28 x 28) grid of pixels. We'd like to flatten each of those (28 x 28) images into a vector of length 28 * 28 = 784, resulting in a *flattened* dataset of shape (60000, 784)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784)\n"
     ]
    }
   ],
   "source": [
    "x_dev_flat = x_dev.reshape(60000, 28 * 28)\n",
    "print(x_dev_flat.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A convenient shortcut is to pass -1 as the \"length\" of one axis, and `numpy` will figure out how long that axis should be based on the length of the other axes and the total number of elements in the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784)\n"
     ]
    }
   ],
   "source": [
    "x_dev_flat = x_dev.reshape(-1, 28 * 28)\n",
    "print(x_dev_flat.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can also *un*-flatten an image with `.reshape`, as shown in the following block. **You will need to do this in these exercises to visualize the filters learned by the models you will train.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAN80lEQVR4nO3df6hcdXrH8c+ncf3DrBpTMYasNhuRWBWbLRqLSl2RrD9QNOqWDVgsBrN/GHChhEr6xyolEuqP0qAsuYu6sWyzLqgYZVkVo6ZFCF5j1JjU1YrdjV6SSozG+KtJnv5xT+Su3vnOzcyZOZP7vF9wmZnzzJnzcLife87Md879OiIEYPL7k6YbANAfhB1IgrADSRB2IAnCDiRxRD83ZpuP/oEeiwiPt7yrI7vtS22/aftt27d281oAesudjrPbniLpd5IWSNou6SVJiyJia2EdjuxAj/XiyD5f0tsR8U5EfCnpV5Ku6uL1APRQN2GfJekPYx5vr5b9EdtLbA/bHu5iWwC61M0HdOOdKnzjND0ihiQNSZzGA03q5si+XdJJYx5/R9L73bUDoFe6CftLkk61/V3bR0r6kaR19bQFoG4dn8ZHxD7bSyU9JWmKpAci4o3aOgNQq46H3jraGO/ZgZ7ryZdqABw+CDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUii4ymbcXiYMmVKsX7sscf2dPtLly5tWTvqqKOK686dO7dYv/nmm4v1u+66q2Vt0aJFxXU///zzYn3lypXF+u23316sN6GrsNt+V9IeSfsl7YuIs+toCkD96jiyXxQRH9TwOgB6iPfsQBLdhj0kPW37ZdtLxnuC7SW2h20Pd7ktAF3o9jT+/Ih43/YJkp6x/V8RsWHsEyJiSNKQJNmOLrcHoENdHdkj4v3qdqekxyTNr6MpAPXrOOy2p9o++uB9ST+QtKWuxgDUq5vT+BmSHrN98HX+PSJ+W0tXk8zJJ59crB955JHF+nnnnVesX3DBBS1r06ZNK6577bXXFutN2r59e7G+atWqYn3hwoUta3v27Cmu++qrrxbrL7zwQrE+iDoOe0S8I+kvauwFQA8x9AYkQdiBJAg7kARhB5Ig7EASjujfl9om6zfo5s2bV6yvX7++WO/1ZaaD6sCBA8X6jTfeWKx/8sknHW97ZGSkWP/www+L9TfffLPjbfdaRHi85RzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtlrMH369GJ948aNxfqcOXPqbKdW7XrfvXt3sX7RRRe1rH355ZfFdbN+/6BbjLMDyRF2IAnCDiRB2IEkCDuQBGEHkiDsQBJM2VyDXbt2FevLli0r1q+44opi/ZVXXinW2/1L5ZLNmzcX6wsWLCjW9+7dW6yfccYZLWu33HJLcV3UiyM7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTB9ewD4JhjjinW200vvHr16pa1xYsXF9e9/vrri/W1a9cW6xg8HV/PbvsB2zttbxmzbLrtZ2y/Vd0eV2ezAOo3kdP4X0i69GvLbpX0bEScKunZ6jGAAdY27BGxQdLXvw96laQ11f01kq6uty0Adev0u/EzImJEkiJixPYJrZ5oe4mkJR1uB0BNen4hTEQMSRqS+IAOaFKnQ287bM+UpOp2Z30tAeiFTsO+TtIN1f0bJD1eTzsAeqXtabzttZK+L+l429sl/VTSSkm/tr1Y0u8l/bCXTU52H3/8cVfrf/TRRx2ve9NNNxXrDz/8cLHebo51DI62YY+IRS1KF9fcC4Ae4uuyQBKEHUiCsANJEHYgCcIOJMElrpPA1KlTW9aeeOKJ4roXXnhhsX7ZZZcV608//XSxjv5jymYgOcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9knulFNOKdY3bdpUrO/evbtYf+6554r14eHhlrX77ruvuG4/fzcnE8bZgeQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtmTW7hwYbH+4IMPFutHH310x9tevnx5sf7QQw8V6yMjIx1vezJjnB1IjrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcHUVnnnlmsX7PPfcU6xdf3Plkv6tXry7WV6xYUay/9957HW/7cNbxOLvtB2zvtL1lzLLbbL9ne3P1c3mdzQKo30RO438h6dJxlv9LRMyrfn5Tb1sA6tY27BGxQdKuPvQCoIe6+YBuqe3XqtP841o9yfYS28O2W/8zMgA912nYfybpFEnzJI1IurvVEyNiKCLOjoizO9wWgBp0FPaI2BER+yPigKSfS5pfb1sA6tZR2G3PHPNwoaQtrZ4LYDC0HWe3vVbS9yUdL2mHpJ9Wj+dJCknvSvpxRLS9uJhx9sln2rRpxfqVV17ZstbuWnl73OHir6xfv75YX7BgQbE+WbUaZz9iAisuGmfx/V13BKCv+LoskARhB5Ig7EAShB1IgrADSXCJKxrzxRdfFOtHHFEeLNq3b1+xfskll7SsPf/888V1D2f8K2kgOcIOJEHYgSQIO5AEYQeSIOxAEoQdSKLtVW/I7ayzzirWr7vuumL9nHPOaVlrN47eztatW4v1DRs2dPX6kw1HdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH2SW7u3LnF+tKlS4v1a665plg/8cQTD7mnidq/f3+xPjJS/u/lBw4cqLOdwx5HdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH2w0C7sexFi8abaHdUu3H02bNnd9JSLYaHh4v1FStWFOvr1q2rs51Jr+2R3fZJtp+zvc32G7ZvqZZPt/2M7beq2+N63y6ATk3kNH6fpL+PiD+X9FeSbrZ9uqRbJT0bEadKerZ6DGBAtQ17RIxExKbq/h5J2yTNknSVpDXV09ZIurpHPQKowSG9Z7c9W9L3JG2UNCMiRqTRPwi2T2ixzhJJS7rsE0CXJhx229+W9Iikn0TEx/a4c8d9Q0QMSRqqXoOJHYGGTGjozfa3NBr0X0bEo9XiHbZnVvWZknb2pkUAdWh7ZPfoIfx+Sdsi4p4xpXWSbpC0srp9vCcdTgIzZswo1k8//fRi/d577y3WTzvttEPuqS4bN24s1u+8886WtccfL//KcIlqvSZyGn++pL+V9LrtzdWy5RoN+a9tL5b0e0k/7EmHAGrRNuwR8Z+SWr1Bv7jedgD0Cl+XBZIg7EAShB1IgrADSRB2IAkucZ2g6dOnt6ytXr26uO68efOK9Tlz5nTSUi1efPHFYv3uu+8u1p966qli/bPPPjvkntAbHNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IIk04+znnntusb5s2bJiff78+S1rs2bN6qinunz66acta6tWrSque8cddxTre/fu7agnDB6O7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQRJpx9oULF3ZV78bWrVuL9SeffLJY37dvX7FeuuZ89+7dxXWRB0d2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUjCEVF+gn2SpIcknSjpgKShiPhX27dJuknS/1ZPXR4Rv2nzWuWNAehaRIw76/JEwj5T0syI2GT7aEkvS7pa0t9I+iQi7ppoE4Qd6L1WYZ/I/Owjkkaq+3tsb5PU7L9mAXDIDuk9u+3Zkr4naWO1aKnt12w/YPu4FusssT1se7i7VgF0o+1p/FdPtL8t6QVJKyLiUdszJH0gKST9k0ZP9W9s8xqcxgM91vF7dkmy/S1JT0p6KiLuGac+W9KTEXFmm9ch7ECPtQp729N425Z0v6RtY4NefXB30EJJW7ptEkDvTOTT+Ask/Yek1zU69CZJyyUtkjRPo6fx70r6cfVhXum1OLIDPdbVaXxdCDvQex2fxgOYHAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ9HvK5g8k/c+Yx8dXywbRoPY2qH1J9NapOnv7s1aFvl7P/o2N28MRcXZjDRQMam+D2pdEb53qV2+cxgNJEHYgiabDPtTw9ksGtbdB7Uuit071pbdG37MD6J+mj+wA+oSwA0k0Enbbl9p+0/bbtm9toodWbL9r+3Xbm5uen66aQ2+n7S1jlk23/Yztt6rbcefYa6i322y/V+27zbYvb6i3k2w/Z3ub7Tds31Itb3TfFfrqy37r+3t221Mk/U7SAknbJb0kaVFEbO1rIy3YflfS2RHR+BcwbP+1pE8kPXRwai3b/yxpV0SsrP5QHhcR/zAgvd2mQ5zGu0e9tZpm/O/U4L6rc/rzTjRxZJ8v6e2IeCcivpT0K0lXNdDHwIuIDZJ2fW3xVZLWVPfXaPSXpe9a9DYQImIkIjZV9/dIOjjNeKP7rtBXXzQR9lmS/jDm8XYN1nzvIelp2y/bXtJ0M+OYcXCarer2hIb7+bq203j309emGR+YfdfJ9OfdaiLs401NM0jjf+dHxF9KukzSzdXpKibmZ5JO0egcgCOS7m6ymWqa8Uck/SQiPm6yl7HG6asv+62JsG+XdNKYx9+R9H4DfYwrIt6vbndKekyjbzsGyY6DM+hWtzsb7ucrEbEjIvZHxAFJP1eD+66aZvwRSb+MiEerxY3vu/H66td+ayLsL0k61fZ3bR8p6UeS1jXQxzfYnlp9cCLbUyX9QIM3FfU6STdU92+Q9HiDvfyRQZnGu9U042p43zU+/XlE9P1H0uUa/UT+vyX9YxM9tOhrjqRXq583mu5N0lqNntb9n0bPiBZL+lNJz0p6q7qdPkC9/ZtGp/Z+TaPBmtlQbxdo9K3ha5I2Vz+XN73vCn31Zb/xdVkgCb5BByRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ/D+f1mbt6t55/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x_dev[0, :].reshape(28, 28), cmap='gray')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5.1: Logistic Regression\n",
    "\n",
    "In the following block, you should:\n",
    "- Train a logistic regression model (i.e. `LogisticRegression()`) to predict the labels of digits in MNIST.\n",
    "- After training a model on the development set (`x_dev_flat`), evaluate its accuracy on the test set.\n",
    "\n",
    "To make predictions on the test set, you'll need to create a flattened version of the test set the same way we created a flattened version of the training set. If you need to review how to train and evaluate a logistic regression model, please consult [computational exercise 3](https://github.com/mengelhard/bsrt_ml4h/blob/master/notebooks/ce3.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "### CREATE A LOGISTIC REGRESSION MODEL AND TRAIN IT ON THE FLATTENED DEVELOPMENT SET ###\n",
    "\n",
    "\n",
    "### USE THE TRAINED MODEL TO MAKE PREDICTIONS ON THE (FLATTENED) TEST SET ###\n",
    "\n",
    "\n",
    "### EVALUATE THE ACCURACY OF THE MODEL'S PREDICTIONS ON THE TEST SET ###\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5.2: Visualize the learned filters\n",
    "\n",
    "The filters of your trained model can be accessed with `.coef_`. If your model is called `lr_model`, for example, then `lr_model.coef_` will be an array of size (10, 784) corresponding to the 10 filters -- one for each digit -- each of which is the same length (784) as the vectorized images.\n",
    "\n",
    "In the following block(s), you should:\n",
    "- access these vectors of coefficients\n",
    "- reshape each of them to have shape (28, 28), which makes it easier to visualize them as filters applied to the handwritten digits\n",
    "- plot each (28 x 28) filter with `plt.imshow` in the same way we plotted the first few handwritten digits at the beginning of the exercise\n",
    "\n",
    "Once you've plotted these filters, consider:\n",
    "- why do the filters look somewhat like the digits they identify?\n",
    "- why do they not look *exactly* like a handwritten digit?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "### ACCESS THE MODEL COEFFICIENTS ###\n",
    "\n",
    "\n",
    "### RESHAPE EACH OF THE 10 COEFFICIENT VECTORS TO HAVE SHAPE (28, 28) ###\n",
    "\n",
    "\n",
    "### PLOT EACH OF THE RESHAPED VECTORS (i.e. FILTERS) ###\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5.3: Multilayer Perceptron\n",
    "\n",
    "In exercises 3 and 4, we'll repeat exercises 1 and 2 with a multilayer perceptron. In the following block, you should:\n",
    "- Train a multilayer perceptron model (i.e. `MLPClassifier()`) to predict the labels of digits in MNIST.\n",
    "- After training a model on the development set (`x_dev_flat`), evaluate its accuracy on the test set.\n",
    "- You may use any architecture (i.e. hidden layer sizes) you like. A single hidden layer with 50 hidden units should be sufficient to improve performance compared to the earlier, logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "### CREATE A MULTILAYER PERCEPTRON MODEL AND TRAIN IT ON THE FLATTENED DEVELOPMENT SET ###\n",
    "\n",
    "\n",
    "### USE THE TRAINED MODEL TO MAKE PREDICTIONS ON THE (FLATTENED) TEST SET ###\n",
    "\n",
    "\n",
    "### EVALUATE THE ACCURACY OF THE MODEL'S PREDICTIONS ON THE TEST SET ###\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5.4: Visualize the MLP filters\n",
    "\n",
    "The filters in the *first fully-connected layer* of your model can be accessed with `.coefs_[0].T`. If your model is called `mlp_model`, for example, then `mlp_model.coefs_[0].T` will be an array of size (M, 784) corresponding to the M filters -- one for each hidden unit in your first hidden layer -- each of which is the same length (784) as the vectorized images.\n",
    "\n",
    "In the following block(s), you should:\n",
    "- access these vectors of coefficients\n",
    "- reshape each of them to have shape (28, 28), which makes it easier to visualize them as filters applied to the handwritten digits\n",
    "- plot each (28 x 28) filter with `plt.imshow` in the same way you plotted the filters of the logistic regression model\n",
    "\n",
    "Once you've plotted these filters, consider:\n",
    "- do these filters look similar to the filters in the logistic regression model?\n",
    "- are they recognizable?\n",
    "- what are these filters \"detecting\", and why do they look the way they do?\n",
    "\n",
    "Note that these questions are just for you to consider, and are **not** a required part of the exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "### ACCESS THE MODEL COEFFICIENTS ###\n",
    "\n",
    "\n",
    "### RESHAPE EACH OF THE M COEFFICIENT VECTORS TO HAVE SHAPE (28, 28) ###\n",
    "\n",
    "\n",
    "### PLOT EACH OF THE RESHAPED VECTORS (i.e. FILTERS) ###\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Once you've completed these exercises, please turn in the assignment as follows:\n",
    "\n",
    "If you're using Anaconda on your local machine:\n",
    "- download your notebook as html (see File > Download as > HTML (.html))\n",
    "- submit the resulting file as your assignment\n",
    "- if .html files are not accepted, .zip the file (i.e. place it in a .zip archive) and submit the .zip file instead\n",
    "\n",
    "If you're using Google Colab:\n",
    "- download your notebook as .ipynb (see File > Download > Download .ipynb)\n",
    "- if you have nbconvert installed, convert it to .html; if not, leave is as .ipynb\n",
    "- submit the resulting file as your assignment\n",
    "- if this file is not accepted, .zip the file (i.e. place it in a .zip archive) and submit the .zip file instead"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf2",
   "language": "python",
   "name": "tf2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}