mnist_sol.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras Neural Networks\n",
    "\n",
    "This is an example of how to use Keras to create a simple neural network. The code for this example was (shamelessly) lifted straight off of the Keras source code [here](https://github.com/fchollet/keras/blob/master/examples/mnist_mlp.py).\n",
    "\n",
    "Keras is a wrapper for TensorFlow. It makes it super easy to build and train a wide range of neural networks with very little code. Here, we'll train our neural network to recognize MNIST digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Conv2D, MaxPool2D, Flatten\n",
    "from keras.optimizers import SGD,Adam\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully Connected Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define hyperparameters for our network\n",
    "`batch_size` := the number of images we show per gradient update \n",
    "\n",
    "`num_classes` := the number of different classes (or bins) that we can place our data into. Determined by the dataset.\n",
    "\n",
    "`epochs` := the number of times we go through the dataset during training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "num_classes = 10 # 10 classes because we have 10 digits (0-9)\n",
    "epochs = 5\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation\n",
    "Keras has many built-in functions such as `to_categorical` that makes data manipulation very easy. Here we download the mnist dataset (using a built-in Keras function) and then preprocess it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 train samples\n",
      "10000 test samples\n"
     ]
    }
   ],
   "source": [
    "# the data, shuffled and split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "# reshape 2d (28x28) image data into 1d vectors (28x28 = 784-d vectors)\n",
    "x_train = x_train.reshape(x_train.shape[0], 784)\n",
    "x_test = x_test.reshape(x_test.shape[0], 784)\n",
    "\n",
    "# set type to float, then clamp data to values between 0-1 instead of integers 0 - 255\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# convert class labels into vectors. Ie 3 -> [0,0,0,1,0,0,0,0,0,0]\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building a network in Keras is incredibly straight forward. Just instantiate a `Sequential` object and then use `.add` to add layers to you network. \n",
    "\n",
    "We're going to make the same network we made in the previous demo with 3 layers: a 784 neuron input, a 30 neuron hidden layer, and a 10 class output layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_7 (Dense)              (None, 30)                23550     \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 10)                310       \n",
      "=================================================================\n",
      "Total params: 23,860\n",
      "Trainable params: 23,860\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(30, activation='relu', input_shape=(784,)))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_7 (Dense)              (None, 30)                23550     \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 10)                310       \n",
      "=================================================================\n",
      "Total params: 23,860\n",
      "Trainable params: 23,860\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "learning_rate = 0.1\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=SGD(lr=learning_rate),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train your model, just use use the `.fit` function. If `verbose=1`, then Keras prints out pretty summaries of how your network is doing while training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 1s - loss: 0.4957 - acc: 0.8633 - val_loss: 0.3058 - val_acc: 0.9107\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 1s - loss: 0.2901 - acc: 0.9175 - val_loss: 0.2570 - val_acc: 0.9289\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 1s - loss: 0.2487 - acc: 0.9295 - val_loss: 0.2261 - val_acc: 0.9358\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 1s - loss: 0.2213 - acc: 0.9375 - val_loss: 0.2037 - val_acc: 0.9411\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 1s - loss: 0.2016 - acc: 0.9434 - val_loss: 0.1835 - val_acc: 0.9476\n",
      "Test loss: 0.18350554651468992\n",
      "Test accuracy: 0.9476\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train,\n",
    "                    batch_size=batch_size,\n",
    "                    epochs=epochs,\n",
    "                    verbose=1,\n",
    "                    validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's get fancier \n",
    "Here we're going to make our network deeper and also a fancy hack that helps to produce a better model known as Dropout.  On top of this, we use another Optimizer known as Adam instead of Stochastic Gradient Descent for training our model. You can find out more about ADAM here: \n",
    "\n",
    "You can read more about [Dropout here](https://medium.com/@amarbudhiraja/https-medium-com-amarbudhiraja-learning-less-to-learn-better-dropout-in-deep-machine-learning-74334da4bfc5), but the TL;DR is that Dropout randomly disables neurons in previous layers while training to regularize the action of the network. [Regularization](https://stats.stackexchange.com/questions/4961/what-is-regularization-in-plain-english) helps to prevent the phenomena of [overfitting](https://elitedatascience.com/overfitting-in-machine-learning)- the case where the network does very well on a training set by poorly on a test set.\n",
    "\n",
    "We will build a network with these layers:\n",
    "* input - 784 neurons\n",
    "* hidden 1 - 512 neurons w/ relu activation\n",
    "* dropout with prob. 0.2\n",
    "* hidden 2 - 512 neurons w/ relu activation\n",
    "* dropout with prob. 0.2\n",
    "* output - 10 neurons w/ softmax activation\n",
    "\n",
    "Again, we'll use a cross_entropy loss and an Adam optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_9 (Dense)              (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 512)               262656    \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 669,706\n",
      "Trainable params: 669,706\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# fancy\n",
    "model = Sequential()\n",
    "model.add(Dense(512, activation='relu', input_shape=(784,)))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(512, activation='relu'))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.summary()\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=Adam(),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 12s - loss: 0.2516 - acc: 0.9247 - val_loss: 0.1255 - val_acc: 0.9604\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 11s - loss: 0.1021 - acc: 0.9696 - val_loss: 0.0769 - val_acc: 0.9755\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 11s - loss: 0.0711 - acc: 0.9772 - val_loss: 0.0819 - val_acc: 0.9749\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 11s - loss: 0.0546 - acc: 0.9825 - val_loss: 0.0661 - val_acc: 0.9799\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 12s - loss: 0.0461 - acc: 0.9850 - val_loss: 0.0664 - val_acc: 0.9802\n",
      "Test loss: 0.0664316877012694\n",
      "Test accuracy: 0.9802\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train,\n",
    "                    batch_size=batch_size,\n",
    "                    epochs=epochs,\n",
    "                    verbose=1,\n",
    "                    validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and as you can see, the network's performance has shot up significantly - despite only making a few changes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction: 2\n",
      "Actual: 2\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADixJREFUeJzt3X2MVGWWx/HfkRn+0MFEpBc7ojRLfElHE2at4CpmZeMy\nEUNE/jHTJshGM4wGk51kElZ6Y8REjNmsM2JcSRghw2xYh00YFI3ZxSUbzUQdKV/GF3QXHZuXDk13\nyxgd30A4+0dfJy12PbeoulW38Hw/Saer7ql776HSP27VfaruY+4uAPGcVnYDAMpB+IGgCD8QFOEH\ngiL8QFCEHwiK8ANBEX4gKMIPBPWddu5s2rRp3tPT085dAqEMDAxodHTU6nlsU+E3s2slrZU0SdKj\n7n5/6vE9PT2qVqvN7BJAQqVSqfuxDb/sN7NJkv5V0kJJvZL6zKy30e0BaK9m3vPPlfSuu//B3Y9I\n+rWkxcW0BaDVmgn/uZL2j7t/IFv2NWa23MyqZlYdGRlpYncAitTys/3uvt7dK+5e6erqavXuANSp\nmfAPSjpv3P0Z2TIAp4Bmwr9L0gVmNsvMJkv6oaTtxbQFoNUaHupz9y/N7A5J/6Wxob6N7v5WYZ0B\naKmmxvnd/WlJTxfUC4A24uO9QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ER\nfiAowg8E1dZLd5/Kjh49WrP2wgsvJNd9/PHHG962JG3dujVZHxoaStZT3D1ZX7p0abK+eHH6so2z\nZ8+uWevtTV/vdfLkyck6msORHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCYpy/Tg899FDN2sqVK5va\n9vnnn5+sn3322cn6kiVLatYWLlyYXHfVqlXJ+lNPPZWsb968OVlPufTSS5P1Rx55JFm/8sorG943\nOPIDYRF+ICjCDwRF+IGgCD8QFOEHgiL8QFBNjfOb2YCkjyUdk/Slu1eKaKoTzZgxo+F1r7/++mQ9\nb6z89NNPb3jfeRYtWpSs79u3L1l/6aWXkvXnn3++Zm3t2rXJdbds2ZKsX3HFFcm6mSXr0RXxIZ+/\ndffRArYDoI142Q8E1Wz4XdIOM3vZzJYX0RCA9mj2Zf9V7j5oZn8h6Rkze8fdnxv/gOw/heVS/mfY\nAbRPU0d+dx/Mfg9L2iZp7gSPWe/uFXevdHV1NbM7AAVqOPxmdoaZTfnqtqQfSHqzqMYAtFYzL/un\nS9qWDad8R9K/u/t/FtIVgJazvOu2F6lSqXi1Wm3b/op07NixmrU9e/Yk180719HKcfyyffrppzVr\nc+bMSa773nvvJesPP/xwsn777bcn699GlUpF1Wq1rg84MNQHBEX4gaAIPxAU4QeCIvxAUIQfCIpL\nd9dp0qRJNWsXX3xxGzs5taSGMc8888ymtt3f35+sRxzqOxkc+YGgCD8QFOEHgiL8QFCEHwiK8ANB\nEX4gKMb50VIffvhhzdrw8HAbO8GJOPIDQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCM86MpeZd+37Bh\nQ83a4OBg0e3gJHDkB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgcsf5zWyjpEWSht39kmzZVElbJPVI\nGpB0o7v/sXVtolPlTaO9cuXKlu375ptvbtm2I6jnyP9LSdeesOxOSTvd/QJJO7P7AE4hueF39+ck\nHT5h8WJJm7LbmyTdUHBfAFqs0ff80939YHZ7SNL0gvoB0CZNn/DzsQ931/yAt5ktN7OqmVVHRkaa\n3R2AgjQa/kNm1i1J2e+aV2J09/XuXnH3SldXV4O7A1C0RsO/XdKy7PYySU8U0w6AdskNv5k9JukF\nSReZ2QEzu1XS/ZIWmNkeSX+X3QdwCskd53f3vhqlawruBfiavr5af3pjHnzwwTZ18u3EJ/yAoAg/\nEBThB4Ii/EBQhB8IivADQXHpbiSNjo4m6+vWrWt4293d3cn6mjVrknUza3jf4MgPhEX4gaAIPxAU\n4QeCIvxAUIQfCIrwA0Exzh/cwMBAsj5v3rxkfWhoqOF979ixI1mfOXNmw9tGPo78QFCEHwiK8ANB\nEX4gKMIPBEX4gaAIPxAU4/zfcvv370/WL7/88mQ97/v8ee65556atQsvvLCpbaM5HPmBoAg/EBTh\nB4Ii/EBQhB8IivADQRF+IKjccX4z2yhpkaRhd78kW7Za0o8kjWQP63f3p1vVJNIGBwdr1vr7+5Pr\nNjuOv2DBgmR91apVNWuTJk1qat9oTj1H/l9KunaC5T939znZD8EHTjG54Xf35yQdbkMvANqomff8\nd5jZ62a20czOKqwjAG3RaPjXSZotaY6kg5IeqPVAM1tuZlUzq46MjNR6GIA2ayj87n7I3Y+5+3FJ\nv5A0N/HY9e5ecfdKV1dXo30CKFhD4Tez8dOrLpH0ZjHtAGiXeob6HpM0X9I0Mzsg6W5J881sjiSX\nNCDpxy3sEUAL5Ibf3fsmWLyhBb2ghn379iXrl112Wc3a4cPpgZq8Oe5vu+22ZP3uu+9O1hnL71x8\nwg8IivADQRF+ICjCDwRF+IGgCD8QFJfu7gCHDh1K1vMur503nJdy1113Jet5Q3k4dXHkB4Ii/EBQ\nhB8IivADQRF+ICjCDwRF+IGgGOdvg7xx+BUrViTrw8PDDe/76quvTtYZx4+LIz8QFOEHgiL8QFCE\nHwiK8ANBEX4gKMIPBMU4fwGOHj2arK9ZsyZZ37ZtW1P7X7ZsWc3ao48+2tS2y/TRRx8l61OmTEnW\nU5cl/+KLL5LrfvLJJ8n6kSNHkvVzzjknWe8EHPmBoAg/EBThB4Ii/EBQhB8IivADQRF+IKjccX4z\nO0/SryRNl+SS1rv7WjObKmmLpB5JA5JudPc/tq7Vcu3du7dm7aabbkqu++KLLza171mzZiXr9913\nX83aaac19/973nj4q6++mqy/8847NWubN29Orvv+++8n6zNnzkzWU//2vM8QDA0NJeuff/55sn7L\nLbck6/fee2/NWrumNa/nL+NLST91915Jfy1phZn1SrpT0k53v0DSzuw+gFNEbvjd/aC7v5Ld/ljS\n25LOlbRY0qbsYZsk3dCqJgEU76ReE5pZj6TvS/qdpOnufjArDWnsbQGAU0Td4Tez70naKukn7v61\nN0zu7ho7HzDResvNrGpm1ZGRkaaaBVCcusJvZt/VWPA3u/tvssWHzKw7q3dLmvAqk+6+3t0r7l7p\n6uoqomcABcgNv419NWqDpLfd/WfjStslffV1smWSnii+PQCtUs9XeudJWirpDTN7LVvWL+l+Sf9h\nZrdK2ivpxta02BmmTp1as/bZZ5+1dN8ffPBBsn7NNdc0vO0FCxYk608++WSyPjAw0PC+m5U3FNhK\nF110UbKeN1x37NixhtctSm743f23kmp9MbrxvzoApeITfkBQhB8IivADQRF+ICjCDwRF+IGguHR3\nnVKXiX722WeT665evTpZ3717d7I+OjqarPf19dWsHT9+PLnurl27kvW86cPz9Pb21qzNnz+/qW2X\nKe+r0pMnT25TJ43jyA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQTHOX4C8qaIfeOCBNnUC1I8jPxAU\n4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwSVG34zO8/M\n/sfMdpvZW2b2D9ny1WY2aGavZT/Xtb5dAEWp52IeX0r6qbu/YmZTJL1sZs9ktZ+7+7+0rj0ArZIb\nfnc/KOlgdvtjM3tb0rmtbgxAa53Ue34z65H0fUm/yxbdYWavm9lGMzurxjrLzaxqZtWRkZGmmgVQ\nnLrDb2bfk7RV0k/c/SNJ6yTNljRHY68MJrxQnbuvd/eKu1e6uroKaBlAEeoKv5l9V2PB3+zuv5Ek\ndz/k7sfc/bikX0ia27o2ARStnrP9JmmDpLfd/WfjlnePe9gSSW8W3x6AVqnnbP88SUslvWFmr2XL\n+iX1mdkcSS5pQNKPW9IhgJao52z/byXZBKWni28HQLvwCT8gKMIPBEX4gaAIPxAU4QeCIvxAUIQf\nCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQ5u7t25nZiKS94xZNkzTatgZOTqf21ql9SfTWqCJ7\nm+nudV0vr63h/8bOzaruXimtgYRO7a1T+5LorVFl9cbLfiAowg8EVXb415e8/5RO7a1T+5LorVGl\n9Fbqe34A5Sn7yA+gJKWE38yuNbP/NbN3zezOMnqoxcwGzOyNbObhasm9bDSzYTN7c9yyqWb2jJnt\nyX5POE1aSb11xMzNiZmlS33uOm3G67a/7DezSZL+T9ICSQck7ZLU5+6729pIDWY2IKni7qWPCZvZ\n30j6k6Rfufsl2bJ/lnTY3e/P/uM8y93/sUN6Wy3pT2XP3JxNKNM9fmZpSTdI+nuV+Nwl+rpRJTxv\nZRz550p6193/4O5HJP1a0uIS+uh47v6cpMMnLF4saVN2e5PG/njarkZvHcHdD7r7K9ntjyV9NbN0\nqc9doq9SlBH+cyXtH3f/gDprym+XtMPMXjaz5WU3M4Hp2bTpkjQkaXqZzUwgd+bmdjphZumOee4a\nmfG6aJzw+6ar3P2vJC2UtCJ7eduRfOw9WycN19Q1c3O7TDCz9J+V+dw1OuN10coI/6Ck88bdn5Et\n6wjuPpj9Hpa0TZ03+/ChryZJzX4Pl9zPn3XSzM0TzSytDnjuOmnG6zLCv0vSBWY2y8wmS/qhpO0l\n9PENZnZGdiJGZnaGpB+o82Yf3i5pWXZ7maQnSuzlazpl5uZaM0ur5Oeu42a8dve2/0i6TmNn/N+T\n9E9l9FCjr7+U9Pvs562ye5P0mMZeBh7V2LmRWyWdLWmnpD2S/lvS1A7q7d8kvSHpdY0Frbuk3q7S\n2Ev61yW9lv1cV/Zzl+irlOeNT/gBQXHCDwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUP8POD5f\nmn7hHg0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1257307f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def predict(n):\n",
    "    \"\"\"Predict the nth test data sample using the model\"\"\"\n",
    "    print(\"Prediction: \" + str(np.argmax(np.round(model.predict(np.expand_dims(x_test[n], 0)), 2))))\n",
    "    print(\"Actual: \" + str(np.argmax(y_test[n])))\n",
    "    plt.imshow(x_test[n].reshape(28,28), cmap='Greys')\n",
    "    plt.show()\n",
    "    \n",
    "predict(2354)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Convolutional Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define hyperparameters for our network\n",
    "`batch_size` := the number of images we show per gradient update \n",
    "\n",
    "`num_classes` := the number of different classes (or bins) that we can place our data into. Determined by the dataset.\n",
    "\n",
    "`epochs` := the number of times we go through the dataset during training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "num_classes = 10 # 10 classes because we have 10 digits (0-9)\n",
    "epochs = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Prep\n",
    "This time, we don't need to reshape our data because we're using a CNN. The CNN performs _better_ with the 2D data because it preserves spatial information. However, we still want to clamp our data between 0 and 255 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# the data, shuffled and split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "# set type to float, then clamp data to values between 0-1 instead of integers 0 - 255\n",
    "x_train = x_train.reshape(list(x_train.shape) + [1])\n",
    "x_test = x_test.reshape(list(x_test.shape) + [1])\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "# convert class labels into vectors. Ie 3 -> [0,0,0,1,0,0,0,0,0,0]\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* input layer : 784 nodes (MNIST images size)\n",
    "* first convolution layer : 5x5x32\n",
    "* first max-pooling layer\n",
    "* second convolution layer : 5x5x64\n",
    "* second max-pooling layer\n",
    "* third fully-connected layer : 1024 nodes\n",
    "* output layer : 10 nodes (number of class for MNIST)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_3 (Conv2D)            (None, 24, 24, 32)        832       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 12, 12, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 8, 8, 64)          51264     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 4, 4, 64)          0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "dense_14 (Dense)             (None, 1024)              1049600   \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 10)                10250     \n",
      "=================================================================\n",
      "Total params: 1,111,946\n",
      "Trainable params: 1,111,946\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(32, (5,5), activation='relu', input_shape=(28,28,1)))\n",
    "model.add(MaxPool2D((2,2)))\n",
    "model.add(Conv2D(64, (5,5), activation='relu'))\n",
    "model.add(MaxPool2D((2,2)))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(1024, activation='relu'))\n",
    "# model.add(Dropout(0.2))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.summary()\n",
    "\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=Adam(),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 87s - loss: 0.1511 - acc: 0.9524 - val_loss: 0.0307 - val_acc: 0.9892\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 90s - loss: 0.0376 - acc: 0.9879 - val_loss: 0.0382 - val_acc: 0.9881\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 86s - loss: 0.0244 - acc: 0.9922 - val_loss: 0.0225 - val_acc: 0.9926\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 79s - loss: 0.0177 - acc: 0.9943 - val_loss: 0.0273 - val_acc: 0.9919\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 85s - loss: 0.0139 - acc: 0.9956 - val_loss: 0.0267 - val_acc: 0.9921\n",
      "Test loss: 0.026710711972102626\n",
      "Test accuracy: 0.9921\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train,\n",
    "                    batch_size=batch_size,\n",
    "                    epochs=epochs,\n",
    "                    verbose=1,\n",
    "                    validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}