Basic_Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Basic Feed Forward Network in Tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Feed Forward Network on MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we build a simple fully-connected network for MNIST. The network will have 2 hidden layers: 784 input neurons (28x28 shaped mnist), 2x layers with 256 hidden neurons , and 10 output neurons ( 1 for each digit)\n",
    "\n",
    "Tensorflow provides a convenient interface for MNIST data. This makes it really easy to test your code on a dataset that is commonly used. The code below shows you how to read MNIST images and store the labels as one-hot vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "MNIST = input_data.read_data_sets(\"../data/mnist\", one_hot = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create placeholders for X and Y. \n",
    "* Note that each MNIST image is 28x28. Additionally, the data will already be flattened into a 784 dimensional vector when we input it into the model\n",
    "* Each label is 10d - a vector element for every possible digit.\n",
    "* Make sure the shapes of the placeholders are defined so a variable number of images and labels can be fed in each batch. *This is what index 0 manages. Just put None instead of a dimension in this piece of the net*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('input'):\n",
    "    X = tf.placeholder (tf.float32, [None, 784])\n",
    "    Y = tf.placeholder (tf.float32, [None, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a weights variable and a biases variable of the appropriate shapes.\n",
    "* Initialize the weights variable from a truncated normal distribution using tf.truncated_normal(...) - this is better than setting weights to zero because it removes symmetry from backpropagation. [Here's a more in depth discussion](https://datascience.stackexchange.com/a/10930)\n",
    "* The bias variable should also be set to a small value, such as 0.1. Do this by using tf.constant(...) and inputting the value and the appropriate shape\n",
    "* When you multiply the feature vector X and the weights variable, the result should be the same shape as the bias tensor so they can be added\n",
    "* Make sure to use tf.matmul() when multiplying matrices. Using \\* will multiply element wise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Declare each layer in the network and the final logits by:\n",
    "* Creating variables for weights and biases of the appropriate sizes\n",
    "* Applying ReLu on $X \\cdot W + b$\n",
    "\n",
    "\n",
    "Network Configurations:\n",
    "* First layer has 784 input features and 256 output features\n",
    "* Second layer has 256 input features and 256 output features\n",
    "* Third layer has 256 input features and 10 output features\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('network'):\n",
    "    W1 = tf.Variable(tf.truncated_normal([784, 256], stddev=0.1))\n",
    "    b1 = tf.Variable(tf.constant(0.1, shape=[1, 256]))\n",
    "    layer1 = tf.nn.relu(tf.matmul(X, W1) + b1)\n",
    "\n",
    "    W2 = tf.Variable(tf.truncated_normal([256, 256], stddev = 0.1))\n",
    "    b2 = tf.Variable(tf.constant(0.1, shape=[1, 256]))\n",
    "    layer2 = tf.nn.relu(tf.matmul(layer1, W2) + b2)\n",
    "\n",
    "    W_out = tf.Variable(tf.truncated_normal([256, 10], stddev = 0.1))\n",
    "    b_out = tf.Variable(tf.constant(0.1, shape=[1, 10]))\n",
    "    logits = tf.matmul(layer2, W_out) + b_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the entropy using tf.nn.softmax_cross_entropy_with_logits. This will apply the softmax function to the logits before calculating the entropy. The loss as the mean over the entropy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('cross_entropy_loss'):\n",
    "    entropy = tf.nn.softmax_cross_entropy_with_logits(logits = logits, labels=Y)\n",
    "    loss = tf.reduce_mean(entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Declare the optimizer as the GradientDescentOptimizer with an appropriate learning rate. Set it to minimize the loss.\n",
    "* Note: When running the optimizer, if the loss is nan or increasing with each epoch, try decreasing the learning rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('optimizer'):\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the accuracy by:\n",
    "* using tf.equal on the predicted label and the true label\n",
    "* casting that to a float and computing the mean over all examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y_pred = tf.nn.softmax(logits)\n",
    "y_pred_cls = tf.argmax(Y_pred, 1)\n",
    "y_cls = tf.argmax(Y, 1)\n",
    "accuracy = tf.reduce_mean(tf.cast(tf.equal(y_pred_cls, y_cls), tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create summaries for Tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.summary.scalar('loss', loss)\n",
    "tf.summary.scalar('accuracy', accuracy)\n",
    "tf.summary.histogram('Weights_1', W1)\n",
    "tf.summary.histogram('Bias_1', b1)\n",
    "tf.summary.histogram('Weights_2', W2)\n",
    "tf.summary.histogram('Bias_2', b2)\n",
    "tf.summary.histogram('Weights_out', W_out)\n",
    "tf.summary.histogram('Bias_out', b_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge all the summaries together so they can called easily"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "summary_op = tf.summary.merge_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start an Interactive Session and initialize all the global variables.\n",
    "* For each epoch, run the optimizer on each X,y pair and sum up the loss over all data points\n",
    "* Print the loss after each epoch\n",
    "\n",
    "We set the batch size to 128 and epochs to 25. Feel free to play around with these variables. Additionally, every 5 epochs we calculate validation accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "epochs = 25\n",
    "sess = tf.InteractiveSession()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "writer = tf.summary.FileWriter('logs/train', graph=tf.get_default_graph())\n",
    "n_batches = (int) (MNIST.train.num_examples/batch_size)\n",
    "for i in range(epochs):\n",
    "    total_loss = 0\n",
    "    for batch in range(n_batches):\n",
    "        X_batch, y_batch = MNIST.train.next_batch(batch_size)\n",
    "        o, l, summary = sess.run([optimizer, loss, summary_op], feed_dict={X: X_batch, Y: y_batch})\n",
    "        total_loss += l\n",
    "        writer.add_summary(summary, i*n_batches + batch)\n",
    "    print(\"Epoch {0}: {1}\".format(i, total_loss))\n",
    "    if i % 5 == 0 and i!= 0:\n",
    "        X_val, y_val = MNIST.validation.next_batch(MNIST.validation.num_examples)\n",
    "        val_accuracy = sess.run(accuracy, feed_dict={X: X_val, Y: y_val})\n",
    "        print(\"\\tVal Accuracy {0}\".format(val_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After training and all validation, you'll want to return your test accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Computing accuracy ...\")\n",
    "X_batch, y_batch = MNIST.test.next_batch(MNIST.test.num_examples)\n",
    "final_accuracy = sess.run(accuracy, feed_dict={X: X_batch, Y: y_batch})\n",
    "\n",
    "print (\"Test Accuracy {0}\".format(final_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "To run Tensorboard, paste this into the terminal:\n",
    "\n",
    "tensorboard --logdir=logs/train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}