Election 2016.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Election 2016: An Exploratory Data Analysis\n",
    "\n",
    "#### Table of Contents\n",
    "1. Environment Setup\n",
    "2. Loading Data\n",
    "3. Scatter Plots\n",
    "4. Boxplots\n",
    "5. Line Graph with Error Bars\n",
    "6. Bubble Chart\n",
    "7. Chloropleth Maps\n",
    "\n",
    "## Environment Setup\n",
    "Information regarding environment setup can be found under Prerequisites on the [README](../master/README.md).\n",
    "\n",
    "## Loading Data\n",
    "We start off by loading the packages that we want to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "#import matplotlib.pyplot as plt\n",
    "from plotly import tools\n",
    "import plotly.plotly as py\n",
    "import plotly.graph_objs as go\n",
    "pd.set_option('display.max_columns', 100) #overrides default to display up to 100 columns in dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's bring in the dataset. We load it into a [Pandas](http://pandas.pydata.org/) dataframe (the preferred tool when working with data in Python) and run some basic commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cycle</th>\n",
       "      <th>branch</th>\n",
       "      <th>type</th>\n",
       "      <th>matchup</th>\n",
       "      <th>forecastdate</th>\n",
       "      <th>state</th>\n",
       "      <th>startdate</th>\n",
       "      <th>enddate</th>\n",
       "      <th>pollster</th>\n",
       "      <th>grade</th>\n",
       "      <th>samplesize</th>\n",
       "      <th>population</th>\n",
       "      <th>poll_wt</th>\n",
       "      <th>rawpoll_clinton</th>\n",
       "      <th>rawpoll_trump</th>\n",
       "      <th>rawpoll_johnson</th>\n",
       "      <th>rawpoll_mcmullin</th>\n",
       "      <th>adjpoll_clinton</th>\n",
       "      <th>adjpoll_trump</th>\n",
       "      <th>adjpoll_johnson</th>\n",
       "      <th>adjpoll_mcmullin</th>\n",
       "      <th>multiversions</th>\n",
       "      <th>url</th>\n",
       "      <th>poll_id</th>\n",
       "      <th>question_id</th>\n",
       "      <th>createddate</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016</td>\n",
       "      <td>President</td>\n",
       "      <td>polls-plus</td>\n",
       "      <td>Clinton vs. Trump vs. Johnson</td>\n",
       "      <td>11/8/16</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/3/2016</td>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>ABC News/Washington Post</td>\n",
       "      <td>A+</td>\n",
       "      <td>2220.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>8.720654</td>\n",
       "      <td>47.00</td>\n",
       "      <td>43.00</td>\n",
       "      <td>4.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>45.20163</td>\n",
       "      <td>41.72430</td>\n",
       "      <td>4.626221</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.washingtonpost.com/news/the-fix/wp...</td>\n",
       "      <td>48630</td>\n",
       "      <td>76192</td>\n",
       "      <td>11/7/16</td>\n",
       "      <td>09:35:33  8 Nov 2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016</td>\n",
       "      <td>President</td>\n",
       "      <td>polls-plus</td>\n",
       "      <td>Clinton vs. Trump vs. Johnson</td>\n",
       "      <td>11/8/16</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/1/2016</td>\n",
       "      <td>11/7/2016</td>\n",
       "      <td>Google Consumer Surveys</td>\n",
       "      <td>B</td>\n",
       "      <td>26574.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>7.628472</td>\n",
       "      <td>38.03</td>\n",
       "      <td>35.69</td>\n",
       "      <td>5.46</td>\n",
       "      <td>NaN</td>\n",
       "      <td>43.34557</td>\n",
       "      <td>41.21439</td>\n",
       "      <td>5.175792</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://datastudio.google.com/u/0/#/org//repor...</td>\n",
       "      <td>48847</td>\n",
       "      <td>76443</td>\n",
       "      <td>11/7/16</td>\n",
       "      <td>09:35:33  8 Nov 2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016</td>\n",
       "      <td>President</td>\n",
       "      <td>polls-plus</td>\n",
       "      <td>Clinton vs. Trump vs. Johnson</td>\n",
       "      <td>11/8/16</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/2/2016</td>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Ipsos</td>\n",
       "      <td>A-</td>\n",
       "      <td>2195.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>6.424334</td>\n",
       "      <td>42.00</td>\n",
       "      <td>39.00</td>\n",
       "      <td>6.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>42.02638</td>\n",
       "      <td>38.81620</td>\n",
       "      <td>6.844734</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://projects.fivethirtyeight.com/polls/2016...</td>\n",
       "      <td>48922</td>\n",
       "      <td>76636</td>\n",
       "      <td>11/8/16</td>\n",
       "      <td>09:35:33  8 Nov 2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016</td>\n",
       "      <td>President</td>\n",
       "      <td>polls-plus</td>\n",
       "      <td>Clinton vs. Trump vs. Johnson</td>\n",
       "      <td>11/8/16</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/4/2016</td>\n",
       "      <td>11/7/2016</td>\n",
       "      <td>YouGov</td>\n",
       "      <td>B</td>\n",
       "      <td>3677.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>6.087135</td>\n",
       "      <td>45.00</td>\n",
       "      <td>41.00</td>\n",
       "      <td>5.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>45.65676</td>\n",
       "      <td>40.92004</td>\n",
       "      <td>6.069454</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://d25d2506sfb94s.cloudfront.net/cumulus_...</td>\n",
       "      <td>48687</td>\n",
       "      <td>76262</td>\n",
       "      <td>11/7/16</td>\n",
       "      <td>09:35:33  8 Nov 2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016</td>\n",
       "      <td>President</td>\n",
       "      <td>polls-plus</td>\n",
       "      <td>Clinton vs. Trump vs. Johnson</td>\n",
       "      <td>11/8/16</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/3/2016</td>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Gravis Marketing</td>\n",
       "      <td>B-</td>\n",
       "      <td>16639.0</td>\n",
       "      <td>rv</td>\n",
       "      <td>5.316449</td>\n",
       "      <td>47.00</td>\n",
       "      <td>43.00</td>\n",
       "      <td>3.00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>46.84089</td>\n",
       "      <td>42.33184</td>\n",
       "      <td>3.726098</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>http://www.gravispolls.com/2016/11/final-natio...</td>\n",
       "      <td>48848</td>\n",
       "      <td>76444</td>\n",
       "      <td>11/7/16</td>\n",
       "      <td>09:35:33  8 Nov 2016</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   cycle     branch        type                        matchup forecastdate  \\\n",
       "0   2016  President  polls-plus  Clinton vs. Trump vs. Johnson      11/8/16   \n",
       "1   2016  President  polls-plus  Clinton vs. Trump vs. Johnson      11/8/16   \n",
       "2   2016  President  polls-plus  Clinton vs. Trump vs. Johnson      11/8/16   \n",
       "3   2016  President  polls-plus  Clinton vs. Trump vs. Johnson      11/8/16   \n",
       "4   2016  President  polls-plus  Clinton vs. Trump vs. Johnson      11/8/16   \n",
       "\n",
       "  state  startdate    enddate                  pollster grade  samplesize  \\\n",
       "0  U.S.  11/3/2016  11/6/2016  ABC News/Washington Post    A+      2220.0   \n",
       "1  U.S.  11/1/2016  11/7/2016   Google Consumer Surveys     B     26574.0   \n",
       "2  U.S.  11/2/2016  11/6/2016                     Ipsos    A-      2195.0   \n",
       "3  U.S.  11/4/2016  11/7/2016                    YouGov     B      3677.0   \n",
       "4  U.S.  11/3/2016  11/6/2016          Gravis Marketing    B-     16639.0   \n",
       "\n",
       "  population   poll_wt  rawpoll_clinton  rawpoll_trump  rawpoll_johnson  \\\n",
       "0         lv  8.720654            47.00          43.00             4.00   \n",
       "1         lv  7.628472            38.03          35.69             5.46   \n",
       "2         lv  6.424334            42.00          39.00             6.00   \n",
       "3         lv  6.087135            45.00          41.00             5.00   \n",
       "4         rv  5.316449            47.00          43.00             3.00   \n",
       "\n",
       "   rawpoll_mcmullin  adjpoll_clinton  adjpoll_trump  adjpoll_johnson  \\\n",
       "0               NaN         45.20163       41.72430         4.626221   \n",
       "1               NaN         43.34557       41.21439         5.175792   \n",
       "2               NaN         42.02638       38.81620         6.844734   \n",
       "3               NaN         45.65676       40.92004         6.069454   \n",
       "4               NaN         46.84089       42.33184         3.726098   \n",
       "\n",
       "   adjpoll_mcmullin multiversions  \\\n",
       "0               NaN           NaN   \n",
       "1               NaN           NaN   \n",
       "2               NaN           NaN   \n",
       "3               NaN           NaN   \n",
       "4               NaN           NaN   \n",
       "\n",
       "                                                 url  poll_id  question_id  \\\n",
       "0  https://www.washingtonpost.com/news/the-fix/wp...    48630        76192   \n",
       "1  https://datastudio.google.com/u/0/#/org//repor...    48847        76443   \n",
       "2  http://projects.fivethirtyeight.com/polls/2016...    48922        76636   \n",
       "3  https://d25d2506sfb94s.cloudfront.net/cumulus_...    48687        76262   \n",
       "4  http://www.gravispolls.com/2016/11/final-natio...    48848        76444   \n",
       "\n",
       "  createddate             timestamp  \n",
       "0     11/7/16  09:35:33  8 Nov 2016  \n",
       "1     11/7/16  09:35:33  8 Nov 2016  \n",
       "2     11/8/16  09:35:33  8 Nov 2016  \n",
       "3     11/7/16  09:35:33  8 Nov 2016  \n",
       "4     11/7/16  09:35:33  8 Nov 2016  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('http://projects.fivethirtyeight.com/general-model/president_general_polls_2016.csv')\n",
    "df.head() #display the first five rows of dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows (polls): 12624\n",
      "Number of columns (data categories): 27\n",
      "\n",
      "Number of empty values for each column:\n",
      "cycle                   0\n",
      "branch                  0\n",
      "type                    0\n",
      "matchup                 0\n",
      "forecastdate            0\n",
      "state                   0\n",
      "startdate               0\n",
      "enddate                 0\n",
      "pollster                0\n",
      "grade                1287\n",
      "samplesize              3\n",
      "population              0\n",
      "poll_wt                 0\n",
      "rawpoll_clinton         0\n",
      "rawpoll_trump           0\n",
      "rawpoll_johnson      4227\n",
      "rawpoll_mcmullin    12534\n",
      "adjpoll_clinton         0\n",
      "adjpoll_trump           0\n",
      "adjpoll_johnson      4227\n",
      "adjpoll_mcmullin    12534\n",
      "multiversions       12588\n",
      "url                     3\n",
      "poll_id                 0\n",
      "question_id             0\n",
      "createddate             0\n",
      "timestamp               0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of rows (polls): \" + str(df.shape[0]))\n",
    "print(\"Number of columns (data categories): \" + str(df.shape[1]))\n",
    "\n",
    "print(\"\\nNumber of empty values for each column:\")\n",
    "print(df.isnull().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that there are 12624 polls and 27 categories of data. Of these, we can subset the dataframe to select only the categories that we're interested in. Let's go ahead and do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type</th>\n",
       "      <th>state</th>\n",
       "      <th>enddate</th>\n",
       "      <th>pollster</th>\n",
       "      <th>grade</th>\n",
       "      <th>samplesize</th>\n",
       "      <th>population</th>\n",
       "      <th>adjpoll_clinton</th>\n",
       "      <th>adjpoll_trump</th>\n",
       "      <th>adjpoll_johnson</th>\n",
       "      <th>adjpoll_mcmullin</th>\n",
       "      <th>poll_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>polls-plus</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>ABC News/Washington Post</td>\n",
       "      <td>A+</td>\n",
       "      <td>2220.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>45.20163</td>\n",
       "      <td>41.72430</td>\n",
       "      <td>4.626221</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>polls-plus</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/7/2016</td>\n",
       "      <td>Google Consumer Surveys</td>\n",
       "      <td>B</td>\n",
       "      <td>26574.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>43.34557</td>\n",
       "      <td>41.21439</td>\n",
       "      <td>5.175792</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>polls-plus</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Ipsos</td>\n",
       "      <td>A-</td>\n",
       "      <td>2195.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>42.02638</td>\n",
       "      <td>38.81620</td>\n",
       "      <td>6.844734</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>polls-plus</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/7/2016</td>\n",
       "      <td>YouGov</td>\n",
       "      <td>B</td>\n",
       "      <td>3677.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>45.65676</td>\n",
       "      <td>40.92004</td>\n",
       "      <td>6.069454</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>polls-plus</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Gravis Marketing</td>\n",
       "      <td>B-</td>\n",
       "      <td>16639.0</td>\n",
       "      <td>rv</td>\n",
       "      <td>46.84089</td>\n",
       "      <td>42.33184</td>\n",
       "      <td>3.726098</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48848</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         type state    enddate                  pollster grade  samplesize  \\\n",
       "0  polls-plus  U.S.  11/6/2016  ABC News/Washington Post    A+      2220.0   \n",
       "1  polls-plus  U.S.  11/7/2016   Google Consumer Surveys     B     26574.0   \n",
       "2  polls-plus  U.S.  11/6/2016                     Ipsos    A-      2195.0   \n",
       "3  polls-plus  U.S.  11/7/2016                    YouGov     B      3677.0   \n",
       "4  polls-plus  U.S.  11/6/2016          Gravis Marketing    B-     16639.0   \n",
       "\n",
       "  population  adjpoll_clinton  adjpoll_trump  adjpoll_johnson  \\\n",
       "0         lv         45.20163       41.72430         4.626221   \n",
       "1         lv         43.34557       41.21439         5.175792   \n",
       "2         lv         42.02638       38.81620         6.844734   \n",
       "3         lv         45.65676       40.92004         6.069454   \n",
       "4         rv         46.84089       42.33184         3.726098   \n",
       "\n",
       "   adjpoll_mcmullin  poll_id  \n",
       "0               NaN    48630  \n",
       "1               NaN    48847  \n",
       "2               NaN    48922  \n",
       "3               NaN    48687  \n",
       "4               NaN    48848  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categories = [ 'type', 'state', 'enddate', 'pollster', 'grade', 'samplesize', 'population',\n",
    "             'adjpoll_clinton', 'adjpoll_trump', 'adjpoll_johnson', 'adjpoll_mcmullin', 'poll_id']\n",
    "df2 = df.loc[:, categories]\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: We've gone ahead and decided to use the adjusted poll data (adjpoll) instead of the raw poll data (rawpoll); this will give us a slight adjustment to account for sampling error.*\n",
    "\n",
    "Awesome! But what is this \"type\" variable? We can tell from `df2.head()` that there's a type called \"polls-plus\", but we can't tell much else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['polls-plus' 'now-cast' 'polls-only']\n"
     ]
    }
   ],
   "source": [
    "print(df2.loc[:,'type'].unique()) #display unique values of the 'type' factor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see three unique types of polls. According to the source of the dataset on [FiveThirtyEight](https://fivethirtyeight.com/features/a-users-guide-to-fivethirtyeights-2016-general-election-forecast/):\n",
    "+ **Polls-plus**: Combines polls with an economic index. Since the economic index implies that this election should be a tossup, it assumes the race will tighten somewhat.\n",
    "+ **Polls-only**: A simpler, what-you-see-is-what-you-get version of the model. It assumes current polls reflect the best forecast for November, although with a lot of uncertainty.\n",
    "+ **Now-cast**: A projection of what would happen in a hypothetical election held today. Much more aggressive than the other models.\n",
    "\n",
    "We want to work with the simple adjusted poll data, not combined with other data. So we're going to take out all the polls that have been adjusted to \"polls-plus\" and \"now-cast\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type</th>\n",
       "      <th>state</th>\n",
       "      <th>enddate</th>\n",
       "      <th>pollster</th>\n",
       "      <th>grade</th>\n",
       "      <th>samplesize</th>\n",
       "      <th>population</th>\n",
       "      <th>adjpoll_clinton</th>\n",
       "      <th>adjpoll_trump</th>\n",
       "      <th>adjpoll_johnson</th>\n",
       "      <th>adjpoll_mcmullin</th>\n",
       "      <th>poll_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>polls-only</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>ABC News/Washington Post</td>\n",
       "      <td>A+</td>\n",
       "      <td>2220.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>45.21947</td>\n",
       "      <td>41.70754</td>\n",
       "      <td>4.606925</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>polls-only</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/7/2016</td>\n",
       "      <td>Google Consumer Surveys</td>\n",
       "      <td>B</td>\n",
       "      <td>26574.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>43.40083</td>\n",
       "      <td>41.14659</td>\n",
       "      <td>5.164047</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>polls-only</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Ipsos</td>\n",
       "      <td>A-</td>\n",
       "      <td>2195.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>42.01984</td>\n",
       "      <td>38.74365</td>\n",
       "      <td>6.816055</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>polls-only</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/7/2016</td>\n",
       "      <td>YouGov</td>\n",
       "      <td>B</td>\n",
       "      <td>3677.0</td>\n",
       "      <td>lv</td>\n",
       "      <td>45.68214</td>\n",
       "      <td>40.90047</td>\n",
       "      <td>6.118311</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>polls-only</td>\n",
       "      <td>U.S.</td>\n",
       "      <td>11/6/2016</td>\n",
       "      <td>Gravis Marketing</td>\n",
       "      <td>B-</td>\n",
       "      <td>16639.0</td>\n",
       "      <td>rv</td>\n",
       "      <td>46.83107</td>\n",
       "      <td>42.27754</td>\n",
       "      <td>3.749071</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48848</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         type state    enddate                  pollster grade  samplesize  \\\n",
       "0  polls-only  U.S.  11/6/2016  ABC News/Washington Post    A+      2220.0   \n",
       "1  polls-only  U.S.  11/7/2016   Google Consumer Surveys     B     26574.0   \n",
       "2  polls-only  U.S.  11/6/2016                     Ipsos    A-      2195.0   \n",
       "3  polls-only  U.S.  11/7/2016                    YouGov     B      3677.0   \n",
       "4  polls-only  U.S.  11/6/2016          Gravis Marketing    B-     16639.0   \n",
       "\n",
       "  population  adjpoll_clinton  adjpoll_trump  adjpoll_johnson  \\\n",
       "0         lv         45.21947       41.70754         4.606925   \n",
       "1         lv         43.40083       41.14659         5.164047   \n",
       "2         lv         42.01984       38.74365         6.816055   \n",
       "3         lv         45.68214       40.90047         6.118311   \n",
       "4         rv         46.83107       42.27754         3.749071   \n",
       "\n",
       "   adjpoll_mcmullin  poll_id  \n",
       "0               NaN    48630  \n",
       "1               NaN    48847  \n",
       "2               NaN    48922  \n",
       "3               NaN    48687  \n",
       "4               NaN    48848  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_po = df2[df2.loc[:,'type']=='polls-only'] #create df_po containing only the polls of type 'polls-only'\n",
    "df_po = df_po.reset_index(drop=True) #reset the dataframe indices, and drop the original indices from memory\n",
    "df_po.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>samplesize</th>\n",
       "      <th>adjpoll_clinton</th>\n",
       "      <th>adjpoll_trump</th>\n",
       "      <th>adjpoll_johnson</th>\n",
       "      <th>adjpoll_mcmullin</th>\n",
       "      <th>poll_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4207.000000</td>\n",
       "      <td>4208.000000</td>\n",
       "      <td>4208.000000</td>\n",
       "      <td>2799.000000</td>\n",
       "      <td>30.000000</td>\n",
       "      <td>4208.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1148.216068</td>\n",
       "      <td>43.322517</td>\n",
       "      <td>42.654425</td>\n",
       "      <td>4.651088</td>\n",
       "      <td>24.508827</td>\n",
       "      <td>45910.899477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2630.856265</td>\n",
       "      <td>7.097772</td>\n",
       "      <td>6.948612</td>\n",
       "      <td>2.472390</td>\n",
       "      <td>5.235812</td>\n",
       "      <td>2864.763228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>35.000000</td>\n",
       "      <td>17.115890</td>\n",
       "      <td>4.488276</td>\n",
       "      <td>-3.677883</td>\n",
       "      <td>11.028320</td>\n",
       "      <td>35362.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>447.500000</td>\n",
       "      <td>40.220230</td>\n",
       "      <td>38.449348</td>\n",
       "      <td>3.130344</td>\n",
       "      <td>23.108497</td>\n",
       "      <td>45151.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>772.000000</td>\n",
       "      <td>44.142125</td>\n",
       "      <td>42.704720</td>\n",
       "      <td>4.366810</td>\n",
       "      <td>25.135225</td>\n",
       "      <td>46384.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1236.500000</td>\n",
       "      <td>46.901398</td>\n",
       "      <td>46.315503</td>\n",
       "      <td>5.763004</td>\n",
       "      <td>27.976062</td>\n",
       "      <td>47741.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>84292.000000</td>\n",
       "      <td>86.713200</td>\n",
       "      <td>72.376610</td>\n",
       "      <td>20.357000</td>\n",
       "      <td>31.574690</td>\n",
       "      <td>48922.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         samplesize  adjpoll_clinton  adjpoll_trump  adjpoll_johnson  \\\n",
       "count   4207.000000      4208.000000    4208.000000      2799.000000   \n",
       "mean    1148.216068        43.322517      42.654425         4.651088   \n",
       "std     2630.856265         7.097772       6.948612         2.472390   \n",
       "min       35.000000        17.115890       4.488276        -3.677883   \n",
       "25%      447.500000        40.220230      38.449348         3.130344   \n",
       "50%      772.000000        44.142125      42.704720         4.366810   \n",
       "75%     1236.500000        46.901398      46.315503         5.763004   \n",
       "max    84292.000000        86.713200      72.376610        20.357000   \n",
       "\n",
       "       adjpoll_mcmullin       poll_id  \n",
       "count         30.000000   4208.000000  \n",
       "mean          24.508827  45910.899477  \n",
       "std            5.235812   2864.763228  \n",
       "min           11.028320  35362.000000  \n",
       "25%           23.108497  45151.750000  \n",
       "50%           25.135225  46384.500000  \n",
       "75%           27.976062  47741.250000  \n",
       "max           31.574690  48922.000000  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_po.describe() #display summary statistics for numerical variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scatter Plots\n",
    "We start with a simple scatter plot, with date plotted on the x-axis and voter percentage plotted on the y-axis. The original dataset contained `startdate`, `enddate`, and `forecastdate`; of these three, we've subsetted only the `enddate` because it's the most accurate representation of the timeframe of each poll."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    11/6/2016\n",
       "1    11/7/2016\n",
       "2    11/6/2016\n",
       "3    11/7/2016\n",
       "4    11/6/2016\n",
       "Name: enddate, dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_po.loc[:,'enddate'].head() #view first 5 'enddate' values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each date is an `object` type; that means that Python will see these as individual discrete variables instead of a continuous variable of dates. To fix this, we use the `to_datetime` function from Pandas on each of the date entries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   2016-11-06\n",
       "1   2016-11-07\n",
       "2   2016-11-06\n",
       "3   2016-11-07\n",
       "4   2016-11-06\n",
       "Name: enddate, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_po.loc[:,'enddate'] = pd.to_datetime(df_po.loc[:,'enddate']) #convert 'enddate' into 'datetime' variables\n",
    "df_po.loc[:, 'enddate'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enddate</th>\n",
       "      <th>adjpoll_clinton</th>\n",
       "      <th>adjpoll_trump</th>\n",
       "      <th>adjpoll_johnson</th>\n",
       "      <th>adjpoll_mcmullin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-11-06</td>\n",
       "      <td>45.21947</td>\n",
       "      <td>41.70754</td>\n",
       "      <td>4.606925</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-11-07</td>\n",
       "      <td>43.40083</td>\n",
       "      <td>41.14659</td>\n",
       "      <td>5.164047</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-11-06</td>\n",
       "      <td>42.01984</td>\n",
       "      <td>38.74365</td>\n",
       "      <td>6.816055</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-11-07</td>\n",
       "      <td>45.68214</td>\n",
       "      <td>40.90047</td>\n",
       "      <td>6.118311</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-11-06</td>\n",
       "      <td>46.83107</td>\n",
       "      <td>42.27754</td>\n",
       "      <td>3.749071</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2016-11-06</td>\n",
       "      <td>49.05626</td>\n",
       "      <td>43.87898</td>\n",
       "      <td>3.018706</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2016-11-06</td>\n",
       "      <td>45.31196</td>\n",
       "      <td>40.80614</td>\n",
       "      <td>4.230162</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2016-11-05</td>\n",
       "      <td>43.68695</td>\n",
       "      <td>40.80897</td>\n",
       "      <td>5.381917</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2016-11-06</td>\n",
       "      <td>45.03026</td>\n",
       "      <td>41.83415</td>\n",
       "      <td>8.034579</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2016-11-07</td>\n",
       "      <td>42.88452</td>\n",
       "      <td>42.18602</td>\n",
       "      <td>6.367243</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     enddate  adjpoll_clinton  adjpoll_trump  adjpoll_johnson  \\\n",
       "0 2016-11-06         45.21947       41.70754         4.606925   \n",
       "1 2016-11-07         43.40083       41.14659         5.164047   \n",
       "2 2016-11-06         42.01984       38.74365         6.816055   \n",
       "3 2016-11-07         45.68214       40.90047         6.118311   \n",
       "4 2016-11-06         46.83107       42.27754         3.749071   \n",
       "5 2016-11-06         49.05626       43.87898         3.018706   \n",
       "6 2016-11-06         45.31196       40.80614         4.230162   \n",
       "7 2016-11-05         43.68695       40.80897         5.381917   \n",
       "8 2016-11-06         45.03026       41.83415         8.034579   \n",
       "9 2016-11-07         42.88452       42.18602         6.367243   \n",
       "\n",
       "   adjpoll_mcmullin  \n",
       "0               NaN  \n",
       "1               NaN  \n",
       "2               NaN  \n",
       "3               NaN  \n",
       "4               NaN  \n",
       "5               NaN  \n",
       "6               NaN  \n",
       "7               NaN  \n",
       "8               NaN  \n",
       "9               NaN  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_po.loc[:, ['enddate', 'adjpoll_clinton', 'adjpoll_trump', 'adjpoll_johnson', 'adjpoll_mcmullin']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot! We start off with a basic scatter plot; API can be found on the [Plotly website](https://plot.ly/python/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~junseopark/14.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#PLOT ALL DATA USING PLOTLY\n",
    "\n",
    "fig = {\n",
    "    'data':[\n",
    "        #Clinton data\n",
    "        {\n",
    "            'name':\"Clinton\", \n",
    "            'x':df_po.loc[:,'enddate'], \n",
    "            'y':df_po.loc[:,'adjpoll_clinton'],\n",
    "            'mode':'markers', 'marker':{'color':'blue', 'opacity':0.1}},\n",
    "        #Trump data\n",
    "        {   \n",
    "            'name':\"Trump\",\n",
    "            'x':df_po.loc[:,'enddate'], \n",
    "            'y':df_po.loc[:,'adjpoll_trump'],\n",
    "            'mode':'markers', 'marker':{'color':'red', 'opacity':0.1}},\n",
    "        #Johnson data\n",
    "        {\n",
    "            'name':\"Johnson\", \n",
    "            'x':df_po.loc[:,'enddate'], \n",
    "            'y':df_po.loc[:,'adjpoll_johnson'],\n",
    "            'mode':'markers', 'marker':{'color':'gold', 'opacity':0.1}},\n",
    "        #McMullin data\n",
    "        {\n",
    "            'name':\"McMullin\", \n",
    "            'x':df_po.loc[:, 'enddate'], \n",
    "            'y':df_po.loc[:,'adjpoll_mcmullin'],\n",
    "            'mode':'markers', 'marker':{'color':'green', 'opacity':0.1}}\n",
    "    ],\n",
    "    #set graph layout\n",
    "    'layout':{\n",
    "        'title':\"Adjusted Poll Data\",\n",
    "        #set x-axis default range from one month before the first observation, until one month after the last observation\n",
    "        'xaxis':{'title':\"Date\", \n",
    "                 'range':[min(df_po.loc[:, 'enddate']) - pd.DateOffset(months=1), \n",
    "                                max(df_po.loc[:, 'enddate']) + pd.DateOffset(months=1)]},\n",
    "        #set y-axis default range from 0 to 100, with tick marks every 10 percentage points\n",
    "        'yaxis':{'title':\"Percentage\", \n",
    "                 'range':[0, 100], 'tick0':0, 'dtick':10},\n",
    "        #set background color\n",
    "        'plot_bgcolor':'ghostwhite', 'paper_bgcolor':'ghostwhite'}\n",
    "}\n",
    "\n",
    "#plot the data\n",
    "py.iplot(fig, filename='Election Poll Data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Graph can be seen [here](https://plot.ly/~junseopark/14/adjusted-poll-data/). \n",
    "\n",
    "It seems like Trump generally began the campaign with lower figures, but there are too many data points toward the end to determine anything conclusive. Let's see what kind of insights we can gain from the `grade` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['A+', 'B', 'A-', 'B-', 'A', nan, 'B+', 'C+', 'C-', 'C', 'D'], dtype=object)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_po.loc[:,'grade'].unique() #display unique values of the 'grade' factor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that there are 11 different `grade` types. That's a lot to work with, so we'll whittle it down to six: A+, A, B, C, D, and N/A. With the exception of A+, we drop the +/- from all the grades, then we'll plot scatterplots for each grade."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Jun Seo\\Anaconda3\\envs\\venv\\lib\\site-packages\\ipykernel\\__main__.py:10: SettingWithCopyWarning:\n",
      "\n",
      "\n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(['A+', 'B', 'A', 'NA', 'C', 'D'], dtype=object)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_grade = df_po\n",
    "for index in range(len(df_grade)):\n",
    "    grade = df_grade.loc[index, 'grade']\n",
    "    if (grade=='A-'):                     #change A- grades to A\n",
    "        df_grade.loc[index, 'grade']='A'\n",
    "    elif (grade=='B+' or  grade=='B-'):   #change B+ and B- grades to B\n",
    "        df_grade.loc[index, 'grade']='B'\n",
    "    elif (grade=='C+' or grade=='C-'):\n",
    "        df_grade.loc[index, 'grade']='C'  #change C+ and C- grades to C\n",
    "df_grade.loc[:, 'grade'][df_grade.loc[:, 'grade'].isnull()] = 'NA'     #change empty grades ('nan') to string 'NA'\n",
    "\n",
    "df_grade.grade.unique() #display unique values of 'grade'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the format of your plot grid:\n",
      "[ (1,1) x1,y1 ]  [ (1,2) x2,y2 ]\n",
      "[ (2,1) x3,y3 ]  [ (2,2) x4,y4 ]\n",
      "[ (3,1) x5,y5 ]  [ (3,2) x6,y6 ]\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~junseopark/10.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#PLOTTING CLINTON, TRUMP BY GRADE\n",
    "\n",
    "#create traces for each grade, for each candidate\n",
    "trace_CAp = go.Scatter(\n",
    "    name=\"Clinton A+\", \n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='A+'], \n",
    "    y=df_grade.loc[:, 'adjpoll_clinton'][df_grade.loc[:, 'grade']=='A'],\n",
    "    mode='markers', marker=dict(color='blue', opacity=0.1))\n",
    "trace_CA = go.Scatter(\n",
    "    name=\"Clinton A\", \n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='A'], \n",
    "    y=df_grade.loc[:, 'adjpoll_clinton'][df_grade.loc[:, 'grade']=='A'],\n",
    "    mode='markers', marker=dict(color='blue', opacity=0.1))\n",
    "trace_CB = go.Scatter(\n",
    "    name=\"Clinton B\", \n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='B'], \n",
    "    y=df_grade.loc[:, 'adjpoll_clinton'][df_grade.loc[:, 'grade']=='B'],\n",
    "    mode='markers', marker=dict(color='blue', opacity=0.1))\n",
    "trace_CC = go.Scatter(\n",
    "    name=\"Clinton C\", \n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='C'], \n",
    "    y=df_grade.loc[:, 'adjpoll_clinton'][df_grade.loc[:, 'grade']=='C'],\n",
    "    mode='markers', marker=dict(color='blue', opacity=0.1))\n",
    "trace_CD = go.Scatter(\n",
    "    name=\"Clinton D\", \n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='D'], \n",
    "    y=df_grade.loc[:, 'adjpoll_clinton'][df_grade.loc[:, 'grade']=='C'],\n",
    "    mode='markers', marker=dict(color='blue', opacity=0.1))\n",
    "trace_CNA = go.Scatter(\n",
    "    name=\"Clinton NA\", \n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='NA'], \n",
    "    y=df_grade.loc[:, 'adjpoll_clinton'][df_grade.loc[:, 'grade']=='NA'],\n",
    "    mode='markers', marker=dict(color='blue', opacity=0.1))\n",
    "\n",
    "trace_TAp = go.Scatter( \n",
    "    name=\"Trump A+\",\n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='A+'], \n",
    "    y=df_grade.loc[:, 'adjpoll_trump'][df_grade.loc[:, 'grade']=='A+'], \n",
    "    mode='markers', marker=dict(color='red', opacity=0.1))\n",
    "trace_TA = go.Scatter( \n",
    "    name=\"Trump A\",\n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='A'], \n",
    "    y=df_grade.loc[:, 'adjpoll_trump'][df_grade.loc[:, 'grade']=='A'], \n",
    "    mode='markers', marker=dict(color='red', opacity=0.1))\n",
    "trace_TB = go.Scatter( \n",
    "    name=\"Trump B\",\n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='B'], \n",
    "    y=df_grade.loc[:, 'adjpoll_trump'][df_grade.loc[:, 'grade']=='B'], \n",
    "    mode='markers', marker=dict(color='red', opacity=0.1))\n",
    "trace_TC = go.Scatter( \n",
    "    name=\"Trump C\",\n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='C'], \n",
    "    y=df_grade.loc[:, 'adjpoll_trump'][df_grade.loc[:, 'grade']=='C'], \n",
    "    mode='markers', marker=dict(color='red', opacity=0.1))\n",
    "trace_TD = go.Scatter( \n",
    "    name=\"Trump D\",\n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='D'], \n",
    "    y=df_grade.loc[:, 'adjpoll_trump'][df_grade.loc[:, 'grade']=='D'], \n",
    "    mode='markers', marker=dict(color='red', opacity=0.1))\n",
    "trace_TNA = go.Scatter( \n",
    "    name=\"Trump NA\",\n",
    "    x=df_grade.loc[:, 'enddate'][df_grade.loc[:, 'grade']=='NA'], \n",
    "    y=df_grade.loc[:, 'adjpoll_trump'][df_grade.loc[:, 'grade']=='NA'], \n",
    "    mode='markers', marker=dict(color='red', opacity=0.1))\n",
    "\n",
    "#create a figure with 3 rows and 2 columns\n",
    "fig = tools.make_subplots(rows=3, cols=2, subplot_titles=('A+', 'A', 'B',\n",
    "                                                          'C', 'D', 'NA'))\n",
    "#apply traces to matching figure in subplot\n",
    "fig.append_trace(trace_CAp, 1, 1)\n",
    "fig.append_trace(trace_TAp, 1, 1)\n",
    "fig.append_trace(trace_CA, 1, 2)\n",
    "fig.append_trace(trace_TA, 1, 2)\n",
    "fig.append_trace(trace_CB, 2, 1)\n",
    "fig.append_trace(trace_TB, 2, 1)\n",
    "fig.append_trace(trace_CC, 2, 2)\n",
    "fig.append_trace(trace_TC, 2, 2)\n",
    "fig.append_trace(trace_CD, 3, 1)\n",
    "fig.append_trace(trace_TD, 3, 1)\n",
    "fig.append_trace(trace_CNA,  3, 2)\n",
    "fig.append_trace(trace_TNA, 3, 2)\n",
    "\n",
    "#create layout: x-axis range over course of data, y-axis from 0 to 100\n",
    "layout = go.Layout(\n",
    "    title=\"Clinton and Trump, By Pollster Grade\",\n",
    "    xaxis1=dict(range=(min(df_grade.loc[:, 'enddate']) - pd.DateOffset(months=1), \n",
    "                       max(df_grade.loc[:, 'enddate']) + pd.DateOffset(months=1))),\n",
    "    xaxis2=dict(range=(min(df_grade.loc[:, 'enddate']) - pd.DateOffset(months=1), \n",
    "                       max(df_grade.loc[:, 'enddate']) + pd.DateOffset(months=1))),\n",
    "    xaxis3=dict(range=(min(df_grade.loc[:, 'enddate']) - pd.DateOffset(months=1), \n",
    "                       max(df_grade.loc[:, 'enddate']) + pd.DateOffset(months=1))),\n",
    "    xaxis4=dict(range=(min(df_grade.loc[:, 'enddate']) - pd.DateOffset(months=1), \n",
    "                       max(df_grade.loc[:, 'enddate']) + pd.DateOffset(months=1))),\n",
    "    xaxis5=dict(range=(min(df_grade.loc[:, 'enddate']) - pd.DateOffset(months=1), \n",
    "                       max(df_grade.loc[:, 'enddate']) + pd.DateOffset(months=1))),\n",
    "    xaxis6=dict(range=(min(df_grade.loc[:, 'enddate']) - pd.DateOffset(months=1), \n",
    "                       max(df_grade.loc[:, 'enddate']) + pd.DateOffset(months=1))),\n",
    "    yaxis1=dict(range=[0,100], tick0=0, dtick=25), \n",
    "    yaxis2=dict(range=[0,100], tick0=0, dtick=25), \n",
    "    yaxis3=dict(range=[0,100], tick0=0, dtick=25), \n",
    "    yaxis4=dict(range=[0,100], tick0=0, dtick=25),\n",
    "    yaxis5=dict(range=[0,100], tick0=0, dtick=25),\n",
    "    yaxis6=dict(range=[0,100], tick0=0, dtick=25),\n",
    "    plot_bgcolor='ghostwhite', paper_bgcolor='ghostwhite')\n",
    "fig['layout'].update(layout)\n",
    "\n",
    "py.iplot(fig, filename='Election Poll Data, By Grade')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to graph](https://plot.ly/~junseopark/10/clinton-and-trump-by-pollster-grade/)\n",
    "\n",
    "Better... We can see that the D grade doesn't have many points, and that the A+/A polls generally ranked Clinton higher until August 2016 — at which point there are still too many points to decipher anything. Boxplots, perhaps?\n",
    "\n",
    "## Boxplots\n",
    "By plotting boxplots of grades, we'll be able to grab a quick view on the distribution of the dataset for each grade. Let's take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~junseopark/16.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#BOXPLOTS BY GRADE\n",
    "\n",
    "trace_Clinton = go.Box(\n",
    "    name=\"Clinton\",\n",
    "    x=df_grade.loc[:,'grade'], \n",
    "    y=df_grade.loc[:, 'adjpoll_clinton'],\n",
    "    marker=dict(color='blue'))\n",
    "trace_Trump = go.Box(\n",
    "    name=\"Trump\",\n",
    "    x=df_grade.loc[:, 'grade'], \n",
    "    y=df_grade.loc[:, 'adjpoll_trump'], \n",
    "    marker=dict(color='red'))\n",
    "\n",
    "data = [trace_Clinton, trace_Trump]\n",
    "layout = go.Layout(\n",
    "    title=\"Boxplots of Election Polls, By Pollster Grade\",\n",
    "    yaxis=dict(title=\"Percentage\", range=[0, 100], tick0=0, dtick=10, zeroline=False), \n",
    "    boxmode='group', #place boxes in groups by grade\n",
    "    plot_bgcolor='ghostwhite', paper_bgcolor='ghostwhite')\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "\n",
    "py.iplot(fig, filename=\"Election Poll Boxplots, by Grade\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to graph](https://plot.ly/~junseopark/16/boxplots-of-election-polls-by-pollster-grade/)\n",
    "\n",
    "It's not in alphabetical order; that's because the first unique grade entry was an A+, second was B+/B/B-, third was A/A-, fourth was NA, etc. We can reorder by creating a dictionary of grades assigned to numbers, then creating a new variable called `graderank` to sort these values. Then we re-plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~junseopark/28.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#reordering data in order of grade\n",
    "grades = {'A+':1, 'A':2, 'B':3, 'C':4, 'D':5, 'NA':6}\n",
    "df_grade.insert(loc=len(df_grade.iloc[0]), column='graderank', value=0)\n",
    "df_grade.loc[:,'graderank'] = df_grade.loc[:, 'grade'].map(grades)\n",
    "df_grade.sort_values(by='graderank', inplace=True)\n",
    "df_grade = df_grade.drop(labels='graderank', axis=1)\n",
    "\n",
    "trace_Clinton = go.Box(\n",
    "    name=\"Clinton\",\n",
    "    x=df_grade.loc[:,'grade'], \n",
    "    y=df_grade.loc[:, 'adjpoll_clinton'],\n",
    "    marker=dict(color='blue'))\n",
    "trace_Trump = go.Box(\n",
    "    name=\"Trump\",\n",
    "    x=df_grade.loc[:, 'grade'], \n",
    "    y=df_grade.loc[:, 'adjpoll_trump'], \n",
    "    marker=dict(color='red'))\n",
    "\n",
    "data = [trace_Clinton, trace_Trump]\n",
    "layout = go.Layout(\n",
    "    title=\"Boxplots of Election Polls, By Ordered Pollster Grade\",\n",
    "    yaxis=dict(title=\"Percentage\", range=[0, 100], tick0=0, dtick=10, zeroline=False), \n",
    "    boxmode='group', #place boxes in groups by grade\n",
    "    plot_bgcolor='ghostwhite', paper_bgcolor='ghostwhite')\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "\n",
    "py.iplot(fig, filename=\"Election Poll Boxplots, by Ordered Pollster Grade\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to graph](https://plot.ly/~junseopark/28/boxplots-of-election-polls-by-ordered-pollster-grade/)\n",
    "\n",
    "D doesn't have enough data points to provide accurate insight, and by nature of NA we don't know the quality of those pollsters. Besides those two grades, we see that as the grade decreases, the distribution widens and there are more outliers. Even so, all the grade letter distributions rank Clinton above Trump. Assuming that predictions were based on poll data, then it makes sense that we were predicting Clinton to win the election.\n",
    "\n",
    "\n",
    "## Line Graph with Error Bars\n",
    "Next, let's take a look at something called a \"Continuous Error Graph.\" We can note that there are two different ways to find the standard deviation - either with Pandas or Numpy. What's the difference?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pandas SD: 7.097772011313118\n",
      "Numpy SD : 7.096928594756007\n"
     ]
    }
   ],
   "source": [
    "print(\"Pandas SD: \" + str(df_po.loc[:,'adjpoll_clinton'].std()))\n",
    "print(\"Numpy SD : \" + str(np.std(df_po.loc[:, 'adjpoll_clinton'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas employs [Bessel's Correction](https://en.wikipedia.org/wiki/Bessel's_correction), which divides the sum of squared residuals by n-1 instead of n to account for sampling error. Since our dataset is a population (compilation of every single poll regarding the 2016 elections), there's no need to make this accomodation. So we'll proceed with Numpy's standard deviation formula:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~junseopark/22.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_po.sort_values(by='enddate', inplace=True)\n",
    "clinton_upper = go.Scatter(\n",
    "    name=\"Clinton Upper Bound\", \n",
    "    x=df_po.loc[:,'enddate'], \n",
    "    y=df_po.loc[:,'adjpoll_clinton'] + np.std(df_po.loc[:,'adjpoll_clinton']), \n",
    "    mode='lines', marker=dict(color='blue'),\n",
    "    line=dict(width=0),\n",
    "    fillcolor='rgba(0, 0, 255, 0.2)',\n",
    "    fill='tonexty')\n",
    "clinton = go.Scatter(\n",
    "    name=\"Clinton Mean\", \n",
    "    x=df_po.loc[:,'enddate'], \n",
    "    y=np.mean(df_po.loc[:,'adjpoll_clinton']), \n",
    "    mode='lines', marker=dict(color='blue'),\n",
    "    line=dict(color='blue'),\n",
    "    fillcolor='rgba(0,0,255,0.2)',\n",
    "    fill='tonexty')\n",
    "clinton_lower = go.Scatter(\n",
    "    name=\"Clinton Lower Bound\",\n",
    "    x=df_po.loc[:,'enddate'],\n",
    "    y=df_po.loc[:,'adjpoll_clinton'] - np.std(df_po.loc[:,'adjpoll_clinton']),\n",
    "    mode='lines', marker=dict(color='blue'),\n",
    "    line=dict(width=0),\n",
    "    fillcolor='rgba(0,0,255,0.2)',\n",
    "    fill='tonexty')\n",
    "\n",
    "trump_upper = go.Scatter(\n",
    "    name=\"Trump Upper Bound\", \n",
    "    x=df_po.loc[:,'enddate'], \n",
    "    y=df_po.loc[:,'adjpoll_trump'] + np.std(df_po.loc[:,'adjpoll_trump']), \n",
    "    mode='lines', marker=dict(color='red'),\n",
    "    line=dict(width=0),\n",
    "    fillcolor='rgba(255, 0, 0, 0.2)',\n",
    "    fill='tonexty')\n",
    "trump = go.Scatter(\n",
    "    name=\"Trump Mean\", \n",
    "    x=df_po.loc[:,'enddate'], \n",
    "    y=np.mean(df_po.loc[:,'adjpoll_trump']), \n",
    "    mode='lines', marker=dict(color='red'),\n",
    "    line=dict(color='red'),\n",
    "    fillcolor='rgba(255, 0, 0, 0.2)',\n",
    "    fill='tonexty')\n",
    "trump_lower = go.Scatter(\n",
    "    name=\"Trump Lower Bound\",\n",
    "    x=df_po.loc[:,'enddate'],\n",
    "    y=df_po.loc[:,'adjpoll_trump'] - np.std(df_po.loc[:,'adjpoll_trump']),\n",
    "    mode='lines', marker=dict(color='red'),\n",
    "    line=dict(width=0),\n",
    "    fillcolor='rgba(255, 0, 0, 0.2)')\n",
    "\n",
    "data = [clinton_lower, clinton, clinton_upper]\n",
    "layout = go.Layout(\n",
    "    title=\"Election Poll Means with Standard Deviation\",\n",
    "    xaxis=dict(range=(min(df_grade.loc[:, 'enddate']) - pd.DateOffset(months=1), \n",
    "                            max(df_grade.loc[:, 'enddate']) + pd.DateOffset(months=1))),\n",
    "    yaxis=dict(title=\"Percentage\", range=[0,100], tick0=0, dtick=10),\n",
    "    plot_bgcolor='ghostwhite', paper_bgcolor='ghostwhite')\n",
    "\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='Election Poll Data, Continuous Error Chart')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to graph](https://plot.ly/~junseopark/22/election-poll-means-with-standard-deviation/)\n",
    "\n",
    "Clearly something's not right here. Let's take a look at a sample code provided on the Plotly API website:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~junseopark/20.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#EXAMPLE CODE FOR STANDARD DEVIATION GRAPH\n",
    "import plotly.plotly as py\n",
    "import plotly.graph_objs as go\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "dftest = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/wind_speed_laurel_nebraska.csv')\n",
    "\n",
    "upper_bound = go.Scatter(\n",
    "    name='Upper Bound',\n",
    "    x=dftest['Time'],\n",
    "    y=dftest['10 Min Sampled Avg']+dftest['10 Min Std Dev'],\n",
    "    mode='lines',\n",
    "    marker=dict(color=\"444\"),\n",
    "    line=dict(width=0),\n",
    "    fillcolor='rgba(68, 68, 68, 0.3)',\n",
    "    fill='tonexty')\n",
    "\n",
    "trace = go.Scatter(\n",
    "    name='Measurement',\n",
    "    x=dftest['Time'],\n",
    "    y=dftest['10 Min Sampled Avg'],\n",
    "    mode='lines',\n",
    "    line=dict(color='rgb(31, 119, 180)'),\n",
    "    fillcolor='rgba(68, 68, 68, 0.3)',\n",
    "    fill='tonexty')\n",
    "\n",
    "lower_bound = go.Scatter(\n",
    "    name='Lower Bound',\n",
    "    x=dftest['Time'],\n",
    "    y=dftest['10 Min Sampled Avg']-dftest['10 Min Std Dev'],\n",
    "    marker=dict(color=\"444\"),\n",
    "    line=dict(width=0),\n",
    "    mode='lines')\n",
    "\n",
    "# Trace order can be important\n",
    "# with continuous error bars\n",
    "data = [lower_bound, trace, upper_bound]\n",
    "\n",
    "layout = go.Layout(\n",
    "    yaxis=dict(title='Wind speed (m/s)'),\n",
    "    title='Continuous, variable value error bars.<br>Notice the hover text!')\n",
    "\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='pandas-continuous-error-bars')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to graph](https://plot.ly/~junseopark/20/continuous-variable-value-error-bars-notice-the-hover-text/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>10 Min Std Dev</th>\n",
       "      <th>Time</th>\n",
       "      <th>10 Min Sampled Avg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.73</td>\n",
       "      <td>2001-06-11 11:00</td>\n",
       "      <td>22.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.98</td>\n",
       "      <td>2001-06-11 11:10</td>\n",
       "      <td>23.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.87</td>\n",
       "      <td>2001-06-11 11:20</td>\n",
       "      <td>23.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.03</td>\n",
       "      <td>2001-06-11 11:30</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3.10</td>\n",
       "      <td>2001-06-11 11:40</td>\n",
       "      <td>20.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   10 Min Std Dev              Time  10 Min Sampled Avg\n",
       "0            2.73  2001-06-11 11:00                22.3\n",
       "1            1.98  2001-06-11 11:10                23.0\n",
       "2            1.87  2001-06-11 11:20                23.3\n",
       "3            2.03  2001-06-11 11:30                22.0\n",
       "4            3.10  2001-06-11 11:40                20.5"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftest.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of individual data points, the sample data is split into bins of 10 minutes each, and takes the mean and standard deviation from every 10 minutes. Of course! There's no way you can find a standard deviation from individual data points, so we'll have to adjust the dataset to be in bins of 5 days."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>mean_c</th>\n",
       "      <th>mean_t</th>\n",
       "      <th>sd_c</th>\n",
       "      <th>sd_t</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-11-06</td>\n",
       "      <td>40.866410</td>\n",
       "      <td>45.800820</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-11-11</td>\n",
       "      <td>44.608547</td>\n",
       "      <td>41.087447</td>\n",
       "      <td>5.778527</td>\n",
       "      <td>3.897990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-11-16</td>\n",
       "      <td>43.626064</td>\n",
       "      <td>43.885286</td>\n",
       "      <td>1.885810</td>\n",
       "      <td>3.093642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-11-21</td>\n",
       "      <td>29.194150</td>\n",
       "      <td>49.732650</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-11-26</td>\n",
       "      <td>47.270880</td>\n",
       "      <td>40.884840</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date     mean_c     mean_t      sd_c      sd_t\n",
       "0 2015-11-06  40.866410  45.800820  0.000000  0.000000\n",
       "1 2015-11-11  44.608547  41.087447  5.778527  3.897990\n",
       "2 2015-11-16  43.626064  43.885286  1.885810  3.093642\n",
       "3 2015-11-21  29.194150  49.732650  0.000000  0.000000\n",
       "4 2015-11-26  47.270880  40.884840  0.000000  0.000000"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#creating bins for dates\n",
    "df_date = df_po.loc[:,['enddate','adjpoll_clinton','adjpoll_trump']]\n",
    "\n",
    "for index in range(len(df_date)):     #change all dates to be on the 1st, 11th, or 21st\n",
    "    d = df_date.loc[index, 'enddate']\n",
    "    if (d.strftime(\"%d\") <= '05'):\n",
    "        df_date.loc[index, 'enddate']=d.replace(day=1)\n",
    "    elif (d.strftime(\"%d\") <= '10'):\n",
    "        df_date.loc[index, 'enddate']=d.replace(day=6)\n",
    "    elif (d.strftime(\"%d\") <= '15'):\n",
    "        df_date.loc[index, 'enddate']=d.replace(day=11)\n",
    "    elif (d.strftime(\"%d\") <= '20'):\n",
    "        df_date.loc[index, 'enddate']=d.replace(day=16)\n",
    "    elif (d.strftime(\"%d\") <= '25'):\n",
    "        df_date.loc[index, 'enddate']=d.replace(day=21)\n",
    "    elif (d.strftime(\"%d\") <= '31'):\n",
    "        df_date.loc[index, 'enddate']=d.replace(day=26)\n",
    "\n",
    "#adding rows to new dataframe df_bins\n",
    "df_bins = pd.DataFrame(columns=('date', 'mean_c', 'mean_t', 'sd_c', 'sd_t'))\n",
    "while (len(df_date) != 0):\n",
    "    row0 = df_date.iloc[0,:]\n",
    "    d = row0.loc['enddate']\n",
    "    mean_c = np.mean(df_date.loc[:,'adjpoll_clinton'][df_date.loc[:,'enddate']==d])\n",
    "    mean_t = np.mean(df_date.loc[:,'adjpoll_trump'][df_date.loc[:,'enddate']==d])\n",
    "    sd_c = np.std(df_date.loc[:,'adjpoll_clinton'][df_date.loc[:,'enddate']==d])\n",
    "    sd_t = np.std(df_date.loc[:,'adjpoll_trump'][df_date.loc[:,'enddate']==d])\n",
    "    df_bins.loc[df_bins.shape[0]] = [d, mean_c, mean_t, sd_c, sd_t]\n",
    "    df_date = df_date[df_date.loc[:,'enddate']!=d]\n",
    "    df_po = df_po.reset_index(drop=True)\n",
    "    \n",
    "#reordering rows by date\n",
    "df_bins.sort_values(by='date', inplace=True)\n",
    "df_bins.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our data resembles the sample Plotly dataset. Now we re-plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~junseopark/26.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#replotting continuous error charts\n",
    "clinton_upper = go.Scatter(\n",
    "    name=\"Clinton Upper Bound\", \n",
    "    x=df_bins.loc[:,'date'], \n",
    "    y=df_bins.loc[:,'mean_c'] + df_bins.loc[:,'sd_c'], \n",
    "    mode='lines', marker=dict(color='blue'),\n",
    "    line=dict(width=0),\n",
    "    fillcolor='rgba(0, 0, 255, 0.2)',\n",
    "    fill='tonexty')\n",
    "clinton = go.Scatter(\n",
    "    name=\"Clinton Mean\", \n",
    "    x=df_bins.loc[:,'date'], \n",
    "    y=df_bins.loc[:,'mean_c'], \n",
    "    mode='lines', marker=dict(color='blue'),\n",
    "    line=dict(color='blue'),\n",
    "    fillcolor='rgba(0,0,255,0.2)',\n",
    "    fill='tonexty')\n",
    "clinton_lower = go.Scatter(\n",
    "    name=\"Clinton Lower Bound\",\n",
    "    x=df_bins.loc[:,'date'],\n",
    "    y=df_bins.loc[:,'mean_c'] - df_bins.loc[:,'sd_c'],\n",
    "    mode='lines', marker=dict(color='blue'),\n",
    "    line=dict(width=0),\n",
    "    fillcolor='rgba(0,0,255,0.2)')\n",
    "\n",
    "trump_upper = go.Scatter(\n",
    "    name=\"Trump Upper Bound\", \n",
    "    x=df_bins.loc[:,'date'], \n",
    "    y=df_bins.loc[:,'mean_t'] + df_bins.loc[:,'sd_t'], \n",
    "    mode='lines', marker=dict(color='red'),\n",
    "    line=dict(width=0),\n",
    "    fillcolor='rgba(255, 0, 0, 0.2)',\n",
    "    fill='tonexty')\n",
    "trump = go.Scatter(\n",
    "    name=\"Trump Mean\", \n",
    "    x=df_bins.loc[:,'date'], \n",
    "    y=df_bins.loc[:,'mean_t'], \n",
    "    mode='lines', marker=dict(color='red'),\n",
    "    line=dict(color='red'),\n",
    "    fillcolor='rgba(255, 0, 0, 0.2)',\n",
    "    fill='tonexty')\n",
    "trump_lower = go.Scatter(\n",
    "    name=\"Trump Lower Bound\",\n",
    "    x=df_bins.loc[:,'date'],\n",
    "    y=df_bins.loc[:,'mean_t'] - df_bins.loc[:,'sd_t'],\n",
    "    mode='lines', marker=dict(color='red'),\n",
    "    line=dict(width=0),\n",
    "    fillcolor='rgba(255, 0, 0, 0.2)')\n",
    "\n",
    "data = [clinton_lower, clinton, clinton_upper, trump_lower, trump, trump_upper]\n",
    "layout = go.Layout(\n",
    "    title=\"Election Poll Means with Standard Deviation (Adjusted)\",\n",
    "    xaxis=dict(range=(min(df_grade.loc[:, 'enddate']) - pd.DateOffset(months=1), \n",
    "                            max(df_grade.loc[:, 'enddate']) + pd.DateOffset(months=1))),\n",
    "    yaxis=dict(title=\"Percentage\", range=[0,100], tick0=0, dtick=10),\n",
    "    plot_bgcolor='ghostwhite', paper_bgcolor='ghostwhite')\n",
    "\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.iplot(fig, filename='Election Poll Data, Continuous Error Chart (Adjusted)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to graph](https://plot.ly/~junseopark/26/election-poll-means-with-standard-deviation-adjusted/)\n",
    "\n",
    "We observe that the standard deviation seems to increase as time goes on. But one more interesting thing: the means and standard deviations essentially overlap towards the end of the election campaign, suggesting that the election might have been tighter than we originally thought."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}