L2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Solving Problems with Python: an Introduction to Data Science\n",
    "\n",
    "## Lesson 2: interacting with data\n",
    "\n",
    "In this lesson you're going to read in some sample data from a text file, then make a plot! You're going to see some stuff that we haven't talked about yet in some of the code samples I've given you below but don't worry--you'll understand it all soon! For now just ensure you can get them to work properly. \n",
    "\n",
    "### Exercise 1: reading in data from a .txt file\n",
    "\n",
    "There are many different ways to do this (including several modules that we can import that accomplish this automatically) but we are going to start with a simple example. In the cell below I am providing you with a simple function called ```getListFromFile``` which takes a file inputted and spits the result out into an array for us to analyze. \n",
    "\n",
    "Before you can use it you need to make sure you have the dow.txt file located in the same place this notebook is running from (or you need to know the explicit filepath to this file). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getListFromFile (inputFile): #input file is the path to the file you want to access, if you put the \n",
    "                            #file in the same place as this notebook you just need to enter the filename\n",
    "    dataList=[] #this initializes an empty list/array\n",
    "    try: #we use try here in case there is an error (see except) -- don't worry too much about this now\n",
    "        source=open(inputFile,\"r\") #open the file\n",
    "        dataList=source.readlines() #read the file\n",
    "        source.close() #close the file\n",
    "    except FileNotFoundError: #if try fails return a value of -1 to indicate error\n",
    "        return-1\n",
    "    return dataList #if try goes well we spit out the resulting data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to run the cell above, then run the cell below to use this function to read in the data from the dow.txt file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "rawData = getListFromFile('dow.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Check:** Does this look right? Open up the text file in a text editor (should happen automatically if you double click on it from the file browser) and compare what's stored in ```rawData``` to what you see in the text file. Are they the same? \n",
    "\n",
    "To display the contents of ```rawData``` in this notebook simply run the following command in the cell below:\n",
    "```python \n",
    "print(rawData)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should notice that the values stored in ```rawData``` look almost right, but there are some funky extra special characters trailing the numbers in the Python version you probably don't see when looking at the file in a text editor. This trailing special character (```\\n```)tells the computer you want to go to a new line. You can confirm this behavior by running the following command in the cell below:\n",
    "```python\n",
    "print(\"hello \\nworld\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#try it!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a *small* problem for us, because we want just the numbers. Computers treat different types of data differently, and right now the computer thinks that what we have in ```rawData``` is a bunch of text, ***not*** a bunch of numbers. We can check this with the ```type()``` function on a sample from the list. \n",
    "\n",
    "Run this code in the cell below:\n",
    "```python \n",
    "type(rawData[0]) #the [0] means we are taking the first item from the list--Python starts counting from 0\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#try it!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Str stands for string, which is computer-speak for text. This is easy for us to fix! We simply need to go through the list and convert each item to a decimal number, which are called **floats** in computer science. In Python we accomplish this by writing ```float(number_you_want_to_convert)```. \n",
    "\n",
    "In the cell below I've written some code that accomplishes this with a **for loop**. This is an important topic we will discuss more later, but for now just run it and investage the resulting ```dowData``` list to make sure that it worked!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "dowData = [] #initialize an empty array for us to collect the numbers in \n",
    "for i in range(len(rawData)): #this means we are going to do the actions below to everything in the list\n",
    "    dowData.append(float(rawData[i])) #take everything from the rawData list, turn it into a decimal, \n",
    "                                     #then add it to our dowData list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print the dowData list and use type() on an item in the list to make sure that the conversion was \n",
    "#successful (it should spit out float and not str)\n",
    "#your code here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2: an introduction to plotting!\n",
    "\n",
    "So we have some data, but now what? As humans we like to look at things, so the next logical step is for us to figure out a way to display this dataset in graphical form. Luckily for us there is a robust library built into Python that can automagically do a lot of what we want. It's called matplotlib! Run the cell below to import it and get it set up for our notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt #the as plt gives matplotlib.pyplot the name plt, so that we don't have\n",
    "                                #to type matplotlib.pyplot everytime we want to use it.\n",
    "%matplotlib inline \n",
    "#the line above helps us display the plots easier/better in our Jupyter notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get started right away by simply telling matplotlib to show us the data (no frills attached). To do this run ```plt.plot(dowData)``` in the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code goes here: \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Formatting:\n",
    "This isn't a very pretty plot right now. Any true scientist knows that you always need to label your axes and have a proper title at the very least! Luckily for us these are commands are very easy, and below I've added the commands you'll need to fix these problems.\n",
    "\n",
    "To add a title: ```plit.title(\"your title here\")```\n",
    "\n",
    "To add a label to the x-axis: ```plt.xlabel(\"your label here\")```\n",
    "\n",
    "To add a label to the y-axis: ```plt.ylabel(\"your label here\")```\n",
    "\n",
    "To explicitly show the plot (if you don't like the Out[#] nonsense): ```plt.show()```)\n",
    "\n",
    "Each of these commands should go on its own line, ie\n",
    "```python\n",
    "plt.plot(myData)\n",
    "plt.title(\"Here's a plot of some data I have!\")\n",
    "plt.xlabel(\"x axis (units)\")\n",
    "plt.ylabel(\"y axis (units)\")\n",
    "plt.show() #not required since we have %matplotlib inline, but still good to know\n",
    "```\n",
    "\n",
    "**Your turn:** Make a new plot of the data with proper labels and a title. The data you are plotting is the daily closing value of the Dow Jones Industrial Average Index (but I'm not telling you over what time period it occurs, as you're going to figure this out yourself soon...). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right now we are just plotting \"y\" values and matplotlib is assuming what our x values are automatically (just numbering them from 1 to the number of bins we have). What if we want to change this? \n",
    "\n",
    "Let's do this now with the help of another module called ```numpy```. Import it by running the following code in the cell below:\n",
    "```python\n",
    "import numpy as np\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import numpy here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```numpy``` has a nice built-in function we are going to use called ```linspace()``` that allows us to easily create a range of numbers automatically. Here's an example of how ```linspace()``` works:\n",
    "```python\n",
    "xValues = np.linspace(startValue, endValue, length) #startValue is where the list/array starts, endValue \n",
    "#where it ends, and length is the number of items the resulting list/array will have.\n",
    "```\n",
    "\n",
    "**Your turn:** Create an array of x values that **starts at 0 and ends at 100** (to represent percentage of time elapsed) using the ```linspace()``` function. The value you pass in for length should be **the same** as the length of the y values (you can find this out by running ```len(dowData)```)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3: more plotting\n",
    "\n",
    "Now that you have lists (of the same size) for both x and y, let's learn how to plot them together and alter the look. To plot two items together using only default values, you can simply run ```plt.plot(xValues, yValues)``` but what if we want to change the color/type of the line, add a label, or otherwise modify it?\n",
    "\n",
    "Here's a more complicated example (you should run it in a cell below and see the output to figure out what each part does):\n",
    "\n",
    "```python\n",
    "plt.plot(xCreatedAbove, dowData, \"b--\", label = \"daily closing value\")\n",
    "plt.title(\"Value of the Dow Jones Industrial Average Index Over Time\")\n",
    "plt.xlabel(\"% time elapsed\")\n",
    "plt.ylabel(\"points\")\n",
    "plt.legend()\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "**Your turn:** Modify your Dow plot to use the new x axis data points you just created, update the x label, and futz around with colors/labels/marker styles. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code goes here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a lot of different options in plots you can pass in, so here's a good cheat sheet you can refer to. \n",
    "\n",
    "**Note:** if image is not displaying the file is not located in the same directory. Locate the file \"matplotlib-cheatsheet.png\" and move it to your notebook directory and it will appear!\n",
    "<img src=\"matplotlib-cheatsheet.png\">\n",
    "\n",
    "Most of these things can be passed in as keyword arguments, for example to explicitly plot a turqoise line we can write the following:\n",
    "```python\n",
    "plt.plot(x, y, color = \"turquoise\") \n",
    "#this also works\n",
    "plt.plot(x, y, c = \"turquoise\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your turn:** Modify your plot by trying at least one new parameter from the cheatsheet above!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4: analyzing the data\n",
    "\n",
    "In the great recession that happened ~a decade ago the stock market dropped precipitously, bottoming out on March 9, 2009. Using this fact and our data above, you are going to figure out what dates this data starts and ends! Recall that each data point was taken one day apart, so if we can find the lowest one (and we know the lowest happened on 3/9/2009) we can do some simple math (Python is a great calculator!) to figure out when the data starts and stops. \n",
    "\n",
    "The easiest way to the location of the lowest data point is to simply check every point, and if it's the lowest then we will spit it out! In Python we can easily get the value of the lowest point by calling the built in ```min()``` function. Run the cell below and print out the value to see how it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "lowestValue = min(dowData) #the correct value is 6547.05 -- make sure you get this!\n",
    "#now print the value to check:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is nice, but we need to know *where* the lowest value is, now just what the number is. To find where it is we will use a loop -- specifically a **for loop**. Simply put, this loop will execute **for the number of times you tell it to**. Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "for i in range(1,5):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This loop creates a variable called ```i``` that starts as 1. Each iteration of the loop prints the value of ```i``` and then increases ```i``` until ```i = 5```, then the loop stops (this is why it only prints out 4 -- once it gets to 5 it doesn't do anything else). We could have used any letter/variable for ```i```, but this is the standard. \n",
    "\n",
    "**Your turn:** Run the cell above again but change something! Make the range bigger/smaller, change ```i``` to ```j``` or ```unicorns``` or whatever else you might want, etc.\n",
    "\n",
    "**Notice the syntax of the for loop:**\n",
    "\n",
    "1. It requires there to be something to iterate (```i``` in our case).\n",
    "\n",
    "2. It must have a range to iterate through (in our case 1 to 5).\n",
    "\n",
    "3. It's a special function, so the end of the for statement **must end with a colon (:)**.\n",
    "\n",
    "4. There must be something indented inside the loop that will be executed each iteration.\n",
    "\n",
    "Recall that to get the \"ith\" item of a list/array in Python, we simply write ```myList[i]```. For example, if we wanted to get the first item (```i = 0```) we would run ```myList[0]```. \n",
    "\n",
    "What if we want to check and see if that item's value matches something? We do this with an **if statement**. Here's a sample if statement continuing with our list example:\n",
    "\n",
    "```python\n",
    "if myList[0] == 0: #we use == to check the value of something, just one = assigns value to a variable\n",
    "    print(\"the first number in this list is 0\") #this is what we want our code to do if it equals 0\n",
    "else:\n",
    "    print(\"the first number in this list is not 0\") #this is what we want our code to do if it doesn't\n",
    "```\n",
    "\n",
    "**Notice the synatax of the if statement:**\n",
    "\n",
    "1. There must be a condition you are checking (in our example we want to see if the first item of the list is zero).\n",
    "\n",
    "2. There must be something indented inside the if statement that will execute if the condition is true (in our case if the first item is zero then we print a message saying so).\n",
    "\n",
    "3. Optionally we can add an **else** statement -- this is what we want to do if the condition **is not** true. Rule number 2 applies to the else statement as well. If you have more than two options in mind you can use **elif condition:** (syntax the same as if but with elif in front) to add more choices for the program. \n",
    "\n",
    "4. It's a special kind of function (like the for loop) so don't forget to end each if/else statement **with a colon (:)**. \n",
    "\n",
    "Here's an example of how we could find the location of the maximum value of a list by combining a for loop and an if statement.\n",
    "```python\n",
    "maxVal = max(list) #this is the biggest number in the list\n",
    "for i in range(0, len(list)): #i will go from 0 to the number of items in my list\n",
    "    if list[i] == maxVal: #this statement checks if the ith element is equal to the maximum value\n",
    "        print(\"The index of the biggest number in this list is:\",i)\n",
    "        break #this kills the loop/if statement -- there's no need for us to keep going once we found it!\n",
    "        #there's no else statement for this if statement because I only care about finding the value...\n",
    "        #if it doesn't find the value I just want it to keep going until it succeeds!\n",
    "```\n",
    "**Your turn:** Using the information/examples in this cell, find both the minimum value encoded in our dowList ***and where it's located***. Use this information to figure out what date the data starts and ends at! For example, if you find that the minimum is the 8th item in the list then the data would start on March 1st 2009. Don't forget that months have different number of days! You can use Python to help in your calculations, it can work pretty much like a normal calculator! \n",
    "\n",
    "Once you know this, modify your plot title or x axis to include the relevant information about what timespan the data is over.\n",
    "\n",
    "**Challenge 1:** Add markers to the plot that indicate the start, stop, and lowest points on the graph -- refer to the matplotlib cheat sheet.\n",
    "\n",
    "**Challenge 2:** Write a specialized function that will automatically figure out the date of an arbitrary number of days before or after 3/9/2009. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code goes here\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}