big-cheat-sheet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Big Cheat Sheet\n",
    "\n",
    "This file summarizes all the coding concepts learned from DataCamp in MA346, as well as those learned in CS230 that remain important in MA346.  It is broken into sections in the order in which we encounter the topics in the course, and the course schedule on [the main page](intro) links to each section from the day on which it's learned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Before Day 2: Review of CS230\n",
    "\n",
    "### [Introduction to Python](https://www.datacamp.com/courses/intro-to-python-for-data-science) (optional, basic review)\n",
    "\n",
    "#### Chapter 1: Python Basics\n",
    "\n",
    "Comments, which are not executed:\n",
    "```python\n",
    "# Start with a hash, then explain your code.\n",
    "```\n",
    "\n",
    "Print simple data:\n",
    "```python\n",
    "print( 1 + 5 )\n",
    "```\n",
    "\n",
    "Storing data in a variable:\n",
    "```python\n",
    "num_friends = 1000\n",
    "```\n",
    "\n",
    "Integers and real numbers (\"floating point\"):\n",
    "```python\n",
    "0, 20, -3192, 16.51309, 0.003\n",
    "```\n",
    "\n",
    "Strings:\n",
    "```python\n",
    "\"You can use double quotes.\"\n",
    "'You can use single quotes.'\n",
    "'Don\\'t forget backslashes when needed.'\n",
    "```\n",
    "\n",
    "Booleans:\n",
    "```python\n",
    "True, False\n",
    "```\n",
    "\n",
    "Asking Python for the type of a piece of data:\n",
    "```python\n",
    "type( 5 ), type( \"example\" ), type( my_data )\n",
    "```\n",
    "\n",
    "Converting among data types:\n",
    "```python\n",
    "str( 5 ), int( \"-120\" ), float( \"0.5629\" )\n",
    "```\n",
    "\n",
    "Basic arithmetic ($+$, $-$, $\\times$, $\\div$):\n",
    "```python\n",
    "1 + 2, 1 - 2, 1 * 2, 1 / 2\n",
    "```\n",
    "\n",
    "Exponents, integer division, and remainders:\n",
    "```python\n",
    "1 ** 2, 1 // 2, 1 % 2\n",
    "```\n",
    "\n",
    "#### Chapter 2: Python Lists\n",
    "\n",
    "Create a list with square brackets:\n",
    "```python\n",
    "small_primes = [ 2, 3, 5, 7, 11, 13, 17, 19, 23 ]\n",
    "```\n",
    "\n",
    "Lists can mix data of any type, even other lists:\n",
    "```python\n",
    "# Sublists are name, age, height (in m)\n",
    "heroes = [ [ 'Harry Potter', 11, 1.3 ],\n",
    "           [ 'Ron Weasley', 11, 1.5 ],\n",
    "           [ 'Hermione Granger', 11, 1.4 ] ]\n",
    "```\n",
    "\n",
    "Accessing elements from the list is zero-based:\n",
    "```python\n",
    "small_primes[0]   # == 2\n",
    "small_primes[-1]  # == 23\n",
    "```\n",
    "\n",
    "Slicing lists is left-inclusive, right-exclusive:\n",
    "```python\n",
    "small_primes[2:4] # == [5,7]\n",
    "small_primes[:4]  # == [2,3,5,7]\n",
    "small_primes[4:]  # == [11,13,17,19,23]\n",
    "```\n",
    "\n",
    "It can even use a \"stride\" to count by something other than one:\n",
    "```python\n",
    "small_primes[0:7:2]     # selects items 0,2,4,6\n",
    "small_primes[::3]       # selects items 0,3,6\n",
    "small_primes[::-1]      # selects all, but in reverse\n",
    "```\n",
    "\n",
    "If indexing gives you a list, you can index again:\n",
    "```python\n",
    "heroes[1][0]        # == 'Ron Weasley'\n",
    "```\n",
    "\n",
    "Modify an item in a list, or a slice all at once:\n",
    "```python\n",
    "some_list[5] = 10\n",
    "some_list[5:10] = [ 'my', 'new', 'entries' ]\n",
    "```\n",
    "\n",
    "Adding or removing entries from a list:\n",
    "```python\n",
    "small_primes += [ 27, 29, 31 ]\n",
    "small_primes = small_primes + [ 37, 41 ]\n",
    "small_primes.append( 43 )  # to add just one entry\n",
    "del( heroes[0] )    # Voldemort's goal\n",
    "del( heroes[:] )    # or, even better, this\n",
    "```\n",
    "\n",
    "Copying or not copying lists:\n",
    "```python\n",
    "# L will refer to the same list in memory as heroes:\n",
    "L = heroes\n",
    "# M will refer to a full copy of the heroes array:\n",
    "M = heroes[:]\n",
    "```\n",
    "\n",
    "#### Chapter 3: Functions and Packages\n",
    "\n",
    "Calling a function and saving the result:\n",
    "```python\n",
    "lastSmallPrime = max( small_primes )\n",
    "```\n",
    "\n",
    "Getting help on a function:\n",
    "```python\n",
    "help( max )\n",
    "```\n",
    "\n",
    "Methods are functions that belong to an object.  (In Python, every piece of data is an object.)\n",
    "\n",
    "Examples:\n",
    "```python\n",
    "name = 'jerry'\n",
    "name.capitalize()             # == 'Jerry'\n",
    "name.count( 'r' )             # == 2\n",
    "flavors = [ 'vanilla', 'chocolate', 'strawberry' ]\n",
    "flavors.index( 'chocolate' )  # == 1\n",
    "```\n",
    "\n",
    "Installing a package from conda:\n",
    "```bash\n",
    "conda install package_name\n",
    "```\n",
    "\n",
    "Ensuring conda forge packages are available:\n",
    "```bash\n",
    "conda config --add channels conda-forge\n",
    "```\n",
    "\n",
    "Installing a package from pip:\n",
    "```bash\n",
    "pip3 install package_name\n",
    "```\n",
    "\n",
    "Importing a package and using its contents:\n",
    "```python\n",
    "import math\n",
    "print( math.pi )\n",
    "# or if you'll use it a lot and want to be brief:\n",
    "import math as M\n",
    "print( M.pi )\n",
    "```\n",
    "\n",
    "Importing just some functions from a package:\n",
    "```python\n",
    "from math import pi, degrees\n",
    "print( \"The value of pi in degrees is:\" )\n",
    "print( degrees( pi ) )        # == 180.0\n",
    "```\n",
    "\n",
    "#### Chapter 4: NumPy\n",
    "\n",
    "Creating NumPy arrays from Python lists:\n",
    "```python\n",
    "import numpy as np\n",
    "a = np.array( [ 5, 10, 6, 3, 9 ] )\n",
    "```\n",
    "\n",
    "Elementise computations are supported:\n",
    "```python\n",
    "a * 2       # == [ 10, 20, 12, 6, 18 ]\n",
    "a < 10      # == [ True, False, True, True, True ]\n",
    "```\n",
    "\n",
    "Use comparisons to subset/select:\n",
    "```python\n",
    "a[a < 10]   # == [ 5, 6, 3, 9 ]\n",
    "```\n",
    "\n",
    "Note: NumPy arrays don't permit mixing data types:\n",
    "```python\n",
    "np.array( [ 1, \"hi\" ] )  # converts all to strings\n",
    "```\n",
    "\n",
    "NumPy arrays can be 2d, 3d, etc.:\n",
    "```python\n",
    "a = np.array( [ [ 1, 2, 3, 4 ],\n",
    "                [ 5, 6, 7, 8 ] ] )\n",
    "a.shape     # == (2,4)\n",
    "```\n",
    "\n",
    "You can index/select with comma notation:\n",
    "```python\n",
    "a[1,3]      # == 8\n",
    "a[0:2,0:2]  # == [[1,2],[5,6]]\n",
    "a[:,2]      # == [3,7]\n",
    "a[0,:]      # == [1,2,3,4]\n",
    "```\n",
    "\n",
    "Fast NumPy versions of Python functions, and some new ones:\n",
    "```python\n",
    "np.sum( a )\n",
    "np.sort( a )\n",
    "np.mean( a )\n",
    "np.median( a )\n",
    "np.std( a )\n",
    "# and others\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Python Data Science Toolbox, Part 1](https://www.datacamp.com/courses/python-data-science-toolbox-part-1) (optional, basic review)\n",
    "\n",
    "#### Chapter 1: Writing your own functions\n",
    "\n",
    "Tuples are like lists, but use parentheses, and are immutable.\n",
    "\n",
    "```python\n",
    "t = ( 6, 1, 7 )        # create a tuple\n",
    "t[0]                   # == 6\n",
    "a, b, c = t            # a==6, b==1, c==7\n",
    "```\n",
    "\n",
    "Syntax for defining a function:\n",
    "\n",
    "(A function that modifies any global variables needs the Python `global` keyword inside to identify those variables.)\n",
    "```python\n",
    "def function_name ( arguments ):\n",
    "    \"\"\"Write a docstring describing the function.\"\"\"\n",
    "    # do some things here.\n",
    "    # note the indentation!\n",
    "    # and optionally:\n",
    "    return some_value\n",
    "    # to return multiple values: return v1, v2\n",
    "```\n",
    "\n",
    "Syntax for calling a function:\n",
    "\n",
    "(Note the distinction between \"arguments\"  and \"parameters.\")\n",
    "```python\n",
    "# if you do not care about a return value:\n",
    "function_name( parameters )\n",
    "# if you wish to store the return value:\n",
    "my_variable = function_name( parameters )\n",
    "# if the function returns multiple values:\n",
    "var1, var2 = function_name( parameters )\n",
    "```\n",
    "\n",
    "#### Chapter 2: Default arguments, variable-length arguments, and scope\n",
    "\n",
    "Defining nested functions:\n",
    "```python\n",
    "def multiply_by ( x ):\n",
    "    \"\"\"Creates a function that multiplies by x\"\"\"\n",
    "    def result ( y ):\n",
    "        \"\"\"Multiplies x by y\"\"\"\n",
    "        return x * y\n",
    "    return result\n",
    "# example usage:\n",
    "df[\"height_in_inches\"].apply(\n",
    "    multiply_by( 2.54 ) )  # result is now in cm\n",
    "```\n",
    "\n",
    "Providing default values for arguments:\n",
    "```python\n",
    "def rand_between ( a=0, b=1 ):\n",
    "    \"\"\"Gives a random float between a and b\"\"\"\n",
    "    return np.random.rand() * ( b - a ) + a\n",
    "```\n",
    "\n",
    "Accepting any number of arguments:\n",
    "```python\n",
    "def commas_between ( *args ):\n",
    "    \"\"\"Returns the args as a string with commas\"\"\"\n",
    "    result = \"\"\n",
    "    for item in args:\n",
    "        result += \", \" + str(item)\n",
    "    return result[2:]\n",
    "commas_between(1,\"hi\",7)    # == \"1,hi,7\"\n",
    "```\n",
    "\n",
    "Accepting a dictionary of arguments:\n",
    "```python\n",
    "def inverted ( **kwargs ):\n",
    "    \"\"\"Interchanges keys and values in a dict\"\"\"\n",
    "    result = {}\n",
    "    for key, value in kwargs.items():\n",
    "        result[value] = key\n",
    "    return result\n",
    "inverted( jim=42, angie=9 )\n",
    "        # == { 42 : 'jim', 9 : 'angie' }\n",
    "```\n",
    "\n",
    "#### Chapter 3: Lambda functions and error handling\n",
    "\n",
    "Anonymous functions:\n",
    "```python\n",
    "lambda arg1, arg2: return_value_here\n",
    "# example:\n",
    "lambda k: k % 2 == 0    # detects whether k is even\n",
    "```\n",
    "\n",
    "Some examples in which anonymous functions are useful:\n",
    "```python\n",
    "list( map( lambda k: k%2==0, [1,2,3,4,5] ) )\n",
    "                   # == [False,True,False,True,False]\n",
    "list( filter( lambda k: k%2==0, [1,2,3,4,5] ) )\n",
    "                   # == [2,4]\n",
    "reduce( lambda x, y: x*y, [1,2,3,4,5] )\n",
    "                   # == 120 (1*2*3*4*5)\n",
    "```\n",
    "\n",
    "Raising errors if users call your functions incorrectly:\n",
    "```python\n",
    "# You can detect problems in advance:\n",
    "def factorial ( n ):\n",
    "    if type( n ) != int:\n",
    "        raise TypeError( \"n must be an int\" )\n",
    "    if n < 0:\n",
    "        raise ValueError( \"n must be nonnegative\" )\n",
    "    return reduce( lambda x,y: x*y, range( 2, n+1 ) )\n",
    "\n",
    "# Or you can let Python detect them:\n",
    "def solve_equation ( a, b ):\n",
    "    \"\"\"Solves a*x+b=0 for x\"\"\"\n",
    "    try:\n",
    "        return -b / a\n",
    "    except:\n",
    "        return None\n",
    "solve_equation( 2, -1 )    # == 0.5\n",
    "solve_equation( 0, 5 )     # == None\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Intermediate Python](https://www.datacamp.com/courses/intermediate-python-for-data-science) (required review)\n",
    "\n",
    "#### Chapter 1: Matplotlib\n",
    "\n",
    "Conventional way to import matplotlib:\n",
    "```python\n",
    "import matplotlib.pyplot as plt\n",
    "```\n",
    "\n",
    "Creating a line plot:\n",
    "```python\n",
    "plt.plot( x_data, y_data )     # create plot\n",
    "plt.show()                     # display plot\n",
    "```\n",
    "\n",
    "Creating a scatter plot:\n",
    "```python\n",
    "plt.scatter( x_data, y_data )  # create plot\n",
    "plt.show()                     # display plot\n",
    "# or this alternative form:\n",
    "plt.plot( x_data, y_data, kind='scatter' )\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "Labeling axes and adding title:\n",
    "```python\n",
    "plt.xlabel( 'x axis label here' )\n",
    "plt.ylabel( 'y axis label here' )\n",
    "plt.title( 'Title of Plot' )\n",
    "```\n",
    "\n",
    "#### Chapter 2: Dictionaries & Pandas\n",
    "\n",
    "Creating a dictionary directly:\n",
    "```python\n",
    "days_in_month = {\n",
    "    \"january\"  : 31,\n",
    "    \"february\" : 28,\n",
    "    \"march\"    : 31,\n",
    "    \"april\"    : 30,\n",
    "    # and so on, until...\n",
    "    \"december\" : 31\n",
    "}\n",
    "```\n",
    "\n",
    "Getting and using keys:\n",
    "```python\n",
    "days_in_month.keys()    # == [\"january\",\n",
    "                        #     \"february\",...]\n",
    "days_in_month[\"april\"]  # == 30\n",
    "```\n",
    "\n",
    "Updating dictionary and checking membership:\n",
    "```python\n",
    "days_in_month[\"february\"] = 29   # update for 2020\n",
    "\"tuesday\" in days_in_month       # == False\n",
    "days_in_month[\"tuesday\"] = 9     # a mistake\n",
    "\"tuesday\" in days_in_month       # == True\n",
    "del( days_in_month[\"tuesday\"] )  # delete mistake\n",
    "\"tuesday\" in days_in_month       # == False\n",
    "```\n",
    "\n",
    "Build manually from dictionary:\n",
    "```python\n",
    "import pandas as pd\n",
    "df = pd.DataFrame( {\n",
    "    \"column label 1\": [\n",
    "        \"this example uses...\",\n",
    "        \"string data here.\"\n",
    "    ],\n",
    "    \"column label 2\": [\n",
    "        100.65,  # and numerical data\n",
    "        -92.04   # here, for example\n",
    "    ]\n",
    "    # and more columns if needed\n",
    "} )\n",
    "df.index = [\n",
    "    \"put your...\",\n",
    "    \"row labels here.\"\n",
    "]\n",
    "```\n",
    "\n",
    "Import from CSV file:\n",
    "```python\n",
    "# if row and column headers are in first row/column:\n",
    "df = pd.read_csv( \"/path/to/file.csv\",\n",
    "                  index_col = 0 )\n",
    "# if no row headers:\n",
    "df = pd.read_csv( \"/path/to/file.csv\" )\n",
    "```\n",
    "\n",
    "Indexing and selecting data:\n",
    "\n",
    "```python\n",
    "df[\"column name\"]    # is a \"Series\" (labeled column)\n",
    "df[\"column name\"].values()\n",
    "                     # extract just its values\n",
    "df[[\"column name\"]]  # is a 1-column dataframe\n",
    "df[[\"col1\",\"col2\"]]  # is a 2-column dataframe\n",
    "df[n:m]              # slice of rows, a dataframe\n",
    "df.loc[\"row name\"]   # is a \"Series\" (labeled column)\n",
    "                     # yes, the row becomes a column\n",
    "df.loc[[\"row name\"]] # 1-row dataframe\n",
    "df.loc[[\"r1\",\"r2\",\"r3\"]]\n",
    "                     # 3-row dataframe\n",
    "df.loc[[\"r1\",\"r2\",\"r3\"],:]\n",
    "                     # same as previous\n",
    "df.loc[:,[\"c1\",\"c2\",\"c3\"]]\n",
    "                     # 3-column dataframe\n",
    "df.loc[[\"r1\",\"r2\",\"r3\"],[\"c1\",\"c2\"]]\n",
    "                     # 3x2 slice of the dataframe\n",
    "df.iloc[[5]]         # is a \"Series\" (labeled column)\n",
    "                     # contains the 6th row's data\n",
    "df.iloc[[5,6,7]]     # 3-row dataframe (6th-8th)\n",
    "df.iloc[[5,6,7],:]   # same as previous\n",
    "df.iloc[:,[0,4]]     # 2-column dataframe\n",
    "df.iloc[[5,6,7],[0,4]]\n",
    "                     # 3x2 slice of the dataframe\n",
    "```\n",
    "\n",
    "#### Chapter 3: Logic, Control Flow, and Filtering\n",
    "\n",
    "Python relations work on NumPy arrays and Pandas Series:\n",
    "```python\n",
    "<, <=, >, >=, ==, !=\n",
    "```\n",
    "\n",
    "Logical operators can combine the above relations:\n",
    "```python\n",
    "and, or, not          # use these on booleans\n",
    "np.logical_and(x,y)   # use these on numpy arrays\n",
    "np.logical_or(x,y)    # (assuming you have imported\n",
    "np.logical_not(x)     # numpy as np)\n",
    "```\n",
    "\n",
    "Filtering Pandas DataFrames:\n",
    "```python\n",
    "series = df[\"column\"]\n",
    "filter = series > some_number\n",
    "df[filter]  # new dataframe, a subset of the rows\n",
    "# or all at once:\n",
    "df[df[\"column\"] > some_number]\n",
    "# combining multiple conditions:\n",
    "df[np.logical_and( df[\"population\"] > 5000,\n",
    "                   df[\"area\"] < 1250 )]\n",
    "```\n",
    "\n",
    "Conditional statements:\n",
    "```python\n",
    "# Take an action if a condition is true:\n",
    "if put_condition_here:\n",
    "    take_an_action()\n",
    "# Take a different action if the condition is false:\n",
    "if put_condition_here:\n",
    "    take_an_action()\n",
    "else:\n",
    "    do_this_instead()\n",
    "# Consider multiple conditions:\n",
    "if put_condition_here:\n",
    "    take_an_action()\n",
    "elif other_condition_here:\n",
    "    do_this_instead()\n",
    "elif yet_another_condition:\n",
    "    do_this_instead2()\n",
    "else:\n",
    "    finally_this()\n",
    "```\n",
    "\n",
    "#### Chapter 4: Loops\n",
    "\n",
    "Looping constructs:\n",
    "```python\n",
    "while some_condition:\n",
    "    do_this_repeatedly()\n",
    "    # as many lines of code here as you like.\n",
    "    # note that indentation is crucial!\n",
    "    # be sure to work towards some_condition\n",
    "    # becoming false eventually!\n",
    "\n",
    "for item in my_list:\n",
    "    do_something_with( item )\n",
    "\n",
    "for index, item in enumerate( my_list ):\n",
    "    print( \"item \" + str(index) +\n",
    "           \" is \" + str(item) )\n",
    "\n",
    "for key, value in my_dict.items():\n",
    "    print( \"key \" + str(key) +\n",
    "           \" has value \" + str(value) )\n",
    "\n",
    "for item in my_numpy_array:\n",
    "    # works if the array is one-dimensional\n",
    "    print( item )\n",
    "\n",
    "for item in np.nditer( my_numpy_array ):\n",
    "    # if it is 2d, 3d, or more\n",
    "    print( item )\n",
    "\n",
    "for column_name in my_dataframe:\n",
    "    work_with( my_dataframe[column_name] )\n",
    "\n",
    "for row_name, row in my_dataframe.iterrows():\n",
    "    print( \"row \" + str(row_name) +\n",
    "           \" has these entries: \" + str(row) )\n",
    "\n",
    "# in dataframes, sometimes you can skip the for loop:\n",
    "my_dataframe[\"column\"].apply( function )  # a Series\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [pandas Foundations](https://www.datacamp.com/courses/pandas-foundations) (required review)\n",
    "\n",
    "#### Chapter 1: Data ingestion & inspection\n",
    "\n",
    "Basic DataFrame/Series tools:\n",
    "\n",
    "```python\n",
    "df.head(5)           # first five rows\n",
    "df.tail(5)           # last five rows\n",
    "series.head(5)       # head, tail also work on series\n",
    "df.info()            # summary of the data types used\n",
    "```\n",
    "\n",
    "Adding details to reading DataFrames from CSV files:\n",
    "\n",
    "```python\n",
    "# if no column headers:\n",
    "df = pd.read_csv( \"/path/to/file.csv\",\n",
    "                  index_col = 0, header = None,\n",
    "                  names = ['column','names','here'] )\n",
    "# if any missing data you want to mark as NaN:\n",
    "# (na_values can be a list of patterns,\n",
    "# or a dict mapping column names to patterns/lists)\n",
    "df = pd.read_csv( \"/path/to/file.csv\",\n",
    "                  na_values = 'pattern to replace' )\n",
    "# and many other options!  (see the documentation)\n",
    "```\n",
    "\n",
    "To get a DataFrame with a date/time index:\n",
    "```python\n",
    "# read as dates any columns that pandas can:\n",
    "df = pd.read_csv( \"/path/to/file.csv\",\n",
    "                  parse_dates = True )\n",
    "# read as dates just the columns you specify:\n",
    "df = pd.read_csv( \"/path/to/file.csv\",\n",
    "                  parse_dates = ['column','names'] )\n",
    "# to use one of those columns as a date/time index:\n",
    "df = pd.read_csv( \"/path/to/file.csv\",\n",
    "                  parse_dates = True,\n",
    "                  index_col = 'Date' )\n",
    "# combine multiple columns to form a date:\n",
    "df = pd.read_csv( \"/path/to/file.csv\",\n",
    "                  parse_dates = [[column,indices]] )\n",
    "```\n",
    "\n",
    "Export to CSV or XLSX file:\n",
    "```python\n",
    "df.to_csv( \"/path/to/output_file.csv\" )\n",
    "df.to_excel( \"/path/to/output_file.xlsx\" )\n",
    "```\n",
    "\n",
    "You can also create a plot from a `Series` or dataframe:\n",
    "```python\n",
    "df.plot()            # or series.plot()\n",
    "plt.show()\n",
    "# or to show each column in a subplot:\n",
    "df.plot( subplots = True )\n",
    "plt.show()\n",
    "# or to plot certain columns:\n",
    "df.plot( x='col name', y='other col name' )\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "A few small ways to customize plots:\n",
    "```python\n",
    "plt.xscale( 'log' )\n",
    "plt.yticks( [ 0, 5, 10, 20 ] )\n",
    "plt.grid()\n",
    "```\n",
    "\n",
    "To create a histogram:\n",
    "```python\n",
    "plt.hist( data, bins=10 )      # 10 is the default\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "To \"clean up\" so you can start a new plot:\n",
    "```python\n",
    "plt.clf()\n",
    "```\n",
    "\n",
    "Write text onto a plot:\n",
    "```python\n",
    "plt.text( x, y, 'Text to write' )\n",
    "```\n",
    "\n",
    "To save a plot to a file:\n",
    "```python\n",
    "# before plt.show(), call:\n",
    "plt.savefig( 'filename.png' )  # or .jpg or .pdf\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Manipulating DataFrames with pandas](https://www.datacamp.com/courses/manipulating-dataframes-with-pandas) (required review)\n",
    "\n",
    "#### Chapter 1: Extracting and transforming data\n",
    "\n",
    "(This builds on the DataCamp Intermediate Python section.)\n",
    "```python\n",
    "df.iloc[5:7,0:4]     # select ranges of rows/columns\n",
    "df.iloc[:,0:4]       # select a range, all rows\n",
    "df.iloc[[5,6],:]     # select a range, all columns\n",
    "df.iloc[5:,:]        # all but the first five rows\n",
    "df.loc['A':'B',:]    # colons can take row names too\n",
    "                     # (but include both endpoints)\n",
    "df.loc[:,'C':'D']    # ...also column names\n",
    "df.loc['D':'A':-1]   # rows by name, reverse order\n",
    "```\n",
    "\n",
    "(This builds on the DataCamp Intermediate Python section.)\n",
    "```python\n",
    "# avoid using np.logical_and with & instead:\n",
    "df[(df[\"population\"] > 5000)\n",
    " & (df[\"area\"] < 1250 )]\n",
    "# avoid using np.logical_or with | instead:\n",
    "df[(df[\"population\"] > 5000)\n",
    " | (df[\"area\"] < 1250 )]\n",
    "# filtering for missing values:\n",
    "df.loc[:,df.all()]  # only columns with no zeroes\n",
    "df.loc[:,df.any()]  # only columns with some nonzero\n",
    "df.loc[:,df.isnull().any()]\n",
    "                    # only columns with a NaN entry\n",
    "df.loc[:,df.notnull().all()]\n",
    "                    # only columns with no NaNs\n",
    "df.dropna( how='any' )\n",
    "                    # remove rows with any NaNs\n",
    "df.dropna( how='all' )\n",
    "                    # remove rows with all NaNs\n",
    "```\n",
    "\n",
    "You can filter one column based on another using these tools.\n",
    "\n",
    "Apply a function to each value, returning a new DataFrame:\n",
    "```python\n",
    "def example ( x ):\n",
    "    return x + 1\n",
    "df.apply( example )   # adds 1 to everything\n",
    "df.apply( lambda x: x + 1 )    # same\n",
    "# some functions are built-in:\n",
    "df.floordiv( 10 )\n",
    "# many operators automatically repeat:\n",
    "df['total pay'] = df['salary'] + df['bonus']\n",
    "# to extend a dataframe with a new column:\n",
    "df['new col'] = df['old col'].apply( f )\n",
    "# slightly different syntax for the index:\n",
    "df.index = df.index.map( f )\n",
    "```\n",
    "\n",
    "You can also map columns through `dict`s, not just functions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Before Day 3\n",
    "\n",
    "### [Manipulating DataFrames with pandas](https://learn.datacamp.com/courses/manipulating-dataframes-with-pandas)\n",
    "\n",
    "#### Chapter 2: Advanced indexing\n",
    "\n",
    "Creating a Series:\n",
    "```python\n",
    "s = pd.Series( [ 5.0, 3.2, 1.9 ] )   # just data\n",
    "s = pd.Series( [ 5.0, 3.2, 1.9 ],    # data with...\n",
    "  index = [ 'Mon', 'Tue', 'Wed' ] )  # ...an index\n",
    "s.index[2:]                          # sliceable\n",
    "s.index.name = 'Day of Week'         # index name\n",
    "```\n",
    "\n",
    "Column headings are also a series:\n",
    "```python\n",
    "df.columns                    # is a pd.Series\n",
    "df.columns.name               # usually a string\n",
    "df.columns.values             # column names array\n",
    "```\n",
    "\n",
    "Using an existing column as the index:\n",
    "```python\n",
    "df.index = df['column name']  # once it's the index,\n",
    "del df['column name']         # it can be deleted\n",
    "```\n",
    "\n",
    "Making an index from multiple columns that, when taken together, uniquely identify rows:\n",
    "```python\n",
    "df = df.set_index( [ 'last_name', 'first_name' ] )\n",
    "df.index.name                 # will be None\n",
    "df.index.names                # list of strings\n",
    "df = df.sort_index()          # hierarchical sort\n",
    "df.loc[('Jones', 'Heide')]    # index rows by tuples\n",
    "df.loc[('Jones', 'Heide'),    # and you can fetch an\n",
    "       'birth_date']          # entry that way, too\n",
    "df.loc['Jones']               # all rows of Joneses\n",
    "df.loc['Jones':'Menendez']    # many last names\n",
    "df.loc[(['Jones','Wu'], 'Heide'), :]\n",
    "      # get both rows: Heide Jones and Heide Wu\n",
    "      # (yes, the colon is necessary for rows)\n",
    "df.loc[(['Jones','Wu'], 'Heide'), 'birth_date']\n",
    "      # get Heide Jones's and Heide Wu's birth dates\n",
    "df.loc[('Jones',['Heide','Henry']),:]\n",
    "      # get full rows for Heide and Henry Jones\n",
    "df.loc[('Jones',slice('Heide','Henry')),:]\n",
    "      # 'Heide':'Henry' doesn't work inside tuples\n",
    "```\n",
    "\n",
    "#### Chapter 3: Rearranging and reshaping data\n",
    "\n",
    "If columns A and B together uniquely identify entries in column C, you can create a new DataFrame showing this:\n",
    "```python\n",
    "new_df = df.pivot( index   = 'A',\n",
    "                   columns = 'B',\n",
    "                   values  = 'C' )\n",
    "# or do this for all columns at once,\n",
    "# creating a hierarchical column index:\n",
    "new_df = df.pivot( index   = 'A',\n",
    "                   columns = 'B' )\n",
    "```\n",
    "\n",
    "You can also invert pivoting, which is called \"melting:\"\n",
    "```python\n",
    "old_df = pd.melt( new_df,\n",
    "    id_vars = [ 'A' ],          # old index\n",
    "    value_vars = [ 'values','of','column','B' ],\n",
    "        # optional...pandas can often infer it\n",
    "    var_name = 'B',       # these two lines just\n",
    "    value_name = 'C' )    # restore column names\n",
    "```\n",
    "\n",
    "Convert hierarchical row index to a hierarchical column index:\n",
    "```python\n",
    "# assume df.index.names is ['A','B','C']\n",
    "df = df.unstack( level = 'B' )  # or A or C\n",
    "# equivalently:\n",
    "df = df.unstack( level = 1 )    # or 0 or 2\n",
    "# and this can be inverted:\n",
    "df = df.stack( level = 'B' )    # for example\n",
    "```\n",
    "\n",
    "To change the nesting order of a hierarchical index:\n",
    "```python\n",
    "df = df.swaplevel( levelindex1, levelindex2 )\n",
    "df = sort_index()               # necessary now\n",
    "```\n",
    "\n",
    "If the pivot column(s) aren't a unique index, use `pivot_table` instead, often with an aggregation function:\n",
    "```python\n",
    "new_df = df.pivot_table(        # this pivot table\n",
    "    index   = 'A',              # is a frequency\n",
    "    columns = 'B',              # table, because\n",
    "    values  = 'C',              # aggfunc is count\n",
    "    aggfunc = 'count' )         # (default: mean)\n",
    "# other aggfuncs: 'sum', plus many functions in\n",
    "# numpy, such as np.min, np.max, np.median, etc.\n",
    "# You can also add column totals at the bottom:\n",
    "new_df = df.pivot_table(\n",
    "    index   = 'A',\n",
    "    columns = 'B',\n",
    "    values  = 'C',\n",
    "    margins = True )            # add column sums\n",
    "```\n",
    "\n",
    "#### Chapter 4: Grouping data\n",
    "\n",
    "Group all columns except column A by the unique values in column A, then apply some aggregation method to each group:\n",
    "```python\n",
    "# example: total number of rows for each weekday\n",
    "df.groupby( 'weekday' ).count()\n",
    "# example: total sales in each city\n",
    "df.groupby( 'city' )['sales'].sum()\n",
    "# multiple column names gives a multi-level index\n",
    "df.groupby( [ 'city', 'state' ] ).mean()\n",
    "# you can group by any series with the same index;\n",
    "# here is an example:\n",
    "series = df['column A'].apply( np.round )\n",
    "df.groupby( series )['column B'].sum()\n",
    "```\n",
    "\n",
    "The `agg` method lets us do even more:\n",
    "```python\n",
    "# you can do multiple aggregations at once;\n",
    "# this, too, gives a multi-level index:\n",
    "df.groupby( 'weekday' ).agg( [ 'max', 'sum' ] )\n",
    "# or you can pass a user-defined function:\n",
    "def sum_of_squares ( series ):\n",
    "    return ( series * series ).sum()\n",
    "df.groupby( 'weekday' )['column name']\n",
    "  .agg( sum_of_squares )\n",
    "# or dictionaries can let us apply different\n",
    "# aggregations to different columns:\n",
    "df.groupby( 'weekday' )[['Quantity Ordered',\n",
    "                         'Total Cost']]\n",
    "  .agg( { 'Quantity Ordered' : 'median',\n",
    "          'Total Cost'       : 'sum' } )\n",
    "```\n",
    "\n",
    "`transform` is just like `apply`, except that it must convert each value into exactly one other, thus preserving shape.\n",
    "```python\n",
    "# example: convert values to zscores\n",
    "from scipy.stats import zscore\n",
    "df.groupby( 'region' )['gdp'].transform( zscore )\n",
    "  .agg( [ 'min', 'max' ] )\n",
    "# example: impute missing values as medians\n",
    "def impute_median(series):\n",
    "    return series.fillna(series.median())\n",
    "grouped = df.groupby( [ 'col B', 'col C' ] )\n",
    "df['col A'] = grouped['col A']\n",
    "              .transform( impute_median )\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Before Day 4: Review of Visualization in CS230\n",
    "\n",
    "### [pandas Foundations](https://www.datacamp.com/courses/pandas-foundations)\n",
    "\n",
    "#### Chapter 2: Exploratory data analysis\n",
    "\n",
    "Plots from DataFrames:\n",
    "```python\n",
    "# any of these can be followed with plt.title(),\n",
    "# plt.xlabel(), etc., then plt.show() at the end:\n",
    "df.plot( x='col name', y='col name', kind='scatter' )\n",
    "df.plot( y='col name', kind='box' )\n",
    "df.plot( y='col name', kind='hist' )\n",
    "df.plot( kind='box' ) # all columns side-by-side\n",
    "df.plot( kind='hist' ) # all columns on same axes\n",
    "```\n",
    "\n",
    "Histogram options: `bins`, `range`, `normed`, `cumulative`, and more.\n",
    "\n",
    "```python\n",
    "df.describe()        # summary statistics\n",
    "# df.describe() makes calls to df.mean(), df.std(),\n",
    "# df.median(), df.quantile(), etc...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Before Day 5\n",
    "\n",
    "### [Intermediate Python](https://www.datacamp.com/courses/intermediate-python-for-data-science)\n",
    "\n",
    "#### Chapter 5: Case Study: Hacker Statistics\n",
    "\n",
    "Uniform random numbers from NumPy:\n",
    "```python\n",
    "np.random.seed( my_int )  # choose a random sequence\n",
    "# (seeds are optional, but ensure reproducibility)\n",
    "np.random.rand()          # uniform random in [0,1)\n",
    "np.random.randint(a,b)    # uniform random in a:b\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Statistical Thinking in Python, Part 1](https://www.datacamp.com/courses/statistical-thinking-in-python-part-1)\n",
    "\n",
    "#### Chapter 1: Graphical Exploratory Data Analysis\n",
    "\n",
    "Plotting a histogram of your data:\n",
    "```python\n",
    "import matplotlib.pyplot as plt\n",
    "plt.hist( df['column of interest'] )\n",
    "plt.xlabel( 'column name (units)' )\n",
    "plt.ylabel( 'number of [fill in]' )\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "To change the $y$ axis to probabilities:\n",
    "```python\n",
    "plt.hist( df['column of interest'], normed=True )\n",
    "```\n",
    "\n",
    "Sometimes there is a sensible choice of where to place bin boundaries, based on the meaning of the $x$ axis.  Example:\n",
    "```python\n",
    "plt.hist( df['column of percentages'],\n",
    "          bins=[0,10,20,30,40,50,60,70,80,90,100] )\n",
    "```\n",
    "\n",
    "Change default plot styling to Seaborn:\n",
    "```python\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "# then do plotting afterwards\n",
    "```\n",
    "\n",
    "If your data has observations as rows and features as columns, with two features of interest in columns A and B, you can create a \"bee swarm plot\" as follows.\n",
    "```python\n",
    "# assuming your dataframe is called df:\n",
    "sns.swarmplot( x='A', y='B', data=df )\n",
    "plt.xlabel( 'explain column A' )\n",
    "plt.ylabel( 'explain column B' )\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "To show a data's distribution as an Empirical Cumulative Distribution Function plot:\n",
    "```python\n",
    "# the data must be sorted from lowest to highest:\n",
    "x = np.sort( df['column of interest'] )\n",
    "# the y values must count evenly from 0% to 100%:\n",
    "y = np.arange( 1, len(x)+1 ) / len(x)\n",
    "# then create and show the plot:\n",
    "plt.plot( x, y, marker='.', linestyle='none' )\n",
    "plt.xlabel( 'explain column of interest' )\n",
    "plt.ylabel( 'ECDF' )\n",
    "plt.margins( 0.02 )  # 2% margin all around\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "Multiple ECDFs on one plot:\n",
    "```python\n",
    "# prepare the data as before, but now repeatedly:\n",
    "# (this could be abstracted into a function)\n",
    "x = np.sort( df['column 1'] )\n",
    "y = np.arange( 1, len(x)+1 ) / len(x)\n",
    "plt.plot( x, y, marker='.', linestyle='none' )\n",
    "x = np.sort( df['column 2'] )\n",
    "y = np.arange( 1, len(x)+1 ) / len(x)\n",
    "# and so on, if there were other columns to plot\n",
    "plt.plot( x, y, marker='.', linestyle='none' )\n",
    "# and so on if there are more data series\n",
    "plt.legend( ('explain x1', 'explain x2'),\n",
    "            loc='lower right')\n",
    "# then label axes and show plot as usual (not shown)\n",
    "```\n",
    "\n",
    "#### Chapter 2: Quantitative Exploratory Data Analysis\n",
    "\n",
    "The mean is the center of mass of the data:\n",
    "```python\n",
    "np.mean( df['column name'] )\n",
    "np.mean( series )\n",
    "```\n",
    "\n",
    "The median is the 50th percentile, or midpoint of the data:\n",
    "```python\n",
    "np.median( df['column name'] )\n",
    "np.median( series )\n",
    "```\n",
    "\n",
    "Or you can compute any percentile:\n",
    "```python\n",
    "quartiles = np.percentile(\n",
    "    df['column name'], [ 25, 50, 75 ] )\n",
    "iqr = quartiles[2] - quartiles[0]\n",
    "```\n",
    "\n",
    "Box plots show the quartiles, the IQR, and the outliers:\n",
    "```python\n",
    "sns.boxplot( x='A', y='B', data=df )\n",
    "# then label axes and show plot as above\n",
    "```\n",
    "\n",
    "Variance measures the spread of the data, the average squared distance from the mean.\n",
    "Standard deviation is its square root.\n",
    "```python\n",
    "np.var( df['column name'] )  # or any series\n",
    "np.std( df['column name'] )  # or any series\n",
    "```\n",
    "\n",
    "Covariance measures correlation between two data series.\n",
    "```python\n",
    "# get a covariance matrix on of these ways:\n",
    "M = np.cov( df['column 1'], df['column 2'] )\n",
    "M = np.cov( series1, series2 )\n",
    "# extract the value you care about, for example:\n",
    "covariance = M[0,1]\n",
    "```\n",
    "\n",
    "The Pearson correlation coefficient normalizes this to $[-1,1]$:\n",
    "```python\n",
    "# same as covariance, but using np.corrcoef instead:\n",
    "np.corrcoef( series1, series2 )\n",
    "```\n",
    "\n",
    "#### Chapter 3: Thinking probabalistically--Discrete variables\n",
    "\n",
    "Recall these random number generation basics:\n",
    "```python\n",
    "np.random.seed( my_int )\n",
    "np.random.random()        # uniform random in [0,1)\n",
    "np.random.randint(a,b)    # uniform random in a:b\n",
    "```\n",
    "\n",
    "Sampling many times from some distribution:\n",
    "```python\n",
    "# if the distribution is built into numpy:\n",
    "results = np.random.random( size=1000 )\n",
    "# if the distribution is not built into numpy:\n",
    "simulation_size = 1000    # or any number\n",
    "results = np.empty( simulation_size )\n",
    "for i in range( simulation_size ):\n",
    "    # generate a random number here, however you\n",
    "    # need to; here is a random example:\n",
    "    value = 1 - np.random.random() ** 2\n",
    "    # store it in the list of results:\n",
    "    results[i] = value\n",
    "```\n",
    "\n",
    "Bernoulli trials with probability $p$:\n",
    "```python\n",
    "success = np.random.random() < p    # one trial\n",
    "num_successes = np.random.binomial(\n",
    "    num_trials, p )                 # many trials\n",
    "# 1000 experiments, each containing 20 trials:\n",
    "results = np.random.binomial( 20, p, size=1000 )\n",
    "```\n",
    "\n",
    "Poisson distribution (size parameter optional):\n",
    "```python\n",
    "samples = np.random.poisson(\n",
    "    mean_arrival_rate, size=1000 )\n",
    "```\n",
    "\n",
    "#### Chapter 4: Thinking probabalistically--Continuous variables\n",
    "\n",
    "Normal (Gaussian) distribution (size parameter optional):\n",
    "```python\n",
    "samples = np.random.normal( mean, std, size=1000 )\n",
    "```\n",
    "\n",
    "Exponential distribution (time between events in a Poisson distribution,\n",
    "size parameter optional again):\n",
    "```python\n",
    "samples = np.random.exponential( mean_wait, size=10 )\n",
    "```\n",
    "\n",
    "You can take an array of numbers generated by simulation and plot it as an ECDF, as covered in the Graphical EDA chapter, earlier in this week."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Introduction to Data Visualization with Python](https://www.datacamp.com/courses/introduction-to-data-visualization-in-python)\n",
    "\n",
    "*NOTE: Only Chapters 1 and 3 are required here.*\n",
    "\n",
    "#### Chapter 1: Customizing Plots\n",
    "\n",
    "Break a plot into an $n\\times m$ grid of subplots as follows:\n",
    "\n",
    "(This is preferable to `plt.axes`, not covered here.)\n",
    "```python\n",
    "# create the grid and begin working on subplot #1:\n",
    "plt.subplot( n, m, 1 )\n",
    "plt.plot( x, y )          # this will create plot #1\n",
    "plt.title( '...' )        # title for plot #1\n",
    "plt.xlabel( '...' )       # ...and any other options\n",
    "# keep the same grid and now work on subplot #2:\n",
    "plt.subplot( n, m, 2 )\n",
    "# any plot commands here for plot 2,\n",
    "# continuing for any further subplots, ending with:\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "```\n",
    "\n",
    "Tweak the limits on the axes as follows:\n",
    "```python\n",
    "plt.xlim( [ min, max ] )  # set x axis limits\n",
    "plt.ylim( [ min, max ] )  # set y axis limits\n",
    "plt.axis( [ xmin, xmax, ymin, ymax ] ) # both\n",
    "```\n",
    "\n",
    "To add a legend to a plot:\n",
    "```python\n",
    "# when plotting series, give each a label,\n",
    "# which will identify it in the legend:\n",
    "plt.plot( x1, y1, label='first series' )\n",
    "plt.plot( x2, y2, label='second series' )\n",
    "plt.plot( x3, y3, label='third series' )\n",
    "# then add the legend:\n",
    "plt.legend( loc='upper right' )\n",
    "# then show the plot as usual\n",
    "```\n",
    "\n",
    "To annotate a figure:\n",
    "```python\n",
    "# add text at some point (here, (10,15)):\n",
    "plt.annotate( 'text', xy=(10,15) )\n",
    "# add text at (10,15) with an arrow to (5,15):\n",
    "plt.annotate( 'text', xytext=(10,15), xy=(5,15),\n",
    "              arrowprops={ 'color' : 'red' } )\n",
    "```\n",
    "\n",
    "Change plot styles globally:\n",
    "```python\n",
    "plt.style.available       # see list of styles\n",
    "plt.style.use( 'style' )  # choose one\n",
    "```\n",
    "\n",
    "#### Chapter 3: Statistical plots with Seaborn\n",
    "\n",
    "Plotting a linear regression line:\n",
    "```python\n",
    "import seaborn as sns\n",
    "sns.lmplot( x='col 1', y='col 2', data=df )\n",
    "```\n",
    "\n",
    "Plotting a linear regression line:\n",
    "```python\n",
    "import seaborn as sns\n",
    "sns.lmplot( x='col 1', y='col 2', data=df )\n",
    "plt.show()\n",
    "# and the corresponding residual plot:\n",
    "sns.residplot( x='col 1', y='col 2', data=df,\n",
    "               color='red' )  # color optional\n",
    "```\n",
    "\n",
    "Plotting a polynomial regression curve of order $n$:\n",
    "```python\n",
    "sns.regplot( x='col 1', y='col 2', data=df,\n",
    "             order=n )\n",
    "# this will include a scatter plot, but if you've\n",
    "# already done one, you can omit redoing it:\n",
    "sns.regplot( x='col 1', y='col 2', data=df,\n",
    "             order=n, scatter=None )\n",
    "```\n",
    "\n",
    "To do multiple regression plots for each value of a categorical variable in column X, distinguished by color:\n",
    "```python\n",
    "sns.lmplot( x='col 1', y='col 2', data=df,\n",
    "            hue='column X', palette='Set1' )\n",
    "# (many other options exist for palette)\n",
    "```\n",
    "\n",
    "Now separate plots into columns, rather than all on one plot:\n",
    "```python\n",
    "sns.lmplot( x='col 1', y='col 2', data=df,\n",
    "            row='column X' )\n",
    "sns.lmplot( x='col 1', y='col 2', data=df,\n",
    "            col='column X' )\n",
    "```\n",
    "\n",
    "Strip plots can visualize univariate distributions, especially useful when broken into categories:\n",
    "```python\n",
    "sns.stripplot( y='data column', x='category column',\n",
    "               data=df )\n",
    "# to add jitter to spread data out a bit in x:\n",
    "sns.stripplot( y='data column', x='category column',\n",
    "               data=df, size=4, jitter=True )\n",
    "```\n",
    "\n",
    "Swarm plots, covered earlier, are very similar, but can also have colors in them to distinguish categorical variables:\n",
    "```python\n",
    "sns.swarmplot( y='data column', x='category 1',\n",
    "               hue='category 2', data=df )\n",
    "# and you can also change the orientation:\n",
    "sns.swarmplot( y='category 1', x='data column',\n",
    "               hue='category 2', data=df,\n",
    "               orient='h' )\n",
    "```\n",
    "\n",
    "Violin plots make curves using kernel density estimation:\n",
    "```python\n",
    "sns.violinplot( y='data column', x='category 1',\n",
    "                hue='category 2', data=df )\n",
    "```\n",
    "\n",
    "Joint plots for visualizing a relationship between two variables:\n",
    "```python\n",
    "sns.jointplot( x='col 1', y='col 2', data=df )\n",
    "# and to add smoothing using KDE:\n",
    "sns.jointplot( x='col 1', y='col 2', data=df,\n",
    "               kind='kde' )\n",
    "# other kind options: reg, resid, hex\n",
    "```\n",
    "\n",
    "Scatter plots and histograms for all numerical columns in `df`:\n",
    "```python\n",
    "sns.pairplot( df )           # no grouping/coloring\n",
    "sns.pairplot( df, hue='A' )  # color by column A\n",
    "```\n",
    "\n",
    "Visualize a covariance matrix with a heatmap:\n",
    "```python\n",
    "M = np.cov( df[['col 1','col 2','col3']], # or more\n",
    "            rowvar=False )   # vars are in columns\n",
    "# (or you can use np.corrcoef to normalize np.cov)\n",
    "sns.heatmap( M )\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Before Day 6\n",
    "\n",
    "### [Merging DataFrames with pandas](https://learn.datacamp.com/courses/merging-dataframes-with-pandas)\n",
    "\n",
    "#### Chapter 1: Preparing data\n",
    "\n",
    "The `glob` module is useful:\n",
    "```python\n",
    "from glob import glob            # built-in module\n",
    "filenames = glob( '*.csv' )      # filename list\n",
    "data_frames = [ pd.read_csv(f)\n",
    "    for f in filenames ]         # import all files\n",
    "```\n",
    "\n",
    "You can reorder the rows in a DataFrame with `reindex`:\n",
    "```python\n",
    "# example: if an index of month or day names were\n",
    "# sorted alphabetically as strings\n",
    "# rather than chronologically:\n",
    "ordered_days = [ 'Mon', 'Tue', 'Wed', 'Thu',\n",
    "                 'Fri', 'Sat', 'Sun' ]\n",
    "df.reindex( ordered_days )\n",
    "# use this to make two dataframes with a common\n",
    "# index agree on their ordering:\n",
    "df1.reindex( df2.index )\n",
    "# in case the indices don't perfectly match,\n",
    "# NaN values will be inserted, which you can drop:\n",
    "df1.reindex( df2.index ).dropna()\n",
    "# or for missing rows, fill with earlier ones:\n",
    "df.reindex( some_series, method=\"ffill\" )\n",
    "# (there is also a bfill, for back-fill)\n",
    "```\n",
    "\n",
    "You can reorder a DataFrame in preparation for reindexing:\n",
    "```python\n",
    "# sort by index, ascending or descending:\n",
    "df = df.sort_index()\n",
    "df = df.sort_index( ascending=False )\n",
    "# sort by a column, ascending or descending:\n",
    "df = df.sort_values( 'column name',      # required\n",
    "                     ascending=False )   # optional\n",
    "```\n",
    "\n",
    "#### Chapter 2: Concatenating data\n",
    "\n",
    "To add one DataFrame onto the end of another:\n",
    "```python\n",
    "big_df = df1.append( df2 )  # top: df1, bottom: df2\n",
    "big_s = s1.append( s2 )     # works for Series, too\n",
    "# This also stacks indices, so you usually want to:\n",
    "big_df = big_df.reset_index( drop=True )\n",
    "```\n",
    "\n",
    "To add many DataFrames or series on top of one another:\n",
    "```python\n",
    "big_df = pd.concat( [ df1, df2, df3 ] )\n",
    "           .reset_index( drop=True )\n",
    "# equivalently:\n",
    "big_df = pd.concat( [ df1, df2, df3 ],\n",
    "                    ignore_index=True )\n",
    "# or add a hierarchical index to disambiguate:\n",
    "big_df = pd.concat( [ df1, df2, df3 ],\n",
    "                    keys=['key1','key2','key3'] )\n",
    "# equivalently:\n",
    "big_df = pd.concat( { key1 : df1,\n",
    "                      key2 : df2,\n",
    "                      key3 : df3 } )\n",
    "```\n",
    "\n",
    "If `df2` introduces new columns, and you want to form rows based on common indices, concat by columns:\n",
    "```python\n",
    "big_df = pd.concat( [ df1, df2 ], axis=1 )\n",
    "# equivalently:\n",
    "big_df = pd.concat( [ df1, df2 ], axis='columns' )\n",
    "# these accept keys=[...] also, or a dict to concat\n",
    "```\n",
    "\n",
    "By default, `concat` performs an \"outer join,\" that is, index sets are unioned.  To intersect them (\"inner join\") do this:\n",
    "```python\n",
    "big_df = pd.concat( [ df1, df2 ], axis=1,\n",
    "                    join='inner' )\n",
    "# equivalently:\n",
    "big_df = df1.join( df2, how='inner' )\n",
    "```\n",
    "\n",
    "### Chapter 3: Merging data\n",
    "\n",
    "Inner joins on non-index columns are done with `merge`.\n",
    "```python\n",
    "# default merges on all columns present\n",
    "# in both dataframes:\n",
    "merged = pd.merge( df1, df2 )\n",
    "# or you can choose your column:\n",
    "merged = pd.merge( df1, df2, on='colname' )\n",
    "# or multiple columns:\n",
    "merged = pd.merge( df1, df2, on=['col1','col2'] )\n",
    "# if the columns have different names in each df:\n",
    "merged = pd.merge( df1, df2,\n",
    "    left_on='col1', right_on='col2' )\n",
    "# to specify meaningful suffixes to replace the\n",
    "# default suffixes _x and _y:\n",
    "merged = pd.merge( df1, df2,\n",
    "    suffixes=['_from_2011','_from_2012'] )\n",
    "# you can also specify left, right, or outer joins:\n",
    "merged = pd.merge( df1, df2, how='outer' )\n",
    "```\n",
    "\n",
    "We often have to sort after merging (maybe by a date index), for which there is `merge_ordered`.  It most often goes with an outer join, so that's its default.\n",
    "```python\n",
    "# instead of this:\n",
    "merged = pd.merge( df1, df2, how='outer' )\n",
    "           .sorted_values( 'colname' )\n",
    "# do this, which is shorter and faster:\n",
    "merged = pd.merge_ordered( df1, df2 )\n",
    "# it accepts same keyword arguments as merge,\n",
    "# plus fill_method, like so:\n",
    "merged = pd.merge_ordered( df1, df2,\n",
    "           fill_method='ffill' )\n",
    "```\n",
    "\n",
    "When dates don't fully match, you can round dates in the right DataFrame up to the nearest date in the left DataFrame:\n",
    "```python\n",
    "merged = pd.merge_asof( df1, df2 )\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Before Day 8\n",
    "\n",
    "### [Streamlined Data Ingestion with pandas](https://learn.datacamp.com/courses/streamlined-data-ingestion-with-pandas)\n",
    "\n",
    "#### Chapter 1: Importing Data from Flat Files\n",
    "\n",
    "Any file whose rows are on separate lines and whose entries are separated by some delimiter can be read with the same `read_csv` function we've already seen.\n",
    "```python\n",
    "df = pd.read_csv( \"my_csv_file.csv\" )   # commas\n",
    "df = pd.read_csv( \"my_tabbed_file.tsv\",\n",
    "                  sep=\"\\t\" )            # tabs\n",
    "```\n",
    "\n",
    "If you only need some of the data, you can save space:\n",
    "```python\n",
    "# choose just some columns:\n",
    "df = pd.read_csv( \"my_csv_file.csv\", usecols=[\n",
    "    \"use\", \"only\", \"these\", \"columns\" ] )\n",
    "# can also give a list of column indices,\n",
    "# or a function that filters column names\n",
    "\n",
    "# choose just the first 100 rows:\n",
    "df1 = pd.read_csv( \"my_csv_file.csv\", nrows=100 )\n",
    "# choose just rows 1001 to 1100,\n",
    "# re-using the column header from df1:\n",
    "df2 = pd.read_csv( \"my_csv_file.csv\",\n",
    "                   nrows=100, skiprows=1000,\n",
    "                   header=None,       # skipped it\n",
    "                   names=list(df1) )  # re-use\n",
    "```\n",
    "\n",
    "If pandas is guessing a column's data type incorrectly, you can specify it manually:\n",
    "```python\n",
    "df = pd.read_csv( \"my_geographic_data.csv\",\n",
    "                  dtype={\"zipcode\":str,\n",
    "                         \"isemployed\":bool} )\n",
    "# to correctly handle bool types:\n",
    "df = pd.read_csv( \"my_geographic_data.csv\",\n",
    "                  dtype={\"zipcode\":str,\n",
    "                         \"isemployed\":bool},\n",
    "                  true_values=[\"Yes\"],\n",
    "                  no_values=[\"No\"] )\n",
    "# note: missing values get coded as True!\n",
    "# (pandas understands True, False, 0, and 1)\n",
    "```\n",
    "\n",
    "If some lines in a file are corrupt, you can ask `read_csv` to skip them and just warn you, importing everything else:\n",
    "```python\n",
    "df = pd.read_csv( \"maybe_corrupt_lines.csv\",\n",
    "                  error_bad_lines=False,\n",
    "                  warn_bad_lines=True )\n",
    "```\n",
    "\n",
    "#### Chapter 2: Importing Data from Excel Files\n",
    "\n",
    "If the spreadsheet is a single table of data without formatting:\n",
    "```python\n",
    "df = pd.read_excel( \"my_table.xlsx\" )\n",
    "# nrows, skiprows, usecols, work as before, plus:\n",
    "df = pd.read_excel( \"my_table.xlsx\",\n",
    "                    usecols=\"C:J,L\" )  # excel style\n",
    "```\n",
    "\n",
    "If a file contains multiple sheets, choose one by name or index:\n",
    "```python\n",
    "df = pd.read_excel( \"my_workbook.xlsx\",\n",
    "                    sheet_name=\"budget\" )\n",
    "df = pd.read_excel( \"my_workbook.xlsx\",\n",
    "                    sheet_name=3 )\n",
    "# (the default is the first sheet, index 0)\n",
    "```\n",
    "\n",
    "Or load all sheets into an ordered dictionary mapping sheet names to DataFrames:\n",
    "```python\n",
    "dfs = pd.read_excel( \"my_workbook.xlsx\",\n",
    "                     sheet_name=None )\n",
    "```\n",
    "\n",
    "Advanced methods of date/time parsing:\n",
    "```python\n",
    "# standard, as seen before:\n",
    "df = pd.read_excel( \"file.xlsx\",\n",
    "                    parse_dates=True )\n",
    "# just some cols, in standard date/time format:\n",
    "df = pd.read_excel( \"file.xlsx\",\n",
    "                    parse_dates=[\"col1\",\"col2\"] )\n",
    "# what if a date/time pair is split over 2 cols?\n",
    "df = pd.read_excel( \"file.xlsx\",\n",
    "                    parse_dates=[\n",
    "                        \"datetime1\",\n",
    "                        [\"date2\",\"time2\"]\n",
    "                    ] )\n",
    "# what if we want to control column names?\n",
    "df = pd.read_excel( \"file.xlsx\",\n",
    "                    parse_dates={\n",
    "                        \"name1\":\"datetime1\",\n",
    "                        \"name2\":[\"date2\",\"time2\"]\n",
    "                    } )\n",
    "# for nonstandard formats, do post-processing,\n",
    "# using a strftime format string, like this example:\n",
    "df[\"col\"] = pd.to_datetime( df[\"col\"],\n",
    "    format=\"%m%d%Y %H:%M:%S\" )\n",
    "```\n",
    "\n",
    "#### Chapter 3: Importing Data from Databases\n",
    "\n",
    "In SQLite, databases are `.db` files:\n",
    "```python\n",
    "# prepare to connect to the database:\n",
    "from sqlalchemy import create_engine\n",
    "engine = create_engine( \"sqlite:///filename.db\" )\n",
    "# fetch a table:\n",
    "df = pd.read_sql( \"table name\", engine )\n",
    "# or run any kind of SQL query:\n",
    "df = pd.read_sql( \"PUT QUERY CODE HERE\", engine )\n",
    "# if the query code is big:\n",
    "query = \"\"\"PUT YOUR SQL CODE\n",
    "           HERE ON AS MANY LINES\n",
    "           AS YOU LIKE;\"\"\"\n",
    "df = pd.read_sql( query, engine )\n",
    "# or get a list of tables:\n",
    "print( engine.table_names() )\n",
    "```\n",
    "\n",
    "#### Chapter 4: Importing JSON Data and Working with APIs\n",
    "\n",
    "From a file or string:\n",
    "```python\n",
    "# from a file:\n",
    "df = pd.read_json( \"filename.json\" )\n",
    "# from a string:\n",
    "df = pd.read_json( string_containing_json )\n",
    "# can specify dtype, as with read_csv:\n",
    "df = pd.read_json( \"filename.json\",\n",
    "                   dtype={\"zipcode\":str} )\n",
    "# also see pandas documentation for JSON \"orient\":\n",
    "# records, columns, index, values, or split\n",
    "```\n",
    "\n",
    "From the web with an API:\n",
    "```python\n",
    "import requests\n",
    "response = requests.get(\n",
    "    \"http://your.api.com/goes/here\",\n",
    "    headers = {\n",
    "        \"dictionary\" : \"with things like\",\n",
    "        \"username\" : \"or API key\"\n",
    "    },\n",
    "    params = {\n",
    "        \"dictionary\" : \"with options as\",\n",
    "        \"required by\" : \"the API docs\"\n",
    "    } )\n",
    "data = response.json()  # ignore metadata\n",
    "result = pd.DataFrame( data )\n",
    "# or possibly some part of the data, like:\n",
    "result = pd.DataFrame( data[\"some key\"] )\n",
    "# (you must inspect it to know)\n",
    "```\n",
    "\n",
    "If the JSON has nested objects, you can flatten:\n",
    "```python\n",
    "from pandas.io.json import json_normalize\n",
    "# instead of this line:\n",
    "result = pd.DataFrame( data[\"maybe a column\"] )\n",
    "# do this:\n",
    "result = json_normalize( data[\"maybe a column\"],\n",
    "                         sep=\"_\" )\n",
    "# (if there is deep nesting, see the record_path,\n",
    "# meta, and meta_prefix options)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Before Day 9\n",
    "\n",
    "### [Introduction to SQL](https://learn.datacamp.com/courses/introduction-to-sql)\n",
    "\n",
    "#### Chapter 1: Selecting columns\n",
    "\n",
    "SQL (\"sequel\") means Structured Query Language.  A SQL database contains tables, each of which is like a DataFrame.\n",
    "\n",
    "```sql\n",
    "-- A single-line SQL comment\n",
    "/*\n",
    "A multi-line\n",
    "SQL comment\n",
    "*/\n",
    "```\n",
    "\n",
    "To fetch one column from a table:\n",
    "```sql\n",
    "SELECT column_name FROM table_name;\n",
    "```\n",
    "\n",
    "To fetch multiple columns from a table:\n",
    "```sql\n",
    "SELECT column1, column2 FROM table_name;\n",
    "SELECT * FROM table_name;   -- all columns\n",
    "```\n",
    "\n",
    "To remove duplicates:\n",
    "```sql\n",
    "SELECT DISTINCT column_name\n",
    "FROM table_name;\n",
    "```\n",
    "\n",
    "To count rows:\n",
    "```sql\n",
    "SELECT COUNT(*)\n",
    "FROM table_name;    -- counts all the rows\n",
    "SELECT COUNT(column_name)\n",
    "FROM table_name;    -- counts the non-\n",
    "     -- missing values in just that column\n",
    "SELECT COUNT(DISTINCT column_name)\n",
    "FROM table_name;    -- # of unique entries\n",
    "```\n",
    "\n",
    "If a result is huge, you may want just the first few lines:\n",
    "```sql\n",
    "SELECT column FROM table_name\n",
    "LIMIT 10;           -- only return 10 rows\n",
    "```\n",
    "\n",
    "#### Chapter 2: Filtering rows\n",
    "\n",
    "(selecting a subset of the rows using the `WHERE` keyword)\n",
    "\n",
    "Using the comparison operators `<`, `>`, `=`, `<=`, `>=`, and `<>`, plus the inclusive range filter `BETWEEN`:\n",
    "```sql\n",
    "SELECT * FROM table_name\n",
    "WHERE quantity >= 100;  -- numeric filter\n",
    "SELECT * FROM table_name\n",
    "WHERE name = 'Jeff';    -- string filter\n",
    "```\n",
    "\n",
    "Using range and set filters:\n",
    "```sql\n",
    "SELECT title,release_year FROM films\n",
    "WHERE release_year BETWEEN 1990 AND 1999;\n",
    "                        -- range filter\n",
    "SELECT * FROM employees\n",
    "WHERE role IN ('Engineer','Sales');\n",
    "                        -- set filter\n",
    "```\n",
    "\n",
    "Finding rows where specific columns have missing values:\n",
    "```sql\n",
    "SELECT * FROM employees\n",
    "WHERE role IS NULL;\n",
    "```\n",
    "\n",
    "Combining filters with `AND`, `OR`, and parentheses:\n",
    "```sql\n",
    "SELECT * FROM table_name\n",
    "WHERE quantity >= 100\n",
    "  AND name = 'Jeff';    -- one combination\n",
    "SELECT title,release_year FROM films\n",
    "WHERE release_year >= 1990\n",
    "  AND release_year <= 1999\n",
    "  AND ( language = 'French'\n",
    "     OR language = 'Spanish' )\n",
    "  AND gross > 2000000;  -- many\n",
    "```\n",
    "\n",
    "Using wildcards (`%` and `_`) to filter strings with `LIKE`:\n",
    "```sql\n",
    "SELECT * FROM employees\n",
    "WHERE name LIKE 'Mac%'; -- e.g., MacEwan\n",
    "SELECT * FROM employees\n",
    "WHERE id NOT LIKE '%00';-- e.g., 352800\n",
    "SELECT * FROM employees\n",
    "WHERE name LIKE 'D_n';  -- e.g., Dan, Don\n",
    "```\n",
    "\n",
    "#### Chapter 3: Aggregate Functions\n",
    "\n",
    "We've seen this function before; it is an aggregator:\n",
    "```sql\n",
    "SELECT COUNT(*)\n",
    "FROM table_name;    -- counts all the rows\n",
    "```\n",
    "Some other aggregating functions:\n",
    "`SUM`, `AVG`, `MIN`, `MAX`.\n",
    "The resulting column name is the function name (e.g., `MAX`).\n",
    "\n",
    "To give a more descriptive name:\n",
    "```sql\n",
    "SELECT MIN(salary) AS lowest_salary,\n",
    "       MAX(salary) AS highest_salary\n",
    "FROM employees;\n",
    "```\n",
    "\n",
    "You can also do arithmetic on columns:\n",
    "```sql\n",
    "SELECT budget/1000 AS budget_in_thousands\n",
    "FROM projects;      -- convert a column\n",
    "SELECT hours_worked * hourly_pay\n",
    "FROM work_log WHERE date > '2019-09-01';\n",
    "                    -- create a column\n",
    "SELECT count(start_date)*100.0/count(*)\n",
    "FROM table_name;    -- percent not missing\n",
    "```\n",
    "\n",
    "#### Chapter 4: Sorting and grouping\n",
    "\n",
    "Sorting happens only after selecting:\n",
    "```sql\n",
    "SELECT * FROM employees\n",
    "ORDER BY name;      -- ascending order\n",
    "SELECT * FROM employees\n",
    "ORDER BY name DESC; -- descending order\n",
    "SELECT name,salary FROM employees\n",
    "ORDER BY role,name; -- multiple columns\n",
    "```\n",
    "\n",
    "Grouping happens after selecting but before sorting.  It is used when you want to apply an aggregate function like `COUNT` or `AVG` not across the whole result set, but to groups within it.\n",
    "```sql\n",
    "-- Compute average salary by role:\n",
    "SELECT role,AVG(salary) FROM employees\n",
    "GROUP BY role;\n",
    "-- How many people are in each division?\n",
    "-- (sorting results by division name)\n",
    "SELECT division,COUNT(*) FROM employees\n",
    "GROUP BY division\n",
    "ORDER BY division;\n",
    "```\n",
    "Every selected column except the one(s) you're aggregating must appear in your `GROUP BY`.\n",
    "\n",
    "To filter by a condition (like with `WHERE` but now applied to each group) use the `HAVING` keyword:\n",
    "```sql\n",
    "-- Same as above, but omit tiny divisions:\n",
    "SELECT division,COUNT(*) FROM employees\n",
    "GROUP BY division\n",
    "HAVING COUNT(*) >= 10\n",
    "ORDER BY division;\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Additional Useful References\n",
    "\n",
    "### [Python Data Science Toolbox, Part 2](https://www.datacamp.com/courses/python-data-science-toolbox-part-2)\n",
    "\n",
    "#### Chapter 1: Using iterators in PythonLand\n",
    "\n",
    "To convert an iterable to an iterator and use it:\n",
    "```python\n",
    "my_iterable = [ 'one', 'two', 'three' ]  # example\n",
    "my_iterator = iter( my_iterable )\n",
    "first_value = next( my_iterator )        # 'one'\n",
    "second_value = next( my_iterator )       # 'two'\n",
    "# and so on\n",
    "```\n",
    "\n",
    "To attach indices to the elements of an iterable:\n",
    "```python\n",
    "my_iterable = [ 'one', 'two', 'three' ]  # example\n",
    "with_indices = enumerate( my_iterable )\n",
    "my_iterator = iter( with_indices )\n",
    "first_value = next( my_iterator )        # (0,'one')\n",
    "second_value = next( my_iterator )       # (1,'two')\n",
    "# and so on; see also \"Looping Constructs\" earlier\n",
    "```\n",
    "\n",
    "To join iterables into tuples, use `zip`:\n",
    "```python\n",
    "iterable1 = range( 5 )\n",
    "iterable2 = 'five!'\n",
    "iterable3 = [ 'How', 'are', 'you', 'today', '?' ]\n",
    "all = zip( iterable1, iterable2, iterable3 )\n",
    "next( all )     # (0,'f','How')\n",
    "next( all )     # (1,'i','are')\n",
    "# and so on, or use this syntax:\n",
    "for x, y in zip( iterable1, iterable2 ):\n",
    "    do_something_with( x, y )\n",
    "```\n",
    "\n",
    "Think of `zip` as converting a list of rows into a list of columns, a \"matrix transpose,\" which is its own inverse:\n",
    "```python\n",
    "row1 = [ 1, 2, 3 ]\n",
    "row2 = [ 4, 5, 6 ]\n",
    "cols = zip( row1, row2 )     # swap rows and columns\n",
    "print( *cols )               # (1,4) (2,5) (3,6)\n",
    "cols = zip( row1, row2 )     # restart iterator\n",
    "undo1, undo2 = zip( *cols )  # swap rows/cols again\n",
    "print( undo1, undo2 )        # (1,2,3) (4,5,6)\n",
    "```\n",
    "\n",
    "Pandas can read CSV files into DataFrames in chunks, creating an iterable out of a file too large for memory:\n",
    "```python\n",
    "import pandas as pd\n",
    "for chunk in pd.read_csv( filename, chunksize=100 ):\n",
    "    process_one_chunk( chunk )\n",
    "```\n",
    "\n",
    "#### Chapter 2: List comprehensions and generators\n",
    "\n",
    "List comprehensions build a list from an output expression and a `for` clause:\n",
    "```python\n",
    "[ n**2 for n in range(3,6) ]      # == [9,16,25]\n",
    "```\n",
    "\n",
    "You can nest list comprehensions:\n",
    "```python\n",
    "[ (i,j) for i in range(3) for j in range(4) ]\n",
    "  # == [(0,0),(0,1),(0,2),(0,3),\n",
    "  #     (1,0),(1,1),(1,2),(1,3),\n",
    "  #     (2,0),(2,1),(2,2),(2,3)]\n",
    "```\n",
    "\n",
    "You can put conditions on the `for` clause:\n",
    "```python\n",
    "[ (i,j) for i in range(3) for j in range(3)\n",
    "        if i + j > 2 ]  # == [ (1,2), (2,1), (2,2) ]\n",
    "```\n",
    "\n",
    "You can put conditions in the output expression:\n",
    "```python\n",
    "some_data = [ 0.65, 9.12, -3.1, 2.8, -50.6 ]\n",
    "[ x if x >= 0 else 'NEG' for x in some_data ]\n",
    "  # == [ 0.65, 9.12, 'NEG', 2.8, 'NEG' ]\n",
    "```\n",
    "\n",
    "A dict comprehension creates a dictionary from an output expression in `key:value` form, plus a `for` clause:\n",
    "```python\n",
    "{ a: a.capitalize() for a in ['one','two','three'] }\n",
    "  # == { 'one':'One', 'two':'Two', 'three':'Three' }\n",
    "```\n",
    "\n",
    "Just like list comprehensions, but with parentheses:\n",
    "```python\n",
    "g = ( n**2 for n in range(3,6) )\n",
    "next( g )                         # == 9\n",
    "next( g )                         # == 16\n",
    "next( g )                         # == 25\n",
    "```\n",
    "\n",
    "You can build generators with functions and `yield`:\n",
    "```python\n",
    "def just_like_range ( a, b ):\n",
    "    counter = a\n",
    "    while counter < b:\n",
    "        yield counter\n",
    "        counter += 1\n",
    "list( just_like_range( 5, 9 ) )   # == [5,6,7,8]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Introduction to Data Visualization with Python](https://www.datacamp.com/courses/introduction-to-data-visualization-in-python)\n",
    "\n",
    "#### Chapter 2: Plotting 2D arrays\n",
    "\n",
    "To plot a bivariate function using colors:\n",
    "```python\n",
    "# choose the sampling points in both axes:\n",
    "u = np.linspace( xmin, xmax, num_xpoints )\n",
    "v = np.linspace( ymin, ymax, num_ypoints )\n",
    "# create pairs from these axes:\n",
    "x, y = np.meshgrid( u, v )\n",
    "# broadcast a function across those points:\n",
    "z = x**2 - y**2\n",
    "# plot it in color:\n",
    "plt.pcolor( x, y, z )\n",
    "plt.colorbar()       # optional but helpful\n",
    "plt.axis( 'tight' )  # remove whitespace\n",
    "plt.show()\n",
    "# optionally, the pcolor call can take a color\n",
    "# map parameter, one of a host of palettes, e.g.:\n",
    "plt.pcolor( x, y, z, cmap='autumn' )\n",
    "```\n",
    "\n",
    "To make a contour plot instead of a color map plot:\n",
    "```python\n",
    "# replace the pcolor line with this:\n",
    "plt.contour( x, y, z )\n",
    "plt.contour( x, y, z, 50 )  # choose num. contours\n",
    "plt.contourf( x, y, z )     # fill the contours\n",
    "```\n",
    "\n",
    "To make a bivariate histogram:\n",
    "```python\n",
    "# for rectangular bins:\n",
    "plt.hist2d( x, y, bins=(xbins,ybins) )\n",
    "plt.colorbar()\n",
    "# with optional x and y ranges:\n",
    "plt.hist2d( x, y, bins=(xbins,ybins),\n",
    "            range=((xmin,xmax),(ymin,ymax)) )\n",
    "# for hexagonal bins:\n",
    "plt.hexbin( x, y,\n",
    "            gridsize=(num_x_hexes,num_y_hexes) )\n",
    "# with optional x and y ranges:\n",
    "plt.hexbin( x, y,\n",
    "            gridsize=(num_x_hexes,num_y_hexes),\n",
    "            extent=(xmin,xmax,ymin,ymax) )\n",
    "```\n",
    "\n",
    "To display an image from a file:\n",
    "```python\n",
    "image = plt.imread( 'filename.png' )\n",
    "plt.imshow( image )\n",
    "plt.axis( 'off' )           # axes don't apply here\n",
    "plt.show()\n",
    "# to collapse a color image to grayscale:\n",
    "gray_img = image.mean( axis=2 )\n",
    "plt.imshow( gray_img, cmap='gray' )\n",
    "# to alter the aspect ratio:\n",
    "plt.imshow( gray_img, aspect=height/width )\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}