pandas_overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Back to Index](index.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A quick overview of working with Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Begin by importing all python modules we will need. For this exercise we will use Pandas.\n",
    "\n",
    "Pandas is a popular open source Python library for data analysis. It introduces two new data structures to Python - Series and DataFrame, both of which are built on top of NumPy (this means it's fast).\n",
    "- [Series](http://pandas.pydata.org/pandas-docs/version/0.15.2/dsintro.html#basicsseries) is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). \n",
    "- [DataFrame](http://pandas.pydata.org/pandas-docs/version/0.15.2/dsintro.html#basics-dataframe) is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects.\n",
    "\n",
    "Pandas is also tightly integrated with [matplotlib](http://pandas.pydata.org/pandas-docs/version/0.15.2/visualization.html) and you can do basic plotting directly from a dataframe. It also provides functionality for applying complex transformations and filters to the data and much more. There are lot of great Pandas tutorials on the web, here is one - [An Introduction to Pandas](http://synesthesiam.com/posts/an-introduction-to-pandas.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# A 'magic' command to display plots inline\n",
    "%matplotlib inline\n",
    "\n",
    "import requests\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Create a Pandas DataFrame\n",
    "\n",
    "We will create a new Pandas DataFrame from a timeseries. Let's get the tiemseries for Annual Average Maximum Temperature for one model (CNRM-CM5) and one scenario (RCP 4.5) for Sacramento County."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "url = 'http://api.cal-adapt.org/api/series/tasmax_year_CNRM-CM5_rcp45/rasters/?pagesize=100&stat=mean&ref=/api/counties/34/'\n",
    "\n",
    "# Make request\n",
    "response = requests.get(url)\n",
    "# Get json from response object\n",
    "data = response.json()\n",
    "\n",
    "# Create a new pandas dataframe. \n",
    "df = pd.DataFrame(data['results'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Explore the data in a DataFrame\n",
    "\n",
    "Uncomment each one and examine the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "len(df)\n",
    "#df.head()\n",
    "#df.tail()\n",
    "#df.columns\n",
    "#df.image\n",
    "#df['image']\n",
    "#df.event.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Apply functions to a DataFrame\n",
    "\n",
    "There are lots useful built-in methods we can use on a specific column, such as `mean()` to get the average. Most of pandas' methods will ignore missing values like NaN. You can also apply functions to all columns in a DataFrame. Try `std()`, `max()`, `min()`, `sum()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.image.mean()\n",
    "#df.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also create custom functions. The `image` values in our dataframe are in Kelvin and we want to change them to degrees Fahrenheit. Let's create a function that does this conversion and apply it to each row in the `image` column using `.apply()`. An intro to [lambda](http://www.diveintopython.net/power_of_introspection/lambda_functions.html) functions in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.image = df.image.apply(lambda x: (x - 273.15) * 9 / 5 + 32)\n",
    "# For simple calculations you dont need to use apply\n",
    "# df.image = (df.image - 273.15) * 9 / 5 + 32\n",
    "\n",
    "df.image.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Indexing\n",
    "\n",
    "Each row in a DataFrame has a unique identifier called the `index`. By default Pandas autogenerates an integer index for each row. But it can be useful to idenitfy each row by other indices. \n",
    "\n",
    "In our DateFrame each row represents max. temp for a year. The date (contained in the `event` field) is a unique identifier. We can tell the DataFrame to use the `event` field as the index field. This creates a timeseries and pandas provides some extra [functionality](https://tomaugspurger.github.io/modern-7-timeseries.html) for working with timeseries data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# First change format of `event` field to datetime\n",
    "df['event'] = pd.to_datetime(df['event'], format='%Y-%m-%d')\n",
    "# Set event field as index\n",
    "df = df.set_index(['event'])\n",
    "# You can index by multiple fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Filter by value\n",
    "hi_temps = df[df.image >= 80]\n",
    "hi_temps.head()\n",
    "\n",
    "# Filter by time. Uncomment the following lines and run the cell again\n",
    "#filtered_df = df['20200101':'20300101']\n",
    "#filtered_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Resampling\n",
    "\n",
    "Resampling is similar to a groupby: you split the time series into groups (10 year bins below), apply a function to each group (mean), and combine the result (one row per group). This creates another data structure commonly used in Pandas called `Series` - a one-dimensional labeled array capable of holding any data type. See the pandas documentation for more examples of [resampling](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.resample.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Only resample rows where year is between 2020 and 2099.\n",
    "# 10AS refers to Annual, Start of Year. Refer to pandas docs for other ways to specify bins\n",
    "decadal_avg = df['20100101':'20990101'].image.resample('10AS').mean()\n",
    "print(type(decadal_avg))\n",
    "decadal_avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The first row of the resampling result above is equivalent to:\n",
    "df['20100101':'20190101'].image.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decadal_stats = df['20100101':'20990101'].image.resample('10AS').agg(['mean', 'max', 'min', 'std'])\n",
    "print(type(decadal_stats))\n",
    "decadal_stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Plotting\n",
    "\n",
    "The plot method on DataFrame is just a simple wrapper around a matplotlib method `plt.plot()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.image.plot()\n",
    "#df.image.hist()\n",
    "#dfP['image'].plot.box()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.image.plot(figsize=(10, 8), color='#348ABD')\n",
    "plt.title(\"CNRM-CM5 RCP 4.5\")\n",
    "plt.ylabel(\"Temperature (degrees F)\")\n",
    "plt.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experiment with more examples [here]( http://pandas.pydata.org/pandas-docs/stable/visualization.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}