BNHM_biodiversityExample-2022.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to IB/ESPM 105 Jupyter Notebook for Natural History Museums"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to setup the notebook (_shift+return in cell to run_ or Press Run button in the menu)\n",
    "Do you see a number in the left margin of the cell below? If so, click on _Kernel->Restart and Clear Output_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install --no-cache-dir shapely\n",
    "!pip install -U folium\n",
    "\n",
    "%matplotlib inline\n",
    "import os\n",
    "import time\n",
    "import folium\n",
    "from datetime import datetime\n",
    "from shapely.geometry import Point, mapping\n",
    "from shapely.geometry.polygon import Polygon\n",
    "import matplotlib as mpl\n",
    "from matplotlib.collections import PatchCollection\n",
    "import matplotlib.pyplot as plot\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "from datascience import *\n",
    "from shapely import geometry as sg, wkt\n",
    "from scripts.espm_module import *\n",
    "import json\n",
    "import random\n",
    "from IPython.core.display import display, HTML\n",
    "import ipywidgets as widgets\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ESPM / IB 105 Natural History Museums and Data Science\n",
    "\n",
    "The goal of this notebook is to demonstrate how to access and integrate diverse data sets to visualize correlations and  patterns of species’ responses to environmental change. We will use programmatic tools (python) to show how to use Berkeley resources such as the biodiversity data from biocollections and online databases, field stations, climate models, and other environmental data. (Specifically we use data from the Berkeley Ecoengine, Cal-Adapt, and museum data via the GBIF API.)\n",
    "\n",
    "Before we begin analyzing and visualizing biodiversity data, this introductory notebook will help familiarize you with the basics of programming in Python.\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "0 - [Jupyter Notebooks](#jupyter)\n",
    "    \n",
    "1 - [Python Basics](#basics)\n",
    "\n",
    "2 - [GBIF API](#gbif)\n",
    "\n",
    "3 - [Comparing California Oak Species](#oak)\n",
    "\n",
    "4 - [Cal-Adapt API](#adapt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 0: Our Computing Environment, Jupyter notebooks  <a id='jupyter'></a>\n",
    "This webpage is called a Jupyter notebook. A notebook is a place to write programs and view their results. \n",
    "\n",
    "### Text cells\n",
    "In a notebook, each rectangle containing text or code is called a *cell*.\n",
    "\n",
    "Text cells (like this one) can be edited by double-clicking on them. They're written in a simple format called [Markdown](http://daringfireball.net/projects/markdown/syntax) to add formatting and section headings.  You don't need to learn Markdown, but you might want to.\n",
    "\n",
    "After you edit a text cell, click the \"run cell\" button at the top that looks like ▶| to confirm any changes. (Try not to delete the instructions of the lab.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code cells\n",
    "Other cells contain code in the Python 3 language. Running a code cell will execute all of the code it contains.\n",
    "\n",
    "To run the code in a code cell, first click on that cell to activate it.  It'll be highlighted with a little green or blue rectangle.  Next, either press ▶| or hold down the `shift` key and press `return` or `enter`.\n",
    "\n",
    "Try running this cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Hello, World!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\N{WAVING HAND SIGN}, \\N{EARTH GLOBE ASIA-AUSTRALIA}!\")\n",
    "print(\"\\N{grinning face}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Python basics <a id='basics'></a>\n",
    "Before getting into the more high level analyses we will do on the GBIF and Cal-Adapt data, we need to cover a few of the foundational elements of programming in Python.\n",
    "\n",
    "#### A. Expressions\n",
    "The departure point for all programming is the concept of the __expression__. An expression is a combination of variables, operators, and other Python elements that the language interprets and acts upon. Expressions act as a set of instructions to be fed through the interpreter, with the goal of generating specific outcomes. See below for some examples of basic expressions. Keep in mind that most of these just map to your understanding of mathematical expressions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'you' + ' and me'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will notice that only the last line in a cell gets printed out. If you want to see the values of previous expressions, you need to call the `print` function on that expression. Functions use parenthesis around their parameters, just like in math!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(2 + 2)\n",
    "\n",
    "print('you' + ' and I')\n",
    "\n",
    "print(12 ** 2)\n",
    "\n",
    "print(6 + 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### B. Variables\n",
    "In the example below, `a` and `b` are Python objects known as __variables__. We are giving an object (in this case, an `integer` and a `float`, two Python data types) a name that we can store for later use. To use that value, we can simply type the name that we stored the value as. Variables are stored within the notebook's environment, meaning stored variable values carry over from cell to cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 4\n",
    "b = 10/5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that when you create a variable, unlike what you previously saw with the expressions, it does not print anything out.\n",
    "\n",
    "We can continue to perform mathematical operations on these variables, which are now placeholders for what we've assigned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('a + b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### C. Lists\n",
    "The following few cells will introduce the concept of __lists__.\n",
    "\n",
    "A list is an ordered collection of objects. They allow us to store and access groups of variables and other objects for easy access and analysis. Check out this [documentation](https://www.tutorialspoint.com/python/python_lists.htm) for an in-depth look at the capabilities of lists.\n",
    "\n",
    "To initialize a list, you use brackets. Putting objects separated by commas in between the brackets will add them to the list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# an empty list\n",
    "lst = []\n",
    "print(lst)\n",
    "\n",
    "# reassigning our empty list to a new list\n",
    "lst = [1, 3, 6, 'lists', 'are' 'fun', 4]\n",
    "print(lst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access a value in the list, put the index of the item you wish to access in brackets following the variable that stores the list. Lists in Python are zero-indexed, so the indicies for `lst` are 0, 1, 2, 3, 4, 5, and 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Elements are selected like this:\n",
    "example = lst[2]\n",
    "\n",
    "# The above line selects the 3rd element of lst (list indices are 0-offset) and sets it to a variable named example.\n",
    "print(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### D. Dictonaries\n",
    "\n",
    "Dictionaries are `key`-`value` pairs. Just like a word dictinary, you have a key that will index a specific definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict = {'python': 'a large heavy-bodied nonvenomous constrictor snake occurring throughout the Old World tropics.'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get a `value` back out by indexing the `key`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict['python']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But like real dictionaries, there can be more than one definition. You can keep a `list`, or even another dictionary within a specific `key`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict = {'python': ['a large heavy-bodied nonvenomous constrictor snake occurring throughout the Old World tropics.',\n",
    "                      'a high-level general-purpose programming language.']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can index the `list` after the `key`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict['python'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict['python'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Part 2: GBIF API<a id='gbif'></a>\n",
    "\n",
    "The Global Biodiversity Information Facility has an API or web service, which we can use to retrieve data. We will demonstrate several different species queries to the [GBIF Web API](http://www.gbif.org/developer/summary).\n",
    "\n",
    "You can think of a Web API as a fancy URL for machines; what do you think the end of this URL means?\n",
    "\n",
    "http://api.gbif.org/v1/occurrence/search?year=1800,1899\n",
    "\n",
    "If you guessed that it limits the search to the years 1800-1899, you're right! Go ahead and click the URL above. You should see something like this:\n",
    "\n",
    "```\n",
    "{\"offset\":0,\"limit\":20,\"endOfRecords\":false,\"count\":5711947,\"results\":[{\"key\":14339704,\"datasetKey\":\"857aa892-f762-11e1-a439-00145eb45e9a\",\"publishingOrgKey\":\"6bcc0290-6e76-11db-bcd5-b8a03c50a862\",\"publishingCountry\":\"FR\",\"protocol\":\"BIOCASE\",\"lastCrawled\":\"2013-09-07T07:06:34.000+0000\",\"crawlId\":1,\"extensions\":{},\"basisOfRecord\":\"OBSERVATION\",\"taxonKey\":2809968,\"kingdomKey\":6,\"phylumKey\":7707728,\"classKey\":196,\"orderKey\":1169,\"familyKey\":7689,\"genusKey\":2849312,\"speciesKey\":2809968,\"scientificName\":\"Orchis militaris L.\",\"kingdom\":\"Plantae\",\"phylum\":\"Tracheophyta\",\"order\":\"Asparagales\",\"family\":\"Orchidaceae\",\"genus\":\"Orchis\",\"species\":\"Orchis \n",
    "```\n",
    "\n",
    "It might look like a mess, but it's not! This is actually very structured data, and can easily be put into a table like format, though often programmers don't do this because it's just as easy to keep it as is. If you know the structure (thanks to the DarwinCore metadata standards), you can found what you're interested in. Check out the api terms here: https://www.gbif.org/developer/occurrence#search\n",
    "\n",
    "You might be able to pick out the curly braces `{` and think this it's a dictionary. You'd be right, except in this format we call it [JSON](https://en.wikipedia.org/wiki/JSON).\n",
    "\n",
    "---\n",
    "\n",
    "## *Aneides flavipunctatus*, Black Salamander\n",
    "\n",
    "![Aneides flavipunctatus by Freiermuth](https://calphotos.berkeley.edu/imgs/512x768/0000_0000/0512/0340.jpeg)\n",
    "\n",
    "When performing data analysis, it is always important to define a question that you seek the answer to. *The goal of finding the answer to this question will ultimately drive the queries and analysis styles you choose to use/write.*\n",
    "\n",
    "For this example, we are going to ask: **where have [*Aneides flavipunctatus*](https://amphibiaweb.org/cgi/amphib_query?where-genus=Aneides&where-species=flavipunctatus) (the Black Salamander) been documented? Are there records at any of our field stations?**\n",
    "\n",
    "The code to ask the API has already been written for us! This is often the case with programming, someone has already written the code, so we don't have to. We'll just set up the `GBIFRequest` object and assign that to the variable `req`, short for \"request\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "req = GBIFRequest()  # creating a request to the API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, so how do we make searches? We can use a Python `dictionary` to create our query parameters. We'll ask for the `scientificName` of the Black Salamander (*Aneides flavipunctatus*):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'scientificName': 'Aneides flavipunctatus'}  # setting our parameters (the specific species we want)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the parameters, we can feed this to our `req` variable to get back all the pages of data. We'll then make sure that each record has a `decimalLatitude`, otherwise we'll thow it out for now. Lastly, we'll print out the first five records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pages = req.get_pages(params)  # using those parameters to complete the request\n",
    "records = [rec for page in pages for rec in page['results'] if rec.get('decimalLatitude')]  # sift out unmapped records\n",
    "records[:5]  # print first 5 records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataFrames\n",
    "\n",
    "JSON is great, but it might be conceptually easier to make this a table. We'll use the popular [`pandas`](http://pandas.pydata.org/) Python library. In `pandas`, a DataFrame is a table that has several convenient features. For example, we can access the columns of the table like we would `dict`ionaries, and we can also treat the columns and rows themselves as Python `list`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df = pd.DataFrame(records)   # converts the JSON above to a dataframe\n",
    "records_df.head()  # prints the first five rows of the dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since each column (or row) above can be thought of as a `list`, that means we can use list functions to interact with them! One such function is the `len` function to get the number of elements in a `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(records_df.columns), len(records_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many columns are there? (hint: first value) How many records/rows? (hint: it's the other value)! That's a lot of information. What variables do we have in the columns?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use two methods from `pandas` to do a lot more. The `value_counts()` method will tabulate the frequency of the row value in a column, and the `plot.barh()` will plot us a horizontal bar chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['country'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "records_df['country'].value_counts().plot.barh()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['county'].value_counts().plot.barh(figsize=(18,10));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['basisOfRecord'].value_counts().plot.barh()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['institutionCode'].value_counts().plot.barh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `groupby()` method allows us to count based one column based on another, and then color the bar chart differently depending on a variable of our choice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df.groupby([\"institutionCode\", \"basisOfRecord\"])['basisOfRecord'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df.groupby([\"institutionCode\", \"basisOfRecord\"])['basisOfRecord'].count().unstack().plot.barh(stacked=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can use `plot.hist()` to make a histogram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['elevation'].plot.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "Question: What does the above plot tell us about the elevation range of this species? In other words is this likely a low elevation species or montane? Type your response below in the blank cell.\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#When were these records collected?\n",
    "records_df['year'].plot.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Mapping\n",
    "\n",
    "Now we can map our latitude and longitude points. We'll want to color code by `institutionCode` so we can see which collection made the observation or has the specimen. We'll use a function that does this automatically, and it will randomly generate a color-- so if you don't like the colors you see, try running the cell again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "color_dict, html_key = assign_colors(records_df, \"institutionCode\")\n",
    "display(HTML(html_key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Did you get an error in the last cell? If so you may have a corrupt or old variable so please run this cell\n",
    "#to re-run your original search\n",
    "#If you did not get an error, skip this cell and keep going!\n",
    "\n",
    "my_req = GBIFRequest()  # creating a request to the API\n",
    "my_params = {'scientificName': 'Aneides flavipunctatus'}  # setting our parameters (the specific species we want)\n",
    "my_pages = my_req.get_pages(my_params)  # using those parameters to complete the request\n",
    "my_records = [rec for page in my_pages for rec in page['results'] if rec.get('decimalLatitude')]  # sift out valid records\n",
    "records_df = pd.DataFrame(my_records) \n",
    "\n",
    "color_dict, html_key = assign_colors(records_df,'institutionCode');\n",
    "display(HTML(html_key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can map the points!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "mapa = folium.Map(location=[37.359276, -122.179626], zoom_start=5) # Folium is a useful library for generating\n",
    "                                                                   # Google maps-like map visualizations.\n",
    "for r in records_df.iterrows():\n",
    "    lat = r[1]['decimalLatitude']\n",
    "    long = r[1]['decimalLongitude']\n",
    "    folium.CircleMarker((lat, long), color=color_dict[r[1]['institutionCode']], popup=r[1]['basisOfRecord']).add_to(mapa)\n",
    "mapa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "To get the boundries for all the reserves, we will need to send a request to get GeoJSON, which is a format for encoding a variety of geographic data structures. With this code, we can request GeoJSON for all reserves and plot ocurrences of the species. First we'll assign the API URL that has the data to a new variable `url`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://ecoengine.berkeley.edu/api/layers/reserves/features/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we make the requests just like we did earlier through the GBIF:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reserves = requests.get(url, params={'page_size': 30}).json()\n",
    "reserves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look closely, this is just bounding boxes for the latitude and longitude of the reserves.\n",
    "\n",
    "There are some reserves that the EcoEngine didn't catch, we'll add the information for \"Blodgett\", \"Hopland\", and \"Sagehen\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "station_urls = {\n",
    "    'Blodgett': 'https://raw.githubusercontent.com/BNHM/spatial-layers/master/wkt/BlodgettForestResearchStation.wkt',\n",
    "    'Hopland': 'https://raw.githubusercontent.com/BNHM/spatial-layers/master/wkt/HoplandResearchAndExtensionCenter.wkt',\n",
    "    'Sagehen': 'https://raw.githubusercontent.com/BNHM/spatial-layers/master/wkt/SagehenCreekFieldStation.wkt'\n",
    "}\n",
    "reserves['features'] += [{'type': \"Feature\", 'properties': {\"name\": name}, 'geometry': mapping(wkt.loads(requests.get(url).text))}\n",
    "                for name, url in station_urls.items()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see all the station names by indexing the `name` value for the reserves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[r['properties']['name'] for r in reserves['features']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can send this `geojson` directly to our mapping library `folium`. You'll have to zoom in, but you should see blue outlines areas, there are the reserves!:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "mapb = folium.Map(location=[37.359276, -122.179626], zoom_start=5) \n",
    "\n",
    "wkt = folium.features.GeoJson(reserves)\n",
    "mapb.add_child(wkt)\n",
    "for r in records_df.iterrows():\n",
    "    lat = r[1]['decimalLatitude']\n",
    "    long = r[1]['decimalLongitude']\n",
    "    folium.CircleMarker((lat, long), color=color_dict[r[1]['institutionCode']], popup=r[1]['basisOfRecord']).add_to(mapb)\n",
    "mapb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "We can also find out which stations have how many our species. First we'll have to add a column to our `DataFrame` that makes points out of the latitude and longitude coordinates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_point(row):\n",
    "    return Point(row['decimalLongitude'], row['decimalLatitude'])\n",
    "\n",
    "records_df[\"point\"] = records_df.apply(lambda row: make_point (row),axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can write a little function to check whether that point is in one of the stations, and if it is, we'll add that station in a new column called `station`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def in_station(reserves, row):\n",
    "    \n",
    "    reserve_polygons = []\n",
    "\n",
    "    for r in reserves['features']:\n",
    "        name = r['properties']['name']\n",
    "        poly = sg.shape(r['geometry'])\n",
    "        reserve_polygons.append({\"id\": name,\n",
    "                                 \"geometry\": poly})\n",
    "    \n",
    "    sid = False\n",
    "    for r in reserve_polygons:\n",
    "        if r['geometry'].contains(row['point']):\n",
    "            sid = r['id']\n",
    "            sid = r['id']\n",
    "    if sid:\n",
    "        return sid\n",
    "    else:\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now apply this function to the `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df[\"station\"] = records_df.apply(lambda row: in_station(reserves, row),axis=1)\n",
    "in_stations_df = records_df[records_df[\"station\"] != False]\n",
    "len(in_stations_df.columns), len(in_stations_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "Question: How many species records occur at one of the field stations or reserves? (write your answer in the blank cell below)\n",
    "    </div>"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if this corresponds to what we observed on the map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_stations_df.groupby([\"species\", \"station\"])['station'].count().unstack().plot.barh(stacked=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "**Exercise**\n",
    "Edit the code below to search for a different species (replace the scientificName value of 'Melanerpes formicivorus') with another you're interested in, then adapt some of the graphing cells above to explore your data! Use the examples from previous cells to copy and paste the code. Try the California Newt \"Taricha torosa\"!\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "my_req = GBIFRequest()  # creating a request to the API\n",
    "my_params = {'scientificName': 'Melanerpes formicivorus', 'basisOfRecord':'Preserved_Specimen'}  # search parameters (requesting a species and only Preserved specimens to filter out all the observations)\n",
    "my_pages = my_req.get_pages(my_params)  # using those parameters to complete the request\n",
    "my_records = [rec for page in my_pages for rec in page['results'] if rec.get('decimalLatitude')]  # sift out valid records\n",
    "my_records_df = pd.DataFrame(my_records)  # make a dataframe\n",
    "my_records_df.tail()  # print last 5 rows\n",
    "\n",
    "len(my_records_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "color_dict, html_key = assign_colors(my_records_df, \"institutionCode\")\n",
    "display(HTML(html_key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapc = folium.Map([37.359276, -122.179626], zoom_start=5)\n",
    "\n",
    "points = folium.features.GeoJson(reserves)\n",
    "mapc.add_child(points)\n",
    "for r in my_records_df.iterrows():\n",
    "    lat = r[1]['decimalLatitude']\n",
    "    long = r[1]['decimalLongitude']\n",
    "    folium.CircleMarker((lat, long), color=color_dict[r[1]['institutionCode']]).add_to(mapc)\n",
    "mapc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see if your search results occur on a field station, run the following cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_point(row):\n",
    "    return Point(row['decimalLongitude'], row['decimalLatitude'])\n",
    "\n",
    "my_records_df[\"point\"] = my_records_df.apply(lambda row: make_point (row),axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_records_df[\"station\"] = my_records_df.apply(lambda row: in_station(reserves, row),axis=1)\n",
    "in_stations_df = my_records_df[my_records_df[\"station\"] != False]\n",
    "len(in_stations_df.columns), len(in_stations_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many records occur in field reserves according to the number of records above?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_stations_df.groupby([\"species\", \"station\"])['station'].count().unstack().plot.barh(stacked=True)\n",
    "in_stations_df.head() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "**Exercise**: Edit the code to search for other species, map the records and see if you can find specimens from field reserves. See if you can figure which code snippets you would use to see which field reserves have these records. Which code snippets would you use to plot other aspects of these records?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Part 3: Comparing California Oak species:<a id='oak'></a>\n",
    "\n",
    "\n",
    "| ![quercus douglassi](https://upload.wikimedia.org/wikipedia/commons/thumb/6/6d/Large_Blue_Oak.jpg/220px-Large_Blue_Oak.jpg)  | ![quercus lobata](https://upload.wikimedia.org/wikipedia/commons/thumb/8/86/Valley_Oak_Mount_Diablo.jpg/220px-Valley_Oak_Mount_Diablo.jpg) | ![quercus durata](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b8/Quercusduratadurata.jpg/220px-Quercusduratadurata.jpg) | ![quercus agrifolia](https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/Quercus_agrifolia_foliage.jpg/220px-Quercus_agrifolia_foliage.jpg) |\n",
    "|:---:|:---:|:---:|:---:|\n",
    "| *Quercus douglassi* | *Quercus lobata* | *Quercus durata* | *Quercus agrifolia*|\n",
    "\n",
    "\n",
    "California has high species diversity in its native oaks. Let's search for these four species of oak using our `GBIF` API function and collect the observations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "species_records = []\n",
    "\n",
    "species = [\"Quercus douglasii\", \"Quercus lobata\", \"Quercus durata\", \"Quercus agrifolia\"]\n",
    "\n",
    "for s in species:\n",
    "    req = GBIFRequest()  # creating a request to the API\n",
    "    params = {'scientificName': s}  # setting our parameters (the specific species we want)\n",
    "    pages = req.get_pages(params)  # using those parameters to complete the request\n",
    "    records = [rec for page in pages for rec in page['results'] if rec.get('decimalLatitude')]  # sift out valid records\n",
    "    species_records.extend(records)\n",
    "    time.sleep(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert this JSON to a `DataFrame` again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df = pd.DataFrame(species_records)\n",
    "records_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many records do we have now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(records_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how they're distributed among our different species queries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "records_df['scientificName'].value_counts().plot.barh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can group this again by `collectionCode`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df.groupby([\"scientificName\", \"collectionCode\"])['collectionCode'].count().unstack().plot.barh(stacked=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also map these like we did with the first species search above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "color_dict, html_key = assign_colors(records_df, \"species\")\n",
    "display(HTML(html_key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapc = folium.Map([37.359276, -122.179626], zoom_start=5)\n",
    "\n",
    "points = folium.features.GeoJson(reserves)\n",
    "mapc.add_child(points)\n",
    "for r in records_df.iterrows():\n",
    "    lat = r[1]['decimalLatitude']\n",
    "    long = r[1]['decimalLongitude']\n",
    "    folium.CircleMarker((lat, long), color=color_dict[r[1]['species']]).add_to(mapc)\n",
    "mapc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the same code we wrote earlier to see which reserves have which species of oak:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df[\"point\"] = records_df.apply(lambda row: make_point (row),axis=1)\n",
    "records_df[\"station\"] = records_df.apply(lambda row: in_station(reserves, row),axis=1)\n",
    "in_stations_df = records_df[records_df[\"station\"] != False]\n",
    "in_stations_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can make a bar graph like we did before, grouping by `species` and stacking the bar based on `station`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_stations_df.groupby([\"species\", \"station\"])['station'].count().unstack().plot.barh(stacked=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "Question: Searching and mapping multiple species can be useful. Which species would you expect to find at more than one field station? Besides multiple species of *Quercus* which co-occur, what other combination of species might be useful to search and visualize their data in this way? Write your response in the cell below.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ---\n",
    "\n",
    "# Part 4: Cal-Adapt API<a id='adapt'></a>\n",
    "<a target=_blank href=\"https://calphotos.berkeley.edu/cgi/img_query?enlarge=0000+0000+0715+1756\"> ![California Damsels dragonfly](https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0715/1756.jpeg)</a>Let's work with California Damsels dragonfly (*Argia agrioides*) data from the GBIF API and climate data from Cal-Adapt. (image from CalPhotos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "req = GBIFRequest()  # creating a request to the API\n",
    "params = {'scientificName': 'Argia agrioides'}  # setting our parameters (the specific species we want)\n",
    "pages = req.get_pages(params)  # using those parameters to complete the request\n",
    "records = [rec for page in pages for rec in page['results'] if rec.get('decimalLatitude')]  # sift out valid records\n",
    "records[:5]  # print first 5 records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll make a `DataFrame` again for later use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df = pd.DataFrame(records)\n",
    "records_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will use the [Cal-Adapt](http://api.cal-adapt.org/api/) web API to work with time series raster data. It will request an entire time series for any geometry and return a Pandas `DataFrame` object for each record in all of our *Argia agrioides* records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "req = CalAdaptRequest()\n",
    "records_g = [dict(rec, geometry=sg.Point(rec['decimalLongitude'], rec['decimalLatitude']))\n",
    "             for rec in records]\n",
    "ca_df = req.concat_features(records_g, 'gbifID')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the first five rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ca_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(ca_df.columns), len(ca_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks like the time series data we want for each record (the unique ID numbers as the columns). Each record has the projected temperature in Fahrenheit for 273 years (every row!). We can plot predictions for few random records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a line plot using the first 9 columns of dataframe\n",
    "ca_df.iloc[:,:9].plot()\n",
    "\n",
    "# Use matplotlib to title your plot.\n",
    "plot.title('Argia agrioides - %s' % req.slug)\n",
    "\n",
    "# Use matplotlib to add labels to the x and y axes of your plot.\n",
    "plot.xlabel('Year', fontsize=18)\n",
    "plot.ylabel('Degrees (Fahrenheit)', fontsize=16)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like temperature is increasing across the board wherever these observations are occuring. We can calculate the average temperature for each year across observations in California:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmax_means = ca_df.mean(axis=1)\n",
    "tmax_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's happening to the average temperature that *Argia agrioides* is going to experience in the coming years across California?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmax_means.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is there a temperature at which the *Argia agrioides* cannot survive? Is there one in which they particularly thrive?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "What if we look specifically at the field stations and reserves? We can grab our same code that checked whether a record was within a station, and then map those `gbifID`s back to this temperature dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df[\"point\"] = records_df.apply(lambda row: make_point (row),axis=1)\n",
    "records_df[\"station\"] = records_df.apply(lambda row: in_station(reserves, row),axis=1)\n",
    "in_stations_df = records_df[records_df[\"station\"] != False]\n",
    "in_stations_df[['gbifID', 'station']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall the column headers of our `ca_df` are the `gbifID`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ca_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we subset the temperature dataset for only the observations that occurr within the bounds of a reserve or field station:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "station_obs = [str(id) for id in list(in_stations_df['gbifID'])]\n",
    "ca_df[station_obs]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's graph these observations from Santa Cruz Island against the average temperature across California where this species was observed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot.plot(tmax_means)\n",
    "plot.plot(ca_df[station_obs])\n",
    "\n",
    "# Use matplotlib to title your plot.\n",
    "plot.title('Argia agrioides and temperatures in Santa Cruz Island')\n",
    "\n",
    "# Use matplotlib to add labels to the x and y axes of your plot.\n",
    "plot.xlabel('Year', fontsize=18)\n",
    "plot.ylabel('Degrees (Fahrenheit)', fontsize=16)\n",
    "plot.legend([\"CA Average\", \"Santa Cruz Island\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does this tell you about Santa Cruz Island? As time goes on and the temperature increases, might Santa Cruz Island serve as a refuge for *Argia agrioides*?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To save a copy of your Jupyter Notebook with its output, use File-->Download as--> PDF as HTML (.PDF)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}