02 Using Jupyter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Jupyter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "[Jupyter](http://jupyter.org/) is an interactive computing environment that allows users to create heterogeneous documents called *notebooks* that can mix executable code, [markdown](https://en.wikipedia.org/wiki/Markdown) text with [MathJax](https://www.mathjax.org/), multimedia, static and interactive charts, and more. A notebook is typically a complete and self-contained record of a computation, and can be converted to various formats and shared with others. Jupyter thus supports a form of [literate programming](https://en.wikipedia.org/wiki/Literate_programming). Jupyter is an extremely popular tool for doing data science in Python due to its interactive nature, good support for iterative and experimental computation, and ability to create a finished artifact combining both scientific text (with math) and code. It's easiest to start to understand this by looking at an [example](http://nbviewer.jupyter.org/url/norvig.com/ipython/Economics.ipynb) of a finished notebook.\n",
    "\n",
    "Jupyter the application combines three components:\n",
    "<!-- TEASER_END -->\n",
    "\n",
    "- **The notebook application**: An front-end interactive application for writing and running code interactively and authoring notebook documents. There are multiple different front-ends available, running on the CLI, natively, or in a browser. The most commonly used front-end is [Jupyter Notebook](http://jupyter.org/) which runs in the browser, although it is due to be replaced by the newer [JupyterLab](https://github.com/jupyterlab/jupyterlab) which is currently in beta. As of this writing (April 2018) I recommend using JupyterLab on Mac and Linux but Jupyter Notebook on Windows, as there seem to be some issues still with getting JupyterLab running on Windows. \n",
    "- **Kernels**: These are separate processes started by the notebook application that runs the user’s code in a given language and returns output back to the notebook application. The kernel also handles things like computations for interactive widgets, tab completion and introspection. The original kernel was Python and the old version of Jupyter that was for Python only was called IPython; the name was changed to Jupyter to remove the language dependency as kernels now exist for a [number of different languages](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels), and IPython is used now to refer to the Python environment for Jupyter. In this overview we are only going to consider that environment.\n",
    "- **Notebook documents**: Self-contained documents (JSON text in files with .ipynb extension) that contain a representation of all content visible in the notebook application, including inputs and outputs of the computations, text, images, and more. A notebook can be *running* which means it is attached to a kernel. Each notebook document has its own kernel.\n",
    "\n",
    "Notebooks consist of a linear sequence of cells of various types:\n",
    "\n",
    "- **Code cells**: live code that is run in the kernel, and the output from running that code\n",
    "- **Markdown cells**: text cells formatted with mrkdown with embedded LaTeX (MathJax) equations\n",
    "- **Heading cells**: Up to 6 hierarchical levels of header text (I usually don't use these and just put my headings in markdown cells, but using heading cells can be useful for outline views of a document or table of contents)\n",
    "- **Raw cells**: Unformatted text that is included, without modification, when notebooks are converted to different formats using the conversion utility [nbconvert](https://nbconvert.readthedocs.io/en/latest/index.html)\n",
    "\n",
    "Jupyter documentation is available at https://jupyter-notebook.readthedocs.io/en/stable/. You can read more about the architecture of Jupyter [here](http://jupyter.readthedocs.io/en/latest/architecture/how_jupyter_ipython_work.html).\n",
    "\n",
    "## Installing and Running JupyterLab (Mac/Linux)\n",
    "\n",
    "On Mac and Linux, if you have the Anaconda distribution of Python, install JupyterLab with:\n",
    "\n",
    "    conda install -c conda-forge jupyterlab\n",
    "    \n",
    "or with pip:\n",
    "\n",
    "    pip install jupyterlab\n",
    "    \n",
    "and run with:\n",
    "\n",
    "    jupyter lab\n",
    "    \n",
    "Jupyter Lab will open in a web browser tab displaying a main menu along the top, a file browser on the left listing the notebooks and files in the current directory, and a tabbed interface on the right for open notebooks, terminals, etc.\n",
    "\n",
    "![https://jupyterlab.readthedocs.io/en/stable/_images/interface_jupyterlab.png](https://jupyterlab.readthedocs.io/en/stable/_images/interface_jupyterlab.png)\n",
    "\n",
    "\n",
    "## Running Jupyter Notebook (Windows)\n",
    "\n",
    "On Windows, use Jupyter Notebook, which is installed as part of the Anaconda distribution, and can be launched directly from Anaconda Navigator or from an Anaconda Python console with:\n",
    "\n",
    "    jupyter notebook\n",
    " \n",
    "This should open a web browser pointing at the notebook dashboard, which displays the notebooks and files in the current directory.\n",
    "\n",
    "![https://jupyter-notebook.readthedocs.io/en/stable/_images/dashboard_files_tab.png](https://jupyter-notebook.readthedocs.io/en/stable/_images/dashboard_files_tab.png)\n",
    "\n",
    "Jupyter Notebook uses separate browser tabs for each notebook and the filebrowser rather than integrating everything within one browser tab like JupyterLab; nonetheless is should be fairly straightforward to follow along).\n",
    "\n",
    "\n",
    "\n",
    "## Basic Usage\n",
    "\n",
    "The top of the notebook list displays clickable breadcrumbs of the current directory. By clicking on these breadcrumbs or on sub-directories in the notebook list, you can navigate your file system.\n",
    "\n",
    "To create a new notebook, use the menu option \"File - New - Notebook\" (JupyterLab), click on the “New” button at the top of the list and select a kernel from the dropdown menu. Which kernels are listed depend on what’s installed on the server.\n",
    "\n",
    "The file manager shows a green dot next to running notebooks (as seen below). Notebooks remain running until you explicitly shut them down; closing the notebook’s page is not sufficient.\n",
    "\n",
    "To view the running notebooks and shutdown a notebook, click the \"Running\" tab on the left side of the window, and then the \"Shutdown\" button next to the notebook in question.\n",
    "\n",
    "To delete, duplicate, or rename a notebook right-click on it in the file browser; a context menu will show with these options along with others.\n",
    "\n",
    "If you create a new notebook or open an existing one, you will be taken to the notebook user interface within a tab, which allows you to run code and author notebook documents interactively. The notebook UI has a toolbar at the top, followed by the notebook area below, which consists of one or more *cells*.\n",
    "\n",
    "The notebook UI is modal. If you click in a cell or press ENTER you enter \"edit mode\" and can type into the cell. If you click outside a cell or press ESC you will be in \"command mode\" which allows you to edit the notebook structure.\n",
    "\n",
    "You can execute a code cell or render a markdown cell by pressing Shift-ENTER. Focus will move to the next cell (a new cell will be created if you executed the last cell).\n",
    "\n",
    "If for some reason your notebook kernel hangs (e.g. waiting on some I/O that never happens, or due to a long running process), you can interrupt the execution by selecting \"Interrupt Kernel\" from the Kernel menu."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial\n",
    "\n",
    "These tutorials are for the old Jupyter Notebook UX, not Jupyter Lab, but are still useful as there is much in common.\n",
    "\n",
    "Basics: https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Notebook%20Basics.html\n",
    "\n",
    "http://jakevdp.github.io/blog/2017/03/03/reproducible-data-analysis-in-jupyter/\n",
    "\n",
    "## Markdown\n",
    "\n",
    "Jupyter supports GitHub-flavored markdown; see https://guides.github.com/pdfs/markdown-cheatsheet-online.pdf for a quick reference.\n",
    "\n",
    "\n",
    "## Re-ordering, Inserting, Deleting and Executing Cells\n",
    "\n",
    "![https://jupyter-notebook.readthedocs.io/en/stable/_images/menubar_toolbar.png](https://jupyter-notebook.readthedocs.io/en/stable/_images/menubar_toolbar.png)\n",
    "\n",
    "The notebook toolbar, shown above for Jupyter Notebook but similar in Jupyter Lab, has options to save the notebook, add (+), delete (cut) cells, copy cells, paste cells, move cells up or down in the notebook, run the cell, interrupt or restart the kernel, and change the cell type.\n",
    "\n",
    "To run a cell that has focus, use Shift-Enter. The output of the execution will be added to the notebook. Try it now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "1+2+3+4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More generally, you can use Python `print` statements to print info to the cell output, or you can put a Python expression (often just a variable name) at the end of the code cell and have that print automatically. It's possible to extend this latter functionality to multiple expressions by executing this code in a cell:\n",
    "\n",
    "```python\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "```\n",
    "\n",
    "or, if you want this to happen automatically every time you start Jupyter, you can edit or create the file `~/.ipython/profile_default/ipython_config.py` and add the contents:\n",
    "\n",
    "```python\n",
    "c = get_config()\n",
    "\n",
    "# Run all nodes interactively\n",
    "c.InteractiveShell.ast_node_interactivity = \"all\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "1+2+3+4;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of execution of the most recently executed cell is assigned to a special variable, '_':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can put a semicolon at the end of code in a cell to suppress the output. Try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "1+2+3+4;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the cells are marked with `In` and `Out`, and a count. The count allows you to keep track of the order in which cells were executed. `Out` is the result of execution, unless the execution of the cell resulted in some output to `stdout`/`stderr`; in this case that is shown instead with no `Out` label. Output to `stderr` is shown in red:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "print('hello', file=sys.stderr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`In` and `Out` are also variables that contain the history of execution (`In` is a list/array of strings, while `Out` is a dictionary/hash table):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(In)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(Out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Executing Shell Commands\n",
    "\n",
    "You can execute a shell command in a cell by starting it with '!'. For example, if your notebook relies on certain packages, you may want to start with a cell that uses shell commands to pip install the dependencies.\n",
    "\n",
    "Try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# On Mac or Linux\n",
    "!ls -l | head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# On Windows\n",
    "!dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's possible to assign this to a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# On Mac or Linux\n",
    "x = !(ls -l | head)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# On Windows\n",
    "x = !dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checkpoints and Saving\n",
    "\n",
    "The notebook is saved automatically periodically. Expplicitly saving the notebook from the toolbar or file menu actually creates a time-stamped \"checkpoint\", and you can revert to a saved checkpoint from the file menu. \n",
    "\n",
    "## Tips and Tricks\n",
    "\n",
    "\n",
    "You can write math with MathJax. For example, in a markdown cell, the text:\n",
    "\n",
    "    $$ P(A \\mid B) = \\frac{P(B \\mid A) \\, P(A)}{P(B)} $$\n",
    "\n",
    "    Pi is $\\pi$ okay?\n",
    "\n",
    "will render as shown below when the cell is \"executed\":"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ P(A \\mid B) = \\frac{P(B \\mid A) \\, P(A)}{P(B)} $$\n",
    "\n",
    "Pi is $\\pi$ okay?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Surrounding with single \\$ signs renders the math inline while double \\$ renders as a separate block.\n",
    "\n",
    "You can get help on a Python function (view its _docstring_) by following it with `?` in Jupyter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.path.exists?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That requires you to execute the code; you can do the same without executing the whole cell by typing Shift-TAB after the function name. You can also use the built-in Python function `help`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(os.path.exists)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This works on your own functions too so writing docstrings is always recommended.\n",
    "\n",
    "You can go a step further and use two ?? to get the source code of a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.concat??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter supports tab-completion with the TAB key (or on every keypress with the hinterland extension; see the section on extensions later to learn how to enable that).\n",
    "\n",
    "Try it below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from itertools import combinatio\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cell and Line Magics\n",
    "\n",
    "There are numerous special commands called Ipython Magics that can be used to control things in Jupyter. These are either *line magics* that start with `%` or *cell magics* that start with `%%`. A line magic consists of a single line, while a cell magic consists of everything from the `%%` to the end of the cell.\n",
    "\n",
    "`%load` can load code from external scripts. We will use that for hiding/loading the answers to some exercises.\n",
    "\n",
    "`%run` will let you run an external script or another notebook.\n",
    "\n",
    "`%%time` at the start of a cell will time the execution of the cell and print a summary when done.  `%%timeit` will run the code repeatedly (100,000 times by default) and then show the mean of the top 3 times.\n",
    "\n",
    "`%env` can be used to set environment variable values.\n",
    "\n",
    "`%%writefile` writes the contents of a cell to a file.\n",
    "\n",
    "`%pycat` shows the syntax-highlighted contents of the specified Python file in a pop-up window.\n",
    "\n",
    "`%%pdb` runs the contents of the cell under control of the Python debugger.\n",
    "\n",
    "You can use `%lsmagic` to see all available magics.\n",
    "\n",
    "More documentation on magics is available here: http://ipython.readthedocs.io/en/stable/interactive/magics.html\n",
    "\n",
    "A very common one for data science is `%matplotlib inline`; this is necessary if using the matplotlib or Seaborn plottting libraries to make sure the plots appear as cell outputs in the notebook. If you have a retina Mac you can use retina-resolution for plots by executing `%config InlineBackend.figure_format = 'retina'`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom Magics\n",
    "\n",
    "You can easily create your own magics; they are just Python functions. You just need to import the appropriate Python decorators and then annotate your function. We're getting ahead of ourselves but a quick example should illustrate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.core.magic import register_line_magic\n",
    "\n",
    "@register_line_magic\n",
    "def greet(line):\n",
    "    print(f'Hello {line}!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%greet Dave"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read more here: http://ipython.readthedocs.io/en/stable/config/custommagics.html\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Jupyter Display System\n",
    "\n",
    "Jupyter can display many different types of output from cells, not just text. This can be determined by the MIME type of the result, but you can use explicit control too with the [`IPython.display`](http://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html) module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Image\n",
    "\n",
    "display(Image('https://www.python.org/static/community_logos/python-logo.png'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "# a talk about IPython at Sage Days at U. Washington, Seattle.\n",
    "YouTubeVideo('1j_HxD4iLn8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extensions\n",
    "\n",
    "Jupyter is highly extensible and there are many extensions available. To start using extensions, install the main contributed ones. With conda, run:\n",
    "\n",
    "    conda install -c conda-forge jupyter_contrib_nbextensions\n",
    "    \n",
    "else with pip, run:\n",
    "\n",
    "    pip install jupyter_contrib_nbextensions\n",
    "    \n",
    "Restart Jupyter. In Jupyter Notebook, in the Edit menu you should see a new option `nbextensions config`. This will open up a new tab where you can easily enable or disable extensions with a checkbox. There doesn't yet seem to be an equivalent option in JupyterLab for setting these so for now just use Jupyter Notebook for changing which extensions you want enabled.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Going Further\n",
    "\n",
    "More good tips here: https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/\n",
    "\n",
    "RISE is an extension that allows you to create a slide deck in Jupyter and present it. Your deck can include live code that you execute, so it is great for Python programming talks :-). See https://github.com/damianavila/RISE for more details.\n",
    "\n",
    "Here's an example, which is an overview of Jupyter :-) http://quasiben.github.io/dfwmeetup_2014/#/\n",
    "\n",
    "For more advanced users, Jupyter can be extended and customized in multiple ways; you can read about them here: https://mindtrove.info/4-ways-to-extend-jupyter-notebook/\n",
    "\n",
    "Diffing notebooks for use with SCMs like git can be tricky as they are complex JSON files. A tool to help is [nbdime](http://nbdime.readthedocs.io/en/stable/) which can do smart diffs and 3-way merges.\n",
    "\n",
    "You can find a gallery of interesting notebooks at https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}