hdf5-is-for-lovers.ipynb

{
 "metadata": {
  "celltoolbar": "Slideshow",
  "name": "",
  "signature": "sha256:bdc79b5a6edfb675f279a9bb176e15082242a060b77906e32910178d510351cb"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "<div style=\"text-align:center;\">\n",
      "<h1>SciPy 2014 - Tutorials</h1>\n",
      "<h2>HDF5 is for Lovers</h2>\n",
      "</div>\n",
      "\n",
      "\n",
      "<div style=\"text-align:center;\">\n",
      "July 6th, 2014, SciPy 2014\n",
      "<br/>\n",
      "Anthony Scopatz\n",
      "<br/>\n",
      "Engineering Physics Dept., The University ot Wisconsin-Madison\n",
      "<br/>\n",
      "scopatz@gmail.com\n",
      "</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "skip"
      }
     },
     "source": [
      "## Cheat Sheet\n",
      "\n",
      "**Center:**  ``<div style=\"text-align:center;\"></div>``\n",
      "    \n",
      "**Right:**   ``<div style=\"text-align:right;\"></div>``\n",
      "    \n",
      "**Center H3:**  ``<div style=\"text-align:center;\"><h3></h3></div>``\n",
      "\n",
      "**Right H3:**  ``<div style=\"text-align:right;\"><h3></h3></div>``\n",
      "\n",
      "**Grey & Small:** ``<div style=\"color:grey;font-size:0.75em;\"></div>``\n",
      "    \n",
      "**Center Image:**  ``<div style=\"text-align:center;\"><img src=\"\" width=\"80%\" /></div>``"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!rm db.h5 temp.h5"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "skip"
      }
     },
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "What is HDF5?\n",
      "==============================\n",
      "HDF5 stands for (**H**)eirarchical (**D**)ata (**F**)ormat (**5**)ive."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "It is supported by the lovely people at HDFGroup."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "At its core HDF5 is binary file type specification."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "However, what makes HDF5 great is the numerous libraries written to interact \n",
      "with files of this type and their *extremely rich* feature set."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "<div style=\"text-align:center;\">\n",
      "**Which you will learn today!**\n",
      "</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "A Note on the Format\n",
      "=================================\n",
      "Intermixed, there will be:\n",
      "\n",
      "* Slides\n",
      "* Interactive Hacking\n",
      "* Exercises"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Feel free to:\n",
      "\n",
      "* Ask questions at anytime \n",
      "* Explore at your own pace."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Class Makeup\n",
      "==============================\n",
      "By a show of hands, how many people have used:\n",
      "\n",
      "* HDF5 before?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* PyTables?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* h5py?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* the HDF5 C API?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* SQL?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Other binary data formats? "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Setup\n",
      "==============================\n",
      "\n",
      "Please clone the repo:\n",
      "\n",
      "```\n",
      "$ git clone git://github.com/scopatz/hdf5-is-for-lovers.git\n",
      "```\n",
      "\n",
      "Or download a tarball from https://github.com/scopatz/hdf5-is-for-lovers"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Warm up exercise\n",
      "===============================\n",
      "In IPython:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "import tables as tb\n",
      "\n",
      "f = tb.open_file('temp.h5', 'a')\n",
      "heart = np.ones(42, dtype=[('rate', int), ('beat', float)])\n",
      "f.create_table('/', 'heart', heart)\n",
      "f.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "-"
      }
     },
     "source": [
      "Or run ``python exer/warmup.py``"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Warm up exercise\n",
      "===============================\n",
      "You should see in ViTables:\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/warmup.png\" width=\"70%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "A Brief Introduction\n",
      "===========================\n",
      "For persisting structured numerical data, binary formats are superior\n",
      "to plaintext."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "For one thing, they are often smaller:\n",
      "\n",
      "```\n",
      "# small ints       # med ints \n",
      "42   (4 bytes)     123456   (4 bytes)\n",
      "'42' (2 bytes)     '123456' (6 bytes)\n",
      "\n",
      "# near-int floats  # e-notation floats\n",
      "12.34   (8 bytes)  42.424242E+42   (8 bytes)\n",
      "'12.34' (5 bytes)  '42.424242E+42' (13 bytes)\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "A Brief Introduction\n",
      "===========================\n",
      "For another, binary formats are often faster for I/O because ``atoi()`` and ``atof()``\n",
      "are expensive."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "However, you often want some thing more than a binary chunk of data in a file."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "**NOTE:** This is the mechanism behind ``numpy.save()`` and ``numpy.savez()``."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "A Brief Introduction\n",
      "===========================\n",
      "Instead, you want a *database* with the ability to store many datasets, user-defined\n",
      "metadata, optimized I/O, and the ability to query its contents."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Unlike SQL, where every dataset lives in a flat namespace, HDF allows datasets to \n",
      "live in a nested tree structure."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "In effect, HDF5 is a file system within a file.  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "(More on this later.)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "A Brief Introduction\n",
      "===========================\n",
      "\n",
      "Basic dataset classes include:\n",
      "\n",
      "* Array"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* CArray (chunked array)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* EArray (extendable array)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* VLArray (variable length array)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Table (structured array w/ named fields)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "All of these must be composed of atomic types."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "A Brief Introduction\n",
      "===========================\n",
      "There are six atomic types supported by PyTables:\n",
      "\n",
      "- bool: Boolean (true/false) types. 8 bits."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "- int: Signed integer types. 8, 16, 32 (default) and 64 bits."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "- uint: Unsigned integers. 8, 16, 32 (default) and 64 bits."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "- float: Floating point types. 16, 32 and 64 (default) bits."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "- complex: Complex number. 64 and 128 (default) bits."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "- string: Raw string types. 8-bit positive multiples."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "A Brief Introduction\n",
      "===========================\n",
      "Other elements of the hierarchy may include:\n",
      "\n",
      "* Groups (dirs)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Links"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* File Nodes"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Hidden Nodes"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "PyTables docs may be found at http://pytables.github.com/"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Opening Files\n",
      "============================="
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import tables as tb\n",
      "f = tb.open_file('db.h5', 'a')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* *'r'*: Read-only; no data can be modified.\n",
      "* *'w'*: Write; a new file is created (an existing file with the\n",
      "  same name would be deleted).\n",
      "* *'a'*: Append; an existing file is opened for reading and writing,\n",
      "  and if the file does not exist it is created.\n",
      "* *'r+'*: It is similar to 'a', but the file must already exist."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Using the Hierarchy\n",
      "==============================\n",
      "In HDF5, all nodes stem from a root (\"``/``\" or ``f.root``)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "In PyTables, you may access nodes as attributes on a Python object, namely\n",
      "``f.root.a_group.some_data``.  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "This is known as natural naming."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Creating new nodes must be done on the file handle:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f.create_group('/', 'a_group', \"My Group\")\n",
      "f.root.a_group"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "/a_group (Group) 'My Group'\n",
        "  children := []"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Creating Datasets\n",
      "==============================\n",
      "The two most common datasets are Tables & Arrays."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Appropriate create methods live on the file handle:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# integer array\n",
      "f.create_array('/a_group', 'arthur_count', [1, 2, 5, 3])"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "/a_group/arthur_count (Array(4,)) ''\n",
        "  atom := Int64Atom(shape=(), dflt=0)\n",
        "  maindim := 0\n",
        "  flavor := 'python'\n",
        "  byteorder := 'little'\n",
        "  chunkshape := None"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# tables need descriptions\n",
      "dt = np.dtype([('id', int), ('name', 'S10')])\n",
      "knights = np.array([(42, 'Lancelot'), (12, 'Bedivere')], dtype=dt)\n",
      "f.create_table('/', 'knights', dt)\n",
      "f.root.knights.append(knights)"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Reading Datasets\n",
      "==============================\n",
      "Arrays and Tables try to preserve the original flavor that they were created with. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f.root.a_group.arthur_count[:]"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "[1, 2, 5, 3]"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "type(f.root.a_group.arthur_count[:])"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "list"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "type(f.root.a_group.arthur_count)"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "tables.array.Array"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Reading Datasets\n",
      "==============================\n",
      "So if they come from NumPy arrays, they may be accessed in a numpy-like fashion \n",
      "(slicing, fancy indexing, masking)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f.root.knights[1]"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "(12, b'Bedivere')"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f.root.knights[:1]"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "array([(42, b'Lancelot')], \n",
        "      dtype=[('id', '<i8'), ('name', 'S10')])"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mask = (f.root.knights.cols.id[:] < 28)\n",
      "f.root.knights[mask]"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "array([(12, b'Bedivere')], \n",
        "      dtype=[('id', '<i8'), ('name', 'S10')])"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f.root.knights[([1, 0],)]"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 13,
       "text": [
        "array([(12, b'Bedivere'), (42, b'Lancelot')], \n",
        "      dtype=[('id', '<i8'), ('name', 'S10')])"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Data is accessed in a *memory mapped*-like interface."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "\n",
      "<h3>exer/peaks_of_kilimanjaro.py</h3>\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/noneshallpass.jpg\" width=\"50%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "\n",
      "<h3>sol/peaks_of_kilimanjaro.py</h3>\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/theblackknighttriumph.jpg\" width=\"45%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Hierarchy Layout\n",
      "===============================\n",
      "Suppose there is a big table of like-things:\n",
      "\n",
      "```python\n",
      "    # people:  name,            profession,    home\n",
      "    people = [('Arthur',        'King',        'Camelot'), \n",
      "              ('Lancelot',      'Knight',      'Lake'), \n",
      "              ('Bedevere',      'Knight',      'Wales'), \n",
      "              ('Witch',         'Witch',       'Village'), \n",
      "              ('Guard',         'Man-at-Arms', 'Swamp Castle'),\n",
      "              ('Ni',            'Knight',      'Shrubbery'),\n",
      "              ('Strange Woman', 'Lady',        'Lake'),\n",
      "              ...\n",
      "              ]\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "It is tempting to throw everyone into a big ``people`` table."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Hierarchy Layout\n",
      "===============================\n",
      "However, a search over a class of people can be eliminated by splitting \n",
      "these tables up:\n",
      "\n",
      "```python\n",
      "    knight = [('Lancelot',      'Knight',      'Lake'),\n",
      "              ('Bedevere',      'Knight',      'Wales'), \n",
      "              ('Ni',            'Knight',      'Shrubbery'),\n",
      "              ]\n",
      "\n",
      "    others = [('Arthur',        'King',        'Camelot'), \n",
      "              ('Witch',         'Witch',       'Village'), \n",
      "              ('Guard',         'Man-at-Arms', 'Swamp Castle'),\n",
      "              ('Strange Woman', 'Lady',        'Lake'),\n",
      "              ...\n",
      "              ]\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Hierarchy Layout\n",
      "===============================\n",
      "The profession column is now redundant:\n",
      "\n",
      "```python\n",
      "    knight = [('Lancelot', 'Lake'),\n",
      "              ('Bedevere', 'Wales'), \n",
      "              ('Ni',       'Shrubbery'),\n",
      "              ]\n",
      "\n",
      "    others = [('Arthur',        'King',        'Camelot'), \n",
      "              ('Witch',         'Witch',       'Village'), \n",
      "              ('Guard',         'Man-at-Arms', 'Swamp Castle'),\n",
      "              ('Strange Woman', 'Lady',        'Lake'),\n",
      "              ...\n",
      "              ]\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Hierarchy Layout\n",
      "===============================\n",
      "Information can be embedded implicitly in the hierarchy as well:\n",
      "\n",
      "    root\n",
      "      | - England\n",
      "      |     | - knight\n",
      "      |     | - others\n",
      "      |\n",
      "      | - France\n",
      "      |     | - knight\n",
      "      |     | - others"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Hierarchy Layout\n",
      "===============================\n",
      "Why bother pivoting the data like this at all?"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Fewer rows to search over."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Fewer rows to pull from disk."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Fewer columns in description."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Ultimately, it is all about *speed*, especially for big tables."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Access Time Analogy\n",
      "==============================\n",
      "\n",
      "If a processor's access of L1 cache is analogous to you finding a \n",
      "word on a computer screen (3 seconds), then"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Accessing L2 cache is getting a book from a bookshelf (15 s)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Accessing main memory is going to the break room, get a candy bar, \n",
      "and chatting with your co-worker (4 min)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Accessing a (mechanical) HDD is leaving your office, leaving your building, \n",
      "wandering the planet for a year and four months to return to your desk with \n",
      "the information finally made available.\n",
      "\n",
      "<div style=\"color:grey;font-size:0.75em;\">\n",
      "Thanks K. Smith & http://duartes.org/gustavo/blog/post/what-your-computer-does-while-you-wait\n",
      "</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Starving CPU Problem\n",
      "===============================\n",
      "Waiting around for access times prior to computation is known as the \n",
      "*Starving CPU Problem*.\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/starving_cpu.png\" width=\"80%\" /></div>\n",
      "\n",
      "<div style=\"color:grey;font-size:0.75em;\">\n",
      "Francesc Alted. 2010. Why Modern CPUs Are Starving and What Can Be Done about It. IEEE Des. Test 12, 2 (March 2010), 68-71. DOI=10.1109/MCSE.2010.51 http://dx.doi.org/10.1109/MCSE.2010.51\n",
      "</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Tables\n",
      "===============================\n",
      "Tables are a high-level interface to extendable arrays of structs.  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Sort-of."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "In fact, the struct / dtype / description concept is only a convenient way to assign \n",
      "meaning to bytes:\n",
      "\n",
      "    |  ids  |       first       |        last       |\n",
      "    |-------|-------------------|-------------------|\n",
      "    | | | | | | | | | | | | | | | | | | | | | | | | | "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Tables\n",
      "===============================\n",
      "Data types may be nested (though they are stored in flattened way).\n",
      "\n",
      "```python\n",
      "dt = np.dtype([('id', int), \n",
      "               ('first', 'S5'),\n",
      "               ('last',  'S5'),\n",
      "               ('parents', [\n",
      "                    ('mom_id', int),\n",
      "                    ('dad_id', int),\n",
      "                ]),\n",
      "            ])\n",
      "\n",
      "people = np.fromstring(np.random.bytes(dt.itemsize * 10000), dt)\n",
      "f.create_table('/', 'random_peeps', people)\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Tables\n",
      "===============================\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/random_peeps.png\" width=\"70%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Tables\n",
      "===============================\n",
      "Python already has the ability to dynamically declare the size of \n",
      "descriptions.  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "This is accomplished in compiled languages through normal memory allocation \n",
      "and careful byte counting:\n",
      "\n",
      "```\n",
      "typedef struct mat {\n",
      "  double mass;\n",
      "  int atoms_per_mol;\n",
      "  double comp [];\n",
      "} mat;\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Tables\n",
      "===============================\n",
      "\n",
      "    // use mat struct from before\n",
      "\n",
      "    size_t mat_size = sizeof(mat) + sizeof(double)*comp_size;\n",
      "    hid_t desc = H5Tcreate(H5T_COMPOUND, mat_size);\n",
      "    hid_t comptype = H5Tarray_create2(H5T_NATIVE_DOUBLE, 1, nuc_dims);\n",
      "\n",
      "    // make the data table type\n",
      "    H5Tinsert(desc, \"mass\", HOFFSET(mat, mass), H5T_NATIVE_DOUBLE);\n",
      "    H5Tinsert(desc, \"atoms_per_mol\", HOFFSET(mat, atoms_per_mol), H5T_NATIVE_DOUBLE);\n",
      "    H5Tinsert(desc, \"comp\", HOFFSET(mat, comp), comp_type);\n",
      "\n",
      "    // make the data array for a single row, have to over-allocate\n",
      "    mat * mat_data  = new mat[mat_size];\n",
      "\n",
      "    // ...fill in data array...\n",
      "\n",
      "    // Write the row\n",
      "    H5Dwrite(data_set, desc, mem_space, data_hyperslab, H5P_DEFAULT, mat_data);"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "\n",
      "<h3>exer/boatload.py</h3>\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/noneshallpass.jpg\" width=\"50%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "\n",
      "<h3>sol/boatload.py</h3>\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/theblackknighttriumph.jpg\" width=\"45%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Chunking\n",
      "===============================\n",
      "Chunking is a feature with no direct analogy in NumPy."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "<div style=\"text-align:center;\">\n",
      "*Chunking is the ability to split up a dataset into smaller blocks of equal or lesser rank.*\n",
      "</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Extra metadata pointing to the location of the chunk in the \n",
      "file and in dataspace must be stored."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "By chunking, sparse data may be stored efficiently and \n",
      "datasets may extend infinitely in all dimensions."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "<div style=\"color:grey;font-size:0.75em;\">\n",
      "**Note:** Currently, PyTables only allows one extendable dim.\n",
      "</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Chunking\n",
      "===============================\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/dset_contiguous.jpg\" width=\"30%\"/></div>\n",
      "\n",
      "<br /><br />\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/dset_chunked.jpg\" width=\"30%\" /></div>\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Chunking\n",
      "===============================\n",
      "All I/O happens by chunk.  This is important for:\n",
      "\n",
      "* edge chunks may extend beyond the dataset"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* default fill values are set in unallocated space"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* reading and writing in parallel"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* small chunks are good for accessing some of data"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* large chunks are good for accessing lots of data"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Chunking\n",
      "===============================\n",
      "Any chunked dataset allows you to set the chunksize.\n",
      "\n",
      "```python\n",
      "f.create_table('/', 'omnomnom', data, chunkshape=(42,42))\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "For example, a 4x4 chunked array could have a 3x3 chunksize."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "However, it could not have a 12x12 chunksize, since the ranks must be \n",
      "less than or equal to that of the array."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Manipulating the chunksize is a great way to fine-tune an application."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Chunking\n",
      "===============================\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/dset_contiguous4x4.jpg\" /><h3>Contiguous 4x4 Dataset</h3></div>\n",
      "<br/><br/>\n",
      "<div style=\"text-align:center;\"><img src=\"img/dset_chunked4x4.jpg\" /><h3>Chunked 4x4 Dataset</h3></div>    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Chunking\n",
      "===============================\n",
      "Note that the addresses of chunks in dataspace (memory) has \n",
      "no bearing on their arrangement in the actual file.\n",
      "<br/><br/>\n",
      "<div style=\"text-align:center;\"><img src=\"img/dset_address_space.jpg\" width=\"100%\" /><h3>Dataspace (top) vs File (bottom) Chunk Locations</h3></div>    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "In-Core vs Out-of-Core\n",
      "===============================\n",
      "Calculations depend on the current memory layout."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Recall access time analogy (wander Earth for 16 months)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "**Definitions:**"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Operations which require all data to be in memory are *in-core* and may be memory bound (NumPy)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Operations where the dataset is external to memory are *out-of-core* (or *in-kernel*) and may be CPU bound."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "In-Core Operations\n",
      "==============================\n",
      "Say, ``a`` and ``b`` are arrays sitting in memory:\n",
      "\n",
      "```python\n",
      "a = np.array(...)\n",
      "b = np.array(...)\n",
      "c = 42 * a + 28 * b + 6\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "The expression for ``c`` creates three temporary arrays!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "For ``N`` operations, ``N-1`` temporaries are made."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Wastes memory and is slow.  Pulling from disk is slower."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "In-Core Operations\n",
      "==============================\n",
      "A less memory intensive implementation would be an element-wise\n",
      "evaluation:\n",
      "\n",
      "```python\n",
      "c = np.empty(...)\n",
      "for i in range(len(c)):\n",
      "    c[i] = 42 * a[i] + 28 * b[i] + 6\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "<div style=\"font-size:1.5em;\">But if ``a`` and ``b`` were HDF5 arrays on disk, individual element access time would kill you.</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "<div style=\"font-size:1.5em;\">Even with in memory NumPy arrays, there are problems with gratuitous Python type checking. </div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Out-of-Core Operations\n",
      "===============================\n",
      "\n",
      "<div style=\"font-size:1.5em;\">Say there was a virtual machine (or kernel) which could be fed arrays and perform specified operations.</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "<div style=\"font-size:1.5em;\">Giving this machine only chunks of data at a time, it could function on infinite-length data using only finite memory.</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "```python\n",
      "for i in range(0, len(a), 256):\n",
      "    r0, r1 = a[i:i+256], b[i:i+256]\n",
      "    multiply(r0, 42, r2)\n",
      "    multiply(r1, 28, r3)\n",
      "    add(r2, r3, r2); add(r2,  6, r2)\n",
      "    c[i:i+256] = r2\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Out-of-Core Operations\n",
      "===============================\n",
      "This is the basic idea behind numexpr, which provides a general \n",
      "virtual machine for NumPy arrays."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "This problem lends itself nicely to parallelism.  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Numexpr has low-level multithreading, avoiding the GIL."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "PyTables implements a ``tb.Expr`` class which backends to the numexpr VM\n",
      "but has additional optimizations for disk reading and writing."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "The full array need never be in memory."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Out-of-Core Operations\n",
      "===============================\n",
      "Fully out-of-core expression example:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "shape = (10, 10000)\n",
      "f = tb.open_file(\"/tmp/expression.h5\", \"w\")\n",
      "\n",
      "a = f.create_carray(f.root, 'a', tb.Float32Atom(dflt=1.), shape)\n",
      "b = f.create_carray(f.root, 'b', tb.Float32Atom(dflt=2.), shape)\n",
      "c = f.create_carray(f.root, 'c', tb.Float32Atom(dflt=3.), shape)\n",
      "out = f.create_carray(f.root, 'out', tb.Float32Atom(dflt=3.), shape)\n",
      "\n",
      "expr = tb.Expr(\"a*b+c\")\n",
      "expr.set_output(out)\n",
      "d = expr.eval()\n",
      "\n",
      "print(\"returned-->\", repr(d))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "returned--> /out (CArray(10, 10000)) ''\n",
        "  atom := Float32Atom(shape=(), dflt=3.0)\n",
        "  maindim := 0\n",
        "  flavor := 'numpy'\n",
        "  byteorder := 'little'\n",
        "  chunkshape := (1, 10000)\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Querying\n",
      "===============================\n",
      "The most common operation is asking an existing dataset\n",
      "whether its elements satisfy some criteria.  \n",
      "This is known as *querying*.  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Because querying is so common PyTables defines special methods on \n",
      "Tables.\n",
      "\n",
      "```python\n",
      "tb.Table.where(cond)\n",
      "tb.Table.get_where_list(cond)\n",
      "tb.Table.read_where(cond)\n",
      "tb.Table.where_append(dest, cond)\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Querying\n",
      "===============================\n",
      "The conditions used in ``where()`` calls are strings which are \n",
      "evaluated by numexpr.  These expressions must return boolean\n",
      "values."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "They are executed in the context of table itself combined with \n",
      "``locals()`` and ``globals()``."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "The ``where()`` method itself returns an iterator over all \n",
      "matched (hit) rows:\n",
      "\n",
      "```python\n",
      "for row in table.where('(col1 < 42) & (col2 == col3)'):\n",
      "    # do something with row\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Querying\n",
      "===============================\n",
      "For a speed comparison, here is a complex query using regular Python:\n",
      "\n",
      "```python\n",
      "result = [row['col2'] for row in table if (\n",
      "              ((row['col4'] >= lim1 and row['col4'] < lim2) or\n",
      "              ((row['col2'] > lim3 and row['col2'] < lim4])) and\n",
      "              ((row['col1']+3.1*row['col2']+row['col3']*row['col4']) > lim5)\n",
      "              )]\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "And this is the equivalent out-of-core search:\n",
      "\n",
      "```python\n",
      "result = [row['col2'] for row in table.where(\n",
      "             '(((col4 >= lim1) & (col4 < lim2)) | '\n",
      "             '((col2 > lim3) & (col2 < lim4)) &   '\n",
      "             '((col1+3.1*col2+col3*col4) > lim5)) ')]\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Querying\n",
      "===============================\n",
      "<div style=\"text-align:center;\"><img src=\"img/where_compare_10Mrow.png\" width=\"77%\" /><h3>Complex query with 10 million rows. Data fits in memory.</h3></div>    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Querying\n",
      "===============================\n",
      "<div style=\"text-align:center;\"><img src=\"img/where_compare_1Grow.png\" width=\"77%\" /><h3>Complex query with 1 billion rows. Too big for memory.</h3></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "\n",
      "<h3>exer/crono.py</h3>\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/noneshallpass.jpg\" width=\"50%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "\n",
      "<h3>sol/crono.py</h3>\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/theblackknighttriumph.jpg\" width=\"45%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "A more general way to solve the starving CPU problem is through\n",
      "*compression*."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Compression is when the dataset is piped through a zipping algorithm\n",
      "on write and the inverse unzipping algorithm on read."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Each chunk is compressed independently, so chunks end up with a\n",
      "varying number bytes."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Has some storage overhead, but may drastically reduce file sizes for \n",
      "very regular data."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "At first glance this is counter-intuitive. (*Why?*)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Compression/Decompression is clearly more CPU\n",
      "intensive than simply blitting an array into memory."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "However, because there is *less total information* to transfer, \n",
      "the time spent unpacking the array can be far less than moving \n",
      "the array around wholesale."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "This is kind of like power steering, you can either tell wheels\n",
      "how to turn manually or you can tell the car how you want the wheels\n",
      "turned."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "Compression is a guaranteed feature of HDF5 itself."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "At minimum, HDF5 requires zlib."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "The compression capabilities feature a plugin architecture which \n",
      "allow for a variety of different algorithms, including user defined ones!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "PyTables supports:\n",
      "\n",
      ".. container:: align-center\n",
      "\n",
      "* zlib (default)\n",
      "* lzo\n",
      "* bzip2 \n",
      "* blosc\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "Compression is enabled in PyTables through *filters*.\n",
      "\n",
      "```python\n",
      "# complevel goes from [0,9]\n",
      "filters = tb.Filters(complevel=5, complib='blosc', ...)\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "```python\n",
      "# filters may be set on the whole file,\n",
      "f = tb.open_file('/path/to/file', 'a', filters=filters)\n",
      "f.filters = filters\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "```python\n",
      "# filters may also be set on most other nodes\n",
      "f.create_table('/', 'table', desc, filters=filters)\n",
      "f.root.group._v_filters = filters\n",
      "```"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Filters only act on chunked datasets."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "Tips for choosing compression parameters:"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* A mid-level (5) compression is sufficient. No need to go all the way up (9)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Use zlib if you must guarantee complete portability."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Use blosc all other times.  It is optimized for HDF5."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "*But why?* (I don't have time to go into the details of blosc. However\n",
      "here are some justifications...)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "<div style=\"text-align:center;\"><img src=\"img/compressed-recordsize-zlib.png\" width=\"70%\" /><h3>Comparison of different compression levels of zlib.</h3></div>    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "<div style=\"text-align:center;\"><img src=\"img/create-chunksize-15GB.png\" width=\"70%\" /><h3>Creation time per element for a 15 GB EArray and different chunksizes.</h3></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "<div style=\"text-align:center;\"><img src=\"img/filesizes-chunksize-15GB.png\" width=\"80%\" /><h3>File sizes for a 15 GB EArray and different chunksizes.</h3></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "<div style=\"text-align:center;\"><img src=\"img/seq-chunksize-15GB.png\" width=\"70%\" /><h3>Sequential access time per element for a 15 GB EArray and different chunksizes.</h3></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Compression\n",
      "===============================\n",
      "<div style=\"text-align:center;\"><img src=\"img/random-chunksize-15GB.png\" width=\"70%\" /><h3>Random access time per element for a 15 GB EArray and different chunksizes.</h3></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "\n",
      "<h3>exer/spam_filter.py</h3> \n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/noneshallpass.jpg\" width=\"50%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "\n",
      "<h3>sol/spam_filter.py</h3>\n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/theblackknighttriumph.jpg\" width=\"45%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Other Python Data Structures\n",
      "===============================\n",
      "Overwhelmingly, numpy arrays have been the in-memory data\n",
      "structure of choice."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Using lists or tuples instead of arrays follows analogously."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "It is data structures like sets and dictionaries which do not\n",
      "quite map.  "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "However, as long as all elements may be cast into the same atomic type, \n",
      "these structures can be stored in HDF5 with relative ease."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Sets\n",
      "===============================\n",
      "Example of serializing and deserializing sets:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s = {1.0, 42, 77.7, 6E+1, True}\n",
      "f.create_array('/', 's', [float(x) for x in s])"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "/s (Array(4,)) ''\n",
        "  atom := Float64Atom(shape=(), dflt=0.0)\n",
        "  maindim := 0\n",
        "  flavor := 'python'\n",
        "  byteorder := 'little'\n",
        "  chunkshape := None"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "set(f.root.s)"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 16,
       "text": [
        "{1.0, 42.0, 60.0, 77.7}"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "**exer/dict_table.py** \n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/noneshallpass.jpg\" width=\"50%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Exercise\n",
      "===============================\n",
      "**sol/dict_table.py** \n",
      "\n",
      "<div style=\"text-align:center;\"><img src=\"img/theblackknighttriumph.jpg\" width=\"45%\" /></div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "What Was Missed\n",
      "==============================\n",
      "* Walking Nodes\n",
      "* File Nodes\n",
      "* Indexing\n",
      "* Migrating to / from SQL\n",
      "* HDF5 in other database formats\n",
      "* Other Databases in HDF5\n",
      "* HDF5 as a File System"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Acknowledgements\n",
      "===============================\n",
      "Many thanks to everyone who made this possible!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "* The HDF Group "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* The PyTables Developers"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* The NumPy Developers"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* h5py, the symbiotic project"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "<div style=\"text-align:center;\">**Shameless Plug:** *We are always looking for more hands. Join Now!*</div>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Questions\n",
      "===============================\n",
      "<div style=\"text-align:center;\"><img src=\"img/xkcd_bobby_tables.jpg\" width=\"50%\" /></div>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!make"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "skip"
      }
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "ipython nbconvert --to slides hdf5-is-for-lovers.ipynb \r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[NbConvertApp] Using existing profile dir: '/home/scopatz/.ipython/profile_default'\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[NbConvertApp] Converting notebook hdf5-is-for-lovers.ipynb to slides\r\n",
        "[NbConvertApp] Support files will be in hdf5-is-for-lovers_files/\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[NbConvertApp] Loaded template slides_reveal.tpl\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/home/scopatz/.local/lib/python3.4/site-packages/IPython/nbconvert/filters/markdown.py:78: UserWarning: Node.js 0.9.12 or later wasn't found.\r\n",
        "Nbconvert will try to use Pandoc instead.\r\n",
        "  \"Nbconvert will try to use Pandoc instead.\")\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[NbConvertApp] Writing 280957 bytes to hdf5-is-for-lovers.slides.html\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "cp classy.css reveal.js/css/theme/ \r\n",
        "sed -i 's:reveal.js/css/theme/simple.css:reveal.js/css/theme/classy.css:' hdf5-is-for-lovers.slides.html \r\n",
        "sed -i 's:class=\"fragment\" class=\"text_cell_render:class=\"fragment text_cell_render:' hdf5-is-for-lovers.slides.html \r\n",
        "sed -i 's/.rendered_html ul{list-style:disc;margin:1em 2em;}/.rendered_html ul{list-style:disc;margin:0em 2em;}/' hdf5-is-for-lovers.slides.html \r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sed -i '/<\\!-- Social buttons -->/,/<\\!-- End of social buttons -->/d' hdf5-is-for-lovers.slides.html \r\n"
       ]
      }
     ],
     "prompt_number": 18
    }
   ],
   "metadata": {}
  }
 ]
}