L05-2-Bootstrap.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div id=\"instructions\" style=\"border-radius: 5px; background-color:#f5f5f5;\" >\n",
    "<h1>Instructions</h1>\n",
    "<p>Look for the <b>2 Your Turn</b> sections to complete the code and/or answer questions.<p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 5 - Bootstrap and Resample Methods\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Bootstrap and resampling are widely applicable statistical methods which relax many of the assumptions of classical statistics. Resampling methods draw heavily on the CLT. Specifically resampling methods:\n",
    "\n",
    "- Allow computation of statistics from limited data\n",
    "- Compute statistics from multiple subsamples of dataset\n",
    "- Minimal distribution assumptions\n",
    "- Computationally intensive\n",
    "\n",
    "Commonly used resampling methods include:\n",
    "\n",
    "- Randomization or permutation methods (e.g., Fisher's exact test)\n",
    "- Bootstrap: resample with equivalent size and replacement\n",
    "- Jackknife: leave one out resampling\n",
    "- Cross validation: resample into folds without replacement\n",
    "\n",
    "\n",
    "## Pitfalls\n",
    "\n",
    "But, there is no magic involved. When using resampling methods always keep in mind the several pitfalls:\n",
    "- If sample is biased, the resample statistic is biased.\n",
    "- Sample variance and CIs are no better than what the sample allows.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## History\n",
    "\n",
    "### Randomization and permutation methods\n",
    "\n",
    "Randomization and permutation methods were pioneered by Fisher as early as 1911. Fisher fully developed the theory in his 1935 book. Scalability of these methods remain limited, even with modern computers. \n",
    "\n",
    "### Cross-validation\n",
    "\n",
    "Cross-validation was originally proposed by Kurtz in 1948. Mosier extended the method to double cross validation in 1951. The modern method of nested or multicross-validation were introduced by Krus and Fuller in 1982. Today, cross-validation is widely used in the testing of machine learning models. \n",
    "\n",
    "### Jackknife Methods\n",
    "\n",
    "Maurice Quenouille originally suggested this method in 1949. The jackknife was fully developed\n",
    "by John W. Tukey, who gave the method its name, in 1958. Tukey saw that method as a simple tool useful for many purposes like a pocket knife. \n",
    "\n",
    "\n",
    "### Bootstrap \n",
    "\n",
    "<img src=\"https://library.startlearninglabs.uw.edu/DATASCI410/img/Efron1980.jpg\" style=\"float: left; height: 150px; padding-left: 10px;\" title=\"Efron's 1980 book.\" />\n",
    "The bootstrap method was first suggested by Efron and Hinkley in 1978 and further developed by Efron in 1979. A full treatment was provided in Efron's 1980 book. \n",
    "\n",
    "With increased computing power, use of bootstrap methods continues to expand. Check out <a title=\"YouTube: Interview with Bradley Efron, BBVA Foundation Frontiers of Knowledge Award in Basic Sciences\" href=\"https://www.youtube.com/embed/H2tOhMaXWvI\" target=\"newwindow\">Efron himself explaining</a> the usefulness of bootstrap method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview of the Bootstrap\n",
    "\n",
    "The bootstrap method:\n",
    "\n",
    "- Re-compute statistic many times with sample \n",
    "- Randomly sample (e.g., Bernoulli sample) data with replacement\n",
    "- Subsamples have the same size as original sample\n",
    "- Works with any statistic … in principle\n",
    "\n",
    "For example, you can compute the bootstrap mean as:\n",
    "\n",
    "$$Meanboot = \\frac{\\Sigma_i mean(sample_i)}{nsample}\\\\\n",
    "where,\\ for\\ example\\ with\\ 10\\ samples,\\\\\n",
    "sample_i = X_1 + X_2 + X_3 + X_4 + X_5 + X_6 + X_7 + X_8 + X_1 + X_5$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bootstrap Example\n",
    "\n",
    "Let's try a first example. You will bootstrap the means of the heights of different populations from Galton's height data. This is an example of a parametric bootstrap estimate. Parametric because our model has a parameter, the mean, we are trying to estimate. \n",
    "\n",
    "As a first step, the code in the cell below divides the data set by male and female adult children, and then plots the distributions with the means of the two populations. Run this code and examine the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "import numpy\n",
    "from matplotlib import pyplot\n",
    "import seaborn\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>father</th>\n",
       "      <th>mother</th>\n",
       "      <th>midparentHeight</th>\n",
       "      <th>children</th>\n",
       "      <th>childNum</th>\n",
       "      <th>gender</th>\n",
       "      <th>childHeight</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>family</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>001</th>\n",
       "      <td>78.5</td>\n",
       "      <td>67.0</td>\n",
       "      <td>75.43</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>73.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>001</th>\n",
       "      <td>78.5</td>\n",
       "      <td>67.0</td>\n",
       "      <td>75.43</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>69.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>001</th>\n",
       "      <td>78.5</td>\n",
       "      <td>67.0</td>\n",
       "      <td>75.43</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>69.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>001</th>\n",
       "      <td>78.5</td>\n",
       "      <td>67.0</td>\n",
       "      <td>75.43</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>female</td>\n",
       "      <td>69.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>002</th>\n",
       "      <td>75.5</td>\n",
       "      <td>66.5</td>\n",
       "      <td>73.66</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>73.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        father  mother  midparentHeight  children  childNum  gender  \\\n",
       "family                                                                \n",
       "001       78.5    67.0            75.43         4         1    male   \n",
       "001       78.5    67.0            75.43         4         2  female   \n",
       "001       78.5    67.0            75.43         4         3  female   \n",
       "001       78.5    67.0            75.43         4         4  female   \n",
       "002       75.5    66.5            73.66         4         1    male   \n",
       "\n",
       "        childHeight  \n",
       "family               \n",
       "001            73.2  \n",
       "001            69.2  \n",
       "001            69.0  \n",
       "001            69.0  \n",
       "002            73.5  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "families = pandas.read_csv('https://library.startlearninglabs.uw.edu/DATASCI410/Datasets/GaltonFamilies.csv', \n",
    "                            index_col=0)\n",
    "families.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can select a subset of rows from a dataframe that match a given condition like so (the condition is gender in this case):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(481, 453)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "male = families[families.gender == 'male']\n",
    "female = families[families.gender == 'female']\n",
    "\n",
    "len(male), len(female)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can get the min or max of a series of values within a \n",
    "dataframe like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "56.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "families.childHeight.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZAElEQVR4nO3dfbBkdX3n8fdHENQRBJwBkWdSgItWiTpBjYagBAUqAjGiUIRMotZoItmSkkRcjeJGavEhhbvlw4qKTjTyIBEhG6NMSCVWNqLMEFQgPIkDjDPOjAgrghEHv/tHn4HmevveO3Nv3/71nferqqvP+fV5+N7DOfPh/Pr0OakqJElqzRNGXYAkSZMxoCRJTTKgJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElNMqCkOZbk7Ul+kOSBJLcmOSbJzkk+nGRd9/pwkp276Y9OsjbJ25JsTLI+yR/1Le+EJDd3y/tBkrNH99dJ88eAkuZQksOAM4Ffr6pdgFcCa4B3Ai8CjgCeCxwJvKtv1mcATwP2Ad4AfDTJ7t1nnwbe1C3vOcA/Df0PkRpgQElz6xFgZ+DwJE+sqjVV9T3gdOC/V9XGqtoEvBc4o2++X3Sf/6KqvgL8FDis77PDk+xaVfdV1fXz9+dIo2NASXOoqu4A3gqcC2xMckmSZwLPBO7qm/Surm2Le6tqc9/4Q8BTu+HfA04A7kryL0lePKTypaYYUNIcq6ovVNVLgQOAAt4PrOvGt9i/a5vJ8q6rqpOAPYEvA5fNacFSowwoaQ4lOSzJy7sLIP4T+Bm9br+LgXclWZJkMfBu4PMzWN5OSU5P8rSq+gXwk2550oK346gLkBaYnYHzgf9C77ujfwOWAz8GdgW+0033ReB9M1zmGcBHkuwA3Ar8/lwWLLUqPrBQktQiu/gkSU2aNqCSXNT9ePDGvrZzux8M3tC9Tuj77B1J7uh+oPjKYRUuSVrYpu3iS3IUvd9k/HVVPadrOxf4aVV9aMK0h9P7MvhIepfQ/iNwaFX5pa4kaatMewZVVV+n9wXvTJwEXFJVP6+q7wN30AsrSZK2ymyu4jszyR8Aq4C3VdV99G7Tcm3fNGu7tl+RZDm9q5tYtGjRC571rGfNohTNyOrVjw2/4AWjq2M7duemBwE4eMmiEVeyALl/j63Vq1f/qKqWTGzf1oD6OPCX9H6E+JfAXwGvBzLJtJP2IVbVhcCFAEuXLq1Vq1ZtYymasfT953F7j8TrPvENAC59kzeDmHPu32MryV2TtW/TVXxVtaGqHqmqXwKf5LFuvLXAfn2T7ssMfy0vSVK/bQqoJHv3jf4usOUKv6uAU7tHCxwEHAJ8a3YlSpK2R9N28SW5GDgaWJxkLfAe4OgkR9DrvlsDvAmgqm5KchlwM7AZeItX8EmStsW0AVVVp03S/Okppj8POG82RUnbkwtW3jbqEh511rGHjroE6VHeSUKS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktSkaZ8HJS1Uo3gO09r7fjaydUvjxjMoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKTpg2oJBcl2Zjkxr62PZKsTHJ7975732fvSHJHkluTvHJYhUuSFraZnEF9FjhuQts5wDVVdQhwTTdOksOBU4Fnd/N8LMkOc1atJGm7MW1AVdXXgR9PaD4JWNENrwBO7mu/pKp+XlXfB+4AjpybUiVJ25Nt/Q5qr6paD9C979m17wPc0zfd2q7tVyRZnmRVklWbNm3axjIkSQvVXF8kkUnaarIJq+rCqlpaVUuXLFkyx2VIksbdtgbUhiR7A3TvG7v2tcB+fdPtC6zb9vIkSdurbQ2oq4Bl3fAy4Mq+9lOT7JzkIOAQ4FuzK1GStD2a9nEbSS4GjgYWJ1kLvAc4H7gsyRuAu4FTAKrqpiSXATcDm4G3VNUjQ6pdkrSATRtQVXXagI+OGTD9ecB5sylKkiTvJCFJapIBJUlqkgElSWqSASVJapIBJUlq0rRX8Ulz6YKVt426BEljwjMoSVKTDChJUpPs4pP0qJa6YM869tBRl6AR8wxKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktSkWd3NPMka4AHgEWBzVS1NsgdwKXAgsAZ4bVXdN7syJUnbm7k4g3pZVR1RVUu78XOAa6rqEOCablySpK0yjC6+k4AV3fAK4OQhrEOStMDNNqAKuDrJ6iTLu7a9qmo9QPe+5yzXIUnaDs32ibovqap1SfYEVia5ZaYzdoG2HGD//fefZRmSpIVmVmdQVbWue98IXAEcCWxIsjdA975xwLwXVtXSqlq6ZMmS2ZQhSVqAtjmgkixKssuWYeAVwI3AVcCybrJlwJWzLVKStP2ZTRffXsAVSbYs5wtV9dUk1wGXJXkDcDdwyuzLlCRtb7Y5oKrqTuC5k7TfCxwzm6IkSfJOEpKkJhlQkqQmGVCSpCYZUJKkJs32h7oaAxesvA2AsyZpk6RWeQYlSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkjeLldSkrb2h8TBvhnzWsYfO6fI0M55BSZKaZEBJkppkQEmSmmRASZKa5EUSQ+ITayVpdoZ2BpXkuCS3JrkjyTnDWo8kaWEaSkAl2QH4KHA8cDhwWpLDh7EuSdLCNKwuviOBO6rqToAklwAnATcPaX2SNDR22U9u2L8PS1XN/UKT1wDHVdUbu/EzgBdW1Zl90ywHlnejhwG3zsGqFwM/moPlzDfrnl/WPb+se36NY90HVNWSiY3DOoPKJG2PS8KquhC4cE5XmqyqqqVzucz5YN3zy7rnl3XPr3GtezLDukhiLbBf3/i+wLohrUuStAANK6CuAw5JclCSnYBTgauGtC5J0gI0lC6+qtqc5Ezga8AOwEVVddMw1jXBnHYZziPrnl/WPb+se36Na92/YigXSUiSNFve6kiS1CQDSpLUpLEIqCS7Jbk8yS1J/iPJi5PskWRlktu7990HzDuyWy4NqPuD3fh3klyRZLcB865J8t0kNyRZ1UDd5yb5QVfPDUlOGDBva9v70r6a1yS5YcC8I9neSQ7rq++GJD9J8tbW9+8p6m56/56i7qb37ynqbnr/nrWqav4FrADe2A3vBOwGfAA4p2s7B3j/JPPtAHwPOLib79vA4SOu+xXAjl3b+yeru/tsDbC4oe19LnD2NPM1t70nfP5XwLtb294Ttt8PgQPGYf8eUHfz+/eAupvfvyere0J70/v3tryaP4NKsitwFPBpgKp6uKrup3frpBXdZCuAkyeZ/dFbLlXVw8CWWy4N3aC6q+rqqtrcTXYtvd+INWOK7T0TzW3vvs8DvBa4eD7q2UbHAN+rqrtofP+e4NG6W9+/J+jf3jPR3Pbe0jAm+/dWaz6g6P3fyibgM0n+PcmnkiwC9qqq9QDd+56TzLsPcE/f+NqubT4Mqrvf64F/GDB/AVcnWZ3ebaHmy1R1n9l13Vw0oMup5e39m8CGqrp9wPyj2t79TuWxf2Ba37/79dfdr8X9u9/Eulvev/tNtr3HYf/eauMQUDsCzwc+XlXPAx6k1+UxE9PecmmIpqw7yTuBzcDfDJj/JVX1fHp3hH9LkqOGXO8Wg+r+OPBrwBHAenrdCRM1u72B05j6/y5Htb0BSO8H7ScCX9ya2SZpm9ffjQyqu+H9G5i07tb3714Bg/eTpvfvbTUOAbUWWFtV3+zGL6f3D9GGJHsDdO8bB8w7qlsuDaqbJMuA3wFOr66DeKKqWte9bwSuoNe9MB8mrbuqNlTVI1X1S+CTA+ppdXvvCLwauHTQzCPc3lscD1xfVRu68db37y0m1t36/r3F4+oeg/17i8m29zjs39uk+YCqqh8C9yQ5rGs6ht5jO64ClnVty4ArJ5l9ZLdcGlR3kuOAtwMnVtVDk82bZFGSXbYM0/vi+cZ5KHuquvfum+x3B9TT3Pbuhn8buKWq1k427yi3d5+J/wfc9P7d53F1t75/95lYd9P7d5/JzpTGYf/eNqO+SmMmL3qn3auA7wBfBnYHng5cA9zeve/RTftM4Ct9854A3Ebv6pt3NlD3HfT6sW/oXv97Yt30vk/5dve6qZG6Pwd8t2u7Cth7HLZ31/5Z4M0Tpm1pez8FuBd4Wl/bOOzfk9U9Dvv3ZHWPw/79K3V37U3v37N5easjSVKTmu/ikyRtnwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKGkKST6b5H1DXse5ST4/zHVI48iAkhqXZE2S3x51HdJ8M6CkBSw9HucaS+64Up8kz0tyfZIHklwKPKlr3z3J/0myKcl93fC+ffM97ixnYrddkj9IcleSe5P8xSRnRTsl+etuvTclWdrN9zlgf+Dvkvw0yZ937S9K8m9J7k/y7SRH963rn5Ocl+T/Ag8BByf5wyR3dsv/fpLTh7H9pLlkQEmd7jHeX6b3dNU9gC8Cv9d9/ATgM8AB9ALjZ8BHZrjcw4GPAacDewNPA/aZMNmJwCXAbvSe6PoRgKo6A7gbeFVVPbWqPpBkH+Dvgfd1dZ4N/G2SJX3LOwNYDuwCbAL+F3B8Ve0C/Aa9p91KTTOgpMe8CHgi8OGq+kVVXQ5cB1BV91bV31bVQ1X1AHAe8FszXO5rgL+rqn+tqoeBdwMTH2X9r1X1lap6hF5APneK5f0+vcd5f6WqfllVK+k96v6Evmk+W1U3VdVmYDPwS+A5SZ5cVeur6qYZ1i6NjAElPeaZwA+qqj887gJI8pQkn+i66X4CfB3YLckOM1zuPVtGquoh4N4J0/ywb/gh4ElJdhywvAOAU7ruvfuT3A+8lN7Z2Rb963sQeB3wZmB9kr9P8qwZ1C2NlAElPWY9sE+S9LXt372/DTgMeGFV7Qoc1bVvmfZB4Cl98z1jwnL7v696MvD0rahr4tnWPcDnqmq3vteiqjp/0DxV9bWqOpZeiN0CfHIr1i+NhAElPeYb9LrD/muSHZO8Gjiy+2wXet873Z9kD+A9E+a9ATg1yRO7Cxxe0/fZ5cCrkvxG9z3Xe3ks2GZiA3Bw3/jnu+W9MskOSZ6U5Oj+izb6JdkryYlJFgE/B34KPLIV65dGwoCSOt33Q68G/hC4j1632Je6jz8MPBn4EXAt8NUJs/8F8GvdfO8FvtC33JuAP6V3EcR64AFgI72wmIn/Abyr6847u6ruAU4C/hu9CyDuAf6MwcfzE+idAa4Dfkzvu7M/meG6pZHJ47vbJQ1bkqcC9wOHVNX3R1yO1CzPoKR5kORV3YUWi4APAd8F1oy2KqltBpQ0P06i18W2DjgEOLXsvpCmZBefJKlJnkFJkpo06IeA82rx4sV14IEHjroMzcbq1Y8Nv+AFo6tD3LnpQQAOXrJoxJVoTi3gY2z16tU/qqolE9ub6OJbunRprVq1atRlaDb6f9vawD61PXvdJ74BwKVvevGIK9GcWsDHWJLVVbV0YrtdfJKkJhlQkqQmGVCSpCY1cZGENAoXrLxt1CUAcNaxh466hEe1sk2gre2i0fAMSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUpGkDKslFSTYmubGvbY8kK5Pc3r3v3vfZO5LckeTWJK8cVuGSpIVtJmdQnwWOm9B2DnBNVR0CXNONk+Rw4FTg2d08H0uyw5xVK0nabkwbUFX1deDHE5pPAlZ0wyuAk/vaL6mqn1fV94E7gCPnplRJ0vZkW7+D2quq1gN073t27fsA9/RNt7Zr+xVJlidZlWTVpk2btrEMSdJCNdcXSWSStkkfXFJVF1bV0qpaumTJrzynSpK0ndvWgNqQZG+A7n1j174W2K9vun2BddteniRpe7WtAXUVsKwbXgZc2dd+apKdkxwEHAJ8a3YlSpK2R9M+biPJxcDRwOIka4H3AOcDlyV5A3A3cApAVd2U5DLgZmAz8JaqemRItUuSFrBpA6qqThvw0TEDpj8POG82RUmS5J0kJElNMqAkSU3yke/SiM31Y9bX3vezoSxXmm+eQUmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkppkQEmSmmRASZKaZEBJkpq042xmTrIGeAB4BNhcVUuT7AFcChwIrAFeW1X3za5MSdL2Zi7OoF5WVUdU1dJu/Bzgmqo6BLimG5ckaasMo4vvJGBFN7wCOHkI65AkLXCzDagCrk6yOsnyrm2vqloP0L3vOct1SJK2Q7P6Dgp4SVWtS7InsDLJLTOdsQu05QD777//LMuQJC00szqDqqp13ftG4ArgSGBDkr0BuveNA+a9sKqWVtXSJUuWzKYMSdICtM0BlWRRkl22DAOvAG4ErgKWdZMtA66cbZGSpO3PbLr49gKuSLJlOV+oqq8muQ64LMkbgLuBU2ZfpiRpe7PNAVVVdwLPnaT9XuCY2RQlSZJ3kpAkNWm2V/FJ0lBcsPK2UZfwqLOOPXTUJWyXPIOSJDXJgJIkNcmAkiQ1yYCSJDXJiyQ0r1r64ltS2zyDkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXJH+pK0jRa+IH5WaMuYAQ8g5IkNcmAkiQ1yS6+7cB8dE/0dz+00B0iafx5BiVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSASVJapIBJUlqkgElSWqSd5KQpDHTyt1azjr20KEu3zMoSVKTDChJUpOG1sWX5DjgfwI7AJ+qqvOHta4tWjntheGf+krSQjeUgEqyA/BR4FhgLXBdkquq6uZhrK9FLYWlJI2jYXXxHQncUVV3VtXDwCXASUNalyRpAUpVzf1Ck9cAx1XVG7vxM4AXVtWZfdMsB5Z3o4cBt87BqhcDP5qD5YyK9Y/WuNcP4/83WP9ojar+A6pqycTGYX0HlUnaHpeEVXUhcOGcrjRZVVVL53KZ88n6R2vc64fx/xusf7Raq39YXXxrgf36xvcF1g1pXZKkBWhYAXUdcEiSg5LsBJwKXDWkdUmSFqChdPFV1eYkZwJfo3eZ+UVVddMw1jXBnHYZjoD1j9a41w/j/zdY/2g1Vf9QLpKQJGm2vJOEJKlJBpQkqUljG1BJ1iT5bpIbkqzqa//TJLcmuSnJB0ZZ41Qmqz/JEUmu3dKW5MhR1zmVJLsluTzJLUn+I8mLk+yRZGWS27v33Udd5yAD6v9gN/6dJFck2W3UdQ4yWf19n52dpJIsHmWNUxlU/xgdw5PtP2NzDCc5rKtzy+snSd7a1DFcVWP5AtYAiye0vQz4R2DnbnzPUde5lfVfDRzfDZ8A/POo65zmb1gBvLEb3gnYDfgAcE7Xdg7w/lHXuZX1vwLYsWt7/7jV3w3vR+8Cpbsm7mMtvQZs/3E6hierf6yO4b6/ZQfgh8ABLR3DY3sGNcAfA+dX1c8BqmrjiOvZWgXs2g0/jYZ/O5ZkV+Ao4NMAVfVwVd1P75ZWK7rJVgAnj6K+6Qyqv6qurqrN3WTX0vsNX3Om2P4AFwB/zoQfx7dkivrH4hieov6xOYYnOAb4XlXdRUPH8DgHVAFXJ1nd3TYJ4FDgN5N8M8m/JPn1EdY3ncnqfyvwwST3AB8C3jGq4mbgYGAT8Jkk/57kU0kWAXtV1XqA7n3PURY5hUH193s98A/zX9qMTFp/khOBH1TVt0dc33QGbf9xOYYH1f9WxucY7ncqcHE33MwxPM4B9ZKqej5wPPCWJEfR+13X7sCLgD8DLksy2W2XWjBZ/X8MnFVV+wFn0f3fWaN2BJ4PfLyqngc8SK87YFxMWX+SdwKbgb8ZTXnTmqz+c4F3Au8eYV0zNWj7j8sxPKj+cTqGAehupnAi8MVR1zLR2AZUVa3r3jcCV9C7g/pa4EvV8y3gl/RufticAfUvA77UTfLFrq1Va4G1VfXNbvxyegfshiR7A3TvTXbRMLh+kiwDfgc4vbqO+AYNqv8g4NtJ1tDrnrw+yTNGU+KUBtU/LsfwoPrH6Rje4njg+qra0I03cwyPZUB1XRm7bBmm98X2jcCXgZd37YfS++KyuTsLT1H/OuC3usleDtw+mgqnV1U/BO5JcljXdAxwM71bWi3r2pYBV46gvGkNqj+9B22+HTixqh4aWYHTGFD/9VW1Z1UdWFUH0vtH9PndtE2ZYv/5MmNwDE9R/9gcw31O47HuPWjoGB7LO0kkOZjeWQf0TrW/UFXndaeqFwFHAA8DZ1fVP42mysGmqP+l9J5CvCPwn8CfVNXqEZU5rSRHAJ+i94/IncAf0fufnsuA/YG7gVOq6sejqnEqA+q/DtgZuLeb7NqqevNICpzGZPVX1X19n68BllZVc//Aw8Dt/yBjcAzDwPqfzXgdw08B7gEOrqr/17U9nUaO4bEMKEnSwjeWXXySpIXPgJIkNcmAkiQ1yYCSJDXJgJIkNcmAkiQ1yYCSJDXp/wNMxL1TqHPe2wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_hist(x, p=5):\n",
    "    # Plot the distribution and mark the mean\n",
    "    pyplot.hist(x, alpha=.5)\n",
    "    pyplot.axvline(x.mean())\n",
    "    # 95% confidence interval    \n",
    "    pyplot.axvline(numpy.percentile(x, p/2.), color='red', linewidth=3)\n",
    "    pyplot.axvline(numpy.percentile(x, 100-p/2.), color='red', linewidth=3)\n",
    "    \n",
    "def plot_dists(a, b, nbins, a_label='pop_A', b_label='pop_B', p=5):\n",
    "    # Create a single sequence of bins to be shared across both\n",
    "    # distribution plots for visualization consistency.\n",
    "    combined = pandas.concat([a, b])\n",
    "    breaks = numpy.linspace(\n",
    "        combined.min(), \n",
    "        combined.max(), \n",
    "        num=nbins+1)\n",
    "\n",
    "    pyplot.subplot(2, 1, 1)\n",
    "    plot_hist(a)\n",
    "    pyplot.title(a_label)\n",
    "    \n",
    "    pyplot.subplot(2, 1, 2)\n",
    "    plot_hist(b)\n",
    "    pyplot.title(b_label)\n",
    "    \n",
    "    pyplot.tight_layout()\n",
    "    \n",
    "\n",
    "plot_dists(male.childHeight, female.childHeight, 20, a_label='sons', b_label='daughters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the distributions of the heights of the sons and daughters overlap. But, are the means significantly different?\n",
    "\n",
    "### Bootstrap the Means\n",
    "\n",
    "Pandas has built-in support for generating bootstrap samples from a given dataframe. We'll use the `.sample()` method of our two dataframes to draw a single bootstrap sample like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>father</th>\n",
       "      <th>mother</th>\n",
       "      <th>midparentHeight</th>\n",
       "      <th>children</th>\n",
       "      <th>childNum</th>\n",
       "      <th>gender</th>\n",
       "      <th>childHeight</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>family</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>68.0</td>\n",
       "      <td>64.5</td>\n",
       "      <td>68.830</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>078</th>\n",
       "      <td>70.0</td>\n",
       "      <td>64.2</td>\n",
       "      <td>69.668</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>female</td>\n",
       "      <td>60.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>053</th>\n",
       "      <td>71.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>69.520</td>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>female</td>\n",
       "      <td>63.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>165</th>\n",
       "      <td>67.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>68.600</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>190</th>\n",
       "      <td>65.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>67.600</td>\n",
       "      <td>9</td>\n",
       "      <td>6</td>\n",
       "      <td>female</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        father  mother  midparentHeight  children  childNum  gender  \\\n",
       "family                                                                \n",
       "139       68.0    64.5           68.830         1         1  female   \n",
       "078       70.0    64.2           69.668         5         5  female   \n",
       "053       71.0    63.0           69.520         9         9  female   \n",
       "165       67.0    65.0           68.600         3         2  female   \n",
       "190       65.0    65.0           67.600         9         6  female   \n",
       "\n",
       "        childHeight  \n",
       "family               \n",
       "139            62.0  \n",
       "078            60.1  \n",
       "053            63.0  \n",
       "165            62.0  \n",
       "190            62.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "female.sample(frac=1, replace=True).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take the mean of our bootstrap sample in the same way as our full dataset dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "64.2055187637969"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "female.sample(frac=1, replace=True).childHeight.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll take many (`n_replicas`) bootstrap samples and plot the distribution of sample means as well as the mean of the sample means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZ+klEQVR4nO3de7CkdX3n8ffHAZFrQGdQGEYHtgZWtKLgLMHoEkrCimwirko5lrK4wSK7wQ0QXRei8bIJG9y11GxFk7BiZL1AiFqKxopOUGPcGHEGQWcYwVGQGRmd8Rbwsgr43T/6GaY5nMM5c/p096/Peb+quk73r59f9+fpc/mc5+mnu1NVSJLUmkeMO4AkSdOxoCRJTbKgJElNsqAkSU2yoCRJTbKgJElNsqAkSU2yoCRJTbKgJElNsqCkBZbkvyb5VpJ7ktya5LQk+yV5W5K7utPbkuzXLX9qku1JXplkZ5IdSf5D3+2dmeSW7va+leRV41s7aXQsKGkBJTkOeAXwr6rqYODZwB3Aa4CTgacCTwFOAl7bN/VxwC8BK4HzgLcnOay77krgt7vbezLwqaGviNQAC0paWPcD+wHHJ9m3qu6oqq8DLwH+W1XtrKpdwBuBc/rm3dtdf29VfRz4EXBc33XHJzmkqn5QVTeObnWk8bGgpAVUVVuBi4A3ADuTXJPkSOBI4Jt9i36zG9vte1V1X9/lnwAHdedfAJwJfDPJ3yd5+pDiS02xoKQFVlXvr6pnAk8ACngTcFd3ebfHd2Nzub0vVtVZwOHAh4FrFzSw1CgLSlpASY5L8qzuAIj/B/yU3m6/q4HXJlmRZDnwOuC9c7i9RyZ5SZJfqqp7gbu725MWvX3GHUBaZPYDLgeeSO+5o38Ezge+DxwCfLlb7q+BP5rjbZ4D/GmSZcCtwEsXMrDUqviBhZKkFrmLT5LUJAtKktQkC0qS1CQLSpLUpCaO4lu+fHmtXr163DEm38aNe84/7Wnjy6GBfWPXjwE4ZsWBY06iveLv4Lxs3Ljxu1W1Yup4E0fxrV27tjZs2DDuGJMv2XO+ge+r5u9Ff/F5AP7qt33TiIni7+C8JNlYVWunjruLT5LUJAtKktQkC0qS1KQmDpKQlrK3rr/tIWPbf/DTGa8bpotPP3ak9yc9HLegJElNsqAkSU2yoCRJTbKgJElNsqAkSU2yoCRJTbKgJElNsqAkSU3yhbpaskb9IlhJe8ctKElSkywoSVKTLChJUpMsKElSkywoSVKTLChJUpMsKElSkywoSVKTZi2oJKuSfDrJliSbk1zYjT86yfokX+u+HtY359IkW5PcmuTZw1wBSdLiNJctqPuAV1bVE4GTgQuSHA9cAlxfVWuA67vLdNetA54EnAG8I8myYYSXJC1esxZUVe2oqhu78/cAW4CVwFnAVd1iVwHP686fBVxTVT+rqtuBrcBJC5xbkrTI7dVzUElWAycAXwAeW1U7oFdiwOHdYiuBbX3TtndjkiTN2ZwLKslBwAeBi6rq7odbdJqxmub2zk+yIcmGXbt2zTWGJGmJmFNBJdmXXjm9r6o+1A1/J8kR3fVHADu78e3Aqr7pRwF3Tb3NqrqiqtZW1doVK1bMN78kaZGay1F8Aa4EtlTVW/quug44tzt/LvCRvvF1SfZLcjSwBrhh4SJLkpaCuXwe1DOAc4CvJLmpG/t94HLg2iTnAXcCZwNU1eYk1wK30DsC8IKqun+hg0uSFrdZC6qqPsf0zysBnDbDnMuAywbIJUla4nwnCUlSkywoSVKTLChJUpMsKElSkywoSVKTLChJUpPm8jooSUvEW9ffNu4ID7j49GPHHUFj5haUJKlJFpQkqUkWlCSpSRaUJKlJFpQkqUkWlCSpSRaUJKlJFpQkqUkWlCSpSRaUJKlJFpQkqUm+F59GqqX3epPUNregJElNsqAkSU2yoCRJTbKgJElNsqAkSU2yoCRJTbKgJElNmrWgkrwryc4km/rGHp1kfZKvdV8P67vu0iRbk9ya5NnDCi5JWtzmsgX1buCMKWOXANdX1Rrg+u4ySY4H1gFP6ua8I8myBUsrSVoyZi2oqvos8P0pw2cBV3XnrwKe1zd+TVX9rKpuB7YCJy1MVEnSUjLf56AeW1U7ALqvh3fjK4Ftfctt78YeIsn5STYk2bBr1655xpAkLVYLfZBEphmr6Rasqiuqam1VrV2xYsUCx5AkTbr5FtR3khwB0H3d2Y1vB1b1LXcUcNf840mSlqr5FtR1wLnd+XOBj/SNr0uyX5KjgTXADYNFlCQtRbN+3EaSq4FTgeVJtgOvBy4Hrk1yHnAncDZAVW1Oci1wC3AfcEFV3T+k7JKkRWzWgqqqF89w1WkzLH8ZcNkgoSRJ8p0kJElNsqAkSU2yoCRJTZr1OShJGoe3rr9t3BEecPHpx447wpLkFpQkqUkWlCSpSRaUJKlJFpQkqUkeJLEEtPRksyTNlVtQkqQmWVCSpCZZUJKkJllQkqQmWVCSpCZZUJKkJllQkqQmWVCSpCb5Ql1JmsVcX+x+8Tzm7K2l9M7qbkFJkppkQUmSmuQuviEZx/vfjWL3giSNiltQkqQmWVCSpCZZUJKkJllQkqQmDe0giSRnAH8CLAPeWVWXD+u+JGmpaOkAqGG/JmsoBZVkGfB24HRgO/DFJNdV1S3DuL/dWvrGSZIGM6xdfCcBW6vqG1X1c+Aa4Kwh3ZckaRFKVS38jSYvBM6oqpd3l88BfqWqXtG3zPnA+d3F44BbFzzIcCwHvjvuEPMwqblhcrNPam6Y3OyTmhsmN/tC5H5CVa2YOjis56AyzdiDmrCqrgCuGNL9D02SDVW1dtw59tak5obJzT6puWFys09qbpjc7MPMPaxdfNuBVX2XjwLuGtJ9SZIWoWEV1BeBNUmOTvJIYB1w3ZDuS5K0CA1lF19V3ZfkFcAn6B1m/q6q2jyM+xqDidst2ZnU3DC52Sc1N0xu9knNDZObfWi5h3KQhCRJg/KdJCRJTbKgJElNsqD6JDk0yQeSfDXJliRPT/LUJP+U5KYkG5KcNMPcC5NsSrI5yUUN5H5Kks8n+UqSjyY5ZIa5ZyS5NcnWJJeMMvcCZH9Xkp1JNk1K7iSrkny6m7M5yYUTkvtRSW5IcnOX+42jzD1I9r75y5J8KcnHJiV3kju6ZW5KsmGUuRcg+0Pm7nWAqvLUnYCrgJd35x8JHAp8EnhON3Ym8Jlp5j0Z2AQcQO/Ak78D1ow59xeBX+vGfgv4w2nmLQO+DhzTzbsZOL6Bx3zW7N11pwAnApsa+VmZy2N+BHBid/5g4LZRPuYD5A5wUHd+X+ALwMmT8Jj3zf894P3AxyYlN3AHsHyUeRcw+0Pm7vX9j2vFWzsBhwC30x040jf+CeBF3fkXA++fZu7Z9N4Qd/flPwBePebcd7PnIJhVwC3TzH068Im+y5cClzbwmM+avW/Z1Yy4oBYid9+cjwCnT1Juev+I3Ujv3WEm4jGn91rM64FnjbKgFiD32ApqwL8t087d25O7+PY4BtgF/GW3G+CdSQ4ELgL+Z5JtwJvp/RGfahNwSpLHJDmA3pbWqmmWG2XuTcBzu2XOniHPSmBb3+Xt3dioDJJ9nBYkd5LVwAn0tkZGYaDc3S6ym4CdwPqqGlVuGPwxfxvwauAXww46xaC5C/hkko3pvT3cKA2Sfaa5e8WC2mMferuL/qyqTgB+DFwC/Cfg4qpaBVwMXDl1YlVtAd4ErAf+lt6usvvGnPu3gAuSbKS3K+nn08yd9S2phmyQ7OM0cO4kBwEfBC6qqruHHxkYMHdV3V9VT6W3NXJSkiePJHXPvLMn+Q1gZ1VtHGHe3Qb9WXlGVZ0IPKdb/pQRZN5tkOwzzd0749h0bPEEPA64o+/yvwb+Bvhn9mzOBrh7Drf134HfGWfuKcscC9wwzdxx7+Kbd/a+61cz+l18A+Wm9xzOJ4Dfm6TcU5Z7PfCqScgO/DG9vQN3AN8GfgK8t/Xc09zWGyboMZ917lxObkF1qurbwLYkx3VDpwG30HsPwV/rxp4FfG26+UkO774+Hng+cPVQA3dmyt2X5xHAa4E/n2b6WN+SasDsYzNI7iShtxW+pareMqLIwMC5VyQ5tDu/P/DrwFdHkRsGy15Vl1bVUVW1mt7P+Keq6qWt505yYJKDd58H/g293WsjMeBjPtPf070O4WlPyz8V2AB8GfgwcBjwTGAjvd12XwCe1i17JPDxvrn/0H0DbgZOayD3hfSOELsNuJw9W4FTc5/ZLfN14DWNPOZzzX41sAO4l95/yOe1nrv7eapu3k3d6cwJyP3LwJe6eZuA103Sz0rfbZzK6I/im+9jfgy9vyc3A5sn8PfzIXP39v59qyNJUpPcxSdJapIFJUlqkgUlSWqSBSVJapIFJUlqkgUlSWqSBSVJapIFJUlqkgUlSWqSBSVJapIFJUlqkgUlSWqSBSU9jCTvTvJHQ76PNyR57zDvQ5pEFpTUuCR3JPn1ceeQRs2Ckhax9Ph7ronkD67UJ8kJSW5Mck+SvwIe1Y0fluRjSXYl+UF3/qi+eQ/aypm62y7Jv0/yzSTfS/IH02wVPTLJ/+nud3OStd289wCPBz6a5EdJXt2Nn5zkH5P8MMnNSU7tu6/PJLksyf+l9/HmxyR5WZJvdLd/e5KXDOPxkxaSBSV1uo+9/zDwHuDRwF8DL+iufgTwl8AT6BXGT4E/nePtHg+8A3gJcATwS8DKKYs9F7gGOBS4bvdtV9U5wJ3Ab1bVQVX1P5KsBP4G+KMu56uADyZZ0Xd75wDnAwcDu4D/BTynqg4GfpXeJ/lKTbOgpD1OBvYF3lZV91bVB4AvAlTV96rqg1X1k6q6B7gM+LU53u4LgY9W1eeq6ufA6+h97Hu/z1XVx6vqfnoF+ZSHub2X0vto7Y9X1S+qaj29j9Y+s2+Zd1fV5qq6D7gP+AXw5CT7V9WOqto8x+zS2FhQ0h5HAt+qqv7y+CZAkgOS/EW3m+5u4LPAoUmWzfF2t+2+UFU/Ab43ZZlv953/CfCoJPvMcHtPAM7udu/9MMkPgWfS2zrbrf/+fgy8CPiPwI4kf5PkX84htzRWFpS0xw5gZZL0jT2++/pK4DjgV6rqEOCUbnz3sj8GDuib97gpt9v/fNX+wGP2ItfUra1twHuq6tC+04FVdflMc6rqE1V1Or0S+yrwv/fi/qWxsKCkPT5Pb3fY7ybZJ8nzgZO66w6m97zTD5M8Gnj9lLk3AeuS7Nsd4PDCvus+APxmkl/tnud6I3uKbS6+AxzTd/m93e09O8myJI9Kcmr/QRv9kjw2yXOTHAj8DPgRcP9e3L80FhaU1OmeH3o+8DLgB/R2i32ou/ptwP7Ad4F/Av52yvQ/AP5FN++NwPv7bncz8J/pHQSxA7gH2EmvLObij4HXdrvzXlVV24CzgN+ndwDENuC/MPPv8yPobQHeBXyf3nNnvzPH+5bGJg/e3S5p2JIcBPwQWFNVt485jtQst6CkEUjym92BFgcCbwa+Atwx3lRS2ywoaTTOoreL7S5gDbCu3H0hPSx38UmSmuQWlCSpSTO9EHCkli9fXqtXrx53jMVp48Y955/2tPHl0Iy+sevHAByz4sAxJ9Gs/H0aio0bN363qlZMHW9iF9/atWtrw4YN446xOPW/5rSB77Ue6kV/8XkA/uq3nz7mJJqVv09DkWRjVa2dOu4uPklSkywoSVKTLChJUpOaOEhCGoe3rr9t3BEkPQy3oCRJTbKgJElNsqAkSU3yOShpzLb/4KdAG8+JXXz6seOOID3ALShJUpMsKElSkywoSVKTLChJUpMsKElSkywoSVKTZi2oJKuSfDrJliSbk1zYjT86yfokX+u+HtY359IkW5PcmuTZw1wBSdLiNJctqPuAV1bVE4GTgQuSHA9cAlxfVWuA67vLdNetA54EnAG8I8myYYSXJC1esxZUVe2oqhu78/cAW4CVwFnAVd1iVwHP686fBVxTVT+rqtuBrcBJC5xbkrTI7dVzUElWAycAXwAeW1U7oFdiwOHdYiuBbX3TtndjkiTN2ZwLKslBwAeBi6rq7odbdJqxh3w2cpLzk2xIsmHXrl1zjSFJWiLmVFBJ9qVXTu+rqg91w99JckR3/RHAzm58O7Cqb/pRwF1Tb7OqrqiqtVW1dsWKFfPNL0lapOZyFF+AK4EtVfWWvquuA87tzp8LfKRvfF2S/ZIcDawBbli4yJKkpWAu72b+DOAc4CtJburGfh+4HLg2yXnAncDZAFW1Ocm1wC30jgC8oKruX+jgkqTFbdaCqqrPMf3zSgCnzTDnMuCyAXJJkpY430lCktQkP7BQI9XCh/JJmgxuQUmSmmRBSZKaZEFJkppkQUmSmmRBSZKaZEFJkppkQUmSmuTroCQ9oKXXqV18+rHjjqAxcwtKktQkC0qS1CQLSpLUJAtKktQkC0qS1CQLSpLUJAtKktQkC0qS1CQLSpLUJAtKktQkC0qS1CQLSpLUJAtKktQkC0qS1CQLSpLUJAtKktQkC0qS1CQLSpLUJAtKktSkWQsqybuS7EyyqW/s0UnWJ/la9/WwvusuTbI1ya1Jnj2s4JKkxW0uW1DvBs6YMnYJcH1VrQGu7y6T5HhgHfCkbs47kixbsLSSpCVj1oKqqs8C358yfBZwVXf+KuB5fePXVNXPqup2YCtw0sJElSQtJfN9DuqxVbUDoPt6eDe+EtjWt9z2buwhkpyfZEOSDbt27ZpnDEnSYrXQB0lkmrGabsGquqKq1lbV2hUrVixwDEnSpJtvQX0nyREA3ded3fh2YFXfckcBd80/niRpqZpvQV0HnNudPxf4SN/4uiT7JTkaWAPcMFhESdJStM9sCyS5GjgVWJ5kO/B64HLg2iTnAXcCZwNU1eYk1wK3APcBF1TV/UPKLklaxGYtqKp68QxXnTbD8pcBlw0SSpIk30lCktQkC0qS1CQLSpLUJAtKktQkC0qS1CQLSpLUpFkPM5ekcXjr+tvGHeEBF59+7LgjLEkW1BLS0i+8JM3GXXySpCZZUJKkJllQkqQmWVCSpCZZUJKkJllQkqQmWVCSpCZZUJKkJllQkqQmWVCSpCZZUJKkJllQkqQmWVCSpCZZUJKkJllQkqQmWVCSpCb5gYVD0sqHA1487gDSIrD79/niacZGbSl9uq9bUJKkJllQkqQmWVCSpCZZUJKkJg3tIIkkZwB/AiwD3llVlw/rviRpqWjlACwY/gEbQ9mCSrIMeDvwHOB44MVJjh/GfUmSFqdhbUGdBGytqm8AJLkGOAu4ZUj3B7T1n4UkaTCpqoW/0eSFwBlV9fLu8jnAr1TVK/qWOR84v7t4HHDrggcZvuXAd8cdYkCTvg6Tnh9chxZMen6Y7HV4QlWtmDo4rC2oTDP2oCasqiuAK4Z0/yORZENVrR13jkFM+jpMen5wHVow6flhcazDVMM6im87sKrv8lHAXUO6L0nSIjSsgvoisCbJ0UkeCawDrhvSfUmSFqGh7OKrqvuSvAL4BL3DzN9VVZuHcV9jNtG7KDuTvg6Tnh9chxZMen5YHOvwIEM5SEKSpEH5ThKSpCZZUJKkJllQM0hyaJIPJPlqki1Jnp7kD5N8OclNST6Z5MgZ5l6cZHOSTUmuTvKoCct/YZd9c5KLRhy9P8dD1qHvulclqSTLZ5h7RpJbk2xNcsnoUj8kxyDr8K4kO5NsGl3ih2SYV/4kq5J8upuzOcmFo03+oCzzXYdHJbkhyc3dOrxxtMkfyDHvn6FumWVJvpTkY6NJvICqytM0J+Aq4OXd+UcChwKH9F3/u8CfTzNvJXA7sH93+VrgZROU/8nAJuAAegfR/B2wppXvQXd+Fb0DcL4JLJ9m3jLg68Ax3bybgeMnaR26ZU4BTgQ2jSP7gN+DI4ATu/MHA7dN2veA3us5D+rO7wt8ATh5UvL3zf894P3Ax8b1czTfk1tQ00hyCL0/DlcCVNXPq+qHVXV332IHMuXFx332AfZPsg+9P/QjfQ3YgPmfCPxTVf2kqu4D/h74d8POPNVM69Bd/Vbg1cz8+D/wVltV9XNg91ttjdSA60BVfRb4/pBjzmiQ/FW1o6pu7M7fA2yh98/bSA24DlVVP+ou7tudRnpU2aA/Q0mOAv4t8M7hJh0OC2p6xwC7gL/sNo3fmeRAgCSXJdkGvAR43dSJVfUt4M3AncAO4J+r6pOjiw4MkJ/e1tMpSR6T5ADgTB78outRmXYdkjwX+FZV3fwwc1cC2/oub2cMfxwZbB1asCD5k6wGTqC3BTJqA61Dt3vsJmAnsL6qRr0Og34P3kavxH4x5JxDYUFNbx96u1b+rKpOAH4MXAJQVa+pqlXA+4BXTJ2Y5DB6/60fDRwJHJjkpaMK3pl3/qraArwJWA/8Lb3dY/eNKHe/6dbhDcBrmL5Y+836VlsjMsg6tGDg/EkOAj4IXDRlC35UBlqHqrq/qp5K791wTkry5OFFnda88yf5DWBnVW0cdshhsaCmtx3Y3vff0gfo/ZD0ez/wgmnm/jpwe1Xtqqp7gQ8Bvzq0pNMbJD9VdWVVnVhVp9DbxfS1oSWd2UzrcDRwc5I76P3RuDHJ46aZ28JbbQ2yDi0YKH+SfemV0/uq6kOjifwQC/I96HarfQY4Y5hhpzFI/mcAz+2WuQZ4VpL3jiT1ArGgplFV3wa2JTmuGzoNuCXJmr7Fngt8dZrpdwInJzkgSbq5W4YaeIoB85Pk8O7r44HnA1cPMe60ZliHG6vq8KpaXVWr6f3yntgt26+Jt9oacB3GbpD83c/+lcCWqnrLKHP3G3AdViQ5tDu/P71/Pqf9nRmWQfJX1aVVdVS3zDrgU1U16r05gxn1URmTcgKeCmwAvgx8GDiM3n+Dm7qxjwIru2WPBD7eN/eN9H6QNwHvAfabsPz/QO+zu24GTmvpezDl+jvojl6aZh3OpHfk2NeB10zoOlxN73nMe+n9ETpvUvIDz6S3W/XLwE3d6cxJ+h4Avwx8qZu3CXjdJOWfssypTOBRfL7VkSSpSe7ikyQ1yYKSJDXJgpIkNcmCkiQ1yYKSJDXJgpIkNcmCkiQ16f8DYqAMyvQNFiQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_replicas = 1000\n",
    "\n",
    "female_bootstrap_means = pandas.Series([\n",
    "    female.sample(frac=1, replace=True).childHeight.mean()\n",
    "    for i in range(n_replicas)])\n",
    "\n",
    "male_bootstrap_means = pandas.Series([\n",
    "        male.sample(frac=1, replace=True).childHeight.mean()\n",
    "        for i in range(n_replicas)])\n",
    "\n",
    "plot_dists(male_bootstrap_means, female_bootstrap_means, \n",
    "           nbins=80, a_label='sons', b_label='daughters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distribution of the bootstrap means do not overlap at all. Evidently, the **difference is significant**. \n",
    "\n",
    "## Your Turn 1\n",
    "Compute and plot the bootstrap distribution of the mean of the heights of the fathers for male children. Compare this estimate to the bootstrap estimate of the mean of the heights of the sons (already computed). Are these means significantly different? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the ???\n",
    "father_bootstrap_means = pandas.Series([\n",
    "    ???.sample(frac=1, replace=True).???.mean()\n",
    "    for i in range(n_replicas)])\n",
    "\n",
    "#Replace the ???\n",
    "plot_dists(???, ???, \n",
    "           nbins=80, a_label='???', b_label='???')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bootstrap Difference in Means\n",
    "\n",
    "In the previous example you computed and compared two bootstrap distributions of the means from the height data. It is actually quite easy to bootstrap a statistic such as the difference in means. \n",
    "\n",
    "In the cell below, we generate bootstrap samples from the full male+female dataset and then difference in the means of male and female `childHeight` for each sample, generating a distribution of sample mean `diffs`. \n",
    "\n",
    "Run the code and examine the plotted results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPI0lEQVR4nO3df4xlZ13H8ffHLjYKVYqd1rK7upXsqm0iRcaiqcZCU1rwx5YEcFFLJTWLsVVoSKTlDyExm2AiFI0WXaChRKBspLWrwcJSf1TCjzJba9vt0rppN+2wm+4AjRA0Nbv9+sc9TS/bmb135t47s/PM+5VM7rnPec493ydz9zPnPvecs6kqJElt+b6VLkCSNH6GuyQ1yHCXpAYZ7pLUIMNdkhq0bqULADjjjDNq06ZNK13G2rZ377PLL3/5ytXRqEfmvgvAT0w9f4UrWSPWyPt5796936iqqfnW5WQ4FXJ6erpmZmZWuoy1LXl2+SR4T7TmN/7mSwB86q2/sMKVrBFr5P2cZG9VTc+3zmkZSWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lq0Elx+wHpZHbDnodHfo3ZJ/930a917SVbRt6v1i6P3CWpQYa7JDXIcJekBhnuktSggeGeZGOSf0myP8m+JG/r2t+T5OtJ7u1+Xtu3zfVJDiR5KMmlkxyAJOm5hjlb5ijwjqq6J8lpwN4ke7p1N1TVn/V3TnIusA04D3gx8PkkW6rq2DgLlyQtbOCRe1Udrqp7uuXvAPuB9SfYZCtwS1U9VVWPAgeAC8ZRrCRpOIuac0+yCXgZ8JWu6Zok9yW5KcnpXdt64PG+zWaZ549Bku1JZpLMzM3NLb5ySdKChg73JC8APg28vaq+DXwQeAlwPnAYeN8zXefZ/Dn/z1VV7ayq6aqanpqa978AlCQt0VDhnuR59IL941V1K0BVPVFVx6rqaeBDPDv1Mgts7Nt8A3BofCVLkgYZ5myZAB8B9lfV+/vaz+7r9jrggW55N7AtyalJzgE2A3ePr2RJ0iDDnC1zIXAFcH+Se7u2dwFvSnI+vSmXg8BbAapqX5JdwIP0zrS52jNlJGl5DQz3qvoC88+jf+YE2+wAdoxQlyRpBF6hKkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSg9YN6pBkI/Ax4EeBp4GdVfXnSV4EfArYBBwE3lhVT3bbXA9cBRwD/rCqPjuR6qWG3bDn4RXb97WXbFmxfWs8BoY7cBR4R1Xdk+Q0YG+SPcDvAHdW1XuTXAdcB7wzybnANuA84MXA55NsqapjkxmC1oKVDDppNRo4LVNVh6vqnm75O8B+YD2wFbi563YzcHm3vBW4paqeqqpHgQPABWOuW5J0Aouac0+yCXgZ8BXgrKo6DL0/AMCZXbf1wON9m812bce/1vYkM0lm5ubmllC6JGkhQ4d7khcAnwbeXlXfPlHXedrqOQ1VO6tquqqmp6amhi1DkjSEocI9yfPoBfvHq+rWrvmJJGd3688GjnTts8DGvs03AIfGU64kaRgDwz1JgI8A+6vq/X2rdgNXdstXArf3tW9LcmqSc4DNwN3jK1mSNMgwZ8tcCFwB3J/k3q7tXcB7gV1JrgIeA94AUFX7kuwCHqR3ps3VnikjSctrYLhX1ReYfx4d4OIFttkB7BihLknSCLxCVZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBg0M9yQ3JTmS5IG+tvck+XqSe7uf1/atuz7JgSQPJbl0UoVLkhY2zJH7R4HL5mm/oarO734+A5DkXGAbcF63zY1JThlXsZKk4QwM96q6C/jWkK+3Fbilqp6qqkeBA8AFI9QnSVqCUebcr0lyXzdtc3rXth54vK/PbNf2HEm2J5lJMjM3NzdCGZKk4y013D8IvAQ4HzgMvK9rzzx9a74XqKqdVTVdVdNTU1NLLEOSNJ8lhXtVPVFVx6rqaeBDPDv1Mgts7Ou6ATg0WomSpMVaUrgnObvv6euAZ86k2Q1sS3JqknOAzcDdo5UoSVqsdYM6JPkkcBFwRpJZ4N3ARUnOpzflchB4K0BV7UuyC3gQOApcXVXHJlK5JGlBA8O9qt40T/NHTtB/B7BjlKIkSaPxClVJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaNPB+7pLWnhv2PLwi+732ki0rst8WeeQuSQ0y3CWpQU7LaFFW6uO6pMXxyF2SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWrQwHBPclOSI0ke6Gt7UZI9Sf6rezy9b931SQ4keSjJpZMqXJK0sGGO3D8KXHZc23XAnVW1Gbize06Sc4FtwHndNjcmOWVs1UqShjIw3KvqLuBbxzVvBW7ulm8GLu9rv6WqnqqqR4EDwAXjKVWSNKylzrmfVVWHAbrHM7v29cDjff1mu7bnSLI9yUySmbm5uSWWIUmaz7i/UM08bTVfx6raWVXTVTU9NTU15jIkaW1barg/keRsgO7xSNc+C2zs67cBOLT08iRJS7HUcN8NXNktXwnc3te+LcmpSc4BNgN3j1aiJGmxBv43e0k+CVwEnJFkFng38F5gV5KrgMeANwBU1b4ku4AHgaPA1VV1bEK1S5IWMDDcq+pNC6y6eIH+O4AdoxQlSRqNV6hKUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktSgdaNsnOQg8B3gGHC0qqaTvAj4FLAJOAi8saqeHK1MSdJijOPI/ZVVdX5VTXfPrwPurKrNwJ3dc0nSMprEtMxW4OZu+Wbg8gnsQ5J0AqOGewGfS7I3yfau7ayqOgzQPZ4534ZJtieZSTIzNzc3YhmSpH4jzbkDF1bVoSRnAnuSfG3YDatqJ7ATYHp6ukasQ5LUZ6Qj96o61D0eAW4DLgCeSHI2QPd4ZNQiJUmLs+RwT/L8JKc9swy8GngA2A1c2XW7Erh91CIlSYszyrTMWcBtSZ55nU9U1R1JvgrsSnIV8BjwhtHLlCQtxpLDvaoeAV46T/s3gYtHKUqSNJpRv1CVpLG5Yc/DY3mdaxf5mtdesmUs+z2ZePsBSWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGuT93Fehcd3zut9i738t6eTmkbskNchwl6QGGe6S1CDDXZIaZLhLUoM8W0bSmreSZ4hde8mWibyuR+6S1CDDXZIaZLhLUoMMd0lqkOEuSQ3ybJkReA8WSScrj9wlqUGGuyQ1aGLhnuSyJA8lOZDkukntR5L0XBOZc09yCvBXwCXALPDVJLur6sFJ7M+5b0n6XpM6cr8AOFBVj1TV/wG3AFsntC9J0nFSVeN/0eT1wGVV9bvd8yuAV1TVNX19tgPbu6c/CTw09kKW7gzgGytdxAQ5vtXN8a1u4xzfj1fV1HwrJnUqZOZp+56/IlW1E9g5of2PJMlMVU2vdB2T4vhWN8e3ui3X+CY1LTMLbOx7vgE4NKF9SZKOM6lw/yqwOck5Sb4f2AbsntC+JEnHmci0TFUdTXIN8FngFOCmqto3iX1NyEk5XTRGjm91c3yr27KMbyJfqEqSVpZXqEpSgwx3SWrQmg/3JKck+Y8k/7jA+ouS3JtkX5J/W+76RnWi8SX54ST/kOQ/u/G9ZSVqXKokB5Pc3/1+ZuZZnyR/0d0C474kP7sSdS7VEOP7rW5c9yX5YpKXrkSdSzVofH39fi7Jse76mVVhmLFNOlu85S+8DdgP/NDxK5K8ELiR3gVZjyU5c5lrG4cFxwdcDTxYVb+WZAp4KMnHu6uKV4tXVtVCF4S8Btjc/bwC+GD3uJqcaHyPAr9cVU8meQ29L+paGt8ztzL5U3onZ6w2C45tObJlTR+5J9kA/Arw4QW6/CZwa1U9BlBVR5artnEYYnwFnJYkwAuAbwFHl6m85bAV+Fj1fBl4YZKzV7qocamqL1bVk93TL9O7nqQ1fwB8GlhV//aGMPFsWdPhDnwA+CPg6QXWbwFOT/KvSfYmefOyVTYeH+DE4/tL4KfpXWB2P/C2qlqo78mogM91v5vt86xfDzze93y2a1stBo2v31XAPy1DTeN0wvElWQ+8DvjrZa9sdIN+dxPPljU7LZPkV4EjVbU3yUULdFsHvBy4GPgB4EtJvlxVJ/1tKIcc36XAvcCrgJcAe5L8e1V9e1mKHN2FVXWo+0i7J8nXququvvUDb4Nxkhs0PgCSvJJeuP/islc4mkHj+wDwzqo61vtwuaoMGtvEs2UtH7lfCPx6koP07lr5qiR/e1yfWeCOqvpuN3d2F7BavrQaZnxvoffRsKrqAL053J9a3jKXrqoOdY9HgNvo3Y2036q+DcYQ4yPJz9CbdttaVd9c3gpHM8T4poFbuvfw64Ebk1y+nDUu1ZDvzYlmy5oN96q6vqo2VNUmerdH+Oeq+u3jut0O/FKSdUl+kN6XVfuXudQlGXJ8j9E7ciDJWfTuzvnIsha6REmen+S0Z5aBVwMPHNdtN/Dm7qyZnwf+u6oOL3OpSzLM+JL8GHArcMVq+DTZb5jxVdU5VbWpew//HfD7VfX3y13rYg353px4tqzZaZmFJPk9gKr666ran+QO4D5689Yfrqrjf0mrSv/4gD8BPprkfnpTGO880ZkLJ5mzgNu6j+vrgE9U1R3Hje8zwGuBA8D/0PuksloMM74/Bn6E3hEtwNFVdDfFYca3Wg0c23Jki7cfkKQGrdlpGUlqmeEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGvT/5j2LEu3nlG8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "diffs = []\n",
    "for i in range(n_replicas):\n",
    "    sample = families.sample(frac=1.0, replace=True)\n",
    "    male_sample_mean = sample[sample.gender == 'male'].childHeight.mean()\n",
    "    female_sample_mean = sample[sample.gender == 'female'].childHeight.mean()\n",
    "    diffs.append(male_sample_mean - female_sample_mean)\n",
    "diffs = pandas.Series(diffs)\n",
    "\n",
    "plot_hist(diffs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distribution of the **difference in means is far from zero**. As before, we can infer that the means of the two populations are **significantly different**. \n",
    "\n",
    "But, is the distribution of the difference in means normal, as implied by the CLT? Run the code in the cell below to create a Q-Q normal plot of the bootstrap distribution of the difference in means and examine the results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Quantiles of standard Normal vs. bookstrapped mean')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA6P0lEQVR4nO3deZxV8x/H8de7aV+IyhIqtFBkC9mj7Pu+RQpRP3spRNqLsvslSaJGtmQJyfKrbPkp2otfKBFaiFS0fX5/nHPrzsy9M3ea7p25dz7Px2Mec+/Z7ufce875nO/3e873yMxwzjlXepUp7gCcc84VL08EzjlXynkicM65Us4TgXPOlXKeCJxzrpTzROCcc6WcJ4Ikk/SXpL3C1yMk9Unx5/eRtFzSL6n83IJImijpmm24vIWSWm2r5aWSpB6SRqXw8+pJMkllk7Bsk1R/Wy+3JEn175UKGZcIJF0laZakNZJ+kTRY0vYp+uw8Bzczq2pm36Xi82PEswfQCWhsZrsUct60PbDmFiZgk3RY1LD6kvwmmhKgOE6QXE4ZlQgkdQLuA24HtgeaA/WACZLKFWNoxaUusMLMlhZ3INtKEc5ifwO2ycEmGWfSLj7/vpMvYxKBpO2AnsCNZjbezNab2ULgImBP4LJwuhxnH5JaSPox6v0dkr6VtErSXEnnRo27StLHkgZJ+l3S95JODcf1BY4BHg+rgx4Ph8ctKks6Q9J0SSslfSqpadS4rpJ+CuP4WlLLOMvYXtJzkpZJWiTpbkllwrP594DaYTwjYsxbU9K48PN/k/RROO9IoA7wZjhvl3D6l8NS1h+SJktqErWsEZL+LemtMObPJe0dNf5ESfPDeR8HFDVub0kfSloRVmNlS6oeNX5h+H3MBFZLKivpinB9V0jqFuu7yeVZoKmk4+J8j7UlvRF+DwskXRs1roekVySNkvQncFVY+usT/m5/SXpTUo0w9j8lfSGpXtQyHpG0OBw3TdIxCcSMpHmSzoh6Xzb8jg6WVDGMaUX4G34haedElhtqJ2mJpJ/Dk6jIZ1SQ9HA4bkn4ukLU+GvD7+i38DurHSf2o8N1Pl6BhyQtDbeBmZL2k9QeuBzoEvkew3lj/eYF7ZufSHosXP58Re0z4e/VX9J/w/GvS9oxanzz8LdcKWmGpBZR4/aUNCn83PeAmvG+UIXHE0ldwnX9WdI5kk6T9E34nd0VNX2ZqPVaIemlXHFt9T5XKGaWEX/AKcAGoGyMcc8C2eHrEUCfqHEtgB+j3l8I1CZIkhcDq4Fdw3FXAeuBa4EsoAOwBFA4fiJwTa7PNqB+7s8GDgaWAoeHy2oDLAQqAI2AxUDtcNp6wN5x1vs54HWgWjjdN8DVsdYtxrz9gSFAufDvmKh1WQi0yjV9u/BzKgAPA9Ojxo0gOOs+DCgLZAMvhONqAn8CF4Sfc2v4W10Tjq8PnBgutxYwGXg4atkLgenAHkAloDHwF3BsOM+D4fJaxVnPEQSlgZuAj6M+06KmmQQMBioCBwLLgJbhuB7h734OwXZRKfytFwB7E5Q+54bffatw/Z8DnolafmugRjiuE/ALUDFq+aPixN6dcNsN358OzA9fXwe8CVQm2IYOAbZLYF+pR7BdjgaqAPuH69sqHN8LmALsFP4enwK9w3EnAMsJtt8KwGPA5NzbO3AywTZ8WDj8ZGAaUJ3gJGBftuxXI4jaJ2P95gnumxsItq1y4fg/gB2j9s2fgP3CdR4T+c6B3YAVwGnhsk8M39cKx39GsI1VINjmVuXze7UI4+gexnFt+N0+T7DvNAH+BvYKp78l/K53D5f/JDC6qPtcoY+fyTowp/qPYEf7Jc64AcCEWBsdBR8spwNnR21sC6LGVQ43/F2iNrZEE8EThDtX1LRfA8cR7EhLCQ4q5fKJLQv4h6ANIDLsOmBiguvWiyCJ1I8xbiFxDqzh+Orhum0ftW7DosafxpYD1pXAlKhxAn7M/V1FjT8H+CpXLO2i3neP3uAJdux18eJlSyKoAPwAnEpUIiA42GwEqkXN0x8YEb7uQdTBLuq37hb1/gHgnaj3ZxK108aI6XfggKjlxzuw1Cc48FQO32cD3cPX7QgO0k0Lua/UC3+7faKG3Q88Hb7+FjgtatzJwMLw9dPA/VHjqhIkyXpR2/udwCJg/6jpTiBIlM2BMrF+nxjbX7sC1mM6OffNzSdl4bD/AldE/V4DosY1DreZLKArMDLXst8lODmrQ3BgrxI17vl8fq8WwFogK3xfLfxODo+aZhpwTvh6HuEJR/h+1/D7jHVCW50E97nC/mVM1RDBWUpNxa5P3JUgKxdI0pXaUl2zkuAMIroouPnqGzNbE76suhXx1gU6RT4n/Kw9CEoBCwjOFHoASyW9EKf4XRMoT7DTRSwiOMNJxECCs9oJkr6TdEe8CSVlSRoQFmH/JNhRIzFERF+ZtIYt30ttgrNDYPPRd/N7STuF6/hTuOxR5C1+L456nXt5qwnO4PJlZv8AvcM/RY2qDfxmZquihuX+HqM/P+LXqNdrY7zfvF1I6hRW8/wR/tbbk08VQ1TMCwgOFmdKqgycRXAgAhhJcMB6IazCuV+FawuLXqdFBN8D4f/c21TMcWb2F8F3H/1d3QK8ZGazoqb7EHgc+Dfwq6ShCqpzE40vkX3zp0hmjxF37uUtIjhjr0mwL16Ya188muC4URv4PdzGoufNzwoz2xi+Xhv+j7dt1AXGRn3uPIKTkp2LuM8VSiYlgs8Izo7Pix4oqQrBGeCkcNBqgjP5iF2ipq0LPAXcANQws+rAbHIeNPJjBU+y2WKgr5lVj/qrbGajAczseTM7mmBDMYJG8NyWE5w91I0aVoegCFxwsGarzKyTme1FcAZ7W1S9au51uQw4m6CUsj3BWSUk9t38TJDkghkkRb8nOPs2gjPb7QhKd7mXGx1P7uVVJqh2ScQzBPGfGzVsCbCjpGpRw3J/j4X5bXNQ0B7QlaC9aodwu/qDxLer0cClBN//3DA5YEE7WE8zawwcCZxBUPpKVPRvUIfgeyD8n3ubijku3L9qkPO7uhA4R9It0R9mZo+a2SEE1SMNCS7qgPjf7ebhCe6bu4XbVqy4Ie/6rifYhxYTlAii98UqZjaAYFvbIVzP6Hm3lcXAqbk+u6KZ/UTR9rlCyZhEYGZ/EDQWPybpFEnlFDTWvUzwY2eHk04HTpO0o6RdCM5eIqoQbHzLACS1JTjrSNSvwF4JTvsUcL2kwxWoIul0SdUkNZJ0goIGur8JziA25l5AeNbxEtA3nK8ucBvBGXWBFDRW1w93nj/Dz4h8Tu51qUaQaFcQJNJ+Ca4nwFtAE0nnhSW2m4hKwOGy/wJWStqNLQeIeF4BzlDQGFmeoIoroW3ZzDYQlLS6Rg1bTFDF0l9BA2xT4Gq2bDNFVY2gemEZUFZSd6Cgs+FoLwAnEbRJRUoDKGiE3V9SFsHvt54Y20k+7pFUOWyAbAu8GA4fDdwtqZakmgRVcZFt6nmgraQDw+2zH/C5BRdmRCwBWgI3SeoYxnpouK2XIzgZ+5v421osieybO4WfWU7ShQTtEG9HjW8tqXF44tALeCXch0YRlLhODs/CKypo9N3dzBYBU4GekspLOprgpGlbGUKw/9YN16uWpLPDcUXZ5wolYxIBgJndD9wFDCKoV/2e4AtsFVW0GwnMIChmTWDLxo+ZzSWo6/2MYOPcH/ikECE8Alyg4IqiRwuIdSpBQ9LjBPXFCwjqOSGoyx5AkMB+IdjA78q7FABuJNixvgM+JthRhycYbwPgfYKD8GfAYDObGI7rT3AwWCmpM0Hj5yKCM7+5BA1cCTGz5QRniQMINuoG5PxeexI0Pv5BkDReLWB5c4B/EazrzwTf34/5zZPL6HC+aJcSnHEtAcYC95rZe4VYZn7eBd4hqCNfRHAQjFXVFJOZ/Uzw+xxJ1PZKkExfIUgC8whKvaMAJA2RNKSARU8i2O4+AAaZ2YRweB+Cg99MYBbwZTgMM/sAuIegsfVngsbyS2LE/ANBMuiq4N6a7QhOfn4Pv4MVBPspBO0OjcNt7bU430Ei++bnBNvWcqAvcIGZRVcZjiSoV/+F4KKAm8JlLyY4876LINEsJjgZiRwfLyO4qOM34F6CfWFbeQR4g6B6dhXBfnV4OG6r97nCilwhkpEktSM4yBwVbpjOuQwk6SqCiw+OjjN+IkED77BUxpUuMvpGDTMbLmk9wZmUJwLnnIshoxMBgJmNLO4YnHOuJMvoqiHnnHMFy6jGYuecc4WXdlVDNWvWtHr16hV3GM45l1amTZu23MxqxRqXdomgXr16TJ06tbjDcM65tCIp7h3RXjXknHOlnCcC55wr5TwROOdcKeeJwDnnSjlPBM45V8p5InDOuVLOE4FzzpVyngiccy4TLF0Kt94KKwp8WF8engiccy6drV0L/ftD/frw2GMwcWKhF+GJwDnn0tGmTTBqFDRqBHfdBccfD7Nnw/nnF3pRngiccy7dTJoEhx0GV1wBtWrBhx+SfdHr1DtlH8qUgXr1ILsQD1r1ROCcc+ni66/hnHOgRQv49Vd47jn44guylxxP+/awaBGYBf/bt088GXgicM65km75crjxRthvP/jgA+jbN0gKV1wBZcrQrRusWZNzljVroFu3xBaf1N5HJS0keIj8RmCDmTWLMU0L4GGgHLDczI5LZkzOOZcuRj/zNwtve5SOK/tSlb/4tmV7Gmb3gJ13zjHdD3EexBtveG6pKBEcb2YHxkkC1YHBwFlm1gS4MAXxOOdciZGdHdTp56jbN+Pjf43myKv34c6VXfmIY9ifWRz02RNkv79znmXUqRN72fGG51bcVUOXAa+a2Q8AZra0mONxzrmkyX3Q79iRPHX7z1z9McsbNOfowZfxu1WnJe9zJuOYR+O41T19+0LlyjmHVa4cDE9EshOBARMkTZPUPsb4hsAOkiaG01wZayGS2kuaKmnqsmXLkhqwc85trVgH+sj7mjWhXbucB/0hQ7bU7e/NAl7hfN7/5xg2LPyRtjzDIUzjQ1rm+IxY1T2XXw5Dh0LduiAF/4cODYYnIqkPr5dU28yWSNoJeA+40cwmR41/HGgGtAQqAZ8Bp5vZN/GW2axZM/MnlDnnSprs7ODsPnejbUF2ZAX30JuODGYd5bmPrjzEbdSsW4VFMZ4pVrcuLFxY+PgkTYtVRQ9JLhGY2ZLw/1JgLHBYrkl+BMab2WozWw5MBg5IZkzOObetZWdDmzaFSwLl+YfbeIAF1OdGHmMEV1GfBfThHmrWrVLk6p7CSFoikFRFUrXIa+AkYHauyV4HjpFUVlJl4HBgXrJics65bS1SEti4MdE5jAt5iXnsywN0ZgrNOYAZXMdQfmWXzQf7olb3FEYyLx/dGRgrKfI5z5vZeEnXA5jZEDObJ2k8MBPYBAwzs9zJwjnnSpzs7KDhNlb1TTzN+YwHuY0jmMJM9uesCu+ye7uT+Ott0A/BVT6RJADB/2Qc+HNLWiIws++IUc1jZkNyvR8IDExWHM45t60Vtj1gT75jAHdwES/zq3bhGhvGh3Wuone/rJQc6AuS1BvKnHMu00TaAwqqCsrKgpvb/M4+L/ehzarH2KByzDznXpo+15lhVaumJtgEeSJwzrkEJdoesH2ldbx//mCaje0Ff62Edm0p36sXTXfbLSVxFlZx31DmnHNpIbErg4zrao5hcbXGNBt1KxxyCHz1FTz9NJTQJABeInDOuQIlUhI4tsLnvFy3Ezt98wk0bgwj3oZTTgku+SnhvETgnHP5KKgkUI/vGa1LmfRPc3Za+T948kmYMQNOPTUtkgB4icA55+LKrySwPSu5i37czCNklcuCLndDly5QrVrqAy0iTwTOORdDvKuDyrKe6xnCvfRkR37j+2OuZO/n+8DuuxdPoNuAVw055xzBgb9mzaA2R4LWrXMnAeNsXmMOTXiMm5hd5gDG95nG3pNHpHUSAE8EzrlSLPrg37o1rFgRe7pDmMpEWvAa57KRLM4u8yY/Pfs+p3U7KLUBJ4knAudcqZSdDW3bxj/4A+zBD4ykNVM5lH2ZRwcGc3ilWVz03Blc3jo9GoIT4W0EzrlSJzsbrrwSNm2KPb4af3In/bmVhzBEP+5kAHewJms7nn0qNf3/pJInAudcqdKxIzzxROxxZVnPtTxFD3qwE8sYSWu60ZfF1EGCkc9mXhIATwTOuVIkfhIwzmAc99OFfZnPJI7lNN5mGsFzXCS4/vrMTALgbQTOuVIgOxuqVo2dBA7kKz6gJW9yFmXYxNm8Rgsm8lWZIAnUrQsjR8LgwSkOOoU8ETjnMlLuK4JWr845fncWM4I2TOMQmjKTG3iMpprNRaPOxkxs3Bg8W3jhwswtCUR41ZBzLuNErghavz7vuKqsoiv30YkHKMMmBnI7/biLv8tvz/DhmX/Qj8UTgXMu49x8c94kkMUGruZpetGdnVnK81zKXfRjEfWoWhWGDymdSQC8asg5lyEi7QBS7nsDjFN5mxkcwJNczzc05HCmcDnPs4h6dOgAq1aV3iQAngiccxkgcl9A7naApsxgAifxNqdTnnWcxxiOZTL/5XAAOnTI7EbgRHkicM6lrUgpoHXrnDeH7coSnqYdX3EQB/MlN/MwTZjDWM4DRNWqMGqUJ4EIbyNwzqWVjh1hyJDgip7cqvAXtzOQzgyiLBt4kNvoSzdWsgMANWrAI4+U7mqgWDwROOfSRpMmMHdu3uFl2EhbnqE397Arv/AiF3En/fmevYCg3WDkSE8A8XjVkHOuxMvOhgoVYieBk3iX6RzIMK7le/bkCD7lEl7cnAQgs+8K3hY8ETjnSrRWrYI2gHXrcg7fj1m8wym8yylUYTUX8hJH8QlTOGLzNN4WkBivGnLOlVixqoJ24Wd60Z12DOdPtuM2HuDf/It1VNg8TdmyMGKElwIS5SUC51yJEn0/QHQSqMxq7qEX/6MBbXiWR7mJvfmWh7gtRxKoWtWTQGEltUQgaSGwCtgIbDCzZnGmOxSYAlxsZq8kMybnXMnVqhV88EHOYWXYyJU8Rx/uZjeW8ArncwcD+Jb6W6YpA9dd51VAWysVVUPHm9nyeCMlZQH3Ae+mIBbnXAkUr3volrzPIDpzIDP4nMO4mBf5hKO3jG8J77+fwkAzVEmoGroRGAMsLe5AnHOp1apVUAWUOwk0Zg7jOJ33OZHqrOQSRtOcKTmSQIcOngS2lWQnAgMmSJomqX3ukZJ2A84FhuS3EEntJU2VNHXZsmVJCtU5lyrZ2UF1Tu5qoJ34lSe4npk05Sg+4XbuZx/m8yKXAMEzgsuW9SuBtrVkVw0dZWZLJO0EvCdpvplNjhr/MNDVzDZK8R8EbWZDgaEAzZo1i3E/oXMuXWRnwxVX5LwzuBJruJWHuIMBVORv/s2/6EV3VlAzx7yNG8OcOSkOuBRIaiIwsyXh/6WSxgKHAdGJoBnwQpgEagKnSdpgZq8lMy7nXPHI3RgsNtGaUfSlG3vwI2M5h67cx/9omGde7yAueZJWNSSpiqRqkdfAScDs6GnMbE8zq2dm9YBXgI6eBJzLLJG7gqWcSaAF/2EqzXiONvzCLhzHRM5jbI4kkJUVVAOZeRJIpmSWCHYGxoZn+2WB581svKTrAcws33YB51x6i3clUCPmcz9dOIs3WUQdLiObF7gEy3Ve6lcEpU7SEoGZfQccEGN4zARgZlclKxbnXOrEagMAqMkyetCD63iSNVTmDvrzCDfzN5VyTOdVQKnnXUw457aJeCWAiqzlZh7hLvpRmTU8yXX05F6WsVOO6cqXh3/+SVGwLgdPBM65Iot1R7DYxKWMph93UZcfeIMz6cL9fM0+MZcxfHgKAnUxlYQbypxzaSxWEjiGyXzO4WTTmhXU4Hg+5GzeiJkEKlYMGoS9b6Di44nAObdVmjTJeyVQA77hVc5lMsexKz9zJc/SjKlM5Pg883foELQjrF3rSaC4eSJwzhVKx455ewatwXIe4Sbm0IRWvE83+tCQbxjJlXmuBookAG8QLjm8jcA5l5BYVUAV+JsbeYxu9KUaq3iKa7mXnixl5zzz+9VAJZcnAudcgSpXDqpwtjAu5kX6cyd7spC3OI3bGcg8GueZ1xNAyedVQ865uCLVQNFJ4Eg+4TOO4AUu5U+2oxXvcQZv5UkCtWt7FVC6KDARSNpbUoXwdQtJN0mqnvTInHPFJtIQHH1fwN4s4GUu4BOOZg8W05bhHMyXfECrPPM3bgw//ZTCgF2RJFIiGANslFQfeBrYE3g+qVE554pF5PkA0Q3BO/AbD3Irc2nMKYynOz1pyDeMoC2byMoxf6SLaO8hNL0k0kawycw2SDoXeNjMHpP0VbIDc86lVvnysH591Hv+4V/8m3vozXb8yXDa0Z1e/MKueeatXh1+/z11sbptK5ESwXpJlwJtgHHhsHLJC8k5lyqRKiApOgkYF/Ayc2nMg3Ticw7nQKbTnqdiJoEOHTwJpLtEEkFb4Aigr5l9L2lPYFRyw3LOJVOsKiCA5nzGJxzFy1zEaqpwMuM5lfHMZv88y/DuoTNHgVVDZjZXUlegTvj+e2BAsgNzziVH3ktBYU++oz93cjEv8TO7cDXDGMFVedoAACpVgjVrUhSsS4lErho6E5gOjA/fHyjpjSTH5ZzbxiKlgOgkUJ3fGUhn5rEvZ/ImPelOA/7HcK6OmQQ6dPAkkIkSaSzuQfCIyYkAZjY9rB5yzpVwse4GBijHOjrwBN3pxQ78zgiu4h56s4Td8kzrJYDMl0gbwQYz+yPXMH+AvHMlXFZWrCRgnMurzKEJj3ALX3IwB/EVVzM8ZhLwEkDpkEgimC3pMiBLUgNJjwGfJjku59xWilQBbdqUc/ih/JdJHMernM86ynMab3ESE5iZ90GCfldwKZNIIrgRaAL8A4wG/gRuSWJMzrmttMMOeUsBdVlINpfxXw6nEV9zHUM4gBm8w2mANk8nbbkSyO8KLl0SuWpoDdAt/HPOlVA77AArV255vz0ruZP+3MwjGKIP3biPrvxFtTzzNm7sdwOXZnETgaQ3yactwMzOSkpEzrlCyd0gXJb1XMeT9KAHO/IbI7mCbvTlJ3aPOb/3DuryKxEMSlkUzrmtkrNbCOMs3uB+utCIb/iQ4+nMIL7i4JjztmwJ77+fslBdCRY3EZjZpFQG4pwrHG2p3ucQpjKIzrRgEvNpxJm8wTjOILoNAPzg72LLr2roJTO7SNIsYlQRmVnTpEbmnMujSZOc3ULswQ/0pRtXMIql1KIDgxnGNWzI1R1YmTKwcWOKg3VpI7+qoZvD/2ekIhDnXP6iu4aoxp/cwQBu5SGE0Y87uY+u/Mn2eebznkFdQeJePmpmP4cvO5rZoug/oGNqwnPOwZYkkMUGrucJFlCfu+jPGM6nId/QjX4xk0ClSp4EXMESuY/gxBjDTk1k4ZIWSpolabqkqTHGXy5pZvj3qaS8d7Y4V8oF/QMZpzOOWezPE3RkHvvSjC+4glEsDvqDzKNxY78r2CUmvzaCDgRn/ntJmhk1qhrwSSE+43gzWx5n3PfAcWb2u6RTgaHA4YVYtnMZKbot4EC+YhCdacmHfE1DzuY13uAscjcER9Su7TeEucLJr0TwPHAm8Eb4P/J3iJm13hYfbmafmlmk4DoF4lzo7FwpEXlY/Ny5sBs/8gxXMY1DOIAZ3Mij7Mds3uBs4iWBDh08CbjCy+/y0T+AP4BLJWUBO4fTV5VU1cx+SGD5BkyQZMCTZjY0n2mvBt5JPHTnMkd0CaAqq+jC/XTiAbLYyCA604+7+IPqcef3HkJdURTYxYSkGwi6ov4ViHRjZUAil48eZWZLJO0EvCdpvplNjvEZxxMkgqPjxNAeaA9Qp07s+lDn0tFuu8GSJcHrLDbQjuH0oju78CujuYQ76c8i6sWd368IcttCIs8juAVoZGYrCrtwM1sS/l8qaSzBcw1yJAJJTYFhwKnxPiMsSQwFaNasmXeB7dJe7juCT2E8A7md/ZjDxxzF2bzOfwtoLvMrgty2kshVQ4sJqogKRVIVSdUir4GTgNm5pqkDvApcYWbfFPYznEs3kYfFR5JAU2bwLifzDqdRkb85jzEcw0cFJoGWLb0qyG07iZQIvgMmSnqLoCtqAMzswQLm2xkYq+A++LLA82Y2XtL14fxDgO5ADWBwON0GM2tW6LVwroTr2BGeeGLL+11ZQm/uoS3PsJLq3MJDDKYj6ymf73K8LcAlQyKJ4Ifwr3z4lxAz+w7yPvEiTACR19cA1yS6TOfSUVbWlofEVGY1tzOQ2xlIOdbzELfSh7tZyQ75LsP7CHLJlMjzCHqmIhDnMk10Q3AZNnIVI+jNPdTmZ17kIu6kP9+zV77L8HsCXCokctVQLaALwVPKKkaGm9kJSYzLubQW3TPoiUxgEJ1pyiw+oznnM4YpHJHv/F4CcKmUSGNxNjAf2BPoCSwEvkhiTM6ltUgSaMJs3uZUJnAyVfmLC3mJI/k03yTQsmXwqEhPAi6VEkkENczsaWC9mU0ys3ZA8yTH5VzaqVw5SAI78wtP0p4ZHEBzpnAbD7Av83iFC4l3R7CZJwBXfBJJBJGrnX+WdLqkg/CuIJzbLJIAWLuGu+nNAupzFSN4lJuozwIe4jbWUSHmvI0bBwnAueKUyFVDfSRtD3QCHgO2A25NalTOpYHIw+LLsJE2jKQv3diNJYzhPLpyH99SP+68fkewK0kSuWpoXPjyD+D45IbjXMkX/YCYE/iAQXTmIKbzOYdxMS/ySeyeUjbzEoAraRK5augZYj+qsl1SInKuhIpOAPswj4Hczhm8xULqcgmjeYmLsHxqW/1SUFdSJVI1NC7qdUXgXGBJcsJxrmSJvhcAYCd+pQc9uJanWE0VunAfj3IT/2y5sjqmxo1hzpwkB+vcVkqkamhM9HtJowG/tsFltFat4IMPtryvyFpu5SHuYACVWMsTdKAn97KCmgUua9QouPzyJAbrXBElctVQbg0gzrPxnEtzkQfDRJKA2ERrRvINDelHNz7kBPZjNjfxWIFJYNSooD3Ak4Ar6RJpI1hF0Eag8P8vQNckx+VcSkWuAIp2HBN5gE4cwpdM5RBaM4rJHFfgsrwx2KWbAksEZlbNzLaL+t8wd3WRc+lMypkEGjGf1zibiRxPLZZxOaM4jP8WmAQ6dPAk4NJTvolAUiVJ10h6MPy7TFLCPZA6V5JFqoEiarKMx7iB2ezH8fyHO+lHI77meS7P92qgSBXQ4MEpCNq5JIhbNSRpf+BNYBIwjaBq6GTgVkknAp3N7O6UROncNhZ9NVAF/uZmHuEu+lGF1TzJdfTkXpaxU9z5y5WDdetSFKxzSZZfG8GjwLVm9l70QEmtCJ405hfDubQT/ZB4sYlLeIH+3EldfuANzqQr9zGffePO772CukyUXyLYNXcSADCz9yWtJ7ifwLm0kLsx+Gg+4gE6cRhf8CUH0ZZn+A/xe1b3m8FcJsuvjaCMpDw9ZUmqSNATqT8wz5V4rVrlbAyuz/8Yw3l8xLHUZglX8izNmBo3CVSvHtT/exJwmSy/RPAcMEZSvciA8PVLwMjkhuVc0VWuvOV+gB1ZwcPczFwacyLv0Y0+NOQbRnJlzIbgyHMBvGM4VxrETQRm1gcYD0yWtFzScoKG4/fMrHeqAnSusCKlgLVroTz/0IlBfMve3MDjPM3VNOB/9KMba6kcc/5Ro7wdwJUu+d5QZmaPA49Lqha+X5WSqJzbCtENwWBcxEsM4A72ZCFvcyq3M5C5NIk7f5kysHFjSkJ1rkRJqIsJM1vlScCVVE2aBCWASBI4gk/5jCN4kUv4k+04kQmcztv5JoHGjT0JuNJra/oacq5EyM7OmQD24lte4kI+5Sjq8ANtGc7BfMn7nBh3GZEnhHnPoK40S6QbaudKlJxVQLADv3E3fbiBx1lPObrTkwfoxBqqxF2G9wjq3BaJdDpXmeAxlXXM7FpJDYBGUU8ucy5loruEKMc6/sW/uYfeVGclw2nHPfTmF3aNO3+lSrDGL3x2LodEqoaeAf4Bjgjf/wj0SVpEzsUQaQcIGOfzCnNpzEPcxhccyoFM51qG5ZsEWrb0JOBcLIkkgr3N7H5gPYCZrSXod8i5pNttt5ztAIczhY85mle4kLVU4mTGcwrvMoumcZcRaQfwS0Kdiy2RRLBOUiXC5xZL2pughFAgSQslzZI0XdLUGOMl6VFJCyTNlHRwoaJ3GStSAoh0DFeP7xnNJUzhCPbiO67hKQ5kOhM4Oe4yIt1Ce0Owc/lLpLH4XoIby/aQlA0cBVxViM843syWxxl3KsETzxoAhwNPhP9dKZWdDa1bb3m/PSvpRl9u4lE2kkVPujOQ21lN1bjL8I7hnCucRJ5Z/J6kL4HmBFVCN+dzYC+ss4HnzMyAKZKqS9rVzH7eRst3aSJ3AijHOq5nCPfSkx34nRFcxT30Zgm7xV2Gdwzn3NaJWzUk6eDIH1AX+BlYAtQpRBWOARMkTZPUPsb43YDFUe9/DIfljqW9pKmSpi5btizBj3bpIHIvwJYkYJzDWGazH49yM19xEAfzJVczPG4SiFQBeRJwbuvkVyJ4IJ9xBvn02bvFUWa2RNJOwHuS5pvZ5KjxsRqd8zzsz8yGAkMBmjVr5g8DzBC5u4Zuxhc8QCeO5SPmsi+n8RbvcCrxrk3wh8M4t23k1+nc8fn8JZIEMLMl4f+lwFjgsFyT/AjsEfV+d4JSh8tgkVJAJAnUYRGjuJwvOIxGfM11DKEpM3mH04iXBDp08CTg3LZS4FVDkipKuk3Sq5LGSLolfCZBQfNViXRWJ6kKcBLBk82ivQFcGV491Bz4w9sHMlvHjluqgbbjD/pzB1/TiPN4lT50owH/YyjXsTFGYVXy5wM7lwyJXDX0HLAKeCx8fynB8wguLGC+nYGxCu4CKgs8b2bjJV0PYGZDgLeB04AFwBqgbWFXwKWPyHOCy7Ke9gylBz2oxXKe4wq60ZcfcxQOc+rQwQ/+ziVLIomgkZkdEPX+P5JmFDSTmX0HHBBj+JCo1wb8K5FAXfra0jeQcSZvcj9d2Iev+Q8t6MQDfEX8aw8aN/b7AJxLtkRuKPsqrLYBQNLhwCfJC8lliujuoQ9mGh9yAm9wNgBn8gYn8GHcJOC9gjqXOokkgsOBT8O7hBcCnwHHhXcMz0xqdC6tRA78kb+5c2F3FvMsVzKNZuzHbDryb/ZnFuM4k1gNwZ4AnEu9RKqGTkl6FC5ttWq15bnA0arxJ125j9t4EGH05w4GcAd/sn3M5filoM4Vn0TuLF4kaQeCyzzLRg3/MpmBuZItXgLIYgPXMIye3MvOLGUUl9ONvvxA3bjL8q6hnSteiTyPoDdB30LfsuVmr0RvKHMZJl4CAOM03mYgt9OYeUzmGM5gHFM5NN/leWOwc8Uvkaqhiwi6ovaCeynVsSM88UT88QcwnUF0phUf8A0NOIexvM7Z5NdbufcL5FzJkUhj8WygepLjcCVMx45bGn3jJYHa/MRw2vIlB3Mg07mRR2nCHF7nHOIlgUhjsCcB50qOREoE/QkuIZ1N1HMIzOyspEXlik38qp8tqvAXXbifzgwii40MojP9uIs/8jlf8Cog50quRBLBs8B9wCxgU3LDccWloOofgDJspB3D6c097MKvjOYS7qIfC9kzz7T+TADn0kciiWC5mT2a9EhcsYl0/ZCfkxnPQG5nf2bzMUdxNq/z31zPEPKzfufSUyJtBNMk9Zd0RK5nFLg0lJ0NVavmvPErvySwPzMZz8mM51QqsZbzeYVj+ChHEmjZ0m8Ccy6dJVIiOCj83zxqmF8+moays+GKK4KDdkF2ZQm96E5bnuEPtucWHmIwHVlP+c3TePWPc5khkRvKjk9FIC65cj8KMp7KrKYzg+jC/ZRjPQ9zC33pxu/sCEDFijBsGFx+eZIDds6lTCIlAiSdDjQBNj+HwMx6JSsot20lciVQGTbShmfpw93U5mde5gLuYADfsbef+TuX4RJ5MM0Q4GLgRoKLwy+EfPoLcCVCdjbUrBm0ARSUBFrxHl9yMMO5mkXU5Ug+4SJe5uQOe2PmScC5TJdIieBIM2sqaaaZ9ZT0APBqsgNzWy+REgBAY+YwkNs5jXf4jj25iBd5mQupXVuY3/DlXKmRyFVDa8P/ayTVBtZDjAvHXYmQSBLYmV8YwnXMpClH8BmdGMS+zOMVXUSHDvK7fp0rZRIpEYyTVB0YCHxJcMXQU8kMyhVOdjbcfDOsWJH/dJVYw208SFfuoyJ/8xg3MvnYe3h1Ug0eSE2ozrkSKJGrhnqHL8dIGgdUNLM/khuWS1QiJQCxiSsYSV+6sTs/MYbzuIMB1G3ZwOv/nXPxq4YkHSppl6j3VwIvAb0l7ZiK4Fz+EkkCx/Mh0ziEZ7mKJdTmGCZzSdkx9BjlScA5F8ivjeBJYB2ApGOBAcBzwB/A0OSH5vLTsWP+SWAf5vEGZ/IhLdmB37mU5zmCKezf4RjWr/f7AJxzW+SXCLLM7Lfw9cXAUDMbY2b3APWTH5qLJzsbhgyJPa4WS/k3HZnF/hzLZLpwHwdWmM8Zoy5lo5Vh8ODUxuqcK/nyayPIklTWzDYALYH2Cc7nkiS/RuGKrOUWHuZO+lOZNTxBB3rRnQNa1mKlVwE55/KR3wF9NDBJ0nKCS0g/ApBUn6B6yKVQdja0aQMbN+YcLjZxGc/Tj7uow2Je42y6ch8LyjTiuuvwEoBzrkBxE4GZ9ZX0AbArMMFsc1dlZQjuMnYpEq+zuGOZxAN0ohnTmMbBXMlzlG3Zgq+9BOCcK4R8bygzsylmNtbMVkcN+8bMvkx+aKVbdBcRrVvnTAIN+ZqxnMMkWrATS2nNSA7lCxp3aOFXAjnnCi2RO4uLRFKWpK/CexByj9te0puSZkiaI6ltsuNJB9nZ0LZt3raAmizjUW5kNvtxAh9yJ/1oxNdk05oda3hDsHNu66Si0fdmYB6wXYxx/wLmmtmZkmoBX0vKNrN1KYirxOrWDdav3/K+An9zE4/Sjb5UYTVDaU8PerCMnQAoXx4eeaSYgnXOpb2klggk7Q6cDgyLM4kB1SQJqAr8BmxIZkwlUXQ1kASLFkXGGJcwmvnsw/10ZTLHsj+z+BeDNyeBqlVh+HC/L8A5t/WSXTX0MNCF+A+9fxzYF1gCzAJuNrM800pqL2mqpKnLli1LVqzFIl410FF8zBSaM5rL+J0daMn7nMWbzGffzdN06ACrVnkScM4VTdISgaQzgKVmNi2fyU4GpgO1gQOBxyXlqUIys6Fm1szMmtWqVSsZ4aZcpBTQunXOaqD6/I9XOJ+POYbd+Ik2jOAQpvEhLTdPU7UqjBrll4Y657aNZJYIjgLOkrQQeAE4QdKoXNO0BV61wALge2CfJMZUIsQqBezICh7iFubSmJN5l7vpTUO+4TnaYOHPVKNGkAC8FOCc25aS1lhsZncCdwJIagF0NrPcT839geCu5Y8k7Qw0Ar5LVkwlRXRjcHn+4QYe5276sB1/MoxruJee/Mrm/v6oWxcWLiyeWJ1zmS/lXUVIuh7AzIYAvYERkmYRPAazq5ktT3VMqZSdHWkMNi7kZQZwB3vxPW9zKl24nznsl2P68uWhb99iCdU5V0qkJBGY2URgYvh6SNTwJcBJqYihJMjOhvbt4Qg+5QE6cQRTmEFTTmQC73Ninulr1AguC/VqIOdcMnnncSmSnQ09r/yWZzbdyUW8zBJ2pR1P8yxt2EQWEJz9+6WgzrlUS/qdxaVddjbsvePv/NK6E7M37cvpvMW99KAB/+MZ2m1OAjVqeBJwzhUPLxEk0ehn1zHj2sF8sb4X1VnJcNrRnV78TO0c03ljsHOuOHkiSAYzePVVjrimK5du+JYJnEhnBjGLpnkmrVzZG4Odc8XLq4a2tc8/h2OOgQsu4K8NFTmFdziZCTGTQFYWDB3q1UHOueLliWBb+f57uPRSaN6cXz9dwLUM5UCm8y6nxJy8cmV49llPAs654udVQ0W1ciX068fGhx7hnw1ZDOIeBtrt/EW1uLP4ZaHOuZLEE8HWWr8+eIJ8z57Yb7/xfJk23EEflrBbvrONGuUJwDlXsnjVUGGZwWuvQZMmcNNNcMABnL7Ll1y58ZkCk0Ddup4EnHMljyeCwpg6FVq0gHPPhbJl+U+ncdRb8D7v/HxggbP61UHOuZLKE0EiFi0K+os+9FCWfzyPDjzBzr/O5JTHTmfRDypw9ho1/Oog51zJ5W0E+fnjDxgwAB56iA2bxINl76LPhq6sYrvgWWoF8EZh51w68EQQy/r18NRT0KMHLFsGV1zBMR/2ZcpPeyQ0e926QTWQJwDnXDrwRBDNDMaNgy5dYP58ft3nOK7OeoC3Rx2CWWKL8O4inHPpxtsIIr78Elq2hLPOAjMm3vo6ey36D2/9kngS8AZh51w68kSweDG0aYM1a8aKSbO4gcepv3YWFzx3FmvW5t8QXK5c0A4gBSUBbxB2zqWj0ls1tGoV3HcfPPAAGzcaD2d1odeGO/mT7YMHaOZDgjp1vB3AOZcZSl8i2LABnn4auneHpUvhsss4blJfPvmpXkKzexuAcy7TlJ5EYAbvvAO33w5z5wY9hI4bB4ceyqcJVpB5G4BzLhOVjjaCGTPgpJPg9NNh3Tp49VWYNAkOPRQIqnliqVEjKAF4G4BzLpNldiL46Sdo1w4OOii4KuiRR2DOnKCLCG1pCO7bNzjbj1a5cjD5woWwaVPw35OAcy4TZWYi+OsvuPdeaNgweGhwp07w7bdBJ3Hly+eZ/PLLg7N9P/t3zpVGmZUINm6EYcOgQQPo1QvOPBPmz4eBA6F6dSDIC/XqQZkywf/s7GDWyy/3s3/nXOmUOY3F774LnTvD7Nlw5JEwdiw0b55jkuxsaN8e1qwJ3i9aFLwHP/A750qvzCgRzJoFp5wSHOFffhk+/jhPEgDo1m1LEohYsyYY7pxzpVVmlAj23x9efx1OPhkqVIg72Q9xbhSLN9w550qDzCgRQNBHUD5JAOJfJhpvuHPOlQZJTwSSsiR9JWlcnPEtJE2XNEfSpGTGEu8yUb9JzDlXmqWiRHAzMC/WCEnVgcHAWWbWBLgwmYH4ZaLOOZdXUhOBpN2B04FhcSa5DHjVzH4AMLOlyYwH/DJR55zLLdklgoeBLsCmOOMbAjtImihpmqQrY00kqb2kqZKmLlu2LEmhOudc6ZS0RCDpDGCpmU3LZ7KywCEEpYaTgXskNcw9kZkNNbNmZtasVq1ayQnYOedKqWRePnoUcJak04CKwHaSRplZ66hpfgSWm9lqYLWkycABwDdJjMs551yUpJUIzOxOM9vdzOoBlwAf5koCAK8Dx0gqK6kycDhxGpadc84lR8pvKJN0PYCZDTGzeZLGAzMJ2hGGmdnsVMfknHOlmSzRJ7OXEM2aNbOpU6cWdxjOOZdWJE0zs2axxmXOncXOOee2iicC55wr5TwROOdcKeeJwDnnSrmMSATxnjrmnHOuYGn/PAJ/6phzzhVN2pcI/KljzjlXNGmfCPypY845VzRpnwj8qWPOOVc0aZ8I/KljzjlXNGmfCPypY845VzRpf9UQBAd9P/A759zWSfsSgXPOuaLxROCcc6WcJwLnnCvlPBE451wp54nAOedKubR7QpmkZcCiBCevCSxPYjiplmnrA5m3Tr4+JVtpXp+6ZlYr1oi0SwSFIWlqvEezpaNMWx/IvHXy9SnZfH1i86oh55wr5TwROOdcKZfpiWBocQewjWXa+kDmrZOvT8nm6xNDRrcROOecK1imlwicc84VwBOBc86VchmfCCT1ljRT0nRJEyTVLu6YikLSQEnzw3UaK6l6ccdUFJIulDRH0iZJaXtZn6RTJH0taYGkO4o7nqKSNFzSUkmzizuWopK0h6T/SJoXbms3F3dMRSGpoqT/SpoRrk/PIi8z09sIJG1nZn+Gr28CGpvZ9cUc1laTdBLwoZltkHQfgJl1LeawtpqkfYFNwJNAZzObWswhFZqkLOAb4ETgR+AL4FIzm1usgRWBpGOBv4DnzGy/4o6nKCTtCuxqZl9KqgZMA85J199HkoAqZvaXpHLAx8DNZjZla5eZ8SWCSBIIVQHSOvOZ2QQz2xC+nQLsXpzxFJWZzTOzr4s7jiI6DFhgZt+Z2TrgBeDsYo6pSMxsMvBbccexLZjZz2b2Zfh6FTAP2K14o9p6FvgrfFsu/CvScS3jEwGApL6SFgOXA92LO55tqB3wTnEH4dgNWBz1/kfS+ECTySTVAw4CPi/mUIpEUpak6cBS4D0zK9L6ZEQikPS+pNkx/s4GMLNuZrYHkA3cULzRFqyg9Qmn6QZsIFinEi2R9UlzijEsrUuemUhSVWAMcEuumoK0Y2YbzexAghqBwyQVqfouIx5VaWatEpz0eeAt4N4khlNkBa2PpDbAGUBLS4NGnkL8PunqR2CPqPe7A0uKKRYXQ1iXPgbINrNXizuebcXMVkqaCJwCbHXDfkaUCPIjqUHU27OA+cUVy7Yg6RSgK3CWma0p7ngcEDQON5C0p6TywCXAG8UckwuFjatPA/PM7MHijqeoJNWKXC0oqRLQiiIe10rDVUNjgEYEV6YsAq43s5+KN6qtJ2kBUAFYEQ6akuZXQZ0LPAbUAlYC083s5GINaitIOg14GMgChptZ3+KNqGgkjQZaEHRz/Ctwr5k9XaxBbSVJRwMfAbMIjgMAd5nZ28UX1daT1BR4lmBbKwO8ZGa9irTMTE8Ezjnn8pfxVUPOOefy54nAOedKOU8EzjlXynkicM65Us4TgXPOlXKeCFyxkFQj7BF2uqRfJP0Uvl4pKaWdgUk6R1LjqPe9JBX6JjhJ9eL11impiaQPJX0j6VtJPSVt8/0vv3WRNDGde3h1yeOJwBULM1thZgeGt8kPAR4KXx/Ilmu9txlJ+d1Ffw6w+eBpZt3N7P1t+NmVCG4wG2BmDYH9CTqqS0Z3yOeQxHVxmckTgSuJsiQ9Ffa1PiE8kCJpb0njJU2T9JGkfcLhdSV9ED6j4QNJdcLhIyQ9KOk/wH2x5pd0JMEd5wPDEsne4XwXhMs4VNKnYd/v/5VULTzz/0jSl+HfkQWsz2XAJ2Y2ASC8I/wG4PbwM3pI6hyZOOyHqV74+rUw3jmS2kdN81fYmeIMSVMk7VzQukSTdJKkz8L4Xw774UHSAElzw+9yUOF/OpeOPBG4kqgB8G8za0Jwt/H54fChwI1mdgjQGRgcDn+coN/8pgSd8D0atayGQCsz6xRrfjP7lOBs/fawhPJtZMawu4gXCfp6P4DgVv61BD0+nmhmBwMX5/q8WJoQ9IG/Wfg5lVTwg4XahfE2A26SVCMcXoXgrvIDgMnAtfmtSzRJNYG7w+/lYGAqcJukHYFzgSbhd9mngNhchsiITudcxvnezKaHr6cB9cIz1iOBl4OuY4Cgqw2AI4DzwtcjgfujlvWymW0sYP54GgE/m9kXsOXZFpKqAI9LOhDYSJBs8iNi90Yaq9fS3G4Ku+GAoGO7BgTdi6wDxoXDpxE8FCdRzQmqjz4Jv4vywGfAn8DfwDBJb0Ut32U4TwSuJPon6vVGoBJB6XVl2I5QkOiD7urwf2Hmj4h3AL+VoP+dA8Ll/l3AcuYAx+ZYsLQXsDzsPXIDOUvnFcNpWhCUQo4wszUKepmsGE6zPqrn2Y0Ubl8WQR/2l+YZIR0GtCToOO8G4IRCLNelKa8acmkhPBv/XtKFEPQoKemAcPSnBAcuCB4+9HEh518FVIvxsfOB2pIODeepFjY6b09QUtgEXEHQ+Vd+soGjo67eqURQnRTpDn0hcHA47mBgz3D49sDvYRLYh+BMviDx1iXaFOAoSfXDz6wsqWFYato+7IztFoKGe1cKeCJw6eRy4GpJMwjOsiMPtrkJaCtpJsGBOd7VOPHmfwG4XdJXkvaOTBw+dvJi4LFwnvcIzsgHA20kTSGoFlpNPsxsLUEjbjdJ3wDLCRqPIw8VGgPsqOCJUx0Inn8MMB4oG65Xb4IDeEFirkuueJYBVwGjw2VPAfYhSCDjwmGTCEo+rhTw3kedSzFJ5wAPAseb2aJiDsc5TwTOOVfaedWQc86Vcp4InHOulPNE4JxzpZwnAuecK+U8ETjnXCnnicA550q5/wMk6sjpBqw7tgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Is the bootstrapped distribution Normal?\n",
    "fig = sm.qqplot(diffs, line='s')\n",
    "pyplot.title('Quantiles of standard Normal vs. bookstrapped mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The points on the Q-Q Normal plot are nearly on a strait line. Apparently, the bootstrap distribution of the difference in means does conform to the CLT.\n",
    "\n",
    "## Your Turn 2\n",
    "In the cell below, create and run the code to compare the difference in means of adult sons to their fathers. Is the difference significant? Also, check if the distribution in the difference in means is approximately normal. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Replace the ???\n",
    "diffs = []\n",
    "for i in range(n_replicas):\n",
    "    sample = families.sample(frac=1.0, replace=True)\n",
    "    male_sample_mean = sample[sample.gender == '???'].???.mean()\n",
    "    father_sample_mean = sample[sample.gender == '???'].???.mean()\n",
    "    diffs.append(male_sample_mean - father_sample_mean)\n",
    "diffs = pandas.Series(diffs)\n",
    "\n",
    "plot_hist(diffs)\n",
    "\n",
    "fig = sm.qqplot(diffs, line='s')\n",
    "pyplot.title('Quantiles of standard Normal vs. bootstrapped mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bootstraping Other Statistics\n",
    "\n",
    "The bootstrap method can be applied to most any statistic. The code in the cell below computes the difference in **medians** of the sons and the fathers. The median is an order statistic, and the values of a median are discretized by the quanta of the samples. Run this code and examine the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAThUlEQVR4nO3df6zdd33f8eerTgis0JEsN5mx3TlFzoaDhoE7jzXTlBJYQkB1kJrObDCrymQqJVvIqrUJ/aP0D0usKw2atrAZiHBXSrAEbVxE1xqXjKFC3Gtmgp2QxJAoubEVXyAVPzRls/PeH+fr5WDfH8f3/LiXT54P6ep8z+d8vuf7utf3vO7X3/Pjm6pCktSWn1rpAJKk0bPcJalBlrskNchyl6QGWe6S1KALVjoAwKWXXlobN25c6RgalUOHXlh+4xtXLscEfHvuRwD83NRPr3AS/UQa8rFy6NCh71TV1Hy3ZTW8FHJ6erpmZmZWOoZGJXlheRX8fo3TP/uvXwHg0+/9RyucRD+RhnysJDlUVdPz3eZhGUlqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGmS5S1KDLHdJatDAHz+QZA0wAzxdVe9IcgnwaWAj8ATwy1X1bDf3TuBm4DTwb6rqz0acWyvkrv2PLjnn9vOcP4jb33rlSO5HerE4nz3324CH+67fARyoqk3Age46STYD24GrgOuBu7s/DJKkCRmo3JOsB94OfKxveBuwp1veA9zYN35vVT1XVY8Dx4CtI0krSRrIoHvuHwZ+HXi+b+zyqjoB0F1e1o2vA57qmzfbjf2YJDuTzCSZmZubO9/ckqRFLFnuSd4BnKyqQ0vNPbPKPGPnfNxZVe2uqumqmp6amvcTKyVJyzTIE6pXA7+Y5AbgpcDPJPkD4Jkka6vqRJK1wMlu/iywoW/99cDxUYaWJC1uyT33qrqzqtZX1UZ6T5T+RVW9G9gH7Oim7QDu65b3AduTXJTkCmATcHDkySVJCxrmTEwfBPYmuRl4ErgJoKqOJtkLPAScAm6pqtNDJ5UkDey8yr2q7gfu75a/C1y7wLxdwK4hs0mSlsl3qEpSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGjTICbJfmuRgkq8nOZrkt7vxDyR5Osnh7uuGvnXuTHIsySNJrhvnNyBJOtcgZ2J6DnhzVf0wyYXAl5P8aXfbXVX1u/2Tk2ymd67Vq4BXAV9IcqWn2pOkyRnkBNlVVT/srl7YfdUiq2wD7q2q56rqceAYsHXopJKkgQ10zD3JmiSHgZPA/qp6oLvp1iQPJrknycXd2Drgqb7VZ7uxs+9zZ5KZJDNzc3PL/w4kSecYqNyr6nRVbQHWA1uTvBb4CPBqYAtwAvhQNz3z3cU897m7qqaranpqamoZ0SVJCzmvV8tU1V8D9wPXV9UzXek/D3yUFw69zAIb+lZbDxwfPqokaVCDvFpmKskru+WXAW8Bvplkbd+0dwJHuuV9wPYkFyW5AtgEHBxpaknSogZ5tcxaYE+SNfT+GOytqs8l+W9JttA75PIE8F6AqjqaZC/wEHAKuMVXykjSZC1Z7lX1IPD6ecbfs8g6u4Bdw0WTJC2X71CVpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBg1yDtWXJjmY5OtJjib57W78kiT7kzzWXV7ct86dSY4leSTJdeP8BiRJ5xpkz/054M1V9TpgC3B9kjcBdwAHqmoTcKC7TpLNwHbgKuB64O7u/KuSpAlZstyr54fd1Qu7rwK2AXu68T3Ajd3yNuDeqnquqh4HjgFbRxlakrS4gY65J1mT5DBwEthfVQ8Al1fVCYDu8rJu+jrgqb7VZ7uxs+9zZ5KZJDNzc3NDfAuSpLMNVO5VdbqqtgDrga1JXrvI9Mx3F/Pc5+6qmq6q6ampqYHCSpIGc16vlqmqvwbup3cs/ZkkawG6y5PdtFlgQ99q64HjwwaVJA1ukFfLTCV5Zbf8MuAtwDeBfcCObtoO4L5ueR+wPclFSa4ANgEHR5xbkrSICwaYsxbY073i5aeAvVX1uSRfAfYmuRl4ErgJoKqOJtkLPAScAm6pqtPjiS9Jms+S5V5VDwKvn2f8u8C1C6yzC9g1dDpJ0rL4DlVJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDRrks2UkrYC79j+6Ytu+/a1Xrti2NRruuUtSgyx3SWqQ5S5JDbLcJalBlrskNWiQ0+xtSPLFJA8nOZrktm78A0meTnK4+7qhb507kxxL8kiS68b5DUiSzjXISyFPAb9WVV9L8grgUJL93W13VdXv9k9OshnYDlwFvAr4QpIrPdWeJE3OknvuVXWiqr7WLf8AeBhYt8gq24B7q+q5qnocOAZsHUVYSdJgzuuYe5KN9M6n+kA3dGuSB5Pck+Tibmwd8FTfarPM88cgyc4kM0lm5ubmzj+5JGlBA5d7kpcDnwHeV1XfBz4CvBrYApwAPnRm6jyr1zkDVburarqqpqemps43tyRpEQOVe5IL6RX7J6vqswBV9UxVna6q54GP8sKhl1lgQ9/q64Hjo4ssSVrKIK+WCfBx4OGq+r2+8bV9094JHOmW9wHbk1yU5ApgE3BwdJElSUsZ5NUyVwPvAb6R5HA39n7gXUm20Dvk8gTwXoCqOppkL/AQvVfa3OIrZSRpspYs96r6MvMfR//8IuvsAnYNkUuSNATfoSpJDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktQgy12SGmS5S1KDLHdJapDlLkkNGuQcqhuSfDHJw0mOJrmtG78kyf4kj3WXF/etc2eSY0keSXLdOL8BSdK5BtlzPwX8WlW9BngTcEuSzcAdwIGq2gQc6K7T3bYduAq4Hrg7yZpxhJckzW/Jcq+qE1X1tW75B8DDwDpgG7Cnm7YHuLFb3gbcW1XPVdXjwDFg64hzS5IWcV7H3JNsBF4PPABcXlUnoPcHALism7YOeKpvtdlu7Oz72plkJsnM3NzcMqJLkhYycLkneTnwGeB9VfX9xabOM1bnDFTtrqrpqpqempoaNIYkaQADlXuSC+kV+yer6rPd8DNJ1na3rwVOduOzwIa+1dcDx0cTV5I0iEFeLRPg48DDVfV7fTftA3Z0yzuA+/rGtye5KMkVwCbg4OgiS5KWcsEAc64G3gN8I8nhbuz9wAeBvUluBp4EbgKoqqNJ9gIP0XulzS1VdXrUwSVJC1uy3Kvqy8x/HB3g2gXW2QXsGiKXJGkIvkNVkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktSgQU6zd0+Sk0mO9I19IMnTSQ53Xzf03XZnkmNJHkly3biCS5IWNsie+yeA6+cZv6uqtnRfnwdIshnYDlzVrXN3kjWjCitJGsyS5V5VXwK+N+D9bQPurarnqupx4BiwdYh8kqRlGOaY+61JHuwO21zcja0DnuqbM9uNnSPJziQzSWbm5uaGiCFJOttyy/0jwKuBLcAJ4EPd+Hwn0q757qCqdlfVdFVNT01NLTOGJGk+yyr3qnqmqk5X1fPAR3nh0MsssKFv6nrg+HARJUnna1nlnmRt39V3AmdeSbMP2J7koiRXAJuAg8NFlCSdrwuWmpDkU8A1wKVJZoHfAq5JsoXeIZcngPcCVNXRJHuBh4BTwC1VdXosySVJC1qy3KvqXfMMf3yR+buAXcOEkiQNx3eoSlKDLHdJatCSh2Wk1eCu/Y+u2LZvf+uVK7Ztabncc5ekBlnuktQgy12SGmS5S1KDLHdJapDlLkkNstwlqUGWuyQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDVqy3JPck+RkkiN9Y5ck2Z/kse7y4r7b7kxyLMkjSa4bV3BJ0sIG2XP/BHD9WWN3AAeqahNwoLtOks3AduCqbp27k6wZWVpJ0kCWLPeq+hLwvbOGtwF7uuU9wI194/dW1XNV9ThwDNg6mqiSpEEt95j75VV1AqC7vKwbXwc81Tdvths7R5KdSWaSzMzNzS0zhiRpPqN+QjXzjNV8E6tqd1VNV9X01NTUiGNI0ovbcsv9mSRrAbrLk934LLChb9564Pjy40mSlmO55b4P2NEt7wDu6xvfnuSiJFcAm4CDw0WUJJ2vC5aakORTwDXApUlmgd8CPgjsTXIz8CRwE0BVHU2yF3gIOAXcUlWnx5RdkrSAJcu9qt61wE3XLjB/F7BrmFCSpOH4DlVJapDlLkkNWvKwjCS17q79j67Idm8f43275y5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIaZLlLUoMsd0lqkOUuSQ2y3CWpQZa7JDXIcpekBlnuktSgoT4VMskTwA+A08CpqppOcgnwaWAj8ATwy1X17HAxJUnnYxR77r9QVVuqarq7fgdwoKo2AQe665KkCRrH57lvo3fOVYA9wP3Ab4xhO5Ias1Kfq96iYffcC/jzJIeS7OzGLq+qEwDd5WXzrZhkZ5KZJDNzc3NDxpAk9Rt2z/3qqjqe5DJgf5JvDrpiVe0GdgNMT0/XkDkkSX2G2nOvquPd5Ungj4CtwDNJ1gJ0lyeHDSlJOj/LLvckP53kFWeWgX8KHAH2ATu6aTuA+4YNKUk6P8Mclrkc+KMkZ+7nD6vqvyf5K2BvkpuBJ4Gbho8pSTofyy73qvo28Lp5xr8LXDtMKEnScHyHqiQ1yHKXpAZZ7pLUIMtdkhpkuUtSgyx3SWqQ5S5JDbLcJalBlrskNchyl6QGjeNkHVJTFjuBxOyz/3vJOdJKcM9dkhrknrukc/g/kZ987rlLUoOa2HNfqb2M29965YpsV5KW4p67JDWoiT33leL/GCStVmPbc09yfZJHkhxLcse4tiNJOtdYyj3JGuA/A28DNgPvSrJ5HNuSJJ1rXHvuW4FjVfXtqvo/wL3AtjFtS5J0llTV6O80+SXg+qr6V9319wD/sKpu7ZuzE9jZXf27wCMjD7J8lwLfWekQi1jt+cCMo7Da88Hqz7ja88FwGf9OVU3Nd8O4nlDNPGM/9lekqnYDu8e0/aEkmamq6ZXOsZDVng/MOAqrPR+s/oyrPR+ML+O4DsvMAhv6rq8Hjo9pW5Kks4yr3P8K2JTkiiQvAbYD+8a0LUnSWcZyWKaqTiW5FfgzYA1wT1UdHce2xmRVHi7qs9rzgRlHYbXng9WfcbXngzFlHMsTqpKkleXHD0hSgyx3SWqQ5Q4kuSTJ/iSPdZcXLzJ3TZL/leRzqylfkg1Jvpjk4SRHk9w2oWyLfsxEev5jd/uDSd4wiVznke9fdLkeTPKXSV43yXyDZOyb9w+SnO7eR7Kq8iW5Jsnh7nfvf0wy3yAZk/zNJH+S5Otdxl+ZcL57kpxMcmSB20f/OKmqF/0X8DvAHd3yHcC/X2TuvwX+EPjcasoHrAXe0C2/AngU2DzmXGuAbwE/B7wE+PrZ2wRuAP6U3nsf3gQ8MMGf2yD5fh64uFt+2yTzDZqxb95fAJ8Hfmk15QNeCTwE/Gx3/bLV9jME3n/mcQNMAd8DXjLBjP8EeANwZIHbR/44cc+9Zxuwp1veA9w436Qk64G3Ax+bTKz/b8l8VXWiqr7WLf8AeBhYN+Zcg3zMxDbg96vnq8Ark6wdc66B81XVX1bVs93Vr9J7T8YkDfpRHf8a+AxwcpLhGCzfPwc+W1VPAlTVasxYwCuSBHg5vXI/NamAVfWlbpsLGfnjxHLvubyqTkCvJIHLFpj3YeDXgecnlOuMQfMBkGQj8HrggTHnWgc81Xd9lnP/oAwyZ1zOd9s309t7mqQlMyZZB7wT+C8TzHXGID/DK4GLk9yf5FCSfzmxdD2DZPxPwGvovZnyG8BtVTXpx/FiRv44edF8nnuSLwB/e56bfnPA9d8BnKyqQ0muGWG0M/c/VL6++3k5vT2891XV90eRbbHNzTN29mtrB5kzLgNvO8kv0Cv3fzzWRPNsep6xszN+GPiNqjrd2/GcqEHyXQC8EbgWeBnwlSRfrapJnfBgkIzXAYeBNwOvBvYn+Z8TeIwMauSPkxdNuVfVWxa6LckzSdZW1Ynuv0Lz/bfyauAXk9wAvBT4mSR/UFXvXiX5SHIhvWL/ZFV9dhS5ljDIx0ys5EdRDLTtJH+f3qG2t1XVdyeU7YxBMk4D93bFfilwQ5JTVfXHqyTfLPCdqvoR8KMkXwJeR+95n0kYJOOvAB+s3gHuY0keB/4ecHAyEZc0+sfJJJ/4WK1fwH/gx5+w/J0l5l/DZJ9QXTIfvb/8vw98eIK5LgC+DVzBC09kXXXWnLfz408UHVxl+X4WOAb8/Ar97i2Z8az5n2CyT6gO8jN8DXCgm/s3gCPAa1dZxo8AH+iWLweeBi6d8L/1RhZ+QnXkj5OJfWOr+Qv4W90v52Pd5SXd+KuAz88zf9LlvmQ+eocTCniQ3n8/DwM3TCDbDfT20L4F/GY39qvAr3bLoXfilm/RO9Y5PeF/26XyfQx4tu9nNrMCv3+LZjxr7kTLfdB8wL+j94qZI/QOCa6qn2H3WPnz7nfwCPDuCef7FHAC+L/09tJvHvfjxI8fkKQG+WoZSWqQ5S5JDbLcJalBlrskNchyl6QGWe6S1CDLXZIa9P8A/yo9isFap3YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "median_diffs = []\n",
    "for i in range(n_replicas):\n",
    "    sample = families.sample(frac=1.0, replace=True)\n",
    "    male_sample_median = sample[sample.gender == 'male'].childHeight.median()\n",
    "    father_sample_median = sample[sample.gender == 'male'].father.median()\n",
    "    median_diffs.append(male_sample_median - father_sample_median)\n",
    "median_diffs = pandas.Series(median_diffs)\n",
    "\n",
    "plot_hist(median_diffs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The chart shows that the bootstrap distribution of the difference in medians. The **confidence interval overlaps zero**, so we must accept the null hypothesis that the difference in medians is zero.\n",
    "\n",
    "You can also plot the Q-Q Normal plot of the bootstrap distribution of the difference in medians. Run the code in the cell below and examine the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Quantiles of standard Normal vs. bootstrapped median')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA76UlEQVR4nO3dd7wU1fnH8c+XJkKwgYmiAhakxgJXVBINBiSKRrDkp4TYFREI1oiRRDGAwYIVkaCxIYoFW4JGUBQLGAVEioASKSKoICKiIu35/XHmyrDs7t177+7dW57367Wv3Zkz5czu7D57zpk5R2aGc845V1zV8p0B55xzFZMHEOeccyXiAcQ551yJeABxzjlXIh5AnHPOlYgHEOeccyXiAaSckLRO0n7R6wclDS7j/Q+WtErSZ2W536JIek3SBVnc3mJJnbK1vbIkaaCkR8pwf00kmaQaZbXPyiDb52wx9rvN5yXpRUln53KfVTaASDpH0mxJ30n6TNIISTuX0b63O8HM7Cdm9nFZ7D9JfvYBrgBamtkexVy3wv4gJ4oCt0lqF5t3gCS/WSrLovf5gAyXrTTnWFkys+PN7KFc7qNKBhBJVwA3An8CdgaOAJoAEyTVzGPW8qUx8KWZfZHvjGRLKf41rwayUvrzf+5lw9/n/KlyAUTSTsD1wB/N7D9mttHMFgP/B+wL/D5abptqJEkdJC2LTV8t6X+SvpH0gaSTY2nnSHpT0i2SvpK0SNLxUdoQ4ChgeFRtNTyan/IfmaQTJc2UtEbSFEkHxdL6S/o0yscCSR1TbGNnSQ9LWilpiaS/SKoW/bObCDSM8vNgknUbSPp3tP/Vkt6I1h0NNAL+Fa17VbT8k1Gp7mtJr0tqFdvWg5LuljQ+yvN/Je0fSz9W0vxo3eGAYmn7S5ok6cuoum2MpF1i6Yuj92MW8K2kGpLOjI73S0kDkr03CR4CDpL0qxTvY0NJz0fvw0JJF8bSBkp6StIjktYC50SlzcHR57ZO0r8k1Y/yvlbSu5KaxLZxh6RPorTpko7KIM9ImifpxNh0jeg9aiOpdpSnL6PP8F1JP8tku5HzJC2XtCL681W4jx0k3R6lLY9e7xBLvzB6j1ZH71nDaP7r0SLvR+/J6cU5x7S1quZ8SUuBSdF2izrvRkqaGJ13kyU1jqWbpH6SPo7et5slVYulnxe9x19Jeilh3ZTnbJLPaWCUz0eifMyWdKCkP0v6IvrsO8eW31nSP6P3/tPoXKoepVVX+I1ZJelj4ISEff1Y06HMvjtXSpoVHcfjkmoXcV6AmVWpB3AcsAmokSTtIWBM9PpBYHAsrQOwLDb9O6AhIQifDnwL7BmlnQNsBC4EqgMXA8sBRemvARck7NuAAxL3DbQBvgAOj7Z1NrAY2AFoBnwCNIyWbQLsn+K4HwaeA+pFy30InJ/s2JKs+3dgJFAzehwVO5bFQKeE5c+L9rMDcDswM5b2IOFffjugBjAGGBulNQDWAqdF+7ks+qwuiNIPAI6Ntrs78Dpwe2zbi4GZwD7AjkBLYB1wdLTOrdH2OqU4zgcJpY9+wJuxfVpsmcnACKA2cAiwEugYpQ2MPvduhPNix+izXgjsTyjtfhC9952i438YeCC2/T8A9aO0K4DPgNqx7T+SIu/XEp270fQJwPzo9UXAv4A6hHOoLbBTBt+VJoTz8jGgLvDz6Hg7Rel/A94Gfhp9HlOAQVHar4FVhPN3B+Au4PVk53txz7FYvh6O8rVjhufdN7Fz4Y7CzziWn1eB3QgB60O2nnfdos+wRfS5/AWYksk5m+Q9HQisB34T+/wXAQOi9S8EFsWWfxb4R3ScPwXeAS6K0noB8wnn+25R/o3ot43Y7wyZfXfeIfym7QbMA3oVeY7k6oe6vD4IX9DPUqQNBSbEf0xiaR1I/yM7E+gavT4HWBhLqxN9sHskfrDJvlBsG0DuIfpSxpZdAPwqOim+IPwY1UyTt+rAD4Q2jsJ5FwGvZXhsfyMEnwOSpC0mxQ9ylL5LdGw7x47tvlh6F7b+0J0FvB1LE7As8b2KpXcD3kvIy3mx6WuJglM0XRfYkCq/bA0gOwBLgeOJBRDCF3UzUC+2zt+BB6PXA4n9SMY+6wGx6WHAi7Hp3xL7oUuSp6+Ag2PbTxVADiD8QNaJpscA10avzyP8uB9UzO9Kk+izax6bdxPwz+j1/4AusbTfAIuj1/8Eboql/YQQXJsknu/FPcdi+dqvmOfd2IT8bAb2ieXnuFh6b+CV6PWLRH+2oulqwHeEqt/inrMDgYkJn/86oHo0XS/Kyy7Azwjf2x1jy3cHXo1eTyL2Iw90JkUAyfC784eEz3lkUedIlavCIvwraqDk9aZ7Ev5hFUnSWdparbQGaE34N1Lox6uZzOy76OVPSpDfxsAVhfuJ9rUPodSxELiUcFJ+IWlsYTVBggZALWBJbN4SYK8M83Az4R/YhKiIf3WqBaNi9VCF6r21hBOzMA+F4ld6fcfW96UhoUQF/Pir/eO0pJ9Gx/hptO1HErZLfPkk2/sW+DLtkYblfgAGRY94dURDYLWZfRObl/g+xvdf6PPY6++TTP94Xki6Iqoq+Tr6rHdm+2NMlueFhH+Nv5VUBzgJeDRKHg28BIyNqppuUvHa+uLHtITwPhA9J55TSdPMbB3hvU91zmV8jiXLV4bnXfxcWEcoCTdMlp5wLI2BO2Lfv9WE82IvijhnU0j8/FeZ2ebYNIRzojGhVLIitu9/EEoiJO6bbT+LbWT43Un1vUypKgaQqYSofkp8pqS6hH+ck6NZ3xJKDoX2iC3bGLgX6AvUN7NdgDmkqftMYMXI7yfAEDPbJfaoY2aPAZjZo2b2S8LJZoSLAxKtIvz7axyb1wj4NKPMmn1jZleY2X6Ef0yXa2tbS+Kx/B7oSigV7Uz4twiZvTcrCMExrCApPk34t2+Ef9I7EUqTiduN5ydxe3UI1UOZeICQ/5Nj85YDu0mqF5uX+D4W57PdhkJ7R39Ce9yu0Xn1NZmfV48R/qF2BT6IggoW2vmuN7OWQHvgRMI/50zFP4NGhPeB6DnxnEqaFn2/6pPinCvmOUaS+Zmcd/Fz4SeEqprlydITjuUTQrVR/Du4o5lNoehztjQ+IfxWNYjtdyczK2zb2WbfUZ5TyeS7U2xVLoCY2deERvS7JB0nqaZCI+aThB/aMdGiM4EuknaTtAfhn36huoQPYyWApHMJJZBMfQ7sl+Gy9wK9JB2uoK6kEyTVk9RM0q8VGi7XE/69bE7cQPTv5glgSLReY+Bywr+QIik04h8QfTnWRvso3E/isdQjnPRfEgLwDRkeJ8B4oJWkU6ISYj9igTva9jpgjaS9CFfRpfMUcKKkX0qqRagmyeicN7NNhJJd/9i8TwhVQX9XaJg+CDifredMadUj1J+vBGpIuhbYqRjrjyVUY1zM1tIHko6R9POo8XUt4c/EdudJGn+VVEehUfpc4PFo/mPAXyTtLqkBocqw8Jx6FDhX0iHR+XkD8F8LF6xAwnlTzHMsmUzOuy6xc2FQlJ/4P/g/SdpV4bL2S2LHORL4c3T8hQ3bv4vSijpnS8zMVgATgGGSdlK4qGB/bb3A4wmgn6S9Je0KpCu1Ffe7k5EqF0AAzOwm4BrgFkK98SLCSdcpquaAUOx/n1AUnsDWkwkz+4BQlz2VcHL/HHirGFm4AzhN4YqOO4vI6zRCw9pwQn34QkIbC4S6+qGEwPcZoWh7TYpN/ZFQqvoYeJPwBb8/w/w2BV4mnIBTgRFm9lqU9nfCj8gaSVcSGgWXEP5pfkBoZM2Ima0iXJwwlPBD0JRt39frCY2yXxO+uE8Xsb25QB/Csa4gvH/L0q2T4LFovbjuhH+3y4FngOvMbGIxtpnOS4T69g8J7+F6iq4O+VH0gzOVUMp4PJa0ByGYriVUc00m+qFXuDJpZBGbnkw4714BbjGzCdH8wcA0YBYwG5gRzcPMXgH+CowjvIf7A2fEtjkQeCg6b/6P4p1jyWRy3j0KXEeogmoL9EhIfw6YTvjzOJ7QjoOZPUMo2Y+Nqn/mEGorMjlnS+ssQvXzB4Tz9ylCVTuEP5cvEX6nZpD++1Cs706mCq9yqNIknUd4g39hZkvznR/nXHYpXJ6+zMz+kiLdgKaF1X4uM34DDmBm90vaSPjn5gHEOecy4AEkYmaj850H55yrSLwKyznnXIlUyUZ055xzpVelqrAaNGhgTZo0yXc2nHOuQpk+ffoqM9s9cX6VCiBNmjRh2rRp+c6Gc85VKJKS3uXuVVjOOedKxAOIc865EvEA4pxzrkQ8gDjnnCsRDyDOOedKxAOIc865EvEA4pxzrkQ8gDjnXFX3wQfQvz8Us2srDyDOOVdVffcd/PnPcPDBcO+9sGhRsVb3AOKcc1XRv/4FLVvC0KHQowcsWAD7ZTpQauABxDnnqpKlS6FbNzjpJKhbFyZPhgcfhN236+qqSHkNIJLul/SFpDkp0ntImhU9pkg6OJa2WNJsSTMleQdXzjmXzsaNcNNN0KIFTJgQSh7vvQdHH13iTea7M8UHCWN9P5wifRHwKzP7StLxwCjg8Fj6MdGYxM4551J58024+GKYMyeUPO68Exo3LvVm81oCMbPXCQPcp0qfYmZfRZNvA3uXScacc64yWLUKzjsPjjoK1q6FZ5+F557LSvCAitUGcj7wYmzagAmSpkvqmWolST0lTZM0beXKlTnPpHPO5d2WLXDffdCsGYweDVddFS7V7do1q7vJdxVWRiQdQwggv4zN/oWZLZf0U2CipPlRiWYbZjaKUPVFQUGBj9/rnKvcZs0K1VVTpoSSx4gR0Lp1TnZV7ksgkg4C7gO6mtmXhfPNbHn0/AXwDNAuPzl0zrlyYN06uPJKaNMGPvwQHnggXGGVo+AB5TyASGoEPA2caWYfxubXlVSv8DXQGUh6JZdzzlVqZvD00+HqqmHD4NxzYf58OOcckHK667xWYUl6DOgANJC0DLgOqAlgZiOBa4H6wAiFN2KTmRUAPwOeiebVAB41s/+U+QE451w+LVoEffvCCy/Az38OY8fCL35RZrvPawAxs+5FpF8AXJBk/sfAwduv4ZxzVcCGDXDLLTBoEFSvHl736wc1a5ZpNipEI7pzzrnIa6+FRvL58+GUU+D222GfffKSlXLdBuKccy7yxRdw1llwzDHwww8wfjyMG5e34AEeQJxzrnzbsgVGjgz3dIwdC9dcE+4o79Il3znzKiznnCu33nsPevWCd96BDh3CPR0tWuQ7Vz/yEohzzpU3a9fCJZdAQQEsXhzuJp80qVwFD/ASiHPOlR9m8OSTcOml8NlnofQxZAjsumu+c5aUBxDnnCsPFi6EPn1CV+uHHho6PmxXvjvY8Cos55zLp/Xr4frrQ5cjU6fCHXeENo9yHjzASyDOOZc/EyeGUsdHH8Hpp8Ott0LDhvnOVca8BOKcc2VtxQro3h06dw6X6b70UrhEtwIFD/AA4pxzZWfzZrjrLmjePHSAeN114Z6Ozp3znbMS8Sos55wrC9Omhauqpk+HTp3g7rvhwAPznatS8RKIc87l0po1oZ2jXTv49FN47LFwpVUFDx7gAcQ553LDDMaMCdVVI0eGbtfnz4czzsj5OB1lxauwnHMu2xYsgN69w93jhx0WOj5s2zbfucq6vJZAJN0v6QtJSUcTVHCnpIWSZklqE0s7TtKCKO3qssu1c66s9e4d/rSX98eO+p5B+is/ND+INZOm05u7qTV9KipoS40aYZlUzw0ahEe1atCkSTjmJk3CdGLamDHhUZheOK9QurRExVl2O2aWtwdwNNAGmJMivQvwIiDgCOC/0fzqwP+A/YBawPtAy6L217ZtW3POVSwXX2wW6oPK9+M4XrCF7GcGNpoe9jNW5GxfNWua1aq17bw6dcweeSQ86tRJnpYo02WBaZbkN1UhLX8kNQH+bWbbjfwu6R/Aa2b2WDS9gDAEbhNgoJn9Jpr/ZwAz+3u6fRUUFNi0adOymX3nXI7VqBGufi2v9mIZt3MppzGO+TSjNyN4lV/nJS+NG4fnJUuSpy1evO28Jk0yW1bSdAvDiW+jvDei7wV8EpteFs1LNX87knpKmiZp2sqVK3OWUedcbpTX4FGdTVzKbcyjBScwngEM5mDez1vwAFi6NDxSpWUyL938ROU9gCS7VMHSzN9+ptkoMysws4Ldd989q5lzzuVe9er5zsH2jmAq0yjgNi7nDY6iFXO5gQFsYIe85qtRo/BIlZbJvHTzE5X3ALIMiI/XuDewPM1851wl07NnvnOw1a6s5h/0ZCrtacAqTuUpTmA8i9ivTPNRsybUqrXtvDp1Qs/vQ4aE18nSEhVn2aSSNYyU5YPQnpGqEf0Etm1EfyeaXwP4GNiXrY3orYralzeiO1cx5b8hfYudxYP2BQ1sI9XtFi63n7A27TrVqoXn6tUze65fPzwks8aNwzE3bhymE9MKG8sL0wvnFUqXliiTZSmPjeiSHiM0ijcAPgeuA2oCmNlISQKGA8cB3wHnmtm0aN0uwO2EK7LuN7MiY6Y3ojvnim3u3HBN7euvwxFHhJsCDz4437kqU6ka0fN6I6GZdS8i3YA+KdJeAF7IRb6cc47vvoNBg+CWW6BePRg1Cs4/P9ww4QC/E90557b3r3/BH/8YrnE9+2y4+Wbwi3C246HUOVeudeqUcLf3jsW8W7o4li6Fbt3gpJOgbl2YPBkefNCDRwoeQJxz5VanTvDKK9vOW78ezjory0Fk40a46SZo0SL0lDt0KLz3Hhx9dBZ3Uvl4AHHOlVuJwaPQli0wYECWdvLmm9CmDfTvHyLWvHnhdeJ1sm47HkCccxVSpndLp7RqFZx3Hhx1FKxdC88+C889t7U/EFckDyDOuQop07ult7NlC9x3HzRrBqNHw1VXwQcfQNeuWc1fVeABxDlX7hR2355KtWrFuFs6btasUOK48EJo1Sq0c9x4Y2gwd8XmAcQ5V6707g333JM6XYKHH4YePYqx0XXr4MorQ1vHhx/CAw+EK6xab9cJuCsGvw/EOVeujBqVPt2sGMHDDJ55Bi65BJYtgwsuCFdY1a9f6nw6L4E458qZrHXfvmgRnHginHoq7LorvPUW3HuvB48s8gDinCtXSt19+4YNcMMN0LJlqKa65RaYPh3at89K/txWHkCcc+VKhw7p0zt2TJP42muho8MBA6BLl3BPxxVXhP7PXdZ5AHHOlRtjxoROb1Pp2BFefjlJwhdfhNvTjzkGfvgBxo+HceNgn32SLOyyxQOIc67cGDAg9CqSTOPGSYLHli2he/VmzWDsWLjmGpgzJ5Q+XM75VVjOuXIj3d3l26W99x706gXvvBPqvUaMCH1ZuTLjJRDnXLmR7u7yH9PWrg2X5RYUwOLF4W7ySZM8eORBXgOIpOMkLZC0UNLVSdL/JGlm9JgjabOk3aK0xZJmR2k+zKBzlcCQIcmvwqpVC4YMNnjiCWjeHO66Cy66CObPhz/8If1t6y5n8laFJak6cDdwLLAMeFfS82b2QeEyZnYzcHO0/G+By8xsdWwzx5jZqjLMtnMux6pV2/5ekKtPW0iP0X1CV+uHHho6PmzXLi/5c1vlswTSDlhoZh+b2QZgLJCuN7PuwGNlkjPnXF4kNqLvwHqu5XquebQ1TJ0Kd9wR2jw8eJQL+QwgewGfxKaXRfO2I6kOcBwwLjbbgAmSpkvqmWonknpKmiZp2sqVK7OQbedcrsQbyjsxkVkcxPUM5Bm6heqqfv2ghl/7U17kM4Akq7S0FMv+FngrofrqF2bWBjge6CMp6dBhZjbKzArMrGB3H5bSuXKtUSPYgxU8Sncm0hlhdOYlrm48Fho2zHf2XIIiA4ik/SXtEL3uIKmfpF2ysO9lQPwun72B5SmWPYOE6iszWx49fwE8Q6gSc86VQ61abTuuebJHdW3mt0vuYj7NOYWnGch1/JzZTK7VuWRdt7ucy6QEMg7YLOkA4J/AvsCjWdj3u0BTSftKqkUIEs8nLiRpZ+BXwHOxeXUl1St8DXQG5mQhT865LGvVKozXlE4B7/IO7biLfrxDO37ObK5nID9Qmw0byiafrvgyCSBbzGwTcDJwu5ldBuxZ2h1H2+wLvATMA54ws7mSeknqFVv0ZGCCmX0bm/cz4E1J7wPvAOPN7D+lzZNzLvvSBY+dWcNw+vBfDqchyzmDx+jMBD7iwG2Wy9r45y6rMmmN2iipO3A2oS0CICs9k5nZC8ALCfNGJkw/CDyYMO9j4OBs5ME5lw/G73mUYVzB7qxkOH35K4NYy85Jly71+OcuJzIpgZwLHAkMMbNFkvYFHslttpxzldWBLOBlOjGGP7CURrTjHS7hzpTBA0ox/rnLqSIDSHRjX39gRjS9yMyG5jpjzrmKb8yYra9r8z1/46/M4iDaMp3e3M2RTGUGbdNuo8Tjn7ucy+QqrN8CM4H/RNOHSNqusds55xIVtl0cx4vMoTV/ZTBP8juaM5976M0W0o8eVbt2CcY/d2UmkzaQgYRLZF8DMLOZUTWWc86ltXnJMp7kUk5jHPNpxq95hVf5NRLYlnznzpVWJgFkk5l9rW07K0t1w59zzsGmTXDXXczTtVS3TQxgMLdwJRvYAfA2jcoikwAyR9LvgeqSmgL9gCm5zZZzrsKaOpWFnS/mgHXv8zrH05fhLGK/H5O9TaPyyOQqrD8CrYAfCHeDrwUuzWGenHMV0erV0LMntG9P7XWrOJWnOIHx2wQPCIMIPvBAnvLosqrIEoiZfQcMiB7OObcts9DSfeWV8NVXDONyBjKQddRLucorr5Rh/lzOpAwgkv5FmrYOMzspJzlyzlUcc+dC797w+utwxBEwciRXHuL3+FYV6Uogt5RZLpxzFcu338KgQTBsGNSrB6NGwfnnhwYOV2WkDCBmNrksM+KcqyD+9S/44x9hyRI4+2y4+WaIDZXQsWPRVVQdO+Y4j65MpPy7IOmJ6Hm2pFmJj7LLonOuXFi6FLp1g5NOgrp1YfJkePDBbYIHwMsvpw8QHTuGZVzFl64K65Lo+cSyyIhzrpzauBFuuw2uvz40mA8dCpddBrVqpVzFA0TVkLIEYmYrope9zWxJ/AH0LpvsOefy6s03oU0b6N8fOnWCefPC6zTBw1UdmbR4HZtk3vHZzohzrhxZtQrOOw+OOgrWroVnn4XnnoPGjfOdM1eOpGsDuVjSbKBZQvvHIiArbSCSjpO0QNJCSVcnSe8g6WtJM6PHtZmu61xl07t30cPClvZRTVu4QPfx5e7N2PjAaG7kKuou/QB161ribdaosW2vvK7ySNcG8ijwIvB3IP4D/Y2ZrS7tjiVVB+4mlHCWAe9Kej7qPj7uDTM7sYTrOlcp9O4N99yT2338nFncw8X8gim8zlH0ZgRzaV3q7W7eDGeeGV57r7qVS7o2kK/NbLGZdSf8SG8k3Fj4E0nZ6AqtHbDQzD42sw3AWKBrGazrXIUzalTutl2XddzMlcygDQfyIefwAL9iclaCRyEzH5a2MspkPJC+wOfARGB89Ph3Fva9F/BJbHpZNC/RkZLel/SipFbFXBdJPSVNkzRt5cqVWci2c2Vv8+ZcbNU4maeZRwuuZBgPcC7Nmc9DnAOoqJWLzYelrXwy6Y33UqCZmX2Z5X0nO0MTu06ZATQ2s3WSugDPAk0zXDfMNBsFjAIoKCjwbuhdhVS9enaDSBMWMZy+nMALvM9BnM7jTKV99naQhHfhXvlkchXWJ8DXOdj3MmCf2PTewPL4Ama21szWRa9fAGpKapDJus5VJj17Zmc7NdnAn7mBD2jJr5jM5QyjLdNzHjwk78K9MsqkBPIx8Jqk8YQu3QEws1tLue93gabR6IafAmcAv48vIGkP4HMzM0ntCAHvS2BNUes6V5mMGBGeS9OQ3oFXGUFvWjCfcZzCpdzOsm3+h+VG9erw0EPegF4ZZRJAlkaPWtEjK8xsU9S+8hJQHbjfzOZK6hWljwROAy6WtAn4HjjDzAxIum628uZceTRixNZAUiyffx66Wn/kEdh3Xxg+nlO7dOHUrOfQVTUKv8dVQ0FBgU2bNi3f2XCubGzZEi7f+vOfQ++5f/pTuBSqTp1858xVMJKmm1lB4vwiSyCSdgeuIoxKWLtwvpn9Oqs5dM5lz3vvQa9e8M470KFDKLq0aJHvXLlKJpNG9DHAfGBf4HpgMaH9wjlXhlq12vYO71atkiy0di1ccgkUFMDixTB6NEya5MHD5UQmAaS+mf0T2Ghmk83sPOCIHOfLORfTqhV8kNDPwgcfxIKIGTzxBDRvDnfdBRddBPPnwx/+EKKNczmQSSP6xuh5haQTCJfL7p27LDnnEiUGj23mL1wIffrAhAlw6KGh48N27coye66KyiSADJa0M3AFcBewE3BZTnPlnCvSDqynPzdC67+H7tXvuCN0mlUjk6+1c6VX5JlmZoXdlnwNHJPb7DjnEu2VpJOeTkzkbvpwIB9Bt9Ph1luhYcOyz5yr0jK5CusBknQTErWFOOdyaK+9YHmsj4U9WMGtXE53xvIRB3BK3Zd4emzn/GXQVWmZlHXjHSfWBk7Guw1xrkwUBo9qbKY3IxjMX6jNegZyHUO5mk3ra6ffgHM5lEkV1rj4tKTHAB/x2LkyUsC7jKQXbZnBBI6lD3ezkKYhMSe99DqXmUwu403UFPB+NZ3LtTVrGE4f/svhNGQ5Z/AYv+GlrcGD0M+Uc/mSSRvIN4Q2EEXPnwH9c5wv56ouM3j0UbjiCnqxkuH05a8MYi07b7dotnrpda4kMqnCqlcWGXHOAQsWhEtxJ01i/k6H0YPxzKBt0kUvvriEnSs6lyVpA4ikHYEeQMto1jTgqWgYWedctnz/PdxwA9x0E+y4I9x9N636XMQWUtdRefBw+ZayDUTSz4F5wFGE/q+WAL8B3pK0i6TBZZJD5yq7F1+E1q1h8GD43e9CFyS9e6cNHs6VB+lKIHcCF5rZxPhMSZ2AOYCPv+FcaSxbBpdeCuPGQbNm8Mor8Gvv5NpVHOmuwtozMXgAmNnLhP6xTi7tziUdJ2mBpIWSrk6S3kPSrOgxRdLBsbTFkmZLminJB/lwFcemTXDbbaGH3PHjQ8nj/fe3Cx4dO6beRLo058pKugBSTdIOiTMl1Sb0zPtdaXYsqTpwN3A8oY2lu6SWCYstAn5lZgcBg4BRCenHmNkhyQY6ca5cmjo1dLV++eVw1FEwd24Y5GmH7b5qvPxy8kDRsWNIcy7f0gWQh4FxkpoUzohePwGMzsK+2wELzezjqFF+LNA1voCZTTGzr6LJt/FegF1FtXp1uOa2fXtYtQqeeiqUPvbbL+1qL78cruqNPzx4uPIiZQAxs8HAf4DXJa2StAqYDEw0s0FZ2PdewCex6WXRvFTOB16MZxGYIGm6pJRXw0vqKWmapGkrV64sVYadKzYzeOih0MZx//2h5DFvHpx6qo/T4Sq8tJfxmtlwYLiketH0N1ncd7JvT9IB2iUdQwggv4zN/oWZLZf0U2CipPlm9vp2GzQbRVT1VVBQUHUGgHf5N3duuKfj9dfhyCPhnnvg4IOLXs+5CiKjrkzM7JssBw8IJY59YtN7k6STRkkHAfcBXc3sy1ielkfPXwDPEKrEnMu/b7+Fq69mY+tDWP36bC5kFNWmvokOOXibIWlL8kg6jK1zeVKSvrCy5V2gqaR9JdUCzgCejy8gqRHwNHCmmX0Ym1+3sFQkqS7QmXBpsXP59fzz4Vf+xhsZQw+asYD7uBDL0ldtm2FsncuzvA1dZmabJPUFXgKqA/eb2VxJvaL0kcC1QH1ghEJ98aboiqufAc9E82oAj5rZf/JwGM4FS5dCv37w3HPQsiVHM5k3ODonu0o1vK1zZU1m6ZsFJNUhDGfbyMwulNQUaBYbqbDCKCgosGnT/JYRl0UbN4Z7Oq6/PjSYX3cdXHYZ2qFWTndbxNfWuaySND3Z7RKZlKsfAH4AjoymlwHejYlzb74JbdpA//7QqVO4uqp//zA+uXNVQCYBZH8zu4lw9zlm9j3Jr6ByrmpYtQrOOy/cCLh2LTz7bKi6atz4x0V22SV3u2+ZeLutc3mSSQDZEPXKawCS9ieUSJyrWrZsgfvuC/d0jB4NV10VGiS6dt1u0eHDc5OFli3D1cHOlQeZNKJfR7ihcB9JY4BfAOfkMlPOlTuzZoUBOKZMCSWPESNCD7opDBiQfH7jxrB4cW6y6FxZy2RAqYmSZgBHEKquLjGzVTnPmXPlwbp1MHAg3H477LorPPAAnH12kXeRL11avPnOVUQpA4ikNgmzVkTPjSQ1MrMZucuWc3lmBs88A5dcErpdv+ACGDoU6tfPaPVGjWDJkuTznass0pVAhqVJM8AHLnCV06JF0LcvvPACHHQQPP546AQxA717hx5LkqlTB4YMyWI+ncuzlAHEzI4py4w4l3cbNsAtt8CgQVC9OgwbFm4OrJHZ/bbpggfA+vXQo0eW8upcOVDkNyMa/6M3oSNDA94ARprZ+hznzbmy8+qrIQLMnw+nnBLaPPbZp8jV4kYljlaTYMuWkmfPufIok8t4HwZaAXcBwwmDP2VjPBDn8u/zz+HMM8NogD/8EMboGDeu2MEDYPPmHOTPuXIsk7J5MzOL90H9qqT3c5Uh58rEli2hyPDnP4fec6+5Jlx7W6dOiTdZvboHEVe1ZFICeU/SEYUTkg4H3spdlpzLsffeC+NzXHwxHHJIGI98yJBSBQ8IAw6mU7NmqTbvXLmTSQA5HJgiabGkxcBU4FeSZkualdPcOZdNa9eGy3ILCsLdfKNHw6RJ0KJFVjY/YkSIScnUrBna6J2rTDLpjbdxunQzS3K1e/nkvfFWUWbw5JNw6aXw2WfQq1cocey6a75z5lyFkKo33kzuRF8iaVfC6IE1YvP9RkJX/i1cCH36wIQJcOihoePDdj54pXPZUGQVlqRBwCzgTsLNhcOAW7Kxc0nHSVogaaGkq5OkS9KdUfqs+N3xRa1blY0ZAw0alG7o1Ir+qK31XKfrWd+0NWsnTKUfd1DjvXfQ4e3KNB9eyHGVWSZXYf0foUv3rNbgSqoO3A0cSxhj5F1Jz5tZfLy144Gm0eNw4B7g8AzXrZLGjIFzzw3jHFVVnZjI3fThQD5iLKdzObeygoZ5ycuaNSGIfPVVXnbvXE5l0og+B9glB/tuByw0s4+j4DQWSOwXuyvwsAVvA7tI2jPDdaukAQOqbvDYgxU8Sncm0hlhdOYlujM2b8Gj0Jo1ed29czmTSQnk74RLeecQGwfEzE4q5b73Aj6JTS8jlDKKWmavDNcFQFJPoCdAoyrQk11V7O21GpvpzQgG8xdqs56BXMdQruYHauc7a85VapkEkIeAG4HZQDY7Y0jWH3biJWGplslk3TDTbBQwCsJVWMXJYEWUqhfYyqqAdxlJL9oygwkcSx/uZiFN850t56qETALIKjO7Mwf7Xka4sqvQ3sDyDJeplcG6VdKQIVWjDWRn1jCEAVzMPXzOzzidsTzB/1EeR1vO5fC2zuVTJm0g0yX9XdKRktoUPrKw73eBppL2lVQLOAN4PmGZ54GzoquxjgC+NrMVGa5bJfXoEcY8ynDYigrI+D1jmE9zejGS4fSlOfN5gtMpr8HDG9BdZZVJCeTQ6PmI2LxSjwdiZpsk9QVeAqoD95vZXEm9ovSRwAtAF2Ah8B1wbrp1S5OfyqRHj0rabfiCBaHH3EmT4LDD4J7x9Gvbln75zpdzVVSRd6JXJn4negX1/fdwww1w002w447h9UUXhd4LnXM5V+I70aOVTyB06f7jZS1m9rfsZc+5FF58MYwO+PHHoVh1yy2wxx75zpVzjszuRB8JnA78kVDJ/Dsgbf9YzpXasmVw2mnQpUvoifCVV+CRRzx4OFeOZNKI3t7MzgK+MrPrgSPZ9goo57Jn0ya47bbQQ+748TB4cOhu/delanJzzuVAJlVY30fP30lqCHwJ7Ju7LLkqa+rU0B/6++/D8cfD8OGw3375zpVzLoVMSiD/lrQLcDMwA1gMPJbDPLmqZvXqMBpT+/awahU89VQofXjwcK5cy6Q790HRy3GS/g3UNrOvc5stVyWYwcMPw5VXhpslLr8cBg6EevXynTPnXAZSlkAkHSZpj9j0WcATwCBJu5VF5lwlNncudOgA55wDTZvC9OkwbJgHD+cqkHRVWP8ANgBIOhoYCjwMfE3Ut5Rzxfbtt3D11WEs8tmzYdQoePNNOPjgfOfMOVdM6aqwqpvZ6uj16cAoMxtHqMqamfOcucrn+eehX7/Q2+M554QbA3ffPd+5cs6VULoSSHVJhQGmIzAplpbRDYjOAaGP+W7doGtXqFsXJk8OHXZ58HCuQksXCB4DJktaRbiU9w0ASQcQqrGcS2/jxnBPx/XXhwbzoUPhssugVq1858w5lwUpSyBmNgS4AngQ+KVt7TSrGuGudFdOpBoDvUGDkJYXb74JbdpA//7QqRPMmxdee/BwrtJIWxUVDSObOO/D3GXHFVe6MdC//BLOOy+8LrPeeVetgquuClVUjRrBs8+GqivnXKWTyY2Erhwragz0DRvCMjm3ZQvcdx80awajR4cg8sEHHjycq8S8MbyCy2QM9JyPkz5rVuiCZMoUOOooGDECWrfO8U6dc/mWlxKIpN0kTZT0UfS8a5Jl9pH0qqR5kuZKuiSWNlDSp5JmRo8uZXsE5UejRtlZpkTWrQt3kbdpAx9+GKqtJk/24OFcFZGvKqyrgVfMrCnwSjSdaBNwhZm1IIyG2EdSy1j6bWZ2SPR4IfdZLp+GDAm9nadSq1ZYJqvM4OmnQ4+5w4aFRpj588O9HSp/w8o653IjXwGkK/BQ9PohoFviAma2wsxmRK+/AeYBe5VVBiuKdGOg168P99+f5Qb0RYvgxBPh1FNht93grbfg3nsr8yDszrkU8hVAfmZmKyAECuCn6RaW1IQwNvt/Y7P7Spol6f5kVWBVSY8e4eIns20fq1ZlMXhs2BCGkm3ZMlRTDRsW+q9q3z5LO3DOVTQ5CyCSXpY0J8mjWJflSPoJMA641MzWRrPvAfYHDgFWAMPSrN9T0jRJ01auXFmyg6nqXn019FU1YACccEKorrr8cqjh12A4V5XlLICYWScza53k8RzwuaQ9AaLnL5JtQ1JNQvAYY2ZPx7b9uZltNrMtwL1AuzT5GGVmBWZWsHsl6DpjzBho0iQ0NVSrluObBj//HM48M4wG+MMPYYyOp56CvffO8o6ccxVRvqqwngfOjl6fDTyXuIAkAf8E5pnZrQlpe8YmTwbm5Cif5cqYMWHcpSVLwvSPfQOw9abBrASRzZvhnnugeXN4/HG45hqYMyeMT+6ccxFZ/FeorHYq1SeMLdIIWAr8zsxWR0Pm3mdmXST9ktD/1mxgS7TqNWb2gqTRhOorI4yQeFFhm0o6BQUFNm3atKwfT1lp0mRr8EilcWNYvLgUO3nvPejVC955J4zXMWJEuNrKOVdlSZpuZgWJ8/NSiW1mXxJ6+E2cvxzoEr1+E0h6TaiZnZnTDJZTOb1pcO1a+OtfwzjkDRqEu8l79PDLcp1zKXlXJhVITm4aNIMnngjVVXfdBRddFBrJ//AHDx7OubQ8gFQgQ4ZAnTqp04t90+DChXDccXD66bDHHvD226HKatcqfVW0cy5DHkAqkB49wgiwjRuH6XgBoVg3Da5fH8boaN0apk6FO+4IbR7tUl7M5pxz2/EL+SuYHj1KeXPgxInQpw989FEoedx6KzRsmLX8OeeqDi+BVBUrVkD37tC5c2j3eOklGDvWg4dzrsQ8gFR2mzeHxvHmzUMHiNddB7Nnh0DinHOl4FVYldm774Z7OmbMgGOPhbvvhqZN850r51wl4SWQymjNmtDOcfjhoepq7NhQZeXBwzmXRR5AKhOz0JdJ8+YwciT07Qvz5oXGcr+nwzmXZV6FVVnMnx9KHZMmwWGHhY4P27bNd66cc5WYl0Aquu+/D12QHHRQGJ/j7rvDvR0ePJxzOeYlkIrsxRdDNdXHH4ebQ265JdxR7pxzZcBLIBXRsmVw2mmhe/WaNeGVV+CRRzx4OOfKlAeQimTTJrjtttC9+vjxMHgwvP9+GPDJOefKmFdhVRRTp8LFF4eAcfzxodv1/fbLd66cc1WYl0DKu9WrwzCE7dvDqlVhSNnx4z14OOfyLi8BRNJukiZK+ih6Ttp/uKTFkmZLmilpWnHXL0rh+OLVqoXndMPBpls2WVrivN69t043aBAeqV5LUKO6cbYeYmX9Zmy6935u0+XU+3QeNU4/FVVTkfl1zrmcM7MyfwA3AVdHr68Gbkyx3GKgQUnXT3y0bdvWCj3yiFmdOmbh7rvwqFMnzE+UbtlkaTVrmtWqte284jxaMsde42gzsLc40g5iZtLlUuXXOeeyCZhmSX5T8zUm+gKgg5mtkLQn8JqZNUuy3GKgwMxWlWT9RPEx0VONL55sTPF0y0LR45Rnqg7f8lcGcQXD+IZ69OdG/sn5WJqCYqnHQHfOuSKkGhM9XwFkjZntEpv+ysy2q4aStAj4CjDgH2Y2qjjrR2k9gZ4AjRo1arsk+rWvVi38j99+ediyZdt56ZaF5GnF9Vue50760YQlPMA5XMVNrGL3ItdLll/nnMumVAEkZ1dhSXoZSHZjwoBibOYXZrZc0k+BiZLmm9nrxclHFHRGQSiBFM5v1Ch5ySHZmOJFLVuaEsg+LOVO+tGN55hLS45mMm9wdMbrF3sMdOecy5KcNaKbWScza53k8RzweVT1RPT8RYptLI+evwCeAQrHXM1o/XSSjS9ep07yMcXTLZssrWbNMD55OjXYyJ+4iXm0oDMT6M9QDuW9YgWPVPl1zrmykK/LeJ8Hzo5enw08l7iApLqS6hW+BjoDczJdvyjx8cWl8DxqVPLhYtMtmyztgQfC+OTxeRdfvHX6hJ3eYFb1Q7mJ/rxesxPtd5nHzerPTvVrUb9+2Gf16umf0+XXOefKQr7aQOoDTwCNgKXA78xstaSGwH1m1kXSfoRSB4SqtkfNbEi69Yvab7wRPS9WrYKrrgoRplEjuPNO6No1f/lxzrkMlHkbSDpm9iXQMcn85UCX6PXHwMHFWb/c2rIlFEn694e1a0MQufZaqFs33zlzzrkS865Mcm3WrFB/NWUKHHUUjBgBrVvnO1fOOVdq3pVJrqxbB1deCW3awIcfhmqryZM9eDjnKg0vgWSbGTzzDFxySeh2/YILYOhQfmwdd865SsJLINm0aBGceCKceirsthu89Rbce68HD+dcpeQBJBs2bIAbboCWLUM11bBhYXjZ9u3znTPnnMsZr8IqrVdfDV3tzp8fSh633w57753vXDnnXM55CaSkPv8czjwzjAb4ww9hjI6nnvLg4ZyrMjyAFNfmzXDPPdC8OTz+OAwYAHPmhPHJnXOuCvEqrOKYMSPc0/HOO9ChQ7ino0WLfOfKOefywksgmVi7NlyWe9hhYfCN0aNh0iQPHs65Ks1LIOmYwZNPwqWXwmefQa9eofvbXUs0gq5zzlUqHkBSWbgQ+vSBCRPg0EPh2WehXbsiV3POuarCq7ASrV8P118fuhyZOhXuuCO0eXjwcM65bXgJJG7ixFDq+OgjOP10uPVWaNgw37lyzrlyyUsgACtWQPfu0LlzaPd46SUYO9aDh3POpZGXACJpN0kTJX0UPW/XKi2pmaSZscdaSZdGaQMlfRpLK/lNGM8+G+7pePppuO46mD07BBLnnHNp5asEcjXwipk1BV6JprdhZgvM7BAzOwRoC3zH1hEKAW4rTDezF0qck2bN4Je/DDcDDhwItWuXeFPOOVeV5KsNpCvQIXr9EPAa0D/N8h2B/5nZkqznpEWL0A2Jc865YslXCeRnZrYCIHr+aRHLnwE8ljCvr6RZku5PVgVWSFJPSdMkTVu5cmXpcu2cc+5HOQsgkl6WNCfJo2sxt1MLOAl4Mjb7HmB/4BBgBTAs1fpmNsrMCsysYPfddy/+gTjnnEsqZ1VYZtYpVZqkzyXtaWYrJO0JfJFmU8cDM8zs89i2f3wt6V7g39nIs3POuczlqwrreeDs6PXZwHNplu1OQvVVFHQKnQzMyWrunHPOFSlfAWQocKykj4Bjo2kkNZT04xVVkupE6U8nrH+TpNmSZgHHAJeVTbadc84VystVWGb2JeHKqsT5y4EusenvgO0GFDezM3OaQeecc0XyO9Gdc86ViAcQ55xzJSIzy3ceyoyklUCmNyM2AFblMDtlzY+nfPPjKf8q2zEV53gam9l290FUqQBSHJKmmVlBvvORLX485ZsfT/lX2Y4pG8fjVVjOOedKxAOIc865EvEAktqofGcgy/x4yjc/nvKvsh1TqY/H20Ccc86ViJdAnHPOlYgHEOeccyXiASQNSYOiMUdmSpogqUIPki7pZknzo2N6RtIu+c5TaUj6naS5krZIqrCXV0o6TtICSQslbTc6Z0USjc/zhaRK0cGppH0kvSppXnSuXZLvPJWGpNqS3pH0fnQ815dqe94GkpqkncxsbfS6H9DSzHrlOVslJqkzMMnMNkm6EcDM0o0EWa5JagFsAf4BXGlm0/KcpWKTVB34kNBp6DLgXaC7mX2Q14yVkKSjgXXAw2bWOt/5Ka2o5+89zWyGpHrAdKBbBf58BNQ1s3WSagJvApeY2dsl2Z6XQNIoDB6RukCFjrZmNsHMNkWTbwN75zM/pWVm88xsQb7zUUrtgIVm9rGZbQDGEoZ8rpDM7HVgdb7zkS1mtsLMZkSvvwHmAXvlN1clZ8G6aLJm9Cjx75oHkCJIGiLpE6AHcG2+85NF5wEv5jsTjr2AT2LTy6jAP1CVmaQmwKHAf/OclVKRVF3STMJAfhPNrMTHU+UDSFFD75rZADPbBxgD9M1vbouWyVDCkgYAmwjHVK5la2jkckxJ5lXokm5lJOknwDjg0oSaiQrHzDab2SGEGoh2kkpc1ZiX8UDKk3RD7yZ4FBgPXJfD7JRaUccj6WzgRKCjVYAGsGJ8PhXVMmCf2PTewPI85cUlEbUVjAPGmFni4HYVlpmtkfQacBwlHNW1ypdA0pHUNDZ5EjA/X3nJBknHAf2Bk6LBulz+vQs0lbSvpFrAGYQhn105EDU6/xOYZ2a35js/pSVp98KrLyXtCHSiFL9rfhVWGpLGAc0IV/osAXqZ2af5zVXJSVoI7AB8Gc16u4JfVXYycBewO7AGmGlmv8lrpkpAUhfgdqA6cL+ZDclvjkpO0mNAB0JX4Z8D15nZP/OaqVKQ9EvgDWA24XcA4BozeyH1WuWXpIOAhwjnWjXgCTP7W4m35wHEOedcSXgVlnPOuRLxAOKcc65EPIA455wrEQ8gzjnnSsQDiHPOuRLxAOIqHEn1ox6SZ0r6TNKn0es1ksq0kztJ3SS1jE3/TVKxb36U1CRVD7aSWkmaJOlDSf+TdL2krH930x2LpNcqco/HLjc8gLgKx8y+NLNDou4YRgK3Ra8PYeu1+lkjKV2PDd2AH390zexaM3s5i/vekXBj4VAzOxD4OaEDxlx0K96NHB6Lq3w8gLjKprqke6OxDiZEP8BI2l/SfyRNl/SGpObR/MaSXonGSHlFUqNo/oOSbpX0KnBjsvUltSf0UHBzVALaP1rvtGgbh0maEo298I6kelFJ4w1JM6JH+yKO5/fAW2Y2ASDqQaAv8KdoHwMlXVm4cNRPWJPo9bNRfudK6hlbZl3USej7kt6W9LOijiVOUmdJU6P8Pxn1E4WkoZI+iN7LW4r/0bmKxgOIq2yaAnebWSvC3emnRvNHAX80s7bAlcCIaP5wwtgVBxE6l7wztq0DgU5mdkWy9c1sCqF08KeoRPS/whWjbkkeJ4y1cDChy4jvCT2gHmtmbYDTE/aXTCvCGBQ/ivazo4oeEOy8KL8FQD9J9aP5dQm9EBwMvA5cmO5Y4iQ1AP4SvS9tgGnA5ZJ2A04GWkXv5eAi8uYqgSrfmaKrdBaZ2czo9XSgSfQPuT3wZOjaCAhdugAcCZwSvR4N3BTb1pNmtrmI9VNpBqwws3dh69gykuoCwyUdAmwmBKl0RPLeeZP14puoX9TdC4QOG5sSurHZAPw7mj+dMJhVpo4gVHO9Fb0XtYCpwFpgPXCfpPGx7btKzAOIq2x+iL3eDOxIKGmvidpJihL/sf42ei7O+oVS/fBfRugj6uBou+uL2M5c4OhtNiztB6yKelPdxLY1CbWjZToQSj1Hmtl3Cr2u1o6W2RjriXkzxfsdEGEMie7bJUjtgI6EDiH7Ar8uxnZdBeRVWK7Si/79L5L0Owg9rEo6OEqeQvjBgzBo2JvFXP8boF6S3c4HGko6LFqnXtQYvzOhZLIFOJPQqV06Y4Bfxq6G2pFQ7VU4rMBioE2U1gbYN5q/M/BVFDyaE0oORUl1LHFvA7+QdEC0zzqSDoxKaTtHnQxeSrigwVVyHkBcVdEDOF/S+4R/9YUDUvUDzpU0i/CDnurqplTrjwX+JOk9SfsXLhwNT3s6cFe0zkRCCWAEcLaktwnVV9+Shpl9T2jcHiDpQ2AVoVG9cDCwccBuCiPMXUwYXx3gP0CN6LgGEX74i5L0WBLysxI4B3gs2vbbQHNC4Pl3NG8yoaTlKjnvjde5CkRSN+BW4BgzW5Ln7LgqzgOIc865EvEqLOeccyXiAcQ551yJeABxzjlXIh5AnHPOlYgHEOeccyXiAcQ551yJ/D+yZ0/0+8OM4AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Is the bootstrapped distribution Normal?\n",
    "fig = sm.qqplot(median_diffs, line='s')\n",
    "pyplot.title('Quantiles of standard Normal vs. bootstrapped median')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep in mind that the **CLT only applies to sample means**, not medians or any other statistic. None the less, you can see that the Q-Q normal plot of the bootstrap distribution of the difference in medians tends down the center diagonal of the plot. The zig-zag behavior arises from the quantization inherent in the median."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Jackknife\n",
    "\n",
    "The jackknife is another resampling method. The jackknife is related to the bootstrap, but is a bit more restrictive in its application. Specifically, the jackknife:\n",
    "\n",
    "- Re-computes a statistic N times with leave-one-out sample, where N is the sample size.\n",
    "- Uses Random leave one (or n) out sampling.\n",
    "- Only works for statistics with continuous derivatives.\n",
    "\n",
    "For example, you can compute the bootstrap mean as:\n",
    "\n",
    "$$MeanJK = \\frac{\\Sigma_i mean(sample_i)}{nsample}\\\\\n",
    "where,\\ for\\ example\\ with\\ 10\\ samples,\\\\\n",
    "sample_i = X_1 + X_2 + X_3 + X_4 + X_5 + X_6 + X_8 + X_9 + X_{10}$$\n",
    "\n",
    "A simple example of using a jackknife estimation of a mean is shown in the cell below. Run this code, examine the results and compare them to those obtained with the bootstrap. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have 481 male rows and 453 female rows\n"
     ]
    }
   ],
   "source": [
    "print(\"We have %d male rows and %d female rows\" % (len(male), len(female)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the index of each row within a set (either `male` or `female`) to leave each of the observations (rows) out, one at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Re-number the rows in each set starting from zero or else the row indices\n",
    "# would still match the original `families` dataframe we started with.\n",
    "male = male.reset_index(drop=True)\n",
    "female = female.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can drop any particular row(s) we like and compute our jackknife sample mean on the remaining rows.\n",
    "\n",
    "One trick to selecting all rows except for a given row index is the following, where we drop the row at index==3 (note the left-most column values skip 3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>father</th>\n",
       "      <th>mother</th>\n",
       "      <th>midparentHeight</th>\n",
       "      <th>children</th>\n",
       "      <th>childNum</th>\n",
       "      <th>gender</th>\n",
       "      <th>childHeight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>78.5</td>\n",
       "      <td>67.0</td>\n",
       "      <td>75.43</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>73.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>75.5</td>\n",
       "      <td>66.5</td>\n",
       "      <td>73.66</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>73.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>75.5</td>\n",
       "      <td>66.5</td>\n",
       "      <td>73.66</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>72.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>75.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72.06</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>70.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>75.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>72.06</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>68.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   father  mother  midparentHeight  children  childNum gender  childHeight\n",
       "0    78.5    67.0            75.43         4         1   male         73.2\n",
       "1    75.5    66.5            73.66         4         1   male         73.5\n",
       "2    75.5    66.5            73.66         4         2   male         72.5\n",
       "4    75.0    64.0            72.06         5         1   male         70.5\n",
       "5    75.0    64.0            72.06         5         2   male         68.5"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "male.drop(male.iloc[[3]].index).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then compute a summary statistic of our choice on the rows that were not dropped like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "69.23041666666668"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "male.drop(male.iloc[[3]].index).childHeight.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now use this strategy to leave each of the rows out, one at a time, and collect the summary statistics (child height mean) for each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    481.000000\n",
       "mean      69.234096\n",
       "std        0.005466\n",
       "min       69.213750\n",
       "25%       69.230417\n",
       "50%       69.234167\n",
       "75%       69.237708\n",
       "max       69.253333\n",
       "dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "male_jk_means = pandas.Series([\n",
    "        male.drop(male.iloc[[i]].index).childHeight.mean()\n",
    "        for i in range(len(male))])\n",
    "\n",
    "male_jk_means.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do the same for the gender==female subset of our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    453.000000\n",
       "mean      64.103974\n",
       "std        0.005212\n",
       "min       64.089823\n",
       "25%       64.100885\n",
       "50%       64.104204\n",
       "75%       64.107522\n",
       "max       64.121903\n",
       "dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "female_jk_means = pandas.Series([\n",
    "        female.drop(male.iloc[[i]].index).childHeight.mean()\n",
    "        for i in range(len(female))])\n",
    "\n",
    "female_jk_means.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross Validation\n",
    "\n",
    "Cross validation is a leave-many-out resampling technique. Cross validation uses multiple non-overlapping subsamples to train and test a data set. Basic cross validation uses the following steps:\n",
    "\n",
    "- Divide dataset into N subsamples \n",
    "- N – 1 Folds train model \n",
    "- One Fold evaluate model\n",
    "\n",
    "The basic concept of cross validation is illustrated below. \n",
    "\n",
    "<img src=\"https://library.startlearninglabs.uw.edu/DATASCI410/img/CrossValidation.jpg\" style=\"height: 200px;\" title=\"Cross validation illustrated\" />\n",
    "\n",
    "Model performance is performed by looking at aggregated summary statistics across the the trained models using the test data for each resample. The resampling reduces bias in the model performance statistics. Most importantly, cross validation reduces unpleasant surprises when a model is placed in production. \n",
    "\n",
    "The jackknife is an extreme end member of the family of cross validation methods. With the jackknife, the number of folds equals the number of data points. \n",
    "\n",
    "There are several variations on cross validation. The most commonly used in **nest cross validation**. In nested cross validation, cross validation is performed on one or more models to compare performance.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<div id=\"reminder\" style=\"border-radius: 5px; background-color:#f5f5f5;\" >\n",
    "<h3>Reminder</h3>\n",
    "<p>Use this notebook to answer the quiz questions related to the <b>Your Turn</b> sections.<p>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}