index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Using fastText to classify reddit submissions\n",
    "\n",
    "## A quick and dirty tutorial\n",
    "\n",
    "Will Radford - @wejradford"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# fastText\n",
    "\n",
    "Concerted programme of research from Facebook towards practical text classification. Three key papers from 2016:\n",
    "* [Enriching Word Vectors with Subword Information](https://arxiv.org/abs/1607.04606)\n",
    "* [Bag of Tricks for Efficient Text Classification](https://arxiv.org/abs/1607.01759)\n",
    "* [FastText.zip: Compressing text classification models](https://arxiv.org/abs/1612.03651)\n",
    "\n",
    "Open source at [GitHub](https://github.com/facebookresearch/fastText) and [elsewhere](https://fasttext.cc/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# How does it work?\n",
    "\n",
    "![title](https://github.com/wejradford/wejradford.github.io/raw/master/fastTextArch.png)\n",
    "\n",
    "* Lowest layer maps n-grams to `d`-dimension word embeddings (à la word2vec) _smushed_ together with averaging.\n",
    "* Hidden layer maps each dimension to a weight.\n",
    "* Top layer is the label."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Mixing neural and shallow models\n",
    "\n",
    "* Word embeddings for a **task-specific** representation\n",
    "* Linear models for **fast** learning and prediction\n",
    "  \n",
    "$\\rightarrow$ **Faster and hopefully better than bag-of-words**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Dealing with large vocabulary\n",
    "\n",
    "* Word n-grams give more context than words alone\n",
    "```python\n",
    ">>> tokens('the cat sat on the mat')\n",
    "['the', 'cat', 'sat', 'on', 'the', 'mat']\n",
    ">>> ngrams('the cat sat on the mat')\n",
    "['START_the', 'the_cat', 'cat_sat', 'sat_on', 'on_the', 'the_mat', 'mat_END']\n",
    "```\n",
    "* But:\n",
    "  * Much sparser features (more zeros than non-zeros)\n",
    "  * Need to maintain the mappings from `sat_on` to `feature_12345`, which costs memory and slows multicore learning\n",
    "\n",
    "* _The hashing trick_ (see also [vowpal wabbit](https://github.com/JohnLangford/vowpal_wabbit) maps n-grams into a constrained memory space\n",
    "\n",
    "$\\rightarrow$ **More memory-efficient and parallelisable learning**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Classifying in large label spaces\n",
    "\n",
    "* Linear classifiers run at $O(fk)$ where:\n",
    "  * $f$ the number of features (i.e. dimensions)\n",
    "  * $k$ is the number of labels\n",
    "* Hierarchical softmax reduces this to $O(f\\log_{2}k)$ by building a tree of labels arranged by probability.\n",
    "\n",
    "$\\rightarrow$ **Faster with some accuracy cost**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Compressing models\n",
    "\n",
    "* Large models limit the deployment environment and increase load time\n",
    "* _Vector quantisation_ lossily compresses the model\n",
    "\n",
    "$\\rightarrow$ **Smaller models at some accuracy cost**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Handling unseen words\n",
    "\n",
    "* Bag-of-words limits the model to handling those words seen in training (maybe not for hashing...).\n",
    "* Learn **character** n-grams embeddings to handle morphology (e.g. \"**mak**e\", \"**mak**ing\", \"**mak**o\") and unknown words.\n",
    "* Pre-initialising word embeddings with those learned from different corpora.\n",
    "\n",
    "$\\rightarrow$ **More robust models**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A toy task\n",
    "\n",
    "* Reddit submissions collected at http://files.pushshift.io/reddit/submissions/\n",
    "* Downloadable as `json`\n",
    "  * `title` The title of the submission\n",
    "  * `subreddit` The subreddit it was submitted to\n",
    "  \n",
    "$\\rightarrow$ Given the `title`, predict which `subreddit` it belongs to."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Installation\n",
    "\n",
    "* Install [homebrew](https://brew.sh) if you're on Mac OSX\n",
    "* Install python3 and some packaging utilities:\n",
    "```bash\n",
    "brew install python3\n",
    "pip3.6 install -U pip wheel\n",
    "```\n",
    "* Clone the [fastText](https://github.com/facebookresearch/fastText) repo, and build `fastText`:\n",
    "```bash\n",
    "git clone https://github.com/facebookresearch/fastText.git\n",
    "cd fastText\n",
    "make\n",
    "```\n",
    "* If you want to use the python bindings (we don't here):\n",
    "```bash\n",
    "cd python\n",
    "pip3.6 install numpy scipy pybind11\n",
    "python3.6 setup bdist_wheel\n",
    "pip3 install dist/*whl\n",
    "```\n",
    "* Install the notebook environment:\n",
    "```bash\n",
    "pip3 install jupyter seaborn ujson nltk sklearn\n",
    "jupyter notebook\n",
    "```\n",
    "* Fetch the data:\n",
    "```bash\n",
    "wget http://files.pushshift.io/reddit/submissions/RS_2017-08.bz2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Preprocessing\n",
    "\n",
    "* Take each post\n",
    "* Find the first sentence\n",
    "* Apply a `really dumb tokeniser to split words !`\n",
    "* Use the subreddit topic as the label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Streaming through reddit submissions\n",
    "\n",
    "* First 50K used to find the top-20 common labels (plus `other`), and for training embeddings\n",
    "* Next 10K for training\n",
    "* Next 10K for development\n",
    "* Next 1K for test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Training the model\n",
    "\n",
    "* Wrap command-line `fastText` using some python\n",
    "* Call and predict, returning:\n",
    "  * Precision\n",
    "  * Recall\n",
    "  * F1\n",
    "  * Training time\n",
    "  * Model size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Sanity-check baseline \n",
    "\n",
    "* Using `scikit-learn`, nothing fancy\n",
    "* Bag of words (unigrams and bigrams)\n",
    "* Logistic regression prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "% matplotlib inline\n",
    "import bz2\n",
    "from collections import Counter\n",
    "import lzma\n",
    "import os\n",
    "import pprint\n",
    "import re\n",
    "import subprocess\n",
    "import time\n",
    "\n",
    "import ujson as json\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "sns.set_context('talk')\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import metrics\n",
    "from sklearn.externals import joblib\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Tokenisation expression.\n",
    "EXP = re.compile('\\w+|\\$[\\d\\.]+|\\S+')\n",
    "FINAL = {'.', '?', '!'}\n",
    "\n",
    "\n",
    "def get_first_sentence(s):\n",
    "    \"\"\" A very naïve method for finding\n",
    "    the first sentence.\n",
    "    \n",
    "    Returns:\n",
    "        A whitespace-normalised version of the sentence.\n",
    "    \"\"\"\n",
    "    tokens = []\n",
    "    for m in EXP.finditer(s.replace('\\n', ' ')):\n",
    "        t = m.group(0)\n",
    "        if not t.strip():\n",
    "            continue\n",
    "        if t in FINAL or len(tokens) == 50:\n",
    "            break\n",
    "        tokens.append(t)\n",
    "    return ' '.join(tokens)\n",
    "\n",
    "\n",
    "def write_data(data, fname, write_label=True):\n",
    "    \"\"\" Writes data to file in fastText format. \"\"\"\n",
    "    with open(fname, 'w') as f:\n",
    "        for label, text in data:\n",
    "            if write_label:\n",
    "                f.write('__label__{} {}\\n'.format(label, text))\n",
    "            else:\n",
    "                f.write('{}\\n'.format(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def iter_reddit(fname):\n",
    "    \"\"\" Yields tuples of (created, subreddit, sentence). \"\"\"\n",
    "    _open = open\n",
    "    if fname.endswith('.xz'):\n",
    "        _open = lzma.open\n",
    "    elif fname.endswith('.bz2'):\n",
    "        _open = bz2.open\n",
    "    with _open(fname) as f:\n",
    "        read = 0\n",
    "        for l in f:\n",
    "            l = json.loads(l)\n",
    "            # pprint.pprint(l)\n",
    "            sentence = get_first_sentence(l['title'])\n",
    "            if not sentence:\n",
    "                continue\n",
    "            subreddit = l.get('subreddit')\n",
    "            if subreddit is None:\n",
    "                continue\n",
    "            yield l['created_utc'], subreddit, sentence\n",
    "            read += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def build_data(fname):\n",
    "    \"\"\" Passes through the data, collecting and returning some datasets. \"\"\"\n",
    "    burnin = []\n",
    "    train = []\n",
    "    dev = []\n",
    "    test = []\n",
    "    burnin_n = 100000\n",
    "    train_n = 10000\n",
    "    dev_n = 1000\n",
    "    test_n = 1000\n",
    "    subreddits = Counter()\n",
    "    top_20_subreddits = None\n",
    "    print('Finding {} sentences for burnin'.format(burnin_n))\n",
    "    for i, (created, subreddit, body) in enumerate(iter_reddit(fname)):\n",
    "        if 'dirty' in subreddit.lower():\n",
    "            continue\n",
    "        # Get a count of common subreddits.\n",
    "        if i < burnin_n:\n",
    "            subreddits[subreddit] += 1\n",
    "            burnin.append(body)\n",
    "            continue\n",
    "        elif not top_20_subreddits:\n",
    "            top_20_subreddits = set(k for k, _ in subreddits.most_common(20))\n",
    "            print('Found top 20 subreddits {}'.format(top_20_subreddits))\n",
    "        elif subreddit not in top_20_subreddits:\n",
    "            continue\n",
    "        \n",
    "        if i % 1000 == 0:\n",
    "            print('Collected train/dev/test {}/{}/{}'.format(len(train), len(dev), len(test)))\n",
    "\n",
    "        if len(train) < train_n:\n",
    "            train.append((subreddit, body))\n",
    "        elif len(dev) < dev_n:\n",
    "            dev.append((subreddit, body))\n",
    "        elif len(test) < test_n:\n",
    "            test.append((subreddit, body))\n",
    "        else:\n",
    "            break\n",
    "    print('Collected train/dev/test {}/{}/{}'.format(len(train), len(dev), len(test)))\n",
    "    return burnin, train, dev, test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finding 100000 sentences for burnin\n",
      "Found top 20 subreddits {'newsbotbot', 'AskReddit', 'GlobalOffensiveTrade', 'RocketLeagueExchange', 'aww', 'Overwatch', 'AutoNewspaper', 'gonewild', 'MemezForDayz', 'gameofthrones', 'The_Donald', 'business', 'Showerthoughts', 'gaming', 'pics', 'funny', 'videos', 'Ice_Poseidon', 'news', 'me_irl'}\n",
      "Collected train/dev/test 0/0/0\n",
      "Collected train/dev/test 931/0/0\n",
      "Collected train/dev/test 3011/0/0\n",
      "Collected train/dev/test 4037/0/0\n",
      "Collected train/dev/test 5511/0/0\n",
      "Collected train/dev/test 5663/0/0\n",
      "Collected train/dev/test 6523/0/0\n",
      "Collected train/dev/test 7155/0/0\n",
      "Collected train/dev/test 7946/0/0\n",
      "Collected train/dev/test 8147/0/0\n",
      "Collected train/dev/test 9690/0/0\n",
      "Collected train/dev/test 10000/542/0\n",
      "Collected train/dev/test 10000/801/0\n",
      "Collected train/dev/test 10000/948/0\n",
      "Collected train/dev/test 10000/1000/1000\n"
     ]
    }
   ],
   "source": [
    "def setup_data(clean=False):\n",
    "    \"\"\" Sets up the data -- CAUTION: this can take a bit... \"\"\"\n",
    "    if clean and os.path.exists('train.json'):\n",
    "        os.remove('train.json')\n",
    "    if not os.path.exists('train.json'):\n",
    "        burnin, train, dev, test = build_data('/Users/admin/fastText-tute/RS_2017-08.bz2')\n",
    "        for data, fname in ((burnin, 'burnin.json'), (train, 'train.json'), \n",
    "                            (dev, 'dev.json'), (test, 'test.json')):\n",
    "            with open(fname, 'w') as f:\n",
    "                f.write(json.dumps(data))\n",
    "    else:\n",
    "        burnin = json.loads(open('burnin.json').read())\n",
    "        train = json.loads(open('train.json').read())\n",
    "        dev = json.loads(open('dev.json').read())\n",
    "        test = json.loads(open('test.json').read())\n",
    "    return burnin, train, dev, test\n",
    "\n",
    "        \n",
    "burnin, train, dev, test = setup_data(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def fastText(train, test, name='fastText', ngrams=2, epoch=90, bucket=200000, dim=50,\n",
    "             maxn=1,\n",
    "             vectors=None, quantize=False, report=True):\n",
    "    \"\"\" Trains supervised fastText models. \"\"\"\n",
    "    train_fname = 'train.txt'\n",
    "    test_fname = 'test.txt'\n",
    "    model_fname = 'model'\n",
    "    write_data(train, train_fname)\n",
    "    write_data(test, test_fname)\n",
    "    cmd = 'fastText/fasttext'\n",
    "    args = [\n",
    "        'supervised',\n",
    "        '-input {}'.format(train_fname),\n",
    "        '-output {}'.format(model_fname),\n",
    "        '-wordNgrams {}'.format(ngrams),\n",
    "        '-epoch {}'.format(epoch),\n",
    "        '-bucket {}'.format(bucket),\n",
    "        '-dim {}'.format(dim),\n",
    "        '-maxn {}'.format(maxn),\n",
    "        '-loss hs',\n",
    "    ]\n",
    "    if vectors:\n",
    "        args.append('-pretrainedVectors {}'.format(vectors))\n",
    "    start = time.time()\n",
    "    output = subprocess.getoutput('{} {} 2> model.log'.format(cmd, ' '.join(args)))\n",
    "    if quantize:\n",
    "        cmd = 'fastText/fasttext'\n",
    "        args = [\n",
    "            'quantize',\n",
    "            '-input {}'.format(train_fname),\n",
    "            '-output {}'.format(model_fname),\n",
    "            '-qnorm -retrain -epoch 1 -cutoff 100000',\n",
    "        ]\n",
    "        output = subprocess.getoutput('{} {} 2> quant.log'.format(cmd, ' '.join(args)))\n",
    "    elapsed = time.time() - start\n",
    "    model_fname = 'model.bin' if not quantize else 'model.ftz'\n",
    "    test_fname = 'test.txt'\n",
    "    cmd = 'fastText/fasttext predict'\n",
    "    args = [\n",
    "        model_fname,\n",
    "        test_fname,\n",
    "    ]\n",
    "    output = subprocess.getoutput('{} {} 2> predict.log'.format(cmd, ' '.join(args)))\n",
    "    Y_test = [i[0] for i in test]\n",
    "    pred = [l.strip().replace('__label__', '') for l in output.strip().split('\\n')]\n",
    "    stat = os.stat(model_fname)\n",
    "    if report:    \n",
    "        print(metrics.classification_report(Y_test, pred))\n",
    "    return {\n",
    "        'p': metrics.precision_score(Y_test, pred, average='micro'),\n",
    "        'r': metrics.recall_score(Y_test, pred, average='micro'),\n",
    "        'f': metrics.f1_score(Y_test, pred, average='micro'),\n",
    "        'time': elapsed,\n",
    "        'size': stat.st_size,\n",
    "        'name': name,\n",
    "        'ngrams': ngrams, \n",
    "        'epoch': epoch, \n",
    "        'bucket': bucket, \n",
    "        'dim': dim,\n",
    "        'maxn': maxn,\n",
    "        'vectors': vectors is not None,\n",
    "        'quantize': quantize,\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def to_xy(data):\n",
    "    \"\"\" Transforms a list of (label, text) tuples to lists as per sklearn. \"\"\"\n",
    "    X, Y = [], [] \n",
    "    for label, text in data:\n",
    "        X.append(text)\n",
    "        Y.append(label)\n",
    "    return X, Y\n",
    "\n",
    "\n",
    "def baseline(train, test, name='baseline', penalty='l2', C=10, report=True):\n",
    "    \"\"\" Trains a baseline model. \"\"\"\n",
    "    c = Pipeline([\n",
    "        ('vec', HashingVectorizer(ngram_range=(1, 2))),\n",
    "        ('clf', LogisticRegression(penalty=penalty, C=C)),\n",
    "    ])\n",
    "    X_train, Y_train = to_xy(train)\n",
    "    start = time.time()\n",
    "    c.fit(X_train, Y_train)\n",
    "    elapsed = time.time() - start\n",
    "    X_test, Y_test = to_xy(test)\n",
    "    pred = c.predict(X_test)\n",
    "    model_fname = 'model.sklearn'\n",
    "    joblib.dump(c, model_fname)\n",
    "    stat = os.stat(model_fname)\n",
    "    if report:\n",
    "        print(metrics.classification_report(Y_test, pred))\n",
    "    return {\n",
    "        'p': metrics.precision_score(Y_test, pred, average='micro'),\n",
    "        'r': metrics.recall_score(Y_test, pred, average='micro'),\n",
    "        'f': metrics.f1_score(Y_test, pred, average='micro'),\n",
    "        'time': elapsed,\n",
    "        'size': stat.st_size,\n",
    "        'name': name,\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Manual hyper-parameter optimisation\n",
    "\n",
    "Train on `train`, measure on `dev`.\n",
    "\n",
    "## Baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "l2\t0.1\n",
      "l2\t1\n",
      "l2\t10\n",
      "l2\t100\n"
     ]
    }
   ],
   "source": [
    "baseline_results = []\n",
    "for p in ('l2',):\n",
    "    for c in (0.1, 1, 10, 100):\n",
    "        print('{}\\t{}'.format(p, c))\n",
    "        baseline_results.append(baseline(train, dev, \n",
    "                                         name='baseline-{}-{}'.format(p, c), \n",
    "                                         penalty=p, C=c, report=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.69\tbaseline-l2-100\t27.691969871520996\n",
      "0.68\tbaseline-l2-10\t22.016381978988647\n",
      "0.63\tbaseline-l2-1\t14.492698907852173\n",
      "0.49\tbaseline-l2-0.1\t10.064432859420776\n"
     ]
    }
   ],
   "source": [
    "for r in sorted(baseline_results, key=lambda d: d['f'], reverse=True):\n",
    "    print('{:.2f}\\t{}\\t{}'.format(r['f'], r['name'], r['time']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## fastText"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "dim_results = []\n",
    "for epoch in (80, 90, 100):\n",
    "    for dim in (50, 100, 200):\n",
    "        for bucket in (100000, 200000, 300000):\n",
    "            name = 'fastText-e{}-d{}-b{}'.format(epoch, dim, bucket)\n",
    "            dim_results.append(fastText(train, dev, name, \n",
    "                                            bucket=bucket, epoch=epoch, \n",
    "                                            ngrams=2, dim=dim, \n",
    "                                            report=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.68\tfastText-e80-d200-b300000\t16.47849178314209\n",
      "0.68\tfastText-e100-d50-b100000\t5.685117959976196\n",
      "0.68\tfastText-e90-d100-b300000\t10.012304782867432\n",
      "0.68\tfastText-e80-d200-b100000\t11.206809043884277\n",
      "0.68\tfastText-e80-d50-b100000\t4.583491086959839\n"
     ]
    }
   ],
   "source": [
    "for r in list(sorted(dim_results, key=lambda i: i['f'], reverse=True))[:5]:\n",
    "    print('{:.2f}\\t{}\\t{}'.format(r['f'], r['name'], r['time']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "results = []\n",
    "results.append(baseline(train, test, name='baseline', penalty='l2', C=10, report=False))\n",
    "results.append(fastText(train, test, \n",
    "                        name='fastText', \n",
    "                        ngrams=2,\n",
    "                        epoch=90, dim=50, bucket=300000, report=False))\n",
    "results.append(fastText(train, test, \n",
    "                        name='fastText-quantized', \n",
    "                        ngrams=2,\n",
    "                        epoch=90, dim=50, bucket=300000, report=False, quantize=True))\n",
    "df = pd.DataFrame(results, columns=['name', 'time', 'size', 'p', 'r', 'f'])\n",
    "df['size'] /= 1000000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Count</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1700</td>\n",
       "      <td>AutoNewspaper</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1248</td>\n",
       "      <td>AskReddit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1085</td>\n",
       "      <td>GlobalOffensiveTrade</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>740</td>\n",
       "      <td>news</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>691</td>\n",
       "      <td>RocketLeagueExchange</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>637</td>\n",
       "      <td>newsbotbot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>473</td>\n",
       "      <td>The_Donald</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>472</td>\n",
       "      <td>business</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>459</td>\n",
       "      <td>videos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>389</td>\n",
       "      <td>Showerthoughts</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Count                 Label\n",
       "16   1700         AutoNewspaper\n",
       "11   1248             AskReddit\n",
       "1    1085  GlobalOffensiveTrade\n",
       "6     740                  news\n",
       "4     691  RocketLeagueExchange\n",
       "7     637            newsbotbot\n",
       "17    473            The_Donald\n",
       "5     472              business\n",
       "15    459                videos\n",
       "8     389        Showerthoughts"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_stats = pd.DataFrame([{'Label': k, 'Count': v} for k, v in \n",
    "              Counter((i[0] for i in train)).items()]).sort_values('Count', ascending=False)\n",
    "data_stats[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Count</th>\n",
       "      <th>Label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>349</td>\n",
       "      <td>funny</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>278</td>\n",
       "      <td>me_irl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>272</td>\n",
       "      <td>gameofthrones</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>262</td>\n",
       "      <td>gaming</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>225</td>\n",
       "      <td>aww</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>207</td>\n",
       "      <td>pics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>172</td>\n",
       "      <td>Overwatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>151</td>\n",
       "      <td>MemezForDayz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>140</td>\n",
       "      <td>gonewild</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>49</td>\n",
       "      <td>Ice_Poseidon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>ShittyCarMod</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Count          Label\n",
       "2     349          funny\n",
       "3     278         me_irl\n",
       "9     272  gameofthrones\n",
       "14    262         gaming\n",
       "13    225            aww\n",
       "10    207           pics\n",
       "18    172      Overwatch\n",
       "19    151   MemezForDayz\n",
       "12    140       gonewild\n",
       "20     49   Ice_Poseidon\n",
       "0       1   ShittyCarMod"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_stats[10:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ShittyCarMod</td>\n",
       "      <td>Maserati with an Aerodynamic Spoiler</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GlobalOffensiveTrade</td>\n",
       "      <td>[Store] Double moses M9 #355, Dual side blue g...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>funny</td>\n",
       "      <td>Adam Mišík – Tak pojď</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>me_irl</td>\n",
       "      <td>me 🌞irl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RocketLeagueExchange</td>\n",
       "      <td>[ps4] [h] crimsom mantis [w] offers key prefer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>business</td>\n",
       "      <td>Edible Prints on Cake</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>funny</td>\n",
       "      <td>RT @naukhaizs: #pakistan #karachi #lahore #isl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>news</td>\n",
       "      <td>10 ảnh avatar buồn , thất tình , tâm trạng dàn...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>newsbotbot</td>\n",
       "      <td>@washingtonpost: For Facebook , erasing hate s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Showerthoughts</td>\n",
       "      <td>Russia vs USA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Label                                               Text\n",
       "0          ShittyCarMod               Maserati with an Aerodynamic Spoiler\n",
       "1  GlobalOffensiveTrade  [Store] Double moses M9 #355, Dual side blue g...\n",
       "2                 funny                              Adam Mišík – Tak pojď\n",
       "3                me_irl                                            me 🌞irl\n",
       "4  RocketLeagueExchange     [ps4] [h] crimsom mantis [w] offers key prefer\n",
       "5              business                              Edible Prints on Cake\n",
       "6                 funny  RT @naukhaizs: #pakistan #karachi #lahore #isl...\n",
       "7                  news  10 ảnh avatar buồn , thất tình , tâm trạng dàn...\n",
       "8            newsbotbot  @washingtonpost: For Facebook , erasing hate s...\n",
       "9        Showerthoughts                                      Russia vs USA"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_df = pd.DataFrame([{'Text': t, 'Label': l} for l, t in train[:10]])\n",
    "data_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Results\n",
    "\n",
    "Comparing models trained on `train`, hyperparameters tuned on `development` and evaluated on `test`:\n",
    "* `baseline` model with L2 regularisation, C10\n",
    "* `fastText` model with epoch 90, dimensions=50\n",
    "* `fastText-quantized` model as above but quantized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>time</th>\n",
       "      <th>size</th>\n",
       "      <th>p</th>\n",
       "      <th>r</th>\n",
       "      <th>f</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>baseline</td>\n",
       "      <td>21.397510</td>\n",
       "      <td>176.164019</td>\n",
       "      <td>0.692</td>\n",
       "      <td>0.692</td>\n",
       "      <td>0.692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>fastText</td>\n",
       "      <td>6.789109</td>\n",
       "      <td>64.886967</td>\n",
       "      <td>0.689</td>\n",
       "      <td>0.689</td>\n",
       "      <td>0.689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fastText-quantized</td>\n",
       "      <td>54.206348</td>\n",
       "      <td>3.623718</td>\n",
       "      <td>0.702</td>\n",
       "      <td>0.702</td>\n",
       "      <td>0.702</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 name       time        size      p      r      f\n",
       "0            baseline  21.397510  176.164019  0.692  0.692  0.692\n",
       "1            fastText   6.789109   64.886967  0.689  0.689  0.689\n",
       "2  fastText-quantized  54.206348    3.623718  0.702  0.702  0.702"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def plot(df):\n",
    "    fig = plt.figure(figsize=(20, 8))\n",
    "    ax = fig.add_subplot(111)\n",
    "    ax2 = ax.twinx()\n",
    "    df.plot(kind='bar', x='name', y='size', ax=ax, color='blue', alpha=0.5, position=0, width=0.3)\n",
    "    df.plot(kind='bar', x='name', y='time', ax=ax2, color='red', alpha=0.5, position=1, width=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Size vs Time\n",
    "\n",
    "* Left-to-right: `baseline`, `fastText`, `fastText-quantized`\n",
    "* `fastText` smaller and quicker to train than `baseline`\n",
    "* `fastText-quantized` smaller again, but much slower to train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABK0AAAJeCAYAAAB27/ZfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X20nmddJ/rvL9lp89Z3QFqLtZ5O6wicVAojawa0CKKD\nBwQUT1FBOuvwOtSBKR4GDx6LL2Wc+rK0DohdM5YqWh2BWXCGoQgyQ1HBI4UgHJpKa2l96YiBNk3S\nl+zkd/7YT5gnu0n3TvIk+0r6+ax1r/3c13Xf1/17Vla722+u67qruwMAAAAAI1m10gUAAAAAwGJC\nKwAAAACGI7QCAAAAYDhCKwAAAACGI7QCAAAAYDhCKwAAAACGI7QCAAAAYDhCKwAAAACGI7QCAAAA\nYDhzK13AoVi1alWvW7dupcsAAAAAOG7s3Lmzu3uYCU7HZGi1bt267NixY6XLAAAAADhuVNV9K13D\ntGHSMwAAAADYS2gFAAAAwHCEVgAAAAAMR2gFAAAAwHCOyY3YAQAAAJKku7Nr167s2bPna2179uzJ\n/Px8unsFKxvL3Nxc1q1bl1Wrjp35S0IrAAAA4JjU3bn77ruzZs2azM0tRBy7d+9Od2f16tWpqhWu\ncBy7du3K3/3d3+XMM888ZoIroRUAAABwTNq1a1fWrFmTjRs3JlkIsfbs2ZPVq1evcGVj6u7cd999\n2bBhw0qXsizHRrQGAAAAsMiePXu+NsMqWQhlzK46sBNOOCHz8/MrXcayCa0AAAAAGI7QCgAAAIDh\nCK0AAAAAVsCVV16Z5z73uQ9pv+mmm7J9+/YVqGgsdSy+/nHDhg29Y8eOlS4DAAAAWEH3339/kmTt\n2rVJFva4SpJVP/3T+1x3771H5vn3Xn7FPudnnfXw11988cV51rOelTe/+c1HpqAl7Ny5M7t27cop\np5yy3/6q2tndw+zSbqYVAAAAAMMRWgEAAAAcYa997Wtz44035md+5meycePGXHDBBXnDG96Qpz71\nqbnpppuyefPmPO5xj8vP/uzP5slPfnI2btyYJz7xiXnve9+bK6+8Mo973OOycePGPP/5z8/NN9/8\ntXE//elP59nPfnYe9ahH5dGPfnRe9rKX5d4jNbXsKBNaAQAAABxhv/Zrv5anP/3p+cmf/Mls3749\nmzdvzr333pv169fnSU96Uh7/+Mdn1apVeec735k3vvGNueOOO7Jp06a84Q1vyO23356bb745N910\nUz72sY/lYx/7WJLkzjvvzPd93/floosuyh133JE///M/zxe/+MW89rWvXeFvOxtCKwAAAICjrKrS\n3dm9e3d2796dubm5VFVe8YpX5Nxzz82aNWvyQz/0Q7ntttvycz/3c1mzZk3uv//+PO1pT8sXvvCF\nJMkf/MEfZPXq1XnrW9+a9evX55xzzsnll1+e66+/Prt3717hb3j45la6AAAAAIBHmhNPPDGnnXZa\ndu3alc2bN2f9+vXZs2dPzjzzzK9ds379+qxevTqnn356br755px22mk5/fTTv7b870tf+lLuvPPO\nnHTSSQ8Z/6677srXf/3XH7XvcyQIrQAAAACOglWr9l3wtm7duqxfvz4XXnhhvvzlL2d+fv5rb0Cc\nduutt2bt2rU5++yz92k/++yzc9555+2zx9XxZFnLA6vqkqq6saq2VdX8or6fqKrti46uql+duub2\nqrp/0TVPnPWXAQAAABjVYx/72Hzxi19Mktx///25//77092pqqxevfqA9+3Zsyfnnntuqmqf9he9\n6EW577778pa3vCXbtm3L7t27s2XLllx//fVH9HscLcudafXVJG9Lsi7Jb0x3dPeVSa7ce15V5ye5\nOclvLxrj/+juxW3AwK64YqUr4Ejy5wsAHDP8h8vx6xH2Z/v6178+l156aU499dScddZZecYznpHt\n27fnM5/5TE488cTMzc09ZDZWkuzYsSOf+cxnkiRbt27N+vXrkyTnnHNO3ve+9+XNb35zzjvvvOzc\nuTOPecxjcskll+SSSy45qt/tSFhWaNXdNyRJVV28jMtfkeTT3f1nh1EXAAAAwKFZFIY9dMen2TjY\ncZ/ylKfkc5/73AH777jjjn3OL7744szP77PgLe9///v3Od+0adND2o4XM317YFWdmORlSd6xn+5f\nqqqvVNVnquqVhzD2GVV1flWd392HWyoAAAAAA5v1Ruw/kOSEJL+zqP1Hk3wqyQNJLk5y/eTVjvsL\ntw7ksiQ/leQhKSMAAAAAx5eZzrRK8sok7+ru7dON3f3fu3t7d+/q7j9M8ktJfuQgx746yQVJLpib\n89JDAAAAgOPZzNKfqvqWJE/PwoyopexJUkteNaW7tybZmiQbNmw46PoAAAAAOHYsa6ZVVa2uqrVZ\nWPqXqlo7OaaDp1cm+UR3b1507zlV9YzJ9aur6juSvD7J783oOwAAAABwnFnuTKuXJPnNqfP7Jj/P\nTXJ7Va1L8tIkr9vPvRuysBzwvCSd5I4kP93dv3ZIFQMAAAAkWbVqVR588MGvnVdV9uzZs4IVje3B\nBx/MmjVrVrqMZVtWaNXd1ya59mH670ty2gH6/r8k33oItQEAAAAc0Jo1a7Jjx45s3749q1evTlVl\n9+7d6e5UVfZdIPbI9uCDD2bHjh0588wzV7qUZbOjOQAAAHBMqqqceuqp2bVr19dmWK1evTp79uzJ\n/Px8unuFKxzHmjVrcuaZZ2bVqlm/k+/IEVoBAAAAx6yqygknnLDSZXAEHDvxGgAAAACPGEIrAAAA\nAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIr\nAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABg\nOEIrAAAAAIYjtAIAAADgkFTVtVW1q6q2Tx2vWXTNS6vq1qraWVWfrKqLljO20AoAAACAw/HO7t44\ndbxtb0dVPS3J25O8OslpSd6d5ANVdfJSgwqtAAAAAEiSqqrzp44zZjDmy5O8p7s/1N0PJLkqyQNJ\nXrDUjUIrAAAAAJJkLsmWqeOyZd73/VX1laq6paquqqqNU32bknxq70l3d5JPT9ofltAKAAAAgCSZ\nT3LB1HH1Mu65Osk3J3lUFmZPfUeSa6b6T0pyz6J77k6y5PLAuWU8HAAAAIDjX3f3LQd5w6emTj9f\nVa9P8t+q6mWT5YD3Jjll0W2nJrl1qbHNtAIAAABgVvZMftbk5+YkT9rbWVWV5MJJ+8MSWgEAAABw\nSKrqkqo6dfL5HyX5xSTv6+77J5dck+SFVfXMqjohyeVJ1iZ571JjC60AAAAAOFSvSnJbVe1I8qEk\nn0hy6d7O7v54ktdkIby6J8kPJnlOd29bamB7WgEAAABwSLr74mVcc12S6w52bDOtAAAAABiO0AoA\nAACA4QitAAAAABiO0AoAAACA4QitAAAAABiO0AoAAACA4QitAAAAABiO0AoAAACA4QitAAAAABiO\n0AoAAACA4QitAAAAABiO0AoAAACA4QitAAAAABiO0AoAAACA4QitAAAAABiO0AoAAACA4QitAAAA\nABiO0AoAAACA4QitAAAAABjOskKrqrqkqm6sqm1VNb+o7+Kq6qraPnX8yaJrzquqD1fVjqr666q6\nfJZfAgAAAIDjy9wyr/tqkrclWZfkN/bTv7u7N+7vxqpaneT9ST6c5HlJvjnJB6vqr7v79w6+ZAAA\nAACOd8uaadXdN3T37ya57RCe8e1Jzknypu7e2d03JXlHklcdwlgAAAAAPALMak+r1VV1Z1XdVVX/\npao2TfVtSnJLd2+fartp0r5sVXVGVZ1fVed39yxqBgAAAGBQswitbk5yYZJzs7D077NJ/qiqzpr0\nn5TknkX33J3k5IN8zmVJtiTZMj8/v9S1AAAAABzDDju06u67untzd893993d/aYkX0nyzyeX3Jvk\nlEW3nZpk20E+6uokFyS5YG5uuVtxAQAAAHAsOlLpz54kNfm8Ocn5VbWhu3dM2r510r5s3b01ydYk\n2bBhw6zqBAAAAGBAy5ppVVWrq2ptkhMm52snR1XVd1bVeVW1qqo2VtUVSb4uyQ2T2z+W5EtJrqyq\ndVV1YZJXZmEzdgAAAAB4iOUuD3xJkvuyEEStnny+LwtvBdyU5CNZWAZ4W5KnJvmu7r4zSbp7d5Ln\nJnlCFmZKfSDJVd19/ey+BgAAAADHk2UtD+zua5Nce4DuX54cD3f/F5M882AKAwAAAOCRaxZvDwQA\nAACAmRJaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAADAc\noRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAA\nADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFa\nAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAA\nwxFaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAADAcoRUAAAAAwxFaAQAAAHDYqmpV\nVf1JVXVVnT3V/tKqurWqdlbVJ6vqouWMJ7QCAAAAYBZen2TndENVPS3J25O8OslpSd6d5ANVdfJS\ngwmtAAAAADgsVXV+ktckecOirpcneU93f6i7H0hyVZIHkrxgqTHnZl4lAAAAAMeimoRPe23t7q3L\nuGlVkv+YhcDq7kXdm5Jcu/eku7uqPj1pf1hmWgEAAACQLExu2jJ1XLbM+/5Vkru6+7376TspyT2L\n2u5OsuTyQDOtAAAAAEiS+SRPmDpfziyr85JcnuTJB7jk3iSnLGo7NcmtS40ttAIAAAAgWVi9d8tB\n3vO0JI9O8rmqSv7nqr7PVtWbk2xO8qS9F9fCRRcmec9SAwutAAAAADhUv5/kw1PnZyf50yTPTnJz\nks8m+WBVvTPJjUl+LMnaJPtbSrgPoRUAAAAAh6S7dybZufe8qvZmTXd19/YkH6+q1yS5JsmZSf4i\nyXO6e9tSYwutAAAAAJiJ7r49SS1quy7JdQc7lrcHAgAAADAcoRUAAAAAw1lWaFVVl1TVjVW1rarm\nF/U9p6r+qKr+oaq+Ornu6Yuu6araWVXbp47FrzsEAAAAgCTLn2n11SRvS/K6/fSdluTqJOdl4RWH\nv5Pkv1bV4xZd9+zu3jh13HOoRQMAAABwfFvWRuzdfUOSVNXF++l716Kmt1fVTyV5SpI7D7fAvarq\njCRnJMm6detmNSwAAAAAA5r5nlZV9cQkj8rCKwyn/afJEsJPVtULD2Hoy5JsSbJlfn5+qWsBAAAA\nOIbNNLSqqsckeXeSX+juv5zqelaSc5OcneSXkryrqr7nIIe/OskFSS6Ym1vWBDEAAAAAjlEzS3+q\n6qwkf5jkQ0neNN3X3R+ZOv29qnpWkh9O8sHljt/dW5NsTZINGzYcdr0AAAAAjGsmM62q6huT3Jjk\nv3b3a7u7l7hlT5KaxbMBAAAAOP4sa6ZVVa1OsibJCZPztZOuB7KwZO/DSa7t7jfv594nJFmf5DNJ\nOsn3JnlJkksOt3gAAAAAjk/LnWn1kiT3JbkhyerJ5/uSnJPkjUm+Psnrqmr71PHDk3sfneQ3k3w1\nyd8neXOSf9Hd75vd1wAAAADgeLKsmVbdfW2Saw/QfenkONC9H03y+IMtDAAAAIBHrpm+PRAAAAAA\nZkFoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYA\nAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBw\nhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAA\nAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdo\nBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAA\nDEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYA\nAAAADEdoBQAAAMBwlhVaVdUlVXVjVW2rqvn99H9PVX2+qu6rqs9V1bMX9Z9XVR+uqh1V9ddVdfms\nvgAAAAAAx5/lzrT6apK3JXnd4o6q+qYk70ny1iSnTH6+t6q+cdK/Osn7k3whyaOTPC/JG6vqfz/M\n2gEAAAA4Ti0rtOruG7r7d5Pctp/uH03yqe7+7e5+sLvfleSmSXuSfHuSc5K8qbt3dvdNSd6R5FWH\nXz4AAAAAx6NZ7Gm1KcmnFrXdNGnf239Ld28/QP+yVNUZVXV+VZ3f3YdcLAAAAADjm0VodVKSexa1\n3Z3k5GX2L9dlSbYk2TI//5BttQAAAAA4jswitLo3C3tZTTs1ybZl9i/X1UkuSHLB3NzcwdYIAAAA\nwDFkFqHV5iRPWtT2rZP2vf3nV9WGA/QvS3dv7e5buvuWqjrkYgEAAACYnar6uar6q6raVlV/X1V/\nUFXfMNX/0qq6tap2VtUnq+qi5Yy7rNCqqlZX1dokJ0zO106OSnJdkidX1Yurak1VvTjJRUneObn9\nY0m+lOTKqlpXVRcmeWUWNmMHAAAA4Nj2W0ku7O6Tk3xjkjuSXJ8kVfW0JG9P8uokpyV5d5IPVNWS\n20Ytd6bVS5Lcl+SGJKsnn+9Lck5335rkhUnenIUlf29O8oLuvj1Junt3kucmeUKSrUk+kOSq7r5+\nmc8GAAAAYFDdfXN3793PvJLsycIWT0ny8iTv6e4PdfcDSa5K8kCSFyw17rI2h+rua5Nc+zD9H0zy\nwYfp/2KSZy7nWQAAAACsiKqq86fOt3b31mXe+ENZmFF1cpL5JP960rUpU5lSd3dVfXrS/rBmsacV\nAAAAAMe+uSRbpo7Llntjd/9Od5+S5MwkVyT5i0nXSUnuWXT53VkIt5YsBgAAAADms7C9017LmmU1\nrbvvqqprktw22Yz93iSnLLrs1CS3LjWW0AoAAACAZGH13i0zGGcuyYYkZyXZnORJezsmL/W7MMl7\nlhrE8kAAAAAADklVraqq11bVYybnZyf590luT3JzkmuSvLCqnllVJyS5PMnaJO9damyhFQAAAACH\n4zlJPldVO5J8MsnOJM/q7vnu/niS12QhvLonyQ8meU53b1tqUMsDAQAAADgk3b0nC6HVw11zXZLr\nDnZsM60AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7Q\nCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAA\nGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0A\nAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDh\nCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAA\nAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGM7cLAapqs8nOWeqaXWStUku\nSnJyko8m2THV/9nu/qezeDYAAAAAx5+ZhFbd/fjp86r6uSTP7+6bquriJLu7e+MsngUAAADA8W/m\nywOrai7Jv0jyjlmPDQAAAMAjw5HY0+r5SU5Jct1U2+qqurOq7qqq/1JVmw520Ko6o6rOr6rzu3tm\nxQIAAAAwniMRWr0yye91992T85uTXJjk3CTfnOSzSf6oqs46yHEvS7IlyZb5+flZ1QoAAADAgGYa\nWlXV/5LkmUl+fW9bd9/V3Zu7e7677+7uNyX5SpJ/fpDDX53kgiQXzM3NZCsuAAAAAAY16/TnlUk2\nd/cnl7huT5I6mIG7e2uSrUmyYcOGQ6sOAAAAgGPCzGZaVdUJSV6WqVlWk/bvrKrzqmpVVW2sqiuS\nfF2SG2b1bAAAAACOL7NcHvjCJGuTvGtR+6YkH0lyb5Lbkjw1yXd1950zfDYAAAAAx5GZLQ/s7uuT\nXL+f9l9O8suzeg4AAAAAx78j8fZAAAAAADgsQisAAAAAhiO0AgAAAGA4QisAAAAAhiO0AgAAAGA4\nQisAAAAAhiO0AgAAAGA4cytdAMe4K65Y6Qo4oq5Y6QIAAAB4hDLTCgAAAIDhCK0AAAAAGI7QCgAA\nAIDhCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIBDUlU/X1Wfr6ptVfW3VXVNVZ2+6JqXVtWt\nVbWzqj5ZVRctZ2yhFQAAAACHaneSH0lyRpJNSc5Ocu3ezqp6WpK3J3l1ktOSvDvJB6rq5KUGFloB\nAAAAkCRVVedPHWcsdUN3/0R3f7q7d3X3l5P8SpKLpy55eZL3dPeHuvuBJFcleSDJC5YaW2gFAAAA\nQJLMJdkydVx2CGM8M8nmqfNNST6196S7O8mnJ+1LFgMAADNxxRUrXQFHij9bgEeE+SRPmDrfejA3\nV9X3J3lVku+Yaj4pyT2LLr07yZLLA4VWAAAAACQLE6FuOZQbq+pFSd6R5HndfdNU171JTll0+alJ\nbl1qTMsDAQAAADhkVXVpFgKr53b3Rxd1b07ypKlrK8mF2XcJ4X4JrQAAAAA4JFX1Y0l+Icl3d/cf\n7+eSa5K8sKqeWVUnJLk8ydok711qbMsDAQAAADhUv5KFvbA+ujCJakF3b5z8/HhVvSYL4dWZSf4i\nyXO6e9tSAwutAAAAADgk3V3LuOa6JNcd7NiWBwIAAAAwHKEVAAAAAMMRWgEAAAAwHKEVAAAAAMMR\nWgEAAAAwHKEVAAAAAMMRWgEAAAAwHKEVAAAAAMMRWgEAAAAwHKEVAAAAAMMRWgEAAAAwHKEVAAAA\nAMMRWgEAAAAwHKEVAAAAAMMRWgEAAAAwHKEVAAAAAMMRWgEAAAAwHKEVAAAAAMMRWgEAAAAwHKEV\nAAAAAMMRWgEAAAAwHKEVAAAAAMMRWgEAAAAwHKEVAAAAAMMRWgEAAAAwnJmEVlV1bVXtqqrtU8dr\nFl3z0qq6tap2VtUnq+qiWTwbAAAAgOPPLGdavbO7N04db9vbUVVPS/L2JK9OclqSdyf5QFWdPMPn\nAwAAAHCcOFrLA1+e5D3d/aHufiDJVUkeSPKCo/R8AAAAAI4hswytvr+qvlJVt1TVVVW1capvU5JP\n7T3p7k7y6Un7slTVGVV1flWdv3A7AAAAAMerWYVWVyf55iSPysLsqe9Ics1U/0lJ7ll0z91JDmZ5\n4GVJtiTZMj8/f+iVAgAAADC8mYRW3f2p7v4f3b2nuz+f5PVJfqCqTpxccm+SUxbddmqSbQfxmKuT\nXJDkgrm5ucOuGQAAAIBxHak9rfZMftbk5+YkT9rbWVWV5MJJ+7J099buvqW7b1m4HQAAAIDj1UxC\nq6q6pKpOnXz+R0l+Mcn7uvv+ySXXJHlhVT2zqk5IcnmStUneO4vnAwAAAHB8mdVMq1clua2qdiT5\nUJJPJLl0b2d3fzzJa7IQXt2T5AeTPKe7D2Z5IAAAAACPEDPZHKq7L17GNdcluW4WzwMAAADg+Hak\n9rQCAAAAgEMmtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIA\nAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYj\ntAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAA\nAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIrAAAAAIYjtAIAAABgOEIr\nAAAAAIYjtAIAAADgkFXVJVV1Y1Vtq6r5/fR/T1V9vqruq6rPVdWzlzOu0AoAAACAw/HVJG9L8rrF\nHVX1TUnek+StSU6Z/HxvVX3jUoMKrQAAAAA4ZN19Q3f/bpLb9tP9o0k+1d2/3d0Pdve7ktw0aX9Y\nczOuEwAAAIBjU1XV+VPnW7t762GOuSnJpxa13TRpf1hmWgEAAACQLExu2jJ1XDaDMU9Kcs+itruT\nnLycYgAAAABgPskTps4Pd5ZVktybhb2spp2aZNtSNwqtAAAAAEiS7u5bZjzm5iTPWNT2rUk+stSN\nlgcCAAAAcMiqanVVrU1ywuR87eSoJNcleXJVvbiq1lTVi5NclOSdS41rphUAAAAAh+MlSX5z6vy+\nyc9zu/vWqnphkl9M8h+z8IbBF3T37UsNKrQCAAAA4JB197VJrn2Y/g8m+eDBjmt5IAAAAADDEVoB\nAAAAMByhFQAAAADDEVoBAAAAMByhFQAAAADDEVoBAAAAMByhFQAAAADDEVoBAAAAMJyZhFZV9fNV\n9fmq2lZVf1tV11TV6VP9L6uqPVW1fer43Vk8GwAAAIDjz6xmWu1O8iNJzkiyKcnZSa5ddM1t3b1x\n6njxjJ4NAAAAwHFmbhaDdPdPTJ1+uap+Jcnvz2LsvarqjCyEYlm3bt0shwYAAABgMEdqT6tnJtm8\nqO1xVXVXVd1ZVddX1bkHOeZlSbYk2TI/Pz+TIgEAAAAY08xDq6r6/iSvSvKvppo/luSJSc5K8pQk\n9yf5w6racBBDX53kgiQXzM3NZIIYAAAAAIOaafpTVS9K8o4kz+vum/a2d/dtU5fdVVUvT3JPkqcm\n+chyxu7urUm2JsmGDQeTdQEAAABwrJnZTKuqujQLgdVzu/ujS1zek6Nm9XwAAAAAjh8zCa2q6seS\n/EKS7+7qr+moAAAYs0lEQVTuP95P//dW1dm14PQk/z7JPyT5xCyeDwAAAMDxZVYzrX4lyclJPlpV\n2/ceU/0XJ/mzJNuTfD4LbwH8ru7e/pCRAAAAAHjEm8meVt39sMv8uvvHk/z4LJ4FAAAAwPHPa/gA\ngKPriitWugKOqCtWugAA4Dgxs43YAQAAAGBWhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBw\nhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAA\nAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdo\nBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAA\nDEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYA\nAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBwhFYAAAAADEdoBQAAAMBw\n5la6AAAAAOCR6YorVroCRmamFQAAAADDEVoBAAAAMByhFQAAAADDEVoBAAAAMJyjFlpV1eqquqqq\nvlxV91bVu6vqUUfr+QAAAADM3pHKfI7mTKt/k+T7knxbkrMnbb91FJ8PAAAAwOwdkcxn7nAHOAiv\nSPLT3X1bklTV/5nki1V1Tnd/aambq+qMJGckybp1645ooQAAAACPQFVV50+db+3urcu477AynwMW\n092Heu/yH1J1apKvJvnW7v7MVPs9SV7S3e9bxhhXJPmpyWknue8IlAr8T5WFYHs+C//MAcBS/O4A\n4GD4vTGedVn4c9nrLd19xcPdMIvM50CO1kyrkyY/71nUfneSk5c5xtVJfmfyeblJH3CIJun6liRP\n6O5bVroeAMbndwcAB8PvjfFMr3KbWE72MovMZ7+OVmh17+TnKYvaT02ybTkDTEIqQRUAAADAEXCI\n2cthZz4HclQ2Yu/uu5PckeRJe9uq6puykLh99mjUAAAAAMBsHcnM52i+PfA3kryxqs6tqpOT/HyS\nG7r79qNYA7B8W5O8JWY4ArB8fncAcDD83jh+HJHM56hsxJ4kVbU6C0W/LMmJSf4wySu6+x+OSgEA\nAAAAzNyRynyOWmgFAAAAAMt1NJcHAgAAAMCyCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgAAAIDh\nCK0AAAAAGI7QCgAAAIDhCK0AAAAAGI7QCgCAQ1JV33aA9n9ytGsBAI4/1d0rXQMwoKqqJI/t7r9b\n6VoAGFNVbevuk/fT/pXuPn0lagJgPFX1hSRLhg/d/S1HoRyOIXMrXQAwlqramORXkvxwkt1JNlTV\n85Ns6u63rGhxAIymHtJQtT7JnhWoBYBx/cLU529I8uok1yX5qyTnJnlJkl9fgboYnNAKWOwXk3xd\nkn+W5MOTtv83yZVJhFYApKr+Igt/Y762qj67qPvMJDce/aoAGFV3/4e9n6vqj5I8r7s/MdX2n5K8\ndSVqY2yWBwL7qKq/SfIt3X3P9PKOqrq7u09d4fIAGEBVvTILs6x+Ocnrprr2JLkryQ3d/eBK1AbA\n2KpqW5LTunv3VNtckq/sb8k5j2xmWgGLrUpy33TDZMng9pUpB4DRdPc7koU9Srr7v690PQAcU/4y\nyb9M8qtTba9OcuvKlMPIvD0QWOzjSd60qO2yJB9dgVoAGNtZ+2usqsW/RwBgrx9L8tNVtaWqbqiq\nLUl+Jgv/zwH7sDwQ2EdVPS7JH01Oz0myJckJSb6zu/9mxQoDYDhVdVeS9yR5XXc/WFVnJPntJOd4\nAxQAB1JVpyd5QZKvT/I3Sf5zd29d2aoYkdAKeIiqOjHJc5N8Y5IvJfl/uvu+h70JgEecqjoryfVJ\n1ie5Kgtvh/pvSV7V3TtWsDQA4DggtAIA4JBNNs/9sySbkvx6d//LFS4JgIFV1aok/zrJjyb5uu5+\nTFV9V5JvmH7LICQ2YgcWqapKckmSJyc5abqvu1+xIkUBMKSqOjXJdUk2JPm/krypqr7U3f9uZSsD\nYGA/leR5Sd6a5G2TttuS/HwSoRX7MNMK2EdV/XqSFyX5SJJ9lnZ096UrUhQAQ6qq25P8aZKXd/f2\nqnpikt9Pcmt3/28rWhwAQ6qqv0ry9O7+66r6SnefPpl99Q/dffpK18dYzLQCFntRkn/S3V45C8BS\n/l137/1b8nT3X1TVU5L8xgrWBMDYNib520Vtc0keXIFaGNyqlS4AGM7OJHesdBEAjG86sJpq297d\nP7QS9QBwTPhMkh9e1PaiJH++ArUwOMsDgX1U1WVJHpPk/27/ggBgCVX1A1nYTPex3f2UqvpnSU7v\n7vevcGkADKiqLkzy0SR/nORZSd6f5NuTPLO7P7eStTEeoRWwj6r6yyTnZGHG1d9P93X3+StSFABD\nqqrXJvnxJO9I8sbuPqWqHp/kmu7+pytbHQCjqqqzklya5NwkX0rym9391ytbFSMSWgH7qKofPVBf\nd7/zaNYCwNiq6pYk39fdX6iqr3b3aVW1Osn/6O5HrXR9AIynqk7o7ofsX1VVa7p710rUxLiEVgAA\nHJK9b32a/lxVc0n+rrsfvcLlATCgqtqd5HeTXDodUlXVtu4+eeUqY0TeHgikqr6tuz85+XzA5Rzd\n/SdHryoAjgFfqKrv7u4bptqelcSeJAAcyANJHpvkhqp6fndvm7TXCtbEoMy0AlJV93b3SZPPew5w\nWXf36qNYFgCDq6pnJPnPSX4rC3uTvC0Lm7K/sLs/vpK1ATCmqtqW5PQkv5nkwiTf091/Y6YV+7Nq\npQsAVt7ewGryedUBDoEVAPvo7o8m+Y4kJyb50yz8T8h3C6wAeDjdPd/dL8nCmwM/UVX/60rXxJjM\ntAIA4KBU1b/p7n+70nUAcOyZXuUxOX9lkrcm2djdJ6xcZYxIaAWkqn5iOdd195VHuhYAxmcJBwCH\nqqre1N1vXdT2vUl+pLtfvEJlMSihFZCq+ugyLuvu/s4jXgwAw1v8t+QAAEeC0AoAgINSVduT/OM8\nzJueuvuOo1cRACOrqqu6+8cnn3/1QNd1948dvao4FsytdAHAmKrqrCTf0N2fWOlaABjO+iS3H6Cv\nknQSL/AAYK9HTX1+9P/f3r3HWlaWdxz//maAgREGZsbxMsIwIJe20RipYNRW2oIlMRatIEirrbbx\nblubGiLW1oCRgAQ0iMZabK0IKBQj0KKIEai3KmpQNNRLA6OigDB3B0SZp3/sNfWcwxkuylnve5jv\nJ9lZa71r7TW//c+Z7Ge/77OapdC840wrSdMkeQxwAfAHwJaq2j3J8cDhVfXatukkST0YZlodfH/X\nVNUtI8WRJEmPUAtaB5DUnbOBm5j8AvLzYewzwHOaJZIk9WZrVd1yf6/WASVJfUpy6XbGPzZ2FvXP\nmVaSpklyG7BvVd2dZG1VLRvGN1TVno3jSZI6YCN2SdKvantPoJ363UPaxp5Wkmb6GTP+NiRZBqxt\nE0eS1KFTAZKsrqqbZ55Msm9VrRk9lSSpW0leOOwuTPLHTH+Yx4HAuvFTqXfOtJI0TZJzgXuB1wO3\nVdWyJO9m8vfi9W3TSZJ64q/lkqQHK8mPh93HALdPObUVuBU4papmXTqoHZdFK0nTDLOqPg4cAuwK\nbAa+Djy/qta3zCZJ6stsywST7AzcWlXLG8WSJHUsyZVVdVTrHJofLFpJmlWS3wb2A9YAXyn/WEiS\nBkmuAgr4PeCaGadXAT+oKh/gIUmSfi32tJI0q6r6KvDVJPsB+wDfbxxJktSPzw3bw4HPTxnftsTj\n4tETSZLmhSS7Aq8BngZMm61bVUc3CaVuWbSSNE2SfwXOrarPJzkB+DBQSf6sqi5oHE+S1IGqOhkg\nyY1VdVHrPJKkeeUDwDOBy/GHcT0AlwdKmmZokLh/Vd2V5Drg7cAG4N1V9aS26SRJPUlyILCuqu5I\nshg4kcnDPM6oqrvbppMk9SjJOuA3q+rW1lnUP2daSZpp8VCwWgo8Ebi0qirJPq2DSZK6cyHwMuAO\n4DQmPa7uAZ4AvLpZKklSz9YNL+kBLWgdQFJ3bklyOHA88NmhYLUE+EXjXJKk/uwPfGvYPwY4Gjhq\n2EqSNJt/AM5K8qjWQdQ/Z1pJmukU4Comv5Q/dxg7Eri+WSJJUq8CLExyALClqm4GSLLH/b5LkrQj\nex+wG/DqJD+deqKqlrSJpF7Z00rSfSTZDaCq7hqOHwMscN25JGmqJJ8E1gCPA26uqr9Jshq4tqr2\nbZlNktSnJEdt71xVXTlmFvXPopUkSZJ+JUn2BU5lMjv3jVV1Z5LjgKdW1Ult00mSpPnOopWkaZKs\nAN4FHAGsmHquqhY2CSVJkiTpESPJ/sCzmXzfyLbxqnpHs1Dqkj2tJM10NvB44C+ZPBXqBOBNwEUt\nQ0mS+pRkOXAo9/3i8aFmoSRJ3UpyDHA+8F3gYODbwG8A/w1YtNI0zrSSNE2S24AnV9XtSdZX1V5J\nVgH/XlWHtc4nSepHkiOBS5gsD9wLWD9sb6qqg1pmkyT1Kck3gDOq6rwk66pqaZLXAE+oqre0zqe+\nWLSSNE2StcDyqqokPwYOqKqfJtno0zwkSVMl+QpwYVWdOeWLxz8Cm6vqrNb5JEn9SbIR2HP4vrHt\n/46dgTVVtbJ1PvVlQesAkrrzHeCQYf/rwJuTnAjc1i6SJKlTBzLpgwi/XBp4OvCGNnEkSfPARmCP\nYf/2JAcBewK7t4ukXtnTStJMbwZ2nbL/EWAJ8IpmiSRJvdoCLBq2dw7LydcBS5umkiT17BrgWOBf\nmCwx/xTwc+CqhpnUKZcHSrqPJLsDzwP2Bn4I/GdVbWqbSpLUmySXAJdU1QVJzgF+B/gZsKGq/rBt\nOklS75IsAF7OZObVuVW1uXEkdcailaRpkjwNuAK4C/g+sArYDXhuVX2lZTZJUntJrquqQ4f904GT\nq2pLkl2Bv2PyxeOsqrq9ZU5JkjT/WbSSNE2SLzP51fz0KWMnAi/a9iVFkrTjSrIeWDo00PUhHZKk\nhyTJ2ds7V1V/PWYW9c+ilaRpkmxi8mXkF1PGdgLWVdUe23+nJGlHkORKYDFwI/Ay4IOzXVdVrxwv\nlSRpvkhy4YyhlcDTgcuq6rgGkdQxG7FLmul64EnDdpsnzziWJO24Xgy8Gth/ON65YRZJ0jxTVSfM\nHEtyPPCsBnHUOWdaSSLJn0w5fCLwKuBcYA2wGvgL4P1V9bbx00mSepXksqo6unUOSdL8NjRk/0lV\nLW+dRX2xaCWJJDc9iMuqqvZ/4MskSZIk6cFLcgRwXlWtbJ1FfXF5oCSqar/WGSRJkiQ98iW5AZg6\ne+ZRTJ5YfmKbROqZRStJkiRJkjSWc2YcbwK+VlX/0yKM+ubyQEmSJEmSJHXHmVaSJEmSJGkUSY57\nMNdV1UVznUX9c6aVJEmSJEkaRZIfAI8FFgJrgaXAVuBWIMNlVVWr2iRUTxa0DiBJkiRJknYYZwLn\nAUuragWwHPg34Myq2md4WbAS4EwrSZIkSZI0kiQ/BA6oqrunjC0GvlNVe7dLph4500qSJEmSJI1l\nF2DFjLFHA4saZFHnbMQuSZIkSZLG8lHgiiRvB9YAq4GThnFpGpcHSpIkSZKkUSTZBXgr8FJgJfBj\nJj2uTq6qn7XMpv5YtJIkSZIkSVJ37GklSZIkSZJGl+Ss1hnUN2daSZIkSZKk0SXZWFVLWudQv5xp\nJUmSJEmSWkjrAOqbRStJkiRJktTCl1sHUN8sWkmSJEmSpFEkefq2/ao6Ysr4YW0SqWf2tJIkSZIk\nSaPYXh+rJGuralmLTOqXM60kSZIkSdJY7tPHKsliYGuDLOrcTq0DSJIkSZKkR7YkNwAF7JrkGzNO\nPx747Pip1DuLVpIkSZIkaa6dw2SW1TuB90wZ3wrcClzZIpT6Zk8rSZIkSZI0iiSHV9W1rXNofrCn\nlSRJkiRJGsvK2QaTnDR2EPXPopUkSZIkSRrLO5O8N8kuAEmWJ/kE8NLGudQhi1aSJEmSJGkshwBP\nAr6Q5HjgeuAO4NCmqdQle1pJkiRJkqTRJNkJ+DLwFOB9VfW6xpHUKWdaSZIkSZKkUSTZC/gY8Cjg\n74GXJDmxbSr1yplWkiRJkiRpFEluBr4IvKKqNid5MnAR8L9V9bym4dQdZ1pJkiRJkqSxvKOqTqiq\nzQBVdQOTflYb28ZSj5xpJUmSJEmSpO4400qSJEmSJI0mybFJLk9y3XD8rCR/1DqX+mPRSpIkSZIk\njSLJ64EzmfS1OmgYXg+c1CyUuuXyQEmSJEmSNIok3wGeX1U3JllXVUuTLARuq6pHt86nvjjTSpIk\nSZIkjeXRVXXjsL9tFk2m7Ev/z6KVJEmSJEkay41JjpoxdiTwzRZh1LedWgeQJEmSJEk7jLcAH09y\nHrAoyRnAnwMvbBtLPXKmlSRJkiRJGkVVXQ0cDixi0ox9GXBUVX2uaTB1yUbskiRJkiRpTiV5U1Wd\n1jqH5heLVpIkSZIkaU4l2VhVS1rn0Pzi8kBJkiRJkjTX0jqA5h8bsUuSJEmSpLmWJPtwP8Wrqvr+\niHk0D7g8UJIkSZIkzakkW4HtFSACVFUtHDGS5gFnWkmSJEmSpLm2BTi4dQjNLxatJEmSJEnSXNta\nVbe0DqH5xUbskiRJkiRprtmIXQ+ZRStJkiRJkjTXTgVIsnq2k0n2HTOM5gcbsUuSJEmSpFEk2VhV\nS2YZX1tVy1pkUr+caSVJkiRJksZyn2WCSXZm+08W1A7MRuySJEmSJGlOJbmKSWFqUZJPzTi9Cvja\n+KnUO4tWkiRJkiRprn1u2B4OfH7K+FbgVuDi0ROpe/a0kiRJkiRJo0hyXFVd1DqH5geLVpIkSZIk\naRRJDgTWVdUdSRYDJwL3AmdU1d1t06k3NmKXJEmSJEljuRB43LB/GvBC4PnAu5olUrecaSVJkiRJ\nkkaRZC2wvKoqyS3As4BNwA1VtbJtOvXGRuySJEmSJGksARYmOQDYUlU3AyTZo2kqdcmilSRJkiRJ\nGsuXgPcwWSJ4BUCS1cDadpHUK3taSZIkSZKksbwK2J1JkeqUYeww4IJmidQte1pJkiRJkiSpOy4P\nlCRJkiRJo0myHDgUWMGkxxUAVfWhZqHUJWdaSZIkSZKkUSQ5ErgEuAfYC1g/bG+qqoNaZlN/7Gkl\nSZIkSZLGchpwSlWtADYP27cB72sbSz1yppUkSZIkSRpFkg3Asqq6N8n6qtorySLgu1W1qnU+9cWZ\nVpIkSZIkaSxbgEXD/p1JVgG7AEvbRVKvLFpJkiRJkqSxfAF4wbD/CeAy4NPAF5slUrcsWkmSJEmS\npDmT5Loph98DPj7svxG4GLgaeMnYudQ/e1pJkiRJkqQ5k2Q9sLSqKsnGqlrSOpPmh51aB5AkSZIk\nSY9oXwL+K8mNwK5J3j/bRVX1ynFjqXcuD5QkSZIkSXPpxcAVQIbjnbfzkqZxeaAkSZIkSRpFksuq\n6ujWOTQ/WLSSJEmSJElSd1weKEmSJEmSpO5YtJIkSZIkSVJ3LFpJkiRJkiSpOxatJEmSJEmS1B2L\nVpIkSZIkSeqORStJkqQZklyT5J+SvDXJbUnuTPKBJIuH888ZrlmbZEOSa5McNuMeleR1SS5O8tMk\na5Icm2TPJOcn2ZTkpiTHzHjf45N8ePg3NyS5OskhY35+SZKkHli0kiRJmt3xwB7As4E/BY4B/nY4\ntzvwXuAZwDOBbwOfTLJ8xj3eAlwOPGXYngd8BPg08FTgUuBDSVYAJNkNuBpYCDwHOBS4DvhMksfN\nyaeUJEnqVKqqdQZJkqSuJLkGWFJVh0wZ+2fggKr6/VmuD3A78IaqOn8YK+DMqnrjcLwUWAucU1V/\nNYztAWwEXlBVlyZ5OfBW4IlVde+U+38L+GBVnTEnH1iSJKlDzrSSJEma3fUzjn8EPBYgyX5Jzkvy\nvSQbgU3AcmDfGe/5+radqloH3AN8Y8rYJmDLtvsCTwP2BjYk2bztBRwMHPCwfTJJkqR5YKfWASRJ\nkjp1z4zj4pc/+P0Hk5lVrwV+OFx7LbDLjPf8fJZ7zDa27b4LgG8Cx86SZ+ODDS5JkvRIYNFKkiTp\nIRj6Vv0Wk6WAVw1jK4GHo+fUV5n0z1pXVXc+DPeTJEmat1weKEmS9NCsA34CvDLJQUmeAXwUuOth\nuPf5TGZuXZ7kiCSrkzwjyclJfvdhuL8kSdK8YdFKkiTpIaiqrcCLgIOY9Kf6IPBuJsWmX/fedzF5\nWuE3gQ8zeSrhR4d/60e/7v0lSZLmE58eKEmSJEmSpO4400qSJEmSJEndsWglSZIkSZKk7li0kiRJ\nkiRJUncsWkmSJEmSJKk7Fq0kSZIkSZLUHYtWkiRJkiRJ6o5FK0mSJEmSJHXHopUkSZIkSZK6Y9FK\nkiRJkiRJ3fk/SvxitNPYkIwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11035d518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Performance \n",
    "\n",
    "* very similar performance\n",
    "* quantisation makes little impact on F1 score\n",
    "\n",
    "$\\rightarrow$ **significant difference, probably not...**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x11035d8d0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABI8AAAJfCAYAAAD2G/YXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XuUpXV95/vPtymuchVEJQiNOhBz4sIL6jrHZIy5TFwa\no5MxF4OOejwSTaLLzBjFXAZ1JiphNKMmGTQmkRDNxWAcL/Ggc5LMMpM4AgbBHAccsWXkRNGG7obg\nhYbv+aN2j2Wlf/Su7up6qqjXa61ae9dvP0/tr39Yi37X73l2dXcAAAAAYG+2TD0AAAAAAOuXeAQA\nAADAkHgEAAAAwJB4BAAAAMCQeAQAAADAkHgEAAAAwJB4BAAAAMDQXPGoqn6iqj5aVbuqavccx59T\nVR+vqjuq6rNV9awDHxUAAACAtTbvzqNbk/xmkpfu68CqOi7Jh5JcluSEJC9McnFV/e/7OyQAAAAA\n06junv/gqu9J8p+7e+Eejnleklcl2dqzH15VlybZ3d3PW8F7nZjkxNm327t7+9yDAgAAALAqhhHo\nAJyd5G/7W6vUJ5I8e4U/58VJLkiSI488Muecc84qjQcAAADAVVdd9ZXuvt++jjsY8eiYJDuXre1I\ncuwKf85bkrwrSc4888zrrrzyylUYDQAAAIAkqarPz3Pcwfi0tduSHLds7fgku1byQ7p7e3df393X\nLywcjMYFAAAAwL4cjHj0ySSPWLb2yNk6AAAAABvIXPGoqg6pqiOSHDb7/ojZV+3l8D9Ncp+q+vmq\nOqyqvi/JjyR526pNDQAAAMCamHfn0bOTfDXJ5UkOmT3/apLTq+q7q+r2qjotSbp7R5InJ/nRLN77\n6LeSvLC7/2a1hwcAAADg4JrrZkLd/Y4k7xi8vC3J0cuOvyLJYw9gLgAAAIA1c/fdd+fOO+/Mt354\n/MZUVTn00EOzZcvq3K3InagBAACATe22227L7t27c/jhh69acJnSXXfdlTvuuCMLCws55phjDvjn\niUcAAADAprV79+50d0444YSpR1lVRx11VHbu3Jndu3fnQD/FfuPnNAAAAID9tHv37hx66KFTj3FQ\nHHbYYdm9e/cB/xzxCAAAANjU9v5h8hvfav3vEo8AAAAAGBKPAAAAABhyw2wAAACAZbae/8E1eZ9t\nr3/KmrzPgbDzCAAAAIAh8QgAAACAIfEIAAAAYIPYunVrXvOa1+S7vuu7cvTRR+ecc87JFVdccVDf\nUzwCAAAA2EAuvvjivOlNb8ott9ySZzzjGXnyk5+cXbt2HbT3E48AAAAANpDnP//5efSjH53DDjss\nr3jFK3LkkUfmAx/4wEF7P/EIAAAAYAPZunXr/3peVTnttNPyhS984aC9n3gEAAAAsIFs27btfz3v\n7tx444059dRTD9r7iUcAAAAAG8jv/M7v5BOf+ETuvPPOXHTRRbnjjjvylKc85aC938JB+8kAAAAA\nG9S21x+8GHOgzjvvvLzkJS/J1VdfnbPOOisf/OAHc9xxxx209xOPAAAAADaQhzzkIbngggvW7P1c\ntgYAAADAkHgEAAAAwJDL1gAAAAA2iKWftLZW7DwCAAAAYEg8AgAAAGBIPAIAAABgSDwCAAAAYEg8\nAgAAAGDIp60BAAAALPeq49bofXauzfscADuPAAAAABgSjwAAAAA2qDvvvPOgv4d4BAAAALBBbN26\nNa95zWvyxCc+MUcffXQuu+yyg/6e7nkEAAAAsIH81m/9Vt73vvflEY94RL72ta8d9PcTjwAAAAA2\nkBe84AV55CMfmSQ58sgjD/r7uWwNAAAAYAPZunXrmr6feAQAAACwgWzZsrY5RzwCAAAAYEg8AgAA\nAGDIDbMBAAAAlnvVzqkn2Ktt27at+XvaeQQAAADAkHgEAAAAwJB4BAAAAMCQeAQAAADAkHgEAAAA\nwJB4BAAAAMCQeAQAAADAkHgEAAAAwJB4BAAAAMDQwtQDAAAAAKw3D7/k4WvyPtc+59oVHX/LLbfk\nmc98Zj72sY/loQ99aK666qqDNNk3iUcAAAAAG8TFF1+c22+/Pdu3b8/CwtpkHZetAQAAAGwQN9xw\nQx72sIetWThKxCMAAACADeGpT31qLrnkklxyySU5+uijc8EFF6zJ+7psDQAAAGADeP/735/nPve5\nWVhYyNvf/vY1e187jwAAAAAYEo8AAAAAGBKPAAAAABgSjwAAAAAYcsNsAAAAgGWufc61U4+wbohH\nAAAAABvEO97xjjV/T5etAQAAADAkHgEAAAAwJB4BAAAAMCQeAQAAAJtad089wkGxWv+7xCMAAABg\n01pYWMidd9459RgHxTe+8Y0sLBz4Z6X5tDUAAABg01pYWEhV5dZbb83hhx+eqkpVTT3WfuvudHe+\n/vWvZ2FhQTwCAAAAOFDHHHNM7r777tx5550b/hK2qsqWLVty3HHHZcuW1bngTDwCAAAANr0tW7bk\n8MMPn3qMdck9jwAAAAAYEo8AAAAAGBKPAAAAABgSjwAAAAAYEo8AAAAAGBKPAAAAABgSjwAAAAAY\nEo8AAAAAGBKPAAAAABgSjwAAAAAYEo8AAAAAGBKPAAAAABgSjwAAAAAYmiseVdUhVXVRVX25qm6r\nqsuq6qR7OP5lVfXZ2bGfqaqfXr2RAQAAAFgr8+48Oj/J05I8Lsmps7VL93ZgVf1wklcnObe7j0ny\nL5NcVFU/cICzAgAAALDG5o1H5yW5sLtv6O6dSV6e5ElVdfpejn1okk9298eSpLv/Jsk1Sc5eyWBV\ndWJVnVlVZ+7evXslpwIAAACwSvYZj6rq+CSnJblqz1p3fzbJruw9CP1hkmOr6vFVtaWqvjvJmUn+\n7xXO9uIk1yW57uabb17hqQAAAACshoU5jjlm9rhz2fqOJMfu5fibk/xJkr/IN+PUS7v7Uyuc7S1J\n3pUkJ5988nUrPBcAAACAVTDPZWu3zR6PW7Z+fBZ3Hy33y0l+Mskjkhyaxd1JP1dVz1/JYN29vbuv\n7+7rFxbmaVwAAAAArLZ9xqPu3pHkxiSP2rNWVQ/O4q6ja/ZyyqOT/Gl3/7+96O+SvDfJU1dnZAAA\nAADWyrw3zH5bkldU1RlVdWySC5Nc3t3b9nLsf03y9Kr6J0lSVQ9L8vQsuWcSAAAAABvDvNeDvT7J\nCUmuSHJ4ko8keVaSVNW5Sd7a3UfPjr0oi5e4faSqTkpyS5J3z34GAAAAABtIdffUM+zTOeec01de\neeXUYwAAAADca1TVVd19zr6Om/eyNQAAAAA2IfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHx\nCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEI\nAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgA\nAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAA\nAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIChhakHAAAA4J5tPf+D\nU48wiW1H/OTUI0zi4WecNvUIk7j2OddOPQIDdh4BAAAAMCQeAQAAADAkHgEAAAAw5J5H7JVrqjcX\n11QDAAAwYucRAAAAAEN2HgHAvZidpJuLnaQAwMFg5xEAAAAAQ+IRAAAAAEPiEQAAAABD4hEAAAAA\nQ+IRAAAAAEPiEQAAAABD4hEAAAAAQ+IRAAAAAEPiEQAAAABD4hEAAAAAQ+IRAAAAAEPiEQAAAABD\n4hEAAAAAQ+IRAAAAAEPiEQAAAABD4hEAAAAAQ+IRAAAAAEPiEQAAAABD4hEAAAAAQ+IRAAAAAEPi\nEQAAAABD4hEAAAAAQ+IRAAAAAEPiEQAAAABD4hEAAAAAQ+IRAAAAAEPiEQAAAABD4hEAAAAAQ+IR\nAAAAAEPiEQAAAABD4hEAAAAAQ+IRAAAAAEPiEQAAAABD4hEAAAAAQ3PFo6o6pKouqqovV9VtVXVZ\nVZ10D8efXFWXVNX2qtpVVVdX1SmrNzYAAAAAa2HenUfnJ3laksclOXW2duneDqyqI5L8P0m+keSs\nJMcnOTfJ7Qc0KQAAAABrbt54dF6SC7v7hu7emeTlSZ5UVafv5djnZDEY/XR3f6W77+7uv+vuXSsZ\nrKpOrKozq+rM3bt3r+RUAAAAAFbJPuNRVR2f5LQkV+1Z6+7PJtmV5Oy9nPLEJJ9J8o7ZZWv/vap+\nbj9me3GS65Jcd/PNN+/H6QAAAAAcqHl2Hh0ze9y5bH1HkmP3cvxJWQxIH0/ywCTPSvKLVXXuCmd7\nSxYvezvr5JNPXuGpAAAAAKyGeeLRbbPH45atH5/F3Ud7O/6m7n5Td3+ju69M8vtZvGfS3Lp7e3df\n393XLywsrORUAAAAAFbJPuNRd+9IcmOSR+1Zq6oHZ3HX0TV7OeXqJL23H7WfMwIAAAAwkXlvmP22\nJK+oqjOq6tgkFya5vLu37eXYdyQ5sap+pqoOqaqzs/hpa+9ZjYEBAAAAWDvzxqPXJ3l/kiuS3JTk\nkCzeyyhVdW5V3b7nwO7+fJInJ/m/snhZ258keVV3/9Eqzg0AAADAGpjrZkLdfVeSl82+lr/2ziTv\nXLb2l0keuQrzAQAAADCheXceAQAAALAJiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAA\nAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAA\nDIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAM\niUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJ\nRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlH\nAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcA\nAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAA\nAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAA\nAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAA\nDIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADM0Vj6rqkKq6qKq+XFW3\nVdVlVXXSHOe9qKq6qn7pwEcFAAAAYK3Nu/Po/CRPS/K4JKfO1i69pxOq6vQk/zrJtfs9HQAAAACT\nmjcenZfkwu6+obt3Jnl5kifNAtHIbyf5xSS37M9gVXViVZ1ZVWfu3r17f34EAAAAAAdon/Goqo5P\nclqSq/asdfdnk+xKcvbgnJ9K8g/d/UcHMNuLk1yX5Lqbb775AH4MAAAAAPtrnp1Hx8wedy5b35Hk\n2OUHV9VpSX4pyU8f2Gh5S5Kzkpx18sknH+CPAgAAAGB/zBOPbps9Hrds/fgs7j5a7u1J/l1333Qg\ng3X39u6+vruvX1hYOJAfBQAAAMB+2mc86u4dSW5M8qg9a1X14CzuOrpmL6f8QJLXVtVXquorSR6f\n5JVV9dHVGRkAAACAtTLvlp63JXlFVf1Fku1JLkxyeXdv28uxD1r2/buTfDTJG/Z3SAAAAACmMW88\nen2SE5JckeTwJB9J8qwkqapzk7y1u49Oku7+wtITq+rrSXZ195dWa2gAAAAA1sZc8ai770rystnX\n8tfemeSd93Du9+zvcAAAAABMa54bZgMAAACwSYlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJ\nRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlH\nAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcA\nAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAA\nAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAA\nAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAA\nDIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAM\niUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJ\nRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlH\nAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAyJRwAAAAAMiUcAAAAADIlHAAAAAAzNFY+q6pCquqiq\nvlxVt1XVZVV10uDYJ1fVn1fVV6rq1qr6aFV99+qODQAAAMBamHfn0flJnpbkcUlOna1dOjj2hCRv\nSfLQJPdL8q4kH6qqBx3AnAAAAABMYN54dF6SC7v7hu7emeTlSZ5UVacvP7C739ndf9rdO7p7d3f/\nxyS3J3nMSgarqhOr6syqOnP37t0rORUAAACAVbLPeFRVxyc5LclVe9a6+7NJdiU5e47zH57kpCTX\nrnC2Fye5Lsl1N9988wpPBQAAAGA1zLPz6JjZ485l6zuSHHtPJ1bVyUkuS/Lvu/szK5ztLUnOSnLW\nySefvMJTAQAAAFgN88Sj22aPxy1bPz6Lu4/2qqpOSfIXST6c5JUrHay7t3f39d19/cLCwkpPBwAA\nAGAV7DMedfeOJDcmedSetap6cBZ3HV2zt3OqamuSjyb5UHf/bHf3agwLAAAAwNqa94bZb0vyiqo6\no6qOTXJhksu7e9vyA6vq25P8VZI/6O6XrdqkAAAAAKy5eePR65O8P8kVSW5KckiSZyVJVZ1bVbcv\nOfYVSb4tyUur6vYlX+eu4twAAAAArIG5bibU3Xclednsa/lr70zyziXfPy/J81ZrQAAAAACmM+/O\nIwAAAAA2IfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHx\nCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEI\nAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgA\nAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAA\nAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAA\ngCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACA\nIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh\n8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHx\nCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgCHxCAAAAIAh8QgAAACAIfEI\nAAAAgCHxCAAAAIAh8QgAAACAIfEIAAAAgKG54lFVHVJVF1XVl6vqtqq6rKpOuofjn1RVf1dVX62q\nT1XVP1u9kQEAAABYK/PuPDo/ydOSPC7JqbO1S/d2YFU9OMl7krwuyXGzxz+tqq0HMigAAAAAa2/e\neHRekgu7+4bu3pnk5UmeVFWn7+XY5yS5qrt/v7u/0d3vTPKJ2frcqurEqjqzqs7cvXv3Sk4FAAAA\nYJVUd9/zAVXHJ7k1ySO7++ol6zuTPLu737fs+Pcm2dbdL12y9qYkD+ruH5l7sKpXJblg9u0dST49\n77mwAR2S5P5JvpTkrolnAWD/+X0OcO/g9zmbxendfb99HbQwxw86Zva4c9n6jiTHDo7f27H/2xzv\ntdRbkrxr9nx7d29f4fmwYVTVmUmuS/I93X391PMAsH/8Pge4d/D7HL7VPPHottnjccvWj0+ya3D8\nvMcOzWKRYAQAAAAwoX3e86i7dyS5Mcmj9qzNbop9bJJr9nLKJ5ceO/PI2ToAAAAAG8i8N8x+W5JX\nVNUZVXVskguTXN7d2/Zy7O8lOaeqnllVh1bVM5M8OsklqzIx3DttT/Lq2G0HsNH5fQ5w7+D3OSyx\nzxtmJ0lVHZLFYPTcJIcn+UiS87r7K1V1bpK3dvfRS45/UpI3JHlwkhuS/Fx3f3j1xwcAAADgYJor\nHgEAAACwOc172RoAAAAAm5B4BAAAAMCQeAQAAADAkHgEAAAAwJB4BAAAAMCQeAQAAADAkHgEAAAA\nwJB4BAAAAMCQeAQAsJ+q6nGD9ceu9SwAAAdLdffUMwAAbEhVtau7j93L+i3dfd8pZgIAWG0LUw8A\nm11VVZIHdPffTz0LACtW/2ih6qgkd08wCwArUFWfTrLP3RTd/R1rMA6sa+IRTKSqjk7ypiTnJrkr\nyX2q6ulJzu7uV086HAD3qKquzeI/OI6oqmuWvfzAJB9d+6kAWKF/v+T5aUlelOT3knwuyRlJnp3k\n4gnmgnXHZWswkap6a5JvS3JBkv/c3SdU1bcl+Yi/bgCsb1X1U1ncdfRrSV665KW7k3wxyeXd/Y0p\nZgNg5arqz5P8Qnd/bMna45K8rru/d7rJYH0Qj2AiVXVTku/o7p1L741RVTu6+/iJxwNgDlX1hO7+\nL1PPAcCBqapdSU7o7ruWrC0kuWVv97aDzcanrcF0tiT56tKF2aVst08zDgD74ZS9LVbVK9d6EAAO\nyGeS/MyytRcl+ewEs8C6Y+cRTKSq3p3kU9396j07j2b/2PiO7n721PMBsG9V9cUk70ny0u7+RlWd\nmOT3k5zuEmSAjaOqHp/kg0m+lGRbkq1J7p/kh7r7r6abDNYH8QgmUlUPSvLns29PT3JdksOSfG93\n3zTZYADMrapOSfKHSY5KclEWb776l0le2N3/MOFoAKxQVd03yT/P4n1Jb0ry3u7ePu1UsD6IRzCh\nqjo8yVOz+JeNzyf5QHd/9R5PAmBdmd0T4+NJzk5ycXcvv+wBAGBDE48AAPZTVR2fxY91PivJ7yZ5\nZZJf6e5fnXQwAFakqrYk+VdJnpPk/t19clX9QJLTuvu3p50OpicewUSqqpL8RJJzkhyz9LXuPm+S\noQBYkaraluRvkrygu2+vqocn+eMkn+3uH5p0OADmVlWvTvLDWbwE+Te7+/iqekiSd3f3o6adDqbn\n09ZgOv8xya8neVCSQ5d9AbAx/Gp3P7O7b0+S7r42yWOS7Jp2LABW6F8meWp3vyvJ3bO1z2Xx9hKw\n6dl5BBOpqu1JHtvdPv4TAAAmVFVfzuLlancv+STkw5Lc2N0PmHo+mJqdRzCdO5LcOPUQAByYqnpG\nVb2/qq6Yff/4qnrq1HMBsCJXJzl32dqPJrlygllg3RGPYDq/muTfzO59BMAGVFU/m+QNWbzv0Zmz\n5R1ZvHE2ABvHzyd5c1V9IMlRVfXuJG9Mcv60Y8H64LI1mEhVfSbJ6VncgXTz0te6+8y9ngTAulJV\n1yd5Wnd/uqpu7e4TquqQJF/q7pOmng+A+VXVKUmel+SMJJ9P8rvd/YVpp4L1YWHqAWAT+3dTDwDA\nATupuz89e77nL3K15DkAG0BVHdbd/1+SX1m2fmh33znRWLBuiEcwke6+ZOoZADhgn66qH+zuy5es\nfX+ST001EAD75atV9QdJnrcsFm1PcuxEM8G6IR7BGqqqx3X3f5s9/z9Gx3X3X6/dVAAcgF9K8t6q\nujTJ4VV1UZLnJPmRaccCYIW+nuQBSS6vqqd3967ZuvuTQtzzCNZUVd3W3cfMnt89OKy7+5A1HAuA\nA1BVj0jyM/nmPTJ+vbv/dtqpAFiJqtqV5L5JfjfJI5I8qbtvqqpd3W3nEZueeAQAsEJVdX53v37q\nOQBYHUsjUVW9NsmzkzwlyV+JRyAeAQCsmL9EA9y7LL1CYPb9TyV5XZKju/uw6SaD9cE9j2ANVdUv\nzHNcd7/2YM8CwAFxDwyAe5dv+e/v7n5rVX0hybMmmgfWFTuPYA1V1V/McVh39/ce9GEA2G9VdXuS\nh+UeIlJ337h2EwEAHDziEQDACs0+9GD0H1EVH34AsO5V1UXd/fOz528eHdfdL1m7qWB9ctkaTKyq\nTklyWnd/bOpZAJjbHUnOmnoIAA7ISUue32+yKWADsPMIJlJVJyd5V5LvTXJHdx9dVT+e5And/dPT\nTgfAPXHDbABgM9ky9QCwib05yeey+FeOO2drf57kByabCIB5uWE2wL1IVf2nwfp71noWWI9ctgbT\neWKS07v7a1XVSdLdX57tSAJgfXttklTV1u7etvzFqjq9uz+/5lMBsL+eOFj/nrUcAtYr8Qim8/Us\n+/9gVd03yS3TjAPAvLr7dbOn1yTZ2+Vrf5vkvms3EQD7o6p+ZPb0kKr65/nWnaX/JMmtaz8VrD/i\nEUznw0neUFU/u2Tt1Uk+ONE8AKzcP7p8raoOzfiT2ABYX35j9nhEkt9csn53ki8m+VdrPhGsQ+IR\nTOflSd6bxb9mHFFVO5J8MsnTJp0KgH2qqo9kMRAdXlUfXvbyaUk+sfZTAbBS3f3AJKmqy7v7B6ee\nB9Yr8QhbQSPTAAAOEklEQVQm0t23JPmnVfXoJGck+XySK9tHIAJsBH81e3xCkv+6ZH3PX6rfveYT\nAbDfhCO4Z+XfqbA+VNUZSe7q7hunngWA+VTVj3X3H089BwAHpqqOSPKiJOckOWbpa939w5MMBevI\nlqkHgM2qqn63qh4/e/7MJP8jyQ1V9ZPTTgbACvxtVZ2UJFV1VFW9qqp+efaPEAA2jt9O8pIk25P8\n3bIv2PTsPIKJVNXfJ3lwd3+1qq5I8itJdiZ5S3d/57TTATCPqroyyXO7+1NV9eYsfqTzN7J4GfIL\nJx0OgLlV1a1JHtbdX5x6FliPxCOYSFXt7O7jquqEJJ9NcmJ39571qecDYN+q6pZ88/f3TUken+S2\nJNd29ynTTgfAvKrqhizGo69PPQusRy5bg+ncVFVPSPLjST46+4fHsUl2TzwXAPOrJIdU1bcnuaO7\nt3X39iy7XwYA694vJ3ljVd1n6kFgPfJpazCd1yT5SBYvb3jybO37k1w92UQArNR/S/IbSR6Q5M+S\npKq2JrllupEA2A8XJzkyyQur6h+WvtDdx04zEqwfLluDCVXVkUnS3V+dfX9yki2utQbYGKrq9CSv\nzeIfAl7W3dur6seSPLK7XzntdADMq6p+cPRad1++lrPAeiQeAQAAADDksjWYSFXdL8l/SPJ9Se63\n9LXuPmSSoQBYsao6Mcljsvi7vPasd/fvTTYUACtWVQ9O8k/zj3+f/+pkQ8E6IR7BdN6c5IFJnp/k\nD5I8M8n5Sf54yqEAmF9VfX+Sy7J42drxSXbMHj+XRDwC2CCq6l8keWeSzyQ5K8l1Sb49yceSiEds\nei5bg4lU1ZeSPLy7b66qHd19fFWdluRPuvuxU88HwL5V1ZVJ/qC731BVt3b3CVX1b5Lc3t1vnHo+\nAOZTVdckuai7L13y+/xFSb6tu39p6vlgauIRTKSqbklyYnd3Vf19kod29z9U1S6f6ACwMVTVziT3\n7e67lvwh4PAkn+nu06aeD4D5VNWuJMfN/tt8Tzw6NMnnu/uUqeeDqblsDaZzfZJHJbkqySeT/MLs\nHyFfmnQqAFbijiSHzx63z3aQ3prkhEmnAmCldiU5ZvZ4c1WdmeSWJEdPOhWsE+IRTOcXkhyx5Pkf\nJjk2yQsmmwiAlfrrJE9P8q4kH0ryviRfT/I3Uw4FwIr9ZZJnJPmdLN7L7sNJ7kzykQlngnXDZWsw\noao6OskPJTk1yReSfLC7b5t2KgDuSVVd0d2PmT2/MMmru/uOqjoiyb/O4l+u39jdN085JwD7p6q2\nJHleFn+fv727b594JJiceAQTqapzkvxZkq8muTHJaUmOTPLk7r5yytkAGKuqHUlOmN0Xw33qAIB7\nPfEIJlJVH09yWXdfuGTt5Ul+dM9ftAFYf6rq8iRHJfl0kucmecfejuvu89ZuKgAORFW9efRad79k\nLWeB9WjL1APAJvawJG9YtvbGJN8+wSwAzO8nsrhztGbfHzr4AmDjuN+yr7OTnJfkAVMOBeuFG2bD\ndK5O8p2zxz0evux7ANaZ7r41yeuSpKru393Pm3gkAA5Qdz9z+VpV/XiSx08wDqw7LluDNVRVP7nk\n24ck+akkb0/y+SRbk/yfSd7W3f927acDAAD2mN04+8vdfeLUs8DUxCNYQ1X1uTkO6+5+8EEfBgAA\nGKqq70tyaXefMvUsMDWXrcEa6u4zpp4BAAD4VlV1bZKlOyvuk8VPQ375NBPB+iIeAQAAsNn9+rLv\nb0vyie7+71MMA+uNy9YAAAAAGLLzCAAAgE2tqn5snuO6+48P9iywHtl5BAAAwKZWVf8zyf2THJLk\nliQnJLk7yReT1Oyw7u7TppkQprVl6gEAAABgYm9IcmmSE7r7fklOTHJJkjd094NmX8IRm5adRwAA\nAGxqVfWFJA/t7q8tWTsqyfXdfep0k8H6YOcRAAAAm91hSe63bO2kJIdPMAusO26YDQAAwGb3R0n+\nrKp+Jcnnk2xN8srZOmx6LlsDAABgU6uqw5JckOTZSU5J8vdZvAfSq7v761POBuuBeAQAAADAkHse\nAQAAwExVvXHqGWC9sfMIAAAAZqpqV3cfO/UcsJ7YeQQAAADfVFMPAOuNeAQAAADf9PGpB4D1RjwC\nAABgU6uqx+153t3ft2T9sdNMBOuLex4BAACwqY3uc1RVt3T3faeYCdYTO48AAADY7P7RfY6q6qgk\nd08wC6w7C1MPAAAAAFOoqmuTdJIjquqaZS8/MMlH134qWH/EIwAAADarX8/irqNfS/IbS9bvTvLF\nJJdPMRSsN+55BAAAwKZWVU/o7v8y9RywXrnnEQAAAJvdKXtbrKpXrvUgsB6JRwAAAGx2v1ZVv1lV\nhyVJVZ1YVR9K8uyJ54J1QTwCAABgs3tUku9M8tdV9eNJrk7ylSSPmXQqWCfc8wgAAIBNr6oWknw8\nydlJLu7un5l4JFg37DwCAABgU6uq45O8J8l9kvxikmdV1cunnQrWDzuPAAAA2NSqalv+//buP/T2\nu64D+PPZNq+KW+4XmtRctCYIIokOpqREG/unUnSVURD9oYImJEgoCOICGQ5BMMOKYOxHNUewubBI\ncRqmyCg0F0OdTEOlWbt3u5tTJturP865ePl6z0XZ3f0c+TwecDifz/t8zvs8P39+X9/3+/VJPpfk\nDTPzSNsXJflIkq/NzG8sGg72gJVHAAAArN37Zub3ZuaRJJmZL2XT7+josrFgP1h5BAAAAMBOVh4B\nAACwem2vbntH27u2569o+5tL54J9oHgEAADAqrX94yTvz6bv0aXb4QeTvHOxULBHbFsDAABg1dp+\nJcmrZ+aetkdm5ty2ZyS5f2YuWDofLM3KIwAAANbugpm5Z3t8bIVFjzuGVVM8AgAAYO3uaXvVgbEr\nkty9RBjYN2cuHQAAAAAW9q4kt7W9Mcmhttcl+cMkr102FuwHK48AAABYtZm5M8mrkhzKpmn2eUmu\nmpnPLBoM9oSG2QAAAKxS23fMzLVL54B9p3gEAADAKrU9OjPnLJ0D9p1tawAAAKxVlw4APw00zAYA\nAGCt2vYXcpIi0sz892nMA3vJtjUAAABWqe0TSXb9UdwkMzNnnMZIsJesPAIAAGCtHk3ygqVDwL5T\nPAIAAGCtnpiZby0dAvadhtkAAACslYbZ8GNQPAIAAGCt3pskbS8+0Ydtn386w8C+0jAbAACAVWt7\ndGbOOcH44Zk5b4lMsE+sPAIAAGDtfmT7WtuzsvtJbLAqGmYDAACwSm0/nk2B6FDbfznw8UVJ/uP0\np4L9o3gEAADAWn1m+/6qJP923PgTSf4nya2nPRHsIT2PAAAAWLW2vzMzH1k6B+wrxSMAAABWre0v\nJzkyM//X9plJ/jTJ40mum5nvL5sOlqdhNgAAAGv3d0meuz2+Nslrk7w6yQcWSwR7xMojAAAAVq3t\n4STnz8y0/VaSVyR5OMmXZuZ5y6aD5WmYDQAAwNo1yRltL0ny6Mx8PUnanr1oKtgTikcAAACs3eeT\nfCibrWsfS5K2Fyc5vFwk2B96HgEAALB2b0ryrGyKRddsxy5L8reLJYI9oucRAAAAADvZtgYAAMDq\ntT0/ycuSXJhND6QkyczcsFgo2BNWHgEAALBqba9I8g9JHkvy7CQPbt/vm5lLl8wG+0DPIwAAANbu\n2iTXzMyFSR7Zvv9Zkg8vGwv2g5VHAAAArFrbh5KcNzOPt31wZp7d9lCSr87MRUvng6VZeQQAAMDa\nPZrk0Pb4gbYXJXlaknOXiwT7Q/EIAACAtftsktdsj/8pyUeTfCLJ5xZLBHtE8QgAAIDVaXvXcaf3\nJrlte/z2JLcmuTPJH5zuXLCP9DwCAABgddo+mOTcmZm2R2fmnKUzwb46c+kAAAAAsIDPJ/nXtvck\neXrbvzrRRTPzxtMbC/aPbWsAAACs0euTfCxJt+dn7XjB6tm2BgAAwKq1/ejM/NbSOWBfKR4BAAAA\nsJNtawAAAADspHgEAAAAwE6KRwAAAADspHgEAAAAwE6KRwAAAADspHgEAKxe20+1/cu27257f9sH\n2v5N22duP79ye83htg+1/XTbyw7MMW3f0vbWtt9t+422V7f92bY3t3247X1tX3fgez/X9qbtbz7U\n9s62Lzmd9w8AcDKKRwAAG7+b5Owkr0zy+0lel+Rt28+eleQvklye5OVJvpzkn9uef2COdyW5I8mL\nt+83Jvn7JJ9I8itJbk9yQ9sLk6TtM5LcmeSMJFcmeVmSu5J8su1zn5K7BAD4CXVmls4AALCotp9K\ncs7MvOS4sb9OcsnM/NoJrm+S7yT5k5m5eTs2Sd4/M2/fnp+b5HCSP5+Zt27Hzk5yNMlrZub2tn+U\n5N1JfmlmHj9u/v9Kcv3MXPeU3DAAwE/AyiMAgI0vHDj/dpLnJEnbX2x7Y9t72x5N8nCS85M8/8B3\nvnjsYGaOJHksyX8eN/ZwkkePzZvkpUl+PslDbR859krygiSXnLI7AwB4Es5cOgAAwJ547MD55If/\naPvHbFYavTnJN7fXfjrJ0w585wcnmONEY8fm/Zkkdye5+gR5jv64wQEAnkqKRwAAJ7Hta/TCbLao\nfXw79rwkp6In0b9n01/pyMw8cArmAwA45WxbAwA4uSNJ/jfJG9te2vbyJLck+d4pmPvmbFYy3dH2\n19te3Pbytu9p+6unYH4AgCdN8QgA4CRm5okkv53k0mz6F12f5IPZFH2e7Nzfy+bpbncnuSmbp7jd\nsv2tbz/Z+QEATgVPWwMAAABgJyuPAAAAANhJ8QgAAACAnRSPAAAAANhJ8QgAAACAnRSPAAAAANhJ\n8QgAAACAnRSPAAAAANhJ8QgAAACAnf4fPAt8//aSISoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ffd0940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.plot(kind='bar', x='name', y=['p', 'r', 'f'], ylim=(0, 1), figsize=(20, 8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What I didn't look at here\n",
    "\n",
    "* Sub-word embeddings for performance on unknown words.\n",
    "* Not using hierarchical softmax to give slower, but better models.\n",
    "* Pre-initialising with other word embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Conclusions\n",
    "\n",
    "`fastText` is worth trying.\n",
    "* Works relatively well for text (see also `*Sem` \"embed all the things\")\n",
    "* Can have small model sizes (for mobile or serverless platforms)\n",
    "* Trains quickly enough to tune and spend time getting more data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Thanks!"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}