180412_AUX_CH6b.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "補助本「ゼロから作るDeep Learning」\n",
    "3章～7章までを輪読会の補足として、読んで行きます。\n",
    "Eli Kaminuma 2018.1.31\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 補助本6章 [2018.4.12]\n",
    "# 6章　学習に関するテクニック\n",
    "  \n",
    "- 目次\n",
    "```\n",
    "6.4 正則化\n",
    "6.4.1 過学習\n",
    "6.4.2 Weight decay\n",
    "6.4.3 Dropout\n",
    "6,5 ハイパーパラメータの検証\n",
    "6.5.1 検証データ\n",
    "6.5.2 ハイパーパラメータの最適化\n",
    "6.5.3 ハイパーパラメータ最適化の実装\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4 正則化\n",
    "- 機械学習の問題では、過学習（overfitting）が問題になる\n",
    "- 過学習とは\n",
    "  - 訓練データだけに適応しすぎる\n",
    "  - 訓練データにない他データにうまく対応できない状態\n",
    "- 機械学習は汎化性能を目指す\n",
    "  - まだ見ぬデータも正しく識別できるモデルが良い\n",
    "\n",
    "### 6.4.1 過学習\n",
    "-　過学習が起きる原因\n",
    "  - ● パラメータを大量に持ち、表現力の高いモデルである\n",
    "  - ● 訓練データが少ない\n",
    "  \n",
    "- わざと過学習を発生させる。\n",
    "   - MNIST訓練データを本来60,000 個から300 個に限定\n",
    "   - ネットワークの複雑性を高め7 層\n",
    "   - 各層のニューロンの個数は100 個、活性化関数はReLU\n",
    "- 該当ファイルはch06/overfit_weight_decay.py\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#前準備\n",
    "import numpy as np\n",
    "import sys, os, time\n",
    "from collections import OrderedDict # 順序付きDict\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "sys.path.append(os.pardir) # load_mnistするために親ディレクトリを追加。必要に応じて変更 or dataset.mnistを別途import\n",
    "from dataset.mnist import load_mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# データの読み込み\n",
    "(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True, one_hot_label=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## これまでの章で使った関数のインポート。 \n",
    "## 本来は下記の2つのインポート文でも良いが、参照しやすいように記載\n",
    "# from common.layers import *\n",
    "# from common.gradient import numerical_gradient\n",
    "\n",
    "# 3.2 章\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))    \n",
    "\n",
    "# 4.4章より, 微分を使った勾配算出用 \n",
    "def numerical_gradient(f, x):\n",
    "    h = 1e-4 # 0.0001\n",
    "    grad = np.zeros_like(x)\n",
    "    \n",
    "    it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite'])\n",
    "    while not it.finished:\n",
    "        idx = it.multi_index\n",
    "        tmp_val = x[idx]\n",
    "        x[idx] = float(tmp_val) + h\n",
    "        fxh1 = f(x) # f(x+h)\n",
    "        \n",
    "        x[idx] = tmp_val - h \n",
    "        fxh2 = f(x) # f(x-h)\n",
    "        grad[idx] = (fxh1 - fxh2) / (2*h)\n",
    "        \n",
    "        x[idx] = tmp_val # 値を元に戻す\n",
    "        it.iternext()   \n",
    "        \n",
    "    return grad\n",
    "\n",
    "# 3章より\n",
    "def softmax(x):\n",
    "    if x.ndim == 2:\n",
    "        x = x.T\n",
    "        x = x - np.max(x, axis=0)\n",
    "        y = np.exp(x) / np.sum(np.exp(x), axis=0)\n",
    "        return y.T \n",
    "\n",
    "    x = x - np.max(x) # オーバーフロー対策\n",
    "    return np.exp(x) / np.sum(np.exp(x))\n",
    "\n",
    "# 4章より\n",
    "def cross_entropy_error(y, t):\n",
    "    if y.ndim == 1:\n",
    "        t = t.reshape(1, t.size)\n",
    "        y = y.reshape(1, y.size)\n",
    "        \n",
    "    # 教師データがone-hot-vectorの場合、正解ラベルのインデックスに変換\n",
    "    if t.size == y.size:\n",
    "        t = t.argmax(axis=1)\n",
    "             \n",
    "    batch_size = y.shape[0]\n",
    "    return -np.sum(np.log(y[np.arange(batch_size), t] + 1e-7)) / batch_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 過学習を再現するために、学習データを削減\n",
    "x_train = x_train[:300]\n",
    "t_train = t_train[:300]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:201, train acc:1.0, test acc:0.7793\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import numpy as np\n",
    "sys.path.append('../')\n",
    "from saitobook.common.multi_layer_net import MultiLayerNet\n",
    "from saitobook.common.optimizer import *\n",
    "\n",
    "network = MultiLayerNet(input_size=784, hidden_size_list=[100, 100, 100,100, 100, 100], output_size=10)\n",
    "optimizer = SGD(lr=0.01) # 学習係数0.01 のSGD でパラメータ更新\n",
    "max_epochs = 201\n",
    "train_size = x_train.shape[0]\n",
    "batch_size = 100\n",
    "train_loss_list = []\n",
    "train_acc_list = []\n",
    "test_acc_list = []\n",
    "iter_per_epoch = max(train_size / batch_size, 1)\n",
    "epoch_cnt = 0\n",
    "for i in range(1000000000):\n",
    "    batch_mask = np.random.choice(train_size, batch_size)\n",
    "    x_batch = x_train[batch_mask]\n",
    "    t_batch = t_train[batch_mask]\n",
    "    grads = network.gradient(x_batch, t_batch)\n",
    "    optimizer.update(network.params, grads)\n",
    "    if i % iter_per_epoch == 0:\n",
    "        train_acc = network.accuracy(x_train, t_train)\n",
    "        test_acc = network.accuracy(x_test, t_test)\n",
    "        train_acc_list.append(train_acc)\n",
    "        test_acc_list.append(test_acc)\n",
    "        epoch_cnt += 1\n",
    "        if epoch_cnt >= max_epochs:\n",
    "            break\n",
    "            \n",
    "\n",
    "print(\"epoch:\" + str(epoch_cnt) + \", train acc:\" + str(train_acc) + \", test acc:\" + str(test_acc))\n",
    "\n",
    "      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VdW5+PHvm5mMhCRMCZAwgyggAVGBgkoFnMBZAZW2\norVa+7vVSlun1t5bq1evtXVCxVqh4lRELU4oSK0MBgggMoUhkBBIyEDmef3+WIcQQgIHyD7nJOf9\nPM95OGft4bzZCfvde6211xJjDEoppRRAgLcDUEop5Ts0KSillGqgSUEppVQDTQpKKaUaaFJQSinV\nQJOCUkqpBpoUlF8TkZ4iUioigd6ORSlfoElB+QUR2SMiFa4EcOTV3Riz1xgTaYypa6Xv6S0iH4lI\niYgcEpEnTrDurSKyVkSKRSRLRJ4QkaBGyzuJyCIRKRORTBG5uTViVOpENCkof3KFKwEcee1vzZ2L\nSAjwOfAl0BVIAuafYJNw4BdAPHAecDFwX6PlzwHVQBdgOvCCiJzVmjEr1VTQyVdRqv0SkWRgNxBs\njKkVkRTgdWA4sBrYBsQYY2a4sbvbgP3GmKcblW1saWVjzAuNPmaLyAJggiuuCOAaYIgxphT4WkQW\nAzOBOe79dEqdOr1TUOpY/wDWAHHAo9iTsLtGA3tE5GNX1dFyETn7FLYfB2x2ve8P1BpjtjdavgHQ\nOwXlKE0Kyp+8LyJFrtf7TReKSE9gJPCwMabaGPM18MEp7D8JuBF4FugO/AtY7KpWOiER+RGQCvyv\nqygSKG6yWjEQdQrxKHXKNCkofzLVGNPR9ZrazPLuQIExprxR2b5T2H8F8LUx5mNjTDX2BB8HDBKR\n6Y0auD9uvJGITAX+CEw2xhxyFZcC0U32HwOUnEI8Sp0yTQpKHZUDdBKR8EZlPU5h+41As8MOG2MW\nNGrgnnykXEQmAS9jG8E3NdpkOxAkIv0alQ3laPWSUo7QpKCUizEmE0gDHhWREBE5H7jiFHYxHxgt\nIpe4nnv4BXAI2NLcyiJyEbAAuMYYs6ZJLGXAP4Hfi0iEiIwBrgTeONWfS6lToUlBqWNNB84H8oE/\nAG8BVUcWuqp/xja3oTFmGzADeBEoBK4CrnRVJTXnIWyV0JIWqpbuAjoAudgG8J8aY/ROQTlKdJId\npVomIm8BW40xj3g7FqU8Qe8UlGpEREaKSB8RCXDV918FHNdTSan2yrGkICLzRCRXRL5rYbmIyLMi\nkiEiG0XkXKdiUeoUdAWWY3v/PIutslnv1YiU8iDHqo9EZBz2P9bfjTFDmlk+BbgHmIJ9xP/Pxpjz\nHAlGKaWUWxy7UzDGrAAKTrDKVdiEYYwxq4COItLNqXiUUkqdnDfHPkrk2AeDslxlOU1XFJHZwGyA\niIiIEQMHDvRIgEodKq2ivProAKqhQQHERYYSFCCUVdVSWF5DfSvfbRsDJZU1xzzwEBESRGCAUFpV\nS70xBIoQGRZETV39MfGp9i0hMpSuMWGnte3atWsPGWMSTrZemxgQzxgzF5gLkJqaatLS0rwckfIl\n5dW1lFTW0iXa/mfJLa7kR69/y9RhifxkbG/2HCqjsLyaTdmH+cfqvSTHRTB9dE8iQ4/++dfUGRat\nz2L1rgKuGpbI2P7xLFqXzRurMhnYKZyQoADqjWFXXhmlAUJoUAAV1XV06RBMQlRoq/9MI3rGcusF\nyUSFBfHBhv18uGE/tfWGs7pHM+XsbixOz2b7wVJCAgO4+txEJg3pSmCAtHocyrdEhAYRHRZ8WtuK\nSKZb6znZJdU1AuVHLbQpvAQsN8a86fq8DRhvjDnuTqExTQr+ZfWufFbtKiAiNJAbRx17Igc4WFzJ\n9FdWsyuvlCuHduen4/vyx4+3sHxbHgBndY9m8/6jQwidnRhDZn4ZxZW1x31XSFAA5yTGkJZZ2FB2\n+9gUfjNlECL2hLv9YAmL1mdTVVNP74QIrh2RRFiwzs+jfJ+IrDXGpJ5sPW/eKXwA3C0iC7ENzYdP\nlhCUf6muredn/1jPoVL77NiHG3OYOboXi9OzqXBVmWQWlFNeVcsNI3uyOD2b99PtFAmPXDGY7QdL\nWbrlIPdfOoDB3aOJjwhlSGI0pVW1rNtbdFy1z5DuMSREhZKRW8q+wnJiOgQzvEfHhoQA0L9LFA9M\n0upL1X452fvoTWA8dgKRg8AjQDCAMeZFsf/T/gpMAsqBWcaYk94C6J1C+1VTV8+6zEJG9IolKDCA\nxenZ3LswnddmjaS2zvCzBeuorqsnOS6cxNgOAIQGBfLzi/sxrEdHCsuqeX3lHsqr6/j15IHHnMyV\n8nfu3im0uSeaNSm0T5U1dfx0/lqWbcsjKbYDd/ygD4vWZVFQVs2XvxxPQICQvq+I/NIqJgzoTIDW\nnyt1StpC9ZHyE1mF5by8YhfLt+c1VNnER4by9PXDSI4L56vteTz9+XY2ZR/mzh/0YfXufB563z7z\n+OBlgxoSwLAeHb32MyjlLzQpqDP2/f5i0jILEBHG90+gc3Qon20+SGF5NRv2HWZxejYAFw3s3NBQ\nvHx7Hte/tJLOUaFs3l9M95gwnrv5XKac3Q1jDKt2FbB8ey43jerpzR9NKb+jSUGdtsKyau57ZwNf\nbM1tKAsMEGLDgzlUagcGDQsOYMboXswe15vuHTs0rLfjYAkzX11DRXUdT1x7DlOHJRISZJ+lFBHO\n7xPH+X3iPPsDKaU0KagTO1xeQ1VtHZ2jj31gJrekkpmvrGF3fhn3/bA/14xIoqqmnvmrMtlbUM6M\n0b04q3s04SFBdAg5vstmvy5RrPjVBIICRNsHlPIhmhRUi+rrDdNfXcXWnBIuO6cbSbEd6BUXwajk\nTsz627ccLK7kb7eN5IK+8Q3bPHj5YLf3f+TOQCnlOzQpqBb9a1MO32UXM2FAAl9uyaW8po66ettQ\nHBUWxBs/HsWIXp28HKVSqjVpUlDH+X5/MRl5pfzf59sZ0CWKV24dSWCAYIzhm535vLcuix9dmMKQ\nxBhvh6qUamWaFPzcweJKnl+W0TCoWnZRBd/szG9Y/uqtqQ1j6ogIF/aN58JG1UVKqfZFk4Ifq6s3\n/PzN9azbW0hCpB3ULSw4kF9NGsDEQV3oEBJIUmy4l6NUSnmSJgU/kV9a1TDS5hHbD5awencBT1x7\nDten9vBidEopX6FJwQ/sL6pg+iur2X2o7Lhl04Ynct2IJC9EpZTyRZoU2pnKmjqyiyrokxAJQHFl\nDTe9vIqC0moWzh7NWd2jj1k/6jTHZldKtU+aFNqZ55fv5NkvdjAqpRM/m9CXxenZ7Cso5607zmdk\nsnYfVUqdmCaFNq66tp7gQGkYJvqbjEN0iwljX0E5t85bA8DPL+qrCUEp5RZNCm1YbkklFz/1Fb3j\nI7j/0oGkJseyMeswt12YzH0/HMD767PZkVvCPRf383aoSqk2QpNCG7ZwzT5KKms5VFrNrL+t4dkb\nh1NdV8+IXrGEBAVw/UjtUaSUOjU6+EwbVVNXz4LVmYzrn8AbPx5FTZ3hdx9+D0Bqr1gvR6eUaqs0\nKbRRn3x3gIPFVdwyuhe9EyIZ2y+eA8WV9E6IIM71IJpSSp0qTQpt0NrMAn7zz030SYhgwsDOANx6\nfjKgdwlKqTOjbQptzMHiSma+uoYu0WG88ePzGsYlmjCwM9PP68m1+iCaUuoMaFJoYxas3ktFTR3z\nbht5zExmgQHCf08724uRKaXaA60+akOqa+t5c81exvdPICU+wtvhKKXaIU0Kbcgnmw+QV1LFLRck\nezsUpVQ7pUnBBy3ZlMOba/YeV/7Gyj30igvnB/0SPB+UUsovaJuCD3p+eQab9xeTEh/B6N5xgJ0N\n7ds9hTx42SCd6F4p5Ri9U/Ax9fWGjNxSjIFfvr2BovJqAN5YtYew4ACuG6FPKSulnKNJwcdkF1VQ\nWVPPjSN7kFdSxY1zV/Hu2izeX7+fqcMSiQnXoa6VUs7RpOBjMvJKAbj63CRevS2VPfll3PfOBrp1\nDOOu8X29HJ1Sqr3TNgUfk3HQJoV+nSOJjQjh3TsvIOdwJRcN7NzwoJpSSjlFk4KPycgtJS4ihNiI\nEACGJMYwJDHGy1EppfyFVh/5mB25JfTtHOntMJRSfkqTgg8xxvY80qSglPIWTQo+JK+0iuLKWvpp\nUlBKeYkmBR+yNacEgH5dorwciVLKX2lS8CFrMwsJEDgnSRuWlVLe4WhSEJFJIrJNRDJEZE4zy2NE\n5EMR2SAim0VklpPx+Lq0zAIGdo0mKkwfUFNKeYdjSUFEAoHngMnAYOAmERncZLWfAd8bY4YC44Gn\nRCTEqZh8WW1dPev3FjEyWWdOU0p5j5N3CqOADGPMLmNMNbAQuKrJOgaIEhEBIoECoNbBmHxScWUN\nW3JKKK+uIzW5k7fDUUr5MScfXksE9jX6nAWc12SdvwIfAPuBKOAGY0x90x2JyGxgNkDPnj0dCdZb\nvtqex22vreFs1wNqqXqnoJTyIm83NF8KpAPdgWHAX0UkuulKxpi5xphUY0xqQkL7mkvg7bR9GAMb\nsw6T2LED3WI6nHwjpZRyiJN3CtlA43Gek1xljc0CHjfGGCBDRHYDA4E1DsblM0qraln6/UGmn9eT\nhKhQukaHeTskpZSfczIpfAv0E5EUbDK4Ebi5yTp7gYuBf4tIF2AAsMvBmHzK598foKq2nmnDE7Ut\nQamTqauFgEAQgapSCAqFQId76lWXQXW5/d4Osfa7jYGqEvv9AcFQkgOmHqK6QWAQ1FZDbYUtM8b1\nbz2Ex9n9VBZDfa3dX1UJFO6B4mxIHguhkbBrOQSHQ1RXKDlol5k6GHKNsz+ri2NJwRhTKyJ3A58C\ngcA8Y8xmEbnTtfxF4DHgbyKyCRDgAWPMIadi8iVrMwt4cfkuEjt24Nye2o6gHFBTaU9iQaH25FZR\nCNGJtgygtgoqD9tlNeX25BcaCZ0H2eU5G+Fwlj2BhYRDTE+I6gJhrudoNr1rT3L71sChHVBZBKk/\nhmE32ZPZuz+yJ8eivdCpDySlwrDp0GWwXb5rGWz50H5PeKw9cd70FsQkQvo/YO3f7H7rqkECoKoY\nfnsAgjvAv34J378P3YZCZGcIDLVxXf60je0/f4aCXRDUwW478DJIGglBIZC/E16/EmKSoGNPqK+B\nrDS49H9g8JWwdxW8MwtqK6Gi4OjxvPFNGDgFNr4Ni2bbsoAge3wAZi+H7sNh3euw5L7jfx/3boDY\nZPj2FfjidxAUZr8DQAJhTqZ9v+4N+O7dY7eN69f2kwKAMWYJsKRJ2YuN3u8HfuhkDL5mze4Cnvps\nG6t3FxAbHsx/Tztbp9dsD2oq4Lv3oP8kiIg/Wm6MfQW00HyX8QVsXmSvFrsPg+G32BNtXB+7fOeX\n9sSdlQbZ6+xJ9ryfwtAboLwAPv4VIPY7u55tT6ZTX4COPWDlX2DF/9oTVc5GqCmzJ6V71gMGnjkH\nSg8cG8+AKXDTm/b9G9OgvMk12vAZcNVzNoG892NbFhQGCQPslW9QqC2rq7LfERIB/S6FQ9thzVxI\nHmOTQk46LLoDIhIgZZy9YpYAuw3YhBUQbE/SIZFQX2evtINdbW4jboXwTpCz4WjiiGjU3liwG7b+\ny+6nrhpWPQdnXQ3XvWbjTBkLRftg3yr7++kxyh4zsMv7XgSBITZxhEbb/XQfZpcnngsTH7M/Y3UZ\nxPSwySE2xS7vcZ5NMBLQ6CV2vwD9fmiPU/F+G3OnFOjYC0JdIxlc+t82AZTlQlR3iO4GkV2a//tx\ngNjq/LYjNTXVpKWleTuM07I4PZv/ensDCZGh3D6uNzeN6kF4iI5e7jOMsVfT4SeoytvztT35dz0H\nBl4OkQn2Snj+NfbEF50E016EXhfaRJCzEV4aa/9Th0bbqobuw2Hi7+3J5fOHYd3f7ck6Z4O9Wu51\nIcxaAmX58FR/eyUaGArdzrFXw8NnwFnTbJJ425VEinPsSSo6EX6yFKK7w/71sGEh7F0J3YZB58E2\nwYx/wP4sG9+xV/chEba6IiTCVoF0HWKX7/jcnogDQ+xJu2iv3W/KWHuSLtxjyzsPOpoMTqS2ChB7\ntV5VYk/K8f08UAVUbn9nGBg+8+idkp8RkbXGmNSTrqdJwTO+3nGImfNWMyq5E6/eNpLIUE0GHlFb\nDWtfs1dyvccfvQJvqr4OXhoHB7+D7ufaKomSHHsSGXW7vaqbO8FeWQeG2KvP4HD40SeQMMienPtf\nCv9+Gg7vhaE3w7QXbDVJ2qu2Xriy2F4Z7vwSbloInQe66qY72Lro/J2w+yuIHwDJF9oklZNuE0WX\ns+3JtCXV5ZC7xd4tnGg95bfcTQp6ZvKQl1bspEtUGH+bNYoOIYHeDqd92PE5fHgvXPI7OOe645cf\nzoZ3boWsb21VxOVP26Sweq49UY+63VYpdIi1V9tDrrbVJxlL7VVsVJejV8DB4dDvEug6FM6daU/g\nGxdClyF225sX2vWGXG2rLbLX2SQQ1QUm/ObYuIw5erUa2mjww7g+xyYtEXtX4Y6QcEga4d66Sp2A\n3il4wK68Ui566it+ObE/91zcz9vh+LbSXNsIJwFw5V9OvG7BLnh2uL1yn/qCrYZIGmXrYDNXwlsz\nbEPeVc9Bz9G2Cmf9G/DBPRDR2dbZAlz9SvNJRal2RO8UfMgbqzIJDhRuHNW+nsY+bTWVUJZ3tGEP\n7NVz+gL49Le2J8zI2215XQ2seBLOvcX2yti70tabT30eOvWGX+2Gly862uh501s2KeRutvXhN8yH\nhP5Hv6fHaLuvKU9BtqvxtqLw2Kt3pfyY3ik4rK7eMOIPnzO2XwJ/ucnNqoD2oLYa/vOMPYlf+kdb\nf15eAJ/8GrZ8YK/gr3vd9i7Znw7vzrJX/j3Phyv+bHuzAOxeAX+favtpAySOsFf8VzxrG3nBNsju\nXwcdOkGXsyA4DMoO2V4xoTphkVKgdwo+Y/P+wxSV13DJoM7eDsUzXploe6WYetudMTTa9nAxxvbQ\nObDJ1snnbLRX911WQVi07Rlz4S9sw27j7psp4+COr2wvmsFTocfI478zIg76TWxSFn/8ekqpk9Kk\n4LD/ZOQDcH6fOC9H0gp2fG6v0jsPsv3fSw/C5CeO7e3S6wK7vKbCdpvsM+Fo3/KJv7cNookj7F3D\nlg9sFZAI3Lig5e/terZ9KaUcp0nBYd/sPET/LpF0jmoD4xqVHLT196NuP1p9c0R1mX1QavITtrvm\nmrn2ac+SHFvv3+t8GHsfTPxdy/tPGXv0fXgnGHGbIz+GUur0eXuU1HatsqaOb/cUcEGfNlKV8eVj\n8O3Ltr/+yufsWDP1dfDF7+GLx2ydf0ikHZPlrlUw6XHY/ol9SCo4XBtqlWoH9E7BQev2FlJZU8+Y\nvj6eFKrLXVU8BobNsD2DPnX1rR90JaTNsz10ksfaOwKw/e9H/9SWdeptq4WUUm2e3ik46NvdhYjA\nqN4+NgJqdbkd6AzsE7h/TLS9da56Dq76K9z8lu3KeWCTHdbghvl2GIaLHjp+X12HaEJQqh3ROwUH\nbcwqom9CJNFhDo/t4o7CTIjtZd8vvssOwtZ9uK36SR57tHvnkSqgQVfYF9hBzO7d4PmYlVIep3cK\nDjHGsCGriHOSOno7EFj5PPzlXNix1JaNugMmPGgHURswBWYu8m6MSimfoXcKDskuquBQaTXDesR4\nL4jaKvjXf8H6+XZEz56jbXmv8+1rnGvMd20gVkq5aFJwyIZ9hwEY2sNLdwqlufDWTDte/Lhfwfhf\nHz+mvyYDpVQTWn3kkA1ZRYQEBjCwa7TzX5aVBv9+yo4K+tZM+0Txjs/s+PzXvgYX/bblSV6UUqoR\nvVNwyIZ9RQzqHk1IkMMn4+IcWHCt7TL61ZN2Qpbxv7YTsaSMs9MNKqWUm/Ty0QGVNXVszDrMsCQH\n2xPq6+0rsguMvgum/K8d/O2SR+x0h6AJQSl1yvROwQGrduVTUVPHDwYknHzl02GM7VZ69rXQ9xL4\nwa9s+cifaDuBUuqM6J2CA77cmktYcIBzw1ukL4ANb0LetmPLNSEopc6QJoVWZozhiy25jOmbQFiw\nA9Nu5m2DJffb9oLz7mz9/Sul/JomhVa29UAJ2UUVzsyfUFMB78yyg89Nm2vnBlZKqVakbQqt7P30\nbETgooEOJIX18+00k9Pfs1NOKqVUK9Ok0IpySyp5/Zs9XDm0O52jHZg/YeRP7FPJOuGMUsohWn3U\niv76ZQa1dYb/d0n/k698KoyBkgO2IVkTglLKQZoUWsm+gnLeXLOX60f2IDk+ovV2vH89zB0Pfx4K\n+9a03n6VUqoZWn3USp5ZuoMAEX5+Ub8z31lVKZg6O+n9ojvt08qXPQWJqWe+b6WUOgG9U2gFOw6W\nsGh9FrdekEzXmFZoS5h/NTzRB3Ytg7ytdnKb4TN0/CKllOP0TqEVvJ+eTYAId/6gz5nvrKII9q22\n79e8DKExMOSaM9+vUkq5QZNCK8gqrKB7xw50igg5853t/NL+O+sTqCqB/pfqdJdKKY/RpNAK9hdV\n0O1Mq41yt4Kph+2fQodYSBoJgfrrUUp5lp51WsH+okrOS+l0+juorYaXxtlnEKK62VnSNCEopbxA\nzzxnqLaungPFlXTv2OH0d/L9+1BXBaN/CgMm2+cSlFLKCxztziIik0Rkm4hkiMicFtYZLyLpIrJZ\nRL5yMh4n5JZUUVdvTj8pGAMr/wrxA6DfpbZMRztVSnmJY3cKIhIIPAdMBLKAb0XkA2PM943W6Qg8\nD0wyxuwVEQcGDHJWzuEKALp3PM02hbR5dtrMK/6sXU6VUl7n5FloFJBhjNlljKkGFgJXNVnnZuCf\nxpi9AMaYXAfjcUR2USUAiad7p1C4G/pOhKE3tWJUSil1epxsU0gE9jX6nAWc12Sd/kCwiCwHooA/\nG2P+3nRHIjIbmA3Qs6dvTDG5NrOAlTvzCQq0ebXb6SaFiY/ZeZUDg1sxOqWUOj3ebmgOAkYAFwMd\ngJUissoYs73xSsaYucBcgNTUVJ9ohZ2/ai+L1mcztl880WFBRIa6eSh3/xsqiyBjKQyeCn0maEJQ\nSvkMt85kIvJP4FXgY2NMvZv7zgZ6NPqc5CprLAvIN8aUAWUisgIYCmzHx209UALAv3ccYmDXKPc2\nKjkIC66DWtsOQefBNikopZSPcLdN4Xls/f8OEXlcRAa4sc23QD8RSRGREOBG4IMm6ywGxohIkIiE\nY6uXtrgZk9fU1NWzM7e04bPb7Qlf/x/UVcO1r8EVz0Lqjx2KUCmlTo9bdwrGmKXAUhGJAW5yvd8H\nvAzMN8bUNLNNrYjcDXwKBALzjDGbReRO1/IXjTFbROQTYCNQD7xijPmuVX4yB+05VEZ1XT2jUjqx\nZneB+91R4/rABXfDkKudDVAppU6T220KIhIHzABmAuuBBcAY4FZgfHPbGGOWAEualL3Y5POTwJOn\nErS3Hak6+sUl/bjjjbUM7OZm9dGo2x2MSimlzpy7bQqLgAHAG8AVxpgc16K3RCTNqeB81bYDJQQG\nCCN6xfLNnIuICDnJYSwvgB2f2dFOtVFZKeXD3L1TeNYYs6y5BcYYv5v5ZdvBElLiIwgNCiQ0KPDk\nG6yZC8v/CN2GQeeBzgeolFKnyd2G5sGup48BEJFYEbnLoZh83rYDJQxwt8dRdRmsfhEGTNGEoJTy\nee4mhduNMUVHPhhjCgG/rCBP31fE3oJyBnZxMyms+7udTnPM/3M2MKWUagXuJoVAkaOjtLnGNWqF\nGWXalvR9RUx/eRU9O4VzXWqPk29QWw3f/BV6XQg9RjkfoFJKnSF32xQ+wTYqv+T6fIerzK/MX5VJ\nYIDw9h3nuzcXc3E2hEbpXYJSqs1wNyk8gE0EP3V9/hx4xZGIfFhuSRXJ8RHuJQSATinw0290KGyl\nVJvh7sNr9cALrpffOlRS5f60mzUVdnrNkAhng1JKqVbkVpuCiPQTkXdF5HsR2XXk5XRwvuZQaRXx\nkaHurbx5ETzZF/J3OhuUUkq1Incbml/D3iXUAhOAvwPznQrKF9XXG/LLqomPcrN9ffP7EB4HnXo7\nG5hSSrUid5NCB2PMF4AYYzKNMY8ClzkXlu8pLK+mrt6Q4M6dQmke7PwCzpqq7QlKqTbF3YbmKhEJ\nwI6Sejd2COxI58LyPYdKqwGIj3IjKaTPtxPnDL/F4aiUUqp1uXuncC8QDvwcOynODOxAeH4jr6QK\n4ORtCsbA2tftswkJ/T0QmVJKtZ6T3im4HlS7wRhzH1AKzHI8Kh90qNQmhYST3SmIwI0LoKbSA1Ep\npVTrOmlSMMbUicgYTwTjy9y+UwDocpbD0SillDPcrT5aLyIfiMhMEbn6yMvRyHzModIqQoICiA47\nQR4tzYP3bodDGZ4LTCmlWpG7SSEMyAcuAq5wvS53KihflFdaRUJkKHKi3kTp82HT2/ahNaWUaoPc\nfaLZL9sRGssrqSI+8gTPKJQXwLfztIFZKdWmuTvz2muAaVpujPlRq0fkow6VVpPYsYUhLopz4O9X\nQulBmPqcZwNTSqlW5O5zCh81eh8GTAP2t344vutQaRVDk2KaX7j6BSjYBbcshmS/b5NXSrVh7lYf\nvdf4s4i8CXztSEQ+qK7ekH+icY/G/QqG3gSdB3k2MKWUamXuNjQ31Q/o3JqB+LItOcXUG1puUwiN\n1ISglGoX3B0ltUREio+8gA+xcyy0e1tyirll3hoSokK5ZHCX41eoqYTPH4GcjZ4PTimlWpm71Udu\nTkjc/vzlyx0YY3j7jgtIig0/foX8DPjPM9D1bOh2jucDVEqpVuTuncI0EYlp9LmjiEx1LizfsTO3\njBG9YkmJb2GynEPb7b/x2g1VKdX2udum8Igx5vCRD8aYIuARZ0LyHfX1ht35ZS0nBHAlBYH4fh6L\nSymlnOJuUmhuPXe7s7ZZ+w9XUF1bT++EE4wSfmg7dOwJwR08F5hSSjnE3aSQJiJPi0gf1+tpYK2T\ngfmCXXllACe+Uyjer1VHSql2w92r/XuAh4C3sE82fw78zKmgfMXuQzYp9G4uKVSX2zGOZn0MNRUe\njkwppZzhbu+jMmCOw7H4nN2HyogMDWp+DoVl/w0xPWD0nRDSTK8kpZRqg9ztffS5iHRs9DlWRD51\nLizfsDNU1TNAAAAXIUlEQVSvlJT4iOZHRt25DLZ/7PmglFLKQe62KcS7ehwBYIwpxA+eaN59qIWe\nR+UFkLtZxzlSSrU77iaFehHpeeSDiCTTzKip7UllTR3ZRRX0TmgmKexbbf/teYFng1JKKYe529D8\nW+BrEfkKEGAsMNuxqHzA5v3FGAN9OzfTHTXzPxAYAokjPB+YUko5yN2G5k9EJBWbCNYD7wPtusvN\nRxv3ExIUwLj+CccvDI+HIddAcAvzKyilVBvl7iQ7PwHuBZKAdGA0sBI7PeeJtpsE/BkIBF4xxjze\nwnojXfu70RjzrtvRO6Su3vDRxhwmDEggOiz4+BXG/MLzQSmllAe426ZwLzASyDTGTACGA0Un2kBE\nAoHngMnAYOAmERncwnp/Aj47hbgdtWpXPnklVVw1LPH4hYWZUF/n+aCUUsoD3E0KlcaYSgARCTXG\nbAUGnGSbUUCGMWaXMaYaWAhc1cx69wDvAbluxuK4jzbmEBkaxEUDm3SwMgYWXAtv3+KdwJRSymHu\nJoUs13MK7wOfi8hiIPMk2yQC+xrvw1XWQEQSsVN7vnCiHYnIbBFJE5G0vLw8N0M+fd/nFDO0Rwxh\nwYHHLtjzbzvW0cDLHI9BKaW8wa2kYIyZZowpMsY8ih3u4lWgNYbOfgZ4wBhTf5Lvn2uMSTXGpCYk\nNNPw28r25pfRK66ZrqjfvQeh0XDWNMdjUEopbzjlkU6NMV+5uWo20KPR5yRXWWOpwELXE8PxwBQR\nqTXGvH+qcbWWw+U1FJbXkBzXzNAVedugyxAdEVUp1W6d7hzN7vgW6CciKSISAtwIfNB4BWNMijEm\n2RiTDLwL3OXNhACQWWAHwWv2TuHQdp03QSnVrjk2J4IxplZE7gY+xXZJnWeM2Swid7qWv+jUd5+J\nPfnlACQ3TQrGwHV/gw6xng9KKaU8xNGJcowxS4AlTcqaTQbGmNucjMVdma7hsnt2alJ9JAIp47wQ\nkVJKeY6T1UdtUmZBOV2jw+gQ0qTnUc5G2PIh1NV4JzCllPIATQpNZOaX0au5RuaNb8G7PwbRQ6aU\nar/0DNfEnvzy49sTAA7tgLi+EBB4/DKllGonNCk0UlZVS15JFT2bu1PQnkdKKT+gSaGRzJZ6HtVW\nQVEmxPf3QlRKKeU5mhQaycgrBaBP5yZJYf96MPXQ7RwvRKWUUp7jaJfUtiYjt5QA4fgpOHuOhp+u\nhE69vROYUkp5iCaFRjJyS+gVF0FoUDONyV2OG/VbKaXaHa0+aiQjt5Q+CU2m39ywEN77CVSVeico\npZTyIE0KLrV19ew+VHb8nMxpr0HOBghpppuqUkq1M5oUXDILyqmpM/RrnBTyd8K+VTDsZjvMhVJK\ntXOaFFwycm310DF3ChvetE8wn3ODl6JSSinP0qTgciQp9DmSFOrrbXtC7wkQ3d2LkSmllOdo7yOX\njNxSuseEERnqOiTVpXaGtZQfeDcwpZTyIE0KLpuyDzOwW/TRgrBo+OFj3gtIKaW8QKuPgMKyajJy\nSxnRq9EEOrlb7PAWSinlRzQpAGszCwEYmdzJFtTXwas/hI8f8GJUSinleZoUgG8zCwgJDOCcpBhb\ncGATVBVDrwu8G5hSSnmYJgUgbU8hQxKjCQt2DW+R+Y39t9eF3gtKKaW8wO+TQmVNHZuyDh+tOgLY\nvQJikyEm0WtxKaWUN/h9UkjbU0h1XT2pR5JCdRnsWgb9LvVuYEop5QV+3yV1wepMOoYHM7ZfvC0I\n6gC3fADhcd4NTCmlvMCvk0LO4Qo++/4gPxmbcrQ9ISAAep7n3cCUUspL/Lr66B+r91JvDDPO62UL\n6mrg09/Cwe+9G5hSSnmJXyeFL7bkcn7vOHp0CrcFe1fCyr9CwS7vBqaUUl7it0nBGENmfhkDukYd\nLdzxGQSGQO/x3gpLKaW8ym+TQn5ZNWXVdfQ6cpcAsONz+2xCaGTLGyqlVDvmt0khM78MgF7xrhnV\nivZC3lboN9GLUSmllHf5bVLYc6gcgOQ4V1Io2GW7ofb7oRejUkop7/LbLqmZ+WUEBgiJHTvYgt7j\n4b4ddqY1pZTyU357BtyTX05ixw6EBAWAMbYwIFDnYlZK+TW/TQqZ+WX0inM1Muekw5+HQlaad4NS\nSikv89uksCe//GhS2PM1FO6BaB0ATynl3/wyKRSVV3O4ouZoI/Puf0NcX4ju5t3AlFLKyxxNCiIy\nSUS2iUiGiMxpZvl0EdkoIptE5BsRGepkPEdk5tueR73iIqCu1j7JnDzWE1+tlFI+zbGkICKBwHPA\nZGAwcJOIDG6y2m7gB8aYs4HHgLlOxdPYtgMlAPRJiICcDXaWteQxnvhqpZTyaU7eKYwCMowxu4wx\n1cBC4KrGKxhjvjHGFLo+rgKSHIynQXpWEdFhQbb6KCQCht4EfS7yxFcrpZRPczIpJAL7Gn3OcpW1\n5MfAx80tEJHZIpImIml5eXlnHNiGfUWck9SRgACBzgNh2osQ3unkGyqlVDvnEw3NIjIBmxQeaG65\nMWauMSbVGJOakJBwRt9VWVPHtgMlDO0RAwW74eDmo88pKKWUn3MyKWQDPRp9TnKVHUNEzgFeAa4y\nxuQ7GA8Am/cXU1tvGJrUEVa9AC9fBNWlTn+tUkq1CU4mhW+BfiKSIiIhwI3AB41XEJGewD+BmcaY\n7Q7G0mDDviIAhiVFw5YPoO8lEBp1kq2UUso/ODb2kTGmVkTuBj4FAoF5xpjNInKna/mLwMNAHPC8\n2OElao0xqU7FBLAxq4iu0WF0Lt0CJTkw8HInv04ppdoURwfEM8YsAZY0KXux0fufAD9xMoamNmUf\n5uykGNi6BCQQ+l/qya9XSimf5lejpFbX1rMnv5xJQ7pCxufQ6wLtdaSUD6qpqSErK4vKykpvh9Lm\nhIWFkZSURHBw8Glt71dJITO/jLp6Q7/OUTB+CZQe9HZISqlmZGVlERUVRXJyMqIjF7vNGEN+fj5Z\nWVmkpKSc1j58okuqp2Tk2l5GfTtH2ik34/p4OSKlVHMqKyuJi4vThHCKRIS4uLgzusPyq6Sww5UU\nBmx+Br591cvRKKVORBPC6TnT4+ZXSSEjt5ReHYMJTnsZDmzydjhKKeVz/C4pTIzJguoS6DPB2+Eo\npXxUUVERzz///GltO2XKFIqKilo5Is/xm6RQV2/YmVfKuIBNdh7mlHHeDkkp5aNOlBRqa2tPuO2S\nJUvo2LGjE2F5hN/0PsourKCqtp5Bleuh+3DoEOvtkJRSbvjdh5v5fn9xq+5zcPdoHrnirBaXz5kz\nh507dzJs2DAmTpzIZZddxkMPPURsbCxbt25l+/btTJ06lX379lFZWcm9997L7NmzAUhOTiYtLY3S\n0lImT57MmDFj+Oabb0hMTGTx4sV06NDhmO/68MMP+cMf/kB1dTVxcXEsWLCALl26UFpayj333ENa\nWhoiwiOPPMI111zDJ598wm9+8xvq6uqIj4/niy++aNVj4zdJISPPzqEQHBkHKed6ORqllC97/PHH\n+e6770hPTwdg+fLlrFu3ju+++66hq+e8efPo1KkTFRUVjBw5kmuuuYa4uLhj9rNjxw7efPNNXn75\nZa6//nree+89ZsyYccw6Y8aMYdWqVYgIr7zyCk888QRPPfUUjz32GDExMWzaZNs/CwsLycvL4/bb\nb2fFihWkpKRQUFDQ6j+73ySFzlFhzBzdC7l0IXQ4vYc6lFKed6Irek8aNWrUMX3/n332WRYtWgTA\nvn372LFjx3FJISUlhWHDhgEwYsQI9uzZc9x+s7KyuOGGG8jJyaG6urrhO5YuXcrChQsb1ouNjeXD\nDz9k3LhxDet06tT6D9/6TZvCkMQYHps6hBhNCEqp0xAREdHwfvny5SxdupSVK1eyYcMGhg8f3uyz\nAaGhoQ3vAwMDm22PuOeee7j77rvZtGkTL730ktef4vabpADAN3+F50ZDjT46r5RqWVRUFCUlJS0u\nP3z4MLGxsYSHh7N161ZWrVp12t91+PBhEhPt/GOvv/56Q/nEiRN57rnnGj4XFhYyevRoVqxYwe7d\nuwEcqT7yr6RwYKOdjzk4zNuRKKV8WFxcHBdeeCFDhgzh/vvvP275pEmTqK2tZdCgQcyZM4fRo0ef\n9nc9+uijXHfddYwYMYL4+PiG8gcffJDCwkKGDBnC0KFDWbZsGQkJCcydO5err76aoUOHcsMNN5z2\n97ZETBubdSw1NdWkpaWd3sYvjoHILjDjvdYNSinVqrZs2cKgQYO8HUab1dzxE5G17kxN4D93CnW1\nkLcdOusfmlJKtcR/kkLhbqirgs6DvR2JUkr5LP9JCgiccwN012cUlFKqJX7znALxfeHqud6OQiml\nfJof3SkopZQ6GU0KSimlGmhSUEqpJs5k6GyAZ555hvLy8laMyHM0KSilVBP+nBT8p6FZKdV2vXbZ\n8WVnTYVRt0N1OSy47vjlw26G4dOhLB/evuXYZbP+dcKvazp09pNPPsmTTz7J22+/TVVVFdOmTeN3\nv/sdZWVlXH/99WRlZVFXV8dDDz3EwYMH2b9/PxMmTCA+Pp5ly5Yds+/f//73fPjhh1RUVHDBBRfw\n0ksvISJkZGRw5513kpeXR2BgIO+88w59+vThT3/6E/PnzycgIIDJkyfz+OOPn+rROyWaFJRSqomm\nQ2d/9tln7NixgzVr1mCM4corr2TFihXk5eXRvXt3/vUvm2QOHz5MTEwMTz/9NMuWLTtm2Ioj7r77\nbh5++GEAZs6cyUcffcQVV1zB9OnTmTNnDtOmTaOyspL6+no+/vhjFi9ezOrVqwkPD3dkrKOmNCko\npXzfia7sQ8JPvDwi7qR3Bifz2Wef8dlnnzF8+HAASktL2bFjB2PHjuWXv/wlDzzwAJdffjljx449\n6b6WLVvGE088QXl5OQUFBZx11lmMHz+e7Oxspk2bBkBYmB2fbenSpcyaNYvw8HDAmaGym9KkoJRS\nJ2GM4de//jV33HHHccvWrVvHkiVLePDBB7n44osb7gKaU1lZyV133UVaWho9evTg0Ucf9fpQ2U1p\nQ7NSSjXRdOjsSy+9lHnz5lFaWgpAdnY2ubm57N+/n/DwcGbMmMH999/PunXrmt3+iCMJID4+ntLS\nUt59992G9ZOSknj//fcBqKqqory8nIkTJ/Laa681NFpr9ZFSSnlB46GzJ0+ezJNPPsmWLVs4//zz\nAYiMjGT+/PlkZGRw//33ExAQQHBwMC+88AIAs2fPZtKkSXTv3v2YhuaOHTty++23M2TIELp27crI\nkSMblr3xxhvccccdPPzwwwQHB/POO+8wadIk0tPTSU1NJSQkhClTpvA///M/jv7s/jV0tlKqTdCh\ns8+MDp2tlFKqVWhSUEop1UCTglLKJ7W1qm1fcabHTZOCUsrnhIWFkZ+fr4nhFBljyM/Pb3jO4XRo\n7yOllM9JSkoiKyuLvLw8b4fS5oSFhZGUlHTa22tSUEr5nODgYFJSUrwdhl9ytPpIRCaJyDYRyRCR\nOc0sFxF51rV8o4joXJlKKeVFjiUFEQkEngMmA4OBm0RkcJPVJgP9XK/ZwAtOxaOUUurknLxTGAVk\nGGN2GWOqgYXAVU3WuQr4u7FWAR1FpJuDMSmllDoBJ9sUEoF9jT5nAee5sU4ikNN4JRGZjb2TACgV\nkW2nGVM8cOg0t3WSr8YFvhubxnVqNK5T0x7j6uXOSm2iodkYMxeYe6b7EZE0dx7z9jRfjQt8NzaN\n69RoXKfGn+NysvooG+jR6HOSq+xU11FKKeUhTiaFb4F+IpIiIiHAjcAHTdb5ALjF1QtpNHDYGJPT\ndEdKKaU8w7HqI2NMrYjcDXwKBALzjDGbReRO1/IXgSXAFCADKAdmORWPyxlXQTnEV+MC341N4zo1\nGtep8du42tzQ2UoppZyjYx8ppZRqoElBKaVUA79JCicbcsODcfQQkWUi8r2IbBaRe13lj4pItoik\nu15TvBDbHhHZ5Pr+NFdZJxH5XER2uP6N9XBMAxodk3QRKRaRX3jjeInIPBHJFZHvGpW1eHxE5Neu\nv7dtInKph+N6UkS2uoaPWSQiHV3lySJS0ei4vejhuFr8vXn5eL3VKKY9IpLuKvfk8Wrp3ODZvzFj\nTLt/YRu6dwK9gRBgAzDYS7F0A851vY8CtmOHAXkUuM/Lx2kPEN+k7Algjuv9HOBPXv49HsA+hOPx\n4wWMA84FvjvZ8XH9TjcAoUCK6+8v0INx/RAIcr3/U6O4khuv54Xj1ezvzdvHq8nyp4CHvXC8Wjo3\nePRvzF/uFNwZcsMjjDE5xph1rvclwBbsU9y+6irgddf714GpXozlYmCnMSbTG19ujFkBFDQpbun4\nXAUsNMZUGWN2Y3vYjfJUXMaYz4wxta6Pq7DPAHlUC8erJV49XkeIiADXA2868d0ncoJzg0f/xvwl\nKbQ0nIZXiUgyMBxY7Sq6x3W7P8/T1TQuBlgqImtdQ4sAdDFHnx05AHTxQlxH3Mix/1m9fbyg5ePj\nS39zPwI+bvQ5xVUV8pWIjPVCPM393nzleI0FDhpjdjQq8/jxanJu8OjfmL8kBZ8jIpHAe8AvjDHF\n2BFiewPDsGM/PeWFsMYYY4ZhR6/9mYiMa7zQ2HtWr/RhFvsA5JXAO64iXzhex/Dm8WmJiPwWqAUW\nuIpygJ6u3/N/Af8QkWgPhuRzv7cmbuLYCw+PH69mzg0NPPE35i9JwaeG0xCRYOwvfYEx5p8AxpiD\nxpg6Y0w98DIO3TqfiDEm2/VvLrDIFcNBcY1c6/o319NxuUwG1hljDrpi9Prxcmnp+Hj9b05EbgMu\nB6a7Tia4qhryXe/XYuuh+3sqphP83nzheAUBVwNvHSnz9PFq7tyAh//G/CUpuDPkhke46ixfBbYY\nY55uVN54yPBpwHdNt3U4rggRiTryHttQ+R32ON3qWu1WYLEn42rkmCs4bx+vRlo6Ph8AN4pIqIik\nYOcMWeOpoERkEvAr4EpjTHmj8gSxc50gIr1dce3yYFwt/d68erxcLgG2GmOyjhR48ni1dG7A039j\nnmhV94UXdjiN7dhM/1svxjEGe/u3EUh3vaYAbwCbXOUfAN08HFdvbE+GDcDmI8cIiAO+AHYAS4FO\nXjhmEUA+ENOozOPHC5uUcoAabP3tj090fIDfuv7etgGTPRxXBra++cjf2Iuuda9x/X7TgXXAFR6O\nq8XfmzePl6v8b8CdTdb15PFq6dzg0b8xHeZCKaVUA3+pPlJKKeUGTQpKKaUaaFJQSinVQJOCUkqp\nBpoUlFJKNdCkoJTDRGS8iHzk7TiUcocmBaWUUg00KSjlIiIzRGSNa/Czl0QkUERKReT/XOPbfyEi\nCa51h4nIKjk6X0Gsq7yviCwVkQ0isk5E+rh2Hyki74qd42CB6+lVRORx1/j5G0Xkf730oyvVQJOC\nUoCIDAJuAC40dvCzOmA69mnqNGPMWcBXwCOuTf4OPGCMOQf7hO6R8gXAc8aYocAF2CdnwY54+Qvs\nGPi9gQtFJA471MNZrv38wdmfUqmT06SglHUxMAL41jXr1sXYk3c9RwdImw+MEZEYoKMx5itX+evA\nONfYUYnGmEUAxphKc3TcoTXGmCxjB4JLx07echioBF4VkauBhjGKlPIWTQpKWQK8bowZ5noNMMY8\n2sx6pzsuTFWj93XYWdFqsaOEvosdzfST09y3Uq1Gk4JS1hfAtSLSGRrmxe2F/T9yrWudm4GvjTGH\ngcJGE67MBL4ydrasLBGZ6tpHqIiEt/SFrnHzY4wxS4D/Bwx14gdT6lQEeTsApXyBMeZ7EXkQ+ExE\nArAjaP4MKANGuZblYtsdwA5h/KLrpL8LmOUqnwm8JCK/d+3juhN8bRSwWETCsHcq/9XKP5ZSp0xH\nSVXqBESk1BgT6e04lPIUrT5SSinVQO8UlFJKNdA7BaWUUg00KSillGqgSUEppVQDTQpKKaUaaFJQ\nSinV4P8DJA2z2KKyuUwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1621de99320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# グラフの描画\n",
    "\n",
    "import matplotlib.pylab as plt\n",
    "import numpy as np\n",
    "\n",
    "markers = {'train': 'o', 'test': 's'}\n",
    "x = np.arange(len(train_acc_list))\n",
    "plt.plot(x, train_acc_list, label='train acc')\n",
    "plt.plot(x, test_acc_list, label='test acc', linestyle='--')\n",
    "plt.xlabel(\"epochs\")\n",
    "plt.ylabel(\"accuracy\")\n",
    "plt.ylim(0, 1.0)\n",
    "plt.legend(loc='lower right')\n",
    "plt.title('Fig. 6-20')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 過学習 vs 汎化\n",
    "- train_acc_list、test_acc_list \n",
    "    - エポック単位\n",
    "    - 訓練データを見終わった単位の認識精度が格納\n",
    "-（train_acc_list、test_acc_list）をグラフとして描画\n",
    "    - 結果は図6-20  \n",
    "    - 認識精度(train)は、100 エポック以上でほぼ100% \n",
    "    - 認識精度(test)は低い\n",
    "- 認識精度のtrain-test間の大きな隔たり\n",
    "    - 訓練データだけに適応しすぎてしまった結果\n",
    "    - <b>汎化性能</b>が低い＝汎用データ（test）へ適応できず\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "### 6.4.2 Weight decay\n",
    "\n",
    "- <b>Weight decay（荷重減衰）</b>\n",
    "   - 過学習抑制でよく用いられる手法\n",
    "   - 大きな重みを持つことに対してペナルティを課す\n",
    "   - 過学習は、重みパラメータが大きな値を取ることが原因の場合が多い\n",
    "   - 損失関数の値を小さくする\n",
    "       - たとえば、重みの2 乗ノルム（L2 ノルム）を損失関数に加算\n",
    "       - 重みが大きくなることを抑制\n",
    "   - Weight decay　$\\frac{1}{2}\\lambda W^{2}$\n",
    "       - W = 重み\n",
    "       - $\\lambda$ = 正則化の強さをコントロールするハイパーパラメータ\n",
    "       - $\\lambda$を大きく設定すれば、重みを取ることに対して\n",
    "           強いペナルティを課す\n",
    "       - $\\frac{1}{2}$ = $\\frac{1}{2}\\lambda W^{2}$の微分の結果を$\\lambda W$ にするための調整用の定数\n",
    "       \n",
    "   - 全ての重みに$\\frac{1}{2}\\lambda W^{2}$を損失関数に加算\n",
    "   - 重みの勾配計算\n",
    "       - これまでの誤差逆伝播法による結果に、正則化項の微分$\\lambda W$を加算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 正則化項\n",
    "\n",
    "- L2 ノルムは、各要素の2 乗和に対応\n",
    "- 重 み : $W = (w_{1},w_{2}, ·  ,w_{n})$ \n",
    "- L2ノルム = $sqrt(w_{1}^{2} + w_{2}^{2} +...+w_{n}^{2})$\n",
    "- L2 ノルムの他\n",
    "    - L1ノルム = $|w_{1}|+|w_{2}|+...+|w_{n}|$\n",
    "    - L∞ノルム = 各要素の絶対値の中で最大のもの\n",
    "\n",
    "- ここでは一般的によく用いられるL2ノルムだけを実装(図6-21)\n",
    "- common/multi_layer_net.py\n",
    "- ch06/overfit_weight_decay.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:0, train acc:0.12, test acc:0.1258\n",
      "epoch:1, train acc:0.14, test acc:0.1411\n",
      "epoch:2, train acc:0.14666666666666667, test acc:0.1431\n",
      "epoch:3, train acc:0.15, test acc:0.1567\n",
      "epoch:4, train acc:0.16666666666666666, test acc:0.1655\n",
      "epoch:5, train acc:0.18, test acc:0.1727\n",
      "epoch:6, train acc:0.21, test acc:0.1837\n",
      "epoch:7, train acc:0.21333333333333335, test acc:0.1865\n",
      "epoch:8, train acc:0.24666666666666667, test acc:0.195\n",
      "epoch:9, train acc:0.25, test acc:0.2037\n",
      "epoch:10, train acc:0.28, test acc:0.2125\n",
      "epoch:11, train acc:0.31333333333333335, test acc:0.2301\n",
      "epoch:12, train acc:0.3566666666666667, test acc:0.2436\n",
      "epoch:13, train acc:0.3933333333333333, test acc:0.2705\n",
      "epoch:14, train acc:0.4066666666666667, test acc:0.2878\n",
      "epoch:15, train acc:0.4266666666666667, test acc:0.3085\n",
      "epoch:16, train acc:0.4533333333333333, test acc:0.3332\n",
      "epoch:17, train acc:0.4766666666666667, test acc:0.3554\n",
      "epoch:18, train acc:0.5133333333333333, test acc:0.3768\n",
      "epoch:19, train acc:0.53, test acc:0.392\n",
      "epoch:20, train acc:0.5433333333333333, test acc:0.4005\n",
      "epoch:21, train acc:0.5466666666666666, test acc:0.4125\n",
      "epoch:22, train acc:0.5633333333333334, test acc:0.4287\n",
      "epoch:23, train acc:0.5766666666666667, test acc:0.4417\n",
      "epoch:24, train acc:0.6, test acc:0.4573\n",
      "epoch:25, train acc:0.59, test acc:0.4628\n",
      "epoch:26, train acc:0.5933333333333334, test acc:0.4608\n",
      "epoch:27, train acc:0.6166666666666667, test acc:0.4734\n",
      "epoch:28, train acc:0.62, test acc:0.4911\n",
      "epoch:29, train acc:0.6233333333333333, test acc:0.4898\n",
      "epoch:30, train acc:0.6433333333333333, test acc:0.5009\n",
      "epoch:31, train acc:0.6466666666666666, test acc:0.5065\n",
      "epoch:32, train acc:0.6533333333333333, test acc:0.507\n",
      "epoch:33, train acc:0.65, test acc:0.5254\n",
      "epoch:34, train acc:0.6533333333333333, test acc:0.5225\n",
      "epoch:35, train acc:0.6533333333333333, test acc:0.5333\n",
      "epoch:36, train acc:0.6566666666666666, test acc:0.5286\n",
      "epoch:37, train acc:0.6466666666666666, test acc:0.5363\n",
      "epoch:38, train acc:0.6566666666666666, test acc:0.548\n",
      "epoch:39, train acc:0.6633333333333333, test acc:0.551\n",
      "epoch:40, train acc:0.6666666666666666, test acc:0.5484\n",
      "epoch:41, train acc:0.6933333333333334, test acc:0.5513\n",
      "epoch:42, train acc:0.6833333333333333, test acc:0.556\n",
      "epoch:43, train acc:0.6766666666666666, test acc:0.5627\n",
      "epoch:44, train acc:0.6933333333333334, test acc:0.5636\n",
      "epoch:45, train acc:0.6866666666666666, test acc:0.564\n",
      "epoch:46, train acc:0.7233333333333334, test acc:0.5746\n",
      "epoch:47, train acc:0.7266666666666667, test acc:0.5891\n",
      "epoch:48, train acc:0.7433333333333333, test acc:0.5985\n",
      "epoch:49, train acc:0.7633333333333333, test acc:0.6014\n",
      "epoch:50, train acc:0.75, test acc:0.6013\n",
      "epoch:51, train acc:0.76, test acc:0.6052\n",
      "epoch:52, train acc:0.76, test acc:0.6073\n",
      "epoch:53, train acc:0.75, test acc:0.6077\n",
      "epoch:54, train acc:0.7633333333333333, test acc:0.6151\n",
      "epoch:55, train acc:0.75, test acc:0.6072\n",
      "epoch:56, train acc:0.7233333333333334, test acc:0.5912\n",
      "epoch:57, train acc:0.7766666666666666, test acc:0.6186\n",
      "epoch:58, train acc:0.7933333333333333, test acc:0.6247\n",
      "epoch:59, train acc:0.8033333333333333, test acc:0.6251\n",
      "epoch:60, train acc:0.81, test acc:0.6328\n",
      "epoch:61, train acc:0.8033333333333333, test acc:0.6315\n",
      "epoch:62, train acc:0.8033333333333333, test acc:0.6398\n",
      "epoch:63, train acc:0.8, test acc:0.6348\n",
      "epoch:64, train acc:0.8166666666666667, test acc:0.6475\n",
      "epoch:65, train acc:0.82, test acc:0.6476\n",
      "epoch:66, train acc:0.8133333333333334, test acc:0.65\n",
      "epoch:67, train acc:0.8133333333333334, test acc:0.6523\n",
      "epoch:68, train acc:0.8266666666666667, test acc:0.6489\n",
      "epoch:69, train acc:0.8233333333333334, test acc:0.6603\n",
      "epoch:70, train acc:0.8266666666666667, test acc:0.6612\n",
      "epoch:71, train acc:0.84, test acc:0.668\n",
      "epoch:72, train acc:0.83, test acc:0.6601\n",
      "epoch:73, train acc:0.82, test acc:0.6615\n",
      "epoch:74, train acc:0.8233333333333334, test acc:0.6644\n",
      "epoch:75, train acc:0.81, test acc:0.6674\n",
      "epoch:76, train acc:0.8166666666666667, test acc:0.6646\n",
      "epoch:77, train acc:0.8133333333333334, test acc:0.6653\n",
      "epoch:78, train acc:0.82, test acc:0.6693\n",
      "epoch:79, train acc:0.8266666666666667, test acc:0.6706\n",
      "epoch:80, train acc:0.8533333333333334, test acc:0.6835\n",
      "epoch:81, train acc:0.8433333333333334, test acc:0.6779\n",
      "epoch:82, train acc:0.8466666666666667, test acc:0.6833\n",
      "epoch:83, train acc:0.8333333333333334, test acc:0.6735\n",
      "epoch:84, train acc:0.84, test acc:0.6829\n",
      "epoch:85, train acc:0.85, test acc:0.6872\n",
      "epoch:86, train acc:0.8433333333333334, test acc:0.6847\n",
      "epoch:87, train acc:0.84, test acc:0.6863\n",
      "epoch:88, train acc:0.85, test acc:0.6885\n",
      "epoch:89, train acc:0.85, test acc:0.6896\n",
      "epoch:90, train acc:0.8366666666666667, test acc:0.6816\n",
      "epoch:91, train acc:0.8533333333333334, test acc:0.6929\n",
      "epoch:92, train acc:0.8266666666666667, test acc:0.6797\n",
      "epoch:93, train acc:0.8533333333333334, test acc:0.6946\n",
      "epoch:94, train acc:0.8566666666666667, test acc:0.6943\n",
      "epoch:95, train acc:0.8633333333333333, test acc:0.7008\n",
      "epoch:96, train acc:0.86, test acc:0.6997\n",
      "epoch:97, train acc:0.8566666666666667, test acc:0.6996\n",
      "epoch:98, train acc:0.8533333333333334, test acc:0.6989\n",
      "epoch:99, train acc:0.8766666666666667, test acc:0.709\n",
      "epoch:100, train acc:0.85, test acc:0.7009\n",
      "epoch:101, train acc:0.8533333333333334, test acc:0.7073\n",
      "epoch:102, train acc:0.8533333333333334, test acc:0.7068\n",
      "epoch:103, train acc:0.8733333333333333, test acc:0.7133\n",
      "epoch:104, train acc:0.86, test acc:0.6964\n",
      "epoch:105, train acc:0.8633333333333333, test acc:0.7067\n",
      "epoch:106, train acc:0.8666666666666667, test acc:0.7081\n",
      "epoch:107, train acc:0.8466666666666667, test acc:0.6994\n",
      "epoch:108, train acc:0.8633333333333333, test acc:0.7084\n",
      "epoch:109, train acc:0.8766666666666667, test acc:0.7183\n",
      "epoch:110, train acc:0.8666666666666667, test acc:0.717\n",
      "epoch:111, train acc:0.8533333333333334, test acc:0.7096\n",
      "epoch:112, train acc:0.8633333333333333, test acc:0.7157\n",
      "epoch:113, train acc:0.86, test acc:0.7188\n",
      "epoch:114, train acc:0.8666666666666667, test acc:0.7189\n",
      "epoch:115, train acc:0.8733333333333333, test acc:0.7165\n",
      "epoch:116, train acc:0.8733333333333333, test acc:0.7228\n",
      "epoch:117, train acc:0.8666666666666667, test acc:0.7258\n",
      "epoch:118, train acc:0.8633333333333333, test acc:0.7114\n",
      "epoch:119, train acc:0.86, test acc:0.7138\n",
      "epoch:120, train acc:0.8666666666666667, test acc:0.7218\n",
      "epoch:121, train acc:0.87, test acc:0.7099\n",
      "epoch:122, train acc:0.87, test acc:0.7211\n",
      "epoch:123, train acc:0.8666666666666667, test acc:0.7237\n",
      "epoch:124, train acc:0.8833333333333333, test acc:0.7277\n",
      "epoch:125, train acc:0.87, test acc:0.7237\n",
      "epoch:126, train acc:0.8833333333333333, test acc:0.7184\n",
      "epoch:127, train acc:0.8666666666666667, test acc:0.7218\n",
      "epoch:128, train acc:0.8666666666666667, test acc:0.7197\n",
      "epoch:129, train acc:0.85, test acc:0.7093\n",
      "epoch:130, train acc:0.8733333333333333, test acc:0.7295\n",
      "epoch:131, train acc:0.87, test acc:0.7278\n",
      "epoch:132, train acc:0.8733333333333333, test acc:0.7196\n",
      "epoch:133, train acc:0.8666666666666667, test acc:0.7165\n",
      "epoch:134, train acc:0.8733333333333333, test acc:0.728\n",
      "epoch:135, train acc:0.87, test acc:0.7265\n",
      "epoch:136, train acc:0.8733333333333333, test acc:0.7292\n",
      "epoch:137, train acc:0.87, test acc:0.7234\n",
      "epoch:138, train acc:0.87, test acc:0.7219\n",
      "epoch:139, train acc:0.8766666666666667, test acc:0.72\n",
      "epoch:140, train acc:0.8866666666666667, test acc:0.7298\n",
      "epoch:141, train acc:0.8766666666666667, test acc:0.7292\n",
      "epoch:142, train acc:0.8766666666666667, test acc:0.7319\n",
      "epoch:143, train acc:0.88, test acc:0.7283\n",
      "epoch:144, train acc:0.8733333333333333, test acc:0.7217\n",
      "epoch:145, train acc:0.8766666666666667, test acc:0.7269\n",
      "epoch:146, train acc:0.8966666666666666, test acc:0.7303\n",
      "epoch:147, train acc:0.8966666666666666, test acc:0.7296\n",
      "epoch:148, train acc:0.89, test acc:0.7263\n",
      "epoch:149, train acc:0.8866666666666667, test acc:0.728\n",
      "epoch:150, train acc:0.8966666666666666, test acc:0.7346\n",
      "epoch:151, train acc:0.8966666666666666, test acc:0.7405\n",
      "epoch:152, train acc:0.8833333333333333, test acc:0.7302\n",
      "epoch:153, train acc:0.87, test acc:0.7194\n",
      "epoch:154, train acc:0.8933333333333333, test acc:0.7277\n",
      "epoch:155, train acc:0.8866666666666667, test acc:0.7284\n",
      "epoch:156, train acc:0.8733333333333333, test acc:0.7153\n",
      "epoch:157, train acc:0.8866666666666667, test acc:0.7241\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:158, train acc:0.8733333333333333, test acc:0.7212\n",
      "epoch:159, train acc:0.8766666666666667, test acc:0.725\n",
      "epoch:160, train acc:0.88, test acc:0.7262\n",
      "epoch:161, train acc:0.8833333333333333, test acc:0.7272\n",
      "epoch:162, train acc:0.8933333333333333, test acc:0.7319\n",
      "epoch:163, train acc:0.8833333333333333, test acc:0.7322\n",
      "epoch:164, train acc:0.8833333333333333, test acc:0.7254\n",
      "epoch:165, train acc:0.8966666666666666, test acc:0.7317\n",
      "epoch:166, train acc:0.8833333333333333, test acc:0.7231\n",
      "epoch:167, train acc:0.8866666666666667, test acc:0.7303\n",
      "epoch:168, train acc:0.88, test acc:0.7218\n",
      "epoch:169, train acc:0.8833333333333333, test acc:0.73\n",
      "epoch:170, train acc:0.8966666666666666, test acc:0.7419\n",
      "epoch:171, train acc:0.8966666666666666, test acc:0.74\n",
      "epoch:172, train acc:0.88, test acc:0.7344\n",
      "epoch:173, train acc:0.8933333333333333, test acc:0.728\n",
      "epoch:174, train acc:0.8866666666666667, test acc:0.7321\n",
      "epoch:175, train acc:0.8933333333333333, test acc:0.7345\n",
      "epoch:176, train acc:0.8966666666666666, test acc:0.7371\n",
      "epoch:177, train acc:0.89, test acc:0.7343\n",
      "epoch:178, train acc:0.8866666666666667, test acc:0.7353\n",
      "epoch:179, train acc:0.8866666666666667, test acc:0.73\n",
      "epoch:180, train acc:0.89, test acc:0.7289\n",
      "epoch:181, train acc:0.8933333333333333, test acc:0.7333\n",
      "epoch:182, train acc:0.8933333333333333, test acc:0.7357\n",
      "epoch:183, train acc:0.9066666666666666, test acc:0.7339\n",
      "epoch:184, train acc:0.8833333333333333, test acc:0.726\n",
      "epoch:185, train acc:0.89, test acc:0.7378\n",
      "epoch:186, train acc:0.8866666666666667, test acc:0.7261\n",
      "epoch:187, train acc:0.89, test acc:0.737\n",
      "epoch:188, train acc:0.8866666666666667, test acc:0.7332\n",
      "epoch:189, train acc:0.8866666666666667, test acc:0.7349\n",
      "epoch:190, train acc:0.89, test acc:0.7369\n",
      "epoch:191, train acc:0.88, test acc:0.7314\n",
      "epoch:192, train acc:0.8833333333333333, test acc:0.7258\n",
      "epoch:193, train acc:0.88, test acc:0.7244\n",
      "epoch:194, train acc:0.8866666666666667, test acc:0.7296\n",
      "epoch:195, train acc:0.89, test acc:0.7273\n",
      "epoch:196, train acc:0.8866666666666667, test acc:0.7347\n",
      "epoch:197, train acc:0.8933333333333333, test acc:0.7332\n",
      "epoch:198, train acc:0.8866666666666667, test acc:0.7341\n",
      "epoch:199, train acc:0.8833333333333333, test acc:0.7262\n",
      "epoch:200, train acc:0.89, test acc:0.7225\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd81PX9wPHXO3sQkhBmEvYeIkuGIE4KuEDr3lqLtmrt\nr0rF1lpHW2mpC6vi3lsRFwKyEUQMO0CAsJNAFiQkZOc+vz++R8i4u1xC7i7j/Xw88kjuO9/3TfJ9\n3/czxRiDUkopBeDn6wCUUko1HpoUlFJKVdCkoJRSqoImBaWUUhU0KSillKqgSUEppVQFTQqqxROR\nLiKSLyL+vo5FKV/TpKBaFBHZLyKF9iSQLyL5QJkxppUxpryex2wnIh+KSK6IHBORD1xse4mI/Cgi\nOSJyREReF5GISuuvEZE1IlIgIsvrE49Sp0OTgmqJLrMngZNfaad5vLnAEaAL0B74r4ttI4F/ALFA\nfyAOmFVp/VHgOWDmacakVL0E+DoApXxNRLoB+4BAY0yZiHQH3gGGAj8DO4FIY8xNDvb9FdAZOK/S\nk8ZGZ+cyxnxY6WWBiLwGPF5p/WL7ce88nfekVH3pk4JSNX0IrANigMeAm11sOxorabwjItki8ouI\nnFuHc40HttU3UKUamiYF1RLNs5fp54jIvMorRKQLcBbwqDGmxBjzI/C1i2PFA78ClgEdgaeBr0Sk\nbW1BiMgE4Fbg0Xq+D6UanCYF1RJNNcZE2b+mVlsXCxw1xhRUWnbIxbEKgf3GmDeMMaXGmI/t248V\nkXMqVWhXeRoQkdFYTyRXGWN2NcB7UqpBaFJQqqrDQBsRCau0rLOL7bcA1YcaNgDGmFWVKrMHnlwp\nIkOxnj7uMMYsaaC4lWoQmhSUqsQYcwBIAB4TkSARGQNcVnkbe7PW2+wvvwSiReRWEfEXkauwipRW\nOzq+iAwCFgD3GWO+cbDeX0RCsBqB+IlIiIgENtT7U6o2mhSUqulGYAyQjdV89BOgGEBEgrAqoNcC\nGGOOApcDDwK5wAxgijEmy8mxHwDaAW84KVq6GatI6mXgHPvPrzXou1PKBdFJdpRyTUQ+AZKMMX8X\nkXHAPcaY630dl1KeoElBqWpE5CysTmT7sFoWzQPGGGOc9j9QqrnwWPGRiLwpIhkikuhkvYjIbBFJ\nFpEtIjLMU7EoVUcdgeVAPjAb+J0mBNVSeOxJQUTGY/1TvWuMGeRg/cXAfcDFwCjgeWPMKI8Eo5RS\nyi0ee1IwxqzEegR3ZgpWwjDGmLVAlIh08lQ8SimlaufLsY/iqNopKMW+7HD1DUVkGjANIDw8fHi/\nfv28EqBSSjUX69evzzLGtKttuyYxIJ4x5lXgVYARI0aYhIQEH0eklFJNi4gccGc7X/ZTSKVqT9F4\n+zKllFI+4suk8DVwi70V0mgg1xhTo+hIKaWU93is+EhEPgLOA9qKSArwdyAQwBgzB5iP1fIoGSgA\nbvdULEoppdzjsaRQW49PY7WFvcdT51dKKVV3OvaRUkqpCpoUlFJKVdCkoJRSqoImBaWUUhU0KSil\nlKqgSUEppVQFTQpKKaUqaFJQSilVQZOCUkqpCpoUlFJKVdCkoJRSqoImBaWUUhU0KSillKqgSUEp\npU5DabkNm834OowG0ySm41RKNS/zNqYya+FO0nIKiY0KZfrEvkwdGtfkzltWbuPy/61mSOconrry\njAaM1Hc0KSilvGrexlQenruVwtJyAFJzCnl47lYAjyaG0z2vo4RSUmZjx+HjpB4r4IkpAwn0d174\ncjoJyZtJVIuPlFJ1Nm9jKmNnLqX7jO8YO3Mp8zY6nl49K7+Y8/+7nKFPLOLG19dy7EQJsxburLgx\nn1RYWs6shTsByC0opbwOxTEnissoqnS85Ix8Jj67ku1px6ts5+q8R0+UVBQB7UrP4/6PN3LZCz9y\nMLug4v0+PHcrqTmFGKyE8tAXW/jX/B20Cg7geFEZ6w8ccxqjo/0fnrvV6XUrK7eRnV9cr31Plz4p\nKKXqpC6fuL/ZnMa+rBNcNTyerzencd2ra0nLKXR43LScQopKyzn/6eX0ateKN24bQURIoMPzn/zU\n3DEyhOKycuKiwvjid2cTFODHfxYksTM9j38vSOKdO0ZWOb6z845+agnje7flsjNjefCzzQT5++Hv\nJ1z9yhpeunG4w4RSXGajuMzGKzcP574PN7JkRzo2m6Gk3MY5vduxYlcGAOf1ae8yIZ28Zgn7j3I4\nt4ize8Zw9/vr2ZKSy5ybHZ+7+r4NSZOCUqpO6nKT+npzGv07tea/V5/JlUPjmPbeepw9A3SKDGHN\nniyOnihh3Ymj3PzGOj69awy70vN4c/U+/nXFGSxIPFIlIR3OLQLg6IlcnvlhFxMGtGfR9nR6tgtn\nxa5MXliym+2Hj3PP+b2IjQol1UFiCPAXMLB4RwaLd2QwvGs0r90ygsy8Ym5+42d+/fIap9dCgIkD\nOzKqRxs++eUQr/+4D2MgIjiAvOIyALrFhDk8L1gJKTOvmPs+2sDavUcB8PcTBOjSJoxp7yZQWu74\nijlLcqdLk4JSzdRT3+9g7d6jfHH3GJ5bvJulSRl8c984/P0EqH85tatP3IeOWsUtnduEcTC7gI0H\nc5gxuR8AZ/dqy4rp5/Hw3C0sScqsUUQ0vGs0S3ZkEB7kz+NTBvHgZ5v5JOEQ8zamsv7AMfp3bM3b\na/bXSEgAYUH+zFmxhzkr9hATHsSnd43h4tmrePqHXfgJ/Lg7iwGxEQ5vzqXlhnvP70mfjhH8tCeL\nRy4ZQHhwAG3Cg1j64Hl8+PMB/rtoFyVlthr7xkaFAnBR/w6s2p3F+D7tuGxwJ5btzGDiwI4AfLEh\nlbScIkrKa+4fHOjHta/8xOHcIv5+2QC6tw3ns4QUrhoRz7Au0Tzw6SYW78hweL1PnruhiTFNqynV\niBEjTEJCgq/DUKpCwv6jHCsoZcKADnXeNzWnkGcX7WTN3mwO5xQ1WAVkbmEpo/+1hMLScv54UW9e\nWraHknIbr90yggkDOtQoAgIIDfTnqSvPqPXcY2cudXhzDQn0o6TMRlhQAK/eMpyFiUd456cDrJ5x\nAXHVbmDV424dEkBabhHBAX4M6xLNyzcN4+o5P7Et7TiFpeVEhQUiwLGCUocxCfD4lIHkFZUxukcb\nhndtw9aUXNJyCxkY25pb3ljHoWMFDO8SzYGjBRzJta515+hQdqbnsXz6+USG1iyqqhyvq+tVXFbO\ngsQjTBrUkeAAf7f29xMI8hcC/f156/azGNGtjcNzz16yixeWJld5YnD3d1XlGomsN8aMqHU7TQpK\nnZ5LZq8i5VghG/42oeJTOMDcDSms2p3FM9eciYg43HfisyvYmZ5fZVn1f/gVuzKZ+X0SxaXlXNCv\nPQ9f3B9/P3F4owkO8OPJKQPJKy7nyW+3E2cvMgny96N1aCD9O0Xw3m9GOb2xBwf4sfTB89hw4Bjv\nrNnP1SPiST1WyNp9R5k6JI5zerflqfk7mJ94pMa+EcH+XHtWF5buzGBv5gnr2pzRiRdvHFbrNUxM\nzeXSF34E4D9XDeaaEZ35eW821766lq4xYTx77RCufMl5MU5cVCirZ1zgdP3JyuiYVsFVlheVllNQ\nUk6b8KBaYzzdFkDV60IemtSPCQM6OIyroc8NmhSU8orDuYWMeWopAN/cO44z4iMBMMZw4TMr2Jt5\ngpdvHMbkMzpRWm5j1e5MhndtQ2RoID/tyeb619Y6PO7Jm9yCxMPc99FGOrcJo3N0GCt2ZXLZmbH8\n64pBTHpulcMbuwDR4UF0bxvOHy/qzc1vrOOOsd2JDgvk6R928cgl/fnHdzucvqe2rYLIPlFCq+AA\n8oqscvGuMWEcsLfEARjRNYq03CKHTzdZ+cX8b2ky5/Ztx3l92jlNiNXd8+EGvt96mJ//chHtIqyb\n5Csr9jCkcxSjesTw9up9bEnJYX7iEYpKTxXF1OdTc0ukSUG1OJ5sy22M4Z01+zlWUMotY7pWfLJ7\nf+0BHpmXCMCMyf24+9yewKlPvgF+QteYMG4Z041XV+4lNaeQW8d05bHLB3LVnJ9cNmOceeUZ/HVe\nImfGR/LW7SOJDA3k5eV7+PeCJCJCTt2wHfETeOnG4Uwa1JHVyVkM7xpNXlEZ581axomSmmXyJ7WL\nCMZmMwyOj+SlG4eTmJZLdFgQPduF8/O+oxw8WkDrkEB+NaADfn7u3ezdlVtYyu70PKfFKCf5quNb\nU6dJQbUop1NGXhubzfDvBUm8snIvYJWdXz+yC9PG9+CvXyaSnJFPSVk5xwpKKSmzERsVSt8OrVi5\nO4snpw6qaK45vGs0xhgOHi3g5ZuGc/Wcn4gMDSS30HE5OcCYHjG8fusIwoNPtQnZmpLLS8uT+d5B\nEQ5YTxlLHzzXYdl2fnEZezPzufx/qxGByv/+J6/X5DM6EuTv5/YnfNU0aFJQLYqzMvLaypod+XF3\nFm0jgujXsTWrdmfy+DfbSc7I55YxXbl5dFfmrNjLvE2p+Nlvqmf3jGH1nuwarWkGdIrguz+cw3tr\nD9CnQwSjurdh0fZ07npvPd1iwsjOL+GRS/rz2DfbqyQzEfjDBb0IDQrgtrO7ERJY8+YO8MqKZGYt\n3EWZre4VkHe8/QtLkzI4edvXT9zNn7tJQZukqmbBVTPJk0rLbTz6VSKje8QwZYh189tw8Bj/WZBE\nSZmNW8/uxnl92/PbdxM4s3MkH/12NA9+tpngAH9euH4olw7uhIjw9DVn8seLevPaqr3M33qYHUfy\nHPbAPXK8GBHhljHdKpad17cdESEB7M8u4Lazu3HtyC4EB/oza+FOUnMK8fcT/jK5H785p0et7/mu\nc3vRoXVovYpSbhnTlaVJGVwyuBP/u6H2imDVcmhSUM2Cs45Jfn5CQUkZYUEBzF6ym4/WHeKjdYfI\nzi/hymFx3PPBBspshrJyG3/8ZFNFccov+46yLe046ceLeerKM7jszNgqx+3cJownpgziiSmD6D7j\nO4cxHTtRUmNZcIA/kwZ25LP1Kdw0uitg9QKu7yf0+u47vnc7bju7mz4ZqBp07CPVKOUWlvLy8j3k\nFTkvb69s+sS+BAdU/XMWoNxmeGv1ftbuzebFZclcMTSOCQM68MS32xn372VWr9XRXSgoKa9Svl5u\nYMYXWwAY27Oty3M760TkbPn0SX1567az6NW+lVvvzRP8/ITHLh/IkM5RPotBNU76pKAalZMtS05+\n6l+y4wif/25srftNHRrH4h1H+HbLEQToEBnCn3/Vl/mJh5mzfA/lxtAtJpwnpgwkLCiAb7ek8dbq\n/Vx+Zixv/LiPYge9VRPTjhMfHUqXmDCX554+sa/DSu7pE/s63L59RAjt+4XU+p6U8gVNCqrRcNSC\nKOFADo99nchjlw+qdf+cgjL6dohg4f+Nr1jWP7Y1F89eRe/2rXj/N6MqBlibMiSuol7hyW+3Oz1m\nbU8JcGoQOG0mqZoDTQqqUXht5V5mL93tcFybd346wO/O60WH1s4/XR89UcIv+49y46iuVZb379Sa\nb+4dR5eYMFo7GHETnNdHAJzdK8at+E+nXkCpxkTrFJTXFZWWk5xxamiHxNRc/jl/h9POWMbA3e+v\nZ8r/fuSeDzfUWJ9xvIjrXv0JA0wdGltj/aC4SKcJAazin9BqzT6DA/wYFNua8/u1d/NdKdU86JOC\nanCuepx+vO4g/120i6z8Yh69dAB3jOvOfxftJDI0kEB/ISu/ZoudyNBANh7MISoskM0pudwwMgt/\nP+FgdgEjukVzx9u/kJFXzNu3ncXg+LpXnGrxj1KnaOc11aBc9Swe3jWa8bOWMaxLNK1DAli2M5N+\nHSNIOpLHjMn96Ng6xOm+A2NbEx8dxoVPL8fPT0g/XlQxamRESABv3z6S4V2jvf5+lWoqGkXnNRGZ\nBDwP+AOvG2NmVlsfCbwPdLHH8l9jzFuejEl5lqsJWC4d3Ak/Ef53w1DatQrmmR92sfNIHgNjI7l1\nTDdCg/wrjuHsE/v9F/XmoS+2MqJrNL8/vyfztx7htrO7MSgu0qvvU6nmymNPCiLiD+wCJgApwC/A\n9caY7ZW2+QsQaYx5SETaATuBjsaYmmUIdvqk0Lh1c9KRCyAqLJAxPWJ4+abh9T6+zWZYtjODMT1j\nCAvS0k+l3OXuk4InK5pHAsnGmL32m/zHwJRq2xggQqyRt1oBRwHnQz+qRu3oiRJcDZyZU1BaZciH\n+vDzEy7s30ETglIe4sn/rDjgUKXXKcCoatv8D/gaSAMigGuNMTV6EYnINGAaQJcuXTwSrDo9xhj+\nMncrgtVyp3JnsNBAP24c1ZUOrUMY3cP1sMhKKd/y9ceticAm4AKgJ/CDiKwyxhyvvJEx5lXgVbCK\nj7weparVxkM5LNh2hOkT+xIXVb9B2pRSvufJpJAKdK70Ot6+rLLbgZnGqthIFpF9QD9gnQfjUh7w\n3k8HiAi2hnoODw7QJKBUE+XJOoVfgN4i0l1EgoDrsIqKKjsIXAggIh2AvsBeD8akPCArv5jvthzm\n18Pjq0wGo5Rqejz2H2yMKRORe4GFWE1S3zTGbBORu+3r5wBPAm+LyFasQS0fMsZkeSom5RmfJaRQ\nUm6rGApaKdV0efRjnTFmPjC/2rI5lX5OA37lyRiU521JyaFHu3CfDgWtlGoYOvaROm0HjxbQtY3r\n4aWVUk2DJgVVw2Nfb+PRrxIBa/C6LSk57Mm0BrBLyylk4rMr+XlvdsX2B48W0FmTglLNgtYKqipy\nCkr44OcDlJYb5m89XGWAur9fNoCkw3nsTM/jH9/t4Ot7x3K8sIy8ojK6aFJQqlnQpKCq+D7xSMVA\nc5UTgp/AP77dDiL0bt+Kram5LEg8Qny0lQxOfleq0ZrVG05k1Fwe3h6m7/Z+PI2UFh+pKr7alEqA\ng7EqbMYakyQ4wI8P7hxFr/ateGFpMoeOFQDok4Jq/BwlBFfLWyh9UlAVjuQW8fO+ozgbI9EYeO83\no2jfOoSrh8fz1PdJJOw/BkDnNo4nqVeqhpb4ib0JvWdNCqrCpwmHMAY6tA4m/XhxjfWxUaEVcxaM\n7WXNXfzlxhSiwwIr5j5WTcTp3KRO9wbXFD6xF+fDiUxo09167e57zt4DAcEQGQ9Fx8HYIDSqabxn\nO00KCoDSchsf/HyA8X3aceXQOIeT3Uyf2Lfi9YBOrYkKC+RYQSlnxutcBk3O6dykfHmDc3ZzDouB\nu1dD607Wa5sN/OpQOm4rtx6F9yyFrZ9C0ndQWgCxQyGklpv6Z7fByLugXV94/UIIjoC7VsGbEyE3\nBYbd6n4cxfmw9EnwC7ASy4lMGHytdWwv0aSgAPhhezrpx4v559SuXDSgA+B6shs/P+HsnjHM33qE\neK1P8I3aPr2WFUN5KQTXo1NheRlkbIOY3pCeCJlJcMY1EBjier+Pb4QLH7VuYsbAroWw4V3wD4SO\nZ8CIO2o/d34GZOyAgBDoUm1gZWc354JseKYfXPQ4hETCD4/C5S9A3DBY+g8Y9yfX5/zwWusGnrkD\nQqOtG3GbHpD4BRQfd73v/h+t99n1bCjKhcJj8NoFcHQPdDsH1r7oev/CHPjlNYjsAhvfgwOrwT8I\nyoqs9aueBWoMHu2xoidNCi2cMYavN6cxa+FO4qNDKyaqnzo0rtZB7c7u2Zb5W49oJfPpqEtRTHEe\nbPoI9q2AS591/el14/uw5EkoyoGz/wDj/g+C6vB7WvoErH4ea/QZeyXT+nfgyldd77dvpXVDvHw2\nJLwF+1dBRKx17u3zYPlTrvc/sAbevwpKT1ivf7sMYnrB3uXQ60LX+/a/HBb/3fo5MAzm/Q7C2kLu\nQTi4FvwCwVZac7+gVpD8A0R1havegn6XQkCQtW7sH6zvj7l4Gr57NbxxESQvhlG/s65/4hdWYrny\nVasYaWZn5/u/Mh5yDpx6fcWrMOhKK1kUZMNL1WccsPPQk5kmhRZuyY4M7v94E73bt+KJKYPwdzVL\nTjXje7fD30/o00GHt6g3Vzf2Az9BebF1UywthPevhJyD1vqoWsaZ+uoeiBsOXcfAyv9A0rcw5AbY\n8glEurhBAaRugJ9ehL4XQ4dBENUZAkLhmz/AC8Nc73vPOnhvKnx+BwSGW8lr6M3Wk0LGDtj0Aax5\nwfn+b19iJZFr3oW5v4XFj4Gfv1WsE1hLUrv6bVj0iPXzmHvg9QnWp/ZLnoYFf7ESwtSXretQXVay\nVVxT25OQIxEd4KYvIeENOG+G9YQW1cVKxgAhrV3vL35wxyIQsfbtfo61vFU768vLPDYdp6fodJwN\n66nvd/Dmj/vY9vgkggLq3kJ5X9YJOkeHEuCvrZsdMsb6Z3fG1SfQygJCrWKgq9+2ngK2zYOyQufb\n3/QF9LjAKlffvRi+nGZ96ux0JpzIhuMprs8XGg33rofwmFPLjh2A3Ytg/oMu3k8uFBy1njKG3Ajt\n+tTcxtnTUWCYlTxumw8dB1mJaeFfrHXj/wz56bDhHdfnrizviHWTje5qJZW8dBhyvfP9XXH1e6p+\nXkdcPRE+uKv+fyPunNvO3ek49Umhhduedpw+HSLqlRAAurcNb+CImiBn//D+wdC2D0xbDv4O/tX2\nLnd93Gvft27OB9dC5k44/2GrnLtVB9j8set9e1106ufeF8HvfrKKKOLPsipQn+oMprzmfiHR0HsC\nDJhSNSGAdXMd+VtY8R/nNziAsDYw4XHnsbkqBy8tOvVp/aw7Ycun0G0cXPBXa5mrpFBdRMdTP/e8\nwP39HAlv7/o916aRNTt1RZNCC2aMYXvacS7s7+YftqrJZnNeBFReDOlbYfdCiOgEK2dB+jYYe79V\n/v3l71wfu/9l1vdu46oub9vbOsbq59yPM6KD9QUQFA63z4c3J8HZ98Kv/uH+ccCzN7jKxTcBwVZC\nrfwp+nRvzvXVhG7qp0uTQguWfryY7BMlDOhUS5mnsuQchKN7rU/rgWFWBWXCm673iYi1ytBzDkJ5\nCYS3g/nTrUrYwqP1j2XC47Dpw/rfILuMhrtXWe+lMaterNKCbs4VvJwINSm0YNvSrPLIgXHaz6BW\n+1bCR9dDSX7d9htxOyz7J4g/3LnYqjR+/ULrCWLKS1ZFan3/4U/3BtnxjNPbX3mHlxOhJoUW4Jkf\ndjGsSxTn9bVuNAezC3ht1V78/QQR6N8cnhQachiBLZ9BZJzV7vyj661KyvISaNsXJjwBeWlWBWbx\ncasfwIp/Oz/WsFtg7csw+vdWm3mAW76yWvj0vxSG3li32JTyME0KzVxGXhGzl+wmIiSABX8cT1xU\nKM8t2cXcDamAVVHcqjnMq+xOL1tjrI5BYW2tzlXH9ludjn74m3XTry4kymrn3/cSaN8PxtxrVaJW\n5yopRHSEB5Ks8vGTWsdaX0o1Qs3gbqBc+WmPNRlOYUk5D3y6ieevG8q3mw8ztlcMmw/lMsI+llGz\nl5tqtd3fu8x6HRJp9T51pSgHQtvAr1+zKmedqa3Mt3JCUKqR06TQzK1OzqJ1SACPXDqAP3++hUtf\n+JGSchuPXz6Qdq1CCPB3v7Nak7XlM1gz26oknjQTEGvohtgh0ON81x2yRt3tOiFAy6z8VM2WJoVm\nat7GVGYtTCI1p4iQQD+C/P14cspA/vbVNs7uGUOv9hG+DtF75t5pfb/uI+h3cd32HTWt4eNRqhHT\npNAMzduYWmWU06JSGw/P3cpTV57BF787m/joRjr3gbuVxbZya+iDkw6scX3c0DZWhW9dEwJYnceU\nakE0KTRDsxburDLsNUBhaTmzFu5k9YzT7NnpSa4qi202qz5g3WtWZ7AuZ1sjaB7Zag294Ex4e/i/\nbacGOFNKuaRJoRlKzXE8Jk6ak+VNwv+GW3UCYW2tT/17V8CPz0KrjnD+I1bP3MB6PgH5qpesUo2Q\nJoUmrNxmyCsq5f21B8jKL+Fvlw5g5e5Mp9vHRjXSYiN3RHeDc2fAwKlWax5jrFmtKhcj1ZdWFCtV\nQZNCI2ZVFlsT3XSKDKFtqyCyT5Ry1fB4Vu7OZOPBnCrbBwX48fn6FGIjQzhaUEJR6amJOarPnOZz\nxsDhTbDtS8hIsgZqc+XmL6u+FrF6CSulGpQmhUaqemVxWm4RablFxEeF8PyS3cRHh3Lv+b0ICfTj\n/H7teWFJMq+u3EtwgB+f3jWOxNTjLmdO8wljrElQUhKsWa5yDljTDoa3s+oJlFI+p0mhkXJUWQxg\nEFbPuIAOEcFV5jD415VnkJ5XxA0ju9CrfQS92kf4Jgm4akF06TPWOPudhljj7oyfDv0usTqS7VoA\nX93reJA4LdtXyms0KTRSziqF03IKiXNQN9AmPIgvfz/W02HVzlULoh/+bo0fdOeSmvML9LvE+lJK\n+ZROl9VIOasUbtKVxUf3WAPKOZpwRinVKGhSaKSmT+yLf7Wx5BtdZXFdTZ0DfSb6OgqllAv6ka2R\nGt0jBn8/CPb3p7CkvPFUFjuTmwrf/cn1NvWdH1cp5TWaFBqZA9knmLNiL1+sT8Fm4IM7RzG0SyMf\naiEjCd6bCsV1nIBGKdXoaPFRI7LzSB4XPbOCL9ancNWIeJY8cG7jSwgnsuDbP1mJ4KTl/4LSQrhj\ngfOWQtqCSKkmwaNPCiIyCXge8AdeN8bMdLDNecBzQCCQZYw515MxNWbfbE7DZmD5g+fSuU2Y7wJx\n1qw0rK01OcyRLVYT0t/8YE1Iv28V9J0MHQdp72ClmjiPPSmIiD/wIjAZGABcLyIDqm0TBbwEXG6M\nGQhc7al4moLFO9IZ3jXatwkBnDcrLciCjB0w8V9QdBw+vRkytll9C7qP926MSimP8GTx0Ugg2Riz\n1xhTAnwMTKm2zQ3AXGPMQQBjjJO7UfOUmJpLcoZVDp9yrICkI3lc1N/HxSwZO1yvv2MhjLkHJv4D\nUtfDsn9Zy7ud4/nYlFIe58mkEAccqvQ6xb6ssj5AtIgsF5H1InKLowOJyDQRSRCRhMxM5wO+NSXG\nGH77bgI3vLaWYydKWJZk5cML+3fwXVBbP4eXRrveJn649X3wtRAWAzvnQ3R3iOrs+fiUUh7n64rm\nAGA4cAkwEfibiPSpvpEx5lVjzAhjzIh27dp5O0aP2JZ2nMO5RWTkFXPX++t5ZeVeurcNp2e7Vr4J\nKCsZvrnhhfgRAAAdHUlEQVQfOteSFE4KDIURv7F+7q5PCUo1F24lBRGZKyKXiEhdkkgqUPnjY7x9\nWWUpwEJjzAljTBawEjizDudospbsyEAEpo3vwbp9RwkO8OOfUwf5JhhjYN7d4B8IV73h/n5n3QlR\nXWDgFZ6LTSnlVe62PnoJuB2YLSKfAW8ZY3bWss8vQG8R6Y6VDK7DqkOo7CvgfyISAAQBo4Bn3Q2+\nKVuSlM7QzlE8PLkfVw6Lo0/7CPz8pPYdPWHPUkj5BS59DiLj3Z90JqID/HGrd2JUSnmFW0nBGLMY\nWCwikcD19p8PAa8B7xtjSh3sUyYi9wILsZqkvmmM2SYid9vXzzHG7BCRBcAWwIbVbDWxQd5ZI5Z+\nvIgtKblMn9gXEaFfx9a+DWjlf6F1HAyx52xtVqpUi+V2PwURiQFuAm4GNgIfAOOAW4HzHO1jjJkP\nzK+2bE6117OAWXUJuqn7LMGqf58wwIeVyiftXQ4H18Dk/1gzmimlWjS3koKIfAn0Bd4DLjPGHLav\n+kREEjwVXHOUU1DCKyv3clH/9vTpEOHbYEpOWJXL0d2teY+VUi2eu08Ks40xyxytMMaMaMB4mr1X\nVu4lv7iMB37lo9FOM5IgbQPkZ8D+H+HYfrjtu/pPeq+UalbcTQoDRGSjMSYHQESigeuNMS95LrTm\nJ6+olHfX7OfSwbH07+SDeoQd38Bnt4PNXgUUGA7nzoBu47wfi1KqUXI3KfzWGPPiyRfGmGMi8lus\nVknKTV9uTOVESTm/GdfdsydyNnYRQPxZMOUlawyjYB/1iVBKNVruJgV/ERFjjIGKcY2CPBdW8zFv\nYyqzFu4kLacQfz+hc3QoQzpHee6ENpvzhABw85cQ7OO6DKVUo+VuZ7QFWJXKF4rIhcBH9mXKhXkb\nU3l47lZScwoxQJnNcOR4EfM2Vu/D14CWPul6vSYEpZQL7iaFh4BlwO/sX0uAP3sqqOZi1sKdFJaW\nV1lWWm6YtbC2fn/1lJEEa2Z75thKqRbB3c5rNuBl+5dyU1pOYZ2WnxabDeY/CEGtoCin4Y+vlGoR\n3O2n0Bt4CmtehJCTy40xPTwUV7PQMTKEw7lFNZbHRjVQ888TWfDRddD9XCjKhf2r4LLZ8M0fGub4\nSqkWx93io7ewnhLKgPOBd4H3PRVUc3FW15pTaYYG+jN94mn2Ucg7AuVlsGCGNafBqv/CL6/BmHth\n+K06JaZSqt7cbX0UaoxZYm+BdAB4TETWA496MLYmrdxm2HAoh57twikqtZGWU0hsVCjTJ/Zl6tDq\n00rUQfYeeHEUhLWB/HSrn0H8CGtynDH3Wtvo2EVKqXpyNykU24fN3m0f5C4V0EbuLixLyiDlWCEv\n3TiMi8/o1HAH3vgemHJo3x/a9YVzHoCAIOg9oeHOoZRqsdxNCvcDYcAfgCexipBu9VRQzcG7aw/Q\nsXVIww56ZyuHzR9Drwlw46cNd1yllLKrNSnYO6pda4x5EMjHmldBuVBUWs7q5Cx+e04PAv3rObmd\nq17Jk/9T/+CUUsqFWu9YxphyrCGylZt2Hsmj3GYY0jmy/gdx1Su5z6T6H1cppVxwt/hoo4h8DXwG\nnDi50Bgz1yNRNXHb0o4DMDD2NJKCKwE6wohSyjPcTQohQDZwQaVlBtCk4MC2tFwiQgKIj9bhqJVS\nTYu7PZq1HqEOtqUdZ0Cn1ojUc87l9e80bEBKKeUmd3s0v4X1ZFCFMeaOBo+oiSu3GZKOHOeGkV3r\nd4CSAlj0t4YNSiml3ORu8dG3lX4OAa4A0ho+nKZvb2Y+RaU2BsbWcxKd7V9BcS6ERDkew0h7JSul\nPMjd4qMvKr8WkY+AHz0SURN3spJ5QH2TwoZ3oE1PuG891Lf4SSml6qmejejpDehHVgd+3pdNq+AA\nerWvR4fv9G1w8Cdr/CJNCEopH3C3TiGPqnUKR7DmWFCV2GyGJTsyGN+nbf06rS1+DIIjYejNDR6b\nUkq5w93iI52uyw3b0o6TkVfMhf3qMLRFziHY9CH4B8LuRTDhCWuwO6WU8gF3nxSuAJYaY3Ltr6OA\n84wx8zwZXFOzeEc6fgLn93OzZG3fKvjsVijItl5HdYGRd3kuQKWUqoW7rY/+boz58uQLY0yOiPwd\n0KRQyZKkdIZ1iaZNuBs9jksL4bPbILQN3PYdFB6DiE4QGFLrrkop5SnuJgVHBeTu7tsiZOUXk5h6\n3P0JdLZ8AgVZcPVb1jDYSinVCLhbG5ogIs+ISE/71zPAek8G1tSs2WMVAY3r1bb2jW02+OlF6DgY\nup3j4ciUUsp97iaF+4AS4BPgY6AIuMdTQTVFa5KzaB0SwKA4NwbB27UAsnZZM6Vp01OlVCPibuuj\nE8AMD8fSpP2YnMXoHjH4+9Vyk7eVw5InoE0PGHSld4JTSik3ufWkICI/2FscnXwdLSILPRdW03Iw\nu4CUY4WMdafoaMsnkLkDLnzUaoaqlFKNiLvFR22NMRUD8RhjjqE9miv8mJwFwNheMa43LC+D5U9B\n7FAYMNULkSmlVN24mxRsItLl5AsR6YaDUVNbqgXbjhAfHUrPdrUMbbHre8g5COP+pHUJSqlGyd1m\npX8FfhSRFYAA5wDTPBZVE5KVX8zq5CzuGt+j9vkT1s6ByC7Q92LvBKeUUnXk1pOCMWYBMALYCXwE\nPAAUejCuJmP+1sOU2wxThsS53vDwFjjwI4y8E/y1i4dSqnFyd5iLO4H7gXhgEzAa+Imq03M62m8S\n8DzgD7xujJnpZLuz7Me7zhjzudvRNwJfb0qjX8cI+nZ0MDzUrN5wIqPqsh8ehTX/g+m7vROgUkrV\ngbt1CvcDZwEHjDHnA0MBBzPAnCIi/sCLwGRgAHC9iAxwst2/gUV1iLtRSEzNJeHAMedPCdUTQm3L\nlVLKx9xNCkXGmCIAEQk2xiQBtY3nMBJINsbsNcaUYHV6m+Jgu/uAL4Amd6ectXAnUWGB3DS6S+0b\nK6VUE+BuUkix91OYB/wgIl8BB2rZJw44VPkY9mUVRCQOa2rPl10dSESmiUiCiCRkZma6GbJn/bw3\nmxW7Mvn9eT2JCNH+Bkqp5sHdHs1X2H98TESWAZHAggY4/3PAQ8YYm6uWO8aYV4FXAUaMGNEomsJ+\nuTGV1iEB3DKmm69DUUqpBlPnZjDGmBVubpoKdK70Ot6+rLIRwMf2hNAWuFhEyprCPA2JabkMjo8i\nJNDf8QZlxd4NSCmlGkB952h2xy9AbxHpLiJBwHXA15U3MMZ0N8Z0M8Z0Az4Hft8UEkJpuY1dR/IZ\nGNva+UZbPnG+Llw7gyulGiePNZg3xpSJyL3AQqwmqW8aY7aJyN329XM8dW5PS87Ip6TcxgBnScEY\nq9lpx8Fw10rtvayUajI82ovKGDMfmF9tmcNkYIy5zZOxNKRtaccBGBjrZJjs9G2QtRMufU4TglKq\nSfFk8VGztS0tl9BAf7q3DXe8wY5vAIF+l3o1LqWUOl2aFOphW9px+nWKcD53wo6voevZ0KqddwNT\nSqnTpEmhjmw2w460484rmbOSIWM79L/cu4EppVQD0KRQR6k5heQVl9G/k5OkkPSN9b2/Fh0ppZoe\nTQp1lJyZD0CfDg4GwAOrPiF2GETGezEqpZRqGJoU6mhPhpUUejmaUCc3BVLXQ//LvByVUko1DE0K\ndbQ7PZ+Y8CCiw4Nqrkz6zvqu9QlKqSZKk0IdJWfm06u9k2k3t38N7QdA217eDUoppRqIJoU6MMaQ\nnOEkKRQchYNroN8l3g9MKaUaiCaFOsjMLya3sJTejpLC3mVgbNB7ovcDU0qpBqJJoQ6ST1Yyt3fQ\n8mj3YgiNhrhhXo5KKaUajiaFOqhoeVT9ScFmg+TF0PMC8HMylLZSSjUBmhTqYFd6PhHBAXRoHVx1\nRfpWa97lXhN8E5hSSjUQTQp1sCU1l4FxrakxS9yuRdb3nhd4PyillGpAmhTcVFJmY0facc7sHFVz\n5fZ50HkURHTwfmBKKdWANCm4KenIcUrKbZwZXy0pZO6C9EQYeIXjHZVSqgnRpOCmzYdyAGo+KWyf\nBwgMmOL9oJRSqoFpUnDDvI2p/Gt+EgDXzFnDvI2pp1YmzoUuY6B1rI+iU0qphuPR6Tibg3kbU3l4\n7lYKS8sBSM0p4uG5WwGY2j4dMnfAJU/7MkSllGow+qRQi1kLd1YkhJMKS8uZtXAnrH8LAsPgjKt9\nFJ1SSjUsTQq1SMspdLg8Lycbtn4Bg34NIZFejkoppTxDk0ItYqNCHS6/udUvUHoCRtzh5YiUUspz\nNCnUYvrEvvhV66sWGujPzW22Q0wvHetIKdWsaFKoxSWDO+EvQniwPwLERYUyc2o/Oh7bAD3O83F0\nSinVsLT1US22px2n1GZ49teDuXSwvdnpwbVW0VH3c30bnFJKNTB9UqjFun1HARjRtc2phXtXAALd\nxvkmKKWU8hB9UnCi3GZ4ZF4inyUcok+HVnSMDDm1ct9K6DQYwto4P4BSSjVB+qTgxC/7j/LRuoNc\nMTSOd+8YdWpFYQ6krNOiI6VUs6RPCk6cHOvo4Yv70yY86NSKNS9AeQkMvsZHkSmllOfok4ITW1Jy\n6dwmtGpCOJEFa1+2RkTteIbvglNKKQ/RpODEpkM5NYfJXjMbygrhvL/4JiillPIwTQoOZOYVk5pT\nyJDKw2SXFcOG96DfJdCuj++CU0opD9I6BQe2pFj1CYPjo2BWb2v+5ZN2fAOPRUJ4e5i+20cRKqWU\nZ+iTggMbD+bgJzAornXVhFCZs+VKKdWEeTQpiMgkEdkpIskiMsPB+htFZIuIbBWRNSJypifjqU1x\nWTn3fbSRF5cnM6xLNGFB+iCllGpZPHbXExF/4EVgApAC/CIiXxtjtlfabB9wrjHmmIhMBl4FRtU8\nmnes3JXFN5vTuH1sN+67oLevwlBKKZ/x5EfhkUCyMWYvgIh8DEwBKpKCMWZNpe3XAvEejKdWCfuP\nEuTvx0OT+hES6A/F+b4MRymlvM6TxUdxwKFKr1Psy5z5DfC9oxUiMk1EEkQkITMzswFDrOqX/Uc5\nIz7SSgg2G8y722PnUkqpxqhRVDSLyPlYSeEhR+uNMa8aY0YYY0a0a9fOIzEUlZazNTWXEd2irQWr\nnrZaGgW1crxDeHuPxKGUUr7kyeKjVKBzpdfx9mVViMhg4HVgsjEm24PxuLQlJZfScsPI+FaQ8BYs\n+wcMvhaueAVEaj+AUko1A55MCr8AvUWkO1YyuA64ofIGItIFmAvcbIzZ5cFYavXL/qN0k8Ocv+D/\n4EQ6xJ8Flz2vCUEp1aJ4LCkYY8pE5F5gIeAPvGmM2SYid9vXzwEeBWKAl8S6+ZYZY0Z4KiZXViRl\n8HzYW/iVF8GNn0PPC8DP3xehKKWUz3i0Ib4xZj4wv9qyOZV+vhO405MxuGN1chY9Ur7gzMBEuHg2\n9J7g65CUUsonWnzvLGMMS75+jycD38bWbTx+w27xdUhKKQ8oLS0lJSWFoqIiX4fiUSEhIcTHxxMY\nGFiv/Vt8Uti4egEP5f6TvKg+tLn2Pa1DUKqZSklJISIigm7duiHN9P/cGEN2djYpKSl07969Xsdo\nFE1Sfeb4YXos+z3pEkOr33wDoVG176OUapKKioqIiYlptgkBQESIiYk5raehlpsUjKHws98SWHaC\nZUOeJah1W19HpJTysOacEE463ffYYpNCye5lhB5axTPl1zLpggt8HY5SSjUKLTIp/JScxY4PZ5Bm\n2mCG30aH1iG+Dkkp1cjM25jK2JlL6T7jO8bOXMq8jTX63tZJTk4OL730Up33u/jii8nJyTmtc9dF\ni0sKO76fQ5v3zuNMdlI05gEevWK4r0NSSjUy8zam8vDcraTmFGKA1JxCHp679bQSg7OkUFZW5nK/\n+fPnExXlvfrO5t/6qNrMaf0BBGxBrejxq9/5LCyllO88/s02tqcdd7p+48EcSsptVZYVlpbz58+3\n8NG6gw73GRDbmr9fNtDpMWfMmMGePXsYMmQIgYGBhISEEB0dTVJSErt27WLq1KkcOnSIoqIi7r//\nfqZNmwZAt27dSEhIID8/n8mTJzNu3DjWrFlDXFwcX331FaGhofW4As41/ycFJzOk+ZXka49lpZRD\n1RNCbcvdMXPmTHr27MmmTZuYNWsWGzZs4Pnnn2fXLmuEnzfffJP169eTkJDA7Nmzyc6uORTc7t27\nueeee9i2bRtRUVF88cUX9Y7Hmeb/pKCUUtW4+kQPMHbmUlJzCmssj4sK5ZO7xjRIDCNHjqzSl2D2\n7Nl8+eWXABw6dIjdu3cTExNTZZ/u3bszZMgQAIYPH87+/fsbJJbKmv+TglJK1dH0iX0JDaxakhAa\n6M/0iX0b7Bzh4eEVPy9fvpzFixfz008/sXnzZoYOHeqwr0FwcHDFz/7+/rXWR9SHPikopVQ1U4da\n84HNWriTtJxCYqNCmT6xb8Xy+oiIiCAvL8/hutzcXKKjowkLCyMpKYm1a9fW+zynS5OCUko5MHVo\n3GklgepiYmIYO3YsgwYNIjQ0lA4dOlSsmzRpEnPmzKF///707duX0aNHN9h566rZJ4VsooihZhtf\na7lSSnnPhx9+6HB5cHAw33/vcDbiinqDtm3bkpiYWLH8wQcfbPD4oAUkhVVT1vDw3K0UlpZXLAsN\n9OepK89gqg/jUkqpxqjZJwVPlA0qpVRz1eyTAjR82aBSSjVX2iRVKaVUBU0KSimlKmhSUEopVaFF\n1CkopVSdVBtIs0J4e5i+u16HzMnJ4cMPP+T3v/99nfd97rnnmDZtGmFhYfU6d13ok4JSSlXnZCBN\np8vdUN/5FMBKCgUFBfU+d13ok4JSquX5fgYc2Vq/fd+6xPHyjmfA5JlOd6s8dPaECRNo3749n376\nKcXFxVxxxRU8/vjjnDhxgmuuuYaUlBTKy8v529/+Rnp6OmlpaZx//vm0bduWZcuW1S9uN2lSUEop\nL5g5cyaJiYls2rSJRYsW8fnnn7Nu3TqMMVx++eWsXLmSzMxMYmNj+e677wBrTKTIyEieeeYZli1b\nRtu2np9LXpOCUqrlcfGJHoDHIp2vu/270z79okWLWLRoEUOHDgUgPz+f3bt3c8455/DAAw/w0EMP\ncemll3LOOeec9rnqSpOCUkp5mTGGhx9+mLvuuqvGug0bNjB//nweeeQRLrzwQh599FGvxqYVzUop\nVV14+7otd0PlobMnTpzIm2++SX5+PgCpqalkZGSQlpZGWFgYN910E9OnT2fDhg019vU0fVJQSqnq\n6tns1JXKQ2dPnjyZG264gTFjrFncWrVqxfvvv09ycjLTp0/Hz8+PwMBAXn75ZQCmTZvGpEmTiI2N\n9XhFsxhjPHqChjZixAiTkJDg6zCUUk3Mjh076N+/v6/D8ApH71VE1htjRtS2rxYfKaWUqqBJQSml\nVAVNCkqpFqOpFZfXx+m+R00KSqkWISQkhOzs7GadGIwxZGdnExISUu9jaOsjpVSLEB8fT0pKCpmZ\nmb4OxaNCQkKIj4+v9/6aFJRSLUJgYCDdu3f3dRiNnkeLj0RkkojsFJFkEZnhYL2IyGz7+i0iMsyT\n8SillHLNY0lBRPyBF4HJwADgehEZUG2zyUBv+9c04GVPxaOUUqp2nnxSGAkkG2P2GmNKgI+BKdW2\nmQK8ayxrgSgR6eTBmJRSSrngyTqFOOBQpdcpwCg3tokDDlfeSESmYT1JAOSLyM56xtQWyKrnvp7U\nWOOCxhubxlU3GlfdNMe4urqzUZOoaDbGvAq8errHEZEEd7p5e1tjjQsab2waV91oXHXTkuPyZPFR\nKtC50ut4+7K6bqOUUspLPJkUfgF6i0h3EQkCrgO+rrbN18At9lZIo4FcY8zh6gdSSinlHR4rPjLG\nlInIvcBCwB940xizTUTutq+fA8wHLgaSgQLgdk/FY3faRVAe0ljjgsYbm8ZVNxpX3bTYuJrc0NlK\nKaU8R8c+UkopVUGTglJKqQotJinUNuSGF+PoLCLLRGS7iGwTkfvtyx8TkVQR2WT/utgHse0Xka32\n8yfYl7URkR9EZLf9e7SXY+pb6ZpsEpHjIvJHX1wvEXlTRDJEJLHSMqfXR0Qetv+97RSRiV6Oa5aI\nJNmHj/lSRKLsy7uJSGGl6zbHy3E5/b35+Hp9Uimm/SKyyb7cm9fL2b3Bu39jxphm/4VV0b0H6AEE\nAZuBAT6KpRMwzP5zBLALaxiQx4AHfXyd9gNtqy37DzDD/vMM4N8+/j0eweqE4/XrBYwHhgGJtV0f\n++90MxAMdLf//fl7Ma5fAQH2n/9dKa5ulbfzwfVy+Hvz9fWqtv5p4FEfXC9n9wav/o21lCcFd4bc\n8ApjzGFjzAb7z3nADqxe3I3VFOAd+8/vAFN9GMuFwB5jzAFfnNwYsxI4Wm2xs+szBfjYGFNsjNmH\n1cJupLfiMsYsMsaU2V+uxeoD5FVOrpczPr1eJ4mIANcAH3ni3K64uDd49W+spSQFZ8Np+JSIdAOG\nAj/bF91nf9x/09vFNHYGWCwi6+1DiwB0MKf6jhwBOvggrpOuo+o/q6+vFzi/Po3pb+4O4PtKr7vb\ni0JWiMg5PojH0e+tsVyvc4B0Y8zuSsu8fr2q3Ru8+jfWUpJCoyMirYAvgD8aY45jjRDbAxiCNfbT\n0z4Ia5wxZgjW6LX3iMj4yiuN9czqkzbMYnWAvBz4zL6oMVyvKnx5fZwRkb8CZcAH9kWHgS723/Of\ngA9FpLUXQ2p0v7dqrqfqBw+vXy8H94YK3vgbaylJoVENpyEigVi/9A+MMXMBjDHpxphyY4wNeA0P\nPTq7YoxJtX/PAL60x5Au9pFr7d8zvB2X3WRggzEm3R6jz6+XnbPr4/O/ORG5DbgUuNF+M8Fe1JBt\n/3k9Vjl0H2/F5OL31hiuVwBwJfDJyWXevl6O7g14+W+spSQFd4bc8Ap7meUbwA5jzDOVllceMvwK\nILH6vh6OK1xEIk7+jFVRmYh1nW61b3Yr8JU346qkyic4X1+vSpxdn6+B60QkWES6Y80Zss5bQYnI\nJODPwOXGmIJKy9uJNdcJItLDHtdeL8bl7Pfm0+tldxGQZIxJObnAm9fL2b0Bb/+NeaNWvTF8YQ2n\nsQsr0//Vh3GMw3r82wJssn9dDLwHbLUv/xro5OW4emC1ZNgMbDt5jYAYYAmwG1gMtPHBNQsHsoHI\nSsu8fr2wktJhoBSr/PY3rq4P8Ff739tOYLKX40rGKm8++Tc2x77tr+2/303ABuAyL8fl9Pfmy+tl\nX/42cHe1bb15vZzdG7z6N6bDXCillKrQUoqPlFJKuUGTglJKqQqaFJRSSlXQpKCUUqqCJgWllFIV\nNCko5WEicp6IfOvrOJRyhyYFpZRSFTQpKGUnIjeJyDr74GeviIi/iOSLyLP28e2XiEg7+7ZDRGSt\nnJqvINq+vJeILBaRzSKyQUR62g/fSkQ+F2uOgw/svVcRkZn28fO3iMh/ffTWlaqgSUEpQET6A9cC\nY401+Fk5cCNWb+oEY8xAYAXwd/su7wIPGWMGY/XQPbn8A+BFY8yZwNlYPWfBGvHyj1hj4PcAxopI\nDNZQDwPtx/mHZ9+lUrXTpKCU5UJgOPCLfdatC7Fu3jZODZD2PjBORCKBKGPMCvvyd4Dx9rGj4owx\nXwIYY4rMqXGH1hljUow1ENwmrMlbcoEi4A0RuRKoGKNIKV/RpKCURYB3jDFD7F99jTGPOdiuvuPC\nFFf6uRxrVrQyrFFCP8cazXRBPY+tVIPRpKCUZQlwlYi0h4p5cbti/Y9cZd/mBuBHY0wucKzShCs3\nAyuMNVtWiohMtR8jWETCnJ3QPm5+pDFmPvB/wJmeeGNK1UWArwNQqjEwxmwXkUeARSLihzWC5j3A\nCWCkfV0GVr0DWEMYz7Hf9PcCt9uX3wy8IiJP2I9xtYvTRgBfiUgI1pPKnxr4bSlVZzpKqlIuiEi+\nMaaVr+NQylu0+EgppVQFfVJQSilVQZ8UlFJKVdCkoJRSqoImBaWUUhU0KSillKqgSUEppVSF/wda\n7d0+PYpG4gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1621e95b0b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## 導入方法：weight_decay_lambda=weight_decay_lambda\n",
    "##  順伝播：0,5*self.weight_decay_lambda*np.sum(W**2)\n",
    "## 逆伝播： (lambda * W )\n",
    "\n",
    "# coding: utf-8\n",
    "import os\n",
    "import sys\n",
    "\n",
    "sys.path.append('../')  # 親ディレクトリのファイルをインポートするための設定\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from saitobook.common.multi_layer_net import MultiLayerNet\n",
    "from saitobook.common.optimizer import SGD\n",
    "\n",
    "\n",
    "# 過学習を再現するために、学習データを削減\n",
    "x_train = x_train[:300]\n",
    "t_train = t_train[:300]\n",
    "\n",
    "# weight decay（荷重減衰）の設定 =======================\n",
    "#weight_decay_lambda = 0 # weight decayを使用しない場合\n",
    "weight_decay_lambda = 0.1\n",
    "# ====================================================\n",
    "\n",
    "network = MultiLayerNet(input_size=784, hidden_size_list=[100, 100, 100, 100, 100, 100], output_size=10,\n",
    "                        weight_decay_lambda=weight_decay_lambda)\n",
    "optimizer = SGD(lr=0.01)\n",
    "\n",
    "max_epochs = 201\n",
    "train_size = x_train.shape[0]\n",
    "batch_size = 100\n",
    "\n",
    "train_loss_list = []\n",
    "train_acc_list = []\n",
    "test_acc_list = []\n",
    "\n",
    "iter_per_epoch = max(train_size / batch_size, 1)\n",
    "epoch_cnt = 0\n",
    "\n",
    "for i in range(1000000000):\n",
    "    batch_mask = np.random.choice(train_size, batch_size)\n",
    "    x_batch = x_train[batch_mask]\n",
    "    t_batch = t_train[batch_mask]\n",
    "\n",
    "    grads = network.gradient(x_batch, t_batch)\n",
    "    optimizer.update(network.params, grads)\n",
    "\n",
    "    if i % iter_per_epoch == 0:\n",
    "        train_acc = network.accuracy(x_train, t_train)\n",
    "        test_acc = network.accuracy(x_test, t_test)\n",
    "        train_acc_list.append(train_acc)\n",
    "        test_acc_list.append(test_acc)\n",
    "\n",
    "        print(\"epoch:\" + str(epoch_cnt) + \", train acc:\" + str(train_acc) + \", test acc:\" + str(test_acc))\n",
    "\n",
    "        epoch_cnt += 1\n",
    "        if epoch_cnt >= max_epochs:\n",
    "            break\n",
    "\n",
    "\n",
    "# 3.グラフの描画==========\n",
    "markers = {'train': 'o', 'test': 's'}\n",
    "x = np.arange(max_epochs)\n",
    "plt.plot(x, train_acc_list, marker='o', label='train', markevery=10)\n",
    "plt.plot(x, test_acc_list, marker='s', label='test', markevery=10)\n",
    "plt.xlabel(\"epochs\")\n",
    "plt.ylabel(\"accuracy\")\n",
    "plt.ylim(0, 1.0)\n",
    "plt.legend(loc='lower right')\n",
    "plt.title('Fig,6-21')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 6.4.3 Dropout\n",
    "\n",
    "- 過学習抑制のWeight Decayの問題\n",
    "   - Weight decay=損失関数に対して重みのL2 ノルムを加算\n",
    "   - モデルが複雑になるとWeight decayは対応が困難になる。\n",
    "- 過学習抑制には<b>Dropout[14] 手法</b>がよく用いられる\n",
    "   - Dropout はニューロンをランダムに消去しながら学習する手法\n",
    "   - 訓練時に隠れ層のニューロンをランダムに選出して消去\n",
    "   - 消去ニューロンは、信号の伝達が行われない(図6-22 )\n",
    "   - テスト時には全ニューロン計算も<u>訓練時の消去割合を乗算</u>して出力\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dropoutの実装\n",
    "- 特徴\n",
    "    - 順伝播のたびにself.maskに消去ニューロンをFalseで格納\n",
    "        - self.mask はxと同形状配列をランダムに生成\n",
    "        - ランダム値が<u>dropout_ratio よりも大きい要素だけをTrueとする</u>\n",
    "    - 逆伝播の際の挙動はReLU と同じ\n",
    "        - 順伝播で信号を通したニューロンは、逆伝播の際に伝わる信号をそのまま通す\n",
    "        - 順伝播で信号を通さなかったニューロンは、逆伝播では信号がストップ\n",
    "    - 以降でDropoutの効果を確かめるために、MNIST データセットで検証\n",
    "        - ソースコードはch06/overfit_dropout.py \n",
    "        - Trainerクラスを利用して実装簡略化\n",
    "    - 7 層ネットワーク（各層のニューロンの個数は100 個、活性化関数はReLU）\n",
    "         - 片方にはDropout適用、もう片方にはDropout を使用しない\n",
    "         - 結果図6-23\n",
    "         - Dropoutで訓練データとテストデータの認識精度の隔たり小さく\n",
    "         - 訓練データが100% の認識精度に到達することもなくなる（過学習抑制）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Trainer.pyの中身\n",
    "# coding: utf-8\n",
    "import sys, os\n",
    "#sys.path.append(os.pardir)  # 親ディレクトリのファイルをインポートするための設定\n",
    "import numpy as np\n",
    "from saitobook.common.optimizer import *\n",
    "\n",
    "class Trainer:\n",
    "    \"\"\"ニューラルネットの訓練を行うクラス\n",
    "    \"\"\"\n",
    "    def __init__(self, network, x_train, t_train, x_test, t_test,\n",
    "                 epochs=20, mini_batch_size=100,\n",
    "                 optimizer='SGD', optimizer_param={'lr':0.01}, \n",
    "                 evaluate_sample_num_per_epoch=None, verbose=True):\n",
    "        self.network = network\n",
    "        self.verbose = verbose\n",
    "        self.x_train = x_train\n",
    "        self.t_train = t_train\n",
    "        self.x_test = x_test\n",
    "        self.t_test = t_test\n",
    "        self.epochs = epochs\n",
    "        self.batch_size = mini_batch_size\n",
    "        self.evaluate_sample_num_per_epoch = evaluate_sample_num_per_epoch\n",
    "\n",
    "        # optimzer\n",
    "        optimizer_class_dict = {'sgd':SGD, 'momentum':Momentum, 'nesterov':Nesterov,\n",
    "                                'adagrad':AdaGrad, 'rmsprpo':RMSprop, 'adam':Adam}\n",
    "        self.optimizer = optimizer_class_dict[optimizer.lower()](**optimizer_param)\n",
    "        \n",
    "        self.train_size = x_train.shape[0]\n",
    "        self.iter_per_epoch = max(self.train_size / mini_batch_size, 1)\n",
    "        self.max_iter = int(epochs * self.iter_per_epoch)\n",
    "        self.current_iter = 0\n",
    "        self.current_epoch = 0\n",
    "        \n",
    "        self.train_loss_list = []\n",
    "        self.train_acc_list = []\n",
    "        self.test_acc_list = []\n",
    "\n",
    "    def train_step(self):\n",
    "        batch_mask = np.random.choice(self.train_size, self.batch_size)\n",
    "        x_batch = self.x_train[batch_mask]\n",
    "        t_batch = self.t_train[batch_mask]\n",
    "        \n",
    "        grads = self.network.gradient(x_batch, t_batch)\n",
    "        self.optimizer.update(self.network.params, grads)\n",
    "        \n",
    "        loss = self.network.loss(x_batch, t_batch)\n",
    "        self.train_loss_list.append(loss)\n",
    "        if self.verbose: print(\"train loss:\" + str(loss))\n",
    "        \n",
    "        if self.current_iter % self.iter_per_epoch == 0:\n",
    "            self.current_epoch += 1\n",
    "            \n",
    "            x_train_sample, t_train_sample = self.x_train, self.t_train\n",
    "            x_test_sample, t_test_sample = self.x_test, self.t_test\n",
    "            if not self.evaluate_sample_num_per_epoch is None:\n",
    "                t = self.evaluate_sample_num_per_epoch\n",
    "                x_train_sample, t_train_sample = self.x_train[:t], self.t_train[:t]\n",
    "                x_test_sample, t_test_sample = self.x_test[:t], self.t_test[:t]\n",
    "                \n",
    "            train_acc = self.network.accuracy(x_train_sample, t_train_sample)\n",
    "            test_acc = self.network.accuracy(x_test_sample, t_test_sample)\n",
    "            self.train_acc_list.append(train_acc)\n",
    "            self.test_acc_list.append(test_acc)\n",
    "\n",
    "            if self.verbose: print(\"=== epoch:\" + str(self.current_epoch) + \", train acc:\" + str(train_acc) + \", test acc:\" + str(test_acc) + \" ===\")\n",
    "        self.current_iter += 1\n",
    "\n",
    "    def train(self):\n",
    "        for i in range(self.max_iter):\n",
    "            self.train_step()\n",
    "\n",
    "        test_acc = self.network.accuracy(self.x_test, self.t_test)\n",
    "\n",
    "        if self.verbose:\n",
    "            print(\"=============== Final Test Accuracy ===============\")\n",
    "            print(\"test acc:\" + str(test_acc))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train loss:2.2983773533733474\n",
      "=== epoch:1, train acc:0.1, test acc:0.0914 ===\n",
      "train loss:2.29541878750506\n",
      "train loss:2.3154481528346325\n",
      "train loss:2.344262193849066\n",
      "=== epoch:2, train acc:0.10666666666666667, test acc:0.0968 ===\n",
      "train loss:2.3204846950992883\n",
      "train loss:2.2913937228211143\n",
      "train loss:2.304509346159207\n",
      "=== epoch:3, train acc:0.12, test acc:0.1041 ===\n",
      "train loss:2.328238665969377\n",
      "train loss:2.2987441510551645\n",
      "train loss:2.3081551362400305\n",
      "=== epoch:4, train acc:0.12333333333333334, test acc:0.1159 ===\n",
      "train loss:2.3104913885256217\n",
      "train loss:2.29659504034014\n",
      "train loss:2.290787673517686\n",
      "=== epoch:5, train acc:0.13333333333333333, test acc:0.1215 ===\n",
      "train loss:2.3188628277443906\n",
      "train loss:2.295529818200188\n",
      "train loss:2.2637998596685436\n",
      "=== epoch:6, train acc:0.14666666666666667, test acc:0.132 ===\n",
      "train loss:2.2890342512415685\n",
      "train loss:2.2793699082397025\n",
      "train loss:2.2942650123530584\n",
      "=== epoch:7, train acc:0.16666666666666666, test acc:0.1424 ===\n",
      "train loss:2.300318728391712\n",
      "train loss:2.3056400832668396\n",
      "train loss:2.3010912900272613\n",
      "=== epoch:8, train acc:0.17, test acc:0.1491 ===\n",
      "train loss:2.2876076247296337\n",
      "train loss:2.275650876115618\n",
      "train loss:2.2918369014502327\n",
      "=== epoch:9, train acc:0.18, test acc:0.1554 ===\n",
      "train loss:2.2792178653831296\n",
      "train loss:2.27697977609496\n",
      "train loss:2.2838130722557097\n",
      "=== epoch:10, train acc:0.19333333333333333, test acc:0.159 ===\n",
      "train loss:2.303893740797976\n",
      "train loss:2.2687041946582465\n",
      "train loss:2.282059879118129\n",
      "=== epoch:11, train acc:0.19666666666666666, test acc:0.1634 ===\n",
      "train loss:2.2829660319209846\n",
      "train loss:2.273897665504434\n",
      "train loss:2.2908167288705896\n",
      "=== epoch:12, train acc:0.19333333333333333, test acc:0.1685 ===\n",
      "train loss:2.273527081555196\n",
      "train loss:2.274281927167968\n",
      "train loss:2.281617820454152\n",
      "=== epoch:13, train acc:0.2, test acc:0.1693 ===\n",
      "train loss:2.295697657578737\n",
      "train loss:2.282917860063466\n",
      "train loss:2.2962013461707462\n",
      "=== epoch:14, train acc:0.2, test acc:0.1718 ===\n",
      "train loss:2.2722456113486573\n",
      "train loss:2.2824441071290575\n",
      "train loss:2.27722803354231\n",
      "=== epoch:15, train acc:0.2, test acc:0.1716 ===\n",
      "train loss:2.268924511406341\n",
      "train loss:2.2561449060973326\n",
      "train loss:2.2821235047481823\n",
      "=== epoch:16, train acc:0.19666666666666666, test acc:0.1698 ===\n",
      "train loss:2.256575043368181\n",
      "train loss:2.286612188419763\n",
      "train loss:2.272723224485307\n",
      "=== epoch:17, train acc:0.21, test acc:0.1714 ===\n",
      "train loss:2.2657397583623573\n",
      "train loss:2.267021628683729\n",
      "train loss:2.280321482995468\n",
      "=== epoch:18, train acc:0.20666666666666667, test acc:0.1688 ===\n",
      "train loss:2.2645530103717233\n",
      "train loss:2.2614026588337\n",
      "train loss:2.277358968498003\n",
      "=== epoch:19, train acc:0.21, test acc:0.1663 ===\n",
      "train loss:2.2798027080789187\n",
      "train loss:2.2848617574709875\n",
      "train loss:2.2872983007867798\n",
      "=== epoch:20, train acc:0.21666666666666667, test acc:0.1644 ===\n",
      "train loss:2.265673728909872\n",
      "train loss:2.285910577551747\n",
      "train loss:2.270007385001835\n",
      "=== epoch:21, train acc:0.21333333333333335, test acc:0.1668 ===\n",
      "train loss:2.2650772345732024\n",
      "train loss:2.2535792360630302\n",
      "train loss:2.2594590667783847\n",
      "=== epoch:22, train acc:0.21333333333333335, test acc:0.1657 ===\n",
      "train loss:2.268030747474674\n",
      "train loss:2.255017616609208\n",
      "train loss:2.283291542130339\n",
      "=== epoch:23, train acc:0.21666666666666667, test acc:0.1684 ===\n",
      "train loss:2.245503481103492\n",
      "train loss:2.250167229064196\n",
      "train loss:2.253636862617962\n",
      "=== epoch:24, train acc:0.20333333333333334, test acc:0.1653 ===\n",
      "train loss:2.275565206270328\n",
      "train loss:2.2618285923433503\n",
      "train loss:2.263955477013617\n",
      "=== epoch:25, train acc:0.2, test acc:0.1653 ===\n",
      "train loss:2.265779047872599\n",
      "train loss:2.266927060097034\n",
      "train loss:2.242227879593471\n",
      "=== epoch:26, train acc:0.20666666666666667, test acc:0.1688 ===\n",
      "train loss:2.2616371254749414\n",
      "train loss:2.2535428994374813\n",
      "train loss:2.257950727116611\n",
      "=== epoch:27, train acc:0.21, test acc:0.1722 ===\n",
      "train loss:2.264093047929348\n",
      "train loss:2.2696797824630717\n",
      "train loss:2.2547327933002665\n",
      "=== epoch:28, train acc:0.21333333333333335, test acc:0.1733 ===\n",
      "train loss:2.2621850619913246\n",
      "train loss:2.2428817637411713\n",
      "train loss:2.274439665683517\n",
      "=== epoch:29, train acc:0.22, test acc:0.1734 ===\n",
      "train loss:2.251542115911268\n",
      "train loss:2.261215130617555\n",
      "train loss:2.26559687741823\n",
      "=== epoch:30, train acc:0.21333333333333335, test acc:0.174 ===\n",
      "train loss:2.241558742584142\n",
      "train loss:2.2433893852559854\n",
      "train loss:2.265297134621338\n",
      "=== epoch:31, train acc:0.21666666666666667, test acc:0.1821 ===\n",
      "train loss:2.2684764847022634\n",
      "train loss:2.248080206166485\n",
      "train loss:2.245535814877662\n",
      "=== epoch:32, train acc:0.22333333333333333, test acc:0.1825 ===\n",
      "train loss:2.2638437918832706\n",
      "train loss:2.250740117719114\n",
      "train loss:2.268421289759071\n",
      "=== epoch:33, train acc:0.22, test acc:0.1859 ===\n",
      "train loss:2.246411717304589\n",
      "train loss:2.246480596809091\n",
      "train loss:2.23368842044915\n",
      "=== epoch:34, train acc:0.21, test acc:0.1846 ===\n",
      "train loss:2.2218494896422616\n",
      "train loss:2.209991211106202\n",
      "train loss:2.242664563003096\n",
      "=== epoch:35, train acc:0.20666666666666667, test acc:0.1814 ===\n",
      "train loss:2.241121849622181\n",
      "train loss:2.2432992850157683\n",
      "train loss:2.209302079199528\n",
      "=== epoch:36, train acc:0.20333333333333334, test acc:0.1776 ===\n",
      "train loss:2.226542697172787\n",
      "train loss:2.222570647874042\n",
      "train loss:2.2354952661715206\n",
      "=== epoch:37, train acc:0.21333333333333335, test acc:0.1797 ===\n",
      "train loss:2.246681139382837\n",
      "train loss:2.2481417529243495\n",
      "train loss:2.2399303923530476\n",
      "=== epoch:38, train acc:0.21666666666666667, test acc:0.1864 ===\n",
      "train loss:2.246098129359508\n",
      "train loss:2.22980561308209\n",
      "train loss:2.2303268335943476\n",
      "=== epoch:39, train acc:0.21666666666666667, test acc:0.1897 ===\n",
      "train loss:2.211803154532376\n",
      "train loss:2.2491314706304992\n",
      "train loss:2.230072231893374\n",
      "=== epoch:40, train acc:0.21666666666666667, test acc:0.1897 ===\n",
      "train loss:2.2288120510677083\n",
      "train loss:2.226361996969188\n",
      "train loss:2.232555810452252\n",
      "=== epoch:41, train acc:0.21333333333333335, test acc:0.1873 ===\n",
      "train loss:2.2415394277096783\n",
      "train loss:2.2097052460463185\n",
      "train loss:2.2432739684811795\n",
      "=== epoch:42, train acc:0.21333333333333335, test acc:0.1882 ===\n",
      "train loss:2.237713407807166\n",
      "train loss:2.2164711978637968\n",
      "train loss:2.2384573703038897\n",
      "=== epoch:43, train acc:0.22333333333333333, test acc:0.1874 ===\n",
      "train loss:2.2286104892729752\n",
      "train loss:2.1894338153784965\n",
      "train loss:2.1772679708345244\n",
      "=== epoch:44, train acc:0.21, test acc:0.1791 ===\n",
      "train loss:2.2486896522834146\n",
      "train loss:2.2327864306420566\n",
      "train loss:2.224354944305801\n",
      "=== epoch:45, train acc:0.21333333333333335, test acc:0.1868 ===\n",
      "train loss:2.1963028706602534\n",
      "train loss:2.2519592553960406\n",
      "train loss:2.2419592018629135\n",
      "=== epoch:46, train acc:0.21666666666666667, test acc:0.1926 ===\n",
      "train loss:2.2268584797994784\n",
      "train loss:2.2151339950861417\n",
      "train loss:2.2070685544283806\n",
      "=== epoch:47, train acc:0.22666666666666666, test acc:0.1931 ===\n",
      "train loss:2.2426119161665867\n",
      "train loss:2.23673896977574\n",
      "train loss:2.2163932522213026\n",
      "=== epoch:48, train acc:0.22333333333333333, test acc:0.1949 ===\n",
      "train loss:2.2292031936090138\n",
      "train loss:2.218847240371582\n",
      "train loss:2.2278090363583885\n",
      "=== epoch:49, train acc:0.22333333333333333, test acc:0.1949 ===\n",
      "train loss:2.2221594550131387\n",
      "train loss:2.206838045046633\n",
      "train loss:2.1883188040031007\n",
      "=== epoch:50, train acc:0.22333333333333333, test acc:0.1932 ===\n",
      "train loss:2.236458808532823\n",
      "train loss:2.242114848755206\n",
      "train loss:2.2450684024551\n",
      "=== epoch:51, train acc:0.23, test acc:0.1973 ===\n",
      "train loss:2.216105436475415\n",
      "train loss:2.2217707669696782\n",
      "train loss:2.228561576566907\n",
      "=== epoch:52, train acc:0.23, test acc:0.2016 ===\n",
      "train loss:2.201421045626703\n",
      "train loss:2.22998100468048\n",
      "train loss:2.1996477424065626\n",
      "=== epoch:53, train acc:0.23333333333333334, test acc:0.2033 ===\n",
      "train loss:2.2234666424348712\n",
      "train loss:2.226774656233995\n",
      "train loss:2.1736757982982295\n",
      "=== epoch:54, train acc:0.23666666666666666, test acc:0.2056 ===\n",
      "train loss:2.15686552864911\n",
      "train loss:2.203471773997851\n",
      "train loss:2.221982307034068\n",
      "=== epoch:55, train acc:0.23666666666666666, test acc:0.2053 ===\n",
      "train loss:2.180570764390508\n",
      "train loss:2.1912103052375644\n",
      "train loss:2.1917531921660984\n",
      "=== epoch:56, train acc:0.24, test acc:0.2056 ===\n",
      "train loss:2.1940005406518437\n",
      "train loss:2.181257993480995\n",
      "train loss:2.203025908482988\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== epoch:57, train acc:0.24333333333333335, test acc:0.2054 ===\n",
      "train loss:2.2011013587788533\n",
      "train loss:2.202878659902505\n",
      "train loss:2.197609112468784\n",
      "=== epoch:58, train acc:0.24333333333333335, test acc:0.2057 ===\n",
      "train loss:2.1873710037738845\n",
      "train loss:2.212730579678651\n",
      "train loss:2.1870679554735704\n",
      "=== epoch:59, train acc:0.24333333333333335, test acc:0.2067 ===\n",
      "train loss:2.2238312155883433\n",
      "train loss:2.2067761657164473\n",
      "train loss:2.191031884063711\n",
      "=== epoch:60, train acc:0.24666666666666667, test acc:0.2075 ===\n",
      "train loss:2.2062941952436645\n",
      "train loss:2.1666399057331587\n",
      "train loss:2.1907601437681627\n",
      "=== epoch:61, train acc:0.24666666666666667, test acc:0.2079 ===\n",
      "train loss:2.1748644811023614\n",
      "train loss:2.1384893593561984\n",
      "train loss:2.1556453238027298\n",
      "=== epoch:62, train acc:0.24666666666666667, test acc:0.2048 ===\n",
      "train loss:2.1414799790537513\n",
      "train loss:2.202593889071415\n",
      "train loss:2.1345823902546748\n",
      "=== epoch:63, train acc:0.24666666666666667, test acc:0.2045 ===\n",
      "train loss:2.1665406476691875\n",
      "train loss:2.1574173274248816\n",
      "train loss:2.1791258825602124\n",
      "=== epoch:64, train acc:0.25333333333333335, test acc:0.2043 ===\n",
      "train loss:2.0999497358185737\n",
      "train loss:2.1869979858707715\n",
      "train loss:2.165404388725694\n",
      "=== epoch:65, train acc:0.24666666666666667, test acc:0.2023 ===\n",
      "train loss:2.2146229371203545\n",
      "train loss:2.160802179301194\n",
      "train loss:2.1829838377947715\n",
      "=== epoch:66, train acc:0.25, test acc:0.2048 ===\n",
      "train loss:2.169387692837447\n",
      "train loss:2.1453830524746618\n",
      "train loss:2.172052788169877\n",
      "=== epoch:67, train acc:0.24333333333333335, test acc:0.2032 ===\n",
      "train loss:2.1721483400983237\n",
      "train loss:2.1670862223487\n",
      "train loss:2.159034170664134\n",
      "=== epoch:68, train acc:0.24333333333333335, test acc:0.2036 ===\n",
      "train loss:2.1566482208703146\n",
      "train loss:2.172533697874337\n",
      "train loss:2.101706508490058\n",
      "=== epoch:69, train acc:0.23333333333333334, test acc:0.2006 ===\n",
      "train loss:2.180436920810539\n",
      "train loss:2.1277874194151827\n",
      "train loss:2.0786487069296236\n",
      "=== epoch:70, train acc:0.24, test acc:0.2006 ===\n",
      "train loss:2.17686304941371\n",
      "train loss:2.1549658738651876\n",
      "train loss:2.181011263401789\n",
      "=== epoch:71, train acc:0.24333333333333335, test acc:0.2034 ===\n",
      "train loss:2.147032814038035\n",
      "train loss:2.2032807878562854\n",
      "train loss:2.174726483842508\n",
      "=== epoch:72, train acc:0.25, test acc:0.2074 ===\n",
      "train loss:2.103072874354317\n",
      "train loss:2.106334529116194\n",
      "train loss:2.1936695153427404\n",
      "=== epoch:73, train acc:0.25, test acc:0.2083 ===\n",
      "train loss:2.1922669725290866\n",
      "train loss:2.169347819750277\n",
      "train loss:2.139693756233038\n",
      "=== epoch:74, train acc:0.25, test acc:0.2114 ===\n",
      "train loss:2.207989886231537\n",
      "train loss:2.154188979556807\n",
      "train loss:2.1270634914788062\n",
      "=== epoch:75, train acc:0.25, test acc:0.2142 ===\n",
      "train loss:2.123104940147536\n",
      "train loss:2.1406441272968006\n",
      "train loss:2.1762569439080113\n",
      "=== epoch:76, train acc:0.25333333333333335, test acc:0.2151 ===\n",
      "train loss:2.142003544786947\n",
      "train loss:2.182966664394937\n",
      "train loss:2.188114472274996\n",
      "=== epoch:77, train acc:0.25666666666666665, test acc:0.2163 ===\n",
      "train loss:2.157371200624059\n",
      "train loss:2.1426033121902854\n",
      "train loss:2.094487682703811\n",
      "=== epoch:78, train acc:0.25333333333333335, test acc:0.2174 ===\n",
      "train loss:2.1406061813543955\n",
      "train loss:2.144125834035493\n",
      "train loss:2.1146745117577477\n",
      "=== epoch:79, train acc:0.25333333333333335, test acc:0.2159 ===\n",
      "train loss:2.1530671851259235\n",
      "train loss:2.0822628616708867\n",
      "train loss:2.0658220119653823\n",
      "=== epoch:80, train acc:0.25333333333333335, test acc:0.2155 ===\n",
      "train loss:2.0468628294364724\n",
      "train loss:2.1055663014771424\n",
      "train loss:2.205983252925946\n",
      "=== epoch:81, train acc:0.26, test acc:0.2188 ===\n",
      "train loss:2.1256598948661316\n",
      "train loss:2.094951833594006\n",
      "train loss:2.0836299457672474\n",
      "=== epoch:82, train acc:0.26, test acc:0.2201 ===\n",
      "train loss:2.124351850339513\n",
      "train loss:2.162730167973186\n",
      "train loss:2.169011622633463\n",
      "=== epoch:83, train acc:0.26, test acc:0.221 ===\n",
      "train loss:2.101533645595896\n",
      "train loss:2.1172228479635824\n",
      "train loss:2.0991807259965976\n",
      "=== epoch:84, train acc:0.2633333333333333, test acc:0.2228 ===\n",
      "train loss:2.0760836341517854\n",
      "train loss:2.1206284606242223\n",
      "train loss:2.0987622675339854\n",
      "=== epoch:85, train acc:0.2633333333333333, test acc:0.2244 ===\n",
      "train loss:2.0505597478062025\n",
      "train loss:2.0615730376996084\n",
      "train loss:2.120156300065695\n",
      "=== epoch:86, train acc:0.2633333333333333, test acc:0.2226 ===\n",
      "train loss:2.0824528532166564\n",
      "train loss:2.1750431082126593\n",
      "train loss:2.1759793492679207\n",
      "=== epoch:87, train acc:0.27, test acc:0.2233 ===\n",
      "train loss:2.039623099110441\n",
      "train loss:2.1532944469149022\n",
      "train loss:2.1179149087794342\n",
      "=== epoch:88, train acc:0.26666666666666666, test acc:0.2226 ===\n",
      "train loss:2.105687686070851\n",
      "train loss:2.0776309566416957\n",
      "train loss:2.138735598795604\n",
      "=== epoch:89, train acc:0.2633333333333333, test acc:0.2235 ===\n",
      "train loss:2.0826043307545343\n",
      "train loss:2.038287734008538\n",
      "train loss:2.0498441079084917\n",
      "=== epoch:90, train acc:0.26666666666666666, test acc:0.2222 ===\n",
      "train loss:2.0704180951002713\n",
      "train loss:2.0560020585233705\n",
      "train loss:2.0475583385440337\n",
      "=== epoch:91, train acc:0.2633333333333333, test acc:0.2225 ===\n",
      "train loss:1.9919029596456652\n",
      "train loss:2.0494720714212615\n",
      "train loss:2.1083179951261637\n",
      "=== epoch:92, train acc:0.26666666666666666, test acc:0.223 ===\n",
      "train loss:2.0887786341981402\n",
      "train loss:2.0630952723190648\n",
      "train loss:2.1144715752240444\n",
      "=== epoch:93, train acc:0.27, test acc:0.224 ===\n",
      "train loss:1.983423163740836\n",
      "train loss:2.0875079091068143\n",
      "train loss:2.051844332039078\n",
      "=== epoch:94, train acc:0.2633333333333333, test acc:0.2223 ===\n",
      "train loss:2.0754402306277124\n",
      "train loss:2.1181498704115382\n",
      "train loss:2.0954569299551724\n",
      "=== epoch:95, train acc:0.27666666666666667, test acc:0.224 ===\n",
      "train loss:2.102328435413722\n",
      "train loss:2.149545907201353\n",
      "train loss:2.0743407961680242\n",
      "=== epoch:96, train acc:0.2833333333333333, test acc:0.2266 ===\n",
      "train loss:2.089117227436851\n",
      "train loss:2.0478064898068684\n",
      "train loss:1.9844090426006145\n",
      "=== epoch:97, train acc:0.28, test acc:0.2261 ===\n",
      "train loss:2.089366332707394\n",
      "train loss:2.134221249650484\n",
      "train loss:2.07680658425815\n",
      "=== epoch:98, train acc:0.2833333333333333, test acc:0.23 ===\n",
      "train loss:2.0385608338851693\n",
      "train loss:2.0697308298758936\n",
      "train loss:2.0074111760461797\n",
      "=== epoch:99, train acc:0.2833333333333333, test acc:0.2288 ===\n",
      "train loss:2.0425990395549594\n",
      "train loss:2.0348867586360253\n",
      "train loss:2.0736202055038877\n",
      "=== epoch:100, train acc:0.29, test acc:0.2333 ===\n",
      "train loss:2.051971057599968\n",
      "train loss:1.9853166759309624\n",
      "train loss:2.0220942510992925\n",
      "=== epoch:101, train acc:0.2866666666666667, test acc:0.2335 ===\n",
      "train loss:2.0905949142549276\n",
      "train loss:2.079513097330752\n",
      "train loss:2.043161859545691\n",
      "=== epoch:102, train acc:0.29333333333333333, test acc:0.2356 ===\n",
      "train loss:2.0491451474231606\n",
      "train loss:2.0090998427375495\n",
      "train loss:2.083149849887957\n",
      "=== epoch:103, train acc:0.29, test acc:0.2328 ===\n",
      "train loss:2.072026588853539\n",
      "train loss:1.9811810610752576\n",
      "train loss:1.9967691408834298\n",
      "=== epoch:104, train acc:0.2966666666666667, test acc:0.2342 ===\n",
      "train loss:2.0364615171614555\n",
      "train loss:1.9921705014173592\n",
      "train loss:2.0341382254488964\n",
      "=== epoch:105, train acc:0.3, test acc:0.236 ===\n",
      "train loss:2.0706162116149556\n",
      "train loss:2.062870451028073\n",
      "train loss:2.0520182014887753\n",
      "=== epoch:106, train acc:0.30333333333333334, test acc:0.2398 ===\n",
      "train loss:2.029375856440734\n",
      "train loss:2.064204083302194\n",
      "train loss:2.003776273276357\n",
      "=== epoch:107, train acc:0.31, test acc:0.247 ===\n",
      "train loss:2.025744089347484\n",
      "train loss:2.0194978223055746\n",
      "train loss:1.9025335652863617\n",
      "=== epoch:108, train acc:0.31333333333333335, test acc:0.2475 ===\n",
      "train loss:2.034450168692243\n",
      "train loss:1.97897616909254\n",
      "train loss:2.0780069127312397\n",
      "=== epoch:109, train acc:0.32, test acc:0.2509 ===\n",
      "train loss:2.0270352063100883\n",
      "train loss:1.9907559367013297\n",
      "train loss:2.043010648953837\n",
      "=== epoch:110, train acc:0.3233333333333333, test acc:0.2554 ===\n",
      "train loss:1.9612129951958224\n",
      "train loss:2.0930008941266203\n",
      "train loss:1.9994084425266425\n",
      "=== epoch:111, train acc:0.3333333333333333, test acc:0.2602 ===\n",
      "train loss:1.9193996839532952\n",
      "train loss:2.012018263884105\n",
      "train loss:2.027561760099055\n",
      "=== epoch:112, train acc:0.33, test acc:0.2611 ===\n",
      "train loss:2.0405900712511356\n",
      "train loss:1.9534907847750813\n",
      "train loss:2.0979154656727625\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== epoch:113, train acc:0.33, test acc:0.2615 ===\n",
      "train loss:1.936097935242457\n",
      "train loss:2.050682472472182\n",
      "train loss:1.9807773835885762\n",
      "=== epoch:114, train acc:0.33, test acc:0.2615 ===\n",
      "train loss:1.9879862000662667\n",
      "train loss:2.0178493706315295\n",
      "train loss:2.087769841205029\n",
      "=== epoch:115, train acc:0.3333333333333333, test acc:0.263 ===\n",
      "train loss:2.0896590974485822\n",
      "train loss:1.9898942370131931\n",
      "train loss:2.0397113432342358\n",
      "=== epoch:116, train acc:0.33666666666666667, test acc:0.2637 ===\n",
      "train loss:1.8997413613430356\n",
      "train loss:1.9611846668757713\n",
      "train loss:1.9974416089155207\n",
      "=== epoch:117, train acc:0.33, test acc:0.2605 ===\n",
      "train loss:1.9289347177141096\n",
      "train loss:2.086232848496385\n",
      "train loss:2.037552241985765\n",
      "=== epoch:118, train acc:0.3333333333333333, test acc:0.2637 ===\n",
      "train loss:2.0998821378075347\n",
      "train loss:1.9865751417759645\n",
      "train loss:2.039674473941974\n",
      "=== epoch:119, train acc:0.34, test acc:0.2676 ===\n",
      "train loss:1.998944593362514\n",
      "train loss:1.9237626925298819\n",
      "train loss:2.0261485918802062\n",
      "=== epoch:120, train acc:0.3433333333333333, test acc:0.2696 ===\n",
      "train loss:1.8708549998767952\n",
      "train loss:1.9939058536958583\n",
      "train loss:1.9505484568253855\n",
      "=== epoch:121, train acc:0.3433333333333333, test acc:0.27 ===\n",
      "train loss:2.013320418791257\n",
      "train loss:1.9308388753244714\n",
      "train loss:1.9614628433314059\n",
      "=== epoch:122, train acc:0.3433333333333333, test acc:0.2687 ===\n",
      "train loss:1.917644682737403\n",
      "train loss:1.985492529942523\n",
      "train loss:2.0051482498864313\n",
      "=== epoch:123, train acc:0.34, test acc:0.2676 ===\n",
      "train loss:2.036899525186287\n",
      "train loss:1.980040629088137\n",
      "train loss:1.9478112076336591\n",
      "=== epoch:124, train acc:0.34, test acc:0.2653 ===\n",
      "train loss:1.987905872906612\n",
      "train loss:2.0986170427180175\n",
      "train loss:1.9007165964385055\n",
      "=== epoch:125, train acc:0.3433333333333333, test acc:0.2664 ===\n",
      "train loss:1.9233146957742449\n",
      "train loss:1.9332412325480834\n",
      "train loss:1.9869819107369062\n",
      "=== epoch:126, train acc:0.34, test acc:0.2679 ===\n",
      "train loss:1.9058938047047773\n",
      "train loss:1.98004172365378\n",
      "train loss:1.9811972773643731\n",
      "=== epoch:127, train acc:0.34, test acc:0.271 ===\n",
      "train loss:1.9139612826815684\n",
      "train loss:1.9381459132040602\n",
      "train loss:2.090128857681712\n",
      "=== epoch:128, train acc:0.3433333333333333, test acc:0.2738 ===\n",
      "train loss:1.8436812271772838\n",
      "train loss:1.9313183256602995\n",
      "train loss:2.0030724770213744\n",
      "=== epoch:129, train acc:0.35, test acc:0.2754 ===\n",
      "train loss:2.003081872493936\n",
      "train loss:1.8549712389083746\n",
      "train loss:1.9470826443356068\n",
      "=== epoch:130, train acc:0.3466666666666667, test acc:0.2767 ===\n",
      "train loss:1.9051581210827695\n",
      "train loss:1.8868845653603938\n",
      "train loss:1.829240624816185\n",
      "=== epoch:131, train acc:0.3433333333333333, test acc:0.2746 ===\n",
      "train loss:1.910825146754179\n",
      "train loss:1.9384364108805578\n",
      "train loss:2.0228681574056764\n",
      "=== epoch:132, train acc:0.35, test acc:0.2786 ===\n",
      "train loss:1.8953538435888833\n",
      "train loss:1.9607990469060796\n",
      "train loss:2.0091566762913002\n",
      "=== epoch:133, train acc:0.3566666666666667, test acc:0.2812 ===\n",
      "train loss:1.962847628537649\n",
      "train loss:1.9618288783982158\n",
      "train loss:1.8096956608324504\n",
      "=== epoch:134, train acc:0.3466666666666667, test acc:0.2771 ===\n",
      "train loss:2.0081634972472564\n",
      "train loss:1.9158904747415642\n",
      "train loss:2.0356623808859817\n",
      "=== epoch:135, train acc:0.3466666666666667, test acc:0.2758 ===\n",
      "train loss:1.8160885767325112\n",
      "train loss:1.8724786148932155\n",
      "train loss:1.9551226661543517\n",
      "=== epoch:136, train acc:0.3466666666666667, test acc:0.2697 ===\n",
      "train loss:1.9788615673693124\n",
      "train loss:1.9758512767247771\n",
      "train loss:1.8629633606203653\n",
      "=== epoch:137, train acc:0.3466666666666667, test acc:0.2718 ===\n",
      "train loss:1.9784470558766207\n",
      "train loss:1.913895976843874\n",
      "train loss:1.9458460169741039\n",
      "=== epoch:138, train acc:0.35333333333333333, test acc:0.2757 ===\n",
      "train loss:1.9226052289873068\n",
      "train loss:1.9670885019292987\n",
      "train loss:1.8552365073462864\n",
      "=== epoch:139, train acc:0.35333333333333333, test acc:0.2763 ===\n",
      "train loss:1.9616437534888727\n",
      "train loss:1.9411875364539386\n",
      "train loss:1.9389812454922266\n",
      "=== epoch:140, train acc:0.3433333333333333, test acc:0.2787 ===\n",
      "train loss:1.943608215749951\n",
      "train loss:1.9765464646011572\n",
      "train loss:1.8979038398725854\n",
      "=== epoch:141, train acc:0.35, test acc:0.2827 ===\n",
      "train loss:1.9414638249113745\n",
      "train loss:1.933337461053589\n",
      "train loss:1.9618088400213145\n",
      "=== epoch:142, train acc:0.35, test acc:0.2825 ===\n",
      "train loss:1.9585318792499329\n",
      "train loss:1.9733857478150014\n",
      "train loss:1.894692686475086\n",
      "=== epoch:143, train acc:0.3566666666666667, test acc:0.2833 ===\n",
      "train loss:1.9824335037442504\n",
      "train loss:1.8957477447350641\n",
      "train loss:1.985085290117629\n",
      "=== epoch:144, train acc:0.36, test acc:0.2885 ===\n",
      "train loss:1.8775992121798513\n",
      "train loss:1.9027748363668993\n",
      "train loss:1.8585072900176163\n",
      "=== epoch:145, train acc:0.36333333333333334, test acc:0.2908 ===\n",
      "train loss:1.7809588940646974\n",
      "train loss:1.811701430324745\n",
      "train loss:1.9755311051902154\n",
      "=== epoch:146, train acc:0.36333333333333334, test acc:0.2904 ===\n",
      "train loss:1.746685692653466\n",
      "train loss:1.9240325963061127\n",
      "train loss:1.8602767828800735\n",
      "=== epoch:147, train acc:0.37666666666666665, test acc:0.2939 ===\n",
      "train loss:1.8783749796693048\n",
      "train loss:1.9438882281095797\n",
      "train loss:1.8148286173080408\n",
      "=== epoch:148, train acc:0.38, test acc:0.2918 ===\n",
      "train loss:1.8517011833968204\n",
      "train loss:1.899279505533415\n",
      "train loss:1.8102676137797349\n",
      "=== epoch:149, train acc:0.38, test acc:0.2894 ===\n",
      "train loss:1.867090282702055\n",
      "train loss:1.845199150659826\n",
      "train loss:1.9411539834054985\n",
      "=== epoch:150, train acc:0.38, test acc:0.2935 ===\n",
      "train loss:1.7980838809417838\n",
      "train loss:1.950080607006031\n",
      "train loss:1.8162263756910146\n",
      "=== epoch:151, train acc:0.38, test acc:0.2962 ===\n",
      "train loss:1.9337414751220658\n",
      "train loss:1.9803226509580028\n",
      "train loss:1.916971465882264\n",
      "=== epoch:152, train acc:0.37, test acc:0.2973 ===\n",
      "train loss:1.9294637021948595\n",
      "train loss:1.8687318866015632\n",
      "train loss:1.8127094926314344\n",
      "=== epoch:153, train acc:0.38, test acc:0.3009 ===\n",
      "train loss:1.9424140946679906\n",
      "train loss:1.8835279486829788\n",
      "train loss:1.9102952035297898\n",
      "=== epoch:154, train acc:0.37, test acc:0.3006 ===\n",
      "train loss:1.8322965302450618\n",
      "train loss:1.782900048052647\n",
      "train loss:1.871133834182159\n",
      "=== epoch:155, train acc:0.38333333333333336, test acc:0.3053 ===\n",
      "train loss:1.876154649912278\n",
      "train loss:1.880881434123086\n",
      "train loss:1.7478782196836227\n",
      "=== epoch:156, train acc:0.38, test acc:0.3082 ===\n",
      "train loss:1.7681265946334952\n",
      "train loss:1.8489052955763847\n",
      "train loss:1.8643050768928964\n",
      "=== epoch:157, train acc:0.38666666666666666, test acc:0.3099 ===\n",
      "train loss:1.8671101199134177\n",
      "train loss:1.8771221849528201\n",
      "train loss:1.8999991728840737\n",
      "=== epoch:158, train acc:0.38333333333333336, test acc:0.3091 ===\n",
      "train loss:1.7753978139811426\n",
      "train loss:1.9357730537480324\n",
      "train loss:1.9619056710661658\n",
      "=== epoch:159, train acc:0.38666666666666666, test acc:0.3112 ===\n",
      "train loss:1.8335681447259216\n",
      "train loss:1.9125910113264735\n",
      "train loss:1.87656097539252\n",
      "=== epoch:160, train acc:0.38666666666666666, test acc:0.3131 ===\n",
      "train loss:1.9491278914821846\n",
      "train loss:1.87085948917885\n",
      "train loss:1.815380824039932\n",
      "=== epoch:161, train acc:0.38333333333333336, test acc:0.3129 ===\n",
      "train loss:1.8568117713746215\n",
      "train loss:1.8457212491776374\n",
      "train loss:1.85602216558416\n",
      "=== epoch:162, train acc:0.39, test acc:0.319 ===\n",
      "train loss:1.829137700302236\n",
      "train loss:1.8025896150299037\n",
      "train loss:1.866414404759229\n",
      "=== epoch:163, train acc:0.39666666666666667, test acc:0.3206 ===\n",
      "train loss:1.835228428977326\n",
      "train loss:1.9515569591299475\n",
      "train loss:1.8375726506364787\n",
      "=== epoch:164, train acc:0.3933333333333333, test acc:0.3256 ===\n",
      "train loss:1.839218002658635\n",
      "train loss:1.9467138790672933\n",
      "train loss:1.7875323756570236\n",
      "=== epoch:165, train acc:0.4033333333333333, test acc:0.3245 ===\n",
      "train loss:1.8604875435878017\n",
      "train loss:1.9300484535402398\n",
      "train loss:1.9320027228624448\n",
      "=== epoch:166, train acc:0.41, test acc:0.3309 ===\n",
      "train loss:1.7281773163842624\n",
      "train loss:1.7122970005249791\n",
      "train loss:1.7818346502347302\n",
      "=== epoch:167, train acc:0.41333333333333333, test acc:0.3314 ===\n",
      "train loss:1.7857069839379043\n",
      "train loss:1.8779801437426682\n",
      "train loss:1.7363252347520979\n",
      "=== epoch:168, train acc:0.4066666666666667, test acc:0.3291 ===\n",
      "train loss:1.8322289385753927\n",
      "train loss:1.8535193669431467\n",
      "train loss:1.9252300002114995\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== epoch:169, train acc:0.3933333333333333, test acc:0.3247 ===\n",
      "train loss:1.7785273883489867\n",
      "train loss:1.7601762139198538\n",
      "train loss:1.7668995807871304\n",
      "=== epoch:170, train acc:0.41, test acc:0.3283 ===\n",
      "train loss:1.955907661337071\n",
      "train loss:1.8420334816304753\n",
      "train loss:1.8813158390390172\n",
      "=== epoch:171, train acc:0.4166666666666667, test acc:0.333 ===\n",
      "train loss:1.878493745718377\n",
      "train loss:1.771954352719621\n",
      "train loss:1.792785853971757\n",
      "=== epoch:172, train acc:0.41333333333333333, test acc:0.3336 ===\n",
      "train loss:1.8802397392071297\n",
      "train loss:1.839068894848875\n",
      "train loss:1.957319240881883\n",
      "=== epoch:173, train acc:0.42, test acc:0.3412 ===\n",
      "train loss:1.8438881482471448\n",
      "train loss:1.857305793824296\n",
      "train loss:1.5990632603216741\n",
      "=== epoch:174, train acc:0.41333333333333333, test acc:0.3378 ===\n",
      "train loss:1.7876939281070046\n",
      "train loss:1.736479381560854\n",
      "train loss:1.7392869527136818\n",
      "=== epoch:175, train acc:0.41333333333333333, test acc:0.3365 ===\n",
      "train loss:1.7207082173516843\n",
      "train loss:1.796194276789069\n",
      "train loss:1.8458910350409985\n",
      "=== epoch:176, train acc:0.41, test acc:0.3374 ===\n",
      "train loss:1.7686746116805478\n",
      "train loss:1.8086553828307546\n",
      "train loss:1.8498328552504062\n",
      "=== epoch:177, train acc:0.4166666666666667, test acc:0.343 ===\n",
      "train loss:1.7786300485684663\n",
      "train loss:1.680323508414119\n",
      "train loss:1.75268817906565\n",
      "=== epoch:178, train acc:0.4166666666666667, test acc:0.3403 ===\n",
      "train loss:1.7874723498272098\n",
      "train loss:1.7965881105548538\n",
      "train loss:1.7605181190898243\n",
      "=== epoch:179, train acc:0.4166666666666667, test acc:0.3483 ===\n",
      "train loss:1.7300391780836106\n",
      "train loss:1.8558834562779554\n",
      "train loss:1.7075023677496455\n",
      "=== epoch:180, train acc:0.43333333333333335, test acc:0.3467 ===\n",
      "train loss:1.8113674985613988\n",
      "train loss:1.8109248210839342\n",
      "train loss:1.7746322886838666\n",
      "=== epoch:181, train acc:0.43666666666666665, test acc:0.3552 ===\n",
      "train loss:1.7503130933645188\n",
      "train loss:1.7624362081594622\n",
      "train loss:1.8542884911400213\n",
      "=== epoch:182, train acc:0.43666666666666665, test acc:0.357 ===\n",
      "train loss:1.72692715138191\n",
      "train loss:1.737638758968572\n",
      "train loss:1.7090048537947318\n",
      "=== epoch:183, train acc:0.44, test acc:0.358 ===\n",
      "train loss:1.5549667732484485\n",
      "train loss:1.752184763842617\n",
      "train loss:1.6799683249088662\n",
      "=== epoch:184, train acc:0.44, test acc:0.3603 ===\n",
      "train loss:1.7594851393622903\n",
      "train loss:1.6160873508879474\n",
      "train loss:1.7596243189866274\n",
      "=== epoch:185, train acc:0.44, test acc:0.3637 ===\n",
      "train loss:1.7612561096934198\n",
      "train loss:1.7212700030402959\n",
      "train loss:1.7026595892026186\n",
      "=== epoch:186, train acc:0.44, test acc:0.3665 ===\n",
      "train loss:1.6823349002126435\n",
      "train loss:1.7527902040269996\n",
      "train loss:1.6610738543423529\n",
      "=== epoch:187, train acc:0.44, test acc:0.3658 ===\n",
      "train loss:1.7455845413676203\n",
      "train loss:1.634015919741745\n",
      "train loss:1.676567625437069\n",
      "=== epoch:188, train acc:0.4533333333333333, test acc:0.3709 ===\n",
      "train loss:1.6843497882581473\n",
      "train loss:1.7888618996992443\n",
      "train loss:1.7677089833241346\n",
      "=== epoch:189, train acc:0.45666666666666667, test acc:0.3722 ===\n",
      "train loss:1.7382788202833805\n",
      "train loss:1.7590140332544126\n",
      "train loss:1.6834136763036414\n",
      "=== epoch:190, train acc:0.45, test acc:0.3712 ===\n",
      "train loss:1.7272837438353466\n",
      "train loss:1.6646576062208058\n",
      "train loss:1.5256543251478383\n",
      "=== epoch:191, train acc:0.45, test acc:0.3701 ===\n",
      "train loss:1.670468018129953\n",
      "train loss:1.7156344900527976\n",
      "train loss:1.7506542713997582\n",
      "=== epoch:192, train acc:0.44333333333333336, test acc:0.3669 ===\n",
      "train loss:1.7725499491138799\n",
      "train loss:1.6298065779738227\n",
      "train loss:1.6461901316765923\n",
      "=== epoch:193, train acc:0.44333333333333336, test acc:0.3678 ===\n",
      "train loss:1.7648366994720097\n",
      "train loss:1.7032811454515455\n",
      "train loss:1.7549041490591675\n",
      "=== epoch:194, train acc:0.44666666666666666, test acc:0.3698 ===\n",
      "train loss:1.6279189849484235\n",
      "train loss:1.743519259022007\n",
      "train loss:1.7328635999900164\n",
      "=== epoch:195, train acc:0.44666666666666666, test acc:0.3694 ===\n",
      "train loss:1.6938468933764725\n",
      "train loss:1.696880223520902\n",
      "train loss:1.6984169046224653\n",
      "=== epoch:196, train acc:0.45, test acc:0.372 ===\n",
      "train loss:1.7979117397528128\n",
      "train loss:1.657850995980607\n",
      "train loss:1.6406231398777487\n",
      "=== epoch:197, train acc:0.4533333333333333, test acc:0.3738 ===\n",
      "train loss:1.7156965638547228\n",
      "train loss:1.6094605130412207\n",
      "train loss:1.779605110045336\n",
      "=== epoch:198, train acc:0.45666666666666667, test acc:0.3743 ===\n",
      "train loss:1.795854433175214\n",
      "train loss:1.7388667569073033\n",
      "train loss:1.727819527610982\n",
      "=== epoch:199, train acc:0.45666666666666667, test acc:0.3765 ===\n",
      "train loss:1.802926637074317\n",
      "train loss:1.7615337914974873\n",
      "train loss:1.7357657602233474\n",
      "=== epoch:200, train acc:0.47, test acc:0.3817 ===\n",
      "train loss:1.6626976606799642\n",
      "train loss:1.6079177411437695\n",
      "train loss:1.7058312155351587\n",
      "=== epoch:201, train acc:0.47, test acc:0.3832 ===\n",
      "train loss:1.665101588410026\n",
      "train loss:1.7286044612707736\n",
      "train loss:1.700110339050581\n",
      "=== epoch:202, train acc:0.4766666666666667, test acc:0.3842 ===\n",
      "train loss:1.7079093431445855\n",
      "train loss:1.6477463466561781\n",
      "train loss:1.6366993029793733\n",
      "=== epoch:203, train acc:0.4766666666666667, test acc:0.387 ===\n",
      "train loss:1.6918191076472546\n",
      "train loss:1.631200005466953\n",
      "train loss:1.6839370090982764\n",
      "=== epoch:204, train acc:0.48, test acc:0.3853 ===\n",
      "train loss:1.706278433594608\n",
      "train loss:1.596910817838531\n",
      "train loss:1.6712001514697\n",
      "=== epoch:205, train acc:0.5, test acc:0.3917 ===\n",
      "train loss:1.631616972495885\n",
      "train loss:1.5734184464718661\n",
      "train loss:1.6509913847727686\n",
      "=== epoch:206, train acc:0.49, test acc:0.3924 ===\n",
      "train loss:1.6428858571276823\n",
      "train loss:1.6997298590345804\n",
      "train loss:1.749919679730983\n",
      "=== epoch:207, train acc:0.5066666666666667, test acc:0.3991 ===\n",
      "train loss:1.6288196775865338\n",
      "train loss:1.6574319144446832\n",
      "train loss:1.6864214867951908\n",
      "=== epoch:208, train acc:0.5166666666666667, test acc:0.4002 ===\n",
      "train loss:1.6588321863033604\n",
      "train loss:1.7351419155207677\n",
      "train loss:1.6992223784349718\n",
      "=== epoch:209, train acc:0.5166666666666667, test acc:0.4032 ===\n",
      "train loss:1.6777888037549369\n",
      "train loss:1.599169542176418\n",
      "train loss:1.6516213369108563\n",
      "=== epoch:210, train acc:0.52, test acc:0.4064 ===\n",
      "train loss:1.6997173227022742\n",
      "train loss:1.6258120151905688\n",
      "train loss:1.6712782896386165\n",
      "=== epoch:211, train acc:0.52, test acc:0.4089 ===\n",
      "train loss:1.6550649926185403\n",
      "train loss:1.5727478303583178\n",
      "train loss:1.6762819151271031\n",
      "=== epoch:212, train acc:0.52, test acc:0.4093 ===\n",
      "train loss:1.5672100972649403\n",
      "train loss:1.6282865543267255\n",
      "train loss:1.4855616785053305\n",
      "=== epoch:213, train acc:0.5266666666666666, test acc:0.4109 ===\n",
      "train loss:1.5333482565292593\n",
      "train loss:1.654550725659757\n",
      "train loss:1.7189386072402726\n",
      "=== epoch:214, train acc:0.5266666666666666, test acc:0.4141 ===\n",
      "train loss:1.7022872106474733\n",
      "train loss:1.6303230927518373\n",
      "train loss:1.4788787526086926\n",
      "=== epoch:215, train acc:0.54, test acc:0.4165 ===\n",
      "train loss:1.6243587056059556\n",
      "train loss:1.619035115022088\n",
      "train loss:1.6255316808722744\n",
      "=== epoch:216, train acc:0.5366666666666666, test acc:0.419 ===\n",
      "train loss:1.681061215470631\n",
      "train loss:1.447163252152802\n",
      "train loss:1.5656621167490257\n",
      "=== epoch:217, train acc:0.54, test acc:0.4202 ===\n",
      "train loss:1.5809050363030437\n",
      "train loss:1.6223719629097553\n",
      "train loss:1.5627400448031687\n",
      "=== epoch:218, train acc:0.55, test acc:0.4222 ===\n",
      "train loss:1.515948338382405\n",
      "train loss:1.6638463304002857\n",
      "train loss:1.4473775523983141\n",
      "=== epoch:219, train acc:0.5466666666666666, test acc:0.4212 ===\n",
      "train loss:1.5744790129004786\n",
      "train loss:1.6269897600507084\n",
      "train loss:1.5888225171955128\n",
      "=== epoch:220, train acc:0.55, test acc:0.4243 ===\n",
      "train loss:1.505855661609657\n",
      "train loss:1.5251216133699608\n",
      "train loss:1.512519210803584\n",
      "=== epoch:221, train acc:0.54, test acc:0.423 ===\n",
      "train loss:1.6097303196567645\n",
      "train loss:1.5296713229217058\n",
      "train loss:1.4717269337457282\n",
      "=== epoch:222, train acc:0.55, test acc:0.4265 ===\n",
      "train loss:1.5918062465798761\n",
      "train loss:1.6182056206242457\n",
      "train loss:1.4615667483526102\n",
      "=== epoch:223, train acc:0.5433333333333333, test acc:0.428 ===\n",
      "train loss:1.4629014450444402\n",
      "train loss:1.556291030765759\n",
      "train loss:1.570539836290929\n",
      "=== epoch:224, train acc:0.5566666666666666, test acc:0.4304 ===\n",
      "train loss:1.4699622774511942\n",
      "train loss:1.543908610150491\n",
      "train loss:1.5526367906481664\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== epoch:225, train acc:0.56, test acc:0.4358 ===\n",
      "train loss:1.3915568324175798\n",
      "train loss:1.5236033179585453\n",
      "train loss:1.513853185728192\n",
      "=== epoch:226, train acc:0.56, test acc:0.4397 ===\n",
      "train loss:1.4874375300861329\n",
      "train loss:1.5646918843372486\n",
      "train loss:1.559151265447551\n",
      "=== epoch:227, train acc:0.5766666666666667, test acc:0.4466 ===\n",
      "train loss:1.515809836773829\n",
      "train loss:1.3266842452782823\n",
      "train loss:1.5150887770709118\n",
      "=== epoch:228, train acc:0.57, test acc:0.4418 ===\n",
      "train loss:1.5586866995394277\n",
      "train loss:1.5967389353194408\n",
      "train loss:1.4475426822853907\n",
      "=== epoch:229, train acc:0.57, test acc:0.4434 ===\n",
      "train loss:1.4128778146302432\n",
      "train loss:1.6640803488487372\n",
      "train loss:1.482440026712454\n",
      "=== epoch:230, train acc:0.5666666666666667, test acc:0.4428 ===\n",
      "train loss:1.5889948413834893\n",
      "train loss:1.5709922124286695\n",
      "train loss:1.4964897146655192\n",
      "=== epoch:231, train acc:0.5833333333333334, test acc:0.4496 ===\n",
      "train loss:1.5601211300809552\n",
      "train loss:1.458404260787435\n",
      "train loss:1.516974762128761\n",
      "=== epoch:232, train acc:0.5966666666666667, test acc:0.4558 ===\n",
      "train loss:1.6006153245032622\n",
      "train loss:1.4987694587502636\n",
      "train loss:1.665009383932876\n",
      "=== epoch:233, train acc:0.5933333333333334, test acc:0.4619 ===\n",
      "train loss:1.5219772273580634\n",
      "train loss:1.5938970682895925\n",
      "train loss:1.6390632653149584\n",
      "=== epoch:234, train acc:0.5933333333333334, test acc:0.4636 ===\n",
      "train loss:1.5031639124515923\n",
      "train loss:1.542843774096454\n",
      "train loss:1.6390134155981233\n",
      "=== epoch:235, train acc:0.5866666666666667, test acc:0.4639 ===\n",
      "train loss:1.4275915158600305\n",
      "train loss:1.496671180361303\n",
      "train loss:1.4137906044521917\n",
      "=== epoch:236, train acc:0.5866666666666667, test acc:0.4619 ===\n",
      "train loss:1.3918030304933395\n",
      "train loss:1.467177207828499\n",
      "train loss:1.435517587883605\n",
      "=== epoch:237, train acc:0.5833333333333334, test acc:0.4644 ===\n",
      "train loss:1.5132949131061957\n",
      "train loss:1.5014968913127285\n",
      "train loss:1.4438889817255134\n",
      "=== epoch:238, train acc:0.5766666666666667, test acc:0.4653 ===\n",
      "train loss:1.4182297895897937\n",
      "train loss:1.5430304096320153\n",
      "train loss:1.4724311410527497\n",
      "=== epoch:239, train acc:0.58, test acc:0.469 ===\n",
      "train loss:1.4629030817848077\n",
      "train loss:1.4668987435976784\n",
      "train loss:1.499837657171305\n",
      "=== epoch:240, train acc:0.58, test acc:0.4716 ===\n",
      "train loss:1.4348451349204898\n",
      "train loss:1.5594010089547672\n",
      "train loss:1.4740783979231387\n",
      "=== epoch:241, train acc:0.5833333333333334, test acc:0.4736 ===\n",
      "train loss:1.5611195212095947\n",
      "train loss:1.4153413144250724\n",
      "train loss:1.4229844211180704\n",
      "=== epoch:242, train acc:0.58, test acc:0.4739 ===\n",
      "train loss:1.485478410915989\n",
      "train loss:1.340948171960759\n",
      "train loss:1.337648383583049\n",
      "=== epoch:243, train acc:0.5833333333333334, test acc:0.4759 ===\n",
      "train loss:1.3763812700493228\n",
      "train loss:1.4454289035002446\n",
      "train loss:1.366977343741677\n",
      "=== epoch:244, train acc:0.5866666666666667, test acc:0.4764 ===\n",
      "train loss:1.3376050976973648\n",
      "train loss:1.3108404745386348\n",
      "train loss:1.5333346155711904\n",
      "=== epoch:245, train acc:0.5866666666666667, test acc:0.4782 ===\n",
      "train loss:1.4470663109678243\n",
      "train loss:1.5321354331184653\n",
      "train loss:1.4157747067420217\n",
      "=== epoch:246, train acc:0.5866666666666667, test acc:0.4819 ===\n",
      "train loss:1.5395666381831266\n",
      "train loss:1.4653344479868085\n",
      "train loss:1.4047591200427518\n",
      "=== epoch:247, train acc:0.61, test acc:0.4862 ===\n",
      "train loss:1.436371962741445\n",
      "train loss:1.3770157827064928\n",
      "train loss:1.3671637263935503\n",
      "=== epoch:248, train acc:0.5933333333333334, test acc:0.486 ===\n",
      "train loss:1.3874159556284082\n",
      "train loss:1.3900724793712171\n",
      "train loss:1.2366704132590303\n",
      "=== epoch:249, train acc:0.5933333333333334, test acc:0.4871 ===\n",
      "train loss:1.4150924920267522\n",
      "train loss:1.478823155819215\n",
      "train loss:1.2910929108354128\n",
      "=== epoch:250, train acc:0.5966666666666667, test acc:0.4868 ===\n",
      "train loss:1.3762525807984316\n",
      "train loss:1.2917178991226044\n",
      "train loss:1.4211240841799104\n",
      "=== epoch:251, train acc:0.6166666666666667, test acc:0.4906 ===\n",
      "train loss:1.2700917302251797\n",
      "train loss:1.3352321580504845\n",
      "train loss:1.348157167556376\n",
      "=== epoch:252, train acc:0.6, test acc:0.4877 ===\n",
      "train loss:1.2703944547272066\n",
      "train loss:1.3375772611906225\n",
      "train loss:1.3036106126181963\n",
      "=== epoch:253, train acc:0.5966666666666667, test acc:0.4881 ===\n",
      "train loss:1.5277021309300585\n",
      "train loss:1.3839089009173906\n",
      "train loss:1.4609979942220677\n",
      "=== epoch:254, train acc:0.5966666666666667, test acc:0.4912 ===\n",
      "train loss:1.3872063337853264\n",
      "train loss:1.4605950343665686\n",
      "train loss:1.3289874416196232\n",
      "=== epoch:255, train acc:0.6033333333333334, test acc:0.4922 ===\n",
      "train loss:1.4036938966830146\n",
      "train loss:1.3670736806040713\n",
      "train loss:1.4945944588644402\n",
      "=== epoch:256, train acc:0.5933333333333334, test acc:0.4925 ===\n",
      "train loss:1.4748343738962142\n",
      "train loss:1.3175399399505363\n",
      "train loss:1.319413753431804\n",
      "=== epoch:257, train acc:0.6, test acc:0.4939 ===\n",
      "train loss:1.3384108598291036\n",
      "train loss:1.2836557800603772\n",
      "train loss:1.360346831420225\n",
      "=== epoch:258, train acc:0.5966666666666667, test acc:0.4946 ===\n",
      "train loss:1.3686988856484006\n",
      "train loss:1.3594253099379634\n",
      "train loss:1.4163299131054554\n",
      "=== epoch:259, train acc:0.5966666666666667, test acc:0.4958 ===\n",
      "train loss:1.3550529950344241\n",
      "train loss:1.4499821302779197\n",
      "train loss:1.439338884235315\n",
      "=== epoch:260, train acc:0.6033333333333334, test acc:0.4988 ===\n",
      "train loss:1.2302222100537186\n",
      "train loss:1.3039361354469186\n",
      "train loss:1.4557030829874251\n",
      "=== epoch:261, train acc:0.6, test acc:0.4998 ===\n",
      "train loss:1.4185231974806578\n",
      "train loss:1.2682503183650764\n",
      "train loss:1.271668298158923\n",
      "=== epoch:262, train acc:0.6, test acc:0.5007 ===\n",
      "train loss:1.3776206078065543\n",
      "train loss:1.2386752914875248\n",
      "train loss:1.3704588352805902\n",
      "=== epoch:263, train acc:0.6033333333333334, test acc:0.5016 ===\n",
      "train loss:1.4715216560647997\n",
      "train loss:1.2763408110061025\n",
      "train loss:1.4054438346783464\n",
      "=== epoch:264, train acc:0.6066666666666667, test acc:0.5027 ===\n",
      "train loss:1.356635172695922\n",
      "train loss:1.255036909879539\n",
      "train loss:1.2923308061465997\n",
      "=== epoch:265, train acc:0.6133333333333333, test acc:0.5055 ===\n",
      "train loss:1.464906051696959\n",
      "train loss:1.2179322860072914\n",
      "train loss:1.1297570407201392\n",
      "=== epoch:266, train acc:0.6133333333333333, test acc:0.5045 ===\n",
      "train loss:1.2673603786754635\n",
      "train loss:1.2311516529354958\n",
      "train loss:1.3665695339892359\n",
      "=== epoch:267, train acc:0.61, test acc:0.5067 ===\n",
      "train loss:1.2247221739070406\n",
      "train loss:1.1244548844045976\n",
      "train loss:1.175329907054143\n",
      "=== epoch:268, train acc:0.62, test acc:0.5079 ===\n",
      "train loss:1.3154183704847917\n",
      "train loss:1.2789860606283956\n",
      "train loss:1.3844320379143369\n",
      "=== epoch:269, train acc:0.62, test acc:0.513 ===\n",
      "train loss:1.1857382460931536\n",
      "train loss:1.1099607561431564\n",
      "train loss:1.3167777735466724\n",
      "=== epoch:270, train acc:0.6166666666666667, test acc:0.515 ===\n",
      "train loss:1.1340919300195413\n",
      "train loss:1.3884278858627337\n",
      "train loss:1.347506824191168\n",
      "=== epoch:271, train acc:0.6166666666666667, test acc:0.5167 ===\n",
      "train loss:1.2552424588212496\n",
      "train loss:1.314832041836045\n",
      "train loss:1.3718947906848147\n",
      "=== epoch:272, train acc:0.62, test acc:0.5186 ===\n",
      "train loss:1.218469927153287\n",
      "train loss:1.296583131729483\n",
      "train loss:1.2343633329605692\n",
      "=== epoch:273, train acc:0.6133333333333333, test acc:0.516 ===\n",
      "train loss:1.3387190498922539\n",
      "train loss:1.248826263963914\n",
      "train loss:1.213608537917425\n",
      "=== epoch:274, train acc:0.6133333333333333, test acc:0.5183 ===\n",
      "train loss:1.1828693704097848\n",
      "train loss:1.2493329247719007\n",
      "train loss:1.3106784778694112\n",
      "=== epoch:275, train acc:0.6166666666666667, test acc:0.5185 ===\n",
      "train loss:1.2883308768761392\n",
      "train loss:1.2685763709974927\n",
      "train loss:1.2155436449846624\n",
      "=== epoch:276, train acc:0.6133333333333333, test acc:0.521 ===\n",
      "train loss:1.2406147376566945\n",
      "train loss:1.0587230895702158\n",
      "train loss:1.2867389408999677\n",
      "=== epoch:277, train acc:0.6166666666666667, test acc:0.5241 ===\n",
      "train loss:1.3277351376855477\n",
      "train loss:1.199827258970863\n",
      "train loss:1.161425595208553\n",
      "=== epoch:278, train acc:0.6166666666666667, test acc:0.524 ===\n",
      "train loss:1.1206773713248126\n",
      "train loss:1.2102704586904647\n",
      "train loss:1.0475020165733415\n",
      "=== epoch:279, train acc:0.62, test acc:0.522 ===\n",
      "train loss:1.2195651841622321\n",
      "train loss:1.2934161398686883\n",
      "train loss:1.1923658007983287\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== epoch:280, train acc:0.6233333333333333, test acc:0.5234 ===\n",
      "train loss:1.2112655434996287\n",
      "train loss:1.060208829906273\n",
      "train loss:1.3433193033990072\n",
      "=== epoch:281, train acc:0.6233333333333333, test acc:0.5232 ===\n",
      "train loss:1.2268317822981356\n",
      "train loss:1.135710370235763\n",
      "train loss:1.2192471441468322\n",
      "=== epoch:282, train acc:0.6266666666666667, test acc:0.5214 ===\n",
      "train loss:1.1411863519089953\n",
      "train loss:1.2012004170692332\n",
      "train loss:1.1702574955731435\n",
      "=== epoch:283, train acc:0.6266666666666667, test acc:0.5229 ===\n",
      "train loss:1.2612216922438784\n",
      "train loss:1.2184325672540803\n",
      "train loss:1.3301222167953264\n",
      "=== epoch:284, train acc:0.6266666666666667, test acc:0.5292 ===\n",
      "train loss:1.246122634567428\n",
      "train loss:1.1611204939214839\n",
      "train loss:1.1267010839293432\n",
      "=== epoch:285, train acc:0.6266666666666667, test acc:0.5294 ===\n",
      "train loss:1.162717609731534\n",
      "train loss:1.1510134056014036\n",
      "train loss:1.124095591778488\n",
      "=== epoch:286, train acc:0.6266666666666667, test acc:0.529 ===\n",
      "train loss:1.247062416487153\n",
      "train loss:1.136062312029753\n",
      "train loss:1.1922837374618185\n",
      "=== epoch:287, train acc:0.6266666666666667, test acc:0.5324 ===\n",
      "train loss:1.153388544247697\n",
      "train loss:1.1388577103700168\n",
      "train loss:1.0774078004313292\n",
      "=== epoch:288, train acc:0.63, test acc:0.5317 ===\n",
      "train loss:1.224730216962432\n",
      "train loss:1.1678214092134858\n",
      "train loss:1.2066246355846295\n",
      "=== epoch:289, train acc:0.63, test acc:0.5333 ===\n",
      "train loss:1.1114502234959622\n",
      "train loss:1.1443012041154954\n",
      "train loss:1.130939080427581\n",
      "=== epoch:290, train acc:0.63, test acc:0.5339 ===\n",
      "train loss:1.072389700861621\n",
      "train loss:1.0047136703939574\n",
      "train loss:1.1625758156075554\n",
      "=== epoch:291, train acc:0.63, test acc:0.5303 ===\n",
      "train loss:1.0949011143399217\n",
      "train loss:1.1937736408540178\n",
      "train loss:1.1449807005519355\n",
      "=== epoch:292, train acc:0.6266666666666667, test acc:0.5352 ===\n",
      "train loss:1.1269748240988249\n",
      "train loss:1.0704854240818875\n",
      "train loss:1.0165935388471357\n",
      "=== epoch:293, train acc:0.63, test acc:0.5339 ===\n",
      "train loss:1.1541483196843954\n",
      "train loss:1.049943633969688\n",
      "train loss:1.001096882433819\n",
      "=== epoch:294, train acc:0.63, test acc:0.5363 ===\n",
      "train loss:1.0190808256262047\n",
      "train loss:1.1017624542586597\n",
      "train loss:1.044710357649188\n",
      "=== epoch:295, train acc:0.6333333333333333, test acc:0.5361 ===\n",
      "train loss:1.0811262542309188\n",
      "train loss:1.0803451176647245\n",
      "train loss:1.0834570078417736\n",
      "=== epoch:296, train acc:0.63, test acc:0.5349 ===\n",
      "train loss:0.9818517598882388\n",
      "train loss:0.9737795170764106\n",
      "train loss:1.1611645509834323\n",
      "=== epoch:297, train acc:0.63, test acc:0.5377 ===\n",
      "train loss:1.2918743900813752\n",
      "train loss:1.1017927125805096\n",
      "train loss:1.0335963490345703\n",
      "=== epoch:298, train acc:0.6366666666666667, test acc:0.5379 ===\n",
      "train loss:0.9557445942255963\n",
      "train loss:0.8615832841549604\n",
      "train loss:1.0687554307913316\n",
      "=== epoch:299, train acc:0.6333333333333333, test acc:0.537 ===\n",
      "train loss:1.1318963998633813\n",
      "train loss:1.1073150926448123\n",
      "train loss:1.0318033062300909\n",
      "=== epoch:300, train acc:0.6333333333333333, test acc:0.5383 ===\n",
      "train loss:1.1308597943759002\n",
      "train loss:1.0816858881973657\n",
      "train loss:1.0179595265764505\n",
      "=== epoch:301, train acc:0.63, test acc:0.5385 ===\n",
      "train loss:1.2536230009560327\n",
      "train loss:1.1767894417799762\n",
      "=============== Final Test Accuracy ===============\n",
      "test acc:0.5412\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VOXZx/HvnYUkkBiWsAdkEVlUBMEdFdxYtKKtdaut\ntVqsS9W+FpfWutS2Ymm12lbRKr71dW0Bd2RzQ1FkB9n3JQlCWBIIJCHL8/4xk2FIZiaTkMlMkt/n\nunI5c84zZ+7juTj3nGc15xwiIiIAcdEOQEREYoeSgoiI+CgpiIiIj5KCiIj4KCmIiIiPkoKIiPhE\nLCmY2UQz22lmy4PsNzN7xszWm9kyMzslUrGIiEh4Ivmk8L/AiBD7RwK9vH9jgOciGIuIiIQhYknB\nOTcb2BOiyGjgFecxF2hpZh0jFY+IiFQvIYrf3RnY5vc+y7tte+WCZjYGz9MELVq0GNSnT596CVBE\npLFYuHDhLudc2+rKRTMphM059wLwAsDgwYPdggULohyRiEjDYmZbwikXzd5H2UAXv/eZ3m0iIhIl\n0UwK7wE/8fZCOgPId85VqToSEZH6E7HqIzN7AxgKZJhZFvAwkAjgnJsATAVGAeuBg8CNkYpFRETC\nE7Gk4Jy7tpr9Drg9Ut8vIiI1pxHNIiLio6QgIiI+SgoiIuKjpCAiIj5KCiIi4qOkICIiPkoKIiLi\no6QgIiI+SgoiIuKjpCAiIj5KCiIi4qOkICIiPkoKIiLio6QgIiI+SgoiIuKjpCAiIj5KCiIi4qOk\nICIiPkoKIiLio6QgIiI+SgoiIuKjpCAiIj5KCiIi4qOkICIiPkoKIiLio6QgIiI+SgoiIuKjpCAi\nIj5KCiIi4qOkICIiPkoKIiLio6QgIiI+SgoiIuKjpCAiIj5KCiIi4hPRpGBmI8xsjZmtN7P7A+xP\nN7P3zWypma0wsxsjGY+IiIQWsaRgZvHAP4GRQD/gWjPrV6nY7cBK59zJwFDgr2bWLFIxiYhIaJF8\nUjgNWO+c2+icOwS8CYyuVMYBaWZmQCqwByiNYEwiIhJCJJNCZ2Cb3/ss7zZ//wD6AjnAt8Bdzrny\nygcyszFmtsDMFuTm5kYqXhGRJi/aDc3DgSVAJ2AA8A8zO6ZyIefcC865wc65wW3btq3vGEVEmoxI\nJoVsoIvf+0zvNn83AlOcx3pgE9AngjGJiEgIkUwK84FeZtbd23h8DfBepTJbgQsAzKw90BvYGMGY\nREQkhIRIHdg5V2pmdwDTgXhgonNuhZn9wrt/AvAY8L9m9i1gwH3OuV2RiklEREKLWFIAcM5NBaZW\n2jbB73UOcHEkYxARkfBFu6FZRERiiJKCiIj4KCmIiIiPkoKIiPgoKYiIiI+SgoiI+CgpiIiIj5KC\niIj4KCmIiIiPkoKIiPgoKYiIiI+SgoiI+CgpiIiIj5KCiIj4KCmIiIiPkoKIiPgoKYiIiI+SgoiI\n+CgpiIiIj5KCiIj4KCmIiIiPkoKIiPgoKYiIiI+SgoiI+CgpiIiIj5KCiIj4KCmIiIiPkoKIiPgk\nRDsAEZGm6p3F2YyfvoacvEI6tUxh7PDeXD6ws2//vE176J7Rgjnrd4UsV5eUFERE6lB1N3r/cg9M\n+ZbCkjIAsvMKuX/KMt5elMWY83pSUlbOT1+eT4tm8RSVlFPmnK/cA1O+BYhIYjDn/aKGYvDgwW7B\nggXRDkNEpIrKN3qApIQ4bhvWk/OObwfA+p0FPDVzLTl5hYS6+ybEGd0yWrBtz0GKS8ur7O/cMoU5\n958fdmxmttA5N7i6cnpSEJFGKdxf7HV5vPHT1xyREACKS8t5auY6npq5zretf2Y62XmFAb/HgLEj\nepN3sITrTz+W88Z/GrBcTpDPHy0lBRFpdAJVzdS2ymXB5j3cN2kZm3YfoNz7076iqgfgspM7ERdn\nFJeWBb3RA7z801MBaJYQx2ndWzN0/GcBy3dqmcJtQ4874n2wcpGg3kci0ugE+sVeWFLG+Olrwvp8\nUUkZH6/aweKte3n8o9Vs9EsIh8uU89u3v+WkR6Zz66sL6fu7aUGP17llCsP6tGNYn3acfVwGifFx\njB3em5TE+CPKpSTGM3Z47yO2hVuurkT0ScHMRgBPA/HAi865cQHKDAX+BiQCu5xz50UyJhFpvD5f\nmwsEr1oJtL1ytdCvLz6eD5Zt5+PVO6v9vgOHyujTIY2Pln/HlYMyAfhgWQ5FJYfbAILdwCueWKqr\nkgq3XF2JWEOzmcUDa4GLgCxgPnCtc26lX5mWwFfACOfcVjNr55wLeSXU0CwiwVzw188oLi3HOUd2\nXlGV/Z1bJvPxPUNJ9v7y9lQzLaPQ7yaeEGeUljvuueh4ikrLmL95L9v2HGR7ftXjtT8mibkPXMCO\nfcV0SE/2HbO+buA1EQsNzacB651zG70BvQmMBlb6lbkOmOKc2wpQXUIQEQmmoLiUjbsO4Bz8ZlQf\nxk9fQ0nZ4R+9KYlxpKck8r2/f8mLNwwmJ6+I8dNXH5EQAErLHc2bxXP7sOOIizMgcK+ilMR4HhjZ\nFzPzJQTw/LKPhSRQW5FMCp2BbX7vs4DTK5U5Hkg0s8+ANOBp59wrlQ9kZmOAMQBdu3aNSLAiEhue\n+XgdPdq24NL+nWr0uZU5+6io+IiPi6NPhzSWZ+/zdfvMSEti5fb9AIx8+gsOHioLfCCg8FCZLyFA\n/VfhRFO0ex8lAIOAC4AU4Gszm+ucW+tfyDn3AvACeKqP6j1KEYm4dxZn86epq9i5vxiAX725hP+5\nuDe3Du1ZpVygm/O32fkAZKQ24w8frsQ5uOXcHvzqouN5cuZavli3ixvPbs+X63axPreAqwZnMmVR\nNqWVW5AJ3LOnoT8BhCuspGBmU4CXgI+cc1VHUQSWDXTxe5/p3eYvC9jtnDsAHDCz2cDJeNoiRKSJ\nCFQ9U+YcT0xbzZ4DxYw6qSMDu7YK2NX03klL+XBZDtv3FdH+mCSeunoAn6/N5eTMlow8sQNmxm9G\n9fUdd92O/WzefZCL+rXnrJ4ZAauFItWzpyEIq6HZzC4EbgTOAP4LvOycC9m3y8wS8NzcL8CTDOYD\n1znnVviV6Qv8AxgONAPmAdc455YHO64amkUan7PHfRKwL35ivKfR14A/XnESf/9kHTkBGpAB4uOM\n607rymOXn1ij747VhuG6VqcNzc65WcAsM0sHrvW+3gb8C3jVOVcS4DOlZnYHMB1Pl9SJzrkVZvYL\n7/4JzrlVZjYNWAaU4+m2GjQhiEjjsmRbHse3Tw3ahbS0zLHskYu57bVFPPzuCg6VBa6oMGDDn0bV\nKoamUi0UrrDbFMysDXA98GNgMfAaMAS4ARga6DPOuanA1ErbJlR6Px4YX5OgRaTh27GviO8/O4cL\n+7anU8vkgF1IO7VMIS05kb9dPYCRT3/BngOHwm4DkNoJa0Szmb0NfAE0B77nnLvMOfeWc+6XQGok\nAxSRxunztbmUO5ixcgf7i0qr7Pev22+TmsQnvx7KEz84qV5H9zZF4T4pPOOcCzgrUzh1VCIilX2+\nNpd2aUnccFY3Fm7Zyyerd5KalMCB4tKAdfupSQn8YFAX4uPimkQbQLSEmxT6mdli51wegJm1wjM6\n+dnIhSYijVF2XiF/nb6GT1bt5NL+Hbl9mGfyt+XZ+fRo24LmzULfltQGEFnhToj384qEAOCc2wv8\nPDIhiUhjlX+whO/9/UumrfiOfp2O4ZrTDg9GPbFzerUJQSIv3CsQb2bmvP1XvfMaNYtcWCLSkJWU\nlfP6N1s5vUdrJny2gW827WF7fhFpyQnsLypl8q1nMujY1tEOUwIINylMA94ys+e972/xbhMRqWLS\nwiwefm9Fle37i0oxYOvug0oK4RrfCw4EmBauRTsYu67q9qMUblK4D08iuNX7fibwYp1HIyINlnOO\nuRv30D8znRdmb6RPhzS27jlYZY4hB/xlxlquOCUzOoHGinBv9oHKhNp+lMIdvFYOPOf9ExGp4uNV\nO7n5lcOzDUy4fhC3vrowYNlILSXZoIS62c9/EZa8DvmVZwaKvHDnPuoFPA70A3xzxDrnekQoLhFp\nYN5Z4rmB9c9M56Yh3RlxYod6X0oy6oL++m8Ldy6GfTkw+y9ANdMLfXgPtOoGvS6Exa9GItKgwq0+\nehl4GHgKGIZnHiQt5SnSRPnPF9QxPZmzjstg1qod/PiMY4+Ye2js8N6NY8K5o67qyYXHvdVlzdKg\nWYvQ33fbXGjVHRKTYzYppDjnPvb2QNoCPGJmC4GHIhibiMSgyjOV5uQXMWlhFnEGPxx8ZDtBo1mH\nIFRVT1kpzHse5lZTu37RY1B2CAb8CI7pCI+kBy/brm/wfREWblIoNrM4YJ13krtsNL2FSJNTVu54\n7IOVR/zyr9AhPZn+mS2rbG/0g82ePQN2r4Pu50L+tuDlzr6zdsdv0S74U0oEhJsU7sIz79GdwGN4\nqpBuiEhEIhKzXvtmC7sPHAq4b3uQKa0bvRYZcMFD0Pd78GjVpBj8c2He7CPQ7TSUapOCd6Da1c65\nXwMFeNoTRKSJKSkr5/nPN5IYb0esfVyhQTYeV9dWUF4OOYtDH+NntRyyVc83+3BVmxScc2VmNqQ+\nghGR2PXxqp1k5xVy05DuvP7N1obfeAyh2wqePxcOHfRUDYWrnqt6IiHc6qPFZvYenlXXDlRsdM5N\niUhUIhIzcvIKmfjlJnYVFJOWnMADI/twUuf02G48rotRwAkpkNgchvwKZj4EB3cFPp6/GP31XxPh\nJoVkYDdwvt82BygpiDRykxdm8eKXmwAYdVIHEuLjYr/xONQTQHkZ7FgBW+aEPsZN0w+/Hvijuost\nxoU7olntCCJN1Pwte32vzzu+bRQjqSO/b0O1g8easHBHNL9MgP+Lzrmf1XlEIhIzysodi7bspU+H\nNPIOljCsTwzXjTsH8/4F274JXe7csZ7Rwj2HwZPRGw8Qq8KtPvrA73UycAWQU/fhiEgsWf3dPgqK\nS/nFeT1jo7oo1DQSx10ES1+HY6qZaO/83/p9ruE3DNe1cKuPJvu/N7M3gC8jEpGIRI3/9BXNk+Jp\n0SyehDjj9B4xMs11qGkklr4OQx+A8+4Lf7xAI2gYrmu1XeaoF9B0U6lII+SZvmIZhSXlABwoLuNg\ncRm3nNeDjukNYAzCVa9Av9Ge13oCqLVw2xT2c2Sbwnd41lgQkUZi/PQ1voRQwQHvL93O/SNjoO59\nz8bQ+ysSAugJ4CiEW32UFulARCS6gq1xEJW1D9bPgoX/C3lbof1JsH0J7Fhe/3E0QeE+KVwBfOKc\ny/e+bwkMdc69E8ngROToFRSXMmnhNv41e1PQwWZFJWW0atGMPQHmNaqX6SuCNSDHJ0LuGkjPhAsf\ngVmPRD6WJi7cNoWHnXNvV7xxzuWZ2cOAkoJIDFuWlcf1L37DvqJS37bsvEIemPIt4JnBdNX2fdz5\nxuKACaHepq8I1oBcVgIPZkNcIsTFwdfPqq0gwsJNCoEW1KltI7WI1JM/fLiKguLSKtsLS8p4Ytpq\n9hw4xLiPVpPePJGnrxnAd/mFvPL11vqbvqK83HOzDyUh6fBrtRVEXLg39gVm9iTwT+/724HAi6+K\nSL3x70Ja+Saek1fIvE17gn52e34Rv/9gJRf2bccTP+hPm1TPzfeW846rl9jZuwVeughOvLJ+vk/C\nEm5S+CXwO+AtPB0SZuJJDCISJZVXQMvOK+Se/yzltblb6NqmBdv2HASg/TFJ7NhXXOXzackJ/OWH\nJ3Nxv/aYWb3GTmkxvHObZ3zB3H9WX17qTbi9jw4A90c4FhGpAU8X0iNXQCtzjkVb88jJ9yx4c8XA\nzpx3fNsq6yQnJ8bx2OgTGX5Ch3qNGYDiAnjtStj6NVz+HGDwzi/qPw4JKNzeRzOBHzrn8rzvWwFv\nOueGRzI4EQkuWFfRcueYc//5VbZHZarrYL2KAK6cCCf+wPN65kNqQI4R4VYfZVQkBADn3F4z09US\niZBQbQUVOqQnsz2/6hKYgbqQRm2q62AJAQ4nBFADcgypptnfp9zMula8MbNuaO5ZkYioaCvIzivE\n4WkruH/KMt5ZnH1EuUv6d6zy2ZhYAe3AbljyOrxxbXTjkFoJ90nht8CXZvY5YMA5wJiIRSXShAVq\nKygqKefP01Zz+cDOzF6by+y1uSzcupfEOKNtWhLb84vqr1ooWJVQ8zbQe5QnIbiy6mcrlZgUbkPz\nNDMbjCcRLMYzaC0KY99FGqecvEIe+2Alw/q0Cz7dRH4R9/xnKe8vzQGDxDjjqlO78McrTqrfYINV\nCR3cDcveglNvhpN+CJ0Hwe9b1W9sctTCbWi+GbgLyASWAGcAX3Pk8pyBPjcCeBqIB150zo0LUu5U\n7/Gucc5NCjt6kUYgd38xl/1jDrsKivl41U7apiWxc3/VLqRJCXFMWZxFRmoS0+8+l9YtmkUh2mr8\nchG07BLtKOQohNumcBdwKrDFOTcMGAjkhfqAmcXjGew2EugHXGtm/YKUewKYUYO4RRqN37z9LfuK\nSnjt5tNJTU4gIa7qmIGUxHie+EF/3rt9CG+OOSM2EwJUTQjBeg+pV1HMCrdNocg5V2RmmFmSc261\nmVXXmnUasN45txHAzN4ERgMrK5X7JTAZT9IRaTJ2FRSzMfcAM1fuYOzw3px9XAaXD+jMxDmbAM+g\ns537iuu3C2kouWvh/Ttr9hn1Kmpwwk0KWd6ZUd8BZprZXmBLNZ/pDGzzPwZwun8BM+uMZ2nPYYRI\nCmY2Bm/DdteuXYMVE4l5/l1NK6SnJHLDWd0AGD2gExPnbOLYNs35fOyw+g0u6FKX7eCm6TDpZ7Av\nu+p+aVTCbWi+wvvyETP7FEgHptXB9/8NuM85Vx5qmL1z7gXgBYDBgwerK6zEnFDjCpxzmBlTFmbx\nm3e+pchvIRsDhh7fltQkzz/F/pnpDDq2Fef0yqj/kwi61OVOeGag5/W1b8J7d2qgWSNW45lOnXOf\nh1k0G/CvYMz0bvM3GHjTmxAygFFmVqp1GqQhCTQH0QNTvuW7/EIu6Nue61/6hgFdWjJz5Q7KK/2k\nccCCLXt9782MybeeVY/Rh6nXcM96Bu37qUqokYvk9NfzgV5m1h1PMrgGuM6/gHOue8VrM/tf4AMl\nBGkocvcXM3HOJl75anOVcQWFJWWMm7aGcdPWkJwYFzAhVIj4ymahqoXCvcFfMQGat67buCQmRSwp\nOOdKzewOYDqeLqkTnXMrzOwX3v0TIvXdIpG2Y18RlzzzJXkHD1Ea7G4PZKQm8fvRJ3BK11Z8/7k5\n5OSFNy1FnQpVLQRw6ABkLQh9DCWEJiOiC+U456YCUyttC5gMnHM/jWQsInVp8qIsdhUU894dZ3Pr\nq4vIDvBrPykhjgUPXuh7f+/wPlVmK434tBQbq6ntnfUIzPsXHCqIXAzSoIQ7TkFE/Ly3JIdTurak\nf2ZLxg7vTUpifJUyV596ZJ/9ywd25vHvn0TnlikY0LllCo9//6S672rqHCx5w9Mg/PpVoct++RQc\nPxx+NBmaB2ncVgNyk6IlNUXC5Jzj9XlbeWrmWnYVHOLRy04A8N3U/Xsf/erCXlw5uOrI3jqdrTTo\nYvfNoOwQJDaHdn0hZ3HwY9y5BFp7m/bu3VA3cUmDpqQgTdrugmK+2rCbS/t3rLL6mH830w7pybRp\n0YzlOfs4o0drrj2tNT8YdHjCt6hMTR10sftDMOovMPgmMINHWwY/RuvuwfdJk6SkII1GOGsQALy9\nMIu/zFxLTl4hifFxHCorJzU5gWG9PdUkq7bv4843FrEh94Cvx9D2/CK25xdx+YBOPHnVAOICTEUR\nU077+eHXLdppXIGETUlBGoVgYwWAIxLD7a8t5MNvv/O9P1TmGUj28LvLKRzZF4C/zFjD5l0HAnYh\nnb95b+wnhMo0rkBqQElBYl44TwCB1iAoLCnjDx+uJDuvkJuGdGf+5j1HJAR/W/cUcttriwBIiLPo\njSmA6scVrJsFezdFPg5pkpQUJKaFegIYdVJHSsvLad4sIWCXUIBdBYcYP30NL8zeyL6ikqDfY8C0\nu88FoFXzRK549quAx4z4mAIIPa5g+WSYMgbKSyMfhzRJ6pIqMS3YE8AfP1zFRU99zsVPzWb6isC/\n/gHSkhOY+NPBDD+hPXcMO46O6ckBy3VqmULvDmn07pBGu2OSA3YzjfiYAhfGtF6TfgZtesHoZyE5\nSAOy2grkKOhJQWJasOqa3IJiOsYns3N/Mbf830IyUptRUFx6xGRzKYnxPDb6RM7v057z+7QHoGfb\n1LAGkAXqZhqx6atLCuGNa2DLV9DplNBlr/svHHsWJKXCwB/VfSzS5CkpSEzr1DIlYDVO82bxTLv7\nXN5fmsOyrDweGNmXz9fmVnsTr8nNPuLdTEuKYNbDnikmshfA4J/Bmo9Cf+b4iyMXjwhgLpxH1hgy\nePBgt2BBNfO0SKPx6PsreHnO5iO2pSTGR2YkcKQFa0DG4Nxfw/kPwv4d8Nfjgx/jkfyIhSeNm5kt\ndM4Nrq6c2hQkZq3bsZ/Xv9lKl9YptD8mKbJTQ9SHYA3IOE9CAEhrryUsJapUfSQxacm2PO54fREt\nkhKYfOtZtEsL3EAc80qLYcsciKvBPzWNK5AoUlKQmPNdfhE3TJxHalICE396auwnhGDVQknHeOYh\nOrir/mMSqSVVH0lMyckr5JZXF3KotJxXbz6dAV1CzNsTK4JVCxXvg9Y9PEtYXvff+o1JpJb0pCBR\nU3mk8ogT2vPfhVmUlTueunoA3TNaRDvEo/ezaRBXdVptkVilpCBREWik8ktzNtOlVQqv3nw6x7aJ\nckIINdXETdNh52r44q/Qe0To4/gnBE1MJw2AkoJExRPTVlcZqQxQUu6inxAg9FQTzwz0vG6eAZ/8\nIfxjqgFZGgAlBak3/5m/jWPbNGfNjv1sz6+6VjHAjiDb60xdLGJ/8R8htT30uQR2r4Pnz63bGEWi\nSElB6sUHy3K4d/Iy33sDAg2brPWEc+He7EM9AZSXQ1wYfS/OuuPw644nq1pIGhUlBYm44tIyHnp3\nBSdnpnPWcRn063gMZeWubhexD3Wzr7BzdehjPHMy3PA+bPm6Zt+taiFpRJQUJGzB1jUoLi3jqZnr\nGNq7LWf0aHNEuRZJCZzZozV7DhziyatOZmjvI389VzsHUV1U9xTvhznPeBapD6UwH54/D4rywjuu\nSCOkpCBhCdRb6L7Jy9iy5wB7Cg7x76+38PzsDZzRozVzN+zxVQ0VFJcyc9VOWjSL5+zjMo44ZlgT\nzoV6AshaCM1bewaIhfJEN8/6A/2vhmVvBS/3g3/BrEdgyN3w9T/hQG7VMqoSkkZOSUHCEmhdg+LS\ncp6a6fm1fuWgTBLijDfnbwt8AIPE+DoeK/ni+Z7pIxKr6a105h3Q51LocmropHD8cM8fwJBf1V2c\nIg2IkkIjFaiqZ/SATpiFv77wwi17eWH2Bm44q1vIZShfu/l0Tu/emoT4ON6avy1gA/LB4qrdT4/a\nVf8Hqz+Efdmw+Yvg5S569PBrNQqLhKSk0AgFquoZO2kpD727nLduOZOycseJndN9ZSuSR5vUZow+\nuROn92jDt9n5PPvZBsqdY/qKHUG/q3PLlCOqhYKtfxCRZSz7Xeb5g9BtD/7UKCwSkpJCIxSoqqek\nzFFSVsolz3xBuYMJ159CUUn5EcljV8EhXpqzmZe86xeMHtCJB0b2ZcbK75i3aQ/TV3xHSdnh54BA\nvYXGDu9dN72KFr8KS14Pv7xu9iJ1QkmhkSgpK+cfn6znvaU5QRexB+jT4RhKysq5+60llJW7I27y\nFdqmJfHmmDPo2TYVgJ+c2Y2fnNktaO8jf2GvbBbql/21b8B7v/RMJpfYAkoOBC4nInVOK681MJVv\nzD8b0o2khHgmLcxiybY8zumVwYLNewNOIdG5ZQpz7j+fbXsO8vdP1vGfBVkBv8OATeMuieyJPJIe\nfF96F3DlcNtcSD4msnGINBHhrrymJ4UGJFBbwWMfrALgmOQE/nHdQC7t36lKOTiyCqdL6+b8+cqT\neW9JDkWl5VW+56jq/+tiXEF5qedpQQlBpN4pKTQABcWlFJWUBWwrAGiXlsTse4eRnOiZkTPcKpy7\nLuzF+OlrKPd7WAxa/18X00jkbYX8LJjxYOgTvvUrz/gDEal3Sgq1FE79un+57LxCMlKTePCSviHL\neXoBJXHNqZn8engfAG59dSHzN++hqKTqr3qA3P3FvoRQIZyBYbcOPY6O6SlhnUfIm/3nf/ZMEZ0c\nokoI4G/9AQfN24Qup4QgEjVKCrXgqZ5ZRqH3Jp2dV8gDU74FDv9KLyt3/OSlb5i3eY+vMXdXQTFj\nJy2tUu6tBVv5/fsrfTf9XQXF/OPTDfTISKVnu1S+WLeLLq1T2LYncAPy0VT3VJs8Sgphz6bQB/n0\nj+F92Rm3QudBcOxZ8GTf8IMUkXqjpFALj3+0ypcQKhR6q3cqbrATPt/AnA27q3y2pMzxxLTVXD6w\nM0u35XHXm4vZvPtgwO959IOVdExPJi05gal3nsPHq3aG193zaOv1922Huc/C1q8hZwmUl4Qu/+t1\nkNjcM2fQUycELzfi8SNj0SAykZgT0aRgZiOAp4F44EXn3LhK+38E3Ienw8t+4Fbn3NJIxlRbZeWO\n52dvYEBmS3bsKw5YJievkKKSMv40dRWvfL0l6LG25xdRVFLGza8sIDEu+Ajj/MIS4gz+fu1A0pIT\nw+/uGc6MoRA8eWCe6SMyB8OZt0PG8fDubUHjJNV7I09KDV6mMo0rEIlJEUsKZhYP/BO4CMgC5pvZ\ne865lX7FNgHnOef2mtlI4AXg9EjFFI5gbQVz1u/iz9PWhPxsRmoSV074iuXZ+7hpSHc++nY7OUEW\njbni2a/I3V/Mazefzr2TlgUcW5CcGMf0u8+l3THJng3je3H5gZ1cDpAMFAHvAh8mw4k/gIN7oHBP\n6BPcsRK2zYVOA4MnDxzc/g206Xl4U6ik4E9PACINWiSfFE4D1jvnNgKY2ZvAaMCXFJxzX/mVnwtk\nRjCeagWbHuKLdbnsPeipQhlzbg9Sk+J57rONVXoC7Ss6RG5BMROuH8SIEztwUuf0KtU9yYlxDD2+\nHTNX7aBUyfPVAAAPGklEQVR/Zjpn9WwTdBTw498/6XBCgOA38dIi2PgZpLSG5q1Cn+RzZ4b3P8M/\nIUD4N3s9AYg0aJFMCp0B/ykzswj9FHAT8FGgHWY2BhgD0LVr17qKr4pg00NMXpQNeGYC/c0oTwNp\n19YtjniiuPiEdryzOIebhnRlxIkdgNBdQzfkFpCWlICZcfmsoVwev9NTyeZvVjsY6L3J7qrmZvs/\nfg9goQaGXfgo9Dwfdq+HSTeGPqY/3exFmoSYaGg2s2F4ksKQQPudcy/gqVpi8ODBERuCHWwmUAOG\nn9CBn5/Tw7ctUK+d311yAnH+bQTBqntmtaNnuH37Z/8FFrwM+wKPPq6xIXd7/tuxf82Sgog0CXU8\nwf0RsoEufu8zvduOYGb9gReB0c65qt116lFqUuAc2allChN+PIjeHdJCfj6ucqNxOA2+awI+HB32\nyWOeqpyLw+z2CcHr71WvLyLViOSTwnygl5l1x5MMrgGu8y9gZl2BKcCPnXNrIxhLtZxzJCUYBw4R\n3gjfo7VnI5SVwH9/Grpc1zPh+skQnwgzfhvescOt6lGjsIhUErGk4JwrNbM7gOl4assnOudWmNkv\nvPsnAA8BbYBnvYu/lIYzYVMkLNyyl10HSrj61Ey+XLe7+hG+R+v586BZC0///tLAPZQA+N7TnoQA\ndX8TVzuBiFSiWVK9fv7KAh7f+H0yyK+6M7mlpyF3/3fQsis82S/04LB1M2HdDJj3QvAvPO5Cz5PC\nBQ/BixcEL/dIgHhERGpIs6TWwJRFWcxcuYN/JQe5ARflweNdwJVBm+NCtxW8fSssfR2aVTOQ6/rJ\nh1+rGkck4kpKSsjKyqKoKMSTeSOQnJxMZmYmiYmJtfp8k08KC7fs5X/+s5RTu7WC70IUPP0XkNbB\nsyJYKEvfgHPvhfPuDf1E4U/VOCIRl5WVRVpaGt26davRWuUNiXOO3bt3k5WVRffu3Wt1jCafFOas\n34UZvPiTU+HPIQqO+JPnv2fcBo+FmOXzhveh+zme17rZi8SMoqKiRp0QAMyMNm3akJubW+tjRLJL\naoOwYMteerdPI710V3gfiK8mj1YkBBGJOY05IVQ42nNs0kmhrNyxaMteBndNh7fHRDscEZGoa9JJ\nYdX2fRQUl3JN6buwaTYkBRmcVrkNQIPDRBq9dxZnc/a4T+h+/4ecPe4T3llcZextjeTl5fHss8/W\n+HOjRo0iLy/vqL67Jpp0m8L0Fd9xfFwWJ6x+GvqNhh/+G8J59FJbgUijFmhyzMoLadVURVK47bYj\nZxwuLS0lISH4rXjq1Km1+r7aarJJwTnHu0tyeKzVx1hxIlzyVHgJQUQavEffX8HKnH1B9y/emseh\nsqoLad07aRlvzNsa8DP9Oh3Dw98LvsjU/fffz4YNGxgwYACJiYkkJyfTqlUrVq9ezdq1a7n88svZ\ntm0bRUVF3HXXXYwZ46nS7tatGwsWLKCgoICRI0cyZMgQvvrqKzp37sy7775LSkrtV14MpMlWHy3a\nupfCPTkMKfwEBv4IWlSzbrCINBmVE0J128Mxbtw4evbsyZIlSxg/fjyLFi3i6aefZu1azww/EydO\nZOHChSxYsIBnnnmG3burTgW3bt06br/9dlasWEHLli2ZPHlylTJHq8k+Kfxr9iZuSv6YuPJSOP3W\naIcjIvUo1C96gLPHfRJw4avOLVN465Yw1ySpxmmnnXbEWIJnnnmGt99+G4Bt27axbt062rQ58sdq\n9+7dGTBgAACDBg1i8+bNdRKLvyb5pDB7bS6fr9zCDQmzsN4jIeO4aIckIjFk7PDepCQeucBJXU+O\n2aJFC9/rzz77jFmzZvH111+zdOlSBg4cGHDkdVJSku91fHw8paWldRZPhSb3pPDNxt3c8PI3/CP1\ndVJK8uGsX0Y7JBGJMWGvh14DaWlp7N+/P+C+/Px8WrVqRfPmzVm9ejVz586t9fccrSaXFN5asI1R\nScu4pGQmnHMPHHtWtEMSkRgUaCGto9GmTRvOPvtsTjzxRFJSUmjfvr1v34gRI5gwYQJ9+/ald+/e\nnHHGGXX2vTXVJGZJfWdxti/jA0xK/xuDmm2Du5dXP0JZRBqFVatW0bdv32iHUS8CnWu4s6Q2+jaF\niv7G2XmFOKAjuxhQNJ81HS9TQhARqaTR3xXPefcsVsXneZb58dNu7RuEngFPRKTpafRPCm0IPDy8\nFcEHroiINFWNPimIiEj4lBRERMRHSUFERHwafUOziEiNje8VfCndWs6SnJeXx+uvv15lltRw/O1v\nf2PMmDE0b968Vt9dE43/SUFrH4hITQVKCKG2h6G26ymAJykcPHiw1t9dE43/SUFrH4hIZR/dD999\nW7vPvnxJ4O0dToKR44J+zH/q7Isuuoh27drxn//8h+LiYq644goeffRRDhw4wFVXXUVWVhZlZWX8\n7ne/Y8eOHeTk5DBs2DAyMjL49NNPaxd3mBp/UhARiQHjxo1j+fLlLFmyhBkzZjBp0iTmzZuHc47L\nLruM2bNnk5ubS6dOnfjwww8Bz5xI6enpPPnkk3z66adkZGREPE4lBRFpekL8ogfgkfTg+2788Ki/\nfsaMGcyYMYOBAwcCUFBQwLp16zjnnHO45557uO+++7j00ks555xzjvq7akpJQUSknjnneOCBB7jl\nlluq7Fu0aBFTp07lwQcf5IILLuChhx6q19gaf0OziEhNRaCDiv/U2cOHD2fixIkUFBQAkJ2dzc6d\nO8nJyaF58+Zcf/31jB07lkWLFlX5bKTpSUFEpLIIdFDxnzp75MiRXHfddZx5pmcVt9TUVF599VXW\nr1/P2LFjiYuLIzExkeeeew6AMWPGMGLECDp16hTxhuYmMXW2iIimztbU2SIiUkNKCiIi4qOkICJN\nRkOrLq+Noz1HJQURaRKSk5PZvXt3o04Mzjl2795NcnJyrY+h3kci0iRkZmaSlZVFbm5utEOJqOTk\nZDIzM2v9eSUFEWkSEhMT6d69e7TDiHkRrT4ysxFmtsbM1pvZ/QH2m5k9492/zMxOiWQ8IiISWsSS\ngpnFA/8ERgL9gGvNrF+lYiOBXt6/McBzkYpHRESqF8knhdOA9c65jc65Q8CbwOhKZUYDrziPuUBL\nM+sYwZhERCSESLYpdAa2+b3PAk4Po0xnYLt/ITMbg+dJAqDAzNbUMqYMYFctPxtrdC6xqbGcS2M5\nD9C5VDg2nEINoqHZOfcC8MLRHsfMFoQzzLsh0LnEpsZyLo3lPEDnUlORrD7KBrr4vc/0bqtpGRER\nqSeRTArzgV5m1t3MmgHXAO9VKvMe8BNvL6QzgHzn3PbKBxIRkfoRseoj51ypmd0BTAfigYnOuRVm\n9gvv/gnAVGAUsB44CNwYqXi8jroKKoboXGJTYzmXxnIeoHOpkQY3dbaIiESO5j4SEREfJQUREfFp\nMkmhuik3Yp2ZbTazb81siZkt8G5rbWYzzWyd97+toh1nZWY20cx2mtlyv21B4zazB7zXaI2ZDY9O\n1IEFOZdHzCzbe12WmNkov32xfC5dzOxTM1tpZivM7C7v9gZ1bUKcR4O7LmaWbGbzzGyp91we9W6v\n32vinGv0f3gaujcAPYBmwFKgX7TjquE5bAYyKm37M3C/9/X9wBPRjjNA3OcCpwDLq4sbz3QoS4Ek\noLv3msVH+xyqOZdHgF8HKBvr59IROMX7Og1Y6425QV2bEOfR4K4LYECq93Ui8A1wRn1fk6bypBDO\nlBsN0Wjg397X/wYuj2IsATnnZgN7Km0OFvdo4E3nXLFzbhOeXmmn1UugYQhyLsHE+rlsd84t8r7e\nD6zCM5tAg7o2Ic4jmJg8DwDnUeB9m+j9c9TzNWkqSSHYdBoNiQNmmdlC77QfAO3d4XEd3wHtoxNa\njQWLu6Fep196Z/md6Pdo32DOxcy6AQPx/DJtsNem0nlAA7wuZhZvZkuAncBM51y9X5OmkhQagyHO\nuQF4Zpa93czO9d/pPM+TDa5/cUON289zeKolB+CZs+uv0Q2nZswsFZgM3O2c2+e/ryFdmwDn0SCv\ni3OuzPvvPBM4zcxOrLQ/4tekqSSFBj+dhnMu2/vfncDbeB4Td1TMKuv9787oRVgjweJucNfJObfD\n+w+5HPgXhx/fY/5czCwRz430NefcFO/mBndtAp1HQ74uAM65POBTYAT1fE2aSlIIZ8qNmGVmLcws\nreI1cDGwHM853OAtdgPwbnQirLFgcb8HXGNmSWbWHc86G/OiEF/Y7Mip3q/Ac10gxs/FzAx4CVjl\nnHvSb1eDujbBzqMhXhcza2tmLb2vU4CLgNXU9zWJdot7ff3hmU5jLZ4W+t9GO54axt4DTy+DpcCK\niviBNsDHwDpgFtA62rEGiP0NPI/vJXjqPG8KFTfwW+81WgOMjHb8YZzL/wHfAsu8/0g7NpBzGYKn\nGmIZsMT7N6qhXZsQ59HgrgvQH1jsjXk58JB3e71eE01zISIiPk2l+khERMKgpCAiIj5KCiIi4qOk\nICIiPkoKIiLio6QgEmFmNtTMPoh2HCLhUFIQEREfJQURLzO73juf/RIze947OVmBmT3lnd/+YzNr\n6y07wMzmeidce7tiwjUzO87MZnnnxF9kZj29h081s0lmttrMXvOOxMXMxnnXAlhmZn+J0qmL+Cgp\niABm1he4GjjbeSYkKwN+BLQAFjjnTgA+Bx72fuQV4D7nXH88I2crtr8G/NM5dzJwFp4R0OCZvfNu\nPHPg9wDONrM2eKZgOMF7nD9E9ixFqqekIOJxATAImO+duvgCPDfvcuAtb5lXgSFmlg60dM597t3+\nb+Bc7/xUnZ1zbwM454qccwe9ZeY557KcZ4K2JUA3IB8oAl4ys+8DFWVFokZJQcTDgH875wZ4/3o7\n5x4JUK6288IU+70uAxKcc6V4Zu+cBFwKTKvlsUXqjJKCiMfHwJVm1g586+Iei+ffyJXeMtcBXzrn\n8oG9ZnaOd/uPgc+dZ+WvLDO73HuMJDNrHuwLvWsApDvnpgK/Ak6OxImJ1ERCtAMQiQXOuZVm9iAw\nw8zi8MyEejtwAM9iJw/imcf+au9HbgAmeG/6G4Ebvdt/DDxvZr/3HuOHIb42DXjXzJLxPKn8Tx2f\nlkiNaZZUkRDMrMA5lxrtOETqi6qPRETER08KIiLioycFERHxUVIQEREfJQUREfFRUhARER8lBRER\n8fl/YIZDUaW7g2EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1621f7036d8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#\n",
    "# class dropoutに実装\n",
    "#\n",
    "# coding: utf-8\n",
    "import os\n",
    "import sys\n",
    "sys.path.append('../')  # 親ディレクトリのファイルをインポートするための設定\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from dataset.mnist import load_mnist\n",
    "from saitobook.common.multi_layer_net_extend import MultiLayerNetExtend\n",
    "from saitobook.common.trainer import Trainer\n",
    "\n",
    "(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True)\n",
    "\n",
    "# 過学習を再現するために、学習データを削減\n",
    "x_train = x_train[:300]\n",
    "t_train = t_train[:300]\n",
    "\n",
    "# Dropuoutの有無、割り合いの設定 ========================\n",
    "use_dropout = True  # Dropoutなしのときの場合はFalseに\n",
    "dropout_ratio = 0.2\n",
    "# ====================================================\n",
    "\n",
    "network = MultiLayerNetExtend(input_size=784, hidden_size_list=[100, 100, 100, 100, 100, 100],\n",
    "                              output_size=10, use_dropout=use_dropout, dropout_ration=dropout_ratio)\n",
    "trainer = Trainer(network, x_train, t_train, x_test, t_test,\n",
    "                  epochs=301, mini_batch_size=100,\n",
    "                  optimizer='sgd', optimizer_param={'lr': 0.01}, verbose=True)\n",
    "trainer.train()\n",
    "\n",
    "train_acc_list, test_acc_list = trainer.train_acc_list, trainer.test_acc_list\n",
    "\n",
    "# グラフの描画==========\n",
    "markers = {'train': 'o', 'test': 's'}\n",
    "x = np.arange(len(train_acc_list))\n",
    "plt.plot(x, train_acc_list, marker='o', label='train', markevery=10)\n",
    "plt.plot(x, test_acc_list, marker='s', label='test', markevery=10)\n",
    "plt.xlabel(\"epochs\")\n",
    "plt.ylabel(\"accuracy\")\n",
    "plt.ylim(0, 1.0)\n",
    "plt.legend(loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dropoutとアンサンブル学習\n",
    "\n",
    "- アンサンブル学習\n",
    "   - 複数のモデルを個別に学習させ、推論時には、その複数の出力を平均\n",
    "   - アンサンブル学習を行うことで、ニューラルネットワークの認識精度が数%向上\n",
    "\n",
    "- アンサンブル学習はDropout と近い関係\n",
    "    - Dropoutは推論時には、ニューロン出力に消去割合（ex.0.5）を乗算\n",
    "    - モデルの平均を取る\n",
    "    - つまりDropout は、アンサンブル学習と同効果を（擬似的に）1ネットワークで実現"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 6.5 ハイパーパラメータの検証\n",
    "\n",
    "- ハイパーパラメータ\n",
    "    - 各層のニューロン数\n",
    "    - バッチサイズ\n",
    "    - 学習係数\n",
    "    - Weight decay\n",
    "- 適切な値にすると予測性能が向上する。\n",
    "- 最適値の探索は試行錯誤が必要になる\n",
    "\n",
    "#### 6.5.1 検証データ(validation data)\n",
    "\n",
    "- 検証データ（validation data）とは\n",
    "   - ハイパーパラメータ調整専用のデータ\n",
    "　 - テストデータを使ってハイパーパラメータの性能を評価してはいけない\n",
    "   - テストデータに対して過学習を起こすから\n",
    "   - 訓練データは、パラメータ（重みやバイアス）の学習に使う\n",
    "   \n",
    "- データセットの種類別\n",
    "   - 訓練データ・検証データ・テストデータの3種類＝＞検証データ使う\n",
    "   - 訓練データ・テストデータの2種類＝＞自分で検証データ構築\n",
    "\n",
    "- 検証データの作成方法\n",
    "   - MNIST データセットの場合、検証データ作成方法\n",
    "   - 訓練データの中から20% 程度を検証データとして分離\n",
    "\n",
    "   \n",
    "\n",
    "\n",
    "#### 6.5.3 ハイパーパラメータ最適化の実装\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from saitobook.common.util import shuffle_dataset\n",
    "# データの読み込み\n",
    "(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True, one_hot_label=True)\n",
    "# 訓練データをシャッフル\n",
    "x_train, t_train = shuffle_dataset(x_train, t_train)\n",
    "# 検証データの分割\n",
    "validation_rate = 0.20\n",
    "validation_num = int(x_train.shape[0] * validation_rate)\n",
    "x_val = x_train[:validation_num]\n",
    "t_val = t_train[:validation_num]\n",
    "x_train = x_train[validation_num:]\n",
    "t_train = t_train[validation_num:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.5.2 ハイパーパラメータの最適化\n",
    "\n",
    "- 最適化を行う上で重要なポイント\n",
    "   -ハイパーパラメータの「良い値」範囲を徐々に絞り込んでいく  \n",
    "   - 最初はおおまかに範囲を設定\n",
    "   - その範囲の中からランダムにハイパーパラメータを選び出し（サンプリングし）\n",
    "   - サンプリングした値で認識精度の評価\n",
    "   - 複数回繰り返し、認識精度の結果を観察\n",
    "   - 結果からハイパーパラメータの「良い値」の範囲を狭めていく\n",
    "   \n",
    "- ニューラルネットワークのハイパーパラメータの最適化手法\n",
    "   - グリッドサーチ(規則的な探索)は使わない\n",
    "   - ランダムにサンプリングして探索するほうが良い結果になる[15] \n",
    "   - 複数ハイパーパラメータのうち、認識精度に与える影響度がハイパーパラメータごとに異なるから\n",
    "\n",
    "- 最適化の手順\n",
    "　　- ステップ0＝ハイパーパラメータの範囲を設定する。\n",
    "    - ステップ1＝設定されたハイパーパラメータの範囲から、ランダムにサンプリング\n",
    "    - ステップ2＝ステップ1 でサンプリングされたハイパーパラメータの値を使用して学習を行い、検証データで認識精度を評価する（ただし、エポックは小さく設定）。\n",
    "    - ステップ3＝ステップ1 とステップ2 をある回数（100 回など）繰り返し、それらの認識精度の結果から、ハイパーパラメータの範囲を狭める。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# coding: utf-8\n",
    "import sys, os\n",
    "sys.path.append('../')  # 親ディレクトリのファイルをインポートするための設定\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from dataset.mnist import load_mnist\n",
    "from saitobook.common.multi_layer_net import MultiLayerNet\n",
    "from saitobook.common.util import shuffle_dataset\n",
    "from saitobook.common.trainer import Trainer\n",
    "\n",
    "#(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True)\n",
    "\n",
    "# 高速化のため訓練データの削減　(＜＜時間がかかるので)\n",
    "#x_train = x_train[:500]\n",
    "#t_train = t_train[:500]\n",
    "\n",
    "# 検証データの分離\n",
    "#validation_rate = 0.20\n",
    "#validation_num = x_train.shape[0] * validation_rate\n",
    "#x_train, t_train = shuffle_dataset(x_train, t_train)\n",
    "#x_val = x_train[:validation_num]\n",
    "#t_val = t_train[:validation_num]\n",
    "#x_train = x_train[validation_num:]\n",
    "#t_train = t_train[validation_num:]\n",
    "\n",
    "\n",
    "def __train(lr, weight_decay, epocs=50):\n",
    "    network = MultiLayerNet(input_size=784, hidden_size_list=[100, 100, 100, 100, 100, 100],\n",
    "                            output_size=10, weight_decay_lambda=weight_decay)\n",
    "    trainer = Trainer(network, x_train, t_train, x_val, t_val,\n",
    "                      epochs=epocs, mini_batch_size=100,\n",
    "                      optimizer='sgd', optimizer_param={'lr': lr}, verbose=False)\n",
    "    trainer.train()\n",
    "\n",
    "    return trainer.test_acc_list, trainer.train_acc_list\n",
    "\n",
    "\n",
    "# ハイパーパラメータのランダム探索======================================\n",
    "optimization_trial = 100\n",
    "results_val = {}\n",
    "results_train = {}\n",
    "for _ in range(optimization_trial):\n",
    "    # 探索したハイパーパラメータの範囲を指定===============\n",
    "    weight_decay = 10 ** np.random.uniform(-8, -4)\n",
    "    lr = 10 ** np.random.uniform(-6, -2)\n",
    "    # ================================================\n",
    "\n",
    "    val_acc_list, train_acc_list = __train(lr, weight_decay)\n",
    "    print(\"val acc:\" + str(val_acc_list[-1]) + \" | lr:\" + str(lr) + \", weight decay:\" + str(weight_decay))\n",
    "    key = \"lr:\" + str(lr) + \", weight decay:\" + str(weight_decay)\n",
    "    results_val[key] = val_acc_list\n",
    "    results_train[key] = train_acc_list\n",
    "\n",
    "# グラフの描画========================================================\n",
    "print(\"=========== Hyper-Parameter Optimization Result ===========\")\n",
    "graph_draw_num = 20\n",
    "col_num = 5\n",
    "row_num = int(np.ceil(graph_draw_num / col_num))\n",
    "i = 0\n",
    "\n",
    "for key, val_acc_list in sorted(results_val.items(), key=lambda x:x[1][-1], reverse=True):\n",
    "    print(\"Best-\" + str(i+1) + \"(val acc:\" + str(val_acc_list[-1]) + \") | \" + key)\n",
    "\n",
    "    plt.subplot(row_num, col_num, i+1)\n",
    "    plt.title(\"Best-\" + str(i+1))\n",
    "    plt.ylim(0.0, 1.0)\n",
    "    if i % 5: plt.yticks([])\n",
    "    plt.xticks([])\n",
    "    x = np.arange(len(val_acc_list))\n",
    "    plt.plot(x, val_acc_list)\n",
    "    plt.plot(x, results_train[key], \"--\")\n",
    "    i += 1\n",
    "\n",
    "    if i >= graph_draw_num:\n",
    "        break\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}