171106_p241.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "II部は「ニューラルネットワークと深層学習」を学びます。\n",
    "深層学習の中心的モデルは、幾層にも連結したニューラルネットワークモデルです。特に深層学習の代表的モデルである畳み込みニューラルネットワーク（Convolutional Neural Network:CNN）について学習していきます。\n",
    "\n",
    "9章からGoogleのDeep Learningフレームワーク[Tensorflow](https://www.tensorflow.org/)を利用します。\n",
    "Anaconda環境(注1)へTensorflowをインストールするステップは次の通りです。\n",
    "\n",
    "注1:非GPU搭載マシンのLOCAL PCの場合。\n",
    "***\n",
    "Tensorflowのセットアップ\n",
    "1. Anaconda promptを起動。\n",
    "2. pip install tensorflowでインストール。\n",
    "3. conda listで、インストール済ライブラリのリストを表示。\n",
    "```\n",
    "tensorflow                1.3.0                     <pip>\n",
    "tensorflow-tensorboard    0.1.4                     <pip>\n",
    "```\n",
    "上記2件がリストに表示されていれば準備OKです。\n",
    "\n",
    "4. tensorflowのupgrade処理(適宜)。\n",
    "　　　- the most recent stable branch r1.3(注：17-11-04時点).\n",
    "      - [versions of tensorflow](https://www.tensorflow.org/versions/)\n",
    "\n",
    "```\n",
    "pip install --ignore-installed --upgrade tensorflow\n",
    "```\n",
    "\n",
    "conda listで下記表示されます。\n",
    "```\n",
    "tensorflow                1.4.0                     <pip>\n",
    "tensorflow                1.3.0                     <pip>\n",
    "tensorflow-tensorboard    0.4.0rc2                  <pip>\n",
    "tensorflow-tensorboard    0.1.4                     <pip>\n",
    "```\n",
    "\n",
    "5. jupyter notebookの起動ディレクトリに移動。\n",
    "6. mkdir tf_logs; mkdir logs;  でtensorboardのログを保存するディレクトリ作成。\n",
    "輪読本コードは、相対パスでディレクトリを指定している事に注意。\n",
    "7. mkdir images; 画像保存用ディレクトリ作成\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "- TensorFlowはオープンソースソフトウェアライブラリ\n",
    "- Pythonで計算定義すると、最適化されたC ++コードを実行\n",
    "- グラフをチャンクに分割して実行する機能で、複数のCPUまたはGPUを使った並列計算が可能（図9-2参照）\n",
    "\n",
    "|図9_1|図9_2|\n",
    "|:-:|:-:|\n",
    "|<img width=\"280\" alt=\"F9_1.png\" src=\"./images/F9_1.PNG\">|<img width=\"280\" alt=\"F9_2.png\" src=\"./images/F9_2.PNG\">|\n",
    "\n",
    "- TensorFlowは分散コンピューティングもサポート(巨大なNeural Networkを分割計算)\n",
    "- 並列計算は、何百サーバー対応可\n",
    "- 数十億のインスタンスの訓練セット、何百万のパラメータ\n",
    "　　　- インスタンスの例：[iris dataset](http://archive.ics.uci.edu/ml/datasets/Iris)(150 instances, 4 attributes)\n",
    "- Google Brainチームが開発(Googleクラウドスピーチ、Googleフォト、Google検索に応用)\n",
    "- 2015年11月にオープンソース化\n",
    "- Popular open source libraries of Deep Learning\n",
    "      - Caffe\n",
    "      - DeepLearning4J\n",
    "      - H2O\n",
    "      - MXNet\n",
    "      - Tensorflow\n",
    "      - Theano\n",
    "      - Torch\n",
    "      - Chainer(日本PFN)\n",
    "- 特徴\n",
    "  - プラットフォーム(Windows, Linux,MacOSX, iOS and Android)\n",
    "  - python API called TF.Learn 2 \n",
    "     - tensorflow.contrib.learn compatible with Scikit-Learn. \n",
    "     - 昔はScikit Flow(skflow)ライブラリとして提供。\n",
    "     - skflowは0,80からtensorflowに統合。\n",
    "  - 高レベルAPIsがTensorFlowをバックエンドとして組み込まれる。 \n",
    "   　- Keras (NN Architectureを簡単に定義できる.Popular Library)\n",
    "     - Pretty Tensor\n",
    "  - C++ APIも高性能操作用に提供.\n",
    "  - コスト関数最小化の為のパラメータ検索用機能\n",
    "     - TensorFlowが定義した関数の勾配を自動計算\n",
    "     - AutoDiff機能と呼ばれる\n",
    "  - TensorBoardと呼ばれる視覚化ツール\n",
    "     - 計算グラフの表示\n",
    "     - 学習曲線の表示\n",
    "  - Googleはtensorflowの[クラウドサービス](https://cloud.google.com/ml/)提供.\n",
    "     - [1,000 Cloud TPUs](https://www.tensorflow.org/tfrc/)\n",
    "     - 180 teraflops of ML acceleration\n",
    "  - Forum,Q&A,コミュニティ作成tensorflowコード\n",
    "     - resources page on https://www.tensorflow.org/, \n",
    "     - https://github.com/jtoy/awesome-tensorflow. \n",
    "     - To ask technical questions, http://stackoverflow.com/ \"tensorflow\" tag.                  \n",
    "     - google group: [tensorflow forum](http://goo.gl/N7kRF9)\n",
    "  - 日本のコミュニティ\n",
    "     - 東京 [TensorFlow User Group Tokyo](https://tfug-tokyo.connpass.com/)\n",
    "     - 大阪 [Soleil Data Dojo](https://soleildatadojo.connpass.com/)\n",
    "     - 宇都宮 [TensorFlow User Group Utsunomiya](https://tfug-utsunomiya.connpass.com/)\n",
    "    \n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9章ではTensorflowの基礎を一通り実行\n",
    "  - 9章の項目\n",
    "    - installation \n",
    "    - creating,\n",
    "    - running, \n",
    "    - saving \n",
    "    - visualizing simple computational graphs.  \n",
    "- GPUバージョンのTensorflowをインストールする場合\n",
    "    - Cuda Toolkit and cuDNN (available on https://developer.nvidia.com/) \n",
    "    - Setting a couple environment variables. \n",
    "    - See the installation page on https://www.tensorflow.org/.\n",
    "    - インストールが難しい(バージョン組合せに試行錯誤必要)\n",
    "    - NIGスパコンチームがQsub環境GPU版を準備中"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [9-1] Tensorflow用のSetup p.244"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#- 作図用の関数を準備。[MatplotLib plots figures inline/Saving figures]\n",
    "#- ホーム直下のimagesディレクトリを作成しておく。\n",
    "\n",
    "# To support both python 2 and python 3\n",
    "from __future__ import division, print_function, unicode_literals\n",
    "\n",
    "#----- Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "#--------- to make this notebook's output stable across runs\n",
    "def reset_graph(seed=42):\n",
    "    tf.reset_default_graph()\n",
    "    tf.set_random_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "\n",
    "#----- To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "#-------- Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"tensorflow\"\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True):\n",
    "    path = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID, fig_id + \".png\")\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-2] Creating your first graph and running it in a session p.245\n",
    "p.245\n",
    "\n",
    "Tensorflowはデータフローグラフ表現で、計算を行います。\n",
    "データフローグラフとは、データの流れをエッジとノードで表現したグラフです。\n",
    "  - 楕円node＝計算(operation)\n",
    "  - 円node=定数(operation)\n",
    "  - エッジ=Tensor\n",
    "\n",
    "参考LINK\n",
    " - [Tensor(n次元多次元配列)の解説](https://qiita.com/rneuo/items/54c88088941bb5c47d17)\n",
    " - [Opノード、エッジ、セッション、変数の説明](https://qiita.com/yanosen_jp/items/70e6d6afc36e1c0a3ef3)\n",
    " - [a+b, 最急降下法のグラフ例](http://developers.gnavi.co.jp/entry/tensorflow-deeplearning-2_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 図9_1の計算グラフを生成\n",
    "# - 変数初期化せず\n",
    "# - グラフを評価するには\n",
    "#    - ①TensorFlowセッションを開き、\n",
    "#    - ②変数を初期化、\n",
    "#    - ③fを評価、が必要\n",
    "import tensorflow as tf\n",
    "import tensorboard as tb   # -> (注：追加）\n",
    "\n",
    "reset_graph()\n",
    "\n",
    "x = tf.Variable(3, name=\"x\")\n",
    "y = tf.Variable(4, name=\"y\")\n",
    "f = x*x*y + y + 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42\n"
     ]
    }
   ],
   "source": [
    "# Tensorflowセッションを開き、変数初期化、fを評価、セッションを閉じる\n",
    "# （セッションを閉じる=リソースを解放）\n",
    "\n",
    "sess = tf.Session()\n",
    "sess.run(x.initializer)\n",
    "sess.run(y.initializer)\n",
    "result = sess.run(f)\n",
    "print(result)\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# withブロックの中で読みやすくなる\n",
    "#   - default sessionとして登録\n",
    "#   - x.initializer.run（）= tf.get_default_session（）.run（x.initializer）\n",
    "#   - f.eval（）= tf.get_default_session（）.run（f）\n",
    "#   - withブロック終わりでsession自動的に閉じる\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    x.initializer.run()\n",
    "    y.initializer.run()\n",
    "    result = f.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# global_variable_initializers（）関数\n",
    "# - 単一変数毎にinitializerを手動でrunする代用 \n",
    "# - 実際には実行されないことに注意\n",
    "# - グラフにすべての変数を初期化するノードが作成される\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    result = f.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42\n"
     ]
    }
   ],
   "source": [
    "# JupyterやPythonシェル中で、InteractiveSessionを使いたい場合\n",
    "#   - Session vs InteractiveSession\n",
    "#   - 作成時に自動でデフォルトセッションとして設定される\n",
    "#   - withブロックは必要なし\n",
    "# 　- セッションを手動で終了必要\n",
    "# \n",
    "init = tf.global_variables_initializer()\n",
    "sess = tf.InteractiveSession()\n",
    "init.run()\n",
    "result = f.eval()\n",
    "print(result)\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlowプログラムは、通常、2つの部分に分割される。\n",
    " - 1. 構築フェーズ＝計算グラフ構築\n",
    " - 2. 実行フェーズ＝run \n",
    "\n",
    "構築フェーズでは、通常、MLモデルの計算グラフが作成され、訓練を行う。 \n",
    "実行フェーズは、一般に、モデルパラメータを改善していく、\n",
    "繰り返し訓練の評価ループ(例えば、ミニバッチごとの1ステップ)を実行する。\n",
    "\n",
    "```\n",
    "*mini batch法とは\n",
    "　訓練データからランダムにデータの塊を呼び出して学習する方式\n",
    "   \n",
    "   epoch数＝訓練データの学習回数\n",
    "   \n",
    "   例 datasize＝1500, batch_size=100,epoch=10\n",
    "        --> 15回パラメータ更新してデータが一巡。\n",
    "   　　 --> epoch=10で10回繰り返し。全計算で15x10=150回のパラメータ更新。\n",
    "　　例：datasize 100, batch_size=32の場合、データの区切りは下記。\n",
    "　　　　0〜31\n",
    "　　　　32〜63\n",
    "　　　　64〜95\n",
    "　　　　66〜99\n",
    "    \n",
    "* epochと精度の参考\n",
    "http://st-hakky.hatenablog.com/entry/2017/01/17/165137\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-3] Managing graphs　p.246\n",
    "### 計算グラフの管理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# どんなノードを生成しても自動的にデフォルトグラフに追加される\n",
    "#\n",
    "reset_graph()\n",
    "\n",
    "x1 = tf.Variable(1)\n",
    "x1.graph is tf.get_default_graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a1= True \n",
      "a2= False\n"
     ]
    }
   ],
   "source": [
    "# ほとんどの場合は追加で問題ない。\n",
    "# 　- 複数の独立グラフを管理したいかもしれない。\n",
    "#　 - 新グラフ生成用に、withブロックで、default graphを保存しておく\n",
    "\n",
    "graph = tf.Graph() # new graph\n",
    "with graph.as_default(): # 新グラフのdefaule graphをwithで一時利用にする\n",
    "    x2 = tf.Variable(2)\n",
    "\n",
    "a1=(x2.graph is graph)\n",
    "a2=(x2.graph is tf.get_default_graph()) #1時利用x2 vs default\n",
    "print(\"a1=\", a1, \"\\na2=\",a2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Jupyter（またはPythonシェル）では、同じコマンドを何回も\n",
    "実行します。結果として、多くの複製ノードを含むデフォルト\n",
    "グラフを作る可能性があります。\n",
    "\n",
    "解決策として次の２策がある。\n",
    "- 1 Jupyterカーネルを再起動\n",
    "- 2 <code>tf.reset_default_graph</code>実行でdefault graphをreset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-4] Lifecycle of a node value p.247"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "15\n"
     ]
    }
   ],
   "source": [
    "#ノードを評価する時、TensorFlowはノードの依存集合を自動で決定\n",
    "# 以下事例\n",
    "w = tf.constant(3)\n",
    "x = w + 2\n",
    "y = x + 5\n",
    "z = x * 3\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    print(y.eval())  # 10\n",
    "    print(z.eval())  # 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ノード値ライフサイクルの特徴\n",
    "- yの評価グラフで、TensorFlowはyがwに依存することを自動検出\n",
    " - 最初にw、次にx、そしてyを評価して、yの値を返す\n",
    " - z評価でも同様に計算される。wとxはyの値を再利用せず、再度計算する。\n",
    " \n",
    " - グラフの実行の間に、sessionで使う変数以外は全ノードはドロップされる(12章で説明あり)。\n",
    "\n",
    "変数のライフサイクル\n",
    " - イニシャライザが実行されるとライフ開始\n",
    " - セッションが終了するとライフ終了\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "15\n"
     ]
    }
   ],
   "source": [
    "# 2回x,wを計算するのが望ましくない場合は、withで一つのグラフとしてy,zを定義する。\n",
    "with tf.Session() as sess:\n",
    "    y_val, z_val = sess.run([y, z])\n",
    "    print(y_val)  # 10\n",
    "    print(z_val)  # 15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "＜参考情報＞\n",
    "1. シングルプロセスのTensorFlow\n",
    " - 複数セッションは共有されない\n",
    " - 同じグラフを再利用しても共有されない\n",
    " - 各セッションは全変数の都度コピーを持つ\n",
    "\n",
    "2. Distributed TensorFlowで（12章)\n",
    " - 複数セッションで同じ変数を共有できる\n",
    " - 変数の状態はセッションではなくサーバーに格納される"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-5] Linear Regression with Tensorflow p.248\n",
    "テンソルフローでの線形回帰"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow操作（operationsの略＝ops）\n",
    " - 任意の数の入力、任意の数の出力を生成\n",
    " - 例えば、加算演算と乗算演算それぞれ2つの入力を取り、1つの出力を生成\n",
    " - 定数と変数には入力が無い（ソース操作と呼ばれます）。\n",
    " - 入力と出力は、テンソルと呼ばれる多次元配列\n",
    "     - \"テンソルの流れ\"というツール名の由来\n",
    " - NumPy配列と同様に、テンソルは型と形状を持つ\n",
    " - 実際には、Python APIのテンソルは単にNumPy ndarrayで表される\n",
    " - 通常は浮動小数点数を含むが、文字列（任意のバイト配列）を渡すために使用可\n",
    " - 上記例では、テンソルは単なるスカラー値だが、もちろん、多次元配列で実行可\n",
    " - 例：次コードは、2次元配列を操作\n",
    "     - カリフォルニア住宅のデータセット（第2章で紹介）に線形回帰を実行  \n",
    "     - コードの工程\n",
    "         1. データセットのフェッチ\n",
    "         2. 全訓練インスタンスに、バイアス入力特徴（x0 = 1）追加。NumPyで実行。\n",
    "         3. データを保持する2つのTensorFlow定数ノードXおよびyを作成\n",
    "         4. ターゲット4を使用しており、Tensor-シータを定義するフロー。\n",
    "         5. 関数（transpose（転置）, matmul(行列乗算),matrix_inverse（逆行列））は、                 \n",
    "            すぐに計算を実行せずノードをグラフ中に作成、実行\n",
    "         6. 正規方程式（第4章を参照）でthetaを定義\n",
    "         7. 最後にセッションを作成し、thetaを評価する\n",
    " - 注意：housing.targetは、thetaを計算する為に、列ベクトルにreshapeする必要があります。numpyの多次元配列を一次元配列にしたいときは、1行へ変換：reshape(-1,),または1列に変換:reshape(-1,1)を使います。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Normal Equation　p.248\n",
    "4章で出てくるthetaを計算して評価する\n",
    "\n",
    "$$\n",
    "theta = （X^{T}*X)^{-1} * X^{T} * ｙ \n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import tarfile\n",
    "from six.moves import urllib\n",
    "DOWNLOAD_ROOT = \"https://raw.githubusercontent.com/ageron/handson-ml/master/\"\n",
    "HOUSING_PATH = \"datasets/housing\"\n",
    "HOUSING_URL = DOWNLOAD_ROOT + HOUSING_PATH + \"/housing.tgz\"\n",
    "def fetch_housing_data(housing_url=HOUSING_URL, hpath=HOUSING_PATH):\n",
    "    os.makedirs(hpath, exist_ok=True)\n",
    "    tgz_path = os.path.join(hpath, \"housing.tgz\")\n",
    "    urllib.request.urlretrieve(housing_url, tgz_path)\n",
    "    housing_tgz = tarfile.open(tgz_path)\n",
    "    housing_tgz.extractall(path=hpath)\n",
    "    housing_tgz.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "def load_housing_data(housing_path=HOUSING_PATH):\n",
    "    csv_path = os.path.join(housing_path, \"housing.csv\")\n",
    "    return pd.read_csv(csv_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': array([[   8.3252    ,   41.        ,    6.98412698, ...,    2.55555556,\n",
      "          37.88      , -122.23      ],\n",
      "       [   8.3014    ,   21.        ,    6.23813708, ...,    2.10984183,\n",
      "          37.86      , -122.22      ],\n",
      "       [   7.2574    ,   52.        ,    8.28813559, ...,    2.80225989,\n",
      "          37.85      , -122.24      ],\n",
      "       ..., \n",
      "       [   1.7       ,   17.        ,    5.20554273, ...,    2.3256351 ,\n",
      "          39.43      , -121.22      ],\n",
      "       [   1.8672    ,   18.        ,    5.32951289, ...,    2.12320917,\n",
      "          39.43      , -121.32      ],\n",
      "       [   2.3886    ,   16.        ,    5.25471698, ...,    2.61698113,\n",
      "          39.37      , -121.24      ]]), 'target': array([ 4.526,  3.585,  3.521, ...,  0.923,  0.847,  0.894]), 'feature_names': ['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude'], 'DESCR': 'California housing dataset.\\n\\nThe original database is available from StatLib\\n\\n    http://lib.stat.cmu.edu/\\n\\nThe data contains 20,640 observations on 9 variables.\\n\\nThis dataset contains the average house value as target variable\\nand the following input variables (features): average income,\\nhousing average age, average rooms, average bedrooms, population,\\naverage occupation, latitude, and longitude in that order.\\n\\nReferences\\n----------\\n\\nPace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,\\nStatistics and Probability Letters, 33 (1997) 291-297.\\n\\n'}\n",
      "\n",
      "\n",
      "housing.data(type)= <class 'numpy.ndarray'> \n",
      "\n",
      "housing.data(m,n)= 20640 8 \n",
      "\n",
      "housing.target(type)= <class 'numpy.ndarray'> \n",
      "\n",
      "<class 'sklearn.datasets.base.Bunch'>\n"
     ]
    }
   ],
   "source": [
    "# tensorflowの場合Normal Equation使う\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.datasets import fetch_california_housing\n",
    "\n",
    "reset_graph()\n",
    "\n",
    "#------housingデータ取得--------------------\n",
    "housing = fetch_california_housing()\n",
    "print(housing)\n",
    "print(\"\\n\\nhousing.data(type)=\",type(housing.data), \"\\n\") \n",
    "\n",
    "m, n = housing.data.shape #行数、列数\n",
    "print(\"housing.data(m,n)=\",m, n, \"\\n\") \n",
    "\n",
    "print(\"housing.target(type)=\",type(housing.target), \"\\n\") \n",
    "#m2, n2 = housing.target.dim() #行数、列数\n",
    "#print('housing.target(m,n)=',m2, n2, \"\\n\") \n",
    "\n",
    "print(type(housing))\n",
    "\n",
    "#housingデータの構造\n",
    "#   housing.data\n",
    "#   housing.target\n",
    "#   housing.feature_names\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJsAAANhCAYAAABAWbrsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X+0XWV56PvvI7GFJhDlx9nDmx5JoVJ6QgjVjGpva7vv\nBfWKh5ZB6G2u6CX2coN10HokLU2HaKLQIzlCb4cChVwURH5cBQIeBD1DTtmlp7ZarISM3OYwEANF\nZRSQkuwEoul97h9zLlxZWWvvrL1+zLnW/n7GWCN7vu+caz1v1t5zrvms90dkJpIkSZIkSVI/vKrq\nACRJkiRJkjQ+TDZJkiRJkiSpb0w2SZIkSZIkqW9MNkmSJEmSJKlvTDZJkiRJkiSpb0w2SZIkSZIk\nqW9MNkmSJEmSJKlvTDZJXYqIoyPi7ojYExFPRsS7q45JkjRaIuKiiHg4IvZFxE1VxyNJGi0R8dMR\n8ZnyfmR3RDwSEe+sOi6pYUHVAUgj6BrgR8AEcBpwX0Rszczt1YYlSRoh3wcuB94BHFFxLJKk0bMA\n+CfgN4CngDOBL0bE8szcWWVgEkBkZtUxSCMjIhYCLwCnZOZjZdnNwPczc32lwUmSRk5EXA78bGau\nqToWSdJoi4hHgY9l5l1VxyI5jE7qzknA/kaiqbQVWFZRPJIkSZLmuYiYoLhXcbSFasFkk9SdRcCu\nlrJdwJEVxCJJkiRpnouIVwO3Ap/LzB1VxyOBySapW9PAUS1li4HdFcQiSZIkaR6LiFcBn6eYU/ai\nisORXmGySerOY8CCiHhDU9kK7K4qSZIkaYgiIoDPUCxctCozf1xxSNIrTDZJXcjMPcAW4OMRsTAi\nfg34TYpvEyRJOiQRsSAiDgcOAw6LiMMjwlWCJUnd+AvgF4GzMvOlqoORmplskrr3AYplqv8ZuA34\nvcy0Z5MkqRuXAi8B64H3lD9fWmlEkqSRERHHAxcCpwHPRMR0+Tiv4tAkACIzq45BkiRJkiRJY8Ke\nTZIkSZIkSeobk02SJEmSJEnqG5NNkiRJkiRJ6huTTZIkSZIkSeqbsVti99hjj82lS5ceULZnzx4W\nLlxYTUCzqGtsdY0LjG2ujG1u6hDbt771recy87hKg5hn2l1LWtXhd2NQbNvoGuf22bbeeC0ZvkO5\nlrSq+++58fXG+HpT5/jqHBv0L76uriWZOVaPN73pTdnqwQcfPKisLuoaW13jyjS2uTK2ualDbMDD\nWYPz63x6tLuWtKrD78ag2LbRNc7ts2298VpSz2tJq7r/nhtfb4yvN3WOr86xZfYvvm6uJQ6jkyRJ\nkiRJUt+YbJIkSZIkSVLfmGySJEmSNBIi4qKIeDgi9kXETS11PxMR10bEcxHxYkQ81FQXEbEpIp4v\nH5siIprql0bEgxGxNyJ2RMQZLc/97oh4MiL2RMQ9EXH0wBsrSSPMZJMkSZKkUfF94HLgs23qNgNH\nA79Y/vuhprq1wNnACuBU4Czgwqb624FvA8cAHwbujIjjACJiGXA98F5gAtgLXNu3FknSGDLZJEmS\nJGkkZOaWzLwHeL65PCJOBn4TWJuZz2bmv2bmt5p2OR+4KjOfzszvAVcCa8pjTwLeCGzIzJcy8y7g\nUWBVeex5wL2Z+VBmTgMfAc6JiCMH11JJGm0mmyRJkiSNul8GngQ+Vg6j2xYRq5rqlwFbm7a3lmWN\nuicyc/cM9a8cm5nfAfYBJ/W3CZI0PhZUHYAkSZIk9ehngVOAu4D/AfgV4L6I+H8z8x+BRcCLTfvv\nAhaV8za11jXql5Q/d6o/qGdTRKylGLLHxMQEU1NTXTVienq662OGyfh6Y3y9qXN8dY4NqonPZJMk\nSZKkUfcS8GPg8szcD/xVRDwIvB34R2AaOKpp/8XAdGZmRLTWNeobPZ1mq39FZm6mmDuKlStX5uTk\nZFeNmJqaottjhsn4emN8valzfHWODaqJr+dhdFWtCCFJkiRJpUfblGXTz9spJgdvWFGWNepOaJmD\nqbX+lWMj4kTgp4DHeoxZksZWP3o2NVaEeAdwREvd5vI1fhH4IXBaU13zihAJfA34LnBdWX878LfA\nmeXjzoh4Q2Y+24eYx9rS9fe1Ld95xbuGHIkkSZLmqt1nuvn+eS4iFlDcXxwGHBYRhwP7gYeAp4A/\niYhPAG8G/ifgkvLQm4GLI+L+cnsd8CmAzHwsIh4BNkTEpRT3HssphuQB3Ar8bUS8FfgH4DJgS8sc\nTwPnZ3xJo6Tnnk0VrQghSZIkaf65lGLI3HrgPeXPl2bmj4HfokgUvQj838D/npk7yuOuB+4FtpWP\nL5dlDauBlcALwCeAcxtfcmfmduD9FEmnfwYWAh8YXBMlafQNcs6m5hUh3gv8ANhYJo6gtxUhDjDb\nRHx1nqxrELGtW76/bXk3rzPf/s/6xdjmxtgkSdKhyMyNwMYOddspJgZvV5cUvZwu6VC/E5ic4XVv\nA27rJlZJms8GmWwa5IoQB5htIr46T9Z1qLF10212Tad9z5v9dbqNqwrGNjfGNjd1jk2SJEmS6qjn\nYXQzaF4R4keZ+VdAY0UImGFFiDZ1jfqhjouWJEmSJElSdwaZbBrkihCSJEmSJEmqoZ6TTRGxoFwF\n4pUVIcpVIppXhFgQEb9KsSLEfykPbawIsSQillCsCHETFCtCAI0VIQ6PiHM4cEUISZIkSZIk1VA/\nejYNfUUISZIkSZIk1VPPE4RXtSKEJEmSJEmS6meQczZJkiRJkiRpnjHZJEmSJEmSpL4x2SRJkiRJ\nkqS+6XnOJlVr6fr7qg5BkiRJkiTpFfZskiRJkiRJUt/Ys2keadcLaucV76ogEkmSJEmSNK7s2SRJ\nkiRJkqS+MdkkSZIkSZKkvjHZJEmqXERcFBEPR8S+iLipqfwtEfG1iPhhRDwbEXdExOua6iMiNkXE\n8+VjU0REU/3SiHgwIvZGxI6IOGPITZMkSZLmHZNNkqQ6+D5wOfDZlvLXApuBpcDxwG7gxqb6tcDZ\nwArgVOAs4MKm+tuBbwPHAB8G7oyI4/ofviRJkqQGk02SpMpl5pbMvAd4vqX8K5l5R2buysy9wNXA\nrzbtcj5wVWY+nZnfA64E1gBExEnAG4ENmflSZt4FPAqsGnyLJEmSpPnL1egkSaPk14HtTdvLgK1N\n21vLskbdE5m5u0P9ASJiLUVPKSYmJpiampoxkOnp6Vn3GVW2bXSNc/vmY9vWLd9/UNm4/h9IksaL\nySZJ0kiIiFOBjwK/1VS8CHixaXsXsKict6m1rlG/pN3zZ+ZmiiF7rFy5MicnJ2eMZ2pqitn2GVW2\nbXSNc/vmY9vWrL/voLKd5x2833wSERdR9GBdDtyemWva7PNR4GPA2zLzgbIsgCuAC8rdbgDWZ2aW\n9Usphmm/GXgKuKhxbFn/buATwLHA14Dfzcwf9r2BkjQmHEYnSaq9iPh54CvABzPzr5uqpoGjmrYX\nA9PlzUNrXaN+N5KkUdVpjj8AIuJE4LeBH7RUzXmOv4hYBlwPvBeYAPYC1/anOZI0nkw2SZJqLSKO\nBx4ALsvMz7dUb6e4cWhYwU+G2W0HToiIIzvUS5JGTKc5/ppcA/wx8KOW8l7m+DsPuDczH8rMaeAj\nwDkt1xdJUhOH0UmSKhcRCyiuSYcBh0XE4cB+im+Q/xK4OjOva3PozcDFEXF/ub0O+BRAZj4WEY8A\nGyLiUuBMimEXdw20MZKkSkTEbwP7MvP+YtTcAXqZ428Z8PVGRWZ+JyL2AScB32qJoav5/1rNNDdZ\nuzm8YLjzeNV97jTj643xzV2dY4Nq4jPZJEmqg0uBDU3b76GYbyOBE4CNEbGxUZmZi8ofry/rt5Xb\nN5RlDauBm4AXKObgODczn+1/+JKkKpW9jP4j8LYOu/Qyx1+n+oN6NnU7/1+rmeYmazeHFwx3Hq+6\nz51mfL0xvrmrc2xQTXw9D6OLiIsi4uGI2BcRN3XY56MRkRFxRlNZRMSmiHi+fGyKpq8gImJpRDwY\nEXsjYkfzsZKk8ZKZGzMzWh4bM/Nj5c+Lmh9Nx2VmXpKZR5ePSxqTvZb1OzNzMjOPyMxfaJ7sVZI0\nVjYCn8/MnR3qe5njzzkAJalL/ZizaeiT9EmSJElSk9OBP4iIZyLiGeDfAl+MiD8u63uZ4++AY8v7\nm58CHut7KyRpTPScbKpokj5JkiRJ80xELCjn9Xtljr9y3r/TgVOA08rH9ym+yL6mPLQxx9+SiFhC\nMcffTVDM8Qc05vg7PCLO4cA5/m4FzoqIt0bEQuAyYEvLHE+SpCYDnbNpgJP0tb7OjBPx1XmyrkON\nrdOEgL3q9Nr//MMX+fStXzqofPmSxQOJoxvj8H5Wwdjmps6xSZI0D7Wd4y8zNzbvFBH/CrxQrh4H\nPczxl5nbI+L9FEmnYyhWSH1f/5okSeNnYMmmAU/Sd4DZJuKr82RdhxpbpwkBe9VpQsFP3/olrtp2\n8K/HMCcg7GQc3s8qGNvc1Dk2SZLmmzKptPEQ9lvasp3AJeWj3f47gckZnu824LZDjVOS5rt+zNnU\nyUYGN0mfJEmSJEmSamiQyaZBTtInSZIkSZKkGup5GF05Id8CmibpA/ZTJJte3bTr3wMXA18ptxuT\n9N1fbq8DPgXFJH0R0Zik71LgTA6cpE+SJEmSJEk11I85m4Y+Sd84WzqguZkkSZIkSZKGoedkU1WT\n9EmSJEmSJKl+BjlnkyRJkiRJkuaZfgyj0xwtXX8f65bvZ41D5yRJkiRJ0piwZ5MkSZIkSZL6xmST\nJEmSJEmS+sZkkyRJkiRJkvrGZJMkSZIkSZL6xmSTJEmSJEmS+sZkkyRJkiRJkvrGZJMkSZIkSZL6\nxmSTJEmSJEmS+sZkkyRJkiRJkvrGZJMkSZIkSZL6ZkHVAahaS9ff17Z83fIhByJJkiTNIiIuAtYA\ny4HbM3NNWf4W4DLgTcC/AlPAH2TmD8r6AK4ALiif6gZgfWZmWb8UuBF4M/AUcFFmPtD0uu8GPgEc\nC3wN+N3M/OHAGipJI86eTZKkykXERRHxcETsi4ibWupOj4gdEbE3Ih6MiOOb6iIiNkXE8+VjU3lD\n0ahfWh6zt3yOM4bYLElS/30fuBz4bEv5a4HNwFLgeGA3RfKoYS1wNrACOBU4C7iwqf524NvAMcCH\ngTsj4jiAiFgGXA+8F5gA9gLX9rFNkjR2TDZJkuqg7c1DRBwLbAE+AhwNPAx8oWmXOd88SJJGT2Zu\nycx7gOdbyr+SmXdk5q7M3AtcDfxq0y7nA1dl5tOZ+T3gSooeUkTEScAbgQ2Z+VJm3gU8Cqwqjz0P\nuDczH8rMaYpr0jkRceTgWipJo81hdJKkymXmFoCIWAn8bFPVOcD2zLyjrN8IPBcRJ2fmDppuHsr6\nKykSUNc13Ty8PTNfAu6KiA9S3DxcN5yWSZIq8uvA9qbtZcDWpu2tZVmj7onM3D1D/dcbFZn5nYjY\nB5wEfKv5RSNiLcV1iImJCaamproKenp6uuMx65bvb1ve7Wv0Yqb46sD4emN8c1fn2KCa+Ew2SZLq\n7ICbg8zcExGPl+U7Wuvp7uZBkjSGIuJU4KPAbzUVLwJebNreBSwqh1631jXql3Q4tlF/UM+mzNxM\nMZyPlStX5uTkZFexT01N0emYNR3mWt15Xnev0YuZ4qsD4+uN8c1dnWODauLrOdlU1SR9kqR5YRHw\nbEtZ8wf8Xm4eDtDtt9F1/warF7ZtdI1z++Zj29r1ZBnX/4N+iYifB74CfDAz/7qpaho4qml7MTCd\nmRkRrXWN+t0djm2tlyS16EfPpsY8G+8Ajmgqb0zS91+A/RTjpm8E/peyvnmejaRY1eG7/GRow+3A\n3wJnlo87I+INmdl60yFJGl/d3gB0c/NwgG6/ja77N1i9sG2ja5zbNx/b1q4nyzB7sYyacgGJB4DL\nMvPzLdXbKe47vllur+Anw+y2AydExJFNvWFXALe2HNt4nROBnwIe63sjJGlM9DxBeEWT9EmS5ofW\nD/gLgRM58AZhRdP+bW8eOtRLkkZMRCyIiMOBw4DDIuLwsmwJ8JfA1ZnZbl6+m4GLI2JJue864CaA\nzHwMeATYUD7fORSjNu4qj70VOCsi3lpehy4DtrQM05YkNRnmnE39nKTvALMNfahrt+t1y/czcUTn\nyf6q1CmuOvw/1vX9BGObK2NTRCyguCa9cvNA0Sv2buCTEbEKuA/YAGwtJweHn9w83F9urwM+BcXN\nQ0Q0bh4upegl23zzIEkaPZdSXAsa3gN8jGKkxAnAxnIxCQAyc1H54/Vl/bZy+4ayrGE1RfLpBYop\nPM5tjKjIzO0R8X6KpNMxFL2n3tfPRknSuBlKsmkAk/QdYLahD3Xtdr1m/X2sW76fq7bVb572TnHV\noet2Xd9PMLa5MjbR4eYhMzeWiaargVuAb1DcEDTM+eZBkjR6MnMjsLFD9cdmOC6BS8pHu/qdwOQM\nx98G3HZoUUqSBp7lGNAkfZKkMTLTzUO5OMTJHep6unmQJEmS1H89z9k0k0OcpK/BeTYkSZIkSZJG\nXM/Jpoom6ZMkSZIkSVIN9WMY3dAn6ZMkSZIkSVI99ZxsqmqSPkmSJEmSJNXPQOdskiRJkiRJ0vxi\nskmSJEmSJEl9Y7JJkiRJkiRJfWOySZIkSZIkSX1jskmSJEmSJEl9Y7JJkiRJkiRJfWOySZIkSZIk\nSX1jskmSJEmSJEl9Y7JJkiRJkiRJfWOySZIkSZIkSX1jskmSJEmSJEl9Y7JJkiRJkiRJfbOg6gAk\nSZI0+pauv69t+c4r3jXkSCRJUtXs2SRJkiRpJETERRHxcETsi4ibWupOj4gdEbE3Ih6MiOOb6iIi\nNkXE8+VjU0REU/3S8pi95XOc0fLc746IJyNiT0TcExFHD7yxkjTCTDZJkiRJGhXfBy4HPttcGBHH\nAluAjwBHAw8DX2jaZS1wNrACOBU4C7iwqf524NvAMcCHgTsj4rjyuZcB1wPvBSaAvcC1fW6XJI0V\nk02SJEmSRkJmbsnMe4DnW6rOAbZn5h2Z+TKwEVgRESeX9ecDV2Xm05n5PeBKYA1ARJwEvBHYkJkv\nZeZdwKPAqvLY84B7M/OhzJymSGidExFHDqyhkjTiep6zKSIuojhRLwduz8w1TXWnA9cArwe+AazJ\nzCfLugCuAC4od78BWJ+ZWdYvBW4E3gw8BVyUmQ/0Gq8kSZKksbMM2NrYyMw9EfF4Wb6jtb78eVnT\nsU9k5u4Z6r/e9NzfiYh9wEnAt5qDiIi1FL2omJiYYGpqqqtGTE9Pdzxm3fL9bcu7fY1ezBRfHRhf\nb4xv7uocG1QTXz8mCG90ZX0HcESjsKkr6wXAvcBlFF1Z31Lu0tyVNYGvAd8Frivrbwf+FjizfNwZ\nEW/IzGf7ELMkSZKk8bEIaL1P2AUc2VT/YkvdovIL8Na6Rv2SDse2PvcrMnMzsBlg5cqVOTk52VUj\npqam6HTMmk6T8J/X3Wv0Yqb46sD4emN8c1fn2KCa+HpONmXmFoCIWAn8bFPVK11Zy/qNwHMRcXJm\n7qCpK2tZfyVFAuq6pq6sb8/Ml4C7IuKDFF1Zr2PEdFqdRZIkadT4uUY1NQ0c1VK2GNjdoX4xMJ2Z\nGRHdHttaL0lq0Y+eTZ0MsivrAWbrrlp1l7ZOXV4BJo6Yub4qneKqQ9fAqt/PmRjb3BibZlMOrb4W\n+BVgH3An8B8yc38vQ7YlSWNjO8WX2QBExELgxLK8Ub8C+Ga5vaKl7oSIOLLp/mMFcGvLsY3nPhH4\nKeCx/jdDksbDIJNNg+zKeoDZuqtW3aWtU5dXKBI6V20b5NswN53iGmY33U6qfj9nYmxzY2w6BNdS\nXFNeB7yGYuj1ByLiNnobsi1JGiERsYDiHuYw4LCIOBzYD9wNfDIiVgH3ARuAreWICoCbgYsj4v5y\nex3wKYDMfCwiHgE2RMSlFFN4LAfuKve9FfjbiHgr8A8U15otLV+MS5KaDDLLMciurKpIu67zO694\nVwWRSJpnfg64ulxh6JmI+CpFb9c5D9muoA2SpN5dSpFIangP8LHM3Fgmmq4GbqHo6bq6ab/rgROA\nbeX2DWVZw2rgJuAFisWJzm3MFZuZ2yPi/RRJp2OAB4D39bdZkjReBplsGmRXVknS/PLnwO9ExBTw\nWuCdFEtPTzL3IdsH6HYFoXEeYmnbRtcw2tft8P9+xTPO712nttV1SoMqZeZGYGOHugeAkzvUJXBJ\n+WhXv5PimtLpdW8DbusmVkmaz3pONlXUlVWSNL88RJEI2kVxvfkccA/w75njkO3WeZu6XUFonIdY\n2rbRNYz2zTQ9QDv9GoI/zu9dp7a1+7+uw5QGkiTN5lV9eI5LgZeA9RTdWF8CLi27na4C/pSiO+ov\nc3BX1nspurJuA77MwV1ZV5bHfoKmrqySpPkjIl4FfJVibqaFwLEUvZs20cOQ7UHGLEmSJM1nPfds\nqqorqyRp3jiaYqW5qzNzH7AvIm4ELqfoETvXIduSJEmSBqAfPZskSRqYzHyOYgW590fEgoh4DUWC\n6VGKIdunRMSqchh3pyHbSyJiCcWQ7ZuG3ghJkiRpHhnkBOGSJPXLORSThK8H/hX4S+BDmflsj6sP\nqSbarXYKrngqSZI0ikw2qWfeIEgatMx8hA5Dq3sZsi1JkiSp/xxGJ0mSJEmSpL4x2SRJkiRJkqS+\nMdkkSZIkSZKkvjHZJEmSJEmSpL4x2SRJkiRJkqS+MdkkSZIkSZKkvjHZJEmSJEmSpL4x2SRJkiRJ\nkqS+MdkkSZIkSZKkvjHZJEmSJEmSpL5ZUHUAkiRJnSxdf1/b8p1XvGvIkUiSJOlQmWySJElSW52S\nfVJdRcRS4FrgV4B9wJ3Af8jM/RFxOnAN8HrgG8CazHyyPC6AK4ALyqe6AVifmdn0vDcCbwaeAi7K\nzAeG0ihJGkEOo5MkSZI0Lq4FngVeB5wG/AbwgYg4FtgCfAQ4GngY+ELTcWuBs4EVwKnAWcCFTfW3\nA98GjgE+DNwZEccNtCWSNMJMNkmSJEkaFz8HfCEzX87MZ4CvAsuAc4DtmXlHZr4MbARWRMTJ5XHn\nA1dl5tOZ+T3gSmANQEScBLwR2JCZL2XmXcCjwKohtkuSRsrAh9ENqiurJEmSJLX4c+B3ImIKeC3w\nToreTJPA1sZOmbknIh6nSETtKP/d2vQ8W8syyn+fyMzdHepfERFrKXpJMTExwdTUVFfBT09Pdzxm\n3fL9bcu7fY1ezBRfHRhfb4xv7uocG1QT3zDmbGruyvoa4GsUXVlvo+jKegFwL3AZRVfWt5THNXdl\nzfK47wLXDSFmSZIkSaPnIYr7iF3AYcDngHuAf09xT9JsF3Bk+fMi4MWWukXlF+CtdY36Ja0vnpmb\ngc0AK1euzMnJya6Cn5qaotMxazotmHBed6/Ri5niqwPj643xzV2dY4Nq4htGsunngKvL7qrPRMRB\nXVkBImIj8FxEnJyZO2jqylrWX0lx4TDZNCJcQUiSJEnDEhGvohg2txn4HymSRJ8FNgHTwFEthywG\nGr2VWusXA9OZmREx27GSpBbDSDYNoivrAWbrrlp1l7ZOXV4BJo6Yub4qg4yr1/ei6vdzJsY2N8Ym\nSZL64GiK6Tmuzsx9wL6IuBG4HPgUxZfZAETEQuBEYHtZtJ1iRMU3y+0VLXUnRMSRTUPpVgC3DrAt\nkjTShpFs6ntX1tZ5m2brrlp1l7ZOXV6hSOhctW0Yb0N3BhlXr119q34/Z2Jsc2NskiSpV5n5XER8\nF3h/RFxFcT9xPsVk3ncDn4yIVcB9wAZgazmiAuBm4OKIuL/cXkeRoCIzH4uIR4ANEXEpcCawHLhr\nSE2TpJEz0NXomrqybgEWAsdS9G7qqSvrIGOWJEmSNLLOoRhJ8SzwOPBj4EOZ+SzF6nF/CrwA/DKw\nuum46ynmkd1WPr5cljWsBlaWx34COLd8TklSG4PuUjOorqySJEmSdIDMfIRiuo52dQ8AJ3eoS+CS\n8tGufmen55UkHWygPZsy8zmKFeTeHxELIuI1HNiV9ZSIWBURh9O5K+uSiFhC0ZX1pkHGK0mSJEmS\npN4MNNlUGlRXVkmSJEmSJNXMwGemHlRXVknS/BIRqyl6wb4eeAZYk5l/HRGnA9eU5d8oy58sjwng\nCuCC8mluANY7/5+qsLTNgiE7r3hXBZEcrF1skiRJczWMnk2SJPUkIt5GsbjE+yhWLf114ImIOJZi\nEYqPUMwT+DDwhaZD1wJnU8z7dypwFnDh8CKXJEmS5h+TTZKkUfAx4OOZ+XeZ+f9l5vcy83sUQ7W3\nZ+YdmfkysBFYERGNXrPnA1dl5tPl/lcCayqIX5IkSZo3Bj6MTpKkXkTEYRTLTf/niHgcOBy4B/gj\nYBmwtbFvZu4p91kG7GitL39e1uF11lL0hGJiYoKpqakZ45qenp51n1FVRdvWLd/f1f5zja/K961d\nG/sdy1zb1+3/fzf61cb5+Dc3jN8ZSZIGwWSTJKnuJoBXA+cCb6VYaOJLwKXAIooFKJrtohhqR1n/\nYkvdooiI1nmbMnMzsBlg5cqVOTk5OWNQU1NTzLbPqKqibWu6nDNo53mTc3qdKt+3dm2cazs6mWv7\nuv3/70a/2jgf/+aG8TsjSdIgmGzS0NV5glRJtfRS+e+nM/MHABHxZxTJpoeAo1r2XwzsLn+ebqlf\nDEw7QbgkSZI0OM7ZJEmqtcx8AXgaaE4QNX7eTjH5NwARsRA4sSw/qL78eTuSJEmSBsZkkyRpFNwI\n/H5E/JuIeC3wIeDLwN3AKRGxKiIOBzYAWzNzR3nczcDFEbEkIpYA64Cbhh++JEmSNH84jE6SNAou\nA44FHgNeBr4I/GlmvhwRq4CrgVuAbwCrm467HjgB2FZu31CWacTVeUh2u9gkSZLmE5NNkqTay8wf\nAx8oH611DwAndzgugUvKhyRMhkmSpMFzGJ0kSZIkSZL6xp5NfeQ3hZIkSZIkab6zZ5MkSZIkSZL6\nxp5NkiRJGpg6T+YuSZIGw55NkiRJkiRJ6huTTZIkSZLGRkSsjoh/jIg9EfGdiHhrWX56ROyIiL0R\n8WBEHN90TETEpoh4vnxsiohoql9aHrO3fI4zqmibJI0Kk02SJEmSxkJEvA3YBLwPOBL4deCJiDgW\n2AJ8BDimytFaAAAgAElEQVQaeBj4QtOha4GzgRXAqcBZwIVN9bcD3waOAT4M3BkRxw20MZI0wpyz\nSZIkDZWrt0oaoI8BH8/Mvyu3vwcQEWuB7Zl5R7m9EXguIk7OzB3A+cBVmfl0WX8lRQLquog4CXgj\n8PbMfAm4KyI+CKwCrhte0yRpdAylZ9MgurJKkiRJUkNEHAasBI6LiMcj4umIuDoijgCWAVsb+2bm\nHuDxspzW+vLn5ronMnN3h3pJUouB92xq6sr6O8A3gdeV5Y2urBcA9wKXUXRlfUt5aHNX1gS+BnwX\nvz2QJEltdOox5cpn0rwxAbwaOBd4K/Bj4EvApcAi4NmW/XdRDLWjrH+xpW5R+WV3a12jfklrAGUP\nqrUAExMTTE1NddWA6enpjsesW76/bXm3r9GLmeKrA+PrjfHNXZ1jg2riG8Ywur53ZR1CzJIkSZJG\ny0vlv5/OzB8ARMSfUSSbHgKOatl/MdDorTTdUr8YmM7MjIjWutZjX5GZm4HNACtXrszJycmuGjA1\nNUWnY9Z0Sqif191r9GKm+OrA+HpjfHNX59igmvgGmmxq6sr6nyPiceBw4B7gj2jTlbXcZxmwo7We\nGbqqzvYNwrCyeJ2+bZjJxBFzO27Qhh1XN+9PnbPGxjY3xiZpvrI3ltQ/mflCRDxNMSrileLy3+0U\nX2YDEBELgRPL8kb9CoqRGJQ/N9edEBFHNg2lWwHc2vdGSNKYGHTPpoF0Zc3M5gvIrN8gDCuL1+nb\nhpmsW76fq7bVb572YcfVzTcydc4aG9vcGJskSeqTG4Hfj4ivUtx7fAj4MnA38MmIWAXcB2wAtpYj\nKgBuBi6OiPvL7XXApwAy87GIeATYEBGXAmcCy4G7htQmSRo5g84mDKQr6+DCVVX8ZleSNGrqsqpe\ncxzrlu9/5csvr6Gapy4DjgUeA14Gvgj8aWa+XCaargZuAb4BrG467nrgBGBbuX1DWdawGrgJeAF4\nCjg3M1u/OJcklQaabBpgV1ZJkqSx5Zcw0txk5o+BD5SP1roHgJM7HJfAJeWjXf1OYLJfcUrSuBvG\nOKm+d2WVJEk6VPb8kSRJGq5hJJsG1ZVVkiRpXulm6F5dhvlJkqT5Z+DJpkF1ZZUkSZIkSVL91G8Z\nNEmSpCFo1/On09A6ewlJkiQduldVHYAkSZIkSZLGh8kmSZIkSZIk9Y3JJkmSJEmSJPWNySZJ0siI\niDdExMsRcUtT2ekRsSMi9kbEgxFxfFNdRMSmiHi+fGyKiKgmekmSJGl+MNkkSRol1wB/39iIiGOB\nLcBHgKOBh4EvNO2/FjgbWAGcCpwFXDisYCVJkqT5yNXoJEkjISJWA/8CfB34+bL4HGB7Zt5R7rMR\neC4iTs7MHcD5wFWZ+XRZfyVFAuq6IYc/L7mCmyRJ0vxkzyZJUu1FxFHAx4GLW6qWAVsbG5m5B3i8\nLD+ovvx5GZIkSZIGxp5NGkntvi3fecW7KohE0pBcBnwmM59umXJpEfBsy767gCOb6l9sqVsUEZGZ\n2XxQRKyl6PXExMQEU1NTMwY0PT096z6jql9tW7d8f+/B9NnEETPH9elbv9S2fN3yQUXUX7O1ry7m\n8vs1H//m2r2X4/p/IEkaLyabJEm1FhGnAWcAv9Smeho4qqVsMbC7Q/1iYLo10QSQmZuBzQArV67M\nycnJGeOamppitn1GVb/atqaGw+jWLd/PVdvG9+PPyLRv2562xTN9cTQf/+ba/Q3tPO/g/SRJqpsR\n+DQiSZrnJoGlwFNlr6ZFwGER8e8o5l46v7FjRCwETgS2l0XbKSYH/2a5vaKpTpIkSdIAOGeTJKnu\nNlMkkE4rH9cB9wHvAO4GTomIVRFxOLAB2FpODg5wM3BxRCyJiCXAOuCmIccvSZIkzSv2bJIk1Vpm\n7gX2NrYjYhp4OTOfLbdXAVcDtwDfAFY3HX49cAKwrdy+oSyTJEmSNCAmmyRJIyUzN7ZsPwCc3GHf\nBC4pH5Jqrt0CINDdIiD9eA5JktQbh9FJkiRJkiSpb+zZJEmSpLHXqcdTO/aCGm0R8QaK4dN3ZuZ7\nyrLTgWuA11MMuV6TmU+WdQFcAVxQPsUNwPrGyqURsRS4EXgz8BRwUdmrVpLUwdB6NkXEGyLi5Yi4\npans9IjYERF7I+LBiDi+qS4iYlNEPF8+NpUXAkmSJEnq5Brg7xsbEXEssAX4CHA08DDwhab91wJn\nU6xYeipwFnBhU/3twLeBY4APA3dGxHEDjF+SRt4wh9H1+6QvSZIkSa+IiNXAvwD/tan4HGB7Zt6R\nmS8DG4EVEdGY7+984KrMfDozvwdcCawpn+8k4I3Ahsx8KTPvAh4FVg2jPZI0qoYyjK7ppP914OfL\n4ldO+uU+G4HnIuLkcsnqV076Zf2VFAmo64YRsyRJkqTRERFHAR8H/md+MiQOYBmwtbGRmXsi4vGy\nfEdrffnzsqZjn8jM3R3qW2NYS3HPwsTEBFNTU121YXp6uuMx65bvb1ve7Wv0Yqb46sD4emN8c1fn\n2KCa+AaebBrQSV+SJEmSml0GfCYzn26ZfWMR8GzLvruAI5vqX2ypW1RO4dFa16hf0i6AzNwMbAZY\nuXJlTk5OdtWAqakpOh2zptNKi+d19xq9mCm+OjC+3hjf3NU5NqgmvmH0bOr7Sb8xWV/DbN8gDCuL\n1+nbhplMHDG34watLnF9+tYvHVRWxHbwvnXIJNc5o21sc1Pn2CRJUiEiTgPOAH6pTfU0cFRL2WJg\nd4f6xcB0ZmZEzHasJKmNgSabBnXSb32i2b5BGFYWr9O3DTNZt3w/V22r36KAdY0LOsc2zG91Oqlz\nRtvY5qbOsUmSpFdMAkuBp8ovuBcBh0XEv6OYhuP8xo4RsRA4EdheFm2nmCf2m+X2ipa6EyLiyKah\ndCuAWwfVEEkaB4OeIHySn5z0nwH+EFgVEf/AT07qwIwn/Ybmk74kSZIkNWymuJc4rXxcB9wHvAO4\nGzglIlZFxOHABmBrOU8swM3AxRGxJCKWAOuAmwAy8zHgEWBDRBweEecAy4G7htYySRpBg+66shn4\nf5q2/5Ai+fR75fYnI2IVxYWg00n//nJ7HfCpAccrSZKkmlm6/j7WLd9/QC/ynVe8q8KIVDeZuRfY\n29guh7+9nJnPlturgKuBW4BvAKubDr8eOAHYVm7fUJY1rKZIPr0APAWc23heSVJ7A002DfikL0mS\namDpHIaRS9IgZebGlu0HgJM77JvAJeWjXf1OihEbkqRDNNRJefp50pckSdL8NcgkZ7vntieVJEmH\nbtBzNkmSJEmSJGkeqedyYyPAIQOSJEmSJEkHs2eTJEmSJEmS+sZkkyRJkiRJkvrGYXSSJM1zrUPD\n1y3f77JLkiRJmjOTTZIk6SCd5iZ0RS5JkiTNxmSTxoY3RpIkSZpv2n0G9vOvpKqZbJIkqUZMnEv1\n5N+mJEmHzmSTJEk6ZJ1uuCVJkqQGV6OTJEmSJElS39izSWPPceySJEmSJA2PPZskSbUWET8dEZ+J\niCcjYndEPBIR72yqPz0idkTE3oh4MCKOb6qLiNgUEc+Xj00REdW0RJIkSZofTDZJkupuAfBPwG8A\ni4FLgS9GxNKIOBbYAnwEOBp4GPhC07FrgbOBFcCpwFnAhcMLXZIkSZp/HEYnSaq1zNwDbGwq+nJE\nfBd4E3AMsD0z7wCIiI3AcxFxcmbuAM4HrsrMp8v6KykSUNcNrwXD5yTekiRJqpLJJknSSImICeAk\nYDvwe8DWRl1m7omIx4FlwI7y361Nh28ty9o971qKRBQTExNMTU3NGMf09PSs+8zFuuX725Z381qd\nnuNQTRzR+3PU1Ti3Dca7fXVtWz/OA53OJ+3aO4jzjiRJ/WaySZI0MiLi1cCtwOcyc0dELAKebdlt\nF3Bk+fMi4MWWukUREZmZzQdl5mZgM8DKlStzcnJyxlimpqaYbZ+5WNOhV9LO8w79tTo9x6Fat3w/\nV20bz48I49w2GO/21bZt2/a0Le5mMZJO55N2f8vdnAskSapKDa/YkiQdLCJeBXwe+BFwUVk8DRzV\nsutiYHeH+sXAdGuiqSoOd5Ok/omInwauBc6gmMfvO8CfZOZXyvrTgWuA1wPfANZk5pNlXQBXABeU\nT3cDsL5xvYiIpcCNwJuBp4CLMvOBoTRMkkbQQCcIdwUhSVI/lOf/zwATwKrM/HFZtZ1i8u/GfguB\nE8vyg+rLn7cjSRpHg1xQ4nbg2xRzBX4YuDMijhtoayRphA26Z1PzCf8p4EyKE/5yim+bt1B8e3Av\ncBnFCf8t5bHNJ/wEvgZ8lzGf1FWS1NZfAL8InJGZLzWV3w18MiJWAfcBG4Ct5eTgADcDF0fE/eX2\nOuBTQ4p5KOwdJdVTu7/NbobWqXuDWlAiIk4C3gi8vbwG3RURHwRW4b2JJLU10GSTKwhJknpV9nq9\nENgHPNPUyfXCzLy1TDRdDdxCMSxiddPh1wMnANvK7RvKMknSmOvjghLLgCcyc3eH+ubX7GqxiVYz\nLT7RzQT5g5pIflCLY/SL8fXG+OauzrFBNfENdc6mqlYQGsR/bL9WQ6nryip1jQv6E9t8vAAb29zU\nObb5opxPo+Mw6nLOjJM71CVwSfmQJM0T/VxQok1do35J6+t2u9hEq5kWn+hm8YdBTSQ/qMUx+sX4\nemN8c1fn2KCa+IaWbKpyBaFB/Mf2utJPQ11XVqlrXNCf2ObjBdjY5qbOsWl+cbicJB2afi8oERGz\nHStJajHQCcIbxnEFIUmSJEn1MqAFJbYDJ0TEkR3qJUktBp5scgUhSZIkSUPSWFDirDYLSpwSEasi\n4nA6LyixJCKWUCwocRNAZj4GPAJsiIjDI+IcYDlw11BaJEkjaBjjpFxBSLXTaTiKq8RIkiSNpgEv\nKLGaIvn0AsUq2+dmZuuUIJKk0kCTTeOwgpBzZEiSJKkfOn2uXLd8f9/mA53PBrmgRGbuBCZ7DvIQ\neP8haRwMNNnkCkIaF+0u+vaCkiRJkiTpYPVcbkyqiN8kSZIkSZLUm6GsRidJkiRJkqT5wZ5N0hy1\n6wW1bvn+4QzmlyRJkiSppuzZJEmSJEmSpL4x2SRJkiRJkqS+cRid1GedJhl39TpJkiRJ0nxgzyZJ\nkiRJkiT1jckmSZIkSZIk9Y3D6KQhaTe8zqF1kiRJkqRxY88mSZIkSZIk9Y09myRJ6pNOCwRIkiRJ\n84k9myRJkiRJktQ39mySKtSpF8Sg5nIa9utJkiRJkuYfk00lhz5I6icnhJckSZI0XzmMTpIkSZIk\nSX1jzyaphvox3K31OdYt30+nP/luevbZO0eSJEmSNBOTTdIIqcPQLOd9kiRJkiTNpNbJpog4GvgM\n8HbgOeBPMvO2aqOS6sX5xqSZeS2RJPXKa4kkdafWySbgGuBHwARwGnBfRGzNzO3VhiXpUHXTG8te\nUxoQryWSpF6N1LXEz1SSqlbbZFNELARWAadk5jTw3yLiS8B7gfWVBifpIN30sGo3n9SaGY4fdu+t\ndh/E/NA2mryWSJJ6NU7XEj/PSBqWyMyqY2grIn4J+JvM/JmmsnXAZGae1bLvWmBtufkLwH9vebpj\nKbq71lFdY6trXGBsc2Vsc1OH2I7PzOMqjmEk9fla0qoOvxuDYttG1zi3z7b1xmvJHA34WtKq7r/n\nxtcb4+tNneOrc2zQv/gO+VpS255NwCJgV0vZLuDI1h0zczOwudMTRcTDmbmyv+H1R11jq2tcYGxz\nZWxzU+fYdEj6di1pNc6/G7ZtdI1z+2ybKjSwa0mruv8uGF9vjK83dY6vzrFBNfG9apgv1qVp4KiW\nssXA7gpikSSNJq8lkqReeS2RpC7VOdn0GLAgIt7QVLYCqOUkfJKkWvJaIknqldcSSepSbZNNmbkH\n2AJ8PCIWRsSvAb8JfH4OTzfnrqxDUNfY6hoXGNtcGdvc1Dk2zaLP15JW4/y7YdtG1zi3z7apEgO+\nlrSq+++C8fXG+HpT5/jqHBtUEF9tJwgHiIijgc8CbwOeB9Zn5m3VRiVJGiVeSyRJvfJaIkndqXWy\nSZIkSZIkSaOltsPoJEmSJEmSNHpMNkmSJEmSJKlvxjrZFBFHR8TdEbEnIp6MiHdXHRNARPx0RHym\njGl3RDwSEe+sOq5mEfGGiHg5Im6pOpZmEbE6Iv6xfE+/ExFvrTomgIhYGhH3R8QLEfFMRFwdEQsq\niuWiiHg4IvZFxE0tdadHxI6I2BsRD0bE8XWILSLeEhFfi4gfRsSzEXFHRLyu6rha9vloRGREnDGs\nuFRPdb22zEWdzxe9mu1aOwbtu6W83uyKiMci4oKmupFuW0O7zyKj3raImCrbNF0+/ntT3Ui3Tb2p\n07VllM6fdT5PdLpvqUN8M927DDu+uX4WicKmiHi+fGyKiBhWfDHL/UvV8bXsc9B9zDDiG+tkE3AN\n8CNgAjgP+IuIWFZtSAAsAP4J+A1gMXAp8MWIWFphTK2uAf6+6iCaRcTbgE3A+4AjgV8Hnqg0qJ+4\nFngWeB1wGsV7+4GKYvk+cDnFJJaviIhjKVZS+QhwNPAw8IU6xAa8lmKFhKXA8cBu4MYaxAVARJwI\n/DbwgyHGpPqq67VlLup8vuhVx2vtmLTvCuCEzDyKYlWsyyPiTWPStoYDPouMUdsuysxF5eMXYKza\nprmr07VllM6ftTxPdLpvqUt8dLh3qSi+uX4WWQucDawATgXOAi4cVnzMfv9SdXzAjPcxg48vM8fy\nASykOGGf1FR2M3BF1bF1iPdRYFXVcZSxrAa+CGwEbqk6nqa4vg78H1XH0SG2fwTObNr+JHB9xTFd\nDtzUtL0W+HrT9kLgJeDkqmNrU/9GYHdd4gK+CpwJ7ATOqPJ99VHtY9SuLV20q7bniz6381Fg1bi1\nD/gFig+R/+u4tK3dZ5FxaBswBVzQpnzk2+ajp9+L2l9b6nj+rPN5otN9S43ia3vvUmV83X4WKf+P\n1zbV/y7wd8OKr039AfcvdYmPDvcxw4hvnHs2nQTsz8zHmsq2ArX79jkiJiji3V6DWI4CPg5cXHUs\nzSLiMGAlcFxEPB4RT5fdPY+oOrbSnwO/ExE/ExFLgHdS/GHXyTKKvwEAMnMP8Dg1/Jug+Pan8r8H\ngIj4bWBfZt5fdSyqhZG5tvRolM4Xh6TlWjsW7YuIayNiL7CDItl0P2PQthk+i4x820qfiIjnIuJv\nImKyLBuXtmluan1tqeP5s87niVnuWyqPr9Tp3qUu8XEIsRxQT/V/M633L5XHN8t9zMDjG+dk0yJg\nV0vZLopujLUREa8GbgU+l5k7qo4HuAz4TGY+XXUgLSaAVwPnAm+l6O75SxTdeuvgIeAUit+xpym6\ned5TaUQHWwS82FJWx7+JU4GPAn9Ug1iOBP4j8MGqY1FtjMS1pQ9G4nxxqNpca8eifZn5AYqY30ox\n1GAf49G2Tp9FxqFtfwycACyhGH5xbznEYRzaprmr7bWlxufPOp8nZrpvqUN80PnepS7xcQixtNbv\nAhYNYl6k2XS4f6k0vkO4jxl4fOOcbJoGjmopW0wxlrIWIuJVwOcpus1eVHE4RMRpwBnA/1V1LG28\nVP776cz8QWY+B/wZRZfASpXv41cpPugvBI6lGMO7qcq42hiFv4mfB74CfDAz/7rqeCi6ZX8+M3dW\nHIfqo/Z/R30yNu3scK0dm/Zl5r9m5n8Dfhb4PUa8bbN8FhnptgFk5jcyc3dm7svMzwF/Q/FZZuTb\npp7U8v2v6/lzBM4TM923VB7fLPculcfXZLZYWusXA9NZjgkblhnuX6qObyMz38cMPL5xTjY9BiyI\niDc0la2gPkNzAvgMReZ7VWb+uOKQACYpJjh7KiKeAf4QWBUR/1BlUACZ+QJF1r35l3+oJ5IZHA28\nHri6/PD4PMXkcJUnwlpsp/gbACAiFgInUp+/ieOBB4DLMvPzVcdTOh34gyhW6XgG+LcUE2T+ccVx\nqTq1vrb0Ua3PF4dqhmvtWLSvxQJ+0oZRbtsknT+LjHrb2kkgGM+26dDV7tpS8/PnJDU+T8xy31J5\nfMx871KH+Bpmi+WAeir4m5nl/qXq+Ga7jxl4fGObbCrHdG4BPh4RCyPi1yhWa6nLTexfAL8InJWZ\nL82285BspvgDPq18XAfcB7yjyqCa3Aj8fkT8m4h4LfAh4MsVx0T5bcV3gfdHxIKIeA1wPsVEikNX\nxnA4cBhwWEQcHsVSpncDp0TEqrJ+A7B1mMM3O8VWjhX/S4qL3nXDime2uChO0qfwk7+J71Os0nDN\nsGNUPYzAtaUrdT5f9Emna+1It6+8Dq6OiEURcVhEvAP434D/yoi3jZk/i4x02yLiNRHxjqZr33kU\nc3x8lRFvm3pT02tLnc+fo3Ce6HTfUnl8s9y7DD2+Hj6L3AxcHBFLynuJdcBNw4rvEO5fKo2P2e9j\nBh/foGZDr8ODImt7D7AHeAp4d9UxlXEdT5Hdfpmi+1rjcV7VsbXEuZF6rUb3aoplOv8FeAb4FHB4\n1XGVsZ1GscLMC8BzFCtjTFT4vmXLY2NZdwbFRLIvlfEurUNsFBePbPl7mK46rjb77cTV6Ob9o67X\nljm2pbbniz60bcZr7Si3DzgO+KvyergL2Ab8n031I9u2Nm094LPIKLetfN/+nmIIyL8Afwe8bRza\n5qMvvx+1ubaM2vmzjucJZrhvqUl8He9dhh0fc/wsQtEr9D8BPywf/wmIYcXHLPcvVcfXZr+dHLga\n3cDji/KFJEmSJEmSpJ6N7TA6SZIkSZIkDZ/JJkmSJEmSJPWNySZJkiRJkiT1jckmSZIkSZIk9Y3J\nJkmSJEmSJPWNySZJkiRJkiT1jckmSZIkSZIk9Y3JJkmSJP3/7N1/sF31ed/798fIjrhIyOZHzs1V\nblBFjcmVhVx8YmfasXOmJvXEKTeMRBvZ2IHcmxGOh9Y3qCF0gEgOdm1NobcNmIDGPzC2YDBIhgJ2\nZkzKrtM4dYMDgipRNBgLAja2TAjoICDGfu4fe22y75aOdI7O0jl7i/drZs3Z+/us79rPGjPaPs9Z\n3+crSZLUGotNkiRJkiRJao3FJkmSJEmSJLXGYpMkSZIkSZJaY7FJkiRJkiRJrbHYJEmSJEmSpNZY\nbJIkSZIkSVJrLDZJkiRJkiSpNRabJEmSJEmS1BqLTZIkSZIkSWqNxSZJkiRJkiS1xmKTJEmSJEmS\nWmOxSZIkSZIkSa2x2CRJkiRJkqTWWGySJEmSJElSayw2SZIkSZIkqTUWmyRJkiRJktQai02SJEmS\nJElqjcUmSZIkSZIktcZikyRJkiRJklpjsUmSJEmSJEmtsdgkSZIkSZKk1lhskiRJkiRJUmssNkmS\nJEmSJKk1FpskSZIkSZLUGotNkiRJkiRJao3FJkmSJEmSJLXGYpMkSZIkSZJaY7FJmqEkX0jyVJLn\nkuxK8hvznZMkaTQleWOSF5N8Yb5zkSSNliSd5jtksjn+ar5zknosNkkz9wlgeVUdD/yfwEeTvHWe\nc5IkjaZPAn8230lIkkbWRVW1qDneNN/JSD0Wm6QZqqr/WVX7em+b49R5TEmSNIKSrAX+Fvij+c5F\nkiSpTRabpMOQ5Lok+4CdwHeBL89zSpKkEZLkeOD3gIvnOxdJ0kj7eJIfJPmTJBPznYzUY7FJOgxV\n9SFgMfAOYBvw0vxmJEkaMVcCn66qJ+Y7EUnSyPodYDmwFNgM3JXEFRcaChabpMNUVT+qqv8G/DTw\nm/OdjyRpNCR5C3AW8P/Ody6SpNFVVd+oqr1V9VJVfQ74E+A9852XBLBgvhOQjgILsGeTJGn6JoBl\nwONJABYBxyT5P6rqzHnMS5I02grIfCchgU82STOS5CeTrE2yKMkxSd4NvBebu0qSpm8z3T9SvKU5\nrgfuAd49n0lJkkZHktcneXeShUkWJDkPeCfwh/OdmwQ+2STNVNFdMnc93WLtY8D/U1X/eV6zkiSN\njGZH096upiSZBF6sqj3zl5UkacS8FvgocDrwI7obF51TVbvmNSupkaqa7xwkSZIkSZJ0lHAZnSRJ\nkiRJklpjsUmSJEmSJEmtsdgkSZIkSZKk1lhskiRJkiRJUmuOut3oTjrppFq2bNm0z3/++ec57rjj\njlxCLRiFHME82zYKeY5CjjD6eX7zm9/8QVWdPA8pvWrN9LukZ9j/Wxvm/IY5Nxju/IY5Nxju/IY5\nN2g3P79L5t7R+l0ylVHM25znzijmbc77m9F3SVUdVcdb3/rWmon77rtvRufPh1HIsco82zYKeY5C\njlWjnydwfw3Bv6+vpmOm3yU9w/7f2jDnN8y5VQ13fsOcW9Vw5zfMuVW1m5/fJX6XHGmjmLc5z51R\nzNuc9zeT7xKX0UmSJEmSJKk1FpskSZIkSZLUGotNkiRJkiRJao3FJkmSJEmSJLXGYpMkSZIkSZJa\nY7FJkiRJkiRJrbHYJEmSJEmSpNZYbJIkSZIkSVJrLDZJkiRJkiSpNQvmO4Fht+zSew44vvsTvzzH\nmUiSRpXfJZKk6Xr4yWe54ADfG35nSBolPtkkSZIkSZKk1lhskiRJkiRJUmssNkmSJEkaekl+Ismn\nkzyWZG+SB5P8Ul/8XUl2JtmX5L4kp/TFkmRTkqebY1OS9MWXNXP2Ndc4a+Cz39d87vNJ7khywtzc\ntSSNJotNkiRJkkbBAuCvgV8AlgCXA19sCkUnAduAK4ATgPuBW/vmrgPOAVYBZwBnAxf2xW8BHgBO\nBC4Dbk9yMkCSFcANwAeAMWAfcN2RuUVJOjpYbJIkSZI09Krq+araWFW7q+rHVXU38G3grcBqYEdV\n3VZVLwIbgVVJTm+mnw9cXVVPVNWTwFXABQBJTgPOBDZU1QtVtRV4CFjTzD0PuKuqvlZVk3QLWquT\nLJ6L+5akUeRudJIkSZJGTpIx4DRgB/CbwPZerKqeT/IIsALY2fzc3jd9ezNG8/PRqtp7kPjX+679\nrSQvNZ/9zYGc1tF9ioqxsTE6nc6M72vsWFi/8uX9xg/nWnNpcnJy6HMcZM5zZxTzNufZsdgkSZIk\nacBfKNQAACAASURBVKQkeS2wBfhcVe1MsgjYM3Dac0Dv6aNFwLMDsUVN36bBWC++dIq5g9d+RVVt\nBjYDjI+P18TExAzuquuaLXdy9cP7/5q2+7yZX2sudTodDud+55M5z51RzNucZ8dldJIkSZJGRpLX\nAJ8H/g64qBmeBI4fOHUJsHeK+BJgsqrqMOYOxiVJAyw2SZIkSRoJzZNIn6bbqHtNVf2wCe2g2/y7\nd95xwKnN+H7x5nV/bPlAD6bBeP+1TwVeB+xq4ZYk6ag0rWJTkouS3J/kpSQ3TnHO7yap/m1C3WJU\nkiRJUov+APhZ4OyqeqFv/EvAm5OsSbIQ2ABsr6qdTfwm4OIkS5MsBdYDNwJU1S7gQWBDkoVJVgMr\nga3N3C3A2Une0RSxrgS2DfR4kiT1me6TTd8BPgp85kDBprr/L4DvDoTcYlSSJEnSrCU5he7vEm8B\nnkoy2RznVdUeurvHfQx4BngbsLZv+g3AXcDDzXF3M9azFhhv5n4cOLe5JlW1A/gg3aLT94HjgA8d\nqfuUpKPBtBqEV9U2gCTjwE8f4JRPAr/D/sWgV7YYbeZfRbcAdX3fFqP/rPmrxNYkH6b7JXE9fVuM\nNnOvAP4yyWL/iiBJkiS9ulTVY0AOEr8XOH2KWAGXNMeB4ruBiYNc+2bg5ulnK0mvbrPejS7JvwBe\nqqov962Q6xn6LUYPtTXggbYdhbndenSYti88GPNs1yjkOQo5gnlKkiRJ0lyaVbGpaaL374BfnOKU\nod9i9FBbA15w6T0HHJ/LrUeHafvCgzHPdo1CnqOQI5inJEmSJM2l2e5GtxH4fPPY6YG4xagkSZIk\nSdKryGyLTe8C/nWSp5I8BfzvwBeT/E4Td4tRSZIkSZKkV5FpFZuSLGi2ED0GOKbZEnQB3WLTm+nu\nCPEWurvWXUi3YTi4xagkSZIkSdKrynR7Nl0ObOh7/37gI1W1sf+kJD8CnqmqyWboBmA53e1FAT7F\n/luM3kh3i9HHGdhiNElvi9ETgXuBX59mvpIkSZIkSZoH0yo2NUWljdM4b9nAe7cYlSRJkiRJehWZ\nbc8mSZJakWRZki8neabpBXhts2SbJO9KsjPJviT3JTmlb16SbErydHNsanY97b/ufc3cnUnOmo/7\nkyRJkl4tLDZJkobFdcAe4Kfo9gH8BeBDSU4CtgFXACcA9wO39s1bB5xDd1OJM4Cz6fYP7LkFeIDu\nkuzLgNuTnHxE70SSJEl6FbPYJEkaFv8AuLWqXqyqp4A/BFYAq4EdVXVbVb1Id1n3qiSnN/POB66u\nqieq6kngKuACgCSnAWcCG6rqharaCjwErJnD+5IkSZJeVabbIFySpCPtPwK/mqQDvAH4JbpPM00A\n23snVdXzSR6hW4ja2fzc3ned7c0Yzc9HB3Yy7Y+/Isk6uk9JMTY2RqfTmfENTE5OHnDe+pUvH/D8\nw/mM2Zgqv2EwzLnBcOc3zLnBcOc3zLnB8OcnSdJULDZJkobF1+gWe54DjgE+B9wB/HO6y+v6PQcs\nbl4vAp4diC1q+jYNxnrxpYMfXlWbgc0A4+PjNTExMeMb6HQ6HGjeBZfec8Dzd58388+YjanyGwbD\nnBsMd37DnBsMd37DnBsMf36SJE3FZXSSpHmX5DV0l81tA44DTqL7dNMmYBI4fmDKEqD3tNJgfAkw\n2eyIeqi5kiRJklpmsUmSNAxOAH4GuLaqXqqqp4HPAu8BdtBt/g1AkuOAU5txBuPN6/7Y8iSLp4hL\nkiRJapnFJknSvKuqHwDfBj6YZEGS19Nt/P0Q8CXgzUnWJFkIbAC2V9XOZvpNwMVJliZZCqwHbmyu\nuwt4ENiQZGGS1cBKYOsc3p4kSZL0qmKxSZI0LFbTbQq+B3gE+CHwW1W1h+7ucR8DngHeBqztm3cD\ncBfwcHPc3Yz1rAXGm7kfB85trilJkiTpCLDYJEkaClX1YFVNVNUbquqkqvqXVfW9JnZvVZ1eVcc2\n5+zum1dVdUlVndAclzT9mnrx3c2cY6vqTVV17zzcniSpBUkuSnJ/kpeS3Ng3fl6Syb5jX5JK8tYm\nvjHJDwfOWd43f1mS+5p5O5OcNfC570vyWJLnk9yR5IQ5u2lJGkEWmyRJkiSNiu8AHwU+0z9YVVuq\nalHvAD4EPAr8ed9pt/afU1WP9sVuAR4ATgQuA25PcjJAkhV0n5j9ADAG7AOuOzK3J0lHB4tNkiRJ\nkkZCVW2rqjuApw9x6vnATf1Puk4lyWnAmcCGqnqhqrbS7Rm4pjnlPOCuqvpaVU0CVwCrBzafkCT1\nWTDfCUiSJElSW5KcArwT+L8GQmcn+Rvgu3R3P/2DZnwF8GhV7e07d3sz3ot/vReoqm8leQk4Dfjm\nwGevA9YBjI2N0el0Zpz/2LGwfuXL+40fzrXm0uTk5NDnOMic584o5m3Os2OxSZIkSdLR5NeAP66q\nb/eNfRHYDHwPeDuwNcnfVtUtwCLg2YFrPAcsbV5PFd/vyaaq2tx8DuPj4zUxMTHj5K/ZcidXP7z/\nr2m7z5v5teZSp9PhcO53Ppnz3BnFvM15dlxGJ0mSJOlo8mvA5/oHquovquo7VfWjqvo68J+Ac5vw\nJHD8wDWWAHunGZckDbDYJEmSJOmokOSfAP8bcPshTi0gzesdwPKBHkyrmvFefFXfZ5wKvA7Y1UbO\nknQ0stgkSZIkaSQkWZBkIXAMcEyShUn615ydD2wd6L9Ekl9J8oZ0vQ34MHAnQFXtAh4ENjTXWw2s\nBLY207fQ7ff0jiTHAVcC2wY/Q5L09yw2SZIkSRoVlwMvAJcC729eXw7QFKH+JQNL6BprgUfoLn27\nCfhEVX1uID4OPAN8HDi3qvYAVNUO4IN0i07fB44DPtT2jUnS0cQG4ZIkSZJGQlVtBDZOEXsReP0U\nsfce4rq7gYmDxG8Gbp5elpKkaT3ZlOSiJPcneSnJjX3jP5/kq0n+JsmeJLcl+am+eJJsSvJ0c2xK\nkr74siT3JdmXZGeSswY+931JHkvyfJI7kpzQwj1LkiRJkiTpCJnuMrrvAB8FPjMw/ga6W3suA06h\n+1jqZ/vi64Bz6DbUOwM4G7iwL34L8ABwInAZcHuSkwGSrABuAD4AjAH7gOumma8kSZIkSZLmwbSK\nTVW1raruAJ4eGP9KVd1WVc9V1T7gWuCf9J1yPnB1VT1RVU8CVwEXACQ5DTgT2FBVL1TVVuAhYE0z\n9zzgrqr6WlVNAlcAqwd2iZAkSZIkSdIQabtn0zv5+y1CAVYA2/veb2/GerFHB3ZxGIx/vReoqm8l\neQk4Dfhm/4cmWUf3KSrGxsbodDrTTnhycvKg569f+fIBx2fyGbN1qByHhXm2axTyHIUcwTwlSZIk\naS61VmxKcgbwu8Cv9A0vAp7te/8csKjp2zQY68WXTjG3F9/vyaaq2kx3OR/j4+M1MTEx7bw7nQ4H\nO/+CS+854Pju86b/GbN1qByHhXm2axTyHIUcwTwlSZIkaS5Nt2fTQSX5h8BXgA9X1R/3hSaB4/ve\nLwEmq6oOEOvF904xdzAuSZIkSZKkITPrYlOSU4B7gSur6vMD4R10m4P3rOLvl9ntAJYP9GAajL8y\nN8mpwOuAXbPNWZIkSZIkSUfGtIpNSRYkWQgcAxyTZGEzthT4L8C1VXX9AabeBFycZGlz7nrgRoCq\n2gU8CGxorrcaWAlsbeZuAc5O8o4kxwFXAtsGejxJkiRJkiRpiEy3Z9PlwIa+9+8HPgIUsBzYmGRj\nL1hVi5qXNzTxh5v3n2rGetbSLT49AzwOnFtVe5pr7EjyQbpFpxPpPj3169PMV5IkSZIkSfNgWsWm\nqtoIbJwi/JGDzCvgkuY4UHw3MHGQ+TcDN08nR0mSJEmSJM2/VhqES5IkSZIkSWCxSZIkSZIkSS2y\n2CRJkiRJkqTWWGySJEmSJElSayw2SZIkSZIkqTUWmyRJkiRJktQai02SJEmSRkKSi5Lcn+SlJDf2\njS9LUkkm+44r+uJJsinJ082xKUkG5t+XZF+SnUnOGvjc9yV5LMnzSe5IcsKc3LAkjSiLTZIkSZJG\nxXeAjwKfmSL++qpa1BxX9o2vA84BVgFnAGcDF/bFbwEeAE4ELgNuT3IyQJIVwA3AB4AxYB9wXWt3\nJElHIYtNkiRJkkZCVW2rqjuAp2c49Xzg6qp6oqqeBK4CLgBIchpwJrChql6oqq3AQ8CaZu55wF1V\n9bWqmgSuAFYnWTz7O5Kko5PFJkmSJElHi8eSPJHks0lO6htfAWzve7+9GevFHq2qvQeJvzK3qr4F\nvASc1nbyknS0WDDfCUiSJEnSLP0A+DngQbpL4T4JbAHe3cQXAc/2nf8csKjp2zQY68WXTjG3F9/v\nyaYk6+gu2WNsbIxOpzPjGxk7FtavfHm/8cO51lyanJwc+hwHmfPcGcW8zXl2LDZJkiRJGmnN8rb7\nm7ffS3IR8N0ki5snliaB4/umLAEmq6qSDMZ68d6TToeK9+exGdgMMD4+XhMTEzO+l2u23MnVD+//\na9ru82Z+rbnU6XQ4nPudT+Y8d0Yxb3OeHZfRSZIkSTraVPOz9/vODrrNwXtWNWO92PKBHkyD8Vfm\nJjkVeB2wq+WcJemoYbFJkiRJ0khIsiDJQuAY4JgkC5uxtyd5U5LXJDkR+H2gU1W95W83ARcnWZpk\nKbAeuBGgqnbRXX63obneamAlsLWZuwU4O8k7khwHXAlsG+jxJEnqY7FJkiRJ0qi4HHgBuBR4f/P6\ncmA58Id0l7b9T7oNvN/bN+8G4C7g4ea4uxnrWQuMA88AHwfOrao9AFW1A/gg3aLT94HjgA8dkbuT\npKOEPZskSZIkjYSq2ghsnCJ8y0HmFXBJcxwovhuYOMj8m4Gbp5elJMknmyRJkiRJktQai02SJEmS\nJElqjcUmSZIkSZIktWZaxaYkFyW5P8lLSW4ciL0ryc4k+5Lcl+SUvliSbErydHNsSpK++LJmzr7m\nGmcNXPt9SR5L8nySO5KcMMv7lSQNsSRrk/xl8+/+t5K8oxk/Yt81kiRJkto13SebvgN8FPhM/2CS\nk4BtwBXACcD9wK19p6wDzgFWAWcAZwMX9sVvAR4ATgQuA25PcnJz7RV0d4j4ADAG7AOum/6tSZJG\nSZJfBDYBvw4sBt4JPHokv2skSZIktW9axaaq2lZVdwBPD4RWAzuq6raqepHuzhCrkpzexM8Hrq6q\nJ6rqSeAq4AKAJKcBZwIbquqFqtoKPASsaeaeB9xVVV+rqkm6v2SsTrL4MO9VkjTcPgL8XlX996r6\ncVU92Xx3HMnvGkmSJEktWzDL+SuA7b03VfV8kkea8Z2D8eb1ir65j1bV3oPEv9537W8leQk4Dfhm\nfxJJ1tH9yzZjY2N0Op1p38Dk5ORBz1+/8uUDjs/kM2brUDkOC/Ns1yjkOQo5gnmOgiTHAOPAf26+\nRxYCdwC/zZH9runP4bC/S3qm+t9wGL5LYLj/Gxvm3GC48xvm3GC48xvm3GD485MkaSqzLTYtAvYM\njD1Hd/lDL/7sQGxR00tjMNaLL51i7uC1X1FVm4HNAOPj4zUxMTHtG+h0Ohzs/AsuveeA47vPm/5n\nzNahchwW5tmuUchzFHIE8xwRY8BrgXOBdwA/BO4ELufIfte8YjbfJT1T/W84DN8lMNz/jQ1zbjDc\n+Q1zbjDc+Q1zbjD8+UmSNJXZ7kY3CRw/MLYE2DtFfAkwWVV1GHMH45Kko8cLzc9rquq7VfUD4D8A\n7+HIftdIkiRJatlsi0076DZkBSDJccCpzfh+8eZ1f2z5QA+mwXj/tU8FXgfsmmXOkqQhU1XPAE8A\n1T/c/DyS3zWSJEmSWjatYlOSBUkWAscAxyRZmGQB8CXgzUnWNPENwPaq2tlMvQm4OMnSJEuB9cCN\nAFW1C3gQ2NBcbzWwEtjazN0CnJ3kHc0vFlcC2wb6bkiSjh6fBf5Vkp9M8gbgt4C7ObLfNZIkSZJa\nNt0nmy6nu8ThUuD9zevLq2oP3R19PgY8A7wNWNs37wbgLuDh5ri7GetZS7ch7DPAx4Fzm2tSVTuA\nD9ItOn0fOA740IzvUJI0Kq4E/ozuE6x/CTwAfOxIftdIkiRJat+0GoRX1Ua6W00fKHYvcPoUsQIu\naY4DxXcDEwf53JuBm6eToyRptFXVD+n+UWG/Pywcye8aSZIkSe2abc8mSZIkSZIk6RUWmyRJkiRJ\nktQai02SJEmSJElqjcUmSZIkSSMhyUVJ7k/yUpIb+8Z/PslXk/xNkj1JbkvyU33xjUl+mGSy71je\nF1+W5L4k+5LsTHLWwOe+L8ljSZ5PckeSE+bkhiVpRFlskiRJkjQqvgN8FPjMwPgbgM3AMuAUYC/w\n2YFzbq2qRX3Ho32xW+jugnoicBlwe5KTAZKsoLvL6QeAMWAfcF2bNyVJR5tp7UYnSZIkSfOtqrYB\nJBkHfrpv/Cv95yW5Fviv07lmktOAM4F/VlUvAFuTfBhYA1wPnAfcVVVfa86/AvjLJIurau/s70qS\njj4+2SRJkiTpaPNOYMfA2NnNMrsdSX6zb3wF8OhA4Wh7M96Lb+8FqupbwEvAae2nLUlHB59skiRJ\nknTUSHIG8LvAr/QNf5HuMrvvAW+n+/TS31bVLcAi4NmByzwHLG1eTxVffIDPXgesAxgbG6PT6cw4\n/7FjYf3Kl/cbP5xrzaXJycmhz3GQOc+dUczbnGfHYpMkSZKko0KSfwh8BfhwVf1xb7yq/qLvtK8n\n+U/AuXR7NU0Cxw9cagndvk9MI/6KqtpMt6jF+Ph4TUxMzPgertlyJ1c/vP+vabvPm/m15lKn0+Fw\n7nc+mfPcGcW8zXl2XEYnSZIkaeQlOQW4F7iyqj5/iNMLSPN6B7A8Sf+TSqv4+2V4O5r3vc85FXgd\nsKuNvCXpaGSxSZIkSdJISLIgyULgGOCYJAubsaXAfwGurarrDzDvV5K8IV1vAz4M3AlQVbuAB4EN\nzfVWAyuBrc30LXT7Pb0jyXHAlcA2m4NL0tRcRidJkiRpVFwObOh7/37gI3SfVFoObEyysResqkXN\ny7XAZ4CfAJ4APlFVn+u7zlrgRuAZ4HHg3Kra01xjR5IP0i06nUj36alfb/vGJOloYrFJkiRJ0kio\nqo3AxinCHznIvPce4rq7gYmDxG8Gbj5UfpKkLpfRSZIkSZIkqTUWmyRJkiRJktQai02SJEmSJElq\njcUmSZIkSZIktcZikyRJkiRJklrTSrEpybIkX07yTJKnklybZEETe1eSnUn2JbkvySl985JkU5Kn\nm2NTkgxc975m7s4kZ7WRryRJkiRJko6Mtp5sug7YA/wU8BbgF4APJTkJ2AZcAZwA3A/c2jdvHXAO\nsAo4AzgbuLAvfgvwAHAicBlwe5KTW8pZkiRJkiRJLWur2PQPgFur6sWqegr4Q2AFsBrYUVW3VdWL\nwEZgVZLTm3nnA1dX1RNV9SRwFXABQJLTgDOBDVX1QlVtBR4C1rSUsyRJkiRJklrWVrHpPwK/muR/\nSbIU+CX+vuC0vXdSVT0PPNKMMxhvXvfHHq2qvVPEJUmSJEmSNGQWtHSdr9FdEvcccAzwOeAO4J/T\nXV7X7zlgcfN6EfDsQGxR07dpMNaLLx388CTrms9nbGyMTqcz7cQnJycPev76lS8fcHwmnzFbh8px\nWJhnu0Yhz1HIEcxTkiRJkubSrItNSV5D9ymmzcA/plsk+gywCZgEjh+YsgToPa00GF8CTFZVJTnU\n3FdU1ebm8xkfH6+JiYlp59/pdDjY+Rdces8Bx3efN/3PmK1D5TgszLNdo5DnKOQI5ilJkiRJc6mN\nZXQnAD8DXFtVL1XV08BngfcAO+g2/wYgyXHAqc04g/HmdX9seZLFU8QlSZIkSZI0ZGZdbKqqHwDf\nBj6YZEGS19Nt/P0Q8CXgzUnWJFkIbAC2V9XOZvpNwMVJlja9ntYDNzbX3QU8CGxIsjDJamAlsHW2\nOUuSJEmSJOnIaKtB+Gq6TcH30G0A/kPgt6pqD93d4z4GPAO8DVjbN+8G4C7g4ea4uxnrWQuMN3M/\nDpzbXFOSJEmSJElDqJUG4VX1IDAxRexe4PQpYgVc0hwHiu+e6rqSJEmSJEkaPm092SRJkiRJkiRZ\nbJIkSZIkSVJ7LDZJkiRJGglJLkpyf5KXktw4EHtXkp1J9iW5L8kpfbEk2ZTk6ebYlCR98WXNnH3N\nNc4auPb7kjyW5PkkdyQ54YjfrCSNMItNkiRJkkbFd4CPAp/pH0xyErANuAI4AbgfuLXvlHXAOcAq\n4AzgbODCvvgtwAPAicBlwO1JTm6uvYLuJkYfAMaAfcB1Ld+XJB1VLDZJkiRJGglVta2q7gCeHgit\nBnZU1W1V9SKwEViVpLdR0fnA1VX1RFU9CVwFXACQ5DTgTGBDVb1QVVuBh+juqg1wHnBXVX2tqibp\nFrRWJ1l8xG5UkkZcK7vRSZKkmVt26T37je3+xC/PQyaSNPJWANt7b6rq+SSPNOM7B+PN6xV9cx+t\nqr0HiX+979rfSvIScBrwzf4kkqyj+xQVY2NjdDqdGd/I2LGwfuXL+40fzrXm0uTk5NDnOMic584o\n5m3Os2OxSZIkSdKoWwTsGRh7DljcF392ILao6ds0GOvFl04xd/Dar6iqzcBmgPHx8ZqYmJjRTQBc\ns+VOrn54/1/Tdp8382vNpU6nw+Hc73wy57kzinmb8+y4jE6SJEnSqJsEjh8YWwLsnSK+BJisqjqM\nuYNxSdIAn2ySJKklDz/5LBccYGmcJOmI20G3LxMASY4DTm3Ge/FVwP9o3q8aiC1PsrhvKd0qYMvA\n3N61TwVeB+xq/zYk6ejgk02SJEmSRkKSBUkWAscAxyRZmGQB8CXgzUnWNPENwPaq2tlMvQm4OMnS\nJEuB9cCNAFW1C3gQ2NBcbzWwEtjazN0CnJ3kHU0R60pg20CPJ0lSH4tNkqShkeSNSV5M8oW+sXcl\n2ZlkX5L7kpzSF0uSTUmebo5NTf+NXnxZM2dfc42z5vqeJEmtuhx4AbgUeH/z+vKq2kN397iPAc8A\nbwPW9s27AbgLeLg57m7GetYC483cjwPnNtekqnYAH6RbdPo+cBzwoSNze5J0dHAZnSRpmHwS+LPe\nmyQnAduA36D7S8KVwK3AzzenrAPOobu8oYCvAt8Grm/itwB/CrynOW5P8sbeLxCSpNFSVRuBjVPE\n7gVOnyJWwCXNcaD4bmDiIJ97M3DzTHKVpFczn2ySJA2FJGuBvwX+qG94NbCjqm6rqhfp/oKxKknv\nl4nzgaur6omqehK4Crigud5pwJnAhqp6oaq2Ag/R/cu3JEmSpCPEJ5skSfMuyfHA7wH/lO5TTD0r\ngO29N1X1fJJHmvGdg/Hm9Yq+uY8O9NTojw/msI7uk1KMjY3R6XRmfB9jx8L6lS/PeF6/w/nc6Zqc\nnDyi15+NYc4Nhju/Yc4Nhju/Yc4Nhj8/SZKmYrFJkjQMrgQ+XVVP9LVcAlgEDC55ew5Y3Bd/diC2\nqOnbNBjrxZceKIGq2gxsBhgfH6+JiYkZ38Q1W+7k6odn99W6+7yZf+50dTodDue+5sIw5wbDnd8w\n5wbDnd8w5wbDn58kSVOx2CRJmldJ3gKcBfyjA4QngeMHxpYAe6eILwEmq6qSHGquJEmSpCPAnk2S\npPk2ASwDHk/yFPBvgDVJ/hzYQbf5NwDNltOnNuMMxpvX/bHlSRZPEZckSZJ0BFhskiTNt810C0hv\naY7rgXuAdwNfAt6cZE2ShcAGYHtV7Wzm3gRcnGRpkqXAeuBGgKraBTwIbEiyMMlqYCWwdc7uTJIk\nSXoVchmdJGleVdU+YF/vfbP87cWq2tO8XwNcC3wB+Aawtm/6DcBy4OHm/aeasZ61dItPzwCPA+f2\nritJkiTpyGjtyaYka5P8ZZLnk3wryTua8Xcl2ZlkX5L7kpzSNydJNiV5ujk2pa8zbJJlzZx9zTXO\naitfSdJwqqqNVfX+vvf3VtXpVXVsVU1U1e6+WFXVJVV1QnNcUlXVF9/dzDm2qt5UVffO8e1IkiRJ\nrzqtFJuS/CKwCfh1ujsEvRN4NMlJwDbgCuAE4H7g1r6p64Bz6PbQOAM4G7iwL34L8ABwInAZcHuS\nk9vIWZIkSZIkSe1r68mmjwC/V1X/vap+XFVPVtWTwGpgR1XdVlUvAhuBVUlOb+adD1xdVU80518F\nXACQ5DTgTGBDVb1QVVuBh4A1LeUsSZIkSZKkls262JTkGGAcODnJI0meSHJtkmOBFcD23rlV9Tzw\nSDPOYLx53R97tKr2ThGXJEmSJEnSkGmjQfgY8FrgXOAdwA+BO4HLgUXAYCPW5+gutaOJPzsQW9T0\nbRqM9eJLBxNIso7ukjzGxsbodDrTTn5ycvKg569f+fIBx2fyGbN1qByHhXm2axTyHIUcwTwlSZIk\naS61UWx6ofl5TVV9FyDJf6BbbPoacPzA+UuA3tNKkwPxJcBkVVWzG9HB5r6iqjbT3Tqb8fHxmpiY\nmHbynU6Hg51/waX3HHB893nT/4zZOlSOw8I82zUKeY5CjmCekiRJkjSXZr2MrqqeAZ4Aqn+4+bmD\nbvNvAJIcB5zajO8Xb173x5YnWTxFXJIkSZIkSUOmrQbhnwX+VZKfTPIG4LeAu4EvAW9OsibJQmAD\nsL2qdjbzbgIuTrI0yVJgPXAjQFXtAh4ENiRZmGQ1sBLY2lLOkiRJkiRJalkby+gArgROAnYBLwJf\nBD5WVS8mWQNcC3wB+Aawtm/eDcBy4OHm/aeasZ61dItPzwCPA+dW1WAPKEmSJEmSJA2JVp5sqqof\nVtWHqur1VfW/VtW/rqoXm9i9VXV6VR1bVRNVtbtvXlXVJVV1QnNcUlXVF9/dzDm2qt5UVfe2ka8k\nSZKko0uSyYHjR0muaWLLktRA/Iq+uUmyKcnTzbGp2bSIvvn3JdmXZGeSs+bjHiVpVLT1ZJMkSZIk\nzZuqWtR7nWQR8BRw28Bpr6+qA203vQ44h26P2AK+CnwbuL6J3wL8KfCe5rg9yRtddSFJB9ZWkCBd\nsAAAIABJREFUzyZJkiRJGhZrgO8DfzzN888Hrq6qJ6rqSeAq4AKAJKcBZwIbquqFqtoKPNR8hiTp\nAHyySZIkSdLR5nzgpv4WHY3HkvSeXPrtqvpBM74C2N533vZmrBd7tKr2ThF/RZJ1dJ+SYmxsjE6n\nM+PEx46F9Sv3f/jqcK41lyYnJ4c+x0HmPHdGMW9znh2LTY1ll94z3ylIkiRJmqUkpwC/APzffcM/\nAH6O7m7XJwKfBLYA727ii4Bn+85/DljU9G0ajPXiSwc/u6o2A5sBxsfHa2JiYsb5X7PlTq5+eP9f\n03afN/NrzaVOp8Ph3O98Mue5M4p5m/PsWGySJEmSdDT5APDfqurbvYGqmgTub95+L8lFwHeTLG6e\nWJoEju+7xhJgsqoqyWCsF9+LJOmA7NkkSZIk6Wjya8DnDnFOb3ld7/ehHXSbg/esasZ6seVJFk8R\nlyQNsNgkSZIk6aiQ5B/TXd5228D425O8KclrkpwI/D7Qqare8ribgIuTLE2yFFgP3AhQVbvoLr/b\nkGRhktXASmDrnNyUJI0gl9FJkiRJOlqcD2wbaOYNsBz4d8BP0u239FXgvX3xG5pzHm7ef6oZ61lL\nt/j0DPA4cG5V7Wk7eUk6WlhskiRJknRUqKoLpxi/BbjlIPMKuKQ5DhTfDUzMPkNJenVwGZ0kSZIk\nSZJaY7FJkiRJkiRJrbHYJEmSJEmSpNZYbJIkSZIkSVJrLDZJkiRJkiSpNRabJEmSJEmS1BqLTZIk\nSZIkSWqNxSZJkiRJkiS1xmKTJEmSJEmSWmOxSZIkSZIkSa1prdiU5I1JXkzyhb6xdyXZmWRfkvuS\nnNIXS5JNSZ5ujk1J0hdf1szZ11zjrLZylSRJkiRJ0pHR5pNNnwT+rPcmyUnANuAK4ATgfuDWvvPX\nAecAq4AzgLOBC/vitwAPACcClwG3Jzm5xXwlSZIkSZLUslaKTUnWAn8L/FHf8GpgR1XdVlUvAhuB\nVUlOb+LnA1dX1RNV9SRwFXBBc73TgDOBDVX1QlVtBR4C1rSRryRJkiRJko6MBbO9QJLjgd8D/inw\nG32hFcD23puqej7JI834zsF483pF39xHq2rvFPHBHNbRfVKKsbExOp3OtPOfnJyk0+mwfuXL054D\nzOgzZquX47Azz3aNQp6jkCOYpyRJkiTNpVkXm4ArgU9X1RN9LZcAFgF7Bs59DljcF392ILao6ds0\nGOvFlx4ogaraDGwGGB8fr4mJiWkn3+l0mJiY4IJL75n2HIDd503/M2arl+OwM892jUKeo5AjmKck\nSZIkzaVZFZuSvAU4C/hHBwhPAscPjC0B9k4RXwJMVlUlOdRcSZIkSZIkDaHZ9myaAJYBjyd5Cvg3\nwJokfw7soNv8G4AkxwGnNuMMxpvX/bHlSRZPEZckSZKk/58knWaH7Mnm+Ku+mDtlS9IcmW2xaTPd\nAtJbmuN64B7g3cCXgDcnWZNkIbAB2F5VO5u5NwEXJ1maZCmwHrgRoKp2AQ8CG5IsTLIaWAlsnWW+\nkiRJko5uF1XVouZ4E7hTtiTNtVkVm6pqX1U91TvoLo17sar2VNUeurvHfQx4BngbsLZv+g3AXcDD\nzXF3M9azFhhv5n4cOLe5piRJkiTNhDtlS9IcaqNB+CuqauPA+3uB06c4t4BLmuNA8d10l+lJkiRJ\n0nR9PMkngL8CLquqDnO0U/ZsdsnuGTuWA+6UPew71o7irrrmPHdGMW9znp1Wi02SJB2OJD8BXEd3\n04kTgG8B/7aqvtLE3wV8EvgZ4BvABVX1WBML8AngN5rLfQq4tPmjBkmWAZ8F3g48Tnd5xb1zcmOS\npLn2O8BfAH9Hd6XEXc2mRnOyU/ZsdsnuuWbLnVz98P6/ps3lbtiHYxR31TXnuTOKeZvz7My2Z5Mk\nSW1YAPw18At0dx+9HPhi05DVPhuSpGmpqm9U1d6qeqmqPgf8CfAeZrFT9jTmSpIGWGySJM27qnq+\nqjZW1e6q+nFV3Q18G3gr9tmQJB2+AoI7ZUvSnHIZnSRp6CQZA06j+3/kf5MR77MxE0dynf0wreMf\nNMy5wXDnN8y5wXDnN8y5wfDnN2ySvJ7ukun/CrwM/CrwTuDDdDcd+vdJ1tDdPXuqnbK/3LxfD/w+\ndHfKTtLbKftyuk9KuVO2JB2ExSZJ0lBJ8lpgC/C5qtqZZOT7bMzEkezJMUzr+AcNc24w3PkNc24w\n3PkNc24w/PkNodcCH6W7QdGP6P5B4pyq2gXQFJquBb5At//f4E7Zy+nukg3d/n+DO2XfSLdo9Tju\nlC1JB2WxSZI0NJK8Bvg83cauFzXDh91nI4l9NiTpVaIp/vzcQeLulC1Jc8SeTZKkodA8ifRpYAxY\nU1U/bEL22ZAkSZJGiMUmSdKw+APgZ4Gzq+qFvvEvAW9OsibJQqbus7E0yVK6fTZuhG6fDaDXZ2Nh\nktXYZ0OSJEk6olxGJ0mad0lOAS4EXgKe6j7kBMCFVbXFPhuSJEnS6LDYJEmad1X1GN2tqaeK22dD\nkiRJGhEuo5MkSZIkSVJrLDZJkiRJkiSpNRabJEmSJEmS1Bp7NkmSNESWXXrPAcd3f+KX5zgTSZIk\n6fD4ZJMkSZIkSZJaY7FJkiRJkiRJrbHYJEmSJEmSpNZYbJIkSZIkSVJrLDZJkiRJkiSpNbMuNiX5\niSSfTvJYkr1JHkzyS33xdyXZmWRfkvuSnNIXS5JNSZ5ujk1J0hdf1szZ11zjrNnmK0mSJEmSpCOn\njSebFgB/DfwCsAS4HPhiUyg6CdgGXAGcANwP3No3dx1wDrAKOAM4G7iwL34L8ABwInAZcHuSk1vI\nWZIkSZIkSUfArItNVfV8VW2sqt1V9eOquhv4NvBWYDWwo6puq6oXgY3AqiSnN9PPB66uqieq6kng\nKuACgCSnAWcCG6rqharaCjwErJltzpIkSZIkSToyFrR9wSRjwGnADuA3ge29WFU9n+QRYAWws/m5\nvW/69maM5uejVbV3inj/Z66j+5QUY2NjdDqdaec7OTlJp9Nh/cqXpz0HmNFnzFYvx2Fnnu0ahTxH\nIUcwT0mSXg2S/ARwHXAW3VUV3wL+bVV9Jckyun8Qf75vyqaqurKZG+ATwG80sU8Bl1ZVNfFlwGeB\ntwOPAxdV1b1H9o4kaXS1WmxK8lpgC/C5qtqZZBGwZ+C054DFzetFwLMDsUXNP/aDsV586eDnVtVm\nYDPA+Ph4TUxMTDvnTqfDxMQEF1x6z7TnAOw+b/qfMVu9HIedebZrFPIchRzBPCVJepXob+/xOPAe\nuu09Vvad8/qqOtBfmfvbexTwVbrFqeub+C3AnzbXfA/d9h5vrKrB33UkSbS4G12S1wCfB/4OuKgZ\nngSOHzh1CbB3ivgSYLL5C8Kh5kqSJEkScMj2Hodiew9JalErTzY1TyJ9GhgD3lNVP2xCO+j+w907\n7zjg1Ga8F18F/I/m/aqB2PIki/uW0q2i++SUJEmSJE1poL1Hz2NJek8u/XZV/aAZn/f2Hj1jx3LA\nFh/DvtR+FNsBmPPcGcW8zXl22lpG9wfAzwJnVdULfeNfAv59kjXAPcAGYHtV7WziNwEXJ/ly8349\n8PsAVbUryYPAhiSX031cdSWwtaWcJUmSJB2Fpmjv8XPAg3R3uv5kE393M2Xe23v0XLPlTq5+eP9f\n0+ayjcfhGMV2AOY8d0Yxb3OenVkXm5KcAlwIvAQ81f33GIALq2pLU2i6FvgC8A1gbd/0G4DlwMPN\n+081Yz1rgRuBZ+iuuz7XddGSJEmSpnKg9h5VNQnc35zyvSQXAd/tW0UxZXuPJLb3kKQZmnWxqaoe\nA3KQ+L3A6VPECrikOQ4U3w1MzDZHSZIkSUe/g7T3GFTNz14PW9t7SFKLWmsQLkmSJEnzrNfe4+z+\n9h5J3p7kTUlek+REuq07OlXVWx7Xa++xNMlSuu09boRuew+6y+82JFmYZDW295Ckg2qrZ5MkSZIk\nzZuDtfcAfgz8O+An6fZb+irw3r7ptveQpBZZbJIkSZI08g7V3gO45SBzbe8hSS1yGZ0kSZIkSZJa\nY7FJkiRJkiRJrXEZnSRJI2DZpfcccHz3J355jjORJEmSDs4nmyRJkiRJktQai02SJEmSJElqjcUm\nSZIkSZIktcZikyRJkiRJklpjsUmSJEmSJEmtsdgkSZIkSZKk1lhskiRJkiRJUmssNkmSJEmSJKk1\nC+Y7AUmSdPiW/X/s3X2UJHd93/v3By0goZWEhWCTrK+1kYIsX7EI48WQxDaTg2xscfFRtDgRlhPp\n5BIR+yjxjTZg5RqyMmAjOSjJscEGXQjiUeZBAgeEnUAuY5vYxpZvtKzXXhQMKyzxEEmWtTt6wou/\n94+qjlutmdmZ7eru6pn365w6M12/qu7vt6q7quc79fvV1bc+bt6ha18yg0gkSZKkhlc2SZIkSZIk\nqTMWmyRJkiRJktQZi02SJEmSJEnqjGM2SZK0wSw3jhPAjT948pQjkSRJ0mbU6yubkpye5CNJHkxy\nZ5IfnXVMkqT54rlEkjQuzyWStD59v7LpLcA3gG3Ac4Bbk+yrqgOzDcu7/0jSHOntuWTa9t/9AJd7\n/pKk4+G5RJLWobfFpiQnA7uBZ1XVEvCZJL8K/CPg6pkGJ0maC55L1malbndrZbFK0kbmuUSS1q+3\nxSbgHOBoVd0xNG8fsDC6YJIrgCvah0tJPr+O1zkDuPd4g3xMHNd18SzL6izGCTPObs1DnPMQI8x/\nnGdOO5ANZFrnkoFev9f+xYTi6+j81+ttR7/j63Ns0O/4+hwbdBuf55Lj14tzyQT/1uhK3z9PyzHm\n6ZnHuI358dZ8LulzsWkrcHhk3mHglNEFq+oG4IbjeZEkt1XVruNZd1rmIUYwzq7NQ5zzECMY5yY3\nlXPJQN/3YZ/j63Ns0O/4+hwb9Du+PscG/Y9vE/FcsgbzGLcxT888xm3M4+nzAOFLwKkj804Djswg\nFknSfPJcIkkal+cSSVqnPheb7gC2JHnm0LzzAQfhkyStlecSSdK4PJdI0jr1tthUVQ8CtwCvS3Jy\nku8Bfhh4T8cvNdZlrlMyDzGCcXZtHuKchxjBODetKZ5LBvq+D/scX59jg37H1+fYoN/x9Tk26H98\nm4LnkjWbx7iNeXrmMW5jHkOqatYxrCjJ6cB/BL4fuA+4uqreP9uoJEnzxHOJJGlcnkskaX16XWyS\nJEmSJEnSfOltNzpJkiRJkiTNH4tNkiRJkiRJ6symLTYlOT3JR5I8mOTOJD8665hGJVlM8kiSpXb6\n/KxjAkhyZZLbkjya5MaRthclOZjkoSSfTnLmjMJcMc4kO5LU0HZdSvLaGcX45CTvaN+DR5LcnuSH\nhtp7sT1Xi7Nn2/O9Sb6W5HCSO5K8YqitF9tytTj7tC21PrM4p6x2jljt/Z7GdUnua6frkmSofUe7\nzkPtc1ywhliO67wwbixJfrTd3g8m+WiaMVXWFNuxPm9Tiu24zwGzjq8n2++4jvnTiG21+Pqw7dRP\n6eHfJ6sdB9r24/6sTSn+Z6Y5V753jmK+JMkft++DP0nyvX2Ouz0mfSLJ/e0x781JtvQp5szoe8ok\nYk7ygiSfTPJnSe5J8qEkf70PMT9OVW3KCbgJ+ACwFfge4AHgvFnHNRLjIvCKWcexTFwXAxcBvwzc\nODT/jHY7/ghwIvBvgd/tYZw7gAK29GBbngxc08b0BOD/AI60j3uzPY8RZ5+257OAp7S/nwt8Dfiu\nPm3LY8TZm23ptO59OvVzykrniGO934FXAp8HvhXYDvwR8M+G2n8H+HfAScBu4M+Bpx8jluM6L4wT\nC3Beexz6vna7vx/4lXXEturnbUqxHfc5oAfx9WH7HdcxfxqxHSO+mW87p35O9PDvk2McB8b6rE0p\n/v8C/Bbw3vZxr2OmGYD+TuAF7fbe3k69jRv4BPCuNq6/BuwH/kWfYmYG31MmGPMPtfGeCjyF5sYF\nv96HmB+Xw7Q+OH2aaA6a3wDOGZr3buDaWcc2EuciPSw2DcX3hpE3/hXAb49s54eBc3sW5w56/Ac9\n8Ln2g9/L7blMnL3cnsC3A18F/kGft+VInL3clk7H3IczOaesdI441vsd+G3giqH2fzL4YgWcAzwK\nnDLU/pus8cvfes8L48QC/Bzw/qG2s9v9cMoaY1v18zbN2EZed03ngB7E16vtxzqO+bPYdqzjWD+r\nfes024k5+fukjWvs49SU4rwE+CBNsWxQbOp7zL8N/J/LzO9t3MAfAxcOPf63wNv6GDNT/J4yqZiX\naX8ucGTkPTTTmAfTZu1Gdw5wtKruGJq3j+a/QX3zxiT3JvlvSRZmHcwxnEezHQGoqgeBL9DP7Qpw\nZ5K7krwzyRmzDgYgyTaa9+cBerw9R+Ic6MX2TPJLSR4CDtJ8sf8EPdyWK8Q50IttqTWb5TlluXPE\nsd7vj2kfifU84ItVdWSF9vWaZCyjz/0nNF+gzllnjCt93qYe2zrPAbOOb2Cm2+84j/lT23bHeazv\nw+dC0zcXf590eJyadJynAq8Drhpp6nPMJwC7gKcn+UJ7fHhzkpP6HDfwH4B/mOQpSbbTXHnz66Mx\n9SzmgT59Zzpe38djz8u9iXmzFpu2AodH5h0GTplBLKv5KeAsmsvfbgA+luTs2Ya0qq00lyEO6+N2\nvRd4HnAmzeXspwDvm2lEQJIntnG8q6oO0tPtuUycvdqeVfUTbQzfC9xC8yW7d9tyhTh7tS21ZrM6\np6x0jjjW+320/TCwte3P3/VnZZKxjBvrsT5vU43tOM4Bs46vF9vvOI/5U9t2x3msn+XnQrPT+79P\nOj5OTdrrgXdU1V0j8/sc8zbgicDLaI4ZzwG+E3jNMnENYutD3L9J0234MHAXcBvw0WViGsTVh5gH\n+vSdad2SPBv4N8Crhmb3JubNWmxaounjOOw0mv7HvVFVn62qI1X1aFW9C/hvwIWzjmsV87Jdl6rq\ntqo6WlVfB64EfiDJzE7mSZ4AvIfm8ukr29m9257LxdnH7VlV36yqz9D0Vf5xergt4fFx9nFbak1m\n8v5a5RxxrHhG208Dlqq5lrrrXCYZy1ixruHzNrXYjvMcMNP4+rT9juOYP9X33XEc62f2udBM9Xrf\nTeA4NTFJngNcAPz7ZZp7GXPr4fbnL1bVV6vqXpqxdcY9t09M+774dZpi+sk0YyB9C3BdX2Me0afv\nTOuS5G8Bvwb8ZFX91lBTb2LerMWmO4AtSZ45NO98Hnv5WR8VMNW7IazTAZrtCECSk2nGCpiH7Qoz\n+jy0VeZ30Pw3Y3dV/UXb1KvtuUqco2a6PUds4a+2WW+25TIGcY7q07bUyvpyThmcI471fn9MO4+N\n9QBw1kiBc5xcJhnL6HOfDTyJZn8cj9HP21RiG+McMOv4Rs1k+41Y6zF/Vu+7tR7r+/S50PT05Vzy\nOBM6Tk3SAs3YaF9O8jXgXwG7k/x/ozH1KGaq6n6aK4OGiy2D3/sa9+nAtwFvbv/5dR/wTpoCWV9j\nHtan70xrluaOeZ8CXl9V7xlp7k/Max3caaNNwK/Q3PHhZHpyt4eR+J4KvJhmVPwtwKXAgwwNGjjD\n2La0cb2R5j8cgxif3m7H3e28n2e2d/xaKc7n0wzU+QTgaTR3/fj0DON8K/C7wNaR+X3bnivF2Yvt\nCTyDZiDIrcAJ7efnQeCH+7QtjxFnL7al03Ht16meU1Y7Rxzr/Q78M5rBPAd3uBm9S8nvAm9q172Y\ntd2N7rjOC+PEQjO+wGGargYns/Idy47rXDCN2Nplj+sc0IP4Zrr9GOOYP41td4z4evHec+rfRE//\nPlnlODDWZ22C8T6F5q5og+lNwIfbeHsZ89Drvw74/fYY8i00d9J7fZ/jBr5I07V/C833k4+0x57e\nxMwMvqdMMObtwJ8A/2qF9WYW8+NimdYHp28TTRX2ozQn/i8DPzrrmEbie3p7oDnSvgF+F/j+WcfV\nxnYNTZV9eLqmbbuAZhDMh2nulLSjb3ECLwe+1O77r9Lc6eOvzSjGM9u4HqG5rHEwXdqn7blanH3Z\nnu1n5jfaz8thmtuu/tOh9r5syxXj7Mu2dDqu/TrVc8qxzhGrvd9prn76eeDP2unngQy172jXeZjm\n1rkXrCGe4zovjBsL8KPt9n4Q+FXg9LXGdqzP25RiO+5zwKzjm/X2Y4xj/pS23XEf66cRn1M/J3r4\n98lqx4G2/bg/a1PM4Rrau9H1PWaaMZt+qT12fA34BeDEPsdNM7bUInA/zZh0HwS29SlmZvQ9ZRIx\nA3vb34c/j0t9iHl0SvuCkiRJkiRJ0tgcB0SSJEmSJEmdsdgkSZIkSZKkzlhskiRJkiRJUmcsNkmS\nJEmSJKkzFpskSZIkSZLUGYtNkiRJkiRJ6ozFJkmSJEmSJHXGYpMkSZIkSZI6Y7FJkiRJkiRJnbHY\nJEmSJEmSpM5YbJIkSZIkSVJnLDZJkiRJkiSpMxabJEmSJEmS1BmLTZIkSZIkSeqMxSZJkiRJkiR1\nxmKTJEmSJEmSOmOxSZIkSZIkSZ2x2CRJkiRJkqTOWGySJEmSJElSZyw2SZIkSZIkqTMWmyRJkiRJ\nktQZi02SJEmSJEnqjMUmSZIkSZIkdcZikyRJkiRJkjpjsUmSJEmSJEmdsdgkSZIkSZKkzlhskiRJ\nkiRJUmcsNkmSJEmSJKkzFpskSZIkSZLUGYtNkiRJkiRJ6ozFJkmSJEmSJHXGYpMkSZIkSZI6Y7FJ\nkiRJkiRJnbHYJEmSJEmSpM5YbJIkSZIkSVJnLDZJxyHJJUn+OMmDSf4kyffOOiZJ0vxIsjQyfTPJ\nL846LkmSpC5smXUA0rxJ8v3AdcA/BH4P+OuzjUiSNG+qauvg9yRbga8BH5pdRJIkSd2x2CSt388A\nr6uq320f3z3LYCRJc2838D+B35p1IJIkSV2wG520DklOAHYBT0/yhSR3JXlzkpNmHZskaW5dBry7\nqmrWgUiSJHXBYpO0PtuAJwIvA74XeA7wncBrZhmUJGk+JTkTeCHwrlnHIkmS1BWLTdL6PNz+/MWq\n+mpV3Qv8O+DCGcYkSZpf/wj4TFV9adaBSJIkdcVik7QOVXU/cBcw3NXBbg+SpOP1j/GqJkmStMFY\nbJLW753AP0/yjCTfAvxL4OMzjkmSNGeS/B1gO96FTpIkbTDejU5av9cDZwB3AI8AHwR+dqYRSZLm\n0WXALVV1ZNaBSJIkdSne+ESSJEmSJEldsRudJEmSJEmSOmOxSZIkSZIkSZ2x2CRJkiRJkqTOWGyS\nJEmSJElSZzbc3ejOOOOM2rFjx6zDmLgHH3yQk08+edZhTI35bmzmu7o/+IM/uLeqnj7BkDRi0ueS\njf6e3+j5wcbP0fzm23L5eS6RJE3This27dixg9tuu23WYUzc4uIiCwsLsw5jasx3YzPf1SW5c3LR\naDmTPpds9Pf8Rs8PNn6O5jfflsvPc4kkaZrsRidJkiRJkqTOWGySJEmSJElSZyw2SZIkSZIkqTMW\nmyRJkiRJktQZi02SJEmSJEnqjMUmSZIkSZIkdcZikyRp5pI8Ock7ktyZ5EiS25P8UNu2I0klWRqa\nXju0bpJcl+S+drouSYbadyT5dJKHkhxMcsEscpQkSZI2iy2zDkCSJJrz0Z8CLwS+DFwIfDDJzqFl\nnlpVR5dZ9wrgIuB8oIBPAl8C3tq23wT8TvucFwIfTvLMqrpnEolIkiRJm51XNkmSZq6qHqyqa6rq\nUFX9ZVV9nKZg9F1rWP0y4Pqququq7gbeBFwOkOQc4LnA3qp6uKpuBj4H7J5IIpIkSZK8smle7Lj6\n1sc83rPzKJePzBs4dO1LphGSJE1Mkm3AOcCBodl3JhlcufSqqrq3nX8esG9ouX3tvEHbF6vqyArt\n0qYz+p0C/O4gSZK6ZbFJktQrSZ4IvA94V1UdTLIVeB5wO/A04C1t+4vbVbYCDww9xWFgaztu02jb\noH37Mq97BU2XPLZt28bi4mJXKT3O0tLSRJ9/1jZ6fjDfOe7Z+fjeqKO5zHN+a2F+kiRNlsUmSVJv\nJHkC8B7gG8CVAFW1BNzWLvL1JFcCX01ySnvF0hJw6tDTnAYsVVUlGW0btB8ZmUdV3QDcALBr165a\nWFjoLK9Ri4uLTPL5Z22j5wfzneNyV0YfunThMY/nOb+1MD9JkiarkzGbkrw3ydeSHE5yR5JXDLW9\nqL37z0Pt3YDOHGrzDkKSJKA5JwDvALYBu6vqL1ZYtNqfg3PYAZrBwQfO56+63x0AzkpyygrtkiRJ\nkjrW1QDh1wJnVdWpwA8Db0jyXUnOAG4BXgucTvOf6Q8MrTd8B6FnAy8FXjnUfhPw32m6Tfw0zR2E\nnt5RzJKkfvll4DuAl1bVw4OZSZ6f5NuTPCHJ04BfABaratA97t3AVUm2J9kO7AFuBKiqO2i63+1N\ncmKSi4GdwM1Ty0qSJEnaZDopNlXVH1bVQ4OH7XQ2cDFwoKo+VFWPANcA5yc5t13WOwhJkmiven0l\n8Bzga0mW2ulS4Czg12m6vv0h8Cjw8qHV3wZ8DNjfTh9v5w1cAuwC7gfeCLysqu6ZbEaSJEnS5tXZ\nmE1JfommUHQSzdVInwB+lqE7BFXVg0m+QHMXoIN0dAehaQ7qOiujg3luO2n5AT7h8YN8bgSbbaBL\n893YNlu+a1FVdwJZZZGbVlm3gFe303Lth4CFMcKTJEmStA6dFZuq6ieS/HPgb9N8qX+U5i5Ao/89\nPgwMxs7o5A5C0xzUdVZGB/Pcs/Mo1+9ffveNDvK5EWy2gS7Nd2PbbPlKkiRJ2ly6GrMJgKr6ZlV9\nBvhW4Md5/B2C4LF3AVrxDkJrWFeSJEmSJEk902mxacgWmjGbHnOHoCQnD81ntB3vICRJkiRJkjTX\nxi42JXlGkkuSbE1yQpIX0wzc+l+BjwDPSrI7yYnAXmBfVR1sV/cOQpIkSZIkSRtIF2M2FU2XubfS\nFK/uBP6vqvpPAEl2A28G3gt8luauQANvo7nL0P728dt5/B2EbqS5g9CX8Q5CkiRJkiRgXBoJAAAg\nAElEQVRJvTZ2sakt/rxwlfZPAeeu0OYdhCRJkiRJkjaQSY3ZJEmSJEmSpE3IYpMkSZIkSZI6Y7FJ\nkiRJkiRJnbHYJEmSJEmSpM5YbJIkSZIkSVJnLDZJkiRJkiSpMxabJEmSJEmS1BmLTZIkSZIkSeqM\nxSZJkiRJkiR1xmKTJEmSJEmSOmOxSZIkSZIkSZ2x2CRJkiRJkqTOWGySJEmSJElSZyw2SZIkSZIk\nqTMWmyRJkiRJktQZi02SJEmSJEnqjMUmSZIkSZIkdcZikyRJkiRJkjpjsUmSJEmSJEmdsdgkSZIk\nSZKkzlhskiRJkiRJUmcsNkmSJEmSJKkzFpskSZIkSZLUGYtNkiRJkiRJ6ozFJkmSJEmSJHVm7GJT\nkicneUeSO5McSXJ7kh9q23YkqSRLQ9Nrh9ZNkuuS3NdO1yXJUPuOJJ9O8lCSg0kuGDdeSZIkSZIk\nTc6Wjp7jT4EXAl8GLgQ+mGTn0DJPraqjy6x7BXARcD5QwCeBLwFvbdtvAn6nfc4LgQ8neWZV3dNB\n3JIkSZIkSerY2Fc2VdWDVXVNVR2qqr+sqo/TFIy+aw2rXwZcX1V3VdXdwJuAywGSnAM8F9hbVQ9X\n1c3A54Dd48YsSZIkSZKkyejiyqbHSLINOAc4MDT7ziSDK5deVVX3tvPPA/YNLbevnTdo+2JVHVmh\nffg1r6C5Sopt27axuLjYQSb9smfnYy8M23bS4+cNbMT8l5aWNmReKzHfjW2z5bsWSZ4M/BJwAXA6\n8CfAv66qX2vbXwS8Bfg24LPA5VV1Z9sW4FrgFe3TvR24uqqqbd8BvBN4Ps0VuFdW1aemkpgkSZK0\nCXVabEryROB9wLuq6mCSrcDzgNuBp9H8ofA+4MXtKluBB4ae4jCwtf3DYbRt0L599HWr6gbgBoBd\nu3bVwsJCVyn1xuVX3/qYx3t2HuX6/cvvvkOXLkwhoulaXFxkI+7XlZjvxrbZ8l2j1bpkLwG30BST\nPga8HvgA8IJ2XbtkS5IkST3S2d3okjwBeA/wDeBKgKpaqqrbqupoVX29nf8DSU5pV1sCTh16mtOA\npfa/0aNtg/YjSJI2lGN0yb4YOFBVH6qqR4BrgPOTnNuubpdsSZIkqUc6ubKpvRLpHcA24MKq+osV\nFq3256DIdYDmP9G/1z4+n7/qfncAOCvJKUNd6c6nuTJKkrSBjXTJ/nGGulxX1YNJvkDTrfogc9gl\ne6N3pdzo+cF857hcN/zRXOY5v7UwP0mSJqurbnS/DHwHcEFVPTyYmeT5wJ8D/wP4FuAXgMWqGnSP\nezdwVZJPtI/3tMtQVXckuR3Ym+Q1NF0fdgI3dxSzJKmHVuiSPdrl7TAwuEp27rpkb/SulBs9P5jv\nHEe75sPju+DPc35rYX6SJE3W2MWmJGcCrwQeBb7WfLeHdt5fAj8HPIPmy/0ngZcPrf424Cxgf/v4\n7e28gUuAG4H7acbweJljbEjSxrVcl2yO3a16xS7ZSeySLUmSJE3Z2MWm9m5AWWWRm1ZZt4BXt9Ny\n7YeAhTHCkyTNiVW6ZB+gGZdpsNzJwNk8ttu1XbIlSZKknuhsgHBJksY06JL90uEu2cBHgGcl2Z3k\nRGAvsK+qDrbtgy7Z25Nsp+mSfSM0XbJp7oi6N8mJSS7GLtmSJEnSRHU1ZpMkScdttS7ZVfW+JLuB\nNwPvBT5L0816wC7ZkiRJUo9YbJIkzdyxumRX1aeAc1dos0u2JEmS1CN2o5MkSZIkSVJnLDZJkiRJ\nkiSpMxabJEmSJEmS1BmLTZIkSZIkSeqMxSZJkiRJkiR1xmKTJEmSJEmSOmOxSZIkSZIkSZ2x2CRJ\nkiRJkqTOWGySJEmSJElSZyw2SZIkSZIkqTMWmyRJkiRJktQZi02SJEmSJEnqjMUmSZIkSZIkdcZi\nkyRJkiRJkjqzZdYBSJIkqXs7rr511iFIkqRNyiubJEmSJEmS1BmLTZIkSZIkSeqM3egkSZI2udEu\nd3t2HuXyq2/l0LUvmVFEkiRpnnllkyRJkiRJkjpjsUmSJEmSJEmdsdgkSZIkSZKkzlhskiRJkiRJ\nUmfGLjYleXKSdyS5M8mRJLcn+aGh9hclOZjkoSSfTnLmUFuSXJfkvna6LkmG2ne06zzUPscF48Yr\nSZIkSZKkyeniyqYtwJ8CLwROA14DfLAtFJ0B3AK8FjgduA34wNC6VwAXAecDzwZeCrxyqP0m4L8D\nTwN+Gvhwkqd3ELMkSZIkSZImYOxiU1U9WFXXVNWhqvrLqvo48CXgu4CLgQNV9aGqegS4Bjg/ybnt\n6pcB11fVXVV1N/Am4HKAJOcAzwX2VtXDVXUz8Dlg97gxS5IkSZIkaTK2dP2ESbYB5wAHgB8H9g3a\nqurBJF8AzgMOtj/3Da2+r51H+/OLVXVkhfbh17yC5ioptm3bxuLiYlfp9MaenUcf83jbSY+fN7AR\n819aWtqQea3EfDe2zZavJEmSpM2l02JTkicC7wPeVVUHk2wF7hlZ7DBwSvv7VuCBkbat7bhNo22D\n9u2jr1tVNwA3AOzatasWFhbGzKR/Lr/61sc83rPzKNfvX373Hbp0YQoRTdfi4iIbcb+uxHw3ts2W\nryRJkqTNpbO70SV5AvAe4BvAle3sJeDUkUVPA46s0H4asFRVtYZ1JUmSJEmS1DOdFJvaK5HeAWwD\ndlfVX7RNB2gG/x4sdzJwdjv/ce3t78NtZyU5ZYV2SZIkSZIk9UxXVzb9MvAdwEur6uGh+R8BnpVk\nd5ITgb3Avqo62La/G7gqyfYk24E9wI0AVXUHcDuwN8mJSS4GdgI3dxSzJEmSJEmSOjb2mE1JzgRe\nCTwKfK25yAmAV1bV+5LsBt4MvBf4LHDJ0OpvA84C9reP397OG7iEpvh0P/Bl4GVVNToGlCRJkiRJ\nknpi7GJTVd0JZJX2TwHnrtBWwKvbabn2Q8DCuDFKkiRJkiRpOjobIFySJEmSJEmy2CRJmrkkVya5\nLcmjSW4cmr8jSSVZGppeO9SeJNclua+drstQf+52/U8neSjJwSQXTDk1SZIkadMZuxudJEkd+Arw\nBuDFwEnLtD+1qo4uM/8K4CKau5UW8EngS8Bb2/abgN8BLmynDyd5puP/SZIkSZPjlU2SpJmrqluq\n6qPAfetc9TLg+qq6q6ruBt4EXA6Q5BzgucDeqnq4qm4GPgfs7i5ySZIkSaO8skmSNA/uTDK4culV\nVXVvO/88YN/QcvvaeYO2L1bVkRXaHyPJFTRXSrFt2zYWFxe7i37E0tLSRJ9/1jZ6fjAfOe7ZudzF\ngGuz7aRm/b7neLzmYf+NY6PnJ0nqP4tNkqQ+uxd4HnA78DTgLcD7aLrbAWwFHhha/jCwtR23abRt\n0L59uReqqhuAGwB27dpVCwsL3WSwjMXFRSb5/LO20fOD+cjx8qtvPe519+w8yvX7t3Do0oXuAuqR\nedh/49jo+UmS+s9ikySpt6pqCbitffj1JFcCX01ySnvF0hJw6tAqpwFLVVVJRtsG7UeQJEmSNDGO\n2SRJmifV/hycvw7QDA4+cH47b9B2VpJTVmiXJEmSNAEWmyRJM5dkS5ITgROAE5Kc2M57fpJvT/KE\nJE8DfgFYrKpB97h3A1cl2Z5kO7AHuBGgqu6g6X63t32+i4GdwM1TTk+SJEnaVOxG1zM7xhhfQZLm\n2GuAvUOPfwz4GeDzwM8Bz6AZb+mTwMuHlnsbcBawv3389nbewCU0xaf7gS8DL6uqe7oPX5IkSdKA\nxSZJ0sxV1TXANSs037TKegW8up2Waz8ELIwVnCRJkqR1sRudJEmSJEmSOmOxSZIkSZIkSZ2x2CRJ\nkiRJkqTOWGySJEmSJElSZyw2SZIkSZIkqTMWmyRJkiRJktQZi02SJEmSJEnqjMUmSZIkSZIkdcZi\nkyRJkiRJkjqzZdYBSJKk+bbj6lsf83jPzqNcfvWtHLr2JTOKSJIkSbNksUmSJK3ZaGFJkiRJGmU3\nOkmSJEmSJHXGK5ukKVnuagC7mEiSJEmSNhqvbJIkSZIkSVJnxi42JbkyyW1JHk1y49D8HUkqydLQ\n9Nqh9iS5Lsl97XRdkoys/+kkDyU5mOSCcWOVJEmSJEnSZHXRje4rwBuAFwMnLdP+1Ko6usz8K4CL\ngPOBAj4JfAl4a9t+E/A7wIXt9OEkz6yqezqIWZIkSZIkSRMw9pVNVXVLVX0UuG+dq14GXF9Vd1XV\n3cCbgMsBkpwDPBfYW1UPV9XNwOeA3ePGK0mSJEmSpMmZxgDhdyYZXLn0qqq6t51/HrBvaLl97bxB\n2xer6sgK7Y+R5AqaK6XYtm0bi4uL3UU/ZXt2LncR2ONtO2nlZec5/5UsLS3NfV7L7a+VctoI+a6H\n+UqSJEnSxjHJYtO9wPOA24GnAW8B3kfT3Q5gK/DA0PKHga3tuE2jbYP27cu9UFXdANwAsGvXrlpY\nWOgmgxm4fJk7li1nz86jXL9/+d136NKFDiPqh8XFReZ5v8Ly+3alfbUR8l0P85UkSZKkjWNixaaq\nWgJuax9+PcmVwFeTnNJesbQEnDq0ymnAUlVVktG2QfsRJEmSJEmS1Ftjj9m0DjXymgdoBgcfOL+d\nN2g7K8kpK7RLkiRJkiSph8YuNiXZkuRE4ATghCQntvOen+TbkzwhydOAXwAWq2rQPe7dwFVJtifZ\nDuwBbgSoqjtout/tbZ/vYmAncPO48UqSJEmSJGlyuuhG9xpg79DjHwN+Bvg88HPAM2jGW/ok8PKh\n5d4GnAXsbx+/vZ03cAlN8el+4MvAy6rqng7ilSRJkiRJ0oSMXWyqqmuAa1ZovmmV9Qp4dTst134I\nWBgrOEmSJEmSJE3VNMdskiRJkiRJ0gZnsUmSJEmSJEmdsdgkSZIkSZKkzlhskiRJkiRJUme6uBud\nJEmStKwdV9/6uHmHrn3JDCKRJEnT4pVNkiRJkiRJ6ozFJkmSJEmSJHXGYpMkSZIkSZI6Y7FJkiRJ\nkiRJnXGAcEnSzCW5Ergc2AncVFWXD7W9CHgL8G3AZ4HLq+rOti3AtcAr2sXfDlxdVdW27wDeCTwf\n+DJwZVV9auIJSRuEg3tLkqTjYbFJktQHXwHeALwYOGkwM8kZwC00xaSPAa8HPgC8oF3kCuAi4Hyg\ngE8CXwLe2rbfBPwOcGE7fTjJM6vqngnnM/eWKzJIkiRJa2E3OknSzFXVLVX1UeC+kaaLgQNV9aGq\negS4Bjg/yblt+2XA9VV1V1XdDbyJ5gopkpwDPBfYW1UPV9XNwOeA3RNPSJIkSdrEvLJJktRn5wH7\nBg+q6sEkX2jnHxxtb38/b2jdL1bVkRXaHyPJFTRXSrFt2zYWFxc7SuHxlpaWJvr8Xdiz8+hxr7vt\npGb9vuc4js2yD5ez3ryXe55Zb7t52H/j2Oj5SZL6z2KTJKnPtgKjXd4OA6cMtT8w0ra1HctptG3Q\nvn25F6qqG4AbAHbt2lULCwtjBb6axcVFJvn8Xbh8jG50e3Ye5fr9Wzh06UJ3AfXMZtmHy1nvfl0u\njlm/N+Zh/41jo+cnSeo/i02SpD5bAk4dmXcacGSF9tOApaqqJMdaVxO20rhPDjAtSZK0sTlmkySp\nzw7QDP4NQJKTgbPb+Y9rb38fbjsrySkrtEuSJEmaAItNkqSZS7IlyYnACcAJSU5MsgX4CPCsJLvb\n9r3Avqo62K76buCqJNuTbAf2ADcCVNUdwO3A3vb5LgZ2AjdPNTlJkiRpk7HYJEnqg9cADwNXAz/W\n/v6aqrqH5u5xPwvcD3w3cMnQem8DPgbsb6ePt/MGLgF2teu+EXhZ+5ySJEmSJsQxmyRJM1dV1wDX\nrND2KeDcFdoKeHU7Ldd+CFjoIERJkiRJa2SxSZIkSVPl4PGSJG1sdqOTJEmSJElSZyw2SZIkSZIk\nqTMWmyRJkiRJktQZi02SJEmSJEnqjAOESzO00gCpN/7gyVOORJIkSZKkbox9ZVOSK5PcluTRJDeO\ntL0oycEkDyX5dJIzh9qS5Lok97XTdUky1L6jXeeh9jkuGDdWSZIkSZIkTVYX3ei+ArwB+I/DM5Oc\nAdwCvBY4HbgN+MDQIlcAFwHnA88GXgq8cqj9JuC/A08Dfhr4cJKndxCvJEmSJEmSJmTsYlNV3VJV\nHwXuG2m6GDhQVR+qqkeAa4Dzk5zbtl8GXF9Vd1XV3cCbgMsBkpwDPBfYW1UPV9XNwOeA3ePGK0mS\nJEmSpMmZ5JhN5wH7Bg+q6sEkX2jnHxxtb38/b2jdL1bVkRXaHyPJFTRXSrFt2zYWFxc7SmH69uw8\nuqbltp208rLznP9KlpaWepfX/rsfWHb+zu2nLTt/rfsW+pnvJJmvJEmSJG0ckyw2bQXuGZl3GDhl\nqP2Bkbat7bhNo22D9u3LvVBV3QDcALBr165aWFgYK/BZunyFAaNH7dl5lOv3L7/7Dl260GFE/bC4\nuEjf9utK+2ql7b/WfQvNAOF9y3eS+rh/J2mz5StJkiRpc+lizKaVLAGnjsw7DTiyQvtpwFJV1RrW\nlSRJkiRJUg9Nsth0gGbwbwCSnAyc3c5/XHv7+3DbWUlOWaFdkiRJkiRJPTR2sSnJliQnAicAJyQ5\nMckW4CPAs5Lsbtv3Avuq6mC76ruBq5JsT7Id2APcCFBVdwC3A3vb57sY2AncPG68kiRJkiRJmpwu\nrmx6DfAwcDXwY+3vr6mqe2juHvezwP3AdwOXDK33NuBjwP52+ng7b+ASYFe77huBl7XPKUmSJEmS\npJ4ae4DwqroGuGaFtk8B567QVsCr22m59kPAwrjxSZIkSZIkaXomOWaTJEmSJEmSNhmLTZIkSZIk\nSeqMxSZJkiRJkiR1xmKTJEmSJEmSOmOxSZIkSZIkSZ2x2CRJkiRJkqTObJl1AH234+pbl51/6NqX\nTDkSSZIkSZKk/rPYJEmSZs5/7kiSJG0cFpskSVJvWYSSJEmaP47ZJEmSJEmSpM5YbJIkSZIkSVJn\nLDZJkiRJkiSpMxabJEmSJEmS1BmLTZIkSZIkSeqMxSZJkiRJkiR1xmKTJEmSJEmSOrNl1gFIkiRp\nfuy4+tZl5x+69iVTjkSSJPWVVzZJknovyWKSR5IstdPnh9pelORgkoeSfDrJmUNtSXJdkvva6bok\nmU0WkiRJ0uZgsUmSNC+urKqt7fTtAEnOAG4BXgucDtwGfGBonSuAi4DzgWcDLwVeOdWoJUmSpE3G\nbnRSx1bqXiBpIi4GDlTVhwCSXAPcm+TcqjoIXAZcX1V3te1voilAvXVG8UqSJEkbnsWmDcixFCRt\nUG9Mci3weeCnq2oROA/YN1igqh5M8oV2/sHR9vb385Z78iRX0BSi2LZtG4uLixNIobG0tDTR5+/C\nnp1Hj3vdbSetvv5yua/39Wa9/Tb7PlzOSttjPc8zrW06D/tvHBs9P0lS/1lskiTNg58C/gj4BnAJ\n8LEkzwG2AveMLHsYOKX9fSvwwEjb1iSpqhpeqapuAG4A2LVrVy0sLHSdw/+yuLjIJJ+/C5ePcZXm\nnp1HuX7/yl8xDl26MPbrLfcc07TZ9+FyVton64ljWvt1HvbfODZ6fpKk/rPYJG1Qy13h5tVtmldV\n9dmhh+9K8nLgQmAJOHVk8dOAI+3vo+2nAUujhSZJkiRJ3XGAcEnSPCogwAGawb8BSHIycHY7n9H2\n9vcDSJIkSZoYi02SpF5L8tQkL05yYpItSS4Fvg/4deAjwLOS7E5yIrAX2NcODg7wbuCqJNuTbAf2\nADfOIA1JkiRp05h4sSnJYpJHkiy10+eH2l6U5GCSh5J8OsmZQ21Jcl2S+9rpuiSZdLySpN55IvAG\nmrGZ7gX+OXBRVd1RVfcAu4GfBe4HvptmTKeBtwEfA/a308fbeZIkSZImZFpjNl1ZVW8fnpHkDOAW\n4BU0fwi8HvgA8IJ2kSuAi2i6PBTwSeBLeLtqSdpU2oLS81Zp/xRw7gptBby6nSRJkiRNwSy70V0M\nHKiqD1XVI8A1wPlJBn8wXAZcX1V3VdXdwJuAy2cSqSRJkiRJktZkWlc2vTHJtcDngZ+uqkXgPGDf\nYIGqejDJF9r5B0fb29/PW+7Jk1xBcyUU27ZtY3FxsbPA9+w8uuz8Ll9jLa83attJa192YFIxT8PS\n0lLv4l/v9l+P9eS7/+4Hlp2/Z+fj5/VtGw70cf9O0mbLV9LmsNxdUCVJ0uY0jWLTTwF/BHyDZhyN\njyV5DrCVZvyNYYeBU9rftwIPjLRtTZLRW1ZX1Q3ADQC7du2qhYWFzoK/fIUvTocu7e411vJ6o/bs\nPMr1+9e3+yYV8zQsLi7S5X7twlr31fG48QdPXnO+64mjr++BPu7fSdps+UqSJEnaXCZebKqqzw49\nfFeSlwMXAkvAqSOLnwYcaX8fbT8NWBotNEmSJM3CSlfyHLr2JVOORJIkqV+m1Y1uWAEBDtCMywRA\nkpOBs9v5tD/PB36vfXz+UJs0lr7/gbD/7geWvWKpL/FJkiar7+cpSZKk1Uy02JTkqcDzgd8AjgL/\nEPg+4CdpblH9b5PsBm4F9gL7qupgu/q7gauSfKJ9vAf4hUnGO22ObSBJ0vFZ7hxqIUaSJKkfJn1l\n0xOBN9DckvqbNAN/X1RVdwC0haY3A+8FPkszptPA24CzgP3t47e38yRJkiRJktRTEy02VdU9wPNW\naf8UTSFqubYCXt1OkiRpg/DK3r/itpAkSRvRLMZs6iW/7EmSpC443pIkSdrsLDZJkrTJ+Q8XSZIk\ndekJsw5AkiRJkiRJG4fFJkmSJEmSJHXGbnTSHLGriyRplOcGSZLUN17ZJEmSJEmSpM54ZZMkSdqU\nlrsiyDvGSZIkjc8rmyRJkiRJktQZi02SJEmSJEnqjN3oJEnShrDSQNl2jZMkSZoui00d6vuXXMem\nOLa+70NJ0vp5tzZJkqTpstikDc8/MiRJkiRJmh6LTcfJAoYkSZIkSdLjOUC4JEmSJEmSOuOVTZJW\n5DhfktQYPR7u2XmUWXyN8spqSZI0Dyw2aWI20mDbfrmXJEmSJGltLDZpbH0pxPQljnnktpOkyfNY\nK0mSNguLTZIkbRIWO9R3G+mqaEmSNjOLTdIm4h+akrQ6j5OSJEnjs9i0ya33S3Vf/rPoHwOSJEmS\nJPWTxaYpsDCizWxSd7Szq4UkSZIk9ZPFJknrYpFHktQHno8kSeovi03qtcEXyT07j3K5V4hpDSZ1\nJZU0jtH35eCY5ntT6ifPJZIkjcdik9aliy6BfoGTtFHZbVqaPa94kiRp9iw2SeqEf2RLs2EBX5uB\n5xhJkuZLr4tNSU4H3gH8AHAv8K+r6v2zjUqT4JdIweTeB/6Xe3PbjOcSj6mSJEmapV4Xm4C3AN8A\ntgHPAW5Nsq+qDsw2LEnjWGn8mo3Eq016pbfnkvW8TywgSeMZ/gwdz7hpk/zHhf8UkSRtNL0tNiU5\nGdgNPKuqloDPJPlV4B8BV880OEkbgn+8b3zzeC7xfSlNj583SZImI1U16xiWleQ7gf9WVU8ZmrcH\nWKiql44sewVwRfvw24HPTy3Q2TmDpjvIZmG+G5v5ru7Mqnr6pILZyHp8Ltno7/mNnh9s/BzNb74t\nl5/nEknS1PT2yiZgK3B4ZN5h4JTRBavqBuCGaQTVF0luq6pds45jWsx3YzNfTVAvzyUb/T2w0fOD\njZ+j+c23jZ6fJKn/njDrAFaxBJw6Mu804MgMYpEkzSfPJZIkSdKU9bnYdAewJckzh+adD8x8QFdJ\n0tzwXCJJkiRNWW+LTVX1IHAL8LokJyf5HuCHgffMNrLe2FTdBjHfjc58NRE9Ppds9PfARs8PNn6O\n5jffNnp+kqSe6+0A4QBJTgf+I/D9wH3A1VX1/tlGJUmaJ55LJEmSpOnqdbFJkiRJkiRJ86W33egk\nSZIkSZI0fyw2SZIkSZIkqTMWm3ouyXuTfC3J4SR3JHnFMsv8mySV5IJZxNil1fJN8pQkv5Tk3iQP\nJPnNWcbahWPk+w+S/HGSI0n+KMlFs4y1S0memeSRJO8dmveiJAeTPJTk00nOnGWMXRrNN8kLknwy\nyZ8luSfJh5L89VnHqe4luTLJbUkeTXLjSNua3gdJntQeC+6aWuBrNE5+SV6V5A/bY9yXkrxq6gkc\nw5j5Jcl1Se5rp+uSZOpJrOIY+T0pyYeTHGq/YyyMtD85yVuTfL3dBh9Lsn2a8a/FODm2yzw3yW8m\nWWpz/clpxb4W4+Y3tFwvjzGSpPllsan/rgXOqqpTae6g9IYk3zVoTHI28CPAV2cUX9dWy/cG4HTg\nO9qf/3I2IXZq2XzbL+zvBa4CTgVeBbw/yTNmF2qn3gL8/uBBkjNo7hj2Wpp9exvwgdmENhGPyRf4\nFpr38w7gTOAI8M7ph6Up+ArwBpoByket9X3wKuCeCcU3rnHyC/CP2+V+ELgyySWTDPY4jJPfFcBF\nwPnAs4GXAq+cYKzHY7X8AD4D/BjwtWXafhL42zS5/Q3gfuAXJxDjuI47x/bc9OvA24CnAX8L+C+T\nCfO4jbMPB/p8jJEkzaktsw5Aq6uqPxx+2E5nA3/QznsL8FPAL005tIlYKd8kD9IUY761qg637X8w\nuv68WWX/bgH+vKp+rW27td0GZwP/c7pRdqv9Y/LPgd+m+eIOcDFwoKo+1C5zDXBvknOr6uBMAu3I\ncvkO7dfBMm8GfmP60WnSquoWgCS7gG8daTvm+yDJ36T5Q/Eq4P+ZaLDHYZz8qurnh5o/n+RXgb8L\n/MrEAl6nMfffZcD1VXVX2/4mmgLUWycZ83ocI79vAP+hbf/mMqv/TeA/V9XX22U+APy7iQZ8HMbM\n8SqaHN/XPn4U+OPJRbt+Y+bX+2OMJGl+eWXTHEjTdewh4CDNFUyfaOf/CPBoVX1ilvF1bYV8vxu4\nE/iZNN3o9ifZPcs4u7JCvrcBf5zkpUlOSNOF7lHgczMMdWxJTgVeR/Oldth5wL7Bg6p6EPhCO39u\nrZLvqO8DDkw+IvXccu+DXwT+b+Dh6YfTuRXf5233su9dqX1OjOb3mONa+/tcH/L601EAAATWSURB\nVNNGvAP4u0n+RpKnAJcCv3aMdebNC4A/S/LbSf5n21Xw22YdVMc20jFGktQjFpvmQFX9BHAKzRfx\nW4BHk5wC/BzNZewbynL50vy37lnAAzSX618JvCvJd8wqzq4sl29VfRN4N3ATTf7vB17ZFmHm2euB\ndwz+0z9kK82+HXaYZrvMs5Xy/V+SPBv4NzTdGLRJLfc+SPL3gROq6iMzC6wja3ifX0PznWQuu5Ou\nkN/oce0wsLVv4zaN4X8AfwrcTZPbd9AU1zeSb6W5Qu0ngW8DvkRzXt4QNtIxRpLUPxab5kRVfbOq\nPkPzxefHab6Yv6eqDs0yrklZJt+Hgb8A3lBV36iq3wA+DfzADMPszGi+aQZ7/3lgAXgS8ELg7Ume\nM7sox9PGfgHw75dpXqIZm2rYaTRjoMylY+Q7WOZv0VwJ8JNV9VvTik3dSLLYDrq73PSZdTzP494H\nSU6mOQb8i8lEv6a4JpbfSPuVNGM3vaSqHu0ug2PGNen8Ro9rpwFLVVXdZHDMuDrJbxVvAU6kGcvo\nZJp/lkz1yqYp5Pgw8JGq+v2qegT4GeDvJDmtg+c+pknm14djjCRpY3PMpvmzhWbcnhcC35rkJ9r5\nTwc+mOS6qrpuZtF1b5Dvf1qmbSpf2KdskO+TgN+sqtva+b+f5LM0xYvbZxXcmBZoBtL9cvuP/a3A\nCUn+d5oxTC4bLNh+CT6b+e5Ss8AK+VbVc9Pcbe9TwOur6j0zi1LHraoWxn2OVd4Hz6R5//xW+/55\nEnBakq8BL5jGPxomnN+g/Z8AVwPft9oVgJMwhfwO0AwO/nvt4/OZ4jGti/yO4TnAT1fVnwEk+UXg\ndUnOqKp7J/zawFRy/ByP/a4x1e8dE85v5scYSdLG5pVNPZbkGUkuSbK1HbfnxcDLgf8KvIimW9lz\n2ukrNHe5ecvMAh7TMfL9TeDLwL9OsiXJ3wX+HvCfZxjyWI6R7+8D3zO4kinJd9J0s5vnMZtuoCkg\nDd6zbwVuBV4MfAR4VpLdSU4E9gL75nxw8BXzTXO3wf8XeHNV9WawYHWvPV6dCJxAU2w8McmWtm21\n98EfAv8bf/X+eQXw9fb3P51W/McyRn4kuZSmO/j3V9UXpxn3Wo2TH01X6KuSbG+X3QPcOKXQ12S1\n/Nr2J7ftAE9q2wfdAH8f+MdJTkvyROAngK9Mq9C0VmPm+E7g7yd5Tpvja4HPVNVot++ZGSO/uTjG\nSJLmWFU59XSiuVrpN2juZHUY2A/80xWWPfT/t3fHJg1FURiA/zeDtZApMkEWcAzBzgVSZwIH0Alc\nwEJwA0uxsDYBEQuLZ3EfEoKFwnlcle+DU4ST4vy8kOKQe5Nk1XvmOfOmXax6l+Q1yX2Sk94zz5z3\nLO2S7JckD0nOe89cnH+d5HLv9SrtkvS3JDdJFr1nnCtv2jJtTDtm81m9Z1SzPffxoNY//Ryk/VLu\nqXeeynxp99+8H/QvemcqzDekHVN6nmqTZOid6bv5pv7jF/3F1DtKcpX2D6m7JLdJlr0zVWac+qdp\n91Jtk1wnOe6dqTLf3vt+5XeMUkqpv1vDOP7Hk0gAAAAA9OAYHQAAAABlLJsAAAAAKGPZBAAAAEAZ\nyyYAAAAAylg2AQAAAFDGsgkAAACAMpZNAAAAAJSxbAIAAACgzAciBLAgX16HnAAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x205ccdb7438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#------housingデータをpandas.Dataframeに変換したらp.49の関数で確認可能--------------------\n",
    "#\n",
    "housing2 = pd.DataFrame(housing.data)\n",
    "housing2.hist(bins=50, figsize=(20,15))\n",
    "plt.show()\n",
    "\n",
    "#housing2.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 20640 entries, 0 to 20639\n",
      "Data columns (total 8 columns):\n",
      "0    20640 non-null float64\n",
      "1    20640 non-null float64\n",
      "2    20640 non-null float64\n",
      "3    20640 non-null float64\n",
      "4    20640 non-null float64\n",
      "5    20640 non-null float64\n",
      "6    20640 non-null float64\n",
      "7    20640 non-null float64\n",
      "dtypes: float64(8)\n",
      "memory usage: 1.3 MB\n"
     ]
    }
   ],
   "source": [
    "housing2.info() #info()=header--wk, describe()=mean table, head()=raw table --not wk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(20640, 1)\n",
      "np.ones((m, 1)) type= <class 'numpy.ndarray'>\n",
      "[[ 1.]\n",
      " [ 1.]\n",
      " [ 1.]\n",
      " ..., \n",
      " [ 1.]\n",
      " [ 1.]\n",
      " [ 1.]]\n",
      "X type= <class 'tensorflow.python.framework.ops.Tensor'>\n",
      "Y type= <class 'tensorflow.python.framework.ops.Tensor'>\n"
     ]
    }
   ],
   "source": [
    "tmp=np.ones((m, 1))\n",
    "print(tmp.shape) #行列数\n",
    "print('np.ones((m, 1)) type=',type(tmp)) #型\n",
    "print(tmp) #1列の1データ\n",
    "\n",
    "housing_data_plus_bias = np.c_[np.ones((m, 1)), housing.data] #列で連結\n",
    "\n",
    "X = tf.constant(housing_data_plus_bias, dtype=tf.float32, name=\"X\")\n",
    "print('X type=',type(X))\n",
    "y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name=\"y\")#1列に変換\n",
    "print('Y type=',type(y))\n",
    "XT = tf.transpose(X) #転置\n",
    "theta = tf.matmul(tf.matmul(tf.matrix_inverse(tf.matmul(XT, X)), XT), y) \n",
    "# theta =（XT*X)^(-1)*XT*ｙ\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    theta_value = theta.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ -3.74651413e+01],\n",
       "       [  4.35734153e-01],\n",
       "       [  9.33829229e-03],\n",
       "       [ -1.06622010e-01],\n",
       "       [  6.44106984e-01],\n",
       "       [ -4.25131839e-06],\n",
       "       [ -3.77322501e-03],\n",
       "       [ -4.26648885e-01],\n",
       "       [ -4.40514028e-01]], dtype=float32)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "theta_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensorflow vs NumPy\n",
    "\n",
    "主な利点は、TensorFlowはGPUカード上でこの計算\n",
    "(正規方程式を直接計算すること)を自動実行することです。\n",
    "（GPUサポート付きのTensorFlowをインストールしたい場合は、第12章を参照）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] Compare with pure NumPy vs Tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ -3.69419202e+01]\n",
      " [  4.36693293e-01]\n",
      " [  9.43577803e-03]\n",
      " [ -1.07322041e-01]\n",
      " [  6.45065694e-01]\n",
      " [ -3.97638942e-06]\n",
      " [ -3.78654265e-03]\n",
      " [ -4.21314378e-01]\n",
      " [ -4.34513755e-01]]\n"
     ]
    }
   ],
   "source": [
    "# NumPyで計算する場合 [（XT*X)^(-1)*XT*ｙ ]\n",
    "# np.dot:内積\n",
    "# np.linalg.inv:逆行列\n",
    "X = housing_data_plus_bias\n",
    "y = housing.target.reshape(-1, 1)\n",
    "theta_numpy = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y) \n",
    "\n",
    "print(theta_numpy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[2] Compare with Scikit-Learn vs Tensorflow\n",
    "\n",
    "- [sklearn線形回帰の参考ページ](https://qiita.com/kento1109/items/3bfa934fbdfcef8ef186)\n",
    "- [MATLAB線形回帰の説明ページ](https://jp.mathworks.com/help/matlab/data_analysis/linear-regression.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ -3.69419202e+01]\n",
      " [  4.36693293e-01]\n",
      " [  9.43577803e-03]\n",
      " [ -1.07322041e-01]\n",
      " [  6.45065694e-01]\n",
      " [ -3.97638942e-06]\n",
      " [ -3.78654265e-03]\n",
      " [ -4.21314378e-01]\n",
      " [ -4.34513755e-01]]\n"
     ]
    }
   ],
   "source": [
    "# Scikit-Learnで計算する場合\n",
    "# 線形回帰モデル\n",
    "# np.r_ :　行で連結\n",
    "# lin_reg.intercept_:切片（線形回帰モデルの独立項）y=ax+bの'b'\n",
    "# lin_reg.coef_:係数 y=ax+bの'a'\n",
    "from sklearn.linear_model import LinearRegression \n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(housing.data, housing.target.reshape(-1, 1)) #線形回帰\n",
    "\n",
    "print(np.r_[lin_reg.intercept_.reshape(-1, 1), lin_reg.coef_.T])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-6] Implementing Batch Gradient Descent p.249"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batch Gradient Descent（第4章で紹介した）をNormal Equationの代わりに\n",
    "使ってみましょう。 まず、勾配を手動で計算します。\n",
    "TensorFlowのAutoDiff機能を使用して、TFが自動的に勾配を\n",
    "計算できるようにします。最後に、TensorFlowの2つのOptimizerを使用します。\n",
    "\n",
    "```\n",
    "Gradient Descentを使用する場合は、入力特徴ベクトルを\n",
    "最初にnormalizeすることが重要だった事を思い出してください。\n",
    "TensorFlow、NumPy、Scikit-LearnのStandardScaler、\n",
    "または他のライブラリにもnormalize機能があります。 \n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n",
      "2\n",
      "185760\n"
     ]
    }
   ],
   "source": [
    "#StandardScalerでデータをNormalize\n",
    "# np.c_: np.concatenate 連結\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "scaled_housing_data = scaler.fit_transform(housing.data)\n",
    "scaled_housing_data_plus_bias = np.c_[np.ones((m, 1)), scaled_housing_data]\n",
    "\n",
    "print(type(scaled_housing_data_plus_bias))\n",
    "\n",
    "print(scaled_housing_data_plus_bias.ndim)\n",
    "scaled_housing_data_plus_bias.reshape\n",
    "print(scaled_housing_data_plus_bias.size)\n",
    "\n",
    "\n",
    "#scaled_housing_data_plus_bias.describe()\n",
    "#print('housing_bias.reshape=',m1,n1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  1.00000000e+00   6.60969987e-17   5.50808322e-18   6.60969987e-17\n",
      "  -1.06030602e-16  -1.10161664e-17   3.44255201e-18  -1.07958431e-15\n",
      "  -8.52651283e-15]\n",
      "[ 0.38915536  0.36424355  0.5116157  ..., -0.06612179 -0.06360587\n",
      "  0.01359031]\n",
      "0.111111111111\n",
      "(20640, 9)\n"
     ]
    }
   ],
   "source": [
    "print(scaled_housing_data_plus_bias.mean(axis=0))\n",
    "print(scaled_housing_data_plus_bias.mean(axis=1))\n",
    "print(scaled_housing_data_plus_bias.mean())\n",
    "print(scaled_housing_data_plus_bias.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-6-1] Manually computing the gradients p.249\n",
    "手動で勾配を計算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "次のコードは、いくつかの新しい要素を除いて、かなり自明であるはずです。\n",
    "```\n",
    "•random_uniform（）関数\n",
    "　　- ランダム値を含むテンソルを生成。\n",
    "    - グラフに1つノードを生成。\n",
    "　  - NumPyのrand関数に類似\n",
    "•assign（）関数\n",
    "　  - 変数に新しい値を割り当てる\n",
    "    - 1つノードを作成\n",
    "　　- 今回はBatch Gradient Descentステップを実装します\n",
    "```\n",
    "$$\n",
    "Batch~Gradient~Descent : \\theta^{next step} =\\theta - \\eta\\nabla _{\\theta}MSE(\\theta)\n",
    "$$\n",
    "```\n",
    "•メインループ \n",
    "    - n_epoch回数分、訓練ステップを繰り返し実行します\n",
    "    - 100回の反復ごとに、平均2乗誤差（MSE）をプリントアウト\n",
    "```\n",
    "\n",
    "$$\n",
    "MSE = \\frac{1}{n}\\sum_{t=1}^{n}( y\\_pred_{t} - y_{t} )^2   \n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 MSE = 9.16154\n",
      "Epoch 100 MSE = 0.714501\n",
      "Epoch 200 MSE = 0.566705\n",
      "Epoch 300 MSE = 0.555572\n",
      "Epoch 400 MSE = 0.548812\n",
      "Epoch 500 MSE = 0.543636\n",
      "Epoch 600 MSE = 0.539629\n",
      "Epoch 700 MSE = 0.536509\n",
      "Epoch 800 MSE = 0.534068\n",
      "Epoch 900 MSE = 0.532147\n"
     ]
    }
   ],
   "source": [
    "# 手動で計算する例:AutoDiffを使わない版\n",
    "reset_graph()\n",
    "\n",
    "n_epochs = 1000 #反復数\n",
    "learning_rate = 0.01\n",
    "\n",
    "# tf.random_uniform  一様乱数生成,seedを変更すると生成パターンを変更できる\n",
    "# theta = 乱数\n",
    "# y_pred = Xと乱数の乗算\n",
    "#\n",
    "X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name=\"X\")\n",
    "y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name=\"y\")\n",
    "theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name=\"theta\")\n",
    "y_pred = tf.matmul(X, theta, name=\"predictions\") \n",
    "\n",
    "\n",
    "\n",
    "error = y_pred - y #予測誤差\n",
    "mse = tf.reduce_mean(tf.square(error), name=\"mse\") #MSE=平均2乗誤差\n",
    "gradients = 2/m * tf.matmul(tf.transpose(X), error) #勾配計算\n",
    "training_op = tf.assign(theta, theta - learning_rate * gradients)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        if epoch % 100 == 0:\n",
    "            print(\"Epoch\", epoch, \"MSE =\", mse.eval())\n",
    "        sess.run(training_op)\n",
    "    \n",
    "    best_theta = theta.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2.06855249],\n",
       "       [ 0.88740271],\n",
       "       [ 0.14401658],\n",
       "       [-0.34770882],\n",
       "       [ 0.36178368],\n",
       "       [ 0.00393812],\n",
       "       [-0.04269557],\n",
       "       [-0.66145277],\n",
       "       [-0.63752776]], dtype=float32)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-6-2 ] Using autodiff p.250"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AutoDiffを使って勾配計算を自動化します\n",
    "\n",
    "***\n",
    "\n",
    "上記の手動計算コードはコスト関数（MSE）から導かれる。\n",
    "線形回帰の場合は簡単ですが、Deep Neural Networkで使おうとしたら、\n",
    "エラーを起こしやすいでしょう。\n",
    "↓\n",
    "そこで自動化「symbolic differentiation」\n",
    "↓\n",
    "自動的に偏微分の方程式を計算する。\n",
    "が、結果のコードは必ずしも効率的ではない。\n",
    "\n",
    "関数f(x) = exp (exp (exp (x)))の事例\n",
    "　- f '(x) = exp (x)×exp (exp(x))×exp( exp( exp( x)))\n",
    "  - このf(x)とf '(x)を直接コーディングは非効率　[※ exp(x)を9回call]\n",
    "　- 効率的な解決策は、\n",
    " 　　　- 最初にexp(x)\n",
    "       - 次にexp(exp(x))\n",
    "       - 次にexp(exp(exp(x)))\n",
    "       - 3項分の計算を全て返す\n",
    "  - 3番目が直接の解になっている[※ exp(x)を3回callのみ]\n",
    "\n",
    "***\n",
    "  \n",
    "関数が任意のコードで定義されていると事態は悪化します。\n",
    "次の関数の偏微分を計算するための方程式（またはコード）を見つける事は出来ますか?\n",
    "\n",
    "```\n",
    "def my_func(a, b):\n",
    "    z = 0\n",
    "    for i in range(100):\n",
    "        z = a * np.cos(z + i) + z * np.sin(b - i)\n",
    "    return z\n",
    "```\n",
    "TFのAutoDiffの場合は、自動で偏微分を計算できる。\n",
    "```\n",
    "    gradients = tf.gradients(mse, [theta])[0]\n",
    "```\n",
    "前回のgradient descentのコードを上記に置き換えるだけで良い。\n",
    "tf.gradients関数には、Op（この場合はMSE）と変数リスト(theta)を入力します。\n",
    "Opsのリストを変数ごとに1つ作成し、Opの勾配を変数毎に計算します。\n",
    "つまり、gradientsノードはthetaに関してMSEの勾配ベクトルを計算します。\n",
    "\n",
    "勾配の自動計算法は4つの方法がある(表9-2)\n",
    "TensorFlowはリバースモードのAutoDiffを使用しています。\n",
    "多くの入力があり、出力が少ない場合、つまりNeural Networkのケースで\n",
    "精度が上がる。TFは、出力のすべての偏微分を計算します。\n",
    "$$n_{outputs} + 1$$グラフの横断で、入力全てに関して出力値の偏微分を計算します。\n",
    "\n",
    "\n",
    "Table 9_2\n",
    "\n",
    "| Technique | Nb of graph traversals | Accuracy | Supports arbitrary code | Comment |\n",
    "|:-------------|:------------|:-------------|:---------|:---------|\n",
    "| Numerical differentiation | ninputs + 1 | Low | Yes | Trivial to implement |\n",
    "| Symbolic differentiation| N/A | High | No | Builds a very different graph |\n",
    "| Forward-mode autodiff | ninputs | High | Yes | Uses dual numbers |\n",
    "| Reverse-mode autodiff | noutputs + 1 | High | Yes |  Implemented by TensorFlow |\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# tensorflowのautodiffの場合\n",
    "reset_graph()\n",
    "\n",
    "n_epochs = 1000\n",
    "learning_rate = 0.01\n",
    "\n",
    "X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name=\"X\")\n",
    "y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name=\"y\")\n",
    "theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name=\"theta\")\n",
    "y_pred = tf.matmul(X, theta, name=\"predictions\")\n",
    "error = y_pred - y\n",
    "mse = tf.reduce_mean(tf.square(error), name=\"mse\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gradients = tf.gradients(mse, [theta])[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 MSE = 9.16154\n",
      "Epoch 100 MSE = 0.714501\n",
      "Epoch 200 MSE = 0.566705\n",
      "Epoch 300 MSE = 0.555572\n",
      "Epoch 400 MSE = 0.548812\n",
      "Epoch 500 MSE = 0.543636\n",
      "Epoch 600 MSE = 0.539629\n",
      "Epoch 700 MSE = 0.536509\n",
      "Epoch 800 MSE = 0.534068\n",
      "Epoch 900 MSE = 0.532147\n",
      "Best theta:\n",
      "[[ 2.06855249]\n",
      " [ 0.88740271]\n",
      " [ 0.14401658]\n",
      " [-0.34770882]\n",
      " [ 0.36178368]\n",
      " [ 0.00393811]\n",
      " [-0.04269556]\n",
      " [-0.66145277]\n",
      " [-0.6375277 ]]\n"
     ]
    }
   ],
   "source": [
    "training_op = tf.assign(theta, theta - learning_rate * gradients)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        if epoch % 100 == 0:\n",
    "            print(\"Epoch\", epoch, \"MSE =\", mse.eval())\n",
    "        sess.run(training_op)\n",
    "    \n",
    "    best_theta = theta.eval()\n",
    "\n",
    "print(\"Best theta:\")\n",
    "print(best_theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以下、本には掲載されていない事例[A-1]\n",
    "How could you find the partial derivatives of the following function with regards to `a` and `b`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#掲載なし事例(ref.p.250)[A-2]\n",
    "def my_func(a, b):\n",
    "    z = 0\n",
    "    for i in range(100):\n",
    "        z = a * np.cos(z + i) + z * np.sin(b - i)\n",
    "    return z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.21253923284754914"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#掲載なし事例[A-3]\n",
    "my_func(0.2, 0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#掲載なし事例[A-4]\n",
    "reset_graph()\n",
    "\n",
    "a = tf.Variable(0.2, name=\"a\")\n",
    "b = tf.Variable(0.3, name=\"b\")\n",
    "z = tf.constant(0.0, name=\"z0\")\n",
    "for i in range(100):\n",
    "    z = a * tf.cos(z + i) + z * tf.sin(b - i)\n",
    "\n",
    "grads = tf.gradients(z, [a, b])\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-0.212537\n",
      "[-1.1388494, 0.19671395]\n"
     ]
    }
   ],
   "source": [
    "#掲載なし事例 [A-5]\n",
    "# Let's compute the function at $a=0.2$ and $b=0.3$, and the partial \n",
    "# derivatives at that point with regards to $a$ and with regards to $b$:\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    print(z.eval())\n",
    "    print(sess.run(grads))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-6-3] Using a optimizer : `GradientDescentOptimizer`　p.251"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TensorFlow computes the gradients. \n",
    "# TensorFlowはa Gradient Descent optimizer以外にもoptimizersがある。\n",
    "# 使いたい場合は、(gradients = ...,training_op = ...) 行を置き換えるだけ。\n",
    "\n",
    "reset_graph()\n",
    "\n",
    "n_epochs = 1000\n",
    "learning_rate = 0.01\n",
    "\n",
    "X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name=\"X\")\n",
    "y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name=\"y\")\n",
    "theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name=\"theta\")\n",
    "y_pred = tf.matmul(X, theta, name=\"predictions\")\n",
    "error = y_pred - y\n",
    "mse = tf.reduce_mean(tf.square(error), name=\"mse\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
    "training_op = optimizer.minimize(mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 MSE = 9.16154\n",
      "Epoch 100 MSE = 0.714501\n",
      "Epoch 200 MSE = 0.566705\n",
      "Epoch 300 MSE = 0.555572\n",
      "Epoch 400 MSE = 0.548812\n",
      "Epoch 500 MSE = 0.543636\n",
      "Epoch 600 MSE = 0.539629\n",
      "Epoch 700 MSE = 0.536509\n",
      "Epoch 800 MSE = 0.534068\n",
      "Epoch 900 MSE = 0.532147\n",
      "Best theta:\n",
      "[[ 2.06855249]\n",
      " [ 0.88740271]\n",
      " [ 0.14401658]\n",
      " [-0.34770882]\n",
      " [ 0.36178368]\n",
      " [ 0.00393811]\n",
      " [-0.04269556]\n",
      " [-0.66145277]\n",
      " [-0.6375277 ]]\n"
     ]
    }
   ],
   "source": [
    "# 異なるOptimizerを使いたたい時は、1行変更するだけで良い\n",
    "# momentum optimizerは、勾配降下法より速く収束する。\n",
    "# 11章でoptimizerを定義する。\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        if epoch % 100 == 0:\n",
    "            print(\"Epoch\", epoch, \"MSE =\", mse.eval())\n",
    "        sess.run(training_op)\n",
    "    \n",
    "    best_theta = theta.eval()\n",
    "\n",
    "print(\"Best theta:\")\n",
    "print(best_theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using a momentum optimizer\n",
    "momentum optimizerの使い方(本に掲載無し)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# momentum optimizer事例：本掲載なし[B-1]\n",
    "reset_graph()\n",
    "\n",
    "n_epochs = 1000\n",
    "learning_rate = 0.01\n",
    "\n",
    "X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name=\"X\")\n",
    "y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name=\"y\")\n",
    "theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name=\"theta\")\n",
    "y_pred = tf.matmul(X, theta, name=\"predictions\")\n",
    "error = y_pred - y\n",
    "mse = tf.reduce_mean(tf.square(error), name=\"mse\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# momentum optimizer事例：本掲載なし[B-2]\n",
    "optimizer = tf.train.MomentumOptimizer(learning_rate=learning_rate,\n",
    "                                       momentum=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# momentum optimizer事例：本掲載なし[B-3]\n",
    "training_op = optimizer.minimize(mse)\n",
    "\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best theta:\n",
      "[[ 2.06855798]\n",
      " [ 0.82962859]\n",
      " [ 0.11875337]\n",
      " [-0.26554456]\n",
      " [ 0.30571091]\n",
      " [-0.00450251]\n",
      " [-0.03932662]\n",
      " [-0.89986444]\n",
      " [-0.87052065]]\n"
     ]
    }
   ],
   "source": [
    "# momentum optimizer事例：本掲載なし[B-4]\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        sess.run(training_op)\n",
    "    \n",
    "    best_theta = theta.eval()\n",
    "\n",
    "print(\"Best theta:\")\n",
    "print(best_theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-7] Feeding data to the training algorithm p.252\n",
    "\n",
    "####\n",
    "Mini-batch gradient descentを実装するために、上のコードを修正する\n",
    " - mini-batchで各iterationでXとyを置き換える。\n",
    " - placeholderノードだと簡単。\n",
    " - これらのノードは計算実行せず、Tensorflowに訓練データを渡す\n",
    " \n",
    "Placeholderノード生成\n",
    " - placeholder() function とtensorデータ型指定\n",
    " - データ型Noneも指定可能\n",
    " - a placeholder node A, and also a node B = A + 5. \n",
    "    - we pass a feed_dict to the eval() method that specifies the value of A.\n",
    "    - Aはrank2(= 2dim), 3 columns,it can have any number of rows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [9-7-1] Placeholder nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 6.  7.  8.]]\n"
     ]
    }
   ],
   "source": [
    "reset_graph()\n",
    "\n",
    "A = tf.placeholder(tf.float32, shape=(None, 3))\n",
    "B = A + 5\n",
    "with tf.Session() as sess:\n",
    "    B_val_1 = B.eval(feed_dict={A: [[1, 2, 3]]})\n",
    "    B_val_2 = B.eval(feed_dict={A: [[4, 5, 6], [7, 8, 9]]})\n",
    "\n",
    "print(B_val_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  9.  10.  11.]\n",
      " [ 12.  13.  14.]]\n"
     ]
    }
   ],
   "source": [
    "print(B_val_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Chips\n",
    "You can actually feed the output of any operations, not just placeholders.\n",
    "In this case TensorFlow does not try to evaluate these\n",
    "operations, it uses the values you feed it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-7-2] Mini-batch Gradient Descent　p.252\n",
    "Mini-batch Gradient Descentの実装には、既存コードを微調整するだけで済む。 \n",
    " - 構築段階でXとYをプレースホルダノードへ変更"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_epochs = 1000\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=(None, n + 1), name=\"X\")\n",
    "y = tf.placeholder(tf.float32, shape=(None, 1), name=\"y\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name=\"theta\")\n",
    "y_pred = tf.matmul(X, theta, name=\"predictions\")\n",
    "error = y_pred - y\n",
    "mse = tf.reduce_mean(tf.square(error), name=\"mse\")\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
    "training_op = optimizer.minimize(mse)\n",
    "\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_epochs = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Then define the batch size and compute the total number of batches:\n",
    "batch_size = 100\n",
    "n_batches = int(np.ceil(m / batch_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最後に、実行段階で、ミニバッチを1つずつ取得。\n",
    "互いに依存するノードを評価する時にfeed_dictパラメータからXとyの値を指定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fetch_batch(epoch, batch_index, batch_size):\n",
    "    np.random.seed(epoch * n_batches + batch_index)  # not shown in the book\n",
    "    indices = np.random.randint(m, size=batch_size)  # not shown\n",
    "    X_batch = scaled_housing_data_plus_bias[indices] # not shown\n",
    "    y_batch = housing.target.reshape(-1, 1)[indices] # not shown\n",
    "    return X_batch, y_batch\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        for batch_index in range(n_batches):\n",
    "            X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})\n",
    "\n",
    "    best_theta = theta.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2.07033372],\n",
       "       [ 0.86371452],\n",
       "       [ 0.12255151],\n",
       "       [-0.31211874],\n",
       "       [ 0.38510373],\n",
       "       [ 0.00434168],\n",
       "       [-0.01232954],\n",
       "       [-0.83376896],\n",
       "       [-0.80304712]], dtype=float32)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "シータを評価するときにXとYの値を渡す必要はない。(どちらにも依存しない為)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [9-8] Saving and restoring a model p.253\n",
    "\n",
    "- モデル訓練後、パラメータをディスクに保存しておく\n",
    "- チェックポイントを定期的に保存可能（訓練中コンピュータクラッシュ後に続行する為）\n",
    "- TensorFlowは、チェックポイント後からモデル復元する。\n",
    "- Saverノードを構築段階の最後に作成、実行フェーズでsave()method呼び出し\n",
    "- セッションとチェックポイントファイルのパスを渡す\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 MSE = 9.16154\n",
      "Epoch 100 MSE = 0.714501\n",
      "Epoch 200 MSE = 0.566705\n",
      "Epoch 300 MSE = 0.555572\n",
      "Epoch 400 MSE = 0.548812\n",
      "Epoch 500 MSE = 0.543636\n",
      "Epoch 600 MSE = 0.539629\n",
      "Epoch 700 MSE = 0.536509\n",
      "Epoch 800 MSE = 0.534068\n",
      "Epoch 900 MSE = 0.532147\n"
     ]
    }
   ],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_epochs = 1000                                                                       # not shown in the book\n",
    "learning_rate = 0.01                                                                  # not shown\n",
    "\n",
    "X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name=\"X\")            # not shown\n",
    "y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name=\"y\")            # not shown\n",
    "theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name=\"theta\")\n",
    "y_pred = tf.matmul(X, theta, name=\"predictions\")                                      # not shown\n",
    "error = y_pred - y                                                                    # not shown\n",
    "mse = tf.reduce_mean(tf.square(error), name=\"mse\")                                    # not shown\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)            # not shown\n",
    "training_op = optimizer.minimize(mse)                                                 # not shown\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        if epoch % 100 == 0:\n",
    "            print(\"Epoch\", epoch, \"MSE =\", mse.eval())                                # not shown\n",
    "            save_path = saver.save(sess, \"./tmp/my_model.ckpt\")\n",
    "        sess.run(training_op)\n",
    "    \n",
    "    best_theta = theta.eval()\n",
    "    save_path = saver.save(sess, \"./tmp/my_model_final.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2.06855249],\n",
       "       [ 0.88740271],\n",
       "       [ 0.14401658],\n",
       "       [-0.34770882],\n",
       "       [ 0.36178368],\n",
       "       [ 0.00393811],\n",
       "       [-0.04269556],\n",
       "       [-0.66145277],\n",
       "       [-0.6375277 ]], dtype=float32)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "モデル復元も簡単\n",
    "- 構築段階最後にSaverを作成\n",
    "- 実行フェーズの開始時に、init nodeで変数初期化の代わりにrestore（）メソッドを呼出\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./tmp/my_model_final.ckpt\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./tmp/my_model_final.ckpt\")\n",
    "    best_theta_restored = theta.eval() # not shown in the book"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(best_theta, best_theta_restored)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "デフォルトでは、Saverは全変数をリストアする。\n",
    "\n",
    "しかし使い勝手を考えて、thetaを”weights”など異なる名前でrestoreしておく。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "saver = tf.train.Saver({\"weights\": theta})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##以下、本に無い解説\n",
    "\n",
    "Saverはグラフ構造も保存する。\n",
    "- グラフ構造自体が、2番目ファイルとして拡張子.metaで保存される。 \n",
    "- tf.train.import_meta_graph（）関数を使用して、グラフ構造を復元可能。\n",
    "- この関数は、保存ファイルのグラフをデフォルトのグラフにロードする。\n",
    "\n",
    "By default the saver also saves the graph structure itself in a second file with the extension `.meta`. You can use the function `tf.train.import_meta_graph()` to restore the graph structure. This function loads the graph into the default graph and returns a `Saver` that can then be used to restore the graph state (i.e., the variable values):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./tmp/my_model_final.ckpt\n"
     ]
    }
   ],
   "source": [
    "reset_graph()\n",
    "# notice that we start with an empty graph.\n",
    "\n",
    "saver = tf.train.import_meta_graph(\"./tmp/my_model_final.ckpt.meta\")  # this loads the graph structure\n",
    "theta = tf.get_default_graph().get_tensor_by_name(\"theta:0\") # not shown in the book\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./tmp/my_model_final.ckpt\")  # this restores the graph's state\n",
    "    best_theta_restored = theta.eval() # not shown in the book"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(best_theta, best_theta_restored)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "つまり、グラフを構築のPythonコードが無くても、事前に訓練されたモデルをインポートできることを意味する。\n",
    "\n",
    "- モデルを調整して保存しておくと便利\n",
    "- モデルを構築したコードのバージョンを検索することなく、保存モデルを読み込める。\n",
    "\n",
    "This means that you can import a pretrained model without having to have the corresponding Python code to build the graph. This is very handy when you keep tweaking and saving your model: you can load a previously saved model without having to search for the version of the code that built it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "nav_menu": {
   "height": "603px",
   "width": "616px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}