hw15_reinforcement_learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hw15_reinforcement_learning.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Iallen520/lhy_DL_Hw/blob/master/hw15_reinforcement_learning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fp30SB4bxeQb",
        "colab_type": "text"
      },
      "source": [
        "# **Homework 15 - Reinforcement Learning**\n",
        "\n",
        "若有任何問題，歡迎來信至助教信箱 ntu-ml-2020spring-ta@googlegroups.com\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yXsnCWPtWSNk",
        "colab_type": "text"
      },
      "source": [
        "## 前置作業\n",
        "\n",
        "首先我們需要安裝必要的系統套件及 PyPi 套件。\n",
        "gym 這個套件由 OpenAI 所提供，是一套用來開發與比較 Reinforcement Learning 演算法的工具包（toolkit）。\n",
        "而其餘套件則是為了在 Notebook 中繪圖所需要的套件。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5e2bScpnkVbv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "!apt update\n",
        "!apt install python-opengl xvfb -y\n",
        "!pip install gym[box2d] pyvirtualdisplay piglet"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M_-i3cdoYsks",
        "colab_type": "text"
      },
      "source": [
        "接下來，設置好 virtual display，並引入所有必要的套件。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nl2nREINDLiw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "from pyvirtualdisplay import Display\n",
        "virtual_display = Display(visible=0, size=(1400, 900))\n",
        "virtual_display.start()\n",
        "\n",
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from IPython import display\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import torch.nn.functional as F\n",
        "from torch.distributions import Categorical"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "He0XDx6bzjgC",
        "colab_type": "text"
      },
      "source": [
        "最後，引入 OpenAI 的 gym，並建立一個 [Lunar Lander](https://gym.openai.com/envs/LunarLander-v2/) 環境。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N_4-xJcbBt09",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "import gym\n",
        "env = gym.make('LunarLander-v2')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NrkVvTrvWZ5H",
        "colab_type": "text"
      },
      "source": [
        "## 什麼是 Lunar Lander？\n",
        "\n",
        "“LunarLander-v2” 這個環境是在模擬登月小艇降落在月球表面時的情形。\n",
        "這個任務的目標是讓登月小艇「安全地」降落在兩個黃色旗幟間的平地上。\n",
        "> Landing pad is always at coordinates (0,0).\n",
        "> Coordinates are the first two numbers in state vector.\n",
        "\n",
        "![](https://gym.openai.com/assets/docs/aeloop-138c89d44114492fd02822303e6b4b07213010bb14ca5856d2d49d6b62d88e53.svg)\n",
        "\n",
        "所謂的「環境」其實同時包括了 agent 和 environment。\n",
        "我們利用 `step()` 這個函式讓 agent 行動，而後函式便會回傳 environment 給予的 observation/state（以下這兩個名詞代表同樣的意思）和 reward。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bIbp82sljvAt",
        "colab_type": "text"
      },
      "source": [
        "### Observation / State\n",
        "\n",
        "首先，我們可以看看 environment 回傳給 agent 的 observation 究竟是長什麼樣子的資料："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rsXZra3N9R5T",
        "colab_type": "code",
        "outputId": "f0604c7f-48d6-49cb-8921-6407b3f6c4f9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(env.observation_space)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Box(8,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ezdfoThbAQ49",
        "colab_type": "text"
      },
      "source": [
        "`Box(8,)` 說明我們會拿到 8 維的向量作為 observation，其中包含：垂直及水平座標、速度、角度、加速度等等，這部分我們就不細說。\n",
        "\n",
        "### Action\n",
        "\n",
        "而在 agent 得到 observation 和 reward 以後，能夠採取的動作有："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p1k4dIrBAaKi",
        "colab_type": "code",
        "outputId": "11c1a4d8-6455-451f-9999-16a5eec8fa63",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(env.action_space)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Discrete(4)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dejXT6PHBrPn",
        "colab_type": "text"
      },
      "source": [
        "`Discrete(4)` 說明 agent 可以採取四種離散的行動：\n",
        "- 0 代表不採取任何行動\n",
        "- 2 代表主引擎向下噴射\n",
        "- 1, 3 則是向左右噴射\n",
        "\n",
        "接下來，我們嘗試讓 agent 與 environment 互動。\n",
        "在進行任何操作前，建議先呼叫 `reset()` 函式讓整個「環境」重置。\n",
        "而這個函式同時會回傳「環境」最初始的狀態。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pi4OmrmZgnWA",
        "colab_type": "code",
        "outputId": "b1fbd5cf-dfcd-40a1-f5e8-4ea078aec347",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "initial_state = env.reset()\n",
        "print(initial_state)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ 4.1179656e-04  1.4151570e+00  4.1696824e-02  1.8830016e-01\n",
            " -4.7039185e-04 -9.4449501e-03  0.0000000e+00  0.0000000e+00]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uBx0mEqqgxJ9",
        "colab_type": "text"
      },
      "source": [
        "接著，我們試著從 agent 的四種行動空間中，隨機採取一個行動"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vxkOEXRKgizt",
        "colab_type": "code",
        "outputId": "e4921d68-698f-46fe-f608-27fd18c4c78c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "random_action = env.action_space.sample()\n",
        "print(random_action)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mns-bO01g0-J",
        "colab_type": "text"
      },
      "source": [
        "再利用 `step()` 函式讓 agent 根據我們隨機抽樣出來的 `random_action` 動作。\n",
        "而這個函式會回傳四項資訊：\n",
        "- observation / state\n",
        "- reward\n",
        "- 完成與否\n",
        "- 其餘資訊"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E_WViSxGgIk9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "observation, reward, done, info = env.step(random_action)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FdieGq7NuBIm",
        "colab_type": "text"
      },
      "source": [
        "第一項資訊 `observation` 即為 agent 採取行動之後，agent 對於環境的 observation 或者說環境的 state 為何。\n",
        "而第三項資訊 `done` 則是 `True` 或 `False` 的布林值，當登月小艇成功著陸或是不幸墜毀時，代表這個回合（episode）也就跟著結束了，此時 `step()` 函式便會回傳 `done = True`，而在那之前，`done` 則保持 `False`。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yK7r126kuCNp",
        "colab_type": "code",
        "outputId": "3b232447-c2a2-498e-a0ae-fe16c3396a05",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(done)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "False\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GKdS8vOihxhc",
        "colab_type": "text"
      },
      "source": [
        "### Reward\n",
        "\n",
        "而「環境」給予的 reward 大致是這樣計算：\n",
        "- 小艇墜毀得到 -100 分\n",
        "- 小艇在黃旗幟之間成功著地則得 100~140 分\n",
        "- 噴射主引擎（向下噴火）每次 -0.3 分\n",
        "- 小艇最終完全靜止則再得 100 分\n",
        "\n",
        "> Reward for moving from the top of the screen to landing pad and zero speed is about 100..140 points.\n",
        "> If lander moves away from landing pad it loses reward back.\n",
        "> Episode finishes if the lander crashes or comes to rest, receiving additional -100 or +100 points.\n",
        "> Each leg ground contact is +10.\n",
        "> Firing main engine is -0.3 points each frame.\n",
        "> Solved is 200 points."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vxQNs77hi0_7",
        "colab_type": "code",
        "outputId": "44c84a9f-499a-4e3d-d1d8-e565791a5f89",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(reward)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2.30736201823902\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mhqp6D-XgHpe",
        "colab_type": "text"
      },
      "source": [
        "### Random Agent\n",
        "\n",
        "最後，在進入實做之前，我們就來看看這樣一個 random agent 能否成功登陸月球："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y3G0bxoccelv",
        "colab_type": "code",
        "outputId": "67f637a7-9c90-404c-d53c-6c1d54ba30a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        }
      },
      "source": [
        "env.reset()\n",
        "\n",
        "img = plt.imshow(env.render(mode='rgb_array'))\n",
        "\n",
        "done = False\n",
        "while not done:\n",
        "    action = env.action_space.sample()\n",
        "    observation, reward, done, _ = env.step(action)\n",
        "\n",
        "    img.set_data(env.render(mode='rgb_array'))\n",
        "    display.display(plt.gcf())\n",
        "    display.clear_output(wait=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAATWElEQVR4nO3df6zldZ3f8edrAdGuZgFlyTAzFq2z\na8imO8gtYtY0SCOLZLawibWSZiWWdGwCiSamLWyTimn3j01WaY0bsrOBChsXlvqjTCZtFJHG9g/B\niyLyQ9ZxxTDDyLjLDyWmbMF3/zifyx5n7p177rn33HM/5zwfycn5fj/f7/d8P5/hzGs+vM/3e06q\nCklSP35p2h2QJK2NwS1JnTG4JakzBrckdcbglqTOGNyS1JmJBXeSS5M8nuRgkusmdR5JmjeZxHXc\nSU4C/hJ4N3AI+AZwZVU9uuEnk6Q5M6kZ9wXAwar6q6r6W+AO4PIJnUuS5srJE3rd7cCTQ+uHgLev\ntHMSb9+UpGNUVZZrn1RwryrJXmDvtM4vSb2aVHAfBnYOre9oba+oqn3APnDGLUlrMaka9zeAXUne\nlORVwPuB/RM6lyTNlYnMuKvqpSTXAl8CTgJuqapHJnEuSZo3E7kccM2dsFQiScdZ6cNJ75yUpM4Y\n3JLUGYNbkjpjcEtSZwxuSeqMwS1JnTG4JakzBrckdcbglqTOGNyS1BmDW5I6Y3BLUmcMbknqjMEt\nSZ0xuCWpMwa3JHXG4JakzhjcktSZdf3mZJIngJ8CLwMvVdVCkjOAvwDOAZ4A3ldVz66vm5KkJRsx\n435XVe2uqoW2fh1wT1XtAu5p65KkDTKJUsnlwK1t+VbgigmcQ5Lm1nqDu4AvJ3kgyd7WdlZVHWnL\nPwLOWuc5JElD1lXjBt5ZVYeT/Cpwd5LvDm+sqkpSyx3Ygn7vctskSStL1bK5uvYXSm4AXgD+FXBR\nVR1Jsg34X1X166scuzGdkKQZUlVZrn3sUkmSX07yuqVl4BLgYWA/cFXb7SrgrnHPIUk63tgz7iRv\nBr7YVk8G/ryq/iDJ64E7gTcCP2RwOeAzq7yWM25JOsZKM+4NK5Wsh8EtScfb8FKJJGk6DG5J6ozB\nLUmdMbglqTMGtyR1xuCWpM4Y3JLUGYNbkjpjcEtSZwxuSeqMwS1JnTG4JakzBrckdcbglqTOGNyS\n1BmDW5I6Y3BLUmcMbknqjMEtSZ1ZNbiT3JLkaJKHh9rOSHJ3ku+159Nbe5J8KsnBJA8ledskOy9J\n82iUGfdngEuPabsOuKeqdgH3tHWA9wC72mMvcNPGdFOStGTV4K6qrwHPHNN8OXBrW74VuGKo/bYa\n+DpwWpJtG9VZSdL4Ne6zqupIW/4RcFZb3g48ObTfodZ2nCR7kywmWRyzD5I0l05e7wtUVSWpMY7b\nB+wDGOd4SZpX4864n14qgbTno639MLBzaL8drU2StEHGDe79wFVt+SrgrqH2D7SrSy4Enh8qqUiS\nNkCqTlylSHI7cBHwBuBp4GPAfwfuBN4I/BB4X1U9kyTApxlchfIz4INVtWoN21KJJB2vqrJc+6rB\nvRkMbkk63krB7Z2TktQZg1uSOmNwS1JnDG5J6ozBLUmdMbglqTMGtyR1xuCWpM4Y3JLUGYNbkjpj\ncEtSZwxuSeqMwS1JnTG4JakzBrckdcbglqTOGNyS1BmDW5I6s2pwJ7klydEkDw+13ZDkcJIH2+Oy\noW3XJzmY5PEkvz2pjkvSvBrlx4L/MfACcFtV/UZruwF4oar+6Jh9zwVuBy4Azga+AvxaVb28yjn8\nzUlJOsbYvzlZVV8DnhnxPJcDd1TVi1X1A+AggxCXJG2Q9dS4r03yUCulnN7atgNPDu1zqLUdJ8ne\nJItJFtfRB0maO+MG903APwB2A0eAT6z1BapqX1UtVNXCmH2QpLk0VnBX1dNV9XJV/Rz4U/6uHHIY\n2Dm0647WJknaIGMFd5JtQ6u/CyxdcbIfeH+SU5O8CdgF3L++LkqShp282g5JbgcuAt6Q5BDwMeCi\nJLuBAp4APgRQVY8kuRN4FHgJuGa1K0okSWuz6uWAm9IJLweUpOOMfTmgJGlrMbglqTMGtyR1xuCW\npM4Y3JLUGYNbkjpjcEtSZwxuSeqMwS1JnTG4JakzBrckdcbglqTOGNyS1BmDW5I6Y3BLUmcMbknq\njMEtSZ0xuCWpM6sGd5KdSe5N8miSR5J8uLWfkeTuJN9rz6e39iT5VJKDSR5K8rZJD0KS5skoM+6X\ngI9W1bnAhcA1Sc4FrgPuqapdwD1tHeA9DH7dfRewF7hpw3stSXNs1eCuqiNV9c22/FPgMWA7cDlw\na9vtVuCKtnw5cFsNfB04Lcm2De+5JM2pNdW4k5wDnAfcB5xVVUfaph8BZ7Xl7cCTQ4cdam3Hvtbe\nJItJFtfYZ0maayMHd5LXAp8HPlJVPxneVlUF1FpOXFX7qmqhqhbWcpwkzbuRgjvJKQxC+7NV9YXW\n/PRSCaQ9H23th4GdQ4fvaG2SpA0wylUlAW4GHquqTw5t2g9c1ZavAu4aav9Au7rkQuD5oZKKJGmd\nMqhynGCH5J3A/wa+A/y8Nf8+gzr3ncAbgR8C76uqZ1rQfxq4FPgZ8MGqOmEdO8mayiySNA+qKsu1\nrxrcm8HglqTjrRTc3jkpSZ0xuCWpMwa3JHXG4JakzhjcktQZg1uSOmNwS1JnDG5J6ozBLWndnrry\nSp668sppd2NuGNySNozhvTkMbkkb5uzbb592F+aC31UiSVuU31UiaVPVtddOuwszy+CWtOGWQtvw\nngyDW9LE5NOfnnYXZpI1bknaoqxxS9KMMLglqTOj/FjwziT3Jnk0ySNJPtzab0hyOMmD7XHZ0DHX\nJzmY5PEkvz3JAUjSvBnlx4K3Aduq6ptJXgc8AFwBvA94oar+6Jj9zwVuBy4Azga+AvxaVb18gnNY\n45akY4xd466qI1X1zbb8U+AxYPsJDrkcuKOqXqyqHwAHGYS4JGkDrKnGneQc4DzgvtZ0bZKHktyS\n5PTWth14cuiwQ5w46CUAqorFxWn3Yvr8M9BqTh51xySvBT4PfKSqfpLkJuA/AtWePwH8yzW83l5g\n79q6q3mwXHAtLGx+P6ZppfDu/c9hz54bXlk+cOCGFffTiY0U3ElOYRDan62qLwBU1dND2/8UONBW\nDwM7hw7f0dp+QVXtA/a1461x64RmNcjWqtd/1JYCe+HsobnaHsN7XKsGd5IANwOPVdUnh9q3VdWR\ntvq7wMNteT/w50k+yeDDyV3A/Rvaa0ldGJ5h/0JoL60b3mMZZcb9W8DvAd9J8mBr+33gyiS7GZRK\nngA+BFBVjyS5E3gUeAm45kRXlEij6GFWuRm22p/DSqWPZWfY2jDe8q4to6p44IFsuXDabIuLWy+g\nlwwHNfxdMC8+tQ8YhPeePTesKbAXn9rnrHsFK10OaHBry6gqBpU5bTUrzaCXAnvYOLNsw3t5KwX3\nyFeVSJpvC2fvPS6ol8J2rbPs5V7bevfoDG5JI1lpVrze0J4H41Q2Fk5QLzO4Ja1qM2bCvc+6N7Ps\n7LcDSlqXAwduYPGpfcvWu8dx7Aeg01JVa3psJoNb0rpt1Cx54ey9LJy9d6rhPY0gXiuDW9KGWJp5\n92haM+dxGdySNsy0wvtPzj9/zcf0FtbDDG5pztx7ySXce8klE3v9zQ7vpdAeNbx7DethBrc0pyYZ\n3usxqZtxZiGwl3g5oDSn3vXlL0/stQ8cuAH2rP0uynFC+0MPPHDC7bMS1sMMbmnOTDKw12MjZ9qz\nGNbDDG5JE7E064bVZ94bFdqzHthLrHFLmphRwni9od3z1SHjcsYtaaKGZ95Lhr+watzQnqegPpZf\n66otw691nW2j3A05aohvhdyatIWFBRYXF/1aV0krGzUM1/OP64lq3atd+z0PYT0qg1vaQB89/3w+\nscrladO23gAc9/jf+Z2Pj3VjjoF9vFVLJUleDXwNOJVB0H+uqj6W5E3AHcDrgQeA36uqv01yKnAb\ncD7wN8A/r6onVjmH/2XUdalkz/nn8+tD61slvA29fp2oVDLKVSUvAhdX1W8Cu4FLk1wI/CFwY1W9\nBXgWuLrtfzXwbGu/se0nzaw97Vbrx9v6NEJ7K3zVqDbPqsFdAy+01VPao4CLgc+19luBK9ry5W2d\ntv2fpNdplDSCA0NBPenQNqAFI9a4k5zEoBzyFuCPge8Dz1XVS22XQ8D2trwdeBKgql5K8jyDcspf\nb2C/NYN6/vf9wAQC20DWSkYK7qp6Gdid5DTgi8Bb13viJHsBf6hOgCElrcWa7pysqueAe4F3AKcl\nWQr+HcDhtnwY2AnQtv8Kgw8pj32tfVW1UFUr/yKmZsJW+9knqXerBneSM9tMmySvAd4NPMYgwN/b\ndrsKuKst72/rtO1fLf9mzjRDWdpco5RKtgG3tjr3LwF3VtWBJI8CdyT5T8C3gJvb/jcDf5bkIPAM\n8P4J9FubxOCVtp5Vg7uqHgLOW6b9r4ALlmn/v8A/25DeaVMYzlJfvHNyThnWUr/8Wtc5ZGhLfXPG\nPQcMamm2GNwzyrCWZpfBPUMMa2k+GNwdM6il+WRwd8awluRVJR0xtCWBM+4tzaCWtByDewsxqCWN\nYkuUSs5vvyAyr/wyJklrsWVm3MPB1fMX6o/KoJY0ri0T3MNmNcQNa0kbYUuUSk5kVsJuVsYhafq2\n5Iz7WEuh19vs27CWNAldBPeSXkooBrakSeoquIdttRA3rCVtlm6De9i0QtywljQNo/xY8KuT3J/k\n20keSfLx1v6ZJD9I8mB77G7tSfKpJAeTPJTkbZMexLDNuCba664lTdMoM+4XgYur6oUkpwD/J8n/\nbNv+TVV97pj93wPsao+3Aze15001qQ80DWxJ07bqjLsGXmirp7THidLrcuC2dtzXgdOSbFt/V8ez\nNDteT+BuxGtI0kYZ6TruJCcleRA4CtxdVfe1TX/QyiE3Jjm1tW0Hnhw6/FBrm7q1hK9hLWmrGim4\nq+rlqtoN7AAuSPIbwPXAW4F/BJwB/Lu1nDjJ3iSLSRZ//OMfr7Hb67NSKBvWknqwpjsnq+o54F7g\n0qo60sohLwL/Fbig7XYY2Dl02I7Wduxr7auqhapaOPPMM8fr/QYwrCX1ZpSrSs5Mclpbfg3wbuC7\nS3XrDD79uwJ4uB2yH/hAu7rkQuD5qjoykd5L0hwa5aqSbcCtSU5iEPR3VtWBJF9NciYQ4EHgX7f9\n/wdwGXAQ+BnwwY3vtiTNr1WDu6oeAs5bpv3iFfYv4Jr1d02StJwt/+2AkqRfZHBLUmcMbknqjMEt\nSZ0xuCWpMwa3JHXG4JakzhjcktQZg1uSOmNwS1JnDG5J6ozBLUmdMbglqTMGtyR1xuCWpM4Y3JLU\nGYNbkjpjcEtSZwxuSeqMwS1JnTG4JakzBrckdSZVNe0+kOSnwOPT7seEvAH462l3YgJmdVwwu2Nz\nXH35+1V15nIbTt7snqzg8apamHYnJiHJ4iyObVbHBbM7Nsc1OyyVSFJnDG5J6sxWCe590+7ABM3q\n2GZ1XDC7Y3NcM2JLfDgpSRrdVplxS5JGNPXgTnJpkseTHExy3bT7s1ZJbklyNMnDQ21nJLk7yffa\n8+mtPUk+1cb6UJK3Ta/nJ5ZkZ5J7kzya5JEkH27tXY8tyauT3J/k221cH2/tb0pyX+v/XyR5VWs/\nta0fbNvPmWb/V5PkpCTfSnKgrc/KuJ5I8p0kDyZZbG1dvxfXY6rBneQk4I+B9wDnAlcmOXeafRrD\nZ4BLj2m7DrinqnYB97R1GIxzV3vsBW7apD6O4yXgo1V1LnAhcE37b9P72F4ELq6q3wR2A5cmuRD4\nQ+DGqnoL8Cxwddv/auDZ1n5j228r+zDw2ND6rIwL4F1VtXvo0r/e34vjq6qpPYB3AF8aWr8euH6a\nfRpzHOcADw+tPw5sa8vbGFynDvAnwJXL7bfVH8BdwLtnaWzA3wO+CbydwQ0cJ7f2V96XwJeAd7Tl\nk9t+mXbfVxjPDgYBdjFwAMgsjKv18QngDce0zcx7ca2PaZdKtgNPDq0fam29O6uqjrTlHwFnteUu\nx9v+N/o84D5mYGytnPAgcBS4G/g+8FxVvdR2Ge77K+Nq258HXr+5PR7Zfwb+LfDztv56ZmNcAAV8\nOckDSfa2tu7fi+PaKndOzqyqqiTdXrqT5LXA54GPVNVPkryyrdexVdXLwO4kpwFfBN465S6tW5I9\nwNGqeiDJRdPuzwS8s6oOJ/lV4O4k3x3e2Ot7cVzTnnEfBnYOre9obb17Osk2gPZ8tLV3Nd4kpzAI\n7c9W1Rda80yMDaCqngPuZVBCOC3J0kRmuO+vjKtt/xXgbza5q6P4LeCfJnkCuINBueS/0P+4AKiq\nw+35KIN/bC9ght6LazXt4P4GsKt98v0q4P3A/in3aSPsB65qy1cxqA8vtX+gfep9IfD80P/qbSkZ\nTK1vBh6rqk8Obep6bEnObDNtkryGQd3+MQYB/t6227HjWhrve4GvViucbiVVdX1V7aiqcxj8Pfpq\nVf0LOh8XQJJfTvK6pWXgEuBhOn8vrsu0i+zAZcBfMqgz/vtp92eM/t8OHAH+H4Na2tUMaoX3AN8D\nvgKc0fYNg6tovg98B1iYdv9PMK53MqgrPgQ82B6X9T424B8C32rjehj4D639zcD9wEHgvwGntvZX\nt/WDbfubpz2GEcZ4EXBgVsbVxvDt9nhkKSd6fy+u5+Gdk5LUmWmXSiRJa2RwS1JnDG5J6ozBLUmd\nMbglqTMGtyR1xuCWpM4Y3JLUmf8PeYFBhumKn2kAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5paWqo7tWL2",
        "colab_type": "text"
      },
      "source": [
        "## Policy Gradient\n",
        "\n",
        "現在來搭建一個簡單的 policy network。\n",
        "我們預設模型的輸入是 8-dim 的 observation，輸出則是離散的四個動作之一："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J8tdmeD-tZew",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class PolicyGradientNetwork(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.fc1 = nn.Linear(8, 16)\n",
        "        self.fc2 = nn.Linear(16, 16)\n",
        "        self.fc3 = nn.Linear(16, 4)\n",
        "\n",
        "    def forward(self, state):\n",
        "        hid = torch.tanh(self.fc1(state))\n",
        "        hid = torch.tanh(self.fc2(hid))\n",
        "        return F.softmax(self.fc3(hid), dim=-1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ynbqJrhIFTC3",
        "colab_type": "text"
      },
      "source": [
        "再來，搭建一個簡單的 agent，並搭配上方的 policy network 來採取行動。\n",
        "這個 agent 能做到以下幾件事：\n",
        "- `learn()`：從記下來的 log probabilities 及 rewards 來更新 policy network。\n",
        "- `sample()`：從 environment 得到 observation 之後，利用 policy network 得出應該採取的行動。\n",
        "而此函式除了回傳抽樣出來的 action，也會回傳此次抽樣的 log probabilities。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zZo-IxJx286z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class PolicyGradientAgent():\n",
        "\n",
        "    def __init__(self, network):\n",
        "        self.network = network\n",
        "        self.optimizer = optim.SGD(self.network.parameters(), lr=0.001)\n",
        "\n",
        "    def learn(self, log_probs, rewards):\n",
        "        loss = (-log_probs * rewards).sum()\n",
        "\n",
        "        self.optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        self.optimizer.step()\n",
        "\n",
        "    def sample(self, state):\n",
        "        action_prob = self.network(torch.FloatTensor(state))\n",
        "        action_dist = Categorical(action_prob)\n",
        "        action = action_dist.sample()\n",
        "        log_prob = action_dist.log_prob(action)\n",
        "        return action.item(), log_prob"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ehPlnTKyRZf9",
        "colab_type": "text"
      },
      "source": [
        "最後，建立一個 network 和 agent，就可以開始進行訓練了。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GfJIvML-RYjL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "network = PolicyGradientNetwork()\n",
        "agent = PolicyGradientAgent(network)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ouv23glgf5Qt",
        "colab_type": "text"
      },
      "source": [
        "## 訓練 Agent\n",
        "\n",
        "現在我們開始訓練 agent。\n",
        "透過讓 agent 和 environment 互動，我們記住每一組對應的 log probabilities 及 reward，並在成功登陸或者不幸墜毀後，回放這些「記憶」來訓練 policy network。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vg5rxBBaf38_",
        "colab_type": "code",
        "outputId": "a2ac2226-a803-41c2-dcc2-9b338ad591d0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "agent.network.train()  # 訓練前，先確保 network 處在 training 模式\n",
        "EPISODE_PER_BATCH = 5  # 每蒐集 5 個 episodes 更新一次 agent\n",
        "NUM_BATCH = 400        # 總共更新 400 次\n",
        "\n",
        "avg_total_rewards, avg_final_rewards = [], []\n",
        "\n",
        "for batch in range(NUM_BATCH):\n",
        "\n",
        "    log_probs, rewards = [], []\n",
        "    total_rewards, final_rewards = [], []\n",
        "\n",
        "    # 蒐集訓練資料\n",
        "    for episode in range(EPISODE_PER_BATCH):\n",
        "        \n",
        "        state = env.reset()\n",
        "        total_reward, total_step = 0, 0\n",
        "\n",
        "        while True:\n",
        "\n",
        "            action, log_prob = agent.sample(state)\n",
        "            next_state, reward, done, _ = env.step(action)\n",
        "\n",
        "            log_probs.append(log_prob)\n",
        "            state = next_state\n",
        "            total_reward += reward\n",
        "            total_step += 1\n",
        "\n",
        "            if done:\n",
        "                final_rewards.append(reward)\n",
        "                total_rewards.append(total_reward)\n",
        "                rewards.append(np.full(total_step, total_reward))  # 設定同一個 episode 每個 action 的 reward 都是 total reward\n",
        "                break\n",
        "\n",
        "    # 紀錄訓練過程\n",
        "    avg_total_reward = sum(total_rewards) / len(total_rewards)\n",
        "    avg_final_reward = sum(final_rewards) / len(final_rewards)\n",
        "    avg_total_rewards.append(avg_total_reward)\n",
        "    avg_final_rewards.append(avg_final_reward)\n",
        "    print(\"Batch {},\\tTotal Reward = {:.1f},\\tFinal Reward = {:.1f}\".format(batch+1, avg_total_reward, avg_final_reward))\n",
        "\n",
        "    # 更新網路\n",
        "    rewards = np.concatenate(rewards, axis=0)\n",
        "    rewards = (rewards - np.mean(rewards)) / (np.std(rewards) + 1e-9)  # 將 reward 正規標準化\n",
        "    agent.learn(torch.stack(log_probs), torch.from_numpy(rewards))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Batch 1,\tTotal Reward = -198.1,\tFinal Reward = -100.0\n",
            "Batch 2,\tTotal Reward = -192.3,\tFinal Reward = -100.0\n",
            "Batch 3,\tTotal Reward = -166.5,\tFinal Reward = -100.0\n",
            "Batch 4,\tTotal Reward = -174.3,\tFinal Reward = -100.0\n",
            "Batch 5,\tTotal Reward = -230.8,\tFinal Reward = -100.0\n",
            "Batch 6,\tTotal Reward = -116.0,\tFinal Reward = -100.0\n",
            "Batch 7,\tTotal Reward = -98.2,\tFinal Reward = -100.0\n",
            "Batch 8,\tTotal Reward = -299.3,\tFinal Reward = -100.0\n",
            "Batch 9,\tTotal Reward = -267.0,\tFinal Reward = -100.0\n",
            "Batch 10,\tTotal Reward = -218.8,\tFinal Reward = -80.5\n",
            "Batch 11,\tTotal Reward = -147.4,\tFinal Reward = -100.0\n",
            "Batch 12,\tTotal Reward = -201.6,\tFinal Reward = -100.0\n",
            "Batch 13,\tTotal Reward = -217.0,\tFinal Reward = -100.0\n",
            "Batch 14,\tTotal Reward = -207.3,\tFinal Reward = -100.0\n",
            "Batch 15,\tTotal Reward = -124.5,\tFinal Reward = -100.0\n",
            "Batch 16,\tTotal Reward = -179.6,\tFinal Reward = -100.0\n",
            "Batch 17,\tTotal Reward = -294.4,\tFinal Reward = -100.0\n",
            "Batch 18,\tTotal Reward = -154.1,\tFinal Reward = -100.0\n",
            "Batch 19,\tTotal Reward = -267.8,\tFinal Reward = -100.0\n",
            "Batch 20,\tTotal Reward = -179.1,\tFinal Reward = -100.0\n",
            "Batch 21,\tTotal Reward = -150.6,\tFinal Reward = -100.0\n",
            "Batch 22,\tTotal Reward = -135.9,\tFinal Reward = -100.0\n",
            "Batch 23,\tTotal Reward = -366.1,\tFinal Reward = -100.0\n",
            "Batch 24,\tTotal Reward = -190.9,\tFinal Reward = -100.0\n",
            "Batch 25,\tTotal Reward = -176.3,\tFinal Reward = -100.0\n",
            "Batch 26,\tTotal Reward = -147.8,\tFinal Reward = -100.0\n",
            "Batch 27,\tTotal Reward = -210.0,\tFinal Reward = -100.0\n",
            "Batch 28,\tTotal Reward = -239.0,\tFinal Reward = -100.0\n",
            "Batch 29,\tTotal Reward = -200.9,\tFinal Reward = -100.0\n",
            "Batch 30,\tTotal Reward = -261.4,\tFinal Reward = -100.0\n",
            "Batch 31,\tTotal Reward = -202.3,\tFinal Reward = -100.0\n",
            "Batch 32,\tTotal Reward = -218.8,\tFinal Reward = -100.0\n",
            "Batch 33,\tTotal Reward = -191.6,\tFinal Reward = -100.0\n",
            "Batch 34,\tTotal Reward = -189.6,\tFinal Reward = -100.0\n",
            "Batch 35,\tTotal Reward = -178.4,\tFinal Reward = -100.0\n",
            "Batch 36,\tTotal Reward = -287.2,\tFinal Reward = -100.0\n",
            "Batch 37,\tTotal Reward = -204.8,\tFinal Reward = -100.0\n",
            "Batch 38,\tTotal Reward = -122.7,\tFinal Reward = -100.0\n",
            "Batch 39,\tTotal Reward = -216.2,\tFinal Reward = -100.0\n",
            "Batch 40,\tTotal Reward = -178.6,\tFinal Reward = -100.0\n",
            "Batch 41,\tTotal Reward = -135.2,\tFinal Reward = -100.0\n",
            "Batch 42,\tTotal Reward = -192.2,\tFinal Reward = -100.0\n",
            "Batch 43,\tTotal Reward = -88.3,\tFinal Reward = -100.0\n",
            "Batch 44,\tTotal Reward = -188.3,\tFinal Reward = -100.0\n",
            "Batch 45,\tTotal Reward = -139.1,\tFinal Reward = -100.0\n",
            "Batch 46,\tTotal Reward = -160.6,\tFinal Reward = -100.0\n",
            "Batch 47,\tTotal Reward = -221.7,\tFinal Reward = -100.0\n",
            "Batch 48,\tTotal Reward = -156.1,\tFinal Reward = -100.0\n",
            "Batch 49,\tTotal Reward = -114.9,\tFinal Reward = -100.0\n",
            "Batch 50,\tTotal Reward = -281.6,\tFinal Reward = -100.0\n",
            "Batch 51,\tTotal Reward = -256.8,\tFinal Reward = -100.0\n",
            "Batch 52,\tTotal Reward = -183.5,\tFinal Reward = -100.0\n",
            "Batch 53,\tTotal Reward = -161.0,\tFinal Reward = -100.0\n",
            "Batch 54,\tTotal Reward = -131.1,\tFinal Reward = -100.0\n",
            "Batch 55,\tTotal Reward = -171.7,\tFinal Reward = -100.0\n",
            "Batch 56,\tTotal Reward = -185.6,\tFinal Reward = -100.0\n",
            "Batch 57,\tTotal Reward = -243.1,\tFinal Reward = -100.0\n",
            "Batch 58,\tTotal Reward = -195.2,\tFinal Reward = -100.0\n",
            "Batch 59,\tTotal Reward = -109.6,\tFinal Reward = -100.0\n",
            "Batch 60,\tTotal Reward = -191.2,\tFinal Reward = -100.0\n",
            "Batch 61,\tTotal Reward = -210.0,\tFinal Reward = -100.0\n",
            "Batch 62,\tTotal Reward = -158.5,\tFinal Reward = -100.0\n",
            "Batch 63,\tTotal Reward = -112.6,\tFinal Reward = -100.0\n",
            "Batch 64,\tTotal Reward = -171.8,\tFinal Reward = -100.0\n",
            "Batch 65,\tTotal Reward = -179.0,\tFinal Reward = -100.0\n",
            "Batch 66,\tTotal Reward = -245.1,\tFinal Reward = -100.0\n",
            "Batch 67,\tTotal Reward = -259.2,\tFinal Reward = -100.0\n",
            "Batch 68,\tTotal Reward = -175.7,\tFinal Reward = -100.0\n",
            "Batch 69,\tTotal Reward = -245.9,\tFinal Reward = -100.0\n",
            "Batch 70,\tTotal Reward = -104.2,\tFinal Reward = -100.0\n",
            "Batch 71,\tTotal Reward = -157.4,\tFinal Reward = -100.0\n",
            "Batch 72,\tTotal Reward = -123.7,\tFinal Reward = -100.0\n",
            "Batch 73,\tTotal Reward = -191.3,\tFinal Reward = -100.0\n",
            "Batch 74,\tTotal Reward = -193.0,\tFinal Reward = -100.0\n",
            "Batch 75,\tTotal Reward = -151.9,\tFinal Reward = -100.0\n",
            "Batch 76,\tTotal Reward = -169.6,\tFinal Reward = -100.0\n",
            "Batch 77,\tTotal Reward = -185.9,\tFinal Reward = -100.0\n",
            "Batch 78,\tTotal Reward = -160.6,\tFinal Reward = -100.0\n",
            "Batch 79,\tTotal Reward = -198.8,\tFinal Reward = -100.0\n",
            "Batch 80,\tTotal Reward = -227.8,\tFinal Reward = -100.0\n",
            "Batch 81,\tTotal Reward = -142.4,\tFinal Reward = -100.0\n",
            "Batch 82,\tTotal Reward = -175.3,\tFinal Reward = -100.0\n",
            "Batch 83,\tTotal Reward = -241.5,\tFinal Reward = -100.0\n",
            "Batch 84,\tTotal Reward = -124.6,\tFinal Reward = -100.0\n",
            "Batch 85,\tTotal Reward = -133.5,\tFinal Reward = -100.0\n",
            "Batch 86,\tTotal Reward = -177.2,\tFinal Reward = -100.0\n",
            "Batch 87,\tTotal Reward = -166.4,\tFinal Reward = -100.0\n",
            "Batch 88,\tTotal Reward = -168.2,\tFinal Reward = -100.0\n",
            "Batch 89,\tTotal Reward = -134.9,\tFinal Reward = -100.0\n",
            "Batch 90,\tTotal Reward = -148.4,\tFinal Reward = -100.0\n",
            "Batch 91,\tTotal Reward = -148.8,\tFinal Reward = -100.0\n",
            "Batch 92,\tTotal Reward = -129.5,\tFinal Reward = -100.0\n",
            "Batch 93,\tTotal Reward = -157.4,\tFinal Reward = -100.0\n",
            "Batch 94,\tTotal Reward = -143.4,\tFinal Reward = -100.0\n",
            "Batch 95,\tTotal Reward = -152.2,\tFinal Reward = -100.0\n",
            "Batch 96,\tTotal Reward = -222.6,\tFinal Reward = -100.0\n",
            "Batch 97,\tTotal Reward = -146.1,\tFinal Reward = -100.0\n",
            "Batch 98,\tTotal Reward = -184.0,\tFinal Reward = -100.0\n",
            "Batch 99,\tTotal Reward = -150.8,\tFinal Reward = -100.0\n",
            "Batch 100,\tTotal Reward = -114.0,\tFinal Reward = -100.0\n",
            "Batch 101,\tTotal Reward = -143.4,\tFinal Reward = -100.0\n",
            "Batch 102,\tTotal Reward = -168.0,\tFinal Reward = -100.0\n",
            "Batch 103,\tTotal Reward = -153.3,\tFinal Reward = -100.0\n",
            "Batch 104,\tTotal Reward = -138.0,\tFinal Reward = -100.0\n",
            "Batch 105,\tTotal Reward = -120.3,\tFinal Reward = -100.0\n",
            "Batch 106,\tTotal Reward = -159.0,\tFinal Reward = -100.0\n",
            "Batch 107,\tTotal Reward = -170.8,\tFinal Reward = -100.0\n",
            "Batch 108,\tTotal Reward = -105.6,\tFinal Reward = -100.0\n",
            "Batch 109,\tTotal Reward = -168.1,\tFinal Reward = -100.0\n",
            "Batch 110,\tTotal Reward = -120.3,\tFinal Reward = -100.0\n",
            "Batch 111,\tTotal Reward = -176.0,\tFinal Reward = -100.0\n",
            "Batch 112,\tTotal Reward = -248.9,\tFinal Reward = -100.0\n",
            "Batch 113,\tTotal Reward = -174.5,\tFinal Reward = -100.0\n",
            "Batch 114,\tTotal Reward = -152.3,\tFinal Reward = -100.0\n",
            "Batch 115,\tTotal Reward = -182.1,\tFinal Reward = -100.0\n",
            "Batch 116,\tTotal Reward = -215.4,\tFinal Reward = -100.0\n",
            "Batch 117,\tTotal Reward = -215.9,\tFinal Reward = -100.0\n",
            "Batch 118,\tTotal Reward = -152.9,\tFinal Reward = -100.0\n",
            "Batch 119,\tTotal Reward = -108.6,\tFinal Reward = -100.0\n",
            "Batch 120,\tTotal Reward = -123.4,\tFinal Reward = -100.0\n",
            "Batch 121,\tTotal Reward = -166.1,\tFinal Reward = -100.0\n",
            "Batch 122,\tTotal Reward = -179.7,\tFinal Reward = -100.0\n",
            "Batch 123,\tTotal Reward = -126.8,\tFinal Reward = -100.0\n",
            "Batch 124,\tTotal Reward = -122.2,\tFinal Reward = -100.0\n",
            "Batch 125,\tTotal Reward = -145.5,\tFinal Reward = -100.0\n",
            "Batch 126,\tTotal Reward = -114.0,\tFinal Reward = -100.0\n",
            "Batch 127,\tTotal Reward = -185.2,\tFinal Reward = -100.0\n",
            "Batch 128,\tTotal Reward = -130.8,\tFinal Reward = -100.0\n",
            "Batch 129,\tTotal Reward = -169.4,\tFinal Reward = -100.0\n",
            "Batch 130,\tTotal Reward = -95.4,\tFinal Reward = -100.0\n",
            "Batch 131,\tTotal Reward = -151.1,\tFinal Reward = -100.0\n",
            "Batch 132,\tTotal Reward = -133.9,\tFinal Reward = -100.0\n",
            "Batch 133,\tTotal Reward = -110.3,\tFinal Reward = -100.0\n",
            "Batch 134,\tTotal Reward = -160.5,\tFinal Reward = -100.0\n",
            "Batch 135,\tTotal Reward = -181.8,\tFinal Reward = -100.0\n",
            "Batch 136,\tTotal Reward = -159.2,\tFinal Reward = -100.0\n",
            "Batch 137,\tTotal Reward = -122.6,\tFinal Reward = -100.0\n",
            "Batch 138,\tTotal Reward = -175.5,\tFinal Reward = -100.0\n",
            "Batch 139,\tTotal Reward = -211.1,\tFinal Reward = -100.0\n",
            "Batch 140,\tTotal Reward = -109.2,\tFinal Reward = -100.0\n",
            "Batch 141,\tTotal Reward = -156.1,\tFinal Reward = -100.0\n",
            "Batch 142,\tTotal Reward = -118.7,\tFinal Reward = -100.0\n",
            "Batch 143,\tTotal Reward = -197.9,\tFinal Reward = -100.0\n",
            "Batch 144,\tTotal Reward = -145.1,\tFinal Reward = -100.0\n",
            "Batch 145,\tTotal Reward = -115.9,\tFinal Reward = -100.0\n",
            "Batch 146,\tTotal Reward = -103.9,\tFinal Reward = -100.0\n",
            "Batch 147,\tTotal Reward = -148.0,\tFinal Reward = -100.0\n",
            "Batch 148,\tTotal Reward = -174.0,\tFinal Reward = -100.0\n",
            "Batch 149,\tTotal Reward = -110.2,\tFinal Reward = -100.0\n",
            "Batch 150,\tTotal Reward = -165.5,\tFinal Reward = -100.0\n",
            "Batch 151,\tTotal Reward = -120.3,\tFinal Reward = -100.0\n",
            "Batch 152,\tTotal Reward = -175.1,\tFinal Reward = -100.0\n",
            "Batch 153,\tTotal Reward = -136.2,\tFinal Reward = -100.0\n",
            "Batch 154,\tTotal Reward = -117.4,\tFinal Reward = -100.0\n",
            "Batch 155,\tTotal Reward = -84.5,\tFinal Reward = -100.0\n",
            "Batch 156,\tTotal Reward = -186.3,\tFinal Reward = -100.0\n",
            "Batch 157,\tTotal Reward = -144.6,\tFinal Reward = -100.0\n",
            "Batch 158,\tTotal Reward = -202.3,\tFinal Reward = -100.0\n",
            "Batch 159,\tTotal Reward = -170.1,\tFinal Reward = -100.0\n",
            "Batch 160,\tTotal Reward = -124.9,\tFinal Reward = -100.0\n",
            "Batch 161,\tTotal Reward = -100.7,\tFinal Reward = -100.0\n",
            "Batch 162,\tTotal Reward = -175.7,\tFinal Reward = -100.0\n",
            "Batch 163,\tTotal Reward = -183.4,\tFinal Reward = -100.0\n",
            "Batch 164,\tTotal Reward = -136.1,\tFinal Reward = -100.0\n",
            "Batch 165,\tTotal Reward = -135.9,\tFinal Reward = -100.0\n",
            "Batch 166,\tTotal Reward = -124.1,\tFinal Reward = -100.0\n",
            "Batch 167,\tTotal Reward = -171.8,\tFinal Reward = -100.0\n",
            "Batch 168,\tTotal Reward = -202.8,\tFinal Reward = -100.0\n",
            "Batch 169,\tTotal Reward = -117.8,\tFinal Reward = -100.0\n",
            "Batch 170,\tTotal Reward = -121.6,\tFinal Reward = -100.0\n",
            "Batch 171,\tTotal Reward = -115.1,\tFinal Reward = -100.0\n",
            "Batch 172,\tTotal Reward = -182.4,\tFinal Reward = -100.0\n",
            "Batch 173,\tTotal Reward = -148.4,\tFinal Reward = -100.0\n",
            "Batch 174,\tTotal Reward = -118.4,\tFinal Reward = -100.0\n",
            "Batch 175,\tTotal Reward = -141.7,\tFinal Reward = -100.0\n",
            "Batch 176,\tTotal Reward = -102.1,\tFinal Reward = -100.0\n",
            "Batch 177,\tTotal Reward = -154.0,\tFinal Reward = -100.0\n",
            "Batch 178,\tTotal Reward = -152.5,\tFinal Reward = -100.0\n",
            "Batch 179,\tTotal Reward = -150.4,\tFinal Reward = -100.0\n",
            "Batch 180,\tTotal Reward = -160.5,\tFinal Reward = -100.0\n",
            "Batch 181,\tTotal Reward = -146.6,\tFinal Reward = -100.0\n",
            "Batch 182,\tTotal Reward = -171.6,\tFinal Reward = -100.0\n",
            "Batch 183,\tTotal Reward = -156.8,\tFinal Reward = -100.0\n",
            "Batch 184,\tTotal Reward = -111.7,\tFinal Reward = -100.0\n",
            "Batch 185,\tTotal Reward = -128.3,\tFinal Reward = -100.0\n",
            "Batch 186,\tTotal Reward = -130.2,\tFinal Reward = -100.0\n",
            "Batch 187,\tTotal Reward = -157.3,\tFinal Reward = -100.0\n",
            "Batch 188,\tTotal Reward = -129.2,\tFinal Reward = -100.0\n",
            "Batch 189,\tTotal Reward = -125.0,\tFinal Reward = -100.0\n",
            "Batch 190,\tTotal Reward = -105.5,\tFinal Reward = -100.0\n",
            "Batch 191,\tTotal Reward = -159.5,\tFinal Reward = -100.0\n",
            "Batch 192,\tTotal Reward = -171.1,\tFinal Reward = -100.0\n",
            "Batch 193,\tTotal Reward = -141.5,\tFinal Reward = -100.0\n",
            "Batch 194,\tTotal Reward = -184.1,\tFinal Reward = -100.0\n",
            "Batch 195,\tTotal Reward = -170.7,\tFinal Reward = -100.0\n",
            "Batch 196,\tTotal Reward = -143.3,\tFinal Reward = -100.0\n",
            "Batch 197,\tTotal Reward = -125.3,\tFinal Reward = -100.0\n",
            "Batch 198,\tTotal Reward = -110.3,\tFinal Reward = -100.0\n",
            "Batch 199,\tTotal Reward = -110.9,\tFinal Reward = -100.0\n",
            "Batch 200,\tTotal Reward = -110.5,\tFinal Reward = -100.0\n",
            "Batch 201,\tTotal Reward = -103.3,\tFinal Reward = -100.0\n",
            "Batch 202,\tTotal Reward = -131.1,\tFinal Reward = -100.0\n",
            "Batch 203,\tTotal Reward = -144.6,\tFinal Reward = -100.0\n",
            "Batch 204,\tTotal Reward = -195.1,\tFinal Reward = -100.0\n",
            "Batch 205,\tTotal Reward = -130.5,\tFinal Reward = -100.0\n",
            "Batch 206,\tTotal Reward = -130.7,\tFinal Reward = -100.0\n",
            "Batch 207,\tTotal Reward = -127.7,\tFinal Reward = -100.0\n",
            "Batch 208,\tTotal Reward = -124.4,\tFinal Reward = -100.0\n",
            "Batch 209,\tTotal Reward = -95.8,\tFinal Reward = -100.0\n",
            "Batch 210,\tTotal Reward = -138.3,\tFinal Reward = -100.0\n",
            "Batch 211,\tTotal Reward = -128.8,\tFinal Reward = -100.0\n",
            "Batch 212,\tTotal Reward = -120.1,\tFinal Reward = -100.0\n",
            "Batch 213,\tTotal Reward = -96.4,\tFinal Reward = -100.0\n",
            "Batch 214,\tTotal Reward = -114.8,\tFinal Reward = -100.0\n",
            "Batch 215,\tTotal Reward = -151.4,\tFinal Reward = -100.0\n",
            "Batch 216,\tTotal Reward = -110.7,\tFinal Reward = -100.0\n",
            "Batch 217,\tTotal Reward = -123.5,\tFinal Reward = -100.0\n",
            "Batch 218,\tTotal Reward = -108.5,\tFinal Reward = -100.0\n",
            "Batch 219,\tTotal Reward = -90.9,\tFinal Reward = -100.0\n",
            "Batch 220,\tTotal Reward = -155.3,\tFinal Reward = -100.0\n",
            "Batch 221,\tTotal Reward = -107.0,\tFinal Reward = -100.0\n",
            "Batch 222,\tTotal Reward = -102.8,\tFinal Reward = -100.0\n",
            "Batch 223,\tTotal Reward = -119.0,\tFinal Reward = -100.0\n",
            "Batch 224,\tTotal Reward = -113.7,\tFinal Reward = -100.0\n",
            "Batch 225,\tTotal Reward = -182.6,\tFinal Reward = -100.0\n",
            "Batch 226,\tTotal Reward = -173.6,\tFinal Reward = -100.0\n",
            "Batch 227,\tTotal Reward = -166.5,\tFinal Reward = -100.0\n",
            "Batch 228,\tTotal Reward = -110.9,\tFinal Reward = -100.0\n",
            "Batch 229,\tTotal Reward = -103.9,\tFinal Reward = -100.0\n",
            "Batch 230,\tTotal Reward = -128.9,\tFinal Reward = -100.0\n",
            "Batch 231,\tTotal Reward = -97.9,\tFinal Reward = -100.0\n",
            "Batch 232,\tTotal Reward = -122.3,\tFinal Reward = -100.0\n",
            "Batch 233,\tTotal Reward = -93.5,\tFinal Reward = -100.0\n",
            "Batch 234,\tTotal Reward = -127.6,\tFinal Reward = -100.0\n",
            "Batch 235,\tTotal Reward = -130.5,\tFinal Reward = -100.0\n",
            "Batch 236,\tTotal Reward = -121.7,\tFinal Reward = -100.0\n",
            "Batch 237,\tTotal Reward = -94.6,\tFinal Reward = -100.0\n",
            "Batch 238,\tTotal Reward = -109.2,\tFinal Reward = -100.0\n",
            "Batch 239,\tTotal Reward = -96.6,\tFinal Reward = -100.0\n",
            "Batch 240,\tTotal Reward = -149.1,\tFinal Reward = -100.0\n",
            "Batch 241,\tTotal Reward = -132.3,\tFinal Reward = -100.0\n",
            "Batch 242,\tTotal Reward = -104.5,\tFinal Reward = -100.0\n",
            "Batch 243,\tTotal Reward = -152.5,\tFinal Reward = -100.0\n",
            "Batch 244,\tTotal Reward = -128.8,\tFinal Reward = -100.0\n",
            "Batch 245,\tTotal Reward = -110.3,\tFinal Reward = -100.0\n",
            "Batch 246,\tTotal Reward = -95.2,\tFinal Reward = -100.0\n",
            "Batch 247,\tTotal Reward = -146.5,\tFinal Reward = -100.0\n",
            "Batch 248,\tTotal Reward = -117.2,\tFinal Reward = -100.0\n",
            "Batch 249,\tTotal Reward = -104.5,\tFinal Reward = -100.0\n",
            "Batch 250,\tTotal Reward = -96.0,\tFinal Reward = -100.0\n",
            "Batch 251,\tTotal Reward = -78.5,\tFinal Reward = -100.0\n",
            "Batch 252,\tTotal Reward = -125.5,\tFinal Reward = -100.0\n",
            "Batch 253,\tTotal Reward = -132.7,\tFinal Reward = -100.0\n",
            "Batch 254,\tTotal Reward = -101.5,\tFinal Reward = -100.0\n",
            "Batch 255,\tTotal Reward = -139.4,\tFinal Reward = -100.0\n",
            "Batch 256,\tTotal Reward = -106.9,\tFinal Reward = -100.0\n",
            "Batch 257,\tTotal Reward = -182.4,\tFinal Reward = -100.0\n",
            "Batch 258,\tTotal Reward = -115.3,\tFinal Reward = -100.0\n",
            "Batch 259,\tTotal Reward = -142.7,\tFinal Reward = -100.0\n",
            "Batch 260,\tTotal Reward = -188.1,\tFinal Reward = -100.0\n",
            "Batch 261,\tTotal Reward = -108.0,\tFinal Reward = -100.0\n",
            "Batch 262,\tTotal Reward = -150.4,\tFinal Reward = -100.0\n",
            "Batch 263,\tTotal Reward = -113.6,\tFinal Reward = -100.0\n",
            "Batch 264,\tTotal Reward = -127.7,\tFinal Reward = -100.0\n",
            "Batch 265,\tTotal Reward = -135.2,\tFinal Reward = -100.0\n",
            "Batch 266,\tTotal Reward = -102.1,\tFinal Reward = -100.0\n",
            "Batch 267,\tTotal Reward = -127.7,\tFinal Reward = -100.0\n",
            "Batch 268,\tTotal Reward = -171.2,\tFinal Reward = -100.0\n",
            "Batch 269,\tTotal Reward = -174.8,\tFinal Reward = -100.0\n",
            "Batch 270,\tTotal Reward = -112.3,\tFinal Reward = -100.0\n",
            "Batch 271,\tTotal Reward = -81.9,\tFinal Reward = -100.0\n",
            "Batch 272,\tTotal Reward = -156.7,\tFinal Reward = -100.0\n",
            "Batch 273,\tTotal Reward = -107.0,\tFinal Reward = -100.0\n",
            "Batch 274,\tTotal Reward = -76.8,\tFinal Reward = -100.0\n",
            "Batch 275,\tTotal Reward = -101.4,\tFinal Reward = -100.0\n",
            "Batch 276,\tTotal Reward = -118.2,\tFinal Reward = -100.0\n",
            "Batch 277,\tTotal Reward = -156.1,\tFinal Reward = -100.0\n",
            "Batch 278,\tTotal Reward = -117.8,\tFinal Reward = -100.0\n",
            "Batch 279,\tTotal Reward = -112.7,\tFinal Reward = -100.0\n",
            "Batch 280,\tTotal Reward = -134.4,\tFinal Reward = -100.0\n",
            "Batch 281,\tTotal Reward = -168.6,\tFinal Reward = -100.0\n",
            "Batch 282,\tTotal Reward = -145.7,\tFinal Reward = -100.0\n",
            "Batch 283,\tTotal Reward = -121.1,\tFinal Reward = -100.0\n",
            "Batch 284,\tTotal Reward = -113.2,\tFinal Reward = -100.0\n",
            "Batch 285,\tTotal Reward = -145.0,\tFinal Reward = -100.0\n",
            "Batch 286,\tTotal Reward = -150.2,\tFinal Reward = -100.0\n",
            "Batch 287,\tTotal Reward = -154.5,\tFinal Reward = -100.0\n",
            "Batch 288,\tTotal Reward = -164.4,\tFinal Reward = -100.0\n",
            "Batch 289,\tTotal Reward = -85.7,\tFinal Reward = -100.0\n",
            "Batch 290,\tTotal Reward = -93.0,\tFinal Reward = -100.0\n",
            "Batch 291,\tTotal Reward = -100.8,\tFinal Reward = -100.0\n",
            "Batch 292,\tTotal Reward = -100.3,\tFinal Reward = -100.0\n",
            "Batch 293,\tTotal Reward = -138.9,\tFinal Reward = -100.0\n",
            "Batch 294,\tTotal Reward = -96.5,\tFinal Reward = -100.0\n",
            "Batch 295,\tTotal Reward = -105.0,\tFinal Reward = -100.0\n",
            "Batch 296,\tTotal Reward = -98.7,\tFinal Reward = -100.0\n",
            "Batch 297,\tTotal Reward = -127.7,\tFinal Reward = -100.0\n",
            "Batch 298,\tTotal Reward = -110.6,\tFinal Reward = -100.0\n",
            "Batch 299,\tTotal Reward = -123.8,\tFinal Reward = -100.0\n",
            "Batch 300,\tTotal Reward = -137.9,\tFinal Reward = -100.0\n",
            "Batch 301,\tTotal Reward = -139.7,\tFinal Reward = -100.0\n",
            "Batch 302,\tTotal Reward = -110.1,\tFinal Reward = -100.0\n",
            "Batch 303,\tTotal Reward = -97.6,\tFinal Reward = -100.0\n",
            "Batch 304,\tTotal Reward = -149.9,\tFinal Reward = -100.0\n",
            "Batch 305,\tTotal Reward = -114.2,\tFinal Reward = -100.0\n",
            "Batch 306,\tTotal Reward = -64.1,\tFinal Reward = -100.0\n",
            "Batch 307,\tTotal Reward = -105.0,\tFinal Reward = -100.0\n",
            "Batch 308,\tTotal Reward = -180.7,\tFinal Reward = -100.0\n",
            "Batch 309,\tTotal Reward = -130.7,\tFinal Reward = -100.0\n",
            "Batch 310,\tTotal Reward = -159.2,\tFinal Reward = -100.0\n",
            "Batch 311,\tTotal Reward = -110.1,\tFinal Reward = -100.0\n",
            "Batch 312,\tTotal Reward = -110.6,\tFinal Reward = -100.0\n",
            "Batch 313,\tTotal Reward = -141.9,\tFinal Reward = -100.0\n",
            "Batch 314,\tTotal Reward = -146.5,\tFinal Reward = -100.0\n",
            "Batch 315,\tTotal Reward = -91.3,\tFinal Reward = -100.0\n",
            "Batch 316,\tTotal Reward = -180.3,\tFinal Reward = -100.0\n",
            "Batch 317,\tTotal Reward = -108.7,\tFinal Reward = -100.0\n",
            "Batch 318,\tTotal Reward = -133.8,\tFinal Reward = -100.0\n",
            "Batch 319,\tTotal Reward = -129.7,\tFinal Reward = -100.0\n",
            "Batch 320,\tTotal Reward = -122.9,\tFinal Reward = -100.0\n",
            "Batch 321,\tTotal Reward = -111.6,\tFinal Reward = -100.0\n",
            "Batch 322,\tTotal Reward = -124.3,\tFinal Reward = -100.0\n",
            "Batch 323,\tTotal Reward = -97.8,\tFinal Reward = -100.0\n",
            "Batch 324,\tTotal Reward = -124.2,\tFinal Reward = -100.0\n",
            "Batch 325,\tTotal Reward = -128.1,\tFinal Reward = -100.0\n",
            "Batch 326,\tTotal Reward = -97.9,\tFinal Reward = -100.0\n",
            "Batch 327,\tTotal Reward = -129.6,\tFinal Reward = -100.0\n",
            "Batch 328,\tTotal Reward = -151.0,\tFinal Reward = -100.0\n",
            "Batch 329,\tTotal Reward = -108.1,\tFinal Reward = -100.0\n",
            "Batch 330,\tTotal Reward = -84.5,\tFinal Reward = -100.0\n",
            "Batch 331,\tTotal Reward = -111.1,\tFinal Reward = -100.0\n",
            "Batch 332,\tTotal Reward = -121.2,\tFinal Reward = -100.0\n",
            "Batch 333,\tTotal Reward = -110.6,\tFinal Reward = -100.0\n",
            "Batch 334,\tTotal Reward = -110.9,\tFinal Reward = -100.0\n",
            "Batch 335,\tTotal Reward = -127.0,\tFinal Reward = -100.0\n",
            "Batch 336,\tTotal Reward = -79.1,\tFinal Reward = -100.0\n",
            "Batch 337,\tTotal Reward = -114.4,\tFinal Reward = -100.0\n",
            "Batch 338,\tTotal Reward = -103.8,\tFinal Reward = -100.0\n",
            "Batch 339,\tTotal Reward = -104.1,\tFinal Reward = -100.0\n",
            "Batch 340,\tTotal Reward = -112.8,\tFinal Reward = -100.0\n",
            "Batch 341,\tTotal Reward = -100.8,\tFinal Reward = -100.0\n",
            "Batch 342,\tTotal Reward = -81.4,\tFinal Reward = -100.0\n",
            "Batch 343,\tTotal Reward = -124.8,\tFinal Reward = -100.0\n",
            "Batch 344,\tTotal Reward = -108.9,\tFinal Reward = -100.0\n",
            "Batch 345,\tTotal Reward = -105.8,\tFinal Reward = -100.0\n",
            "Batch 346,\tTotal Reward = -106.5,\tFinal Reward = -100.0\n",
            "Batch 347,\tTotal Reward = -112.4,\tFinal Reward = -100.0\n",
            "Batch 348,\tTotal Reward = -125.1,\tFinal Reward = -100.0\n",
            "Batch 349,\tTotal Reward = -95.9,\tFinal Reward = -100.0\n",
            "Batch 350,\tTotal Reward = -89.7,\tFinal Reward = -100.0\n",
            "Batch 351,\tTotal Reward = -97.5,\tFinal Reward = -100.0\n",
            "Batch 352,\tTotal Reward = -114.2,\tFinal Reward = -100.0\n",
            "Batch 353,\tTotal Reward = -129.7,\tFinal Reward = -100.0\n",
            "Batch 354,\tTotal Reward = -103.3,\tFinal Reward = -100.0\n",
            "Batch 355,\tTotal Reward = -117.6,\tFinal Reward = -100.0\n",
            "Batch 356,\tTotal Reward = -126.2,\tFinal Reward = -100.0\n",
            "Batch 357,\tTotal Reward = -125.6,\tFinal Reward = -100.0\n",
            "Batch 358,\tTotal Reward = -86.3,\tFinal Reward = -100.0\n",
            "Batch 359,\tTotal Reward = -91.2,\tFinal Reward = -100.0\n",
            "Batch 360,\tTotal Reward = -77.5,\tFinal Reward = -100.0\n",
            "Batch 361,\tTotal Reward = -108.3,\tFinal Reward = -100.0\n",
            "Batch 362,\tTotal Reward = -147.5,\tFinal Reward = -100.0\n",
            "Batch 363,\tTotal Reward = -108.7,\tFinal Reward = -100.0\n",
            "Batch 364,\tTotal Reward = -110.0,\tFinal Reward = -100.0\n",
            "Batch 365,\tTotal Reward = -133.0,\tFinal Reward = -100.0\n",
            "Batch 366,\tTotal Reward = -64.1,\tFinal Reward = -100.0\n",
            "Batch 367,\tTotal Reward = -135.9,\tFinal Reward = -100.0\n",
            "Batch 368,\tTotal Reward = -120.9,\tFinal Reward = -100.0\n",
            "Batch 369,\tTotal Reward = -104.7,\tFinal Reward = -100.0\n",
            "Batch 370,\tTotal Reward = -155.9,\tFinal Reward = -100.0\n",
            "Batch 371,\tTotal Reward = -122.9,\tFinal Reward = -100.0\n",
            "Batch 372,\tTotal Reward = -116.6,\tFinal Reward = -100.0\n",
            "Batch 373,\tTotal Reward = -100.7,\tFinal Reward = -100.0\n",
            "Batch 374,\tTotal Reward = -131.9,\tFinal Reward = -100.0\n",
            "Batch 375,\tTotal Reward = -96.5,\tFinal Reward = -100.0\n",
            "Batch 376,\tTotal Reward = -70.6,\tFinal Reward = -100.0\n",
            "Batch 377,\tTotal Reward = -163.0,\tFinal Reward = -100.0\n",
            "Batch 378,\tTotal Reward = -115.1,\tFinal Reward = -100.0\n",
            "Batch 379,\tTotal Reward = -137.8,\tFinal Reward = -100.0\n",
            "Batch 380,\tTotal Reward = -93.5,\tFinal Reward = -100.0\n",
            "Batch 381,\tTotal Reward = -79.1,\tFinal Reward = -100.0\n",
            "Batch 382,\tTotal Reward = -80.8,\tFinal Reward = -100.0\n",
            "Batch 383,\tTotal Reward = -105.6,\tFinal Reward = -100.0\n",
            "Batch 384,\tTotal Reward = -117.4,\tFinal Reward = -100.0\n",
            "Batch 385,\tTotal Reward = -139.5,\tFinal Reward = -100.0\n",
            "Batch 386,\tTotal Reward = -87.3,\tFinal Reward = -100.0\n",
            "Batch 387,\tTotal Reward = -95.8,\tFinal Reward = -100.0\n",
            "Batch 388,\tTotal Reward = -121.0,\tFinal Reward = -100.0\n",
            "Batch 389,\tTotal Reward = -110.1,\tFinal Reward = -100.0\n",
            "Batch 390,\tTotal Reward = -128.2,\tFinal Reward = -79.6\n",
            "Batch 391,\tTotal Reward = -96.8,\tFinal Reward = -100.0\n",
            "Batch 392,\tTotal Reward = -93.1,\tFinal Reward = -100.0\n",
            "Batch 393,\tTotal Reward = -118.6,\tFinal Reward = -100.0\n",
            "Batch 394,\tTotal Reward = -121.0,\tFinal Reward = -100.0\n",
            "Batch 395,\tTotal Reward = -79.9,\tFinal Reward = -100.0\n",
            "Batch 396,\tTotal Reward = -132.6,\tFinal Reward = -100.0\n",
            "Batch 397,\tTotal Reward = -126.3,\tFinal Reward = -100.0\n",
            "Batch 398,\tTotal Reward = -97.3,\tFinal Reward = -100.0\n",
            "Batch 399,\tTotal Reward = -105.9,\tFinal Reward = -100.0\n",
            "Batch 400,\tTotal Reward = -88.1,\tFinal Reward = -100.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vNb_tuFYhKVK",
        "colab_type": "text"
      },
      "source": [
        "### 訓練結果\n",
        "\n",
        "訓練過程中，我們持續記下了 `avg_total_reward`，這個數值代表的是：每次更新 policy network 前，我們讓 agent 玩數個回合（episodes），而這些回合的平均 total rewards 為何。\n",
        "理論上，若是 agent 一直在進步，則所得到的 `avg_total_reward` 也會持續上升，直至 250 上下。\n",
        "若將其畫出來則結果如下："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wZYOI8H10SHN",
        "colab_type": "code",
        "outputId": "1f1cae34-df0c-4bd5-9d89-c5ed57f4a603",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        }
      },
      "source": [
        "plt.plot(avg_total_rewards)\n",
        "plt.title(\"Total Rewards\")\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOx9d7gdVbn++83scmr6CYH0QAISOqEp\nTUSIIqLeq2LvyhXsXn+oV0W5lisqFiwXu1cQUURQUHqXAIEEElpIAgkppJwkJzltl5n1+2Pmm1mz\nZs3es885+7S93ufJk7NnZs+sPXvPu771fo2EEDAwMDAwaCxYIz0AAwMDA4PhhyF/AwMDgwaEIX8D\nAwODBoQhfwMDA4MGhCF/AwMDgwaEIX8DAwODBoQhf4OGBxE1EZEgolkjPZbBgIi+RUS/GOlxGIwN\nGPI3GJUgom7pn0tEfdLrd1R571IiWjuEY1lGRP3+tXcQ0bVE1DFU5zcwGAkY8jcYlRBCtPE/ABsB\nnCttu2oEhvRBfywHA5gO4FsjMAYAABFZRGSeXYNBwfyADMYkiKiZiH5MRFuJaBMRXUZEWSKaCuB6\nAAuklcJUInoFET1ERHuIaAsRXU5EmVqvK4TYBeBGAEdJY7GJ6EtEtJ6IdhLRVUQ0yd/3RyK60P/7\nQF9e+oD/ejERveT/3UFE//BXFruI6AYi2l+6xjIi+hoRPQSgF8ABRHQQET1ARPuI6B8AJkvHtxLR\nNf659vifPdhvYGDI32Cs4qsAjgBwOIBjAZwO4HNCiE4AbwSwXlopdAIoAbgIwFQApwA4F8AHa72o\nL/e8AYAsK30WwFkATgYwy7/W5f6+e/yxAcBpANYDOFV6fY//twXgZwDmAJjvb+NzMN4J4N0A2gFs\nA/AnAPf6n+k7AN4lHftBABkAMwFMg/fZi7V9WoPxDEP+BmMV7wDwFSHETiHENgD/jSj5RSCEeFgI\n8YgQwhFCrAPwC3jkmxb/S0R7AWwH0AzgU9K+CwBcLITYIoTohzcxvZWICB65M9mfCk8uOsV/HZC/\nEGKbEOIGIUSfEKILwDc14/uFEOJZIUQJ3gTxMgBfFUIUhRB3APindGwJQAeAA4UQZf+z99TweQ3G\nOQz5G4w5+KQ6A8AGafMGeFZu0nsO9WWVbT6JfxmeRZwWHxFCTABwjH/tA6SxzAZwsy+v7AGwAt6z\nNRXAUwAsIjoU3srgegDdRDQP3mRwj3+ediL6FRFt9Md3q2Z8L0p/HwBghz/ZyPeA8Uv/3H/2ZbFv\nEJFdw+c1GOcw5G8w5iC8UrQvAZgrbZ4DYDMfonnbzwE8Bs8SngDgawBoANdeAeDbAH4kjWUzgDOE\nEJOkf03+qkTAk2beCaBfCLETHil/GJ4s85R/6ovhSUbH+eM7SzM++XNtBTCNiJqUe8DjLAghviyE\nOATeJPNmAOfX+nkNxi8M+RuMVfwBwFd8Z+50AF8E8Ht/3zYA04moTTq+HUCXEKKbiBYD+NAgrv0L\nAAcR0dn+658B+BYRzQYAIppOROdKx98DT3Nnff9u//W9Iqyp3g7PkbuHiKYB+K8qY1gD4FkAXyKi\nHBG9EsBS3klEZ/qrHQvAXgBlAO6APq3BuIQhf4Oxii/Ds5qfBLASwAPwLHIAeBxeRM4GX4qZAk+j\n/yARdQP4MYA/DvTCQog+AFcA+JK/6dsAbgdwJxHtA/AvePIQ4x545H6v//peAG3Sa8Bz2E4D0Ang\nfgA3VxmDAPAWAK8EsAvA5xBOfoAngd0AYB+A1f75BvyZDcYfyDRzMTAwMGg8GMvfwMDAoAFhyN/A\nwMCgAVE38ieiS4hoMxGt9P+9Vtr3eSJaS0TPSk4zAwMDA4NhQs3p7TXiciHEd+QNfrzz+QAWw4tV\nvp2IFgkhnDqPxcDAwMDAR73JX4fzAFwjhCgAeN6vvng8gAcrvWnatGli3rx5wzA8AwMDg/GBRx99\ndKcQQluBtt7kfxERvRvAcgCfEULshheCtkw6ZhMSMjOJ6MPwkmEwZ84cLF++vM7DNTAwMBg/IKIN\nSfsGpfkT0e1EtFrz7zwAPwVwILzqh1sBfLfW8wshrhRCLBFCLOnoMOXTDQwMDIYKg7L8hRBnpjmO\niH4O4O/+y83waqEwZiFMyzcwMDAwGAbUM9pnf+nlG+FlGQJe5uX5RJQnovkAFgJ4uF7jMDAwMDCI\no56a/7eJ6Ch4xaheAPARABBCPElE18JLzS8DuNBE+hgYGBgML+pG/kKISrXVvw7g6/W6toGBgYFB\nZZgMXwMDA4MGhCF/AwMDgwaEIX8DA4MRQ7Hs4trlL2KsVRfesqcPdz6zbaSHMSgY8jcwMBgxXHHX\nWnzuz0/g709sHemh1ITXX3E/3v+bsZ10asjfwMBgxLCzuwAA2NNXGuGR1Iad3cWRHsKgYcjfwMBg\nxBA0KR5jsg/DdcfmuAFD/gYGBiMIizz6H6scWk4Y+Iu7etE1ylczhvwNDAxGDD73jzmHL8NJIP9T\nvn0XXn/F/cM8mtpgyN/AwGDEwLLP2KR+oOy6ifs2dPYO40hqhyF/AwODEQONcdlHZ/mPlVWMIX8D\nA4NEdPWVsKunfpEtY132KTnxcfeXklcDowmG/A0MDBKx5L9vwzGX3la381MY7zNqUSy7+MQ1K7B2\n+77YPp3l31MsD8ewBg1D/gYG4xDLX9iFeRffhMdf3DOo8+gs26EEW/71uM6Lu3rxrX88M+hVxWMb\nd+OGlVvwhb+sju3Taf69hWiRYtcV+OrfnsTTW/dWvZbrirqutGQY8jcwGIe45cmXAAAPru8c4ZFU\nBtv9hfLQV3X/6FWP4Wf3rMOabd3Btq7eEnprtMz5+KacHduns/x7S9Hzb97Th18/8AJe84P7qn7O\nH9zxHI659Dbs2FeoaYwDgSF/A4NxiN6iRzKtGsKqBiEEzvju3bh+xaahHlYMjm+VF8tDr5PzOWWC\nPvJrt+LV37u3pvP0FJLvpS7On4/PZzx67ZQs+eekiUiHf672Jm3OfK4nDPkbjDk4rsC+/tGdQDPS\n6Ct5BNScq71lR8kRWL+jB5+59vFgW70csiXHI+ihIP89vcXgfABgWRxJFB375j19tZ3XT9Zqzcfv\npc7y7ytGyX9XT0jkPCknQfhBr5z8Vk8Y8jcYc/ivv67C4ZfcmphgYxASUNaunUR44sjYIT0UUpCz\nEALL1nfWNFGUfa0/zfmrXfuor92G//xTOGH53B8j/1rR6VvhzVmN5a/xVbDDN+8f3ynVAarmDOaf\n9DBwvyF/g7GHPzz8IoDKCTaNjh6f/HXkVA39PvnnJPKvZrECwPINu3H+lcvw8PO7Ul+rOESWP4dX\n/nXllmCb7bM/TyzyqiAtLrnxSXz/9ucA6H9vum2q5b+7NyT/3oIDIQSWv6C/RzxRDYdhY8i/zvj1\nA8/j0Q3pHwaD9DDcn4x+n4AGQni6VUNPobqTtNs/5tlt8ZDIJHCUT3EA49RdWwYnkBX8iWFff+0h\nmL/51wvB332aCVCr+bPlr9H8e4pl/HXlZvz7zx7EDSs3xy/on46/tx/d8Rze/vNlNY87DQz51xlf\n/dtT+LefPjjSwxiXcMZoYtBwgCNOBkT+vuVPkvaQxvJ3fCJfv6Mn9bVK5aGx/HWTE8s+HGGzt0Kh\ntase2oDfSkTPmNySDf7m+yKjkubf5Ms+u7qLwUTQWyhjd483juUv7I69ly1/nhTX7+zBxl31KRNh\nyN9gzGIsa/6f+/PjOO/HD9Tt/ExAA9HSWfYpSxNHmsQlnozX7agc0SKDJyddCOT6Hd345j+eTuVD\n0I2Pnab9KSz/L16/Gl+58cnY9kX7tQd/6zJ3tZp/IS77zJ7S4o/TQUd7HoDXDUyFq1j+e/tKmNic\njR03FKg9FMDAYJRgLNdSv3Z5+jDKTbt7Ma0tH1iSadAXyD613yO2cOXJVU1c0oGPX7c9PfkXA/KP\nE+v7f/MIXujsxTtPmBuQZxKYcC3JUWqz7MOWf0KEmCwZFcsuchm9TZzW8lfj/Dt7ipgxoQkv7OxB\nb7EcFLGTo44e27gbfUVHsvy9+9FVR/I3lr/BmMVgozjGAoQQOPl/7sJH/u9RAMDNq7ZibQK57thX\nCMiotzRwzT+w/CViS2P58/FbuvpTJ1JVCvXkbZZVPfSFZR9ZquI/Q8tfT/7PvhRm3r64OyqxFB0X\npyychtMP7gjui4xKGb78XXT1ljCpJYuWnI2eghNIXVu7+oP3vOkn/8I7fvFQ0NOGVxSG/A3GPD7/\nl1WYd/FNQ3rO0az5v9TVj/f++mHs7S9BCIF/rds5oFh5tojvWbMDgJe1eub37olNALt6ijju67fj\nO7c+CyEEun2JYyBael8xnhyVhswdiQjX7+hBf8nBsy9Vdv6WKoR68vebJuqRJyd5nrBilr/+Mzy1\nJST/5xV/RaHkIp+x0Jy1teSvtfz9VRePv1B20ZS10ZrPoLdYjlj1gF4mK0rHTGgaheRPRG8moieJ\nyCWiJcq+zxPRWiJ6lojOlrYv9betJaKLB3P9gUCI4audYRDiDw9vHPJzjuZonx/d+RzufnYHbly5\nBX97Yive/vOH8KcapB4A2N1TxANrdwav5cnjzO/dg+17Q8uR9eO7n92Bzp5iYIUPleXfnUL2kfXv\ndTu68aW/rsbZ379Xm636zEt78fcntlS0/JNUvULZwd+f2AIhBO5/bie6+kqh5S9NFZbFnyfUz3V4\nSbqP63dGibjouMhnbDRlba3so5PVeKLkW1/wpaSWnI2eohP5TsqOi/v8id37zCI4xwd/+wi27ytg\nYssoJH8AqwG8CUAkX5qIDgVwPoDFAJYC+AkR2URkA/gxgNcAOBTA2/xjhw0/uXsdjrn0Nmztqi3L\nz2D0YTRb/mwRZizCJl9KWOcTS9oVwLt+9RA+8NvlwWvVOpZfs27dns9EmogMJIRSR3K9KUI9ZRlu\n/Y4ePO3LKedd8QBuemJr5Nil378PF129IiB93Tj5dKp1fefT23HR1Svw2MbdeOcvH8JHr3o0nJwq\nWP7s8FU1/b6ii/Z8Bm35TESK4ffmMpZH/kXN6kQzQ/H3wj6pYtlBzrY8y79QjkwYvSUnyCDOZ6zg\nHj65eS9uf3o7AIxO2UcI8bQQ4lnNrvMAXCOEKAghngewFsDx/r+1Qoj1QogigGv8Y4cNN/pJIBxu\nVU+M1Rrl9cRQ3hPZ4fvP1VurSgzDCSYFy6LA8cihkGnCJgFg9eZoFUj1fTLZMrG15m28KIUGDkT2\n0ckbPSnGzCsF2yL84I7nsG67J6Fs3tOHC69+TPueSpY//1ZU3w47brn42bMv7QsmJ1ki4pWIGu2j\nEnZfyUFTzsakliy6eqO8UCyHsk+hiubvuAK/uG99kNTF+4pOsuXfW3CC8RXKbrBakA2bCaOR/Ctg\nJoAXpdeb/G1J27Ugog8T0XIiWr5jx46kw2oCO31a87UXvKoVYzkUsV4YytK98v294PeP4ezvpyvY\ntaGzB7+4b/2QjUMHHptNFGSa8gMtN/YWQmB3ShlS1d1lEmHCaWvKYkNnL4iAKa05lBwX8y6+CZdo\nwhhl3PTEVtz/nCcx6ZKZ0iR58Wd+w1HeIy2vICYlSBeh5h+/JpO+mkjFk+Buiai7i2FeA08aRSWM\ndI9/jxxXRH47fcUymrMe+cvZuN57Xd/yt6pG+/z9iS3475uexoqNe/zxh58xZ1tozUU1f8DzVciT\nbZ//Ofb2hfd7QlN9gjKrkj8R3U5EqzX/6m6xCyGuFEIsEUIs6ejoGJJzDiTLb6CohyzhuiJVXfDR\nirQadF/RwfUrNlVcKQz0/r795w/hv296OkLCA8XjL+7Be371cMxy5bFlbIrJF/J1f3rPOhx96W0R\n/R4IqzsyiOKkXCyHn5+JrS2fwYZdPZgxoQlt+Uwwrt9oEphkXHj1Y3jnLx8CoJd9KiVIMdjS/vLr\nDo0R1ryprdr3VLL8+X6pIb1M/p2BL4GCyckV8XIOf350E37zwPORZCn5en0lB81ZG5NbctjdW8KP\n71qLi697AkDU4Vt2Rez3K09Mcg0fHj//y2UstOQz6C04Udmn4ETIn1dYcljqiMk+QogzhRCHaf7d\nUOFtmwHMll7P8rclbR827JN+JPVGPRySv3rgebzmB/cNa8mIkuNGHI+DQdpaM1+5cTU+9cfH8djG\n5GYkoaZa240OyDfFUJ7btg8v7EzOWP3oVY/hnjU7sKEzegyTgitEEIlScuLkf/ltawAA26X67dv2\n9uOC3z8aOR8hLvts7erDvItvwp+Wv4hdPaFu/OKuXsye0oJcxhqQsaMmM01tzQW6dCUEqx2bgqJm\njHZpMpCtZSY+XbRPMGkKlfy9zySXTZDzEHiS5N/Fvv4yLvnbU1i+Icyo7eorBd9ZX8lFc87GxOYs\nuvpKuOyWZ3HNI55AwZJNs1/OWZXENu3uw1mX34PNe/rQr6xeHFcEY8jaFlpzNno0ln+1ybZNU010\nKFAv2edGAOcTUZ6I5gNYCOBhAI8AWEhE84koB88pfGOdxlARw6HH18PyX725CwAiTr164/u3r8E7\nfvFQTQW7kpDWAfmMr99Xqm7oSJERtSBIpEkxO7/68ntx+nfuTtzPiTqqMSFPTGyV8v8y+fOEsEeS\nMHRJUhZRjPx5BfjnRzcF0lHJcbF9XwEzJjQha1sRC/Ifq6JO1ySoZLTfhKaYFq4Dfx82UZDdyihI\nE4ocbddboQAdb1H38Xvk88iJWpVyHPj39OUbVuO0y+7GL+9/Hv1F2fIPz1l2XDiuQD5jB5OZem9W\nbdqDNdu6sW57d2zSlMk/l7HQls9gb1+U/HsV2Ychh6W2j9JQzzcS0SYAJwG4iYhuAQAhxJMArgXw\nFIB/ArhQCOEIIcoALgJwC4CnAVzrHzvsGA7Lvx6av+3Hrw2nP4Edd5UaTDy6YRe27+tP3M9IW4mT\nyVCtay4/OHwP0jgjZfCcPJAwyCSoE5Ajk78/PiZ9ndwkk85zCeTfp2SOMhFlbSt4f8lxsWNfAR3t\neeRsimjH/3HVY4GT9LpHN+Grf9M/eioZzZjYlEoiCyx/i4KImvnTWnHGIdMjWa9ylyq2+HWGEp9v\nzbZ90QmjwLKPt40ouipi56/OvzRzUjMA4NantgEArnl4oyf7sMNX+pxMwCz7AF4UjuxQ3+E/EyXH\njTmEHSECYyeXsdDRnkdfyYlM9D0FB32ashFs+f/47cfg4Bntsf1DgcFG+1wvhJglhMgLIfYTQpwt\n7fu6EOJAIcTBQoh/SNtvFkIs8vd9fTDXHxzqT54DKT9ww8rNFWPiucruSDiTKyXb/NtPH8Rrf3Bf\n1XOUygJPbIpLOfMuvgn/9ddVwWvWsItlF09u6QoISSYmnkfSOCNlBJZ/eeD3cFdPMZKco+rx/P0U\nJMufrXCdFS3LKmt0VTEpLvvwa9uigPz39JbQW3QwrS2PXMaKlTTgFe9n/vQ4fv3AC9rPpn6WaW05\n7Omr7pRmCz1jEfIZjyxztkec8jl3aIwIxxUQQqCrt4S7ntmO/lJY6uDT1z6Osy6/J/zcpbjlL1vk\n3QXPmtbJgbMmNwd/Hzt3Mtbt6EZnd8F3+OYgz0GcO8EOXwB4328ewSnfvis4Zuc+nnRFLAvadUPy\nz9sWpk/wavrIZR3ufnYHtuzpQ0bJYubv7eUHTo19hqFCw2b4DovmPwDZ5xPXrMTn/7IqcT9b/rpS\nsvVGtSvu7K5OEH94ZCNef8UDuPXJl2L7fr8snPTY6tq2tx/n/PB+/OefPQecvLRma5HJVX2AksBf\niypBlRw3Ume9kjR4yY1P4lXflQhJJX8p4oTrzuztK2HexTfh6zc/HTvfnp5qln94jTNfth+AcCLM\n2hREvnCcekd73pN9VItdc4tUI0XWrnO2hcktuYi1mgTHdUHkhbey7MMx8ut29ODor92Kff2lSGcr\nIPzeHFfgD49sxPt+8wguuvqxCBHzb2vVpi4850+OTP4Ez0hg6/zj16zAIV/6ZyR5i30OsyaHNYLO\nPWJ/uMIrR9GcszFJcay+5N/LfMYOJjMVOyXLf7dyjxwhaf4ZwvT2JgCen4Ank+se24S127sxtS0X\neS/7apoH0IYzLRqW/IcjBH+wmv8/V2/FL+9/PrItk9Carp6o1lWolhUOW/1py9TyMvxRn5Rlyz+o\nY6M0z6iGMIQwSv6X37YG//6zB7FqU5d/rWRZiK3+9758HoC4Fhx0qCq5ktafvEKRLX9dZI1FFFjP\nfE3Z8uf3cPLitLYcsrYVk8TUFWN3oRxxtgohIlZ61iZMbMmiUHbRX3Lw47vWBqUmgLAgGeAZJPz7\nlMm/xSew3b0lbNnTH0uWYtIuS+05t+zp1/7Gz73i/sAf1OlPIkQe+e/nW9Yv7ooncHK0kex4PmVR\nGEHYnLUxuTVK/lv9ySOXsRKLvcmZ1OqKznEkzd+2MV2q5jmpOUr2U1rzsXMTpf9NDwQNRf4ycYhh\nkX3i24QQ2JjSWfvXFVtw1bINkW0cMz6QDk3VcP9zO3Huj+6vWQtPcuL+4eGN+PS1KyPb1HK3OsgW\nNz88vEWOB3cVy1+NMJEha7lJsg/XoefiXklVIAGP7M85Yn988JT53muFZHmcRccN5ICuCtLJL+9/\nHp/1WxDqVnWuEAHZT2j2CIytw4zk2GULuaM9ryUs9Xdz2FduwffvWBO8Ljpu5PvMZqwg1HDLnj5c\ndsuzeM+vHgYAvLCzB2/6yb/w9ZufAuAZO+yj4WvnbCtivXZ2F2ITZZO/33GFlJjlxMhfNTJkTb/o\nuJg3TR9OCgDfe8uROOOQ6ThxQSijLJjWGozTi/aJEvJL/kSaz1hVSdiz/JVQTyHQXfC+l1zGwvQJ\nnuVfdgValDyjqa3RawPehER17OfYUOQfIYBhqAujs/x/ft96nHrZXali9UuOG4tIseto+f+/657A\nqs1dwXI3LZLI//N/WYW/PBaN5GXHaNZO/untk0vs+ufmjxuRfQKHr3d8U8IDevOqrTjyq7cGFj3f\nOXXcbKEyyVZqEr+7p4gpLTm0+A3SVYdvrxRu2FPBASnjz496tX/KmvvZX3LRWyyDKIz+4N+zECK2\nSvEcvvH7ccq378Klf38qsu13/woNjL6iE1kdZG0rsFLv9S1+JkwuLrfRt7QdR7b87eBYuRT1zp5i\nzKEsW/78nXjkHx37rt7kybNYdjG1NR/rWdyWz+BNx8zEwv3a8av3HhcpkkZE6GjzLO6mrB1p3AIA\nW/dUt/yD6zsiIo1lbS86ixs5ZW3ChKZMuCJSvpspCeRfTzQU+cuOweGx/OPXeHBdJwCkqi1UdNyY\npcYPVz00fzYykiaWpPmmqMgGlcCWf6WHSV4+q1p61OHLlr+/mtA8LGu27cN3bvEqkPA9T4r2YQs1\nyLJMiJF3XIE9fSVMbs2FE4ZCaGzdeuSfLhqp1T9X0iTR1VdCc9YOCI6lHvUeEQFTWnKJ91iVEvuU\nJCP5+jnbCrJz73jGqzWzaL82AOH9PGBiaNGycZLPhrKPTGKd3YXYKqnVj2N3pCQqXex7klHiuGEm\nrhod9qFTFuB7bzkq/DzKPWEZqCXnOXxlbKnB8i+UnECG8s4Xjc3PZSwQUeD0zSiTlI78a+nfMBA0\nFPnLfDksmr+GoPnBqmT5hse6MSKwJOfYUIMfHDVKotrKUyZR3UMrIyy9qz/pJ69ZEUlM4rA9nqxl\nDVuN89c9oGddfi/W+0labU2ZSFlpdWJlEgqyLBPCG7v6ShACmNKSRT5jgSjsmcuQE416iuWYRdqu\nSdxZPHMigOQQ1F09RbTk7MBqZKlHbbTSls8gY1uxa6ZBb6EcWXlkbQpkn0d8vwt/dy/48iXvd1yB\njD82neYPeOGZ6m+kLc+WvxtIcTp/S1IocaHkoFh2kJfIf5rvQFXJXv2N8HfenLVjmbTsO6jk8GWs\n29ETGXOr4qjl6071tX31+dc5duvp7AUajvzDh32kHL78YGesNOQvYk5JLhJWjy5WzMdsSd75zDb8\n7z3rgnv1f8tewN+f2BJ7nzxZVCuXzeeWCU7+LH9duQUrXwxDQdmi3ra3gG/c/HSEkPlt3SlWE0A0\n0UgdAxDKRjzhJGXH8mec3JoDEaE5a8esbya4QtlBT6Ecs+zksX7ktAUAovKHDnt6S2jO2QFxsOyj\nyiFMtmkMDBU9RSdy/awdyjZMbtwr4Hl/UpVj9Zl8mezytmL59xRj94qt5GqW/7a9+jyT/rITZOLy\nb/jAjjZ//NEJUL0nMvnbFkXKUmyWQj2rWf5q+LJK3Dnbe83ZujHyz9qRMFQAkUmzHmgo8pet5XpF\ny/QWy4ElqpNA+MFKY5WVNLKPVUfZhx9cfvDe/5vl+OY/ngn2L1u/CxddvSL2Ppn8d/d4zUuuX6Gv\nXV8M6q5449/Y2RubJKNVD0MCvvLe9RFC5kmjNyjZUfme7FPyAVTNn+8pyz1J5M+Ovcm+TNCSs2Oy\nDxPclq5+uCIaYghEV1NvO24Ojp07OZjoSwnlKnb1FNGSzQSSAZMx17hha5fJttpkqINaeCxrx4mP\ns2k5YouluETNX3H4qpo/F1ksO6JiBniS7FNyPJ9HzrZw6kIvgmeBT/7qc8L3hD8Tr8B4+2R/kpYL\n0aWRfZ7cshe2RYHfoFVZ2WUz3n0JyZ8iZRuasha+9LpodXsj+wwhIpZ/na7x/65bhU9e40W46H7H\n/GClscqKZTdmnfLDdf/anRELeSjAfKRaXdVkHzlUsLOngDuf2Y5P/fHxiu/hekGnXnYXrlecwtH0\n9+hYZKe9muEr36pNu3tRdlzMmxqSbrdC5uq95decYJYU7cOlFNiab85Fk5jktP71fkjoQT4Z6ZDL\nWLAtCib6pLITG3f1YqofwhkZj+8jYUmBSUPn8K2G3oLi8M1YgX7P6A5WRt51+fcS0fwl2Ue1/GOa\nv8by12HT7sp+slzGwvfeeiRu+vjJQVil2oeASZ7JmQmYPwPH+h8sNW73yL86ES+Y1hrce9VZy99F\nq2T5P/zFV+HtJ8wB4CkRZy+egQcuPgOH+/KfcfgOIeRnSrUSV2/uwryLbwoe1oFi+97+IINRp8sH\nfUlThHAVHTdmufDD9eiG3XjDjx+o+P7Lb1uDt/zvg6nGDUg9T2sslxApVFVwsGx9Z2S/bgVUctwg\n4mmFMonJ5KCSvxxOpyZ5ceGiBKwAACAASURBVBvBzXv6cPL/3IUf3vEcXAEc4qfHc9gdQ11V8XfD\nMfdMbmryWGD5t4aWtjxmeYwcennQ9Cj5f/LMRcHfWV+f59+LPK5Lzj0Uv3j3En/8Zcyd2pJoOMiT\nEZ+3VniFx2SHL8WIr9ePCGJfQ2D5u26wKpFDPWWrmUM95Z8/E6JXNTPZLFu1eQ9m+OGSOnj+hQwW\nHzAxWE2oeQ78W+QoH742Gwbs9H3Z/hOC9+QzdqpV1OTWXPBcq5Y/v58dzFnbGytPNvzbmzmpOQw/\nrTP516dc3ChFJc3/+hWe9XnH09uDJeNA4LgikCN0MkS5wj4VJb+wlBAiiPe1U2axAsAP7ngOACLv\nrwRSZJ+0kGWfouPgcT+kkpHk+A65SUk8ksIm1RBKuWwu32e17SBP4Ms37PYKc/kPkWr5qxJDMbD8\nPdJ/Zuu+4LzyPWRLm5f4zblMRPbhcsxt+UxgJS/oCGPQf/+BEyC7fHK2BduyUHYdCCEiE74rosv/\n2VNaYFsEi+JZ6lMV2UeNJU+DvqIT8TPpZB/Au5fsvOfaNDrLnyhqHHR2FzF9QhMmNmeD+8xEXc3y\nX7OtG+88cQ7OP24O1u3oxieuieaQyCsdJl/19zNzUjM+deYivOkYr+fA+0+eh2XrO3He0QcACOWe\nQ6R6OhNbsqnIX/6eVL0+tPx5Yo5OkvJvkY2Nejt8G4r8nQj5R58cpsbB+gIcIYLr6EiPIylSkX+Z\nswcFcr5mWAP3B+jsKWJaWzyDUAWfWyX/apFF8g+3WHZjJZB1/gnvIdc/UDJJq5a/XBpALqEgjzMk\n5xzW7+gJiEjtQ6sSDdfH39NbRLHs4kFpBVN2RfDAdvWVkLUpJNmsHYSHAl7K/iEz2jGtLY/71+7E\ntLZcJIxQJZJshpCxyIt2USxfV4igFAAAzJniyVhZ24rIbVmbgvh/Jp75CTX0K6G7UI6sPLK2pZWP\ndnQXgsmHJ19XiCAgQV51yKvcfYUyunpLaG/KBOTPDt+y61Ytz71ov3YcNnMi5k5tie3LSvf12LmT\nAQAnHxTtA0JE+MSZC4PXsya34OZPnBK8Zj/OCQum4uNnHIRTF3UEUUBZmyquTJokh3OrJtQTANry\n3rn4HuvIn6+w/8TkVc5QoMFkn2TN3wqSpwZ3DTlLUR/tw5Z/9XMxOcmWWC1zEz+0LyqlFB55YRe+\nd2u8+2bg8C06kXtVLTkpYvmX3URHqoyS4yZOgGwt52xLQ/7FgNyY7IP75I+TNflJLVmUXSGRf1T2\nUR2rPO4Nu3rxuwdfQG/RwTFzJkWuAXjkP6EpG6wEWnJhtM+abfuwYuMevOmYmcGDPWtyS8R6zmWs\nSJx31rY88peiu+b72aqzJjdHJAgmf5WQJzRlg2uwBbpwv9pXsIWyG7P8LYuC6/GEJzef4WqWZSe0\n/JkEhQBOXdSBz561CJ9/zSEAPFlOTraSv89q2eWLfC2+vSmL95w0F684KMzYzUv35JAZE/DU187G\nOUfsX8OnD8NW25sy+PRZB2PJvCnBPr4H6uTN25uydvC5Y9E+Afl723nifv2RB2BCUwZvPjZsc8IV\nAOTvvR5oLPKXuEYNlQx+rIN0BZcdEZCaLhwzzFj19q3a1IWf3L1Wey4+VibfWianmX7omFpH580/\nexA/vFN/TcCz5GTpRddiT4b8wBbK8Qil0y+7K/aeotRuT50D2PKf0JyJyz49xcA64/vMkw9PBhyK\nObE5C1eIQLPuVpx/6qRUKruYOakZE5uz+JYf5cTldOXvYG9fKRITzg7fh5/fhbMu91pJnnfUzIAU\nZk9piVjv+YyFrKT7ZCxCxtf8ebX3jhPm4NqPnISzF89Aaz4TyBBM/mqSUGs+E6xMmKDnDsDyLyrf\nH58ziFP3pSW5aFpfoPmLYFzkr6UFPKnyojMWBhNad6EcIX+WaF5/xQN4bOOeWIy8jJfNCAnxq+cd\nhgtPPyh4rZKymmiVBi8/cCpOW9QRK/IGhEmEE5T6+hxl1ZS1gs+tyrP8W2jzNX/+zc6a3IInLjk7\n4hPie2vIfwjhVLL8ffYfbASoK8L+oJVkH95z7hX349v/jFvhgGzRSjHxNQzwgElhFcFK51fP3Vt0\nIlm2any8ikJE849b/rpqn6Wy0EZDAZ40kPPjy1XLv7O7GOiy/H5+kJjM2eEu4N07jlZRZR+d5j+l\nNYfZk1uCc/GDrlr+7RI5tDdlsbe/hE1+TaCLXnkQ9pvQJFn+zRGnKUf3MIjI1/xFEOmTz1g4fv6U\nYHXxx4+chN+877hAPtLFq3OC1UAdvrZFQZABG0Nqxi7XoOGY+5acHYScepp/8jWnStIj1ycC4hJJ\npYblE5USDG1SXP5AQltVnLBgKn77/uODeymDCVwee3tTJrjfTVkbJy7wVgpqwhjfR/6s1QwqIFz9\n1QsNRf6yzq+SKD+Lg02eKrsisM51p2KrSr2OWs/FlcIFIw7AGsbHhMOktK+/VDGShh/ivpITCanU\ntdiToco+aQrDlRw3kMV0ln9T1qunokueCix/bpjCMptPnCxJlMouXBFard39quwTj/bJ2hRZsjMR\nydbw3v5y5OGe1pbDrp5icB/OP95bwjMZzZ7cEgmXzGmyb7OB5u8nASrkM7E5i9MPnh4er+xvy4fJ\nX7Lj8QuvPQQHdqQjkXzGCn4ffM/CpC3vnFMC8vfu8dS2XMTy548lyz6MaVLZYrk7leqYbk9oWM7E\nKkOOlR9IaGst4O+QJZqmrFfuWr7vl77hMPzzk6cEFUYZPInzZFXpmfrB+Ufho6cfWFNwx0DQUOQf\n0eAVwuEfucqtf39iS1DAKg1c17P8//jIRrzt58ti+9naVK+js0L5mEg2bA1zk1ry+PBLbsWZ3wtr\n0CdVouwvOZFM2mptEmVJpK/kpFo9lWTZR432KZTRLJUxUMHWX+DwLYfRPpv39OH2p70aNJ4V6ybK\nPjGHr58l2qIhf/nYvX2lSCbotLY8XBHWdufs7YD8p0Qt/3zWipG7bREcR0QaolSCSgwtuUzwHnli\n+fCpB+JDpyyoeC6GN9n6RfL8CYQvw5MBh7dyqYWprXkp1FPEMtfl7zZi+cvkr4Q0qrIK4Dlwf/qO\nY2Pbh9ryrwS+B6962XS88K1zcMiMCZjckg2kriY/H+CQGRMSQ7l5sqpE/ucdNROfW3rIEI8+jsYi\nfyWETgYF5B+NCLro6hV47Q+rd6hilH3y/+6taxL387llqFEOchZkOaL5p2d/VRMHol2E1KiewPIv\nRi3/aoXJitISNm1XraIfxgpoLP9CGS25TOLDPEmqJQOEk4/jCvzID2/1trtwXdnyV8hfSaYqll3k\nMnaE/DkDVJV9ZMtf1cGDWHdbsvylz5K37Ri5Z2xCSXJ4ViOyeH2cTHA/4gRcGff/v1fi6g+egJxU\n/1+1/Hk8/LnZrzKtLUr+ocM3LqO2RibVkLTViVBn+X/09AODiUdGxPKvM/mr2cFHz5mEExdMDT6j\nXFhQ9ckw2L9RLappONBQ5B81/PWhnvJWuSxvWjg++auTy33P7Qj2q9cB4paA/HBHo30GR/4yVIue\nLX9V9lHb06kI+pTaVqzIWBJKTugbUT+7J/skW/78AKmfr+x6Ne9nTW7GzEnNKJa96JlQ81fIXyP7\n5GxCczas98IPPE8wQgjP8pfJ38+sZR1cjtO2CDhgUnPFaB/vPZYf7aIncBW8auOJqiVnS+QfPXc1\ng2HW5Ba8/KBpyEmyD1v+FMg+evKf2poLvsuylOSlA1FY/kAmbXW8Os0/SQKRE6Hqb/nbkf+/cu5i\nfP61Lwvuuyy3JVn+/H3pmrYPNxqK/CtZ/qHDN9zBP/BK0Qe6azhCxEj6Xb98GGu3h71Z1QcybvmH\nrwca7RM0EU/Q4GXZR64J31eKtqRTdXcVPPbWvB0j2CSUym5A+qrzq+i4aM4m11Dnh0wN9eRQwZac\nR9osnXEIYDXZp6TIPi250ELnY3v9wmeq5g+EOjhbsm8/fg5+8o5jgnK+jFzGipG7V95B1vwryz78\nWaZJmaq8krGV9y7aL10D8JxtBSu3powq+0QjXTjZjlc9/SUnYvm/5rAZaMtngvIFjAtf6UXnyPdD\nJXad7JM0GUbua701f6lkhQx+luWILv5M7U0ZPP6Vs4Lt/NlOqmNv3rRoKPJXJR0ZgcNX2s5p+mqd\n70oo+xm+OmtLjnpRSbxQdiNjkom5Vtnnpa5+9BbLwTWSLf/wGvIE0V90omUUqiZ5eftbcpmqEwXg\nPUQlJyR/Xfne5lxySr0a5y+HenpOW8+hyvcwY1vajFi1YmrRcf20ezsYAycO3bByM8674v5gRRSV\nfTwC5sJjPGHMntKCpYfF48xti+Kyj0Uou2F2b1oi6/Br2LTlM4n+guPmTcF1/3FS1XPJDnYmMjXa\npzXvrWa4kih/9r6SNylyktcBk5qx+qtnxyaeD5w8H1e+61i89bgwrl2d6HTVLNM4P+vZ8hCIyz6M\nwPKX/Dp8H1pzmchvpTWfwZ2fOQ3/829H1HWsadBQGb6Vyjtwkpe8vTMo3Zsceqa7RhJByySsHlMo\nRzsXybKPrE2nsfxP/OYdOGLWxIBAio6rjRL62T3rMHtKC+ZPa40QcG+pXLE0s1ouwpNLvAJgaSz/\njvY8io4bJAfplsDNWbvq0pnvYUHpJ5C1vUxLXlHYFiFjWyiWvQbjrz/yANzx9PYgo1f9HBztk5Pi\n8X9+3/MAwtWgbJ1Oas7CImD7vqjsUwkx2cf2Qz1TWv4Mdjy3SKsunZU8L0XMfy4TWv6qdi8nkHEI\nbs62An1etfyTQEQ4a/GMyDb1fukm/TT3Y7gcvir586Mlyz58H3T3YzDlY4YSDWX5y6v8mOYfWP7h\ntiBTtLkGy99x4ThCS7Zqo2wZxbIbsbBlQixVeF8SntjUFdHEddEF/1rXGRR+k6WX3qKDPb3FROtT\nXQkU/S5KsmxQCdPa8ig7QpJ94mNrytqRdH11HwB84+Zn8M/VW1Fy3CDCpa/keKRth/q1bGlPa8vj\nB+cfjY72fFXZhxAvvc0GgRxlYlkUacCdxkpVQzUzFkUyXNP0ewDCRKbmrB2sZHTlwtOcT86o5olX\nlX3yUlvG1rwt1fp3IkletUDNDcjaFhYfMEE5ZjSQv7fqUR3UoeYfl33q2IJ30BjU3SKiNxPRk0Tk\nEtESafs8IuojopX+v59J+44lolVEtJaIfkj17FCsQLa21aq5vEun+euiDJIQaP6afZEG8mqop0L+\nEdkn4qtIL/qzWlQsu4kOph37Crjizudw6d+fBuA97L0FB7t6ipiRUFsklhnrk29ekg3Uh1dGWz7j\nyz7Jln9T1o6k68uQnXwX/P4xCBFOCH2+RZq1rWD1lLEoeBj5vV6dFu9HsHpzF+56djsKvmTU7BOq\nQPxB53LPqjTBafsZi1IV0VPJzA7I35d9Mukei6B8c8YKZB9dopXqB9BBln14Jaw2Z5Ebm7TmM4G/\nY9PuPt/yr51SVMs/a1s4bl40pj+bcvKqJ3IJpZ35uZX3WRUs/9GCwd6t1QDeBOBezb51Qoij/H8X\nSNt/CuBDABb6/5YOcgypodbzv3fNDsy7+CZs2t0bfIGuhvz5wf7xXWtx5zPbKl7D4QxfDUfL0TU6\nzV/OQ4jIPpKFmiJ/KgBPZCXHRX+FjMLv3LoGf3vc69A1pTWPnmIZu3tLieQfy0lgyz8Tyj4fO+Mg\nfPfNR2rfz8TLFr/OJ5G1kx2+Ok2Yt3myj1eLhu+hRSQl4ljB+ZloX/ej+/G+Xz+CYtlFPmNF4s5V\nQmHNXy23y+dPa/nGCS+sq+TtT/dospmRsy186NQFWNDRirMX71f1ejrIcf62avlnWfIIrf3WXAZH\nzJoEi4DHNu5B2RWprqNCJcisTfjIaQtwqFTeYDRY/s1ZO2LdM7SWP1fhHcWm/6DulhDiaSGEvjaB\nBkS0P4AJQohlwmOm3wF4w2DGUAsilr8Q+OPyFwF4P9yQ/MPj2enJb7vslmfx/t8sr3gNx3f26iz0\nSBcqnewjOXaHIs5fdojqnKo6TGnNorfooLO7kFg7PVYH3wnJn2WfXMbC7CnxyouAR5RFRwRlI3Sy\nT85fSeigK3XL8kdf0QnGwufP2KHl3xRY/lZQMptRUhy+/F4ZYSVKPfmnsVABxFYHbDHzaijtJBKu\nFCwc2NGGOz9zeiSZKjx/CvKUJkRV8+dJMB+x/G205TNYtF87Vmz0ymen6VOhQqf57z+xOVJtczRo\n/u95+Vx8801xRy0bbTrN3xrHln8lzCeiFUR0DxHxtzgTgNzfb5O/TQsi+jARLSei5Tt27Bj0gCKa\nv4iGd4aNNMKD2PKvpd5PUpw/ECV/IaIVDItO1PLvTxnnX+l1EO3jhLLPJeceil+/77jE8U9pzcFx\nBfb2lxNLyqqlKNhRmvOdloBHhkmEkw2ifZJln6xtoUXT5BzQt7cLZB/f4Zu1KWr5x8ifsGx9Jw78\nws3BOVzhd55iYhdxbV4tQyx/JiCdvKIDEyBb/tUkjBsvegWufNexQYJdtTo+aSxQ2ccSk32yYYx7\nPtD8vXtwzNzJWLlxDwplt2bL/7CZE2K/E90EknbyqicOmt6OpYfNiG13K2j+Y9ryJ6LbiWi15t95\nFd62FcAcIcTRAD4N4GoiqrlEnRDiSiHEEiHEko6OjupvqIJISWchwsQuEerYRcmq5cxWgXjcftL5\nXeGRiK6csxw+6Yoo6RXK0SYaUdknOc7/6oc34uT/uTMYXySXQUqi4mvNndqKV0o1YlRMlRyX+yVY\n/kXHxU/vXodf3Lc+eK3KNBnL0hLBO0+c41uYcqhnnPwzthVL+2foVgSB7CNr/kVJ87ej5G9bpF9x\n+N2ggtcq+fd536G6+uDJpZJcc2BHa6zgFyMjOay915UfzSNmTcJZi2eEln+V43UW6FmH7oe7P3t6\n8Fr2sfDhanmHXMYKGt1zkbJXHDgN+wpl7Owu1DT5rf/Ga3HjhSfHiF0XWlxpUvnZO4/Fqw6ZnsrX\nUg/wsy5r/mPB8q8a6imEOLPWkwohCgAK/t+PEtE6AIsAbAYwSzp0lr9tWKBq/nJsP+8rOy5++68X\n8IqDpgb6pyuqFzcDooSv+wF3SuGTQkQJvlh2I05oWaYpR0I9o+d9bls3Nu3uQ2/RQWs+ExmDHO3D\n51P7saqYIjm3p7XntQ0syo7A//zTK3n8wVMWSJq/nG1JsaX6D84/CucdNRP/+afHUSq7weSbJPsk\nWf46pxtr8F7TFe8zBpa/RYEcw8S1YqO+/3E12YdrHiXKPhXI7/ZPn5a4j8mNv6e0FvRXzj0U+YyF\nkxdOS3W8jOPnT8E8qXKkPHkHmr/SmSsa7eOT/0FTQeT9pmux/JOcorr+D5Us/6WHzdBa5MOFShm+\ndV6MDAp1GRoRdRCR7f+9AJ5jd70QYiuAvUR0oh/l824AN9RjDDqoxCjXHwmaZzsuvnLjkzjnh/cH\n7emEiBdB057frUz+uyJJXgL9xWg1TJnk5WSpSpY/T1DcdzYyBqnTFTt8dZKJDDmyaUpLTmtRqslR\nju/ok4/1GpSo4Yw+QWZ8zV9TtTQ4ViFhGTrLX7bEWfMPrytF+/jH/dc5L8N7Xz4v5sDLZaxgm4Be\n9snaFNseyD4VSIooORKINX+esNLq13OntuKn7zy26veqv2ZyfD0TMwWWvy/7ZMP7wy0JJ7XkAufs\nwDT/6GdVZUVgYP2Ihwu8wpYNq6Ak9liWfSqBiN5IRJsAnATgJiK6xd91KoAniGglgD8DuEAIscvf\n91EAvwCwFsA6AP8YzBhqgcwxQkTr+bCVzERbKLtBaz4BEenRmoRqmbBy4pQrRMTyL8Qsf73DV5Wf\nuBAXhyDqZB8hEEvbT8JUifwnt2YjxaoYanJUyXGR0cg+unBGADHZRwfVApdRSfbh98pkYUvkz5//\nXSfNwyWvXxwjzZxNCH8ZcUu+q6+kbayd84+rhaS++abDg8xbXmH0S+Gp9UbM0SqNnWvwcDLbjIlN\naMnZaG/KBhOB3KSck8iGItqnVst/pHHBaQcCiEaA8XM6pmWfShBCXA/ges326wBcl/Ce5QAOG8x1\nB4pYG0cKxhRYszIhBeSc0vLX/WhlyJq/Kvuo7fOSHL6q7NPrk3qXxvKXh8POZrbajp83Bc939mDH\nvrAnLhCVfaa05rREq1r+ZdfrMxstXqYvYQCEoZ5q9Olv33883vOrh4NjkjoxVZJ9AK8nrmzX2FY8\n1DM8V/R1e1MWMyc1oyVn4/8tPThu+feVtOPi42ohqbcdH9a9CWWfdJr/UEAlJnnyfteJ83DwjAl4\n14lzAQDnHL4/Tj5oGtrymdDylyZcbhw0EJJWfydqNJnumNGEj71qIT72qoWRbbx4GbeW/1iDozh8\n5e5dYYXJeLMTV4hU5F+t0coeqVKmQPScxXK0p20lh+/MSc34xhsPj4yxq7eE+5/bGTlW/rysVbOl\ne+0FJ+H/PnB8bIwRy78lpw2rVOPyy46LjBW1/HXRPrZkHRfLLvqV88jO0KxtJRbU00ki8jjzyirE\nJsnyV86pTiT8mZ/62lIsPWz/GPl39Za0K5Igzn+AJMXjSxvtMxSoVFYhn7XwgZPnB9ssiwJJUNX8\nAa+WDwDsVRrmpIE6CekSBEez5a8DP3uj2fJvKPJXa/uEso+IxMQz2AoTiNdP16Ga5R+p7eOq0T5u\nJBQ1WtgtavkThQ8uk/8fHnkR7/zlQ/jdgy9EjmXwQynLHDqi4ge8Jecl83RoYsblcs8AguQembAy\ndrxsMSOXsYK+BzJkiSUr1djRvV+FKvvkpHN5ln9U9mF89PQDI6+nKNncuYwVKT9cdFytvh46fAf2\nSKlO6oGUSagVurIKjEqTWF6J9gGA/Sd65M9lrQeKOz5zGs48VJekNraoip89Y/mPEqhJXnL3rnJg\n+cdLKQtRvZuVev40x8pF0FSHr2wVq20cLQojabjW/qpNXvTKhs6wWXt/yQkimvb2RWUfQF8GgMmP\nWyVO14R7yuWehfC6T2VsUix/ij2wvDJqS4jika27rE0RyzLpOEZU9qmg+SvEff7xc3DLJ08NXuuK\n+KnhmTrLn8sxDNRCtQPZp7Zon8FAvUZeWbklQW/5e78Trmw6UCQVoBtrlj9DlxE8WjB6R1YHqEle\nPCk7bliITVdqwBUisNIrTeTVLH8ZshMWAIqOE3X4RqJ9olU9ZTLjFYJMyIyeohM8qIHln0m2/LM2\nBVUaeRKYqqlrtEfyXZQcr+m46vDN2fE4f74/cqcm+RDZSsraltaxmoRmJTZfTljyCrt5r5s1D6NM\n5pM15bvV5iK6FUmaUM9KkDX/rJ2uPtBAEcTwV5B9Kq085MJuDLb8B/r5GUkkP5o1fx1OXDAVHzx5\n/qgo3ZyEhijp3NldwP1rd8Ysf36+yo4rWf5xeUcglFcqWUSOxlGVBNnyb83ZKJRcbZIXUby8A1E4\nDrWKpswZxbKL9tYceosObljp1e6RH3j1AW/K2GjO2iAK5R8dGe6RJpr+soOyI5DVyD7qSsgJLP+Q\nTCe35IL8B8uKkn+S5a+DavnL8Eo66y1/IC4ZqVDbClbS/AdqoWYk2afeEkc+Y/vXSY72qTQGubAb\no6M9j2+96XCcsmjwyZg6jGbtXAfbIvzX6w4d6WFURENY/p//yyp84pqVWLcjbMTu0ZD3g5L1Z23X\nKykyp5IjTpfVmwSBsBvT5Nac33Uq6vC1yA+LjJR38GKpA8u/ii+iUvy3SlT5rA0iQmsugyl+qN8U\njQwiRy2FddytCEl6so8+c1MuhzypJTy/FbH8KUKyf7vo5MTPAUQJOWeTxvJPJv9qk4zaWUoX7ZMZ\nQKhn5P3++Dq7C5H7Uw+wdV4pzr+i5i8VdpNx/vFzMNN3/BqMfjQE+XOYo6yHe9E+3t9yM/GCpgCa\nHJlTaTnsqHWiK8AVAj2FMpqznrVdLLsRC58LlGVtK2b5W5LDV82+VSFrjuqDqVp3bNGdfnAHXn6g\nlzEq16lnRCz/ohuUdOaWfoBX4CwpbV/W/OWVhSr7yBr04bMmJn1EAFFS93oLSA5fCmUfHflX6wAl\nNxsH9LJPbpCWP7/vhc5eHFTnZh98rUrRPpV+55wDUEuTI4PRh4Ygfy5N/PzOnmCbrPmXKzQTB7zI\nnDSldmspt+wKoLvglWTIZaxYSed+v0BZxqZYtI/n8NWPQ3U7yGR3l1THBYgTFU8UV7z9GLzFb7PH\n2v/x86bg3v98JQDF8i977fsyNkXqAsklFQBg/rRWnOZLAu0Ryz8nvScci9eNKz2RNqvRPgqRhbV9\n4vet2nViln+FaJ+BihOyVr5wv+Eh/5jmr2RoJ+HsxTPw1wtfEej8BmMTDUH+7JiKZtiG1TpLUkVN\nncNXQEQ0+CSoyU+VIHzNv70pg3zGQqHsxDp5cYEyeUJyhUdWSctytUiaahHLUJ1zuuSpY+ZOwhmH\nTMdXz1scWIOy5d9XdII4f9nyB6LkctdnTw/8CFHLPyRWNdqnFrTlo+Qvk5euqqcOOuc2ALx5yazI\na9UBzNcEtG0cUkGOvFo4vb7kz/Ka+huSw1wryT5Z28JRsyfVZ3AGw4aGcPjqmorLsf2liOWvOVaE\n59DVHWFUK+8gw3U92ac1b2NSSw4vdfVHO3mVHLQ3ZTFjYhM27orLVUkPpxqSyo5Q3aSVZPnLyGds\n/Oq9Xglozgbeo2j+HOc/VSMR6ZCk+cuyT60Zrmolzsi5LCsg16QIors+ezomJVTcXHzARLzwrXMw\n7+KbAHitIFXwZFVL+W8Z8vd50PT2CkcOHmHRsej3LxP6WA2tNEiPhiB/XXauHNvvlRoQwXYZuYwV\nKb+sSz1n1EL+7PBtzWUwtTWHp7bsjZVmyGYIB01vwz1rvF4GD63vxLPb9mFScy5Rk+Uy1AwmdJ1V\nG9f8K4dWMsHJSV59JceP809OylIhOwqjsk/4mViCWLp4BuZ3VG4+3pS1IiuFXMaKkLBloaLDF/Bk\nqbRQVzh8TWAwln84efgo1gAAIABJREFU/gVVPu9gkVRrPmNbmNaWx87uwoiVRzYYPjQE+ffoSjMo\nDVySInW8EgMiOEepgrRTk+UvgG6/YcrUtjw6ewqxPIGsbeGg6W3486Ob0NVbwluvXAYAmDIrF5EJ\nchkrkKvUiY5PqWauAtEYe6B6Qgpb43KRu/6Si5Lr1iTTyEQX6X4Usfy9v3/2rmODbf/3geO1Ppem\nrB2x/L3SEtK4LUsK9Ry80jlNQ/6B7DNA01++f7qs6qFE4PDVfGd3fPo0rN/ZHdtuMP7QEOTfp8nO\nlS3/ohMvNcBoyWUghL5qJkMIge/eugYd7ekfWiEEeopltDV5ln/JEehSErVythXov2t37Au2q5r/\n9PY8Nu3uAwD0lqKfdY7fSvHTrz44NgbVuqtm+fM1ZX7rLZYhxMBlAjnSRo3zV3HKQn0MeVPGjrSc\nVEsu25LlX0viWBJ08lZI/gM7pzyZ1zumnU+vy/Ce2JLF0XMm1/X6BqMDDUH+qhQCcOtGv568lOSl\nojlnQyCMcCk5XlcvmTh3dhdxxV1raxoTh3q25jOBjLB9XzQ1PpexMNdPd5d1fzlpCYiSv2r5T2zO\n4oVvnZNqTFUtfw0pcZLZQOPb80o8PqOWwmZNWSsqGUkNR7zzWsGqRVeiulboZJ9A8x+g8DOcGax8\nr8Za1qzB0KIhyF9OhOKOQ66INnBJqsiZ9TNV5UYsjh/ayFDLIqeBK7z8g7Z8JpBk1PNk7bBxRkmq\noe85fENylFcc6kRXi0VezfKXz8X3ca+fQ8FEcvWHTsD2Gop7VZN90kB1DueU0hA2yUleg5d9dLWJ\ncoO0/PnzDkc1z6BL1xjQ9ZcunoEnt3aN9DDGJRqC/OUSCBnLa0soIDl8XZFo+WcsghDALinCxYtr\nD49RLfY0KPvNTNrymSB6ZHuM/MPkJHl8quwj99pVM35rMe6qESP5mcWOK9CWz2BffznIUmYC5uSw\najh+/hSUHFeRfcL9tawk+F7MnNSMzXv6/M8ikb9fJK4paw0Jueqcofz5aynuFzmnnyFw9Jz6h1CO\nxkiex770au1vVfb5GAwtGoL85VBPm8lfhIRfKidb/rZF6Cs66C+5mNaWw87uIkpKWd9tez3yb2/K\nBNnE1dDtW+gtOVuSfeKWP1uEcg6BRdEHeHoFX0Mt+nGasgJM/q05n/wVyz8trv2I18GKI5mAqOVf\nC0nzPbr4NYfgY39YgentTbF6/u8+aS5OW9RRtyiWwYZ6zpjYhMvfeiTOODheznio8MDFZ6CvWMbH\n/rASwMAnqnpAF5BgUF+Me/IXQkRi3z1L2vUbuIQ9ZHUJWhmLQATs7PFI2QuDK8acvi91efsnt+RS\nkz9X6sxYFPzwY5q/VBlTDjG1KOrQ1MWdy8emwVuWzML7XjG/6nEZi1CEF4aaz1iS5Z98nQMmxstC\nM5oSNP9aZB92XJ575AE498gDAEQb69gWYVJLLhJWOhDccOErEleIg5V9AOCNR8+qftAgwOU9+Kcz\nmsjfYPgx7jN8C2UXrgjjsJlfXL8OPeBn+GoeasvyFuOdvt7P8opaT2ebT9q1OPuY/G3bQj5jwyKg\nV9HrPcufZR/Z8qcIUbbkM/jCaw/RFtVK20zihPlTK04iwfkCZ6HnVN3nl4rOJpS9ePzLZ+H2z5yW\neD7ZAStb5dVkn38/NiRK3apDXvEMlcxx5OxJOHauPhJmLFWd5N9ELaHJBuMP4578WfJR67MIQMrw\n1cf520SwiIKkJpZX1FXCNr+BRamc/DC9/MCpkdc8gTBxZWwrVpohm5Es/4jmH40Lz2csfPjUA3Hk\n7Hjxs7QqR1pLm8djW4TmrB2sdJLeP7Elm9iLF0guqlYtb+A7bz4S3/ZrpVcj9+HQuIOWoANO8xo+\ncN2mWZNbRngkBiOJcS/7sLN3YnMGO7sLwaPpypq/I7S1+Fn2YUyf4JO/dGxXXwkrXtzjn0efAHbP\nf56ODZ29+Ne6zmBbYPn7xJS1KNbTVi6LrMo+kbBIn0B1Onla4kt/nHeNjEVoyoayz0AJNinjNo02\nzxNONX/DcJA/D3csGNNvP34O3n78HJPF2+BoGMs/aMXnP5xChBZ8UoavJ/uEDwiXH5ZJ/vfLNgQF\n43RF4QBg7tTWiIxBFB4rW/7qMjyfCcsiy5a/RdGwzDzHsGtCNdMSX1qHbTheQlPWDsh/KOL8awVL\nYtU+43DEs/MVBprhO5wgqm+nMIOxgXFP/lv80D8uP8uPppA0/6QMX9uiSI1eJir52K1dfZjcksXk\nlmysEcylbzgMHz/jIABRGcMiilv+EnlysTMua+yFp0Y1/1zGwvn+8r3dl7TyAyhXHH7WdD8FtrZt\nX/MfaLQPYzDkn9XcOx2GI56d7/NYsPwNDIAGkH1WbfYSRI6cPRE3rdoaRDh40T5hbR9dFIdtUST2\nmElGdvj2Fh205DIolJ0Y+b/rxLnB33IiEkGO9on3fp3cksOe3lKkNaDcZIadi9980+F4xwlzcdjM\nCQD0RJrW4Vuz5W8RsjZh3yAt/0ollquBCXc0WP7hJG4saoOxgUFZ/kR0GRE9Q0RPENH1RDRJ2vd5\nIlpLRM8S0dnS9qX+trVEdPFgrp8GqzZ3YcG01iDMj8nfFYrD1xUxzdymqOzD2nqk127RQXPOhkVU\nMcxPJiDP8vcO1hXZmixZ/vx/v1Rqmk9FRDh81sSABHWyT1reSxutElSE9B2+LF8NRW2fWsHfX7UJ\nbjgicY6YOREfPnUBvn/+0XW/loHBUGCwss9tAA4TQhwBYA2AzwMAER0K4HwAiwEsBfATIrKJyAbw\nYwCvAXAogLf5x9YNqzd34bCZE2NJOAKKw9cVMavNVhy+TP5xy9+uKi2omj9b/nxNDpXkmHQAQSvC\nmOWfcC21WQuQnvjSW/6ywzecbGqJy4+cbxAZtzwJ26PA2rYswhde+zLTw9ZgzGBQso8Q4lbp5TIA\n/+7/fR6Aa4QQBQDPE9FaAMf7+9YKIdYDABFd4x/71GDGkYRi2cVx86bg1EUdAWmFmn/YmIUt/+ac\nDSjZwNGm4r7l7yiWf9auKdxQp/mHDcDDxuWh5U+RJjNJ5K8rz5BW7641Kogtf8ZAZR8Vf//YyanL\nZfAqLjsMlr2BwXjDUGr+7wfwR//vmfAmA8YmfxsAvKhsPyHphET0YQAfBoA5c+bUPKBcxsIP3+Yt\nw/+5eiuAMBpDLu9QdtnyV2Qf1fL393/zH89gX38Jt33qNPSWyuhoy6OavzTq8PWczICs+Yf/M6lm\nM5LmL0USJfG5rlxxWk5ObflL4ZX5SOXMoSHgw2ZOBFC5WTsjlM7GfdyCgcGQoyr5E9HtAGZodn1R\nCHGDf8wXAZQBXDWUgxNCXAngSgBYsmTJoOIomFwd12uDKCd59RbLKOs0f4XQWFZZ6cf1L/jCzQCA\ncw7fv6qFHXH4EqFUVi3/MFafO2KFztVoH99ky1+n+dfL8o9WzkzK8K0nuKnK/GkmWcnAoFZUJX8h\nxJmV9hPRewG8DsCrRBjkvBnAbOmwWf42VNheV4RVFz0Ck5O8+n09XdXMbSUeWqepA17N/2pOR1ma\nIAr16kDu8ffLtejZss3ahEIp7vDVjUNFWvLXdcjSHxda/rLMNFDNfzB45cHT8ev3HodTF+mbvPz2\n/cfjofWd2n0GBo2OQck+RLQUwOcAnCaE6JV23QjgaiL6HoADACwE8DC8KMeFRDQfHumfD+DtgxlD\nWmStqOzCoZ4tOTtIBFMdvlZCqKeKlpxd1bEqW/6Von28Gv4eiXN5ZlX2SSL0Fg35D32Gr2/521GH\n70iEOBIRXnnI9MT9py3qwGkJE4OBQaNjsJr/FQDyAG7zLeRlQogLhBBPEtG18By5ZQAXCiEcACCi\niwDcAsAG8CshxJODHEMqROPsyW/j6GK/CU3Y0OnNWyq5Zyw50DOZ4NJY/hlV81cyfGXnLsspXOsn\na1uK5l9d9uFmK2ml+PS1fcJon2alW5aBgcHYwWCjfQ6qsO/rAL6u2X4zgJsHc92BQCZuIj/U0xGY\n1pYPyF+tKW5ZUdknKSa9JZupWkBN1sRJF+0jTQLNvpzSL1n+ajcyHWQytojgCFHXaJ+I7GMibgwM\nxhQaxlxT4+yF38C9QypjzM3OGTYhtewjk6IOquXP/gY12ieXsTDTr7bIY8vYViTOPymZTNb8eRjV\nyJ9315oJrMb5D1Wop4GBwfBg3Jd3YGSU2jpc10fufztbIX+PmMP36UIpAV/2YYetbaHPjTeMly1j\nIkJJsuoBifxtC2e+bDr+913H4gxfz84omn9S8bCWbPh1eisWMeSav1rbR90+EMye0owXd/UN+P0G\nBga1o2HIP2L5I8yw5RaKADBrcjQ707JCy9gifTQNgEiGb9Ym+OX/I5DlIy/OX63qGU4CRISzF4fR\ntR75hxOKrgIpADTlZKcyX1d7aDgueGGvg9X8ByP73PHp001XKQODYUbjkL8VjbbhUEuZwFqVpiNy\nYTciSixC5pG/93eacgXaDF+WfzR+hYwdtfyTKkeqmr98/moYmOYvW/4Dl32SQmgNDAzqh4Z56iKW\nLQHFcjTUEojXwbEtKyjsZpFnlesifjh3QD7f1AoNqQkhgctlHYCwnk9k7JYV6RWQZCXryL+ecf6y\nj6SWhusGBgYjj4ax/FXNX26gfvScSZgxoSnm9LQplE1YtmnO2ig50SbtxbIbkGzGIvz6vcfhkP3b\nE8ciS0Cq5q9znKqSSpLmn1Gc2vxZK4F87/dA4vznTG3BY196NXb1FI31bmAwxtAw5J9TiFFuoH79\nR18BAPjX2p2R99iWFZK/v605Z2Nvf5T8501rDUjRqpJ4BCBSByjQ0NnyT5B9ZKRpvB1a/lUP9cdR\nm8OXj5/SmouFyBoYGIx+NIy5pmbYsoySl7fHZJ/QSrcky1/Gz9+9BEfNnhRa/ikcp1bNln90W5pu\nUfxR6qf5N8xPx8BgXKJhnuBIqCUQOFCzmagcJMOWMnz57So5s+5tKZp/Eo6dOzlyHVlD150fiE8o\nafrEBpZ/lfFw4lqtvgGT1GVgMLbRMLJPNMkrDJ2Ut6u8aytZuUCcJIO49xTJUmv++zWwLcKrv3dP\n7P1hVc/4+9UJJU1UZNJ4VVz3Hy/HrU9uS63ZV0tmMzAwGBtoGPKXyYpIsvwVOSjyHgqbv4SO3+h5\n5Q5c6nVUMMHK52BLmklfV7dHLZecFOcvw0oxGQHAy/afgJftP6Hq+RjqSsXAwGBsomFkHxlyYTXZ\n4lXJ34rIPnpyzyikncYilq+j5gfouFptU5hO8+dxVT+2FhjL38BgfKAhyZ8QJk3lIrKPQv5SPf8k\nB2oo+6S3iOWwUErxPrVNYSXN/9D9J6C9KVOzwzctjOVvYDA+0JDkL1v+asE3GYQwxJMoauEzapF9\n1Ovojo0Wkebj1GifZPK/6eMnY9UlZ6fW/GsFr1Bsk9RlYDCm0XBP8NLFMxSHbzzskkGEmOWv8jVb\n/mmjfbzzxq1nJnTd29WsYteNH6Oem+eLoeZotQS1gYHB2ETDOHwB4PlvvhYAcMq37wpKJEeifTRW\nsprhqx7D769FZtEdy8a8VvNXzpnO4ZvsQB4MMkbzNzAYF2go8ifJCcqav9ygRSVKAoH8eB/V8csI\nyDCw5tMVdgP0xdB0ZK0elybOn5T/hwrG8jcwGB9oKPJnECgoqZyt4PAlAqygsFtUTlHfYyn/V4LW\n8q9wvEq0tUT7pDm2FhjL38BgfKAhyT/SnSsS6hk9TjbCdYSdtePRQKks4gqav06lUTN809S+5/Ok\nWSXUAnb0pq0CamBgMDrRkE+wLK1klWqfypExzX/+tNZgr0yAcmG3aqio+WuEGmP5GxgYDDUalPzD\nvyvF+XvHRpOl/uucQ3HpeYsBxMtEA7XH+VcaG0O1smup7TPUHbKM5m9gMD7QmOQv/V0pw5co7uht\nytp4zeH7A9D7C9RsXB1qTcAaiOzz/pPnAQAOmNRc+cAakanhcxoYGIxeNKjmL8s+cknn6HGEaA/f\n4D2aypZB+YcUsg9pIoNOOnAqAOC0RR2x42MlnSvE+TPeetwcvPW4OdUPrBHG8jcwGB9oSPKPFlaT\nkry0ln88Xt4OWi/GVw1pSJGPkC3/Y+ZMxvpvvFYbLTQQy79eyCgZzQYGBmMTg5J9iOgyInqGiJ4g\nouuJaJK/fR4R9RHRSv/fz6T3HEtEq4hoLRH9kIY6CykFmKhzthUh9ZjsA7mBe7g9qG8TyQ72z1GL\n5q+QetJ7JzZnI68/eebCqteoF9SG8wYGBmMTg32CbwNwmBDiCABrAHxe2rdOCHGU/+8CaftPAXwI\nwEL/39JBjmHAUGvY68hXVyMna2tknxrkEObNtNKJ3Az+hW+dg6WH7Z/qffUAR0cZy9/AYGxjUOQv\nhLhVCMENbZcBmFXpeCLaH8AEIcQy4YWs/A7AGwYzhoGAiVytmVMpzl/eZVteCKhO9qmlpHNa63k0\n9chdOL0di/Zrw9ypLSM9FAMDg0FgKNfu7wfwD+n1fCJaQUT3ENEp/raZADZJx2zyt2lBRB8mouVE\ntHzHjh1DNlAmdbVlYizDF3Ij9Oi+rGVFZZ8ayL+W2v8AMLU1n+q44cCcqS249VOnYVrb6BmTgYFB\n7ajq8CWi2wHM0Oz6ohDiBv+YLwIoA7jK37cVwBwhRCcRHQvgr0S0uNbBCSGuBHAlACxZsmTIvJyh\n5W9pt8tI6uCVsSliuddU1VM6RxpMaG5Iv7yBgUEdUZVVhBBnVtpPRO8F8DoAr/KlHAghCgAK/t+P\nEtE6AIsAbEZUGprlbxtWMJHHNP9YtE+8kxcjY1G0HHRNSV7+e1Ja/iPgEzcwMBjnGGy0z1IAnwPw\neiFEr7S9g4hs/+8F8By764UQWwHsJaIT/SifdwO4YTBjGOC4AUSze4GkDN/o/4ysbUUtf84HGGSG\nr4GBgcFwYLB6whUA8gBu8wl1mR/ZcyqArxFRCYAL4AIhxC7/PR8F8BsAzfB8BP9QT1pvMOVmM9Ud\nvkmaf8amaFG4GqJ9dEleBgYGBsOJQZG/EOKghO3XAbguYd9yAIcN5rqDBfOzqvnr6vnzTBHr4GVZ\nkd66QXmHVBm+3v8tebuGURsYGBgMHRrSk0gJDl/tsdDrPlmbEqJ90jRz8f5vzaW//df9x0lYt6Mn\n9fEGBgYGldCQ5M/kq2r+KjzZJ/oexuwpLZg9uSVyLJAugoclpFos/2PnTsGxc6ekPt7AwMCgEhqS\n/NmaV6N94sfJhd2ipP7b9x0feV1LPX/HL7Jfi+VvYGBgMJRoSPYJk7zSxOT7hd2U7WpUTy3VLvv9\n/sEtOaP5GxgYjAwaMtwkKcNXd5yVYPmrqKW8Q1/Rq4jRmm/IudfAwGAUoCHJP6jqWUX2OWzmxGCm\nqKbm1ET+JQeAsfwNDAxGDg1J/kzkzdlk8r3rs6fjvKNmBnJPNfLnRUQ6y98jf6P5GxgYjBQakvzl\nloxJ4EbtQYavprG67pxpyL+/5Gv+Js7fwMBghNCQ5M9oyqaJyU9XgmEgso+x/A0MDEYKDUn+HGrZ\nlKlueaetvlNLtE9v4PA1lr+BgcHIoCHJv+iHWlaSfRhJhd1U1FLSOZB9jOVvYGAwQmhI8i8w+aeI\ntklbTrmWZi4MI/sYGBiMFBqU/D3NvalKqCdQ3eJnMOfXUqa52YR6GhgYjBAalPxrkH2QMs6/hvIO\njGp5BgYGBgb1QkOyT6FUu+ZfDUEnrxQlIxbt15bupAYGBgZ1QkOKzkXHI/9KSV4MK2Wcfy2F3f78\nHy9HV2+p6nEGBgYG9UJDkn/Bj7NPE+dfjfSD4wLNv/o5JzRlMaEpm+q8BgYGBvVAQ8o+bPkPZain\nXUOop4GBgcFIoyHJv+T4SV5pLP+Uov+CjjYs6GgNykIYGBgYjGY0pOzDyA9hhu/MSc248zOnD2o8\nBgYGBsOFhrT8GWni7GuI3DQwMDAYM2ho8k+j+dcSt29gYGAwVtDY5D+EGb4GBgYGYwmDJn8iupSI\nniCilUR0KxEd4G8nIvohEa319x8jvec9RPSc/+89gx3DQJHRtHH84duOxg/fdnTwOmzmYmYBAwOD\n8YOhcPheJoT4EgAQ0ccBfBnABQBeA2Ch/+8EAD8FcAIRTQHwFQBLAAgAjxLRjUKI3UMwlkHj9Uce\nEN1gSN/AwGAcYtCWvxBir/SyFR6hA8B5AH4nPCwDMImI9gdwNoDbhBC7fMK/DcDSwY6jXggzfA0M\nDAzGD4Yk1JOIvg7g3QC6ALzS3zwTwIvSYZv8bUnbhw2ffvUi3PfcjlTHps3wNTAwMBhLSGX5E9Ht\nRLRa8+88ABBCfFEIMRvAVQAuGqrBEdGHiWg5ES3fsSMdWafBx1+1EH+64OUpxxD938DAwGA8IJXl\nL4Q4M+X5rgJwMzxNfzOA2dK+Wf62zQBOV7bfnXDdKwFcCQBLliwRumPqDVOtwcDAYDxiKKJ9Fkov\nzwPwjP/3jQDe7Uf9nAigSwixFcAtAM4ioslENBnAWf62UQkj+xgYGIxHDIXm/y0iOhiAC2ADvEgf\nwFsBvBbAWgC9AN4HAEKIXUR0KYBH/OO+JoTYNQTjqA+Mw9fAwGAcYtDkL4T4t4TtAsCFCft+BeBX\ng732cMCQvoGBwXhEQ2f4pgGXdzBJXgYGBuMJhvyrwHC+gYHBeIQh/yow5G9gYDAeYci/CgLZZ4TH\nYWBgYDCUMORvYGBg0IAw5F8FFDh8R3ggBgYGBkMIQ/5VYDjfwMBgPMKQfxWYTl4GBgbjEYb8qyDk\nfjMJGBgYjB8Y8q8CQ/kGBgbjEYb8q8A4fA0MDMYjDPlXgSF9AwOD8QhD/lUgRqSLgIGBgUF9Ycg/\nJcwCwMDAYDzBkL+BgYFBA8KQf1V4uo/R/g0MDMYTDPlXgdH8DQwMxiMM+aeE6eVrYGAwnmDI38DA\nwKABYci/CozqY2BgMB5hyD8ljMPXwMBgPMGQv4GBgUEDwpB/FZhoHwMDg/EIQ/5VIEycv4GBwTjE\noMifiC4loieIaCUR3UpEB/jbTyeiLn/7SiL6svSepUT0LBGtJaKLB/sBDAwMDAxqx2At/8uEEEcI\nIY4C8HcAX5b23SeEOMr/9zUAICIbwI8BvAbAoQDeRkSHDnIMwwIT529gYDCeMCjyF0LslV62onpk\n5PEA1goh/n979xsjR13Hcfz98a49jBCxtIGmbaBtGgkYrU0lEAkY/NdW42nsg0tI5IEJodpEY4y2\naaL4gERN/I9KUCuo1aL4ryElCraJj2wp0JarUDihpjSVqxhQEwPWfn0w36PDZfeu172d2e58Xsnm\nZn4zd/O57+5+b+e3c3dPR8TLwHZguJMM3eY5fzPrRx3P+Uu6TdJR4EZe/cr/GkkHJN0v6cocWwQc\nLe3zbI61+9o3S9onad+JEyc6jdoZv/A3sz4ybfOX9KCk0Ra3YYCI2BIRS4BtwMb8tEeASyPiLcC3\ngN+cTbiIuDMiVkfE6gULFpzNlzAzsxYGp9shIt51hl9rG7AT+Hx5Oigidkr6jqT5wDFgSelzFudY\nz/Ksj5n1o06v9llRWh0GnsjxS5T//FbSVXmc54GHgBWSlkqaC4wAOzrJUBXP+phZP5n2lf80vijp\njcAp4K/ALTm+Htgg6STwH2AkIgI4KWkj8DtgANgaEYc6zGBmZjPUUfOPiA+3Gb8duL3Ntp0U00Pn\nhPDlPmbWh/wbvmdI/hVfM+sjbv5nyK3fzPqJm7+ZWQO5+U/DU/5m1o/c/KcxOFBM+AwNulRm1j86\nvdSz76258hJuuX45G65fXncUM7NZ4+Y/jcGB17Bp7eV1xzAzm1WeyzAzayA3fzOzBnLzNzNrIDd/\nM7MGcvM3M2sgN38zswZy8zczayA3fzOzBtK58vfqJZ2g+IcxZ2M+8PdZjDNbnGtmnGtmejUX9G62\nfst1aUS0/Afo50zz74SkfRGxuu4ckznXzDjXzPRqLujdbE3K5WkfM7MGcvM3M2ugpjT/O+sO0IZz\nzYxzzUyv5oLezdaYXI2Y8zczs1dryit/MzMrcfM3M2ugvm7+ktZIOixpTNKmmrMckfSYpP2S9uXY\nPEkPSHoqP76hoixbJY1LGi2Ntcyiwjezhgclrao4162SjmXd9ktaV9q2OXMdlvTeLuZaImm3pD9L\nOiTpEzlea82myFVrzSSdJ2mvpAOZ6ws5vlTSnjz+PZLm5vhQro/l9ssqznWXpGdK9VqZ45U99vN4\nA5IelXRfrne3XhHRlzdgAPgLsAyYCxwArqgxzxFg/qSxLwObcnkT8KWKslwHrAJGp8sCrAPuBwRc\nDeypONetwKdb7HtF3qdDwNK8rwe6lGshsCqXLwCezOPXWrMpctVas/y+z8/lOcCerMPPgZEcvwPY\nkMsfA+7I5RHgni7Vq12uu4D1Lfav7LGfx/sU8FPgvlzvar36+ZX/VcBYRDwdES8D24HhmjNNNgzc\nnct3Ax+s4qAR8UfgH2eYZRj4URT+BFwoaWGFudoZBrZHxEsR8QwwRnGfdyPX8Yh4JJf/BTwOLKLm\nmk2Rq51Kapbf979zdU7eArgBuDfHJ9droo73Au+UpApztVPZY1/SYuB9wPdzXXS5Xv3c/BcBR0vr\nzzL1E6PbAvi9pIcl3ZxjF0fE8Vz+G3BxPdGmzNILddyYp91bS1NjteTKU+y3Urxq7JmaTcoFNdcs\npzD2A+PAAxRnGS9ExMkWx34lV25/EbioilwRMVGv27JeX5M0NDlXi8yz7evAZ4BTuX4RXa5XPzf/\nXnNtRKwC1gIfl3RdeWMU53A9cd1tL2UBvgssB1YCx4Gv1BVE0vnAL4FPRsQ/y9vqrFmLXLXXLCL+\nFxErgcUUZxeXV52hlcm5JL0J2EyR723APOCzVWaS9H5gPCIervK4/dz8jwFLSuuLc6wWEXEsP44D\nv6Z4Qjw3cRqZH8fryjdFllrrGBHP5RP2FPA9Tk9TVJpL0hyKBrstIn6Vw7XXrFWuXqlZZnkB2A1c\nQzFtMtji2K+NTZ6EAAABiElEQVTkyu2vB56vKNeanD6LiHgJ+CHV1+vtwAckHaGYnr4B+AZdrlc/\nN/+HgBX5jvlcijdGdtQRRNLrJF0wsQy8BxjNPDflbjcBv60jX2qXZQfwkbzy4WrgxdJUR9dNmmP9\nEEXdJnKN5JUPS4EVwN4uZRDwA+DxiPhqaVOtNWuXq+6aSVog6cJcfi3wbor3I3YD63O3yfWaqON6\nYFeeSVWR64nSD3BRzKuX69X1+zEiNkfE4oi4jKJP7YqIG+l2vWbz3epeu1G8W/8kxXzjlhpzLKO4\nyuIAcGgiC8U83R+Ap4AHgXkV5fkZxXTAfynmEj/aLgvFlQ7fzho+BqyuONeP87gH80G/sLT/lsx1\nGFjbxVzXUkzpHAT2521d3TWbIletNQPeDDyaxx8FPld6HuyleKP5F8BQjp+X62O5fVnFuXZlvUaB\nn3D6iqDKHvuljO/g9NU+Xa2X/7yDmVkD9fO0j5mZteHmb2bWQG7+ZmYN5OZvZtZAbv5mZg3k5m9m\n1kBu/mZmDfR/eLSd+Qi+jrwAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mV5jj4dThz0Y",
        "colab_type": "text"
      },
      "source": [
        "另外，`avg_final_reward` 代表的是多個回合的平均 final rewards，而 final reward 即是 agent 在單一回合中拿到的最後一個 reward。\n",
        "如果同學們還記得環境給予登月小艇 reward 的方式，便會知道，不論**回合的最後**小艇是不幸墜毀、飛出畫面、或是靜止在地面上，都會受到額外地獎勵或處罰。\n",
        "也因此，final reward 可被用來觀察 agent 的「著地」是否順利等資訊。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "txDZ5vlGWz5w",
        "colab_type": "code",
        "outputId": "ffbecb9d-e408-4cbc-8bed-4c1bddffc8c9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        }
      },
      "source": [
        "plt.plot(avg_final_rewards)\n",
        "plt.title(\"Final Rewards\")\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEICAYAAABF82P+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAeuElEQVR4nO3de5QdZZnv8e8vRBC5hUuDkSQENRwX\nikRpI+M4gjEKBD2BEZwwjuIFMoKso7jkCAeXC+bIHEU8GHWAFcNVGUFAJSqOgje8ELCDISSIJFw8\nIQkQQUAUgpDn/FFvJ9Vt7d3p7tR+Kzu/j2vbtd+6PV1N9rPfS9WriMDMzGywMbkDMDOzZnKCMDOz\nSk4QZmZWyQnCzMwqOUGYmVklJwgzM6vkBGFdS9IkSU9J2mYzHOsySZ/eHHHlImmypJA0NncstmVw\ngrAtnqQHJD2dkkH/6yUR8f8iYseIeL7m879P0vPpvE9KukPS2+s8p1knOEFYt3hHSgb9r9UdPv8t\nEbEjMA64ALhK0rgOx7CBawm2OThBWNca3KQi6aeS/rekX0r6k6QfStqjtP01kh6S9ISkmyW9crjn\njIj1wFeBHYAppWMfLOlXkh5PNYxDU/mbJd1Z2u5GSb8uvf+5pKPS8umS7k2x3yXp6NJ270u/1/mS\nHgXOkrSNpPMk/UHSfcCRg67P+yTdl453v6R3D/f3te7mBGFbm38G3g/sCWwLfLy07vsUH+p7ArcD\nVw734Km/4/3AX4Hfp7K9ge8BnwZ2S+e8TlIPsBCYImkPSS8AXg28RNJOkrYHeoGfp8PfC/wDsAtw\nNvA1SeNLp389cB+wF3AOcCLwduA16TjHlOLcAfgicERE7AS8AVg83N/XupsThHWLb6dv549L+nab\n7S6NiHsi4mngG8DU/hURcUlE/Cki1gFnAQdK2mUTz3+wpMeBZ4DzgH+JiEfSun8BboiIGyJifUTc\nCPQBM1McvwbeBBwE3AH8Evh74GBgeUQ8muK7JiJWp2NcDSwHppViWB0RX4qI59Jx3wV8ISJWRsRj\nwP8ZFPN64FWSto+INRGxbBN/V9tKOEFYtzgqIsal11FttnuotPwXYEcovvlL+kxqwnkSeCBtsweb\nZmFEjAN2BRZQfNPvtw9wbCmBPQ68Eej/9v8z4FCKJPEz4KfAIen1s/6DSHqvpMWlY7xqUHwrB8X0\nkkFlv+9fiIg/A/8EfAhYI+l7kl6xib+rbSWcIMwK/wzMAmZQNOFMTuUazkEi4ingJOA9kl6TilcC\nXy0lsHERsUNEfCatH5wgfsagBCFpH+ArwCnA7ikZLR0U3+BHM68BJpbeTxoU6w8i4q0UierudHyz\nDZwgzAo7AeuAR4EXAf8+0gOl5pz5wKdS0deAd0g6LNVUXijpUEkT0vpfAf+NornottTUsw9Fn8LN\naZsdKBLAWgBJ76eoQbTzDeB/SJogaVfg9P4VkvaSNCv1RawDnqJocjLbwAnCrHAFRRPMKuAuis7j\n0fgCMFPSqyNiJUXt5H9RfMCvBE4j/ftLzT23A8si4tm0/y3A7/v7MSLiLuDzqfxh4ACKvop2vgL8\ngKJf43bgm6V1Y4CPAauBxyhqKyeN4ve1LiRPGGRmZlVcgzAzs0pOEGZmVskJwszMKjlBmJlZpa56\noNcee+wRkydPzh2GmdkWZdGiRX+IiJ7B5V2VICZPnkxfX1/uMMzMtiiSfl9V7iYmMzOr5ARhZmaV\nnCDMzKySE4SZmVVygjAzs0pOEGZmVskJwszMKjlBmJk13K33Pcryh//U8fM6QZiZNdyZ317KBT+9\nt+PnrS1BSJoqaWGaQ7dP0rRULklflLRC0hJJr22x/0GS7kzbfVHSsKZ+NDPrFs89v56/Pt/5Cf/q\nrEGcC5wdEVMppl48N5UfAUxJrznAhS32vxA4sbTt4TXGambWWMHfTjjeCXUmiAB2Tsu7UExtCMXU\ni1dEYSEwTtL48o7p/c4RsTCKKe+uAI6qMVYzs8aKTBmizof1fRT4gaTzKBLRG1L53hRz8vZ7MJWt\nKZXtncoHb/M3JM2hqIkwadKkzRK4mVmTRPpfp40qQUi6CXhxxaozgbcAp0bEdZLeBVwMzBjN+apE\nxDxgHkBvb68n2DazrhORahEdNqoEEREtP/AlXQF8JL29BpifllcBE0ubTkhlZatSebttzMy2CrkS\nRJ19EKuBQ9LydGB5Wl4AvDeNZjoYeCIiys1LpPdPSjo4jV56L3B9jbG2tPZP63j4yWdynNrMbIMt\nrolpCCcCcyWNBZ4h9RMANwAzgRXAX4D39+8gaXEa9QRwMnAZsD3w/fTquLMWLOPxp5/lyhMOznF6\nMzMiYstrYmonIn4BHFRRHsCHW+wztbTcB7yqrvg21ZPP/JWnnnkudxhmthXrxmGuXcM932aWUzf2\nQXSFXH8YM7N+kakO4QQxhFzjj83M+rkG0VCuQZhZbu6DaCgnCDPLrfgcchNT47iByczyy/M55AQx\nhFyZ28ysn/sgGsqpwcxycx9EU7kPwswyK+6kdh9E43iYq5nllusTyAliCB7FZGa5uQ+ioXK1/ZmZ\n9YvI05LhBDGE9Zna/szM+gWuQTRShGsQZpaZm5iaKVfmNjPrVzR1d0kTk6SpkhZKWiypT9K0VP5u\nSUsk3SnpV5IObLH/ZZLuT/svljS1aruOcBOTmWXWbRMGnQucHRHflzQzvT8UuB84JCL+KOkIYB7w\n+hbHOC0irq0pvk3mTmozyy3X51BdCSKAndPyLhTzUxMRvyptsxCYUNP5NxsPczWz3CJThqirD+Kj\nwOckrQTOA86o2OaDtJ9n+pzUHHW+pO1abSRpTmrG6lu7du3ooq7gG+XMLLdcn0MjThCSbpK0tOI1\nCzgJODUiJgKnAhcP2vfNFAniEy0OfwbwCuB1wG5ttiMi5kVEb0T09vT0jPTXack1CDPLLdfn0Iib\nmCJiRqt1kq4APpLeXgPML617dXp/REQ82uLYa9LiOkmXAh8faZyj5QRhZrl128P6VgOHpOXpwHIA\nSZOAbwLviYh7Wu0saXz6KeAoYGlNcQ7JucHMsss07UBdndQnAnMljQWeAeak8k8BuwMXFJ/9PBcR\nvQCSbgBOiIjVwJWSegABi4EP1RTnkHI9RdHMrF+untBaEkRE/AI4qKL8BOCEFvvMLC1PryOukXJ6\nMLOc/LC+hnIfhJnl1m19EF3Dw1zNLLfI9E3VCWIIrkGYWW6uQTSUH7VhZrm5D6Khcj0ky8ysbIu6\nk3prEaX/NzPrtP5h9q5BNJH7IMwso/7PHyeIBnIfhJnlFIN+dpITxBB8J7WZ5bSxicl9EI3jGoSZ\n5ZTz88cJYgi+D8LMcnIfRIMFbmIys3xiw1hKNzE1ToSbmMwsH9cgGizXXLBmZmUexdRQzg9mlsvG\nGkQXNTFJmippoaTFkvokTUvlh0p6IpUvlvSpFvvvK+lWSSskXS1p27pibcfDXM0sp8j4PIc6axDn\nAmdHxFSKmeTOLa37eURMTa9/a7H/Z4HzI+LlwB+BD9YYa0tuYTKznCLjnXJ1JogAdk7Lu1DMU71J\n0lzU04FrU9HlFHNTd5yHuZpZE3TNlKPJR4EfSDqPIhG9obTu7yTdQZE0Ph4RywbtuzvweEQ8l94/\nCOxddRJJc0hzXk+aNGkzhl9YH54wyMzy2VCByPBNdVQJQtJNwIsrVp0JvAU4NSKuk/Qu4GJgBnA7\nsE9EPCVpJvBtYMpIY4iIecA8gN7e3s1+BQPXIMwsnw2P2shw7lEliIiY0WqdpCuAj6S31wDz0z5P\nlva/QdIFkvaIiD+Udn8UGCdpbKpFTABWjSbWkfJ9EGaW08YaROfPXWcfxGrgkLQ8HVgOIOnFqY+B\nNLJpDEVC2CCKlPkT4JhUdDxwfY2xtuEMYWb5bBjmmuGDqM4+iBOBuZLGAs+Q+gkoPvRPkvQc8DQw\nOyUEJN0AnBARq4FPAFdJ+jTwG4omqo4rahDOEGaWScY7qWtLEBHxC+CgivIvA19usc/M0vJ9wLS6\n4ttU7oMws5w23AfRZU1MXSHC9QczyyfnF1QniCEUNQinCDPLI+cwVyeIIUTAeucHM8sk5zBXJ4gh\nuPZgZjl16zDXrpCzemdmlnOYqxPEUDIOMTMz8yimBsv4IEUzs41fUjOc2gliCBs6iFyFMLMM3AfR\nYK5BmFlOGxOD+yAaJ+eE4WZm7oNosI3T/TlDmFnnhfsgmss1CDPLyXdSN5jzgpnl5Dupm8w1CDPL\nKGcrhhPEENwHYWZN0DVNTJKmSlooabGkvjRzHJJOS2WLJS2V9Lyk3Sr2v0zS/aVtp9YR56ZwH4SZ\n5ZSzk7quCYPOBc6OiO9LmpneHxoRnwM+ByDpHcCpEfFYi2OcFhHX1hTfJvN9EGaWU2S8lbquJqYA\ndk7Lu1DMTz3YccDXazr/ZuM7qc0sp24c5vpR4HOSVgLnAWeUV0p6EXA4cF2bY5wjaYmk8yVtV1Oc\nQ3INwsxy2iKHuUq6KfUjDH7NAk6iaD6aCJwKXDxo93cAv2zTvHQG8ArgdcBuwCfaxDEn9XP0rV27\ndqS/TkvugzCznHK2Xoy4DyIiZrRaJ+kK4CPp7TXA/EGbzKZN81JErEmL6yRdCny8zbbzgHkAvb29\n9V1JJwgzyyBnK0ZdTUyrgUPS8nRgef8KSbukdde32lnS+PRTwFHA0pribKucuT3M1cxyyNmKUdco\nphOBuZLGAs8Ac0rrjgZ+GBF/Lu8g6QbghIhYDVwpqQcQsBj4UE1xtlX+g7iJyczyyHcvVi0JIiJ+\nARzUYt1lwGUV5TNLy9PriGu4osWymVmn+E7qhlpfbmJyFcLMMujGPoiuMKCJKV8YZrYVi4wZwgmi\njXKbnysQZpZDzufBOUG0MbAG4QxhZp3nPogtgfODmWXQjY/a6ArugzCz3DbOSe0mpkZxH4SZ5eYa\nREO5D8LMmsJ9EA0z4EY55wczyyDnZ48TRBsDn8VkZtZ5A5u6O/tJ5ATRxsAahFOEmXVezmfCOUG0\n4Yf1mVluOZ8J5wTRjpOCmWUWGZ8J5wTRhoe5mllurkE0VDkprHeGMLMM3AfRUJ4PwszyyzezZW0J\nQtKBkm6RdKek70jaubTuDEkrJP1O0mEt9t9X0q1pu6slbVtXrK3kbPszM4PurUHMB06PiAOAbwGn\nAUjaH5gNvBI4HLhA0jYV+38WOD8iXg78EfhgjbFWcg3CzHLL+dlTZ4LYD7g5Ld8IvDMtzwKuioh1\nEXE/sAKYVt5RkoDpwLWp6HLgqBpjreRhrmaWW7fWIJZRJAOAY4GJaXlvYGVpuwdTWdnuwOMR8Vyb\nbQCQNEdSn6S+tWvXbpbA+4XrEGaW2cAnOmxBfRCSbpK0tOI1C/gAcLKkRcBOwLObI+DBImJeRPRG\nRG9PT89mPnj5PJv30GZmmyLnM+HGjmbniJgxxCZvA5C0H3BkKlvFxtoEwIRUVvYoME7S2FSLqNqm\ndq4/mFluOeelqXMU057p5xjgk8BFadUCYLak7STtC0wBbivvG0Wd6ifAManoeOD6umJtxX0QZpZb\ntz6s7zhJ9wB3A6uBSwEiYhnwDeAu4L+AD0fE8wCSbpD0krT/J4CPSVpB0SdxcY2xVoqM44/NzICB\nTd0dPvWompjaiYi5wNwW684Bzqkon1lavo9Bo5s6zTUIM8stZx+E76RuwxMGmVlukbEz1AmijZzD\ny8zMIG9TtxNEG25iMrPcuvVGOTMzGyU/7ruhXIMws9w8YVBDrXcfhJll5hpEQ3kUk5ll5z6IZho4\nisnMrPM8iqmhBtYgnCLMrPN8H0RD5XxIlpkZdOnD+rpDefRAxjDMbKvlR2001MA/hjOEmeXlPogG\n8SgmM8tt4H0QnT23E0Qb7oMws9x8H0RDhfsgzCyzgU906IImJkkHSrpF0p2SviNp51T+VkmLUvki\nSdNb7H+WpFWSFqfXzKrt6pbzD2NmVui+Jqb5wOkRcQDwLeC0VP4H4B2p/Hjgq22OcX5ETE2vG2qK\nsy03MZlZbjm/m9aVIPYDbk7LNwLvBIiI30TE6lS+DNhe0nY1xTBqbmIys9y6cZjrMmBWWj4WmFix\nzTuB2yNiXYtjnCJpiaRLJO3a6kSS5kjqk9S3du3a0UU9yMAahDOEmXVezs+hEScISTdJWlrxmgV8\nADhZ0iJgJ+DZQfu+Evgs8K8tDn8h8DJgKrAG+HyrOCJiXkT0RkRvT0/PSH+doTk/mFkGOVsyxo50\nx4iYMcQmbwOQtB9wZH+hpAkU/RLvjYh7Wxz74dL2XwG+O9I4R8N9EGaWW9c9akPSnunnGOCTwEXp\n/TjgexQd2L9ss//40tujgaV1xDkU90GYWW45HxpaVx/EcZLuAe4GVgOXpvJTgJcDnyoNYe1PJvMl\n9abtzk1DYZcAbwZOrSnOttwHYWa55Zx2YMRNTO1ExFxgbkX5p4FPt9jnhNLye+qIa7j8qA0za5Ju\nGcXUFTxhkJnllvOhoU4QbZT/FOtdhTCzDHL2hTpBtJFzJiczM+jCUUzdI99csGZmMPiZcJ09txNE\nGzn/MGZmMPhx3+6DaAyPYjKz3DxhUEP5Tmozy60bH9bXFQZmbqcIM8tgS3xY39bAg5jMLDcPc20o\nd1KbWW7dOGFQVwjXIcwsM/dBNJRrEGaW2xY5YdDWwKOYzCw390E0lOeDMLPc/KiNhvJ8EGaWW9dN\nGCTpQEm3pEl/viNp51Q+WdLTpcmCLmqx/26SbpS0PP3ctY44h+I7qc0su4zTDtRVg5hPMa3oARTz\nT59WWndvRExNrw+12P904EcRMQX4UXrfcZ4Pwsxy68ZRTPsBN6flG4F3DnP/WcDlafly4KjNFNew\n5KzamZlBd04YtIziQx7gWGBiad2+kn4j6WeS/qHF/ntFxJq0/BCwV6sTSZojqU9S39q1a0cd+ADO\nCWaW2Rb5sD5JN0laWvGaBXwAOFnSImAn4Nm02xpgUkS8BvgY8J/9/ROtRHF1Wl6WiJgXEb0R0dvT\n0zPSX6f62B7FZGaZ5bxdd+xId4yIGUNs8jYASfsBR6Z91gHr0vIiSfdSNEf1Ddr3YUnjI2KNpPHA\nIyONczQ8isnMcuu6CYMk7Zl+jgE+CVyU3vdI2iYtvxSYAtxXcYgFwPFp+Xjg+jriHIrvpDaz3Lpu\nmCtwnKR7gLuB1cClqfxNwBJJi4FrgQ9FxGMAkuZL6k3bfQZ4q6TlwIz0vuM8zNXMcss5mnLETUzt\nRMRcYG5F+XXAdS32OaG0/CjwljpiGw4PczWzJumKJqZu4WGuZpabH9bXUH5Yn5nlNiApuAbRJM4Q\nZpaXH9bXUB7mama5deOjNrqCRzGZWW7ug2go90GYWW6eMKih/KgNM8vNfRAN5T4IM2uSbrmTuiu4\nD8LMcst5w64TRBu+k9rMchvw5dR9EA3lKoSZZTAwP7iJqTE8isnMcuu6x313C49iMrPcPMy1ocp/\njPXOEGaWgYe5NpQnDDKz3LpxwqCukHMuWDMzYMC3066oQUg6UNItku6U9B1JO6fyd0taXHqtlzS1\nYv+zJK0qbTezjjiHMmCYq6sQZpZBNz6sbz5wekQcAHwLOA0gIq6MiKkRMRV4D3B/RCxucYzz+7eN\niBtqirMtpwQzy21gUuiOJqb9gJvT8o3AOyu2OQ64qqbzbxYDaxAZAzGzrVY3jmJaBsxKy8cCEyu2\n+Sfg622OcYqkJZIukbRrq40kzZHUJ6lv7dq1I4+4gp/FZGa5bZGjmCTdJGlpxWsW8AHgZEmLgJ2A\nZwft+3rgLxGxtMXhLwReBkwF1gCfbxVHRMyLiN6I6O3p6Rnpr1N97AHn2ayHNjPbJDk/h8aOdMeI\nmDHEJm8DkLQfcOSgdbNpU3uIiIf7lyV9BfjuCMMcFd9JbWa5dd2EQZL2TD/HAJ8ELiqtGwO8izb9\nD5LGl94eDbSqadTKd1KbWW7d2AdxnKR7gLuB1cClpXVvAlZGxH3lHSTNl9Sb3p6bhsguAd4MnFpT\nnG25D8LMssvYkjHiJqZ2ImIuMLfFup8CB1eUn1Bafk8dcQ2X+yDMLLecHz2+k7odZwUzyyznDbtO\nEG3kfAaKmRnk/Z7qBNGGH9ZnZrl146M2uoKnHDWz3LpumGu3cCe1meXWjcNcu4KHuZpZkzhBNIhr\nEGaW2xb5LKatgfsgzKxJPMy1qVyFMLMMcn5RdYJoww/rM7PcMs4X5ATRjh/WZ2a5eZhrQ3kUk5nl\n5mGuDeVRTGaWm0cxNZT7IMwsNz9qo6HcB2FmuW2xfRCSjpW0TNL60mQ//evOkLRC0u8kHVYqPzyV\nrZB0eovjbifp6rTNrZImjybOker/w0jugzCzXAIpLW1hNYilwD8CN5cLJe1PMe/0K4HDgQskbSNp\nG+A/gCOA/Slmntu/4rgfBP4YES8Hzgc+O8o4R2VMkSHMzDouIn0GsYXNKBcRvwVQf3rbaBZwVUSs\nA+6XtAKYltat6J9uVNJVadu7KvY/Ky1fC3xZkqKm2wi/9KPlLLhj9d+UP/bnZwEYI7i6byU/vvuR\nOk5vZtbSQ08+wxjB88AXf7ScK371QOV2//6PB/C6ybtt1nPXMuUosDewsPT+wVQGsHJQ+etb7L8S\nICKek/QEsDvwh8EbSpoDzAGYNGnSiILt2Wk7puy1Y+W6fffYgRdtO5Zlq58Y0bHNzEZjyl47ctA+\nu7Hqj0/z0JNPt9xu+xdss9nPPWSCkHQT8OKKVWdGxPWbPaJhioh5wDyA3t7eEdUwZk+bxOxpI0su\nZmbdasgEEREzRnDcVcDE0vsJqYw25VX7PyhpLLAL8OgI4jAzsxGqa5jrAmB2Go20LzAFuA34NTBF\n0r6StqXoyF7QYv/j0/IxwI/r6n8wM7Nqo+qDkHQ08CWgB/iepMURcVhELJP0DYrO5+eAD0fE82mf\nU4AfANsAl0TEslT+b0BfRCwALga+mjq3H6NIJGZm1kHqpi/mvb290dfXlzsMM7MtiqRFEdE7uNx3\nUpuZWSUnCDMzq+QEYWZmlZwgzMysUld1UktaC/x+hLvvQcWd2g3guIavqbE5ruFxXMMzmrj2iYie\nwYVdlSBGQ1JfVS9+bo5r+Joam+MaHsc1PHXE5SYmMzOr5ARhZmaVnCA2mpc7gBYc1/A1NTbHNTyO\na3g2e1zugzAzs0quQZiZWSUnCDMzq+QEAUg6XNLvJK2QdHrmWB6QdKekxZL6Utlukm6UtDz93LUD\ncVwi6RFJS0tllXGo8MV0/ZZIem2H4zpL0qp0zRZLmllad0aK63eSDqsxromSfiLpLknLJH0klWe9\nZm3iynrNJL1Q0m2S7khxnZ3K95V0azr/1WlaANLUAVen8lslTe5wXJdJur90vaam8o79t5/Ot42k\n30j6bnpf7/WKiK36RfHY8XuBlwLbAncA+2eM5wFgj0Fl5wKnp+XTgc92II43Aa8Flg4VBzAT+D4g\n4GDg1g7HdRbw8Ypt909/z+2AfdPfeZua4hoPvDYt7wTck86f9Zq1iSvrNUu/945p+QXArek6fAOY\nncovAk5KyycDF6Xl2cDVNV2vVnFdBhxTsX3H/ttP5/sY8J/Ad9P7Wq+XaxAwDVgREfdFxLPAVcCs\nzDENNgu4PC1fDhxV9wkj4maKuTg2JY5ZwBVRWAiMkzS+g3G1Mgu4KiLWRcT9wAqKv3cdca2JiNvT\n8p+A31LMrZ71mrWJq5WOXLP0ez+V3r4gvQKYDlybygdfr/7reC3wFknqYFytdOy/fUkTgCOB+em9\nqPl6OUEU/1hWlt4/SPt/QHUL4IeSFkmak8r2iog1afkhYK88obWMownX8JRUxb+k1ASXJa5UnX8N\nxbfPxlyzQXFB5muWmksWA48AN1LUVh6PiOcqzr0hrrT+CWD3TsQVEf3X65x0vc6XtN3guCpi3ty+\nAPxPYH16vzs1Xy8niOZ5Y0S8FjgC+LCkN5VXRlFnzD42uSlxJBcCLwOmAmuAz+cKRNKOwHXARyPi\nyfK6nNesIq7s1ywino+IqRRz008DXtHpGKoMjkvSq4AzKOJ7HbAb8IlOxiTp7cAjEbGok+d1goBV\nwMTS+wmpLIuIWJV+PgJ8i+IfzsP91db085FM4bWKI+s1jIiH0z/q9cBX2Ngk0tG4JL2A4kP4yoj4\nZirOfs2q4mrKNUuxPA78BPg7iiaa/qmQy+feEFdavwvwaIfiOjw11UVErAMupfPX6++B/y7pAYpm\n8OnAXGq+Xk4Q8GtgShoNsC1Fh86CHIFI2kHSTv3LwNuApSme49NmxwPX54ivTRwLgPemER0HA0+U\nmlVqN6jN92iKa9Yf1+w0omNfYApwW00xiGIu9d9GxP8trcp6zVrFlfuaSeqRNC4tbw+8laJ/5CfA\nMWmzwder/zoeA/w41cg6EdfdpSQvinb+8vWq/e8YEWdExISImEzxGfXjiHg3dV+vzdnDvqW+KEYi\n3EPRBnpmxjheSjGC5A5gWX8sFG2HPwKWAzcBu3Uglq9TND38laJt84Ot4qAYwfEf6frdCfR2OK6v\npvMuSf8wxpe2PzPF9TvgiBrjeiNF89ESYHF6zcx9zdrElfWaAa8GfpPOvxT4VOnfwG0UnePXANul\n8hem9yvS+pd2OK4fp+u1FPgaG0c6dey//VKMh7JxFFOt18uP2jAzs0puYjIzs0pOEGZmVskJwszM\nKjlBmJlZJScIMzOr5ARhZmaVnCDMzKzS/wcmREU80NDigAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u2HaGRVEYGQS",
        "colab_type": "text"
      },
      "source": [
        "## 測試"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5yFuUKKRYH73",
        "colab_type": "code",
        "outputId": "b18389e2-83e2-430a-90cb-8c34f3f93014",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "agent.network.eval()  # 測試前先將 network 切換為 evaluation 模式\n",
        "\n",
        "state = env.reset()\n",
        "\n",
        "img = plt.imshow(env.render(mode='rgb_array'))\n",
        "\n",
        "total_reward = 0\n",
        "\n",
        "done = False\n",
        "while not done:\n",
        "    action, _ = agent.sample(state)\n",
        "    state, reward, done, _ = env.step(action)\n",
        "\n",
        "    total_reward += reward\n",
        "\n",
        "    img.set_data(env.render(mode='rgb_array'))\n",
        "    display.display(plt.gcf())\n",
        "    display.clear_output(wait=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAATBklEQVR4nO3df6zdd33f8eerSQisoCYBN3JsZ4Hh\nDkXV5uC7EFQ0hUjQEJk5lRgNm4rF0C6TggRatC3ppOFq6x+VCmyIKpurZDgVS5rxY3GtVRBCJtY/\nSLAhhPwgxZSg2DUxbUggQkvn8N4f53Ozw/W177k/js/9nPN8SEfn+/18v99zPh/fk5c/ed/P1ydV\nhSSpH78w6Q5IklbG4JakzhjcktQZg1uSOmNwS1JnDG5J6szYgjvJtUmeSHIkyc3jeh9JmjUZxzru\nJOcAfw68DTgKfA14T1U9tu5vJkkzZlwz7iuBI1X1F1X1N8BdwO4xvZckzZRzx/S6W4CnhvaPAm86\n3clJvH1TkhapqizVPq7gXlaSeWB+Uu8vSb0aV3AfA7YN7W9tbS+pqn3APnDGLUkrMa4a99eA7Ule\nm+RlwA3AgTG9lyTNlLHMuKvqZJIPAl8AzgFur6pHx/FekjRrxrIccMWdsFQiSac43S8nvXNSkjpj\ncEtSZwxuSeqMwS1JnTG4JakzBrckdcbglqTOGNyS1BmDW5I6Y3BLUmcMbknqjMEtSZ0xuCWpMwa3\nJHXG4JakzhjcktQZg1uSOmNwS1Jn1vSdk0meBH4CvAicrKq5JBcBfwxcBjwJvLuqfrS2bkqSFqzH\njPutVbWjquba/s3AfVW1Hbiv7UuS1sk4SiW7gf1tez9w/RjeQ5Jm1lqDu4AvJjmcZL61XVxVx9v2\nD4CL1/gekqQha6pxA2+pqmNJfhm4N8m3hw9WVSWppS5sQT+/1DFJ0umlaslcXfkLJXuB54F/Dlxd\nVceTbAb+V1X93WWuXZ9OSNIUqaos1b7qUkmSX0zyqoVt4O3AI8ABYE87bQ9wz2rfQ5J0qlXPuJO8\nDvh82z0X+G9V9btJXg3cDVwKfJ/BcsBnlnktZ9yStMjpZtzrVipZC4Nbkk617qUSSdJkGNyS1BmD\nW5I6Y3BLUmcMbknqjMEtSZ0xuCWpMwa3JHXG4JakzhjcktQZg1uSOmNwS1JnDG5J6ozBLUmdMbgl\nqTMGtyR1xuCWpM4Y3JLUGYNbkjqzbHAnuT3JiSSPDLVdlOTeJN9pzxe29iT5RJIjSR5O8sZxdl6S\nZtEoM+5PAdcuarsZuK+qtgP3tX2AdwDb22MeuHV9uilJWrBscFfVV4BnFjXvBva37f3A9UPtd9TA\nV4ELkmxer85KklZf4764qo637R8AF7ftLcBTQ+cdbW2nSDKf5FCSQ6vsgyTNpHPX+gJVVUlqFdft\nA/YBrOZ6SZpVq51xP71QAmnPJ1r7MWDb0HlbW5skaZ2sNrgPAHva9h7gnqH297bVJVcBzw2VVCRJ\n6yBVZ65SJLkTuBp4DfA08BHgfwB3A5cC3wfeXVXPJAnwSQarUH4KvK+qlq1hWyqRpFNVVZZqXza4\nzwaDW5JOdbrg9s5JSeqMwS1JnTG4JakzBrckdcbglqTOGNyS1BmDW5I6Y3BLUmcMbknqjMEtSZ0x\nuCWpMwa3JHXG4JakzhjcktQZg1uSOmNwS1JnDG5J6ozBLUmdWTa4k9ye5ESSR4ba9iY5luSh9rhu\n6NgtSY4keSLJr4+r45I0q0b5suB/CDwP3FFVv9ra9gLPV9XvLzr3cuBO4ErgEuBLwK9U1YvLvIff\nOSlJi6z6Oyer6ivAMyO+z27grqp6oaq+BxxhEOKSpHWylhr3B5M83EopF7a2LcBTQ+ccbW2nSDKf\n5FCSQ2vogyTNnNUG963A3wF2AMeBj670BapqX1XNVdXcKvsgSTNpVcFdVU9X1YtV9TPgD/n/5ZBj\nwLahU7e2NknSOllVcCfZPLT7G8DCipMDwA1Jzk/yWmA78ODauihJGnbucickuRO4GnhNkqPAR4Cr\nk+wACngS+ABAVT2a5G7gMeAkcONyK0okSSuz7HLAs9IJlwNK0ilWvRxQkrSxGNyS1BmDW5I6Y3BL\nUmcMbknqjMEtSZ0xuCWpMwa3JHXG4JakzhjcktQZg1uSOmNwS1JnDG5J6ozBLUmdMbglqTMGtyR1\nxuCWpM4Y3JLUmWWDO8m2JPcneSzJo0k+1NovSnJvku+05wtbe5J8IsmRJA8neeO4ByFJs2SUGfdJ\n4Kaquhy4CrgxyeXAzcB9VbUduK/tA7yDwbe7bwfmgVvXvdeSNMOWDe6qOl5VX2/bPwEeB7YAu4H9\n7bT9wPVtezdwRw18FbggyeZ177kkzagV1biTXAZcATwAXFxVx9uhHwAXt+0twFNDlx1tbYtfaz7J\noSSHVthnSZppIwd3klcCnwU+XFU/Hj5WVQXUSt64qvZV1VxVza3kOkmadSMFd5LzGIT2p6vqc635\n6YUSSHs+0dqPAduGLt/a2iRJ62CUVSUBbgMer6qPDR06AOxp23uAe4ba39tWl1wFPDdUUpEkrVEG\nVY4znJC8BfjfwLeAn7Xm32ZQ574buBT4PvDuqnqmBf0ngWuBnwLvq6oz1rGTrKjMIkmzoKqyVPuy\nwX02GNySdKrTBbd3TkpSZwxuSeqMwS1JnTG4JakzBrckdcbglqTOGNyS1BmDW5I6Y3BLUmcMbknq\njMEtSZ0xuCWpMwa3JHXG4JakzhjcktQZg1uSOmNwS1JnDG5J6swoXxa8Lcn9SR5L8miSD7X2vUmO\nJXmoPa4buuaWJEeSPJHk18c5AEmaNaN8WfBmYHNVfT3Jq4DDwPXAu4Hnq+r3F51/OXAncCVwCfAl\n4Feq6sUzvIffOSlJi6z6Oyer6nhVfb1t/wR4HNhyhkt2A3dV1QtV9T3gCIMQlyStgxXVuJNcBlwB\nPNCaPpjk4SS3J7mwtW0Bnhq67ChnDnoJgKri0KFJ92Ly/DPQcs4d9cQkrwQ+C3y4qn6c5Fbg3wPV\nnj8K/LMVvN48ML+y7moWLBVcc3Nnvx+TdLrwnrU/By1tpOBOch6D0P50VX0OoKqeHjr+h8DBtnsM\n2DZ0+dbW9nOqah+wr11vjVtnZJAN+JeaYLRVJQFuAx6vqo8NtW8eOu03gEfa9gHghiTnJ3ktsB14\ncP26LEmzbZQZ968BvwV8K8lDre23gfck2cGgVPIk8AGAqno0yd3AY8BJ4MYzrSiRRuGscsA/B8EI\nywHPSicslYjBLycPH87Mh9OhQwa0Bk63HNDg1oZRVQwqcxq3XTt3AnDw8OEJ90Rnsup13JKm10KA\nqy8Gt9S5Xbv2rviahZm2M+4+jbyOW9LGsprAHmZo98vgljqxOKjnLhm6f20XHDz488c1vQxuaYM6\nY1BrprmqRBvGrK8qWWtQH/rLfc66p8zpVpU445bW4P63vx2At37xi2t6nYXQHses+qadO/mo9eyp\n4qoSaR0sBPgkzV0yf8qs/aa23O+mnTtf2lb/DG5J6ow1bm0Ys1zjXq9SyaG/3Af8/AoTSyX98s5J\naQYsDv5du/byxOZ3Tqg3Ghdn3NowZnnGDYOQXa9fTi7MvBdeb6mZuDY+Z9zSBrNr19413/14JgcP\n7j0lwHtRVS89dKoNEdw7d+70B6SZsjC7XmolyHpaHN7jfK/VGg7ppcJ6qeOznhcbIrgXzPoPQ7Pn\nbNw0887jf7Jhw3st/83PcoBvuBtwFv8gZrnmqem0nrXs05m7ZB52DUIbBs9/0toXjk2i3j2OoD3d\na05zdmy44F5s+IcyzT8IzYbh2e7i2fbBg3th1+rr0Quz6uHXe2e76eYDhw8Dh196/bMR3pOeDU9z\noC8b3EleDnwFOL+d/5mq+kj7IuC7gFcDh4Hfqqq/SXI+cAewE/hr4Der6sn16Kwhrmkwd8n8upRI\nlgrqxT6waP32wl8Opzt/rSYd1qPYqIG+uF9zZ/j+ulFm3C8A11TV80nOA/4syZ8C/xL4eFXdleQ/\nA+8Hbm3PP6qq1ye5Afg94DdXNZIzMMTVm7WWSEYJ6lGsZ2D3ENSj6mksywZ3DUbzfNs9rz0KuAb4\nJ619P7CXQXDvbtsAnwE+mSQ1xj8VQ1w9OdNse7hcsl5BvZ56CrdpNlKNO8k5DMohrwf+APgu8GxV\nnWynHAW2tO0twFMAVXUyyXMMyil/tY79Pi1DvF/T/vMaLlMsZ6P8E60G9cY0UnBX1YvAjiQXAJ8H\n3rDWN04yD8wDXHrppWt9uSUtfOimPRCmxWyFxEfW6RzNohWt466qZ4H7gTcDFyRZCP6twLG2fQzY\nBtCO/xKDX1Iufq19VTVXVXObNm1aZfdH7vdMr/ncqLyhQlqdZYM7yaY20ybJK4C3AY8zCPB3tdP2\nAPe07QNtn3b8y+Osb6+UYTFZ/tlLazdKqWQzsL/VuX8BuLuqDiZ5DLgryX8AvgHc1s6/DfijJEeA\nZ4AbxtDvdeMNP+NnSEvra5RVJQ8DVyzR/hfAlUu0/x/gH69L7yZglJAx3JdnWEvjs+HvnNyIDPdT\nGdTS2WNwj8lSQTaNYW5gS2efwX0WjRpyGz3gDWtpsgzuDWgjBrxhLW0cBnfHxhnwBrW0cRncM8AQ\nlqbLhvoGHEnS8gxuSeqMwS1JnTG4JakzBrckdcbglqTOGNyS1BmDW5I6Y3BLUmcMbknqjMEtSZ0x\nuCWpM6N8WfDLkzyY5JtJHk3yO639U0m+l+Sh9tjR2pPkE0mOJHk4yRvHPQhJmiWj/OuALwDXVNXz\nSc4D/izJn7Zj/6qqPrPo/HcA29vjTcCt7VmStA6WnXHXwPNt97z2ONO/E7obuKNd91XggiSb195V\nSRKMWONOck6Sh4ATwL1V9UA79LutHPLxJOe3ti3AU0OXH21tkqR1MFJwV9WLVbUD2ApcmeRXgVuA\nNwD/ALgI+DcreeMk80kOJTn0wx/+cIXdlqTZtaJVJVX1LHA/cG1VHW/lkBeA/wpc2U47Bmwbumxr\na1v8Wvuqaq6q5jZt2rS63kvSDBplVcmmJBe07VcAbwO+vVC3zuALDa8HHmmXHADe21aXXAU8V1XH\nx9J7SZpBo6wq2QzsT3IOg6C/u6oOJvlykk1AgIeAf9HO/5/AdcAR4KfA+9a/25I0u5YN7qp6GLhi\nifZrTnN+ATeuvWuSpKV456QkdcbglqTOGNyS1BmDW5I6Y3BLUmcMbknqjMEtSZ0xuCWpMwa3JHXG\n4JakzhjcktQZg1uSOmNwS1JnDG5J6ozBLUmdMbglqTMGtyR1xuCWpM4Y3JLUGYNbkjpjcEtSZwxu\nSepMqmrSfSDJT4AnJt2PMXkN8FeT7sQYTOu4YHrH5rj68reratNSB8492z05jSeqam7SnRiHJIem\ncWzTOi6Y3rE5rulhqUSSOmNwS1JnNkpw75t0B8ZoWsc2reOC6R2b45oSG+KXk5Kk0W2UGbckaUQT\nD+4k1yZ5IsmRJDdPuj8rleT2JCeSPDLUdlGSe5N8pz1f2NqT5BNtrA8neePken5mSbYluT/JY0ke\nTfKh1t712JK8PMmDSb7ZxvU7rf21SR5o/f/jJC9r7ee3/SPt+GWT7P9ykpyT5BtJDrb9aRnXk0m+\nleShJIdaW9efxbWYaHAnOQf4A+AdwOXAe5JcPsk+rcKngGsXtd0M3FdV24H72j4Mxrm9PeaBW89S\nH1fjJHBTVV0OXAXc2H42vY/tBeCaqvr7wA7g2iRXAb8HfLyqXg/8CHh/O//9wI9a+8fbeRvZh4DH\nh/anZVwAb62qHUNL/3r/LK5eVU3sAbwZ+MLQ/i3ALZPs0yrHcRnwyND+E8Dmtr2ZwTp1gP8CvGep\n8zb6A7gHeNs0jQ34W8DXgTcxuIHj3Nb+0ucS+ALw5rZ9bjsvk+77acazlUGAXQMcBDIN42p9fBJ4\nzaK2qfksrvQx6VLJFuCpof2jra13F1fV8bb9A+Ditt3leNv/Rl8BPMAUjK2VEx4CTgD3At8Fnq2q\nk+2U4b6/NK52/Dng1We3xyP7j8C/Bn7W9l/NdIwLoIAvJjmcZL61df9ZXK2Ncufk1KqqStLt0p0k\nrwQ+C3y4qn6c5KVjvY6tql4EdiS5APg88IYJd2nNkuwCTlTV4SRXT7o/Y/CWqjqW5JeBe5N8e/hg\nr5/F1Zr0jPsYsG1of2tr693TSTYDtOcTrb2r8SY5j0Fof7qqPteap2JsAFX1LHA/gxLCBUkWJjLD\nfX9pXO34LwF/fZa7OopfA/5RkieBuxiUS/4T/Y8LgKo61p5PMPjL9kqm6LO4UpMO7q8B29tvvl8G\n3AAcmHCf1sMBYE/b3sOgPrzQ/t72W++rgOeG/ldvQ8lgan0b8HhVfWzoUNdjS7KpzbRJ8goGdfvH\nGQT4u9ppi8e1MN53AV+uVjjdSKrqlqraWlWXMfjv6MtV9U/pfFwASX4xyasWtoG3A4/Q+WdxTSZd\nZAeuA/6cQZ3x3066P6vo/53AceD/MqilvZ9BrfA+4DvAl4CL2rlhsIrmu8C3gLlJ9/8M43oLg7ri\nw8BD7XFd72MD/h7wjTauR4B/19pfBzwIHAH+O3B+a3952z/Sjr9u0mMYYYxXAwenZVxtDN9sj0cX\ncqL3z+JaHt45KUmdmXSpRJK0Qga3JHXG4JakzhjcktQZg1uSOmNwS1JnDG5J6ozBLUmd+X9X5YVH\nswOMfAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Aex7mcKr0J01",
        "colab_type": "code",
        "outputId": "0ffbf013-226f-40f0-faae-735f48493ed3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(total_reward)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-86.50240373839777\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wUBtYXG2eaqf",
        "colab_type": "text"
      },
      "source": [
        "## 參考資料\n",
        "\n",
        "以下是一些有用的參考資料。\n",
        "建議同學們實做前，可以先參考第一則連結的上課影片。\n",
        "在影片的最後有提到兩個有用的 Tips，這對於本次作業的實做非常有幫助。\n",
        "\n",
        "- [DRL Lecture 1: Policy Gradient (Review)](https://youtu.be/z95ZYgPgXOY)\n",
        "- [ML Lecture 23-3: Reinforcement Learning (including Q-learning) start at 30:00](https://youtu.be/2-JNBzCq77c?t=1800)\n",
        "- [Lecture 7: Policy Gradient, David Silver](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf)\n"
      ]
    }
  ]
}