MVA_DL_3_TOULEMONT_MATTHIEU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DQN project MVA \n",
    "\n",
    "Student : Matthieu Toulemont"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You may need to install [OpenCV](https://pypi.python.org/pypi/opencv-python) and [scikit-video](http://www.scikit-video.org/stable/).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Toulemont/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import keras\n",
    "import numpy as np\n",
    "import io\n",
    "import base64\n",
    "from IPython.display import HTML, Video\n",
    "import skvideo.io\n",
    "import cv2\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from keras.models import Sequential,model_from_json\n",
    "from keras.layers.core import Dense, Flatten\n",
    "from keras.optimizers import sgd\n",
    "from keras.layers import Conv2D, MaxPooling2D, Activation, AveragePooling2D,Reshape,BatchNormalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MiniProject #3: Deep Reinforcement Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Notations__: $E_p$ is the expectation under probability $p$. Please justify each of your answer and widely comment your code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a reinforcement learning algorithm, we modelize each step $t$ as an action $a_t$ obtained from a state $s_t$, i.e. $\\{(a_{t},s_{t})_{t\\leq T}\\}$ having the Markov property. We consider a discount factor $\\gamma \\in [0,1]$ that ensures convergence. The goal is to find among all the policies $\\pi$, one that maximizes the expected reward:\n",
    "\n",
    "\\begin{equation*}\n",
    "R(\\pi)=\\sum_{t\\leq T}E_{p^{\\pi}}[\\gamma^t r(s_{t},a_{t})] \\> ,\n",
    "\\end{equation*}\n",
    "\n",
    "where: \n",
    "\\begin{equation*}p^{\\pi}(a_{0},a_{1},s_{1},...,a_{T},s_{T})=p(a_{0})\\prod_{t=1}^{T}\\pi(a_{t}|s_{t})p(s_{t+1}|s_{t},a_{t}) \\> .\n",
    "\\end{equation*}\n",
    "\n",
    "We note the $Q$-function:\n",
    "\n",
    "\\begin{equation*}Q^\\pi(s,a)=E_{p^{\\pi}}[\\sum_{t\\leq T}\\gamma^{t}r(s_{t},a_{t})|s_{0}=s,a_{0}=a] \\> .\n",
    "\\end{equation*}\n",
    "\n",
    "Thus, the optimal Q function is:\n",
    "\\begin{equation*}\n",
    "Q^*(s,a)=\\max_{\\pi}Q^\\pi(s,a) \\> .\n",
    "\\end{equation*}\n",
    "\n",
    "In this project, we will apply the deep reinforcement learning techniques to a simple game: an agent will have to learn from scratch a policy that will permit it maximizing a reward."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The environment, the agent and the game"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```Environment``` is an abstract class that represents the states, rewards, and actions to obtain the new state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Environment(object):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "\n",
    "    def act(self, act):\n",
    "        \"\"\"\n",
    "        One can act on the environment and obtain its reaction:\n",
    "        - the new state\n",
    "        - the reward of the new state\n",
    "        - should we continue the game?\n",
    "\n",
    "        :return: state, reward, game_over\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"\n",
    "        Reinitialize the environment to a random state and returns\n",
    "        the original state\n",
    "\n",
    "        :return: state\n",
    "        \"\"\"\n",
    "        pass\n",
    "    \n",
    "    def draw(self,e,folder):\n",
    "        \"\"\"\n",
    "        Visualize in the console or graphically the current state\n",
    "        \"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method ```act``` allows to act on the environment at a given state $s_t$ (stored internally), via action $a_t$. The method will return the new state $s_{t+1}$, the reward $r(s_{t},a_{t})$ and determines if $t\\leq T$ (*game_over*).\n",
    "\n",
    "The method ```reset``` simply reinitializes the environment to a random state $s_0$.\n",
    "\n",
    "The method ```draw``` displays the current state $s_t$ (this is useful to check the behavior of the Agent).\n",
    "\n",
    "We modelize $s_t$ as a tensor, while $a_t$ is an integer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of the ```Agent``` is to interact with the ```Environment``` by proposing actions $a_t$ obtained from a given state $s_t$ to attempt to maximize its __reward__ $r(s_t,a_t)$. We propose the following abstract class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Agent(object):\n",
    "    def __init__(self, epsilon=0.1, n_action=4):\n",
    "        self.epsilon = epsilon\n",
    "        self.n_action = n_action\n",
    "    \n",
    "    def set_epsilon(self,e):\n",
    "        self.epsilon = e\n",
    "\n",
    "    def act(self,s,train=True):\n",
    "        \"\"\" This function should return the next action to do:\n",
    "        an integer between 0 and 4 (not included) with a random exploration of epsilon\"\"\"\n",
    "        if train:\n",
    "            if np.random.rand() <= self.epsilon:\n",
    "                a = np.random.randint(0, self.n_action, size=1)[0]\n",
    "            else:\n",
    "                a = self.learned_act(s)\n",
    "        else: # in some cases, this can improve the performance.. remove it if poor performances\n",
    "            a = self.learned_act(s)\n",
    "\n",
    "        return a\n",
    "\n",
    "    def learned_act(self,s):\n",
    "        \"\"\" Act via the policy of the agent, from a given state s\n",
    "        it proposes an action a\"\"\"\n",
    "        pass\n",
    "\n",
    "    def reinforce(self, s, n_s, a, r, game_over_):\n",
    "        \"\"\" This function is the core of the learning algorithm. \n",
    "        It takes as an input the current state s_, the next state n_s_\n",
    "        the action a_ used to move from s_ to n_s_ and the reward r_.\n",
    "        \n",
    "        Its goal is to learn a policy.\n",
    "        \"\"\"\n",
    "        pass\n",
    "\n",
    "    def save(self):\n",
    "        \"\"\" This function returns basic stats if applicable: the\n",
    "        loss and/or the model\"\"\"\n",
    "        pass\n",
    "\n",
    "    def load(self):\n",
    "        \"\"\" This function allows to restore a model\"\"\"\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "__Question 1__:\n",
    "Explain the function act. Why is ```epsilon``` essential?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The function act returns the next state given a specific action and a given state. \n",
    "\n",
    "\n",
    ">$\\epsilon$ is the probability of exploration. Without it the agent would act solely on the knowledge it has accumulated so far. Even though this can lead to good reward, the agent might miss the opportunity of gaining better rewards via action or states it has never seen. \n",
    "\n",
    "\n",
    ">If $\\epsilon$ is close to 1, the agents never uses its knowledge and acts randomly. An intermediate valuee of $\\epsilon$ enables the agent to keep exploring and to use its knowledge to gain better rewards. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### The Game"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```Agent``` and the ```Environment``` work in an interlaced way as in the following (take some time to understand this code as it is the core of the project)\n",
    "\n",
    "```python\n",
    "\n",
    "epoch = 300\n",
    "env = Environment()\n",
    "agent = Agent()\n",
    "\n",
    "\n",
    "# Number of won games\n",
    "score = 0\n",
    "loss = 0\n",
    "\n",
    "\n",
    "for e in range(epoch):\n",
    "    # At each epoch, we restart to a fresh game and get the initial state\n",
    "    state = env.reset()\n",
    "    # This assumes that the games will end\n",
    "    game_over = False\n",
    "\n",
    "    win = 0\n",
    "    lose = 0\n",
    "    \n",
    "    while not game_over:\n",
    "        # The agent performs an action\n",
    "        action = agent.act(state)\n",
    "\n",
    "        # Apply an action to the environment, get the next state, the reward\n",
    "        # and if the games end\n",
    "        prev_state = state\n",
    "        state, reward, game_over = env.act(action)\n",
    "\n",
    "        # Update the counters\n",
    "        if reward > 0:\n",
    "            win = win + reward\n",
    "        if reward < 0:\n",
    "            lose = lose -reward\n",
    "\n",
    "        # Apply the reinforcement strategy\n",
    "        loss = agent.reinforce(prev_state, state,  action, reward, game_over)\n",
    "\n",
    "    # Save as a mp4\n",
    "    if e % 10 == 0:\n",
    "        env.draw(e)\n",
    "\n",
    "    # Update stats\n",
    "    score += win-lose\n",
    "\n",
    "    print(\"Epoch {:03d}/{:03d} | Loss {:.4f} | Win/lose count {}/{} ({})\"\n",
    "          .format(e, epoch, loss, win, lose, win-lose))\n",
    "    agent.save()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The game, *eat cheese*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A rat runs on an island and tries to eat as much as possible. The island is subdivided into $N\\times N$ cells, in which there are cheese (+0.5) and poisonous cells (-1). The rat has a visibility of 2 cells (thus it can see $5^2$ cells). The rat is given a time $T$ to accumulate as much food as possible. It can perform 4 actions: going up, down, left, right. \n",
    "\n",
    "The goal is to code an agent to solve this task that will learn by trial and error. We propose the following environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Environment(object):\n",
    "    def __init__(self, grid_size=10, max_time=500, temperature=0.1):\n",
    "        grid_size = grid_size+4\n",
    "        self.grid_size = grid_size\n",
    "        self.max_time = max_time\n",
    "        self.temperature = temperature\n",
    "\n",
    "        #board on which one plays\n",
    "        self.board = np.zeros((grid_size,grid_size))\n",
    "        self.position = np.zeros((grid_size,grid_size))\n",
    "\n",
    "        # coordinate of the cat\n",
    "        self.x = 0\n",
    "        self.y = 1\n",
    "\n",
    "        # self time\n",
    "        self.t = 0\n",
    "\n",
    "        self.scale=16\n",
    "\n",
    "        self.to_draw = np.zeros((max_time+2, grid_size*self.scale, grid_size*self.scale, 3))\n",
    "\n",
    "\n",
    "    def draw(self,e,folder):\n",
    "        if folder not in os.listdir():\n",
    "            os.mkdir(folder)\n",
    "        skvideo.io.vwrite(os.path.join(folder,str(e) + '.mp4'), self.to_draw)\n",
    "\n",
    "    def get_frame(self,t):\n",
    "        b = np.zeros((self.grid_size,self.grid_size,3))+128\n",
    "        b[self.board>0,0] = 256\n",
    "        b[self.board < 0, 2] = 256\n",
    "        b[self.x,self.y,:]=256\n",
    "        b[-2:,:,:]=0\n",
    "        b[:,-2:,:]=0\n",
    "        b[:2,:,:]=0\n",
    "        b[:,:2,:]=0\n",
    "        \n",
    "        b =  cv2.resize(b, None, fx=self.scale, fy=self.scale, interpolation=cv2.INTER_NEAREST)\n",
    "\n",
    "        self.to_draw[t,:,:,:]=b\n",
    "\n",
    "\n",
    "    def act(self, action):\n",
    "        \"\"\"This function returns the new state, reward and decides if the\n",
    "        game ends.\"\"\"\n",
    "\n",
    "        self.get_frame(int(self.t))\n",
    "\n",
    "        self.position = np.zeros((self.grid_size, self.grid_size))\n",
    "\n",
    "        self.position[0:2,:]= -1\n",
    "        self.position[:,0:2] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "\n",
    "        self.position[self.x, self.y] = 1\n",
    "        if action == 0:\n",
    "            if self.x == self.grid_size-3:\n",
    "                self.x = self.x-1\n",
    "            else:\n",
    "                self.x = self.x + 1\n",
    "        elif action == 1:\n",
    "            if self.x == 2:\n",
    "                self.x = self.x+1\n",
    "            else:\n",
    "                self.x = self.x-1\n",
    "        elif action == 2:\n",
    "            if self.y == self.grid_size - 3:\n",
    "                self.y = self.y - 1\n",
    "            else:\n",
    "                self.y = self.y + 1\n",
    "        elif action == 3:\n",
    "            if self.y == 2:\n",
    "                self.y = self.y + 1\n",
    "            else:\n",
    "                self.y = self.y - 1\n",
    "        else:\n",
    "            RuntimeError('Error: action not recognized')\n",
    "\n",
    "        self.t = self.t + 1\n",
    "        reward = self.board[self.x, self.y]\n",
    "        self.board[self.x, self.y] = 0\n",
    "        game_over = self.t > self.max_time\n",
    "        state = np.concatenate((self.board.reshape(self.grid_size, self.grid_size,1),\n",
    "                        self.position.reshape(self.grid_size, self.grid_size,1)),axis=2)\n",
    "        state = state[self.x-2:self.x+3,self.y-2:self.y+3,:]\n",
    "\n",
    "        return state, reward, game_over\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"This function resets the game and returns the initial state\"\"\"\n",
    "\n",
    "        self.x = np.random.randint(3, self.grid_size-3, size=1)[0]\n",
    "        self.y = np.random.randint(3, self.grid_size-3, size=1)[0]\n",
    "\n",
    "\n",
    "        bonus = 0.5*np.random.binomial(1,self.temperature,size=self.grid_size**2)\n",
    "        bonus = bonus.reshape(self.grid_size,self.grid_size)\n",
    "\n",
    "        malus = -1.0*np.random.binomial(1,self.temperature,size=self.grid_size**2)\n",
    "        malus = malus.reshape(self.grid_size, self.grid_size)\n",
    "\n",
    "        self.to_draw = np.zeros((self.max_time+2, self.grid_size*self.scale, self.grid_size*self.scale, 3))\n",
    "\n",
    "\n",
    "        malus[bonus>0]=0\n",
    "\n",
    "        self.board = bonus + malus\n",
    "\n",
    "        self.position = np.zeros((self.grid_size, self.grid_size))\n",
    "        self.position[0:2,:]= -1\n",
    "        self.position[:,0:2] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.board[self.x,self.y] = 0\n",
    "        self.t = 0\n",
    "\n",
    "        state = np.concatenate((\n",
    "                               self.board.reshape(self.grid_size, self.grid_size,1),\n",
    "                        self.position.reshape(self.grid_size, self.grid_size,1)),axis=2)\n",
    "\n",
    "        state = state[self.x - 2:self.x + 3, self.y - 2:self.y + 3, :]\n",
    "        return state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following elements are important because they correspond to the hyper parameters for this project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parameters\n",
    "size = 13\n",
    "T=200\n",
    "temperature=0.3\n",
    "epochs_train=10 # set small when debugging\n",
    "epochs_test=1 # set small when debugging\n",
    "\n",
    "# display videos\n",
    "def display_videos(name, prefix):\n",
    "    video = io.open(os.path.join(prefix,name), 'r+b').read()\n",
    "    encoded = base64.b64encode(video)\n",
    "    return '''<video alt=\"test\" controls>\n",
    "                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
    "             </video>'''.format(encoded.decode('ascii'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Question 2__ Explain the use of the arrays ```position``` and ```board```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The array **board** represents the reward with : \n",
    "<ul>\n",
    "    <li>**-1** for poison</li>\n",
    "    <li>**0.5** for food</li>\n",
    "    <li>**0** otherwise</li>\n",
    "</ul>\n",
    "\n",
    "Whereas **position** represents the location of the agent with : \n",
    "\n",
    "<ul>\n",
    "    <li>**-1** for outside walls</li>\n",
    "    <li>**0** Otherwise</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "__Question 3__ Implement a random Agent (only ```learned_act``` needs to be implemented):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RandomAgent(Agent):\n",
    "    def __init__(self):\n",
    "        super(RandomAgent, self).__init__()\n",
    "        pass\n",
    "\n",
    "    def learned_act(self, s):\n",
    "        return np.random.randint(0,self.n_action, size = 1)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "__Question 4__ Visualize the game moves. You need to fill in the following function for the evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(agent,env,epochs,prefix=''):\n",
    "    # Number of won games\n",
    "    score = 0\n",
    "    if prefix not in os.listdir():\n",
    "        os.mkdir(prefix)\n",
    "    for e in range(epochs):\n",
    "        \n",
    "        ##### FILL IN HERE\n",
    "        state = env.reset()\n",
    "        \n",
    "        win, lose, terminated = 0,0, False\n",
    "        \n",
    "        while not terminated:\n",
    "            action = agent.act(state, train=False)\n",
    "            next_state, reward, terminated = env.act(action)\n",
    "            state = next_state\n",
    "            if reward <=0:\n",
    "                lose = lose - reward\n",
    "            else:\n",
    "                win = win + reward\n",
    "        # Save as a mp4\n",
    "        env.draw(prefix+str(e), folder=prefix)\n",
    "        # Update stats\n",
    "        score = score + win-lose\n",
    "        if e%10==0:\n",
    "            print(\"Win/lose count {}/{}. Average score ({})\"\n",
    "                  .format(win, lose, score/(1+e)))\n",
    "    print('Final score: '+str(score/epochs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Win/lose count 5.0/11.0. Average score (-6.0)\n",
      "Final score: -6.0\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAF/xtZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1MiByMjg1NCBlOWE1OTAzIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNyAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MToweDExMSBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MCBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAAK5ZYiEADv//vb8/AptUwn/LZ/+iL/lb+9P2a61uFE7M7QacwPQC/3cd/Xi1bCrc27LcdG8bJkAU3Roif8hrvvgUoqkfgUzWsTzDZhGHbEeGtwo+34nAez8F521GIdohDtwirxPK3nfH5cGwRMMJWzQqm/QgcbWCkSKqSjevVnXi1Q0r8iKITUhclAHEKhh7uj8AMgY86FcQjBJEUR2Xj58q7ystkk170GogjNtuBms+lADzY9CZIyFT24HCQwRxz87+B4P3trltnKC9yTmmeSCZkN5I63Cd47Vde54f3PwJvJ4y0asnnCaieOdCDwS6LXD5r1zKtB3+Z4Cjsp0k224CfgNp4TdBsKC1sqpIqpYtbfKRceGdOD7bw1bWmzqDq9SePU1niTpRwSLYlym8scucoiOLsVYIbcsbIV0GkbXJX1xlpNKIpdybJIIn5Ww9+IEeFJn/X0XB5vJF3LWgBPsWRTyhgukt0gUxJAmtr4EUokgCyGLpWAwQqaKcndhAUw1phVocMsj4UwFgW2G5lVcnC968PjFI5NwtkUl8FkmAE9vfHYer+OtA1x1YoNaiPDhQztcNKxSyBg3KRHx/wuMbsWxj6QgSAdKMWOhWbYEM8aYzKvhbN8n5L/+WtNjNYb9+KiEWqZesPw1SZrCM2+vCWy3QAihGH4OMrg62A3GWKlLo5xw2c+P8CiruOqvqjhpYc8tO2tX4/vl7YrN1Jz1aOZbUcHgAGDYtondLuBf3M+J2PlvEROtHM2+HMla+Nh07qbpzGooAa3dMAf9gj7PCiwISDXNrgTrRRCjsCIdK6mXILpqeFDtA4xAWYkx+7aLjgpPEVFTH0APmwt0sLhzliW8/ygwWpXzkR+95MgZAOy1Z/KjHQiejGrFgCQPBOal8WtikJvWR3Kzs47eHA/gipBOKVXWdCAKiQAAABFBmiJsQ7/+qZYABlvaX9YCQAAAABABnkF5Cv8ACjxtd3klJmChAAAAHEGaRjwhkymEO//+qZYABAfjz+RekfVCyFMNlOAAAAASQZ5kalPC/wAE1n+6zE+zpnt7AAAAEAGeg3RCvwAGmAQuA/KAHeEAAAAQAZ6FakK/AAQXaITcZ9eyyQAAABxBmopJqEFomUwId//+qZYAApfyS/Ls9qFkKXU9AAAAEEGeqEURLC//AAMQq7v877AAAAAPAZ7HdEK/AAQW0YuA/PvAAAAAEAGeyWpCvwAEFk+c60MMRIEAAAAZQZrOSahBbJlMCHf//qmWAAKUE6P99pfekQAAABBBnuxFFSwv/wADEKu7/O+wAAAADwGfC3RCvwACs2jvPOO4gQAAAA8Bnw1qQr8ABBdiPJgevucAAAAgQZsSSahBbJlMCHf//qmWAAKX8kv0FuiefAprqQ7dr4EAAAAVQZ8wRRUsL/8AAxAjwwz0U6Zy2soQAAAAEAGfT3RCvwAEFdqTyvyU8ZAAAAAQAZ9RakK/AALpYR5LmfMPgQAAAB9Bm1VJqEFsmUwId//+qZYAAvHuxcyyz59uXQ3O8gyAAAAAEkGfc0UVLCv/AAS3aG/2G87M7gAAAA8Bn5RqQr8ABLdiPJgevr8AAAAaQZuYSahBbJlMCHf//qmWAAMBBZXGaX9sSMAAAAAPQZ+2RRUsK/8ABNZNw8JBAAAADwGf12pCvwAE2DWBdf5OwQAAABJBm9xJqEFsmUwIb//+p4QAAScAAAAMQZ/6RRUsL/8AALKBAAAADwGeGXRCvwAE+tHdHbfDxwAAABABnhtqQr8AB2zUOf5lvGBBAAAAKEGaAEmoQWyZTAhv//6nhAAGT9YbmWV4wz8CmWzs+BQpT7QeHL97W8EAAAAQQZ4+RRUsL/8AA7X/q5AZQAAAAA8Bnl10Qr8ABR8sGDZjiuEAAAAPAZ5fakK/AAUflA8mCXqBAAAAGkGaQUmoQWyZTAhv//6nhAAJqgCzbbTC+elAAAAAG0GaYknhClJlMCHf/qmWAATn5HQdvaDo6nmPgQAAABJBmoZJ4Q6JlMCHf/6plgAAlYAAAAAMQZ6kRRE8L/8AALKBAAAAEAGew3RCvwAHhWAYjsuzYIEAAAAQAZ7FakK/AAeFYBvZ4+5RgQAAAB1BmspJqEFomUwIb//+p4QADo++zH+JrjVEHZeXswAAABVBnuhFESwv/wAI7jx0zikg1JsClrAAAAAPAZ8HdEK/AAxElm4NkvMXAAAADwGfCWpCvwAMQRfM2zI3RwAAABlBmw1JqEFsmUwIb//+p4QADjewevZnwRbvAAAAEkGfK0UVLCv/ABHenXd39ItJgAAAAA4Bn0xqQr8AEdlddx4KTQAAABpBm1BJqEFsmUwIb//+p4QADd++zH+H1befgQAAABJBn25FFSwr/wALW2AIBTAOh8EAAAAOAZ+PakK/AAtfKVdTqC0AAAAgQZuUSahBbJlMCGf//p4QAHn9ff1Iz/550jYLZpfxq6AAAAASQZ+yRRUsL/8AEtz9y14XcZdBAAAAEAGf0XRCvwAQ4QBztjjTbqAAAAAPAZ/TakK/ABnErYwrNvZAAAAAGUGb1UmoQWyZTAhn//6eEABP/dNjLk2VcpUAAAAZQZv2SeEKUmUwIb/+p4QAE/91P1HGhIdUwAAAABlBmhdJ4Q6JlMCG//6nhAAM77B/hOC3Qq7BAAAAHUGaOUnhDyZTBRE8N//+p4QACDfHT7rSzNTbouVZAAAAEAGeWGpCvwAG6Zua48VbeOAAAAAbQZpdSeEPJlMCG//+p4QABY/dT91pZmpt0XbJAAAAEEGee0URPC//AANMq8b2EXgAAAAOAZ6adEK/AASXcd55xq8AAAAQAZ6cakK/AASXNG80xVuSwQAAABlBmp9JqEFomUwU8N/+p4QABYgUHt7qfttkAAAAEAGevmpCvwAEd2iE3GfXsegAAAAZQZqgSeEKUmUwId/+qZYAAtullcZpf2xMwQAAAB9BmsJJ4Q6JlMFNEw7//qmWAASAo6IFmgO77VQY/3dSAAAAEAGe4WpCvwAHQZ4Q8aGtCoEAAAAYQZrmSeEPJlMCHf/+qZYABIfjz+XaM3OpAAAAEEGfBEURPC//AAVmgQUobxkAAAAPAZ8jdEK/AAtfQDoTkyghAAAADwGfJWpCvwAHQB/VIoEsRwAAABdBmypJqEFomUwId//+qZYABGfjz+TLwQAAAA5Bn0hFESwv/wAFQZU1IAAAABABn2d0Qr8ABxbFYvP4HOzAAAAAEAGfaWpCvwAHFNQ5/mW8ZcEAAAASQZtuSahBbJlMCG///qeEAAEnAAAAFEGfjEUVLC//AANXuF5r44+iylf4AAAADwGfq3RCvwAEldlCk2yWrwAAAA8Bn61qQr8ABJdiPJgevscAAAAaQZuvSahBbJlMCHf//qmWAALtpZXGaX9sSsEAAAAaQZvTSeEKUmUwId/+qZYAAvHvq+9FBpG4QrgAAAAVQZ/xRTRML/8AA3QjjdU6XIkc7f7oAAAAEAGeEHRCvwAE13HeVsofHYEAAAAPAZ4SakK/AAMkRfM2zI8dAAAAGUGaF0moQWiZTAhv//6nhAAF0EMu7fYP2L8AAAAQQZ41RREsL/8AA3Sru/ztOQAAAA8BnlR0Qr8ABJhAHQnJwMAAAAAPAZ5WakK/AAS3YjyYHr6/AAAAHUGaWUmoQWyZTBRMO//+qZYABGCjqEGaBT6MfpzdAAAAEAGeeGpCvwAHFZ8xuhyQebgAAAAYQZp9SeEKUmUwId/+qZYABGfjz+ZsUyQVAAAAFUGem0U0TC//AAVBlit2lDT6LK7f6AAAABABnrp0Qr8ABxOGAyS3+zNBAAAAEAGevGpCvwAEt2I8lzPlgYEAAAASQZqhSahBaJlMCG///qeEAAEnAAAADEGe30URLC//AACygAAAABABnv50Qr8AB0LFYvP4HOpBAAAAEAGe4GpCvwAHQNQ5/mW8YsAAAAAcQZrlSahBbJlMCG///qeEAAio+aprNua8dPtkqQAAABVBnwNFFSwv/wAFQoEU8GFQR+7V7XQAAAAPAZ8idEK/AAS12UKTbJalAAAAEAGfJGpCvwAHFZ4F1/biNcEAAAAZQZsmSahBbJlMCG///qeEAAi3x0x/h9W4JwAAABlBm0dJ4QpSZTAh3/6plgAER+POlnR1PNHBAAAAFkGba0nhDomUwIb//qeEAAVr40/i44AAAAAOQZ+JRRE8L/8AAzirhfAAAAAQAZ+odEK/AAavOTiOy7N5gQAAABABn6pqQr8ABq85O9nj7mWAAAAAGUGbrkmoQWiZTAhv//6nhAAIN8dMf4fVuDkAAAAPQZ/MRREsK/8ABsCNA6PBAAAADQGf7WpCvwAGwsSLfR8AAAAaQZvvSahBbJlMCG///qeEAAyNIn+q3zH4wcEAAAAZQZoTSeEKUmUwIZ/+nhAAMT6+/qGvaPD3tgAAABBBnjFFNEwv/wAHbTqN7BjMAAAADwGeUHRCvwAPjYrGEKuYwQAAABABnlJqQr8ACoUo3mmKts7AAAAAGUGaVEmoQWiZTAhn//6eEAAyMhjn8Oc318kAAAAXQZp1SeEKUmUwIb/+p4QADO+we26XXW8AAAAYQZqWSeEOiZTAhv/+p4QADJ+wevZnwRcbAAAAHUGauEnhDyZTBRE8N//+p4QADE+wf5a6W5wTRP4VAAAAEAGe12pCvwAJ825FXgCgo4EAAAAbQZrcSeEPJlMCG//+p4QABSPdT9zIwtmKEdJEAAAAEEGe+kURPC//AAMQI3e4aUEAAAAPAZ8ZdEK/AAQV2UKTbJbVAAAADwGfG2pCvwACsqNE1JWYgQAAABlBmx9JqEFomUwIZ//+nhAADJ+vu7Tm7jH9AAAAEkGfPUURLCv/AAP5E2SHPtxwwAAAAA4Bn15qQr8AA/gQLMHDJgAAABlBm0BJqEFsmUwIZ//+nhAADE+vu7Tm7jIdAAAAGEGbYUnhClJlMCG//qeEAAMT7B69mfBG1QAAABlBm4JJ4Q6JlMCG//6nhAAElQBZttn2fR1BAAAAGUGbo0nhDyZTAh3//qmWAAOOmQk3Dgo+gRAAAAAdQZvHSeEPJlMCHf/+qZYABbdLMWmZ29X3x1S7mCEAAAAQQZ/lRRE8L/8ABsFSb+bTtQAAAA8BngR0Qr8ACPCAOhOTOMEAAAAPAZ4GakK/AAkuzy3DZtXLAAAAEkGaC0moQWiZTAhv//6nhAABJwAAAAxBnilFESwv/wAAsoAAAAAQAZ5IdEK/AA4tisXn8Dl4wQAAAA8BnkpqQr8ACRKkbrPVoJ4AAAAaQZpMSahBbJlMCHf//qmWAAjCLDdGIRz7HuAAAAAbQZpwSeEKUmUwId/+qZYACQ/Hn5sH13jmuAaPAAAAFEGejkU0TC//ABBceOmcVgneX8pPAAAADwGerXRCvwAWvMncGyXktQAAABABnq9qQr8AFrbcirwBQE6AAAAAGkGas0moQWiZTAh3//6plgAF499X12INxViQAAAAEUGe0UURLCv/AAmuaN5pvetxAAAADgGe8mpCvwAJrKMeiLGcAAAAF0Ga90moQWyZTAhv//6nhAAHn9g/zEuAAAAAFEGfFUUVLC//AASWfIbbsPmIsX2BAAAAEAGfNHRCvwAGSeTeVsoe+8AAAAAQAZ82akK/AAZIFjXvNK0qQQAAABpBmzlJqEFsmUwUTDv//qmWAAO/7S/sWs4kZwAAABABn1hqQr8ABkmbmuPFW33gAAAAEkGbXUnhClJlMCHf/qmWAACVgQAAAAxBn3tFNEwv/wAAsoAAAAAQAZ+adEK/AAPuob2XVfxbQQAAABABn5xqQr8ABkkrYvV2HPlBAAAAHEGbgUmoQWiZTAhv//6nhAAHaB4muNUS/RP8kygAAAAQQZ+/RREsL/8ABHc/c4WdeAAAAA8Bn950Qr8AA/hegMkvBoEAAAAQAZ/AakK/AAZIFjXvNK0qQAAAABpBm8NJqEFsmUwUTDv//qmWAAO/7S/sWs4kZwAAABABn+JqQr8ABiCZJpvpIPaQAAAAGUGb5knhClJlMCHf/qmWAAKTpZXGaX9sVcEAAAASQZ4ERTRMK/8ABBdivYWC/YaBAAAADwGeJWpCvwAEF2eW4bNriwAAABxBmihJqEFomUwU8O/+qZYAApfvq++MKgWimIkfAAAAEAGeR2pCvwAENzRvNMVbmUAAAAAbQZpMSeEKUmUwId/+qZYAArelmLTNAd30Y9jfAAAAEEGeakU0TC//AAM4I3e4ZEEAAAAPAZ6JdEK/AARX0ncGyXr2AAAADwGei2pCvwAEWDQPJgmZgAAAABNBmpBJqEFomUwId//+qZYAAJWBAAAAEEGerkURLC//AAM45Ee7u5EAAAAQAZ7NdEK/AARV1aMkt/trgQAAAA8Bns9qQr8ABFdnluGza3sAAAAcQZrUSahBbJlMCHf//qmWAARAo6IFmgO76MewWgAAABVBnvJFFSwv/wAFHoEE82IbumcttIUAAAAQAZ8RdEK/AARV2pPK/JTwcAAAABABnxNqQr8ABunajlf24jfAAAAAHEGbGEmoQWyZTAhv//6nhAAId8dPutLM1Nui5QkAAAASQZ82RRUsL/8ABR2WK3aVxdAkAAAAEAGfVXRCvwAG6AADJLf7NcEAAAAQAZ9XakK/AASXYjyXM+WGgQAAABJBm1xJqEFsmUwIb//+p4QAAScAAAATQZ96RRUsL/8AA2ES3KZj5iIskQAAABABn5l0Qr8ABLdx3lbKHyGAAAAAEAGfm2pCvwAEteaJkTStRMEAAAAaQZudSahBbJlMCG///qeEAAiqALNts+z58cEAAAAdQZu/SeEKUmUwUVLDf/6nhAANK6tmJ/q7e6n7X9gAAAAQAZ/eakK/AAsSjRMiaVoNwAAAABxBm8FJ4Q6JlMFEw3/+p4QAFPxWqY/1bt9g/Xb5AAAADwGf4GpCvwAQ3YjyYHr37wAAABxBm+VJ4Q8mUwIZ//6eEAB+vXI3Y4YPUebXeMA1AAAAFEGeA0URPC//ABPqASfd1d065PIrAAAAEAGeInRCvwARXcd5Wyh60YEAAAAQAZ4kakK/ABsHajlf24gPwQAAABpBmilLqEIQWiRGCCgH8gH9h4AhX/44QAARcQAAACNBnkdFESwv/wIB3OpL2zMKuYDoGrWoXAlAGWiTwt8ykzScMQAAABABnmZ0Qr8AGrzk4jsuy1mAAAAAJgGeaGpCvwKvY+1BxN2qw0km5aqGByy1u80sa5G2/erSl/OW5IfwAAAMAG1vb3YAAABsbXZoZAAAAAAAAAAAAAAAAAAAA+gAAB+QAAEAAAEAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAsqdHJhawAAAFx0a2hkAAAAAwAAAAAAAAAAAAAAAQAAAAAAAB+QAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAEQAAABEAAAAAAAJGVkdHMAAAAcZWxzdAAAAAAAAAABAAAfkAAABAAAAQAAAAAKom1kaWEAAAAgbWRoZAAAAAAAAAAAAAAAAAAAMgAAAZQAVcQAAAAAAC1oZGxyAAAAAAAAAAB2aWRlAAAAAAAAAAAAAAAAVmlkZW9IYW5kbGVyAAAACk1taW5mAAAAFHZtaGQAAAABAAAAAAAAAAAAAAAkZGluZgAAABxkcmVmAAAAAAAAAAEAAAAMdXJsIAAAAAEAAAoNc3RibAAAAJVzdHNkAAAAAAAAAAEAAACFYXZjMQAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAEQARAASAAAAEgAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABj//wAAAC9hdmNDAfQADf/hABdn9AANkZsoIhHQgAAAAwCAAAAZB4oUywEABWjr48RIAAAAGHN0dHMAAAAAAAAAAQAAAMoAAAIAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAAXYY3R0cwAAAAAAAAC5AAAAAQAABAAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAAQAACAAAAAACAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAAQAACAAAAAACAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAwAABAAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAABAAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAIAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAMAAAQAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACAAAAAACAAACAAAAAAQAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAABxzdHNjAAAAAAAAAAEAAAABAAAAygAAAAEAAAM8c3RzegAAAAAAAAAAAAAAygAABW4AAAAVAAAAFAAAACAAAAAWAAAAFAAAABQAAAAgAAAAFAAAABMAAAAUAAAAHQAAABQAAAATAAAAEwAAACQAAAAZAAAAFAAAABQAAAAjAAAAFgAAABMAAAAeAAAAEwAAABMAAAAWAAAAEAAAABMAAAAUAAAALAAAABQAAAATAAAAEwAAAB4AAAAfAAAAFgAAABAAAAAUAAAAFAAAACEAAAAZAAAAEwAAABMAAAAdAAAAFgAAABIAAAAeAAAAFgAAABIAAAAkAAAAFgAAABQAAAATAAAAHQAAAB0AAAAdAAAAIQAAABQAAAAfAAAAFAAAABIAAAAUAAAAHQAAABQAAAAdAAAAIwAAABQAAAAcAAAAFAAAABMAAAATAAAAGwAAABIAAAAUAAAAFAAAABYAAAAYAAAAEwAAABMAAAAeAAAAHgAAABkAAAAUAAAAEwAAAB0AAAAUAAAAEwAAABMAAAAhAAAAFAAAABwAAAAZAAAAFAAAABQAAAAWAAAAEAAAABQAAAAUAAAAIAAAABkAAAATAAAAFAAAAB0AAAAdAAAAGgAAABIAAAAUAAAAFAAAAB0AAAATAAAAEQAAAB4AAAAdAAAAFAAAABMAAAAUAAAAHQAAABsAAAAcAAAAIQAAABQAAAAfAAAAFAAAABMAAAATAAAAHQAAABYAAAASAAAAHQAAABwAAAAdAAAAHQAAACEAAAAUAAAAEwAAABMAAAAWAAAAEAAAABQAAAATAAAAHgAAAB8AAAAYAAAAEwAAABQAAAAeAAAAFQAAABIAAAAbAAAAGAAAABQAAAAUAAAAHgAAABQAAAAWAAAAEAAAABQAAAAUAAAAIAAAABQAAAATAAAAFAAAAB4AAAAUAAAAHQAAABYAAAATAAAAIAAAABQAAAAfAAAAFAAAABMAAAATAAAAFwAAABQAAAAUAAAAEwAAACAAAAAZAAAAFAAAABQAAAAgAAAAFgAAABQAAAAUAAAAFgAAABcAAAAUAAAAFAAAAB4AAAAhAAAAFAAAACAAAAATAAAAIAAAABgAAAAUAAAAFAAAAB4AAAAnAAAAFAAAACoAAAAUc3RjbwAAAAAAAAABAAAAMAAAAGJ1ZHRhAAAAWm1ldGEAAAAAAAAAIWhkbHIAAAAAAAAAAG1kaXJhcHBsAAAAAAAAAAAAAAAALWlsc3QAAAAlqXRvbwAAAB1kYXRhAAAAAQAAAABMYXZmNTguMTIuMTAw\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize the game\n",
    "env = Environment(grid_size=size, max_time=T,temperature=temperature)\n",
    "\n",
    "# Initialize the agent!\n",
    "agent = RandomAgent()\n",
    "\n",
    "test(agent,env,1,prefix='random')\n",
    "HTML(display_videos('random0.mp4', 'random'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## DQN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us assume here that $T=\\infty$.\n",
    "\n",
    "***\n",
    "__Question 5__ Let $\\pi$ be a policy, show that:\n",
    "\n",
    "\\begin{equation*}\n",
    "Q^{\\pi}(s,a)=E_{(s',a')\\sim p(.|s,a)}[r(s,a)+\\gamma Q^{\\pi}(s',a')]\n",
    "\\end{equation*}\n",
    "\n",
    "Then, show that for the optimal policy $\\pi^*$ (we assume its existence), the following holds: \n",
    "\n",
    "\\begin{equation*}\n",
    "Q^{*}(s,a)=E_{s'\\sim \\pi^*(.|s,a)}[r(s,a)+\\gamma\\max_{a'}Q^{*}(s',a')].\n",
    "\\end{equation*}\n",
    "Finally, deduce that a plausible objective is:\n",
    "\n",
    "\\begin{equation*}\n",
    "\\mathcal{L}(\\theta)=E_{s' \\sim \\pi^*(.|s,a)}\\Vert r+\\gamma\\max\\max_{a'}Q(s',a',\\theta)-Q(s,a,\\theta)\\Vert^{2}.\n",
    "\\end{equation*}\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer **\n",
    "\n",
    "Let $\\pi$ be a policy : \n",
    "\n",
    "**i**\n",
    "$$Q^{\\pi}(s,a)=E[\\sum_{t=0}^{\\infty}\\gamma^{t} r(S_{t},A_{t})|S_{0}=s, A_{0}=a, A_{t} = \\pi(S_{t}), \\forall t \\ge 1]$$\n",
    "\n",
    "\n",
    "$$Q^{\\pi}(s,a)=r(s,a) + \\gamma E[\\sum_{t=1}^{\\infty}\\gamma^{t-1} r(S_{t},A_{t})|S_{0}=s, A_{0}=a, A_{t} = \\pi(S_{t}), \\forall t \\ge 1]$$\n",
    "Using the properties of the conditional expectation we have : \n",
    "$$Q^{\\pi}(s,a)=r(s,a) + \\gamma E_{(s',a')\\sim p(.|s,a)}[E[\\sum_{t=1}^{\\infty}\\gamma^{t-1} r(S_{t},A_{t})|S_{1}=s', A_{1}=a']|S_{0}=s, A_{0}=a, A_{t} = \\pi(S_{t}), \\forall t \\ge 1]$$\n",
    "\n",
    "\n",
    "$$Q^{\\pi}(s,a)=r(s,a) + \\gamma E_{(s',a')\\sim p(.|s,a)}[Q^{\\pi}(s',a')]$$\n",
    "\n",
    "$$Q^{\\pi}(s,a)=E_{(s',a')\\sim p(.|s,a)}[r(s,a) + \\gamma Q^{\\pi}(s',a')]$$\n",
    "\n",
    "**ii**\n",
    "\n",
    "Given the optimal policy $\\pi^*$\n",
    "\n",
    "$$Q^{*}(s,a)=E_{(s',a')\\sim p(.|s,a)}[r(s,a) + \\gamma Q^{*}(s',a')]$$\n",
    "\n",
    "$$Q^{*}(s,a)=E_{(s')\\sim p(.|s,a)}[r(s,a) + \\gamma Q^{*}(s',\\pi^{*}(s'))]$$\n",
    "\n",
    "Then by definition of the optimal policy $Q^{*}(s',\\pi^{*}(s')) = \\max_{a'} Q^{*}(s',a')$ we have \n",
    "\n",
    "$$Q^{*}(s,a)=E_{(s')\\sim p(.|s,a)}[r(s,a) + \\gamma \\max_{a'}Q^{*}(s',a')]$$\n",
    "\n",
    "\n",
    "**iii**\n",
    "\n",
    "We want our $Q$-function estimate to converge towards $Q^*$ which is unknown. However the equation above gives us it one-step estimate which we can use to learn $Q$. Therefore \n",
    "\n",
    "\\begin{equation*}\n",
    "\\mathcal{L}(\\theta)=E_{s' \\sim \\pi^*(.|s,a)}\\Vert r+\\gamma\\max\\max_{a'}Q(s',a',\\theta)-Q(s,a,\\theta)\\Vert^{2}.\n",
    "\\end{equation*}\n",
    "\n",
    "is a plausible objective function. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "***\n",
    "The DQN-learning algorithm relies on these derivations to train the parameters $\\theta$ of a Deep Neural Network:\n",
    "\n",
    "1. At the state $s_t$, select the action $a_t$ with best reward using $Q_t$ and store the results;\n",
    "\n",
    "2. Obtain the new state $s_{t+1}$ from the environment $p$;\n",
    "\n",
    "3. Store $(s_t,a_t,s_{t+1})$;\n",
    "\n",
    "4. Obtain $Q_{t+1}$ by minimizing  $\\mathcal{L}$ from a recovered batch from the previously stored results.\n",
    "\n",
    "***\n",
    "__Question 6__ Implement the class ```Memory``` that stores moves (in a replay buffer) via ```remember``` and provides a ```random_access``` to these. Specify a maximum memory size to avoid side effects. You can for example use a ```list()``` and set by default ```max_memory=100```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Memory(object):\n",
    "    def __init__(self, max_memory=100):\n",
    "        self.max_memory = max_memory\n",
    "        self.memory = list()\n",
    "\n",
    "    def remember(self, m):\n",
    "        self.memory.append(m)\n",
    "        if len(self.memory) > self.max_memory:\n",
    "            del self.memory[0]\n",
    "\n",
    "    def random_access(self):\n",
    "        return self.memory[np.random.randint(0, len(self.memory), size=1)[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The pipeline we will use for training is given below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(agent,env,epoch,prefix=''):\n",
    "    # Number of won games\n",
    "    score = 0\n",
    "    loss = 0\n",
    "    scores = []\n",
    "    for e in range(epoch):\n",
    "        # At each epoch, we restart to a fresh game and get the initial state\n",
    "        state = env.reset()\n",
    "        # This assumes that the games will terminate\n",
    "        game_over = False\n",
    "\n",
    "        win = 0\n",
    "        lose = 0\n",
    "\n",
    "        while not game_over:\n",
    "            # The agent performs an action\n",
    "            action = agent.act(state)\n",
    "\n",
    "            # Apply an action to the environment, get the next state, the reward\n",
    "            # and if the games end\n",
    "            prev_state = state\n",
    "            state, reward, game_over = env.act(action)\n",
    "\n",
    "            # Update the counters\n",
    "            if reward > 0:\n",
    "                win = win + reward\n",
    "            if reward < 0:\n",
    "                lose = lose -reward\n",
    "\n",
    "            # Apply the reinforcement strategy\n",
    "            loss = agent.reinforce(prev_state, state,  action, reward, game_over)\n",
    "\n",
    "        # Save as a mp4\n",
    "        if e % 10 == 0:\n",
    "            env.draw(prefix+str(e), folder=prefix)\n",
    "\n",
    "        # Update stats\n",
    "        score += win-lose\n",
    "        \n",
    "        scores.append(score/(1+e))\n",
    "        \n",
    "\n",
    "        print(\"Epoch {:03d}/{:03d} | Loss {:.4f} | Win/lose count {}/{} ({})\"\n",
    "              .format(e, epoch, loss, win, lose, win-lose))\n",
    "        agent.save(name_weights=prefix+'model.h5',name_model=prefix+'model.json')\n",
    "    plt.plot(scores)\n",
    "    plt.title(\"Average Scores\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "__Question 7__ Implement the DQN training algorithm using a cascade of fully connected layers. You can use different learning rate, batch size or memory size parameters. In particular, the loss might oscillate while the player will start to win the games. You have to find a good criterium."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DQN(Agent):\n",
    "    def __init__(self, grid_size,  epsilon = 0.1, memory_size=100, batch_size = 16,n_state=2):\n",
    "        super(DQN, self).__init__(epsilon = epsilon)\n",
    "\n",
    "        # Discount for Q learning\n",
    "        self.discount = 0.99\n",
    "        \n",
    "        self.grid_size = grid_size\n",
    "        \n",
    "        # number of state\n",
    "        self.n_state = n_state\n",
    "\n",
    "        # Memory\n",
    "        self.memory = Memory(memory_size)\n",
    "        \n",
    "        # Batch size when learning\n",
    "        self.batch_size = batch_size\n",
    "\n",
    "    def learned_act(self, s):\n",
    "        return np.argmax(self.model.predict(s.reshape([1, s.shape[0], s.shape[1], s.shape[2]]))\n",
    "                                                                   [0,:])\n",
    "\n",
    "    def reinforce(self, s_, n_s_, a_, r_, game_over_):\n",
    "        # Two steps: first memorize the states, second learn from the pool\n",
    "\n",
    "        self.memory.remember([s_, n_s_, a_, r_, game_over_])\n",
    "        \n",
    "        input_states = np.zeros((self.batch_size, 5,5,self.n_state))\n",
    "        target_q = np.zeros((self.batch_size, 4))\n",
    "        \n",
    "        for i in range(self.batch_size):\n",
    "            ######## FILL IN\n",
    "            s_, n_s_, a_, r_, game_over_  = self.memory.random_access()\n",
    "            \n",
    "            if game_over_:\n",
    "                input_states[i] = s_\n",
    "                target_q[i,a_] = r_\n",
    "            else:\n",
    "                ######## FILL IN\n",
    "                input_states[i] = s_\n",
    "                target_q[i,a_] = r_ + self.discount*max(self.model.predict(n_s_.reshape([1, s_.shape[0], s_.shape[1], s_.shape[2]])).ravel())\n",
    "        ######## FILL IN\n",
    "        # HINT: Clip the target to avoid exploiding gradients.. -- clipping is a bit tighter\n",
    "        target_q = np.clip(target_q, -4, 4)\n",
    "\n",
    "        l = self.model.train_on_batch(input_states, target_q)\n",
    "\n",
    "\n",
    "        return l\n",
    "\n",
    "    def save(self,name_weights='model.h5',name_model='model.json'):\n",
    "        self.model.save_weights(name_weights, overwrite=True)\n",
    "        with open(name_model, \"w\") as outfile:\n",
    "            json.dump(self.model.to_json(), outfile)\n",
    "            \n",
    "    def load(self,name_weights='model.h5',name_model='model.json'):\n",
    "        with open(name_model, \"r\") as jfile:\n",
    "            model = model_from_json(json.load(jfile))\n",
    "        model.load_weights(name_weights)\n",
    "        model.compile(\"sgd\", \"mse\")\n",
    "        self.model = model\n",
    "\n",
    "            \n",
    "class DQN_FC(DQN):\n",
    "    def __init__(self, *args, lr=0.1,**kwargs):\n",
    "        super(DQN_FC, self).__init__( *args,**kwargs)\n",
    "        \n",
    "        # NN Model\n",
    "        \n",
    "        ####### FILL IN\n",
    "        model = Sequential()\n",
    "        model.add(Flatten(input_shape=(5,5, self.n_state,)))\n",
    "        model.add(Dense(15, activation='relu'))\n",
    "        model.add(Dense(4, activation=\"linear\"))\n",
    "        model.compile(sgd(lr=lr, decay=1e-4, momentum=0.0), \"mse\")\n",
    "        self.model = model\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 000/101 | Loss 0.0457 | Win/lose count 6.5/7.0 (-0.5)\n",
      "Epoch 001/101 | Loss 0.0059 | Win/lose count 2.5/4.0 (-1.5)\n",
      "Epoch 002/101 | Loss 0.0084 | Win/lose count 7.5/9.0 (-1.5)\n",
      "Epoch 003/101 | Loss 0.0095 | Win/lose count 5.0/2.0 (3.0)\n",
      "Epoch 004/101 | Loss 0.0117 | Win/lose count 2.5/4.0 (-1.5)\n",
      "Epoch 005/101 | Loss 0.0144 | Win/lose count 2.5/2.0 (0.5)\n",
      "Epoch 006/101 | Loss 0.0026 | Win/lose count 6.5/8.0 (-1.5)\n",
      "Epoch 007/101 | Loss 0.0054 | Win/lose count 5.0/7.0 (-2.0)\n",
      "Epoch 008/101 | Loss 0.0106 | Win/lose count 7.0/8.0 (-1.0)\n",
      "Epoch 009/101 | Loss 0.0141 | Win/lose count 3.0/4.0 (-1.0)\n",
      "Epoch 010/101 | Loss 0.0171 | Win/lose count 8.5/9.0 (-0.5)\n",
      "Epoch 011/101 | Loss 0.0211 | Win/lose count 5.0/7.0 (-2.0)\n",
      "Epoch 012/101 | Loss 0.0175 | Win/lose count 8.5/6.0 (2.5)\n",
      "Epoch 013/101 | Loss 0.0073 | Win/lose count 7.0/6.0 (1.0)\n",
      "Epoch 014/101 | Loss 0.0086 | Win/lose count 5.5/10.0 (-4.5)\n",
      "Epoch 015/101 | Loss 0.0013 | Win/lose count 4.0/7.0 (-3.0)\n",
      "Epoch 016/101 | Loss 0.0088 | Win/lose count 5.5/10.0 (-4.5)\n",
      "Epoch 017/101 | Loss 0.0113 | Win/lose count 5.5/6.0 (-0.5)\n",
      "Epoch 018/101 | Loss 0.0157 | Win/lose count 7.5/5.0 (2.5)\n",
      "Epoch 019/101 | Loss 0.0155 | Win/lose count 7.0/3.0 (4.0)\n",
      "Epoch 020/101 | Loss 0.0154 | Win/lose count 9.5/7.0 (2.5)\n",
      "Epoch 021/101 | Loss 0.0055 | Win/lose count 8.5/4.0 (4.5)\n",
      "Epoch 022/101 | Loss 0.0086 | Win/lose count 13.0/3.0 (10.0)\n",
      "Epoch 023/101 | Loss 0.0151 | Win/lose count 6.5/3.0 (3.5)\n",
      "Epoch 024/101 | Loss 0.0177 | Win/lose count 8.0/15.0 (-7.0)\n",
      "Epoch 025/101 | Loss 0.0115 | Win/lose count 6.0/4.0 (2.0)\n",
      "Epoch 026/101 | Loss 0.0097 | Win/lose count 10.0/8.0 (2.0)\n",
      "Epoch 027/101 | Loss 0.0092 | Win/lose count 4.0/8.0 (-4.0)\n",
      "Epoch 028/101 | Loss 0.0122 | Win/lose count 7.5/8.0 (-0.5)\n",
      "Epoch 029/101 | Loss 0.0012 | Win/lose count 8.0/7.0 (1.0)\n",
      "Epoch 030/101 | Loss 0.0047 | Win/lose count 6.0/5.0 (1.0)\n",
      "Epoch 031/101 | Loss 0.0146 | Win/lose count 11.0/2.0 (9.0)\n",
      "Epoch 032/101 | Loss 0.0023 | Win/lose count 11.5/3.0 (8.5)\n",
      "Epoch 033/101 | Loss 0.0120 | Win/lose count 3.5/3.0 (0.5)\n",
      "Epoch 034/101 | Loss 0.0156 | Win/lose count 9.0/3.0 (6.0)\n",
      "Epoch 035/101 | Loss 0.0172 | Win/lose count 3.0/3.0 (0.0)\n",
      "Epoch 036/101 | Loss 0.0053 | Win/lose count 8.5/4.0 (4.5)\n",
      "Epoch 037/101 | Loss 0.0030 | Win/lose count 4.0/1.0 (3.0)\n",
      "Epoch 038/101 | Loss 0.0093 | Win/lose count 9.5/4.0 (5.5)\n",
      "Epoch 039/101 | Loss 0.0042 | Win/lose count 5.5/11.0 (-5.5)\n",
      "Epoch 040/101 | Loss 0.0046 | Win/lose count 8.5/4.0 (4.5)\n",
      "Epoch 041/101 | Loss 0.0058 | Win/lose count 3.0/2.0 (1.0)\n",
      "Epoch 042/101 | Loss 0.0030 | Win/lose count 7.0/2.0 (5.0)\n",
      "Epoch 043/101 | Loss 0.0009 | Win/lose count 5.5/5.0 (0.5)\n",
      "Epoch 044/101 | Loss 0.0006 | Win/lose count 4.0/5.0 (-1.0)\n",
      "Epoch 045/101 | Loss 0.0087 | Win/lose count 8.5/5.0 (3.5)\n",
      "Epoch 046/101 | Loss 0.0083 | Win/lose count 11.5/4.0 (7.5)\n",
      "Epoch 047/101 | Loss 0.0027 | Win/lose count 7.0/4.0 (3.0)\n",
      "Epoch 048/101 | Loss 0.0058 | Win/lose count 2.5/0 (2.5)\n",
      "Epoch 049/101 | Loss 0.0054 | Win/lose count 3.0/6.0 (-3.0)\n",
      "Epoch 050/101 | Loss 0.0136 | Win/lose count 3.5/5.0 (-1.5)\n",
      "Epoch 051/101 | Loss 0.0056 | Win/lose count 2.0/2.0 (0.0)\n",
      "Epoch 052/101 | Loss 0.0052 | Win/lose count 6.0/2.0 (4.0)\n",
      "Epoch 053/101 | Loss 0.0162 | Win/lose count 4.0/2.0 (2.0)\n",
      "Epoch 054/101 | Loss 0.0101 | Win/lose count 8.0/1.0 (7.0)\n",
      "Epoch 055/101 | Loss 0.0110 | Win/lose count 8.0/8.0 (0.0)\n",
      "Epoch 056/101 | Loss 0.0052 | Win/lose count 2.5/2.0 (0.5)\n",
      "Epoch 057/101 | Loss 0.0154 | Win/lose count 4.5/6.0 (-1.5)\n",
      "Epoch 058/101 | Loss 0.0052 | Win/lose count 6.5/6.0 (0.5)\n",
      "Epoch 059/101 | Loss 0.0086 | Win/lose count 6.0/5.0 (1.0)\n",
      "Epoch 060/101 | Loss 0.0123 | Win/lose count 4.0/1.0 (3.0)\n",
      "Epoch 061/101 | Loss 0.0056 | Win/lose count 9.5/5.0 (4.5)\n",
      "Epoch 062/101 | Loss 0.0004 | Win/lose count 2.5/3.0 (-0.5)\n",
      "Epoch 063/101 | Loss 0.0030 | Win/lose count 5.0/4.0 (1.0)\n",
      "Epoch 064/101 | Loss 0.0125 | Win/lose count 7.0/3.0 (4.0)\n",
      "Epoch 065/101 | Loss 0.0101 | Win/lose count 3.0/1.0 (2.0)\n",
      "Epoch 066/101 | Loss 0.0074 | Win/lose count 3.5/7.0 (-3.5)\n",
      "Epoch 067/101 | Loss 0.0074 | Win/lose count 2.0/1.0 (1.0)\n",
      "Epoch 068/101 | Loss 0.0108 | Win/lose count 9.5/7.0 (2.5)\n",
      "Epoch 069/101 | Loss 0.0025 | Win/lose count 4.0/3.0 (1.0)\n",
      "Epoch 070/101 | Loss 0.0150 | Win/lose count 6.0/5.0 (1.0)\n",
      "Epoch 071/101 | Loss 0.0116 | Win/lose count 8.0/3.0 (5.0)\n",
      "Epoch 072/101 | Loss 0.0094 | Win/lose count 7.0/3.0 (4.0)\n",
      "Epoch 073/101 | Loss 0.0075 | Win/lose count 5.5/5.0 (0.5)\n",
      "Epoch 074/101 | Loss 0.0141 | Win/lose count 9.0/4.0 (5.0)\n",
      "Epoch 075/101 | Loss 0.0004 | Win/lose count 3.0/1.0 (2.0)\n",
      "Epoch 076/101 | Loss 0.0108 | Win/lose count 7.5/3.0 (4.5)\n",
      "Epoch 077/101 | Loss 0.0081 | Win/lose count 2.0/3.0 (-1.0)\n",
      "Epoch 078/101 | Loss 0.0007 | Win/lose count 6.0/4.0 (2.0)\n",
      "Epoch 079/101 | Loss 0.0146 | Win/lose count 6.5/0 (6.5)\n",
      "Epoch 080/101 | Loss 0.0066 | Win/lose count 7.0/7.0 (0.0)\n",
      "Epoch 081/101 | Loss 0.0032 | Win/lose count 4.5/5.0 (-0.5)\n",
      "Epoch 082/101 | Loss 0.0094 | Win/lose count 7.0/2.0 (5.0)\n",
      "Epoch 083/101 | Loss 0.0034 | Win/lose count 4.0/1.0 (3.0)\n",
      "Epoch 084/101 | Loss 0.0178 | Win/lose count 6.5/5.0 (1.5)\n",
      "Epoch 085/101 | Loss 0.0036 | Win/lose count 4.0/5.0 (-1.0)\n",
      "Epoch 086/101 | Loss 0.0138 | Win/lose count 10.5/7.0 (3.5)\n",
      "Epoch 087/101 | Loss 0.0023 | Win/lose count 5.0/3.0 (2.0)\n",
      "Epoch 088/101 | Loss 0.0102 | Win/lose count 6.5/1.0 (5.5)\n",
      "Epoch 089/101 | Loss 0.0035 | Win/lose count 2.0/2.0 (0.0)\n",
      "Epoch 090/101 | Loss 0.0080 | Win/lose count 11.0/1.0 (10.0)\n",
      "Epoch 091/101 | Loss 0.0042 | Win/lose count 2.0/5.0 (-3.0)\n",
      "Epoch 092/101 | Loss 0.0284 | Win/lose count 5.0/2.0 (3.0)\n",
      "Epoch 093/101 | Loss 0.0111 | Win/lose count 5.0/5.0 (0.0)\n",
      "Epoch 094/101 | Loss 0.0012 | Win/lose count 4.5/4.0 (0.5)\n",
      "Epoch 095/101 | Loss 0.0078 | Win/lose count 5.0/4.0 (1.0)\n",
      "Epoch 096/101 | Loss 0.0082 | Win/lose count 4.5/2.0 (2.5)\n",
      "Epoch 097/101 | Loss 0.0048 | Win/lose count 9.5/3.0 (6.5)\n",
      "Epoch 098/101 | Loss 0.0060 | Win/lose count 3.5/4.0 (-0.5)\n",
      "Epoch 099/101 | Loss 0.0089 | Win/lose count 2.0/2.0 (0.0)\n",
      "Epoch 100/101 | Loss 0.0047 | Win/lose count 3.0/1.0 (2.0)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8lOW9///XJzvZ90BCQiDsi4KGTVBxq0s9otUq1eNSrdqqte1pfx7bntPtdD0931rPOZ621K3uu5UqreIKisgOQiCBJEBCSDJZSCb7TOb6/TF3QgiTjZlkkpnP8/GYhzP33HPf153Bd6587uu+bjHGoJRSKriE+LsBSimlRp6Gv1JKBSENf6WUCkIa/kopFYQ0/JVSKghp+CulVBDS8FdKqSCk4a9GBRH5UETqRSTS323xBRGZIyLvWMd0XES2icgV/m6XUl00/JXfiUgucC5ggKuGaR9hw7HdfvwNWAdkAOnA/UCjL3fgh2NSAUTDX40GtwCbgCeBW7sWisgSEakUkdAey64Rkd3W8xAReVBEikWkVkReEpFk671cETEicoeIHAHet5a/bG2zQUTWi8icHttOEZG/iUijiGwRkZ+LyMc93p8pIutEpE5ECkXkek8HIyKpwGTgz8aYDuvxiTGm57ZWishOa1/FInKZtTxTRNZY+zgoInf2+MxPROQVEXlGRBqB2wb4GURZ69Zaf31sEZGM0/+aVCDR8FejwS3As9bj0q6AMsZsApqBC3useyPwnPX8fuBq4HwgE6gHHum17fOBWcCl1uu/A9Nw98a3W/vs8oi1v/G4fwn1/EUUg7sn/5z12a8A/9fzl0cPtcBB4BkRubp34IrIIuAp4P8DEoHzgEPW288D5dbxXAf8UkQu6vHxlcAr1ueeHeBncCuQAGQDKcDXgVYP7VXByBijD3347QEsBxxAqvV6P/CdHu//HHjceh6HO5wnWa/3ARf1WHeCta0wIBd3GWlKP/tOtNZJAEKtz87ote+Prec3ABt6ff5PwI/72PZE4H+BYsAFrAem9fjcQx4+kw10AnE9lv0KeNJ6/hNgfa/P9PczuB3YCJzh7+9ZH6PvoT1/5W+3Au8YY2qs18/Ro8dtvf6SdSL4S8B2Y8xh671JwOtWSeM47iDsxF1n71LW9UREQkXk11aJpJETve1UIA13YJZ5+qy1r8Vd+7L2dxPuvxJOYYwpN8bcZ4zJsz7bjLu3D+6QL/bwsUygzhhj77HsMJDVR5sG+hk8DbwNvCAiFSLynyIS7qm9KvjoCSPlNyIyDrgeCBWRSmtxJJAoImcaY3YZYwpE5DBwOSeXfMAdhLcbYz7xsO1c62nPaWtvxF02uRh38CfgLpMIYAOcuHvsRdb62b329ZEx5pKhHqcxpkxEHsFd0unaVp6HVSuAZBGJ6/ELIAc42nNzvT7T58/A8lPgp9bPYy1QCDw21GNQgUd7/sqfrsbdS50NzLces4ANuM8DdHkOd237PODlHsv/CPxCRCYBiEiaiKzsZ39xQDvumnw08MuuN4wxncBrwE9EJFpEZvZqw5vAdBG5WUTCrcdCEZnVeycikiQiPxWRqdYJ2VTcJZhN1iqPAV8VkYus97NEZKYxpgx3meZX1snaM4A7OPm8RG99/gxE5AIRmWedMG/EXQ7q7GdbKoho+Ct/uhV4whhzxBhT2fXAXSu/qcdQxueBFcD7PcpDAA8Da4B3RMSOO1wX97O/p3CXUY4CBZwI4y734f5roBJ3yeR53L8ssHriXwBW4e6hVwK/wf2XSm8duM85vIs7dPdY27nN2tZm4KvAQ0AD8BHu8g24TyTnWvt4Hfc5hXX9HFN/P4PxuE8ON+IuB30EPNPPtlQQEWP0Zi5KeSIivwHGG2NuHXBlpcYY7fkrZbHG8Z8hbotwl1xe93e7lBoOesJXqRPicJd6MoFq4P8Bb/i1RUoNEy37KKVUENKyj1JKBaFRW/ZJTU01ubm5/m6GUkqNKdu2basxxqQNtN6oDf/c3Fy2bt3q72YopdSYYl0UOSAt+yilVBDS8FdKqSCk4a+UUkFIw18ppYKQhr9SSgUhDX+llApCGv5KKRWENPyVUqqXTpfhte3l1Dd3+HS77c5OnJ0un27zdGn4K6VUL2/uruBfXtrF157aSpvDN/e/qWlq5+LffcT5v/2QZzYdpt3p3/vqaPgrpVQPxhj+vKGEpOhwth2u54FXdtNzAkyXa+iTYbY7O/n609uw2dtJi4vk3/66hxW//ZDV64upbmzzZfMHbdRO76CUUv6wqaSOPUcb+eU186hv6eC3bxeSmxLNmdmJvLKtnHf3VTFjfBw35Gdz1fwsEsaF97s9Yww/eG0PWw/X88iNZ3HFvPF8fLCG/3nvIL9cu59f/30/505LY1p6LJWNbVQ2tDExaRy/X7VgWI/TJ+EvIo8DVwLVxpi5Ht5fgXte9FJr0WvGmJ/5Yt9KKeVLj24oISUmgi+dlUVkWAgltmb++/2DAKTERHDd2dnsLDvOv7+xl5+/tY+leSmck5fC0impTB8fS2RYKOAO/dKaZl7cUsar28v59sXT+OIZEwA4d1oa505Lo9jWxOvbj/L6jqN8VlrLhIRxjI+PIiMhatiP01c9/ydx33f1qX7W2WCMudJH+1NKKZ87WG3nvf3VfPviaUSFu0P8V1+ax+TUaGaMj2fFjDTCQ0MwxrDnaCOvbCtjw8EaPiy0ASACGXFRZCZGcaSulZqmdgBWzs/k/gunnbK/vLRYvnfpDL536QyMMYjIiB2rT8LfGLNeRHJ9sS2llBopHxXZ+NdXdnNOXgqrFuXw+o5yIsNCuHnJpO51IsJCuK9XcIsI8yYmMG9iAgBVjW18VlpHia2JsrpWjh5vYfnUFBZNTmHR5CTy0mIHDPaRDH4Y2Zr/UhHZBVQA3zPG7O29gojcBdwFkJOTM4JNU0oFm62H6rj76a2kxkbyTkEVr+04CsCNi3NIiY0c0rYy4qO46szM4WjmsBmp8N8OTDLGNInIFcBfgVP+BjLGrAZWA+Tn5+v9JZVSw2LfsUZuf3ILExLG8dLdS4mJDOXN3cdYX2Tj3gum+rt5I2JEwt8Y09jj+VoR+T8RSTXG1IzE/pVSgckYw6fFtWw4WIPD6cLpMiRGh3P78snER50YhdPpMmwuraPY1kRpTTNv7KwgOiKMp+9YRFqcu5d/fX421+dn++tQRtyIhL+IjAeqjDFGRBbhvr6gdiT2rZQKPI5OF3/dcZTHPznEvmONhIUIkWEhhIQITe1OXtxSxm+uPYPzpqfx8YEafv5WAfsr7QCMCw9l5oQ4fnvdGUxMivbzkfiPr4Z6Pg+sAFJFpBz4MRAOYIz5I3Ad8A0RcQKtwCrT86oJpZSy/PD1z3l/fzXX52dz4+IcMuJPHvbY7uzkrqe28VGRjRkZcfzntWdw1fzM7tE5O8uO892XdnLL45uZkxnP3opGspPH8fCq+SyanExGXBQhISN7cnU0ktGawfn5+Ubv4atUcPlgfzVffXILU9JiKLE1ExYifPGMCXz3khnkpETT4XRxz7PbeHdfNT+/ei43Lc7xOEqmzdHJ/3unkDd2VnDH8snctiy3e/x9oBORbcaY/AHX0/BXSo2U/sayN7Y5+MLv1hM/Loy/fXM5x4638cymwzz72RGcLhe3Ls2lvL6Vf+yt5D+unnvScEx1wmDDX6d3UEqNiIYWB197agtOl+F3189ncmrMSe//au0+qu1t/OnmZUSGhZKbGsO/XTmbO8+bwu/eKeKxT0oxBv79ytka/D6gPX+l1LCra+7gnx/9jIPVTUSFh9DpMvzH1XO5ZkEWVY3tfFhYzYOvfc7d50/h+5fP8riNwko7FcdbuWBm+gi3fmzRso9SalSotrfxz49+xuHaFv5089lMz4jj2y/sZPOhOhLGhdPQ6gBgRkYcb9y3rPvErTo9WvZRSvnNxuIa/v55JbuPNrDvWCOhIjxx20LOmZoKwPN3LeHxj0spqrIzJzOeOVkJzMtK0OAfQRr+Simf2nDAxm1PbCEqLIS5WQncdk4uK+dnMiczoXud0BDhzvOm+LGVSsNfKeUzRVV27nlmO9PSY3n560uJi+p/rnvlPxr+SqnT0uF0sXp9MR2dhnPyUshOjuarT2whKiKUx25bqME/ymn4K6WGzN7m4J5nt7PhQA0hAv/93gEAosJDeOnupWQljvNzC9VANPyVCnA2ezsbi2soqrJz9/l5J014djqqG9u47YktFFbZ+e11Z/CFOePZXFrH5tJazpuexhkTE33UcjWcNPyVGma/eKuA3eUNfPvi6SzNSxmx/X5e3sADr+5m37HuSXVpanPy05Wn3Gl10Gqa2rn2jxupbergsVvzWTHDPeb+ktkZXDI7w+s2q5ET4u8GKDUaNbU7+eNHxRystnu1ndqmdp7ceIith+v5yp83cevjm9l2uA6Xy7vra17ZVs4F//Uhb+0+5vH9PUcbuOnRTTS0dPDAZTNYc98ybl4yiac3HaagotHjZwbi6HRx77PbqW5s55mvLe4OfjU2ac9fqR6MMby9t5KfrCmgsrGNdwuqePnrS0/7Fnuvbi/H0Wl485vL2VhcwyMfFHPtHz4lPS6Si2ZlcH3+RBbkJA1pmzuO1POD1z4nNES497ntvLl7PD9dOYf0OPfsl3srGrjp0c+IiwrnhbuWkJ3snrZ4UnIMb31+jB+v2cNLdw/9mH7x1j4+K63jd9efyVlDbLMafTT8lbLY2xx858VdvLuvipnj47h0TgZ/+fQwG4trWWZdnDQUxhie31zGwtwk5mYlMDcrgVWLcnhvXxXrCqpYs/Mor24r58nbF3JO3uC2b7O3841ntpOREMnr9yzj5a3lPPRuEe/tr2Z8fBTJMRGU1jQTExHK83eeCH6AhOhwHrh0Bg++9jlv7Kzg6gVZJ217f2Ujr24rByApJoKk6AgSx4WTEB3O/mN2ntx4iNuXTeZLZ00c8s9CjT4a/krhPol56xNbOFBl5wdXzOT2ZZPpNIa391bx+3eLOCcvZcg95U0ldZTWNPPNC0/cFjA+KpxrFkzkmgUTaWhx8OU/beSup7bx4t1LTroIypOussvx1g5e+8YyUmMj+caKPC6ZncELm49ga2qnrrmD+dmJ/GzlHHJSTr1RyfX52Ty/+Qi/XLuPVkcniePCcRl4YcsRNhyoISI0BBFod7pO+ezSKSn84IqZQ/oZqNFL5/ZRQa/Y1sQtj22mvqWDP/7z2Zw3Pa37vac+PcSP3tjLc19b3D01wWDd//wOPiysZvMPL+5z2oKK461c+4eNOF2G175xzkk99Z6qG9v47su72HCghodXzWfl/CyP6w3GrrLj3PToZzS1O7uXpcdFcus5udy4KIekmAhaOzqpb+mgodXB8RYHLR1OlualEB2h/cXRTid2U2oQmtqdrPjtBwA8ftvCU4Yptjk6Of+3HzApOYYX714y6N5/XXMHS375HjcuzuEnV83pd92iKjvX/WEjIsKKGWmcOy2Ns3ISSY6JIC4qnPf2VfGvr+6m1dHJj/9pDl9ZlHN6B9vruOpbOrqDfV5WIhFhOv4jEOjEbkoNwoeF1dQ0dfDcnYs9jk+PCg/lnhVT+fGavXxaXDvo3v9LW8vo6HSxatHANwSfnhHHc3cu4bGPS9lwwMYbOyu63xMBY2BOZjwPr1rA1PTYwR9cP6LCQ5mQMI4JCXoxVrDS8FdB7d2CKpKiw1k8ue/x9zcszGb1+hIeeHU3a+5bTnJMhMf17G0O1uyq4JVt5ew4cpxFucnMHB8/qHbMzUrgoRvm43IZ9lfaKTjWSEOrg4ZWBwnjwrl5ySTtmSuf0vBXQcvR6eL9/dVcMns8of3c0DsqPJT/u+ksvvynT7n32e08dcciwkNPDuKGVgcr//djDtW2MD0jlh9eMYsv5w99VExIiDA7M57ZmYP7paHU6dKuhApaW0rraGxzDurK1DOzE/nVNfP4tKSWX7y176T3XC7Dd1/aRXl9K09+dSFvf/s87jxvConRnv9CUGo00J6/Clrr9lURGRbCedMHV8e/9uyJ7K1o5PFPSomNDOPrK/KIjQxj9YYS3t1XxY+unK1XvaoxQ8NfBSVjDOsKqlg+NXVIwxd/cMVMbE3t/O8HB3n2s8NcvSCLpz49zBfnTeCry3KHr8FK+ZiWfVRQKqyyU17fysVDnIwsLDSE//nKAl6/5xzOmJjIE58cYlJyNL++dt5pTwGhlD9oz18FpXV7qxCBi2adXplmQU4Sf7l9EQUVjaTGReiNS9SYoz1/FVCMMew4Uo+9zdHveuv2VTE/O7F7MrTTNTsz3uttKOUP2vNXAeWtz49x33M7CAsRFk1O5uJZGdy4OOek6RV2lh1nd3kDD1w2w48tVcq/fNLzF5HHRaRaRPb08b6IyH+LyEER2S0iZ/livyrw7S4/zv3P76DDw0RjvRljWL2+hEkp0dxx7mRqmtr52ZsF3PfcDhyd7s/XNXdw77PbyUocx40+mCZBqbHKV2WfJ4HL+nn/cmCa9bgL+IOP9qsC3J8+KmHNrgp2lh0fcN3NpXXsLm/gznOn8P3LZ/HOd87nP1bO4d19VTzwym6cnS6+9cIObPZ2/vjPZ+s4fBXUfFL2McasF5HcflZZCTxl3LPIbRKRRBGZYIzxfBsipYDGNgfr9lUBsLG4hkWTk/td/88bSkmKDufaHvPN37w0l4ZWB//1ThF7KxooqmriV1+ax7yJ/U+frFSgG6kTvllAWY/X5dayk4jIXSKyVUS22my2EWqaGq3+8XklHU4XidHhbCyu7XfdElsT7+2v4uYlkxgXcfL0yfdeMJW7zptCUVUT1509kVULB55sTalAN1InfD0NgD5lLmljzGpgNbindB7uRqnR7fUdR5mcGsMXZmfw+CeltHQ4+7wg67GPSwkPDeHmpbmnvCcifP/ymVw0M50FOUk6Hl8pRi78y4Ge3a2JQEUf6ypFxfFWNpXW8u2LpjM/J5E/rS9h66H67huttHQ4eXX7UZrbnXQ4Xby6vZxr5meRFhfpcXsiwuIpfc/cqVSwGanwXwPcJyIvAIuBBq33q/6s2VWBMbByfibp8ZGEhwobi2u7w/9PH5Xw8HsHutePjQzjzvOm+Ku5So05Pgl/EXkeWAGkikg58GMgHMAY80dgLXAFcBBoAb7qi/2qwPXXHUdZkJNIbmoMAAuyk9hYXAO477715MZDXDwrg4dXzSciLISwENFyjlJD4KvRPl8Z4H0D3OuLfanAt+9YI/sr7fxs5YnbHy7NS+F/3j9AQ4uDF7ceoaHVwX0XTiUmUq9TVOp06PQOalQxxvBfbxcSERrCF+dN6F6+bGoqLgPrD9j484ZSlk9NZX72qbddVEoNjoa/GlUe+7iU9/ZX8/0rZpISe+Lk7fzsRKLCQ/iPNwuw2du554I8P7ZSqbFPw1+NGtuP1PPrv+/n0jkZ3HZO7knvRYSFsDA3mWp7OwtyElmqI3eU8oqGvxoVGlocfPO5HYxPiOI/rz3T48nbZVPdd9y6d8VUPbmrlJf0bJkaFR56t4iqxjZe+cY5JER7nhv/psU5ZCaOO+05+JVSJ2j4K79rbnfy6rZyrjxjQr8nceOiwrnqzMwRbJlSgUvLPsrv/rrzKPZ2JzcvneTvpigVNDT8lV8ZY3j608PMnhDPWTlJ/m6OUkFDw1/51ZZD9eyvtHPL0kl6ElepEaThr0bU4dpm3tp9rPvOWk9vOkxcVBgr558yw7dSahjpCV81on7+1j7WFVSRkxzNHcsn8489x7h5Se4pc/ArpYaXhr8aMY5OF58W17JkSjJN7U5+vGYvgJ7oVcoPNPzViNlx5DhN7U5uOyeXS+eMZ11BFQ2tDiZbM3cqpUaOhr8aMRsO2AgNEZbmpSIifGHOeH83SamgpSd81YhZX2RjfnYiCeM8X8GrlBo5Gv5qRNQ3d7D7aAPnTkv1d1OUUmj4qxHySXENxtB9G0allH9p+KsRsb7IRnxUGGdkJfi7KUopNPzVCDDGsOFADcumphIWqv/klBoN9P9ENeyKbU0ca2jTko9So4gO9VTD4v39VfxjTyXJMZGU1bUAsHyqnuxVarTQ8FfD4pEPitldfhxB6Oh0MScznuzkaH83Syll0fBXPmeMoajKzqqFOfxs5Rzs7U4iw7TCqNRoouGvfK6ysQ17m5PpGbGICPFRelGXUqONdseUzxVW2gGYnhHn55Yopfqi4a98rqhKw1+p0U7DX/lcYWUT6XGRJMVE+LspSqk++CT8ReQyESkUkYMi8qCH928TEZuI7LQeX/PFftXoVFRlZ8Z47fUrNZp5Hf4iEgo8AlwOzAa+IiKzPaz6ojFmvvV41Nv9qtGp02U4UG3Xko9So5wvev6LgIPGmBJjTAfwArDSB9tVY1BZXQttDhczNPyVGtV8Ef5ZQFmP1+XWst6uFZHdIvKKiGR72pCI3CUiW0Vkq81m80HT1Egr7DrZq2UfpUY1X4S/eFhmer3+G5BrjDkDeBf4i6cNGWNWG2PyjTH5aWk6D8xYdMAK/2npsX5uiVKqP74I/3KgZ09+IlDRcwVjTK0xpt16+WfgbB/sV41ChVVNZCePIyZSrx9UajTzRfhvAaaJyGQRiQBWAWt6riAiE3q8vArY54P9qlGoqNKu9X6lxgCvu2fGGKeI3Ae8DYQCjxtj9orIz4Ctxpg1wP0ichXgBOqA27zdrxp9Opwuim1NXDQr3d9NUUoNwCd/mxtj1gJrey37UY/n3we+74t9qdHrUG0zTpfRYZ5KjQF6ha/yGZ3TR6mxQ8Nf+UxRlZ3QEGFKWoy/m6KUGoCGv/KZwko7k1KiiQoP9XdTlFID0PAPIp8crKG53Tls2y+2Nen4fqXGCA3/IHGw2s5Nj37GC1vKBl75NDg7XRypa2FKmoa/UmOBhn+QeHtvFeDunQ+HsvpWHJ2GKala71dqLNDwDxLrCtzhf6imeVi2X2L9UtGev1Jjg4Z/EKhubGNn2XFCZDjD373dPB3po9SYoOEfBN7dVw3A5XMnUNHQRpuj0+f7KKlpJik6nMRovXuXUmOBhn8QeKegkpzkaL4wJwOAI3UtPt9Hia1JSz5KjSEa/gGuqd3JxoO1XDI7g9wUd0mmdBhKPyU1zXqyV6kxRMM/wK0vstHR6XKHvxXOvq7729sc2Ozt2vNXagzRSdcD3LqCKhKjw8mflERYaAjJMREcqvVt+Hed7NVpHZQaO7TnH8CcnS7e31/NhTPTCQt1f9W5KdEcqvFtzb+kxj3MU0f6KDV2aPgHsEO1LTS0OliWl9q9LDclpt+e/0tbyrpn5xysElszoSFCTrKGv1JjhYZ/ALPZ3XfOnJAQ1b0sNzWGYw1ttHacOtxzfZGNB17dzcPvFQ1pPyU1zWQnjSMiTP85KTVW6P+tAczW5A7/9PjI7mVdJ30P153c+29zdPKjN/YAsOFADc5O16D3U2JrZrKO9FFqTNHwD2DVjW0ApMWe6PlPTuka8XNy3f+PHxVzqLaFmxbnYG9zsrPs+KD24XIZSmt0jL9SY42GfwCzNbUTERZC/LgTg7ompUYDnFT3P1TTzP99WMw/nZnJA5fNJDRE+KjINqh9HGtso83h0pE+So0xGv4BzNbYTlpsJCLSvSw+KpyUmIjusf7GGH68Zi8RoSH8+xdnkTAunLNyEgcd/t0TuqVqz1+psUTDP4DZmtpJi4s8ZXluakz3Vb7v76/moyIb37lkOunx7vLQ+dPT2F3eQI11zqA/OqGbUmOThn8As9n7CP+UGA7XtuDodPGLt/YxJS2GW5ZO6n7//OnpAHx8oGbAfZTYmoiNDPO4H6XU6KXhH8Cq7e2kewz/aCob21i9voSSmmZ+eMUswkNP/FOYkxlPSkzEgKWf2qZ2Pi2pZUpazEmlJaXU6KfhH6AcnS7qmjv6LPsAPLSuiOVTU7lwZvpJ74eECOdNT2N9kQ2Xy3jc/kdFNi57eAOHalr42rlTfH8ASqlhpeEfoGqbOgA8hn/XmHyXMfzblbM89trPn55GbXMHeysaT1re4XTx8zcLuPXxzSRFh/PGfcu46szMYTgCpdRw0vAPUNV29xj/9LioU97LTY0hIjSEVYtymDk+3uPnz52Wioh7/H+Vdb1AxfFWblj9KY9+XMotSyex5r7lzJrg+fNKqdFNZ/UMUF1TO3jq+cdGhrH2W8v7nYsnJTaS25dN5olPSnmnoJLL5k7g4wM2HJ2GR248iy+eMWHY2q6UGn4+6fmLyGUiUigiB0XkQQ/vR4rIi9b7n4lIri/2q/rWX/gDTE2PG3Aunn+/cjYffu8Cblo8iXcLqsiIj2LNfcs0+JUKAF73/EUkFHgEuAQoB7aIyBpjTEGP1e4A6o0xU0VkFfAb4AZv9636Vm2Ff2qsd/fUzUmJ5idXzeGHX5xFWIjoqB6lAoQvev6LgIPGmBJjTAfwArCy1zorgb9Yz18BLhJNkWFls7eTGB1OZFioT7YXHhqiwa9UAPFF+GcBZT1el1vLPK5jjHECDUBK7w2JyF0islVEttpsg5teQHlms7undlBKKU98Ef6euoO9B4cPZh2MMauNMfnGmPy0tDQfNC14Vdvb9KpbpVSffBH+5UB2j9cTgYq+1hGRMCABqPPBvlUfbE2er+5VSinwTfhvAaaJyGQRiQBWAWt6rbMGuNV6fh3wvjHG86WjymvGmD7n9VFKKfDBaB9jjFNE7gPeBkKBx40xe0XkZ8BWY8wa4DHgaRE5iLvHv8rb/aq+2dudtDlcGv5KqT755CIvY8xaYG2vZT/q8bwN+LIv9qUG1jXG39PVvUopBTq9Q0Aa6AIvpZTS8A9A1Rr+SqkBBG34f/uFHaz9/Ji/mzEsTpR9NPyVUp4Fbfi/ufsYf99T6e9mDAubvZ3wUCFhXLi/m6KUGqWCMvwdnS6cLkNxdZO/mzIsqu1tp9y4XSmlegrK8G91dAJQWtPc552qxjKbvZ20eB3po5TqW3CGf4c7/FsdnVRaNyoJJDqvj1JqIEEd/gDFtsAr/ejVvUqpgQRl+Lf0CP8SW7MfW+J7jk4XdS2eb9yulFJdgjL8u2r+ACUB1vOva+7AGB3mqZTqX1CGf5sV/iECxQHW8z9S1wJAZqJTkqmUAAASt0lEQVSe8FVK9S0ow7+r7JOXFntKz/8Hr3/OM5sO+6NZPrHvWCMAsyck+LklSqnRLCjDv6vsMycznoqGNlo6nADUNrXz/OYjPL/5iD+b55WCikaSYyLIiNeyj1Kqb0EZ/m1Wz39ulrt33HXSd8OBGoxx957tbQ6/tc8bBccamT0hXi/wUkr1KyjDv6unPyfTCv8ad/h/VOS+b7DLwPYjx/3TOC84Ol3sr7QzOzPe301RSo1yQRn+rQ4XALMmxCHiHvHjchnWF9m4eFYGoSHCltKxd5fJElszHU4Xsydo+Cul+ueTm7mMNa0dTkQgYVw4E5PGUWxrZm9FI7XNHVwxbzzV9ja2HBp74V9wrAFAe/5KqQEFac+/k3HhoYgIU1LdI34+KqoG4LzpaSzMTWZn2XHanZ0DbGl0KahoJCIshCmpMf5uilJqlAvq8AeYkhZDia2ZDwptzMtKIDU2koW5SbQ7Xew52ujnlg5NwbFGZo6PIyw0KL9WpdQQBGVKtHR0EmWFf15aLK2OTrYdruf86WkA5OcmA4yp0o8xhoKKRq33K6UGJSjDv83RSXTEiZ5/l/NnuMM/NTaSKWkxbB1D4V/Z2EZ9i0Pr/UqpQQnK8G/p6GRcxImeP0BcVBgLshO711k4KZkth+rHzHz/BRVdV/Zq+CulBhaU4d/ao+yTHhdJfFQYy6emnlQrXzg5mYZWBwfHyMRvXeE/U8NfKTUIQTnUs83RSVJMBAAiwhNfXXTKRGgLc5MA2Fxax/SMuBFv41AVHGskNyWa2Mig/EqVUkMUlD3/lo4To30Azp6UxISEcSetk5McTXpcpM/r/k9vOsxlv1/v83JSwbFGrfcrpQYtKMO/1XGi5t8XEeGsnCR2lPlumgeXy/Dn9SXsr7T79PaR9jYHh2tbtN6vlBq0oAz/NsfJPf++zM9J5HBtC3XNHT7Z72eldd3z7fvyDmK7y91X9nbNVaSUUgPxKvxFJFlE1onIAeu/SX2s1ykiO63HGm/26Qu9yz59mW+N/tnlo97/y1vLiAxz/8h9ee/gj4pshIcKiyYn+2ybSqnA5m3P/0HgPWPMNOA967UnrcaY+dbjKi/36RVjDK09xvn3Z15WAiGCT0o/jW0O1u45xrVnTyQuMsyn4f9hYTWLJicToyd7lVKD5G34rwT+Yj3/C3C1l9vzms3ezvm//YDXd5R7fL/d6cIYiBpE+MdEhjE9I44dR+q9btffdlXQ5nBxQ342U9JjfRb+R4+3UlTVxIrp6T7ZnlIqOHgb/hnGmGMA1n/7SqAoEdkqIptEpM9fECJyl7XeVpvNdloNiosK43BtCxXHPZ9QbbVu5BI9iLIPwIKcRHaVHfd6dM5LW8uZOT6OMyYmkJcWQ3G1b2r+Hxa6J6RbYV2drJRSgzFg+IvIuyKyx8Nj5RD2k2OMyQduBH4vInmeVjLGrDbG5Btj8tPSTi/MosJDiYkI7fMkbdctHAca7dNlfnYijW1OSmtPP6wLK+3sKjvOl/OzERHy0mKpbGyjqd152tvs8mGhjazEcUxNj/V6W0qp4DFgkdgYc3Ff74lIlYhMMMYcE5EJQHUf26iw/lsiIh8CC4Di02vywFJiI6ltavf4XtfN26MG2fOfn+0+h73zyPHuqSCGas2uo4SFCNcsyAJOTClRamtm3sTTH6HT4XSx8WANVy/I0ts2KqWGxNuyzxrgVuv5rcAbvVcQkSQRibSepwLLgAIv99uv5JgIavvo+bdZPf/oiMGdHJ2aHktsZBg7vTjpu7eikekZcSRbVxXnWZPJeVv333qojuaOTlbM0Hq/UmpovA3/XwOXiMgB4BLrNSKSLyKPWuvMAraKyC7gA+DXxphhDf/U2AhqmzyHf1fPfzBDPQFCQ4QzJiZ4Ff4HqpqYnnHir4aclGhCQ8Tr8P+wyEZEaAjn5KV4tR2lVPDxamygMaYWuMjD8q3A16znG4F53uxnqFJiIvn8aIPH907U/Af/e29+diKr15fQ5ugcdLmoS1O7k6PHW7kxI6d7WWRYKDnJ0V6H/wf7q1k4OUmHeCqlhiwgr/BNjo2grrkDY04dodPa3fMffGDOz07E6TLsrfD8C6U/B6rsAEzrdULW2xE/R4+3cqC6iQu05KOUOg0BGf4pMRE4Og2NbaeOpml1uJcNdrQPuKd5ANhxZOilnwPV7t5975lB89JiKa1tpvM0h5BuP+y+9mDJFC35KKWGLjDDP9Z9YtXTiJ/WDhfAoK7w7ZIeF0VW4jje21c95LA+UGUnMiyE7OTok5bnpcXS4XRxtL51SNvrUlhpJzREdIinUuq0BGb4x0QCeBzr39Lh7vkPtXZ/x/LJfFpSy3de3Imz0zXozxVVNZGXFktoyMlDMfPSvRvxU1hlJzclesjHoZRSEKjhb/X8azyM+Oka6jnY0T5dbl8+mX+9bCZrdlXwrRd34hjkL4ADVfaTRvp0mZLqXnba4V9pZ+Z4ncJZKXV6AnKYSH89/1ZHJ2EhQkTY0H/vfWNFHqEh8Mu1+2l3dPLQDfOJiwrvc317m4OKhjamebgTWFJMBMkxEacV/s3tTo7UtXDd2ROH/FmllIIA7fl3XUzlqeY/2Omc+3LXeXn8bOUcPii0sfKRTzhY7R7NU2xr4idr9vLQuqLudQ9aJ3t7j/TpcrojfrpOIs8YP/pvL6mUGp0CsucfERZCXFSYx6t82wZxF6+B3LI0l2npcXzz+e2s/N9PmJ+TyCcHawEQgesXZpOVOI4DVZ5H+nTJS4tlXUHVkPdfWOm+WfuMMXBvYaXU6BSQPX+A1NhIj+Hf2uF9+AMszUvhb99czpysBEpszfzLJdN5/Z5zMAZe3eaeTrqoj5E+XWaMj6O2uYMy6+5eg1VY2cS4cPeFYkopdToCsucP7rH+w1H26WlCwjheunvpScuWTU3hpa1l3HfBVA5Uex7p0+W86e6ZSz8srObmpbmD3m9hVSPTM2IJ6WO7Sik1kIDt+SfHeJ7fZzA3b/fG9fnZlNe3sqmkts+RPl2mpMYwKSWa9/d7nAy1T4WV9j5LSUopNRgBG/4p/ZV9hnFs/KVzxhMfFcbjn5T2OdKni4hw4cx0NhbXdk87MZCapnZqmjr0ZK9SyisBG/6psRHUt3SccgeuVsfwhn9UeCgr52fx7j53b36gHvqFM9Npd7rYWFwzqO0XVbpHF+kYf6WUNwI2/JNjIuh0GRpaHSctH+6yD8ANC7O7n/c1zLPLosnJREeEDrr0s98Kf+35K6W8EbDhnxLrvtCrtvnkk77DXfYBmJMZz6wJ8f2O9OkSGRbK8qmpfLC/2uMspL0VVdlJjokg1bqKWSmlTkfghn+M5ykeWh2dQ5rU7XSICD/5p9n86J9m9znSp6eLZqVT0dDW3avvz/5KOzMy4vS2jUoprwRu+Fs9495TPLR0dBI1zOEPsHhKCjctnjSodbvm5B+o9ONyGYqq7FryUUp5LXDD35rfp+dY/06XocPpGvayz1Clx0cxNyueDwYI/6PHW2np6NTwV0p5LWDDPynaPeFaz7LPiZu3j67wB7hwRjrbj9SfcoK6p4Jj1rQOGv5KKS8FbPiHhYaQFB1+UtlnqDdvH0ln5ybjMvR7q8jth+uJCA1h9gQd5qmU8k7Ahj90Xeh1ouzTPZd/xOib1WJupjvQ9x5t7HOdbYfrmZsVrzdwUUp5LaDDv/cUD62neSOXkZASG0lmQhR7+uj5tzs72X20gfzc5BFumVIqEAV0+KfGRpw0xUN32SdidB72nKwE9hz1HP57KxrpcLo4KydphFullApEozMFfSS518yerd01/9FX9gGYm5lASU0zTe3OU97bdqgegLMmJY50s5RSASigwz8lJpLjrY7uG663OtyhOtzTO5yuuVnxGAP7jp1a9992uJ6c5GjS46L80DKlVKAJ6PBPjY3AGKhvcQ+fbO1w/xIYjUM9AeZlJQCcUvoxxrDtSD35k7Tko5TyjYAO/+SYk+f3Gc0nfMF9sVdaXCSf9wr/8vpWbPZ2ztLwV0r5iFfhLyJfFpG9IuISkfx+1rtMRApF5KCIPOjNPoeie4oHa8RPa4e77DOah0rOzYw/Zbjn1sN1AJyt4a+U8hFve/57gC8B6/taQURCgUeAy4HZwFdEZLaX+x2U7sndrBE/raP4Ct8uc7MSOFBtP+nmLtsO1xMbGaZ371JK+YxX4W+M2WeMKRxgtUXAQWNMiTGmA3gBWOnNfgera1rnGru77NM11HNU9/yzEnAZ2F95ove/7fBxFuQkDmqGUKWUGoyRqPlnAWU9Xpdby04hIneJyFYR2Wqz2bzecVJ0OCkxEd0nUFsdnUSEhYzqEJ3b66Svvc1BYWWjlnyUUj414IB3EXkXGO/hrR8aY94YxD48Ja3Hu5YYY1YDqwHy8/MHvrPJQDsWYcmUFD4tqcUYQ1vH8M/l763MhCiSosPZY9X9X91WjstovV8p5VsDhr8x5mIv91EOZPd4PRGo8HKbg7YkL4W3Pj/G4doWWkbgLl7eEhHmZiWw/Ug933t5F69sK2fR5GQWTdZpHZRSvjMSl7puAaaJyGTgKLAKuHEE9gvA0ikpAHxaUjsi9+/1hblZCWw4UMNBWxP3XzSN+y+cSlhoQI/KVUqNMG+Hel4jIuXAUuAtEXnbWp4pImsBjDFO4D7gbWAf8JIxZq93zR68vLQY0uIi+bS4dkTu3+sLV8ydwFk5iTz7tcX8yyXTNfiVUj7nVc/fGPM68LqH5RXAFT1erwXWerOv0yUiLLXq/lPTYsdE+M+bmMBr9yzzdzOUUgEsKLqUS/NSsNnb2V/ZOCbKPkopNdyCI/ytun99i2NM9PyVUmq4BUX4T0qJZny8ezbM0T7UUymlRkJQhL+IsDTP3fvXso9SSgVJ+MOJ0s9ontpBKaVGSvCEf1fPX8NfKaVG5CKvUSE7OZoHLpvBhTPT/d0UpZTyu6AJf4B7Vkz1dxOUUmpUCJqyj1JKqRM0/JVSKghp+CulVBDS8FdKqSCk4a+UUkFIw18ppYKQhr9SSgUhDX+llApCYozX90kfFiJiAw57sYlUoMZHzRkrgu2Yg+14QY85WHhzzJOMMWkDrTRqw99bIrLVGJPv73aMpGA75mA7XtBjDhYjccxa9lFKqSCk4a+UUkEokMN/tb8b4AfBdszBdrygxxwshv2YA7bmr5RSqm+B3PNXSinVBw1/pZQKQgEX/iJymYgUishBEXnQ3+0ZDiKSLSIfiMg+EdkrIt+ylieLyDoROWD9N8nfbfU1EQkVkR0i8qb1erKIfGYd84siEuHvNvqSiCSKyCsist/6vpcG+vcsIt+x/l3vEZHnRSQq0L5nEXlcRKpFZE+PZR6/V3H7byvTdovIWb5oQ0CFv4iEAo8AlwOzga+IyGz/tmpYOIHvGmNmAUuAe63jfBB4zxgzDXjPeh1ovgXs6/H6N8BD1jHXA3f4pVXD52HgH8aYmcCZuI89YL9nEckC7gfyjTFzgVBgFYH3PT8JXNZrWV/f6+XANOtxF/AHXzQgoMIfWAQcNMaUGGM6gBeAlX5uk88ZY44ZY7Zbz+24AyEL97H+xVrtL8DV/mnh8BCRicAXgUet1wJcCLxirRJQxywi8cB5wGMAxpgOY8xxAvx7xn172XEiEgZEA8cIsO/ZGLMeqOu1uK/vdSXwlHHbBCSKyARv2xBo4Z8FlPV4XW4tC1gikgssAD4DMowxx8D9CwIItLvV/x54AHBZr1OA48YYp/U60L7vKYANeMIqdT0qIjEE8PdsjDkK/BdwBHfoNwDbCOzvuUtf3+uw5Fqghb94WBawY1lFJBZ4Ffi2MabR3+0ZTiJyJVBtjNnWc7GHVQPp+w4DzgL+YIxZADQTQCUeT6w690pgMpAJxOAue/QWSN/zQIbl33mghX85kN3j9USgwk9tGVYiEo47+J81xrxmLa7q+nPQ+m+1v9o3DJYBV4nIIdzlvAtx/yWQaJUHIPC+73Kg3BjzmfX6Fdy/DAL5e74YKDXG2IwxDuA14BwC+3vu0tf3Oiy5FmjhvwWYZo0MiMB9omiNn9vkc1at+zFgnzHmdz3eWgPcaj2/FXhjpNs2XIwx3zfGTDTG5OL+Xt83xtwEfABcZ60WaMdcCZSJyAxr0UVAAQH8PeMu9ywRkWjr33nXMQfs99xDX9/rGuAWa9TPEqChqzzkFWNMQD2AK4AioBj4ob/bM0zHuBz3n327gZ3W4wrcNfD3gAPWf5P93dZhOv4VwJvW8ynAZuAg8DIQ6e/2+fhY5wNbre/6r0BSoH/PwE+B/cAe4GkgMtC+Z+B53Oc0HLh79nf09b3iLvs8YmXa57hHQnndBp3eQSmlglCglX2UUkoNgoa/UkoFIQ1/pZQKQhr+SikVhDT8lVIqCGn4K6VUENLwV0qpIPT/AxLZOyzPzrZ1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFm9tZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1MiByMjg1NCBlOWE1OTAzIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNyAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MToweDExMSBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MCBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAALWZYiEADf//vaH+BTZWBP+Wb/9DX/cj9uPrP1xYyEE31qvIejAGS+1H+b/rFFs6Z6UB/fgCJQAc24ZwpHZJ3iSUc9nTsQMIvWPwKXAUp8CmkyM1QcBoKHYFU/rLGGDUA1ikvTEO13IoqVKQT1Cx9s9saG1o9JHceaUhCUAKDXTBEfCv3mGJ0Q8prmNsINx5/CjuK1LWT7r2yIzAw07wj4nyihtIMaR7qXV8vfpn9Hpc98AVxwvJZbKvrTSOZWmhIA/FSxdZcc+ekXaC7ZjZunDQMjeh532ACDCEV3Vrf8omgv2ZFwpoaRttoyXqhy5DPQIuq6E/7hgWkKEQJt04wm5EIWVDFbQKOuH6AgHdPndi5YFbfq/h+Qh87G4wvbyaD6EffFowCQiYd+bFXJlDmUDgAgJc9eIknrAt7A8FEBYZlU9iqj+JJNZMX/OzkRviHAm7464NLfr5GKr6fPthn3PhgoCsNZ50yCpzil/S1evFvN0bn6Cfk/3/IWycMwQB2Y7L2OnrPK4wk9rKX+ZfSMqiypZCHiT6EtOIvACxyvoSgZdf5RsXO5RwzP6Sl7TrAXKVbkyVXSEfgcTpjH85dMxdxqnnca4PuAoA1kBtoOl6ASBRwxvwmFwMWlb3HFGfO4nTwX7zt8pEChHPmH9jcMWgGQBe3XtdObXz+l2WojvCGnZ8ntBEifMIvINzNbafnt0BwFwnRIjjkBNKiHHrPZQIH6SBnAT1E1RpHqfIvtftYoOhjj9glciCv96T5BxFEYrfZ8NMskWnmVqD0RHLNqgwLxLBDZvFsF7VjVRBRvJzqZIrGBbo9Knz7XwOaO98CvdZAAHaMK5H+XpA7xQ36Io8E8ZEWRVz7ttZsHbYMxHDEsC+kXhGU93NtD58BGnPw+RS0EM7wJXXDjHvc1GydVsXIy16xgBHBdd3LPEscrJ9kpLczqeVVyj9YjFkiXSFHS4XQcAAJiBAAAADEGaImxDP/6eEAAEfAAAABABnkF5Cv8ABHbXOF0kpNVxAAAAGEGaQzwhkymEN//+p4QABYfRkZEjo5F/8AAAABhBmmRJ4Q8mUwIb//6nhAADjewevZnwRp8AAAAdQZqISeEPJlMCGf/+nhAAFR9032Atb/Lq+FBimqEAAAARQZ6mRRE8L/8AAzgip6M1j/0AAAAPAZ7FdEK/AARYQB0JycTBAAAAEAGex2pCvwAC1tfOdUzHUkAAAAAZQZrJSahBaJlMCGf//p4QABPeDHP4c5vtCQAAAB1BmutJ4QpSZTBREsM//p4QAB2fXLrY4bPxD/HB4QAAABABnwpqQr8ABnIaXcPtm1fwAAAAGEGbDEnhDomUwIZ//p4QAB3Pf3dpzdxe5gAAABtBmy1J4Q8mUwIZ//6eEAAtXBjn8OfEBTP1/EEAAAAYQZtOSeEPJlMCGf/+nhAARU4Rz9GA7NKXAAAAF0Gbb0nhDyZTAhn//p4QAEVSFXRPMpV1AAAAGUGbkEnhDyZTAhv//qeEABFvo59zIoSHYEAAAAAYQZuxSeEPJlMCG//+p4QAC2e6nH+H1bfTAAAAG0Gb1EnhDyZTAhv//qeEABDUAWbbZ9nzFbdFWQAAABJBn/JFETwr/wAN07UCEjH70UAAAAAOAZ4TakK/AA3TtV0/VYoAAAAZQZoXSahBaJlMCGf//p4QAENEOP54L+SLnQAAAA9BnjVFESwr/wAOKCuG98AAAAAPAZ5WakK/AA4tgS5X+FnBAAAAHEGaWEmoQWyZTAhv//6nhAAa+kT/Vb6qDH/iekEAAAAZQZp5SeEKUmUwIb/+p4QAKf6J/qR0aQ2AQAAAACBBmptJ4Q6JlMFNEw3//qeEAD++ys8+P8TXGqIPW8sxYQAAABABnrpqQr8ANMCxr3mlZw3AAAAAH0Gav0nhDyZTAhv//qeEAJd8OfMssTI7iOGeXl+9LUEAAAAQQZ7dRRE8L/8AWugEq7E9fQAAAA8Bnvx0Qr8AT7oB0JyXpMAAAAAQAZ7+akK/AHmZ4F1/bh9eYAAAABlBmuBJqEFomUwIb//+p4QAmo+Y8jE/y21DAAAAEUGbBEnhClJlMCG//qeEAAEnAAAADEGfIkU0TC//AACygQAAABABn0F0Qr8AexQ3suq/gRfAAAAAEAGfQ2pCvwB7FDexWj7dfkEAAAAZQZtFSahBaJlMCG///qeEAJ6PmPIxP8ttOwAAABlBm2lJ4QpSZTAhv/6nhADy++z7gt3NL2i5AAAAEEGfh0U0TC//AJLQEHLdTWkAAAAQAZ+mdEK/AMjZV3V+O79/wAAAABABn6hqQr8AfdQ3sVo+3XtAAAAAGkGbqkmoQWiZTAhv//6nhADxnGf6rfMfiDZhAAAAGUGby0nhClJlMCHf/qmWAMoJhui3cx+AouAAAAAaQZvvSeEOiZTAhv/+p4QD78BgE1/fiqeny4gAAAAVQZ4NRRE8L/8BW0kvzNuEwD+vcw2ZAAAAEAGeLHRCvwHSRMiOxZijTekAAAAQAZ4uakK/Ad8PBrjwps6vIQAAABxBmjBJqEFomUwId//+qZYCJEw3RLOXQOH9m0vAAAAAFkGaVEnhClJlMCHf/qmWAmPZj8rgg4AAAAAOQZ5yRTRML/8Bb/2tR8EAAAAPAZ6RdEK/AewkDRBc6eEDAAAAEAGek2pCvwHYrW7AqPtwhYAAAAASQZqYSahBaJlMCG///qeEAAEnAAAADEGetkURLC//AACygAAAAA8BntV0Qr8B7CQNEFzp4QMAAAAQAZ7XakK/AditbsCo+3CFgQAAABlBmttJqEFsmUwIb//+p4QES7Mbz4LadtI+AAAAEUGe+UUVLCv/AewPBrjLBfknAAAADgGfGmpCvwHrs6dBwqScAAAAHEGbHEmoQWyZTAh3//6plgIB2qgcPhagn4eIqYEAAAASQZsgSeEKUmUwId/+qZYAAJWBAAAADEGfXkU0TC//AACygAAAAA8Bn310Qr8B0iQNEFzqWRcAAAAPAZ9/akK/AdGF7p1G5dFbAAAAIEGbZEmoQWiZTAh3//6plgHV4hAP7+dRjogW4mqePtP8AAAAEUGfgkURLC//AVFlioJwgkXBAAAAEAGfoXRCvwHSJA1tMoejYMAAAAAPAZ+jakK/AS8NA8mCLNmBAAAAGUGbqEmoQWyZTAh3//6plgHNCf93jz+RYz8AAAAQQZ/GRRUsL/8BUWWKhBPg4QAAABABn+V0Qr8B0mlYtjZUo90xAAAADgGf52pCvwHRhe9UlNkXAAAAE0Gb7EmoQWyZTAh3//6plgAAlYAAAAAMQZ4KRRUsL/8AALKBAAAADwGeKXRCvwHSJA0QXOpZFwAAAA8BnitqQr8B0YXunUbl0VsAAAATQZowSahBbJlMCHf//qmWAACVgQAAAAxBnk5FFSwv/wAAsoEAAAAPAZ5tdEK/AdIkDRBc6lkXAAAADwGeb2pCvwHRhe6dRuXRWwAAABNBmnRJqEFsmUwId//+qZYAAJWAAAAADEGekkUVLC//AACygQAAAA8BnrF0Qr8B0iQNEFzqWRcAAAAPAZ6zakK/AdGF7p1G5dFbAAAAE0GauEmoQWyZTAh3//6plgAAlYEAAAAMQZ7WRRUsL/8AALKAAAAADwGe9XRCvwHSJA0QXOpZFwAAAA8BnvdqQr8B0YXunUbl0VsAAAATQZr8SahBbJlMCHf//qmWAACVgAAAAAxBnxpFFSwv/wAAsoEAAAAPAZ85dEK/AdIkDRBc6lkXAAAADwGfO2pCvwHRhe6dRuXRWwAAABNBmyBJqEFsmUwId//+qZYAAJWBAAAADEGfXkUVLC//AACygAAAAA8Bn310Qr8B0iQNEFzqWRcAAAAPAZ9/akK/AdGF7p1G5dFbAAAAE0GbZEmoQWyZTAh3//6plgAAlYAAAAAMQZ+CRRUsL/8AALKBAAAADwGfoXRCvwHSJA0QXOpZFwAAAA8Bn6NqQr8B0YXunUbl0VsAAAAeQZuoSahBbJlMCHf//qmWAdXiEA/v51GOiBbjAsZ9AAAAEEGfxkUVLC//AVGgRWlE+DkAAAAPAZ/ldEK/AdIkDXXxaK2BAAAAEAGf52pCvwHGCJmm+kg4kXAAAAAaQZvsSahBbJlMCHf//qmWAczUvu8kj/zplbAAAAAVQZ4KRRUsL/8BZUkuTNvkhkFonbQRAAAAEAGeKXRCvwHfQrVeA1dlrYAAAAAPAZ4rakK/Ad6x1RpRKWLiAAAAE0GaMEmoQWyZTAh3//6plgAAlYEAAAAMQZ5ORRUsL/8AALKBAAAADwGebXRCvwHSJA0QXOpZFwAAAA8Bnm9qQr8B0YXunUbl0VsAAAATQZp0SahBbJlMCHf//qmWAACVgAAAAAxBnpJFFSwv/wAAsoEAAAAPAZ6xdEK/AdIkDRBc6lkXAAAADwGes2pCvwHRhe6dRuXRWwAAABNBmrhJqEFsmUwId//+qZYAAJWBAAAADEGe1kUVLC//AACygAAAAA8BnvV0Qr8B0iQNEFzqWRcAAAAPAZ73akK/AdGF7p1G5dFbAAAAE0Ga/EmoQWyZTAh3//6plgAAlYAAAAAMQZ8aRRUsL/8AALKBAAAADwGfOXRCvwHSJA0QXOpZFwAAAA8BnztqQr8B0YXunUbl0VsAAAASQZsgSahBbJlMCG///qeEAAEnAAAADEGfXkUVLC//AACygAAAAA8Bn310Qr8B0iQNEFzqWRcAAAAPAZ9/akK/AdGF7p1G5dFbAAAAGkGbYUmoQWyZTAh3//6plgHz1PynDH6HiKmAAAAAEkGbhUnhClJlMCHf/qmWAACVgQAAAAxBn6NFNEwv/wAAsoAAAAAPAZ/CdEK/Ad8kDRBc6eELAAAADwGfxGpCvwHMSAXWerPRWwAAABxBm8lJqEFomUwId//+qZYCAdmPzPglnKDcE4bNAAAAEEGf50URLC//AVtV1vBAXTEAAAAPAZ4GdEK/AdHnQGSXKMqAAAAAEAGeCGpCvwHRsv1R9A+AeMAAAAAaQZoLSahBbJlMFEw7//6plgH07nn72l9lidkAAAAQAZ4qakK/AdIfhDxoaxiqgAAAABJBmi9J4QpSZTAh3/6plgAAlYAAAAAMQZ5NRTRML/8AALKBAAAADwGebHRCvwHfJA0QXOnhCwAAAA8Bnm5qQr8BzEgF1nqz0VsAAAATQZpzSahBaJlMCHf//qmWAACVgAAAAAxBnpFFESwv/wAAsoAAAAAQAZ6wdEK/AcxIBiOy7KjKgQAAAA8BnrJqQr8BzEgF1nqz0VsAAAATQZq3SahBbJlMCHf//qmWAACVgAAAABRBntVFFSwv/wFW4i9VMcfRZSf6aQAAABABnvR0Qr8B0X8Bklv9bLKAAAAAEAGe9mpCvwHSH4Q8aGsYqoEAAAATQZr7SahBbJlMCHf//qmWAACVgQAAAAxBnxlFFSwv/wAAsoAAAAAPAZ84dEK/Ad8kDRBc6eELAAAADwGfOmpCvwHehe6dRuTxSQAAABNBmz9JqEFsmUwId//+qZYAAJWBAAAADEGfXUUVLC//AACygQAAABABn3x0Qr8BzEgGI7LsqMqAAAAADwGffmpCvwHMSAXWerPRWwAAABNBm2NJqEFsmUwId//+qZYAAJWBAAAADEGfgUUVLC//AACygAAAABABn6B0Qr8BzEgGI7LsqMqBAAAADwGfompCvwHMSAXWerPRWwAAABNBm6dJqEFsmUwId//+qZYAAJWBAAAADEGfxUUVLC//AACygQAAABABn+R0Qr8BzEgGI7LsqMqBAAAADwGf5mpCvwHMSAXWerPRWwAAABNBm+tJqEFsmUwId//+qZYAAJWAAAAADEGeCUUVLC//AACygAAAABABnih0Qr8BzEgGI7LsqMqBAAAADwGeKmpCvwHMSAXWerPRWwAAABNBmi9JqEFsmUwId//+qZYAAJWAAAAADEGeTUUVLC//AACygQAAABABnmx0Qr8BzEgGI7LsqMqBAAAADwGebmpCvwHehe6dRuTxSQAAABNBmnNJqEFsmUwId//+qZYAAJWAAAAADEGekUUVLC//AACygAAAABABnrB0Qr8BzEgGI7LsqMqBAAAADwGesmpCvwHMSAXWerPRWwAAABNBmrdJqEFsmUwId//+qZYAAJWAAAAADEGe1UUVLC//AACygQAAABABnvR0Qr8BzEgGI7LsqMqAAAAADwGe9mpCvwHMSAXWerPRWwAAABNBmvtJqEFsmUwId//+qZYAAJWBAAAADEGfGUUVLC//AACygAAAABABnzh0Qr8BzEgGI7LsqMqBAAAADwGfOmpCvwHMSAXWerPRWwAAABJBmz9JqEFsmUwIb//+p4QAAScAAAAMQZ9dRRUsL/8AALKBAAAAEAGffHRCvwHMSAYjsuyoyoAAAAAPAZ9+akK/Ad6F7p1G5PFJAAAAEkGbY0moQWyZTAhv//6nhAABJwAAAAxBn4FFFSwv/wAAsoAAAAAQAZ+gdEK/AcxIBiOy7KjKgQAAAA8Bn6JqQr8BzEgF1nqz0VsAAAASQZunSahBbJlMCGf//p4QAAR9AAAADEGfxUUVLC//AACygQAAABABn+R0Qr8BzEgGI7LsqMqBAAAADwGf5mpCvwHMSAXWerPRWwAAABtBm+lLqEIQWyRGCCgH8gH9h4BRMK/+OEAAEXAAAAAjAZ4IakK/Aq9j7UHE3arDzNeJoFdnSLQGuGbr8HtGndA7iwYAAAwwbW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAAH5AAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAC1p0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAAH5AAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAARAAAAEQAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAB+QAAAEAAABAAAAAArSbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAAAAyAAABlABVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAAAAAABWaWRlb0hhbmRsZXIAAAAKfW1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAACj1zdGJsAAAAlXN0c2QAAAAAAAAAAQAAAIVhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAARABEABIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAL2F2Y0MB9AAN/+EAF2f0AA2RmygiEdCAAAADAIAAABkHihTLAQAFaOvjxEgAAAAYc3R0cwAAAAAAAAABAAAAygAAAgAAAAAUc3RzcwAAAAAAAAABAAAAAQAABghjdHRzAAAAAAAAAL8AAAABAAAEAAAAAAEAAAYAAAAAAQAAAgAAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAGAAAAAAEAAAIAAAAABgAABAAAAAABAAAIAAAAAAIAAAIAAAAAAQAACAAAAAACAAACAAAAAAIAAAQAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACAAAAAACAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAFiwAAABAAAAAUAAAAHAAAABwAAAAhAAAAFQAAABMAAAAUAAAAHQAAACEAAAAUAAAAHAAAAB8AAAAcAAAAGwAAAB0AAAAcAAAAHwAAABYAAAASAAAAHQAAABMAAAATAAAAIAAAAB0AAAAkAAAAFAAAACMAAAAUAAAAEwAAABQAAAAdAAAAFQAAABAAAAAUAAAAFAAAAB0AAAAdAAAAFAAAABQAAAAUAAAAHgAAAB0AAAAeAAAAGQAAABQAAAAUAAAAIAAAABoAAAASAAAAEwAAABQAAAAWAAAAEAAAABMAAAAUAAAAHQAAABUAAAASAAAAIAAAABYAAAAQAAAAEwAAABMAAAAkAAAAFQAAABQAAAATAAAAHQAAABQAAAAUAAAAEgAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAiAAAAFAAAABMAAAAUAAAAHgAAABkAAAAUAAAAEwAAABcAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAATAAAAEwAAABcAAAAQAAAAEwAAABMAAAAWAAAAEAAAABMAAAATAAAAHgAAABYAAAAQAAAAEwAAABMAAAAgAAAAFAAAABMAAAAUAAAAHgAAABQAAAAWAAAAEAAAABMAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAYAAAAFAAAABQAAAAXAAAAEAAAABMAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAXAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAWAAAAEAAAABQAAAATAAAAFgAAABAAAAAUAAAAEwAAABYAAAAQAAAAFAAAABMAAAAfAAAAJwAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1OC4xMi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "epochs_train = 101\n",
    "prefix='fc_train'\n",
    "env = Environment(grid_size=size, max_time=T, temperature=0.3)\n",
    "agent = DQN_FC(size, lr=.1, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "train(agent, env, epochs_train, prefix=prefix)\n",
    "name = sorted(list(filter(lambda x:  prefix in x and 'mp4' in x, os.listdir(prefix))))[-1]\n",
    "HTML(display_videos(name,prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "__Question 8__ Implement the DQN training algorithm using a CNN (for example, 2 convolutional layers and one final fully connected layer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DQN_CNN(DQN):\n",
    "    def __init__(self, *args,lr=0.1,**kwargs):\n",
    "        super(DQN_CNN, self).__init__(*args,**kwargs)\n",
    "        \n",
    "        ###### FILL IN\n",
    "        model = Sequential()\n",
    "        model.add(Conv2D(15,(3,3), input_shape=(5,5, self.n_state,), activation='relu'))\n",
    "        model.add(Conv2D(8,(3,3), activation='relu'))\n",
    "        model.add(Flatten())\n",
    "        model.add(Dense(15, activation='relu'))\n",
    "        model.add(Dense(4, activation='linear'))\n",
    "        model.compile(sgd(lr=lr, decay=1e-4, momentum=0.0), \"mse\")\n",
    "        self.model = model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 000/101 | Loss 0.0060 | Win/lose count 5.0/6.0 (-1.0)\n",
      "Epoch 001/101 | Loss 0.0173 | Win/lose count 4.0/3.0 (1.0)\n",
      "Epoch 002/101 | Loss 0.0088 | Win/lose count 3.0/7.0 (-4.0)\n",
      "Epoch 003/101 | Loss 0.0072 | Win/lose count 7.5/5.0 (2.5)\n",
      "Epoch 004/101 | Loss 0.0094 | Win/lose count 4.0/2.0 (2.0)\n",
      "Epoch 005/101 | Loss 0.0008 | Win/lose count 6.0/1.0 (5.0)\n",
      "Epoch 006/101 | Loss 0.0051 | Win/lose count 3.0/4.0 (-1.0)\n",
      "Epoch 007/101 | Loss 0.0191 | Win/lose count 7.5/11.0 (-3.5)\n",
      "Epoch 008/101 | Loss 0.0030 | Win/lose count 9.5/10.0 (-0.5)\n",
      "Epoch 009/101 | Loss 0.0037 | Win/lose count 3.0/4.0 (-1.0)\n",
      "Epoch 010/101 | Loss 0.0060 | Win/lose count 4.0/2.0 (2.0)\n",
      "Epoch 011/101 | Loss 0.0074 | Win/lose count 4.0/7.0 (-3.0)\n",
      "Epoch 012/101 | Loss 0.0080 | Win/lose count 6.5/7.0 (-0.5)\n",
      "Epoch 013/101 | Loss 0.0205 | Win/lose count 6.0/1.0 (5.0)\n",
      "Epoch 014/101 | Loss 0.0101 | Win/lose count 5.5/4.0 (1.5)\n",
      "Epoch 015/101 | Loss 0.0067 | Win/lose count 6.0/9.0 (-3.0)\n",
      "Epoch 016/101 | Loss 0.0007 | Win/lose count 2.0/2.0 (0.0)\n",
      "Epoch 017/101 | Loss 0.0021 | Win/lose count 6.5/3.0 (3.5)\n",
      "Epoch 018/101 | Loss 0.0004 | Win/lose count 8.5/6.0 (2.5)\n",
      "Epoch 019/101 | Loss 0.0211 | Win/lose count 7.0/3.0 (4.0)\n",
      "Epoch 020/101 | Loss 0.0101 | Win/lose count 7.5/1.0 (6.5)\n",
      "Epoch 021/101 | Loss 0.0177 | Win/lose count 9.5/3.0 (6.5)\n",
      "Epoch 022/101 | Loss 0.0212 | Win/lose count 13.0/5.0 (8.0)\n",
      "Epoch 023/101 | Loss 0.0180 | Win/lose count 3.5/4.0 (-0.5)\n",
      "Epoch 024/101 | Loss 0.0030 | Win/lose count 3.5/3.0 (0.5)\n",
      "Epoch 025/101 | Loss 0.0019 | Win/lose count 7.0/2.0 (5.0)\n",
      "Epoch 026/101 | Loss 0.0025 | Win/lose count 7.0/4.0 (3.0)\n",
      "Epoch 027/101 | Loss 0.0053 | Win/lose count 8.0/1.0 (7.0)\n",
      "Epoch 028/101 | Loss 0.0149 | Win/lose count 8.0/4.0 (4.0)\n",
      "Epoch 029/101 | Loss 0.0080 | Win/lose count 8.5/2.0 (6.5)\n",
      "Epoch 030/101 | Loss 0.0108 | Win/lose count 8.0/6.0 (2.0)\n",
      "Epoch 031/101 | Loss 0.0220 | Win/lose count 5.0/2.0 (3.0)\n",
      "Epoch 032/101 | Loss 0.0043 | Win/lose count 11.5/1.0 (10.5)\n",
      "Epoch 033/101 | Loss 0.0054 | Win/lose count 8.0/4.0 (4.0)\n",
      "Epoch 034/101 | Loss 0.0120 | Win/lose count 12.0/7.0 (5.0)\n",
      "Epoch 035/101 | Loss 0.0132 | Win/lose count 3.5/2.0 (1.5)\n",
      "Epoch 036/101 | Loss 0.0132 | Win/lose count 4.0/4.0 (0.0)\n",
      "Epoch 037/101 | Loss 0.0043 | Win/lose count 12.5/3.0 (9.5)\n",
      "Epoch 038/101 | Loss 0.0100 | Win/lose count 12.5/4.0 (8.5)\n",
      "Epoch 039/101 | Loss 0.0026 | Win/lose count 9.5/1.0 (8.5)\n",
      "Epoch 040/101 | Loss 0.0068 | Win/lose count 7.5/3.0 (4.5)\n",
      "Epoch 041/101 | Loss 0.0142 | Win/lose count 7.0/5.0 (2.0)\n",
      "Epoch 042/101 | Loss 0.0194 | Win/lose count 7.5/2.0 (5.5)\n",
      "Epoch 043/101 | Loss 0.0027 | Win/lose count 4.0/2.0 (2.0)\n",
      "Epoch 044/101 | Loss 0.0010 | Win/lose count 4.5/2.0 (2.5)\n",
      "Epoch 045/101 | Loss 0.0088 | Win/lose count 6.0/3.0 (3.0)\n",
      "Epoch 046/101 | Loss 0.0032 | Win/lose count 6.5/6.0 (0.5)\n",
      "Epoch 047/101 | Loss 0.0107 | Win/lose count 9.0/3.0 (6.0)\n",
      "Epoch 048/101 | Loss 0.0066 | Win/lose count 15.5/3.0 (12.5)\n",
      "Epoch 049/101 | Loss 0.0059 | Win/lose count 5.5/3.0 (2.5)\n",
      "Epoch 050/101 | Loss 0.0016 | Win/lose count 8.0/4.0 (4.0)\n",
      "Epoch 051/101 | Loss 0.0061 | Win/lose count 12.0/2.0 (10.0)\n",
      "Epoch 052/101 | Loss 0.0017 | Win/lose count 4.0/3.0 (1.0)\n",
      "Epoch 053/101 | Loss 0.0104 | Win/lose count 6.0/2.0 (4.0)\n",
      "Epoch 054/101 | Loss 0.0077 | Win/lose count 2.0/4.0 (-2.0)\n",
      "Epoch 055/101 | Loss 0.0038 | Win/lose count 7.0/3.0 (4.0)\n",
      "Epoch 056/101 | Loss 0.0069 | Win/lose count 5.5/1.0 (4.5)\n",
      "Epoch 057/101 | Loss 0.0088 | Win/lose count 6.0/4.0 (2.0)\n",
      "Epoch 058/101 | Loss 0.0098 | Win/lose count 10.5/4.0 (6.5)\n",
      "Epoch 059/101 | Loss 0.0072 | Win/lose count 8.5/3.0 (5.5)\n",
      "Epoch 060/101 | Loss 0.0084 | Win/lose count 4.0/2.0 (2.0)\n",
      "Epoch 061/101 | Loss 0.0124 | Win/lose count 5.5/3.0 (2.5)\n",
      "Epoch 062/101 | Loss 0.0160 | Win/lose count 4.5/2.0 (2.5)\n",
      "Epoch 063/101 | Loss 0.0030 | Win/lose count 10.0/3.0 (7.0)\n",
      "Epoch 064/101 | Loss 0.0089 | Win/lose count 4.5/1.0 (3.5)\n",
      "Epoch 065/101 | Loss 0.0095 | Win/lose count 7.0/6.0 (1.0)\n",
      "Epoch 066/101 | Loss 0.0026 | Win/lose count 9.5/1.0 (8.5)\n",
      "Epoch 067/101 | Loss 0.0108 | Win/lose count 9.5/4.0 (5.5)\n",
      "Epoch 068/101 | Loss 0.0075 | Win/lose count 4.5/2.0 (2.5)\n",
      "Epoch 069/101 | Loss 0.0097 | Win/lose count 7.5/4.0 (3.5)\n",
      "Epoch 070/101 | Loss 0.0049 | Win/lose count 4.0/3.0 (1.0)\n",
      "Epoch 071/101 | Loss 0.0048 | Win/lose count 1.5/5.0 (-3.5)\n",
      "Epoch 072/101 | Loss 0.0010 | Win/lose count 5.5/3.0 (2.5)\n",
      "Epoch 073/101 | Loss 0.0195 | Win/lose count 10.5/0 (10.5)\n",
      "Epoch 074/101 | Loss 0.0027 | Win/lose count 7.5/2.0 (5.5)\n",
      "Epoch 075/101 | Loss 0.0038 | Win/lose count 3.0/2.0 (1.0)\n",
      "Epoch 076/101 | Loss 0.0080 | Win/lose count 12.0/2.0 (10.0)\n",
      "Epoch 077/101 | Loss 0.0029 | Win/lose count 5.0/5.0 (0.0)\n",
      "Epoch 078/101 | Loss 0.0045 | Win/lose count 5.5/0 (5.5)\n",
      "Epoch 079/101 | Loss 0.0078 | Win/lose count 5.0/3.0 (2.0)\n",
      "Epoch 080/101 | Loss 0.0090 | Win/lose count 4.0/3.0 (1.0)\n",
      "Epoch 081/101 | Loss 0.0046 | Win/lose count 2.5/2.0 (0.5)\n",
      "Epoch 082/101 | Loss 0.0199 | Win/lose count 2.5/1.0 (1.5)\n",
      "Epoch 083/101 | Loss 0.0019 | Win/lose count 8.5/2.0 (6.5)\n",
      "Epoch 084/101 | Loss 0.0134 | Win/lose count 6.0/2.0 (4.0)\n",
      "Epoch 085/101 | Loss 0.0078 | Win/lose count 2.5/2.0 (0.5)\n",
      "Epoch 086/101 | Loss 0.0010 | Win/lose count 3.5/0 (3.5)\n",
      "Epoch 087/101 | Loss 0.0023 | Win/lose count 8.0/2.0 (6.0)\n",
      "Epoch 088/101 | Loss 0.0152 | Win/lose count 9.5/1.0 (8.5)\n",
      "Epoch 089/101 | Loss 0.0018 | Win/lose count 9.0/6.0 (3.0)\n",
      "Epoch 090/101 | Loss 0.0104 | Win/lose count 8.0/4.0 (4.0)\n",
      "Epoch 091/101 | Loss 0.0067 | Win/lose count 6.0/3.0 (3.0)\n",
      "Epoch 092/101 | Loss 0.0070 | Win/lose count 5.5/2.0 (3.5)\n",
      "Epoch 093/101 | Loss 0.0052 | Win/lose count 5.0/3.0 (2.0)\n",
      "Epoch 094/101 | Loss 0.0017 | Win/lose count 6.5/2.0 (4.5)\n",
      "Epoch 095/101 | Loss 0.0045 | Win/lose count 7.0/1.0 (6.0)\n",
      "Epoch 096/101 | Loss 0.0065 | Win/lose count 6.5/3.0 (3.5)\n",
      "Epoch 097/101 | Loss 0.0048 | Win/lose count 14.5/4.0 (10.5)\n",
      "Epoch 098/101 | Loss 0.0103 | Win/lose count 13.5/2.0 (11.5)\n",
      "Epoch 099/101 | Loss 0.0018 | Win/lose count 5.5/5.0 (0.5)\n",
      "Epoch 100/101 | Loss 0.0024 | Win/lose count 7.0/2.0 (5.0)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8XHW9//HXJ5N9afZ0SdKme1m6QSj7jggqsigILoCiXAXXn8vDe/X3U+/1evGK14uKC4iAbCoIiKgoSAu00h1oS9d0SZt0yWTfM5OZ7++PmYS0Tdu0mWQyM+/n45FHM3POnPM9Oe2733zO93yPOecQEZH4kRTtBoiISGQp2EVE4oyCXUQkzijYRUTijIJdRCTOKNhFROKMgl1EJM4o2GVEmdkSM2sys7RotyUSzOwUM/t7+JiazWyNmb0n2u0SGUjBLiPGzCqA8wEHvH+E9pE8Ets9ij8BLwLjgRLg80BrJHcQhWOSOKNgl5F0M7AceAi4pe9NMzvLzPabmWfAe9ea2brw90lm9nUz225mDWb2ezMrCC+rMDNnZreZ2W7g5fD7T4a32WJmr5rZKQO2XWhmfzKzVjNbZWbfNbOlA5bPMbMXzazRzLaY2Q2DHYyZFQFTgfudc77w1zLn3MBtXW1mb4b3td3Mrgi/P8nMngvvo8rMPjXgM982s6fM7FEzawVuPcbPID28bkP4t4ZVZjb+xE+TxBsFu4ykm4HHwl/v7gsf59xyoAO4ZMC6HwYeD3//eeAa4EJgEtAE3HvIti8ETgLeHX79V2AmoV702vA++9wb3t8EQv/BDPxPJotQD/zx8GdvAn428D+GARqAKuBRM7vm0DA1s0XAb4CvAnnABcCu8OIngJrw8XwQ+J6ZXTrg41cDT4U/99gxfga3ALlAOVAIfBroGqS9kqicc/rSV8S/gPMAP1AUfr0Z+NKA5d8Ffh3+PodQ8E4Jv94EXDpg3YnhbSUDFYRKO9OOsu+88Dq5gCf82dmH7Htp+PsPAa8d8vlfAt86wrbLgJ8C24Eg8Cowc8DnfjTIZ8qBAJAz4L3/Ah4Kf/9t4NVDPnO0n8EngH8C86J9nvU1Nr/UY5eRcgvwd+dcffj14wzoKYdfXxe+qHodsNY5Vx1eNgV4JlxmaCYUcgFCde0+e/q+MTOPmd0VLlu08k4vuQgoJhSGewb7bHhfZ/btK7y/jxDq3R/GOVfjnPusc256+LMdhHrpEArw7YN8bBLQ6JxrG/BeNVB6hDYd62fwCPA34LdmttfM/tvMUgZrryQmXaSRiDOzDOAGwGNm+8NvpwF5ZjbfOfeWc26jmVUDV3JwGQZCIfcJ59yyQbZdEf524LSkHyZUyriMUKjnEipdGOAFegn1tLeG1y8/ZF+vOOfedbzH6ZzbY2b3Eiqz9G1r+iCr7gUKzCxnQLhPBmoHbu6QzxzxZxD2HeA74Z/HX4AtwAPHewwSn9Rjl5FwDaHe5cnAgvDXScBrhOrufR4nVEu+AHhywPu/AP7TzKYAmFmxmV19lP3lAD2EauCZwPf6FjjnAsDTwLfNLNPM5hzShueBWWb2MTNLCX+dYWYnHboTM8s3s++Y2Yzwxc0iQmWR5eFVHgA+bmaXhpeXmtkc59weQqWT/wpf+JwH3MbB1wEOdcSfgZldbGZzwxefWwmVaAJH2ZYkGAW7jIRbgAedc7udc/v7vgjVpj8yYDjfE8BFwMsDSjYA9wDPAX83szZCwXnmUfb3G0KljVpgI+8EbZ/PEurF7ydUxniC0H8EhHvQlwM3EupZ7we+T+g3jEP5CNX4XyIUqBvC27k1vK2VwMeBHwEtwCuESioQuihbEd7HM4Rq+C8e5ZiO9jOYQOhCayuhEs0rwKNH2ZYkGHNOD9qQxGJm3wcmOOduOebKIjFIPXaJe+Fx6vMsZBGhMsgz0W6XyEjRxVNJBDmEyi+TgDrgh8Afo9oikRGkUoyISJxRKUZEJM5EpRRTVFTkKioqorFrEZGYtWbNmnrnXPGx1otKsFdUVLB69epo7FpEJGaFb+o7JpViRETijIJdRCTOKNhFROKMgl1EJM4o2EVE4oyCXUQkzijYRUTijIJdRCSCWjr9PLB0Jzu87Yct63t03UjTJGAiIhHS5Qvw8YdWsnZ3M//xPJwzvZCr5k9if0s362tbWFfTwi8/djqnT8kf0XYo2EVEIsAfCHLHY2t4Y08zd103l4YOH4+v2M2/Pr2eJIMZJdlcMKuIjBTPiLdFwS4iMkzBoONrT61j8RYv37t2LjcumgzApy+cTlVdO2X5GWSljV7cKthFRIYhGHR8848beOaNWr5y+Sw+fObk/mWeJGP2hJxRb5OCXUTkBAWCjq//YR1PrqnhMxdN586LZ0S7SYCCXURkyAJBx57GTnyBIL7eIA8s3ckzb9TyhUtn8sXLZmJm0W4ioGAXkQipbe5iUm76mAm3wQSDjoYOH93+AFlpyWSmekj1JIWCOhAk1ZNE+iAXN3d42/nD2hqeWVvL3pbug5Z95fJZfPaSmaN1CEOiYBeRYfvFK9u566+bOWd6If/vqpOZM2EcwaBj+c4GFm+uo7wgk4Xl+cyZmEOKZ+RvnznQ2s0XfvsGb+xuJistmaw0D/5eh7e9h0DwyOPIkwymFmVxyqRcCrJS2e5tZ9uBdva3dpNkcMGsYj536Uxy0pNJ8SRRkpPGwskjO3TxRCjYReLUhtoWdjV0MLUoi6lFWWSmJuOco6c3iCfJIhawL2zYx11/3cwZFfls3NfKe+55jStOncBbe1qobe4iOcnoDYdpRoqHy04ezw2VZZw7vYikpMj37lfubOSOx9bS6evlpkWT6Q0G6ewJ4EkySsalMX5cOukpHjp7eunwBfAHgqQmJ5HqSaK1u5eNe1tZU91EY4eP6SVZnDO9kFNKc3nfvImMH5ce8faOBAW7SBwJBB0vbjzAA0t3sGpX00HLslI9dPkDBB3kpCVzfWU5t5wzhSmFWUPa9s76Dp5YuZu11U1cdvJ4Pnh6GXubu/ji795k4eQ8HrntTLr9Af73pW38fvUeTp+Sz9eumM27T5lAQ4ePtdVNLN/RwPPr9vGnt/ZSmpfBR86azEcWTSE3M2XQfTZ3+tju7WBvcxf7W7qZMzGH82YUDVruaerw8ZvXq/nJy9soL8jk8U+dyazxoz8iZSyw0bi99VCVlZVOj8YTiay+ux6X72ikLD+DW8+p4KxphVQ3dLLD205Tp5/MVA8ZqR627G/jL+v3EXCOq+ZN4u7r55OaPHgPfld9B994dj3LqhrwJBkzS7LZvL+NFI+RnuIhNyOFZ+44l+KctCG1s9sf4MWNB3hi5W7+ub2BzFQPHzqjnGsXlnLqpFySkoyG9h5+tmQ7jyyvxtcbPOjzM0qyueWcCuaV5tLpC9DR08vfN+7nj2/upac3yBWnTOC/r5/HuPTB/7OIZWa2xjlXecz1FOwiscMfCNLW3Utrl5/inLT+m158vUFuf2Q1r2z18h9Xn8qNZ5STfIxSy4HWbn69bCe/fGUHt5w9he9cfeph63T5Alxz7zL2t3bzqfOnckNlOSXj0qmqa+PxFXtYXd3I3dfPP+Ge8dt7W3jgtZ0899ZeeoOOwqxUTp+Sz7Kqerr8AT54ehlXnjqR0vwMirPTWLK1jgeX7WJdTctB28lI8XDtaaXcfPYU5kwYd0JtiQUKdpFhau32j2ivr7HDR2aqZ9BRGIfa09jJnY+vPSjQctKSuXFROTefXcH3X9jM8+v28b1r5x50g8xQ/OefN3L/azv5wQfncX1l+UHLvvbUWzy5poaHP76IC2YVH9d2j0dDew+vbavnla1eVuxoYOGUfP7Pu2YxvTj7sHWdc6yvbcHb1kNGqoeMFA/TS7Ljsod+KAW7yDD8duVuvv70eq5ZMImvvHs2ZfmZQChU2np6hxUitc1d3PPSVp5aU0OyJ4nKKfn9F+imFmZRmp9x0IXNf1bVc+fjawkEHbeeO5WCzBSy0pJ5bVs9f16/r3+Ux79eOYd/uXD6cbenNxDk5l+vZHV1E099+mzmleUB8Ic1NXz5ybf47MUz+Mq7Z5/w8UrkKNhFTlBbt5+LfrCEjFQP3rYeHHDNgkl423p4q6aFpk4fP73pNN47b+JxbbfbH+Duv23hN69Xg8GHF00mOclYtr2BTfta+9dLTjJK8zOYUphFYVYqz721l2lFWdx/cyUVRQdf6Kxt7uKR16spyUnjE+dNPeFjbuzwcdVPltLc6WPG+BzK8jJ4eXMdc8tyefyTZx6zrCOjQ8EucoK+/8Jmfr5kO8999lyKstP44d+38qe39jK1KIv55bls2tfGdm87z9xx7pDnAdnd0MmnH13Dxn2tfKiynC9cNpNJeRn9y5s6fGz3trOzvoNdDR1UN3RS3dDJ7sZOzptZxPc/MI/sEZ5EqqqujV+9tpOapi5qm7vISvPwwC1nxMwQv0SgYBc5AbXNXVx89xLeO3ciP/rQgv73nXP9Q+wOtHbzvp8sJTPVw3N3ntc/VK/LF2DJljr+vH4fS6vqmVIYGgM9KS+DH7ywGYD/vXEBl8wZP/oHJnFhqMGucewiA/zghc0Y8NVDasoDx02PH5fOzz9yGjfdv5w7Hl/DvLI81lY38VZNM93+IIVZqVwyu4TdjZ3c/+oOeoOOUyaN4xcfPZ3ygsxRPiJJRMMOdjNLB14F0sLbe8o5963hbldkpLR2+1m+vYHS/AymFmXhSTLe2N3M0m31PPvmXu68ePpBZZLBVFYU8K2rTuGbz25gxY5GTpk0jpsWTeayk8Zz5tSC/pp0R08vWw+0cdLEcUMa/SISCZHosfcAlzjn2s0sBVhqZn91zi2PwLZFIu6Hf9vCw69X979O8Rj+gMMMzp5WyKeHOLLko2dN6S+1HCm0s9KSx+RcIhLfhh3sLlSk73tqa0r4a/QL9yJD0O0P8Oybe7l0TgnXnVbGroYOWrv9nD45nzOnFh7x1vYjmTbIOGuRaItIjd3MPMAaYAZwr3NuxSDr3A7cDjB58vHdQCESKS9tOkBLl59bz63g/Jkjd8ONSDRFZHCqcy7gnFsAlAGLzOywe5Odc/c55yqdc5XFxfoHJdHx5OoaJuWmc870omg3RWTERPSuA+dcM7AEuCKS2xWJhP0t3by2zcsHTi/DMwLTxYqMFcMOdjMrNrO88PcZwGXA5uFuVyTS/rC2hqCDD55eFu2miIyoSNTYJwIPh+vsScDvnXPPR2C7IhHjnOOpNTUsmlow5PnHRWJVJEbFrAMWRqAtIke1rKqeHfUdnD2tgOnF2cf1bM011U3srO/gjouOf5IskVijO09lzHPO8dOXq/jhi1v73yvJSWNeWS4l49IpyUkjI8VDR08vbT29lOVn8rGzpvQ/OGJPYydfe2odOWnJvGfu8U3cJRKLFOwypnjbevj5ku10+QOcPb2Q06fkc/fftvDMG7Vcu7CUOy+ewapdjSyrqqeqrp03djfT0OEDwAwyUzx0+AI8taaGu6+fh2Hc8uBKfL1BHvz4Gf0PphCJZ5oETMaEbn+AXy/byc8Wb6fbHyAjxUNbT2//8q9cPos7L54xaPnFHwji6w2SkeIhKcn4+9v7+cazG2jq8JGWnMS4jBQe/sSihH3+pcQPTQImMSMQdNzwy9dZV9PCZSeN59/eM4fJBZm8vbeVFTsbmD1hHBce5ek9KZ6kgx5McfkpE1g0tYDv/nkTO7zt/PTDpx1z7heReKJgl6j701t7WVfTwl3XzeXGRe/clTy/PI/55XkntM28zFTuvn5+pJooElP0WBSJqt5AkB//YxtzJuRwwyHP2xSRE6Ngl6j645t72VHfwRcvm0mS7gYViQgFu0SNPxDkxy9v4+SJ47j85AnRbo5I3FCwS9Q8s7aW6oZOvvSuWeqti0SQgl2iwh8I8pPF25hbmstlJ5VEuzkicUXBLlHx3Jt72dPYxecvnXlcUwOIyLEp2GXUBYKOny2pYs6EHPXWRUaAgl1G3Qsb9rPd28FnLxn8TlIRGR4Fu4wq5xw/XVzFtOIsrjxVE3KJjAQFu4yqxVvq2LSvlc9cOF1PMRIZIQp2GVX3Lt5OaV4G1ywsjXZTROKWgl1GzYbaFtZUN/HJ86ceNGmXiESW/nXJqPndqj2kJSdx3UI9c1RkJCnYZVR0+QI8+2Yt75k7kdzMlGg3RySuKdglIoJBx7Nv1LJ2dxPB4OEPb/nL+n20dffyoTM0g6PISNN87DJszjm+9dzbPLK8GoDx49K4/OQJfPqi6ZSGH3Dxu1V7mFqUxZlTC6LZVJGEoGCXYXHO8b2/bOKR5dXcdt5U5pbm8sKG/fx+9R7+umEfv/xYJbkZKazc1cjXr5yjG5JERoGCXU6Yc44fvbiV+1/byc1nT+Gb7z0JM+OahaVU1bXziYdWcdP9y1lQlkdyknHdaRriKDIaVGOXE9LU4eMzj67lxy9Xcf3pZXz7qlMO6o3PKMnm2TvPZUFZHit3NXLpSSWU5KRHscUiiUM9djlur2z18tUn36Kp08fXr5zDp86fNuh86gVZqTzyyUU8/M9depCGyChSsMtx2XqgjVsfXMnMkmwe/PgZnDIp96jrpyV7uP2C6aPUOhEBBbscpxc3HsA5ePSTZ6q0IjJGqcYux2Xx5jrmleUq1EXGMAW7DFlzp4+1u5u4aFZxtJsiIkehYJche3VbPUEHF83RU49ExjIFuwzZks115GemML8sL9pNEZGjULDLkASDjle2erlwVrEekCEyxg072M2s3MwWm9kmM3vbzL4QiYbJ2LK+toWGDh8XqwwjMuZFYrhjL/Bl59xaM8sB1pjZi865jRHYtowRi7fUYQbnz9SFU5Gxbtg9dufcPufc2vD3bcAmQJOCxJnFW7wsKM+jICs12k0RkWOIaI3dzCqAhcCKQZbdbmarzWy11+uN5G5lhK3a1ci6mmYunq0yjEgsiNidp2aWDfwB+KJzrvXQ5c65+4D7ACorKw9/EoNEnXOOPY1dtPX46ekNsqexk4f/uYu1u5vJy0zh/fMnRbuJIjIEEQl2M0shFOqPOeeejsQ2ZfQ9v24fn3vijYPem1yQybevOpnrK8vJStMMFCKxYNj/Ui00V+sDwCbn3P8Mv0kSDcGg4ycvb2N6cRZfffcc0lKSGJeezILyfA1vFIkxkeiCnQt8DFhvZm+G3/s359xfIrBtGSX/2FzH1gPt/OhD87niVE2xKxLLhh3szrmlgLp0Mcw5x8+WVFGWn8FV81RHF4l1uvNUWL6jkTd2N/MvF0wj2aO/EiKxTv+KhZ8tqaIoO43rK8uj3RQRiQAFe4JbubOR17bVc9t5U0lP8US7OSISAQr2BLb1QBu3P7Ka8oIMPnrW5Gg3R0QiRMGeoHY3dPLRX60g1ZPEY7edRU56SrSbJCIRojtOElBdWzcfeWA5vkCQ3//L2UwuzIx2k0QkgtRjT0D3vbKDfc3dPPzxRcwanxPt5ohIhCnYE4w/EOTZN2u59KQS5pfrSUgi8UjBnmAWb66jvt3H9adraKNIvFKwJ5gn19RQlJ3GRbP1wAyReKVgTyDeth5e3lzHB04r1R2mInFM/7oTyLNv1BIIOq6vLIt2U0RkBCnYE4RzjifX7GFBeR4zSjQSRiSeKdgTxNrdzWw90K7eukgCULAngMVb6vjEQ6vIz0zhfZqWVyTu6c7TOBYIOu75xzZ+8vI2Zo/P4RcfPZ3cDE0dIBLvFOxx7J6XtvLjl6v4wGllfPeaU8lI1eyNIolApZgY9ejyaj758Crauv2DLve29XD/azt577yJ3H39PIW6SAJRsMeox1bs5qVNdXzy4dV0+wOHLb93cRW+QJCvXD6b0PPGRSRRKNhjUHOnj837W1lUUcDKXY3c8dha/IFg//Kapk4eX7GbGyrLmFqUFcWWikg0KNhj0IqdjTgHX71iNt+95lRe3lzHZx5dw3ZvOwA//sc2MPjcJTOj3FIRiQZdPI1BK3Y0kp6SxLyyXM6oKKDbH+T7L2zmpR++wgWzilm6zcvHz53KpLyMaDdVRKJAwR6Dlu9o4LTJ+aQlhy6I3nbeVK5eMIlHl1fzyOvVZKUlc8dF06PcShGJFgV7jGnu9LFpfytfumzWQe8XZafxxctm8ekLp9PR00thdlqUWigi0aZgjzErw/X1s6YVDro8PcVDeoqGNookMl08jTHLdzSSlpzE/PLcaDdFRMYoBXuMWbHz4Pq6iMihFOwxpKXTz8Z9rUcsw4iIgII9pqzc1VdfL4h2U0RkDFOwx5DlOxrC9fW8aDdFRMYwBXsMeXWrl8qKfI16EZGjUrDHiD2NnWyra+fi2SXRboqIjHERCXYz+7WZ1ZnZhkhsTw63ZEsdABfPUbCLyNFFqsf+EHBFhLYlg1i8xcuUwkymabZGETmGiAS7c+5VoDES25LDdfsD/HN7PRfPLtHc6iJyTKNWYzez281stZmt9nq9o7XbuPD6jga6/UGVYURkSEYt2J1z9znnKp1zlcXFxaO127iweHMdGSkezpyq8esicmwJMyrmQGs3d/11M77e4LFXHkOcc7y8uY5zZxRqmKOIDEnCBPuLGw/wi1e2s3xHQ7Sbcly2ezuoaeriIg1zFJEhitRwxyeA14HZZlZjZrdFYruRVN/eA8Br22Krvr94s4Y5isjxich87M65myKxnZHkbesL9voot+T4LNlax6zx2ZTqMXciMkQJU4rp67Fv3t9GXWt3lFszNN3+AKt2NXH+TF1sFpGhS5hg97b1UJSdCsDSqtjota+pbsLXG+S8GUXRboqIxJCECfb6dh/nTC+iMCs1ZsoxS6vqSU4yFmmYo4gchwQK9h5KctI4d0YRr22rxzkX7SYd07KqehZOziMrTY+mFZGhS4hg7+jppdMXoCgnjfNnFlHf3sPm/W3RbtZRtXT6WV/bwrkqw4jIcUqIYO+7cFqUndZ/IXKsD3t8fUc9zqFgF5HjlhDB3jfUsTgnjQm56cwanz3m6+zLqhrISvWwQE9LEpHjlBDB/k6PPTQq5vyZxazc2Ui3PxDNZh3Vsqp6zpxWSIonIU6RiERQQqSGt90HhHrsAGdNK6SnN8jbe1uj2awj2tvcxY76Ds6ZXhjtpohIDEqMYG/rwQwKMkM99mnFoYdV7KrviGazjmhZeJz9eTNVXxeR45cQwV7f3kNBZirJ4bJGeX4mSQbVDWMz2JdW1VOUncrs8TnRboqIxKCECHZvW09/GQYgNTmJsvxMdjZ0RrFVg2vt9vPixgNcMkdPSxKRE5MQwV7f3kNRdtpB71UUZY3JUswza2vp9AX46FlTot0UEYlRCRTsqQe9V1GYya76jjF1B6pzjkeWVzO/LJd5ZRrmKCInJu6D3Tl3WCkGoKIwi7aeXho6fFFq2eGW72ikqq5dvXURGZa4D/YOX4Buf/CwUszUorE3MubR5dXkZqRw1fxJ0W6KiMSwuA/2+gF3nQ5UEQ72nWMk2A+0dvO3t/dzQ2WZnm0qIsMS98HuHTBPzEBl+Rl4koxdY2TI429X7qE36PjImSrDiMjwxH2w9/XYDw32FE8S5fkZ7BoDQx7r23t48J87uXBWcf9vEiIiJyrug72vx35oKQbGzpDHf//TRjp7AnzzvSdFuykiEgfiPtjr23pIMijISj1sWUVhVtSHPC7eUsdzb+3ljounM1N3mopIBMR9sHvbfRRkpeFJOvwuzorCTDp8gf5efSS9sGE/B47x0OyOnl6++cwGZpRk85mLpke8DSKSmOI/2NsOvzmpT0X/kMfI1tl/vXQnn350Df/+p41HXe+Hf99KbXMXd103l7RkjYQRkciI+2Cvbz/85qQ+IzGW/W9v7+c//ryRrFQPL246QEuXf9D1mjt9PLqimutPL6OyQg+rFpHIiftg97b1UJw9eLCX5mWQnGTsjNCQxzf3NPOF377B/LI8Hrj1DHy9QV7YsG/QdZ99oxZfb5Bbz62IyL5FRPrEdbA750LzxByhx57sSWJyQWZEeuz+QJDbf7Oakpx0fnVLJWdOLWBaURZPr60dtF2/XbWHuaW5nDIpd9j7FhEZKK6Dvb2nl57e4BFr7ABTCjMjcvfptgPt1LX18OXLZ1GUnYaZcc3CUlbsbKS2ueugdd/c08zm/W3cuKh82PsVETlUXAe79wjTCQxUUZRFdUPnsIc8rq9tBjhoVsZrFpQCobLLQL9btYeMFA/v15wwIjIC4jrY68PPOj30rtOBphZl0eUPUNd25CGPLV1+bntoFdsOtB1xnXU1LeSkJzOlILP/vcmFmVROyeeZN2r7/+No7+nlubf2ctX8ieSkpxzvIYmIHFNcB/tQeuynTBoHhO7+9PUGB13n6bU1/GNzHU+uqTnidtbXtjC3NJekQ8bLX3taKVV17ayraQHg+bf20ukLcOOiycd1LCIiQxXXwb4/fINQSU76Edc5fUoB33jPSfx5/T4++ZvVdPp6D1runOPxFbsBWLKlbtBt+HqDbN7Xxtyywy+EvnfuRFI9SVx97zJO+r8v8K3n3mbW+GwWlutBGiIyMpIjsREzuwK4B/AAv3LO3RWJ7Q7XDm87uRkp5GceveTxqQumkZuRwtefXsfHHljJw59YRHZa6EezprqJbXXtnDxxHBv3tbK3uYtJeRkHfX7rgTZ8gSDzSg8P67zMVH596xm8VdNMU4eP5i4/1y4s1fNMRWTEDDvYzcwD3Au8C6gBVpnZc865o992OQq2e9uZXpw1pBC94YxyctKTufPxtXzrj2/zwxvmA/D4it3kpCXzX9fN5ep7l7Fki5cPn3lwGaWvzDK3dPChi+fNLOK8mUXDPBoRkaGJRClmEVDlnNvhnPMBvwWujsB2j6rbHzjmXCzbvR3MKMke8javnDuRz148gz+sreHP6/bR3Onj+fX7uGZhKfPKcinNyxi0HLO+tpncjBTKCzIG2aqIyOiKRLCXAnsGvK4Jv3cQM7vdzFab2Wqv1zvsnd7/6g7e95OlR1ze0uXH29bD9OKhBzvA5y6dyfzyPP7tmfX8fMl2fL1Bblo0GTPjwtnFLKuqP+wi67qaFuaV5aq8IiJjQiSCfbA0O2xQuHPuPudcpXOusri4eNg73d3YibetB39g8JEsO7ztAMcd7CmeJO750ALmD9kMAAAKNklEQVT8gSC/fHUH88vzODk8cuaiWcV0+AKs3tXYv363P8CW/W1HLMOIiIy2SAR7DTDwFsoyYG8EtntUTZ2hybXau3sHXV5VFw724yjF9KkoyuJbV50MwMfOeudRdefMKCLFYyzZ+s5vHJv3t9EbdMwbZESMiEg0RCLYVwEzzWyqmaUCNwLPRWC7R9XcGbr5qL1n8GDf7u0gNfz4uxPxoTMms/grF/GB096pKmWnJXNGRcFBdfb1teELp2UavigiY8Owg9051wt8FvgbsAn4vXPu7eFu91iawsHe2j34tLjbve1UFGWS7DnxQ5xadPiImotmF7P1QDt7w/O/rK9ppjArlUm5Rx4rLyIymiJyg5Jz7i/OuVnOuenOuf+MxDaP5VilmNBQx+MvwxzLJXNKALjhl69zz0vbWL2riVNLdeFURMaOmLzzNBh0Ry3F+ANBdjd0jkiwzyjJ4f6bK5lSmMmPXtrKjvoO1ddFZEyJyJ2no62tu5ege+f7Q1U3dNIbdEwvyRqR/b/r5PG86+Tx7Gns5MWNB3jf/Ikjsh8RkRMRk8HeV18HaBukx94/ImYEeuwDlRdk8onzpo7oPkREjldMlmIGBvtgNfbtJziGXUQkHsRksDd3vjMSpm2QUTHbve1MzE0nKy0mfyERERmWmAz2g3rsg5Ritns71FsXkYQVo8Ee6qXnpCcfVopxzrGjLjSro4hIIorJYG/u9JFkUJqXcdjFU29bD209vSc0lYCISDyIyWBv6vSRm5HCuPSUw2rsozUiRkRkrIrNYO/wk5+VSnZ68mE19u31HYCCXUQSV2wGe6eP/MzUQWvs3tZukgxKjvIAaxGReBajwe4nPzOF7LTkw+48bez0kZeZSlKS5m4RkcQUk8HeHA7v7PTkwy6eNnX4j/nwahGReBaTwR4qxaSQk5aMrzdIT2+gf1ljh4+CrNQotk5EJLpiLti7/QG6/UHyMlPJSQ/1zAfW2fvq7yIiiSrmgr3vrtP8zFSyw1MGDBwZox67iCS6mJtMpakjNG49PzOl/wJp3wVU51yox65gF5EEFnPB3veAjbzMVByhSdn7gr29pxd/wFGgUoyIJLCYC/bGvlJMVgr+3lCw95Vi+mZ9VI9dRBJZzAV73wRgBZmpdPhCo2Hae0LvNXaEQr8gS8MdRSRxxVywN3e8U4pJSgoFel8ppnHAhVURkUQVg6Ni/GSlekhNTuofFdMX7E39PXYFu4gkrpgL9r67TgHSkpNI8Vh/jb1xQG9eRCRRxVywh4YzhmroZkbOgKl7mzp9eJKMcekxV2ESEYmYGAx2/0E19Oy0d2Z4bOwILTPTBGAikrhiLtgHlmIgHOw979TYNSJGRBJdzAV735S9fXLSk2kdMCpGI2JEJNHFVLD3BoK0dvsP6rEPfNhGk+aJERGJrWBv6fLjHBQM6LEfVIrRPDEiIrEV7E2DTBnQNyomGHQ0dfo1T4yIJLyYCvaBE4D16XugdVt3L4GgU49dRBJeTAV7f4/9kFKMP+DY39oNaJ4YEZFhBbuZXW9mb5tZ0MwqI9WoI2kaZC6YvpuRqhs6DlsmIpKIhttj3wBcB7wagbYc0zulmAE99nCw727sBBTsIiLDuvfeObcJGLU7PZs6/SQnWf/kXwDZaaGQ3xMOdg13FJFEN2o1djO73cxWm9lqr9d7Qtvou+t04H8kOYf22BXsIpLgjtljN7OXgAmDLPqGc+6PQ92Rc+4+4D6AyspKN+QWDvDvV5/K197de9B7fb336sZOUj1JZKV6TmTTIiJx45jB7py7bDQaMhQpnqTDeuR9Pfaaxi7ys1I0AZiIJLyYGu44mJz0UI3dFwjqwqmICMMf7nitmdUAZwN/NrO/RaZZQ5eV9k7pRRdORUSGPyrmGeCZCLXlhKQlhx6T5+sN6sKpiAhxUIqBd25S0jwxIiJxEux9I2PUYxcRiZdg7++xa54YEZH4CHb12EVE+sVFsPcNedRwRxGReAn2cI9dwx1FROIk2Ptq7CrFiIjESbD3TSuQr4unIiLDu0FprLh6QSk56SlkpsbF4YiIDEtcJOGs8TnMGp8T7WaIiIwJcVGKERGRdyjYRUTijIJdRCTOKNhFROKMgl1EJM4o2EVE4oyCXUQkzijYRUTijDnnRn+nZl6g+gQ/XgTUR7A5sUDHnBh0zIlhOMc8xTlXfKyVohLsw2Fmq51zldFux2jSMScGHXNiGI1jVilGRCTOKNhFROJMLAb7fdFuQBTomBODjjkxjPgxx1yNXUREji4We+wiInIUCnYRkTgTU8FuZleY2RYzqzKzr0e7PZFmZuVmttjMNpnZ22b2hfD7BWb2opltC/+ZH+22RpqZeczsDTN7Pvx6qpmtCB/z78wsrh5oa2Z5ZvaUmW0On++z4/08m9mXwn+vN5jZE2aWHm/n2cx+bWZ1ZrZhwHuDnlcL+XE4z9aZ2WmRakfMBLuZeYB7gSuBk4GbzOzk6LYq4nqBLzvnTgLOAu4MH+PXgX8452YC/wi/jjdfADYNeP194EfhY24CbotKq0bOPcALzrk5wHxCxx6359nMSoHPA5XOuVMBD3Aj8XeeHwKuOOS9I53XK4GZ4a/bgZ9HqhExE+zAIqDKObfDOecDfgtcHeU2RZRzbp9zbm34+zZC/9hLCR3nw+HVHgauiU4LR4aZlQHvBX4Vfm3AJcBT4VXi6pjNbBxwAfAAgHPO55xrJs7PM6FHcWaYWTKQCewjzs6zc+5VoPGQt490Xq8GfuNClgN5ZjYxEu2IpWAvBfYMeF0Tfi8umVkFsBBYAYx3zu2DUPgDJdFr2Yj4X+BrQDD8uhBods71hl/H27meBniBB8Plp1+ZWRZxfJ6dc7XA3cBuQoHeAqwhvs9znyOd1xHLtFgKdhvkvbgcq2lm2cAfgC8651qj3Z6RZGbvA+qcc2sGvj3IqvF0rpOB04CfO+cWAh3EUdllMOG68tXAVGASkEWoFHGoeDrPxzJif89jKdhrgPIBr8uAvVFqy4gxsxRCof6Yc+7p8NsH+n5FC/9ZF632jYBzgfeb2S5C5bVLCPXg88K/skP8nesaoMY5tyL8+ilCQR/P5/kyYKdzzuuc8wNPA+cQ3+e5z5HO64hlWiwF+ypgZvgqeiqhCy/PRblNERWuLT8AbHLO/c+ARc8Bt4S/vwX442i3baQ45/7VOVfmnKsgdE5fds59BFgMfDC8Wrwd835gj5nNDr91KbCROD7PhEowZ5lZZvjved8xx+15HuBI5/U54Obw6JizgJa+ks2wOedi5gt4D7AV2A58I9rtGYHjO4/Qr2LrgDfDX+8hVHP+B7At/GdBtNs6Qsd/EfB8+PtpwEqgCngSSIt2+yJ8rAuA1eFz/SyQH+/nGfgOsBnYADwCpMXbeQaeIHQNwU+oR37bkc4roVLMveE8W09oxFBE2qEpBURE4kwslWJERGQIFOwiInFGwS4iEmcU7CIicUbBLiISZxTsIiJxRsEuIhJn/j8ipP2PnRwtuwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFuttZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1MiByMjg1NCBlOWE1OTAzIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNyAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MToweDExMSBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MCBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAALUZYiEADf//vaH+BTZWBP+Wb/9DX/cj9uPrP1xYyEE31qvIejAGS+1H+b/rFFs6Z6UB/fgCJQAc24ZwpJw4v/ApLdW+BTLYTkVLnX1aIxw4jw34T+srwtQbsVYpK9aMqy996n7h2/+3tY17iJ/KHMmAnDLJMkkWqVMrZ53wrmxizbKadzB97Luc8HAd3Np8hsqaf87+qnR0hgkcn6Hu+w5wSKksSk7uQ8mo7yRtUY6VoamOtZZaY8dortb9SzEn3t6YLSvoqnsVHLilMNacdBVRUs7J6vrzyegOcRqMZIBzmhWDTyf6xh6XHxXb2Nth2W3UJt6iy8TBaNyL0DApeaFSXCBflA7feVbAEjZISqUtZbhwHAanpCv3NG/qvYaJs014JedopPyGz1JF8MnM1PZtReDX4BxhuY3Pvv66qh3b5NlxYAlOKJalECXqZ6Jk3iJK9T3/+ojy4NP+WY90QayCJcWO+nlAfrqlTVqyKuwHgPKAdh8yaZzh3p+SIPAzSBJQIFEmrala1tBR3inoVGa75a0vwuOW9WuhRHvKPMoAmZbElv6oTJ8ErApkIcMoekw8wXvwYnBVg6+rj4SrpeKyAj1ZBLYYGMQAu11hVIn/ElSgzX2XVJ73e8zwUA8aBxy01oXqcJ16Zy1bIx9tGkTOyQ4xnIY1wSRfAF5wICuLv8CBjQeku/P+cUBzx3ZlRlpKMeTPidAaBoZ54YetpvbBy4VAUvVa+vDStZyOdBvjarza4UpOL/DQsDAs+/up9/hoV0mwGu9KvchShSfSN7GAS4x0qUv83xuvRQ0vVAN8UegLeeJQGfdgkfqynkioDgOQlMmM56Ym8jxTot+XEs5g5yZutAn2buz9fRk8mr4pkgmpjXy78ZZxbrSCPbMGlOQ2ec3PoBuf2In++WQmGCZBeZcI4RVP8wp27DLzaiRBH9t8pG3fITFTU4AIBBoQ6o4cwAHbQAAACZBmiRsQz/+nhABk/X36bXtv7MrwKa+oMuBSpaFwKZ2Bpnbt8K+YAAAABNBnkJ4hf8APM0BN+uNl9ZKAvUtAAAAEAGeYXRCvwBR8yN9f2mvwCAAAAAQAZ5jakK/AFQpRvNMVbSiwQAAABlBmmVJqEFomUwIZ//+nhABBRDj+eC/kh3FAAAAGEGahknhClJlMCGf/p4QAZuQxz+HOb6zgQAAABlBmqdJ4Q6JlMCG//6nhACi+if6rfMfiFJBAAAAHkGayUnhDyZTBRE8N//+p4QAqHup93m5yeB4N0hiXgAAABABnuhqQr8AhuaN5piraRNAAAAAGUGa6knhDyZTAhv//qeEAGx9g/wnBboSakEAAAAfQZsNSeEPJlMCGf/+nhABu/V35lleJ0cUbroub8s/TAAAABJBnytFETwr/wBdLIb/YbzsqXkAAAAQAZ9MakK/AF0sI8mB69vegQAAABpBm05JqEFomUwIb//+p4QAcb2D/CcFuhJlQQAAABhBm29J4QpSZTAhv/6nhABLR8x5GJ/lt00AAAAgQZuSSeEOiZTAhv/+p4QAS746fejTo5lliZHcGum1/IAAAAASQZ+wRRE8K/8APMEVTC0S/21HAAAAEAGf0WpCvwAqFhHkuZ8lW4EAAAAaQZvVSahBaJlMCG///qeEADSurSCET/Lb6oAAAAAPQZ/zRREsK/8AKy1uGyBAAAAADQGeFGpCvwArPKRb2QMAAAAaQZoWSahBbJlMCG///qeEAFG9E/1W+Y/EUkAAAAAcQZo6SeEKUmUwIZ/+nhAB7/XI3Y3/pvus/RsxVQAAABBBnlhFNEwv/wBNZ61xczphAAAAEAGed3RCvwBppFlXgRXbxYAAAAAPAZ55akK/AGmsWBdf390hAAAAGkGae0moQWiZTAhv//6nhADH0if6rfMfiEHAAAAAIEGanUnhClJlMFESw3/+p4QBPB8zU2bbEbn/6yj8fcTBAAAADwGevGpCvwD+2eW4bNqZMwAAABpBmr5J4Q6JlMCG//6nhAKIoY1PKCtT/FqdgAAAABlBmt9J4Q8mUwId//6plgbXIMz21QA/5yl5AAAAGUGa40nhDyZTAh3//qmWBh9sT/0T3LuMP8EAAAATQZ8BRRE8L/8CAN5upBmDnChFxAAAAA8BnyB0Qr8CrdkLbe5E8fcAAAAPAZ8iakK/Aq9iPJgN9yXHAAAAE0GbJ0moQWiZTAh3//6plgAAlYEAAAAMQZ9FRREsL/8AALKBAAAAEAGfZHRCvwKwQB0IcOQbI2EAAAAQAZ9makK/Aq7Wu7fhyDZGwQAAABNBm2tJqEFsmUwId//+qZYAAJWAAAAADEGfiUUVLC//AACygAAAABABn6h0Qr8CsEAdCHDkGyNhAAAAEAGfqmpCvwKu1ru34cg2RsAAAAATQZuvSahBbJlMCHf//qmWAACVgAAAAAxBn81FFSwv/wAAsoEAAAAQAZ/sdEK/ArBAHQhw5BsjYQAAABABn+5qQr8Crta7t+HINkbBAAAAEkGb80moQWyZTAhv//6nhAABJwAAAAxBnhFFFSwv/wAAsoAAAAAQAZ4wdEK/ArBAHQhw5BsjYQAAABABnjJqQr8Crta7t+HINkbAAAAAGkGaNEmoQWyZTAh3//6plgbXIMutKORIQQ3oAAAAHEGaWEnhClJlMCHf/qmWBh9LoHD/IA/A3F44YEEAAAAQQZ52RTRML/8B6fvX5DRRQAAAABABnpV0Qr8Cj9ojyvyUwZ0xAAAADwGel2pCvwKu1ru5/1yNgQAAABNBmpxJqEFomUwId//+qZYAAJWAAAAADEGeukURLC//AACygQAAABABntl0Qr8Bjs5OI7LsqOmAAAAAEAGe22pCvwKu1ruqxn0S0YEAAAATQZrASahBbJlMCHf//qmWAACVgQAAAAxBnv5FFSwv/wAAsoAAAAAQAZ8ddEK/ArBAHP2BbiyNgAAAABABnx9qQr8Crta7qsZ9EtGBAAAAE0GbBEmoQWyZTAh3//6plgAAlYAAAAAMQZ8iRRUsL/8AALKBAAAAEAGfQXRCvwKwQBz9gW4sjYAAAAAQAZ9DakK/Aq7Wu6rGfRLRgQAAABtBm0hJqEFsmUwIb//+p4QCk9g/w3PsJombOOEAAAAQQZ9mRRUsL/8BJs/QcodGtQAAAA8Bn4V0Qr8CsEAdB1LJaMEAAAAQAZ+HakK/AZMltOvAE/kzgAAAABtBm4lJqEFsmUwId//+qZYAoNSDM//WA/7lZUAAAAAbQZusSeEKUmUwId/+qZYBR79Po2A/1uf5wLPhAAAAEUGfykU0TCv/AZN2n/RyRVEHAAAAEAGf62pCvwGTdU8mB69sz4AAAAATQZvwSahBaJlMCHf//qmWAACVgQAAAAxBng5FESwv/wAAsoEAAAAQAZ4tdEK/ArBAHP2BbiyNgQAAABABni9qQr8Crta7qsZ9EtGAAAAAE0GaNEmoQWyZTAh3//6plgAAlYAAAAAMQZ5SRRUsL/8AALKBAAAAEAGecXRCvwKwQBz9gW4sjYAAAAAQAZ5zakK/Aq7Wu6rGfRLRgAAAABNBmnhJqEFsmUwId//+qZYAAJWBAAAADEGelkUVLC//AACygAAAABABnrV0Qr8CsEAc/YFuLI2BAAAAEAGet2pCvwKu1ruqxn0S0YEAAAATQZq8SahBbJlMCHf//qmWAACVgAAAAAxBntpFFSwv/wAAsoEAAAAQAZ75dEK/ArBAHP2BbiyNgAAAABABnvtqQr8Crta7qsZ9EtGBAAAAE0Ga4EmoQWyZTAh3//6plgAAlYEAAAAMQZ8eRRUsL/8AALKAAAAAEAGfPXRCvwKwQBz9gW4sjYAAAAAQAZ8/akK/Aq7Wu6rGfRLRgQAAABNBmyRJqEFsmUwId//+qZYAAJWAAAAADEGfQkUVLC//AACygQAAABABn2F0Qr8CsEAc/YFuLI2AAAAAEAGfY2pCvwKu1ruqxn0S0YEAAAATQZtoSahBbJlMCHf//qmWAACVgQAAAAxBn4ZFFSwv/wAAsoEAAAAQAZ+ldEK/ArBAHP2BbiyNgQAAABABn6dqQr8Crta7qsZ9EtGAAAAAE0GbrEmoQWyZTAh3//6plgAAlYAAAAAMQZ/KRRUsL/8AALKBAAAAEAGf6XRCvwKwQBz9gW4sjYAAAAAQAZ/rakK/Aq7Wu6rGfRLRgAAAABNBm/BJqEFsmUwId//+qZYAAJWBAAAADEGeDkUVLC//AACygQAAABABni10Qr8CsEAc/YFuLI2BAAAAEAGeL2pCvwKu1ruqxn0S0YAAAAASQZo0SahBbJlMCG///qeEAAEnAAAADEGeUkUVLC//AACygQAAABABnnF0Qr8CsEAc/YFuLI2AAAAAEAGec2pCvwKu1ruqxn0S0YAAAAAdQZp2SahBbJlMFEw3//6nhAKT2D/EVa8KNbmBmLEAAAAQAZ6VakK/AZMjtzrQwvDZQAAAABtBmppJ4QpSZTAhv/6nhAE1+On3Wlmam3RazVkAAAAQQZ64RTRML/8AujLFQgn90QAAABABntd0Qr8A9/FmeV+Smyz4AAAADwGe2WpCvwD+ta7vu93VQQAAABpBmttJqEFomUwIb//+p4QBPEAWbbEYXzNHwAAAABlBmvxJ4QpSZTAh3/6plgFHaQk2tIY+3TZhAAAAEUGbAEnhDomUwIb//qeEAAEnAAAAE0GfPkURPC//Aem7fRYrdbR7RRQAAAAQAZ9ddEK/ArBAHO2MTapIwAAAABABn19qQr8CkFgqm+kgrDKhAAAAHEGbQkmoQWiZTBTw7/6plgFM7S/n9BLOUG4MBTQAAAAQAZ9hakK/AZMjtzrQwvDZQQAAABlBm2VJ4QpSZTAh3/6plgCc/HnSzo6nkT/AAAAAEkGfg0U0TCv/APgDAIBTAOP2YQAAABABn6RqQr8A8rPmN0OSDif5AAAAG0GbqUmoQWiZTAhv//6nhAJJ3U/WLpdAF4cRSQAAABRBn8dFESwv/wEe5hMUOmLupWN5eQAAABABn+Z0Qr8BiZFlXgRXbNSAAAAAEAGf6GpCvwGJJbTrwBP5OIAAAAAdQZvrSahBbJlMFEw3//6nhAE8HzNTZtvm91Pi1d0AAAAPAZ4KakK/AP7Z5bhs2pkzAAAAGkGaDEnhClJlMCHf/qmWAUdpCTa0i0/8Wp2AAAAAEkGaMEnhDomUwId//qmWAACVgQAAAAxBnk5FETwv/wAAsoEAAAAQAZ5tdEK/ArBAHP2BbiyNgQAAABABnm9qQr8Crta7qsZ9EtGAAAAAHEGackmoQWiZTBTw7/6plgFM7S/n9BLOUG4MBTQAAAAQAZ6RakK/AZMjtzrQwvDZQQAAABhBmpZJ4QpSZTAh3/6plgFHlz/t7S+11vQAAAAVQZ60RTRML/8BJs/ZrXvyIfMRWvYsAAAAEAGe03RCvwD38WZ5X5KbLPkAAAAQAZ7VakK/AZN2o5X9uHzNwAAAABNBmtpJqEFomUwId//+qZYAAJWBAAAADEGe+EURLC//AACygQAAABABnxd0Qr8CsEAc/YFuLI2AAAAAEAGfGWpCvwKu1ruqxn0S0YEAAAATQZseSahBbJlMCHf//qmWAACVgAAAAAxBnzxFFSwv/wAAsoEAAAAQAZ9bdEK/ArBAHP2BbiyNgQAAABABn11qQr8Crta7qsZ9EtGAAAAAE0GbQkmoQWyZTAh3//6plgAAlYAAAAAMQZ9gRRUsL/8AALKBAAAAEAGfn3RCvwKwQBz9gW4sjYAAAAAQAZ+BakK/Aq7Wu6rGfRLRgQAAABNBm4ZJqEFsmUwId//+qZYAAJWAAAAADEGfpEUVLC//AACygQAAABABn8N0Qr8CsEAc/YFuLI2BAAAAEAGfxWpCvwKu1ruqxn0S0YEAAAATQZvKSahBbJlMCHf//qmWAACVgQAAAAxBn+hFFSwv/wAAsoAAAAAQAZ4HdEK/ArBAHP2BbiyNgAAAABABnglqQr8Crta7qsZ9EtGBAAAAE0GaDkmoQWyZTAh3//6plgAAlYAAAAAMQZ4sRRUsL/8AALKAAAAAEAGeS3RCvwKwQBz9gW4sjYEAAAAQAZ5NakK/Aq7Wu6rGfRLRgQAAAB1BmlBJqEFsmUwUTDv//qmWAUztL+f0Es5QbgwFNQAAABABnm9qQr8BkyO3OtDC8NlAAAAAGEGadEnhClJlMCHf/qmWAUeXP+3tL7XW9AAAABVBnpJFNEwv/wEmz9mte/Ih8xFa9i0AAAAQAZ6xdEK/APfxZnlfkpss+AAAABABnrNqQr8Bk3ajlf24fM3AAAAAE0GauEmoQWiZTAh3//6plgAAlYEAAAAMQZ7WRREsL/8AALKAAAAAEAGe9XRCvwKwQBz9gW4sjYEAAAAQAZ73akK/Aq7Wu6rGfRLRgQAAABxBmvxJqEFsmUwId//+qZYBTO0v5/QSzlBuDAU0AAAAEEGfGkUVLC//ASbP2bggMPEAAAAPAZ85dEK/ArBAHQdSyWjAAAAAEAGfO2pCvwGTI7c60MLw2UEAAAAZQZsgSahBbJlMCG///qeEAokLKtvRPzgU0QAAABVBn15FFSwv/wEmz9mte/Ih8xFa9iwAAAAQAZ99dEK/APfxZnlfkpss+AAAABABn39qQr8Bk3ajlf24fM3BAAAAEkGbZEmoQWyZTAhv//6nhAABJwAAAAxBn4JFFSwv/wAAsoEAAAAQAZ+hdEK/ArBAHP2BbiyNgAAAABABn6NqQr8Crta7qsZ9EtGBAAAAEkGbqEmoQWyZTAhf//6MsAAEjQAAAAxBn8ZFFSwv/wAAsoEAAAAQAZ/ldEK/ArBAHP2BbiyNgQAAABABn+dqQr8Crta7qsZ9EtGAAAAAGkGb6UuoQhBbJEYIKAfyAf2HgCFf/jhAABFwAAAMOG1vb3YAAABsbXZoZAAAAAAAAAAAAAAAAAAAA+gAAB+QAAEAAAEAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAtidHJhawAAAFx0a2hkAAAAAwAAAAAAAAAAAAAAAQAAAAAAAB+QAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAQAAAAAEQAAABEAAAAAAAJGVkdHMAAAAcZWxzdAAAAAAAAAABAAAfkAAABAAAAQAAAAAK2m1kaWEAAAAgbWRoZAAAAAAAAAAAAAAAAAAAMgAAAZQAVcQAAAAAAC1oZGxyAAAAAAAAAAB2aWRlAAAAAAAAAAAAAAAAVmlkZW9IYW5kbGVyAAAACoVtaW5mAAAAFHZtaGQAAAABAAAAAAAAAAAAAAAkZGluZgAAABxkcmVmAAAAAAAAAAEAAAAMdXJsIAAAAAEAAApFc3RibAAAAJVzdHNkAAAAAAAAAAEAAACFYXZjMQAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAEQARAASAAAAEgAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABj//wAAAC9hdmNDAfQADf/hABdn9AANkZsoIhHQgAAAAwCAAAAZB4oUywEABWjr48RIAAAAGHN0dHMAAAAAAAAAAQAAAMoAAAIAAAAAFHN0c3MAAAAAAAAAAQAAAAEAAAYQY3R0cwAAAAAAAADAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAMAAAQAAAAAAQAABgAAAAABAAACAAAAAAEAAAQAAAAAAQAACAAAAAACAAACAAAAAAIAAAQAAAAAAQAACAAAAAACAAACAAAAAAEAAAgAAAAAAgAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAGAAAAAAEAAAIAAAAAAgAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAIAAAAAAIAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAgAAAAAAgAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAFiQAAACoAAAAXAAAAFAAAABQAAAAdAAAAHAAAAB0AAAAiAAAAFAAAAB0AAAAjAAAAFgAAABQAAAAeAAAAHAAAACQAAAAWAAAAFAAAAB4AAAATAAAAEQAAAB4AAAAgAAAAFAAAABQAAAATAAAAHgAAACQAAAATAAAAHgAAAB0AAAAdAAAAFwAAABMAAAATAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAAB4AAAAgAAAAFAAAABQAAAATAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAHwAAABQAAAATAAAAFAAAAB8AAAAfAAAAFQAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAWAAAAEAAAABQAAAAUAAAAIQAAABQAAAAfAAAAFAAAABQAAAATAAAAHgAAAB0AAAAVAAAAFwAAABQAAAAUAAAAIAAAABQAAAAdAAAAFgAAABQAAAAfAAAAGAAAABQAAAAUAAAAIQAAABMAAAAeAAAAFgAAABAAAAAUAAAAFAAAACAAAAAUAAAAHAAAABkAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAACEAAAAUAAAAHAAAABkAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAgAAAAFAAAABMAAAAUAAAAHQAAABkAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAWAAAAEAAAABQAAAAUAAAAHgAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1OC4xMi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prefix = 'cnn_train'\n",
    "env = Environment(grid_size=size, max_time=T, temperature=0.3)\n",
    "agent = DQN_CNN(size, lr=.1, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "train(agent,env,epochs_train,prefix=prefix)\n",
    "name = sorted(list(filter(lambda x: prefix in x and '.mp4' in x, os.listdir(prefix))))[-1]\n",
    "HTML(display_videos(name, prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "__Question 9__ Test both algorithms and compare their performances. Which issue(s) do you observe? Observe also different behaviors by changing the temperature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test of the CNN\n",
      "Win/lose count 4.5/0.0. Average score (4.5)\n",
      "Win/lose count 4.5/1.0. Average score (4.909090909090909)\n",
      "Win/lose count 4.5/0.0. Average score (4.928571428571429)\n",
      "Win/lose count 6.5/0.0. Average score (5.274193548387097)\n",
      "Win/lose count 1.0/0.0. Average score (5.439024390243903)\n",
      "Final score: 5.34\n",
      "Test of the FC\n",
      "Win/lose count 0/1.0. Average score (-1.0)\n",
      "Win/lose count 0.5/0.0. Average score (0.5909090909090909)\n",
      "Win/lose count 0.5/0.0. Average score (0.30952380952380953)\n",
      "Win/lose count 1.0/0.0. Average score (0.3064516129032258)\n",
      "Win/lose count 0.5/0.0. Average score (0.34146341463414637)\n",
      "Final score: 0.28\n"
     ]
    }
   ],
   "source": [
    "env = Environment(grid_size=size, max_time=T,temperature=0.7)\n",
    "agent_cnn = DQN_CNN(size, lr=.1, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "agent_cnn.load(name_weights='cnn_trainmodel.h5',name_model='cnn_trainmodel.json')\n",
    "\n",
    "agent_fc = DQN_FC(size, lr=.1, epsilon = 0.1, memory_size=2000, batch_size = 32)\n",
    "agent_cnn.load(name_weights='fc_trainmodel.h5',name_model='fc_trainmodel.json')\n",
    "\n",
    "epochs_test = 50\n",
    "print('Test of the CNN')\n",
    "test(agent_cnn,env,epochs_test,prefix='cnn_test')\n",
    "print('Test of the FC')\n",
    "test(agent_fc,env,epochs_test,prefix='fc_test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer**\n",
    "\n",
    "When doing many experiments it seemed the DQN_CNN  performed better than its fully-connected counterpart. \n",
    "\n",
    "However both tend to get stuck in certain places, which is not good.\n",
    "Augmenting the temperature to correct this issue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "The algorithm tends to not explore the map which can be an issue. We propose two ideas in order to encourage exploration:\n",
    "1. Incorporating a decreasing $\\epsilon$-greedy exploration. You can use the method ```set_epsilon```\n",
    "2. Append via the environment a new state that describes if a cell has been visited or not\n",
    "\n",
    "***\n",
    "__Question 10__ Design a new ```train_explore``` function and environment class ```EnvironmentExploring``` to tackle the issue of exploration.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_explore(agent, env,epoch, epsilons=(0.1,0.01), prefix=''):\n",
    "     # Number of won games\n",
    "    score = 0\n",
    "    loss = 0\n",
    "    \n",
    "    exp_start = epsilons[0]\n",
    "    exp_end = epsilons[1]\n",
    "    \n",
    "    update_eps = lambda e: (exp_end - exp_start)/(epoch-1)*e + exp_start\n",
    "    scores = []\n",
    "    for e in range(epoch):\n",
    "        # At each epoch, we restart to a fresh game and get the initial state\n",
    "        state = env.reset()\n",
    "        # This assumes that the games will terminate\n",
    "        game_over = False\n",
    "\n",
    "        win = 0\n",
    "        lose = 0\n",
    "        \n",
    "        epsilon = update_eps(e)\n",
    "        agent.set_epsilon(epsilon)\n",
    "\n",
    "        while not game_over:\n",
    "            # The agent performs an action\n",
    "            \n",
    "            action = agent.act(state)\n",
    "\n",
    "            # Apply an action to the environment, get the next state, the reward\n",
    "            # and if the games end\n",
    "            prev_state = state\n",
    "            state, reward, game_over = env.act(action)\n",
    "\n",
    "            # Update the counters\n",
    "            if reward > 0:\n",
    "                win = win + reward\n",
    "            if reward < 0:\n",
    "                lose = lose -reward\n",
    "\n",
    "            # Apply the reinforcement strategy\n",
    "            loss = agent.reinforce(prev_state, state,  action, reward, game_over)\n",
    "\n",
    "        # Save as a mp4\n",
    "        if e % 10 == 0:\n",
    "            env.draw(prefix+str(e), folder=prefix)\n",
    "\n",
    "        # Update stats\n",
    "        score += win-lose\n",
    "        scores.append(score/(1+e))\n",
    "\n",
    "        print(\"Epoch {:03d}/{:03d} | Loss {:.4f} | Win/lose count {}/{} ({})\"\n",
    "              .format(e, epoch, loss, win, lose, win-lose))\n",
    "        agent.save(name_weights=prefix+'model.h5',name_model=prefix+'model.json')\n",
    "    \n",
    "    plt.plot(scores)\n",
    "    plt.title(\"Average Scores\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EnvironmentExploring(object):\n",
    "    def __init__(self, grid_size=10, max_time=500, temperature=0.5, visited_malus=0.15):\n",
    "        grid_size = grid_size+4\n",
    "        self.grid_size = grid_size\n",
    "        self.max_time = max_time\n",
    "        self.temperature = temperature\n",
    "        self.visited_malus = visited_malus\n",
    "\n",
    "        #board on which one plays\n",
    "        self.board = np.zeros((grid_size,grid_size))\n",
    "        self.position = np.zeros((grid_size,grid_size))\n",
    "        self.visited_cells = np.zeros((grid_size, grid_size))\n",
    "\n",
    "        # coordinate of the cat\n",
    "        self.x = 0\n",
    "        self.y = 1\n",
    "\n",
    "        # self time\n",
    "        self.t = 0\n",
    "\n",
    "        self.scale=16\n",
    "\n",
    "        self.to_draw = np.zeros((max_time+2, grid_size*self.scale, grid_size*self.scale, 3))\n",
    "\n",
    "\n",
    "    def draw(self,e,folder):\n",
    "        if folder not in os.listdir():\n",
    "            os.mkdir(folder)\n",
    "        skvideo.io.vwrite(os.path.join(folder,str(e) + '.mp4'), self.to_draw)\n",
    "\n",
    "    def get_frame(self,t):\n",
    "        b = np.zeros((self.grid_size,self.grid_size,3))+128\n",
    "        b[self.board>0,0] = 256\n",
    "        b[self.board < 0, 2] = 256\n",
    "        b[self.x,self.y,:]=256\n",
    "        b[-2:,:,:]=0\n",
    "        b[:,-2:,:]=0\n",
    "        b[:2,:,:]=0\n",
    "        b[:,:2,:]=0\n",
    "        \n",
    "        b =  cv2.resize(b, None, fx=self.scale, fy=self.scale, interpolation=cv2.INTER_NEAREST)\n",
    "\n",
    "        self.to_draw[t,:,:,:]=b\n",
    "\n",
    "\n",
    "    def act(self, action, train=True):\n",
    "        \"\"\"This function returns the new state, reward and decides if the\n",
    "        game ends.\"\"\"\n",
    "\n",
    "        self.get_frame(int(self.t))\n",
    "\n",
    "        self.position = np.zeros((self.grid_size, self.grid_size))\n",
    "\n",
    "        self.position[0:2,:]= -1\n",
    "        self.position[:,0:2] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "\n",
    "        self.position[self.x, self.y] = 1\n",
    "        if action == 0:\n",
    "            if self.x == self.grid_size-3:\n",
    "                self.x = self.x-1\n",
    "            else:\n",
    "                self.x = self.x + 1\n",
    "        elif action == 1:\n",
    "            if self.x == 2:\n",
    "                self.x = self.x+1\n",
    "            else:\n",
    "                self.x = self.x-1\n",
    "        elif action == 2:\n",
    "            if self.y == self.grid_size - 3:\n",
    "                self.y = self.y - 1\n",
    "            else:\n",
    "                self.y = self.y + 1\n",
    "        elif action == 3:\n",
    "            if self.y == 2:\n",
    "                self.y = self.y + 1\n",
    "            else:\n",
    "                self.y = self.y - 1\n",
    "        else:\n",
    "            RuntimeError('Error: action not recognized')\n",
    "\n",
    "        self.t = self.t + 1\n",
    "        if train:\n",
    "            reward = self.board[self.x, self.y] - self.visited_cells[self.x, self.y]\n",
    "            self.visited_cells[self.x, self.y] = self.visited_malus\n",
    "        else:\n",
    "            reward = self.board[self.x, self.y]\n",
    "        self.board[self.x, self.y] = 0\n",
    "        game_over = self.t > self.max_time\n",
    "        state = np.concatenate((self.visited_cells.reshape(self.grid_size, self.grid_size,1),\n",
    "                               self.board.reshape(self.grid_size, self.grid_size,1),\n",
    "                        self.position.reshape(self.grid_size, self.grid_size,1)),axis=2)\n",
    "        state = state[self.x-2:self.x+3,self.y-2:self.y+3,:]\n",
    "\n",
    "        return state, reward, game_over\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"This function resets the game and returns the initial state\"\"\"\n",
    "\n",
    "        self.x = np.random.randint(3, self.grid_size-3, size=1)[0]\n",
    "        self.y = np.random.randint(3, self.grid_size-3, size=1)[0]\n",
    "\n",
    "\n",
    "        bonus = 0.5*np.random.binomial(1,self.temperature,size=self.grid_size**2)\n",
    "        bonus = bonus.reshape(self.grid_size,self.grid_size)\n",
    "\n",
    "        malus = -1.0*np.random.binomial(1,self.temperature,size=self.grid_size**2)\n",
    "        malus = malus.reshape(self.grid_size, self.grid_size)\n",
    "\n",
    "        self.to_draw = np.zeros((self.max_time+2, self.grid_size*self.scale, self.grid_size*self.scale, 3))\n",
    "\n",
    "\n",
    "        malus[bonus>0]=0\n",
    "\n",
    "        self.board = bonus + malus\n",
    "\n",
    "        self.position = np.zeros((self.grid_size, self.grid_size))\n",
    "        self.position[0:2,:]= -1\n",
    "        self.position[:,0:2] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.position[-2:, :] = -1\n",
    "        self.board[self.x,self.y] = 0\n",
    "        self.t = 0\n",
    "\n",
    "        state = np.concatenate((self.visited_cells.reshape(self.grid_size, self.grid_size,1),\n",
    "                               self.board.reshape(self.grid_size, self.grid_size,1),\n",
    "                        self.position.reshape(self.grid_size, self.grid_size,1)),axis=2)\n",
    "\n",
    "        state = state[self.x - 2:self.x + 3, self.y - 2:self.y + 3, :]\n",
    "        return state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 000/101 | Loss 0.0122 | Win/lose count 8.5/28.99999999999992 (-20.49999999999992)\n",
      "Epoch 001/101 | Loss 0.0268 | Win/lose count 9.349999999999996/34.59999999999986 (-25.249999999999865)\n",
      "Epoch 002/101 | Loss 0.0249 | Win/lose count 9.499999999999996/31.449999999999918 (-21.94999999999992)\n",
      "Epoch 003/101 | Loss 0.0178 | Win/lose count 4.0/33.499999999999886 (-29.499999999999886)\n",
      "Epoch 004/101 | Loss 0.0080 | Win/lose count 7.849999999999997/32.849999999999866 (-24.99999999999987)\n",
      "Epoch 005/101 | Loss 0.0382 | Win/lose count 4.55/36.89999999999986 (-32.34999999999986)\n",
      "Epoch 006/101 | Loss 0.0182 | Win/lose count 9.849999999999996/30.749999999999904 (-20.899999999999906)\n",
      "Epoch 007/101 | Loss 0.0127 | Win/lose count 9.699999999999994/37.299999999999905 (-27.59999999999991)\n",
      "Epoch 008/101 | Loss 0.0433 | Win/lose count 6.949999999999998/38.24999999999987 (-31.299999999999873)\n",
      "Epoch 009/101 | Loss 0.0164 | Win/lose count 5.249999999999999/33.89999999999987 (-28.64999999999987)\n",
      "Epoch 010/101 | Loss 0.0615 | Win/lose count 6.749999999999999/34.999999999999865 (-28.249999999999865)\n",
      "Epoch 011/101 | Loss 0.0388 | Win/lose count 7.699999999999997/36.84999999999988 (-29.149999999999885)\n",
      "Epoch 012/101 | Loss 0.0093 | Win/lose count 7.349999999999997/34.399999999999885 (-27.049999999999887)\n",
      "Epoch 013/101 | Loss 0.0273 | Win/lose count 6.649999999999998/37.549999999999905 (-30.899999999999906)\n",
      "Epoch 014/101 | Loss 0.0281 | Win/lose count 6.599999999999998/31.14999999999989 (-24.54999999999989)\n",
      "Epoch 015/101 | Loss 0.0266 | Win/lose count 5.949999999999998/38.599999999999866 (-32.64999999999987)\n",
      "Epoch 016/101 | Loss 0.0358 | Win/lose count 6.999999999999997/34.14999999999989 (-27.149999999999896)\n",
      "Epoch 017/101 | Loss 0.0167 | Win/lose count 11.149999999999995/37.34999999999987 (-26.19999999999988)\n",
      "Epoch 018/101 | Loss 0.0095 | Win/lose count 11.549999999999994/39.19999999999986 (-27.649999999999867)\n",
      "Epoch 019/101 | Loss 0.0093 | Win/lose count 11.699999999999994/32.899999999999906 (-21.19999999999991)\n",
      "Epoch 020/101 | Loss 0.0181 | Win/lose count 9.949999999999996/32.49999999999991 (-22.549999999999912)\n",
      "Epoch 021/101 | Loss 0.0238 | Win/lose count 3.8500000000000005/32.49999999999988 (-28.649999999999878)\n",
      "Epoch 022/101 | Loss 0.0196 | Win/lose count 9.799999999999995/36.94999999999989 (-27.149999999999892)\n",
      "Epoch 023/101 | Loss 0.0088 | Win/lose count 6.649999999999998/41.29999999999984 (-34.64999999999984)\n",
      "Epoch 024/101 | Loss 0.0240 | Win/lose count 11.199999999999994/33.34999999999989 (-22.149999999999892)\n",
      "Epoch 025/101 | Loss 0.0282 | Win/lose count 6.299999999999998/41.449999999999854 (-35.14999999999986)\n",
      "Epoch 026/101 | Loss 0.0381 | Win/lose count 7.699999999999997/32.84999999999991 (-25.149999999999913)\n",
      "Epoch 027/101 | Loss 0.0301 | Win/lose count 3.8500000000000005/36.499999999999844 (-32.64999999999984)\n",
      "Epoch 028/101 | Loss 0.0108 | Win/lose count 7.699999999999997/33.849999999999866 (-26.14999999999987)\n",
      "Epoch 029/101 | Loss 0.0133 | Win/lose count 4.2/36.34999999999983 (-32.14999999999983)\n",
      "Epoch 030/101 | Loss 0.0178 | Win/lose count 10.499999999999995/33.649999999999906 (-23.149999999999913)\n",
      "Epoch 031/101 | Loss 0.0165 | Win/lose count 3.5000000000000004/35.64999999999988 (-32.14999999999988)\n",
      "Epoch 032/101 | Loss 0.0186 | Win/lose count 4.55/36.19999999999987 (-31.649999999999867)\n",
      "Epoch 033/101 | Loss 0.0191 | Win/lose count 8.049999999999997/35.69999999999984 (-27.649999999999842)\n",
      "Epoch 034/101 | Loss 0.0190 | Win/lose count 5.949999999999998/42.599999999999845 (-36.64999999999985)\n",
      "Epoch 035/101 | Loss 0.0276 | Win/lose count 7.699999999999997/35.84999999999986 (-28.149999999999864)\n",
      "Epoch 036/101 | Loss 0.0410 | Win/lose count 3.5000000000000004/32.64999999999988 (-29.149999999999878)\n",
      "Epoch 037/101 | Loss 0.0194 | Win/lose count 5.599999999999999/42.74999999999985 (-37.14999999999985)\n",
      "Epoch 038/101 | Loss 0.0428 | Win/lose count 5.599999999999999/37.74999999999988 (-32.14999999999988)\n",
      "Epoch 039/101 | Loss 0.0358 | Win/lose count 6.299999999999998/34.449999999999854 (-28.149999999999856)\n",
      "Epoch 040/101 | Loss 0.0359 | Win/lose count 4.55/35.19999999999983 (-30.64999999999983)\n",
      "Epoch 041/101 | Loss 0.0392 | Win/lose count 8.399999999999997/32.5499999999999 (-24.1499999999999)\n",
      "Epoch 042/101 | Loss 0.0189 | Win/lose count 4.2/33.34999999999988 (-29.14999999999988)\n",
      "Epoch 043/101 | Loss 0.0387 | Win/lose count 3.1500000000000004/35.79999999999984 (-32.64999999999984)\n",
      "Epoch 044/101 | Loss 0.0183 | Win/lose count 5.249999999999999/29.899999999999892 (-24.649999999999892)\n",
      "Epoch 045/101 | Loss 0.0185 | Win/lose count 7.349999999999997/32.99999999999989 (-25.649999999999896)\n",
      "Epoch 046/101 | Loss 0.0175 | Win/lose count 8.399999999999997/40.54999999999986 (-32.149999999999864)\n",
      "Epoch 047/101 | Loss 0.0228 | Win/lose count 4.2/33.349999999999866 (-29.149999999999867)\n",
      "Epoch 048/101 | Loss 0.0095 | Win/lose count 4.8999999999999995/37.049999999999876 (-32.14999999999988)\n",
      "Epoch 049/101 | Loss 0.0263 | Win/lose count 5.949999999999998/29.599999999999916 (-23.649999999999917)\n",
      "Epoch 050/101 | Loss 0.0272 | Win/lose count 10.849999999999994/31.499999999999904 (-20.64999999999991)\n",
      "Epoch 051/101 | Loss 0.0267 | Win/lose count 8.049999999999997/36.699999999999875 (-28.649999999999878)\n",
      "Epoch 052/101 | Loss 0.0294 | Win/lose count 9.099999999999996/36.24999999999988 (-27.149999999999885)\n",
      "Epoch 053/101 | Loss 0.0275 | Win/lose count 4.2/34.349999999999866 (-30.149999999999867)\n",
      "Epoch 054/101 | Loss 0.0336 | Win/lose count 5.949999999999998/33.59999999999991 (-27.64999999999991)\n",
      "Epoch 055/101 | Loss 0.0071 | Win/lose count 4.2/33.34999999999986 (-29.14999999999986)\n",
      "Epoch 056/101 | Loss 0.0066 | Win/lose count 8.399999999999997/38.5499999999999 (-30.1499999999999)\n",
      "Epoch 057/101 | Loss 0.0082 | Win/lose count 6.649999999999998/40.29999999999984 (-33.64999999999984)\n",
      "Epoch 058/101 | Loss 0.0360 | Win/lose count 6.999999999999997/35.14999999999985 (-28.149999999999853)\n",
      "Epoch 059/101 | Loss 0.0381 | Win/lose count 5.599999999999999/38.74999999999983 (-33.14999999999983)\n",
      "Epoch 060/101 | Loss 0.0188 | Win/lose count 9.449999999999996/40.09999999999985 (-30.649999999999856)\n",
      "Epoch 061/101 | Loss 0.0091 | Win/lose count 5.249999999999999/34.89999999999986 (-29.649999999999856)\n",
      "Epoch 062/101 | Loss 0.0188 | Win/lose count 9.799999999999995/37.94999999999989 (-28.149999999999892)\n",
      "Epoch 063/101 | Loss 0.0470 | Win/lose count 6.999999999999997/36.14999999999988 (-29.14999999999988)\n",
      "Epoch 064/101 | Loss 0.0166 | Win/lose count 7.699999999999997/40.8499999999999 (-33.149999999999906)\n",
      "Epoch 065/101 | Loss 0.0095 | Win/lose count 7.699999999999997/30.849999999999877 (-23.14999999999988)\n",
      "Epoch 066/101 | Loss 0.0288 | Win/lose count 9.449999999999996/34.09999999999991 (-24.649999999999913)\n",
      "Epoch 067/101 | Loss 0.0311 | Win/lose count 9.449999999999996/32.09999999999989 (-22.649999999999892)\n",
      "Epoch 068/101 | Loss 0.0331 | Win/lose count 4.2/33.34999999999986 (-29.14999999999986)\n",
      "Epoch 069/101 | Loss 0.0064 | Win/lose count 2.8000000000000003/30.949999999999893 (-28.149999999999892)\n",
      "Epoch 070/101 | Loss 0.0154 | Win/lose count 6.999999999999997/36.14999999999988 (-29.14999999999988)\n",
      "Epoch 071/101 | Loss 0.0182 | Win/lose count 5.599999999999999/36.74999999999985 (-31.149999999999853)\n",
      "Epoch 072/101 | Loss 0.0294 | Win/lose count 6.299999999999998/31.449999999999918 (-25.14999999999992)\n",
      "Epoch 073/101 | Loss 0.0153 | Win/lose count 6.649999999999998/37.29999999999984 (-30.649999999999842)\n",
      "Epoch 074/101 | Loss 0.0149 | Win/lose count 2.8000000000000003/34.94999999999984 (-32.14999999999984)\n",
      "Epoch 075/101 | Loss 0.0106 | Win/lose count 5.949999999999998/31.599999999999895 (-25.649999999999896)\n",
      "Epoch 076/101 | Loss 0.0363 | Win/lose count 5.249999999999999/30.899999999999892 (-25.649999999999892)\n",
      "Epoch 077/101 | Loss 0.0095 | Win/lose count 8.049999999999997/33.69999999999987 (-25.64999999999987)\n",
      "Epoch 078/101 | Loss 0.0087 | Win/lose count 2.45/30.09999999999989 (-27.649999999999892)\n",
      "Epoch 079/101 | Loss 0.0075 | Win/lose count 6.999999999999997/31.1499999999999 (-24.149999999999903)\n",
      "Epoch 080/101 | Loss 0.0091 | Win/lose count 7.349999999999997/34.99999999999985 (-27.649999999999853)\n",
      "Epoch 081/101 | Loss 0.0078 | Win/lose count 13.999999999999991/32.149999999999906 (-18.149999999999913)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 082/101 | Loss 0.0188 | Win/lose count 3.8500000000000005/32.499999999999865 (-28.649999999999864)\n",
      "Epoch 083/101 | Loss 0.0174 | Win/lose count 5.949999999999998/31.599999999999874 (-25.649999999999874)\n",
      "Epoch 084/101 | Loss 0.0086 | Win/lose count 6.649999999999998/35.29999999999988 (-28.649999999999885)\n",
      "Epoch 085/101 | Loss 0.0088 | Win/lose count 9.799999999999995/31.94999999999992 (-22.149999999999928)\n",
      "Epoch 086/101 | Loss 0.0151 | Win/lose count 6.649999999999998/30.299999999999887 (-23.64999999999989)\n",
      "Epoch 087/101 | Loss 0.0185 | Win/lose count 3.5000000000000004/33.64999999999989 (-30.149999999999892)\n",
      "Epoch 088/101 | Loss 0.0390 | Win/lose count 8.399999999999997/31.549999999999905 (-23.149999999999906)\n",
      "Epoch 089/101 | Loss 0.0311 | Win/lose count 8.749999999999996/35.39999999999988 (-26.64999999999988)\n",
      "Epoch 090/101 | Loss 0.0110 | Win/lose count 5.949999999999998/38.59999999999985 (-32.64999999999986)\n",
      "Epoch 091/101 | Loss 0.0272 | Win/lose count 5.249999999999999/33.899999999999885 (-28.649999999999885)\n",
      "Epoch 092/101 | Loss 0.0118 | Win/lose count 10.849999999999994/34.49999999999987 (-23.649999999999878)\n",
      "Epoch 093/101 | Loss 0.0084 | Win/lose count 5.949999999999998/29.5999999999999 (-23.6499999999999)\n",
      "Epoch 094/101 | Loss 0.0189 | Win/lose count 5.599999999999999/37.74999999999986 (-32.14999999999986)\n",
      "Epoch 095/101 | Loss 0.0170 | Win/lose count 13.299999999999992/33.449999999999875 (-20.149999999999885)\n",
      "Epoch 096/101 | Loss 0.0214 | Win/lose count 4.2/32.349999999999895 (-28.149999999999896)\n",
      "Epoch 097/101 | Loss 0.0255 | Win/lose count 5.599999999999999/35.74999999999985 (-30.149999999999853)\n",
      "Epoch 098/101 | Loss 0.0242 | Win/lose count 6.999999999999997/31.14999999999991 (-24.149999999999913)\n",
      "Epoch 099/101 | Loss 0.0188 | Win/lose count 5.599999999999999/33.749999999999865 (-28.149999999999867)\n",
      "Epoch 100/101 | Loss 0.0405 | Win/lose count 7.699999999999997/27.849999999999923 (-20.149999999999928)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8XXWd//HXJ/uetE3atHSl+0LBNmxSFmllEQRURAWhAoqMw+jPGWewvzIz8BtHRUYdZ3BUBgSFAio7KNqyL1LoQmnpvrfplqRNs+/5/P64JyWULG3vTW97z/v5eNxH7z3r9+T28c43n3PO95i7IyIiiS8p3g0QEZGjQ4EvIhISCnwRkZBQ4IuIhIQCX0QkJBT4IiIhocAXEQkJBb7EjZm9YmaVZpYe77bEgplNNrP5wTHtN7MlZvapeLdLpIMCX+LCzEYCZwMOXNZH+0jpi+324FlgATAIGAh8E6iO5Q7icEySQBT4Ei/XAQuBB4DZHRPN7Awz221myZ2mfcbMlgfvk8zsu2a20cz2mtnvzax/MG+kmbmZ3Whm24CXgul/CLZZZWavmdnkTtseYGbPmlm1mS0ys++Z2Rud5k8wswVmts/M1prZVV0djJkVAqOA/3X35uD1prt33tblZrYs2NdGM7somD7EzJ4J9rHBzL7WaZ3bzewxM3vIzKqBr/TyM8gIlt0b/JWxyMwGHfnXJIlEgS/xch0wL3hd2BFK7r4QqAPO77Ts1cDDwftvAlcA5wJDgErg5wdt+1xgInBh8Pl5YCyRXvfSYJ8dfh7sr5jIL57Ov3yyifTYHw7W/RLwP51/YXSyF9gAPGRmVxwcsmZ2GvBb4B+BAuAcYEsw+xGgNDieK4Hvm9nMTqtfDjwWrDevl5/BbCAfGAYMAG4GGrpor4SRu+ul11F9ATOAFqAw+LwG+Han+d8Dfh28zyUSyCOCz6uBmZ2WHRxsKwUYSaREdGIP+y4IlskHkoN1xx+07zeC918AXj9o/V8B/9rNtocCdwMbgXbgNWBsp/V+2sU6w4A2ILfTtB8ADwTvbwdeO2idnn4GNwB/BabG+3vW69h7qYcv8TAbmO/uFcHnh+nUsw4+fzY4mftZYKm7bw3mjQCeDMoV+4mEXxuRunmH7R1vzCzZzH4YlD+q+aBXXQgUEQnJ7V2tG+zr9I59Bfu7hshfAx/h7qXufou7jw7WrSPSq4dIsG/sYrUhwD53r+k0bStwQjdt6u1n8CDwF+BRM9tpZj8ys9Su2ivhoxNAclSZWSZwFZBsZruDyelAgZmd7O7vufsqM9sKXMyHyzkQCb8b3P3NLrY9MnjbeQjYq4mURGYRCft8IiUQA8qBViI983XB8sMO2ter7v7Jwz1Od99uZj8nUq7p2NboLhbdCfQ3s9xOoT8c2NF5cwet0+3PIHAHcEfw8/gTsBa473CPQRKPevhytF1BpDc6CTgleE0EXidS1+/wMJFa9TnAHzpN/yXw72Y2AsDMiszs8h72lws0EamxZwHf75jh7m3AE8DtZpZlZhMOasNzwDgzu9bMUoPXqWY28eCdmFk/M7vDzMYEJ1ULiZRXFgaL3Adcb2Yzg/knmNkEd99OpATzg+CE61TgRj58nuFg3f4MzOwTZnZScNK7mkipp62HbUmIKPDlaJsN3O/u29x9d8eLSO37mk6XHT4CnAe81Kn0A/Az4BlgvpnVEAnU03vY32+JlEh2AKv4IIA73EKk17+bSDnkESK/IAh63BcAXyTSE98N3EnkL5KDNRM5h/ACkaB9P9jOV4JtvQNcD/wUqAJeJVKagcjJ4JHBPp4kco5gQQ/H1NPPoJjICd5qIqWeV4GHetiWhIi56wEoIh3M7E6g2N1n97qwyHFGPXwJteA6+6kWcRqRcsqT8W6XSF/QSVsJu1wiZZwhQBnwY+DpuLZIpI+opCMiEhIq6YiIhMQxVdIpLCz0kSNHxrsZIiLHlSVLllS4e1Fvyx1TgT9y5EgWL14c72aIiBxXghsVe6WSjohISCjwRURCQoEvIhISCnwRkZBQ4IuIhIQCX0QkJBT4IiIhkRCBv3N/Az+Zv5bNFXXxboqIyDErIQJ/b20z//XSBtbvqel9YRGRkEqIwM9OTwagrrk1zi0RETl2JUTg52RERoiobdKT3EREupMYgZ8eBH6jevgiIt1JiMDPTE0myaCuSYEvItKdhAh8MyM7LYVaBb6ISLcSIvAhUsdX4IuIdC9hAj87PUUlHRGRHiRU4KuHLyLSvagC38zuMrM1ZrbczJ40s4Jg+gAze9nMas3s7tg0tWe56uGLiPQo2h7+AmCKu08F1gFzgumNwD8D34ly+4csOz1ZPXwRkR5EFfjuPt/dO1J2ITA0mF7n7m8QCf6jIlLD141XIiLdiWUN/wbg+Rhu77DkqoYvItKjlN4WMLMXgOIuZs1196eDZeYCrcC8w22Amd0E3AQwfPjww139gI6Ttu6OmR3xdkREElWvge/us3qab2azgUuBme7uh9sAd78HuAegpKTksNfvkJ2eQlu709TaTkZq8pFuRkQkYUV7lc5FwK3AZe5eH5smHZkD4+morCMi0qVee/i9uBtIBxYEZZSF7n4zgJltAfKANDO7ArjA3VdFub9udR5ArTAnva92IyJy3Ioq8N19TA/zRkaz7cOVrR6+iEiPEuZO244evm6+EhHpWuIEfoZ6+CIiPUmcwA8ec6jAFxHpWsIEfvaBko7uthUR6UrCBL5q+CIiPUuYwM9OiwR+jQJfRKRLCRP4SUlGVlqyevgiIt1ImMAHPfVKRKQnCRX4uekpKumIiHQjoQJfPXwRke4lWOCrhi8i0p2ECvyc9FRqGhX4IiJdSbDAT6auWYEvItKVhAp8PddWRKR7CRX4ORl6rq2ISHcSK/DTUmhubae5tT3eTREROeZE+4jDu8xsjZktN7MnzawgmP5JM1tiZiuCf8+PTXN7lq3xdEREuhVtD38BMMXdpwLrgDnB9Arg0+5+EjAbeDDK/RwSPddWRKR7UQW+u8939450XQgMDaa/6+47g+krgQwz6/MHzeohKCIi3YtlDf8G4Pkupn8OeNfdm7paycxuMrPFZra4vLw8qgaopCMi0r1eH2JuZi8AxV3MmuvuTwfLzAVagXkHrTsZuBO4oLvtu/s9wD0AJSUlfsgt74KeeiUi0r1eA9/dZ/U038xmA5cCM93dO00fCjwJXOfuG6Nt6KHISU8FFPgiIl3pNfB7YmYXAbcC57p7fafpBcAfgTnu/mZ0TTx02UEPXyUdEZGPiraGfzeQCywws2Vm9stg+i3AGOCfg+nLzGxglPvq1QdX6ehuWxGRg0XVw3f3Md1M/x7wvWi2fSQ6TtrWagA1EZGPSKg7bVOTk0hPSdIAaiIiXUiowIdIWUcnbUVEPirhAl9PvRIR6VrCBX5Oeopq+CIiXUjMwFcPX0TkIxIu8LP11CsRkS4lXODnZKSqpCMi0oXEC/z05AM3XrW3O4+8s4399c1xbpWISPwlXOBnp31wlc4r68qY88QK/rRid5xbJSISf4kX+OkpNLS00drWzsNvbwegUj18EZHEC/zc4CEomyrqeGnNHgCqGlri2SQRkWNCwgV+x3g697+5mXaHzNRk1fBFRIhy8LRjUUfgP750B2ePLaS8pon99erhi4gkXA8/Nwj85tZ2rj5tOPmZqQp8ERESMPA7eviFOenMmjSIfllp7G9QSUdEJOECv+MhKFeVDCU1OYmCLPXwRUQgysA3s7vMbI2ZLTezJ4NHG2Jmp3V60tV7ZvaZ2DS3dxOKc5lz8QRuOudEAPKzUtnf0EKnx+2KiIRStD38BcAUd58KrAPmBNPfB0rc/RTgIuBXZnZUThAnJRlfP3c0BVlpABRkptHc2k5Dix57KCLhFlXgu/t8d+8YuGYhMDSYXt9pegYQt+51QVYqgMo6IhJ6sazh3wA83/HBzE43s5XACuDmTr8APsTMbjKzxWa2uLy8PIbNieinwBcRAQ4h8M3sBTN7v4vX5Z2WmQu0AvM6prn72+4+GTgVmGNmGV1t393vcfcSdy8pKiqK/ogOkp8ZKe3oSh0RCbte6+ruPqun+WY2G7gUmOldnBl199VmVgdMARYfaUOPVEdJp0o9fBEJuWiv0rkIuBW4zN3rO00f1XGS1sxGAOOBLdHs60h1BH6lAl9EQi7aK2fuBtKBBWYGsNDdbwZmAN81sxagHfiGu1dEua8j0i9LJR0REYgy8N19TDfTHwQejGbbsZKRmkx6SpJKOiISegl3p21XdLetiEhYAj9T4+mIiIQi8POzUnXSVkRCLxSB3y8rVTV8EQm9UAS+SjoiImEJfJ20FREJR+DnZ6XS1NpOo0bMFJEQC0Xgd9x8VamHmYtIiIUi8AsyNWKmiEgoAj9fQySLiIQj8AuCIZKrdKWOiIRYKAK/X7ZGzBQRCUXgd/TwVdIRkTALReBnpCaRlpKkm69EJNRCEfhmRkGmhlcQkXCL9olXd5nZGjNbbmZPmlnBQfOHm1mtmX0numZGT3fbikjYRdvDXwBMcfepwDpgzkHzfwo8H+U+YqIgK003XolIqEUV+O4+391bg48LgaEd88zsCmATsDKafcRKQWYqVQ3q4YtIeMWyhn8DQW/ezLKJPNz8jhhuPyoq6YhI2PX6TFszewEo7mLWXHd/OlhmLtAKzAvm3QH81N1rg4eb97T9m4CbAIYPH37oLT9MBVkaIllEwq3XwHf3WT3NN7PZwKXATHf3YPLpwJVm9iOgAGg3s0Z3v7uL7d8D3ANQUlLiB8+PlYKsVBpbIiNmZqQm99VuRESOWb0Gfk/M7CIipZtz3b2+Y7q7n91pmduB2q7C/mjqfPNVcb4CX0TCJ9oa/t1ALrDAzJaZ2S9j0KY+UdAxgJrKOiISUlH18N19zCEsc3s0+4gVDZEsImEXijttIXLSFhT4IhJeIQr8SA9/5/6GOLdERCQ+QhP4g/IymDwkjx/PX8vKnVXxbo6IyFEXmsBPTjLum30q+ZmpXH//Ikor63tfSUQkgYQm8AGK8zN44IbTaGhp4yv3L9LomSISKqEKfIBxg3K559oSNpTV8uiibfFujojIURO6wAc4c/QAcjNS2FXVGO+miIgcNaEMfICi3HTKa5ri3QwRkaMmvIGfo8AXkXAJb+DnplNeq8AXkfAId+Crhy8iIRLqwK9taqW+ubX3hUVEEkB4Az8nHYCKGo2eKSLhEN7Az40EfnmtLs0UkXAIbeAXBj181fFFJCxCG/gDD/TwVdIRkXCIKvDN7C4zW2Nmy83sSTMrCKaPNLOG4ClYx+STsPpnp2GmHr6IhEe0PfwFwBR3nwqsA+Z0mrfR3U8JXjdHuZ+YS0lOYkB2mgJfREIjqsB39/nu3nFd40JgaPRNOnoKdbetiIRILGv4NwDPd/o8yszeNbNXzezs7lYys5vMbLGZLS4vL49hc3qnu21FJEx6DXwze8HM3u/idXmnZeYCrcC8YNIuYLi7fwz4e+BhM8vravvufo+7l7h7SVFRUfRHdBiKctOpOKiHf+MDi/idhk0WkQSU0tsC7j6rp/lmNhu4FJjp7h6s0wQ0Be+XmNlGYBywOOoWx1DH8Arujpmxt7aJF9eUsa6shs9PH0ZSksW7iSIiMRPtVToXAbcCl7l7fafpRWaWHLw/ERgLbIpmX32hKCed5rZ2qhsipyHW7q4BYPu+BhZu2hvPpomIxFy0Nfy7gVxgwUGXX54DLDez94DHgJvdfV+U+4q5g++2XRMEflZaMo8u2h63domI9IVeSzo9cfcx3Ux/HHg8mm0fDR2BX1bTxJiBuazZXU1hThqXnDSYRxZtZ399MwVZaXFupYhIbIT2TlvodLdtcOJ27e4axhfn8oVTh9Pc2s5T7+6IZ/NERGIq1IFflJMBRAK/rd1Zu6eGCcV5TBqSx9Sh+Ty6aDvBeWgRkeNeqAM/LzOFtOQkymub2LavnsaWdsYX5wLwhVOHsWZ3DctLq+LcShGR2Ah14JvZgUsz1+yqBmBiceR2gctOHkJGahJPqqwjIgki1IEPUNgR+LtrSDIYOygHgNyMVKYOLWDFDvXwRSQxhD7wi3I6Ar+akYXZZKQmH5g3aXAeq3dV096uOr6IHP8U+LnpVNQ2sXZ3DROC+n2HSUPyqG9uY+u++m7WFhE5fijwc9PZW9fM1n31TCj+8HA/kwZHPq/aWR2PpomIxJQCPzcdd3DnwBU6HcYOyiElyVi1S3V8ETn+KfCDZ9sCHynppKckM2Zgjnr4IpIQFPjB3bZZackM65f1kfmThuSxUoEvIgkg9IHfMbzCuEG5XQ6HPGlwHmU1TXoylogc90If+IVBSWfi4Nwu508aEjlxu3qXevkicnwLfeBnpiXz7VnjuPq0EV3OP3CljgJfRI5zUQ2PnCi+NWtst/MKstI4oSBTJ25F5LgX+h7+oZg4OE89fBE57kX7iMO7zGyNmS03syfNrKDTvKlm9paZrTSzFWaWEX1z42PSkDw2ldfS0NwW76aIiByxaHv4C4Ap7j4VWAfMATCzFOAhIo82nAycB7REua+4mTQ4j3aHtXtq4t0UEZEjFlXgu/t8d28NPi4EhgbvLwCWu/t7wXJ73f247R5PHqIhFkTk+BfLGv4NwPPB+3GAm9lfzGypmf1TdyuZ2U1mttjMFpeXl8ewObEztF8muekpLC/dH++miIgcsV4D38xeMLP3u3hd3mmZuUArMC+YlALMAK4J/v2Mmc3savvufo+7l7h7SVFRUdQH1BfMjFmTBvHMezvZX98c7+aIiByRXgPf3We5+5QuXk8DmNls4FLgGv/gAbClwKvuXuHu9cCfgGl9dRBHw83njqa+uY3f/HVrvJsiInJEor1K5yLgVuCyINg7/AWYamZZwQncc4FV0ewr3sYX5zJr4kAe+Otm6ptbe19BROQYE20N/24gF1hgZsvM7JcA7l4J/ARYBCwDlrr7H6PcV9z9zXljqKxv4ZF3tse7KSIihy2qO23dfUwP8x4icmlmwpg+oh+nj+rPva9v4tozRpCWovvWROT4ocQ6TN/4xBh2VTXy1Ls74t0UEZHDosA/TOeMLWTq0Hzumr+WfXW6YkdEjh8K/MNkZtz5ualU1bcw54nlfHBhkojIsU2BfwQmDs7jOxeO4y8r9/D4UpV2ROT4oMA/QjfOOJHTR/Xn9mdWsn1ffe8riIjEmQL/CCUnGT++6mQMmPPECpV2ROSYp8CPwtB+WfzDBeN4Y0MFr6w9NscBEhHpoMCP0tWnj2BUYTbf/9NqWtva490cEZFuKfCjlJaSxK0XTWB9WS2/X1wa7+aIiHRLgR8DF04eRMmIfvxkwTpqm47eODs6byAih0OBHwNmxtxLJlJR28ScJ1bw2rpyqhr69gFff1m5m6m3z+e+NzYr+EXkkEQ1lo584GPD+/GVj4/kN29t4dn3dgLwhZJh3Hnl1Jjvq66plX99eiUt7e3823OreGN9OXd9/mQKc9Jjvi8RSRx2LPUOS0pKfPHixfFuRlRqGltYXlrF7xZt55n3dvLC35/DmIG5Md3HD55fza9e3cTjf3MmK3dW870/riY7LZlThhUwYkA2Ewfn8vnpw0hKspjuV0SOTWa2xN1LeltOPfwYy81I5awxhYwvzuX593fx0MJt3H7Z5Jhtf0NZLfe9vpkrpw9l+oj+TB/Rn1NH9ufulzewubyORVsqqW1qJTkpiSunD+19gyISGqrh95HCnHQ+ddJgHl9S+qEHpry1cS8PvrXliLbp7tz+zEoy05L57sUTDkyfODiPn189jT9962xW3H4BowqzeWyJxuwXkQ+L9olXd5nZGjNbbmZPmllBMP2a4IEoHa92MzslNk0+flx7xghqmlp5elmkpr9tbz03PbiYf31mJeU1TYe9vZfWlPHGhgq+c8H4buv1Zsbnpp3Awk37PjLkw6byWp3gFQmxaHv4C4Ap7j4VWAfMAXD3ee5+irufAlwLbHH3ZVHu67gzfUQ/JhTn8uBbW2lqbeOWR5bS2ua0O/z5/V2Hvb3739zCkPwMrjl9eI/LfXbaUMzgsSUf3Bfw0po9nP/jV/mD7hUQCa2oAt/d57t7R71iIdBV0fhLwCPR7Od4ZWZce+YIVu2q5qu/Wczy0ir+84unMHZgDs++d3iBv6m8ljc2VHD16cNJSe75axtSkMmMMYU8vrSU9nanvrmVf35qJQAPv7PtiI9HRI5vsazh3wA838X0LxDSwAe44pQTyElP4fX1FVx/1kgunFzMpVOHsGjrPnZXNR7ydua9vY2UJOOqU4cd0vJXTh9KaWUDCzfv5WcvrGfH/gYumTqYZdv3s2Z39ZEejogcx3oNfDN7wcze7+J1eadl5gKtwLyD1j0dqHf393vY/k1mttjMFpeXJ94AZNnpKfzd+WOYNXEgcy6eCMClJw/GHf644tB6+Q3NbTy2pJSLphQzMDfjkNa5cHIxuekp/GT+Ou59YzNXlQzl3y6fQlpyEr9bpBO6ImHUa+C7+yx3n9LF62kAM5sNXApc4x89I/hFeundu/s97l7i7iVFRUVHehzHtK+fO5p7Z5964KHno4tymDQ4j+eW7zyk9Z9dvpOqhha+fMaIQ95nRmoyl548hMVbK8nPTGXOxRPpn53GJycP4sl3d9DU2nZExyIix69or9K5CLgVuMzd6w+alwR8Hng0mn0kqk+fPIR3t+3v8uEpza3tbKmoo6098vvzoYVbGTswh9NH9T+sfXzptGEkGfzLpZPol50GwBdPHcb++hbmr9wDRC71XLRlH40t+gUgkuiivfHqbiAdWGBmAAvd/eZg3jlAqbtvinIfCenSqYO5889r+OOKXdx87mgAqhtbePjtbfz6jc2U1TSRlZbM+OJclpdWccdlkwl+xods6tAC3v3nC8jPSj0w7azRhZxQkMnvFm1n8pA8bnvqff66cS9XTh/Kf3z+5Jgeo4gcWzS0Qhxd/vM3Kd1Xz5iBOTS2trOxrJbaplbOGjOAi6cMZkNZLe+V7qehuY3f33wmeRmpvW/0EPzshfX89IV1pKUkkZ6SxCnDCnh9fQWPfO0Mzhw9ICb7EJGj51CHVlDgx9H8lbv5n1c2kpaSREZqMoPzMrj2zBFMOSG/T/e7q6qBi3/2OjPGFPIvl04iLzOVC376GilJxp++dTYZqcl9un8RiS0FvvTI3T9UInp9fTnX3vcO35w5lr//5Di2VNTx4poyPjG+iBOLcuLYUhHpjQZPkx4dfD7g7LFFXHHKEH7xygZeXL2HlTsj1+r/9q0snv27GQfKSZV1zXzlgUWUjOjHbZdMPOzzCiISPxo8TQ647dJJDMrLIDU5idsumcgvvzyN0soG/vEP7+HuNLa08dXfLua97fu5743NPLhwa7ybLCKHQT18OaAwJ503bj3/Q9PmXNzA9/64mv99fRNLt+5n6bZK/vtLH+PpZTu449lVjCnK4eNjCuPUYhE5HOrhS49unDGKCycP4vt/WsOfV+7mtksm8emTh/DTL5zCiYXZfOPhpWzdWxfvZorIIVDgS4/MjLs+fzLThhdwyyfGcOOMUUDkQS/3zo6cI7ru1+8c1rhAIhIfukpHorJ0WyXX3vs2g/IyePTrZxzyWD8iEjuHepWOevgSlWnD+/HADaexq6qRa/73bVbtrGZLRR3b99VrvB6RY4x6+BITb23cy/UPvENjS/uBaWeNGcC8r54Rx1aJhIOuw5ej6szRA3ju72awcmc1be3Okq2VzHt7G4u27OPUkYc36JuI9A2VdCRmxgzM5fJTTuCz04Yy95LIcMz/8/KGeDdLRAIKfOkTWWkp3HDWSF5eW877O6ri3RyRY9r7O6pYtbPvn0SnwJc+c+2ZI8lNT+EXr2yMd1NEjkk1jS3c8exKLrv7De7885o+359q+NJn8jNTufbMEfzi1Y1sKKtlzEANwiaJa3NFHdUNLQeGHR9SkPmhkWe376vn5bVlVDe0kJqcRLvDA3+NPPviy6eP4DsXju/zNirwpU/dMGMUv35zM19/cDETBueRl5HK8P5ZnD22kEmD80hK0uBrcvxobGljydZKJg7Oo3/wFLmaxhZ++Pwa5r297UPLJicZY4pyGFecy/o9NazZXfOR7U0eksevri3hlGEFR6X9UV2WaWZ3AZ8GmoGNwPXuvt/MUoF7gWlEfqn81t1/0Nv2dFlmYvr94u08/PY2qhtbqG5ooaK2GYiM3fOVj4/glvPHxrmFEibt7c7q3dXsqGxg5/4G6prb+NRJgxlVmN3l8pHHgFbyxNJS/rh8FzVNraQkGeeNH8hZYwbwv69tYld1IzecNYqPjx5AS1s7DS1tbCqvY+XOatburmFY/0xmTRzErImDGFKQSUtbOy1t7eRnpsZkxNmjMh6+mV0AvOTurWZ2J4C732pmVxN5zu0XzSwLWAWc5+5betqeAj8cyqobeW19BU++W8qbG/by3N/N6POHvkj4rNpZzfLS/YwrzmXS4Dza3XlsSSn3v7mFzRUfHf/p3HFFfOm04Qzvn0VuRgqt7c5z7+3ksaWlbN1bT1ZaMhdPGcwnJw1iydZ9PLVsJ+U1TYwZmMOPrpzKtOH94nCUEUf9AShm9hngSne/xsy+BFwNfAbIB94CznD3fT1tQ4EfLtWNLcz44UucNmrAgXF5RKLR2NLGc8t3Me/trby7bf+B6clJRnpKEvXNbZw8NJ9rzhjBhOJcBudn0u7Oo+9sZ97bWymrafrINs88cQBXTh/KxScVk5X2QRW8ta2dtXtqGF2UE/enxMUj8J8FfufuDwUlnQeBmUAW8G13v6eb9W4CbgIYPnz49K1bNcZ6mPz3i+v58YJ1PHPLWUwdenTqmJJ43J1nl+/iR39eQ2llA6OLsrn69BGcO66IjeW1rCitYm9dM1dOP4Fpw/t1WUZpaWtn6dZKKuubqW5opamtnfPGFTGsf1YcjujwxCzwzewFoLiLWXPd/elgmblACfBZd3czOwv4BvAVoB/wOnCxu2/qaV/q4YdPTWMLZ//oZT42rID7rz8t3s2RY0x7u1NR20RORgqZqclUNbSweEsli7bsY8f+BnIzUshJT2Hx1kre3bafiYPz+O7FEzhnbGGonsYWs6G1VokpAAALHElEQVQV3H1WLzuaDVwKzPQPfntcDfzZ3VuAMjN7k8gvhB4DX8InNyOVm845kR/9eS1Lt1XGtQ7ak91VjdQ0ttA/O438zFT21TWzsbyOrXvrGNY/i9NG9Sc1Wbe1xFJVfQvX3f8O722PlGaSDNqDhElLTmJIQQb1zW3UNrVSkJnKj66cyuemDSVZV351K6rLMs3sIuBW4Fx3r+80axtwvpk9RKSkcwbwn9HsSxLX7DNHcu/rm/nBn1bz4I2nx70eerC3Nu7lul+/TUtb938N56ancO74Igpz0tlf30xlfQvZ6cmMHJDNqMJsThqaz/hBuaHqdUajqr6FL9/3Nmt31/CPF44nJcmobWolIzWZkhH9OHlYwTH3/+R4EO1VOhuAdGBvMGmhu99sZjnA/cAkwID73f2u3rankk54/WHxdv7p8eVMPSGfe64rYVBez+Pqt7S1B5e7ZZGfmdrjsu7O6+sreG1dOVedOoxxg3IPuV3b9tZz+c/foH92Gt+cOZbKukiY989O48SibIb3z2LdnlpeWLWHl9aW0djcRkF2Kv2y0qhpbGX7vnpag27poLx0zh1XxFUlwyjRgHJdagtKODf+ZhHrdtfyy2uncf6EQfFu1jHvqJ+0jQUFfrjNX7mb//O7ZeRmpPDDz02lIDOVljanX1YqYzuF9J7qRv7moSUsDa7CGJKfwSnDC7jtkkkMKcg8sFxbu/Pc8p388tVNrN4VGackLSWJ71wwjhtnnNjrn/61Ta187n/+yu7qRp7627O6vU67Jy1t7ZRWNrBo8z5eXVfO6+vLqWlq5duzxnHLJ8aE/sazptY2nl+xm4cWbmXN7hpqm1qBSMnmV9dO5xMTBsa5hccHBb4cl1bvquarv1nMjv0NH5r+8dED+MZ5Y8hMS+ZvHlpCbVMr/3jheBpb2lm7u5oXVpeRkZrML788jZKR/dlUXss/PbacxVsrGTswh6+dcyJnjy3kX59eyfxVe5g2vICLphQzqjCH0UWRskvncktTaxt/O28pL68t5zfXn8aMsbF5UHtdUytzn1zBU8t2cvbYQm67ZBJJBi1tzuD8DPoFd2/2pZa2dhZt3kdWegqD8tIpzEkHoLXNaXcnKy25T0pP5TVNvLmhgu376qlqaKGyvoVX1paxt66ZUYXZnDe+iLyMVHIzUjh91ABOGqp7Mw6VAl+OW1X1LSzaso/kJCM1OYlVu6q49/XNB66RHt4/i3uum86E4rwD62woqznwi+Jz04by5Ls7SE9J4l8+PZnPfuyEAz1pd+epZTv44fNr2FP9wTXXMycM5N+umMKQgky276vnbx9eyvLSKv7t8slce+bImB6fu/PIO9u5/dmVNLd+8MCY7LRk5nxqIlefNrzPev7r99TwD394j+Wl3Y9gmpaSRFFOOsX5GXx89AAunFzM5CF5bNlbz1Pv7uDFNXsYnJ/JmScO4ONjBnR7bqLjMsdX15Xz6rpyVnYaDTIzNZn8zFROGprPdWeO4KzRhaH/aycaCnxJKI0tbTyxdAerdlXxnQvGU5D10Z5wVX0LtzyylNfXVzBr4kC+/5mTGNjDuYCq+hY2763jzQ0V3P3SBpIMvnzGCB5dtJ12d378+ZO5YHJXVyTHxoayWpaX7ic1OYnkJGPe21t5c8NezjixP//+mZMYXRS7weZa29p54K9b+NFf1pKdlszcSybRLyuVPdVNVNQ2kWSQnJSEGVTWNVNW08S2ffW8u62Sdod+WalU1rdgBiUj+lFW08TWvZHrNMYMzOHLpw/ns9OH0tjSxitry3llbRmvr6s4MAzBtBH9OHdcEeeMLWJccQ7pKTrhGksKfAmltnZnfVnNYV8Rs31fPf/3yRW8vr6CyUPy+MU10xk+4OjecOPu/G7Rdv79j6upaWplQnEusyYOYsbYQsYPyj2ick9dUyu/X7yd+97YTGllA7MmDuIHnz2Jotz0Q1p/b20TL64u482NFUwZks+nTx5CcX7kl+iO/Q28tq6cRxdt573t+0lLSTrwF0txXgbnjS/ivPFFnDWmkNyMnk+sS3QU+CKHyd1Ztj1y8048L/krq27k6WU7WbB6D4u37Dtw7fmA7DQG5WXQ1Np24NnBhTlpFOakM3pgDl87+8QDQd7e7sx7Zxv/8Ze1VDW0UDKiH18/dzSzJg7sk/r8itIqnni3lMKcdM4bX8SkwXm6BPUoUuCLJIDKumaWle5nY1kt6/fUUl7bRGZqMumpSeBQUddMRU0T68tqyEhJ5luzxnLe+IHc9tQKFm7ax1ljBvD3nxzP9BHH5g1tEhsKfJEQ2VxRx/97diUvry0HIjeC3XbpRK4qGaaedgjEbGgFETn2jSrM5v7rT+PF1Xt4bV05N583msH5mb2vKKGiwBdJIDMnDmLmRN2ZKl3TaE8iIiGhwBcRCQkFvohISCjwRURCQoEvIhISCnwRkZBQ4IuIhIQCX0QkJI6poRXMrBzYGsUmCoGKGDXneBC24wUdc1jomA/PCHcv6m2hYyrwo2Vmiw9lPIlEEbbjBR1zWOiY+4ZKOiIiIaHAFxEJiUQL/Hvi3YCjLGzHCzrmsNAx94GEquGLiEj3Eq2HLyIi3VDgi4iEREIEvpldZGZrzWyDmX033u3pC2Y2zMxeNrPVZrbSzL4VTO9vZgvMbH3wb0I9vNTMks3sXTN7Lvg8yszeDo73d2aWFu82xpqZFZjZY2a2Jvi+z0zk79nMvh38n37fzB4xs4xE/J7N7NdmVmZm73ea1uX3ahH/FWTacjObFos2HPeBb2bJwM+Bi4FJwJfMbFJ8W9UnWoF/cPeJwBnA3wbH+V3gRXcfC7wYfE4k3wJWd/p8J/DT4HgrgRvj0qq+9TPgz+4+ATiZyPEn5PdsZicA3wRK3H0KkAx8kcT8nh8ALjpoWnff68XA2OB1E/CLWDTguA984DRgg7tvcvdm4FHg8ji3KebcfZe7Lw3e1xAJgROIHOtvgsV+A1wRnxbGnpkNBS4B7g0+G3A+8FiwSEIdL4CZ5QHnAPcBuHuzu+8ngb9nIo9azTSzFCAL2EUCfs/u/hqw76DJ3X2vlwO/9YiFQIGZDY62DYkQ+CcA2zt9Lg2mJSwzGwl8DHgbGOTuuyDySwEYGL+Wxdx/Av8EtAefBwD73b01+JyI3/WJQDlwf1DKutfMsknQ79nddwD/AWwjEvRVwBIS/3vu0N332ie5lgiBb11MS9hrTc0sB3gc+D/uXh3v9vQVM7sUKHP3JZ0nd7Foon3XKcA04Bfu/jGgjgQp33QlqFlfDowChgDZRMoZB0u077k3ffJ/PRECvxQY1unzUGBnnNrSp8wslUjYz3P3J4LJezr+1Av+LYtX+2LsLOAyM9tCpEx3PpEef0Hwpz8k5nddCpS6+9vB58eI/AJI1O95FrDZ3cvdvQV4Avg4if89d+jue+2TXEuEwF8EjA3O6qcROeHzTJzbFHNB/fo+YLW7/6TTrGeA2cH72cDTR7ttfcHd57j7UHcfSeQ7fcndrwFeBq4MFkuY4+3g7ruB7WY2Ppg0E1hFgn7PREo5Z5hZVvB/vON4E/p77qS77/UZ4Lrgap0zgKqO0k9U3P24fwGfAtYBG4G58W5PHx3jDCJ/0i0HlgWvTxGpa78IrA/+7R/vtvbBsZ8HPBe8PxF4B9gA/AFIj3f7+uB4TwEWB9/1U0C/RP6egTuANcD7wINAeiJ+z8AjRM5TtBDpwd/Y3fdKpKTz8yDTVhC5iinqNmhoBRGRkEiEko6IiBwCBb6ISEgo8EVEQkKBLyISEgp8EZGQUOCLiISEAl9EJCT+P3r2N7Jy8TYjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAF6xtZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1MiByMjg1NCBlOWE1OTAzIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNyAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MToweDExMSBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MCBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAAMbZYiEADP//vaG+BTYUyP+T7/8I/+5H7cfWfrixkIJvrVeQ9GAMl8a/5lGYaXpyc8t7R+vTNAEZz8ZS8pY/so8yxXhlWbQ+BTV0nV+BSUJ4ieeJ5SmSomT/5XHy+drTInVVaRMhFVrYDmiHMyOkkhINs/fVjBzqJTx37QhG/VsAGOoXZ/MQtwEc2IF1lDNnSd59BtDad3380NMpMC4lzTAwHQX63xzewd95EWhDHIcBTk/Hy9/BVUYjYr77JOnN0c8asDL1n91O0ddQS+I4I78rvCM4ONFo9DSgvTswlf7WYPo/iIYpfGY4UAlZU1B1wtpCgMnnjqhQ2LdiqcqRYCR4Ue+5QcLal5Q+Wc24xxoti63qAJVwDdNxIP3uknSD/uRAF0YCPHrtvNHXKuipZMYGwjUABMf2oYKV3twahHBmuXsDwAyxYMONI3R2iK7ICQWEd7BDbWLhTyjuYQ044hy/sQzqkn6mnSQePWgV/dbn/Nzs2pcvUAS6UAHNFtCI7ZIqDiUDG7ZOeRve8pHWKg170ef7TXM3T3ROcCkSHSpM+/KXuxerL1shi7SIzzxJ4yomjYRSPoGAjTgPJOxiMztyQfXXYmVDTn88own9np/Tbf/8/k15Myge7AJEVWfUD/cB+ECGJPFyeH0b87AVABEUjyaCso/LGhMImLevSphqmyd5e3rEVb1pnVI8agkE2WbeStHyYAKMSJHq4l4ixgYCkW5ndhR89XUJc1g1SanRSL0Oh+db8M9Jn41bzfZBlnyMdrmJnnNppe6GNXXLMUJcay0JuA6DwUyEBf2lWK351EN2EEgcFEe3Lb+uhF7C2aV9Jwxrm+INAkpVkIPL8Sp5l6o0esvE6il9MiNkXXNy53ZedrXSShqDnV4vx2AX+cpbZYpVYOa97k4gfCRPfErRJWd/8IPED/6iBMmyu7bPDHKe54MpxaaiEZ/haCt0nyNQBwV5U3IL8UmIVsHGPS9diN7rwuRmv7Zb9UFpE0na+YNoPHcE36WdYeC/M9OJX0wFw5mACFq08XAOq7tvmjDOfz9DzY5TEeAAAE1AAAAFEGaIWxDf/6nhABJvo591DW6EpuAAAAAGEGaQjwhkymEN//+p4QAMO6tIIRP8tv7gQAAABlBmmNJ4Q8mUwId//6plgAZSCyuM0v7YETAAAAAKUGah0nhDyZTAhv//qeEADd+y+r4FNfUK/ApUtn4FM7A41dt33rmwhbBAAAAFEGepUURPC//ACC6CP34Msf1irTBAAAADwGexHRCvwArOWDBsxxKrQAAABABnsZqQr8ALXYR5LmfJUyBAAAAG0GayEmoQWiZTAh3//6plgArfyDNAH2A/7qdwAAAABFBmuxJ4QpSZTAhv/6nhAABJwAAAAxBnwpFNEwv/wAAsoEAAAAQAZ8pdEK/AEWEAc0SEdK+mAAAABABnytqQr8ARW1ruA51oLUwAAAAEkGbMEmoQWiZTAhv//6nhAABJwAAAAxBn05FESwv/wAAsoEAAAAQAZ9tdEK/AEWEAc0SEdK+mQAAABABn29qQr8ARW1ruA51oLUwAAAAHUGbckmoQWyZTBRMN//+p4QAfsHia41RJFwp8vPuAAAAEAGfkWpCvwBpnbhNxn16b6UAAAAZQZuTSeEKUmUwIb/+p4QAx9In+pHRpDTQQAAAACFBm7VJ4Q6JlMFNEw7//qmWATfuxcyyz59uCLY67Qd5xMAAAAAQAZ/UakK/AYl2o5X9uHzQQQAAABtBm9lJ4Q8mUwIb//6nhAJp3U/W/XgeDdFMGpAAAAAQQZ/3RRE8L/8BHs/c4WT5uQAAAA8BnhZ0Qr8CkkAdB1LJa0EAAAAQAZ4YakK/AYkmSab6SDiUkAAAABJBmh1JqEFomUwIb//+p4QAAScAAAAQQZ47RREsL/8BHvQRY4CfNwAAABABnlp0Qr8BiZNCJ8WYo1BxAAAAEAGeXGpCvwGJJkmm+kg4lJEAAAAaQZpeSahBbJlMCG///qeEAT346fTwaEhrVMAAAAAdQZpgSeEKUmUwUVLDf/6nhADN+wf5a6W5wTRPkQwAAAAQAZ6fakK/AKg25FXgCf0VgQAAABlBmoFJ4Q6JlMCHf/6plgArvvq+uxBuKgxwAAAAHEGapUnhDyZTAhv//qeEADd+wf5a6QatmKEihEEAAAAQQZ7DRRE8L/8AILP91Fdk4AAAAA8BnuJ0Qr8ALWnKFJtkqycAAAAPAZ7kakK/ABz+cNErnl5bAAAAG0Ga50moQWiZTBTw3/6nhAAi3x0+64mM0UWzVQAAABABnwZqQr8AHQVwa48VbUhhAAAAGkGbCEnhClJlMCHf/qmWAAvHwH/+/ZBuKlpAAAAAEkGbLEnhDomUwId//qmWAACVgAAAAAxBn0pFETwv/wAAsoEAAAAQAZ9pdEK/AAwecnEdl2YugAAAAA8Bn2tqQr8ADB5ybrPVoB8AAAATQZtwSahBaJlMCHf//qmWAACVgQAAAAxBn45FESwv/wAAsoEAAAAQAZ+tdEK/AAwecnEdl2YugQAAAA8Bn69qQr8ADB5ybrPVoB8AAAATQZu0SahBbJlMCHf//qmWAACVgAAAAAxBn9JFFSwv/wAAsoEAAAAQAZ/xdEK/AAwecnEdl2YugAAAAA8Bn/NqQr8ADB5ybrPVoB8AAAASQZv4SahBbJlMCG///qeEAAEnAAAADEGeFkUVLC//AACygAAAABABnjV0Qr8ADB5ycR2XZi6BAAAADwGeN2pCvwAMHnJus9WgHwAAAB5BmjxJqEFsmUwIb//+p4QAFqxWzE/1dveGAmv9C2YAAAAQQZ5aRRUsL/8ADYKvG9gquQAAAA8Bnnl0Qr8ADEJKIUwSCYAAAAAQAZ57akK/ABJdohNxn16lKQAAABlBmn9JqEFsmUwIZ//+nhAAWqvcaF033XHdAAAAEkGenUUVLCv/ABxX4HRS0lg2XAAAABABnr5qQr8AHQVwa48VbUhgAAAAGUGaoEmoQWyZTAhn//6eEABbPdN9FSs18d8AAAAYQZrBSeEKUmUwIZ/+nhAAOj6+/kSI+sPTAAAAGEGa4knhDomUwIZ//p4QACXfEP7ZDH1ibQAAABtBmwNJ4Q8mUwIZ//6eEAAYn2PgKZ+2Qx9Yy4AAAAAXQZskSeEPJlMCG//+p4QABkbUG4DH5A8AAAAZQZtFSeEPJlMCG//+p4QACaoAs220wvnpQQAAABdBm2hJ4Q8mUwIb//6nhAAJt8dPqTwelQAAABJBn4ZFETwr/wAMRDS7vApqbcEAAAAOAZ+nakK/AAxBLjO/De4AAAASQZusSahBaJlMCGf//p4QAAR8AAAADEGfykURLC//AACygQAAABABn+l0Qr8ABQ7KO/AB90vAAAAAEAGf62pCvwAFDso72ePul4AAAAAaQZvtSahBbJlMCG///qeEAAZP2D/CcFuhkMEAAAAYQZoOSeEKUmUwIb/+p4QAA/vsHr2Z8EZ3AAAAGUGaL0nhDomUwId//qmWAAMVUgzQB6S+2DEAAAAbQZpSSeEPJlMCHf/+qZYABMfjz+RekfVBPwpMAAAAEkGecEURPCv/AAeZ+B0JLC3MQAAAABABnpFqQr8AB5gXnOtDDAnBAAAAE0GalkmoQWiZTAh3//6plgAAlYAAAAAMQZ60RREsL/8AALKAAAAAEAGe03RCvwAEyVI78AH3UMEAAAAQAZ7VakK/AATJUjvZ4+6hgAAAABNBmtpJqEFsmUwId//+qZYAAJWBAAAADEGe+EUVLC//AACygQAAABABnxd0Qr8ABMlSO/AB91DAAAAAEAGfGWpCvwAEyVI72ePuoYEAAAATQZseSahBbJlMCHf//qmWAACVgAAAAAxBnzxFFSwv/wAAsoEAAAAQAZ9bdEK/AATJUjvwAfdQwQAAABABn11qQr8ABMlSO9nj7qGAAAAAE0GbQkmoQWyZTAh3//6plgAAlYAAAAAMQZ9gRRUsL/8AALKBAAAAEAGfn3RCvwAEyVI78AH3UMAAAAAQAZ+BakK/AATJUjvZ4+6hgQAAABJBm4ZJqEFsmUwIb//+p4QAAScAAAAMQZ+kRRUsL/8AALKBAAAAEAGfw3RCvwAEyVI78AH3UMEAAAAQAZ/FakK/AATJUjvZ4+6hgQAAABpBm8lJqEFsmUwIb//+p4QABf/YP8JwW6GWQQAAAA9Bn+dFFSwr/wAE1lcCnsAAAAANAZ4IakK/AATYNYeM9gAAABlBmgpJqEFsmUwIb//+p4QAA8/sHr2Z8EaHAAAAGUGaK0nhClJlMCHf/qmWAALt8gzQB6S+2XAAAAASQZpPSeEOiZTAh3/+qZYAAJWAAAAADEGebUURPC//AACygQAAABABnox0Qr8AB0LFYvP4HOpBAAAAEAGejmpCvwAHQNQ5/mW8YsEAAAATQZqTSahBaJlMCHf//qmWAACVgAAAAAxBnrFFESwv/wAAsoAAAAAQAZ7QdEK/AAdCxWLz+BzqQQAAABABntJqQr8AB0DUOf5lvGLAAAAAE0Ga10moQWyZTAh3//6plgAAlYAAAAAMQZ71RRUsL/8AALKBAAAAEAGfFHRCvwAHQsVi8/gc6kAAAAAQAZ8WakK/AAdA1Dn+ZbxiwQAAABNBmxtJqEFsmUwId//+qZYAAJWBAAAADEGfOUUVLC//AACygAAAABABn1h0Qr8AB0LFYvP4HOpBAAAAEAGfWmpCvwAHQNQ5/mW8YsAAAAASQZtfSahBbJlMCG///qeEAAEnAAAADEGffUUVLC//AACygQAAABABn5x0Qr8ABKlSO/AB91NAAAAAEAGfnmpCvwAHQNQ5/mW8YsAAAAAaQZuASahBbJlMCHf//qmWAASBFhujEI59n+EAAAASQZukSeEKUmUwId/+qZYAAJWAAAAADEGfwkU0TC//AACygQAAABABn+F0Qr8AC19AOf1oHMggAAAAEAGf42pCvwAHLUN7FaPuWYEAAAATQZvoSahBaJlMCHf//qmWAACVgQAAAAxBngZFESwv/wAAsoEAAAAQAZ4ldEK/AActQ3suq/hqQQAAABABnidqQr8ABy1DexWj7lmAAAAAEkGaLEmoQWyZTAhv//6nhAABJwAAAAxBnkpFFSwv/wAAsoEAAAAQAZ5pdEK/AActQ3suq/hqQAAAABABnmtqQr8ABy1DexWj7lmAAAAAEkGacEmoQWyZTAhv//6nhAABJwAAAAxBno5FFSwv/wAAsoEAAAAQAZ6tdEK/AActQ3suq/hqQQAAABABnq9qQr8ABy1DexWj7lmAAAAAGkGasUmoQWyZTAhv//6nhAANzSJ/qt8x+LugAAAAKEGa1EnhClJlMCGf/p4QAMn7I6vMssYVPzLJg4DzK35FQ/PdxVbZEcEAAAATQZ7yRTRMK/8AKhZBqQ0OkEGegAAAABABnxNqQr8AKhY8cr+3D+3AAAAAGUGbFUmoQWiZTAhv//6nhAA0rq0dVDbb6oEAAAAdQZs3SeEKUmUwURLDf/6nhAA0/vs+60szU26LYdgAAAAQAZ9WakK/ACxUo3mmKtpu4QAAABlBm1hJ4Q6JlMCG//6nhAAi3x0+o40JDmBBAAAAHEGbeknhDyZTBRU8N//+p4QAFs91P3BTsJomf3AAAAAQAZ+ZakK/ABJZXIq8AUB4gQAAABlBm5tJ4Q8mUwId//6plgAEx+PP37INxV3gAAAAEkGbv0nhDyZTAh3//qmWAACVgQAAAAxBn91FETwv/wAAsoEAAAAQAZ/8dEK/AATqyjvwAfdOQAAAAA8Bn/5qQr8ABR1GiC1HmSkAAAATQZvjSahBaJlMCHf//qmWAACVgQAAAAxBngFFESwv/wAAsoAAAAAPAZ4gdEK/AAUe0d0dt8O/AAAADwGeImpCvwAFHUaILUeZKQAAABxBmidJqEFsmUwIb//+p4QACSj5qms25rx0+2P5AAAAEEGeRUUVLC//AAWKgRWlG40AAAAPAZ5kdEK/AAUe0d55xpSBAAAAEAGeZmpCvwAHl5w17zStGcEAAAAZQZpoSahBbJlMCHf//qmWAATAo50qW7lJwAAAABJBmoxJ4QpSZTAh3/6plgAAlYAAAAAMQZ6qRTRML/8AALKBAAAADwGeyXRCvwAHxbA0POeYgQAAAA8BnstqQr8AB8OcNErnmIEAAAATQZrQSahBaJlMCHf//qmWAACVgQAAAAxBnu5FESwv/wAAsoEAAAAPAZ8NdEK/AAfFsDQ855iBAAAADwGfD2pCvwAHw5w0SueYgQAAABJBmxRJqEFsmUwIb//+p4QAAScAAAAMQZ8yRRUsL/8AALKBAAAADwGfUXRCvwAHxbA0POeYgQAAAA8Bn1NqQr8AB8OcNErnmIEAAAASQZtYSahBbJlMCGf//p4QAAR9AAAADEGfdkUVLC//AACygAAAAA8Bn5V0Qr8AB8WwNDznmIEAAAAPAZ+XakK/AAfDnDRK55iBAAAAGUGbmUmoQWyZTAhv//6nhAAJd8dMf4fVuA0AAAAYQZu6SeEKUmUwIb/+p4QACTfRzQVrMpvnAAAAKEGb3knhDomUwIZ//p4QAE/94G1cCmvqFfMsSwXzLJsHBTfPdn22fS4AAAAUQZ/8RRE8L/8ADEKug/iJgt8nxTEAAAAQAZ4bdEK/AAc/hgMkPJxNgQAAABABnh1qQr8AEF2eOV/biDzAAAAAIEGaAEmoQWiZTBTwz/6eEAC/+yOrzLLPn24LkfjPp1KOAAAAEAGeP2pCvwAn1jxyv7cP8sEAAAAYQZohSeEKUmUwIZ/+nhAAv/r7u05u4t6cAAAAGEGaQknhDomUwIZ//p4QALr7psZcmyrevQAAABhBmmNJ4Q8mUwIZ//6eEAC2fE7Ot0DJECwAAAAYQZqESeEPJlMCGf/+nhAAsfumxlybKt8FAAAAIEGapknhDyZTBRE8M//+nhABnfV35lleJ0cFzDxoA2/xAAAAEAGexWpCvwBWbHjlf24fe0EAAAAaQZrJS+EIQ8kRggoB/IB/YeAIV//+OEAAEXEAAAAnQZ7nRRE8K/8Cr2PtQcTdqsNJJuWqhgculOhbEbt1DsPxwCCGakbAAAAAJAGfCGpCvwKvY+1BxN2qw0km5aqGBy6WkKhKoNE00LcmCpoJnwAAC+Btb292AAAAbG12aGQAAAAAAAAAAAAAAAAAAAPoAAAfkAABAAABAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAALCnRyYWsAAABcdGtoZAAAAAMAAAAAAAAAAAAAAAEAAAAAAAAfkAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAABEAAAARAAAAAAACRlZHRzAAAAHGVsc3QAAAAAAAAAAQAAH5AAAAQAAAEAAAAACoJtZGlhAAAAIG1kaGQAAAAAAAAAAAAAAAAAADIAAAGUAFXEAAAAAAAtaGRscgAAAAAAAAAAdmlkZQAAAAAAAAAAAAAAAFZpZGVvSGFuZGxlcgAAAAotbWluZgAAABR2bWhkAAAAAQAAAAAAAAAAAAAAJGRpbmYAAAAcZHJlZgAAAAAAAAABAAAADHVybCAAAAABAAAJ7XN0YmwAAACVc3RzZAAAAAAAAAABAAAAhWF2YzEAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAABEAEQAEgAAABIAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAY//8AAAAvYXZjQwH0AA3/4QAXZ/QADZGbKCIR0IAAAAMAgAAAGQeKFMsBAAVo6+PESAAAABhzdHRzAAAAAAAAAAEAAADKAAACAAAAABRzdHNzAAAAAAAAAAEAAAABAAAFuGN0dHMAAAAAAAAAtQAAAAQAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAABAAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAABgAAAAABAAACAAAAAAEAAAQAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACAAAAAACAAACAAAAAAYAAAQAAAAAAQAACAAAAAACAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAwAABAAAAAABAAAIAAAAAAIAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAgAAAAAAgAAAgAAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAIAAAAAAIAAAIAAAAAAQAABAAAAAABAAAGAAAAAAEAAAIAAAAAAQAABAAAAAABAAAGAAAAAAEAAAIAAAAAAQAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAgAABAAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAYAAAAAAQAAAgAAAAAEAAAEAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAF0AAAABgAAAAcAAAAHQAAAC0AAAAYAAAAEwAAABQAAAAfAAAAFQAAABAAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAhAAAAFAAAAB0AAAAlAAAAFAAAAB8AAAAUAAAAEwAAABQAAAAWAAAAFAAAABQAAAAUAAAAHgAAACEAAAAUAAAAHQAAACAAAAAUAAAAEwAAABMAAAAfAAAAFAAAAB4AAAAWAAAAEAAAABQAAAATAAAAFwAAABAAAAAUAAAAEwAAABcAAAAQAAAAFAAAABMAAAAWAAAAEAAAABQAAAATAAAAIgAAABQAAAATAAAAFAAAAB0AAAAWAAAAFAAAAB0AAAAcAAAAHAAAAB8AAAAbAAAAHQAAABsAAAAWAAAAEgAAABYAAAAQAAAAFAAAABQAAAAeAAAAHAAAAB0AAAAfAAAAFgAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAAB4AAAATAAAAEQAAAB0AAAAdAAAAFgAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAXAAAAEAAAABQAAAAUAAAAFwAAABAAAAAUAAAAFAAAABYAAAAQAAAAFAAAABQAAAAeAAAAFgAAABAAAAAUAAAAFAAAABcAAAAQAAAAFAAAABQAAAAWAAAAEAAAABQAAAAUAAAAFgAAABAAAAAUAAAAFAAAAB4AAAAsAAAAFwAAABQAAAAdAAAAIQAAABQAAAAdAAAAIAAAABQAAAAdAAAAFgAAABAAAAAUAAAAEwAAABcAAAAQAAAAEwAAABMAAAAgAAAAFAAAABMAAAAUAAAAHQAAABYAAAAQAAAAEwAAABMAAAAXAAAAEAAAABMAAAATAAAAFgAAABAAAAATAAAAEwAAABYAAAAQAAAAEwAAABMAAAAdAAAAHAAAACwAAAAYAAAAFAAAABQAAAAkAAAAFAAAABwAAAAcAAAAHAAAABwAAAAkAAAAFAAAAB4AAAArAAAAKAAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1OC4xMi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Training\n",
    "prefix='cnn_train_explore'\n",
    "env = EnvironmentExploring(grid_size=size, max_time=T, temperature=0.3)\n",
    "agent = DQN_CNN(size, lr=.1, epsilon = 0.1, memory_size=2000, batch_size = 32,n_state=3)\n",
    "train_explore(agent, env, epochs_train, prefix='cnn_train_explore')\n",
    "\n",
    "name = sorted(list(filter(lambda x: prefix in x and '.mp4' in x, os.listdir(prefix))))[-1]\n",
    "HTML(display_videos(name, prefix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Win/lose count 1.0499999999999998/30.699999999999886. Average score (-29.649999999999885)\n",
      "Win/lose count 2.8000000000000003/30.949999999999886. Average score (-28.14999999999989)\n",
      "Final score: -28.14999999999989\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<video alt=\"test\" controls>\n",
       "                <source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAFhhtZGF0AAACrQYF//+p3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1MiByMjg1NCBlOWE1OTAzIC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxNyAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MToweDExMSBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MCBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD00IHRocmVhZHM9NiBsb29rYWhlYWRfdGhyZWFkcz0xIHNsaWNlZF90aHJlYWRzPTAgbnI9MCBkZWNpbWF0ZT0xIGludGVybGFjZWQ9MCBibHVyYXlfY29tcGF0PTAgY29uc3RyYWluZWRfaW50cmE9MCBiZnJhbWVzPTMgYl9weXJhbWlkPTIgYl9hZGFwdD0xIGJfYmlhcz0wIGRpcmVjdD0xIHdlaWdodGI9MSBvcGVuX2dvcD0wIHdlaWdodHA9MiBrZXlpbnQ9MjUwIGtleWludF9taW49MjUgc2NlbmVjdXQ9NDAgaW50cmFfcmVmcmVzaD0wIHJjX2xvb2thaGVhZD00MCByYz1jcmYgbWJ0cmVlPTEgY3JmPTIzLjAgcWNvbXA9MC42MCBxcG1pbj0wIHFwbWF4PTY5IHFwc3RlcD00IGlwX3JhdGlvPTEuNDAgYXE9MToxLjAwAIAAAALfZYiEADP//vaG+BTYUyP+T7/8I/+5H7cfWfrixkIJvrVeQ9GAMl8a/5lGYaXpyc8t7R+vTNAEZz8ZS8pY/sqYnJyuHwKaC5y+ZXhdwG68RGrK6TMpPh4FkiKb9zqGz7VEBkKipqzOKC5G42wlPVA6PHTVPufR4LVq6eecLllWODocmnkElent2oraQ79u2Eu3OhtaJ44CloEcYuspAZbPwIU7ZIgCBl9w3f+nIK6zSWzd5unDfnAy9jpJ6Z3ZAeAqr9uXgpV/WA0/9MsIY/qnsHgKRTiP23KRugxVpsNNeXyuT1OKrgBmmjZ8WXAiwKkjS5uFSDHh7newBUUpv0nBgsuPR5q70cg6plrpOYsU4TdyCBSbjko0sIF9nVy1mOsYAaQ/bkEuu2Fdzx9KjDrCgWVc8w6h5fKguvNbc+A+mvwOOdpl22n3EaFIzZqwueXt9GgzErlJHJ3a352fo/7UjZ3uhk6lvHPCBNewtsUy7yxg2QFG2IqzTO0w6XOB4ih4VY5phvgpgRk98iPUudNED6kVXDNkSenFNbXSwqM1JX/O4BtJBX83uEAGgMRQ2WP1s/q2kSLGV/N8mBEGgGPfIBuvwNO9AD3MgIeMzgAQIdLz9ws6Tz2Ti8xBcmWNvxb2wOzoTk/H6cKstVYeMYkzZ87yXSL9KEjX1YnmL1hcmF4raAP65WHFQ7V3VfIfUmxSu55f6WzyuzUt+7IxSBU5tqZWV9vgTewldlpvVhNKlDtYgfcDjhdnqX+MvHDzGRhgDQMhcP1DjVOmHJe3LNPavIAwoU6Yzdg9EVio6HB9NapktIsnibTEBdLLMIk6KUcaueA62Jk5D4ljZPEBqepz8JzTzkoB1mX2bmCMw+IiTRrlUTPDURFfl6o15BgBnTeuO6iiv5e9PmgSqklgY92pxIEpowlCMvZ3j3OB+FVjn7a2sVdrB3NPX3AN2PFQswMj/wp4EAlS8Qg8VH/0AAN7AAAAE0GaIWxDP/6eEAG54I5/DnN9ZsoAAAAXQZpCPCGTKYQz//6eEAHD9cbe9N91t9MAAAAZQZpjSeEPJlMCG//+p4QAdoHhTrOn3W3ugAAAAB9BmoVJ4Q8mUwURPDP//p4QAuVe5rjn6PuDsv7+ba7hAAAAEAGepGpCvwCa7PLcNm1M+4EAAAAZQZqmSeEPJlMCG//+p4QBJEAWbYgDSGlswQAAACJBmshJ4Q8mUwURPDf//qeEAkEVqmP5b/zHD2qRvg/uhruBAAAAEAGe52pCvwF/dU8mB69s2YAAAAAdQZrqSeEPJlMFPDf//qeEC0PwPBm0GF88JbLJU9IAAAAPAZ8JakK/ApFiPJgN9yXTAAAAGEGbC0nhDyZTAhv//qeEDOtaQRDH5BGEXAAAABdBmyxJ4Q8mUwId//6plgbZOakkKb8peQAAACFBm05J4Q8mUwURPDv//qmWAUztqAf385uKfVCyFMJAs+EAAAAPAZ9takK/AZMlpUigSqIPAAAAEkGbcknhDyZTAh3//qmWAACVgQAAAAxBn5BFETwv/wAAsoAAAAARAZ+vdEK/AP7cd6XbyoRCpbQAAAARAZ+xakK/AP55oloWJSYApo0AAAATQZu2SahBaJlMCHf//qmWAACVgAAAAAxBn9RFESwv/wAAsoAAAAARAZ/zdEK/AP7cd6XbyoRCpbUAAAARAZ/1akK/AP55oloWJSYApowAAAATQZv6SahBbJlMCHf//qmWAACVgQAAAAxBnhhFFSwv/wAAsoEAAAARAZ43dEK/AP7cd6XbyoRCpbQAAAARAZ45akK/AP55oloWJSYApo0AAAATQZo+SahBbJlMCHf//qmWAACVgAAAAAxBnlxFFSwv/wAAsoEAAAARAZ57dEK/AP7cd6XbyoRCpbUAAAARAZ59akK/AP55oloWJSYApowAAAATQZpiSahBbJlMCHf//qmWAACVgAAAAAxBnoBFFSwv/wAAsoEAAAARAZ6/dEK/AP7cd6XbyoRCpbQAAAARAZ6hakK/AP55oloWJSYApo0AAAATQZqmSahBbJlMCHf//qmWAACVgAAAAAxBnsRFFSwv/wAAsoEAAAARAZ7jdEK/AP7cd6XbyoRCpbUAAAARAZ7lakK/AP55oloWJSYApo0AAAATQZrqSahBbJlMCHf//qmWAACVgQAAAAxBnwhFFSwv/wAAsoAAAAARAZ8ndEK/AP7cd6XbyoRCpbQAAAARAZ8pakK/AP55oloWJSYApo0AAAATQZsuSahBbJlMCHf//qmWAACVgAAAAAxBn0xFFSwv/wAAsoAAAAARAZ9rdEK/AP7cd6XbyoRCpbUAAAARAZ9takK/AP55oloWJSYApo0AAAATQZtySahBbJlMCHf//qmWAACVgQAAAAxBn5BFFSwv/wAAsoAAAAARAZ+vdEK/AP7cd6XbyoRCpbQAAAARAZ+xakK/AP55oloWJSYApo0AAAATQZu2SahBbJlMCHf//qmWAACVgAAAAAxBn9RFFSwv/wAAsoAAAAARAZ/zdEK/AP7cd6XbyoRCpbUAAAARAZ/1akK/AP55oloWJSYApowAAAATQZv6SahBbJlMCHf//qmWAACVgQAAAAxBnhhFFSwv/wAAsoEAAAARAZ43dEK/AP7cd6XbyoRCpbQAAAARAZ45akK/AP55oloWJSYApo0AAAATQZo+SahBbJlMCHf//qmWAACVgAAAAAxBnlxFFSwv/wAAsoEAAAARAZ57dEK/AP7cd6XbyoRCpbUAAAARAZ59akK/AP55oloWJSYApowAAAATQZpiSahBbJlMCHf//qmWAACVgAAAAAxBnoBFFSwv/wAAsoEAAAARAZ6/dEK/AP7cd6XbyoRCpbQAAAARAZ6hakK/AP55oloWJSYApo0AAAATQZqmSahBbJlMCHf//qmWAACVgAAAAAxBnsRFFSwv/wAAsoEAAAARAZ7jdEK/AP7cd6XbyoRCpbUAAAARAZ7lakK/AP55oloWJSYApo0AAAATQZrqSahBbJlMCHf//qmWAACVgQAAAAxBnwhFFSwv/wAAsoAAAAARAZ8ndEK/AP7cd6XbyoRCpbQAAAARAZ8pakK/AP55oloWJSYApo0AAAATQZsuSahBbJlMCHf//qmWAACVgAAAAAxBn0xFFSwv/wAAsoAAAAARAZ9rdEK/AP7cd6XbyoRCpbUAAAARAZ9takK/AP55oloWJSYApo0AAAATQZtySahBbJlMCHf//qmWAACVgQAAAAxBn5BFFSwv/wAAsoAAAAARAZ+vdEK/AP7cd6XbyoRCpbQAAAARAZ+xakK/AP55oloWJSYApo0AAAATQZu2SahBbJlMCHf//qmWAACVgAAAAAxBn9RFFSwv/wAAsoAAAAARAZ/zdEK/AP7cd6XbyoRCpbUAAAARAZ/1akK/AP55oloWJSYApowAAAATQZv6SahBbJlMCHf//qmWAACVgQAAAAxBnhhFFSwv/wAAsoEAAAARAZ43dEK/AP7cd6XbyoRCpbQAAAARAZ45akK/AP55oloWJSYApo0AAAATQZo+SahBbJlMCHf//qmWAACVgAAAAAxBnlxFFSwv/wAAsoEAAAARAZ57dEK/AP7cd6XbyoRCpbUAAAARAZ59akK/AP55oloWJSYApowAAAATQZpiSahBbJlMCHf//qmWAACVgAAAAAxBnoBFFSwv/wAAsoEAAAARAZ6/dEK/AP7cd6XbyoRCpbQAAAARAZ6hakK/AP55oloWJSYApo0AAAATQZqmSahBbJlMCHf//qmWAACVgAAAAAxBnsRFFSwv/wAAsoEAAAARAZ7jdEK/AP7cd6XbyoRCpbUAAAARAZ7lakK/AP55oloWJSYApo0AAAATQZrqSahBbJlMCHf//qmWAACVgQAAAAxBnwhFFSwv/wAAsoAAAAARAZ8ndEK/AP7cd6XbyoRCpbQAAAARAZ8pakK/AP55oloWJSYApo0AAAATQZsuSahBbJlMCHf//qmWAACVgAAAAAxBn0xFFSwv/wAAsoAAAAARAZ9rdEK/AP7cd6XbyoRCpbUAAAARAZ9takK/AP55oloWJSYApo0AAAATQZtySahBbJlMCHf//qmWAACVgQAAAAxBn5BFFSwv/wAAsoAAAAARAZ+vdEK/AP7cd6XbyoRCpbQAAAARAZ+xakK/AP55oloWJSYApo0AAAATQZu2SahBbJlMCHf//qmWAACVgAAAAAxBn9RFFSwv/wAAsoAAAAARAZ/zdEK/AP7cd6XbyoRCpbUAAAARAZ/1akK/AP55oloWJSYApowAAAATQZv6SahBbJlMCHf//qmWAACVgQAAAAxBnhhFFSwv/wAAsoEAAAARAZ43dEK/AP7cd6XbyoRCpbQAAAARAZ45akK/AP55oloWJSYApo0AAAATQZo+SahBbJlMCHf//qmWAACVgAAAAAxBnlxFFSwv/wAAsoEAAAARAZ57dEK/AP7cd6XbyoRCpbUAAAARAZ59akK/AP55oloWJSYApowAAAATQZpiSahBbJlMCHf//qmWAACVgAAAAAxBnoBFFSwv/wAAsoEAAAARAZ6/dEK/AP7cd6XbyoRCpbQAAAARAZ6hakK/AP55oloWJSYApo0AAAATQZqmSahBbJlMCHf//qmWAACVgAAAAAxBnsRFFSwv/wAAsoEAAAARAZ7jdEK/AP7cd6XbyoRCpbUAAAARAZ7lakK/AP55oloWJSYApo0AAAATQZrqSahBbJlMCHf//qmWAACVgQAAAAxBnwhFFSwv/wAAsoAAAAARAZ8ndEK/AP7cd6XbyoRCpbQAAAARAZ8pakK/AP55oloWJSYApo0AAAATQZsuSahBbJlMCHf//qmWAACVgAAAAAxBn0xFFSwv/wAAsoAAAAARAZ9rdEK/AP7cd6XbyoRCpbUAAAARAZ9takK/AP55oloWJSYApo0AAAATQZtySahBbJlMCHf//qmWAACVgQAAAAxBn5BFFSwv/wAAsoAAAAARAZ+vdEK/AP7cd6XbyoRCpbQAAAARAZ+xakK/AP55oloWJSYApo0AAAATQZu2SahBbJlMCHf//qmWAACVgAAAAAxBn9RFFSwv/wAAsoAAAAARAZ/zdEK/AP7cd6XbyoRCpbUAAAARAZ/1akK/AP55oloWJSYApowAAAATQZv6SahBbJlMCHf//qmWAACVgQAAAAxBnhhFFSwv/wAAsoEAAAARAZ43dEK/AP7cd6XbyoRCpbQAAAARAZ45akK/AP55oloWJSYApo0AAAATQZo+SahBbJlMCHf//qmWAACVgAAAAAxBnlxFFSwv/wAAsoEAAAARAZ57dEK/AP7cd6XbyoRCpbUAAAARAZ59akK/AP55oloWJSYApowAAAATQZpiSahBbJlMCHf//qmWAACVgAAAAAxBnoBFFSwv/wAAsoEAAAARAZ6/dEK/AP7cd6XbyoRCpbQAAAARAZ6hakK/AP55oloWJSYApo0AAAATQZqmSahBbJlMCHf//qmWAACVgAAAAAxBnsRFFSwv/wAAsoEAAAARAZ7jdEK/AP7cd6XbyoRCpbUAAAARAZ7lakK/AP55oloWJSYApo0AAAATQZrqSahBbJlMCHf//qmWAACVgQAAAAxBnwhFFSwv/wAAsoAAAAARAZ8ndEK/AP7cd6XbyoRCpbQAAAARAZ8pakK/AP55oloWJSYApo0AAAATQZsuSahBbJlMCHf//qmWAACVgAAAAAxBn0xFFSwv/wAAsoAAAAARAZ9rdEK/AP7cd6XbyoRCpbUAAAARAZ9takK/AP55oloWJSYApo0AAAATQZtySahBbJlMCHf//qmWAACVgQAAAAxBn5BFFSwv/wAAsoAAAAARAZ+vdEK/AP7cd6XbyoRCpbQAAAARAZ+xakK/AP55oloWJSYApo0AAAATQZu2SahBbJlMCHf//qmWAACVgAAAAAxBn9RFFSwv/wAAsoAAAAARAZ/zdEK/AP7cd6XbyoRCpbUAAAARAZ/1akK/AP55oloWJSYApowAAAATQZv6SahBbJlMCHf//qmWAACVgQAAAAxBnhhFFSwv/wAAsoEAAAARAZ43dEK/AP7cd6XbyoRCpbQAAAARAZ45akK/AP55oloWJSYApo0AAAATQZo+SahBbJlMCHf//qmWAACVgAAAAAxBnlxFFSwv/wAAsoEAAAARAZ57dEK/AP7cd6XbyoRCpbUAAAARAZ59akK/AP55oloWJSYApowAAAASQZpiSahBbJlMCG///qeEAAEnAAAADEGegEUVLC//AACygQAAABEBnr90Qr8A/tx3pdvKhEKltAAAABEBnqFqQr8A/nmiWhYlJgCmjQAAABJBmqZJqEFsmUwIZ//+nhAABHwAAAAMQZ7ERRUsL/8AALKBAAAAEQGe43RCvwD+3Hel28qEQqW1AAAAEQGe5WpCvwD+eaJaFiUmAKaNAAAAGkGa6UuoQhBbJEYIKAfyAf2HgCFf/jhAABFxAAAAKUGfB0UVLCv/Aq9j7UHE3arDSSblqoYbQafur/oVaTqYedqhPIjRe6dgAAAAIgGfKGpCvwKvY+1BxN2qw0km5aqGxMuKpecafBBE6b9loD0AAAxgbW9vdgAAAGxtdmhkAAAAAAAAAAAAAAAAAAAD6AAAH5AAAQAAAQAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAC4p0cmFrAAAAXHRraGQAAAADAAAAAAAAAAAAAAABAAAAAAAAH5AAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAABAAAAAARAAAAEQAAAAAAAkZWR0cwAAABxlbHN0AAAAAAAAAAEAAB+QAAAEAAABAAAAAAsCbWRpYQAAACBtZGhkAAAAAAAAAAAAAAAAAAAyAAABlABVxAAAAAAALWhkbHIAAAAAAAAAAHZpZGUAAAAAAAAAAAAAAABWaWRlb0hhbmRsZXIAAAAKrW1pbmYAAAAUdm1oZAAAAAEAAAAAAAAAAAAAACRkaW5mAAAAHGRyZWYAAAAAAAAAAQAAAAx1cmwgAAAAAQAACm1zdGJsAAAAlXN0c2QAAAAAAAAAAQAAAIVhdmMxAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAARABEABIAAAASAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGP//AAAAL2F2Y0MB9AAN/+EAF2f0AA2RmygiEdCAAAADAIAAABkHihTLAQAFaOvjxEgAAAAYc3R0cwAAAAAAAAABAAAAygAAAgAAAAAUc3RzcwAAAAAAAAABAAAAAQAABjhjdHRzAAAAAAAAAMUAAAAEAAAEAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAgAABAAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAHHN0c2MAAAAAAAAAAQAAAAEAAADKAAAAAQAAAzxzdHN6AAAAAAAAAAAAAADKAAAFlAAAABcAAAAbAAAAHQAAACMAAAAUAAAAHQAAACYAAAAUAAAAIQAAABMAAAAcAAAAGwAAACUAAAATAAAAFgAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAXAAAAEAAAABUAAAAVAAAAFwAAABAAAAAVAAAAFQAAABcAAAAQAAAAFQAAABUAAAAWAAAAEAAAABUAAAAVAAAAFgAAABAAAAAVAAAAFQAAAB4AAAAtAAAAJgAAABRzdGNvAAAAAAAAAAEAAAAwAAAAYnVkdGEAAABabWV0YQAAAAAAAAAhaGRscgAAAAAAAAAAbWRpcmFwcGwAAAAAAAAAAAAAAAAtaWxzdAAAACWpdG9vAAAAHWRhdGEAAAABAAAAAExhdmY1OC4xMi4xMDA=\" type=\"video/mp4\" />\n",
       "             </video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluation\n",
    "test(agent,env,11,prefix='cnn_test_explore')\n",
    "HTML(display_videos('cnn_test_explore10.mp4', 'cnn_test_explore/'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "__BONUS question__ Use the expert DQN from the previous question to generate some winning games. Train a model that mimicks its behavior. Compare the performances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}