notebooks/tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "incoming-liberia",
   "metadata": {},
   "source": [
    "# Getting Started with RL4UC\n",
    "\n",
    "This notebook will briefly describe how to use RL4UC. It will cover the following elements:\n",
    "\n",
    "- Making an environment\n",
    "- Interacting with the environment\n",
    "- Training and testing an agent"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "weekly-house",
   "metadata": {},
   "source": [
    "## Making an Environment\n",
    "\n",
    "Environments are specified with just a handful of arguments. The key ones are: \n",
    "\n",
    "- `num_gen`: number of controllable generators (must be at least 5)\n",
    "- `dispatch_freq_mins`: length of settlement periods/frequency of decision-making (every 30 minutes by default)\n",
    "- `voll`: the 'value of lost load', determining the magnitude of the penalty for failing to meet demand\n",
    "- `arma_demand` and `arma_wind`: a dictionary specifying the details of the auto-regressive moving average processes determining the forecast errors for demand and wind\n",
    "- `usd_per_kgco2`: the cost in USD of emitting a kilogram of CO2\n",
    "\n",
    "You can manually create an environment with your specifications using `make_env()`, or you can create an environment from a json file using `make_env_from_json()`. The json which must be located in `data/envs`. There are already a few datasets in there for systems ranging from 5 to 30 generators. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "closing-madrid",
   "metadata": {},
   "source": [
    "First we will create an environment manually. All the variables have default values, so we will just specify a few here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "boring-lawsuit",
   "metadata": {},
   "outputs": [],
   "source": [
    "from rl4uc.environment import make_env\n",
    "import numpy as np\n",
    "\n",
    "env = make_env(num_gen=10, dispatch_freq_mins=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "psychological-brave",
   "metadata": {},
   "source": [
    "The generator specifications are all derived from a single dataset, widely used in the UC literature, from: Kazarlis, S.A., Bakirtzis, A.G. and Petridis, V., 1996. A genetic algorithm solution to the unit commitment problem. IEEE transactions on power systems, 11(1), pp.83-92.\n",
    "\n",
    "For 10 generators, the paper specifies quadratic fuel cost curves, cold and hot start costs (RL4UC currently considers hot starts), minimum up/down times and minimum and maximum operating outputs. \n",
    "\n",
    "Here is the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "sporting-platinum",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>min_output</th>\n",
       "      <th>max_output</th>\n",
       "      <th>status</th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>t_min_down</th>\n",
       "      <th>t_min_up</th>\n",
       "      <th>t_max_up</th>\n",
       "      <th>hot_cost</th>\n",
       "      <th>cold_cost</th>\n",
       "      <th>cold_hrs</th>\n",
       "      <th>kgco2_per_mmbtu</th>\n",
       "      <th>usd_per_mmbtu</th>\n",
       "      <th>fuel_type</th>\n",
       "      <th>outage_rate</th>\n",
       "      <th>outage_weibull_shape</th>\n",
       "      <th>max_cost_per_mwh</th>\n",
       "      <th>min_fuel_cost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>150</td>\n",
       "      <td>455</td>\n",
       "      <td>96</td>\n",
       "      <td>0.00048</td>\n",
       "      <td>16.19</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>96</td>\n",
       "      <td>96</td>\n",
       "      <td>1000000</td>\n",
       "      <td>4500</td>\n",
       "      <td>9000</td>\n",
       "      <td>1000000</td>\n",
       "      <td>95</td>\n",
       "      <td>3</td>\n",
       "      <td>coal</td>\n",
       "      <td>0.001</td>\n",
       "      <td>8.0</td>\n",
       "      <td>22.928667</td>\n",
       "      <td>18.606202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>150</td>\n",
       "      <td>455</td>\n",
       "      <td>96</td>\n",
       "      <td>0.00031</td>\n",
       "      <td>17.26</td>\n",
       "      <td>970.0</td>\n",
       "      <td>96</td>\n",
       "      <td>96</td>\n",
       "      <td>1000000</td>\n",
       "      <td>5000</td>\n",
       "      <td>10000</td>\n",
       "      <td>1000000</td>\n",
       "      <td>95</td>\n",
       "      <td>3</td>\n",
       "      <td>coal</td>\n",
       "      <td>0.002</td>\n",
       "      <td>12.0</td>\n",
       "      <td>23.773167</td>\n",
       "      <td>19.532918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "      <td>130</td>\n",
       "      <td>-60</td>\n",
       "      <td>0.00200</td>\n",
       "      <td>16.60</td>\n",
       "      <td>700.0</td>\n",
       "      <td>60</td>\n",
       "      <td>60</td>\n",
       "      <td>1000000</td>\n",
       "      <td>550</td>\n",
       "      <td>1100</td>\n",
       "      <td>1000000</td>\n",
       "      <td>54</td>\n",
       "      <td>3</td>\n",
       "      <td>gas</td>\n",
       "      <td>0.005</td>\n",
       "      <td>3.0</td>\n",
       "      <td>51.640000</td>\n",
       "      <td>22.244615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>20</td>\n",
       "      <td>130</td>\n",
       "      <td>-60</td>\n",
       "      <td>0.00211</td>\n",
       "      <td>16.50</td>\n",
       "      <td>680.0</td>\n",
       "      <td>60</td>\n",
       "      <td>60</td>\n",
       "      <td>1000000</td>\n",
       "      <td>560</td>\n",
       "      <td>1120</td>\n",
       "      <td>1000000</td>\n",
       "      <td>54</td>\n",
       "      <td>3</td>\n",
       "      <td>gas</td>\n",
       "      <td>0.010</td>\n",
       "      <td>1.1</td>\n",
       "      <td>50.542200</td>\n",
       "      <td>22.005069</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>25</td>\n",
       "      <td>162</td>\n",
       "      <td>-72</td>\n",
       "      <td>0.00398</td>\n",
       "      <td>19.70</td>\n",
       "      <td>450.0</td>\n",
       "      <td>72</td>\n",
       "      <td>72</td>\n",
       "      <td>1000000</td>\n",
       "      <td>900</td>\n",
       "      <td>1800</td>\n",
       "      <td>1000000</td>\n",
       "      <td>54</td>\n",
       "      <td>3</td>\n",
       "      <td>gas</td>\n",
       "      <td>0.005</td>\n",
       "      <td>1.3</td>\n",
       "      <td>37.799500</td>\n",
       "      <td>23.122538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>20</td>\n",
       "      <td>80</td>\n",
       "      <td>-36</td>\n",
       "      <td>0.00712</td>\n",
       "      <td>22.26</td>\n",
       "      <td>370.0</td>\n",
       "      <td>36</td>\n",
       "      <td>36</td>\n",
       "      <td>1000000</td>\n",
       "      <td>170</td>\n",
       "      <td>340</td>\n",
       "      <td>1000000</td>\n",
       "      <td>54</td>\n",
       "      <td>3</td>\n",
       "      <td>gas</td>\n",
       "      <td>0.002</td>\n",
       "      <td>4.0</td>\n",
       "      <td>40.902400</td>\n",
       "      <td>27.454600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>25</td>\n",
       "      <td>85</td>\n",
       "      <td>-36</td>\n",
       "      <td>0.00079</td>\n",
       "      <td>27.74</td>\n",
       "      <td>480.0</td>\n",
       "      <td>36</td>\n",
       "      <td>36</td>\n",
       "      <td>1000000</td>\n",
       "      <td>260</td>\n",
       "      <td>520</td>\n",
       "      <td>1000000</td>\n",
       "      <td>54</td>\n",
       "      <td>3</td>\n",
       "      <td>gas</td>\n",
       "      <td>0.001</td>\n",
       "      <td>1.0</td>\n",
       "      <td>46.959750</td>\n",
       "      <td>33.454209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>10</td>\n",
       "      <td>55</td>\n",
       "      <td>-12</td>\n",
       "      <td>0.00413</td>\n",
       "      <td>25.92</td>\n",
       "      <td>660.0</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>1000000</td>\n",
       "      <td>30</td>\n",
       "      <td>60</td>\n",
       "      <td>1000000</td>\n",
       "      <td>73</td>\n",
       "      <td>3</td>\n",
       "      <td>oil</td>\n",
       "      <td>0.004</td>\n",
       "      <td>2.0</td>\n",
       "      <td>91.961300</td>\n",
       "      <td>38.147150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>10</td>\n",
       "      <td>55</td>\n",
       "      <td>-12</td>\n",
       "      <td>0.00222</td>\n",
       "      <td>27.27</td>\n",
       "      <td>665.0</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>1000000</td>\n",
       "      <td>30</td>\n",
       "      <td>60</td>\n",
       "      <td>1000000</td>\n",
       "      <td>73</td>\n",
       "      <td>3</td>\n",
       "      <td>oil</td>\n",
       "      <td>0.001</td>\n",
       "      <td>1.7</td>\n",
       "      <td>93.792200</td>\n",
       "      <td>39.483009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>55</td>\n",
       "      <td>-12</td>\n",
       "      <td>0.00173</td>\n",
       "      <td>27.79</td>\n",
       "      <td>670.0</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>1000000</td>\n",
       "      <td>30</td>\n",
       "      <td>60</td>\n",
       "      <td>1000000</td>\n",
       "      <td>73</td>\n",
       "      <td>3</td>\n",
       "      <td>oil</td>\n",
       "      <td>0.010</td>\n",
       "      <td>1.5</td>\n",
       "      <td>94.807300</td>\n",
       "      <td>40.066968</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   min_output  max_output  status        a      b       c  t_min_down  \\\n",
       "0         150         455      96  0.00048  16.19  1000.0          96   \n",
       "1         150         455      96  0.00031  17.26   970.0          96   \n",
       "2          20         130     -60  0.00200  16.60   700.0          60   \n",
       "3          20         130     -60  0.00211  16.50   680.0          60   \n",
       "4          25         162     -72  0.00398  19.70   450.0          72   \n",
       "5          20          80     -36  0.00712  22.26   370.0          36   \n",
       "6          25          85     -36  0.00079  27.74   480.0          36   \n",
       "7          10          55     -12  0.00413  25.92   660.0          12   \n",
       "8          10          55     -12  0.00222  27.27   665.0          12   \n",
       "9          10          55     -12  0.00173  27.79   670.0          12   \n",
       "\n",
       "   t_min_up  t_max_up  hot_cost  cold_cost  cold_hrs  kgco2_per_mmbtu  \\\n",
       "0        96   1000000      4500       9000   1000000               95   \n",
       "1        96   1000000      5000      10000   1000000               95   \n",
       "2        60   1000000       550       1100   1000000               54   \n",
       "3        60   1000000       560       1120   1000000               54   \n",
       "4        72   1000000       900       1800   1000000               54   \n",
       "5        36   1000000       170        340   1000000               54   \n",
       "6        36   1000000       260        520   1000000               54   \n",
       "7        12   1000000        30         60   1000000               73   \n",
       "8        12   1000000        30         60   1000000               73   \n",
       "9        12   1000000        30         60   1000000               73   \n",
       "\n",
       "   usd_per_mmbtu fuel_type  outage_rate  outage_weibull_shape  \\\n",
       "0              3      coal        0.001                   8.0   \n",
       "1              3      coal        0.002                  12.0   \n",
       "2              3       gas        0.005                   3.0   \n",
       "3              3       gas        0.010                   1.1   \n",
       "4              3       gas        0.005                   1.3   \n",
       "5              3       gas        0.002                   4.0   \n",
       "6              3       gas        0.001                   1.0   \n",
       "7              3       oil        0.004                   2.0   \n",
       "8              3       oil        0.001                   1.7   \n",
       "9              3       oil        0.010                   1.5   \n",
       "\n",
       "   max_cost_per_mwh  min_fuel_cost  \n",
       "0         22.928667      18.606202  \n",
       "1         23.773167      19.532918  \n",
       "2         51.640000      22.244615  \n",
       "3         50.542200      22.005069  \n",
       "4         37.799500      23.122538  \n",
       "5         40.902400      27.454600  \n",
       "6         46.959750      33.454209  \n",
       "7         91.961300      38.147150  \n",
       "8         93.792200      39.483009  \n",
       "9         94.807300      40.066968  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.gen_info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ambient-blanket",
   "metadata": {},
   "source": [
    "When we set `num_gen < 10`, RL4UC takes a subset of these generators by default. When using more than 10 generators, we use duplicates of these generators."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "frozen-accordance",
   "metadata": {},
   "source": [
    "When making an environment from a json, we need to specify the environment's 'name', that is the prefix to '.json'. As mentioned, this must be in the directory `data/envs`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "first-mouth",
   "metadata": {},
   "outputs": [],
   "source": [
    "from rl4uc.environment import make_env_from_json\n",
    "\n",
    "env = make_env_from_json('20gen_carbon1')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "authorized-ethics",
   "metadata": {},
   "source": [
    "The preset environments are aimed at better defining benchmarks that people can compare performance on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "armed-drain",
   "metadata": {},
   "source": [
    "### Training and Testing Modes\n",
    "\n",
    "Environments have two modes: train and test. Training mode is designed for when we want to sample from a **range of episodes**, specified in `env.profiles_df`. In training mode, when we 'reset' with `env.reset()` the environment samples a new episode from `env.profiles_df`. This makes it useful for training agents. In test mode, we are considering the case where `env.profiles_df` is **a single test episode**, and we want to evaluate the performance. Typically the test problem will not have been observed in training.\n",
    "\n",
    "When making an environment in training mode (e.g. `make_env(mode='train')`), the profiles in `data/train_data_10gen.csv` are automatically scaled to the number of generators and available as episodes. These profile are based on data from the GB power system. By contrast, when using a test environment you **must** specify a test profile.\n",
    "\n",
    "Some other differences between training and testing:\n",
    "\n",
    "- In train mode, states are terminal if there is lost load. In testing the episode continues. \n",
    "- In train mode, resetting the environment randomly initialises the generator up/down times. In test mode they are initialised to `env.gen_info.status`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "north-tuner",
   "metadata": {},
   "source": [
    "## Interacting with the environment\n",
    "\n",
    "RL4UC **roughly** follows the OpenAI Gym API. In other words, we act on the environment by executing `env.step(action)`, which returns an observation, reward and an indicator of whether the state is terminal. (Currently the 'info' return is not implemented). We can reset the environment to begin a new episode with `env.reset()`. **An action is a binary sequence of length `env.num_gen`, where 1 indicates turning the generator on, and 2 indicates turning it off.**\n",
    "\n",
    "For instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "enclosed-request",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward: -3504.3211146136528\n",
      "Done: False\n"
     ]
    }
   ],
   "source": [
    "env = make_env_from_json('5gen') # make the environment\n",
    "obs = env.reset() # reset the environment\n",
    "action = np.array([1,1,1,1,0]) # turn all but the last generator on\n",
    "obs, reward, done = env.step(action) # act on the environment\n",
    "\n",
    "print(\"Reward: {}\".format(reward))\n",
    "print(\"Done: {}\".format(done))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mobile-custom",
   "metadata": {},
   "source": [
    "We can see the reward achieved by the action. In addition, we can see whether the environment is 'done': that is, whether it has reached a terminal state. \n",
    "\n",
    "**NOTE: currently (and counter-intuitively) you can still continue to act on an environment after it has reached a terminal state. When training, you should embed each episode in a while loop, that checks for if environment is done (as in the next section).**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "silent-heating",
   "metadata": {},
   "source": [
    "### States\n",
    "\n",
    "States are comprised predominantly of the following components: generator up/down times, demand forecast, wind forecast, timestep. In addition are the forecast errors, produced by the ARMA processes. **Note: while the forecast errors are returned in the observation, you may want to consider whether the agent *should* observe this in a particular problem setting. If you are training an agent to solve the day-ahead problem, it probably *shouldn't* observe them, as that won't be possible when it comes to testing!**\n",
    "\n",
    "Let's see the observed state from the code block above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "operating-buffer",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'status': array([17, 11, 13,  7, -3]),\n",
       " 'demand_forecast': array([492.54669382, 500.01662836, 499.59599127, 493.50400582,\n",
       "        495.27358255, 496.30341818, 493.63454836, 486.70128873,\n",
       "        483.22015418, 484.51107491, 489.55872   , 496.52098909,\n",
       "        533.52254836, 578.71927855, 623.17626764, 646.74644945,\n",
       "        670.69375418, 680.41192145, 684.60378764, 678.04765091,\n",
       "        668.67759709, 657.13183418, 648.77711127, 642.43854545,\n",
       "        638.91389673, 628.18039855, 625.30846255, 622.79914473,\n",
       "        629.50032873, 632.87993018, 641.91637527, 658.72735418,\n",
       "        672.69540655, 695.07620073, 730.55476364, 761.15973818,\n",
       "        767.36776145, 759.46268509, 749.12081455, 729.96006982,\n",
       "        708.53658764, 683.61746618, 654.62251636, 620.30433164,\n",
       "        598.44570764, 568.44993164, 522.45544145, 513.37548218]),\n",
       " 'demand_errors': array([-7.17739011,  0.        ,  0.        ,  0.        ,  0.        ]),\n",
       " 'wind_forecast': array([230.4912456 , 202.50622308, 235.12445511, 259.64549788,\n",
       "        228.16371347, 260.94585621, 256.41099313, 261.08791216,\n",
       "        269.72054308, 275.46834544, 264.6720931 , 113.36064959,\n",
       "        227.46436109, 288.6576942 , 267.04333476, 289.09478944,\n",
       "        298.55790131, 303.6609882 , 279.3257109 , 262.89093001,\n",
       "        269.97187284, 275.92729544, 284.80032874, 295.86976561,\n",
       "        280.82276209, 248.20453006, 267.33837404, 291.06171801,\n",
       "        309.36508103, 315.33143101, 317.38577863, 295.10484894,\n",
       "        288.05668825, 319.94278576, 319.86629409, 323.69087741,\n",
       "        320.56564647, 311.72539531, 298.76552155, 297.20290608,\n",
       "        313.08039055, 314.80691673, 318.79541076, 303.86860844,\n",
       "        277.7849502 , 311.45221079, 262.19157763, 243.01402413]),\n",
       " 'wind_errors': array([9.31601955, 0.        , 0.        , 0.        , 0.        ]),\n",
       " 'timestep': 0}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "headed-sauce",
   "metadata": {},
   "source": [
    "The `status` component gives the up/down times in decision periods. Remember that the generators have minimum up/down time constraints, so the up/down times are a very important component of the state vector."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "focal-audience",
   "metadata": {},
   "source": [
    "## Training an Agent\n",
    "\n",
    "Now we'll look at how to train an agent. \n",
    "\n",
    "In this case we'll look at a simple Q-learning agent. Needless to say, this is a pretty simple solution that could be vastly improved.\n",
    "\n",
    "**Note: this requires Pytorch to run yourself.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "attached-substitute",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "class QAgent(nn.Module):\n",
    "    def __init__(self, env):\n",
    "        super(QAgent, self).__init__()\n",
    "        self.num_gen = env.num_gen\n",
    "        \n",
    "        self.num_nodes = 32\n",
    "        self.gamma = 0.99\n",
    "        self.activation = torch.tanh\n",
    "        \n",
    "        # There are 2N output nodes, corresponding to ON/OFF for each generator\n",
    "        self.n_out = 2*self.num_gen\n",
    "        \n",
    "        self.obs_size = self.process_observation(env.reset()).size\n",
    "        \n",
    "        self.in_layer = nn.Linear(self.obs_size, self.num_nodes)\n",
    "        self.out_layer = nn.Linear(self.num_nodes, self.n_out) \n",
    "        \n",
    "        self.optimizer = optim.Adam(self.parameters(), lr=3e-04)\n",
    "        self.criterion = nn.MSELoss()\n",
    "        \n",
    "                \n",
    "    def process_observation(self, obs):\n",
    "        \"\"\"\n",
    "        Process an observation into a numpy array.\n",
    "        \n",
    "        Observations are given as dictionaries, which is not very convenient\n",
    "        for function approximation. Here we take just the generator up/down times\n",
    "        and the timestep.\n",
    "        \n",
    "        Customise this!\n",
    "        \"\"\"\n",
    "        obs_new = np.concatenate((obs['status'], [obs['timestep']]))\n",
    "        return obs_new\n",
    "    \n",
    "    def forward(self, obs):\n",
    "        x = torch.as_tensor(obs).float()\n",
    "        x = self.activation(self.in_layer(x))\n",
    "        return self.out_layer(x)\n",
    "        \n",
    "    def act(self, obs):\n",
    "        \"\"\"\n",
    "        Agent always acts greedily w.r.t Q-values!\n",
    "        \"\"\"\n",
    "        processed_obs = self.process_observation(obs)\n",
    "\n",
    "        q_values = self.forward(processed_obs)\n",
    "        q_values = q_values.reshape(self.num_gen, 2)\n",
    "        action = q_values.argmax(axis=1).detach().numpy()\n",
    "        \n",
    "        return action, processed_obs\n",
    "    \n",
    "    def update(self, memory, batch_size=None):\n",
    "        \n",
    "        if batch_size == None:\n",
    "            batch_size = memory.capacity\n",
    "        \n",
    "        data = memory.sample(batch_size)\n",
    "        \n",
    "        qs = self.forward(data['obs']).reshape(batch_size, self.num_gen, 2)\n",
    "        \n",
    "        # A bit of complicated indexing here! \n",
    "        # We are using the actions [batch_size, num_gen] to index Q-values\n",
    "        # which have shape [batch_size, num_gen, 2]\n",
    "        m,n = data['act'].shape\n",
    "        I,J = np.ogrid[:m,:n]\n",
    "        qs = qs[I, J, data['act']]\n",
    "        \n",
    "        next_qs = self.forward(data['next_obs']).reshape(batch_size, self.num_gen, 2)\n",
    "        next_acts = next_qs.argmax(axis=2).detach().numpy()\n",
    "        \n",
    "        # The same complicated indexing! \n",
    "        m,n = next_acts.shape\n",
    "        I,J = np.ogrid[:m,:n]\n",
    "        next_qs = next_qs[I, J, next_acts]\n",
    "        \n",
    "        # Recasting rewards into the same shape as next_qs\n",
    "        m,n = next_qs.shape\n",
    "        rews = np.broadcast_to(data['rew'], (self.num_gen,batch_size)).T\n",
    "        rews = torch.as_tensor(rews).float()\n",
    "\n",
    "        td_target = rews + self.gamma * next_qs\n",
    "                \n",
    "        criterion = nn.MSELoss()\n",
    "        loss = criterion(qs, td_target)\n",
    "        \n",
    "        self.optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        self.optimizer.step()\n",
    "\n",
    "class ReplayMemory(object):\n",
    "\n",
    "    def __init__(self, capacity, obs_size, act_dim):\n",
    "        \n",
    "        self.capacity = capacity\n",
    "        self.obs_size = obs_size\n",
    "        self.act_dim = act_dim \n",
    "        \n",
    "        self.act_buf = np.zeros((self.capacity, self.act_dim))\n",
    "        self.obs_buf = np.zeros((self.capacity, self.obs_size))\n",
    "        self.rew_buf = np.zeros(self.capacity)\n",
    "        self.next_obs_buf = np.zeros((self.capacity, self.obs_size))\n",
    "        \n",
    "        self.num_used = 0\n",
    "        \n",
    "    def store(self, obs, action, reward, next_obs):\n",
    "        \"\"\"Store a transition in the memory\"\"\"\n",
    "        idx = self.num_used % self.capacity\n",
    "        \n",
    "        self.act_buf[idx] = action\n",
    "        self.obs_buf[idx] = obs\n",
    "        self.rew_buf[idx] = reward\n",
    "        self.next_obs_buf[idx] = next_obs\n",
    "        \n",
    "        self.num_used += 1\n",
    "    \n",
    "    def sample(self, batch_size):\n",
    "        idx = np.random.choice(np.arange(self.capacity), size=batch_size, replace=False)\n",
    "        \n",
    "        data = {'act': self.act_buf[idx],\n",
    "                'obs': self.obs_buf[idx],\n",
    "                'rew': self.rew_buf[idx],\n",
    "                'next_obs': self.next_obs_buf[idx]}\n",
    "        \n",
    "        return data\n",
    "        \n",
    "    def is_full(self):\n",
    "        return (self.num_used >= self.capacity)\n",
    "    \n",
    "    def reset(self):\n",
    "        self.num_used = 0 \n",
    "        \n",
    "def train():\n",
    "    \n",
    "    MEMORY_SIZE = 200\n",
    "    N_EPOCHS = 500\n",
    "    \n",
    "    env = make_env_from_json('5gen')\n",
    "    agent = QAgent(env)\n",
    "    memory = ReplayMemory(MEMORY_SIZE, agent.obs_size, env.num_gen)\n",
    "    \n",
    "    log = {'mean_timesteps': [],\n",
    "           'mean_reward': []}\n",
    "    \n",
    "    for i in range(N_EPOCHS):\n",
    "        if i % 10 == 0:\n",
    "            print(\"Epoch {}\".format(i))\n",
    "        epoch_timesteps = []\n",
    "        epoch_rewards = []\n",
    "        while memory.is_full() == False:\n",
    "            done = False\n",
    "            obs = env.reset()\n",
    "            timesteps = 0\n",
    "            while not done: \n",
    "                action, processed_obs = agent.act(obs)\n",
    "                next_obs, reward, done = env.step(action)\n",
    "                \n",
    "                next_obs_processed = agent.process_observation(next_obs)\n",
    "                \n",
    "                memory.store(processed_obs, action, reward, next_obs_processed)\n",
    "                \n",
    "                obs = next_obs\n",
    "                \n",
    "                if memory.is_full():\n",
    "                    break\n",
    "                \n",
    "                timesteps += 1\n",
    "                if done:\n",
    "                    epoch_rewards.append(reward)\n",
    "                    epoch_timesteps.append(timesteps)\n",
    "                    \n",
    "        log['mean_timesteps'].append(np.mean(epoch_timesteps))\n",
    "        log['mean_reward'].append(np.mean(epoch_rewards))\n",
    "        \n",
    "        agent.update(memory)\n",
    "        memory.reset()\n",
    "                    \n",
    "    return agent, log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "attempted-burden",
   "metadata": {},
   "source": [
    "The following cell trains the agent. Note that this might take a few minutes to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "existing-highland",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-6-3a0952178698>:83: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /tmp/pip-req-build-4_wdsgbr/torch/csrc/utils/tensor_numpy.cpp:143.)\n",
      "  rews = torch.as_tensor(rews).float()\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10\n",
      "Epoch 20\n",
      "Epoch 30\n",
      "Epoch 40\n",
      "Epoch 50\n",
      "Epoch 60\n",
      "Epoch 70\n",
      "Epoch 80\n",
      "Epoch 90\n",
      "Epoch 100\n",
      "Epoch 110\n",
      "Epoch 120\n",
      "Epoch 130\n",
      "Epoch 140\n",
      "Epoch 150\n",
      "Epoch 160\n",
      "Epoch 170\n",
      "Epoch 180\n",
      "Epoch 190\n",
      "Epoch 200\n",
      "Epoch 210\n",
      "Epoch 220\n",
      "Epoch 230\n",
      "Epoch 240\n",
      "Epoch 250\n",
      "Epoch 260\n",
      "Epoch 270\n",
      "Epoch 280\n",
      "Epoch 290\n",
      "Epoch 300\n",
      "Epoch 310\n",
      "Epoch 320\n",
      "Epoch 330\n",
      "Epoch 340\n",
      "Epoch 350\n",
      "Epoch 360\n",
      "Epoch 370\n",
      "Epoch 380\n",
      "Epoch 390\n",
      "Epoch 400\n",
      "Epoch 410\n",
      "Epoch 420\n",
      "Epoch 430\n",
      "Epoch 440\n",
      "Epoch 450\n",
      "Epoch 460\n",
      "Epoch 470\n",
      "Epoch 480\n",
      "Epoch 490\n"
     ]
    }
   ],
   "source": [
    "agent, log = train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "appointed-miami",
   "metadata": {},
   "source": [
    "And now let's see if the agent managed to improve its performance: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "divided-sarah",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAD4CAYAAADGmmByAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA/yElEQVR4nO3dd5ycVb348c93+2Z7z7aw6b2RJQnVACEEUAJeVKxRuQZRrNd7EbmKYkXRePn5E+VCpPyERAEhYCAkVIGQZNN7stm07b3vbD2/P+bMZrZmk5nd2dn9vl+vefHsec4588yjme+c8pwjxhiUUkqpCxXg6wtQSinl3zSQKKWU8ogGEqWUUh7RQKKUUsojGkiUUkp5JMjXF+ALiYmJJisry9eXoZRSfmXHjh3lxpik7umjMpBkZWWRk5Pj68tQSim/IiKnekvXri2llFIe0UCilFLKIxpIlFJKeUQDiVJKKY9oIFFKKeURDSRKKaU8ooFEKaWURzSQKHUezlQ28tLuAl9fhlLDyqh8IFGpgapqaKGsvpkpKVEAPPjaYV7ZW0RLWwefyM708dUpNTxoi0Spfnz8kQ9Ytvrdzr8r6lsAeGGntkqUctFAolQ/TpQ3AFDT1Nrl7yMldejuoko5aSBRo15Dcxsv7MynvaPvwFBQ1cTGA8UU1zpIigql0nZ5KaU0kCjFQ68f4bt/28O67WcAeD+3nJJaR5c8T394kjuf3gHAx+enA/C5x7b2G3yUGi00kKhRq7mtnfdzy3l5TxEAT3xwAkdrO599bCu3/N/3aWnr6Mz77LYzJEaG8Pxdl/Kd66ZwxaREjpbUs+lgia8uX6lhQwOJGlH+nnOGX2w41Dmm0Z/vrNvNZx/bSnl9M2kxYRwva+BQUS0ARTUOzlQ1dsl/+aREFlwUT1hwIE9+eSGpMWG8uEsH3ZXSQKJGjFMVDfznc3t59N083j5S2m/eWkcrG/YVd/79yUsyae8wvLb/bNq1v32nS5npqdGdx4EBwvTUaE5Xdg02So1GXgkkIvI9ETEikuiWdq+I5IrIERG53i19gYjss+ceFhGx6aEiss6mbxWRLLcyK0XkmH2tdEsfb/Mes2VDvPF5lH96ZtvpzuPC6q5jHE+8f4Jnt53m3aNlfJBbzt4zNV3OXzstBYBX9hYRIPCzW2bxiQUZfGbROF75xhXMzYjhhllju5RJjw2noLppkD6NUv7D4wcSRSQTuA447ZY2A7gdmAmkAZtFZIoxph14BFgFfAhsAJYDrwJ3AFXGmEkicjvwIPApEYkH7geyAQPsEJH1xpgqm2e1MWatiPzJ1vGIp59J+ac9Z6qZlxnLyYoGCu0XvKO1nQARfvzywS55/+O6KYgNGB0GJiVHAlBQ3cQVkxL53OKL+Nziizrzv3T3FT3eLy02nJqmVuqb24gM1Wd71ejljRbJauC/cH7Ju6wA1hpjmo0xJ4BcYKGIpALRxpgtxjkJ/yngFrcyT9rj54BrbWvlemCTMabSBo9NwHJ77hqbF1vWVZcaZYwxHCisZWZaNGkx4Z2B5DP/+yG3/vH9HvnX5ZxhcnIkn110EZ9ffBHhIYHcPDcNgM9felGP/L1JjwsH6HwvpUYrjwKJiNwMFBhj9nQ7lQ6ccfs736al2+Pu6V3KGGPagBogoZ+6EoBqm7d7Xb1d6yoRyRGRnLKysgF/RuUfzlQ2UedoY1Z6DGm2y6m8vpmdp6s5UOgcQP/mtZPJjHd++edXNfHAilld6njoE3NZu2oxy2akDOg902PDAOczJkqNZucMJCKyWUT29/JaAdwH/Ki3Yr2kmX7SL6RMf3X1PGHMo8aYbGNMdlJSUl/ZlJ/aX+gc85iVFkN6bBgFVU28n1sOQPZFcYQHB7Lqqgn8/lPzOsssnpDQpY6QoAAWT0jADtud00UJEQAcL6v3widQyn+ds2PXGLO0t3QRmQ2MB/bYf3gZwE4RWYizdeC+ol0GUGjTM3pJx61MvogEATFApU1f0q3M20A5ECsiQbZV4l6XGmX2F9QQFCBMGRvJnIxYntxyij+8mUtUWBDPrlpMZUMLkaFBnYsv/uTmmR6/Z2JkKCnRoZ0tHqVGqwvu2jLG7DPGJBtjsowxWTi/8C82xhQD64Hb7Uys8cBkYJsxpgioE5HFdozjC8BLtsr1gGtG1m3Am3YcZSOwTETiRCQOWAZstOfesnmxZV11qVHmQGEtk1OiCA0KZNnMFEKCAjhWWs9lExMIDgwgJdrZDRUVFkzeL25k5WVZXnnfmWkxHCisOXdGpdx0dBgqRtASO4PyHIkx5gDwN+Ag8BrwdTtjC+Au4DGcA/DHcc7YAngcSBCRXOC7wPdtXZXAT4Ht9vWATQO4B/iuLZNg61CjyIHCGrK+/0/eOVrGzDTncx5RYcF8+hJng3jR+IQeZQICBtZ1NRCz0qLJLa2npvHcD0Aq5fLbTUdY8LPNVDe2+PpSvMJrcxZtq8T9758DP+8lXw4wq5d0B/CJPupeA6zpJT0PWHhhV6xGgo1uDxDOSjv7wOAPPzqD+ePiuH7m2N6Kec2ymWN5+M1c/rErny9ePn5Q30uNHH/d6nxaIr+qidgx/v/4mz7ZrvxaraOt83hWekzncVBgALfMTyc8JHBQ339WegwzUqN5ZW/RoL6PGlkamp3/vx0pD7RqIFF+Lbf07Iwp9yVMhtJVU5LYk19NY0vbuTOrUc8YQ2u7c4Lp/2w+xvLfv8sf38718VV5RgOJ8ltt7R0cKqolISKEu5ZMJMJHT5cvnhBPa7thx6kqn7z/SLXmvRNs2FfE01tOdk7lHgnOVJ5thRwsquVwcR2P/euEX2+Upus6KL/1+sESKhpa+PPnFwz6WEh/Lr4oDoDdp6vJr2pi6fQUkqJCfXY9I0FNUysPvHJ2WZsxIYEcfGC5D6/IM47WdvKrGpmUHMWHeRVdzn1v2RQeev0odz69g+ysOFZdNdFHV3nhNJAov/Xa/mKSo0JZOn1gT6IPluiwYNJjw/ntpqMAbJxazBNf0jkgF6KxpY2K+hZ+seEQ4OyuPFRUS1NrO23tHQQF+l8nyr8/uZ3Nh5yrUe/476VsyasgMTKEf1uQQb2jjeWzxvLQ60d5/WAJrx8s4cO8SkICA/jT5xf4+MoHTgOJ8lvHSuuZkRZNoBen816oCUkRnQOn7xwto7y+mcRIbZWcr19uOMzTH54CIDEyhJfvvpwN+4v55rO7OFxc12VChT/ILa3rDCIAbxwq5cO8ChZNSODeG6YDzjGTe2+YRlpsON94dhdvHnbmP1xcy7Sxvhn3O1/+F96VAto7DHll9UxKivT1pQAQZIPZnVdNwBjYm1/t2wvyU5sPOXecvGf5NF779lUEBQZw8bhYAHae9r8xqFfd9rwBWPP+CYpqHF2W5xER7vzIRD42N42X776CL9tp5Bu6lR3ONJAov1RQ1URzW0fn8u++dt9N0/nKleP5+jWTEIG9+fq0e19e3FVAaa2jR3p9cxvFtQ6+de1k7loysbNFlx4bTnJUqF9OZsgtqyctJoxffnw2189M4XBxHQCXTuj5oCzA7IwYfvSxGaTHhnO6omEoL9UjGkiUX3rkHed0SV9N+e1uUnIU9900g+iwYCYlRbJPA0mvSmsdfHvdbr7yVE6Pc/sLajAG5mXGdkkXERZcFMdLuwu77GDpD05VNJKVGMGnF47jqinOxWKDAoSJSRH9lkuPC++xOdtwpoFE+Z32DsO67Wf4t4szmJMx/PrMp4yNIteuCFznaOV3m47yzWd34Whtp6PD8MtXD/Gddbv5y/snaGnr8PHVDi3XL/IDhbU8+u5xfrz+QOc92HOmGqDX/01XzHPuFfPEByeG5kK95HRlIxcljAEg+6J4AD63+KJzrjCd4We7b+pgu/I7VY0tdBjnF85Al3wfSuMTInhtfzFVDS1c+qs3cLQ6vyjHxY9hVno0f34nD4B/7CogKDCAzy8e2EZa/qKjw/Cb14+QERfOinnpXXaPPGIDSVuH4RcbDgNw87w0Lh4Xx978GjLjw0noZZLC8lmp3Do/na3dps4OZ3WOViobWhgX72x9TB0bxQtfu4w5A5gwkBYbTnGtw29mqg3/K1Sqm4p650J3w3VW1EUJY2jvMDyz7TSO1g6umJTI4gnx/OGtXP7zub1d8h4vHXl7mfx16ykeefs49/1jP7Pu30h+VSMldkzkUHHPJfdLahw4WtvZfaaauRmxfdZ7UcIYCmscfO/ve/xiFYH3c51Bb3zi2W6si8fFDSgwpMeF095hKO5lLGkgmlracbS2nzujl2iLRPkd1/LbCZHDc7E71xfH4++dIG5MME9+eSGv7C3kw7xK6hxt3DQnlX/atbme+OAkH5ubxgL7UONI8M99Xdcdu+LBtwBYNiOF1w+WcElWHNtPnh04v+uvOzuPv3R5Vp/1ZtmNxJ7bkU+gCN+/YRoBAUJMeLAXr95zq57Koc7Rxr6CGqamRHH1tPPfSG9cvLM77FRFIxlxYwZcrr3D8NDrR3jk7eMkRoaQ89/XdZ6rbGjhcHEtF4+LIyzYu2vQaSBRfqfMBpLEYRpIsmwgcbS288uPzyYwQLh5bhoTkyIprnEwf1wst8xL51hpHb9+7QhPbTk5YgJJnaOVnJNV3LVkIl+8LItH3j7OEx+cBJwrEQB8euE4rpuRQnl9C4++m9el/NxuA+3u0uPCO4/X5ZxhXc4Z5o+L5R9fu9zrn+NCtbR1dH7OjLhw1nzpEkKDzv9Le6Kd1p5XVs/lkxIHXG5PfjWPvH0cgPL6rkvUv3W4lP/4+x5e+/aVXn8+RQOJ8jvDvWsrMTKUv3zpEiYlRZJpf1mKCLPSYzofqLtuRgrXzUjh7SNlFPrRoOq57C+opa3DsGh8PCnRYdx303Sa29p5dtuZzjzXzUghKszZiugeSGam9f0FNz8zlm8vncz8cXGsXLMNgF2nq73/ITzgPkB+y7x00mPD+8ndt5ToUCJCAjledn5TgM9UNnb5u6aptbPFtv1kJdFhQUxJjrqga+qPBhLldyoamgkKEKLDhleXhrurpyYPKF9GbHiPtZeGs335NWTEhRMX0Xtr8GiJczDdNS07ODCA7y+f3hlIPvj+NZ1BxN29N0wjr6yBMSF9fyUFBQbw7aVTAHjha5exetNRdpyqwhgzbCZdnLTPftw6P527r5l0wfWICBOTIzleNvAxNEdrOxu6dSseL6unrd2w6ukcqhtbuWZaslc3dnPxymC7iHxPRIyIJNq/s0SkSUR229ef3PIuEJF9IpIrIg/bLXex2/Kus+lbRSTLrcxKETlmXyvd0sfbvMds2eHZ16G8qrS2mfiIkEH5BzHUXLNzzlQ2Ut88vAeQOzoMH/vDe8z/6SbWvHeC9XsKae/oumLtkZI6YsKDSXZbtDI6/GxwcG157LJovHNK7J0fmciDt80Z8LVcPC6OqyYn0djS3mVPGl87We4MJD+4cbrH4xCTkiI7A/NAPP7eCTYeKOmSdry0ni3HK6i2O3hePfX8x2sGwuNAIiKZwHXA6W6njhtj5tnXV93SHwFW4dzHfTLgWtLzDqDKGDMJWA08aOuPB+4HFuHcDfF+u3c7Ns9qY8xkoMrWoUa4nFNVzOinC8SfpMeF02Hgyl+/xS/tQoXDVZnbHuMPvHKQbz67q8cv4MNFtUxNierSQnA/7r4u2tN3LOLQBa7qmxrrDEpFNYPXNfj3nDM8acd4BuJQUS1RoUFeGb+bkRZNSW0zZXUD29s9z3aDzU6PYYkNGLll9ZyqaCAtJozDP13O5wZpqrk3WiSrgf8CzrmYvoikAtHGmC3Gufj+U8At9vQK4El7/BxwrW2tXA9sMsZUGmOqgE3AcnvuGpsXW9ZVlxqhTlU0cKK8gSVTBueX1VBLjTn7C323fSBvONqXX9P5tP43r53MC1+7DIA3D5dS63D+2i2tdbD7TDWLJsT3KH/Xkol85cqeWxGHBAVc8C6WqTHO8YeiQXoCvLKhhf98bi/3rz8woCfqHa3tvLa/mGunJ3ulq222HU/bX9B1lYSSWgdbjvfsDq1saGZGajQvf+MKnvjSQqakRHK8tIGTFQ1clBBBWHDgoHUBehRIRORmoMAYs6eX0+NFZJeIvCMiV9q0dCDfLU++TXOdOwNgjGkDaoAE9/RuZRKAapu3e11qhHJ92S7qY60if3PpxATuu3E6H5ubNmwH3eub2/jYH97j3+2yJivsA4S3zEvjH7sKmPeT11m/p5Bvrd1Nh4EV83r+M7xn+TTuu2mGV6/LNZB9urKR1/YXUd3Yco4S5+exf52dCDCQcawtxyuodbRxy3zvfA3NTI9BxDkTy9231+7m0//7Ic9uO92la7Gw2tFlZtvEpEjyyurtMi0Dn0J8Ic4ZSERks4js7+W1ArgP+FEvxYqAccaY+cB3gWdEJBroLRy67kRf5843va/PsUpEckQkp6ysrK9sapgrqnH++nTNhvJ3oUGBfOWqCczNiKGqsZWqBu9+GXrD9pOVncciZ7/Ap9kB9Q4D3/v7HrbkVfDVj0wcsoU0U6JDSY0JY837J/jq/9vZZSMsT9U0tbLm/RPcMi+Ni8fFcqio54OU4Nyl0/Xg39YTlQQHSpeVfT0RGRrE9LHRPVofVTZg3vvCPl7df7ZrsbC6qcsssUnJkeSVN1DR0NI5nXiwnDOQGGOWGmNmdX8BecB4YI+InAQygJ0iMtYY02yMqbDldwDHgSk4Ww0ZbtVnAIX2OB/IBBCRICAGqHRP71amHIi1ebvX1dvneNQYk22MyU5KGhndIqNRUXUTUWFBXZbdGAlcDzHe+8I+H19JV23tHV1+mafFhHcOIrt/abW0dZAQEcJ/Xj91yK5NRLh8UiKnKpxTXmvsgLI37M2vxtHawW0LMpmWGs3WE5UU1/TsQrvz6R0s/PlmfrnhEH965ziz02O8+rDflVMS2Xm6iga3iRju40wHC50BrtbRSl1zG2mxZ7tKr3Lr/r1tgfvXrvddcNeWMWafMSbZGJNljMnC+YV/sTGmWESSRCQQQEQm4BxUzzPGFAF1IrLYjnF8AXjJVrkecM3Iug14046jbASWiUicHWRfBmy0596yebFlXXWpEaqg2kFazIXNzR/OLh4XR3hwIK8dKOZ0ReO5CwyRV/cXdy71AXQZ/3DvRpmbEcOt89OHfJOxT2ZnEmKXHPHmrLd9dlxiVnp059pYq54+u2KxMYYfvbSfNw6XUuto48/v5hEfEdJrt54nrpueQmu74Ycv7u9MK6xu4jOLxjElJbJz7TLXQPs4t5b6JVnxvHz3Fay/+3JixwzuhNbBWmvrKmCviOzBORj+VWOMq318F/AYkIuzpfKqTX8cSBCRXJzdYd8HsOV+Cmy3rwfc6roH+K4tk2DrUCNYUU1T52ydkSQuIoRXv+UcSnzjcMk5cg8d1xfUy3dfQUZcOHdfffbZiAy3QPLi1y/nvz/q3TGQgVg4Pp69P17GLfPSOp/h8Iadp6rJjA8ndoxzS9yLx8VyuLiuc0ziWGk9T2051Zk/LDiAnT+8jpWXZXntGgCys+L5xjWTeGFXAa/sLeS9Y+VUNbaSFhPG1LHRnaspnw18XReEnJ0Rw5x+1i/zFq/1D9hWiev4eeD5PvLlALN6SXcAn+ijzBpgTS/peTinBKtRoqjGMST/MHwhKzGCGanRPLTxCLPTY8jO6jn7aaidqmwgNSaM2RkxvHfPNV3OJUacfVbElw8EhgUHMik5khd3F9LY0tbvQ40DsfFAMZsPlXDHFc5ZZsGBAXxm0UV87+97eHrLSVrbDX9537mc/bb7rmX7iSqmp3r/aXGXry2ZxBMfnOTuZ3Z1pqXFhhMVFszLewrZX1DD/vwa4sYEX/CT9J7S1X+V32hobqOyoaXLL+GRZs0XLyEpKpSvP7OTtnbf71VyqqKxS3eJu4AA4ePz0/n5rT1+Fw4515P01zz0jsdb8r59pIzosCDuvWFaZ9q0sc5A8eOXD/LzDYcorHHwyewMkqPCuGlOKhMGcTA7PCSQqSlnA1XcmGDmZMRwy/x0woMDue8f+3jtQDFzM2N9FtA1kCi/4eq6cK0COxKNjQnjP5ZNpaS2ubO7wpdOVZzdmKk3v/vUPD67yPf7qbhmShXXOli96ahHdR0qqmVGWnSX5d4nJUcSHHj2Szp2TDC/vm2uR+9zPibYHRU/lZ3Jrh8tY1JyFDHhwSydkcKe/BpE4Ic+6Fp00UCi/MbJcucg9GDPife1Syc6vxS3+HgNrtMVjZTXNzMlZfC6bbwlIjSos9UQGnThX2vtHYbDxbXMSO061hAWHEhyVJg9DuCJLw1tj7prgdLk6K4LlbqWPFmYFT/oU3z7o4FE+Y3R0CIB55fGhMSIzq1nfeWl3QUA3DA71afXMVDP3XUZi8bHd04HvhD/+688HK0dzBsX2+Nc7BjnYpN//nx2j33lB5trBd/uC5XeMCuVzy4ax49vnjmk19OdBhLlN06WN5AUFUrECHuGpDfJ0aFUNrTQ1NLOl/6yrccyGUPh3WNlzM2M9dkA7vmKDA1ibmYspyob6eg454pNPTQ0t/HwG8dYOj2Fj/YSPH9z21wum5hAtg/2jvnCpVl869rJfP7Srt2I4SGB/PzW2aT5+H8jDSTKbxTXOnz+D2aoxI0JoaqxlZxTlbx1pIyvP7Pz3IV6UVrnYMepyn7zvJ9b3mMv9PYOw/6CWuYP8S9vT01IjKClreOCugVf219MY0s7d35kQq8rS89Ii+aZryz2yQ+Z8JBAvnPdFK/vbOgtGkiU3yirayZpmO6K6G2xY4Ipqm5i7XbnMnMFVU09lmwfiBv/5z3+7ZEtnTPAXt1XxCf/tIVyu5KvMYbPPraVTz36YZdyf3n/BE2t7T2eSxjubpyTyoTECO55fu953a/2DsPa7ae5KGGMT1oc/k4DifIb5fUtJEUNz10RvS12TAgNLe2de7u3dZjzXtTR0dreGTCOljg3SHp+Zz7bTlaS/bPN/OAf+zjSy34XVQ0t/OyfziXt5/cyVjCcRYcF81/Lp5Jf1cTmQwN/sPOzj33I9pNV3HZxxrDZJMufaCBRfqG9w1DZ0Dxst9f1trgxZwdVr5uRAsCJ8gaMMQP+pe3+PMWND/+LF3cVsDWvsvOZi2e2nuaJ90925nEtPugKLj+4cZpPZwJdqKXTUwgPDmTbif679NwdsGtWrbw8a5CuamTTQKL8QlVjCx1m+O7T7m2x4c4uvOSoUH5ol1//wpptjL93A1/76w4OF9eyt9vy4t11n7307XW7qWtu42tLJrL+7ssBOrvOgM4Wj2tXPm+vGzVUggIDyIwP77F/eV9a2jqoc7Tx7aWTh/X2zcOZBhLlF1xdNKMmkNgWSVRYUI+1xTYeKGH57//FzX94vzNt/Z5C/mfzMQCe3Xaa/QU1nK5sJDhQWHXVBK6YlNiZd15mLLPTY0iw+667Nte65rfvUNPYypHintvl+pvMuDGcHkAgefKDk7yw07lFkus5EXX+NJAov1Be59yDwRtbmPqDKPvLODIsmGC3J6xdLQmX9g7DwcJavvnsLlZvPsrLewq594V93PyH9zhd2UhG3Bh+cON0fnzz2aeeM+LCERE+u2gc0HWJ8Sc+OMmGfUVcPM53y214Q2b8GPKrmjq7Av/9ye0s+Okm8qvOBhdjDPevP8D37dL9o2X8bTCM/An5akRwdbeMlum/EaHOaZ6z053jGZdPSqClrYPZ6TH8n0/PZ39BDX9+N4/rVr/DifIGEiNDaGhu5z/+5tystMPAmcrGzg3AJiSeHetwBYjvXDeFa6enMCs9hosviuOBlw+yerNzeZH/vP7sOlP+KDN+DPXNbazbfobSumY2HyoFnNvWRoYGUVLbTHxE1x8l/twC8zUNJMovvLy3kOmp0SNmZ8RzmZMRy6OfX8BH7BIYf/33xZ3nPjY3jbExYfz53bzOZd6f+NJC3jxcyu/c1pk6UFjLl+3gcUCA8PtPzeuyR7yIMNc+J3L11GQKq5u47x/7iQoNYkZa9CB/wsE10a5N5WptJEeFUlrXzJHietZtP8NbR8r41cdndymjLZILp11batiraWxl1+lqbpw11teXMqSWzRxLaFDvD6C5dlQEePqOhcxKj+EztqvK5dIJCXxr6ZTOv2+Zn97vXvc3z01jSkokD31y6BYjHCzdl+C/9eJ0xoQEsnrzUd464txq+/vddqMcLeNvg0FbJGrYO1zsnJo5O8O/Ho4bTImRofzi1tlcOTmxs5WWGBnKgZ9cT31zG4eL61g8Ib7PQNSbqLBgXv/ORwbrkodU962YL5uYyJ/fObtlcFJUKGV1zgkcX1sykfGJEYR4sNjjaKeBRA17rl3gXM8/KKfuLRBwroIbERpESrTOQHr2K4sprm1iXHwEC9yeVr9yciK/vm0OL+wsYOH4eC4ZBhuI+TuPQrCI/FhECkRkt33d6HbuXhHJFZEjInK9W/oCEdlnzz1s925HREJFZJ1N3yoiWW5lVorIMfta6ZY+3uY9ZsuOjik9o8zh4lrixvj3dFQ19C6dmMCt8zM6g8j3lk1hYlIET9+xiNSYcL5+9SQNIl7ijbbcamPMPPvaACAiM4DbgZnAcuCPIuJqYz8CrAIm29dym34HUGWMmQSsBh60dcUD9wOLcG6re7+IuH5ePGjffzJQZetQI8zJ8kYmJEX69XRU5Xt3XzOZN/5jia8vY0QarE7BFcBaY0yzMeYEkAssFJFUINoYs8UYY4CngFvcyjxpj58DrrWtleuBTcaYSmNMFbAJWG7PXWPzYsu66lIjSEmtg7Ex2lWj1HDljUByt4jsFZE1bi2FdOCMW558m5Zuj7undyljjGkDaoCEfupKAKpt3u519SAiq0QkR0RyysrKzv9TKp8wxlBU42Cs9vkrNWydM5CIyGYR2d/LawXObqqJwDygCPitq1gvVZl+0i+kTH919TxhzKPGmGxjTHZSUlJf2dQwU+too6m1XQOJUsPYOWdtGWOWDqQiEflf4BX7Zz6Q6XY6Ayi06Rm9pLuXyReRICAGqLTpS7qVeRsoB2JFJMi2StzrUiNESa0DgBTt2lJq2PJ01pb7fpS3Avvt8XrgdjsTazzOQfVtxpgioE5EFtsxji8AL7mVcc3Iug14046jbASWiUic7TpbBmy0596yebFlXXWpEaKoxhlIUjWQKDVsefocya9FZB7OLqWTwJ0AxpgDIvI34CDQBnzdGNNuy9wFPAGEA6/aF8DjwNMikouzJXK7ratSRH4KbLf5HjDGuDYauAdYKyI/A3bZOtQIst3uKZEZNzqWRlHKH4nzh/3okp2dbXJycnx9Geocmtvamf/AJq6emsz//ezFvr4cpUY9EdlhjMnunq5rAqhhq7jGQWNLO1dPS/b1pSil+qGBRA1bxXZ8RGdsKTW8aSBRw1axnbGlDyMqNbxpIFHDlmvGlgYSpYY3DSRq2CqucRAVGtRjSXCl1PCigUQNW8U1Dn0QUSk/oIFEDUvGGPbmVzM5OfLcmZVSPqWBRA07xhgefiOXwhoHl01K9PXlKKXOQQOJGnYOFNayevNRAK7UQKLUsKeBRA07G/YVAfDklxeSlRjh46tRSp2LBhI1rJTVNfPXrae5emoSH5miy/0r5Q80kKhh5YWd+dQ0tfKDG6f7+lKUUgOkgUQNK0dL6kmOCmVySpSvL0UpNUAaSNSwkltWz+QUnfKrlD/RQKKGDWMMx0vrmZSkgUQpf6KBRA0beeUN1De3abeWUn5GA4kaNtbvLkQElk5P8fWlKKXOg6d7tv9YRApEZLd93WjTs0SkyS39T25lFojIPhHJFZGH7d7t2P3d19n0rSKS5VZmpYgcs6+Vbunjbd5jtmyIJ59H+dbmQyUszIrX1X6V8jPeaJGsNsbMs68NbunH3dK/6pb+CLAKmGxfy236HUCVMWYSsBp4EEBE4oH7gUXAQuB+EYmzZR607z8ZqLJ1KD/U2t7BsZJ65o2L9fWlKKXO05B2bYlIKhBtjNlinJvFPwXcYk+vAJ60x88B19rWyvXAJmNMpTGmCtgELLfnrrF5sWVddSk/k1fWQEt7B9PHRvv6UpRS58kbgeRuEdkrImvcWgoA40Vkl4i8IyJX2rR0IN8tT75Nc507A2CMaQNqgAT39G5lEoBqm7d7XT2IyCoRyRGRnLKysgv6oGrwHCqqBWB6qgYSpfzNOQOJiGwWkf29vFbg7KaaCMwDioDf2mJFwDhjzHzgu8AzIhINSC9vYVxv1ce5803vlTHmUWNMtjEmOylJl94YbvLK6gkQGK9raynld8659ZwxZulAKhKR/wVesWWagWZ7vENEjgNTcLYaMtyKZQCF9jgfyATyRSQIiAEqbfqSbmXeBsqBWBEJsq0S97qUn8mvaiI1JpyQIJ1IqJS/8XTWVqrbn7cC+216kogE2uMJOAfV84wxRUCdiCy2YxxfAF6y5dcDrhlZtwFv2nGUjcAyEYmzXWfLgI323Fs2L7asqy7lZ/KrmkiPC/f1ZSilLoCnm2H/WkTm4exSOgncadOvAh4QkTagHfiqMabSnrsLeAIIB161L4DHgadFJBdnS+R2AGNMpYj8FNhu8z3gVtc9wFoR+Rmwy9ah/FB+VSOLJyb4+jKUUhfAo0BijPl8H+nPA8/3cS4HmNVLugP4RB9l1gBreknPwzklWPmx1vYOimsdZMSN8fWlKKUugHZIK58rqnbQYSAjVru2lPJHGkiUz+WV1wPobohK+SkNJMrnTpQ3ADAhSQOJUv5IA4nyubyyBqLCgkiI0KXSlPJHGkiUz50ob2BCYgR2/U6llJ/RQKJ8rrC6SWdsKeXHNJAonyuvbyYxUru1lPJXGkiUT7W0dVDraCMhMtTXl6KUukAaSJRPVTa0AJCgLRKl/JYGEuVT5fXNACREaItEKX+lgUT5lLZIlPJ/GkiUT1U0uFokGkiU8lcaSJRPVdS7WiTataWUv9JAonwqv6qJiJBAosM83dFAKeUrGkiUT+WVNzA+SZ9qV8qfaSBRPpVXVs/4xEhfX4ZSygMeBxIR+YaIHBGRAyLya7f0e0Uk15673i19gYjss+cetlvuIiKhIrLOpm8VkSy3MitF5Jh9rXRLH2/zHrNldcTWjzha2ymobmKCLh+vlF/zdM/2q4EVwBxjzEzgIZs+A+dWuTOB5cAfXXu4A48Aq3Du4z7Znge4A6gyxkwCVgMP2rrigfuBRTh3Q7zf7t2OzbPaGDMZqLJ1KD+xbvsZjIFpY6N8fSlKKQ942iK5C/iVMaYZwBhTatNXAGuNMc3GmBNALrBQRFKBaGPMFmOMAZ4CbnEr86Q9fg641rZWrgc2GWMqjTFVwCZguT13jc2LLeuqSw1zH+SW85OXD3DttGSum5Hi68tRSnnA00AyBbjSdi+9IyKX2PR04Ixbvnyblm6Pu6d3KWOMaQNqgIR+6koAqm3e7nWpYe6l3YVEhwfzfz4zn6BAHapTyp+dc86liGwGxvZy6j5bPg5YDFwC/E1EJgC9TcEx/aRzAWX6q6sHEVmFs0uNcePG9ZVNDZFDxbXMTItmTIhO+1XK353zp6AxZqkxZlYvr5dwtgJeME7bgA4g0aZnulWTARTa9Ixe0nEvIyJBQAxQ2U9d5UCszdu9rt4+x6PGmGxjTHZSUtK5PrYaRG3tHRwprmP62GhfX4pSygs87VN4Eec4BSIyBQjB+QW/HrjdzsQaj3NQfZsxpgioE5HFdozjC8BLtq71gGtG1m3Am3YcZSOwTETi7CD7MmCjPfeWzYst66pLDWM7TlXR3NbB9FQNJEqNBJ72K6wB1ojIfqAFWGm/4A+IyN+Ag0Ab8HVjTLstcxfwBBAOvGpfAI8DT4tILs6WyO0AxphKEfkpsN3me8AYU2mP7wHWisjPgF22DjWM1Tpa+d5ze0iPDWepDrIrNSKI83t/dMnOzjY5OTm+voxR6ZcbDvHYeyf4252LWXBRvK8vRyl1HkRkhzEmu3u6TpdRQ8YYw8t7ClkyJUmDiFIjiAYSNWRyTlVRWOPghtmpvr4UpZQXaSBRQ6K5rZ2H3zhGVFgQN8zqbTa5UspfaSBRQ+LhN47xr2PlrLpyAhGh+uyIUiOJBhI1JLafqGJWejTfuHayry9FKeVlGkjUoOvoMBwsqmV+Zty5Myul/I4GEjXo8quaqG9uY0aaPoCo1EikgUQNur0F1QDM1ECi1IikgUQNuh2nqggPDtQlUZQaoTSQqEG381QVczNjCNbl4pUakfRfthpUre0dHCyqZW5GrK8vRSk1SDSQqEF1qqKR1nbDlBTdTlepkUoDiRpUx0rqADSQKDWCaSBRg8YYwzPbTiMCk5IjfX05SqlBooFEDZrndxbwr2PlzM2IJTwk0NeXo5QaJBpIlNcdKa6jsaWNtdtOMy5+DC/cdZmvL0kpNYh09TzlVe0dhut//y5pMWEU1jj4ztIpBASIry9LKTWIPG6RiMg3ROSIiBwQkV/btCwRaRKR3fb1J7f8C0Rkn4jkisjDdu927P7u62z6VhHJciuzUkSO2ddKt/TxNu8xWzbE08+jPFNR3wxAYY0DgEuydH0tpUY6jwKJiFwNrADmGGNmAg+5nT5ujJlnX191S38EWAVMtq/lNv0OoMoYMwlYDTxo3yMeuB9YBCwE7hcR17fTg8BqY8xkoMrWoXyotK65y99zMmN9cyFKqSHjaYvkLuBXxphmAGNMaX+ZRSQViDbGbDHOzeKfAm6xp1cAT9rj54BrbWvlemCTMabSGFMFbAKW23PX2LzYsq66lI+U1jlbIpdNTODG2WOJ1L1HlBrxPA0kU4ArbffSOyJyidu58SKyy6ZfadPSgXy3PPk2zXXuDIAxpg2oARLc07uVSQCqbd7udfUgIqtEJEdEcsrKyi7ks6oBKKl1tkge+sRc/vjZBT6+GqXUUDjnz0UR2Qz0tjfqfbZ8HLAYuAT4m4hMAIqAccaYChFZALwoIjOB3kZdjeut+jh3vum9MsY8CjwKkJ2d3Wc+5ZlSG0gSI0N9fCVKqaFyzkBijFna1zkRuQt4wXZTbRORDiDRGFMGuLq7dojIcZytl3wgw62KDKDQHucDmUC+iAQBMUClTV/SrczbQDkQKyJBtlXiXpfykdI6B/ERIYQE6cxypUYLT/+1v4hznAIRmQKEAOUikiQigTZ9As5B9TxjTBFQJyKL7RjHF4CXbF3rAdeMrNuAN22A2ggsE5E4O8i+DNhoz71l82LLuupSPnKstJ7MuHBfX4ZSagh5GkjWABNEZD+wFlhpv+CvAvaKyB6cg+FfNcZU2jJ3AY8BucBx4FWb/jiQICK5wHeB7wPYcj8FttvXA2513QN815ZJsHUoH2lqaWf36WoWT0jw9aUopYaQR1NqjDEtwOd6SX8eeL6PMjnArF7SHcAn+iizBmfQ6p6eh3NKsBoGdpyqoqW9g8UTNZAoNZpoR7bymjNVjQBM1ZV+lRpVNJAor6lsaAEgPkIXGFBqNNFAorymor6FiJBAwoJ1pV+lRhMNJMprKhuaiY/U1ohSo40GEuU1FQ0txEfog4hKjTYaSJTXVDa0kKDjI0qNOhpIlNdUNrToQLtSo5AGEuUVxhgqtEWi1KikgUR5RUVDCy1tHaREh/n6UpRSQ0wDifKKE+UNAExIivDxlSilhpoGEuUVeWX1AExIjPTxlSilhpoGEuUVeeUNhAQGkK4r/yo16mggUV5xtLiOrMQxBAb0tt+YUmok00CiPOZobefDvEpdPl6pUUoDifLY9pOVNLW2c/XUZF9filLKBzSQKI/tOVMNQHZWnG8vRCnlExpIhtgLO/N5/UCxry/Dqw4V1TEufgxRYcG+vhSllA94FEhEZJ2I7LavkyKy2+3cvSKSKyJHROR6t/QFIrLPnnvY7t2OiITa+nJFZKuIZLmVWSkix+xrpVv6eJv3mC07rB+r3nywhO/+bQ+rnt7BAy8fxLkrcd/aOww5Jyu75Gtr76CxpW2wL/W8HCqqZXqqbmal1GjlUSAxxnzKGDPPGDMP59a6LwCIyAzgdmAmsBz4o4i4Nql4BFgFTLav5Tb9DqDKGDMJWA08aOuKB+4HFuHcVvd+EXH1oTwIrDbGTAaqbB3D1qP/yiMlOpT542JZ8/4JzlQ29Zt/44FibvvTFh5/7wQAv3r1MHN/8jozfrSRjo7+g9BQaWpp50RFA9PGRvv6UpRSPuKVri3bqvgk8KxNWgGsNcY0G2NOALnAQhFJBaKNMVuM82f2U8AtbmWetMfPAdfaeq8HNhljKo0xVcAmYLk9d43Niy3rqmvYeXFXAdtOVLLysix+9NEZABwuru23THGNA4DfbDzCrtNV/Omd4zS0tANwtLRucC94gI6U1GEMTE/VQKLUaOWtMZIrgRJjzDH7dzpwxu18vk1Lt8fd07uUMca0ATVAQj91JQDVNm/3unoQkVUikiMiOWVlZef9AQeir66q9g7DT14+wMXjYll5aRaT7Z7mR4r7DwYldc5A0tzWwbfW7u5ybtuJSs8v2AsOFTmD4QwNJEqNWucMJCKyWUT29/Ja4Zbt05xtjQD09lSa6Sf9Qsr0V1fPE8Y8aozJNsZkJyUl9ZXtgn2YV8H4ezdwrKRncNiTX01VYytfvHw8EaFBRIYGkRkfzr6CGo6W1JFf1dhrnSU1DtJiwkiLCeN0ZSM3zUll233XkhIdyq7T1V7/DBficFEtESGBZOgT7UqNWkHnymCMWdrfeREJAj4OLHBLzgcy3f7OAAptekYv6e5l8m2dMUClTV/SrczbQDkQKyJBtlXiXteQe+Tt4wDsOFXV2eJw+dfRckTgykmJnWnLZozl8fdO8PrBElJjwtj03Y8QGdr1f46iGgfpceEsmZrMbzYe4dppySRHhTElJYrjdm0rXztUVMe01GgC9Il2pUYtb3RtLQUOG2Pcu6zWA7fbmVjjcQ6qbzPGFAF1IrLYjnF8AXjJrYxrRtZtwJt2HGUjsExE4uwg+zJgoz33ls2LLeuqa8i5WiJVja09zm0/Wcn0sdHEue3V8b1lU/n0wnF8bclEimsdfPPZXT0G0EtqHYyNCecLl17E95ZN4cbZqQBMTIrkeGn9OWd9DTZjDIeKdcaWUqPdOVskA3A7Xbu1MMYcEJG/AQeBNuDrxph2e/ou4AkgHHjVvgAeB54WkVycLZHbbV2VIvJTYLvN94AxxjVAcA+wVkR+BuyydQy55rZ2imqd4xndu6na2jvYebqK2xZkdEkPDwnklx+fDUDsmGB+seEwOaeqWDg+HoC9+dUUVDdx/ayxRIUFc/c1kzvLTkyOpKGlnaIaB2mxvutSKqhuos7RpjO2lBrlPA4kxpgv9pH+c+DnvaTnALN6SXcAn+ijrjXAml7S83BOCfapomoHrsZBflXXKb3bTlTS2NLOJVnxfZb/9MJx/GbjEd44VNIZSP70znGiw4JZeWlWj/yTkpxLtS/93TvsvX8ZQYG+ea70UJGzFaYztpQa3fTJdi8oqHYGj7gxwZypaqSppb3z3GPvnSAxMoTrZqT0WT4qLJhLsuL517HyzrSS2mamjo3qtcWxcHw8K+al0djSzgfHK7z4SZye3XaaS3/5Bnvzq/vNt+dMNYEBol1bSo1yGki8wNWdtXR6CnllDUz/0Wu8fqAYYwxb8yq4YVYqYcGB/dYxJyOWY6V1tLR1AFBe30xiZGiveQMDhAf/bQ5RoUG8ur/Iux8G2LCviKIaB19/ZieO1vY+820/WcmstGjGhHijh1Qp5a80kHhBQVUTAQK3LxzXmfbj9QcorWumoaWdySnn3jVwemoUre2G3FLnbKzyur4DCUBYcCDzxsWyN7/G8w/QTX5VE1FhQZypbOozULW0dbD7TDXZ/XTZKaVGBw0kHjLG8MHxCjLjxzA/M5bQIOctLaxxkHOyChjY9rMz05zjDIeKamlsaaOhpZ3EqP6XDpuRFs2BwtrO4OMNjtZ2TlU08MXLsogOC+rzwce88nqa2zqYkxHjtfdWSvknDSQeevdYOTmnqvjKlRMICBD23L+Mv3zpEgA22F/zE5MjzllPVkIEoUEBHCqqpbyuBYCkflokcPZp8qW/e4cSO2vMU0dL6ugwMHVsFNlZ8WztI5AcLXEGrykpOj6i1GingcRDL+0qIDosiE9mO5+/DAsOZKb9gv/n3iIiQgIZGx12znqCAgOYNjaKg0W1lNU3A5AY1X8gcZ8J9szW0xf6EbrYeKCYwABh8YQE5mXGklfW0GXygEtuSR0BAhOSzh0klVIjmwYSD3R0GDYdLOH6mWMJCTp7K5OiQom3Dx/OSo/BrpR/TtNTozlUVNu5WOO5WiRpseGc+OWNzMmIYefpqgv8FF39c28Rl01MIDEylMx454wx16w0d0dL6m0rqv9JBEqpkU8DiQcKqpuoa25j/riuOwOKCJfb5VBmpA38GYvpqdFUNbbyj10FjAkJHNAgvYiQHhtOYS9f9gNljOHB1w6z5DdvcbKikaXTnVOVM+LGAL0Hkn0FNfr8iFIK0EDikVy73tWk5J5f+EunO/cvn3oeYwiuoLP5UAmXT0oc8K/91JhwimocF7xkytrtZ3jk7eOcrHBOY75isjMIuhZi7P60fmmtg4LqJuaPi72g91NKjSz6AIAHjpf2HUhunptGUmQoiyckDLi+aWPPBp2rpyYPuFxabBiNLe3UNrURM6brdrcF1U28sqeQ5rYOvnnt5B5lt5+s5Icv7mdeZizXzxzLW0dKmZDoHPdIjgojKEB6PK3v6kbr3hJTSo1OGkg8cLysnviIkM7xEHciwmVuq/0ORFRYMOPix3C6spElUwe+1H1qjLPlUFjT1CWQFNc4uPxXb3b+7QokeWX1PPT6Ea6bkcJ31u0B4DMLx/HJSzK5a8nEzvyBAUJGXDgf5lXQ1t7RuRTLhn3FRIcFMStdu7aUUhpIPJJbWt+57pW3LBwfT3xEyHktxpga65wVll/VxPTUaByt7fzwxf0U1nRtSdQ6WqluaOXjj3xAdWMrG/YVA3Dl5ERumd/7nmBfWzKJ/3p+L5sPlbB8VipVDS28tr+YzywapwPtSilAx0g8kltaP6BnRM7HL26dzbNfWXxeZaamRBETHswzW08B8N6xcv6+I5/3cyuICAkk3Qal3NJ6XtxdQE1TK3+789LO8r/95Nwus87cLZ89FoDTlY3UOlr52l930tLewe0LM3vNr5QafbRFcgGMMfzq1cNUNbYy0cstkr6+0PsTERrEly8fz+rNRymtdfD6QWdL4xe3zuba6ck4Wtv5yG/eZtVTOZTXtzApOZKF4+OZlxnLvoIakqP6fs4lKjSIMSGBFNc08/SWU2zJq2DBRXG6dLxSqpMGkguwJa+CP7+bBzj3BhkOrpmWzOrNR3n/eDlvHCrl5rlpfGaRc+2vjg5DTHgw5fXOJ+aXTHGOv6y7czHNdpHIvogIY6PDKKl1UGr3kH98ZfYgfhKllL/Rrq3z9PqBYn6/+RgAczJiuDhzeMxcmpEWTVRoEA9tPEpFQwvLZp5dtj4gQFhml7H/75um853rpgAQGhRIdFhwr/W5S44OpaTWQW5pPddMSyZ2TP9rgCmlRhePAomIrBOR3fZ1UkR22/QsEWlyO/cntzILRGSfiOSKyMN2y13strzrbPpWEclyK7NSRI7Z10q39PE27zFbdlC/4Rpb2lj19A62najki5dlsf7uK3pMt/WVwADh4xenU1DdxJiQQD4ypeusrx/fPJP/8+n53HHFeCJCz68hOjY6jJ2nqzhcXDeghySVUqOLR11bxphPuY5F5LeA+5rmx40x83op9giwCvgQ2AAsx7nd7h1AlTFmkojcDjwIfEpE4oH7gWzAADtEZL0xpsrmWW2MWWuD1R22/kFR03R2P/ab5qQO1ttcsB/cNJ2xMeEsm5lCVLeWRkRoEB+bm3ZB9SZFheLaTn52uq72q5TqyitjJLZV8UngmnPkSwWijTFb7N9PAbfgDCQrgB/brM8Bf7D1Xg9scu3TLiKbgOUista+32dsmSdt+UEPJJ9emNnv1rm+EhoU2OU5EG+5bUEmHQZWzEvTQKKU6sFbYyRXAiXGmGNuaeNFZJeIvCMiV9q0dCDfLU++TXOdOwNgjGnD2bpJcE/vViYBqLZ5u9fVg4isEpEcEckpKyu7kM9IbZPzrW6afWG/7P3V1LFR/PCjM5iTETvgBSiVUqPHOVskIrIZGNvLqfuMMS/Z408Dz7qdKwLGGWMqRGQB8KKIzAR6+xZyLRDV17nzTe+VMeZR4FGA7OzsC1qUqta2SKLDdbKbUkq5nPMb0RiztL/zIhIEfBxY4FamGWi2xztE5DgwBWerIcOteAZQaI/zgUwg39YZA1Ta9CXdyrwNlAOxIhJkWyXudQ0KV9dWTPjwGGBXSqnhwBtdW0uBw8aYzi4rEUkSkUB7PAGYDOQZY4qAOhFZbMc/vgC4WjXrAdeMrNuAN41zOduNwDIRiROROGAZsNGee8vmxZZ11TUoah22RTKAKbNKKTVaeKOP5na6dmsBXAU8ICJtQDvwVddgOXAX8AQQjnOQ/VWb/jjwtIjk4myJ3A5gjKkUkZ8C222+B9zqugdYKyI/A3bZOgaNa4wkKky7tpRSysXjb0RjzBd7SXseeL6P/DnArF7SHcAn+iizBljTS3oesPD8rvjC1TS1Ehka1LkKrlJKKX2y/bzUOlqJ1taIUkp1oYHkPNQ2tRKtA+1KKdWF/rw+D3MzY4fNIo1KKTVcaCA5D1+/epKvL0EppYYd7dpSSinlEQ0kSimlPKKBRCmllEc0kCillPKIBhKllFIe0UCilFLKIxpIlFJKeUQDiVJKKY+IczX20UVEyoBTvr4OL0jEuS+L0nvRnd6Ps/RedOXJ/bjIGJPUPXFUBpKRQkRyjDHZvr6O4UDvRVd6P87Se9HVYNwP7dpSSinlEQ0kSimlPKKBxL896usLGEb0XnSl9+MsvRddef1+6BiJUkopj2iLRCmllEc0kCillPKIBpJhSkTWiEipiOx3S4sXkU0icsz+N87t3L0ikisiR0Tket9c9eAQkUwReUtEDonIARH5lk0frfcjTES2icgeez9+YtNH5f0AEJFAEdklIq/Yv0fzvTgpIvtEZLeI5Ni0wb0fxhh9DcMXcBVwMbDfLe3XwPft8feBB+3xDGAPEAqMB44Dgb7+DF68F6nAxfY4CjhqP/NovR8CRNrjYGArsHi03g/7Gb8LPAO8Yv8ezffiJJDYLW1Q74e2SIYpY8y7QGW35BXAk/b4SeAWt/S1xphmY8wJIBdYOBTXORSMMUXGmJ32uA44BKQzeu+HMcbU2z+D7cswSu+HiGQANwGPuSWPynvRj0G9HxpI/EuKMaYInF+uQLJNTwfOuOXLt2kjjohkAfNx/goftffDduXsBkqBTcaY0Xw/fg/8F9DhljZa7wU4f1S8LiI7RGSVTRvU+xHkwcWq4UN6SRtx87pFJBJ4Hvi2MaZWpLeP7czaS9qIuh/GmHZgnojEAv8QkVn9ZB+x90NEPgqUGmN2iMiSgRTpJW1E3As3lxtjCkUkGdgkIof7yeuV+6EtEv9SIiKpAPa/pTY9H8h0y5cBFA7xtQ0qEQnGGUT+aox5wSaP2vvhYoypBt4GljM678flwM0ichJYC1wjIv+P0XkvADDGFNr/lgL/wNlVNaj3QwOJf1kPrLTHK4GX3NJvF5FQERkPTAa2+eD6BoU4mx6PA4eMMb9zOzVa70eSbYkgIuHAUuAwo/B+GGPuNcZkGGOygNuBN40xn2MU3gsAEYkQkSjXMbAM2M9g3w9fzzDQV58zL54FioBWnL8a7gASgDeAY/a/8W7578M54+IIcIOvr9/L9+IKnM3tvcBu+7pxFN+POcAuez/2Az+y6aPyfrh9xiWcnbU1Ku8FMAHnLKw9wAHgvqG4H7pEilJKKY9o15ZSSimPaCBRSinlEQ0kSimlPKKBRCmllEc0kCillPKIBhKllFIe0UCilFLKI/8fpOKnsVDs6WwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd \n",
    "\n",
    "pd.Series(log['mean_reward']).rolling(50).mean().plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broadband-shift",
   "metadata": {},
   "source": [
    "Some things you may want to consider when writing an agent:\n",
    "\n",
    "- The action space is combinatorial\n",
    "- Illegal actions are corrected by the environment\n",
    "- Observations and rewards are unnormalised by default"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}