Policy_Iteration_FrozenLake.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 5,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python3.8",
      "language": "python",
      "name": "python3.8"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.9"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "auburn-pittsburgh",
        "tags": []
      },
      "source": [
        "# Lab 1 - FrozenLake MDP: Part 1\n",
        "# Assignment\n",
        "\n",
        "- In this assignment you will learn how to tackle problems with limited state spaces.\n",
        "- In particular we consider the FrozenLake MDP problem.\n",
        "\n",
        "# Outline\n",
        "\n",
        "- Part 0 introduces us to [gym](https://gymnasium.farama.org/), an environment that allows us to test our reinforcement learning algorithms in various problems\n",
        "- In Part 1, you will implement a policy iteration algorithm (HW1)\n",
        "- In Part 2, you will implement Q-Learning and SARSA (in next homework HW2) \n",
        "\n",
        "# Deliverable\n",
        "\n",
        "Regarding the Lab:\n",
        "\n",
        "- Make sure your code runs from top to bottom without any errors.\n",
        "- Your submitted Notebook must contain saved outputs."
      ],
      "id": "auburn-pittsburgh"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "flush-grove",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d2462fea-0442-41b0-cfed-28756b25ada6"
      },
      "source": [
        "import os\n",
        "# You will need numpy and gym. You can try running the following lines to install them\n",
        "# The assignment is tested on Python3.8 so in case you are having installation issues you might \n",
        "# want to try installing that version. \n",
        "\n",
        "# !{os.sys.executable} -m pip install numpy\n",
        "!{os.sys.executable} -m pip install gymnasium\n",
        "# !{os.sys.executable} -m pip install Pillow\n",
        "# !{os.sys.executable} -m pip install ipython\n",
        "!{os.sys.executable} -m pip install pygame\n",
        "import gymnasium as gym\n",
        "import numpy as np\n",
        "from PIL import Image\n",
        "from IPython.display import display"
      ],
      "id": "flush-grove",
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: gymnasium in /usr/local/lib/python3.8/dist-packages (0.27.1)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (2.2.1)\n",
            "Requirement already satisfied: jax-jumpy>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (0.2.0)\n",
            "Requirement already satisfied: importlib-metadata>=4.8.0 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (6.0.0)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (4.4.0)\n",
            "Requirement already satisfied: gymnasium-notices>=0.0.1 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (0.0.1)\n",
            "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.8/dist-packages (from gymnasium) (1.21.6)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.8.0->gymnasium) (3.12.0)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: pygame in /usr/local/lib/python3.8/dist-packages (2.1.2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "medium-dancing"
      },
      "source": [
        "# Part 0 - Introduction to Gym\n",
        "- We look at [FrozenLake-v1 environment](https://gymnasium.farama.org/environments/toy_text/frozen_lake/) in gym. \n",
        "- You don't need to write any code for this part\n",
        "- you should still understand the code to help you solve Part 1 and Part 2"
      ],
      "id": "medium-dancing"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "considerable-swing",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2155cc89-c89a-4948-ac54-d31c156c48c0"
      },
      "source": [
        "# Import the environment we will use in this assignment\n",
        "env=gym.make('FrozenLake-v1',is_slippery=True, render_mode=\"rgb_array\") \n",
        "\n",
        "# Show the model\n",
        "print(f\"Number of States {env.observation_space.n}, Number of Actions {env.action_space.n}\")\n",
        "print(f\"Reward range {env.reward_range}\")"
      ],
      "id": "considerable-swing",
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of States 16, Number of Actions 4\n",
            "Reward range (0, 1)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "severe-conjunction",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "66a4eb71-3d8a-4a92-cce0-5ec5fcfd9f71"
      },
      "source": [
        "env.reset() # reset the environment "
      ],
      "id": "severe-conjunction",
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0, {'prob': 1})"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "endless-cassette",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 273
        },
        "outputId": "3cb7a7e9-c0e2-4839-8aa1-6dfa05ddc853"
      },
      "source": [
        "# visualize the current state\n",
        "display(Image.fromarray(env.render()))"
      ],
      "id": "endless-cassette",
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<PIL.Image.Image image mode=RGB size=256x256 at 0x7F3820121AC0>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAIAAADTED8xAAAeXklEQVR4nO2df4xd1XHHv0bPD7Nm8drYGFeVwdRKKMVuiymoBtEFFP9BhJAMUdUWlSpt3UaRatIIGlUCoSJFTfIHOFVp66BKliBV1YBEMFaBVFmB6xpau6kNrhOtYoEq7wYbsDFdvOut6R/z7ntz79yZO/e9t7985vOHtXvuOefOXvnMzJkz55xFu/cfRxCkSgPAlWvX2JWuWjbV/nnlQLPvQpycmCqUvHPa9Zbxd8cQ8vdMyvI3AFy1bMpukH86Bd+fwcWy6/On1Ip/sso/JuQvJeQve3uRiyrlCIILmEbdBtl4Kh+dNP6oRI5aaao08m3VVuPO7hghv4d05A8LECRNywLU8vnadaiVHJcHxop9EjQu+SgnZA+a/6fpgJA/5O9O/rAAQdI0wIaInEET2pjm5byVDHvZM3rb25N+YWEch/whf9fyhwUIkiYXBSod3ygb03Z9Xi7HIq+jYY9pz3tD/pDfI39YgCBpStYBtLFlz7XtcjnWtToaHs/PLg/5Q35ZEhYgSJoG2GCqO4Zs7r1xuP3zc/8xAt3nI/weXqH+DMmvSRXyF+ovaPnDAgRJ01oHoNEgV+C0KKxcjSOo/I9u2wJg+dc65WQN/u61VwqtqH5ZnLh6NLfL+y4/L9cyVUL+C0P+sABB0rSiQNk6Gf1WHoXlaKNK6n6CSugpzQfy/XTeQqO5bgZ5v+TvNhIS8i9U+cMCBEnTsgB+30uLv5KX/7O7hgAsx6nSl9FTbT6gITPIJb3LXxfNi5WE/PNZ/rAAQdIs2r3/eHtT8yZzc7McuzS2vn73MDLt7mf1nlPI5gNaz5UUNmV3Jz9FLbrTQ3bPlYT8mGv5wwIESZPLBeJxXG3Grc3r//T7EwB+/YqPK1/5b+9dCuBZ9mbZM8ngHMdtupM/ixsAVTpMI+QnFqL8YQGCpCnJBvWPSPL+f2e6G++NWlFEiGcKETSau9NDvWgU0mGExzflsYiQn1hY8ocFCJJm0e79x2/ecDn9oo0VORZ5pifp8ju/Pg3g2Mh7AH7l7odL+/nRi98EsG74CgD/8ucNAM82On6blinEKXh7bxx+H0AX8vOe62o4bX8Tl1C+hT8N+eeP/GEBgqTJzQHssSXH06NfugfAL3/mcmTafeTo9QBGju4B8OBDd7VrPvmtPQCA6wEAbwHY+lcPA/iFn7wP4C/+5gXkV+y4V2fvA+pFft7Ko4e0mvnykL+c+Sl/WIAgaXLZoBJt1JK/TtmdxKNf+n0AD959OYB7v/L3AB5k9V8/Pg7guSe+COC/fnJzuw7vjecScjxrhHXll/hPs9FaZRGPkL/D/Jc/LECQNBWnQ9ujhzQ3QdaAZgWk6TmZ7u94/LxtL3kgNp4osobm0WpnVs4EIb+nvBf5wwIESdOyAHwFzr96J0feD194A8APu2pbd/cQp1/ye3zWuv5uyO+XZPblDwsQJE3J/QAHxpqoM45pVZh2/b6FcQCP75xClvVJUJboI9uaAJazVmX7gzvIfaXaaO5Ffk4v3mTITyws+cMCBEmTiwLxyCt5ddo4ljsyn179RQC3vvo8gEe2nQLAd2HuQRPA0UNDAPZ+biuAe9FZB5A9a9ijuRf5PXhahfzEQpE/LECQNLmT4STaOJajkLJ97sMSAL+5fwrAwINL2k8nnjwL4B+XLAHw5I/2tMu7281ZuFuqL/J78EQtQn7O/Jc/LECQNIt27z/++ZvXePKz7Vw/Wgnmu8N4rr8s5/EfQtMihjyUj94X+T3w7EJ/DDvkb9evL3KOmZA/LECQNLlsUHsGrY2kkVdfArD5Tx4C8Oy3v9Uup5IVl0wDePYbT7TLf+uRhwB874WXANx3z+fb5dpeHk++eC/yc+R9t/7Mk5B/IcofFiBImtYcgBfR6Km7U/PQ3hfaP3/wSQOZ7h9sliecct3vodTLpJPJ+iK/pnv6lSkZ8tvMlfxhAYKkMW6JrJ7Xczbeeg90j016h3VPCeb9tN8yrtesKz/XOvKv6CVLkT8N+TXmSv6wAEHSqDvC5IjxzPR5W4mtCegsMXnDlNazLUnIXyZhyF+UPyxAkDSte4Lt2KrMq5ZPeYnWj8cv9N/u1C4P+UP+ruUPCxAkTW4OoGVZeObXEtlb6Uwc1aO5WkNob/S0CvlTlj8sQJA06i2Rds6dZ33On7NhZ2tIr04S8ttyauUhf1iAIGlK5gCEZ/z5W/ULPprt9cKQfya48OQPCxAkjboS7Fmxs/F4e3Je788itGuG/CG/DdVctHv/8R4FDYKFS7hAQdI0wK6617ATjHqnu8MtkG3ICPl7JGX5GwCuWjZVlWfnza/QxPKvBfrX+dp0Lb8dvZ7/8tvMf/nnw/cPFyhImoobYiR2fgUfwXbE1ybfVm1VuiOJsGMC2Xn2TWR55Bm8pNM2X6f4Fknv8mvE95dvkfjlDwsQJE3J/QB1d3BKvULj294fLPUWR/P/NB0m5fdlhvSKdmdt7/J73hvfv/fvHxYgSJrW6dD0i5xBE57RzFvJsJc9o7e9VenXFsaxIb9H32zdMVJZR/L89uHS8rI3Vnje8f0r60j69f3DAgRJk4sClY5vVM217b2eUpfwOhqenEHtvbK+RGqdgRs2G/Ub09MAJiYAYHr0zdIeCKmZ7AxKTnx/jZn7/mEBgqQxToYrYscK7HKpM2z94ZHKri99wbpap1VniPppIotLTw3cVKizeHoKwOmpZuEtmjayT72M75+rM8PfPyxAkDStc4Hol+5279t4fEetprPngvz81nKCawWP1uFMnCpKO7ByQNQaADAEINNP5K3a2kiTXyO+f6uHvn7/sABB0uRuiZQriFoU2b/n3z4pgErs2G2lr1yQn+PXPRs3nwcwNnbOqJMxCeDEsYvLHzYaAIbWNpFpo48OdaIWXA9p8sf3r6Kf3z8sQJA0uTvCMueqPIrMqRvB0J5mJXYmiRUz4f3wnEFb96xaN1koGWPaa82axQC2rFwC4JWTZ1GmmWQP1OrQvovQ9lwbDQCXbSzXQ9yVje8/V98/LECQNOrJcJ59OrKmH8+ef0JmwEu49+nXPaQzNDTdI1tRHfp31bpOHdJG041GW5KJg/varbR8GyK+/+x8/7AAQdI00Bq+3Actoukerhvq6iGtvraTqG4eOdc9MsLwwIbBQn1N32g1yUPddfhM4Sn3X7H5LDI9JGn/RfH9MXffPyxAkDS5XCAeh9YiBpoeogjApooDMsqRPdc95YuQ2SaZx9nxGqVGIbjuyWkRBal7eM/8KXmlatyaEd+fmM3vHxYgSJqSbNBeNAqPBnhu+tZOMejutEeCvE8ebVhxZgmALRuKGoXrCR5bkLrHE5HQ/NEsOjEJ4J2D1fLH95/N7x8WIEia1slw9Isc69Ir5f6i52YOeaKL51x5+Rb+tF1/vNPc0lI3riuWSN2jeZxUk+rwmlRO2oXrIf4z11j/+2HR971q2dR49gOVxPeXzPT3DwsQJE1uDmDvWLXPhPF4ilrNfHm5trNl4MgsEcLvcWo15dNcD0ruCvVw5AQAPPrYMAB8WC55fH+7pnya66Gr7x8WIEiaXDaoxH+apP80G61VFvHQMha7p67HKWtKveKJZ1Odp14DgGYDAO6cBIBLRXwmvr9dc+a+f1iAIGkqTof2RJE1NI9WO7OyX0jNwfF7nHa2idaDLJ+aHgTw4i0AcPUyADg5AZTtn5LE95/p7x8WIEialgXg2si/+mifH+ZvZbf166eVdwwDAM5C1ys8rkxIb5Lq7HJkp2i6h6DYM3mf7090/iU/e9MavPRuq2Z8f85sfv+wAEHSlNwPcGCsNT66oBdvUtNndc+rGX17CYClK84h0y7cE/VHoAkt2iC9W8n6684CGD1SfIv8tvH9idn//mEBgqTJRYF45Ji8Uk0P+XeU1m1le7GaNiINunNDE8C2w5360uPkcC0i1w61mho891BCUu3c0Pkr5HeI70/M5vcPCxAkTe5kOImmh7rzNT1RC//6JT+SUvYj9YEWmdZ0hlZftpIxDd5K80Rt+Yn4/jP9/cMCBElTkg1qZ6X3gjzXUtuFxNFOKNDkuWQxUHXKpLa3iMclPLpKi1HYaPLH95/97x8WIEiaXDaoHQGom12o6QnPuZZ1TyjgZ1PuuBYAth0eROb/kebYpegk7kF6tI62KumPaWjyx/ef/e8fFiBImpJbImlPUN2zADTd0694hXGC8cmJKVmfPFGa+y9dXq17JPZJNXaeo6btOPILx/fnzM73DwsQJI1xS2RxxNtwrWOfO6DhOYG+EKkYzzekeDnFTMgT3X4UyLICly4v5ploUQV7B1OF7hFrn6QFaQ2SKMhf+pfG9ydm+vuHBQiSRt0RJke8J1LB20psTUYxBHkKjdazdkuhXw9JfVNX98j4htQ9Z9gZBHzfrf0l4/vPzvcPCxAkTeueYDs2rN1XJXVSVYy52q/1306lZbZwPUTsAJDpofMfDwL45Bzg22/KT6MntDVOqXsoErLzNhTkKWRlxvdvM/vfPyxAkDS5OYCWL+6JD0i0e69kzkmVNqq3I4nffUtv2bSmiUwPEdwrJcg31bSLJ64s8w3JA85yJys7iO8/B98/LECQNOotkf67yDX8OSeeU44zqrNlZCyc3kJ6iH7ee3sTWaSC9grlTg/+VHS9yHhtCx5pJrJ4SPHvKuTTI77/3H3/sABB0pTMAYi6c3+7Vb/g2kjLp9eQupM0xMtm5NuG/+1aHr9Ey9OM7z/73z8sQJA06kqw/1xiu4e6GSb+LMi6+ZJcKn/NuZI/vv/syL9o9/7jzgZBcOERLlCQNA0AV66tOIbPTpDqne4O5wAw/u4YQv6eSVn+1i2RVXmC3vwQTSz/WqZ/nbJNyF/K/JffXv2YHfnDBQqSpuKGGImdH6KdOcOfesi3VVuV7qiyCfk99C6/HZPJIvRNFG8y5iWdtvK2Y/4WiV/+sABB0pTcD1B3B6rUK9pJZrbnx9H8P02HhfzzTX5fZlGvaHce++UPCxAkTet0aPrFfzqkLOetZNjLntHb3qr0awvjOOSft/J79P3WHSOVdSTPbx8uLS97ozXzCQsQJE3JyXDST7Ln2vZeValLeB0NT86jZ49syD/78sv6Eqn1B27YbNRvTE8DmJgAgOnRN0t7IKRlsDNYwwIESWOcDFfEjhXY5VJn2PrDI5U/Y157Y8jvea/21K4vffG6Wr9VZ4j6aSJbF5gauKlQZ/H0FIDTU83CWzRrwM9ZCgsQJE3rXCD6pbvTB2w8vqNW09lzyO95C2ZLfrkPi2tlj9bnTJwqSjuwckDUGgAwBCCzDzRbsK1B7AkOgvwtkXIFUYsia7t1ZLmWqcJL7Nhtpa8c8s8r+Tl+3b9x83lU3SyWMQngxLGLyx82GgCG1jaRWYOPDnWiRtwOxBwgCPJ3hGXOVXkUmVM3gqE9zUrsTJJybSf7CfnnVn6es2nr/lXrJgslY8x68HOh6WxQaRlkD9Tq0L6L0J45NBoALttYbgfoa4cFCJJGPRlO8x3rxo81NC9WIjPgPb2F/Db9lZ97/37db98Rpul+2Yrq0L+r1nXqkDWYbjTakkwc3NduFVGgIEADreHLfdAimu7huqGuHtLqazuJjLzCkN+QR6OP8pfCdb+M8Mgb3jV9r9WkGYJ2w0DrPoHNZ5HZAUncEhkE+VwgHofWIgaaHqIIwKaKAzLKkT3XPeWLCPnnVn6Z7ZN5/B2vXWp0wr4VWCJ1P++ZP6VZgbZuEBYgSJqSbNBeNAqPBnhuKpe7jXh5d3oo5CfmSn7y/nm0Z8WZJQC2bChqdHknJCF1vycipM0HsujQJIB3DhalDQsQJE3rZDj6RTthnesS7i96bhaRJ7p4zsWXb+FP2/XH8x2G/HMi/8mJKX6Gj+TGdcUS+z5gWZPq8JpUTtqd2wH+M7cYuRtoAPq74lygIHVycwDPDR8Sv6eo1cyXl2s7Wwbej0bIP9PyEzJLh/B7/FpN+TTXg5I7RD0cOQEAjz42DADs7viwAEHS5LJBJZ4RT/hPs9FaZREPLWPRIuSfW/lt6nr8sqbU6571BKrz1GsA0GwAwJ2TAHBp7AkOAqLidGhPFFlD82i1MytngpDfU95f+aXm5vg9fjvbR+tBlk9NDwJ48RYAuHoZANCt8RRhCwsQJE3LAvAVRP/qo7YntW4ru61HP4X8tgweeXqRv1XzjmEAwFnoep3H9QnpzVOdXY7sIE33ExT7J+///YnOvzTP2bQGL70bFiBIm5L7AQ6MtcZHF/TiTWr6zHNeTchv999dW7/8nNG3lwBYuuIcMu3OZwL+FQBCi/bI2YVk/XVnAYweKb6Ff9uwAEHS5KJAPHJMXqmmh/w7Suu2sr1YWxuF/DYzJz9ZsJ0bmgC2He7Ulx4/h2txuXar1dTguZ8Skmrnhs5fEesAQZA/GU6i6aHufE1P1MK/fsmPpAz5Pcym/FIfaysDms7W6stWMqbEW2kzgTgVIgjKskHtrPRe4NmF9i4kjnZCgawZ8s+t/MQli4GqUz61vV08LuSxFVqMyIbLHxYgSJpcNqgdAaibXajpCc1/ldqI92m/PeTn7+L1Z1N+2n2241oA2HZ4EJn/TZp7l2ITuAfv0fraqrA/psQJCxAkTcktkbQnqO5ZAJru6Ve8ouoEY94q5J9V+U9OTMn6NBOg2MvS5dW6X2KfFGTnmWrWhhMnwwWBdUtkccTbcK1jnzugYeuq0nMKxq1+Qv4OMyp/uyGtV1DMimYC248CWVbm0uXFPB8tqmPvIKvQ/WLtmawQrQETXP6wAEHSqDvC5Ij3RCp4W4mtySiGIE+h0Xq2JQn5yyScEfn5kjDq2AGp7+vqfhlfkrr/DDsDgu97jpXgIMjuCbZjw9p9VVInVcWYq/1a/+1U7fKQf27lb8PXmHn+0g4AmR04//EggE/OAb79vvw2AEJbY5a6nyJRO29DQZ7IBg2CFrk5gJYv7okPSGRvpZEEVGujag2nvdHTKuTvl/z8vTwqtWlNE5kdIPisgKC5gabdPXF9me9JM5Asd7W8VViAIGnUWyI1beTfi+TPOfGccpxRHgMJ+W05tfJ+yc9ryrUIegvZAfp57+1NZJEi2quVO735U9H1IuO1LXikn8jiUcW/i+9nCAsQJE3JHICoO/e3W/ULro0859xLQv5esOX3nG7NJSQN/bK58mDD/3ZtH4Uk9gMEQQt1Jdh/LrHdQ90ME38WpF0z5J9b+W2p/DVnWv5Fu/cfdzYIgguPcIGCpGkAuHJtxTF8doJU73R3OAeA8XfHEPL3TMryt26JrMoTLM8PsaPX/uiEFpl2/jFdy28T8pe9vYQFLX+4QEHSqFEge06dRVibKN5Ey0s6beVttfwtkvyIV+MGpTuqbOz8Fu3MHP7UQ8ivMd/kDwsQJE3J/QDamQX8ab/Q7qzV/D9NB9TyWdt15LsI7SQ2e+bDCfkXivxhAYKkaZ0OTb90EQEAsHXHSBcvfn77cGl52RsrPD9Dfr3PYjlvJcN2dkTC9lalX1vQQyH/HMofFiBImtKT4YCqGbfU+gM3bLZeMz0NYGICAKZH3yztgZCWwc5A5JTqJ6OVVl/bidtdZFpr69njG/LPtPxhAYKkUdcBpN9WV+u36gxRP01k6wJTAzcV6iyengJweqpZeItmDexTL+09snaJVi51lVZHw7/TN+SfTfnDAgRJ0zoXiH6hMSH30XCt7NH6nIlTxZE6sHJA1BoAMAQgsw80W7CtQVvsgvwa3aVheXxfraaz52Tl/97WXy2U3Pf8f1ZKxUtG7r+5/fPwM28Y75LEHCAI8rdE9qL7N24+j6qboTImAZw4drEiUQPA0NomMmvw0aFO1IjbgfYoL8jviSJru41kuZapwkvstYtKXzkp+e+56w8BfGXFIQCf+4PlhT7JJjzxwcbSNxLUluA9vHr/ze22L+z5jlP+sABB0uTuCOM5m7buX7VuslAyxqwHP9eXznaUlkH2QK0O7bsI7ZlDowHgso3ldoBPJbJ1PvqtPIrMqRvB0J5mJcW187oZ8CnIb+t+olX+9CEAz9z7l4Wn9z/3NWdbete3/2FXpfxhAYKkaVkA7v37db99x5Om+2UrqkP/rlrXqUPWYLrRaEsycXBfu5WW70Fovm/d+LGG5oVLZAa8p7cLT37JsZH3AIyOdv4/kBanf9ePfLXwdP364v+lV5/+UDwt/u+y5Q8LECSNuhLMdb+M8MgbujV9r9WkGYJ2QnzrPPjNZ5HZAUnbTz0wBvI7SxYYAOi6k+uGunpUq6/tpDPyahORn0O6f93wFQCAjh0gjU66nEpycZ6Wvu/Ulzbh5JkjAIBiHEmTPyxAkDQ5CyCzfTKPv+NXSY1O2Le6SqTu5z3zpzQrUNcNGDwOrUU8ND1KEbBNFQd8lCN7rntKGXFhyy8hO0BwXc79fl6H/h/ypxwq/+ff+z4AfOe7TpnDAgRJUzIHIO+fR3tWnFkCYMuGokaXd/oRUvd7IkLafCCLDk0CeOeg9ccQvWhEHg3zaDLtFAO+N6quHr2w5afoPkX0f/reqKO/9e2fZP2fvgcA11yxHmXrBh75wwIESdPOBrVGyY3riiX2fa6yJtXhNamctDu3A/xnbjFyN4gAAK5aNjWe/UAl2gnxXBdyf9dzM4o80chzrr98C3/arp+U/Byurf9v7wEAD/zsnwp1slkB9xrWF54+cua3Aey7dVPxj2TY8ocFCJKmZA4gs3QIv8ev1ZRPcz0ouUPUw5ETAPDoY8MA8CFK8dxQorXyeLpazXx5uba2ZeD9aCx0+TWeufsHAIDlyK8NZ6sEHUZHO6sE9PQZUNsfAHjgwz8r1PfIHxYgSBp1JZhT1+OXNaVe96wnUJ2nXgOAZgMA7pwEgEtFfMO/D1XDf5qN1iqLeGgZlxZpyr9r+TcKJaT7d63+AoDPtbQ7q7/6CwAeGKXZAl9L7vSm2QGNsABB0pRYAKm5OX6P38720XqQ5VPTgwBevAUArl4GAHTrt3YHIMcTBdfQPFrtzMqZ4MKTn2I+jw9+F8AxVs51v02rzhkAeHykuOL7GL4K4Muw4kKcsABB0uRPhrtjGABwFrpe53F9QnrzVGeXIztI0/0Exf7J+39/ovMv+amb1uCld1s1uTXwr55qe2rrtrLbevRravJT/J4ga0B7eVes7tTJZwFZ/VAPvE/gx/DteQgLECRNyRxg9O0lAJauOIdMu/OZgH8FgNCiPXJ2IVl/3VkAo0eKb5E6hvu1B8aapXU89OINa/rYc95OCvL/68guANdfvwXAspWdx9xf/+C/fwzg9MkxAK8MvVPo4dipkULbfA+dtm+99YpT/rAAQdLkbojZuaEJYNvhzmPp8XO4Fpdrt1pNDZ77KSGpdm4AdN+OR77JK9X0qH9HbN1Wthdua6N05F/xi5813rICnwWwZfB4ofz1M8NGK+L062PQ/wopf1iAIGlyJ8NxpD7WVgY0na3Vl61kTIm30mYC/EhKbcRreqg7X9kTdfGvv6Ys/0xj/xVc/rAAQdKURIEuWQxUnfKp7e3icSGPrdBiRDb8lgCOlu2oZaXXRZ7Lqe2iktKG/Lw+Re41Wrt+xX8N2jMgT4Lg3I+fK7zLlj8sQJA0JWeD7rgWALYdHkTmf5Pm3qXYBO7Be7S+tirsjylJtBMTOHWzIzU95zmXs+4JCynIv+ff/8d4OnPY8ocFCJKmtSdYagWaCVDsZenyat0vsU8KsvNMNWvDkWOan3Vc9ywDTXf2K95SdQIzb3Xhy5/P2xGsBoDHUZwn8DxQnRFk1uZ3f+MaVl4uf1iAIGlyN8VTvJbm7DQT2H4UyLIyly4vDj0tqmPvIKvQ/WLtmawQrQETbW03XvYnaWfe23CtaZ+boOE5Qb8QaQn5Zw1N/rAAQdKU3BLptwNS39fV/TK+JHX/GXYGBN+3ams1OeI9kQreVmJrMoqhyVNotJ5Tk/9tAMDDf/23AL755T822hIU0c8zUtmK+relImIlOAjyK8F8jZDnb+wAkNmB8x8PAvjkHODb78tvAyC0NWap+ykStfM2FOQpZDWWRrEIroe0u1U8+e5+v9Z/u1Zq8t/1az+PLD5j6+ne+aXPXAPx/6RUtrAAQdKU5ALxu2PJe9u0ponMDhB8VkDQ3EDT7p64vsz3pBlIlrtX2YE64j3xDU9v2omZVdq0WkNrb/S0Wljy52PzFlpv/gxWj/xhAYKkyVkALZZMY47sAP289/YmskgR7dXKnd78qXjPompReKSfyOJRxVFbyEdHmbaw9Ucvp51pGZR+r1oS8ttyauW9yx8WIEianAXwnE7Mxx9p6JfNyLEN1+JaHrlEy3OsEw+p16pf8L8x5Pe36hdS/rAAQdK4Tofm+M8r9nh7Mi7hzxuxa/rltHsI+TUuDPkX7d5/vEdBg2Dh8v/LTEi46XMPqgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "naughty-bread",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 309
        },
        "outputId": "e106d38c-5e0f-45a8-aed2-8255dd7cb73f"
      },
      "source": [
        "# run a policy that chooses actions randomly \n",
        "env.reset()\n",
        "n = 25\n",
        "for i in range(n):\n",
        "    a = env.action_space.sample() # Sample Random Action\n",
        "    state, reward, finished, _, _ = env.step(a)\n",
        "    if finished: break\n",
        "        \n",
        "print(f'Render State after {n} slots')\n",
        "display(Image.fromarray(env.render()))\n",
        "print(f'Reached terminal state? {finished}')"
      ],
      "id": "naughty-bread",
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Render State after 25 slots\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<PIL.Image.Image image mode=RGB size=256x256 at 0x7F382012C670>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAIAAADTED8xAAAfA0lEQVR4nO2df4xd1XHHv0bPD7Nm8Y+agKPWxNRqKI0tlaUQmUIXEP4jVRoVkkqtUN1KqVtBJFSlv/4JskCq0h9S5YqgxkGqVkKJ1BaqBogKSaMVppahXVeyg0ukbZDdyrsFEwMLi73eQv+Yd9+be+fM3Lnv7dr79sznD7R77jnnzh58Z86ZM2fOmmePnEYQ5EoLwLXbttqVrtuw0P15y0h7yYU4M79QKTn5justs6dmEPIPTM7ytwBct2HBblB+ugDfn8HFsuvzp9SKD1ntHxPyJwn5U2+vclmtHEGwimk1bVB8T+mvk74/KpFfrTRVGuW2aqtZZ3eMkN9DPvKHBQiypmMBGs35unWolfwup2aqfRL0XfKvnJA9aPM/TQeE/CF/f/KHBQiypgX2icgVNKF907yct5JuL3tFb8/25Lyw8h2H/CF/3/KHBQiypuQFSn7fSH3Tdn1eLr9FXkfD/qY97w35Q36P/GEBgqxJ7ANo35a91rbL5beu1dHwzPzs8pA/5JclYQGCrGmBfUxNvyEPnrmjVtPZc8jveQtC/lT9sABB1nT2AehrkDtwmhdW7sYRslyL9OAlKT9x/dfcLQ/5Q/6+5Q8LEGRNxwtU7JPRb2kvLKepB0B7WpT03kJfc9MI8pDfLteehvxhAYKs6VgA/9yrqf9VQ5sFSmQEuae3kN8m5CfCAgRZ00Jn8c7ncFW0b5d/W02/Y60++RDkut6IMg/5DXk0Qn4iLECQNWuePXK6ktaCviFPRIf8qv70s+O1r3zq3ydre9b6r5BMyzGI/GM1CT4sQn5iuOQPCxBkTSIatPCkAk2+SNL9t3/8WgC7fmp9ss6x/34fwH03j6OwA3wtL/0G2nzOpj/5CdrLJGxNxuUM+TnDJX9YgCBr1jx75PStO3+CftG+FfktypkcWYAHb/3p2ld+7eX/AvD1F19IPtXewp92y18+/haAJZHfj3a+iUso38KfhvwrR/6wAEHWlNYA9relxXXYrTRkRgDpFZb+Y00GjyS2/B49pNUM+Xk/GitT/rAAQdaUokEl9hc/OFxnFB4DLeLPYnD5/dlstFYhv2Tlyx8WIMiamuzQHi8shzz99j4AsdyXJnh67m9Gq+WsXA5Cfk/5IPKHBQiypmMB+A6cf/dOfnmHTs8COHS6n7ZNTw9xlkp+z5y16Xw35PdLcvHlDwsQZE3ifoCpmTb6jcujWCCyA7/+lT/sln/r0b+oPJVo+sCT72Wp5B9kNhnyE8Mlf1iAIGtKXiDueaVZnfYdyxOZpO9J02tIy6D17Hkqv+ZB5PfgaRXyE8Mif1iAIGtKmeEk2ncsv8LPf+6Xjddwrc9r9nGKB+JuqSWR34PHaxHyc1a+/GEBgqxJRIPKb0i7tU8yfs89xlMenWef4uFoJ/xlzcHltwn5V5/8YQGCrClFg9or6KbReZqe0OZ/2lkeT7x4yM/fxeuH/Lb8YQGCrEncEklnagY5qcm/3aVa79dlAOatQv6Q3yt/WIAga4xbIqtfjA3/auW8bZAoP/60stJPBhWF/CG/9lTKHxYgyBr1RJj8Yjwrfd5WYmsCyiVmnxSzs77ImiF/WcKQvyp/WIAgazr3BNu+VRlXLZ/yEq0fz7zQf7tTtzzkD/n7lj8sQJA1pTWAFm/tWV9LZG/JlTjqv+Z6DaG90dMq5M9Z/rAAQdaot0RqX7P/LI8/ZsOO1pCzOknIb8uplYf8YQGCrEmsAQjP9+dvtVTInNJNJQn5B2H1yR8WIMgadSfYs2Nn45ntyXW9P4rQrhnyh/w2VHPNs0dODyhoEAwvNdmhVz6VS2qTLHcm6v5yExB01W5wqWjB8W9oxf4DWh3/eoZ9/Ida/haA6zYs1MXZeeMrNLH8e4H+fT7CM/PT5Le91xdHfgz5+GMA+VfC+IcXKMiaxmsAO75Cy9nCn3oot1VbpdNMA6izDEU++zaKOPICXtJrW65TfYvEKT/MPyFJjL98i8Qvf1iAIGsS9wM0PcEp9YqWCcye+XG0+Z+mw6T8vsiQQdHurB1cfs97Y/wHH/+wAEHWdLJD0y/+7IqynLeSbi97RW/PVuW81v6+m2rTew9M1taRPP3QeLI89caamfdQj78t/8of/7AAQdYkMsPJeZK91rbPekpdwutoeGIGtffK+hKpdUZu2m3Uby0uApifB4DF6VeSPRBSM9kRlJwYf43lG/+wAEHWGJnhqti+Artc6gxbf3iksuvLuWBTrTOykffQRuGXXhi5pVJz7eICgHcW2pW3aNrIznoZ41+q0/m/sFzjHxYgyJpOXiD6pb/T+zaeuaNW09OzbDUlAuS4ViCt8we/0uv/L79tzVbn364+HdkyImqNANgIoNBPNFu1tVFX8qEefwj5PePvZ7nHPyxAkDWlWyLlDqLmRfaf+bczBVCJ7bu158pcHr/uefzFak2yCRPH57ol759tA7h6+3nx5vMA3j87CuCBO4CKDWm1AGzc1kahjd491vNacD3U/buGevyl/By/7t+1+0MAMzMXjDoF5wG8+frl6YcNxz8sQJA1pTvCislV2ovMaerB0J4WJXYkieUz4f3wmEFN95CmAc6h0Ddnvj8F4JGjAPDw/nEAL5w5B2D6LABs3boWKc105vuTACY2jQG4enuv5rHDl6E7c221AFy1K62H+FQ2n/EnpF2dYdaDRnLPlnUo/l/I8Zc99Df+YQGCrFEzw3nO6ciafjxn/gkZAS/hs8863d+DdMbe/eMAvnj7XgCPsKdkDSaOXwBw8qmp5Hv37hxFsXIgLSWtwWKr1ZVk/ujhblst3oZYfeNPcM1No6Sh6X7Ziur0N/5hAYKsaaHz+fI5aBVN93Dd0FQPafW1k0RN48il7icNweeXnCcOTaCwA9vu/m0Aj+yfNPov7MNcpbzU/+5zKPSQpPsXZTX+BFlOjqbvtZo0woOPf1iAIGtKsUDcD615DDQ9RB6AsfokPQlkz02zfBE0+9xy13i3ZMeN57o/a7qfIF1Cuv/Uv/xt92fJp7/0aaR0D9dV9JSvHDw7oKtj/DnFjL83a5caneC63/4/RXjGv5AB0PcNwgIEWZOIBh1Eo3BvgOemby2LQX/ZHgnyAZPuOfJY1XuzZ/9492eppzlkB/w88lS6/Lr7xgDQ/uXJo/X9DPv4k63j3p7Nc+sA7NlZ1ehcT3PfjtT9Ho+Qth4ovEPp8Q8LEGRNJzMc/SK/dTkr5fNFz80cMqOLJ6+8fAt/2q0/2/tZ8WncN4ZidkheHdu3s3xQ7BCH/q7ZIR//4lfLSty8vVoidb+9NqM6vCbfe+F2gP/MLUZy/CMvUJA7pTWAfWLVzgnjmSlqNcvlaW1ny8BJxW8Cvbk4UOiJI48d6Zb4Z/yypuYv4pFFJ97sleBsuufVPf7+Gb9WUz4t9aDEDlEP2viHBQiyphQNKvFnk/Rns9FaFR4PLWKxf7QZ5x4WBTQIqvXYPw5g+sQ6AO0WANx9HgCuFP6ZPMffX1Pqdc9+AtWhsx/a+IcFCLKmJju0x4usoc1otZyVS4XUHBx7f9EDRQ1J5K4ClSwsjgJ45jYA+MQGADgzD6TOT0lyGH97baB597UeZLk9/mEBgqzpWACujfy7j3b+MH8ru61fPxVRQOeg6xV797cpUvfwKFHyPdPs86353n9pnk3j/NwpIL/x5yVyNk91JhzRQZruJ2rH/7lTYQGCvEncDzA109NPTRlkNqnps6b5aqZfXQdg/eYLKO8BLxXUG8WESrhOolhU8gJx5Niu7vHnKwH/DgCheXvk6kLiGf+wAEHWlLxA3HNMs1JND/lPlDZtZc9iNW1EGvTgzjaAfceBYv5H2pr76fmubdN4Tw7tIsvdZS2zDUl1cGfvr5DjsJrGn5Azfg7X4nLvVqup0d/4hwUIsqaUGU6i6aH+5poer4V//9LOlUDzv/WbxlDM12nGOfi+b0la5k16v+MD6c04KbOQNhPlKTWHd/yhyC/1sbYzoOlsrb5sJX1KvJU9/mEBgqxJRIPaUemDIPNaaqeQOFqGAk2eK9YCul6hfVy5NvCjnRimPKGEPWfV5M9h/AntbBf3C3lsheYjsuHyhwUIsqYUDWp7AJpGF2p6wpPXsmmGAp6b8sANALDv+CiK+R9pDr6zyL03fjtAWp9H+XONVc443SziKLfx5/AZvEfra7vCfp8SJyxAkDWJWyLpTFDTXACa7lkqf4WWwVjzbdNMlNb+6zd5ss6n4bGfUj+R1ic998AdPZ1UvmcgPU+VIzyM4w/gzPxCf+Nv62Y7U5AdZ6pZG05khgsC65bI6hdvw7WOnXdAw5OBvuKpmC23In85+UxoJvrQa0Chg9dvqsaZ0Oz/9scfQyqv2KEHvoTyXN8T4y73Pmk3gPYgCT4+s8pfOizjzxs2HX/Nq2OfIGs8/ifWQRn/yAoR5I56Ikx+8R5PBW8rsTUZ+RBkFhqtZy1W0aOH5HkALbKcYn72MM8Ph98TQ0jdM8dyEPBzt/ZIDsv4o7wljCZ2QOr7prpf+peajn9YgCBrOvcE275h7b4qqZPqfMz181r/7VRaZAvXQ8QBAIUe+vC93lyfPDza2aVf+8evA/i7X/1dlG+OScj81BSK81DTTN+QJ+TgHajIU/FcZTj+H1wAfOd95X6Ltscsdb9n/MMCBFlTWgNoPnWPf0Ci3XslY07qtFGzE0n87lt6y9jWNgo9RNz7PNDbu+3NTaV2IR9RR56nplDeRea8L/K90Qy4iJ2sFTyj8eerAkIbf8Lj15fxnp7xDwsQZI16S6T/LnINf8yJJ8txQX20jPSF01tID3X001fHAYy90Gtb2q/9SHS9RqnJ4J5movCHVP+uSjw98hv/l+5so/AU8RN8Hczx1+hv/MMCBFmTWAMQTdf+dqulgmsjLZ5eQ+rOqT29cqJG/j3jyT6lbJ3+ldxvWpxmbuNPGvp5c+fBZvDxDwsQZI26E+zPS2z30DTCxB8F2TRekkvlr3mp5I/xvzjytwC8fPwtZ5tgORj28R9q+WuyQ698hnr0g0tOC8C122rS8NkBUoPTX3IOALOnZhDyD0zO8nduiayLE/TGh2hi+fcy/fuUXUL+JCtffnv34+LIH16gIGsarwHs+BAt5wx/6qHcVm0lj1PVEvJ7GFx+2ydTeOjbqN5kzEt6beVtx/wtEr/8YQGCrEncD9D0BKrUK1omM3vmx9Hmf5oOC/lXmvy+yKJB0e489ssfFiDImk52aPrFnx1SlvNW0u1lr+jt2aqc11a+45B/xcrv0ff3HpisrSN5+qHxZHnqjdbKJyxAkDWJzHBynmSvte2zqlKX8DoanphHzxnZkP/iyy/rS6TWH7lpt1G/tbgIYH4eABanX0n2QEjLYEewhgUIssbIDFfF9hXY5VJn2PrDI5U/Yl57Y8jvea/21K4v5+JNtX6nzkbqp41iX2Bh5JZKnbWLCwDeWWhX3qJZA55nKSxAkDWdvED0S3/ZB2w8c0etprPnkN/zFlws+eU5LK6VPVqfM/92VdqRLSOi1giAjQAK+0CrBdsaxJngICjfEil3EDUvsnZaR5ZrkSq8xPbd1s6VQ/4VJT/Hr/t37f4QdTeLFZwH8Obrl6cftloANm5ro7AG7x7reY24HYg1QBCU7wgrJldpLzKnqQdDe1qU2JEkaW0n+wn5L638PGbT1v1Xbz9fKZlh1oPnhabcoNIyyB6o1bHDl6G7cmi1AFy1K20HaLTDAgRZo2aG0+aOTf3HGtosViIj4D29hfw2Sys/n/37db99R5im+2UrqkP/vXp7rw5Zg8VWqyvJ/NHD3VbhBQoCtND5fPkctIqme7huaKqHtPraSSIjrjDkN+TRWEL5k3DdLz088oZ3Td9rNWmFoN0w0LlPYPc5FHZAErdEBkE5Foj7oTWPgaaHyAMwVpMgI43suWmWLyLkv7Tyy2ifYsbfm7VLjU7YtwJLpO6X93wWMgD6vkFYgCBrEtGgg2gU7g3w3FQuTxvx8v70UMhPXCr5afbPvT2b59YB2LOzqtG129mk7vd4hLT1QOEdOg/g5NGqtGEBgqzpZIajX7QM61yX8Pmi52YRmdHFkxdfvoU/7dafLXcY8l8S+c/ML/AcPpKbt1dL7PuAZU2qw2tSOWl3bgf4z9xiyHt9rtuwEHmBgtwprQE8N3xI/DNFrWa5PK3tbBl4Pxoh/3LLT8goHcI/49dqyqelHpTYIerhxJsA8PD+cQBg93mGBQiyphQNKvF88YQ/m43WqvB4aBGLFiH/pZXfpumMX9aUet2zn0B16DbodgsA7j4PAFfGmeAgIGqyQ3u8yBrajFbLWbkchPye8qWVX2pujn/Gb0f7aD3I8oXFUQDP3AYAn9gAAHRrPHnYwgIEWdOxAHwH0b/7qJ1JbdrKbuvRTyG/LYNHnkHk79S8axwAcA66Xud+fULO5qnOhCM6SNP9BPn+afb/1nzvv7TOGduK506FBQjyJnE/wNRM5/vog0Fmk5o+8+SrCfnt/vtr65efM/3qOgDrN19Aod35SsC/A0Bo3h65upDsuPEcgOkT1bfwsQ0LEGRNyQvEPcc0K9X0kP9EadNW9izW1kYhv83yyU8W7ODONoB9x3v15Yyfw7W43LvVamrw2E8JSXVwZ++viH2AIChnhpNoeqi/uabHa+Hfv+QpKUN+DxdTfqmPtZ0BTWdr9WUr6VPirbSVQGSFCIJUNKgdlT4IPLrQPoXE0TIUyJoh/6WVn7hiLVCX5VM728X9Qh5bofmIbLj8YQGCrClFg9oegKbRhZqe0OavUhvxPu23h/z8Xbz+xZSfTp8duAEA9h0fRTH/Js09odgEPoP3aH1tV9jvU+KEBQiyJnFLJJ0JapoLQNM9S+WvqMtgzFuF/BdV/jPzC7I+rQTI97J+U73ul9iZguw4U83acCIzXBBYt0RWv3gbrnXsvAMatq5K5imYtfoJ+Xssq/zdhrRfQT4rWgk89BpQRGWu31SN89G8OvYJshrdL/aeyQrRHjDB5Q8LEGSNeiJMfvEeTwVvK7E1GfkQZBYarWdbkpA/JeGyyM+3hNHEDkh931T3S/+S1P1zLAcEP/ccO8FBUNwTbPuGtfuqpE6q8zHXz2v9t1N1y0P+Syt/F77HzOOXDgAo7MCH740C+OAC4Dvvy28DILQ9Zqn7yRN18A5U5Ilo0CDoUFoDaPHiHv+ARPaW9CSgXhvVazjtjZ5WIf9Syc/fy71SY1vbKOwAwVcFBK0NNO3u8evLeE9agRSxq+lWYQGCrFFvidS0kf8skj/mxJPluCDtAwn5bTm18qWSn9eUexH0FrID9PNLd7ZReIrorFYpe/NHous1xms7cE8/Ufijqn8XP88QFiDImsQagGi69rdbLRVcG3ny3EtC/kGw5fdkt+YSkoZ+3tx5sOF/u3aOQhLnAYKgg7oT7M9LbPfQNMLEHwVp1wz5L638tlT+msst/5pnj5x2NgiC1UdMgYKsaQG4dltNGj47QGpw+kvOAWD21AxC/oHJWf7OLZF1cYLp+BDbe+33Tmieaecf07f8NiF/6u0Jhlr+mAIFWaN6gew1deFhbaN6Ey0v6bWVt9Xyt0jKX7zqN0ieqLKx41u0nDn8qYeQX2OlyR8WIMiaxP0AWs4C/nSp0O6s1eZ/mg5oNGft1pHvIrRMbPbKhxPyD4v8YQGCrOlkh6Zf+vAAALj3wGQfL376ofFkeeqNNTM/Q369z2o5byXddrZHwp6tynltRQ+F/JdQ/rAAQdYkM8MBdStuqfVHbtptvWZxEcD8PAAsTr+S7IGQlsGOQOQk9ZPRSquvncTtzzOttfWc8Q35l1v+sABB1qj7AHLe1lTrd+pspH7aKPYFFkZuqdRZu7gA4J2FduUtmjWws17aZ2TtEq1c6iqtjob/pG/IfzHlDwsQZE0nLxD9Qt+EPEfDtbJH63Pm365+qSNbRkStEQAbART2gVYLtjXoil2RX6O/MCzP3Fer6ew5W/n/4d6fr5R8/un/qJWKl0zef2v35/EnXzbeJYk1QBCUb4kcRPfv2v0h6m6GKjgP4M3XL1ckagHYuK2Nwhq8e6znNeJ2oPuVV+T3eJG100ayXItU4SX23kXtXDkr+T/3md8B8PubjwG454ubKn2STfirH+9KvpGgtgTv4bv339pt+0/f+YZT/rAAQdaU7gjjMZu27r96+/lKyQyzHjyvL+V2lJZB9kCtjh2+DN2VQ6sF4KpdaTvAlxLFPh/9lvYic5p6MLSnRUl177xpBHwO8tu6n+iUP3EMwJP3fbXy9P6n/sTZlt7119+aqJU/LECQNR0LwGf/ft1v3/Gk6X7ZiurQf6/e3qtD1mCx1epKMn/0cLeVFu9BaHPfpv5jDW0WLpER8J7eVp/8ktcn3wAwPd3790BanP67Y/LLlac7dlT/LX33ibPiafVfly1/WIAga9SdYK77pYdH3tCt6XutJq0QtAzxnXzwu8+hsAOS7jx1agY070xsMADQdSfXDU31qFZfO0lnxNVmIj+HdP/28Y8BAHp2gDQ66XIqKfl5Ovq+V1/ahDNzJwAAVT+SJn9YgCBrShZARvsUM/7evEpqdMK+1VUidT/vmT+lVYG6b8DgfmjN46HpUfKAjdUk+Egje26apYxY3fJLyA4QXJfzeT+vQ/8O+VMOlf/zb30bAL7xTafMYQGCrEmsAWj2z709m+fWAdizs6rR5Z1+hNT9Ho+Qth4ovEPnAZw8av0xxCAakXvDPJpMy2LAz0Y11aOrW37y7pNH/0dvTDv629H9Sdb/0RsAcP3HdiC1b+CRPyxAkDXdaFDrK7l5e7XEvs9V1qQ6vCaVk3bndoD/zC1G6QYRAMB1GxZmix+oRMsQz3Uhn+96bkaRGY08ef3lW/jTbv2s5Odwbf1/L00B2Pu/f1+pU6wK+KxhR+XpV+Z+A8DhXxyr/pEMW/6wAEHWJNYAMkqH8M/4tZryaakHJXaIejjxJgA8vH8cAM4iieeGEq2VZ6ar1SyXp7W1LQPvR2PY5dd48rPfAwBsQnlvuNgl6DE93dsloKdPgtp+D8Des39cqe+RPyxAkDXqTjCn6Yxf1pR63bOfQHUefxEA2i0AuPs8AFwp/Bv+c6ga/mw2WqvC46FFXFrkKf/Epj+rlJDun7jmCwDu6Wh3Vv+aLwDYO02rBb6X3OtNswMaYQGCrElYAKm5Of4Zvx3to/UgyxcWRwE8cxsAfGIDANCt39odgByPF1xDm9FqOSuXg9UnP/l8Hh39JoDXWTnX/TadOnMA8Ohkdcd3P74M4EFYfiFOWIAga8qZ4e4aBwCcg67XuV+fkLN5qjPhiA7SdD9Bvn+a/b813/svzVPHtuK5U52a3Br4d0+1M7VNW9ltPfo1N/nJf0+QNaCzvJuv6dUpRwFZ/VAPvE/gh/CdeQgLEGRNYg0w/eo6AOs3X0Ch3flKwL8DQGjeHrm6kOy48RyA6RPVt0gdw+e1UzPtZB0Pg8yGNX3sybeTg/z/OjkB4FOf2gNgw5beYz5f//F//hDAO2dmALyw8WSlh9ffnqy0LffQa/uDH7zglD8sQJA1pRtiDu5sA9h3vPdYzvg5XIvLvVutpgaP/ZSQVAd3Avrcjnu+aVaq6VH/idimrexZuK2N8pF/889+0njLZnwSwJ7R05XyQ3PjRivinUMz0P8KKX9YgCBrSpnhOFIfazsDms7W6stW0qfEW2krAZ6SUvviNT3U31zZ43Xx77/mLP9yY/8VXP6wAEHWJLxAV6wF6rJ8ame7uF/IYys0H5ENvyWAo0U7alHpTZF5ObVTVFLakJ/XJ8+9RufUr/inQWcGZCYIzv34eOVdtvxhAYKsSeQGPXADAOw7Popi/k2ae0KxCXwG79H62q6w36ck0TImcJpGR2p6zpOXs2mGhRzk/86//Y/xdPmw5Q8LEGRN50yw1Aq0EiDfy/pN9bpfYmcKsuNMNWvDkd80z3XcNJeBpjuXyt9Sl4GZt1r98pfjdgTXAMCjqK4TeByoziQKa/Obv3Q9K0/LHxYgyJrSTfHkr6U1O60EHnoNKKIy12+qfnqaV8c+QVaj+8XeM1kh2gMmutpuNvUnaTnvbbjWtPMmaHgy6Fc8LSH/RUOTPyxAkDWJWyL9dkDq+6a6X/qXpO6fYzkg+LlVW6vJL97jqeBtJbYmIx+azEKj9Zyb/K8CAP7oa38D4M8f/D2jLUEe/TKTta2of1sqInaCg6C8E8z3CHn8xgEAhR348L1RAB9cAHznffltAIS2xyx1P3miDt6BijyVqMakF4vgeki7W8UT7+6f1/pv18pN/s/8wk+i8M/Yenpwfu5nrof4d5KULSxAkDWJWCB+dyzN3sa2tlHYAYKvCghaG2ja3ePXl/GetAIpYvdqO1C/eI9/w9ObljGzTpvWa2jtjZ5WwyV/2TdvofXmj2D1yB8WIMiakgXQfMn0zZEdoJ9furONwlNEZ7VK2Zs/Eu9ZUy8K9/QThT+q+tVW4tGR0ha2/hgk25kWQemfVUtCfltOrXxw+cMCBFlTsgCe7MT8+yMN/bzpObbhWlyLI5docY5N/CHNWi0V/G8M+f2tlgopf1iAIGtc2aE5/nzFntme9Ev440bsmn457R5Cfo3VIf+aZ4+cHlDQIBhe/h9cv+fzvdRkhAAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reached terminal state? True\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "front-nomination",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 273
        },
        "outputId": "8e47a26d-0949-4028-deda-43ebdf1539cf"
      },
      "source": [
        "env.reset() # Let's reset the state again\n",
        "display(Image.fromarray(env.render()))"
      ],
      "id": "front-nomination",
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<PIL.Image.Image image mode=RGB size=256x256 at 0x7F3820121F10>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAIAAADTED8xAAAeXklEQVR4nO2df4xd1XHHv0bPD7Nm8drYGFeVwdRKKMVuiymoBtEFFP9BhJAMUdUWlSpt3UaRatIIGlUCoSJFTfIHOFVp66BKliBV1YBEMFaBVFmB6xpau6kNrhOtYoEq7wYbsDFdvOut6R/z7ntz79yZO/e9t7985vOHtXvuOefOXvnMzJkz55xFu/cfRxCkSgPAlWvX2JWuWjbV/nnlQLPvQpycmCqUvHPa9Zbxd8cQ8vdMyvI3AFy1bMpukH86Bd+fwcWy6/On1Ip/sso/JuQvJeQve3uRiyrlCIILmEbdBtl4Kh+dNP6oRI5aaao08m3VVuPO7hghv4d05A8LECRNywLU8vnadaiVHJcHxop9EjQu+SgnZA+a/6fpgJA/5O9O/rAAQdI0wIaInEET2pjm5byVDHvZM3rb25N+YWEch/whf9fyhwUIkiYXBSod3ygb03Z9Xi7HIq+jYY9pz3tD/pDfI39YgCBpStYBtLFlz7XtcjnWtToaHs/PLg/5Q35ZEhYgSJoG2GCqO4Zs7r1xuP3zc/8xAt3nI/weXqH+DMmvSRXyF+ovaPnDAgRJ01oHoNEgV+C0KKxcjSOo/I9u2wJg+dc65WQN/u61VwqtqH5ZnLh6NLfL+y4/L9cyVUL+C0P+sABB0rSiQNk6Gf1WHoXlaKNK6n6CSugpzQfy/XTeQqO5bgZ5v+TvNhIS8i9U+cMCBEnTsgB+30uLv5KX/7O7hgAsx6nSl9FTbT6gITPIJb3LXxfNi5WE/PNZ/rAAQdIs2r3/eHtT8yZzc7McuzS2vn73MDLt7mf1nlPI5gNaz5UUNmV3Jz9FLbrTQ3bPlYT8mGv5wwIESZPLBeJxXG3Grc3r//T7EwB+/YqPK1/5b+9dCuBZ9mbZM8ngHMdtupM/ixsAVTpMI+QnFqL8YQGCpCnJBvWPSPL+f2e6G++NWlFEiGcKETSau9NDvWgU0mGExzflsYiQn1hY8ocFCJJm0e79x2/ecDn9oo0VORZ5pifp8ju/Pg3g2Mh7AH7l7odL+/nRi98EsG74CgD/8ucNAM82On6blinEKXh7bxx+H0AX8vOe62o4bX8Tl1C+hT8N+eeP/GEBgqTJzQHssSXH06NfugfAL3/mcmTafeTo9QBGju4B8OBDd7VrPvmtPQCA6wEAbwHY+lcPA/iFn7wP4C/+5gXkV+y4V2fvA+pFft7Ko4e0mvnykL+c+Sl/WIAgaXLZoBJt1JK/TtmdxKNf+n0AD959OYB7v/L3AB5k9V8/Pg7guSe+COC/fnJzuw7vjecScjxrhHXll/hPs9FaZRGPkL/D/Jc/LECQNBWnQ9ujhzQ3QdaAZgWk6TmZ7u94/LxtL3kgNp4osobm0WpnVs4EIb+nvBf5wwIESdOyAHwFzr96J0feD194A8APu2pbd/cQp1/ye3zWuv5uyO+XZPblDwsQJE3J/QAHxpqoM45pVZh2/b6FcQCP75xClvVJUJboI9uaAJazVmX7gzvIfaXaaO5Ffk4v3mTITyws+cMCBEmTiwLxyCt5ddo4ljsyn179RQC3vvo8gEe2nQLAd2HuQRPA0UNDAPZ+biuAe9FZB5A9a9ijuRf5PXhahfzEQpE/LECQNLmT4STaOJajkLJ97sMSAL+5fwrAwINL2k8nnjwL4B+XLAHw5I/2tMu7281ZuFuqL/J78EQtQn7O/Jc/LECQNIt27z/++ZvXePKz7Vw/Wgnmu8N4rr8s5/EfQtMihjyUj94X+T3w7EJ/DDvkb9evL3KOmZA/LECQNLlsUHsGrY2kkVdfArD5Tx4C8Oy3v9Uup5IVl0wDePYbT7TLf+uRhwB874WXANx3z+fb5dpeHk++eC/yc+R9t/7Mk5B/IcofFiBImtYcgBfR6Km7U/PQ3hfaP3/wSQOZ7h9sliecct3vodTLpJPJ+iK/pnv6lSkZ8tvMlfxhAYKkMW6JrJ7Xczbeeg90j016h3VPCeb9tN8yrtesKz/XOvKv6CVLkT8N+TXmSv6wAEHSqDvC5IjxzPR5W4mtCegsMXnDlNazLUnIXyZhyF+UPyxAkDSte4Lt2KrMq5ZPeYnWj8cv9N/u1C4P+UP+ruUPCxAkTW4OoGVZeObXEtlb6Uwc1aO5WkNob/S0CvlTlj8sQJA06i2Rds6dZ33On7NhZ2tIr04S8ttyauUhf1iAIGlK5gCEZ/z5W/ULPprt9cKQfya48OQPCxAkjboS7Fmxs/F4e3Je788itGuG/CG/DdVctHv/8R4FDYKFS7hAQdI0wK6617ATjHqnu8MtkG3ICPl7JGX5GwCuWjZVlWfnza/QxPKvBfrX+dp0Lb8dvZ7/8tvMf/nnw/cPFyhImoobYiR2fgUfwXbE1ybfVm1VuiOJsGMC2Xn2TWR55Bm8pNM2X6f4Fknv8mvE95dvkfjlDwsQJE3J/QB1d3BKvULj294fLPUWR/P/NB0m5fdlhvSKdmdt7/J73hvfv/fvHxYgSJrW6dD0i5xBE57RzFvJsJc9o7e9VenXFsaxIb9H32zdMVJZR/L89uHS8rI3Vnje8f0r60j69f3DAgRJk4sClY5vVM217b2eUpfwOhqenEHtvbK+RGqdgRs2G/Ub09MAJiYAYHr0zdIeCKmZ7AxKTnx/jZn7/mEBgqQxToYrYscK7HKpM2z94ZHKri99wbpap1VniPppIotLTw3cVKizeHoKwOmpZuEtmjayT72M75+rM8PfPyxAkDStc4Hol+5279t4fEetprPngvz81nKCawWP1uFMnCpKO7ByQNQaADAEINNP5K3a2kiTXyO+f6uHvn7/sABB0uRuiZQriFoU2b/n3z4pgErs2G2lr1yQn+PXPRs3nwcwNnbOqJMxCeDEsYvLHzYaAIbWNpFpo48OdaIWXA9p8sf3r6Kf3z8sQJA0uTvCMueqPIrMqRvB0J5mJXYmiRUz4f3wnEFb96xaN1koGWPaa82axQC2rFwC4JWTZ1GmmWQP1OrQvovQ9lwbDQCXbSzXQ9yVje8/V98/LECQNOrJcJ59OrKmH8+ef0JmwEu49+nXPaQzNDTdI1tRHfp31bpOHdJG041GW5KJg/varbR8GyK+/+x8/7AAQdI00Bq+3Actoukerhvq6iGtvraTqG4eOdc9MsLwwIbBQn1N32g1yUPddfhM4Sn3X7H5LDI9JGn/RfH9MXffPyxAkDS5XCAeh9YiBpoeogjApooDMsqRPdc95YuQ2SaZx9nxGqVGIbjuyWkRBal7eM/8KXmlatyaEd+fmM3vHxYgSJqSbNBeNAqPBnhu+tZOMejutEeCvE8ebVhxZgmALRuKGoXrCR5bkLrHE5HQ/NEsOjEJ4J2D1fLH95/N7x8WIEia1slw9Isc69Ir5f6i52YOeaKL51x5+Rb+tF1/vNPc0lI3riuWSN2jeZxUk+rwmlRO2oXrIf4z11j/+2HR971q2dR49gOVxPeXzPT3DwsQJE1uDmDvWLXPhPF4ilrNfHm5trNl4MgsEcLvcWo15dNcD0ruCvVw5AQAPPrYMAB8WC55fH+7pnya66Gr7x8WIEiaXDaoxH+apP80G61VFvHQMha7p67HKWtKveKJZ1Odp14DgGYDAO6cBIBLRXwmvr9dc+a+f1iAIGkqTof2RJE1NI9WO7OyX0jNwfF7nHa2idaDLJ+aHgTw4i0AcPUyADg5AZTtn5LE95/p7x8WIEialgXg2si/+mifH+ZvZbf166eVdwwDAM5C1ys8rkxIb5Lq7HJkp2i6h6DYM3mf7090/iU/e9MavPRuq2Z8f85sfv+wAEHSlNwPcGCsNT66oBdvUtNndc+rGX17CYClK84h0y7cE/VHoAkt2iC9W8n6684CGD1SfIv8tvH9idn//mEBgqTJRYF45Ji8Uk0P+XeU1m1le7GaNiINunNDE8C2w5360uPkcC0i1w61mho891BCUu3c0Pkr5HeI70/M5vcPCxAkTe5kOImmh7rzNT1RC//6JT+SUvYj9YEWmdZ0hlZftpIxDd5K80Rt+Yn4/jP9/cMCBElTkg1qZ6X3gjzXUtuFxNFOKNDkuWQxUHXKpLa3iMclPLpKi1HYaPLH95/97x8WIEiaXDaoHQGom12o6QnPuZZ1TyjgZ1PuuBYAth0eROb/kebYpegk7kF6tI62KumPaWjyx/ef/e8fFiBImpJbImlPUN2zADTd0694hXGC8cmJKVmfPFGa+y9dXq17JPZJNXaeo6btOPILx/fnzM73DwsQJI1xS2RxxNtwrWOfO6DhOYG+EKkYzzekeDnFTMgT3X4UyLICly4v5ploUQV7B1OF7hFrn6QFaQ2SKMhf+pfG9ydm+vuHBQiSRt0RJke8J1LB20psTUYxBHkKjdazdkuhXw9JfVNX98j4htQ9Z9gZBHzfrf0l4/vPzvcPCxAkTeueYDs2rN1XJXVSVYy52q/1306lZbZwPUTsAJDpofMfDwL45Bzg22/KT6MntDVOqXsoErLzNhTkKWRlxvdvM/vfPyxAkDS5OYCWL+6JD0i0e69kzkmVNqq3I4nffUtv2bSmiUwPEdwrJcg31bSLJ64s8w3JA85yJys7iO8/B98/LECQNOotkf67yDX8OSeeU44zqrNlZCyc3kJ6iH7ee3sTWaSC9grlTg/+VHS9yHhtCx5pJrJ4SPHvKuTTI77/3H3/sABB0pTMAYi6c3+7Vb/g2kjLp9eQupM0xMtm5NuG/+1aHr9Ey9OM7z/73z8sQJA06kqw/1xiu4e6GSb+LMi6+ZJcKn/NuZI/vv/syL9o9/7jzgZBcOERLlCQNA0AV66tOIbPTpDqne4O5wAw/u4YQv6eSVn+1i2RVXmC3vwQTSz/WqZ/nbJNyF/K/JffXv2YHfnDBQqSpuKGGImdH6KdOcOfesi3VVuV7qiyCfk99C6/HZPJIvRNFG8y5iWdtvK2Y/4WiV/+sABB0pTcD1B3B6rUK9pJZrbnx9H8P02HhfzzTX5fZlGvaHce++UPCxAkTet0aPrFfzqkLOetZNjLntHb3qr0awvjOOSft/J79P3WHSOVdSTPbx8uLS97ozXzCQsQJE3JyXDST7Ln2vZeValLeB0NT86jZ49syD/78sv6Eqn1B27YbNRvTE8DmJgAgOnRN0t7IKRlsDNYwwIESWOcDFfEjhXY5VJn2PrDI5U/Y157Y8jvea/21K4vffG6Wr9VZ4j6aSJbF5gauKlQZ/H0FIDTU83CWzRrwM9ZCgsQJE3rXCD6pbvTB2w8vqNW09lzyO95C2ZLfrkPi2tlj9bnTJwqSjuwckDUGgAwBCCzDzRbsK1B7AkOgvwtkXIFUYsia7t1ZLmWqcJL7Nhtpa8c8s8r+Tl+3b9x83lU3SyWMQngxLGLyx82GgCG1jaRWYOPDnWiRtwOxBwgCPJ3hGXOVXkUmVM3gqE9zUrsTJJybSf7CfnnVn6es2nr/lXrJgslY8x68HOh6WxQaRlkD9Tq0L6L0J45NBoALttYbgfoa4cFCJJGPRlO8x3rxo81NC9WIjPgPb2F/Db9lZ97/37db98Rpul+2Yrq0L+r1nXqkDWYbjTakkwc3NduFVGgIEADreHLfdAimu7huqGuHtLqazuJjLzCkN+QR6OP8pfCdb+M8Mgb3jV9r9WkGYJ2w0DrPoHNZ5HZAUncEhkE+VwgHofWIgaaHqIIwKaKAzLKkT3XPeWLCPnnVn6Z7ZN5/B2vXWp0wr4VWCJ1P++ZP6VZgbZuEBYgSJqSbNBeNAqPBnhuKpe7jXh5d3oo5CfmSn7y/nm0Z8WZJQC2bChqdHknJCF1vycipM0HsujQJIB3DhalDQsQJE3rZDj6RTthnesS7i96bhaRJ7p4zsWXb+FP2/XH8x2G/HMi/8mJKX6Gj+TGdcUS+z5gWZPq8JpUTtqd2wH+M7cYuRtoAPq74lygIHVycwDPDR8Sv6eo1cyXl2s7Wwbej0bIP9PyEzJLh/B7/FpN+TTXg5I7RD0cOQEAjz42DADs7viwAEHS5LJBJZ4RT/hPs9FaZREPLWPRIuSfW/lt6nr8sqbU6571BKrz1GsA0GwAwJ2TAHBp7AkOAqLidGhPFFlD82i1MytngpDfU95f+aXm5vg9fjvbR+tBlk9NDwJ48RYAuHoZANCt8RRhCwsQJE3LAvAVRP/qo7YntW4ru61HP4X8tgweeXqRv1XzjmEAwFnoep3H9QnpzVOdXY7sIE33ExT7J+///YnOvzTP2bQGL70bFiBIm5L7AQ6MtcZHF/TiTWr6zHNeTchv999dW7/8nNG3lwBYuuIcMu3OZwL+FQBCi/bI2YVk/XVnAYweKb6Ff9uwAEHS5KJAPHJMXqmmh/w7Suu2sr1YWxuF/DYzJz9ZsJ0bmgC2He7Ulx4/h2txuXar1dTguZ8Skmrnhs5fEesAQZA/GU6i6aHufE1P1MK/fsmPpAz5Pcym/FIfaysDms7W6stWMqbEW2kzgTgVIgjKskHtrPRe4NmF9i4kjnZCgawZ8s+t/MQli4GqUz61vV08LuSxFVqMyIbLHxYgSJpcNqgdAaibXajpCc1/ldqI92m/PeTn7+L1Z1N+2n2241oA2HZ4EJn/TZp7l2ITuAfv0fraqrA/psQJCxAkTcktkbQnqO5ZAJru6Ve8ouoEY94q5J9V+U9OTMn6NBOg2MvS5dW6X2KfFGTnmWrWhhMnwwWBdUtkccTbcK1jnzugYeuq0nMKxq1+Qv4OMyp/uyGtV1DMimYC248CWVbm0uXFPB8tqmPvIKvQ/WLtmawQrQETXP6wAEHSqDvC5Ij3RCp4W4mtySiGIE+h0Xq2JQn5yyScEfn5kjDq2AGp7+vqfhlfkrr/DDsDgu97jpXgIMjuCbZjw9p9VVInVcWYq/1a/+1U7fKQf27lb8PXmHn+0g4AmR04//EggE/OAb79vvw2AEJbY5a6nyJRO29DQZ7IBg2CFrk5gJYv7okPSGRvpZEEVGujag2nvdHTKuTvl/z8vTwqtWlNE5kdIPisgKC5gabdPXF9me9JM5Asd7W8VViAIGnUWyI1beTfi+TPOfGccpxRHgMJ+W05tfJ+yc9ryrUIegvZAfp57+1NZJEi2quVO735U9H1IuO1LXikn8jiUcW/i+9nCAsQJE3JHICoO/e3W/ULro0859xLQv5esOX3nG7NJSQN/bK58mDD/3ZtH4Uk9gMEQQt1Jdh/LrHdQ90ME38WpF0z5J9b+W2p/DVnWv5Fu/cfdzYIgguPcIGCpGkAuHJtxTF8doJU73R3OAeA8XfHEPL3TMryt26JrMoTLM8PsaPX/uiEFpl2/jFdy28T8pe9vYQFLX+4QEHSqFEge06dRVibKN5Ey0s6beVttfwtkvyIV+MGpTuqbOz8Fu3MHP7UQ8ivMd/kDwsQJE3J/QDamQX8ab/Q7qzV/D9NB9TyWdt15LsI7SQ2e+bDCfkXivxhAYKkaZ0OTb90EQEAsHXHSBcvfn77cGl52RsrPD9Dfr3PYjlvJcN2dkTC9lalX1vQQyH/HMofFiBImtKT4YCqGbfU+gM3bLZeMz0NYGICAKZH3yztgZCWwc5A5JTqJ6OVVl/bidtdZFpr69njG/LPtPxhAYKkUdcBpN9WV+u36gxRP01k6wJTAzcV6iyengJweqpZeItmDexTL+09snaJVi51lVZHw7/TN+SfTfnDAgRJ0zoXiH6hMSH30XCt7NH6nIlTxZE6sHJA1BoAMAQgsw80W7CtQVvsgvwa3aVheXxfraaz52Tl/97WXy2U3Pf8f1ZKxUtG7r+5/fPwM28Y75LEHCAI8rdE9qL7N24+j6qboTImAZw4drEiUQPA0NomMmvw0aFO1IjbgfYoL8jviSJru41kuZapwkvstYtKXzkp+e+56w8BfGXFIQCf+4PlhT7JJjzxwcbSNxLUluA9vHr/ze22L+z5jlP+sABB0uTuCOM5m7buX7VuslAyxqwHP9eXznaUlkH2QK0O7bsI7ZlDowHgso3ldoBPJbJ1PvqtPIrMqRvB0J5mJcW187oZ8CnIb+t+olX+9CEAz9z7l4Wn9z/3NWdbete3/2FXpfxhAYKkaVkA7v37db99x5Om+2UrqkP/rlrXqUPWYLrRaEsycXBfu5WW70Fovm/d+LGG5oVLZAa8p7cLT37JsZH3AIyOdv4/kBanf9ePfLXwdP364v+lV5/+UDwt/u+y5Q8LECSNuhLMdb+M8MgbujV9r9WkGYJ2QnzrPPjNZ5HZAUnbTz0wBvI7SxYYAOi6k+uGunpUq6/tpDPyahORn0O6f93wFQCAjh0gjU66nEpycZ6Wvu/Ulzbh5JkjAIBiHEmTPyxAkDQ5CyCzfTKPv+NXSY1O2Le6SqTu5z3zpzQrUNcNGDwOrUU8ND1KEbBNFQd8lCN7rntKGXFhyy8hO0BwXc79fl6H/h/ypxwq/+ff+z4AfOe7TpnDAgRJUzIHIO+fR3tWnFkCYMuGokaXd/oRUvd7IkLafCCLDk0CeOeg9ccQvWhEHg3zaDLtFAO+N6quHr2w5afoPkX0f/reqKO/9e2fZP2fvgcA11yxHmXrBh75wwIESdPOBrVGyY3riiX2fa6yJtXhNamctDu3A/xnbjFyN4gAAK5aNjWe/UAl2gnxXBdyf9dzM4o80chzrr98C3/arp+U/Byurf9v7wEAD/zsnwp1slkB9xrWF54+cua3Aey7dVPxj2TY8ocFCJKmZA4gs3QIv8ev1ZRPcz0ouUPUw5ETAPDoY8MA8CFK8dxQorXyeLpazXx5uba2ZeD9aCx0+TWeufsHAIDlyK8NZ6sEHUZHO6sE9PQZUNsfAHjgwz8r1PfIHxYgSBp1JZhT1+OXNaVe96wnUJ2nXgOAZgMA7pwEgEtFfMO/D1XDf5qN1iqLeGgZlxZpyr9r+TcKJaT7d63+AoDPtbQ7q7/6CwAeGKXZAl9L7vSm2QGNsABB0pRYAKm5OX6P38720XqQ5VPTgwBevAUArl4GAHTrt3YHIMcTBdfQPFrtzMqZ4MKTn2I+jw9+F8AxVs51v02rzhkAeHykuOL7GL4K4Muw4kKcsABB0uRPhrtjGABwFrpe53F9QnrzVGeXIztI0/0Exf7J+39/ovMv+amb1uCld1s1uTXwr55qe2rrtrLbevRravJT/J4ga0B7eVes7tTJZwFZ/VAPvE/gx/DteQgLECRNyRxg9O0lAJauOIdMu/OZgH8FgNCiPXJ2IVl/3VkAo0eKb5E6hvu1B8aapXU89OINa/rYc95OCvL/68guANdfvwXAspWdx9xf/+C/fwzg9MkxAK8MvVPo4dipkULbfA+dtm+99YpT/rAAQdLkbojZuaEJYNvhzmPp8XO4Fpdrt1pNDZ77KSGpdm4AdN+OR77JK9X0qH9HbN1Wthdua6N05F/xi5813rICnwWwZfB4ofz1M8NGK+L062PQ/wopf1iAIGlyJ8NxpD7WVgY0na3Vl61kTIm30mYC/EhKbcRreqg7X9kTdfGvv6Ys/0xj/xVc/rAAQdKURIEuWQxUnfKp7e3icSGPrdBiRDb8lgCOlu2oZaXXRZ7Lqe2iktKG/Lw+Re41Wrt+xX8N2jMgT4Lg3I+fK7zLlj8sQJA0JWeD7rgWALYdHkTmf5Pm3qXYBO7Be7S+tirsjylJtBMTOHWzIzU95zmXs+4JCynIv+ff/8d4OnPY8ocFCJKmtSdYagWaCVDsZenyat0vsU8KsvNMNWvDkWOan3Vc9ywDTXf2K95SdQIzb3Xhy5/P2xGsBoDHUZwn8DxQnRFk1uZ3f+MaVl4uf1iAIGlyN8VTvJbm7DQT2H4UyLIyly4vDj0tqmPvIKvQ/WLtmawQrQETbW03XvYnaWfe23CtaZ+boOE5Qb8QaQn5Zw1N/rAAQdKU3BLptwNS39fV/TK+JHX/GXYGBN+3ams1OeI9kQreVmJrMoqhyVNotJ5Tk/9tAMDDf/23AL755T822hIU0c8zUtmK+relImIlOAjyK8F8jZDnb+wAkNmB8x8PAvjkHODb78tvAyC0NWap+ykStfM2FOQpZDWWRrEIroe0u1U8+e5+v9Z/u1Zq8t/1az+PLD5j6+ne+aXPXAPx/6RUtrAAQdKU5ALxu2PJe9u0ponMDhB8VkDQ3EDT7p64vsz3pBlIlrtX2YE64j3xDU9v2omZVdq0WkNrb/S0Wljy52PzFlpv/gxWj/xhAYKkyVkALZZMY47sAP289/YmskgR7dXKnd78qXjPompReKSfyOJRxVFbyEdHmbaw9Ucvp51pGZR+r1oS8ttyauW9yx8WIEianAXwnE7Mxx9p6JfNyLEN1+JaHrlEy3OsEw+p16pf8L8x5Pe36hdS/rAAQdK4Tofm+M8r9nh7Mi7hzxuxa/rltHsI+TUuDPkX7d5/vEdBg2Dh8v/LTEi46XMPqgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "official-china"
      },
      "source": [
        "# Part 1 - MDP and Planning: Implement Policy Iteration \n",
        "- In this part we will focus on methods that assume knowledge of the enivonment dynamics, in partucular you will implement Policy Iteration. \n",
        "- The environment model can be obtained through `env.P`"
      ],
      "id": "official-china"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "spanish-feeding",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "db7a592f-1cba-49de-bf24-bd39741f0831"
      },
      "source": [
        "# No need to change anything here. Try to understand what happens \n",
        "\n",
        "# let's look at a random state-action pair and observe its transition characteristics\n",
        "# you can re-run this cell to get a different state-action pair\n",
        "random_state  = env.observation_space.sample()\n",
        "random_action = env.action_space.sample()\n",
        "# returns a list of tuples (probability,newstate,reward,is_terminal_state)\n",
        "env.P[random_state][random_action] "
      ],
      "id": "spanish-feeding",
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(1.0, 7, 0, True)]"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lucky-place",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a004df00-2aab-41bf-d39a-72d1d0f84d7d"
      },
      "source": [
        "############################\n",
        "# YOUR CODE HERE #\n",
        "# Print all the terminal states in the environment.\n",
        "# you can use env.P\n",
        "############################\n",
        "terminal_states = set([])\n",
        "for state in env.P:\n",
        "    for action in env.P[state]:\n",
        "        for transitions in env.P[state][action]:\n",
        "            if transitions[3]:\n",
        "                terminal_states.add(transitions[1])\n",
        "\n",
        "print(terminal_states)"
      ],
      "id": "lucky-place",
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{5, 7, 11, 12, 15}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rocky-danger",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 273
        },
        "outputId": "966ec098-78e7-4a1e-938f-a636a585e0f2"
      },
      "source": [
        "# Verify your solution (look at the positions where final states are)\n",
        "display(Image.fromarray(env.render()))"
      ],
      "id": "rocky-danger",
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<PIL.Image.Image image mode=RGB size=256x256 at 0x7F38201839A0>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAIAAADTED8xAAAeXklEQVR4nO2df4xd1XHHv0bPD7Nm8drYGFeVwdRKKMVuiymoBtEFFP9BhJAMUdUWlSpt3UaRatIIGlUCoSJFTfIHOFVp66BKliBV1YBEMFaBVFmB6xpau6kNrhOtYoEq7wYbsDFdvOut6R/z7ntz79yZO/e9t7985vOHtXvuOefOXvnMzJkz55xFu/cfRxCkSgPAlWvX2JWuWjbV/nnlQLPvQpycmCqUvHPa9Zbxd8cQ8vdMyvI3AFy1bMpukH86Bd+fwcWy6/On1Ip/sso/JuQvJeQve3uRiyrlCIILmEbdBtl4Kh+dNP6oRI5aaao08m3VVuPO7hghv4d05A8LECRNywLU8vnadaiVHJcHxop9EjQu+SgnZA+a/6fpgJA/5O9O/rAAQdI0wIaInEET2pjm5byVDHvZM3rb25N+YWEch/whf9fyhwUIkiYXBSod3ygb03Z9Xi7HIq+jYY9pz3tD/pDfI39YgCBpStYBtLFlz7XtcjnWtToaHs/PLg/5Q35ZEhYgSJoG2GCqO4Zs7r1xuP3zc/8xAt3nI/weXqH+DMmvSRXyF+ovaPnDAgRJ01oHoNEgV+C0KKxcjSOo/I9u2wJg+dc65WQN/u61VwqtqH5ZnLh6NLfL+y4/L9cyVUL+C0P+sABB0rSiQNk6Gf1WHoXlaKNK6n6CSugpzQfy/XTeQqO5bgZ5v+TvNhIS8i9U+cMCBEnTsgB+30uLv5KX/7O7hgAsx6nSl9FTbT6gITPIJb3LXxfNi5WE/PNZ/rAAQdIs2r3/eHtT8yZzc7McuzS2vn73MDLt7mf1nlPI5gNaz5UUNmV3Jz9FLbrTQ3bPlYT8mGv5wwIESZPLBeJxXG3Grc3r//T7EwB+/YqPK1/5b+9dCuBZ9mbZM8ngHMdtupM/ixsAVTpMI+QnFqL8YQGCpCnJBvWPSPL+f2e6G++NWlFEiGcKETSau9NDvWgU0mGExzflsYiQn1hY8ocFCJJm0e79x2/ecDn9oo0VORZ5pifp8ju/Pg3g2Mh7AH7l7odL+/nRi98EsG74CgD/8ucNAM82On6blinEKXh7bxx+H0AX8vOe62o4bX8Tl1C+hT8N+eeP/GEBgqTJzQHssSXH06NfugfAL3/mcmTafeTo9QBGju4B8OBDd7VrPvmtPQCA6wEAbwHY+lcPA/iFn7wP4C/+5gXkV+y4V2fvA+pFft7Ko4e0mvnykL+c+Sl/WIAgaXLZoBJt1JK/TtmdxKNf+n0AD959OYB7v/L3AB5k9V8/Pg7guSe+COC/fnJzuw7vjecScjxrhHXll/hPs9FaZRGPkL/D/Jc/LECQNBWnQ9ujhzQ3QdaAZgWk6TmZ7u94/LxtL3kgNp4osobm0WpnVs4EIb+nvBf5wwIESdOyAHwFzr96J0feD194A8APu2pbd/cQp1/ye3zWuv5uyO+XZPblDwsQJE3J/QAHxpqoM45pVZh2/b6FcQCP75xClvVJUJboI9uaAJazVmX7gzvIfaXaaO5Ffk4v3mTITyws+cMCBEmTiwLxyCt5ddo4ljsyn179RQC3vvo8gEe2nQLAd2HuQRPA0UNDAPZ+biuAe9FZB5A9a9ijuRf5PXhahfzEQpE/LECQNLmT4STaOJajkLJ97sMSAL+5fwrAwINL2k8nnjwL4B+XLAHw5I/2tMu7281ZuFuqL/J78EQtQn7O/Jc/LECQNIt27z/++ZvXePKz7Vw/Wgnmu8N4rr8s5/EfQtMihjyUj94X+T3w7EJ/DDvkb9evL3KOmZA/LECQNLlsUHsGrY2kkVdfArD5Tx4C8Oy3v9Uup5IVl0wDePYbT7TLf+uRhwB874WXANx3z+fb5dpeHk++eC/yc+R9t/7Mk5B/IcofFiBImtYcgBfR6Km7U/PQ3hfaP3/wSQOZ7h9sliecct3vodTLpJPJ+iK/pnv6lSkZ8tvMlfxhAYKkMW6JrJ7Xczbeeg90j016h3VPCeb9tN8yrtesKz/XOvKv6CVLkT8N+TXmSv6wAEHSqDvC5IjxzPR5W4mtCegsMXnDlNazLUnIXyZhyF+UPyxAkDSte4Lt2KrMq5ZPeYnWj8cv9N/u1C4P+UP+ruUPCxAkTW4OoGVZeObXEtlb6Uwc1aO5WkNob/S0CvlTlj8sQJA06i2Rds6dZ33On7NhZ2tIr04S8ttyauUhf1iAIGlK5gCEZ/z5W/ULPprt9cKQfya48OQPCxAkjboS7Fmxs/F4e3Je788itGuG/CG/DdVctHv/8R4FDYKFS7hAQdI0wK6617ATjHqnu8MtkG3ICPl7JGX5GwCuWjZVlWfnza/QxPKvBfrX+dp0Lb8dvZ7/8tvMf/nnw/cPFyhImoobYiR2fgUfwXbE1ybfVm1VuiOJsGMC2Xn2TWR55Bm8pNM2X6f4Fknv8mvE95dvkfjlDwsQJE3J/QB1d3BKvULj294fLPUWR/P/NB0m5fdlhvSKdmdt7/J73hvfv/fvHxYgSJrW6dD0i5xBE57RzFvJsJc9o7e9VenXFsaxIb9H32zdMVJZR/L89uHS8rI3Vnje8f0r60j69f3DAgRJk4sClY5vVM217b2eUpfwOhqenEHtvbK+RGqdgRs2G/Ub09MAJiYAYHr0zdIeCKmZ7AxKTnx/jZn7/mEBgqQxToYrYscK7HKpM2z94ZHKri99wbpap1VniPppIotLTw3cVKizeHoKwOmpZuEtmjayT72M75+rM8PfPyxAkDStc4Hol+5279t4fEetprPngvz81nKCawWP1uFMnCpKO7ByQNQaADAEINNP5K3a2kiTXyO+f6uHvn7/sABB0uRuiZQriFoU2b/n3z4pgErs2G2lr1yQn+PXPRs3nwcwNnbOqJMxCeDEsYvLHzYaAIbWNpFpo48OdaIWXA9p8sf3r6Kf3z8sQJA0uTvCMueqPIrMqRvB0J5mJXYmiRUz4f3wnEFb96xaN1koGWPaa82axQC2rFwC4JWTZ1GmmWQP1OrQvovQ9lwbDQCXbSzXQ9yVje8/V98/LECQNOrJcJ59OrKmH8+ef0JmwEu49+nXPaQzNDTdI1tRHfp31bpOHdJG041GW5KJg/varbR8GyK+/+x8/7AAQdI00Bq+3Actoukerhvq6iGtvraTqG4eOdc9MsLwwIbBQn1N32g1yUPddfhM4Sn3X7H5LDI9JGn/RfH9MXffPyxAkDS5XCAeh9YiBpoeogjApooDMsqRPdc95YuQ2SaZx9nxGqVGIbjuyWkRBal7eM/8KXmlatyaEd+fmM3vHxYgSJqSbNBeNAqPBnhu+tZOMejutEeCvE8ebVhxZgmALRuKGoXrCR5bkLrHE5HQ/NEsOjEJ4J2D1fLH95/N7x8WIEia1slw9Isc69Ir5f6i52YOeaKL51x5+Rb+tF1/vNPc0lI3riuWSN2jeZxUk+rwmlRO2oXrIf4z11j/+2HR971q2dR49gOVxPeXzPT3DwsQJE1uDmDvWLXPhPF4ilrNfHm5trNl4MgsEcLvcWo15dNcD0ruCvVw5AQAPPrYMAB8WC55fH+7pnya66Gr7x8WIEiaXDaoxH+apP80G61VFvHQMha7p67HKWtKveKJZ1Odp14DgGYDAO6cBIBLRXwmvr9dc+a+f1iAIGkqTof2RJE1NI9WO7OyX0jNwfF7nHa2idaDLJ+aHgTw4i0AcPUyADg5AZTtn5LE95/p7x8WIEialgXg2si/+mifH+ZvZbf166eVdwwDAM5C1ys8rkxIb5Lq7HJkp2i6h6DYM3mf7090/iU/e9MavPRuq2Z8f85sfv+wAEHSlNwPcGCsNT66oBdvUtNndc+rGX17CYClK84h0y7cE/VHoAkt2iC9W8n6684CGD1SfIv8tvH9idn//mEBgqTJRYF45Ji8Uk0P+XeU1m1le7GaNiINunNDE8C2w5360uPkcC0i1w61mho891BCUu3c0Pkr5HeI70/M5vcPCxAkTe5kOImmh7rzNT1RC//6JT+SUvYj9YEWmdZ0hlZftpIxDd5K80Rt+Yn4/jP9/cMCBElTkg1qZ6X3gjzXUtuFxNFOKNDkuWQxUHXKpLa3iMclPLpKi1HYaPLH95/97x8WIEiaXDaoHQGom12o6QnPuZZ1TyjgZ1PuuBYAth0eROb/kebYpegk7kF6tI62KumPaWjyx/ef/e8fFiBImpJbImlPUN2zADTd0694hXGC8cmJKVmfPFGa+y9dXq17JPZJNXaeo6btOPILx/fnzM73DwsQJI1xS2RxxNtwrWOfO6DhOYG+EKkYzzekeDnFTMgT3X4UyLICly4v5ploUQV7B1OF7hFrn6QFaQ2SKMhf+pfG9ydm+vuHBQiSRt0RJke8J1LB20psTUYxBHkKjdazdkuhXw9JfVNX98j4htQ9Z9gZBHzfrf0l4/vPzvcPCxAkTeueYDs2rN1XJXVSVYy52q/1306lZbZwPUTsAJDpofMfDwL45Bzg22/KT6MntDVOqXsoErLzNhTkKWRlxvdvM/vfPyxAkDS5OYCWL+6JD0i0e69kzkmVNqq3I4nffUtv2bSmiUwPEdwrJcg31bSLJ64s8w3JA85yJys7iO8/B98/LECQNOotkf67yDX8OSeeU44zqrNlZCyc3kJ6iH7ee3sTWaSC9grlTg/+VHS9yHhtCx5pJrJ4SPHvKuTTI77/3H3/sABB0pTMAYi6c3+7Vb/g2kjLp9eQupM0xMtm5NuG/+1aHr9Ey9OM7z/73z8sQJA06kqw/1xiu4e6GSb+LMi6+ZJcKn/NuZI/vv/syL9o9/7jzgZBcOERLlCQNA0AV66tOIbPTpDqne4O5wAw/u4YQv6eSVn+1i2RVXmC3vwQTSz/WqZ/nbJNyF/K/JffXv2YHfnDBQqSpuKGGImdH6KdOcOfesi3VVuV7qiyCfk99C6/HZPJIvRNFG8y5iWdtvK2Y/4WiV/+sABB0pTcD1B3B6rUK9pJZrbnx9H8P02HhfzzTX5fZlGvaHce++UPCxAkTet0aPrFfzqkLOetZNjLntHb3qr0awvjOOSft/J79P3WHSOVdSTPbx8uLS97ozXzCQsQJE3JyXDST7Ln2vZeValLeB0NT86jZ49syD/78sv6Eqn1B27YbNRvTE8DmJgAgOnRN0t7IKRlsDNYwwIESWOcDFfEjhXY5VJn2PrDI5U/Y157Y8jvea/21K4vffG6Wr9VZ4j6aSJbF5gauKlQZ/H0FIDTU83CWzRrwM9ZCgsQJE3rXCD6pbvTB2w8vqNW09lzyO95C2ZLfrkPi2tlj9bnTJwqSjuwckDUGgAwBCCzDzRbsK1B7AkOgvwtkXIFUYsia7t1ZLmWqcJL7Nhtpa8c8s8r+Tl+3b9x83lU3SyWMQngxLGLyx82GgCG1jaRWYOPDnWiRtwOxBwgCPJ3hGXOVXkUmVM3gqE9zUrsTJJybSf7CfnnVn6es2nr/lXrJgslY8x68HOh6WxQaRlkD9Tq0L6L0J45NBoALttYbgfoa4cFCJJGPRlO8x3rxo81NC9WIjPgPb2F/Db9lZ97/37db98Rpul+2Yrq0L+r1nXqkDWYbjTakkwc3NduFVGgIEADreHLfdAimu7huqGuHtLqazuJjLzCkN+QR6OP8pfCdb+M8Mgb3jV9r9WkGYJ2w0DrPoHNZ5HZAUncEhkE+VwgHofWIgaaHqIIwKaKAzLKkT3XPeWLCPnnVn6Z7ZN5/B2vXWp0wr4VWCJ1P++ZP6VZgbZuEBYgSJqSbNBeNAqPBnhuKpe7jXh5d3oo5CfmSn7y/nm0Z8WZJQC2bChqdHknJCF1vycipM0HsujQJIB3DhalDQsQJE3rZDj6RTthnesS7i96bhaRJ7p4zsWXb+FP2/XH8x2G/HMi/8mJKX6Gj+TGdcUS+z5gWZPq8JpUTtqd2wH+M7cYuRtoAPq74lygIHVycwDPDR8Sv6eo1cyXl2s7Wwbej0bIP9PyEzJLh/B7/FpN+TTXg5I7RD0cOQEAjz42DADs7viwAEHS5LJBJZ4RT/hPs9FaZREPLWPRIuSfW/lt6nr8sqbU6571BKrz1GsA0GwAwJ2TAHBp7AkOAqLidGhPFFlD82i1MytngpDfU95f+aXm5vg9fjvbR+tBlk9NDwJ48RYAuHoZANCt8RRhCwsQJE3LAvAVRP/qo7YntW4ru61HP4X8tgweeXqRv1XzjmEAwFnoep3H9QnpzVOdXY7sIE33ExT7J+///YnOvzTP2bQGL70bFiBIm5L7AQ6MtcZHF/TiTWr6zHNeTchv999dW7/8nNG3lwBYuuIcMu3OZwL+FQBCi/bI2YVk/XVnAYweKb6Ff9uwAEHS5KJAPHJMXqmmh/w7Suu2sr1YWxuF/DYzJz9ZsJ0bmgC2He7Ulx4/h2txuXar1dTguZ8Skmrnhs5fEesAQZA/GU6i6aHufE1P1MK/fsmPpAz5Pcym/FIfaysDms7W6stWMqbEW2kzgTgVIgjKskHtrPRe4NmF9i4kjnZCgawZ8s+t/MQli4GqUz61vV08LuSxFVqMyIbLHxYgSJpcNqgdAaibXajpCc1/ldqI92m/PeTn7+L1Z1N+2n2241oA2HZ4EJn/TZp7l2ITuAfv0fraqrA/psQJCxAkTcktkbQnqO5ZAJru6Ve8ouoEY94q5J9V+U9OTMn6NBOg2MvS5dW6X2KfFGTnmWrWhhMnwwWBdUtkccTbcK1jnzugYeuq0nMKxq1+Qv4OMyp/uyGtV1DMimYC248CWVbm0uXFPB8tqmPvIKvQ/WLtmawQrQETXP6wAEHSqDvC5Ij3RCp4W4mtySiGIE+h0Xq2JQn5yyScEfn5kjDq2AGp7+vqfhlfkrr/DDsDgu97jpXgIMjuCbZjw9p9VVInVcWYq/1a/+1U7fKQf27lb8PXmHn+0g4AmR04//EggE/OAb79vvw2AEJbY5a6nyJRO29DQZ7IBg2CFrk5gJYv7okPSGRvpZEEVGujag2nvdHTKuTvl/z8vTwqtWlNE5kdIPisgKC5gabdPXF9me9JM5Asd7W8VViAIGnUWyI1beTfi+TPOfGccpxRHgMJ+W05tfJ+yc9ryrUIegvZAfp57+1NZJEi2quVO735U9H1IuO1LXikn8jiUcW/i+9nCAsQJE3JHICoO/e3W/ULro0859xLQv5esOX3nG7NJSQN/bK58mDD/3ZtH4Uk9gMEQQt1Jdh/LrHdQ90ME38WpF0z5J9b+W2p/DVnWv5Fu/cfdzYIgguPcIGCpGkAuHJtxTF8doJU73R3OAeA8XfHEPL3TMryt26JrMoTLM8PsaPX/uiEFpl2/jFdy28T8pe9vYQFLX+4QEHSqFEge06dRVibKN5Ey0s6beVttfwtkvyIV+MGpTuqbOz8Fu3MHP7UQ8ivMd/kDwsQJE3J/QDamQX8ab/Q7qzV/D9NB9TyWdt15LsI7SQ2e+bDCfkXivxhAYKkaZ0OTb90EQEAsHXHSBcvfn77cGl52RsrPD9Dfr3PYjlvJcN2dkTC9lalX1vQQyH/HMofFiBImtKT4YCqGbfU+gM3bLZeMz0NYGICAKZH3yztgZCWwc5A5JTqJ6OVVl/bidtdZFpr69njG/LPtPxhAYKkUdcBpN9WV+u36gxRP01k6wJTAzcV6iyengJweqpZeItmDexTL+09snaJVi51lVZHw7/TN+SfTfnDAgRJ0zoXiH6hMSH30XCt7NH6nIlTxZE6sHJA1BoAMAQgsw80W7CtQVvsgvwa3aVheXxfraaz52Tl/97WXy2U3Pf8f1ZKxUtG7r+5/fPwM28Y75LEHCAI8rdE9qL7N24+j6qboTImAZw4drEiUQPA0NomMmvw0aFO1IjbgfYoL8jviSJru41kuZapwkvstYtKXzkp+e+56w8BfGXFIQCf+4PlhT7JJjzxwcbSNxLUluA9vHr/ze22L+z5jlP+sABB0uTuCOM5m7buX7VuslAyxqwHP9eXznaUlkH2QK0O7bsI7ZlDowHgso3ldoBPJbJ1PvqtPIrMqRvB0J5mJcW187oZ8CnIb+t+olX+9CEAz9z7l4Wn9z/3NWdbete3/2FXpfxhAYKkaVkA7v37db99x5Om+2UrqkP/rlrXqUPWYLrRaEsycXBfu5WW70Fovm/d+LGG5oVLZAa8p7cLT37JsZH3AIyOdv4/kBanf9ePfLXwdP364v+lV5/+UDwt/u+y5Q8LECSNuhLMdb+M8MgbujV9r9WkGYJ2QnzrPPjNZ5HZAUnbTz0wBvI7SxYYAOi6k+uGunpUq6/tpDPyahORn0O6f93wFQCAjh0gjU66nEpycZ6Wvu/Ulzbh5JkjAIBiHEmTPyxAkDQ5CyCzfTKPv+NXSY1O2Le6SqTu5z3zpzQrUNcNGDwOrUU8ND1KEbBNFQd8lCN7rntKGXFhyy8hO0BwXc79fl6H/h/ypxwq/+ff+z4AfOe7TpnDAgRJUzIHIO+fR3tWnFkCYMuGokaXd/oRUvd7IkLafCCLDk0CeOeg9ccQvWhEHg3zaDLtFAO+N6quHr2w5afoPkX0f/reqKO/9e2fZP2fvgcA11yxHmXrBh75wwIESdPOBrVGyY3riiX2fa6yJtXhNamctDu3A/xnbjFyN4gAAK5aNjWe/UAl2gnxXBdyf9dzM4o80chzrr98C3/arp+U/Byurf9v7wEAD/zsnwp1slkB9xrWF54+cua3Aey7dVPxj2TY8ocFCJKmZA4gs3QIv8ev1ZRPcz0ouUPUw5ETAPDoY8MA8CFK8dxQorXyeLpazXx5uba2ZeD9aCx0+TWeufsHAIDlyK8NZ6sEHUZHO6sE9PQZUNsfAHjgwz8r1PfIHxYgSBp1JZhT1+OXNaVe96wnUJ2nXgOAZgMA7pwEgEtFfMO/D1XDf5qN1iqLeGgZlxZpyr9r+TcKJaT7d63+AoDPtbQ7q7/6CwAeGKXZAl9L7vSm2QGNsABB0pRYAKm5OX6P38720XqQ5VPTgwBevAUArl4GAHTrt3YHIMcTBdfQPFrtzMqZ4MKTn2I+jw9+F8AxVs51v02rzhkAeHykuOL7GL4K4Muw4kKcsABB0uRPhrtjGABwFrpe53F9QnrzVGeXIztI0/0Exf7J+39/ovMv+amb1uCld1s1uTXwr55qe2rrtrLbevRravJT/J4ga0B7eVes7tTJZwFZ/VAPvE/gx/DteQgLECRNyRxg9O0lAJauOIdMu/OZgH8FgNCiPXJ2IVl/3VkAo0eKb5E6hvu1B8aapXU89OINa/rYc95OCvL/68guANdfvwXAspWdx9xf/+C/fwzg9MkxAK8MvVPo4dipkULbfA+dtm+99YpT/rAAQdLkbojZuaEJYNvhzmPp8XO4Fpdrt1pNDZ77KSGpdm4AdN+OR77JK9X0qH9HbN1Wthdua6N05F/xi5813rICnwWwZfB4ofz1M8NGK+L062PQ/wopf1iAIGlyJ8NxpD7WVgY0na3Vl61kTIm30mYC/EhKbcRreqg7X9kTdfGvv6Ys/0xj/xVc/rAAQdKURIEuWQxUnfKp7e3icSGPrdBiRDb8lgCOlu2oZaXXRZ7Lqe2iktKG/Lw+Re41Wrt+xX8N2jMgT4Lg3I+fK7zLlj8sQJA0JWeD7rgWALYdHkTmf5Pm3qXYBO7Be7S+tirsjylJtBMTOHWzIzU95zmXs+4JCynIv+ff/8d4OnPY8ocFCJKmtSdYagWaCVDsZenyat0vsU8KsvNMNWvDkWOan3Vc9ywDTXf2K95SdQIzb3Xhy5/P2xGsBoDHUZwn8DxQnRFk1uZ3f+MaVl4uf1iAIGlyN8VTvJbm7DQT2H4UyLIyly4vDj0tqmPvIKvQ/WLtmawQrQETbW03XvYnaWfe23CtaZ+boOE5Qb8QaQn5Zw1N/rAAQdKU3BLptwNS39fV/TK+JHX/GXYGBN+3ams1OeI9kQreVmJrMoqhyVNotJ5Tk/9tAMDDf/23AL755T822hIU0c8zUtmK+relImIlOAjyK8F8jZDnb+wAkNmB8x8PAvjkHODb78tvAyC0NWap+ykStfM2FOQpZDWWRrEIroe0u1U8+e5+v9Z/u1Zq8t/1az+PLD5j6+ne+aXPXAPx/6RUtrAAQdKU5ALxu2PJe9u0ponMDhB8VkDQ3EDT7p64vsz3pBlIlrtX2YE64j3xDU9v2omZVdq0WkNrb/S0Wljy52PzFlpv/gxWj/xhAYKkyVkALZZMY47sAP289/YmskgR7dXKnd78qXjPompReKSfyOJRxVFbyEdHmbaw9Ucvp51pGZR+r1oS8ttyauW9yx8WIEianAXwnE7Mxx9p6JfNyLEN1+JaHrlEy3OsEw+p16pf8L8x5Pe36hdS/rAAQdK4Tofm+M8r9nh7Mi7hzxuxa/rltHsI+TUuDPkX7d5/vEdBg2Dh8v/LTEi46XMPqgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "protective-pendant"
      },
      "source": [
        "\n",
        "### Step A: Implement Policy Evaluation\n"
      ],
      "id": "protective-pendant"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "forty-offering"
      },
      "source": [
        "def policy_evaluation(P, nS, nA, policy, gamma=0.9, tol=1e-3):\n",
        "    \"\"\"Evaluate the value function from a given policy.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    P, nS, nA, gamma:\n",
        "        defined at beginning of file\n",
        "    policy: np.array[nS]\n",
        "        The policy to evaluate. Maps states to actions.\n",
        "    tol: float\n",
        "        Terminate policy evaluation when\n",
        "            max |value_function(s) - prev_value_function(s)| < tol\n",
        "    Returns\n",
        "    -------\n",
        "    value_function: np.ndarray[nS]\n",
        "        The value function of the given policy, where value_function[s] is\n",
        "        the value of state s\n",
        "    \"\"\"\n",
        "\n",
        "    value_function = np.zeros(nS)\n",
        "\n",
        "    ############################\n",
        "    # YOUR IMPLEMENTATION HERE #\n",
        "    step = -1\n",
        "    total_steps = 1000\n",
        "    error = 1e9\n",
        "\n",
        "    while error > tol and step < total_steps:\n",
        "        step += 1\n",
        "        new_value_function = np.zeros(nS)\n",
        "\n",
        "        for state in range(nS):\n",
        "            action = policy[state]\n",
        "            transitions = P[state][action]\n",
        "\n",
        "            for transition in transitions:\n",
        "                probability,newstate,reward,is_terminal_state = transition\n",
        "                new_value_function[state] += probability*(reward + gamma*value_function[newstate])\n",
        "        error = np.max(np.abs(new_value_function - value_function))\n",
        "        value_function = new_value_function\n",
        "\n",
        "    if step == total_steps:\n",
        "        print(\"Policy evaluation didn't converge in given number of step.\")\n",
        "\n",
        "    ############################\n",
        "    return value_function"
      ],
      "id": "forty-offering",
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "stunning-property"
      },
      "source": [
        "#### Evaluate random policies"
      ],
      "id": "stunning-property"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "naked-tooth",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cef53792-d5d7-4f39-bbb2-7eca4008c13a"
      },
      "source": [
        "# Test your policy_evaluation on 5 randomly generated deterministic policies\n",
        "# print the value function of the policies\n",
        "\n",
        "############################\n",
        "# YOUR CODE HERE #\n",
        "\n",
        "for i in range(5):\n",
        "    random_policy = np.random.randint(4, size = 16)\n",
        "    print(f'-------- Policy {i}','-'*30)\n",
        "    value_function = policy_evaluation(env.P, env.observation_space.n,\\\n",
        "                                       env.action_space.n, random_policy, gamma=0.9, tol=1e-3)\n",
        "    print(value_function)\n",
        "\n",
        "############################"
      ],
      "id": "naked-tooth",
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------- Policy 0 ------------------------------\n",
            "[0.01904429 0.00748964 0.0028253  0.         0.02804704 0.\n",
            " 0.04265613 0.         0.04814971 0.11252281 0.14221179 0.\n",
            " 0.         0.18486793 0.43143766 0.        ]\n",
            "-------- Policy 1 ------------------------------\n",
            "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
            "-------- Policy 2 ------------------------------\n",
            "[0.01327497 0.00522332 0.03471471 0.         0.01962175 0.\n",
            " 0.08158298 0.         0.0535326  0.10710167 0.23806366 0.\n",
            " 0.         0.3049442  0.60632301 0.        ]\n",
            "-------- Policy 3 ------------------------------\n",
            "[4.19686110e-03 9.81430290e-03 2.40888933e-02 9.81430290e-03\n",
            " 1.20265560e-03 0.00000000e+00 6.23427285e-02 0.00000000e+00\n",
            " 3.92639400e-04 8.55117000e-05 1.84857879e-01 0.00000000e+00\n",
            " 0.00000000e+00 1.64025000e-05 5.54980006e-01 0.00000000e+00]\n",
            "-------- Policy 4 ------------------------------\n",
            "[0.00227286 0.00070227 0.00227286 0.         0.00573831 0.\n",
            " 0.00573831 0.         0.01772964 0.05462895 0.01772964 0.\n",
            " 0.         0.16561345 0.38796228 0.        ]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ongoing-evaluation"
      },
      "source": [
        "### Step B: Implement Policy Improvement"
      ],
      "id": "ongoing-evaluation"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cooperative-dressing"
      },
      "source": [
        "def policy_improvement(P, nS, nA, value_from_policy, policy, gamma=0.9):\n",
        "    \"\"\"\n",
        "    Given the value function from policy improve the policy.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    P, nS, nA, gamma:\n",
        "        defined at beginning of file\n",
        "    value_from_policy: np.ndarray\n",
        "        The value calculated from the policy\n",
        "    policy: np.array\n",
        "        The previous policy.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    new_policy: np.ndarray[nS]\n",
        "        An array of integers. Each integer is the optimal action to take\n",
        "        in that state according to the environment dynamics and the\n",
        "        given value function.\n",
        "    \"\"\"\n",
        "    new_policy = np.zeros(nS, dtype='int')\n",
        "    ############################\n",
        "    # YOUR IMPLEMENTATION HERE #\n",
        "    ############################\n",
        "\n",
        "    for state in range(nS):\n",
        "        expected_reward = np.zeros(nA)\n",
        "        for action in range(nA):\n",
        "            transitions = P[state][action]\n",
        "\n",
        "            for transition in transitions:\n",
        "                probability,newstate,reward,is_terminal_state = transition\n",
        "                expected_reward[action] += probability*(reward + gamma*value_from_policy[newstate])\n",
        "        \n",
        "        max_expected_reward_action = np.argmax(expected_reward)\n",
        "        new_policy[state] = max_expected_reward_action\n",
        "    \n",
        "    return new_policy"
      ],
      "id": "cooperative-dressing",
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "behavioral-penetration",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ed2f840f-6da8-43a5-e433-77b45941db1a"
      },
      "source": [
        "# Print the value before and after policy improvements for 5 randomly generated policies\n",
        "\n",
        "############################\n",
        "# YOUR CODE HERE #\n",
        "for i in range(5):\n",
        "    random_policy = np.random.randint(4, size = 16)\n",
        "    print(f'-------- Policy {i}','-'*30)\n",
        "    value_function = policy_evaluation(env.P, env.observation_space.n,\\\n",
        "                                       env.action_space.n, random_policy, gamma=0.9, tol=1e-3)\n",
        "    print(value_function) # Print value function before improvement\n",
        "    improved_random_policy = policy_improvement(env.P, env.observation_space.n,\\\n",
        "                                       env.action_space.n, value_function, random_policy, gamma=0.9)\n",
        "    print(f'-------> Policy {i} IMPROVED','-'*21)\n",
        "    improved_value_function = policy_evaluation(env.P, env.observation_space.n,\\\n",
        "                                       env.action_space.n, improved_random_policy, gamma=0.9, tol=1e-3)\n",
        "    print(improved_value_function) # Print value after improvement\n",
        "    print()\n",
        "############################"
      ],
      "id": "behavioral-penetration",
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------- Policy 0 ------------------------------\n",
            "[7.76072988e-04 1.99239078e-03 4.46486410e-03 1.74861463e-03\n",
            " 2.43776142e-04 0.00000000e+00 9.24442076e-03 0.00000000e+00\n",
            " 4.05554307e-02 9.59350669e-02 3.12450259e-02 0.00000000e+00\n",
            " 0.00000000e+00 2.49924276e-01 4.89255580e-01 0.00000000e+00]\n",
            "-------> Policy 0 IMPROVED ---------------------\n",
            "[0.0247247  0.03521485 0.05954041 0.04314273 0.06450289 0.\n",
            " 0.10547769 0.         0.12844342 0.23791923 0.29327002 0.\n",
            " 0.         0.37363349 0.63583836 0.        ]\n",
            "\n",
            "-------- Policy 1 ------------------------------\n",
            "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
            "-------> Policy 1 IMPROVED ---------------------\n",
            "[0.         0.         0.02264904 0.00907875 0.         0.\n",
            " 0.05415996 0.         0.         0.         0.15891205 0.\n",
            " 0.         0.         0.4761811  0.        ]\n",
            "\n",
            "-------- Policy 2 ------------------------------\n",
            "[0.         0.         0.         0.         0.         0.\n",
            " 0.         0.         0.         0.         0.         0.\n",
            " 0.         0.         0.47584333 0.        ]\n",
            "-------> Policy 2 IMPROVED ---------------------\n",
            "[0.         0.         0.03155046 0.01308357 0.         0.\n",
            " 0.07479459 0.         0.         0.0745465  0.2190716  0.\n",
            " 0.         0.24909301 0.58242634 0.        ]\n",
            "\n",
            "-------- Policy 3 ------------------------------\n",
            "[0.01129655 0.01765965 0.03219081 0.02239419 0.00427007 0.\n",
            " 0.05443318 0.         0.03531183 0.11453497 0.18146944 0.\n",
            " 0.         0.16521924 0.43732517 0.        ]\n",
            "-------> Policy 3 IMPROVED ---------------------\n",
            "[0.02544746 0.03629475 0.06140948 0.04450216 0.06166765 0.\n",
            " 0.10051655 0.         0.12096546 0.22295813 0.27473218 0.\n",
            " 0.         0.34933607 0.59361435 0.        ]\n",
            "\n",
            "-------- Policy 4 ------------------------------\n",
            "[0.00837935 0.01348476 0.03829581 0.01576565 0.00304923 0.\n",
            " 0.07809549 0.         0.03747718 0.08580317 0.22357412 0.\n",
            " 0.         0.2494558  0.58278914 0.        ]\n",
            "-------> Policy 4 IMPROVED ---------------------\n",
            "[0.0276176  0.03917841 0.06597253 0.04791505 0.06636779 0.\n",
            " 0.10778494 0.         0.12980604 0.23899397 0.29446224 0.\n",
            " 0.         0.37438447 0.63623513 0.        ]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "independent-tragedy"
      },
      "source": [
        "### Step C: Implement Policy Iteration"
      ],
      "id": "independent-tragedy"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ranking-template"
      },
      "source": [
        "def policy_iteration(P, nS, nA, gamma=0.9, tol=1e-3):\n",
        "    \"\"\" \n",
        "    Run policy iteration for dynamics of P.\n",
        "\n",
        "    You should use your methods: policy_evaluation() and policy_improvement() here\n",
        "\n",
        "    Parameters: \n",
        "    P, nS, nA, gamma: defined at beginning of file\n",
        "    tolerance:        tolerance parameter used in policy_evaluation()\n",
        "    \n",
        "    Returns: \n",
        "    value_function: np.ndarray[nS]\n",
        "    policy:         np.ndarray[nS]\n",
        "    \"\"\"\n",
        "\n",
        "    value_function = np.zeros(nS)\n",
        "    policy = np.zeros(nS, dtype=int)\n",
        "\n",
        "    ############################\n",
        "    # YOUR IMPLEMENTATION HERE #\n",
        "    ############################\n",
        "    total_iterations = 1000\n",
        "    converged = False\n",
        "\n",
        "    for iter in range(total_iterations):\n",
        "        value_function = policy_evaluation(P, nS, nA, policy, gamma, tol)\n",
        "        new_policy = policy_improvement(P, nS, nA, value_function, policy, gamma)\n",
        "        \n",
        "        if np.linalg.norm(new_policy - policy) == 0:\n",
        "            converged = True\n",
        "            # print(\"Converged in \" + str(iter) + \" iterations\")\n",
        "            break\n",
        "        else:\n",
        "            policy = new_policy\n",
        "\n",
        "    if not converged:\n",
        "        print(\"Policy iteration didn't converge in given number of iterations.\")\n",
        "\n",
        "    return value_function, policy"
      ],
      "id": "ranking-template",
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "individual-cookie"
      },
      "source": [
        "#### Call your function for gamma=0.9 and gamma=0.6"
      ],
      "id": "individual-cookie"
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "subsequent-tonight",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "531c6019-8b75-47b3-a258-eb242d3532dc"
      },
      "source": [
        "V_pi_s, p_pi_s = policy_iteration(env.P, env.observation_space.n, env.action_space.n, gamma=0.9, tol=1e-3)\n",
        "V_pi_s, p_pi_s"
      ],
      "id": "subsequent-tonight",
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([0.06237625, 0.05593278, 0.07041162, 0.05147676, 0.08589466,\n",
              "        0.        , 0.11002402, 0.        , 0.14055576, 0.24415129,\n",
              "        0.29725924, 0.        , 0.        , 0.37747618, 0.63771531,\n",
              "        0.        ]), array([0, 3, 0, 3, 0, 0, 0, 0, 3, 1, 0, 0, 0, 2, 1, 0]))"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "military-presence",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7e5dc804-8481-409a-dc6e-1de4a8ea3e14"
      },
      "source": [
        "V_pi_f, p_pi_f = policy_iteration(env.P, env.observation_space.n, env.action_space.n, gamma=0.6, tol=1e-3)\n",
        "V_pi_f, p_pi_f"
      ],
      "id": "military-presence",
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([0.0008448 , 0.00139093, 0.0055168 , 0.00156587, 0.00286293,\n",
              "        0.        , 0.0214272 , 0.        , 0.0119808 , 0.0470272 ,\n",
              "        0.10272853, 0.        , 0.        , 0.12289707, 0.44699307,\n",
              "        0.        ]), array([2, 3, 2, 3, 0, 0, 0, 0, 3, 1, 0, 0, 0, 2, 1, 0]))"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fifth-unknown"
      },
      "source": [
        "#### What do you observe in terms of impact of gamma on the actions taken by the policy?"
      ],
      "id": "fifth-unknown"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GGe3wwuEoKye"
      },
      "source": [
        "Though not always observable with gamma = 0.6 and 0.9, the discount factor has effects on the convergence of a policy during policy iteration. Too small a gamma makes the agent greedy focusing more on immediate reward, causing it to not converge to an optimal solution. If set too high, the value function may converge slowly. As, you can see above the optimal policies are different. I tried with gamma = 0.1 and 1.0 and the change was much more drastic"
      ],
      "id": "GGe3wwuEoKye"
    }
  ]
}