Ex13.ReinforcementLearning.Q-Table_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b9bdaa1f",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float:right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25c621fd",
   "metadata": {},
   "source": [
    "# 13th exercise: <font color=\"#C70039\">First Reinforcement Learning Q-Table learning</font>\n",
    "* Course: AML\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date:   05.01.2024\n",
    "\n",
    "---------------------------------\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information on your tasks (e.g. regarding the set of certain paramaters or a specific computational trick), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation please use English language, too.\n",
    "* This applies to all exercises throughout this course.\n",
    "\n",
    "---------------------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "The tasks that you need to work on within this notebook are always indicated below as bullet points. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done by using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook to Google Colab or use your local machine.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date. \n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully \n",
    "    * add comments whereever you feel it necessary for better understanding\n",
    "    * run the notebook for the first time. \n",
    "4. play with all hypterparameters including the actions and states table.\n",
    "5. add and implement an ϵ-greedy strategy \n",
    "---------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8d8b5efe-2b35-4935-af30-8c270bd3b9a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Imports \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d567ab5-06ac-418f-96ca-892695bf9f61",
   "metadata": {},
   "source": [
    "### Create the possible states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9d53fc92-527b-46c6-bc57-4e68423aa61a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "location_to_state = {\n",
    "    'L1' : 0,\n",
    "    'L2' : 1,\n",
    "    'L3' : 2,\n",
    "    'L4' : 3,\n",
    "    'L5' : 4,\n",
    "    'L6' : 5,\n",
    "    'L7' : 6,\n",
    "    'L8' : 7,\n",
    "    'L9' : 8\n",
    "}\n",
    "\n",
    "state_to_location = dict((state,location) for location, state in location_to_state.items())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ba3fd99",
   "metadata": {},
   "source": [
    "### Create the actions & rewards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "349627df",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "actions = [0,1,2,3,4,5,6,7,8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9ac22125",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "rewards = np.array([[0,1,0,0,0,0,0,0,0],\n",
    "                   [1,0,1,0,1,0,0,0,0],\n",
    "                   [0,1,0,0,0,1,0,0,0],\n",
    "                   [0,0,0,0,0,0,1,0,0],\n",
    "                   [0,1,0,0,0,0,0,1,0],\n",
    "                   [0,0,1,0,0,0,0,0,0],\n",
    "                   [0,0,0,1,0,0,0,1,0],\n",
    "                   [0,0,0,0,1,0,1,0,1],\n",
    "                   [0,0,0,0,0,0,0,1,0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7828a644-6339-4f56-8066-9dd3d629190a",
   "metadata": {},
   "source": [
    "### Def remaining hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "20f112bf-e2e1-42a1-a9b3-df514af058c3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# initialize the parameters\n",
    "gamma = 0.75 # discount factor\n",
    "alpha = 0.9  # learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37973f00-e730-4fd2-b660-0ff268984e61",
   "metadata": {},
   "source": [
    "### Define agent and its attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "36a8c7ba-023b-414c-b007-64e789dcd9f0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "class QAgent():\n",
    "    # initialize everything\n",
    "    def __init__(self, alpha, gamma, location_to_state, actions, rewards, state_to_location):\n",
    "        self.gamma = gamma\n",
    "        self.alpha = alpha\n",
    "        \n",
    "        self.location_to_state = location_to_state\n",
    "        self.actions = actions\n",
    "        self.rewards = rewards\n",
    "        self.state_to_location = state_to_location\n",
    "        \n",
    "        # remember, the Q-value table is of size all actions x all states\n",
    "        M = len(location_to_state)\n",
    "        self.Q = np.zeros((M,M), dtype = None, order = 'C')\n",
    "        \n",
    "    # now, implement the training method for the agent\n",
    "    def training(self, start_location, end_location, iterations):\n",
    "\n",
    "        rewards_new = np.copy(self.rewards)\n",
    "\n",
    "        ending_state = self.location_to_state[end_location]\n",
    "        \n",
    "        rewards_new[ending_state, ending_state] = 999\n",
    "        \n",
    "        # DEBUG\n",
    "        print(rewards_new)\n",
    "\n",
    "        # pick random current state\n",
    "        # iterations = the # of training cycles\n",
    "        for i in range(iterations):\n",
    "            current_state = np.random.randint(0,9)\n",
    "            playable_actions = []\n",
    "\n",
    "            # iterate thru the rewards matrix to get states\n",
    "            # that are really reachable from the randomly chosen\n",
    "            # state and assign only those in a list since they are really playable\n",
    "            for j in range(9):\n",
    "                if rewards_new[current_state, j] > 0:\n",
    "                    playable_actions.append(j)\n",
    "\n",
    "            # choosing next random state\n",
    "            # however, make sure that playable_actions is not empty\n",
    "            if len(playable_actions) != 0:\n",
    "                # Randomly select a value from playable_actions\n",
    "                next_state = np.random.choice(playable_actions)\n",
    "    \n",
    "            # finding the difference in Q, often referred to as temporal difference\n",
    "            # by means of the Bellman's equation\n",
    "            TD = rewards_new[current_state, next_state] + self.gamma * self.Q[next_state, np.argmax(self.Q[next_state,])] - self.Q[current_state, next_state]\n",
    "            # combine with the learning rate\n",
    "            self.Q[current_state, next_state] += self.alpha*TD\n",
    "            # DEBUG\n",
    "            #print(f\"Q[{current_state}, {next_state}]:\", self.Q[current_state, next_state])\n",
    "\n",
    "        route = [start_location]\n",
    "        next_location = start_location\n",
    "\n",
    "        # print the optimal route from start to end\n",
    "        self.get_optimal_route(start_location, end_location, next_location, route, self.Q)\n",
    "\n",
    "    # compute the optimal route\n",
    "    def get_optimal_route(self, start_location, end_location, next_location, route, Q):\n",
    "        while(next_location != end_location):\n",
    "            starting_state = self.location_to_state[start_location]\n",
    "            next_state = np.argmax(Q[starting_state,])\n",
    "            next_location = self.state_to_location[next_state]\n",
    "            route.append(next_location)\n",
    "            start_location = next_location\n",
    "        # DEBUG\n",
    "        print('Q-table:',Q)\n",
    "        print(\"optimal route:\", route)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "25bc2a1b-4291-4560-ba23-0dc145e01154",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[999   1   0   0   0   0   0   0   0]\n",
      " [  1   0   1   0   1   0   0   0   0]\n",
      " [  0   1   0   0   0   1   0   0   0]\n",
      " [  0   0   0   0   0   0   1   0   0]\n",
      " [  0   1   0   0   0   0   0   1   0]\n",
      " [  0   0   1   0   0   0   0   0   0]\n",
      " [  0   0   0   1   0   0   0   1   0]\n",
      " [  0   0   0   0   1   0   1   0   1]\n",
      " [  0   0   0   0   0   0   0   1   0]]\n",
      "Q: [[3995.85082585 2249.39694146    0.            0.            0.\n",
      "     0.            0.            0.            0.        ]\n",
      " [2997.88682523    0.         1687.93581703    0.         1687.85490113\n",
      "     0.            0.            0.            0.        ]\n",
      " [   0.         2249.39058429    0.            0.            0.\n",
      "  1266.91613806    0.            0.            0.        ]\n",
      " [   0.            0.            0.            0.            0.\n",
      "     0.          950.93697686    0.            0.        ]\n",
      " [   0.         2249.39757723    0.            0.            0.\n",
      "     0.            0.         1266.58370155    0.        ]\n",
      " [   0.            0.         1687.93624661    0.            0.\n",
      "     0.            0.            0.            0.        ]\n",
      " [   0.            0.            0.          714.20141576    0.\n",
      "     0.            0.         1266.5836127     0.        ]\n",
      " [   0.            0.            0.            0.         1687.44494851\n",
      "     0.          950.9302772     0.          950.92224578]\n",
      " [   0.            0.            0.            0.            0.\n",
      "     0.            0.         1266.5837104     0.        ]]\n",
      "optimal route: ['L9', 'L8', 'L5', 'L2', 'L1']\n"
     ]
    }
   ],
   "source": [
    "qagent = QAgent(alpha, gamma, location_to_state, actions, rewards, state_to_location)\n",
    "qagent.training('L9', 'L1', 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b023083-1b97-40de-9abd-68c1d7711a2c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2305f31a-215d-44aa-9625-19e18e03f12e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}