notebooks/Using-an-Offline-Solver.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using an Offline Solver\n",
    "In this section, we will walk through using an Offline solver to obtain a policy (typically approximately optimal in practice) for a POMDP. For more details on POMDPs and offline solvers, please consult Chapter 6 of the DMU textbook[1]. The solver we will be using is FIB or Fast Informed Bound[2], implemented [here](https://github.com/JuliaPOMDP/FIB.jl). We will also compare it against a policy that chooses actions at random.\n",
    "\n",
    "[1] Kochenderfer, Mykel J. Decision Making Under Uncertainty: Theory and Application. MIT Press, 2015\n",
    "\n",
    "[2] Smith, Trey, and Reid Simmons. Point-based POMDP Algorithms: Improved Analysis and Implementation. In Conference on Uncertainty in Artificial Intelligence. AUAI Press, 2005.\n",
    "\n",
    "### POMDP Model\n",
    "For this example we will use the Tiger POMDP (defined in [POMDPModels](https://github.com/JuliaPOMDP/POMDPModels.jl)) which is an instance of an explicit POMDP. Please see this [notebook](Defining-a-POMDP-with-the-Explicit-Interface.ipynb) for how to define a POMDP with the explicit interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "using POMDPs\n",
    "using POMDPModels # For the problem\n",
    "using FIB # For the solver\n",
    "using POMDPPolicies # For creating a random policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TigerPOMDP(-1.0, -100.0, 10.0, 0.85, 0.95)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define the POMDP problem with default params\n",
    "pomdp = TigerPOMDP()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FIBSolver(100, 0.001, false)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define the FIB Solver with default params\n",
    "solver = FIBSolver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solving the POMDP\n",
    "This is as simple as calling `solve` on the solver and the pomdp problem. The resulting policy is an `AlphaVectorPolicy` defined [here](https://github.com/JuliaPOMDP/POMDPPolicies.jl/blob/master/src/alpha_vector.jl) in `POMDPPolicies`. We also create the baseline random policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AlphaVectorPolicy{TigerPOMDP,Int64}(TigerPOMDP(-1.0, -100.0, 10.0, 0.85, 0.95), 2, Array{Float64,1}[[86.66333847822722, 86.66333847822722], [92.27096626211252, -17.729033737887477], [-17.729033737887477, 92.27096626211252]], [0, 1, 2])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solve the problem offline and obtain the FIB policy which is an AlphaVectorPolicy\n",
    "fib_policy = solve(solver, pomdp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a TigerPOMDP policy that chooses actions at random\n",
    "rand_policy = RandomPolicy(pomdp);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmarking the Solver Policy\n",
    "We will compare the performance of the FIB policy against that of the Random Policy for the Tiger POMDP. Since we only care about the discounted reward, we can use the rollout simulator defined in [POMDPSimulators](https://github.com/JuliaPOMDP/POMDPSimulators.jl). Checkout this [notebook](https://github.com/JuliaPOMDP/POMDPExamples.jl/blob/master/notebooks/Running-Simulations.ipynb) for ways to use the other simulators as well. We will also need a method to update the belief state of the Tiger POMDP after taking an action and seeing an observation. More information about that is in [BeliefUpdaters](https://github.com/JuliaPOMDP/BeliefUpdaters.jl). The default updater for an `AlphaVectorPolicy` is the `DiscreteUpdater`, which is called when we do not provide an updater argument while calling `simulate`. Finally, we can compare the expected discounted rewards and see how the FIB policy does significantly better than random for this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create and run the rollout simulator\n",
    "using POMDPSimulators\n",
    "rollout_sim = RolloutSimulator(max_steps=10);\n",
    "fib_reward = simulate(rollout_sim, pomdp, fib_policy);\n",
    "rand_reward = simulate(rollout_sim, pomdp, rand_policy);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fib_reward = 1.90223878476758\n",
      "rand_reward = -144.15783588873046\n"
     ]
    }
   ],
   "source": [
    "@show fib_reward;\n",
    "@show rand_reward;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.2.0",
   "language": "julia",
   "name": "julia-1.2"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}