notebooks/Defining-a-POMDP-with-the-Generative-Interface.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining a POMDP with the Generative Interface\n",
    "\n",
    "In this tutorial we will define a version of the Crying Baby Problem [1] with the generative interface of POMDPs.jl. To find out more about the generative interface, please see this section of the POMDPs.jl documentation: [Generative POMDP Interface](https://juliapomdp.github.io/POMDPs.jl/latest/generative#generative_doc-1).\n",
    "\n",
    "[1] Kochenderfer, Mykel J. Decision Making Under Uncertainty: Theory and Application. MIT Press, 2015"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "using POMDPs\n",
    "using Random # for AbstractRNG\n",
    "using POMDPModelTools # for Deterministic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model type\n",
    "\n",
    "We begin by implementing the type that represents the model itself. First, some notes about the Crying Baby POMDP:\n",
    "\n",
    "- The *state* is a Boolean value representing whether the baby is hungry\n",
    "- The *action* is a Boolean indicateting whether the agent is feeding the baby\n",
    "- The *observation* is a Boolean indicating whether the baby is crying\n",
    "- When the baby is hungry, it cries with a high probability and cries with a low probability when it is full\n",
    "- When the baby is fed, it immediately becomes full.\n",
    "- At each step, the baby has a small probability of becoming hungry.\n",
    "\n",
    "A [version of this problem implemented with the explicit interface](https://github.com/JuliaPOMDP/POMDPModels.jl/blob/master/src/CryingBabies.jl) can be found in the [POMDPModels repository](https://github.com/JuliaPOMDP/POMDPModels.jl).\n",
    "\n",
    "The `BabyPOMDP` model type is a subtype of the [`POMDP`](https://juliapomdp.github.io/POMDPs.jl/latest/api#POMDPs.POMDP) abstract type. `POMDP` has three parameters which indicate the types used to represent the state, action, and observation respectively, which are all `Bool` in this case. The type also includes parameters that will be used in the reward and transition definitions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "struct BabyPOMDP <: POMDP{Bool, Bool, Bool}\n",
    "    r_feed::Float64\n",
    "    r_hungry::Float64\n",
    "    p_become_hungry::Float64\n",
    "    p_cry_when_hungry::Float64\n",
    "    p_cry_when_not_hungry::Float64\n",
    "    discount::Float64   \n",
    "end\n",
    "\n",
    "BabyPOMDP() = BabyPOMDP(-5., -10., 0.1, 0.8, 0.1, 0.9);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining `gen`\n",
    "\n",
    "The complete generative model for the `BabyPOMDP` can be defined by implemening the method\n",
    "```julia\n",
    "gen(m::BabyPOMDP, s, a, rng)\n",
    "```\n",
    "where `s` is the current state, `a` is the action, and `rng` is a [random number generator](https://juliapomdp.github.io/POMDPs.jl/latest/generative/#Random-number-generators).\n",
    "This function should return a [`NamedTuple`](https://docs.julialang.org/en/v1/manual/types/index.html#Named-Tuple-Types-1) containing the next state (key `sp`, mnemonic \"s prime\"), observation (key `o`), and reward (key `r`).\n",
    "\n",
    "---\n",
    "\n",
    "Note:\n",
    "It is also possible to implement a separate generative model for the transition, observation, and reward models (or, more generally each node in the [dynamic decision network](https://juliapomdp.github.io/POMDPs.jl/latest/ddns)) with the methods\n",
    "```julia\n",
    "gen(::DDNNode{:sp}, m::BabyPOMDP, s, a, rng)\n",
    "gen(::DDNNode{:o}, m::BabyPOMDP, s, a, sp, rng)\n",
    "reward(m::BabyPOMDP, s, a, sp, o)\n",
    "```\n",
    "but this will not be covered in the present tutorial. See the [generative model section of the POMDPs.jl documentation](https://juliapomdp.github.io/POMDPs.jl/latest/generative/) for more information.\n",
    "\n",
    "---\n",
    "\n",
    "First we will qualitatively describe the state and observation models for the Baby POMDP, then provide the complete implementation below.\n",
    "\n",
    "## State transition model\n",
    "\n",
    "For the crying baby problem, if the baby is fed (`a` = `true`), the baby always becomes full (the next state is `false`). Otherwise, if the baby is already hungry, it remains hungry. If not, a random number between 0 and 1 is generated using `rng`, and if this number is less than the probability of becoming hungry, the next state is hungry.\n",
    "\n",
    "## Observation model\n",
    "\n",
    "The observation model in the crying baby problem is implemented as follows: a random number is generated with `rng` and produces a crying observation (`true`) with high probability if the baby is hungry (`sp` = `true`) and with lower probability if the baby is not hungry.\n",
    "\n",
    "## Reward model\n",
    "\n",
    "For this problem there is a negative reward when the baby is hungry (`s` = `true`) and an additional smaller negative reward when the baby is fed (`a` = `true`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "function POMDPs.gen(m::BabyPOMDP, s, a, rng)\n",
    "    # transition model\n",
    "    if a # feed\n",
    "        sp = false\n",
    "    elseif s # hungry\n",
    "        sp = true\n",
    "    else # not hungry\n",
    "        sp = rand(rng) < m.p_become_hungry\n",
    "    end\n",
    "    \n",
    "    # observation model\n",
    "    if sp # hungry\n",
    "        o = rand(rng) < m.p_cry_when_hungry\n",
    "    else # not hungry\n",
    "        o = rand(rng) < m.p_cry_when_not_hungry\n",
    "    end\n",
    "    \n",
    "    # reward model\n",
    "    r = s*m.r_hungry + a*m.r_feed\n",
    "    \n",
    "    # create and return a NamedTuple\n",
    "    return (sp=sp, o=o, r=r)\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial state distribution\n",
    "\n",
    "The final task for describing the POMDP is specifying the initial state distribution, which can be accomplished by implementing a method of [`initialstate_distribution`](https://juliapomdp.github.io/POMDPs.jl/latest/api#POMDPs.initialstate_distribution).\n",
    "\n",
    "For this example, the baby always starts in the the \"not hungry\" state (`false`). This is implemented using the [`Deterministic` distribution from POMDPModelTools](https://juliapomdp.github.io/POMDPModelTools.jl/latest/distributions#Deterministic-1). The distribution returned by `initialstate_distribution` can be any object that implements all or part of the [POMDPs.jl distribution interface](https://juliapomdp.github.io/POMDPs.jl/latest/interfaces#Distributions-1).\n",
    "\n",
    "Alternatively, one may implement [`initialstate(m::BabyPOMDP, rng::AbstractRNG)`](https://juliapomdp.github.io/POMDPs.jl/latest/api#POMDPs.initialstate), to generate initial states, but many solvers expect at least a sampleable model of the initial state distribution, so `initialstate_distribution` is preferred."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "POMDPs.initialstate_distribution(m::BabyPOMDP) = Deterministic(false)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other methods\n",
    "\n",
    "Certain solvers may require other methods to be implemented. To see which methods are required for a particular solver, [check the solver requirements](https://juliapomdp.github.io/POMDPs.jl/latest/requirements#requirements-1) with [@requirements_info](https://juliapomdp.github.io/POMDPs.jl/latest/api#POMDPs.@requirements_info).\n",
    "\n",
    "[`actions`](https://juliapomdp.github.io/POMDPs.jl/latest/api#POMDPs.actions) is a particularly common requirement. For this crying baby case, since the action is a Boolean variable, a [default implementation of the action space is provided when POMDPModelTools is imported](https://juliapomdp.github.io/POMDPModelTools.jl/latest/convenience#Convenience-1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seeing the model in action\n",
    "\n",
    "The generative POMDP model can now be simulated using the `stepthrough` function in the cell below. For more information on running simulations, see the [simulation example](Running-Simulations.ipynb).\n",
    "\n",
    "A Generative model is all that is required for most online POMDP methods and many offline ones. To find out more about solving POMDPs, please see the [online solution](Using-an-Online-Solver.ipynb) and [offline solution](Using-an-Offline-Solver.ipynb) tutorials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n",
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n",
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n",
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n",
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n",
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n",
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n",
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n",
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n",
      "s = false\n",
      "a = true\n",
      "r = -5.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "using POMDPSimulators\n",
    "using POMDPPolicies\n",
    "\n",
    "m = BabyPOMDP()\n",
    "\n",
    "# policy that maps every input to a feed (true) action\n",
    "policy = FunctionPolicy(o->true)\n",
    "\n",
    "for (s, a, r) in stepthrough(m, policy, \"s,a,r\", max_steps=10)\n",
    "    @show s\n",
    "    @show a\n",
    "    @show r\n",
    "    println()\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.2.0",
   "language": "julia",
   "name": "julia-1.2"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}