03_advanced_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creative machine learning - Advanced neural networks\n",
    "\n",
    "### Author: Philippe Esling (esling@ircam.fr)\n",
    "\n",
    "In this course we will cover\n",
    "1. An introduction to [convolutions](#convolution) and how they can be used in CNN\n",
    "2. Defining a [convolutional Neural Network](#cnn) in Pytorch for image classification \n",
    "3. Coding our own [convolutional layer](#layer)\n",
    "4. An explanation on [recurrent networks](#rnn) in practice\n",
    "5. An exercise on [symbolic music generation](#music) with LSTM in both Pytorch and JAX\n",
    "\n",
    "As usual, we are going to use relatively _low-level_ libraries to perform the first exercises (implementing convolutions). To observe this idea in simple setups, we are going to use the `numpy` library and also initialize the homemade course library `cml` and style for future plotting and exercise. We also set the random generator to a fixed point with `rng = np.random.RandomState(1)`, to ensure reproducibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "        .bk-notebook-logo {\n",
       "            display: block;\n",
       "            width: 20px;\n",
       "            height: 20px;\n",
       "            background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAABx0RVh0U29mdHdhcmUAQWRvYmUgRmlyZXdvcmtzIENTNui8sowAAAOkSURBVDiNjZRtaJVlGMd/1/08zzln5zjP1LWcU9N0NkN8m2CYjpgQYQXqSs0I84OLIC0hkEKoPtiH3gmKoiJDU7QpLgoLjLIQCpEsNJ1vqUOdO7ppbuec5+V+rj4ctwzd8IIbbi6u+8f1539dt3A78eXC7QizUF7gyV1fD1Yqg4JWz84yffhm0qkFqBogB9rM8tZdtwVsPUhWhGcFJngGeWrPzHm5oaMmkfEg1usvLFyc8jLRqDOMru7AyC8saQr7GG7f5fvDeH7Ej8CM66nIF+8yngt6HWaKh7k49Soy9nXurCi1o3qUbS3zWfrYeQDTB/Qj6kX6Ybhw4B+bOYoLKCC9H3Nu/leUTZ1JdRWkkn2ldcCamzrcf47KKXdAJllSlxAOkRgyHsGC/zRday5Qld9DyoM4/q/rUoy/CXh3jzOu3bHUVZeU+DEn8FInkPBFlu3+nW3Nw0mk6vCDiWg8CeJaxEwuHS3+z5RgY+YBR6V1Z1nxSOfoaPa4LASWxxdNp+VWTk7+4vzaou8v8PN+xo+KY2xsw6une2frhw05CTYOmQvsEhjhWjn0bmXPjpE1+kplmmkP3suftwTubK9Vq22qKmrBhpY4jvd5afdRA3wGjFAgcnTK2s4hY0/GPNIb0nErGMCRxWOOX64Z8RAC4oCXdklmEvcL8o0BfkNK4lUg9HTl+oPlQxdNo3Mg4Nv175e/1LDGzZen30MEjRUtmXSfiTVu1kK8W4txyV6BMKlbgk3lMwYCiusNy9fVfvvwMxv8Ynl6vxoByANLTWplvuj/nF9m2+PDtt1eiHPBr1oIfhCChQMBw6Aw0UulqTKZdfVvfG7VcfIqLG9bcldL/+pdWTLxLUy8Qq38heUIjh4XlzZxzQm19lLFlr8vdQ97rjZVOLf8nclzckbcD4wxXMidpX30sFd37Fv/GtwwhzhxGVAprjbg0gCAEeIgwCZyTV2Z1REEW8O4py0wsjeloKoMr6iCY6dP92H6Vw/oTyICIthibxjm/DfN9lVz8IqtqKYLUXfoKVMVQVVJOElGjrnnUt9T9wbgp8AyYKaGlqingHZU/uG2NTZSVqwHQTWkx9hxjkpWDaCg6Ckj5qebgBVbT3V3NNXMSiWSDdGV3hrtzla7J+duwPOToIg42ChPQOQjspnSlp1V+Gjdged7+8UN5CRAV7a5EdFNwCjEaBR27b3W890TE7g24NAP/mMDXRWrGoFPQI9ls/MWO2dWFAar/xcOIImbbpA3zgAAAABJRU5ErkJggg==);\n",
       "        }\n",
       "    </style>\n",
       "    <div>\n",
       "        <a href=\"https://bokeh.org\" target=\"_blank\" class=\"bk-notebook-logo\"></a>\n",
       "        <span id=\"c21c71c9-8c96-4353-987c-60170d437d1e\">Loading BokehJS ...</span>\n",
       "    </div>\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": "(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  const force = true;\n\n  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n    root._bokeh_onload_callbacks = [];\n    root._bokeh_is_loading = undefined;\n  }\n\nconst JS_MIME_TYPE = 'application/javascript';\n  const HTML_MIME_TYPE = 'text/html';\n  const EXEC_MIME_TYPE = 'application/vnd.bokehjs_exec.v0+json';\n  const CLASS_NAME = 'output_bokeh rendered_html';\n\n  /**\n   * Render data to the DOM node\n   */\n  function render(props, node) {\n    const script = document.createElement(\"script\");\n    node.appendChild(script);\n  }\n\n  /**\n   * Handle when an output is cleared or removed\n   */\n  function handleClearOutput(event, handle) {\n    const cell = handle.cell;\n\n    const id = cell.output_area._bokeh_element_id;\n    const server_id = cell.output_area._bokeh_server_id;\n    // Clean up Bokeh references\n    if (id != null && id in Bokeh.index) {\n      Bokeh.index[id].model.document.clear();\n      delete Bokeh.index[id];\n    }\n\n    if (server_id !== undefined) {\n      // Clean up Bokeh references\n      const cmd_clean = \"from bokeh.io.state import curstate; print(curstate().uuid_to_server['\" + server_id + \"'].get_sessions()[0].document.roots[0]._id)\";\n      cell.notebook.kernel.execute(cmd_clean, {\n        iopub: {\n          output: function(msg) {\n            const id = msg.content.text.trim();\n            if (id in Bokeh.index) {\n              Bokeh.index[id].model.document.clear();\n              delete Bokeh.index[id];\n            }\n          }\n        }\n      });\n      // Destroy server and session\n      const cmd_destroy = \"import bokeh.io.notebook as ion; ion.destroy_server('\" + server_id + \"')\";\n      cell.notebook.kernel.execute(cmd_destroy);\n    }\n  }\n\n  /**\n   * Handle when a new output is added\n   */\n  function handleAddOutput(event, handle) {\n    const output_area = handle.output_area;\n    const output = handle.output;\n\n    // limit handleAddOutput to display_data with EXEC_MIME_TYPE content only\n    if ((output.output_type != \"display_data\") || (!Object.prototype.hasOwnProperty.call(output.data, EXEC_MIME_TYPE))) {\n      return\n    }\n\n    const toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n\n    if (output.metadata[EXEC_MIME_TYPE][\"id\"] !== undefined) {\n      toinsert[toinsert.length - 1].firstChild.textContent = output.data[JS_MIME_TYPE];\n      // store reference to embed id on output_area\n      output_area._bokeh_element_id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n    }\n    if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n      const bk_div = document.createElement(\"div\");\n      bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n      const script_attrs = bk_div.children[0].attributes;\n      for (let i = 0; i < script_attrs.length; i++) {\n        toinsert[toinsert.length - 1].firstChild.setAttribute(script_attrs[i].name, script_attrs[i].value);\n        toinsert[toinsert.length - 1].firstChild.textContent = bk_div.children[0].textContent\n      }\n      // store reference to server id on output_area\n      output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n    }\n  }\n\n  function register_renderer(events, OutputArea) {\n\n    function append_mime(data, metadata, element) {\n      // create a DOM node to render to\n      const toinsert = this.create_output_subarea(\n        metadata,\n        CLASS_NAME,\n        EXEC_MIME_TYPE\n      );\n      this.keyboard_manager.register_events(toinsert);\n      // Render to node\n      const props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n      render(props, toinsert[toinsert.length - 1]);\n      element.append(toinsert);\n      return toinsert\n    }\n\n    /* Handle when an output is cleared or removed */\n    events.on('clear_output.CodeCell', handleClearOutput);\n    events.on('delete.Cell', handleClearOutput);\n\n    /* Handle when a new output is added */\n    events.on('output_added.OutputArea', handleAddOutput);\n\n    /**\n     * Register the mime type and append_mime function with output_area\n     */\n    OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n      /* Is output safe? */\n      safe: true,\n      /* Index of renderer in `output_area.display_order` */\n      index: 0\n    });\n  }\n\n  // register the mime type if in Jupyter Notebook environment and previously unregistered\n  if (root.Jupyter !== undefined) {\n    const events = require('base/js/events');\n    const OutputArea = require('notebook/js/outputarea').OutputArea;\n\n    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n      register_renderer(events, OutputArea);\n    }\n  }\n  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  const NB_LOAD_WARNING = {'data': {'text/html':\n     \"<div style='background-color: #fdd'>\\n\"+\n     \"<p>\\n\"+\n     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n     \"</p>\\n\"+\n     \"<ul>\\n\"+\n     \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n     \"<li>use INLINE resources instead, as so:</li>\\n\"+\n     \"</ul>\\n\"+\n     \"<code>\\n\"+\n     \"from bokeh.resources import INLINE\\n\"+\n     \"output_notebook(resources=INLINE)\\n\"+\n     \"</code>\\n\"+\n     \"</div>\"}};\n\n  function display_loaded() {\n    const el = document.getElementById(\"c21c71c9-8c96-4353-987c-60170d437d1e\");\n    if (el != null) {\n      el.textContent = \"BokehJS is loading...\";\n    }\n    if (root.Bokeh !== undefined) {\n      if (el != null) {\n        el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n      }\n    } else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(display_loaded, 100)\n    }\n  }\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n\n    root._bokeh_onload_callbacks.push(callback);\n    if (root._bokeh_is_loading > 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls == null || js_urls.length === 0) {\n      run_callbacks();\n      return null;\n    }\n    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n    root._bokeh_is_loading = css_urls.length + js_urls.length;\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n\n    function on_error(url) {\n      console.error(\"failed to load \" + url);\n    }\n\n    for (let i = 0; i < css_urls.length; i++) {\n      const url = css_urls[i];\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error.bind(null, url);\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }\n\n    for (let i = 0; i < js_urls.length; i++) {\n      const url = js_urls[i];\n      const element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error.bind(null, url);\n      element.async = false;\n      element.src = url;\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  const js_urls = [\"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-mathjax-3.2.0.min.js\", \"https://unpkg.com/@holoviz/panel@1.1.0/dist/panel.min.js\"];\n  const css_urls = [];\n\n  const inline_js = [    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\nfunction(Bokeh) {\n    }\n  ];\n\n  function run_inline_js() {\n    if (root.Bokeh !== undefined || force === true) {\n          for (let i = 0; i < inline_js.length; i++) {\n      inline_js[i].call(root, root.Bokeh);\n    }\nif (force === true) {\n        display_loaded();\n      }} else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    } else if (force !== true) {\n      const cell = $(document.getElementById(\"c21c71c9-8c96-4353-987c-60170d437d1e\")).parents('.cell').data().cell;\n      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n    }\n  }\n\n  if (root._bokeh_is_loading === 0) {\n    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n    run_inline_js();\n  } else {\n    load_libs(css_urls, js_urls, function() {\n      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n      run_inline_js();\n    });\n  }\n}(window));",
      "application/vnd.bokehjs_load.v0+json": ""
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": "(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  var force = true;\n  var py_version = '3.2.0'.replace('rc', '-rc.');\n  var is_dev = py_version.indexOf(\"+\") !== -1 || py_version.indexOf(\"-\") !== -1;\n  var reloading = false;\n  var Bokeh = root.Bokeh;\n  var bokeh_loaded = Bokeh != null && (Bokeh.version === py_version || (Bokeh.versions !== undefined && Bokeh.versions.has(py_version)));\n\n  if (typeof (root._bokeh_timeout) === \"undefined\" || force) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks;\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, js_modules, js_exports, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n    if (js_modules == null) js_modules = [];\n    if (js_exports == null) js_exports = {};\n\n    root._bokeh_onload_callbacks.push(callback);\n\n    if (root._bokeh_is_loading > 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls.length === 0 && js_modules.length === 0 && Object.keys(js_exports).length === 0) {\n      run_callbacks();\n      return null;\n    }\n    if (!reloading) {\n      console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n    }\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n    window._bokeh_on_load = on_load\n\n    function on_error() {\n      console.error(\"failed to load \" + url);\n    }\n\n    var skip = [];\n    if (window.requirejs) {\n      window.requirejs.config({'packages': {}, 'paths': {'ace': '//cdnjs.cloudflare.com/ajax/libs/ace/1.4.7', 'jspanel': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/jspanel', 'jspanel-modal': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/modal/jspanel.modal', 'jspanel-tooltip': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/tooltip/jspanel.tooltip', 'jspanel-hint': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/hint/jspanel.hint', 'jspanel-layout': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/layout/jspanel.layout', 'jspanel-contextmenu': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/contextmenu/jspanel.contextmenu', 'jspanel-dock': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/dock/jspanel.dock', 'gridstack': 'https://cdn.jsdelivr.net/npm/gridstack@7.2.3/dist/gridstack-all', 'notyf': 'https://cdn.jsdelivr.net/npm/notyf@3/notyf.min'}, 'shim': {'ace/ext-language_tools': {'deps': ['ace/ace']}, 'ace/ext-modelist': {'deps': ['ace/ace']}, 'jspanel': {'exports': 'jsPanel'}, 'gridstack': {'exports': 'GridStack'}}});\n      require([\"ace/ace\"], function(ace) {\n\twindow.ace = ace\n\ton_load()\n      })\n      require([\"ace/ext-language_tools\"], function() {\n\ton_load()\n      })\n      require([\"ace/ext-modelist\"], function() {\n\ton_load()\n      })\n      require([\"jspanel\"], function(jsPanel) {\n\twindow.jsPanel = jsPanel\n\ton_load()\n      })\n      require([\"jspanel-modal\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-tooltip\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-hint\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-layout\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-contextmenu\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-dock\"], function() {\n\ton_load()\n      })\n      require([\"gridstack\"], function(GridStack) {\n\twindow.GridStack = GridStack\n\ton_load()\n      })\n      require([\"notyf\"], function() {\n\ton_load()\n      })\n      root._bokeh_is_loading = css_urls.length + 12;\n    } else {\n      root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length + Object.keys(js_exports).length;\n    }\n\n    var existing_stylesheets = []\n    var links = document.getElementsByTagName('link')\n    for (var i = 0; i < links.length; i++) {\n      var link = links[i]\n      if (link.href != null) {\n\texisting_stylesheets.push(link.href)\n      }\n    }\n    for (var i = 0; i < css_urls.length; i++) {\n      var url = css_urls[i];\n      if (existing_stylesheets.indexOf(url) !== -1) {\n\ton_load()\n\tcontinue;\n      }\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }    if (((window['ace'] !== undefined) && (!(window['ace'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js', 'https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js', 'https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['jsPanel'] !== undefined) && (!(window['jsPanel'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/jspanel.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/modal/jspanel.modal.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/tooltip/jspanel.tooltip.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/hint/jspanel.hint.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/layout/jspanel.layout.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/contextmenu/jspanel.contextmenu.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/dock/jspanel.dock.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['GridStack'] !== undefined) && (!(window['GridStack'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/1.1.0/dist/bundled/gridstack/gridstack@7.2.3/dist/gridstack-all.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['Notyf'] !== undefined) && (!(window['Notyf'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/1.1.0/dist/bundled/notificationarea/notyf@3/notyf.min.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    var existing_scripts = []\n    var scripts = document.getElementsByTagName('script')\n    for (var i = 0; i < scripts.length; i++) {\n      var script = scripts[i]\n      if (script.src != null) {\n\texisting_scripts.push(script.src)\n      }\n    }\n    for (var i = 0; i < js_urls.length; i++) {\n      var url = js_urls[i];\n      if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    for (var i = 0; i < js_modules.length; i++) {\n      var url = js_modules[i];\n      if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      element.type = \"module\";\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    for (const name in js_exports) {\n      var url = js_exports[name];\n      if (skip.indexOf(url) >= 0 || root[name] != null) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onerror = on_error;\n      element.async = false;\n      element.type = \"module\";\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      element.textContent = `\n      import ${name} from \"${url}\"\n      window.${name} = ${name}\n      window._bokeh_on_load()\n      `\n      document.head.appendChild(element);\n    }\n    if (!js_urls.length && !js_modules.length) {\n      on_load()\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  var js_urls = [\"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.2.0.min.js\", \"https://cdn.holoviz.org/panel/1.1.0/dist/panel.min.js\"];\n  var js_modules = [];\n  var js_exports = {};\n  var css_urls = [];\n  var inline_js = [    function(Bokeh) {\n      inject_raw_css(\"/*\\n ~ CML // Creative Machine Learning ~\\n mml.css : CSS styling information for Panel and Bokeh\\n \\n This file defines the main CSS styling information for the CML course\\n \\n Author               :  Philippe Esling\\n                        <esling@ircam.fr>\\n*/\\n\\nbody {\\n  display: flex;\\n  height: 100vh;\\n  margin: 0px;\\n  overflow-x: hidden;\\n  overflow-y: hidden;\\n}\\n\\n.bk-root .bk, .bk-root .bk:before, .bk-root .bk:after {\\n  font-family: \\\"Josefin Sans\\\";\\n}\\n\\nimg {\\n  max-width: 100%;\\n}\\n\\n#container {\\n  padding:0px;\\n  height:100vh;\\n  width: 100vw;\\n  max-width: 100vw;\\n}\\n\\n#sidebar .mdc-list {\\n  padding-left: 5px;\\n  padding-right: 5px;\\n}\\n\\n.mdc-drawer-app-content {\\n  flex: auto;\\n  position: relative;\\n  overflow: hidden;\\n}\\n\\n.mdc-drawer {\\n  background: #FAFAFA; /* GRAY 50 */\\n}\\n\\n.mdc-drawer-app-content {\\n  margin-left: 0 !important;\\n}\\n\\n.title-bar {\\n  display: contents;\\n  justify-content: center;\\n  align-content: center;\\n  width: 100%;\\n}\\n\\n.mdc-top-app-bar .bk-menu {\\n  color: black\\n}\\n\\n.app-header {\\n  display: contents;\\n  padding-left: 10px;\\n  font-size: 1.25em;\\n}\\n\\nimg.app-logo {\\n  padding-right: 10px;\\n  font-size: 28px;\\n  height: 30px;\\n  max-width: inherit;\\n  padding-top: 12px;\\n  padding-bottom: 6px;\\n}\\n\\n#app-title {\\n  padding-right: 12px;\\n  padding-left: 12px;\\n}\\n\\n.title {\\n  font-family: \\\"Josefin Sans\\\";\\n  color: #fff;\\n  text-decoration: none;\\n  text-decoration-line: none;\\n  text-decoration-style: initial;\\n  text-decoration-color: initial;\\n  font-weight: 400;\\n  font-size: 2em;\\n  line-height: 2em;\\n  white-space: nowrap;\\n}\\n\\n.main-content {\\n  overflow-y: scroll;\\n  overflow-x: auto;\\n}\\n\\n#header {\\n  position: absolute;\\n  z-index: 7;\\n}\\n\\n#header-items {\\n  width: 100%;\\n  margin-left:15px;\\n}\\n\\n.pn-busy-container {\\n  align-items: center;\\n  justify-content: center;\\n  display: flex;\\n}\\n\\n.mdc-drawer__content {\\n  overflow-x: hidden;\\n}\\n.mdc-drawer__content, .main-content {\\n  padding: 12px;\\n}\\n\\n.main-content {\\n  height: calc(100vh - 88px);\\n  max-height: calc(100vh - 88px);\\n  padding-right: 32px;\\n}\\n\\nbutton.mdc-button.mdc-card-button {\\n  color: transparent;\\n  height: 50px;\\n}\\n\\np.mdc-button {\\n  display: none;\\n}\\n\\ndiv.mdc-card {\\n  border-radius: 0px\\n}\\n\\n.mdc-card .card-header {\\n  display: flex;\\n}\\n\\n.mdc-card-title {\\n  font-family: \\\"Josefin Sans\\\";\\n  font-weight: bold;\\n  align-items: center;\\n  display: flex !important;\\n  position: relative !important;\\n}\\n\\n.mdc-card-title:nth-child(2) {\\n  margin-left: -1.4em;\\n}\\n\\n.pn-modal {\\n  overflow-y: scroll;\\n  width: 100%;\\n  display: none;\\n  position: absolute;\\n  top: 0;\\n  left: 0;\\n}\\n\\n.pn-modal-content {\\n  font-family: \\\"Josefin Sans\\\";\\n  background-color: #0e0e0e;\\n  margin: auto;\\n  margin-top: 25px;\\n  margin-bottom: 25px;\\n  padding: 15px 20px 20px 20px;\\n  border: 1px solid #888;\\n  width: 80% !important;\\n}\\n\\n.pn-modal-close {\\n  position: absolute;\\n  right: 25px;\\n  z-index: 100;\\n}\\n\\n.pn-modal-close:hover,\\n.pn-modal-close:focus {\\n  color: #000;\\n  text-decoration: none;\\n  cursor: pointer;\\n}\\n\\n.custom_button_bokeh button.bk-btn.bk-btn-default {\\n    font-size:48pt;\\n    background-color: #05b7ff;\\n    border-color: #05b7ff;\\n}\");\n    },    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\nfunction(Bokeh) {} // ensure no trailing comma for IE\n  ];\n\n  function run_inline_js() {\n    if ((root.Bokeh !== undefined) || (force === true)) {\n      for (var i = 0; i < inline_js.length; i++) {\n        inline_js[i].call(root, root.Bokeh);\n      }\n      // Cache old bokeh versions\n      if (Bokeh != undefined && !reloading) {\n\tvar NewBokeh = root.Bokeh;\n\tif (Bokeh.versions === undefined) {\n\t  Bokeh.versions = new Map();\n\t}\n\tif (NewBokeh.version !== Bokeh.version) {\n\t  Bokeh.versions.set(NewBokeh.version, NewBokeh)\n\t}\n\troot.Bokeh = Bokeh;\n      }} else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    }\n    root._bokeh_is_initializing = false\n  }\n\n  function load_or_wait() {\n    // Implement a backoff loop that tries to ensure we do not load multiple\n    // versions of Bokeh and its dependencies at the same time.\n    // In recent versions we use the root._bokeh_is_initializing flag\n    // to determine whether there is an ongoing attempt to initialize\n    // bokeh, however for backward compatibility we also try to ensure\n    // that we do not start loading a newer (Panel>=1.0 and Bokeh>3) version\n    // before older versions are fully initialized.\n    if (root._bokeh_is_initializing && Date.now() > root._bokeh_timeout) {\n      root._bokeh_is_initializing = false;\n      root._bokeh_onload_callbacks = undefined;\n      console.log(\"Bokeh: BokehJS was loaded multiple times but one version failed to initialize.\");\n      load_or_wait();\n    } else if (root._bokeh_is_initializing || (typeof root._bokeh_is_initializing === \"undefined\" && root._bokeh_onload_callbacks !== undefined)) {\n      setTimeout(load_or_wait, 100);\n    } else {\n      Bokeh = root.Bokeh;\n      bokeh_loaded = Bokeh != null && (Bokeh.version === py_version || (Bokeh.versions !== undefined && Bokeh.versions.has(py_version)));\n      root._bokeh_is_initializing = true\n      root._bokeh_onload_callbacks = []\n      if (!reloading && (!bokeh_loaded || is_dev)) {\n\troot.Bokeh = undefined;\n      }\n      load_libs(css_urls, js_urls, js_modules, js_exports, function() {\n\tconsole.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n\trun_inline_js();\n      });\n    }\n  }\n  // Give older versions of the autoload script a head-start to ensure\n  // they initialize before we start loading newer version.\n  setTimeout(load_or_wait, 100)\n}(window));",
      "application/vnd.holoviews_load.v0+json": ""
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": "\nif ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {\n  window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}\n}\n\n\n    function JupyterCommManager() {\n    }\n\n    JupyterCommManager.prototype.register_target = function(plot_id, comm_id, msg_handler) {\n      if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        comm_manager.register_target(comm_id, function(comm) {\n          comm.on_msg(msg_handler);\n        });\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        window.PyViz.kernels[plot_id].registerCommTarget(comm_id, function(comm) {\n          comm.onMsg = msg_handler;\n        });\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        google.colab.kernel.comms.registerTarget(comm_id, (comm) => {\n          var messages = comm.messages[Symbol.asyncIterator]();\n          function processIteratorResult(result) {\n            var message = result.value;\n            console.log(message)\n            var content = {data: message.data, comm_id};\n            var buffers = []\n            for (var buffer of message.buffers || []) {\n              buffers.push(new DataView(buffer))\n            }\n            var metadata = message.metadata || {};\n            var msg = {content, buffers, metadata}\n            msg_handler(msg);\n            return messages.next().then(processIteratorResult);\n          }\n          return messages.next().then(processIteratorResult);\n        })\n      }\n    }\n\n    JupyterCommManager.prototype.get_client_comm = function(plot_id, comm_id, msg_handler) {\n      if (comm_id in window.PyViz.comms) {\n        return window.PyViz.comms[comm_id];\n      } else if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        var comm = comm_manager.new_comm(comm_id, {}, {}, {}, comm_id);\n        if (msg_handler) {\n          comm.on_msg(msg_handler);\n        }\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        var comm = window.PyViz.kernels[plot_id].connectToComm(comm_id);\n        comm.open();\n        if (msg_handler) {\n          comm.onMsg = msg_handler;\n        }\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        var comm_promise = google.colab.kernel.comms.open(comm_id)\n        comm_promise.then((comm) => {\n          window.PyViz.comms[comm_id] = comm;\n          if (msg_handler) {\n            var messages = comm.messages[Symbol.asyncIterator]();\n            function processIteratorResult(result) {\n              var message = result.value;\n              var content = {data: message.data};\n              var metadata = message.metadata || {comm_id};\n              var msg = {content, metadata}\n              msg_handler(msg);\n              return messages.next().then(processIteratorResult);\n            }\n            return messages.next().then(processIteratorResult);\n          }\n        }) \n        var sendClosure = (data, metadata, buffers, disposeOnDone) => {\n          return comm_promise.then((comm) => {\n            comm.send(data, metadata, buffers, disposeOnDone);\n          });\n        };\n        var comm = {\n          send: sendClosure\n        };\n      }\n      window.PyViz.comms[comm_id] = comm;\n      return comm;\n    }\n    window.PyViz.comm_manager = new JupyterCommManager();\n    \n\n\nvar JS_MIME_TYPE = 'application/javascript';\nvar HTML_MIME_TYPE = 'text/html';\nvar EXEC_MIME_TYPE = 'application/vnd.holoviews_exec.v0+json';\nvar CLASS_NAME = 'output';\n\n/**\n * Render data to the DOM node\n */\nfunction render(props, node) {\n  var div = document.createElement(\"div\");\n  var script = document.createElement(\"script\");\n  node.appendChild(div);\n  node.appendChild(script);\n}\n\n/**\n * Handle when a new output is added\n */\nfunction handle_add_output(event, handle) {\n  var output_area = handle.output_area;\n  var output = handle.output;\n  if ((output.data == undefined) || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n    return\n  }\n  var id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n  var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n  if (id !== undefined) {\n    var nchildren = toinsert.length;\n    var html_node = toinsert[nchildren-1].children[0];\n    html_node.innerHTML = output.data[HTML_MIME_TYPE];\n    var scripts = [];\n    var nodelist = html_node.querySelectorAll(\"script\");\n    for (var i in nodelist) {\n      if (nodelist.hasOwnProperty(i)) {\n        scripts.push(nodelist[i])\n      }\n    }\n\n    scripts.forEach( function (oldScript) {\n      var newScript = document.createElement(\"script\");\n      var attrs = [];\n      var nodemap = oldScript.attributes;\n      for (var j in nodemap) {\n        if (nodemap.hasOwnProperty(j)) {\n          attrs.push(nodemap[j])\n        }\n      }\n      attrs.forEach(function(attr) { newScript.setAttribute(attr.name, attr.value) });\n      newScript.appendChild(document.createTextNode(oldScript.innerHTML));\n      oldScript.parentNode.replaceChild(newScript, oldScript);\n    });\n    if (JS_MIME_TYPE in output.data) {\n      toinsert[nchildren-1].children[1].textContent = output.data[JS_MIME_TYPE];\n    }\n    output_area._hv_plot_id = id;\n    if ((window.Bokeh !== undefined) && (id in Bokeh.index)) {\n      window.PyViz.plot_index[id] = Bokeh.index[id];\n    } else {\n      window.PyViz.plot_index[id] = null;\n    }\n  } else if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n    var bk_div = document.createElement(\"div\");\n    bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n    var script_attrs = bk_div.children[0].attributes;\n    for (var i = 0; i < script_attrs.length; i++) {\n      toinsert[toinsert.length - 1].childNodes[1].setAttribute(script_attrs[i].name, script_attrs[i].value);\n    }\n    // store reference to server id on output_area\n    output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n  }\n}\n\n/**\n * Handle when an output is cleared or removed\n */\nfunction handle_clear_output(event, handle) {\n  var id = handle.cell.output_area._hv_plot_id;\n  var server_id = handle.cell.output_area._bokeh_server_id;\n  if (((id === undefined) || !(id in PyViz.plot_index)) && (server_id !== undefined)) { return; }\n  var comm = window.PyViz.comm_manager.get_client_comm(\"hv-extension-comm\", \"hv-extension-comm\", function () {});\n  if (server_id !== null) {\n    comm.send({event_type: 'server_delete', 'id': server_id});\n    return;\n  } else if (comm !== null) {\n    comm.send({event_type: 'delete', 'id': id});\n  }\n  delete PyViz.plot_index[id];\n  if ((window.Bokeh !== undefined) & (id in window.Bokeh.index)) {\n    var doc = window.Bokeh.index[id].model.document\n    doc.clear();\n    const i = window.Bokeh.documents.indexOf(doc);\n    if (i > -1) {\n      window.Bokeh.documents.splice(i, 1);\n    }\n  }\n}\n\n/**\n * Handle kernel restart event\n */\nfunction handle_kernel_cleanup(event, handle) {\n  delete PyViz.comms[\"hv-extension-comm\"];\n  window.PyViz.plot_index = {}\n}\n\n/**\n * Handle update_display_data messages\n */\nfunction handle_update_output(event, handle) {\n  handle_clear_output(event, {cell: {output_area: handle.output_area}})\n  handle_add_output(event, handle)\n}\n\nfunction register_renderer(events, OutputArea) {\n  function append_mime(data, metadata, element) {\n    // create a DOM node to render to\n    var toinsert = this.create_output_subarea(\n    metadata,\n    CLASS_NAME,\n    EXEC_MIME_TYPE\n    );\n    this.keyboard_manager.register_events(toinsert);\n    // Render to node\n    var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n    render(props, toinsert[0]);\n    element.append(toinsert);\n    return toinsert\n  }\n\n  events.on('output_added.OutputArea', handle_add_output);\n  events.on('output_updated.OutputArea', handle_update_output);\n  events.on('clear_output.CodeCell', handle_clear_output);\n  events.on('delete.Cell', handle_clear_output);\n  events.on('kernel_ready.Kernel', handle_kernel_cleanup);\n\n  OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n    safe: true,\n    index: 0\n  });\n}\n\nif (window.Jupyter !== undefined) {\n  try {\n    var events = require('base/js/events');\n    var OutputArea = require('notebook/js/outputarea').OutputArea;\n    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n      register_renderer(events, OutputArea);\n    }\n  } catch(err) {\n  }\n}\n",
      "application/vnd.holoviews_load.v0+json": ""
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>*[data-root-id],\n",
       "*[data-root-id] > * {\n",
       "  box-sizing: border-box;\n",
       "  font-family: var(--jp-ui-font-family);\n",
       "  font-size: var(--jp-ui-font-size1);\n",
       "  color: var(--vscode-editor-foreground, var(--jp-ui-font-color1));\n",
       "}\n",
       "\n",
       "/* Override VSCode background color */\n",
       ".cell-output-ipywidget-background:has(> .cell-output-ipywidget-background\n",
       "    > .lm-Widget\n",
       "    > *[data-root-id]),\n",
       ".cell-output-ipywidget-background:has(> .lm-Widget > *[data-root-id]) {\n",
       "  background-color: transparent !important;\n",
       "}\n",
       "</style>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.holoviews_exec.v0+json": "",
      "text/html": [
       "<div id='299f59bc-53e7-4733-bf61-5d22e2748f68'>\n",
       "  <div id=\"d18c7453-1cf0-4100-9f18-4ba649670f4c\" data-root-id=\"299f59bc-53e7-4733-bf61-5d22e2748f68\" style=\"display: contents;\"></div>\n",
       "</div>\n",
       "<script type=\"application/javascript\">(function(root) {\n",
       "  var docs_json = {\"fb56d172-d496-4701-8c1a-514c5b6c4809\":{\"version\":\"3.2.0\",\"title\":\"Bokeh Application\",\"roots\":[{\"type\":\"object\",\"name\":\"panel.models.browser.BrowserInfo\",\"id\":\"299f59bc-53e7-4733-bf61-5d22e2748f68\"},{\"type\":\"object\",\"name\":\"panel.models.comm_manager.CommManager\",\"id\":\"d7d93dd6-f275-4c8e-a05c-6a9b7b08d7d3\",\"attributes\":{\"plot_id\":\"299f59bc-53e7-4733-bf61-5d22e2748f68\",\"comm_id\":\"f9e4fed06f03428e90f709721a303593\",\"client_comm_id\":\"7328429f512e4dba8305f531aa4f3d7f\"}}],\"defs\":[{\"type\":\"model\",\"name\":\"ReactiveHTML1\"},{\"type\":\"model\",\"name\":\"FlexBox1\",\"properties\":[{\"name\":\"align_content\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"align_items\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"flex_direction\",\"kind\":\"Any\",\"default\":\"row\"},{\"name\":\"flex_wrap\",\"kind\":\"Any\",\"default\":\"wrap\"},{\"name\":\"justify_content\",\"kind\":\"Any\",\"default\":\"flex-start\"}]},{\"type\":\"model\",\"name\":\"FloatPanel1\",\"properties\":[{\"name\":\"config\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"contained\",\"kind\":\"Any\",\"default\":true},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"right-top\"},{\"name\":\"offsetx\",\"kind\":\"Any\",\"default\":null},{\"name\":\"offsety\",\"kind\":\"Any\",\"default\":null},{\"name\":\"theme\",\"kind\":\"Any\",\"default\":\"primary\"},{\"name\":\"status\",\"kind\":\"Any\",\"default\":\"normalized\"}]},{\"type\":\"model\",\"name\":\"GridStack1\",\"properties\":[{\"name\":\"mode\",\"kind\":\"Any\",\"default\":\"warn\"},{\"name\":\"ncols\",\"kind\":\"Any\",\"default\":null},{\"name\":\"nrows\",\"kind\":\"Any\",\"default\":null},{\"name\":\"allow_resize\",\"kind\":\"Any\",\"default\":true},{\"name\":\"allow_drag\",\"kind\":\"Any\",\"default\":true},{\"name\":\"state\",\"kind\":\"Any\",\"default\":[]}]},{\"type\":\"model\",\"name\":\"drag1\",\"properties\":[{\"name\":\"slider_width\",\"kind\":\"Any\",\"default\":5},{\"name\":\"slider_color\",\"kind\":\"Any\",\"default\":\"black\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":50}]},{\"type\":\"model\",\"name\":\"click1\",\"properties\":[{\"name\":\"terminal_output\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"debug_name\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"clears\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"FastWrapper1\",\"properties\":[{\"name\":\"object\",\"kind\":\"Any\",\"default\":null},{\"name\":\"style\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"NotificationAreaBase1\",\"properties\":[{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"NotificationArea1\",\"properties\":[{\"name\":\"notifications\",\"kind\":\"Any\",\"default\":[]},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0},{\"name\":\"types\",\"kind\":\"Any\",\"default\":[{\"type\":\"map\",\"entries\":[[\"type\",\"warning\"],[\"background\",\"#ffc107\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-exclamation-triangle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]},{\"type\":\"map\",\"entries\":[[\"type\",\"info\"],[\"background\",\"#007bff\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-info-circle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]}]}]},{\"type\":\"model\",\"name\":\"Notification\",\"properties\":[{\"name\":\"background\",\"kind\":\"Any\",\"default\":null},{\"name\":\"duration\",\"kind\":\"Any\",\"default\":3000},{\"name\":\"icon\",\"kind\":\"Any\",\"default\":null},{\"name\":\"message\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"notification_type\",\"kind\":\"Any\",\"default\":null},{\"name\":\"_destroyed\",\"kind\":\"Any\",\"default\":false}]},{\"type\":\"model\",\"name\":\"TemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"BootstrapTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"MaterialTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]}]}};\n",
       "  var render_items = [{\"docid\":\"fb56d172-d496-4701-8c1a-514c5b6c4809\",\"roots\":{\"299f59bc-53e7-4733-bf61-5d22e2748f68\":\"d18c7453-1cf0-4100-9f18-4ba649670f4c\"},\"root_ids\":[\"299f59bc-53e7-4733-bf61-5d22e2748f68\"]}];\n",
       "  var docs = Object.values(docs_json)\n",
       "  if (!docs) {\n",
       "    return\n",
       "  }\n",
       "  const py_version = docs[0].version.replace('rc', '-rc.')\n",
       "  const is_dev = py_version.indexOf(\"+\") !== -1 || py_version.indexOf(\"-\") !== -1\n",
       "  function embed_document(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
       "    for (const render_item of render_items) {\n",
       "      for (const root_id of render_item.root_ids) {\n",
       "\tconst id_el = document.getElementById(root_id)\n",
       "\tif (id_el.children.length && (id_el.children[0].className === 'bk-root')) {\n",
       "\t  const root_el = id_el.children[0]\n",
       "\t  root_el.id = root_el.id + '-rendered'\n",
       "\t}\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "  function get_bokeh(root) {\n",
       "    if (root.Bokeh === undefined) {\n",
       "      return null\n",
       "    } else if (root.Bokeh.version !== py_version && !is_dev) {\n",
       "      if (root.Bokeh.versions === undefined || !root.Bokeh.versions.has(py_version)) {\n",
       "\treturn null\n",
       "      }\n",
       "      return root.Bokeh.versions.get(py_version);\n",
       "    } else if (root.Bokeh.version === py_version) {\n",
       "      return root.Bokeh\n",
       "    }\n",
       "    return null\n",
       "  }\n",
       "  function is_loaded(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    return (Bokeh != null && Bokeh.Panel !== undefined)\n",
       "  }\n",
       "  if (is_loaded(root)) {\n",
       "    embed_document(root);\n",
       "  } else {\n",
       "    var attempts = 0;\n",
       "    var timer = setInterval(function(root) {\n",
       "      if (is_loaded(root)) {\n",
       "        clearInterval(timer);\n",
       "        embed_document(root);\n",
       "      } else if (document.readyState == \"complete\") {\n",
       "        attempts++;\n",
       "        if (attempts > 200) {\n",
       "          clearInterval(timer);\n",
       "\t  var Bokeh = get_bokeh(root)\n",
       "\t  if (Bokeh == null || Bokeh.Panel == null) {\n",
       "            console.warn(\"Panel: ERROR: Unable to run Panel code because Bokeh or Panel library is missing\");\n",
       "\t  } else {\n",
       "\t    console.warn(\"Panel: WARNING: Attempting to render but not all required libraries could be resolved.\")\n",
       "\t    embed_document(root)\n",
       "\t  }\n",
       "        }\n",
       "      }\n",
       "    }, 25, root)\n",
       "  }\n",
       "})(window);</script>"
      ]
     },
     "metadata": {
      "application/vnd.holoviews_exec.v0+json": {
       "id": "299f59bc-53e7-4733-bf61-5d22e2748f68"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Base imports\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cml.plot import initialize_bokeh\n",
    "from cml.panel import initialize_panel\n",
    "from jupyterthemes.stylefx import set_nb_theme\n",
    "from bokeh.io import show\n",
    "initialize_bokeh()\n",
    "initialize_panel()\n",
    "set_nb_theme(\"onedork\")\n",
    "rng = np.random.RandomState(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"convolution\"></a>\n",
    "## Convolutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In purely mathematical terms, convolution is a function derived from two given functions by integration which expresses how the shape of one is modified by the other. In simpler (discrete) terms, the convolution product of a matrix by a smaller one can be seen as _filtering_ the large matrix. Hence we slide the small matrix over the large one and compute local products at each position. Therefore the convolution operator $\\star$ computes at each position $n$\n",
    "\n",
    "$$\n",
    "(f \\star g)[n]=\\sum _{m=-M}^{M}f[n-m]g[m].\n",
    "$$\n",
    "\n",
    "An example of this operation is shown here\n",
    "\n",
    "<center>\n",
    "<img src=\"images/02_convolution.png\" align=\"center\"/>\n",
    "</center>\n",
    "\n",
    "This operation can be used to _filter_ the image (as in the _gaussian blur_ operator), or _detect_ features (such as edges). \n",
    "\n",
    "Given a 32x32 image with RGB channels, we can represent it as a tensor of shape `(3, 32, 32)` which is (channels, height, width). When we perform convolution, we need a filter that has the same channel depth as the image. For example, we can use a 5x5 filter which is of shape `(3, 5, 5)` and slide it across the image left to right, top to bottom with a stride of 1 to perform convolution. We are going to perform this in numpy, depending on a certain amount of parameters, which define the behavior of our convolution\n",
    "\n",
    "* `height` and `width`: spatial extend of the filters\n",
    "* `stride`: stride size (number of steps to jump to the next position)\n",
    "* `padding`: amount of padding (adding zeros in the original matrix)\n",
    "\n",
    "Here note that we define a simple _horizontal line detector_ kernel, but we will display other kernels that you can toy off with later on "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Properties of our kernel\n",
    "pad = 2\n",
    "stride = 1\n",
    "height, width = 3, 3\n",
    "# Our convolution kernel\n",
    "weight = [\n",
    "[1, 1, 1],\n",
    "[1, 1.5, 1],\n",
    "[1, 1, 1],\n",
    "]\n",
    "weight = np.array(weight)\n",
    "# Final kernel (replicated over channels)\n",
    "kernel = np.repeat(weight[np.newaxis, :], 3, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next up we need to import some image on which to apply our convolution operation. Here, we display a small code to retrieve an image directly from a URL. You can replace the desired image by the URL of your choosing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8133a99a8f75406cae8354254aa2c272",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'4a6f556e-76e8-4c63-839c-3209b23c68d6': {'version…"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from PIL import Image\n",
    "import requests\n",
    "from io import BytesIO\n",
    "from cml.plot import image, center_plot, cml_figure\n",
    "# URL to load our example image from\n",
    "url = \"https://thumbnailer.mixcloud.com/unsafe/300x300/extaudio/6/b/a/b/d576-35c3-48a9-a01e-7dc4f0255e62.jpg\"\n",
    "response = requests.get(url)\n",
    "img = np.array(Image.open(BytesIO(response.content)))\n",
    "img = img.transpose(2, 0, 1)[:, :]\n",
    "plot = center_plot(image(img, title=\"Example image\"))\n",
    "plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_w, x_h = img.shape[1], img.shape[2]\n",
    "# Padding the original image\n",
    "x_pad = np.pad(img, pad_width=((0, 0,), (pad, pad), (pad, pad)), mode='constant', constant_values=0)\n",
    "# We can expect the output size to be\n",
    "h_out = int(1 + (x_h + 2 * pad - height) / stride)\n",
    "w_out = int(1 + (x_w + 2 * pad - width) / stride)\n",
    "# So we will store our result in\n",
    "y = np.zeros((1, h_out, w_out))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing the convolution itself can be performed by using the following loop (which amounts to _slide_ our kernel across the large matrix). As we have seen in the course, it corresponds to computing the convolution equation\n",
    "\n",
    "$$\n",
    "(f \\star g)[n]=\\sum _{m=-M}^{M}f[n-m]g[m].\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sliding the kernel across the matrix\n",
    "weight = [\n",
    "[-1, -2, -1],\n",
    "[0, 0, 0],\n",
    "[1, 2, 1],\n",
    "]\n",
    "for h in range(h_out):\n",
    "    for w in range(w_out):\n",
    "        i, j = h * stride, w * stride\n",
    "        conv_sum = np.sum(x_pad[:, i:i+height, j:j+width] * weight)\n",
    "        y[0, h, w] = conv_sum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can witness the effect of this operation with the following plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "41638d0d6bdb47a89bde54df91f98ce4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'8cb35884-4703-483d-b6e2-aa8975ccec4a': {'version…"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bokeh.layouts import row\n",
    "plot = center_plot(row(image(img, title=\"Input\"), image(y, title=\"Convolved\")))\n",
    "plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we discussed previously, depending on the content (weights) of the kernel, different features are detected by different types of convolutional kernels, we outline here several kernels that you can play with (or even try your own) by modifying the previous code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gaussian blur\n",
    "weight = [\n",
    "[1, 1, 1],\n",
    "[1, 3, 1],\n",
    "[1, 1, 1],\n",
    "]\n",
    "# Sharpen\n",
    "weight = [\n",
    "[0, -1, 0],\n",
    "[-1, 5, -1],\n",
    "[0, -1, 0],\n",
    "]\n",
    "# Outline\n",
    "weight = [\n",
    "[-1, -1, -1],\n",
    "[-1, 8, -1],\n",
    "[-1, -1, -1],\n",
    "]\n",
    "# Detect vertical lines\n",
    "weight = [\n",
    "[-1, 0, 1],\n",
    "[-2, 0, 2],\n",
    "[-1, 0, 1],\n",
    "]\n",
    "# Detect horizontal lines\n",
    "weight = [\n",
    "[1, 2, 1],\n",
    "[0, 0, 0],\n",
    "[-1, -2, -1],\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional Neural Networks (CNN)\n",
    "\n",
    "Convolutional NNs (CNNs) rely on convolution in place of general matrix multiplication. They are specialized for processing data with a known grid-like topology and are among the best performing systems in classification/recognition tasks. Each layer in a CNN consists in a set of $N$ _filters_ called _kernels_, that are convolved across the input. If we denote as $\\{k^l_n\\}_{n\\in[1;N]}$ the set of kernels for layer $l$, these all share a unique _kernel size_. By convolving each one of its $N$ kernels across a d-dimensional input $x$, a convolutional layer produces $N$ d-dimensional outputs called _feature maps_, denoted as $\\{a^l_n\\}_{n\\in[1;N]}$. Hence, the computation of the $n$-th activation map in layer $l$ for input $x$ is defined as:\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "a^l_m = \\sum_{n=1}^{N} k^l_m \\star x_n + b^l_m\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Thus, as depicted in the following Figure, the feature map corresponding to kernel $n$ consists in the sum of the d-dimensional discrete convolutions (denoted by the $\\star$ operator) between the kernel $n$ and each one of the d-dimensional data $\\{x_m\\}_{m\\in[1;M]}$, plus a bias $b$. A convolutional layer is thus a 3-dimensional tensor $h \\in \\mathcal{T}_{N,I,J}(\\mathbb{R})$ where $N$ is the number of features maps while $I$ and $J$ are respectively the _width_ and _height_ of the maps. \n",
    "\n",
    "<img src=\"images/02_cnns.png\" align=\"center\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be seen as replacing our _neurons_ by _feature detectors_ (the convolutional kernels), which will increasingly process the image. In the following, we will first use the high-level interface of `Pytorch` to define a CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing a CNN in `JAX`\n",
    "\n",
    "First, we are going to start by implementing the whole network in pure `JAX`. This will allow us to get a sense of how low-level operations can impact the overall design in the learning steps. We start by importing the required libraries (`JAX`, `Optax`, and their related modules). As we are on an historic path, we will be using the [`MNIST` dataset](https://en.wikipedia.org/wiki/MNIST_database), which is a digit classification dataset used for a long time as a reference in machine learning to test models (this dataset is still widely used in order to perform benchmarking on low-level properties of a network or to perform debugging).\n",
    "\n",
    "### Quick note on `JAX` vs. `Pytorch`\n",
    "\n",
    "In `JAX`, parameters are typically represented as plain numpy arrays or `JAX` DeviceArrays, and transformations on these arrays are expressed as **pure functions**. This approach is called **\"functional programming\"** and is a key feature of `JAX`. One advantage of this approach is that it allows for more explicit control over the state of the model. Since parameters are represented as plain arrays, it is easy to reason about how the state of the model changes over time. Additionally, it can be easier to implement certain optimizations (such as gradient checkpointing) using a functional programming approach.\n",
    "\n",
    "In `PyTorch`, parameters are typically represented as object-oriented modules, which encapsulate both the parameters themselves and the transformations that act on them. This approach is called **\"imperative programming\"** and is a key feature of `PyTorch`. One advantage of the object-oriented approach is that it allows for more flexible and modular code organization. Modules can be easily combined to create complex models, and the encapsulation of parameters and transformations within modules can make the code more readable and easier to reason about.\n",
    "\n",
    "Overall, both approaches have their advantages and disadvantages, and the choice between them may depend on the specific use case and personal preference. However, it is worth noting that `JAX` is generally better suited for large-scale machine learning applications, while `PyTorch` is often preferred for rapid prototyping and experimentation. Note also that several libraries exist for `JAX` that emulates this object-oriented behavior, and we will also be relying on those. Here are a few examples\n",
    "- [Flax](https://flax.readthedocs.io/en/latest/) provides a high-level API for defining neural networks and includes a variety of pre-built models and utilities.\n",
    "- [Stax](https://jax.readthedocs.io/en/latest/_autosummary/jax.experimental.stax.html) provides a functional API for defining neural network architectures with pre-built layers. \n",
    "- [Haiku](https://dm-haiku.readthedocs.io/en/latest/) also provides a high-level API for networks using a functional programming style. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and preprocess the dataset: \n",
    "\n",
    "Next, we are going to load the MNIST dataset and preprocess it by reshaping the input data to be in the format (batch_size, height, width, channels), normalizing the pixel values by dividing by 255, and converting the labels to integer format. Note here that we show how we can get the `MNIST` dataset from `TensorFlow` (in the `Keras` library), but throughout the course, we will be using different sources of data so that you can use any you like in the future. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jax\n",
    "import jax.numpy as jnp\n",
    "from jax import random, jit, grad\n",
    "import optax\n",
    "from tensorflow.keras.datasets import mnist\n",
    "# Obtain the dataset\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
    "# Perform [0, 1] normalization\n",
    "X_train = X_train.reshape(-1, 28, 28, 1).astype(jnp.float32) / 255.0\n",
    "X_test = X_test.reshape(-1, 28, 28, 1).astype(jnp.float32) / 255.0\n",
    "y_train = y_train.astype(jnp.int32)\n",
    "y_test = y_test.astype(jnp.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28, 1)\n",
      "(60000,)\n",
      "9\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'n_example' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[8], line 4\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;28mprint\u001b[39m(y_train\u001b[38;5;241m.\u001b[39mshape)\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28mprint\u001b[39m(y_train[\u001b[38;5;241m4\u001b[39m])\n\u001b[0;32m----> 4\u001b[0m X_train\u001b[38;5;241m.\u001b[39mreshape(\u001b[43mn_example\u001b[49m, \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m1\u001b[39m)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'n_example' is not defined"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(y_train.shape)\n",
    "print(y_train[4])\n",
    "X_train.reshape(n_example, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the CNN architecture\n",
    "\n",
    "Here, we are going to implement the CNN architecture using pure JAX functions. In this example, we use two convolutional layers, each followed by a ReLU activation. **Note that the gradient of the _pooling_ operation being slightly peculiar, we leave it for the following exercise**. Instead, we will rely on _strided convolutions_ to still obtain some dimensionality reduction across the different layers. Afterward, we flatten the output and pass it through a dense layer for classification.\n",
    "\n",
    "**We provide the full code to make this model works, but we strongly encourage you to try to recode the whole network first without looking at the actual solution**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Exercise (course)\n",
    ">   1. Implement the convolution, pooling operation in JAX\n",
    ">   2. Define your CNN based on those defined operations\n",
    ">   3. Train your model on the MNIST dataset\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1 - Implementing the operations\n",
    "\n",
    "Here you first need to define how to perform a convolution operation, which handles the padding and also how to define a linear (dense) layer. You can look into the following JAX operations.\n",
    "- `jax.lax.pad`: This operation pads an array with zeros along one or more dimensions. [Documentation](https://jax.readthedocs.io/en/latest/_autosummary/jax.lax.pad.html)\n",
    "- `jax.lax.conv_general_dilated`: This operation computes a n-dimensional convolution with dilated filter. [Documentation](https://jax.readthedocs.io/en/latest/_autosummary/jax.lax.conv_general_dilated.html)\n",
    "- `jax.nn.relu`: This operation applies the Rectified Linear Unit (ReLU) function element-wise to an input array. [Documentation](https://jax.readthedocs.io/en/latest/_autosummary/jax.nn.relu.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE - JAX Operations\n",
    "######################\n",
    "def conv2d(x, W, cb):\n",
    "    pass\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "######################\n",
    "# Solution: \n",
    "def conv2d(x, W, b, stride=2):\n",
    "    padding_height = max((x.shape[1] - 1) * stride + W.shape[0] - x.shape[1], 0)\n",
    "    padding_width = max((x.shape[2] - 1) * stride + W.shape[1] - x.shape[2], 0)\n",
    "    pad_top = padding_height // 2\n",
    "    pad_bottom = padding_height - pad_top\n",
    "    pad_left = padding_width // 2\n",
    "    pad_right = padding_width - pad_left\n",
    "    padded_x = jax.lax.pad(x, 0.0, ((0, 0, 0), (pad_top, pad_bottom, 0), (pad_left, pad_right, 0), (0, 0, 0)))\n",
    "    return jax.lax.conv_general_dilated(padded_x, W, (stride, stride), ((1, 1), (1, 1)), dimension_numbers=('NHWC', 'HWIO', 'NHWC')) + b\n",
    "\n",
    "def dense(x, W, b):\n",
    "    return jnp.dot(x, W) + b\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2 - Defining the CNN model\n",
    "\n",
    "Then, defining the CNN model should be a quite simple succession of your defined operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE - CNN Model\n",
    "######################\n",
    "def cnn(x, params):\n",
    "    W1, W2, K1 = params\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "######################\n",
    "# Solution: \n",
    "def cnn(params, x):\n",
    "    W1, b1, W2, b2, W3, b3, W4, b4 = params\n",
    "    # First layer\n",
    "    x = conv2d(x, W1, b1)\n",
    "    x = jax.nn.relu(x)\n",
    "    # Second layer\n",
    "    x = conv2d(x, W2, b2)\n",
    "    x = jax.nn.relu(x)\n",
    "    # Third layer\n",
    "    x = x.reshape((x.shape[0], -1))\n",
    "    x = dense(x, W3, b3)\n",
    "    x = jax.nn.relu(x)\n",
    "    # Final (MLP) layer\n",
    "    x = dense(x, W4, b4)\n",
    "    return x\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 3 - Define the loss function\n",
    "\n",
    "Implement a loss function that computes the categorical cross-entropy loss between the predicted logits and the true labels. To do so, you can rely on the following JAX functions\n",
    "- `jax.nn.one_hot`: This operation converts integer labels to a one-hot representation. [Documentation:](https://jax.readthedocs.io/en/latest/_autosummary/jax.nn.one_hot.html)\n",
    "- `optax.softmax_cross_entropy`: This operation computes the cross-entropy loss between a set of predicted probabilities (using softmax) and a set of true labels. [Documentation](https://optax.readthedocs.io/en/latest/api.html#optax.softmax_cross_entropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE - CNN Model\n",
    "######################\n",
    "\n",
    "######################\n",
    "# Solution: \n",
    "\n",
    "def loss(params, x, y):\n",
    "    logits = cnn(params, x)\n",
    "    return jnp.mean(optax.softmax_cross_entropy(logits, jax.nn.one_hot(y, 10)))\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 4 - Initializing the weights and optimizer\n",
    "\n",
    "Here, we implement a loss function that computes the categorical cross-entropy loss between the predicted logits and the true labels. Then, we set up the Adam optimizer using Optax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_params(rng):\n",
    "    # Initialize parameters\n",
    "    rng, *keys = random.split(rng, 5)\n",
    "    W1 = random.normal(keys[0], (3, 3, 1, 32)) * jnp.sqrt(2 / (3 * 3 * 1))\n",
    "    b1 = jnp.zeros(32)\n",
    "    W2 = random.normal(keys[1], (3, 3, 32, 64)) * jnp.sqrt(2 / (3 * 3 * 32))\n",
    "    b2 = jnp.zeros(64)\n",
    "    W3 = random.normal(keys[2], (57600, 128)) * jnp.sqrt(2 / (7 * 7 * 64))\n",
    "    b3 = jnp.zeros(128)\n",
    "    W4 = random.normal(keys[3], (128, 10)) * jnp.sqrt(2 / 128)\n",
    "    b4 = jnp.zeros(10)\n",
    "    return W1, b1, W2, b2, W3, b3, W4, b4\n",
    "# Prepare random generator\n",
    "rng = random.PRNGKey(42)\n",
    "# Obtain the dictionary of parameters\n",
    "params = init_params(rng)\n",
    "# Create our optimizer\n",
    "optimizer = optax.adam(1e-3)\n",
    "# Initialize the optimization state\n",
    "opt_state = optimizer.init(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we provide here the code for the main training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "Epoch 2/10\n",
      "Epoch 3/10\n",
      "Epoch 4/10\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[13], line 16\u001b[0m\n\u001b[1;32m     14\u001b[0m batch_x \u001b[38;5;241m=\u001b[39m X_train[i:i\u001b[38;5;241m+\u001b[39mbatch_size]\n\u001b[1;32m     15\u001b[0m batch_y \u001b[38;5;241m=\u001b[39m y_train[i:i\u001b[38;5;241m+\u001b[39mbatch_size]\n\u001b[0;32m---> 16\u001b[0m params, opt_state \u001b[38;5;241m=\u001b[39m \u001b[43mupdate\u001b[49m\u001b[43m(\u001b[49m\u001b[43mparams\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mopt_state\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_x\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_y\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m<string>:1\u001b[0m, in \u001b[0;36m<lambda>\u001b[0;34m(_cls, count, mu, nu)\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "@jax.jit\n",
    "def update(params, opt_state, x, y):\n",
    "    grads = jax.grad(loss)(params, x, y)\n",
    "    updates, opt_state = optimizer.update(grads, opt_state)\n",
    "    params = optax.apply_updates(params, updates)\n",
    "    return params, opt_state\n",
    "\n",
    "num_epochs = 5\n",
    "batch_size = 128\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    print(f\"Epoch {epoch+1}/{num_epochs}\")\n",
    "    for i in range(0, len(X_train), batch_size):\n",
    "        batch_x = X_train[i:i+batch_size]\n",
    "        batch_y = y_train[i:i+batch_size]\n",
    "        params, opt_state = update(params, opt_state, batch_x, batch_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating our network\n",
    "\n",
    "In the following, we provide a simplistic code for evaluating the results of our network. We take a random batch from the test set and pass it through the network to obtain its infered label and display the results by printing the infered label above each given input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAADEQAAAMFCAYAAADH5cIhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAB7CAAAewgFu0HU+AABgZklEQVR4nOzde5jVZb03/s8MMxxEgcEjCIKpkKZAqRhIAoKFmudDHnaZmqaPu13uamv2tDXM8vl1eLIsrQzQrHSrlCSKioCHREE8i4KQonjgIAdBOc6s3x/7ye241hzhZriH1+u6Ptcl91rf9/cexmx95zvvtcoiohAAAAAAAAAAAAAAAAAZKW/pDQAAAAAAAAAAAAAAADSVQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAABERET37t3j29/+dvztb3+LefPmxTvvvBMbN26M6urquOuuu2o996yzzopCoVA0AAAAALClVLT0BgAAAAAAAABoeaNHj45vfetb0aFDh5KPd+zYcQvvCAAAAADqpxABbBM6d+4c3/jGN4rWx40bFwsWLNjyGwIAAAAAANiK/OxnP4uLL764pbcBAAAAAE2iEAFsE7p06RJXXHFF0fq0adMUIgAAAAAAgG1a37594+tf/3pLbwMAAAAAmkwhAgAAAAAAAGAbdtxxx0V5eXnR+tq1a+P++++PV155JdavXx8LFy5sgd0BAAAAQN0UIgAAAAAAAAC2YQcccEDR2saNG2Pw4MHx1FNPtcCOAAAAAKBxit/mAwAAAAAAAIBtxk477VS09vTTTzdYhrjxxhujrKysaAAAAABgS1GIAAAAAAAAANiGtWvXrmjtH//4RwvsBAAAAACaRiECAAAAAAAAYBtW6lMdli5d2gI7AQAAAICmUYgAAAAAAAAAoJbVq1e39BYAAAAAoEEKEQAAAAAAAADUsn79+pbeAgAAAAA0qKKlNwCQwllnnRXjxo1r8HnTpk0rWhs2bFg8+OCDm39TW4FDDjkkjjzyyPj0pz8d++yzT+y8887Rvn37WL58eSxevDhee+21eOCBB2LSpEkxe/bslt4uAADQTJ07d47jjz8+hg8fHgcccED06tUrdthhh6iuro5ly5bFiy++GI899lhMmDAhZs6cmXw/ffr0iQsuuCBGjRoVPXv2jPbt28c111wT3/rWtxqdseuuu8YJJ5wQhx12WOy3337Rs2fP2H777WPt2rWxePHiWLRoUTz++OMxadKkeOihh2LdunUJvyIAAGj9qqurW3oLW1SHDh3imGOOiREjRsSAAQOid+/e0blz5ygUCrFixYqYO3duzJw5M+6666548MEHo1AotPSWAQCAZnAPxT0UoHUqGGNMa5uzzjqr0FxDhw4tTJgwoWh9w4YNhR122KHRe+jUqVNhw4YNJc/Rq1evJn0911xzTVHGHXfc0ejjzzjjjMLs2bOb9Pcwc+bMwqhRo1r8e2mMMcYYY8zWPL169WrS6+zmaMr1Q48ePQq/+c1vCmvWrGl0/jPPPFM488wzm/X1jx07tijvo1mjR48urF+/vuh5Y8eObdQ59ttvv8Jtt91W5/VVKe+8807hO9/5TqFjx44t/u+IMcYYY4wxW9tsynXM1KlTa2XVdT/mo+c888wzSz7v3nvv3eSv55JLLimZ/dnPfrZRx3ft2rVw9dVXF1auXNnov4d58+YV/vVf/7VQUVHR4t9PY4wxxhhjttZxD2VsUZ57KMYYk2bKA4AiEydOLFqrqKiIIUOGNDrjsMMOi4qK0h/EM2zYsCbtZ+TIkUVrd911V4PH9ezZM6ZOnRp//OMfY999923SOQ866KC45557YsqUKbHbbrs16VgAAGDLO++882L27Nlx/vnnR/v27Rt9XL9+/eLmm2+OGTNmNPm6oT7l5eVxyy23xPe+972orKxs8vFlZWVxxRVXxNNPPx0nn3xynddXpXTt2jV++MMfxiuvvBKf//znm3xuAABg8xo/fny8++67RevDhw+Prl27blL26aefXrT22muvxf3339/gsccee2zMnj07LrnkkujUqVOjz7nXXnvFL3/5y3jhhRfi0EMPbdJ+AQCALc89lP/hHgrQGilEAJRQqhARETF06NBGZ9RXehg+fHijc7p16xb77bdfrbWampq4++676z3uwAMPjJkzZza5fPFRw4cPj1mzZsWgQYM2KQcAAEjn2muvjd/+9rexww47NDvj4IMPjpkzZ8YJJ5ywWfb029/+Nr7whS8069j27dvHhAkT4vLLL2/WjYB/2nnnnePOO++MK664otkZAADApluzZk3cfvvtReuVlZWbdA2y7777Rv/+/YvWb7zxxigUCvUee8kll8Rf/vKX2HXXXZt9/j59+sS0adPioosuanYGAACQlnsopbmHArQmja+FAWTk+eefj5/85Ccf/LlTp05x/vnnFz3v1ltvjddff73W2uuvvx4LFy6M5557Lg444IBaj22uQkRTSgojRowoWnvyySdj0aJFdR6z7777xn333Vfnuyq9//778fjjj8ebb74ZS5YsiY4dO8Zee+0VAwcOjO23377o+d27d48pU6bEUUcdFVOnTm303gEAgPR+/etfx4UXXljn40899VS8+OKLsXTp0qiqqopu3brFQQcdFF26dCl6bseOHePWW2+Nk046Kf72t781e0//9m//Fueee26zji0rK4s77rgjjjrqqDqf8+yzz8a8efPizTffjPLy8thtt93iU5/6VPTu3bvoueXl5XH55ZfHbrvtFhdccEGz9gQAAGy6cePGxTnnnFO0fsopp8Tvf//7ZmWW+nSImpqaGDt2bL3H/cd//EdcffXVdT7+0ksvxTPPPBOLFy+OHXbYIbp16xYDBgwoWZ6oqKiIa6+9NmpqauK6665r+hcBAAAk4x6KeyjAtkEhAmiVZs2aFbNmzfrgz7169SpZiLjuuuviwQcfLJkxceLEokLEgQceGB07doz33nuv3vN37tw5BgwYUOfjvXr1ij333DNeeeWVenMiIkaOHFlyb3Vp27Zt3HLLLSXLEG+//XZ85zvfidtvvz1Wr15d8tiTTjopRo8eHXvvvXetx9q3bx/jx4+PAQMGxIIFCxrcNwAAbAtWrlxZq4zdHJ07d47zzjuv5GOzZs2Kt956q85jL7roopI/yF+/fn387ne/ix/96EfxxhtvFD1eWVkZo0aNiiuvvLLo3VQrKyvjpptuik9+8pPx6quvNu2LiYi99947LrnkkiYf90+XXnppyR/k19TUxLXXXhu//OUvY968eSWP/eQnPxmXXXZZnHzyyUWPffWrX40XX3wxrrnmmmbvDQAAWoNS1zFf+MIXomfPnrXWHn300Xj00Udrrc2fP7/Z53344Ydj/vz5sddee9VaP/zww6Nr166xbNmyJmeedtppRWsPPfRQvfdfPv/5z8ePfvSjovWampq45ZZb4vvf/37MnTu36PHy8vIYNmxYfO973yv5xlc///nPY9asWTFjxoymfREAANBKuYdSzD0UgHQKxhjT2qdXr16FUoYOHVrnMUOGDCl5zGc/+9kGz3fMMceUPPbDzjnnnEbt/fXXXy869qCDDqrz+T/+8Y9Lnm/atGmFnXbaqVHn7NChQ2H8+PElcyZPntzi309jjDHGGGNay5SXlxcmTpxY8rX3m2++Wdh9993rPLZPnz6FNWvWFB23ePHiwqGHHtqo87dt27bw61//uuT5//rXvzZ4/NixY4uOW7p0aa0/r127tnDDDTcUjj322ELPnj0L7du3L1RWVhY6d+5clDdgwIDCunXrijJXrlxZGDFiRKP/Xr/4xS+WzFm7dm1hr732avHvuzHGGGOMMVvbTJ06tej18+WXX97gcWeddVbJ64m6nv+f//mfJZ/f2HsmH54DDzywZNYXv/jFOo/ZcccdC4sXLy46ZvXq1YXjjz++UectKysrXHbZZSXPPWvWrEJZWVmLfz+NMcYYY4xpDeMeSu0891CMMabeafENGGNM8mlOIaK8vLywbNmyomOuuuqqBs/305/+tOT5PuwPf/hDgzl9+/YtOu6tt96q8/l9+vQpbNy4seiYefPmFaqqqpr0d1ZRUVGYNm1ayb2PGjWqxb+nxhhjjDHGtIap69phzZo1hYEDB9Z77N1331103KpVqwr7779/k/cxYcKEkvvo379/vceV+mH+hz3++ONN+uH5I488UpRRXV3d6F9M+vCcd955Jfd0yy23tPj33RhjjDHGmK1ttlQhonfv3oXq6uqi599zzz1N3vNPfvKTopyVK1cWOnToUOcxpX6ZacOGDYWRI0c2+fzXXnttya/9uOOOa/HvpzHGGGOMMa1h3EOpPe6hGGNMvdPiGzDGmOTTnEJERBT+/Oc/Fx3zyCOPNHi+WbNm1Trm4YcfLmodv/766w3mXHTRRUXnHzNmTJ3Pr+uH70OGDGnW31vv3r0L77//flHepEmTWvx7aowxxhhjTO5z9tlnl3z9XigUCmeeeWa9xw4cOLDkcWeddVaz9tKlS5fC8uXLi/Kuu+66eo+r74f506dPL2y//faN3kNd7+76u9/9rtl/x5MmTSrK27BhQ6FHjx4t/v03xhhjjDFma5otVYio61zr169v0hs7lZWVlfyE7d/+9rd1HtOjR4+S74LamK+z1FRUVBRefvnlorzmlDuMMcYYY4wxtcc9lNrjHooxxtQ/5QFAnSZOnFi0dvDBB0eHDh3qPKZz587Rv3//opxHH3201lqPHj1i7733rvf8I0eObNSeIiI6deoUX/rSl4rW77///njkkUfqPU9dXn311fjzn/9ccl877bRTszIBAICIQw89NK677rqSj1199dXxxz/+sd7jL7zwwqK12bNnx0033dSs/axYsSJ++9vfFq0ff/zxzcpbuXJlnHbaabF69epGH/P1r3+9aG39+vXxgx/8oFl7iIi46qqritYqKiri1FNPbXYmAACwaW688caitcrKyjjhhBManfGZz3wmevToUbQ+ZsyYOo8599xzo23btrXWlixZEj/5yU8afd4P27hxY/zsZz8rWj/88MNjhx12aFYmAADgHkop7qEA1E8hAqAekyZNiurq6lprbdu2jcGDB9d5zGGHHRZt2rSptTZlypSYPHly0XOHDx9eZ055eXkMGzas1tr69evjvvvuK/n8o446quQP2H/xi1/UeY7GGDt2bNFamzZt4nOf+9wm5QIAwLZqjz32iPHjx0e7du2KHpswYUJcdtll9R5fWVkZJ554YtH6TTfdFIVCodn7uvXWW4vWdttttwaL3KVcc801sWDBgkY/v02bNiW/pr/97W9Nyvmohx9+OObNm1e0fuSRRzY7EwAA2DS33XZbyV/8OeWUUxqdcdpppxWtvfjii/HYY4816Zg77rgj3nvvvUaf96Nuu+22qKmpqbXWtm3bOOSQQ5qdCQAA2zL3UIq5hwLQMIUIgHosXbo0Zs6cWbQ+dOjQOo/5aMlh5cqV8cQTTzS5EHHQQQdFly5daq09/PDDsWrVqpLPP/TQQ4vWVq5cWWeBorGmT58e7777btH6wIEDNykXAAC2Rdttt13ceeedscsuuxQ99txzz8WZZ57Z4A/kBw8eHJ06dSpaf+CBBzZpb08//XTJ1/4HHnhgk3JqamrqfOemugwYMCA6duxYtH777bc3KaeU+++/v2jtoIMO2uRcAACged57770YP3580fqIESOiqqqqwePbtGkTJ598ctF6qTd4+qdevXrFxz/+8aL1Tb2OWrp0abz44otF6029jgIAANxDqYt7KAANU4gAaMDEiROL1uorRHz0Ux0eeuihqKmpiSeeeCKWL19e73M/bOTIkY3ayz+V+tSKadOmxfr16+s8pjGqq6vj+eefL1rfb7/9NikXAAC2RTfffHMMGDCgaH3JkiVx7LHHNurjket6p9HXXnttk/ZWU1NT8hd5+vTp06ScF154Id5+++0mHVPXp/Dde++9Tcop5amnnipa69KlS+y+++6bnA0AADTPuHHjitYqKyvj+OOPb/DYI444Inbeeedaaxs2bIibbrqpzmNSXUdFRMl7KE29jgIAANxDqYt7KAANq2jpDQBs7SZOnBhXXnllrbWBAwdGu3btYt26dbXWu3TpEv369au1NmXKlIiIKBQKMWXKlDjppJM+eKxbt27Rt2/fmDNnTtF5m1KIKC8vLzpvRMSzzz5bx1fVNPPmzSt6cd2jR4/Nkg0AANuKH/zgB3HCCScUra9fvz5OOumkePXVVxuVs//++5dcv/TSS6O6unpTthhdu3YtWuvWrVuTMv7xj380+byf+tSnitZef/31olJ5c5T6uOeI/76meeONNzY5HwAAaLqpU6fGggULolevXrXWTznllHo/6SEi4vTTTy9au+eee2LRokV1HlPXddRXv/rVOOWUUxqx47rtvffeRWtNvY4CAIBtnXsodXMPBaBhChEADXjqqafizTffjO7du3+w1r59+xg0aFBMmzat1nMPO+ywaNOmTa21qVOnfvDPkydPrlWIiIgYPnx4USHin/kfNm/evJg7d27JPVZVVUVFRfF/0mfPnl33F9YE77zzTtFaly5dNks2AABsC0477bT47ne/W/Kxiy66KB5++OFGZ+2xxx4l1y+++OJm7a0hVVVVTXp+c34Av9NOOxWtpbyeiXBNAwAALe2mm26K733ve7XWRowYEV26dIkVK1aUPKZdu3Zx3HHHFa03VKKo6zrqnHPOadxmm6ip11EAALAtcw+lfu6hADSsvKU3AJCDe+65p2ht6NChRWvDhg2r9eelS5fGM88888GfJ0+eXHTM8OHDi9Y+85nPRPv27Wut3XXXXXXur64X15ujCRwR8d577xWtfXR/AABAaQcddFCMGTOm5GO/+MUv4oYbbmhS3o477rg5ttVo22+/ffJzlLqmSXk9E+GaBgAAWtqNN95YtNa2bds4/vjj6zzm6KOPjs6dO9daW7RoUb33UCJa53UUAAC0Bu6hNMw9FICGKUQANMLEiROL1koVIj5abvjoJ0jMmzcvFixYUGvtoyWKiIiRI0c2ag//VFch4t13363zmKaorKwsWluzZs1myQYAgNasW7du8de//jU6dOhQ9Nh9993XrHck2tI/hG7btm3yc5S6pkl5PRPhmgYAAFra/Pnz45FHHilaP+WUU+o85vTTTy9au/nmm2Pjxo31nqs1XkcBAEDu3ENpHPdQABqmEAHQCPfff3+sX7++1tqnP/3pWi9qq6qqol+/frWeM2XKlKKsj35KxC677BL77bdfrbWPFiJWrVoVDz74YJ37Ky8v/Z/zsrKyOo9pik6dOhWtba6mMQAAtFbt2rWLv/71r7H77rsXPTZnzpw49dRTo6ampsm5hUJhc2yv0SoqKpKfo9Q1TcrrmQjXNAAAsDUo9SkRI0eOjC5duhStb7/99nH00UcXrY8dO7bB87TG6ygAAMiZeyiN5x4KQMMUIgAaYfXq1fHwww/XWuvQoUMMHDjwgz8fdthhRS9AG1OIiKj9yRJdu3aNAQMGFB2zYcOGOvdX14vQj35sdHPtsssuRWsrVqzYLNkAANBajRkzptY1wz8tX748jj322Fi5cmWzct9///2itbfeeivKysqSzIgRI5q1z6YodU2T8nomwjUNAABsDW699daia5y2bdvG8ccfX/TcE044oeidY2fMmBEvvPBCg+cpdR0V8d+/hJXiOmqvvfZq/F8CAABsg9xDaTz3UAAaphAB0EgTJ04sWhs2bFjJf46IePPNN2POnDlFxzzwwANFDeYPFyIOP/zwomLFXXfdVe/e6ipE7LrrrvUe11gf/QSLiP/++gAAgNK++93vxhlnnFG0vnHjxvjCF74Qc+fObXb24sWLi9bqegefXJS6pkl5PVNTUxNvv/32ZskHAACab9WqVfHXv/61aP2UU04pWjvttNOK1saMGdOo85S6jorI/1oKAABy5B5K07iHAtAwhQiARipViBg6dOgH//zRQsTUqVNL5ixZsiSeffbZOnNGjhxZ67Gampq4++67693bsmXLSn6CRP/+/es9rjG222672HvvvYvWn3jiiU3OBgCA1ui4446L0aNHl3zsm9/8Ztx///2blL9gwYKitY4dO8YOO+ywSbktadGiRUVr/fr12yzZpa6L5s6dG6tWrdos+QAAwKYZN25c0drIkSOjS5cuH/x5xx13jCOOOKLWc9asWRO33HJLo85R6joqIqJbt26N3icAALDp3ENpOvdQABqmEAHQSHPnzo158+bVWhs0aFBUVFREVVVV0QvNKVOm1Jk1efLkWn/eaaed4oADDoiIKPootaeeeqrB1m11dXU89dRTReuHHHJIvcc1xogRI6KysrJofcaMGZucDQAArc0BBxwQN998c9GnvkVE3HDDDfGLX/xik8/x0YL1P33yk5/c5OyWMnPmzKK1qqqq6NOnzybllpWVFf3SVITrGQAA2JpMnjw5Fi5cWGutbdu2cdxxx33w55NPPrnoXsX48eNj5cqVjTpHa7yOAgCA3LiH0jzuoQA0TCECoAk++kkNHTt2jIMPPjiGDh1a9GK9KYWIiIjhw4fHHnvsUfRpDKU+maKUv//970VrhxxySPTo0aNRx9flmGOOKVpbt25dyRfbAACwLdt5551jwoQJsf322xc99tBDD8WFF164Wc5T6rV/RO1PnmuOTp06xUsvvRSvvPJKrdl///03Kbcx6vqaTjnllE3KHTRoUOy8886NPh8AALDlFQqF+MMf/lC0/uHrgdNPP73o8TFjxjT6HNOnT4+ampqi9U29jiorK4vHH3+86Dpq1KhRm5QLAACtjXsozeceCkDDFCIAmqBUOWHYsGExbNiwWmuvvPJKvPrqq3XmPPTQQ7Fu3bpaa8OHDy/Zum1sIWLatGlFa+Xl5XH22Wc36vhSOnXqFKeddlrR+p133hmrV69udi4AALQ2lZWVMX78+Ojdu3fRY6+88kqceOKJsXHjxs1yrieffDLefPPNovV/+Zd/2aTcL33pS9G3b9/o3bv3B7PzzjvHSy+9tEm5jfHcc8/FO++8U3JPpd4pqrEuuOCCorW1a9fGbbfd1uxMAABg87vxxhuL1kaOHBmdO3eO7t27x5AhQ2o99uqrr9b7xlQftXTp0pLvcnrSSSdF+/btm77h/+foo4+OgQMH1rqO6t27dzzzzDPNzgQAgNbGPZRN4x4KQMMUIgCaYNq0aUVFgKFDhxYVIhr6IfyaNWti+vTptdYOO+ywokLEokWLGv0xZHfffXfRR0pHRPz7v/977Ljjjo3K+Kjzzz8/dthhh6L13/3ud83KAwCA1ur6668v+gWdiIhVq1bFscceW/IH1Zui1Lun9unTJ0466aRm5XXt2jUuvfTSovVbbrlls92EqE+hUIjf//73Ret9+vSJc845p1mZPXr0KPnuSHfccUcsX768WZkAAEAac+bMiccee6zWWrt27eK4446LL3zhC9GmTZtaj40bN67J5yhVuujcuXNcdNFFTc6KiGjbtm1ceeWVResPPPBAvPXWW83KBACA1sg9lE3jHgpA4xSMMaa1T/fu3QulHHvssU3OuvPOO2tlvPfee4Xq6upaa2eccUaDOZdddlnRftatW1frz2PHjm3S3r7zne+U/Dpvu+22Jn+d++yzT2H16tVFWS+++GKLfz+NMcYYY4zZmubiiy8u+Tq8urq6cMwxxyQ5Z48ePQpr1qwpOueCBQsKVVVVTcoqKysrjB8/vuT+991333qPHTt2bNFxTb2O+ef07NmzsGHDhqK8xYsXFz72sY81Oe/ee+8t+X0ZNGhQi/87Y4wxxhhjzNY2U6dOLXrtfPnllzd43FlnnVXydXdz9nDBBRcU5dx1112FGTNmFF2r9OrVq8n5HTt2LCxatKjoHCtXrizsueeeTc675pprSn7tn/vc51r8+2mMMcYYY8zWMu6h1OYeijHGJJsW34AxxiSf7bbbruSLuL/85S+FsrKyJmWdf/75JbM+rFu3bg3mDBw4sMGck08+uUl7q6qqKrz11lsls6699tpCRUVFo3L23HPPwnPPPVcy5/DDD2/x76cxxhhjjDFby3zuc58r+QPoQqFQuPTSS5Oe+0c/+lHJ886YMaPQpUuXRmW0a9eucPvtt5fMuf766xs8fnP+MD8iCr/5zW9K7mXOnDmN/oWnNm3a1PmLSTfeeGOL/ztjjDHGGGPM1jhbQyGiS5cuRb+0tH79+qLsyZMnN/vr/OpXv1pyv/Pnzy/07NmzURllZWV1XnNMmjSpxb+XxhhjjDHGbC3jHsrYouPcQzHGmDRT9v/+AaDVW7BgQeyxxx5F6y+99FI8/PDD8e6770bbtm3jV7/6VcyZM6fOnB49esTrr79e5+MvvfRS7Lvvvg3up7y8PJYuXRpVVVUlH1+/fn3svPPO8e677zaY9WGjRo2Ke+65p+RjzzzzTFx66aUxefLkkh/Z1rVr1zjjjDNi9OjRJfd18803xxe/+MUm7QcAAFqrvn37xmOPPRZdunQpemzFihVxww03bJbzzJ8/P66//vqi9Xbt2sUTTzwR+++/f9Fjb7/9dlx22WVx6623xvvvv1/0ePv27ePkk0+O0aNHx5577ln0+MKFC+MTn/hEg9cjY8eOjS9/+cu11saNGxdnn312A19VaR07doynn3469t5776LHVq9eHaNHj45x48bFkiVLih4vLy+PI444Ir73ve/FoYceWvT4smXLom/fvrF06dJm7Q0AAFqzqVOnxrBhw2qtXXHFFfH973+/3uPOOuusGDduXNF6WVlZs/Zx6623xqmnnlrvc84888z405/+1Kz8iIj77rsvjjjiiKL1lStXxujRo2PMmDGxYsWKoscrKipi1KhRcdVVV0W/fv2KHl+1alUccMABsWDBgmbvDQAAWgv3UNxDAdiSFCKAbcZvfvObOP/88xt83rBhw+LBBx+s9znPPPNMyR92R0T8+te/josuuqhRexo/fnyccMIJJR+bMmVKjBgxolE5H/Wzn/0sLr744jofX758eTz22GPx9ttvx6pVq2LXXXeN3XffPQYOHBht27YteczcuXPj05/+dCxfvrxZewIAgNbm8ssvjyuuuCL5eaZNmxbDhw8v+Vjv3r1j+vTpsdtuu5V8fO3atfHoo4/GwoULY/ny5dG5c+fo0aNHDB48OLbbbruSx6xatSqGDRsWTz75ZIN729w/zI+IGDhwYDz44IPRvn37ko9XV1fHjBkzYsGCBbFkyZLYfvvto3v37tGvX7/o1q1bnceceOKJMWHChGbvCwAAWrOtpRBx5JFHxt13313n4ytWrIhu3brF2rVrm5UfEVFVVRV///vf63xzqw0bNsTjjz8eCxYsiKVLl8YOO+wQ3bp1i8GDB0fnzp3rPOb444+vd+8AALAtcQ/FPRSALamipTcAsKX88pe/jHPPPTfatGmzyVkTJ06ssxAxZcqURudMnjy5zkLExIkTm7W3iIhvfvOb0aVLlzpfQFdVVcWRRx7Z6Lz58+fHqFGjlCEAAGAr8+qrr8bQoUPj/vvvL/mJeO3bt4/DDz+80XkrV66ME088sVE/yE9lxowZcdJJJ8Vtt91W8oZDmzZtYtCgQTFo0KBG5W3cuDHOP/98P8gHAIAM3HvvvfHWW2/V+Ys6t9xyyyaVISL++02jhg8fHvfee2/079+/6PHKysoYMmRIDBkypFF569ati3POOUcZAgAAtjLuoTTMPRSgtShv6Q0AbCnPP/98fPOb39wsWXWVFWpqamLq1KmNzpk8eXKTz9EYhUIhzjnnnPj2t78d69ata3ZORMRdd90VgwYNildeeWWTcgAAgDTmzp0bBx54YNxzzz2blPP000/H4MGDm1TyTuXuu++Oz3zmM/Hyyy9vUs7ChQvjyCOPjLFjx26mnQEAACnV1NTEzTffXOfjY8aM2SznWbRoUQwePDhuvPHGTcr5xz/+ESNHjow//elPm2VfAADA5uUeSt3cQwFaE4UIYJtyzTXXxMCBA2PcuHHx8ssvx+rVq2PDhg2xfPnymDt3btxyyy2xYMGCBnOmT58ey5YtK1p/9tlnS67XZe7cufHaa68Vrc+fPz/mzJnT6Jy6/OQnP4n+/fvHf/3Xf0V1dXWTjn388cfj2GOPjWOOOSaWLFmyyXsBAADSWbp0aRx11FFx6qmnxvPPP9+kY19++eW48MIL46CDDorZs2cn2mHTPfnkk9GvX7/47ne/2+RrksWLF8fll18e++23X71FdAAAYOtTV0nh+eefj5kzZ26287z//vvx5S9/OUaOHBnTp09v0rFvvPFGXHrppfGJT3wiHnnkkc22JwAAYPNzD6U291CA1qgsIgotvQkA0uvWrVuceOKJcdhhh8UnPvGJ6NGjR3Ts2DFqampi9erVsXDhwpg7d25Mnz49Jk6cuFkKGQAAQMsYPHhwHHfccTFo0KDYZ599oqqqKsrLy2PVqlXxzjvvxOzZs+PJJ5+MSZMmxYwZM1p6uw2qrKyMI488Mo444og48MAD42Mf+1h06dIl2rRpE2vWrIklS5bEP/7xj3jyySdjypQpMXny5CaXwgEAYFvWvn37aNOmTa219evXx4YNG1poR1te//7944QTToghQ4ZE3759Y6eddoqKiopYvXp1LFu2LObMmRNPP/103HvvvfHII4+45gAAgEy5h+IeCtD6KEQAAAAAAAAAAAAAAADZKW/pDQAAAAAAAAAAAAAAADSVQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdio2V1Dbtm2jX79+ERGxePHiqK6u3lzRAABQpzZt2sQuu+wSERHPPvtsrF+/voV3xNbCNQoAAC3BNQqluD4BAKAluD6hLq5RAABoCamuUTZbIaJfv34xc+bMzRUHAABNdvDBB8cTTzzR0ttgK+EaBQCAluYahX9yfQIAQEtzfcKHuUYBAKClbc5rlPLNkgIAAAAAAAAAAAAAALAFbbZPiFi8ePEH/3zRwZfEsrdXbK5oAACoU9fdusSvZv6fiKj9mhRcowAA0BJco1CK6xMAAFqC6xPq4hoFAICWkOoaZbMVIqqrqz/452Vvr4ilbyzbXNEAANAoH35NCq5RAABoaa5R+CfXJwAAtDTXJ3yYaxQAAFra5rxGKd9sSQAAAAAAAAAAAAAAAFuIQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOxUtPQGgK3LO18ZlDT/wm/+JWn+og2dk+bfdv2IpPm7/n5W0vzCunVJ8wEAAAAAAAAAAABgS/EJEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkJ2Klt4AsHU58Pynk+Z/udObSfNr4o2k+d/+7uyk+R8f9JWk+X3/7dWk+dXLlyfNBwCA3Gw8/MCk+cu+/l7S/CcO+lPS/NTalKV9L5DhLxyXNP+Nmd2T5u81+qmk+TVr1ybNBwAAAAAAAPAJEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkJ2Klt4AAP/jpcNvSJo/4fGqpPmX3XZm0vy9fvJS0vzq5cuT5gMAsO35v2N+lTR/38rKpPk1SdPTqylUJ82/b7/xSfNjv7Txnz/kuKT5bU59P2l+9TvLkuYDAAAAAEBdKnbvnjT/vX67J81/c2jaXx/esMuGpPl9f7UmaX5h1gtJ84HNyydEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdipaegPA1mXy3/snzX/phClJ8/tUtk2an7tjOy5Pmn/8l3+VNP8rI4YmzX/l+wcnzW93z8yk+QAArVHNkAFJ8+d/pU3S/L6VXgOy9brr43cmzR/+x1OS5lf8Yq+k+RER7e72v2EAAPJQlvgeWXmH9knzF35l/6T51e2Sxm8R3R9dkzS//OFnk+ZHTXXafAAgO206dUqa/8o30r7G/PppaX/GfnbntPnlid9PvSZqkuY/MzxpfHzll19Pmt/tZ48mzYdtjU+IAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7FS09AaArcveFz+WNP9bN56bNH9N945J81N77bTqpPljDh2XNP8z7Tcmzf9tz2lJ839w9bKk+TMe6pI0v+a995LmAwCUUnbgJ5Lmr/nf7ybNn3PAbUnzvRcF27Kpif/3NffX65PmR0RcdOG/Jc1vd8/MpPkAAGw95v/xk0nzT91vVtL87+8yPWl+xNTE+a3Av6aN7zPhwqT5+131RtL8jQvT5gPAtmjj4QemPcF3FyWNf/rj1yTNp2X1b5s2f+Y30/77c/BhX0qa3/3K9PcoC7NeSH4OaCx35QEAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYqWnoDwLal5unZSfPbPZ00Prl97k6b/6PolzT/q7emzZ89ZFzS/P/c6bmk+YNOuShpftW46UnzAQBKWXXVmqT50w64LWk+LWvqmvZJ81fUbJc0/4SOy5LmU78+lW2Tn2O/0WmvE2f0GJQ0f8ffuU4EgM2pLPHrj2VnHpg0/7Rv3Zs0/5fTRyTN321Km6T51/3wmqT5fSpnJM2vLEv795O79wvrk+b//J20//vdEn5zxNik+TcccFjS/JVDksYDwDZpwdGVSfPn7XtX0vwNhbzfL/y1jWnvwZ3w1HlJ89c/U5U0f8Neaf9+nhl2XdL8Zwf+OWn+hgnVSfMjIoZdnPZ34bb/r8eS5tO65P1ffAAAAAAAAAAAAAAAYJukEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOxUtPQGAGg9yl/cPml+zZBC0vzUBn9tZtL8F8cljQcAcjXwgKTx/2vPu5PmU7/VNeuS5h90+78nze973ZKk+bFkWdL471+4b9L8qRf+OGl+VXn7pPmtwTXd/540f8Z3Hk2aP/p3n0qaDwDbmjWjBiTN//tV1ybNT+1rR72cNP/Q7qclzd+ufGPS/MqytK+/a6Imaf4z65PGx2kPXpA0v2xFZdL8vb/xWNL8LWFG7yOT5hfatU2aH/FO4nwA2PoUDh2QNH/MCdcnzd9QqE6an/o18lPr0r4f+aVfuzhpfreJaX+PKXf9b74waf5Lw29Imp/63/+IiG9fdXPS/F8vPClpftmjzyTNZ8vyCREAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJCdipbeAACtx64zN6Q9wXlp41Pbq/2SpPkvRlXSfABg8ys7+IDk57jgj39Jmn/0diuT5lO/YbPOTZq/98WPJc2vTpqeXo8fPpo0/5TnLk6af8ev/2/S/M7l7ZPmAwB81OpTP500/6T/vC9pfu4WblyTNL/q+2lfX1785qlJ898Z1jNpflniC6xOf057fbhPzEqaT8M2vvpaS28BALa4mqGfTJq/cGiHpPl3nvvjpPm9KtomzU/9ft5PrUub/x8XX5g0v8PEGUnzqd/ut1emPcHwtPFbwpHbLU+af/FZaf8b1CftbT62MJ8QAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2alo6Q0A0Hq0n/xs0vyb3t09af6XOr2RNB8A4KPmnN0h+TmO3m5l8nPQcqYfdGPS/OOGnJ80v/yRp5Pm567932YkzV95bSFpfmdvxdKgge3Sfg/em/SxpPkdR/0jaT4Arc97Jx2SNP+G/+9nSfP3rmyXND+1P6/aNW3+aZ9Nml/2/ItJ8zdu3Jg0v/PN7kEAAGxtul09P2n+XXs8kDQ/om3i/LRe27gmaf6lX7s4aX6HiWl/hk/LWvxJv14NOXFbEgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAslPR0hsAoPWoPnjfpPmf6/hA0vyIDknTf/bw55Lm94kZSfMBYFtU3j/t65vrPntj0nxav8qyNmlPUFaWNh+2cc+ur06a3+lrSeMj7e4B2NLK2rVLfo7y8xcnzd+7Mv3XkLNrfnZK0vxd1yxJml+2V++k+bEo7f6rV6xMmg8AsKW16dQpaf43n/p70vyIiGHtNyTNr/F+1fU64anzkuZ3mzgzaT6t2+4PrkuaX/6V1Pfg8v/vz+kDH0+aP6sV/B3xP3w3AQAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOxUtvQEAWo8Fn++QNL9bm+2S5qfW9p02Lb0FAKCJ5pzXKWn+iA7vJ82n9Xtobduk+RXvrk2aX5M0HbZ+7cuq056gbWXafABalXfO+FTyczy6/7XJz0HdHrvc3399rl/xsaT5Y647Omn+Ltc+mjQfAMhP2YGfSJr/sevnJc0f0j7tz6cjImoSv590jZ+C12vV2zskze+WNJ3WbvXuae/B1UQhcX7+//25fJdZSfOPjYOT5rNl+YQIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADITkVLbwBomjZ775k0f96VnZLmb1jVNmn+t4ZMSpqf2v+9++ik+Z8Z8kLS/J/v9tOk+TXRLml+amNO/1XS/C/ucGHS/IrV6XuU+/zylaT5G996O2k+AK3PLUdfm/gM3qegIesKG5Lm//idA5Pm3/Tkp5Pm7/O7jUnzy555Jmk+9Vv0tcFJ83dt81jSfBrWpzLtz0nmn941aX7v/500HoAtbMfnViU/x2WLDkqaf/Wus5Lm527uhrVJ83++eETS/F/v/vek+V+rWpA0/399J+3PGI4+8Zik+et/uFvS/MrJ/vcLwLanTae0v6Oz8qo1SfN/2v2RpPmtwU/f2T9p/v3/cVjS/DfPXpc0f/v5fn2VrdeQbzze0lsAmsBvXgAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYqWnoDQNOs/lXa/Bf2H5M0vzzKkubXRCFpfmrnn574G5xYebRPmp/793dgu7T7//GoPyfNP77jiqT5ERGDX7woaX6XP7ydNB8A2Pxe31iTNP+x/pVJ8/vErKT5tKyKHrsnzT/1vAeS5rcrS/vvPw17ZePapPm9J6xOmg9A61J44vnk55h9TPek+Xt///yk+W3fSvv6aa8/LE6aX7ZufdL86jcXJc0/Zs9Tkuav69klaf7Xrr81af7f+k5Imv/U79Jen19x1BlJ86tffDlpPgA0S7ddksZPOSDt7xC0Bq9tXJM0/+bbRiTN7znp0aT5vScljYet2qx39kh7gt1mps1vBf5z8cEtvQUy4hMiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANlRiAAAAAAAAAAAAAAAALKjEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOxUtvQGgaXpsv6KltwA008qatUnzvzXltKT5Y65cmDQ/IqJq0cyk+YWk6QC0hPdOPiRpfs82jyTNj+iQOD9/6wptWnoLpFSe9vtb0bN70vyBE+Ynzf/2jrOT5tPyXt/YKe0JZjyXNh8AmqjQMe010I6PVabNv2F60vzqpOn5q56b9vV3xdyk8XH1y6OS5h/d/9ak+Z9sm/a9Fl+6JO1r432+nDQeAJrlxW9VJc0v917JDZr03r5J83te+WjSfNiW9X0i7c8AftrtjqT55VGWNL81vF/+1J8PSppfFWl/zsOWlf+/8QAAAAAAAAAAAAAAwDZHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAAAAsqMQAQAAAAAAAAAAAAAAZEchAgAAAAAAAAAAAAAAyI5CBAAAAAAAAAAAAAAAkB2FCAAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAQHYUIgAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAAAANmpaOkNAE3zje73JT5D3j2p//POJ5LmP7miZ9L8W/ealDQ/d0ur1yTNP/pH306a3/XFdUnz+0ybkTR/Y9J0AGiexZ9K+/p1pzYdkubTsH/5zcVJ83ePR5PmU79XRw9Mmv/82dcmzaf1m5X2Mi5+dO5ZSfPbxJNJ8wFga7Pzn55Jml+TNJ2WtuKLg5LmX/Nx1yf1qXirbUtvAQC2uKpd302aX+MVbIN+/8vPJ83fOaYnzYdNUb7ddknzl5zeP2n+v+z4i6T56f8LmvY+9ysb1ybNj4g4bsYFSfN7j38+ab7/l2xd8v7NZwAAAAAAAAAAAAAAYJukEAEAAAAAAAAAAAAAAGRHIQIAAAAAAAAAAAAAAMiOQgQAAAAAAAAAAAAAAJAdhQgAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAEB2FCIAAAAAAAAAAAAAAIDsKEQAAAAAAAAAAAAAAADZUYgAAAAAAAAAAAAAAACyoxABAAAAAAAAAAAAAABkRyECAAAAAAAAAAAAAADIjkIEAAAAAAAAAAAAAACQHYUIAAAAAAAAAAAAAAAgOwoRAAAAAAAAAAAAAABAdhQiAAAAAAAAAAAAAACA7ChEAAAAAAAAAAAAAAAA2VGIAAAAAAAAAAAAAPj/27v/ULvrOo7jn3vP3XXO6Zw/bmqWw7mrYCq6rBVpEhWNrCkm/SBNjMxSRHMKYRFJBkFIWGaCgvirRFKE/shMkCTvnTa1mS1Xi7lEY/5aXdM2ds7tj0hE3Gp53/fc193j8df96/X9cncZ53POeZ4DAMQRRAAAAAAAAAAAAAAAAHEEEQAAAAAAAAAAAAAAQBxBBAAAAAAAAAAAAAAAEEcQAQAAAAAAAAAAAAAAxBFEAAAAAAAAAAAAAAAAcQQRAAAAAAAAAAAAAABAnKF+3wDMNi+e9Z7S/aXDD5fuV+sM1HZYN9xzUul+5+CXS/cHFw+U7ler/vf98FWXlu4fdM0DpfsAAOy8wWUvlu4/9dX3lu4f9MG/lO5feMg9pftHzHmudP8tnfHS/dbmFO8z251/xfml+/veN1a6D8D0Wv/dZaX7V664sXR/707t89+ttbby8trXUPa5pfbxNzs2OHdu6f6znz22dP+ClbeX7i/drXS+3OH3fqF0f8llD5buA8BMNDmZ/R6R2eDFo7ql+weMLi7d765bX7rP7Lb++iWl+4+deFXpPju24sFzy6/x9tMfK93vla4z2/iGCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIM9fsGYLbpbJ0s3d8yua10f85Ap3S/TfZK59d+6urS/Wq1v516t/x9v9L9g3+5uXQ//fcPALuiyUWvlO7/rffP0v0Fg3NL92eD1cffXHuB42vn8+3e7xug0EdPP7t0v/PSltL96TCy4fHS/W7pOgDT7dQPrCrdXz5vonR/Ovz6ih+U7n9v5Wjp/pbenNL9dPM7m0v3z9u79u9nsA2U7le/BvFct/Y5kj0fLn4Oo+fRMQC7njl3LizdX9bOKN0ff+dNpfvTYd2Ka0r3H1le+yjwr9sWlO5/5a4zS/ffdswzpfsXLbqndD/d8nmrS/fT34d15H3nlO7v84vaM9aiO35Xut9a/r8xs4tviAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACDOUL9vAGabvW4dL93/9qVLS/e/sf+jpfvMbtd+/bTS/fmPrirdBwDyLPnmS6X7z949ULq/wMcUAH002an9T6i35g+l+wAw09x/5btL90/4zOLS/fuPua10fzpcuHBdv28Btusja08t3d/6wwNL9w+444HSfQDYFS28Yaz2AjfUzq9Yelbp/uCGZ0r3W2vtlZ/sWbp/0+G3lO4fOzxRur/801eX7g+22tfhem2ydD9d9e+/+vPav/zUiaX7oxc8Wbrfff6F0v1e6TrMPN56AQAAAAAAAAAAAAAAxBFEAAAAAAAAAAAAAAAAcQQRAAAAAAAAAAAAAABAHEEEAAAAAAAAAAAAAAAQRxABAAAAAAAAAAAAAADEEUQAAAAAAAAAAAAAAABxBBEAAAAAAAAAAAAAAEAcQQQAAAAAAAAAAAAAABBHEAEAAAAAAAAAAAAAAMQRRAAAAAAAAAAAAAAAAHEEEQAAAAAAAAAAAAAAQBxBBAAAAAAAAAAAAAAAEEcQAQAAAAAAAAAAAAAAxBFEAAAAAAAAAAAAAAAAcQQRAAAAAAAAAAAAAABAHEEEAAAAAAAAAAAAAAAQRxABAAAAAAAAAAAAAADEEUQAAAAAAAAAAAAAAABxBBEAAAAAAAAAAAAAAEAcQQQAAAAAAAAAAAAAABBHEAEAAAAAAAAAAAAAAMQRRAAAAAAAAAAAAAAAAHEEEQAAAAAAAAAAAAAAQJyhft8AsHNu/9n7SvfPPXOsdP/AzrzS/XRjWzql+5dd/MXS/fl3rirdBwB4vcm5c0r3T77/vNL9T7zjkdL91lr71sjq8mvArqrXeqX7R9z9pdL9Jb1tpfsAsKtZcPN47QVurX3++ON7nFS631prm08+snR/4hCfBcf/76Bf/aN0f2hsTe3+5MbSfQCA15tc/Xjpfrd0/d+GP/RC6f4pn7+kdP9d59a+znTiXk+U7p82/7nS/ern8NNd9HTt+xDvvve40v3R7z9Zut99/unSfWBqeVYQAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgzlC/bwDYOYu+Nla6f8qGS0r3z7n4rtL9j+2xrnT/hNtXlu6P/mhT6f68P64q3QcAmG69364t3T/sjNL59tjhh9VeoLV2ya21R//TFz5Yun/o0Mul+/t1di/dr/bUtlf6fQtvysuTndL9zb3dSvc/N3ZO6f7o2b8p3QcAwvS6tfMTE6X7rbW214/Ha/dL1wEAAHbOvtfXvs9r/fWl8+3JQ99fun/t6P6l++zY8M8fKt0/tNX+/W8rXQfS+IYIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4gz1+waAmWXf68ZK93963UjtfqvdX9zGS/e7pesAAMw03Sf+VH6NtUtr9y9vx5XuT3xyWen+80cNlO5Xe+t9W/t9C2/K3KcnSve7v19Xur+4PVK6DwAAAAAA/bLtzxtK94eL9wHYdfiGCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIM9fsGAAAAALZnz9vGi/dL5/kvuv2+AQAAAAAAAACi+YYIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAIM7QVA11Op1Xf97ngL2nahYAAHbotY89X/uYFJxRAADoB2cU3ojzCQAA/eB8wvY4owAA0A9VZ5QpCyJGRkZe/fnqh74zVbMAAPA/GxkZaRs3buz3bTBDOKMAANBvzij8h/MJAAD95nzCazmjAADQb1N5RhmckhUAAAAAAAAAAAAAAIBpNNBam5yKoeHh4Xb00Ue31lrbtGlT63a7UzELAAA71Ol0Xv0UmzVr1rStW7f2+Y6YKZxRAADoB2cU3ojzCQAA/eB8wvY4owAA0A9VZ5QpCyIAAAAAAAAAAAAAAACmy2C/bwAAAAAAAAAAAAAAAGBnCSIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOIIIgAAAAAAAAAAAAAAgDiCCAAAAAAAAAAAAAAAII4gAgAAAAAAAAAAAAAAiCOIAAAAAAAAAAAAAAAA4ggiAAAAAAAAAAAAAACAOIIIAAAAAAAAAAAAAAAgjiACAAAAAAAAAAAAAACII4gAAAAAAAAAAAAAAADiCCIAAAAAAAAAAAAAAIA4gggAAAAAAAAAAAAAACCOIAIAAAAAAAAAAAAAAIgjiAAAAAAAAAAAAAAAAOL8C5EKoQmNIbIpAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 2000x10000 with 4 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 386,
       "width": 1570
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from cml.plot import cml_figure_matplotlib\n",
    "test_size = 4\n",
    "# Pick a random batch of input images\n",
    "start_idx = np.random.randint(0, X_test.shape[0] - test_size)\n",
    "x_batch = X_test[start_idx:start_idx+test_size]\n",
    "# Perform inference on test data\n",
    "predicted_probabilities = cnn(params, x_batch)\n",
    "# Get predicted classes\n",
    "predicted_classes = np.argmax(predicted_probabilities, axis=1)\n",
    "# Define class names\n",
    "class_names = [\"zero\", \"one\", \"two\", \"three\", \"four\", \"five\", \"six\", \"seven\", \"eight\", \"nine\"]\n",
    "# Get corresponding class names\n",
    "class_labels = [class_names[i] for i in predicted_classes]\n",
    "# Plot results\n",
    "fig = cml_figure_matplotlib(100, 20); \n",
    "axs = fig.subplots(1, 4)\n",
    "for i in range(test_size):\n",
    "    axs[i].imshow(x_batch[i]); axs[i].set_title(class_labels[i]);\n",
    "    axs[i].set_xticks([]); axs[i].set_yticks([]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cnn\"></a>\n",
    "## Exercise 1 - Defining a CNN in Pytorch\n",
    "\n",
    "Defining a convolutional network in Pytorch is quite easy, as we can rely on the `nn` module, which contains all the required layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous course, we have seen that we could define our network in a very simple way, by using the `Sequential` model definition. Here we define a CNN followed by a MLP, as seen in the previous course"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "# Define the input dimensions\n",
    "in_size = 1000\n",
    "# Use the nn package to define our model and loss function.\n",
    "model = nn.Sequential(\n",
    "    nn.Conv2d(3, 6, 5),\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2, 2),\n",
    "    nn.Conv2d(6, 16, 5),\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2, 2),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(16 * 5 * 5, 120),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(120, 84),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(84, 10),\n",
    "    nn.Softmax(dim=1)\n",
    ")\n",
    "inputs = torch.rand(16, 3, 32, 32)\n",
    "model(inputs);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to implement the networks in Pytorch is to use the `functional` approach. In this version, each layer is seen as a function, that we apply on sucessive inputs. For instance, we can define one layer of fully-connected units and apply it to some inputs as follows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "# Define one layer\n",
    "layer = nn.Linear(100, 10)\n",
    "# Define the non-linearity\n",
    "activation = nn.Sigmoid()\n",
    "# Create some random input\n",
    "inputs = torch.rand(32, 100)\n",
    "# inputs.view(16, -1) => 200\n",
    "# inputs.view(16, -1, 4) => 50\n",
    "# Apply our layers\n",
    "outputs = activation(layer(inputs))\n",
    "# Equivalently, as ReLU is parameter-free\n",
    "outputs = F.relu(layer(inputs))\n",
    "y = (outputs + 3).sum()\n",
    "y.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to make this even cleaner, we can define our own `nn.Module`, which is a `Pytorch` class representing models. To do so, we can define a sub-class, and implement the functions `__init__` (defining our layers) and `forward` (explaining how our forward pass will behave)\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 1.1 - Implement a CNN by **using the `functional` library**\n",
    "\n",
    "> 1. Update the forward propagation and error computation\n",
    "> 2. Update the back-propagation part to learn the weights of both layers.\n",
    "  \n",
    "*For optional questions, please look at the end of this exercise's code boxes for more information*\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class CNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(CNN, self).__init__()\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "    def forward(self, x):\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        \n",
    "        return x\n",
    "\n",
    "model = CNN()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the network\n",
    "\n",
    "In order to test our CNN, we are going to try to perform image classification. To do so, we can use the simplifications for data loading contained in `torchvision`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision\n",
    "import torchvision.transforms as transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `torchvision` package provides pre-coded simplification to download and use the major image datasets, notably `MNIST` and `CIFAR`, which are the baseline datasets for testing image ML models. The output of torchvision datasets are PILImage images of range [0, 1]. We transform them to Tensors of normalized range [-1, 1].\n",
    "\n",
    "In the following code, we are going to load the `CIFAR10` _train_ and _test_ sets. **Note that this code will automatically download the dataset if you did not have it before, and place it in the `data` folder, so this might take a bit of time**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "# Transforms to apply to the images\n",
    "transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
    "# Import the train dataset\n",
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)\n",
    "# Import the test dataset\n",
    "testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)\n",
    "# Classes in the CIFAR dataset\n",
    "classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using your knowledge from the previous course, you can now define an optimization problem, and implement the training loop for your model\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 1.2 - Training our `functional` CNN\n",
    "\n",
    "> 1. Define a `criterion` and `optimizer`\n",
    "> 2. Fill in the training loop to train your model\n",
    "  \n",
    "*For optional questions, please look at the end of this exercise's code boxes for more information*\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "\n",
    "criterion = ...\n",
    "optimizer = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #410819; border-color: #cb2e47\">\n",
    "\n",
    "> **Warning on the training time**\n",
    "\n",
    "> The CIFAR dataset that we are using is a very large one, and given that we are using a deep architecture, the full training of the model will take a prohibitive time on a CPU. We recommend that you either ensure that you use the GPU of your own computer or (if unavailable) try out your code on the Colab version with a GPU runtime.\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 100\n",
    "# Training loop\n",
    "for epoch in range(n_epochs):\n",
    "    running_loss = 0.0\n",
    "    # Go through all batches\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        # Get the inputs; data is a list of [inputs, labels]\n",
    "        inputs, labels = data\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        \n",
    "    #print(f'loss at epoch {epoch}: {running_loss}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that your model is trained, you can test it by feeding some new (unseen) images and see if it is able to classify them correctly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"layer\"></a>\n",
    "## Coding our own convolutional layer\n",
    "\n",
    "Although `Pytorch` comes packed with pre-implemented layers, we can also very easily define our own layers. This will be useful when you start doing research and propose your own way of processing the information. A large advantage of `Pytorch` is that it performs _automatic gradient differentiation_, this means that we simply have to define how the `forward` pass will work, and `Pytorch` will automatically infer the backpropagation equations, without us having to go through any complicated differentiation\n",
    "\n",
    "In the following, we are going to redefine the `Conv2d` layer, by computing the operation ourselves.\n",
    "\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Going further (optional)\n",
    ">   1. Complete the `forward` function to compute a convolution\n",
    ">   2. Do the same for the pooling operation\n",
    ">   3. Rewrite your previous Pytorch CNN using your own modules\n",
    ">   2. Train this model on the MNIST dataset\n",
    "\n",
    "</div>\n",
    "\n",
    "***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class MyConv2d(nn.Module):\n",
    "    def __init__(self, n_channels, out_channels, kernel_size, dilation=1, padding=0, stride=1):\n",
    "        super(MyConv2d, self).__init__()\n",
    "        self.kernel_size = (kernel_size, kernel_size)\n",
    "        self.kernal_size_number = kernel_size * kernel_size\n",
    "        self.out_channels = out_channels\n",
    "        self.dilation = (dilation, dilation)\n",
    "        self.padding = (padding, padding)\n",
    "        self.stride = (stride, stride)\n",
    "        self.n_channels = n_channels\n",
    "        self.weights = nn.Parameter(torch.Tensor(self.out_channels, self.n_channels, self.kernal_size_number))\n",
    "\n",
    "    def forward(self, x):\n",
    "        width = self.compute_width(x)\n",
    "        height = self.compute_height(x)\n",
    "        windows = self.compute_windows(x)\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "        return x  \n",
    "\n",
    "    def compute_windows(self, x):\n",
    "        windows = F.unfold(x, kernel_size=self.kernel_size, padding=self.padding, dilation=self.dilation, stride=self.stride)\n",
    "        windows = windows.transpose(1, 2).contiguous().view(-1, x.shape[1], self.kernal_size_number)\n",
    "        windows = windows.transpose(0, 1)\n",
    "        return windows\n",
    "\n",
    "    def compute_width(self, x):\n",
    "        return ((x.shape[2] + 2 * self.padding[0] - self.dilation[0] * (self.kernel_size[0] - 1) - 1) // self.stride[0]) + 1\n",
    "\n",
    "    def compute_height(self, x):\n",
    "        return ((x.shape[3] + 2 * self.padding[1] - self.dilation[1] * (self.kernel_size[1] - 1) - 1)// self.stride[1]) + 1\n",
    "\n",
    "# Testing the code directly\n",
    "conv = MyConv2d(3, 1, 3)\n",
    "x = torch.randn(1, 3, 24, 24)\n",
    "out = conv(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can use your own `MyConv2d` layer, and use it in real-life scenarios. Hence, you need to first implement the pooling operation and then trying to change your previous model to use your own layer instead. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2 - Recurrent Neural Networks (RNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recurrent neural networks (RNNs) are a family of models designed to process time series and sequential data, which perform remarkably in applications such as speech recognition or machine translation. The ability of RNNs to model correlations between successive computations through recurrent connection make them efficient for temporal problem as they provide a form of _memory_. \n",
    "\n",
    "To model structured sequential data, NNs can be augmented with recurrent loops, which allow to retain information across time steps. Considering a sequence $\\mathbf{X}=\\{\\mathbf{x}_t\\}$, dependencies between elements are managed by having a recurrent hidden state $\\mathbf{h}_t$ at time $t$ in the network. The value of $\\mathbf{h}_t$ at each time depends of the previous time and the input, as depicted in the following figure. \n",
    "\n",
    "<img src=\"images/02_rnn.png\" align=\"center\"/>\n",
    "\n",
    "Formally, each hidden state is updated as\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\mathbf{h}_t = \n",
    "\\begin{cases} \n",
    "\\phi_{\\mathbf{\\mathbf{\\theta}}}(\\mathbf{x}_0) & \\mbox{if } t=0 \\\\ \n",
    "\\phi_{\\mathbf{\\mathbf{\\theta}}}(\\mathbf{h}_{t-1},\\mathbf{x}_t), & \\mbox{otherwise} \n",
    "\\end{cases}\n",
    "\\label{eq:RNNhiddenupdate}\n",
    "\\end{equation}\n",
    "$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementing a simple rnn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a simple one-to-many vanilla recurrent neural network example in functional form. If we were to produce `h[t]`, we need some weight matrices, `h[t-1]`, `x[t]` and a non-linearity `tanh`.\n",
    "\n",
    "$$\n",
    "h_{t} = tanh(W_{hh}h_{t-1} + W_{xh}x_{t} + B_{h})\n",
    "$$\n",
    "\n",
    "Since this is a **one-to-many** network, we'd want to produce an output `y[t]` at every timestep, thus, we need another weight matrix that accepts a hidden state and project it to an output.\n",
    "\n",
    "$$\n",
    "y_{t} = W_{hy}h_{t} + B_{y}\n",
    "$$\n",
    "\n",
    "Now that we know how to use the `Functional` library of `Pytorch`, we are going to implement our own simple RNN layer as previously. This time, we do not provide the content of the `__init__` function, so think carefully of what parameters you will need and how you need to define them.\n",
    "\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 2.1 - Defining a `functional` RNN\n",
    "\n",
    "> 1. Complete the `__init__` function to instantiate all required parameters\n",
    "> 2. Complete the `forward` function to compute the forward pass\n",
    "\n",
    "  \n",
    "*For optional questions, please look at the end of this exercise's code boxes for more information*\n",
    "\n",
    "</div>\n",
    "\n",
    "***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class RecurrentNetwork(nn.Module):\n",
    "    \n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        \"\"\" Defines the required variables \"\"\"\n",
    "        super(RecurrentNetwork, self).__init__()\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        self.state \n",
    "\n",
    "\n",
    "    def init_state()\n",
    "        \n",
    "    def reset_state()\n",
    "    \n",
    "    def forward(self, x ???):\n",
    "        \"\"\" Computes the forward pass \"\"\"\n",
    "        input, state = x\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        output = ...\n",
    "        \n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then forward propagate our information inside our network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ellipsis\n",
      "Ellipsis\n",
      "Ellipsis\n"
     ]
    }
   ],
   "source": [
    "in_size = 4\n",
    "input_vector = torch.ones(16, in_size)\n",
    "simple_network = RecurrentNetwork(in_size, 10, 5)\n",
    "# Notice that same input, but leads to different ouptut at every single time step.\n",
    "print(simple_network(input_vector))\n",
    "print(simple_network(input_vector))\n",
    "print(simple_network(input_vector))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, `Pytorch` also comes packed with some pre-coded recurrent layers. You can go check the documentation to find how to use these."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Music generation with LSTMs \n",
    "\n",
    "In this exercise, we will explore how to use LSTMs (Long Short-Term Memory) for symbolic music generation with the help of `PyTorch`. Music generation with LSTM is a challenging problem where the goal is to train a model to generate new pieces of music that are musically coherent and similar to a given input dataset. However, there are several challenges associated with this task, such as capturing the long-term dependencies and structures that exist in music, dealing with the high dimensionality and complexity of musical data, and balancing the need for creativity and novelty with the desire for consistency and coherence. In addition, there is the issue of evaluating the quality of generated music, which can be subjective and context-dependent. Despite these challenges, recent advancements in deep learning techniques and models such as LSTMs have shown promising results in generating high-quality music, and the field continues to be an active area of research and development.\n",
    "\n",
    "We will provide here the blueprint for adressing this problem by using a \"mid-level\" approach. Compared to the previous problems, we will heavily rely on the pre-defined layers available in `Pytorch` and `JAX` (through the `Flax` library). However, we will still need to define the overall approach to solve this problem, notably for the data pre-processing, model and loss definition and the overall training loop. The major library we will be using is the `torch.nn` as follows "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "Usually, symbolic music scores in computers come in the [MIDI format](https://en.wikipedia.org/wiki/MIDI). MIDI files contain information about the _pitch_, _timing_, and _duration_ of musical notes, as well as other metadata such as tempo and instrument type. However, these are events that only provide information at various time positions. Hence, to train an LSTM model, the MIDI data needs to be converted into a suitable format that can be used as input to the model. This involves several preprocessing steps, such as **quantization**, **encoding**, and **normalization**. \n",
    "- _Quantization_ involves dividing the MIDI data into fixed time intervals and mapping each note to the nearest quantized time step. \n",
    "- _Encoding_ involves converting the quantized MIDI data into a numerical representation, such as a one-hot encoding or a continuous representation using embeddings. \n",
    "- _Normalization_ involves scaling the encoded data to a fixed range to ensure that the LSTM can learn meaningful patterns and avoid saturation or vanishing gradients. \n",
    "\n",
    "Overall, preprocessing the MIDI data is a crucial step in preparing the data for deep learning with LSTMs and can have a significant impact on the quality of the generated music. However, as this is a quite complicated set of operations, we will rather rely on a dataset that has already been preprocessed and is one of the \"golden standard\" in symbolic music generation, the [JSB Chorales dataset](https://github.com/czhuang/JSB-Chorales-dataset)\n",
    "\n",
    "**Note for other input data**\n",
    "If you want to try out your network on other data, and learn how to implement your own data pre-processing methods, we recommend trying out the [Groove dataset](https://www.tensorflow.org/datasets/catalog/groove) from TFDS, which contains a large number of drum performances (as scores) in [MIDI format](https://en.wikipedia.org/wiki/MIDI).\n",
    "\n",
    "### Loading the dataset\n",
    "\n",
    "Thanks to the great work on the [JSB Chorales github](https://github.com/czhuang/JSB-Chorales-dataset), we just need to retrieve a pickled (zipped) array and import it to obtain a prepared dataset for music generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import requests\n",
    "from io import BytesIO\n",
    "from cml.plot import image, center_plot, cml_figure\n",
    "# URL to load the JSB chorales dataset from\n",
    "url = \"https://github.com/czhuang/JSB-Chorales-dataset/raw/master/jsb-chorales-8th.pkl\"\n",
    "response = requests.get(url)\n",
    "with open(\"./data/jsb_chorales.pkl\", \"wb\") as f:\n",
    "    f.write(response.content)\n",
    "with open(\"./data/jsb_chorales.pkl\", \"rb\") as p:\n",
    "    data = pickle.load(p, encoding=\"latin1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observing examples from the training set\n",
    "\n",
    "A very important step before designing a model is to carefully scrutinize the properties of the data corresponding to the task we wish to address.\n",
    "\n",
    "Here, each training example in the JSB Chorales dataset contains a sequence of 16th notes represented as integers, where each integer corresponds to a MIDI note number or a rest. The notes are represented using a simplified version where a note is represented as an integer in the range [0, 127], and a rest is represented as -1. The sequences represent the soprano, alto, tenor, and bass voices in four-part chorales from the J.S. Bach chorale harmonization collection. The chorales are all in 4/4 time, and each example consists of exactly 64 beats (i.e., 256 16th notes). \n",
    "\n",
    "The dataset contains a total of 229 chorales, split into 182 for training, 21 for validation, and 26 for testing. The dataset was created to facilitate research into automatic music composition and machine learning for music."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(74, 70, 65, 58), (74, 70, 65, 58), (75, 70, 58, 55), (75, 70, 60, 55), (77, 69, 62, 50), (77, 69, 62, 50), (77, 70, 62, 55), (77, 69, 62, 55), (75, 67, 63, 48), (75, 69, 63, 48), (74, 70, 65, 46), (74, 70, 65, 46), (72, 69, 65, 53), (72, 69, 65, 53), (72, 69, 65, 53), (72, 69, 65, 53), (74, 70, 65, 46), (74, 70, 65, 46), (75, 69, 63, 48), (75, 67, 63, 48), (77, 65, 62, 50), (77, 65, 60, 50), (74, 67, 58, 55), (74, 67, 58, 53), (72, 67, 58, 51), (72, 67, 58, 51), (72, 65, 57, 53), (72, 65, 57, 53), (70, 65, 62, 46), (70, 65, 62, 46), (70, 65, 62, 46), (70, 65, 62, 46), (72, 69, 65, 53), (72, 69, 65, 53), (74, 71, 53, 50), (74, 71, 53, 50), (75, 72, 55, 48), (75, 72, 55, 50), (75, 67, 60, 51), (75, 67, 60, 53), (74, 67, 60, 55), (74, 67, 57, 55), (74, 65, 59, 43), (72, 63, 59, 43), (72, 63, 55, 48), (72, 63, 55, 48), (72, 63, 55, 48), (72, 63, 55, 48), (75, 67, 60, 60), (75, 67, 60, 60), (77, 70, 62, 58), (77, 70, 62, 56), (79, 70, 62, 55), (79, 70, 62, 53), (79, 70, 63, 51), (79, 70, 63, 51), (77, 70, 63, 58), (77, 70, 60, 58), (77, 70, 62, 46), (77, 68, 62, 46), (75, 67, 58, 51), (75, 67, 58, 51), (75, 67, 58, 51), (75, 67, 58, 51), (74, 67, 58, 55), (74, 67, 58, 55), (75, 67, 58, 53), (75, 67, 58, 51), (77, 65, 58, 50), (77, 65, 56, 50), (70, 63, 55, 51), (70, 63, 55, 51), (75, 65, 60, 45), (75, 65, 60, 45), (74, 65, 58, 46), (74, 65, 58, 46), (72, 65, 57, 53), (72, 65, 57, 53), (72, 65, 57, 53), (72, 65, 57, 53), (74, 65, 58, 58), (74, 65, 58, 58), (75, 67, 58, 57), (75, 67, 58, 55), (77, 65, 60, 57), (77, 65, 60, 53), (74, 65, 58, 58), (74, 65, 58, 58), (72, 67, 58, 51), (72, 67, 58, 51), (72, 65, 57, 53), (72, 65, 57, 53), (70, 65, 62, 46), (70, 65, 62, 46), (70, 65, 62, 46), (70, 65, 62, 46)]\n"
     ]
    }
   ],
   "source": [
    "print(data[\"train\"][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence, based on the observed data, we can start thinking on how we could design an adequate model for processing this data. As we know that this form of data is clearly temporal, we will rely on network composed of LSTM cells. We can cast this task as an _inference_ problem, in which we try to **predict from a given sequence of notes the most probable next note**. Note that this allows us to straightforwardly use our trained network to generate entirely novel pieces of music, we just need to give different starting sequences, and then predict each note one by one (we feed the network with its own results from the past steps).\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Exercice (optional)\n",
    ">   1. Create a model that can train on the proposed input data from JSB Chorales\n",
    ">   2. Use at least one LSTM layer and an output layer for _prediction_\n",
    ">   3. Write the training function for your model\n",
    ">   4. Complete the training loop and launch your model\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Model\n",
    "\n",
    "In this step, you will implement an RNN or LSTM model using PyTorch. Design the architecture of your choice, but make sure it has at least one LSTM layer and an output layer with a softmax activation function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MusicLSTM(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size, num_layers=1):\n",
    "        super(MusicRNN, self).__init__()\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        \n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the Model\n",
    "\n",
    "Now that our model is defined, we can train it on our preprocessed music data. Implement the training function using the appropriate loss function and optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, dataloader, optimizer, criterion, device):\n",
    "    model.train()\n",
    "   \n",
    "    ######################\n",
    "    # YOUR CODE GOES HERE\n",
    "    ######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can define the final training loop for your music generation network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create model, dataloader, optimizer and loss function\n",
    "input_size = 1\n",
    "hidden_size = 512\n",
    "output_size = len(note_to_int)\n",
    "num_layers = 1\n",
    "batch_size = 64\n",
    "# Try to use a GPU if we have one available\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "\n",
    "# Create our model\n",
    "model = ...\n",
    "# Create our optimizer and criterion\n",
    "optimizer = ...\n",
    "criterion = ...\n",
    "\n",
    "# Train the model\n",
    "num_epochs = 50\n",
    "n_steps = 4\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "\n",
    "    ######################\n",
    "    # YOUR CODE GOES HERE\n",
    "    ######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Music Generation\n",
    "With our trained model, we can now generate new music sequences. Implement a function that takes an initial sequence as input and generates a new sequence of notes and chords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_music(model, initial_sequence, length, device):\n",
    "    model.eval()\n",
    "    generated_sequence = []\n",
    "\n",
    "    for _ in range(length):\n",
    "    \n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        pass\n",
    "\n",
    "    return generated_sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listening to the results\n",
    "\n",
    "In order to be able to listen to the beautiful results of your music generation LSTM, we provide here a code that is able to synthesize audio from a MIDI file so that you can directly hear your results :)\n",
    "\n",
    "**Note that we apply this to an example from the training data, you need to change the following code so that it uses your `generate_music` function instead**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing stream\n",
      "Opening player\n",
      "Launching play (may take a while)\n"
     ]
    }
   ],
   "source": [
    "import music21\n",
    "import random\n",
    "# Select a random training example\n",
    "example = random.choice(data['train'])\n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "\n",
    "# Convert the example notes to a music21 stream\n",
    "stream = music21.stream.Stream()\n",
    "print(\"Preparing stream\")\n",
    "for i, voice in enumerate(('soprano', 'alto', 'tenor', 'bass')):\n",
    "    part = music21.stream.Part(id=voice)\n",
    "    for note_f in example:\n",
    "        note = note_f[i]\n",
    "        if note == -1:  # Rest\n",
    "            n = music21.note.Rest()\n",
    "        else:  # Note\n",
    "            n = music21.note.Note(note)\n",
    "        part.append(n)\n",
    "    stream.append(part)\n",
    "# Play the music21 stream\n",
    "print(\"Opening player\")\n",
    "sp = music21.midi.realtime.StreamPlayer(stream)\n",
    "# Launch playing\n",
    "print(\"Launching play (may take a while)\")\n",
    "sp.play()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}