01a_machine_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creative machine learning - Machine Learning\n",
    "\n",
    "### Author: Philippe Esling (esling@ircam.fr)\n",
    "\n",
    "In this course we will cover\n",
    "1. A [first definition](#definition) on the concept of machine learning\n",
    "2. An introduction to a simple problem of [linear regression](#regression)\n",
    "4. A detailed implementation of [simple linear regression](#linear)\n",
    "3. An explanation on [model capacity and overfitting](#capacity)\n",
    "4. Two [exercises](#exercises) to explore regression and classification\n",
    "4. An introduction to the [audio datasets](#audio) that we will use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"definition\"></a>\n",
    "## Defining machine learning\n",
    "\n",
    "In all natural process, there exists complex relations between sets $\\mathcal{X} \\mapsto \\mathcal{Y}$. This can relate some objects with their names, or a cause to a consequence. In most cases, _we do not know the precise relations_ between these sets, all we have is _observations_ such as pairs $(x,y)$, composed of input data $x \\in \\mathcal{X}$, which have a corresponding expected output $y \\in \\mathcal{Y}$. The overarching goal of machine learning is to approximate such _unknown processes_ as a function $\\mathcal{F}_{\\theta}$, which _transforms_ input data $x$ into output data $y$.\n",
    "\n",
    "\n",
    "<center>\n",
    "<img src=\"images/01_machine_learning_basic.png\" align=\"center\"/>\n",
    "</center>\n",
    "\n",
    "Hence, machine learning aims to understand and model the relationship between some (usually complex and high-dimensional) inputs $\\mathbf{x}\\in\\mathcal{X}\\subset\\mathbb{R}^{\\mathcal{X}}$ and outputs $\\mathbf{y}\\in\\mathcal{Y}\\subset\\mathbb{R}^{\\mathcal{Y}}$, given by a set of data examples $\\mathcal{D}=\\left\\{(x_1,y_1),\\cdots,(x_N,y_N)\\right\\}$. This is achieved by defining a parametric model $f_{\\mathbf{\\theta}}\\in\\mathcal{F}$ inside a family of functions $\\mathcal{F}$, which depends on parameters $\\mathbf{\\theta} \\in \\mathbf{\\Theta}$ and that could approximate the underlying relationship. The _learning_ aspect refers to the adjustment of the parameters $\\mathbf{\\theta}$ in order to obtain the best approximation of the given task\n",
    "$$\n",
    "\\begin{equation}\n",
    "f_{\\mathbf{\\theta}}(\\mathbf{x}) = \\bar{\\mathbf{y}}\\approx \\mathbf{y}.\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Hence, the major elements that we have to define in any machine learning problems are\n",
    "1. **Dataset** : $\\mathcal{D}=\\left\\{(x_1,y_1),\\cdots,(x_N,y_N)\\right\\}$. This dataset has to be representative of the relation $f:\\mathcal{X} \\mapsto \\mathcal{Y}$ that we are looking to model\n",
    "2. **Model** : Our parametric approximation $\\bar{\\mathbf{y}} = f_{\\mathbf{\\theta}}(\\mathbf{x})$, where the choice of family $f_{\\mathbf{\\theta}}\\in\\mathcal{F}$ is critical\n",
    "3. **Loss** : $\\mathcal{L}\\left( \\bar{\\mathbf{y}}, \\mathbf{y} \\mid f_{\\theta}, \\theta \\right)$ allows to measure the amount of errors made by our model\n",
    "4. **Optimization** : Method to find $\\theta^{*}\\in\\Theta$ so that our model minimizes the loss\n",
    "$$\\theta^{*}= \\underset{\\theta}{\\text{argmin }} \\mathcal{L}\\left( \\bar{\\mathbf{y}}, \\mathbf{y} \\mid f_{\\theta}, \\theta \\right)$$\n",
    "\n",
    "\n",
    "To observe this idea in simple setups, we are going to use the `numpy` library and also initialize the homemade course library `cml` and style for future plotting and exercise. We also set the random generator to a fixed point with `rng = np.random.RandomState(1)`, to ensure reproducibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "        .bk-notebook-logo {\n",
       "            display: block;\n",
       "            width: 20px;\n",
       "            height: 20px;\n",
       "            background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAABx0RVh0U29mdHdhcmUAQWRvYmUgRmlyZXdvcmtzIENTNui8sowAAAOkSURBVDiNjZRtaJVlGMd/1/08zzln5zjP1LWcU9N0NkN8m2CYjpgQYQXqSs0I84OLIC0hkEKoPtiH3gmKoiJDU7QpLgoLjLIQCpEsNJ1vqUOdO7ppbuec5+V+rj4ctwzd8IIbbi6u+8f1539dt3A78eXC7QizUF7gyV1fD1Yqg4JWz84yffhm0qkFqBogB9rM8tZdtwVsPUhWhGcFJngGeWrPzHm5oaMmkfEg1usvLFyc8jLRqDOMru7AyC8saQr7GG7f5fvDeH7Ej8CM66nIF+8yngt6HWaKh7k49Soy9nXurCi1o3qUbS3zWfrYeQDTB/Qj6kX6Ybhw4B+bOYoLKCC9H3Nu/leUTZ1JdRWkkn2ldcCamzrcf47KKXdAJllSlxAOkRgyHsGC/zRday5Qld9DyoM4/q/rUoy/CXh3jzOu3bHUVZeU+DEn8FInkPBFlu3+nW3Nw0mk6vCDiWg8CeJaxEwuHS3+z5RgY+YBR6V1Z1nxSOfoaPa4LASWxxdNp+VWTk7+4vzaou8v8PN+xo+KY2xsw6une2frhw05CTYOmQvsEhjhWjn0bmXPjpE1+kplmmkP3suftwTubK9Vq22qKmrBhpY4jvd5afdRA3wGjFAgcnTK2s4hY0/GPNIb0nErGMCRxWOOX64Z8RAC4oCXdklmEvcL8o0BfkNK4lUg9HTl+oPlQxdNo3Mg4Nv175e/1LDGzZen30MEjRUtmXSfiTVu1kK8W4txyV6BMKlbgk3lMwYCiusNy9fVfvvwMxv8Ynl6vxoByANLTWplvuj/nF9m2+PDtt1eiHPBr1oIfhCChQMBw6Aw0UulqTKZdfVvfG7VcfIqLG9bcldL/+pdWTLxLUy8Qq38heUIjh4XlzZxzQm19lLFlr8vdQ97rjZVOLf8nclzckbcD4wxXMidpX30sFd37Fv/GtwwhzhxGVAprjbg0gCAEeIgwCZyTV2Z1REEW8O4py0wsjeloKoMr6iCY6dP92H6Vw/oTyICIthibxjm/DfN9lVz8IqtqKYLUXfoKVMVQVVJOElGjrnnUt9T9wbgp8AyYKaGlqingHZU/uG2NTZSVqwHQTWkx9hxjkpWDaCg6Ckj5qebgBVbT3V3NNXMSiWSDdGV3hrtzla7J+duwPOToIg42ChPQOQjspnSlp1V+Gjdged7+8UN5CRAV7a5EdFNwCjEaBR27b3W890TE7g24NAP/mMDXRWrGoFPQI9ls/MWO2dWFAar/xcOIImbbpA3zgAAAABJRU5ErkJggg==);\n",
       "        }\n",
       "    </style>\n",
       "    <div>\n",
       "        <a href=\"https://bokeh.org\" target=\"_blank\" class=\"bk-notebook-logo\"></a>\n",
       "        <span id=\"ac5f9fbc-e51f-4719-ba0c-80ba0421a102\">Loading BokehJS ...</span>\n",
       "    </div>\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": "(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  const force = true;\n\n  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n    root._bokeh_onload_callbacks = [];\n    root._bokeh_is_loading = undefined;\n  }\n\nconst JS_MIME_TYPE = 'application/javascript';\n  const HTML_MIME_TYPE = 'text/html';\n  const EXEC_MIME_TYPE = 'application/vnd.bokehjs_exec.v0+json';\n  const CLASS_NAME = 'output_bokeh rendered_html';\n\n  /**\n   * Render data to the DOM node\n   */\n  function render(props, node) {\n    const script = document.createElement(\"script\");\n    node.appendChild(script);\n  }\n\n  /**\n   * Handle when an output is cleared or removed\n   */\n  function handleClearOutput(event, handle) {\n    const cell = handle.cell;\n\n    const id = cell.output_area._bokeh_element_id;\n    const server_id = cell.output_area._bokeh_server_id;\n    // Clean up Bokeh references\n    if (id != null && id in Bokeh.index) {\n      Bokeh.index[id].model.document.clear();\n      delete Bokeh.index[id];\n    }\n\n    if (server_id !== undefined) {\n      // Clean up Bokeh references\n      const cmd_clean = \"from bokeh.io.state import curstate; print(curstate().uuid_to_server['\" + server_id + \"'].get_sessions()[0].document.roots[0]._id)\";\n      cell.notebook.kernel.execute(cmd_clean, {\n        iopub: {\n          output: function(msg) {\n            const id = msg.content.text.trim();\n            if (id in Bokeh.index) {\n              Bokeh.index[id].model.document.clear();\n              delete Bokeh.index[id];\n            }\n          }\n        }\n      });\n      // Destroy server and session\n      const cmd_destroy = \"import bokeh.io.notebook as ion; ion.destroy_server('\" + server_id + \"')\";\n      cell.notebook.kernel.execute(cmd_destroy);\n    }\n  }\n\n  /**\n   * Handle when a new output is added\n   */\n  function handleAddOutput(event, handle) {\n    const output_area = handle.output_area;\n    const output = handle.output;\n\n    // limit handleAddOutput to display_data with EXEC_MIME_TYPE content only\n    if ((output.output_type != \"display_data\") || (!Object.prototype.hasOwnProperty.call(output.data, EXEC_MIME_TYPE))) {\n      return\n    }\n\n    const toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n\n    if (output.metadata[EXEC_MIME_TYPE][\"id\"] !== undefined) {\n      toinsert[toinsert.length - 1].firstChild.textContent = output.data[JS_MIME_TYPE];\n      // store reference to embed id on output_area\n      output_area._bokeh_element_id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n    }\n    if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n      const bk_div = document.createElement(\"div\");\n      bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n      const script_attrs = bk_div.children[0].attributes;\n      for (let i = 0; i < script_attrs.length; i++) {\n        toinsert[toinsert.length - 1].firstChild.setAttribute(script_attrs[i].name, script_attrs[i].value);\n        toinsert[toinsert.length - 1].firstChild.textContent = bk_div.children[0].textContent\n      }\n      // store reference to server id on output_area\n      output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n    }\n  }\n\n  function register_renderer(events, OutputArea) {\n\n    function append_mime(data, metadata, element) {\n      // create a DOM node to render to\n      const toinsert = this.create_output_subarea(\n        metadata,\n        CLASS_NAME,\n        EXEC_MIME_TYPE\n      );\n      this.keyboard_manager.register_events(toinsert);\n      // Render to node\n      const props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n      render(props, toinsert[toinsert.length - 1]);\n      element.append(toinsert);\n      return toinsert\n    }\n\n    /* Handle when an output is cleared or removed */\n    events.on('clear_output.CodeCell', handleClearOutput);\n    events.on('delete.Cell', handleClearOutput);\n\n    /* Handle when a new output is added */\n    events.on('output_added.OutputArea', handleAddOutput);\n\n    /**\n     * Register the mime type and append_mime function with output_area\n     */\n    OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n      /* Is output safe? */\n      safe: true,\n      /* Index of renderer in `output_area.display_order` */\n      index: 0\n    });\n  }\n\n  // register the mime type if in Jupyter Notebook environment and previously unregistered\n  if (root.Jupyter !== undefined) {\n    const events = require('base/js/events');\n    const OutputArea = require('notebook/js/outputarea').OutputArea;\n\n    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n      register_renderer(events, OutputArea);\n    }\n  }\n  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  const NB_LOAD_WARNING = {'data': {'text/html':\n     \"<div style='background-color: #fdd'>\\n\"+\n     \"<p>\\n\"+\n     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n     \"</p>\\n\"+\n     \"<ul>\\n\"+\n     \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n     \"<li>use INLINE resources instead, as so:</li>\\n\"+\n     \"</ul>\\n\"+\n     \"<code>\\n\"+\n     \"from bokeh.resources import INLINE\\n\"+\n     \"output_notebook(resources=INLINE)\\n\"+\n     \"</code>\\n\"+\n     \"</div>\"}};\n\n  function display_loaded() {\n    const el = document.getElementById(\"ac5f9fbc-e51f-4719-ba0c-80ba0421a102\");\n    if (el != null) {\n      el.textContent = \"BokehJS is loading...\";\n    }\n    if (root.Bokeh !== undefined) {\n      if (el != null) {\n        el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n      }\n    } else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(display_loaded, 100)\n    }\n  }\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n\n    root._bokeh_onload_callbacks.push(callback);\n    if (root._bokeh_is_loading > 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls == null || js_urls.length === 0) {\n      run_callbacks();\n      return null;\n    }\n    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n    root._bokeh_is_loading = css_urls.length + js_urls.length;\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n\n    function on_error(url) {\n      console.error(\"failed to load \" + url);\n    }\n\n    for (let i = 0; i < css_urls.length; i++) {\n      const url = css_urls[i];\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error.bind(null, url);\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }\n\n    for (let i = 0; i < js_urls.length; i++) {\n      const url = js_urls[i];\n      const element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error.bind(null, url);\n      element.async = false;\n      element.src = url;\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  const js_urls = [\"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-mathjax-3.2.0.min.js\", \"https://unpkg.com/@holoviz/panel@1.1.0/dist/panel.min.js\"];\n  const css_urls = [];\n\n  const inline_js = [    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\nfunction(Bokeh) {\n    }\n  ];\n\n  function run_inline_js() {\n    if (root.Bokeh !== undefined || force === true) {\n          for (let i = 0; i < inline_js.length; i++) {\n      inline_js[i].call(root, root.Bokeh);\n    }\nif (force === true) {\n        display_loaded();\n      }} else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    } else if (force !== true) {\n      const cell = $(document.getElementById(\"ac5f9fbc-e51f-4719-ba0c-80ba0421a102\")).parents('.cell').data().cell;\n      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n    }\n  }\n\n  if (root._bokeh_is_loading === 0) {\n    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n    run_inline_js();\n  } else {\n    load_libs(css_urls, js_urls, function() {\n      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n      run_inline_js();\n    });\n  }\n}(window));",
      "application/vnd.bokehjs_load.v0+json": ""
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": "(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  var force = true;\n  var py_version = '3.2.0'.replace('rc', '-rc.');\n  var is_dev = py_version.indexOf(\"+\") !== -1 || py_version.indexOf(\"-\") !== -1;\n  var reloading = false;\n  var Bokeh = root.Bokeh;\n  var bokeh_loaded = Bokeh != null && (Bokeh.version === py_version || (Bokeh.versions !== undefined && Bokeh.versions.has(py_version)));\n\n  if (typeof (root._bokeh_timeout) === \"undefined\" || force) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks;\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, js_modules, js_exports, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n    if (js_modules == null) js_modules = [];\n    if (js_exports == null) js_exports = {};\n\n    root._bokeh_onload_callbacks.push(callback);\n\n    if (root._bokeh_is_loading > 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls.length === 0 && js_modules.length === 0 && Object.keys(js_exports).length === 0) {\n      run_callbacks();\n      return null;\n    }\n    if (!reloading) {\n      console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n    }\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n    window._bokeh_on_load = on_load\n\n    function on_error() {\n      console.error(\"failed to load \" + url);\n    }\n\n    var skip = [];\n    if (window.requirejs) {\n      window.requirejs.config({'packages': {}, 'paths': {'ace': '//cdnjs.cloudflare.com/ajax/libs/ace/1.4.7', 'jspanel': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/jspanel', 'jspanel-modal': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/modal/jspanel.modal', 'jspanel-tooltip': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/tooltip/jspanel.tooltip', 'jspanel-hint': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/hint/jspanel.hint', 'jspanel-layout': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/layout/jspanel.layout', 'jspanel-contextmenu': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/contextmenu/jspanel.contextmenu', 'jspanel-dock': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/dock/jspanel.dock', 'gridstack': 'https://cdn.jsdelivr.net/npm/gridstack@7.2.3/dist/gridstack-all', 'notyf': 'https://cdn.jsdelivr.net/npm/notyf@3/notyf.min'}, 'shim': {'ace/ext-language_tools': {'deps': ['ace/ace']}, 'ace/ext-modelist': {'deps': ['ace/ace']}, 'jspanel': {'exports': 'jsPanel'}, 'gridstack': {'exports': 'GridStack'}}});\n      require([\"ace/ace\"], function(ace) {\n\twindow.ace = ace\n\ton_load()\n      })\n      require([\"ace/ext-language_tools\"], function() {\n\ton_load()\n      })\n      require([\"ace/ext-modelist\"], function() {\n\ton_load()\n      })\n      require([\"jspanel\"], function(jsPanel) {\n\twindow.jsPanel = jsPanel\n\ton_load()\n      })\n      require([\"jspanel-modal\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-tooltip\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-hint\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-layout\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-contextmenu\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-dock\"], function() {\n\ton_load()\n      })\n      require([\"gridstack\"], function(GridStack) {\n\twindow.GridStack = GridStack\n\ton_load()\n      })\n      require([\"notyf\"], function() {\n\ton_load()\n      })\n      root._bokeh_is_loading = css_urls.length + 12;\n    } else {\n      root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length + Object.keys(js_exports).length;\n    }\n\n    var existing_stylesheets = []\n    var links = document.getElementsByTagName('link')\n    for (var i = 0; i < links.length; i++) {\n      var link = links[i]\n      if (link.href != null) {\n\texisting_stylesheets.push(link.href)\n      }\n    }\n    for (var i = 0; i < css_urls.length; i++) {\n      var url = css_urls[i];\n      if (existing_stylesheets.indexOf(url) !== -1) {\n\ton_load()\n\tcontinue;\n      }\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }    if (((window['ace'] !== undefined) && (!(window['ace'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js', 'https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js', 'https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['jsPanel'] !== undefined) && (!(window['jsPanel'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/jspanel.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/modal/jspanel.modal.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/tooltip/jspanel.tooltip.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/hint/jspanel.hint.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/layout/jspanel.layout.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/contextmenu/jspanel.contextmenu.js', 'https://cdn.holoviz.org/panel/1.1.0/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/dock/jspanel.dock.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['GridStack'] !== undefined) && (!(window['GridStack'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/1.1.0/dist/bundled/gridstack/gridstack@7.2.3/dist/gridstack-all.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['Notyf'] !== undefined) && (!(window['Notyf'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/1.1.0/dist/bundled/notificationarea/notyf@3/notyf.min.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    var existing_scripts = []\n    var scripts = document.getElementsByTagName('script')\n    for (var i = 0; i < scripts.length; i++) {\n      var script = scripts[i]\n      if (script.src != null) {\n\texisting_scripts.push(script.src)\n      }\n    }\n    for (var i = 0; i < js_urls.length; i++) {\n      var url = js_urls[i];\n      if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    for (var i = 0; i < js_modules.length; i++) {\n      var url = js_modules[i];\n      if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      element.type = \"module\";\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    for (const name in js_exports) {\n      var url = js_exports[name];\n      if (skip.indexOf(url) >= 0 || root[name] != null) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onerror = on_error;\n      element.async = false;\n      element.type = \"module\";\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      element.textContent = `\n      import ${name} from \"${url}\"\n      window.${name} = ${name}\n      window._bokeh_on_load()\n      `\n      document.head.appendChild(element);\n    }\n    if (!js_urls.length && !js_modules.length) {\n      on_load()\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  var js_urls = [\"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.2.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.2.0.min.js\", \"https://cdn.holoviz.org/panel/1.1.0/dist/panel.min.js\"];\n  var js_modules = [];\n  var js_exports = {};\n  var css_urls = [];\n  var inline_js = [    function(Bokeh) {\n      inject_raw_css(\"/*\\n ~ CML // Creative Machine Learning ~\\n mml.css : CSS styling information for Panel and Bokeh\\n \\n This file defines the main CSS styling information for the CML course\\n \\n Author               :  Philippe Esling\\n                        <esling@ircam.fr>\\n*/\\n\\nbody {\\n  display: flex;\\n  height: 100vh;\\n  margin: 0px;\\n  overflow-x: hidden;\\n  overflow-y: hidden;\\n}\\n\\n.bk-root .bk, .bk-root .bk:before, .bk-root .bk:after {\\n  font-family: \\\"Josefin Sans\\\";\\n}\\n\\nimg {\\n  max-width: 100%;\\n}\\n\\n#container {\\n  padding:0px;\\n  height:100vh;\\n  width: 100vw;\\n  max-width: 100vw;\\n}\\n\\n#sidebar .mdc-list {\\n  padding-left: 5px;\\n  padding-right: 5px;\\n}\\n\\n.mdc-drawer-app-content {\\n  flex: auto;\\n  position: relative;\\n  overflow: hidden;\\n}\\n\\n.mdc-drawer {\\n  background: #FAFAFA; /* GRAY 50 */\\n}\\n\\n.mdc-drawer-app-content {\\n  margin-left: 0 !important;\\n}\\n\\n.title-bar {\\n  display: contents;\\n  justify-content: center;\\n  align-content: center;\\n  width: 100%;\\n}\\n\\n.mdc-top-app-bar .bk-menu {\\n  color: black\\n}\\n\\n.app-header {\\n  display: contents;\\n  padding-left: 10px;\\n  font-size: 1.25em;\\n}\\n\\nimg.app-logo {\\n  padding-right: 10px;\\n  font-size: 28px;\\n  height: 30px;\\n  max-width: inherit;\\n  padding-top: 12px;\\n  padding-bottom: 6px;\\n}\\n\\n#app-title {\\n  padding-right: 12px;\\n  padding-left: 12px;\\n}\\n\\n.title {\\n  font-family: \\\"Josefin Sans\\\";\\n  color: #fff;\\n  text-decoration: none;\\n  text-decoration-line: none;\\n  text-decoration-style: initial;\\n  text-decoration-color: initial;\\n  font-weight: 400;\\n  font-size: 2em;\\n  line-height: 2em;\\n  white-space: nowrap;\\n}\\n\\n.main-content {\\n  overflow-y: scroll;\\n  overflow-x: auto;\\n}\\n\\n#header {\\n  position: absolute;\\n  z-index: 7;\\n}\\n\\n#header-items {\\n  width: 100%;\\n  margin-left:15px;\\n}\\n\\n.pn-busy-container {\\n  align-items: center;\\n  justify-content: center;\\n  display: flex;\\n}\\n\\n.mdc-drawer__content {\\n  overflow-x: hidden;\\n}\\n.mdc-drawer__content, .main-content {\\n  padding: 12px;\\n}\\n\\n.main-content {\\n  height: calc(100vh - 88px);\\n  max-height: calc(100vh - 88px);\\n  padding-right: 32px;\\n}\\n\\nbutton.mdc-button.mdc-card-button {\\n  color: transparent;\\n  height: 50px;\\n}\\n\\np.mdc-button {\\n  display: none;\\n}\\n\\ndiv.mdc-card {\\n  border-radius: 0px\\n}\\n\\n.mdc-card .card-header {\\n  display: flex;\\n}\\n\\n.mdc-card-title {\\n  font-family: \\\"Josefin Sans\\\";\\n  font-weight: bold;\\n  align-items: center;\\n  display: flex !important;\\n  position: relative !important;\\n}\\n\\n.mdc-card-title:nth-child(2) {\\n  margin-left: -1.4em;\\n}\\n\\n.pn-modal {\\n  overflow-y: scroll;\\n  width: 100%;\\n  display: none;\\n  position: absolute;\\n  top: 0;\\n  left: 0;\\n}\\n\\n.pn-modal-content {\\n  font-family: \\\"Josefin Sans\\\";\\n  background-color: #0e0e0e;\\n  margin: auto;\\n  margin-top: 25px;\\n  margin-bottom: 25px;\\n  padding: 15px 20px 20px 20px;\\n  border: 1px solid #888;\\n  width: 80% !important;\\n}\\n\\n.pn-modal-close {\\n  position: absolute;\\n  right: 25px;\\n  z-index: 100;\\n}\\n\\n.pn-modal-close:hover,\\n.pn-modal-close:focus {\\n  color: #000;\\n  text-decoration: none;\\n  cursor: pointer;\\n}\\n\\n.custom_button_bokeh button.bk-btn.bk-btn-default {\\n    font-size:48pt;\\n    background-color: #05b7ff;\\n    border-color: #05b7ff;\\n}\");\n    },    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\nfunction(Bokeh) {} // ensure no trailing comma for IE\n  ];\n\n  function run_inline_js() {\n    if ((root.Bokeh !== undefined) || (force === true)) {\n      for (var i = 0; i < inline_js.length; i++) {\n        inline_js[i].call(root, root.Bokeh);\n      }\n      // Cache old bokeh versions\n      if (Bokeh != undefined && !reloading) {\n\tvar NewBokeh = root.Bokeh;\n\tif (Bokeh.versions === undefined) {\n\t  Bokeh.versions = new Map();\n\t}\n\tif (NewBokeh.version !== Bokeh.version) {\n\t  Bokeh.versions.set(NewBokeh.version, NewBokeh)\n\t}\n\troot.Bokeh = Bokeh;\n      }} else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    }\n    root._bokeh_is_initializing = false\n  }\n\n  function load_or_wait() {\n    // Implement a backoff loop that tries to ensure we do not load multiple\n    // versions of Bokeh and its dependencies at the same time.\n    // In recent versions we use the root._bokeh_is_initializing flag\n    // to determine whether there is an ongoing attempt to initialize\n    // bokeh, however for backward compatibility we also try to ensure\n    // that we do not start loading a newer (Panel>=1.0 and Bokeh>3) version\n    // before older versions are fully initialized.\n    if (root._bokeh_is_initializing && Date.now() > root._bokeh_timeout) {\n      root._bokeh_is_initializing = false;\n      root._bokeh_onload_callbacks = undefined;\n      console.log(\"Bokeh: BokehJS was loaded multiple times but one version failed to initialize.\");\n      load_or_wait();\n    } else if (root._bokeh_is_initializing || (typeof root._bokeh_is_initializing === \"undefined\" && root._bokeh_onload_callbacks !== undefined)) {\n      setTimeout(load_or_wait, 100);\n    } else {\n      Bokeh = root.Bokeh;\n      bokeh_loaded = Bokeh != null && (Bokeh.version === py_version || (Bokeh.versions !== undefined && Bokeh.versions.has(py_version)));\n      root._bokeh_is_initializing = true\n      root._bokeh_onload_callbacks = []\n      if (!reloading && (!bokeh_loaded || is_dev)) {\n\troot.Bokeh = undefined;\n      }\n      load_libs(css_urls, js_urls, js_modules, js_exports, function() {\n\tconsole.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n\trun_inline_js();\n      });\n    }\n  }\n  // Give older versions of the autoload script a head-start to ensure\n  // they initialize before we start loading newer version.\n  setTimeout(load_or_wait, 100)\n}(window));",
      "application/vnd.holoviews_load.v0+json": ""
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": "\nif ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {\n  window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}\n}\n\n\n    function JupyterCommManager() {\n    }\n\n    JupyterCommManager.prototype.register_target = function(plot_id, comm_id, msg_handler) {\n      if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        comm_manager.register_target(comm_id, function(comm) {\n          comm.on_msg(msg_handler);\n        });\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        window.PyViz.kernels[plot_id].registerCommTarget(comm_id, function(comm) {\n          comm.onMsg = msg_handler;\n        });\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        google.colab.kernel.comms.registerTarget(comm_id, (comm) => {\n          var messages = comm.messages[Symbol.asyncIterator]();\n          function processIteratorResult(result) {\n            var message = result.value;\n            console.log(message)\n            var content = {data: message.data, comm_id};\n            var buffers = []\n            for (var buffer of message.buffers || []) {\n              buffers.push(new DataView(buffer))\n            }\n            var metadata = message.metadata || {};\n            var msg = {content, buffers, metadata}\n            msg_handler(msg);\n            return messages.next().then(processIteratorResult);\n          }\n          return messages.next().then(processIteratorResult);\n        })\n      }\n    }\n\n    JupyterCommManager.prototype.get_client_comm = function(plot_id, comm_id, msg_handler) {\n      if (comm_id in window.PyViz.comms) {\n        return window.PyViz.comms[comm_id];\n      } else if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        var comm = comm_manager.new_comm(comm_id, {}, {}, {}, comm_id);\n        if (msg_handler) {\n          comm.on_msg(msg_handler);\n        }\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        var comm = window.PyViz.kernels[plot_id].connectToComm(comm_id);\n        comm.open();\n        if (msg_handler) {\n          comm.onMsg = msg_handler;\n        }\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        var comm_promise = google.colab.kernel.comms.open(comm_id)\n        comm_promise.then((comm) => {\n          window.PyViz.comms[comm_id] = comm;\n          if (msg_handler) {\n            var messages = comm.messages[Symbol.asyncIterator]();\n            function processIteratorResult(result) {\n              var message = result.value;\n              var content = {data: message.data};\n              var metadata = message.metadata || {comm_id};\n              var msg = {content, metadata}\n              msg_handler(msg);\n              return messages.next().then(processIteratorResult);\n            }\n            return messages.next().then(processIteratorResult);\n          }\n        }) \n        var sendClosure = (data, metadata, buffers, disposeOnDone) => {\n          return comm_promise.then((comm) => {\n            comm.send(data, metadata, buffers, disposeOnDone);\n          });\n        };\n        var comm = {\n          send: sendClosure\n        };\n      }\n      window.PyViz.comms[comm_id] = comm;\n      return comm;\n    }\n    window.PyViz.comm_manager = new JupyterCommManager();\n    \n\n\nvar JS_MIME_TYPE = 'application/javascript';\nvar HTML_MIME_TYPE = 'text/html';\nvar EXEC_MIME_TYPE = 'application/vnd.holoviews_exec.v0+json';\nvar CLASS_NAME = 'output';\n\n/**\n * Render data to the DOM node\n */\nfunction render(props, node) {\n  var div = document.createElement(\"div\");\n  var script = document.createElement(\"script\");\n  node.appendChild(div);\n  node.appendChild(script);\n}\n\n/**\n * Handle when a new output is added\n */\nfunction handle_add_output(event, handle) {\n  var output_area = handle.output_area;\n  var output = handle.output;\n  if ((output.data == undefined) || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n    return\n  }\n  var id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n  var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n  if (id !== undefined) {\n    var nchildren = toinsert.length;\n    var html_node = toinsert[nchildren-1].children[0];\n    html_node.innerHTML = output.data[HTML_MIME_TYPE];\n    var scripts = [];\n    var nodelist = html_node.querySelectorAll(\"script\");\n    for (var i in nodelist) {\n      if (nodelist.hasOwnProperty(i)) {\n        scripts.push(nodelist[i])\n      }\n    }\n\n    scripts.forEach( function (oldScript) {\n      var newScript = document.createElement(\"script\");\n      var attrs = [];\n      var nodemap = oldScript.attributes;\n      for (var j in nodemap) {\n        if (nodemap.hasOwnProperty(j)) {\n          attrs.push(nodemap[j])\n        }\n      }\n      attrs.forEach(function(attr) { newScript.setAttribute(attr.name, attr.value) });\n      newScript.appendChild(document.createTextNode(oldScript.innerHTML));\n      oldScript.parentNode.replaceChild(newScript, oldScript);\n    });\n    if (JS_MIME_TYPE in output.data) {\n      toinsert[nchildren-1].children[1].textContent = output.data[JS_MIME_TYPE];\n    }\n    output_area._hv_plot_id = id;\n    if ((window.Bokeh !== undefined) && (id in Bokeh.index)) {\n      window.PyViz.plot_index[id] = Bokeh.index[id];\n    } else {\n      window.PyViz.plot_index[id] = null;\n    }\n  } else if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n    var bk_div = document.createElement(\"div\");\n    bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n    var script_attrs = bk_div.children[0].attributes;\n    for (var i = 0; i < script_attrs.length; i++) {\n      toinsert[toinsert.length - 1].childNodes[1].setAttribute(script_attrs[i].name, script_attrs[i].value);\n    }\n    // store reference to server id on output_area\n    output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n  }\n}\n\n/**\n * Handle when an output is cleared or removed\n */\nfunction handle_clear_output(event, handle) {\n  var id = handle.cell.output_area._hv_plot_id;\n  var server_id = handle.cell.output_area._bokeh_server_id;\n  if (((id === undefined) || !(id in PyViz.plot_index)) && (server_id !== undefined)) { return; }\n  var comm = window.PyViz.comm_manager.get_client_comm(\"hv-extension-comm\", \"hv-extension-comm\", function () {});\n  if (server_id !== null) {\n    comm.send({event_type: 'server_delete', 'id': server_id});\n    return;\n  } else if (comm !== null) {\n    comm.send({event_type: 'delete', 'id': id});\n  }\n  delete PyViz.plot_index[id];\n  if ((window.Bokeh !== undefined) & (id in window.Bokeh.index)) {\n    var doc = window.Bokeh.index[id].model.document\n    doc.clear();\n    const i = window.Bokeh.documents.indexOf(doc);\n    if (i > -1) {\n      window.Bokeh.documents.splice(i, 1);\n    }\n  }\n}\n\n/**\n * Handle kernel restart event\n */\nfunction handle_kernel_cleanup(event, handle) {\n  delete PyViz.comms[\"hv-extension-comm\"];\n  window.PyViz.plot_index = {}\n}\n\n/**\n * Handle update_display_data messages\n */\nfunction handle_update_output(event, handle) {\n  handle_clear_output(event, {cell: {output_area: handle.output_area}})\n  handle_add_output(event, handle)\n}\n\nfunction register_renderer(events, OutputArea) {\n  function append_mime(data, metadata, element) {\n    // create a DOM node to render to\n    var toinsert = this.create_output_subarea(\n    metadata,\n    CLASS_NAME,\n    EXEC_MIME_TYPE\n    );\n    this.keyboard_manager.register_events(toinsert);\n    // Render to node\n    var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n    render(props, toinsert[0]);\n    element.append(toinsert);\n    return toinsert\n  }\n\n  events.on('output_added.OutputArea', handle_add_output);\n  events.on('output_updated.OutputArea', handle_update_output);\n  events.on('clear_output.CodeCell', handle_clear_output);\n  events.on('delete.Cell', handle_clear_output);\n  events.on('kernel_ready.Kernel', handle_kernel_cleanup);\n\n  OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n    safe: true,\n    index: 0\n  });\n}\n\nif (window.Jupyter !== undefined) {\n  try {\n    var events = require('base/js/events');\n    var OutputArea = require('notebook/js/outputarea').OutputArea;\n    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n      register_renderer(events, OutputArea);\n    }\n  } catch(err) {\n  }\n}\n",
      "application/vnd.holoviews_load.v0+json": ""
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>*[data-root-id],\n",
       "*[data-root-id] > * {\n",
       "  box-sizing: border-box;\n",
       "  font-family: var(--jp-ui-font-family);\n",
       "  font-size: var(--jp-ui-font-size1);\n",
       "  color: var(--vscode-editor-foreground, var(--jp-ui-font-color1));\n",
       "}\n",
       "\n",
       "/* Override VSCode background color */\n",
       ".cell-output-ipywidget-background:has(> .cell-output-ipywidget-background\n",
       "    > .lm-Widget\n",
       "    > *[data-root-id]),\n",
       ".cell-output-ipywidget-background:has(> .lm-Widget > *[data-root-id]) {\n",
       "  background-color: transparent !important;\n",
       "}\n",
       "</style>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.holoviews_exec.v0+json": "",
      "text/html": [
       "<div id='970dc25f-8e61-4374-a302-6fd929247b71'>\n",
       "  <div id=\"b07b193c-c367-4881-93a0-037028b21add\" data-root-id=\"970dc25f-8e61-4374-a302-6fd929247b71\" style=\"display: contents;\"></div>\n",
       "</div>\n",
       "<script type=\"application/javascript\">(function(root) {\n",
       "  var docs_json = {\"814bc044-2495-4970-a7ea-d523a725bb35\":{\"version\":\"3.2.0\",\"title\":\"Bokeh Application\",\"roots\":[{\"type\":\"object\",\"name\":\"panel.models.browser.BrowserInfo\",\"id\":\"970dc25f-8e61-4374-a302-6fd929247b71\"},{\"type\":\"object\",\"name\":\"panel.models.comm_manager.CommManager\",\"id\":\"283c5b48-294f-4a4b-b746-bb66a0f3a487\",\"attributes\":{\"plot_id\":\"970dc25f-8e61-4374-a302-6fd929247b71\",\"comm_id\":\"d9bdd1c439fd4c88938fda3847f7eb03\",\"client_comm_id\":\"6013665f62924caa9f90c1e5daa80aac\"}}],\"defs\":[{\"type\":\"model\",\"name\":\"ReactiveHTML1\"},{\"type\":\"model\",\"name\":\"FlexBox1\",\"properties\":[{\"name\":\"align_content\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"align_items\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"flex_direction\",\"kind\":\"Any\",\"default\":\"row\"},{\"name\":\"flex_wrap\",\"kind\":\"Any\",\"default\":\"wrap\"},{\"name\":\"justify_content\",\"kind\":\"Any\",\"default\":\"flex-start\"}]},{\"type\":\"model\",\"name\":\"FloatPanel1\",\"properties\":[{\"name\":\"config\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"contained\",\"kind\":\"Any\",\"default\":true},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"right-top\"},{\"name\":\"offsetx\",\"kind\":\"Any\",\"default\":null},{\"name\":\"offsety\",\"kind\":\"Any\",\"default\":null},{\"name\":\"theme\",\"kind\":\"Any\",\"default\":\"primary\"},{\"name\":\"status\",\"kind\":\"Any\",\"default\":\"normalized\"}]},{\"type\":\"model\",\"name\":\"GridStack1\",\"properties\":[{\"name\":\"mode\",\"kind\":\"Any\",\"default\":\"warn\"},{\"name\":\"ncols\",\"kind\":\"Any\",\"default\":null},{\"name\":\"nrows\",\"kind\":\"Any\",\"default\":null},{\"name\":\"allow_resize\",\"kind\":\"Any\",\"default\":true},{\"name\":\"allow_drag\",\"kind\":\"Any\",\"default\":true},{\"name\":\"state\",\"kind\":\"Any\",\"default\":[]}]},{\"type\":\"model\",\"name\":\"drag1\",\"properties\":[{\"name\":\"slider_width\",\"kind\":\"Any\",\"default\":5},{\"name\":\"slider_color\",\"kind\":\"Any\",\"default\":\"black\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":50}]},{\"type\":\"model\",\"name\":\"click1\",\"properties\":[{\"name\":\"terminal_output\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"debug_name\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"clears\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"FastWrapper1\",\"properties\":[{\"name\":\"object\",\"kind\":\"Any\",\"default\":null},{\"name\":\"style\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"NotificationAreaBase1\",\"properties\":[{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"NotificationArea1\",\"properties\":[{\"name\":\"notifications\",\"kind\":\"Any\",\"default\":[]},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0},{\"name\":\"types\",\"kind\":\"Any\",\"default\":[{\"type\":\"map\",\"entries\":[[\"type\",\"warning\"],[\"background\",\"#ffc107\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-exclamation-triangle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]},{\"type\":\"map\",\"entries\":[[\"type\",\"info\"],[\"background\",\"#007bff\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-info-circle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]}]}]},{\"type\":\"model\",\"name\":\"Notification\",\"properties\":[{\"name\":\"background\",\"kind\":\"Any\",\"default\":null},{\"name\":\"duration\",\"kind\":\"Any\",\"default\":3000},{\"name\":\"icon\",\"kind\":\"Any\",\"default\":null},{\"name\":\"message\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"notification_type\",\"kind\":\"Any\",\"default\":null},{\"name\":\"_destroyed\",\"kind\":\"Any\",\"default\":false}]},{\"type\":\"model\",\"name\":\"TemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"BootstrapTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"MaterialTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]}]}};\n",
       "  var render_items = [{\"docid\":\"814bc044-2495-4970-a7ea-d523a725bb35\",\"roots\":{\"970dc25f-8e61-4374-a302-6fd929247b71\":\"b07b193c-c367-4881-93a0-037028b21add\"},\"root_ids\":[\"970dc25f-8e61-4374-a302-6fd929247b71\"]}];\n",
       "  var docs = Object.values(docs_json)\n",
       "  if (!docs) {\n",
       "    return\n",
       "  }\n",
       "  const py_version = docs[0].version.replace('rc', '-rc.')\n",
       "  const is_dev = py_version.indexOf(\"+\") !== -1 || py_version.indexOf(\"-\") !== -1\n",
       "  function embed_document(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
       "    for (const render_item of render_items) {\n",
       "      for (const root_id of render_item.root_ids) {\n",
       "\tconst id_el = document.getElementById(root_id)\n",
       "\tif (id_el.children.length && (id_el.children[0].className === 'bk-root')) {\n",
       "\t  const root_el = id_el.children[0]\n",
       "\t  root_el.id = root_el.id + '-rendered'\n",
       "\t}\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "  function get_bokeh(root) {\n",
       "    if (root.Bokeh === undefined) {\n",
       "      return null\n",
       "    } else if (root.Bokeh.version !== py_version && !is_dev) {\n",
       "      if (root.Bokeh.versions === undefined || !root.Bokeh.versions.has(py_version)) {\n",
       "\treturn null\n",
       "      }\n",
       "      return root.Bokeh.versions.get(py_version);\n",
       "    } else if (root.Bokeh.version === py_version) {\n",
       "      return root.Bokeh\n",
       "    }\n",
       "    return null\n",
       "  }\n",
       "  function is_loaded(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    return (Bokeh != null && Bokeh.Panel !== undefined)\n",
       "  }\n",
       "  if (is_loaded(root)) {\n",
       "    embed_document(root);\n",
       "  } else {\n",
       "    var attempts = 0;\n",
       "    var timer = setInterval(function(root) {\n",
       "      if (is_loaded(root)) {\n",
       "        clearInterval(timer);\n",
       "        embed_document(root);\n",
       "      } else if (document.readyState == \"complete\") {\n",
       "        attempts++;\n",
       "        if (attempts > 200) {\n",
       "          clearInterval(timer);\n",
       "\t  var Bokeh = get_bokeh(root)\n",
       "\t  if (Bokeh == null || Bokeh.Panel == null) {\n",
       "            console.warn(\"Panel: ERROR: Unable to run Panel code because Bokeh or Panel library is missing\");\n",
       "\t  } else {\n",
       "\t    console.warn(\"Panel: WARNING: Attempting to render but not all required libraries could be resolved.\")\n",
       "\t    embed_document(root)\n",
       "\t  }\n",
       "        }\n",
       "      }\n",
       "    }, 25, root)\n",
       "  }\n",
       "})(window);</script>"
      ]
     },
     "metadata": {
      "application/vnd.holoviews_exec.v0+json": {
       "id": "970dc25f-8e61-4374-a302-6fd929247b71"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Base imports\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cml.plot import initialize_bokeh\n",
    "from cml.panel import initialize_panel\n",
    "from jupyterthemes.stylefx import set_nb_theme\n",
    "from bokeh.io import show\n",
    "initialize_bokeh()\n",
    "initialize_panel()\n",
    "set_nb_theme(\"onedork\")\n",
    "rng = np.random.RandomState(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"regression\"></a>\n",
    "## Simple learning problem\n",
    "\n",
    "Imagine that a certain process somewhere follows the form of a quadratic relationship\n",
    "\n",
    "$$\n",
    " y = a x^{2} + bx + c \n",
    "$$\n",
    "\n",
    "In this case, all the **unknown parameters** are that of a polynomial model, therefore we have $\\theta = \\{a, b, c\\}$. However, this is clearly an ideal (clean) case, whereas in natural observations, there might be some noise in our observations\n",
    "\n",
    "$$\n",
    " y = a x^{2} + bx + c +\\epsilon \\text{ with } \\epsilon \\in [-0.1, 0.1]\n",
    "$$\n",
    "\n",
    "An example of such noisy observations for different parameters is given below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mInit signature:\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpoly1d\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mc_or_r\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvariable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mDocstring:\u001b[0m     \n",
      "A one-dimensional polynomial class.\n",
      "\n",
      ".. note::\n",
      "   This forms part of the old polynomial API. Since version 1.4, the\n",
      "   new polynomial API defined in `numpy.polynomial` is preferred.\n",
      "   A summary of the differences can be found in the\n",
      "   :doc:`transition guide </reference/routines.polynomials>`.\n",
      "\n",
      "A convenience class, used to encapsulate \"natural\" operations on\n",
      "polynomials so that said operations may take on their customary\n",
      "form in code (see Examples).\n",
      "\n",
      "Parameters\n",
      "----------\n",
      "c_or_r : array_like\n",
      "    The polynomial's coefficients, in decreasing powers, or if\n",
      "    the value of the second parameter is True, the polynomial's\n",
      "    roots (values where the polynomial evaluates to 0).  For example,\n",
      "    ``poly1d([1, 2, 3])`` returns an object that represents\n",
      "    :math:`x^2 + 2x + 3`, whereas ``poly1d([1, 2, 3], True)`` returns\n",
      "    one that represents :math:`(x-1)(x-2)(x-3) = x^3 - 6x^2 + 11x -6`.\n",
      "r : bool, optional\n",
      "    If True, `c_or_r` specifies the polynomial's roots; the default\n",
      "    is False.\n",
      "variable : str, optional\n",
      "    Changes the variable used when printing `p` from `x` to `variable`\n",
      "    (see Examples).\n",
      "\n",
      "Examples\n",
      "--------\n",
      "Construct the polynomial :math:`x^2 + 2x + 3`:\n",
      "\n",
      ">>> p = np.poly1d([1, 2, 3])\n",
      ">>> print(np.poly1d(p))\n",
      "   2\n",
      "1 x + 2 x + 3\n",
      "\n",
      "Evaluate the polynomial at :math:`x = 0.5`:\n",
      "\n",
      ">>> p(0.5)\n",
      "4.25\n",
      "\n",
      "Find the roots:\n",
      "\n",
      ">>> p.r\n",
      "array([-1.+1.41421356j, -1.-1.41421356j])\n",
      ">>> p(p.r)\n",
      "array([ -4.44089210e-16+0.j,  -4.44089210e-16+0.j]) # may vary\n",
      "\n",
      "These numbers in the previous line represent (0, 0) to machine precision\n",
      "\n",
      "Show the coefficients:\n",
      "\n",
      ">>> p.c\n",
      "array([1, 2, 3])\n",
      "\n",
      "Display the order (the leading zero-coefficients are removed):\n",
      "\n",
      ">>> p.order\n",
      "2\n",
      "\n",
      "Show the coefficient of the k-th power in the polynomial\n",
      "(which is equivalent to ``p.c[-(i+1)]``):\n",
      "\n",
      ">>> p[1]\n",
      "2\n",
      "\n",
      "Polynomials can be added, subtracted, multiplied, and divided\n",
      "(returns quotient and remainder):\n",
      "\n",
      ">>> p * p\n",
      "poly1d([ 1,  4, 10, 12,  9])\n",
      "\n",
      ">>> (p**3 + 4) / p\n",
      "(poly1d([ 1.,  4., 10., 12.,  9.]), poly1d([4.]))\n",
      "\n",
      "``asarray(p)`` gives the coefficient array, so polynomials can be\n",
      "used in all functions that accept arrays:\n",
      "\n",
      ">>> p**2 # square of polynomial\n",
      "poly1d([ 1,  4, 10, 12,  9])\n",
      "\n",
      ">>> np.square(p) # square of individual coefficients\n",
      "array([1, 4, 9])\n",
      "\n",
      "The variable used in the string representation of `p` can be modified,\n",
      "using the `variable` parameter:\n",
      "\n",
      ">>> p = np.poly1d([1,2,3], variable='z')\n",
      ">>> print(p)\n",
      "   2\n",
      "1 z + 2 z + 3\n",
      "\n",
      "Construct a polynomial from its roots:\n",
      "\n",
      ">>> np.poly1d([1, 2], True)\n",
      "poly1d([ 1., -3.,  2.])\n",
      "\n",
      "This is the same polynomial as obtained by:\n",
      "\n",
      ">>> np.poly1d([1, -1]) * np.poly1d([1, -2])\n",
      "poly1d([ 1, -3,  2])\n",
      "\u001b[0;31mFile:\u001b[0m           ~/Library/Python/3.9/lib/python/site-packages/numpy/__init__.py\n",
      "\u001b[0;31mType:\u001b[0m           type\n",
      "\u001b[0;31mSubclasses:\u001b[0m     orthopoly1d"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "a, b, c = 5, 2, 1\n",
    "poly = np.poly1d([a, b, c])\n",
    "X = np.linspace(0, 1, 100)\n",
    "poly(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters to our function\n",
    "eps = 0.1\n",
    "a, b, c = 5, 2, 1\n",
    "# Generating the corresponding data\n",
    "x = np.linspace(0, 1, 100)\n",
    "poly = np.poly1d([a, b, c])\n",
    "epsilon = np.random.uniform(-eps, eps, x.shape)\n",
    "y = poly(x) + epsilon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4b494ae8e8a4adb8fb6b00a0a0c164b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'77338d71-5c38-4fbd-8a83-589dc74bf243': {'version…"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from cml.plot import center_plot, scatter\n",
    "plot = (center_plot(scatter(x, y, title=\"Simple quadratic problem\", toolbar_location=\"left\")))\n",
    "plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our main problem is that this function can follow different types of parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = [[5, -5, 4], [-2, 1, 0], [0.1, 1, 1]]\n",
    "# Generating the x axis\n",
    "x = np.linspace(0, 1, 100)\n",
    "plots = []\n",
    "for p in range(len(params)):\n",
    "    poly = np.poly1d(params[p])\n",
    "    epsilon = np.random.uniform(-eps, eps, x.shape)\n",
    "    y = poly(x) + epsilon\n",
    "    plots.append(scatter(x, y, title=\"Problem \"+(str(p+1))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20943826d8df4753a3234eb8580eda21",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'2d928e06-eec4-49c8-bff7-a0ff5b796b7a': {'version…"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bokeh.models import Div\n",
    "from bokeh.layouts import column, row\n",
    "plot = center_plot(column(Div(text = \"Observing different problems\", styles={'font-size': '250%'}), row(*plots)))\n",
    "plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In real-life settings, this function can also have different levels of noise, as exemplified in the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = [3, 0, 1]\n",
    "noise_levels = [0.1, 1.0, 8.0]\n",
    "# Generating the x axis\n",
    "x = np.linspace(0, 1, 100)\n",
    "plots = []\n",
    "for p in range(len(noise_levels)):\n",
    "    poly = np.poly1d(params)\n",
    "    epsilon = np.random.uniform(-noise_levels[p],noise_levels[p],x.shape)\n",
    "    y = poly(x) + epsilon\n",
    "    plots.append(scatter(x, y, title=\"Problem \"+(str(p+1))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aa33071183dd417eb1a3ffe099049a2f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'ae09d377-c91a-4991-93e7-aed3f8a76b6a': {'version…"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot = center_plot(column(Div(text = \"Different amounts of noise\", styles={'font-size': '250%'}), row(*plots)))\n",
    "plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarize, we will have some observations of a function, and we would like to optimize a function that gets as close as possible to the real function that generated this data. Here, we plot the real function and also _subsample_ our number of observations (having only a few points to find the corresponding function)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79967c1ebf12446481c23c3f61630875",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'665cabce-02a7-499e-b32e-b67317261174': {'version…"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generating the data and subsampling\n",
    "x_all = np.linspace(0, 1, 100); x_plot = np.linspace(0, 1, 100)\n",
    "rng.shuffle(x_all); x = np.sort(x_all[:50])\n",
    "poly = np.poly1d([3,0,1])\n",
    "# Adding some external noise\n",
    "epsilon = np.random.uniform(-0.2,0.2,x.shape)\n",
    "y = poly(x)+ epsilon\n",
    "p = scatter(x, y, title=\"Learning problem with groundtruth\")\n",
    "p.line(x_plot, poly(x_plot), line_width=6, line_alpha=0.6, color=\"green\", legend_label=r\"True function\")\n",
    "plot = (center_plot(p))\n",
    "plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summarizing our observations (interactive)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating the object\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "160011068ab6480492a2a90ecfa6ae82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'76e6f4b7-d68d-4844-b1a6-7b95aaa297d0': {'version…"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from cml.tasks import RegressionPolynomial\n",
    "explorer = RegressionPolynomial()\n",
    "explorer.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using learning libraries (`scikit-learn`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a first grip on what machine learning does, we will rely on the `scikit-learn` library. This contains already coded models and learning procedure, that will allow us to _learn_ the parameters of this unknown function.\n",
    "\n",
    "Here we already know that we want to use a `PolynomialFeatures` model to perfom `LinearRegression` and that this polynomial should be of degree 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mInit signature:\u001b[0m\n",
      "\u001b[0mLinearRegression\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mfit_intercept\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mcopy_X\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mpositive\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mDocstring:\u001b[0m     \n",
      "Ordinary least squares Linear Regression.\n",
      "\n",
      "LinearRegression fits a linear model with coefficients w = (w1, ..., wp)\n",
      "to minimize the residual sum of squares between the observed targets in\n",
      "the dataset, and the targets predicted by the linear approximation.\n",
      "\n",
      "Parameters\n",
      "----------\n",
      "fit_intercept : bool, default=True\n",
      "    Whether to calculate the intercept for this model. If set\n",
      "    to False, no intercept will be used in calculations\n",
      "    (i.e. data is expected to be centered).\n",
      "\n",
      "copy_X : bool, default=True\n",
      "    If True, X will be copied; else, it may be overwritten.\n",
      "\n",
      "n_jobs : int, default=None\n",
      "    The number of jobs to use for the computation. This will only provide\n",
      "    speedup in case of sufficiently large problems, that is if firstly\n",
      "    `n_targets > 1` and secondly `X` is sparse or if `positive` is set\n",
      "    to `True`. ``None`` means 1 unless in a\n",
      "    :obj:`joblib.parallel_backend` context. ``-1`` means using all\n",
      "    processors. See :term:`Glossary <n_jobs>` for more details.\n",
      "\n",
      "positive : bool, default=False\n",
      "    When set to ``True``, forces the coefficients to be positive. This\n",
      "    option is only supported for dense arrays.\n",
      "\n",
      "    .. versionadded:: 0.24\n",
      "\n",
      "Attributes\n",
      "----------\n",
      "coef_ : array of shape (n_features, ) or (n_targets, n_features)\n",
      "    Estimated coefficients for the linear regression problem.\n",
      "    If multiple targets are passed during the fit (y 2D), this\n",
      "    is a 2D array of shape (n_targets, n_features), while if only\n",
      "    one target is passed, this is a 1D array of length n_features.\n",
      "\n",
      "rank_ : int\n",
      "    Rank of matrix `X`. Only available when `X` is dense.\n",
      "\n",
      "singular_ : array of shape (min(X, y),)\n",
      "    Singular values of `X`. Only available when `X` is dense.\n",
      "\n",
      "intercept_ : float or array of shape (n_targets,)\n",
      "    Independent term in the linear model. Set to 0.0 if\n",
      "    `fit_intercept = False`.\n",
      "\n",
      "n_features_in_ : int\n",
      "    Number of features seen during :term:`fit`.\n",
      "\n",
      "    .. versionadded:: 0.24\n",
      "\n",
      "feature_names_in_ : ndarray of shape (`n_features_in_`,)\n",
      "    Names of features seen during :term:`fit`. Defined only when `X`\n",
      "    has feature names that are all strings.\n",
      "\n",
      "    .. versionadded:: 1.0\n",
      "\n",
      "See Also\n",
      "--------\n",
      "Ridge : Ridge regression addresses some of the\n",
      "    problems of Ordinary Least Squares by imposing a penalty on the\n",
      "    size of the coefficients with l2 regularization.\n",
      "Lasso : The Lasso is a linear model that estimates\n",
      "    sparse coefficients with l1 regularization.\n",
      "ElasticNet : Elastic-Net is a linear regression\n",
      "    model trained with both l1 and l2 -norm regularization of the\n",
      "    coefficients.\n",
      "\n",
      "Notes\n",
      "-----\n",
      "From the implementation point of view, this is just plain Ordinary\n",
      "Least Squares (scipy.linalg.lstsq) or Non Negative Least Squares\n",
      "(scipy.optimize.nnls) wrapped as a predictor object.\n",
      "\n",
      "Examples\n",
      "--------\n",
      ">>> import numpy as np\n",
      ">>> from sklearn.linear_model import LinearRegression\n",
      ">>> X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])\n",
      ">>> # y = 1 * x_0 + 2 * x_1 + 3\n",
      ">>> y = np.dot(X, np.array([1, 2])) + 3\n",
      ">>> reg = LinearRegression().fit(X, y)\n",
      ">>> reg.score(X, y)\n",
      "1.0\n",
      ">>> reg.coef_\n",
      "array([1., 2.])\n",
      ">>> reg.intercept_\n",
      "3.0...\n",
      ">>> reg.predict(np.array([[3, 5]]))\n",
      "array([16.])\n",
      "\u001b[0;31mFile:\u001b[0m           ~/Library/Python/3.9/lib/python/site-packages/sklearn/linear_model/_base.py\n",
      "\u001b[0;31mType:\u001b[0m           ABCMeta\n",
      "\u001b[0;31mSubclasses:\u001b[0m     "
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "?LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.pipeline import make_pipeline\n",
    "# Our data to fit\n",
    "X = x[:, np.newaxis]\n",
    "# Degree of our polynomial\n",
    "degree = 30;\n",
    "# Create our polynomial model for regression\n",
    "model = make_pipeline(PolynomialFeatures(degree), LinearRegression())\n",
    "# Fit the parameters of this model\n",
    "model.fit(X, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have trained the model, we can perform _predictions_ from it, meaning that we can infer the output of the function at values that we did not observe originally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model error : 0.049550050282625414\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48eb818ff7084bfea431c2edc34aadeb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'64d21f38-7cd8-401a-b92a-aa1f7e6efa18': {'version…"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inference points (not observed)\n",
    "X_plot = x_plot[:, np.newaxis]\n",
    "# Predict the values\n",
    "y_plot = model.predict(X_plot)\n",
    "# Compute the error of our model at observed points\n",
    "Y_model_err = np.sqrt(np.mean(np.square(y-model.predict(X))))\n",
    "print(f'Model error : {Y_model_err}')\n",
    "# Plot the result\n",
    "p = scatter(x, y, title=\"Training a scikit-learn model\")\n",
    "p.line(x_plot, poly(x_plot), line_width=6, line_alpha=0.6, color=\"red\", legend_label=r\"Trained model\")\n",
    "plot = (center_plot(p))\n",
    "plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cml.tasks import RegressionPolynomialSolver\n",
    "explorer = RegressionPolynomialSolver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d0380875f5c47818bdab5bb15ab53c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'70454c8e-9c70-4a8c-b0ce-2a4d061201d6': {'version…"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def solve(x, y, degree):\n",
    "    X = x[:, np.newaxis]\n",
    "    # Create our polynomial model for regression\n",
    "    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())\n",
    "    # Fit the parameters of this model\n",
    "    model.fit(X, y)\n",
    "    # Predict the values\n",
    "    x_predict = np.linspace(np.min(x), np.max(x), 200)[:, np.newaxis]\n",
    "    y_model = model.predict(x_predict)\n",
    "    return x_predict[:, 0], y_model #np.array(jnp.zeros(y_model.shape))\n",
    "\n",
    "explorer.solve = solve\n",
    "explorer.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"linear\"></a>\n",
    "# Simple linear regression \n",
    "\n",
    "As discussed previously, regression allows to model the relationships that exist between inputs $\\mathbf{x}\\in\\mathbb{R}^{n}$ and a continuous output $y\\in\\mathbb{R}$. In the case of **linear** regression, we assume that the data that we observe comes from a linear relationship in the input, so that\n",
    "$$y = w_0 + w_1x_1 + w_2x_2 + ... + w_nx_n + \\epsilon$$\n",
    "with $\\epsilon$ exhibiting the *observation noise* (also called *residual error*) that is always present in measurements.\n",
    "\n",
    "To understand how we could learn a model approximating this relationship, we start with the case of *simple* linear regression, where $\\mathbf{x}\\in\\mathbb{R}$. This implies that our model will follow\n",
    "$$\\bar{y} = w_0 + w_1 x$$\n",
    "\n",
    "We will measure the errors made by our model by using the Mean Squared Error (MSE) loss, defined as \n",
    "$$\\mathcal{L}_{MSE}\\left( \\bar{\\mathbf{y}},\\theta \\right) = \\sum_{i=1}^{n} \\left| y_{i} - \\bar{y}_{i} \\right|^{2} = \\sum_{i=1}^{n} \\left| y_{i} - (w_{0} + w_{1} x_{i}) \\right|^{2}$$\n",
    "\n",
    "Then our goal is to find the most adequate set of parameters $\\theta = \\{w_{0}, w_{1}\\}$, which are those that minimize the MSE loss defined previously. Therefore, we aim to obtain\n",
    "$$\\theta^{*}= \\underset{\\theta}{\\text{argmin }} \\mathcal{L}\\left( \\bar{\\mathbf{y}}, \\mathbf{y} \\mid f_{\\theta}, \\theta \\right)$$\n",
    "\n",
    "To do so, we will implement the **gradient descent** algorithm discussed in the course.\n",
    "\n",
    "## Manual implementation - `NumPy`\n",
    "\n",
    "We start by performing a *full manual implementation*, in the sense that we need to manually derive the gradient in order to apply the gradient descent updates. To do so, we will rely on [NumPy](https://numpy.org/), which is a fundamental library for numerical computing offering support for N-dimensional arrays and scientific computing tasks, such as linear algebra, statistical analysis, and matrix manipulation. We strongly encourage you to learn NumPy through the set of [tutorials](https://numpy.org/learn/). For the sake of this introductory tutorial, we will provide the explanation for all of the functions that we will use in this first exercise. After that exercise, we will assume for the rest of the course that knowledge of Numpy should be found online. \n",
    "\n",
    "We start by import libraries (`NumPy`) and set a random seed to ensure that the random number generator produces always a reproducible series of random numbers.\n",
    "\n",
    "**Used functions**\n",
    "- `np.random.seed`: sets the seed for the NumPy random number generator. [Documentation](https://numpy.org/doc/stable/reference/random/generated/numpy.random.seed.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mDocstring:\u001b[0m\n",
      "seed(self, seed=None)\n",
      "\n",
      "Reseed a legacy MT19937 BitGenerator\n",
      "\n",
      "Notes\n",
      "-----\n",
      "This is a convenience, legacy function.\n",
      "\n",
      "The best practice is to **not** reseed a BitGenerator, rather to\n",
      "recreate a new one. This method is here for legacy reasons.\n",
      "This example demonstrates best practice.\n",
      "\n",
      ">>> from numpy.random import MT19937\n",
      ">>> from numpy.random import RandomState, SeedSequence\n",
      ">>> rs = RandomState(MT19937(SeedSequence(123456789)))\n",
      "# Later, you want to restart the stream\n",
      ">>> rs = RandomState(MT19937(SeedSequence(987654321)))\n",
      "\u001b[0;31mType:\u001b[0m      builtin_function_or_method"
     ]
    }
   ],
   "source": [
    "?np.random.seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate a synthetic dataset.\n",
    "\n",
    "For the sake of this exercise, we will generate the data ourselves, so that we know the true values that we are looking for in advance. Hence, we define a linear relationship following\n",
    "$$y = w^{t}_0 + w^{t}_1 x + \\epsilon$$\n",
    "\n",
    "**Used functions**\n",
    "- `np.random.rand`: generates an array of random numbers uniformly distributed over [0, 1). [Documentation](https://numpy.org/doc/stable/reference/random/generated/numpy.random.rand.html)\n",
    "- `np.random.randn`: generates an array of random numbers from the standard normal distribution (mean 0, variance 1). [Documentation](https://numpy.org/doc/stable/reference/random/generated/numpy.random.randn.html)\n",
    "- `np.ones`: generates an array of ones with a specified shape. [Documentation](https://numpy.org/doc/stable/reference/generated/numpy.ones.html)\n",
    "- `np.c_`: concatenates arrays along the second axis. [Documentation](https://numpy.org/doc/stable/reference/generated/numpy.c_.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.   0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1  0.11 0.12 0.13\n",
      " 0.14 0.15 0.16 0.17 0.18 0.19 0.2  0.21 0.22 0.23 0.24 0.25 0.26 0.27\n",
      " 0.28 0.29 0.3  0.31 0.32 0.33 0.34 0.35 0.36 0.37 0.38 0.39 0.4  0.41\n",
      " 0.42 0.43 0.44 0.45 0.46 0.47 0.48 0.49 0.5  0.51 0.52 0.53 0.54 0.55\n",
      " 0.56 0.57 0.58 0.59 0.6  0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69\n",
      " 0.7  0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.8  0.81 0.82 0.83\n",
      " 0.84 0.85 0.86 0.87 0.88 0.89 0.9  0.91 0.92 0.93 0.94 0.95 0.96 0.97\n",
      " 0.98 0.99 1.  ]\n",
      "[2.   2.03 2.06 2.09 2.12 2.15 2.18 2.21 2.24 2.27 2.3  2.33 2.36 2.39\n",
      " 2.42 2.45 2.48 2.51 2.54 2.57 2.6  2.63 2.66 2.69 2.72 2.75 2.78 2.81\n",
      " 2.84 2.87 2.9  2.93 2.96 2.99 3.02 3.05 3.08 3.11 3.14 3.17 3.2  3.23\n",
      " 3.26 3.29 3.32 3.35 3.38 3.41 3.44 3.47 3.5  3.53 3.56 3.59 3.62 3.65\n",
      " 3.68 3.71 3.74 3.77 3.8  3.83 3.86 3.89 3.92 3.95 3.98 4.01 4.04 4.07\n",
      " 4.1  4.13 4.16 4.19 4.22 4.25 4.28 4.31 4.34 4.37 4.4  4.43 4.46 4.49\n",
      " 4.52 4.55 4.58 4.61 4.64 4.67 4.7  4.73 4.76 4.79 4.82 4.85 4.88 4.91\n",
      " 4.94 4.97 5.  ]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "true_w0 = 2\n",
    "true_w1 = 3\n",
    "n_obs = 100\n",
    "x = [0, 1]\n",
    "# Input to model\n",
    "x = np.linspace(x[0], x[1], n_obs + 1)\n",
    "# Output (target)\n",
    "y = true_w1 * x + true_w0\n",
    "# REAL LIFE\n",
    "print(x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mse_loss(y_bar, y_true):\n",
    "    return np.sum((y_true - y_bar) ** 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "true_w0 = 2\n",
    "true_w1 = 3\n",
    "x_min = 0\n",
    "x_max = 1\n",
    "n_obs = 100\n",
    "x = np.linspace(x_min, x_max, n_obs)\n",
    "y = true_w1 * x + true_w0 + eps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mse_loss(y, y_bar):\n",
    "    return np.sum((y - y_bar) ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing the MSE loss\n",
    "\n",
    "We will measure the errors made by our model by using the Mean Squared Error (MSE) loss, defined as \n",
    "$$\\mathcal{L}_{MSE}\\left( \\bar{\\mathbf{y}},\\theta \\right) = \\sum_{i=1}^{n} \\left| y_{i} - \\bar{y}_{i} \\right|^{2} = \\sum_{i=1}^{n} \\left| y_{i} - (w_{0} + w_{1} x_{i}) \\right|^{2}$$\n",
    "\n",
    "This will allow us to evaluate the performances of our model, but is also the basis for the following gradient descent algorithm.\n",
    "\n",
    "**Used functions**\n",
    "- `np.sum`: computes the sum of the provided array (optionally across a provided `axis`). [Documentation](https://numpy.org/doc/stable/reference/generated/numpy.mean.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mse_loss(y, y_bar):\n",
    "    return np.sum((y - y_bar) ** 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mse_loss(y, y_bar):\n",
    "    return np.sum((y - y_bar) ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implement the gradient descent algorithm.\n",
    "\n",
    "As seen in the course Initialize the weight $w_1$ and bias $w_0$ to small random values\n",
    "1. Evaluate the predictions made by our model \n",
    "$$\\hat{y} = w_{1}x + w_0$$\n",
    "2. Compute the mean squared error (MSE) loss between the predicted values and the ground truth labels: \n",
    "$$\\mathcal{L}_{MSE} = \\frac{1}{n} \\sum_{i=1}^n (\\bar{y_i} - y_i)^2$$\n",
    "3. Compute the gradients of the loss with respect to the weight and bias (see derivation in the course)\n",
    "$$\\frac{\\partial \\mathcal{L}}{\\partial w_1} = \\sum_{i} 2 * (\\bar{y}_{i} - y_{i})*x_{i}$$\n",
    "$$\\frac{\\partial \\mathcal{L}}{\\partial w_0} = \\sum_{i} 2 * (\\bar{y}_{i} - y_{i})$$\n",
    "4. Update the weights and bias using the gradients and a learning rate $\\eta$: \n",
    "$$w_1 \\leftarrow w_1 - \\eta \\frac{\\partial \\mathcal{L}}{\\partial w_1}$$ \n",
    "$$w_0 \\leftarrow w_0 - \\eta \\frac{\\partial \\mathcal{L}}{\\partial w_0}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-0.5622875292409727\n",
      "2.9999999999999822\n"
     ]
    }
   ],
   "source": [
    "# Hyper-parameters\n",
    "nb_epochs = 10000\n",
    "eta = 1e-3\n",
    "# Initialize random parameters w0, w1\n",
    "w0 = np.random.randn()\n",
    "w1 = np.random.randn()\n",
    "print(w1)\n",
    "# Complicated model\n",
    "def model(x, params):\n",
    "    w0, w1 = params\n",
    "    return w0 + w1 * x\n",
    "# Loop\n",
    "for epoch in range(nb_epochs):\n",
    "#   1. Compute output\n",
    "    y_bar = model(x, (w0, w1))\n",
    "#   2. Compute loss\n",
    "    loss = mse_loss(y, y_bar)\n",
    "#   3. Compute gradient\n",
    "    gradients = grad(model_loss)(x, y, w1, w0)\n",
    "#   4. Update parameters\n",
    "    w0 = w0 - eta * gradients[0]\n",
    "    w1 = w1 - eta * gradients[w1]\n",
    "print(w1)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.21107226845711613\n",
      "3.4737044007295648\n"
     ]
    }
   ],
   "source": [
    "n_iter = 1000\n",
    "loss = np.zeros(n_iter)\n",
    "eta = 1e-3\n",
    "def model(x, w0, w1):\n",
    "    return x * w1 + w0\n",
    "w0 = np.random.randn()\n",
    "w1 = np.random.randn()\n",
    "for i in range(n_iter):\n",
    "    y_bar = model(x, w0, w1)\n",
    "    l_mse = mse_loss(y, y_bar)\n",
    "    #print(f'w0 = {w0}, w1 = {w1}, loss = {l_mse}')\n",
    "    dldw0 = np.sum(2 * (y_bar - y))\n",
    "    w0 = w0 - eta * dldw0\n",
    "    dldw1 = np.sum(2 * (y_bar - y) * x)\n",
    "    w1 = w1 - eta * dldw1\n",
    "    loss[i] = l_mse\n",
    "import matplotlib.pyplot as plt\n",
    "print(w0)\n",
    "print(w1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.480582228699151\n",
      "0.20689795422754362\n"
     ]
    }
   ],
   "source": [
    "# Parameters\n",
    "n_iter = 1000\n",
    "lr = 0.001\n",
    "def gradient_descent(x, y, n_iter, lr):\n",
    "    # Initialize the parameters\n",
    "    w_1 = np.random.randn()\n",
    "    w_0 = np.random.randn()\n",
    "    # Perform gradient descent\n",
    "    for i in range(n_iter):\n",
    "        # 1. Calculate the predictions\n",
    "        y_bar = w_1 * x + w_0\n",
    "        # 2. Compute the loss\n",
    "        loss = mse_loss(y, y_bar)\n",
    "        # 3. Calculate the gradients\n",
    "        dw_1 = 2 * np.sum((y_bar - y) * x)\n",
    "        dw_0 = 2 * np.sum((y_bar - y))\n",
    "        # 4. Update the parameters\n",
    "        w_1 -= lr * dw_1\n",
    "        w_0 -= lr * dw_0\n",
    "    return w_1, w_0\n",
    "w_1, w_0 = gradient_descent(x, y, n_iter, lr)\n",
    "print(w_1)\n",
    "print(w_0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the results.\n",
    "\n",
    "We can see how well our model fits to the observed data by plotting it against the observed samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f936f6265b7b4cb19a4638a3c20a8117",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'052573b5-c03b-4f12-9ab5-4b49fc119efc': {'version…"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = scatter(x, y, title=\"Simple linear regression\")\n",
    "p.line(x, w_1 * x + w_0, line_width=6, line_alpha=0.6, color=\"red\", legend_label=r\"Learned model\")\n",
    "plot = (center_plot(p))\n",
    "plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observing our solution interactively\n",
    "\n",
    "In the following code, you can observe the behavior of the model by playing interactively with the properties of the original problem. \n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> Note that each change in the properties of the original problem requires to run the gradient descent algorithm entirely each time.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2cb28a3f59934015b9bbdb522169af2f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'75acb6d9-183d-497c-a759-63d495c45e49': {'version…"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from cml.tasks import RegressionLinearSolver\n",
    "explorer = RegressionLinearSolver()\n",
    "def solve(x, y):\n",
    "    w_1, w_0 = gradient_descent(x, y, n_iter, lr)\n",
    "    x_predict = np.linspace(np.min(x), np.max(x), 200)\n",
    "    y_model = w_1 * x_predict + w_0\n",
    "    return np.array(x_predict), np.array(y_model) #np.array(np.zeros(y_model.shape))\n",
    "\n",
    "explorer.solve = solve\n",
    "explorer.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Going further\n",
    "> **Exercise 1 \\[Learning rate\\]**: Experiment with different learning rates (e.g., 0.001, 0.01, 0.1, 1) and observe how they affect the convergence of the gradient descent algorithm. Plot the convergence history (cost function value vs. iteration) for each learning rate.\n",
    "\n",
    "> **Exercise 2 \\[Initialization\\]**: Experiment with different initializations of the coefficients (zeros, random values, etc.) and observe their impact on the convergence of the gradient descent algorithm.\n",
    "\n",
    "> **Exercise 3 \\[Variants\\]**: Implement and compare different variants of gradient descent, such as stochastic gradient descent (SGD) and mini-batch gradient descent. Analyze their convergence properties and computational efficiency.\n",
    "\n",
    "> **Exercise 4 \\[Regularization\\]**: Implement Lasso and Ridge regression with gradient descent by incorporating the regularization terms into the cost function and gradient calculations. Compare their performance with the standard linear regression implementation and analyze their impact on the learned coefficients.\n",
    "\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Discovering automatic differentiation - `JAX`\n",
    "\n",
    "The previous implementation required us to perform manual differentiation of our loss function to understand how to update the parameters. However, large developments have been made in the field of **automatic differentiation**. \n",
    "\n",
    "The recent library [JAX](https://github.com/google/jax) extends NumPy with this automatic differentiation feature (*autograd*), while providing a functional approach to numerical computing, allowing for easy gradient computation and just-in-time (JIT) compilation. Its ability to handle complex and custom gradients makes JAX particularly well-suited for advanced research projects. Similar to NumPy, we strongly encourage you to learn JAX through the set of [tutorials](https://jax.readthedocs.io/en/latest/), but we will also provide here the explanation for all of the functions that we will use in this first exercise. After that, we will assume that knowledge of JAX should be found online. \n",
    "\n",
    "Note that `JAX` has been thought as an extension of `NumPy`, therefore an extremely large portion of its API simply mirrors the `NumPy` functions by adding automatic differentiation features to it.\n",
    "\n",
    "**Used functions**\n",
    "- `random.PRNGKey`: function to generate a key for the pseudorandom number generator. [Documentation](https://jax.readthedocs.io/en/latest/_autosummary/jax.random.PRNGKey.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jax.numpy as jnp\n",
    "from jax import grad, jit, random\n",
    "key = random.PRNGKey(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset.\n",
    "\n",
    "We can simply keep our previous approach to generating the dataset but we will rely on `jnp.ndarray` instead of `np.ndarray`. We also provide the code for splitting the dataset between a `training` and `validation` dataset, as discussed in the course\n",
    "\n",
    "**Used functions**\n",
    "- `random.split`: function to split a PRNGKey into a list of subkeys [Documentation](https://jax.readthedocs.io/en/latest/_autosummary/jax.random.split.html)\n",
    "- `random.uniform`: function to generate an array of uniformly distributed random numbers [Documentation](https://jax.readthedocs.io/en/latest/_autosummary/jax.random.uniform.html)\n",
    "- `random.normal`: function to generate an array of normally distributed random numbers [Documentation](https://jax.readthedocs.io/en/latest/_autosummary/jax.random.normal.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "true_w0 = 2\n",
    "true_w1 = 3\n",
    "key_0, key_1, key = random.split(key, 3)\n",
    "x = random.uniform(key_0, (100,))\n",
    "y = true_w0 + true_w1 * x + (random.normal(key_1, (100,)) * 0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting our dataset\n",
    "train_size = int(0.8 * len(x))\n",
    "x_train, x_valid = x[:train_size], x[train_size:]\n",
    "y_train, y_valid = y[:train_size], y[train_size:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss function\n",
    "\n",
    "Similarly Define the cost function and its gradient.\n",
    "\n",
    "**Used functions**\n",
    "- `jnp.mean`: function to compute the mean of an array [Documentation](https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.mean.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_function_plus_loss(x, y, w_1, w_0):\n",
    "    y_bar = x * w_1 + w_0\n",
    "    # VERY COOOOOOOMPLEX\n",
    "    return jnp.sum((y - y_bar) ** 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<function model_loss at 0x2fe4253a0>\n"
     ]
    }
   ],
   "source": [
    "print(grad(model_loss))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_function(x, y, w_1, w_0):\n",
    "    y_pred = x * w_1 + w_0\n",
    "    residuals = y_pred - y\n",
    "    return jnp.mean(residuals**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Used functions**\n",
    "- `jit`: function to compile a function for faster execution [Documentation](https://jax.readthedocs.io/en/latest/jit.html)\n",
    "- `grad`: function to compute the gradient of a function [Documentation](https://jax.readthedocs.io/en/latest/jax.html#jax.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "grad_loss_function = jit(grad(loss_function, argnums=[2, 3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implement the gradient descent algorithm.\n",
    "\n",
    "Note that nowhere in the code do we need to explicitly define the gradients of the different variables. Yet, the optimization will be performed adequately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gradient Descent coefficients: [3.0639799] [1.9661584]\n"
     ]
    }
   ],
   "source": [
    "def gradient_descent(key, x, y, lr=0.05, n_iter=1000):\n",
    "    m = x.shape\n",
    "    k_0, k_1 = random.split(key, 2)\n",
    "    w_0 = random.normal(k_0, (1,))\n",
    "    w_1 = random.normal(k_1, (1,))\n",
    "    loss_history = []\n",
    "    for _ in range(n_iter):\n",
    "        gradients = grad_loss_function(x, y, w_1, w_0)\n",
    "        #print(gradients[0])\n",
    "        w_1 -= lr * gradients[0]\n",
    "        w_0 -= lr * gradients[1]\n",
    "        loss_history.append(loss_function(x, y, w_1, w_0))\n",
    "    return w_1, w_0, loss_history\n",
    "\n",
    "key_gd, key = random.split(key, 2)\n",
    "w_1, w_0, loss_history = gradient_descent(key_gd, x_train, y_train)\n",
    "print(\"Gradient Descent coefficients:\", w_1, w_0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate our performances\n",
    "\n",
    "Thanks to our previous split of the dataset, we are now able to evaluate the performances of our model on *unseen data*, which is the overarching goal of building such machine learning models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE for Gradient Descent: 0.03851323\n"
     ]
    }
   ],
   "source": [
    "def mean_squared_error(y_true, y_pred):\n",
    "    return np.mean((y_true - y_pred)**2)\n",
    "\n",
    "y_valid_gd = w_1 * x_valid + w_0\n",
    "mse_gd = mean_squared_error(y_valid, y_valid_gd)\n",
    "print(\"MSE for Gradient Descent:\", mse_gd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the results.\n",
    "\n",
    "As previously we can witness our results, and also evaluate our solution with our interactive solver."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c644a5682e724fcd831885b5db38a135",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'97cc0b10-fdb7-45ad-9355-9c3ee5c1a84f': {'version…"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = scatter(np.array(x), np.array(y), title=\"Linear regression\")\n",
    "p.line(np.array(x), np.array(w_1 * x + w_0), line_width=6, line_alpha=0.6, color=\"red\", legend_label=r\"Learned model\")\n",
    "plot = (center_plot(p))\n",
    "plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7318431202f0477591d00690cfc58844",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'9bb03be5-cd1d-4676-9651-f7e591c3a31c': {'version…"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from cml.tasks import RegressionLinearSolver\n",
    "explorer = RegressionLinearSolver()\n",
    "global key\n",
    "key = random.PRNGKey(23)\n",
    "def solve(x, y):\n",
    "    global key\n",
    "    key_gd, key = random.split(key, 2)\n",
    "    w_1, w_0, loss_history = gradient_descent(key_gd, x, y)\n",
    "    x_predict = np.linspace(np.min(x), np.max(x), 200)\n",
    "    y_model = w_1 * x_predict + w_0\n",
    "    return np.array(x_predict), np.array(y_model) #np.array(jnp.zeros(y_model.shape))\n",
    "\n",
    "explorer.solve = solve\n",
    "explorer.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Going further\n",
    "\n",
    "> You are encouraged to experiment in a similar way as outlined for NumPy with different learning rates, maximum number of iterations, and regularization techniques, as well as compare the performance (both accuracy and speed) of the implemented algorithms with `scikit-learn` built-in linear regression functions.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"capacity\"></a>\n",
    "## Understanding model capacity and selection\n",
    "\n",
    "\n",
    "In real-life problem, we are aiming to find the parameters of a model, but we do not really know what is the _real_ function underlying this process. So what we can decide to select _any_ function of _any_ **capacity** (complexity of the function). One of the problem with that, is that if we have a too simple function, it will _underfit_ (it is not complex enough for our observations). On the opposite end, if we have a function which is too complex, it might be able to _fit through all training points exactly_ ... even though there is noise in our observations ! This is examplified in the following\n",
    "\n",
    "<img src=\"images/01_soa_function_families.png\" align=\"center\"/>\n",
    "\n",
    "We can observe this idea and play with it directly by trying to find a function approximating our previous observations with a polynomial function chosen to have a degree inside \\([1,2,8]\\).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Depending on the _capacity_ of the model, what we can observe is that\n",
    "\n",
    "- `capacity too low   -> underfitting   : prediction variance >  noise variance`\n",
    "- `adequate capacity  -> good fit       : prediction variance == noise variance`\n",
    "- `capacity too high  -> overfitting    : prediction variance <  noise variance`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A similar example can be given for a classification problem in two dimensions as follows\n",
    "\n",
    "<img src=\"images/01_underfit.png\" align=\"center\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "\n",
    "In the following, we define the exercises that you should fill for this session. These go further than what we have seen together in the course, but they are based on the exact same principles, simply with slightly more complex definitions. We provide an overall guideline for successfully implementing each of the exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1 - Linear classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> In this exercise, we will implement linear classification using NumPy and JAX. The goal of this exercise is to gain a better understanding of how linear classification works and how to implement it ourselves using our two first libraries of choice, namely NumPy and JAX. To complete this exercise, you will need to have a basic understanding of NumPy and JAX. You can use the resources provided in the previous exercises to learn more about these libraries.\n",
    "\n",
    "We help you out by first defining a simple linear classification problem to solve\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import make_blobs\n",
    "# Properties of the problem\n",
    "n_observations = 200\n",
    "noise = 0.2\n",
    "c1_center = [-2, -1]\n",
    "c2_center = [2, 1]\n",
    "# Create points\n",
    "x_coords, y_class = make_blobs(n_samples=n_observations, centers=[c1_center, c2_center], n_features=2, cluster_std=0.55)\n",
    "x_data = x_coords + (noise * np.random.randn(n_observations, 2))\n",
    "#x_data = x_coords\n",
    "y_classes = y_class\n",
    "y_classes[y_classes == 0] = -1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 1.1 - Implement linear classification with NumPy\n",
    "\n",
    "> 1. Check the training data with two classes that are linearly separable.\n",
    "> 2. Initialize weights and bias.\n",
    "> 3. Implement the forward pass of the linear classifier.\n",
    "> 4. Implement the hinge loss for classification.\n",
    "> 5. Derive the gradients (backward pass) of the linear classifier.\n",
    "> 6. Implement gradient descent to optimize the weights and bias.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Going further (optional)\n",
    "> Try to replace the Hinge loss by the cross-entropy loss in your implementation\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 1.2 - Implement linear classification with JAX\n",
    "\n",
    "> 1. Initialize weights and bias.\n",
    "> 2. Define the loss function using the jax.nn.softmax_cross_entropy_with_logits function.\n",
    "> 3. Implement gradient descent to optimize the weights and bias.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 1.3 - Compare the results of the NumPy and JAX implementations.\n",
    "> 1. Plot the decision boundary for each implementation.\n",
    "> 2. Compare the training time and accuracy of each implementation.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Going further (optional)\n",
    "> Implement a more complex dataset and compare the results of the NumPy and JAX implementations.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2 - Polynomial regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> In this exercise, you will implement polynomial regression using NumPy and JAX. This should be a quite straightforward extension of what we have seen until now. We deliberately removed details of the implementation (list of points to adress) so that you can start defining your first entire machine learning setup by yourself.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 2.1 - Implement polynomial regression using NumPy with gradient descent.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 2.2. Implement polynomial regression using JAX with gradient descent.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 2.3 - Compare the results of both implementations and the speed of the optimization algorithm.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"audio\"></a>\n",
    "# Audio applications\n",
    "\n",
    "In order to test our algorithms on audio and music data, we will work with several datasets. We will both be using well-known state-of-art datasets (through the TF dataset system). But first, we will rely on a simple dataset (`Musclefish`) that should be downloaded on your local computer first from this [link](https://nubo.ircam.fr/index.php/s/ByK4QL7nE4Mq5MA)\n",
    "\n",
    "**The following set of instructions will perform that automatically for you :)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl https://nubo.ircam.fr/index.php/s/ByK4QL7nE4Mq5MA/download/musclefish.zip --output musclefish.zip\n",
    "!unzip musclefish.zip\n",
    "!mkdir data\n",
    "!mv musclefish data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset details\n",
    "\n",
    "  |**Type**|*Origin*|\n",
    "  |-------:|:---------|\n",
    "  |**Classification**|[*MuscleFish*](http://knight.cis.temple.edu/~vasilis/Courses/CIS750/Papers/muscle_fish.pdf) dataset|\n",
    "  |**Music-speech**|[*MIREX Recognition*](http://www.music-ir.org/mirex/wiki/2015:Music/Speech_Classification_and_Detection) set|\n",
    "  |**Source separation**|[*SMC Mirum*](http://smc.inesctec.pt/research/data-2/) dataset|\n",
    "  |**Speech recognition**|[*CMU Arctic*](http://festvox.org/cmu_arctic/) dataset|\n",
    "\n",
    "**Unzip the file and place the `musclefish` folder inside a `data` folder in the root of this notebook**\n",
    "For the first parts of the tutorial, we will mostly rely solely on the classification dataset. In order to facilitate the interactions, we provide a dataset class called `AudioSupervisedDataset` that will allow to import all audio datasets along the tutorials. This class also contains a set of holders and will allow us to perform batch-wise gradient descent.\n",
    "\n",
    "```Python\n",
    "class AudioSupervisedDataset():\n",
    "    \"\"\"\n",
    "    Helper class to import datasets\n",
    "    % class_path  : Path to the dataset (string)\n",
    "    % type       : Type of dataset (string: 'classify', 'plain', 'metadata')\n",
    "    \"\"\" \n",
    "    audio_files\n",
    "    labels\n",
    "    labels_names\n",
    "    num_examples \n",
    "    num_classes \n",
    "```\n",
    "\n",
    "We also provide a simplified function `import_dataset` that is demonstrated below.\n",
    "\n",
    "***\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "**_Exercise_**  \n",
    "\n",
    "  1. Launch the import procedure and check the corresponding structure\n",
    "  2. Code a count function that prints the name and number of examples for each classes \n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recursive search in /tmp/data//musclefish\n",
      "Metal device set to: Apple M1 Pro\n",
      "(16, 44100)\n",
      "(16,)\n"
     ]
    }
   ],
   "source": [
    "from cml.data import import_dataset, AudioSupervisedDataset\n",
    "data_path = \"data/\"\n",
    "batch_size = 16\n",
    "# Instantiate the dataset class\n",
    "train_dataset = import_dataset(data_path, \"musclefish\", 'train', batch_size=batch_size)\n",
    "# Generate a batch of data from the train dataset\n",
    "batch_x, batch_y = next(train_dataset)\n",
    "# Print the shape of the input and output batch tensors\n",
    "print(batch_x.shape)  # (16, 44100)\n",
    "print(batch_y.shape)  # (16,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52\n"
     ]
    }
   ],
   "source": [
    "#%% Q-0.1.2 - Count function to print the number of examples in each class along with class label\n",
    "\n",
    "n_batch = 0\n",
    "train_dataset._reset_generator()\n",
    "for batch_x, batch_y in iter(train_dataset):\n",
    "    n_batch += 1\n",
    "    ######################\n",
    "    # YOUR CODE GOES HERE\n",
    "    ######################\n",
    "\n",
    "print(n_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing\n",
    "\n",
    "We will rely on a set of spectral transforms that allow to obtain a more descriptive view over the audio information. As most of these are out of the scope of the machine learning course, we redirect you to a [signal processing course](https://ccrma.stanford.edu/~jos/sasp/) proposed by [Julius O. Smith](https://ccrma.stanford.edu/~jos/).  \n",
    "\n",
    "The following functions to compute various types of transforms are given as part of the basic audio dataset class, in the `cml.data.audio` package\n",
    "\n",
    "  |**Name**|*Transform*|\n",
    "  |-------:|:----------|\n",
    "  |`stft`       |[Short-term Fourier transform](https://en.wikipedia.org/wiki/Short-time_Fourier_transform)|\n",
    "  |`mel`  |[Mel scale](https://en.wikipedia.org/wiki/Mel_scale) transform|\n",
    "  |`chroma` |[Chromas vector](https://en.wikipedia.org/wiki/Harmonic_pitch_class_profiles)|\n",
    "  |`cqt`        |[Constant-Q](https://en.wikipedia.org/wiki/Constant_Q_transform) transform|\n",
    "\n",
    "In order to perform the various computations, we provide the following function, in the `AudioSupervisedDataset` which performs the different transforms on a complete dataset.  \n",
    "\n",
    "``` Python\n",
    "dataset.transform(index, name)\n",
    "    \"\"\" index   : Specific index in our dataset \"\"\"\n",
    "    \"\"\" name    : Name of the transform to apply \"\"\"\n",
    "\n",
    "# The name can be selected in \n",
    "\"stft\"      # Power spectrum (STFT)\n",
    "\"mel\"       # Spectrum in Mel scale\n",
    "\"chroma\"    # Chroma vectors\n",
    "\"cqt\"       # Constant-Q transform\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "**Exercise**  \n",
    "\n",
    "  1. Launch the transform computation procedure and check the corresponding structure\n",
    "  2. For each class, select a random element and plot its various transforms on a single plot. You should obtain plots similar to those shown afterwards.\n",
    "  3. For each transform, try to spot major pros and cons of their representation.\n",
    "  \n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 0.2 - Pre-process the audio to obtain spectral transforms \n",
    "power_spec = train_dataset.transform(0, \"stft\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%% Q-0.2.2 - Plot the various transforms \n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Features\n",
    "\n",
    "<div markdown = \"1\">\n",
    "\n",
    "As you might have noted from the previous exercice, most spectral transforms have a very high dimensionality, and might not be suited to exhibit the relevant structure of different classes. To that end, we provide a set of functions for computing several spectral features in the `cml.data` package, we redirect interested readers to this [exhaustive article](http://recherche.ircam.fr/anasyn/peeters/ARTICLES/Peeters_2003_cuidadoaudiofeatures.pdf) on spectral features computation.\n",
    "\n",
    "  |**File**|*Transform*|\n",
    "  |-------:|:----------|\n",
    "  |`spectral_centroid`|Spectral centroid|\n",
    "  |`spectral_bandwidth`|Spectral bandwidth|\n",
    "  |`spectral_contrast`|Spectral contrast|\n",
    "  |`spectral_flatness`|Spectral flatness|\n",
    "  |`spectral_rolloff`|Spectral rolloff|\n",
    "\n",
    "Once again, we provide a function to perform the computation of different features on a complete set. Note that for each feature, we obtain the temporal evolution in a vector. Therefore, for further learning tasks, if you wish to obtain simplified spaces, you might need to compute the mean and standard deviation of each feature.\n",
    "\n",
    "``` Python\n",
    "dataset.feature(index, name)\n",
    "    \"\"\" index   : Specific index in our dataset \"\"\"\n",
    "    \"\"\" name    : Name of the feature to obtain \"\"\"\n",
    "\n",
    "# Name can be chosen inside the following\n",
    "\"loudness\"    # Loudness\n",
    "\"centroid\"    # Spectral centroid\n",
    "\"bandwidth\"   # Spectral bandwidth\n",
    "\"contrast\"    # Spectral contrast\n",
    "\"flatness\"    # Spectral flatness\n",
    "\"rolloff\"     # Spectral rolloff\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "**Exercise**\n",
    "\n",
    "  1. Launch the feature computation procedure and check the corresponding structure\n",
    "  2. This time for each class, superimpose the plots of various features on a single plot, along with a boxplot of mean and standard deviations. You should obtain plots similar to those shown afterwards.\n",
    "  3. What conclusions can you make on the discriminative power of each feature ?\n",
    "  4. Perform scatter plots of the mean features for all the dataset, while coloring different classes.\n",
    "  5. What conclusions can you make on the discriminative power of mean features ?\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 0.3 - Compute temporal spectral features\n",
    "power_spec = train_dataset.feature(0, \"centroid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%% Q-0.3.2 - Plot the various features \n",
    "\n",
    "# Use these styles for boxplot\n",
    "boxprops=dict(linewidth=3, color='white')\n",
    "whiskerprops=dict(linewidth=3, color='white')\n",
    "medianprops=dict(linewidth=2.5, color='firebrick')\n",
    "flierprops = dict(markeredgecolor='white', markerfacecolor='firebrick')\n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%% Q-0.3.4 - Observe the distribution of classes for different features\n",
    "\n",
    "# This allows to use 3D rendering in matplotlib\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "plt.figure(figsize=(12,8))\n",
    "# Create a vector of random colors for each class\n",
    "colorVect = np.zeros((3, len(data_struct[\"class_names\"])));\n",
    "for c in range(len(data_struct[\"class_names\"])):\n",
    "    colorVect[:,c] = np.random.rand(3);\n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it for this tutorial, now remember that we can use any form of description (features) as a basis for learning algorithms. We will see in the next tutorial what we an do with these features."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}