Day2/13_parallel_computing/parallel_computing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Parallel and Distributed Computing\n",
    "\n",
    "**Multithreading** refers to the ability of a processor to execute multiple threads concurrently, where each thread runs a process. **Multiprocessing** refers to the ability of a system to run multiple processors concurrently, where each processor can run one or more threads.\n",
    "\n",
    "![](https://miro.medium.com/v2/resize:fit:720/format:webp/1*hZ3guTdmDMXevFiT5Z3VrA.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Multithreading\n",
    "Multi-threading is a programming technique that allows **multiple threads** of execution to run concurrently within a single process. Julia provides built-in support for multi-threading, making it easy to write concurrent code. To use multi-threading in Julia, you can use the Threads standard library.\n",
    "\n",
    "The number of execution threads is controlled either by using the `-t`/`--threads` command line argument \n",
    "\n",
    "```shell\n",
    "julia --threads 10 my_script.jl\n",
    "```\n",
    "\n",
    "or by using the `JULIA_NUM_THREADS` environment variable. This can also be changed in VSCode setting. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When both `JULIA_NUM_THREADS` and `-t`/`--threads` are specified, then `-t`/`--threads` takes precedence.\n",
    "\n",
    "The number of threads can either be specified as an integer (`--threads=4`) or as auto (`--threads=auto`), where auto sets the number of threads to the number of local CPU threads.\n",
    "\n",
    "To check the number of threads available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Threads.nthreads()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "Multithreading in Julia is **super easy**: just put `Threads.@threads` in front of the loop you want to parrallelize.\n",
    "\n",
    "Here is a stupid example where we assign to an array the id of the thread assigned within the loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4.0, 4.0, 5.0, 5.0, 3.0, 3.0, 2.0, 2.0, 1.0, 1.0]\n"
     ]
    }
   ],
   "source": [
    "a = zeros(10)\n",
    "\n",
    "Threads.@threads for i = 1:10\n",
    "    a[i] = Threads.threadid()\n",
    "end\n",
    "println(a)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is another example to make sure that `Threads.@threads` does indeed run code in parrallel: we compare a simple loop against a loop using `Threads.@threads`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  10.022 s (358 allocations: 10.03 KiB)\n"
     ]
    }
   ],
   "source": [
    "myfun() = sleep(1)\n",
    "function myfun_loop()\n",
    "    for i = 1:10\n",
    "        myfun()\n",
    "    end\n",
    "end\n",
    "\n",
    "using BenchmarkTools\n",
    "@btime myfun_loop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  2.004 s (143 allocations: 5.84 KiB)\n"
     ]
    }
   ],
   "source": [
    "function myfun_loop_multithreading()\n",
    "    Threads.@threads for i = 1:10\n",
    "        myfun()\n",
    "    end\n",
    "end\n",
    "\n",
    "@btime myfun_loop_multithreading()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Be careful with race condition!\n",
    "A race condition is a situation that occurs in computer programming when multiple processes or threads access and manipulate shared resources or data concurrently. In this scenario, the outcome of the program is dependent on the order and timing of the execution of each process, which can lead to unexpected and undesirable results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "567\n"
     ]
    },
    {
     "ename": "AssertionError",
     "evalue": "AssertionError: println(length(a)) == 1000",
     "output_type": "error",
     "traceback": [
      "AssertionError: println(length(a)) == 1000\n",
      "\n",
      "Stacktrace:\n",
      " [1] top-level scope\n",
      "   @ ~/ETHZ/PostDoc_ELE/teaching/WSL_workshop_Julia/material/Day2/13_parallel_computing/parallel_computing.ipynb:5"
     ]
    }
   ],
   "source": [
    "a = Float64[]\n",
    "Threads.@threads for i in 1:1000\n",
    "    push!(a, i)\n",
    "end\n",
    "@assert println(length(a)) == 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### `lock`\n",
    "\n",
    "The `lock` function can be used to prevent race condition\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100\n"
     ]
    }
   ],
   "source": [
    "a = []\n",
    "lk = ReentrantLock()\n",
    "Threads.@threads for i in 1:100\n",
    "    x = i^2\n",
    "    lock(lk) do\n",
    "        push!(a, x)\n",
    "    end\n",
    "end\n",
    "println(length(a)) # ==1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Overhead\n",
    "There's a performance benefit to parallelization, but the overhead for starting threads may be an overkill. For multithreading to be worth, you need a reasonably large amount of \"real work\". Conversely, with small works, the parallel version might be slower than the serial version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "N = 2^30\n",
    "x = fill(1.0f0, N)  # a vector filled with 1.0 (Float32)\n",
    "y = fill(2.0f0, N);  # a vector filled with 2.0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  2.104 s (0 allocations: 0 bytes)\n"
     ]
    }
   ],
   "source": [
    "function sequential_add!(y, x)\n",
    "    for i in eachindex(y, x)\n",
    "        @inbounds y[i] += x[i]\n",
    "    end\n",
    "    return nothing\n",
    "end\n",
    "\n",
    "using BenchmarkTools\n",
    "@btime sequential_add!($y, $x)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  1.559 s (34 allocations: 2.73 KiB)\n"
     ]
    }
   ],
   "source": [
    "function parallel_add!(y, x)\n",
    "    Threads.@threads for i in eachindex(y, x)\n",
    "        @inbounds y[i] += x[i]\n",
    "    end\n",
    "    return nothing\n",
    "end\n",
    "\n",
    "@btime parallel_add!($y, $x)\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above, without overhead, we'd expect to have a 5 fold time decrease since we have 5 threads. However, we get much less than that, due to overhead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 👍 How I use multithreading in my simulations\n",
    "\n",
    "I typically have an expensive function that I want to call multiple times with different arguments.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "simul (generic function with 1 method)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "function simul(noise, batch_size)\n",
    "    # do something with noise and batch_size\n",
    "    sleep(1)\n",
    "    return randn()\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "`simul` does some simulation based on the `noise` and `batch_size` parameters, then returns the simulation result.\n",
    "\n",
    "I want to loops through all combinations of the arguments proposed. Let's do so by creating a dictionary `pars` for each combination of arguments, and adding it to an array `pars_arr`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "pars_arr = Dict[]\n",
    "\n",
    "noises = [0.1, 0.2, 0.3]\n",
    "batch_sizes = [1000, 2000, 3000]\n",
    "\n",
    "for noise in noises, batch_size in batch_sizes\n",
    "    pars = Dict()\n",
    "    pars[\"noise\"] = noise\n",
    "    pars[\"batch_size\"] = batch_size\n",
    "    push!(pars_arr, pars)\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We'll also create a `DataFrame` to store the results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><div style = \"float: left;\"><span>0×3 DataFrame</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">Result</th><th style = \"text-align: left;\">noise</th><th style = \"text-align: left;\">batch_size</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Any\" style = \"text-align: left;\">Any</th><th title = \"Any\" style = \"text-align: left;\">Any</th><th title = \"Any\" style = \"text-align: left;\">Any</th></tr></thead><tbody></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& Result & noise & batch\\_size\\\\\n",
       "\t\\hline\n",
       "\t& Any & Any & Any\\\\\n",
       "\t\\hline\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m0×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m Result \u001b[0m\u001b[1m noise \u001b[0m\u001b[1m batch_size \u001b[0m\n",
       "     │\u001b[90m Any    \u001b[0m\u001b[90m Any   \u001b[0m\u001b[90m Any        \u001b[0m\n",
       "─────┴───────────────────────────"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "using DataFrames\n",
    "df_results = DataFrame(\"Result\" => [],\n",
    "                    \"noise\" => [],\n",
    "                    \"batch_size\" => [])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here is how I would run the simulations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r\u001b[32mProgress:  22%|██▎       |  ETA: 0:00:06 ( 0.80  s/it)\u001b[39m\u001b[K"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r\u001b[32mProgress:  67%|██████▋   |  ETA: 0:00:01 ( 0.43  s/it)\u001b[39m\u001b[K"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r\u001b[32mProgress:  78%|███████▊  |  ETA: 0:00:01 ( 0.40  s/it)\u001b[39m\u001b[K"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r\u001b[32mProgress: 100%|██████████| Time: 0:00:02 ( 0.31  s/it)\u001b[39m\u001b[K\n"
     ]
    }
   ],
   "source": [
    "using ProgressMeter\n",
    "progr = Progress(length(pars_arr), showspeed = true, barlen = 10)\n",
    "\n",
    "loc = Threads.ReentrantLock()\n",
    "\n",
    "Threads.@threads for k in 1:length(pars_arr)\n",
    "    p = pars_arr[k]\n",
    "    noise = p[\"noise\"]\n",
    "    batch_size = p[\"batch_size\"]\n",
    "    try\n",
    "        out = simul(noise, batch_size)\n",
    "        lock(loc) do\n",
    "            push!(df_results, (out, noise, batch_size));\n",
    "        end\n",
    "    catch e\n",
    "        println(\"problem with p = $(pars_arr[k])\")\n",
    "        println(e)\n",
    "    end\n",
    "    next!(progr)\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "I like using `ProgressMeter`, to get a sense of where my computation is at."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Atomic operations\n",
    "Note that you can also perform something called atomic operations, see the [dedicated section](https://docs.julialang.org/en/v1/manual/multi-threading/#Atomic-Operations) in Julia documentation. Atomic operations are similar to what you could do with `lock`, although they may be faster but more limited in what you could do.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## Multi-processing\n",
    "\n",
    "### `Distributed`\n",
    "Julia has also a built-in library for distributed parallel computing, called `Distributed`. Although it is generally more difficult to deploy than mulitthreading, it may be useful in certain occasions.  Distributed computing is useful when you have a lot of work that cannot be split among multiple threads and needs to be distributed across multiple machines.\n",
    "\n",
    "Monte Carlo simulations is another good use-case with distributed computing may be useful.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "`julia -p 4` provides `4` worker processes on the local machine. Alternatively, within Julia you can add workers by \n",
    "```julia\n",
    "using Distributed\n",
    "addprocs(4)  # add 4 worker processes\n",
    "```\n",
    "\n",
    "The most straightforward way of performing distributed computing is using  `pmap`. A good tutorial on how to use `pmap` can be found [here](https://github.com/Arpeggeo/julia-distributed-computing).\n",
    "\n",
    "Note that [ClusterManagers.jl](https://github.com/JuliaParallel/ClusterManagers.jl) may be useful for distributed computing.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "#### `MPI.jl`\n",
    "There exists an MPI (Message Passing Interface) interface for the Julia language, provided by the `MPI.jl` package. MPI is a low-level communication protocol that enables message passing between processes running on different nodes in a distributed system. It may be a better choice due to its interoperability, customization options, performance, and scalability on large-scale systems. If you never heard of it, then forget about it!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "## GPU computing\n",
    "\n",
    "Multiple dispatch allows your code to be executed on GPUS! Here is how.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "### GPU programming with CUDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  5.458 ms (4 allocations: 7.63 MiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.6680128f8"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "function myfun(a::AbstractArray, b::AbstractArray)\n",
    "    return sum(a.^2 * b)\n",
    "end\n",
    "\n",
    "# generate CPU arrays\n",
    "a = rand(Float32, 1000, 1000)\n",
    "b = rand(Float32, 1000, 1000)\n",
    "\n",
    "using BenchmarkTools\n",
    "@btime myfun($a, $b) # 820.959 μs (3 allocations: 7.63 MiB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CuDevice(0)\n",
      "CuDevice(1)\n",
      "CuDevice(2)\n",
      "CuDevice(3)\n",
      "CuDevice(4)\n",
      "CuDevice(5)\n",
      "CuDevice(6)\n",
      "CuDevice(7)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CuDevice(1): NVIDIA TITAN RTX"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "using CUDA\n",
    "\n",
    "@assert CUDA.functional()\n",
    "\n",
    "for d in devices()\n",
    "    println(d)\n",
    "end\n",
    "CUDA.device!(1)\n",
    "CUDA.current_device()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000×1000 CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}:\n",
       " 0.834464    0.457023   0.717635     …  0.484234  0.397357   0.0324202\n",
       " 0.288913    0.796186   0.255861        0.265453  0.427988   0.0370142\n",
       " 0.122341    0.97253    0.976035        0.77248   0.706358   0.260713\n",
       " 0.351675    0.446846   0.671586        0.920287  0.500553   0.374924\n",
       " 0.59903     0.312281   0.092035        0.35717   0.295004   0.95406\n",
       " 0.782895    0.702707   0.226901     …  0.995633  0.707188   0.841076\n",
       " 0.831006    0.409586   0.991196        0.725811  0.400745   0.936713\n",
       " 0.680695    0.935281   0.69477         0.928192  0.585817   0.400571\n",
       " 0.658063    0.521759   0.968131        0.439888  0.717022   0.961326\n",
       " 0.352665    0.412369   0.176933        0.404161  0.574798   0.888975\n",
       " ⋮                                   ⋱                       \n",
       " 0.639501    0.0262803  0.377526        0.923417  0.340922   0.543892\n",
       " 0.539695    0.839126   0.24328         0.596974  0.550674   0.181971\n",
       " 0.800403    0.437032   0.109061        0.711812  0.253885   0.357474\n",
       " 0.589502    0.306888   0.832209        0.162344  0.503309   0.357285\n",
       " 0.387903    0.244288   0.0200396    …  0.24908   0.520489   0.72995\n",
       " 0.857638    0.0745547  0.194844        0.978808  0.0127436  0.0997393\n",
       " 0.912032    0.380721   0.000763817     0.986396  0.979483   0.216162\n",
       " 0.00997814  0.787027   0.391444        0.117686  0.491821   0.0242603\n",
       " 0.272318    0.911535   0.829672        0.129895  0.729489   0.61118"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "a_cuda = CUDA.rand(1000, 1000)\n",
    "b_cuda = CUDA.rand(1000, 1000)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  214.489 μs (153 allocations: 7.11 KiB)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1.6638253f8"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@btime myfun($a_cuda, $b_cuda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### GPU programming on MacOS\n",
    "```julia\n",
    "using Metal\n",
    "a_mtl = MtlArray(a)\n",
    "b_mtl = MtlArray(b)\n",
    "\n",
    "@btime myfun($a_mtl, $b_mtl)\n",
    "```\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Additional resources and acknowledgements\n",
    "- [Discourse category :Julia at scale](https://discourse.julialang.org/c/domain/parallel/34)\n",
    "- [Further explanations on Multithreading vs Multiprocessing computing](https://towardsdatascience.com/multithreading-and-multiprocessing-in-10-minutes-20d9b3c6a867)\n",
    "- [Julia multi threading](https://docs.julialang.org/en/v1/manual/multi-threading/)\n",
    "- [CUDA.jl documentation](https://github.com/JuliaGPU/CUDA.jl)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Julia 1.8.5",
   "language": "julia",
   "name": "julia-1.8"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}