13_hardware_acceleration_architecture_overview.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNWBRQkcr+1xpaWxfYNIxpF",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dlsyscourse/lecture13/blob/main/13_hardware_acceleration_architecture_overview.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mpn1ti5Urdsv"
      },
      "source": [
        "# Lecture 13: Hardware Acceleration Implementation\n",
        "\n",
        "In this lecture, we will to walk through backend scafoldings to get us hardware accelerations for needle.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MkXPIjVd90z7"
      },
      "source": [
        "## Select a GPU runtime type\n",
        "In this lecture, we are going to make use of c++ and CUDA to build accelerated linear algebra libraries. In order to do so, please make sure you select a runtime type with GPU and rerun the cells if needed:\n",
        "- Click on the \"Runtime\" tab\n",
        "- Click \"Change runtime type\"\n",
        "- Select GPU\n",
        "\n",
        "After you started the right runtime, you can run the following command to check if there is a GPU available."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5VM6IcuZ-kv6",
        "outputId": "58553884-f279-40b0-8889-b9b42ad2c7fd"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tue Oct  8 01:26:45 2024       \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |\n",
            "|-----------------------------------------+----------------------+----------------------+\n",
            "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                                         |                      |               MIG M. |\n",
            "|=========================================+======================+======================|\n",
            "|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   53C    P8              10W /  70W |      0MiB / 15360MiB |      0%      Default |\n",
            "|                                         |                      |                  N/A |\n",
            "+-----------------------------------------+----------------------+----------------------+\n",
            "                                                                                         \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| Processes:                                                                            |\n",
            "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
            "|        ID   ID                                                             Usage      |\n",
            "|=======================================================================================|\n",
            "|  No running processes found                                                           |\n",
            "+---------------------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qXysoqn-vZuF"
      },
      "source": [
        "## Prepare the codebase\n",
        "\n",
        "To get started, we can clone the related repo from the github."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JjEIRTyr8ajf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c13f17d1-a504-4560-e161-90566afc3963"
      },
      "source": [
        "# Code to set up the assignment\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "%cd /content/drive/MyDrive/\n",
        "!mkdir -p 10714f24\n",
        "%cd /content/drive/MyDrive/10714f24\n",
        "# comment out the following line if you run it for the second time\n",
        "# as you already have a local copy of lecture13\n",
        "#!git clone https://github.com/dlsyscourse/lecture13\n",
        "!rm -rf /content/needle\n",
        "!ln -s /content/drive/MyDrive/10714f24/lecture13 /content/needle"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n",
            "/content/drive/MyDrive\n",
            "/content/drive/MyDrive/10714f24\n",
            "Cloning into 'lecture14'...\n",
            "remote: Enumerating objects: 99, done.\u001b[K\n",
            "remote: Counting objects: 100% (99/99), done.\u001b[K\n",
            "remote: Compressing objects: 100% (71/71), done.\u001b[K\n",
            "remote: Total 99 (delta 31), reused 85 (delta 23), pack-reused 0 (from 0)\u001b[K\n",
            "Receiving objects: 100% (99/99), 53.49 KiB | 1.14 MiB/s, done.\n",
            "Resolving deltas: 100% (31/31), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xe3vClsD9jlq",
        "outputId": "0ff2a35c-4d81-45d7-98bf-45c22fc05279"
      },
      "source": [
        "!python3 -m pip install pybind11"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting pybind11\n",
            "  Downloading pybind11-2.13.6-py3-none-any.whl.metadata (9.5 kB)\n",
            "Downloading pybind11-2.13.6-py3-none-any.whl (243 kB)\n",
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/243.3 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m112.6/243.3 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m243.3/243.3 kB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: pybind11\n",
            "Successfully installed pybind11-2.13.6\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O_RrW38i_JNp"
      },
      "source": [
        "### Build the needle cuda library\n",
        "\n",
        "We leverage pybind to build a c++/cuda library for acceleration. You can type make to build the corresponding library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o0EdAcB19saK",
        "outputId": "28c177ad-c7b2-46fb-eb9f-8f65878597db"
      },
      "source": [
        "%cd /content/needle\n",
        "!make clean\n",
        "!make"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/10714f24/lecture14\n",
            "rm -rf build python/needle/backend_ndarray/ndarray_backend*.so\n",
            "\u001b[0mCMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):\n",
            "  Compatibility with CMake < 3.5 will be removed from a future version of\n",
            "  CMake.\n",
            "\n",
            "  Update the VERSION argument <min> value or use a ...<max> suffix to tell\n",
            "  CMake that the project does not need compatibility with older versions.\n",
            "\n",
            "\u001b[0m\n",
            "-- The C compiler identification is GNU 11.4.0\n",
            "-- The CXX compiler identification is GNU 11.4.0\n",
            "-- Detecting C compiler ABI info\n",
            "-- Detecting C compiler ABI info - done\n",
            "-- Check for working C compiler: /usr/bin/cc - skipped\n",
            "-- Detecting C compile features\n",
            "-- Detecting C compile features - done\n",
            "-- Detecting CXX compiler ABI info\n",
            "-- Detecting CXX compiler ABI info - done\n",
            "-- Check for working CXX compiler: /usr/bin/c++ - skipped\n",
            "-- Detecting CXX compile features\n",
            "-- Detecting CXX compile features - done\n",
            "-- Found Python: /usr/local/bin/python (found version \"3.10.12\") found components: Development Interpreter Development.Module Development.Embed\n",
            "-- Performing Test HAS_FLTO\n",
            "-- Performing Test HAS_FLTO - Success\n",
            "-- Found pybind11: /usr/local/lib/python3.10/dist-packages/pybind11/include (found version \"2.13.6\")\n",
            "\u001b[33mCMake Warning (dev) at CMakeLists.txt:55 (find_package):\n",
            "  Policy CMP0146 is not set: The FindCUDA module is removed.  Run \"cmake\n",
            "  --help-policy CMP0146\" for policy details.  Use the cmake_policy command to\n",
            "  set the policy and suppress this warning.\n",
            "\n",
            "This warning is for project developers.  Use -Wno-dev to suppress it.\n",
            "\u001b[0m\n",
            "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD\n",
            "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success\n",
            "-- Found Threads: TRUE\n",
            "-- Found CUDA: /usr/local/cuda (found version \"12.2\")\n",
            "-- Found cuda, building cuda backend\n",
            "Tue Oct  8 01:30:32 2024       \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |\n",
            "|-----------------------------------------+----------------------+----------------------+\n",
            "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                                         |                      |               MIG M. |\n",
            "|=========================================+======================+======================|\n",
            "|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   45C    P8              10W /  70W |      0MiB / 15360MiB |      0%      Default |\n",
            "|                                         |                      |                  N/A |\n",
            "+-----------------------------------------+----------------------+----------------------+\n",
            "                                                                                         \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| Processes:                                                                            |\n",
            "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
            "|        ID   ID                                                             Usage      |\n",
            "|=======================================================================================|\n",
            "|  No running processes found                                                           |\n",
            "+---------------------------------------------------------------------------------------+\n",
            "-- Autodetected CUDA architecture(s):  7.5\n",
            "-- Configuring done (6.0s)\n",
            "-- Generating done (0.3s)\n",
            "-- Build files have been written to: /content/drive/MyDrive/10714f24/lecture14/build\n",
            "make[1]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[2]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[-25%] \u001b[32mBuilding CXX object CMakeFiles/ndarray_backend_cpu.dir/src/ndarray_backend_cpu.cc.o\u001b[0m\n",
            "[  0%] \u001b[32m\u001b[1mLinking CXX shared module /content/drive/MyDrive/10714f24/lecture14/python/needle/backend_ndarray/ndarray_backend_cpu.cpython-310-x86_64-linux-gnu.so\u001b[0m\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[  0%] Built target ndarray_backend_cpu\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[ 25%] \u001b[34m\u001b[1mBuilding NVCC (Device) object CMakeFiles/ndarray_backend_cuda.dir/src/ndarray_backend_cuda_generated_ndarray_backend_cuda.cu.o\u001b[0m\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[ 50%] \u001b[32m\u001b[1mLinking CXX shared module /content/drive/MyDrive/10714f24/lecture14/python/needle/backend_ndarray/ndarray_backend_cuda.cpython-310-x86_64-linux-gnu.so\u001b[0m\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[ 50%] Built target ndarray_backend_cuda\n",
            "make[2]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[1]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DFxG3p3S1sBq"
      },
      "source": [
        "We can then run the following command to make the path to the package available in colab's environment as well as the PYTHONPATH."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bix8OXLuCOKt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a250842b-c671-4ba1-e539-efa93f7fc35e"
      },
      "source": [
        "%set_env PYTHONPATH /content/needle/python:/env/python\n",
        "import sys\n",
        "sys.path.append(\"/content/needle/python\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "env: PYTHONPATH=/content/needle/python:/env/python\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BBIuE2jc1DaU"
      },
      "source": [
        "## Codebase walkthrough\n",
        "\n",
        "\n",
        "Now click the files panel on the left side. You should be able to see these files\n",
        "\n",
        "Python:\n",
        "- needle/backend_ndarray/ndarray.py\n",
        "- needle/backend_ndarray/ndarray_backend_numpy.py\n",
        "\n",
        "C++/CUDA\n",
        "- src/ndarray_backend_cpu.cc\n",
        "- src/ndarray_backend_cuda.cu\n",
        "\n",
        "The main goal of this lecture is to create an accelerated ndarray library.\n",
        "As a result, we do not need to deal with needle.Tensor for now and will focus on backend_ndarray's implementation.\n",
        "\n",
        "After we build up this array library, we can then use it to power backend array computations in needle.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x1Z8wSsI6PrU"
      },
      "source": [
        "## Creating a CUDA NDArray\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N2bm_WB9uF4V",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "874e2b52-6487-434b-fc94-b36c3fc57733"
      },
      "source": [
        "from needle import backend_ndarray as nd"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using needle backend\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wZGnTUsKF1x1"
      },
      "source": [
        "We can create a CUDA tensor from the data by specifying a device keyword."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1h5iAYFfBRED"
      },
      "source": [
        "x = nd.NDArray([1, 2, 3], device=nd.cuda())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CulMPqJkhkpE"
      },
      "source": [
        "y = x + 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t4UuEs9KAkDR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0b671189-f629-4829-bcfa-56aaeefad557"
      },
      "source": [
        "x.numpy()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1., 2., 3.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WBMvL6QEBtG7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "818f1fdc-69b0-4a95-f6da-36f3ba67a1ab"
      },
      "source": [
        "x.device"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "cuda()"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qJSv7D8NGfAr"
      },
      "source": [
        "y = x + 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yZ7hmyBVGhGd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "60d5e07f-13f7-417e-9557-8abc87656d13"
      },
      "source": [
        "y.device"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "cuda()"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NQVtUgK-f7_y",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff69ffd8-720c-492d-a818-7d1f57dcd63f"
      },
      "source": [
        "y.numpy()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2., 3., 4.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SPjNJfJsf_T9"
      },
      "source": [
        "### Key Data Structures\n",
        "\n",
        "Key data structures in backend_ndarray\n",
        "\n",
        "- NDArray: the container to hold device specific ndarray\n",
        "- BackendDevice: backend device\n",
        "    - mod holds the module implementation that implements all functions\n",
        "    - checkout ndarray_backend_numpy.py for a python-side reference.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HxKF9dcFhTy3"
      },
      "source": [
        "## Trace GPU execution\n",
        "\n",
        "Now, let us take a look at what happens when we execute the following code\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PLLzZzuthhBH"
      },
      "source": [
        "x = nd.NDArray([1, 2, 3], device=nd.cuda())\n",
        "y = x + 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V9NV0JFESkIe",
        "outputId": "011bc9d0-bcd5-4830-8d71-a2c373eb2202"
      },
      "source": [
        "x.device.from_numpy"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<function needle.backend_ndarray.ndarray_backend_cuda.PyCapsule.from_numpy>"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H6vwR3yBRI9F"
      },
      "source": [
        "x = nd.NDArray([1, 2, 3])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "id": "0PxoH_UzRMd3",
        "outputId": "54fd43b1-e18f-4f6d-e92a-4b04744fde90"
      },
      "source": [
        "x.device.from_numpy"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<function needle.backend_ndarray.ndarray_backend_numpy.from_numpy(a, out)>"
            ],
            "text/html": [
              "<div style=\"max-width:800px; border: 1px solid var(--colab-border-color);\"><style>\n",
              "      pre.function-repr-contents {\n",
              "        overflow-x: auto;\n",
              "        padding: 8px 12px;\n",
              "        max-height: 500px;\n",
              "      }\n",
              "\n",
              "      pre.function-repr-contents.function-repr-contents-collapsed {\n",
              "        cursor: pointer;\n",
              "        max-height: 100px;\n",
              "      }\n",
              "    </style>\n",
              "    <pre style=\"white-space: initial; background:\n",
              "         var(--colab-secondary-surface-color); padding: 8px 12px;\n",
              "         border-bottom: 1px solid var(--colab-border-color);\"><b>needle.backend_ndarray.ndarray_backend_numpy.from_numpy</b><br/>def from_numpy(a, out)</pre><pre class=\"function-repr-contents function-repr-contents-collapsed\" style=\"\"><a class=\"filepath\" style=\"display:none\" href=\"#\">/content/needle/python/needle/backend_ndarray/ndarray_backend_numpy.py</a>&lt;no docstring&gt;</pre>\n",
              "      <script>\n",
              "      if (google.colab.kernel.accessAllowed && google.colab.files && google.colab.files.view) {\n",
              "        for (const element of document.querySelectorAll('.filepath')) {\n",
              "          element.style.display = 'block'\n",
              "          element.onclick = (event) => {\n",
              "            event.preventDefault();\n",
              "            event.stopPropagation();\n",
              "            google.colab.files.view(element.textContent, 24);\n",
              "          };\n",
              "        }\n",
              "      }\n",
              "      for (const element of document.querySelectorAll('.function-repr-contents')) {\n",
              "        element.onclick = (event) => {\n",
              "          event.preventDefault();\n",
              "          event.stopPropagation();\n",
              "          element.classList.toggle('function-repr-contents-collapsed');\n",
              "        };\n",
              "      }\n",
              "      </script>\n",
              "      </div>"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xU5PFJJ-iR7J"
      },
      "source": [
        "Have the following trace:\n",
        "\n",
        "backend_ndarray/ndarray.py\n",
        "- `NDArray.__add__`\n",
        "- `NDArray.ewise_or_scalar`\n",
        "- `ndarray_backend_cpu.cc:ScalarAdd`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TxAKyM6yjr_R",
        "outputId": "ba1bfe21-77be-4cd2-f625-f2a3afbcfb57"
      },
      "source": [
        "y.numpy()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2., 3., 4.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F4vqb_a4j2O8"
      },
      "source": [
        "Have the following trace:\n",
        "\n",
        "- `NDArray.numpy`\n",
        "- `ndarray_backend_cpu.cc:to_numpy`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tMiFJmJVlD6j"
      },
      "source": [
        "## Guidelines for Reading C++/CUDA related Files\n",
        "\n",
        "Read\n",
        "- src/ndarray_backend_cpu.cc\n",
        "- src/ndarray_backend_cuda.cu\n",
        "\n",
        "\n",
        "Optional\n",
        "- CMakeLists.txt: this is used to setup the build and likely you do not need to tweak it.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uEpPbwQKkSkZ"
      },
      "source": [
        "## NDArray Data Structure\n",
        "\n",
        "Open up `python/needle/backend_ndarray/ndarray.py`.\n",
        "\n",
        "An NDArray contains the following fields:\n",
        "- handle: The backend handle that build a flat array which stores the data.\n",
        "- shape: The shape of the NDArray\n",
        "- strides: The strides that shows how do we access multi-dimensional elements\n",
        "- offset: The offset of the first element.\n",
        "- device: The backend device that backs the computation\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "875DgxFFACqb"
      },
      "source": [
        "## Transformation as Strided Computation\n",
        "\n",
        "We can leverage the strides and offset to perform transform/slicing with zero copy.\n",
        "\n",
        "- Broadcast: insert strides that equals 0\n",
        "- Tranpose: swap the strides\n",
        "- Slice: change the offset and shape\n",
        "\n",
        "For most of the computations, however, we will call `array.compact()` first to get a contiguous and aligned memory before running the computation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I49fcoiyWYLt"
      },
      "source": [
        "import numpy as np\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qGbICVsb6y98"
      },
      "source": [
        "x = nd.NDArray([1, 2, 3, 4], device=nd.cpu_numpy())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iofcuXso64yk",
        "outputId": "66e792a6-918e-4a01-b237-c8ee97c694a8"
      },
      "source": [
        "x.numpy()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1., 2., 3., 4.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oceIop5P7RHW"
      },
      "source": [
        "We can use strides and shape manipulation to create different views of the same array."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C7zCed7e7B4u"
      },
      "source": [
        "y = nd.NDArray.make(shape=(2, 2), strides=(2, 1), device=x.device, handle=x._handle, offset=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oaEPCvR17OMf",
        "outputId": "eee40587-5da5-4e01-a539-69f7788c638b"
      },
      "source": [
        "y.numpy()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1., 2.],\n",
              "       [3., 4.]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5rNS5MW67XyX"
      },
      "source": [
        "z = nd.NDArray.make(shape=(2, 1), strides=(2, 1), device=x.device, handle=x._handle, offset=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HzhpVtKB7b97",
        "outputId": "fe5e6ac3-3458-4140-a5f6-fae9547f6e99"
      },
      "source": [
        "z.numpy()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[2.],\n",
              "       [4.]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ONkZbUuj6Dx"
      },
      "source": [
        "## CUDA Acceleration\n",
        "\n",
        "Now let us open `src/ndarray_cuda_backend.cu` and take a look at current implementation of GPU ops.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Og8N3iuZiZ4g"
      },
      "source": [
        "## Steps for adding a new operator implementation\n",
        "- Add an implementation in `ndarray_backend_cuda.cu`, expose via pybind\n",
        "- Call into the operator in ndarray.py\n",
        "- Write up testcases"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xV1I7I2lkOJG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5701699e-a11c-4ee3-e98b-a6211908f329"
      },
      "source": [
        "!make"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[0mCMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):\n",
            "  Compatibility with CMake < 3.5 will be removed from a future version of\n",
            "  CMake.\n",
            "\n",
            "  Update the VERSION argument <min> value or use a ...<max> suffix to tell\n",
            "  CMake that the project does not need compatibility with older versions.\n",
            "\n",
            "\u001b[0m\n",
            "-- Found pybind11: /usr/local/lib/python3.10/dist-packages/pybind11/include (found version \"2.13.6\")\n",
            "\u001b[33mCMake Warning (dev) at CMakeLists.txt:55 (find_package):\n",
            "  Policy CMP0146 is not set: The FindCUDA module is removed.  Run \"cmake\n",
            "  --help-policy CMP0146\" for policy details.  Use the cmake_policy command to\n",
            "  set the policy and suppress this warning.\n",
            "\n",
            "This warning is for project developers.  Use -Wno-dev to suppress it.\n",
            "\u001b[0m\n",
            "-- Found cuda, building cuda backend\n",
            "Tue Oct  8 01:33:00 2024       \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |\n",
            "|-----------------------------------------+----------------------+----------------------+\n",
            "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                                         |                      |               MIG M. |\n",
            "|=========================================+======================+======================|\n",
            "|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   55C    P0              29W /  70W |    105MiB / 15360MiB |      0%      Default |\n",
            "|                                         |                      |                  N/A |\n",
            "+-----------------------------------------+----------------------+----------------------+\n",
            "                                                                                         \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| Processes:                                                                            |\n",
            "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
            "|        ID   ID                                                             Usage      |\n",
            "|=======================================================================================|\n",
            "+---------------------------------------------------------------------------------------+\n",
            "-- Autodetected CUDA architecture(s):  7.5\n",
            "-- Configuring done (0.4s)\n",
            "-- Generating done (0.4s)\n",
            "-- Build files have been written to: /content/drive/MyDrive/10714f24/lecture14/build\n",
            "make[1]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[2]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[  0%] Built target ndarray_backend_cpu\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[ 25%] \u001b[32m\u001b[1mLinking CXX shared module /content/drive/MyDrive/10714f24/lecture14/python/needle/backend_ndarray/ndarray_backend_cuda.cpython-310-x86_64-linux-gnu.so\u001b[0m\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[ 50%] Built target ndarray_backend_cuda\n",
            "make[2]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[1]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If we directly run the code block, we will see an error, because ewise mul is not yet implemented"
      ],
      "metadata": {
        "id": "OpWxZKGb_4cJ"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YU870vVVZkzg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 311
        },
        "outputId": "99ae1dbc-7ebd-4229-e3de-50eabafc8d5f"
      },
      "source": [
        "x = nd.NDArray([1,2,3], device=nd.cuda())\n",
        "x * 2"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AttributeError",
          "evalue": "module 'needle.backend_ndarray.ndarray_backend_cuda' has no attribute 'ewise_mul'",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-40-dabe3239c8f9>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNDArray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mx\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/content/needle/python/needle/backend_ndarray/ndarray.py\u001b[0m in \u001b[0;36m__mul__\u001b[0;34m(self, other)\u001b[0m\n\u001b[1;32m    419\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__mul__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mother\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    420\u001b[0m         return self.ewise_or_scalar(\n\u001b[0;32m--> 421\u001b[0;31m             \u001b[0mother\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mewise_mul\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscalar_mul\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    422\u001b[0m         )\n\u001b[1;32m    423\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/needle/python/needle/backend_ndarray/ndarray.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__getattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0menabled\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mAttributeError\u001b[0m: module 'needle.backend_ndarray.ndarray_backend_cuda' has no attribute 'ewise_mul'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VEtbnbvr6Wt7"
      },
      "source": [
        "## Connect back to needle Tensor\n",
        "\n",
        "So far we only played with the `backend_ndarray` subpackage, which is a self-contained ndarray implementation within needle.\n",
        "\n",
        "We can connect the ndarray back to needle as a backend."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JeThSA8zAu_v"
      },
      "source": [
        "import needle as ndl"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dobDH96Ql8SV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cd23f433-8c4c-4eb3-d67f-8b492a703261"
      },
      "source": [
        "x = ndl.Tensor([1,2,3], device=nd.cuda(), dtype=\"float32\")\n",
        "y = ndl.Tensor([2,3,5], device=nd.cuda(), dtype=\"float32\")\n",
        "z = x + y\n",
        "z"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "needle.Tensor([3. 5. 8.])"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ouXpj1v6g3z1",
        "outputId": "ed910149-2d9e-4055-b31d-52fb698e463d"
      },
      "source": [
        "z.device"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "cuda()"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "id": "4827VUz3bwvA",
        "outputId": "94b59197-251e-4be5-8bb3-c0b704809477"
      },
      "source": [
        "type(z.cached_data)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "needle.backend_ndarray.ndarray.NDArray"
            ],
            "text/html": [
              "<div style=\"max-width:800px; border: 1px solid var(--colab-border-color);\"><style>\n",
              "      pre.function-repr-contents {\n",
              "        overflow-x: auto;\n",
              "        padding: 8px 12px;\n",
              "        max-height: 500px;\n",
              "      }\n",
              "\n",
              "      pre.function-repr-contents.function-repr-contents-collapsed {\n",
              "        cursor: pointer;\n",
              "        max-height: 100px;\n",
              "      }\n",
              "    </style>\n",
              "    <pre style=\"white-space: initial; background:\n",
              "         var(--colab-secondary-surface-color); padding: 8px 12px;\n",
              "         border-bottom: 1px solid var(--colab-border-color);\"><b>needle.backend_ndarray.ndarray.NDArray</b><br/>def __init__(other, device=None)</pre><pre class=\"function-repr-contents function-repr-contents-collapsed\" style=\"\"><a class=\"filepath\" style=\"display:none\" href=\"#\">/content/needle/python/needle/backend_ndarray/ndarray.py</a>A generic ND array class that may contain multipe different backends\n",
              "i.e., a Numpy backend, a native CPU backend, or a GPU backend.\n",
              "\n",
              "This class will only contains those functions that you need to implement\n",
              "to actually get the desired functionality for the programming examples\n",
              "in the homework, and no more.\n",
              "\n",
              "For now, for simplicity the class only supports float32 types, though\n",
              "this can be extended if desired.</pre>\n",
              "      <script>\n",
              "      if (google.colab.kernel.accessAllowed && google.colab.files && google.colab.files.view) {\n",
              "        for (const element of document.querySelectorAll('.filepath')) {\n",
              "          element.style.display = 'block'\n",
              "          element.onclick = (event) => {\n",
              "            event.preventDefault();\n",
              "            event.stopPropagation();\n",
              "            google.colab.files.view(element.textContent, 88);\n",
              "          };\n",
              "        }\n",
              "      }\n",
              "      for (const element of document.querySelectorAll('.function-repr-contents')) {\n",
              "        element.onclick = (event) => {\n",
              "          event.preventDefault();\n",
              "          event.stopPropagation();\n",
              "          element.classList.toggle('function-repr-contents-collapsed');\n",
              "        };\n",
              "      }\n",
              "      </script>\n",
              "      </div>"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "74OUUH2REG18"
      },
      "source": [
        "## Write Standalone Python Test Files\n",
        "\n",
        "Now that we have additional c++/cuda libraries in needle, we will need to type make in order to rebuild the library. Additionally, because the colab environment caches the old library, it is inconvenient to use the ipython cells to debug the updated library.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sgLoV-_KHAM3",
        "outputId": "ae19d49d-faa3-4aa7-cd6f-ed61815cfce6"
      },
      "source": [
        "!make"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[0mCMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):\n",
            "  Compatibility with CMake < 3.5 will be removed from a future version of\n",
            "  CMake.\n",
            "\n",
            "  Update the VERSION argument <min> value or use a ...<max> suffix to tell\n",
            "  CMake that the project does not need compatibility with older versions.\n",
            "\n",
            "\u001b[0m\n",
            "-- Found pybind11: /usr/local/lib/python3.10/dist-packages/pybind11/include (found version \"2.13.6\")\n",
            "\u001b[33mCMake Warning (dev) at CMakeLists.txt:55 (find_package):\n",
            "  Policy CMP0146 is not set: The FindCUDA module is removed.  Run \"cmake\n",
            "  --help-policy CMP0146\" for policy details.  Use the cmake_policy command to\n",
            "  set the policy and suppress this warning.\n",
            "\n",
            "This warning is for project developers.  Use -Wno-dev to suppress it.\n",
            "\u001b[0m\n",
            "-- Found cuda, building cuda backend\n",
            "Tue Oct  8 01:33:33 2024       \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |\n",
            "|-----------------------------------------+----------------------+----------------------+\n",
            "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                                         |                      |               MIG M. |\n",
            "|=========================================+======================+======================|\n",
            "|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   58C    P0              30W /  70W |    105MiB / 15360MiB |      0%      Default |\n",
            "|                                         |                      |                  N/A |\n",
            "+-----------------------------------------+----------------------+----------------------+\n",
            "                                                                                         \n",
            "+---------------------------------------------------------------------------------------+\n",
            "| Processes:                                                                            |\n",
            "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
            "|        ID   ID                                                             Usage      |\n",
            "|=======================================================================================|\n",
            "+---------------------------------------------------------------------------------------+\n",
            "-- Autodetected CUDA architecture(s):  7.5\n",
            "-- Configuring done (0.3s)\n",
            "-- Generating done (0.3s)\n",
            "-- Build files have been written to: /content/drive/MyDrive/10714f24/lecture14/build\n",
            "make[1]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[2]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[  0%] Built target ndarray_backend_cpu\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "[ 50%] Built target ndarray_backend_cuda\n",
            "make[2]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n",
            "make[1]: Leaving directory '/content/drive/MyDrive/10714f24/lecture14/build'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dudnLHRqoKY2"
      },
      "source": [
        "\n",
        "We recommend writing separate python files and invoke them from the command line. Create a new file `tests/mytest.py` and write your local tests. This is also a common develop practice in big projects that involves python c++ FFI."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TubIHJrkn4Sk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cf8889e2-605f-4310-d89a-e4a49bc254e4"
      },
      "source": [
        "!python tests/mytest.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "python3: can't open file '/content/drive/MyDrive/10714f24/lecture14/tests/mytest.py': [Errno 2] No such file or directory\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ei0UR-FYoY1-"
      },
      "source": [
        "After we have building the library, we could choose to fully restart the runtime (factory reset runtime) if you want to bring the updated change back to another colab. Note that you will need to save your code changes to the drive or a private github repo."
      ]
    }
  ]
}