14_hardware_acceleration_architecture_overview.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyO4pFqm3zAA4uAu9HavofY3",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dlsyscourse/lecture14/blob/main/14_hardware_acceleration_architecture_overview.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mpn1ti5Urdsv"
      },
      "source": [
        "# Lecture 14: Hardware Acceleration Implementation\n",
        "\n",
        "In this lecture, we will to walk through backend scafoldings to get us hardware accelerations for needle.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MkXPIjVd90z7"
      },
      "source": [
        "## Select a GPU runtime type\n",
        "In this lecture, we are going to make use of c++ and CUDA to build accelerated linear algebra libraries. In order to do so, please make sure you select a runtime type with GPU and rerun the cells if needed:\n",
        "- Click on the \"Runtime\" tab\n",
        "- Click \"Change runtime type\"\n",
        "- Select GPU\n",
        "\n",
        "After you started the right runtime, you can run the following command to check if there is a GPU available."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5VM6IcuZ-kv6",
        "outputId": "f3170387-6181-4c98-8fe9-37fc50a51b62"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mon Oct  9 14:55:46 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   76C    P0    33W /  70W |    105MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qXysoqn-vZuF"
      },
      "source": [
        "## Prepare the codebase\n",
        "\n",
        "To get started, we can clone the related repo from the github."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JjEIRTyr8ajf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "24b4b819-8dd8-45c2-f1e8-082c0b111a02"
      },
      "source": [
        "# Code to set up the assignment\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "%cd /content/drive/MyDrive/\n",
        "!mkdir -p 10714f23\n",
        "%cd /content/drive/MyDrive/10714f23\n",
        "# comment out the following line if you run it for the second time\n",
        "# as you already have a local copy of lecture14\n",
        "# !git clone https://github.com/dlsyscourse/lecture14\n",
        "!rm -rf /content/needle\n",
        "!ln -s /content/drive/MyDrive/10714f23/lecture14 /content/needle"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n",
            "/content/drive/MyDrive\n",
            "/content/drive/MyDrive/10714f23\n",
            "Cloning into 'lecture14'...\n",
            "remote: Enumerating objects: 96, done.\u001b[K\n",
            "remote: Counting objects: 100% (96/96), done.\u001b[K\n",
            "remote: Compressing objects: 100% (67/67), done.\u001b[K\n",
            "remote: Total 96 (delta 29), reused 87 (delta 24), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (96/96), 49.00 KiB | 1.69 MiB/s, done.\n",
            "Resolving deltas: 100% (29/29), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xe3vClsD9jlq",
        "outputId": "fb1367b1-13b1-42ff-ad08-5d677283cb4c"
      },
      "source": [
        "!python3 -m pip install pybind11"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: pybind11 in /usr/local/lib/python3.10/dist-packages (2.11.1)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O_RrW38i_JNp"
      },
      "source": [
        "### Build the needle cuda library\n",
        "\n",
        "We leverage pybind to build a c++/cuda library for acceleration. You can type make to build the corresponding library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o0EdAcB19saK",
        "outputId": "03b49ca1-de48-488a-8d96-b44b0a70ac93"
      },
      "source": [
        "%cd /content/needle\n",
        "!make clean\n",
        "!make"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/10714f23/lecture14\n",
            "rm -rf build python/needle/backend_ndarray/ndarray_backend*.so\n",
            "-- The C compiler identification is GNU 11.4.0\n",
            "-- The CXX compiler identification is GNU 11.4.0\n",
            "-- Detecting C compiler ABI info\n",
            "-- Detecting C compiler ABI info - done\n",
            "-- Check for working C compiler: /usr/bin/cc - skipped\n",
            "-- Detecting C compile features\n",
            "-- Detecting C compile features - done\n",
            "-- Detecting CXX compiler ABI info\n",
            "-- Detecting CXX compiler ABI info - done\n",
            "-- Check for working CXX compiler: /usr/bin/c++ - skipped\n",
            "-- Detecting CXX compile features\n",
            "-- Detecting CXX compile features - done\n",
            "-- Found Python: /usr/local/bin/python (found version \"3.10.12\") found components: Development Interpreter Development.Module Development.Embed \n",
            "-- Performing Test HAS_FLTO\n",
            "-- Performing Test HAS_FLTO - Success\n",
            "-- Found pybind11: /usr/local/lib/python3.10/dist-packages/pybind11/include (found version \"2.11.1\")\n",
            "\u001b[33mCMake Warning (dev) at CMakeLists.txt:55 (find_package):\n",
            "  Policy CMP0146 is not set: The FindCUDA module is removed.  Run \"cmake\n",
            "  --help-policy CMP0146\" for policy details.  Use the cmake_policy command to\n",
            "  set the policy and suppress this warning.\n",
            "\n",
            "This warning is for project developers.  Use -Wno-dev to suppress it.\n",
            "\u001b[0m\n",
            "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD\n",
            "-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success\n",
            "-- Found Threads: TRUE  \n",
            "-- Found CUDA: /usr/local/cuda (found version \"11.8\") \n",
            "-- Found cuda, building cuda backend\n",
            "Mon Oct  9 15:08:22 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   47C    P8    10W /  70W |      0MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n",
            "-- Autodetected CUDA architecture(s):  7.5\n",
            "-- Configuring done (3.5s)\n",
            "-- Generating done (0.2s)\n",
            "-- Build files have been written to: /content/drive/MyDrive/10714f23/lecture14/build\n",
            "make[1]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[2]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[-25%] \u001b[32mBuilding CXX object CMakeFiles/ndarray_backend_cpu.dir/src/ndarray_backend_cpu.cc.o\u001b[0m\n",
            "[  0%] \u001b[32m\u001b[1mLinking CXX shared module /content/drive/MyDrive/10714f23/lecture14/python/needle/backend_ndarray/ndarray_backend_cpu.cpython-310-x86_64-linux-gnu.so\u001b[0m\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[  0%] Built target ndarray_backend_cpu\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[ 25%] \u001b[34m\u001b[1mBuilding NVCC (Device) object CMakeFiles/ndarray_backend_cuda.dir/src/ndarray_backend_cuda_generated_ndarray_backend_cuda.cu.o\u001b[0m\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[ 50%] \u001b[32m\u001b[1mLinking CXX shared module /content/drive/MyDrive/10714f23/lecture14/python/needle/backend_ndarray/ndarray_backend_cuda.cpython-310-x86_64-linux-gnu.so\u001b[0m\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[ 50%] Built target ndarray_backend_cuda\n",
            "make[2]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[1]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DFxG3p3S1sBq"
      },
      "source": [
        "We can then run the following command to make the path to the package available in colab's environment as well as the PYTHONPATH."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bix8OXLuCOKt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "583e1b27-02e7-42de-fa80-32f3c25d9323"
      },
      "source": [
        "%set_env PYTHONPATH /content/needle/python:/env/python\n",
        "import sys\n",
        "sys.path.append(\"/content/needle/python\")"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "env: PYTHONPATH=/content/needle/python:/env/python\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BBIuE2jc1DaU"
      },
      "source": [
        "## Codebase walkthrough\n",
        "\n",
        "\n",
        "Now click the files panel on the left side. You should be able to see these files\n",
        "\n",
        "Python:\n",
        "- needle/backend_ndarray/ndarray.py\n",
        "- needle/backend_ndarray/ndarray_backend_numpy.py\n",
        "\n",
        "C++/CUDA\n",
        "- src/ndarray_backend_cpu.cc\n",
        "- src/ndarray_backend_cuda.cu\n",
        "\n",
        "The main goal of this lecture is to create an accelerated ndarray library.\n",
        "As a result, we do not need to deal with needle.Tensor for now and will focus on backend_ndarray's implementation.\n",
        "\n",
        "After we build up this array library, we can then use it to power backend array computations in needle.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x1Z8wSsI6PrU"
      },
      "source": [
        "## Creating a CUDA NDArray\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N2bm_WB9uF4V",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f46bf72c-c8b1-443e-ba84-5bcd4b8dc8ee"
      },
      "source": [
        "from needle import backend_ndarray as nd"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using needle backend\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2On4NzNggDfN"
      },
      "source": [
        "x = nd.NDArray([1, 2, 3], device=nd.cpu())"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "x = nd.NDArray([1, 2, 3], device=nd.cuda())"
      ],
      "metadata": {
        "id": "0W4C5Zbmao7_"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "x.device"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "elL0Bq-mapF3",
        "outputId": "4be304fe-5a04-447c-c330-47cd52804a94"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "cuda()"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lwXRXHbnN2-s"
      },
      "source": [
        "x = nd.NDArray([1,2,3], device=nd.cuda())"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hSjmK60DOFEi"
      },
      "source": [
        "y = x + 1"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_s59AZm7OFVV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4f365885-3fd1-4bb1-b6c1-96e45cb0b5a2"
      },
      "source": [
        "y"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NDArray([2. 3. 4.], device=cuda())"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZBlbinVMYNhA"
      },
      "source": [
        "y = x + x"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-4RytYKfY6JQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4f49b23a-5b9f-4a78-9b10-678989562574"
      },
      "source": [
        "y"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NDArray([2. 4. 6.], device=cuda())"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wZGnTUsKF1x1"
      },
      "source": [
        "We can create a CUDA tensor from the data by specifying a device keyword."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1h5iAYFfBRED"
      },
      "source": [
        "x = nd.NDArray([1, 2, 3], device=nd.cuda())"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CulMPqJkhkpE"
      },
      "source": [
        "y = x + 1"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t4UuEs9KAkDR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bac99671-d9a1-4983-8dc9-331768e4ab6e"
      },
      "source": [
        "x.numpy()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1., 2., 3.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WBMvL6QEBtG7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1f852562-8b1b-4ec7-bf35-0a8a3f96ce96"
      },
      "source": [
        "x.device"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "cuda()"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qJSv7D8NGfAr"
      },
      "source": [
        "y = x + 1"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yZ7hmyBVGhGd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d35e51be-5dca-4221-b1e9-ed434f3660ae"
      },
      "source": [
        "y.device"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "cuda()"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NQVtUgK-f7_y",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "34201640-3812-419b-f822-729fe08f3f87"
      },
      "source": [
        "y.numpy()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2., 3., 4.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SPjNJfJsf_T9"
      },
      "source": [
        "### Key Data Structures\n",
        "\n",
        "Key data structures in backend_ndarray\n",
        "\n",
        "- NDArray: the container to hold device specific ndarray\n",
        "- BackendDevice: backend device\n",
        "    - mod holds the module implementation that implements all functions\n",
        "    - checkout ndarray_backend_numpy.py for a python-side reference.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HxKF9dcFhTy3"
      },
      "source": [
        "## Trace GPU execution\n",
        "\n",
        "Now, let us take a look at what happens when we execute the following code\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PLLzZzuthhBH"
      },
      "source": [
        "x = nd.NDArray([1, 2, 3], device=nd.cuda())\n",
        "y = x + 1"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V9NV0JFESkIe",
        "outputId": "7a2d70c9-a9a8-4bed-d586-84f14b687dc6"
      },
      "source": [
        "x.device.from_numpy"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<function needle.backend_ndarray.ndarray_backend_numpy.from_numpy(a, out)>"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H6vwR3yBRI9F"
      },
      "source": [
        "x = nd.NDArray([1, 2, 3])"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0PxoH_UzRMd3",
        "outputId": "bf714546-5f6d-4774-9067-6652e356f156"
      },
      "source": [
        "x.device.from_numpy"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<function needle.backend_ndarray.ndarray_backend_numpy.from_numpy(a, out)>"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xU5PFJJ-iR7J"
      },
      "source": [
        "Have the following trace:\n",
        "\n",
        "backend_ndarray/ndarray.py\n",
        "- `NDArray.__add__`\n",
        "- `NDArray.ewise_or_scalar`\n",
        "- `ndarray_backend_cpu.cc:ScalarAdd`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TxAKyM6yjr_R",
        "outputId": "3a249eac-8bb5-4e3d-964c-cdf39bee0dbc"
      },
      "source": [
        "y.numpy()"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2., 3., 4.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F4vqb_a4j2O8"
      },
      "source": [
        "Have the following trace:\n",
        "\n",
        "- `NDArray.numpy`\n",
        "- `ndarray_backend_cpu.cc:to_numpy`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tMiFJmJVlD6j"
      },
      "source": [
        "## Guidelines for Reading C++/CUDA related Files\n",
        "\n",
        "Read\n",
        "- src/ndarray_backend_cpu.cc\n",
        "- src/ndarray_backend_cuda.cu\n",
        "\n",
        "\n",
        "Optional\n",
        "- CMakeLists.txt: this is used to setup the build and likely you do not need to tweak it.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uEpPbwQKkSkZ"
      },
      "source": [
        "## NDArray Data Structure\n",
        "\n",
        "Open up `python/needle/backend_ndarray/ndarray.py`.\n",
        "\n",
        "An NDArray contains the following fields:\n",
        "- handle: The backend handle that build a flat array which stores the data.\n",
        "- shape: The shape of the NDArray\n",
        "- strides: The strides that shows how do we access multi-dimensional elements\n",
        "- offset: The offset of the first element.\n",
        "- device: The backend device that backs the computation\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "875DgxFFACqb"
      },
      "source": [
        "## Transformation as Strided Computation\n",
        "\n",
        "We can leverage the strides and offset to perform transform/slicing with zero copy.\n",
        "\n",
        "- Broadcast: insert strides that equals 0\n",
        "- Tranpose: swap the strides\n",
        "- Slice: change the offset and shape\n",
        "\n",
        "For most of the computations, however, we will call `array.compact()` first to get a contiguous and aligned memory before running the computation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I49fcoiyWYLt"
      },
      "source": [
        "import numpy as np\n",
        "x = nd.NDArray([0,1,2,3,4,5], device=nd.cpu_numpy())"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8H3F_ST1Wcvx",
        "outputId": "f04724fb-56c6-4b16-9f72-76d10014104c"
      },
      "source": [
        "x.numpy()"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0., 1., 2., 3., 4., 5.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "z = nd.NDArray.make(shape=(2, 3),\n",
        "                strides=(3, 1),\n",
        "                device=x.device,\n",
        "                handle=x._handle,\n",
        "                offset=0)\n",
        "z"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CXZavfGnghB1",
        "outputId": "3f53317b-96b0-459e-fece-3becee718bfb"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NDArray([[0. 1. 2.]\n",
              " [3. 4. 5.]], device=cpu_numpy())"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "OyENEwN2i4dg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "b = nd.NDArray.make(shape=(2, 3, 4),\n",
        "                    strides=(3, 1, 0),\n",
        "                    device=z.device,\n",
        "                    handle=z._handle,\n",
        "                    offset=0)\n",
        "b"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xxoqPaTUghMe",
        "outputId": "813af112-5b8f-4e4a-a3b9-908db8f060ba"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NDArray([[[0. 0. 0. 0.]\n",
              "  [1. 1. 1. 1.]\n",
              "  [2. 2. 2. 2.]]\n",
              "\n",
              " [[3. 3. 3. 3.]\n",
              "  [4. 4. 4. 4.]\n",
              "  [5. 5. 5. 5.]]], device=cpu_numpy())"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "LMruIyFbghS_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "VgGDKXXdghdK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N0D4mCqjWh6c",
        "outputId": "acd653e2-9e8b-4df5-c9e2-1cf4563dc01d"
      },
      "source": [
        "y = nd.NDArray.make(shape=(3, 2, 2), strides=(2, 1, 0), device=x.device, handle=x._handle, offset=0)\n",
        "y.numpy()"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[[0., 0.],\n",
              "        [1., 1.]],\n",
              "\n",
              "       [[2., 2.],\n",
              "        [3., 3.]],\n",
              "\n",
              "       [[4., 4.],\n",
              "        [5., 5.]]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qGbICVsb6y98"
      },
      "source": [
        "x = nd.NDArray([1, 2, 3, 4], device=nd.cpu_numpy())"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iofcuXso64yk",
        "outputId": "dba7f35e-a89f-43d4-d3fc-dc7eff3c13ed"
      },
      "source": [
        "x.numpy()"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1., 2., 3., 4.], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oceIop5P7RHW"
      },
      "source": [
        "We can use strides and shape manipulation to create different views of the same array."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C7zCed7e7B4u"
      },
      "source": [
        "y = nd.NDArray.make(shape=(2, 2), strides=(2, 1), device=x.device, handle=x._handle, offset=0)"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oaEPCvR17OMf",
        "outputId": "8b4fe214-96c8-4f3e-c2f0-a545942e1b18"
      },
      "source": [
        "y.numpy()"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1., 2.],\n",
              "       [3., 4.]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5rNS5MW67XyX"
      },
      "source": [
        "z = nd.NDArray.make(shape=(2, 1), strides=(2, 1), device=x.device, handle=x._handle, offset=1)"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HzhpVtKB7b97",
        "outputId": "28553353-73c7-4849-9f06-6ab1ae255c3d"
      },
      "source": [
        "z.numpy()"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[2.],\n",
              "       [4.]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ONkZbUuj6Dx"
      },
      "source": [
        "## CUDA Acceleration\n",
        "\n",
        "Now let us open `src/ndarray_cuda_backend.cu` and take a look at current implementation of GPU ops.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Og8N3iuZiZ4g"
      },
      "source": [
        "## Steps for adding a new operator implementation\n",
        "- Add an implementation in `ndarray_backend_cuda.cu`, expose via pybind\n",
        "- Call into the operator in ndarray.py\n",
        "- Write up testcases"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xV1I7I2lkOJG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f5f6ad2b-8570-490d-cde5-bbc34be411b7"
      },
      "source": [
        "!make"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-- Found pybind11: /usr/local/lib/python3.10/dist-packages/pybind11/include (found version \"2.11.1\")\n",
            "\u001b[33mCMake Warning (dev) at CMakeLists.txt:55 (find_package):\n",
            "  Policy CMP0146 is not set: The FindCUDA module is removed.  Run \"cmake\n",
            "  --help-policy CMP0146\" for policy details.  Use the cmake_policy command to\n",
            "  set the policy and suppress this warning.\n",
            "\n",
            "This warning is for project developers.  Use -Wno-dev to suppress it.\n",
            "\u001b[0m\n",
            "-- Found cuda, building cuda backend\n",
            "Mon Oct  9 15:11:44 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   50C    P0    28W /  70W |    105MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n",
            "-- Autodetected CUDA architecture(s):  7.5\n",
            "-- Configuring done (0.4s)\n",
            "-- Generating done (0.2s)\n",
            "-- Build files have been written to: /content/drive/MyDrive/10714f23/lecture14/build\n",
            "make[1]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[2]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[  0%] Built target ndarray_backend_cpu\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[ 25%] \u001b[32m\u001b[1mLinking CXX shared module /content/drive/MyDrive/10714f23/lecture14/python/needle/backend_ndarray/ndarray_backend_cuda.cpython-310-x86_64-linux-gnu.so\u001b[0m\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[ 50%] Built target ndarray_backend_cuda\n",
            "make[2]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[1]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If we directly run the code block, we will see an error, because ewise mul is not yet implemented"
      ],
      "metadata": {
        "id": "OpWxZKGb_4cJ"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YU870vVVZkzg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 346
        },
        "outputId": "2a69f4d5-a495-494b-f0a9-ef9e3ef226f2"
      },
      "source": [
        "x = nd.NDArray([1,2,3], device=nd.cuda())\n",
        "x * 2"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AttributeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-39-dabe3239c8f9>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNDArray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mx\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/content/needle/python/needle/backend_ndarray/ndarray.py\u001b[0m in \u001b[0;36m__mul__\u001b[0;34m(self, other)\u001b[0m\n\u001b[1;32m    419\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__mul__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mother\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    420\u001b[0m         return self.ewise_or_scalar(\n\u001b[0;32m--> 421\u001b[0;31m             \u001b[0mother\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mewise_mul\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscalar_mul\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    422\u001b[0m         )\n\u001b[1;32m    423\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/content/needle/python/needle/backend_ndarray/ndarray.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__getattr__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0menabled\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mAttributeError\u001b[0m: module 'needle.backend_ndarray.ndarray_backend_cuda' has no attribute 'ewise_mul'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iiQFCpb-ZqX8",
        "outputId": "3ac0f010-ced5-45e7-ef08-019ac5371aa7"
      },
      "source": [
        "!nvprof python test_mul.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==1619== NVPROF is profiling process 1619, command: python3 test_mul.py\n",
            "[3. 6. 9.]\n",
            "==1619== Profiling application: python3 test_mul.py\n",
            "==1619== Profiling result:\n",
            "            Type  Time(%)      Time     Calls       Avg       Min       Max  Name\n",
            " GPU activities:   53.55%  4.8320us         1  4.8320us  4.8320us  4.8320us  needle::cuda::ScalarMulKernel(float const *, float, float*, unsigned long)\n",
            "                   23.40%  2.1120us         1  2.1120us  2.1120us  2.1120us  [CUDA memcpy DtoH]\n",
            "                   23.05%  2.0800us         1  2.0800us  2.0800us  2.0800us  [CUDA memcpy HtoD]\n",
            "      API calls:   99.74%  333.65ms         2  166.83ms  9.9830us  333.64ms  cudaMalloc\n",
            "                    0.13%  421.20us         1  421.20us  421.20us  421.20us  cuDeviceTotalMem\n",
            "                    0.05%  165.97us         2  82.987us  19.197us  146.78us  cudaFree\n",
            "                    0.05%  158.40us       101  1.5680us     140ns  67.406us  cuDeviceGetAttribute\n",
            "                    0.02%  53.548us         2  26.774us  26.006us  27.542us  cudaMemcpy\n",
            "                    0.01%  29.122us         1  29.122us  29.122us  29.122us  cuDeviceGetName\n",
            "                    0.01%  28.830us         1  28.830us  28.830us  28.830us  cudaLaunchKernel\n",
            "                    0.00%  6.1030us         1  6.1030us  6.1030us  6.1030us  cuDeviceGetPCIBusId\n",
            "                    0.00%  1.8340us         2     917ns     222ns  1.6120us  cuDeviceGet\n",
            "                    0.00%  1.6350us         3     545ns     178ns  1.1310us  cuDeviceGetCount\n",
            "                    0.00%     249ns         1     249ns     249ns     249ns  cuDeviceGetUuid\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VEtbnbvr6Wt7"
      },
      "source": [
        "## Connect back to needle Tensor\n",
        "\n",
        "So far we only played with the `backend_ndarray` subpackage, which is a self-contained ndarray implementation within needle.\n",
        "\n",
        "We can connect the ndarray back to needle as a backend."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JeThSA8zAu_v"
      },
      "source": [
        "import needle as ndl"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dobDH96Ql8SV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b74f9b03-86c6-4f37-f22e-d43d9a214bf2"
      },
      "source": [
        "x = ndl.Tensor([1,2,3], device=nd.cuda(), dtype=\"float32\")\n",
        "y = ndl.Tensor([2,3,5], device=nd.cuda(), dtype=\"float32\")\n",
        "z = x + y\n",
        "z"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "needle.Tensor([3. 5. 8.])"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ouXpj1v6g3z1",
        "outputId": "7cb789fa-4c5d-4e16-d05b-6c34dfd547d8"
      },
      "source": [
        "z.device"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "cuda()"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4827VUz3bwvA",
        "outputId": "0e0857b3-bb98-47a9-f512-6212547c1930"
      },
      "source": [
        "type(z.cached_data)"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "needle.backend_ndarray.ndarray.NDArray"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "74OUUH2REG18"
      },
      "source": [
        "## Write Standalone Python Test Files\n",
        "\n",
        "Now that we have additional c++/cuda libraries in needle, we will need to type make in order to rebuild the library. Additionally, because the colab environment caches the old library, it is inconvenient to use the ipython cells to debug the updated library.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sgLoV-_KHAM3",
        "outputId": "d15515c2-ba84-4806-b914-85f484b60cd4"
      },
      "source": [
        "!make"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-- Found pybind11: /usr/local/lib/python3.10/dist-packages/pybind11/include (found version \"2.11.1\")\n",
            "\u001b[33mCMake Warning (dev) at CMakeLists.txt:55 (find_package):\n",
            "  Policy CMP0146 is not set: The FindCUDA module is removed.  Run \"cmake\n",
            "  --help-policy CMP0146\" for policy details.  Use the cmake_policy command to\n",
            "  set the policy and suppress this warning.\n",
            "\n",
            "This warning is for project developers.  Use -Wno-dev to suppress it.\n",
            "\u001b[0m\n",
            "-- Found cuda, building cuda backend\n",
            "Mon Oct  9 15:12:01 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   51C    P0    28W /  70W |    105MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n",
            "-- Autodetected CUDA architecture(s):  7.5\n",
            "-- Configuring done (0.3s)\n",
            "-- Generating done (0.2s)\n",
            "-- Build files have been written to: /content/drive/MyDrive/10714f23/lecture14/build\n",
            "make[1]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[2]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[  0%] Built target ndarray_backend_cpu\n",
            "make[3]: Entering directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[3]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "[ 50%] Built target ndarray_backend_cuda\n",
            "make[2]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n",
            "make[1]: Leaving directory '/content/drive/MyDrive/10714f23/lecture14/build'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dudnLHRqoKY2"
      },
      "source": [
        "\n",
        "We recommend writing separate python files and invoke them from the command line. Create a new file `tests/mytest.py` and write your local tests. This is also a common develop practice in big projects that involves python c++ FFI."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TubIHJrkn4Sk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a421fa96-6bc4-46d7-b06d-feae5e000d1d"
      },
      "source": [
        "!python tests/mytest.py"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "python3: can't open file '/content/drive/MyDrive/10714f23/lecture14/tests/mytest.py': [Errno 2] No such file or directory\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ei0UR-FYoY1-"
      },
      "source": [
        "After we have building the library, we could choose to fully restart the runtime (factory reset runtime) if you want to bring the updated change back to another colab. Note that you will need to save your code changes to the drive or a private github repo."
      ]
    }
  ]
}