05-remote_inference_grpc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f73046ff",
   "metadata": {},
   "source": [
    "# Using the model server to do an inference using gRPC\n",
    "\n",
    "In this notebook we are going to use the gRPC inference endpoint to query our model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60552a1d-4061-4bb5-bd60-16f770515a75",
   "metadata": {},
   "source": [
    "## First, replace the placeholder with the grpcUrl you got at the previous step from the Model Serving configuration\n",
    "\n",
    "The port, the model name and the classes definition file are already filled in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba98525f-df87-40fe-8f86-2c0d99a295c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'yolo'\n",
    "grpc_url = 'change_me' # e.g. 'grpc://modelmesh-serving.nerc-demo-xxx:8033'\n",
    "classes_file = 'coco.yaml'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32e060bb-5620-4365-bb45-d7e07ac093dc",
   "metadata": {},
   "source": [
    "## Make sure we have the libraries we need\n",
    "If you want, you can have a look at requirements.txt to see which libraries are used. One of them is PyTorch, as this is the frameworks that was used to train the model. Some parts of this library are still need to do the image preparation before querying the model.\n",
    "\n",
    "The installation of the libraries can take up to 1mn, please be patient. **There may be Errors or Warnings during this installation, but you can ignore them**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5ac29a8-da58-4e36-a92c-6be1f4c5d32a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade pip\n",
    "!pip install -qr requirements.txt\n",
    "\n",
    "# Install a CPU-only version of Pytorch\n",
    "!pip install -q -r requirements-torch-cpu.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b6c7a78-7bec-4c81-9a53-75ea36470b60",
   "metadata": {},
   "source": [
    "## Import the libraries and module where we have all helper code for doing the inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db9df000-a171-4652-8160-272f81e49612",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import cv2\n",
    "from matplotlib import pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "from remote_infer_grpc import ort_v5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30c9819b",
   "metadata": {},
   "source": [
    "You can also have a look at the file `remote_infer_grpc.py` to get more details on the pre and post-processing that will happen with the data before and after sending it to the inference endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f65acbc-82a2-4a3a-bdb6-2fff1803d518",
   "metadata": {},
   "source": [
    "## Now set the parameters for the inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84370c09-7123-4df1-8d08-740cb86b0649",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. The image you want to analyze\n",
    "image_path='images/bus.jpg' # You can replace this with an image you upload\n",
    "#image_path='images/zidane.jpg' \n",
    "\n",
    "# 2. Confidence threshold, between 0 and 1 (detections with less score won't be retained)\n",
    "conf = 0.4\n",
    "\n",
    "# 3. Intersection over Union Threshold, between 0 and 1 (cleanup overlapping boxes)\n",
    "iou = 0.6"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1339611-0e6b-49b0-ac6d-bd5c1e1fca56",
   "metadata": {},
   "source": [
    "## Launch the inference and display the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d232504-f324-41fa-950d-2f96a5883478",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# extract host\n",
    "pattern1 = r\"\\/\\/(.+?):\\d+\"\n",
    "match1 = re.search(pattern1, grpc_url)\n",
    "if match1:\n",
    "    grpc_host = match1.group(1)\n",
    "\n",
    "# extract port\n",
    "pattern2 = r\"(\\d+)$\"\n",
    "match2 = re.search(pattern2, grpc_url)\n",
    "if match2:\n",
    "    grpc_port = match2.group(1)\n",
    "\n",
    "infer=ort_v5(grpc_host, grpc_port, model_name, 640, classes_file)\n",
    "img, out, result = infer(image_path, conf, iou)\n",
    "print(f'{result}')\n",
    "print('Predictions:')\n",
    "print(out)\n",
    "print('Format: each detection is a float64 array shaped as [top_left_corner_x, top_left_corner_y, bottom_right_corner_x, bottom_right_corner_y, confidence, class_index]')\n",
    "print('The coordinates are relative to a letterboxed representation of the image of size 640x640')\n",
    "img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
    "fig = plt.gcf()\n",
    "fig.set_size_inches(24, 12)\n",
    "plt.axis('off')\n",
    "plt.imshow(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9923faec",
   "metadata": {},
   "source": [
    "## Perfect, we can see that the model serving API is working!\n",
    "\n",
    "You can now get back to the instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abc2e68e-37a7-4876-b7f3-7cc1d869b64c",
   "metadata": {},
   "source": [
    "## Time it on multiple images!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "427cfe3e-0bf3-4437-a224-f8b53327daa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "\n",
    "folder_path = 'images'\n",
    "infer=ort_v5(grpc_host, grpc_port, model_name, 640, classes_file)\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "# Iterate over all files in the folder\n",
    "for filename in os.listdir(folder_path):\n",
    "    # Get the full path of the file\n",
    "    file_path = os.path.join(folder_path, filename)\n",
    "\n",
    "    # Check if the file path is a file (not a directory)\n",
    "    if os.path.isfile(file_path):\n",
    "        # Do the inference on the file\n",
    "        img, out, result = infer(file_path, conf, iou)\n",
    "        print(f'{result}')\n",
    "\n",
    "end_time = time.time()\n",
    "execution_time = end_time - start_time\n",
    "print(f\"Executed in {execution_time:.2f} seconds\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}