From ffe3fceb578295525d20d9e699875b85c0eb4459 Mon Sep 17 00:00:00 2001 From: Sam948-byte Date: Sun, 12 Jan 2025 18:21:12 -0600 Subject: [PATCH 1/6] oh boy documentation! --- docs/source/docs/objectDetection/about-object-detection.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/docs/objectDetection/about-object-detection.md b/docs/source/docs/objectDetection/about-object-detection.md index e7f89755d0..78216f44d8 100644 --- a/docs/source/docs/objectDetection/about-object-detection.md +++ b/docs/source/docs/objectDetection/about-object-detection.md @@ -35,7 +35,7 @@ Photonvision will letterbox your camera frame to 640x640. This means that if you ## Training Custom Models -Coming soon! +There is a [Jupyter Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/yolov8_to_rknn.ipynb) that contains all of the code necessary to train a yolov8 model for upload. ## Uploading Custom Models From 43c0429d4afaa5055dd1544e19f7e581d56e6833 Mon Sep 17 00:00:00 2001 From: Sam948-byte Date: Sun, 12 Jan 2025 18:21:12 -0600 Subject: [PATCH 2/6] oh boy documentation! --- .../objectDetection/about-object-detection.md | 2 +- scripts/yolov8_to_rknn.ipynb | 513 ++++++++++++++++++ 2 files changed, 514 insertions(+), 1 deletion(-) create mode 100644 scripts/yolov8_to_rknn.ipynb diff --git a/docs/source/docs/objectDetection/about-object-detection.md b/docs/source/docs/objectDetection/about-object-detection.md index e7f89755d0..78216f44d8 100644 --- a/docs/source/docs/objectDetection/about-object-detection.md +++ b/docs/source/docs/objectDetection/about-object-detection.md @@ -35,7 +35,7 @@ Photonvision will letterbox your camera frame to 640x640. This means that if you ## Training Custom Models -Coming soon! +There is a [Jupyter Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/yolov8_to_rknn.ipynb) that contains all of the code necessary to train a yolov8 model for upload. ## Uploading Custom Models diff --git a/scripts/yolov8_to_rknn.ipynb b/scripts/yolov8_to_rknn.ipynb new file mode 100644 index 0000000000..13bc69cdaf --- /dev/null +++ b/scripts/yolov8_to_rknn.ipynb @@ -0,0 +1,513 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "578NGV7h3aNh" + }, + "source": [ + "# YOLOv8 Training and Conversion to RKNN" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "axJXpbXE3aNi" + }, + "outputs": [], + "source": [ + "root_path = '/content'\n", + "%cd {root_path}\n", + "import os\n", + "root_path = os.getcwd()\n", + "\n", + "!git clone https://github.com/airockchip/ultralytics_yolov8 ultralytics\n", + "%cd ultralytics\n", + "\n", + "!pip install -e .\n", + "\n", + "from IPython import display\n", + "display.clear_output()\n", + "\n", + "import ultralytics\n", + "ultralytics.checks()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "I7eoco723aNj" + }, + "source": [ + "## Downloading the dataset\n", + "Input your Roboflow API key below. It can be obtained [here](https://app.roboflow.com/settings/api).\n", + "You can use your own dataset, the rest of the notebook should work with any number of classes, as long as the project is of \"object detection\" type." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Zo6xpEG73aNj" + }, + "outputs": [], + "source": [ + "!mkdir {root_path}/datasets\n", + "%cd {root_path}/datasets\n", + "\n", + "!pip install roboflow -q\n", + "\n", + "from roboflow import Roboflow\n", + "\n", + "rf = Roboflow(api_key=\"roboflowKey\")\n", + "project = rf.workspace(\"YOUR_WORKSPACE_HERE\").project(\"YOUR_PROJECT_HERE\")\n", + "dataset = project.version(5).download(\"yolov8\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ZCbLJbb13aNj" + }, + "outputs": [], + "source": [ + "import os\n", + "import re\n", + "from IPython.core.magic import register_line_cell_magic\n", + "\n", + "@register_line_cell_magic\n", + "def writetemplate(line, cell):\n", + " with open(line, 'w') as f:\n", + " f.write(cell.format(**globals()))\n", + " print(\"Wrote successfully to \" + line)\n", + "\n", + "@register_line_cell_magic\n", + "def replaceAllInFile(line, cell):\n", + " filename = line.strip()\n", + " replacements = eval(cell) # Assuming input is a valid Python expression\n", + " with open(filename, 'r') as f:\n", + " file_content = f.read()\n", + " for replaced, with_this in replacements:\n", + " file_content = re.sub(replaced, with_this, file_content)\n", + " with open(filename, 'w') as f:\n", + " f.write(file_content)\n", + " print(f\"Replaced successfully in {filename}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_FfsnYbu3aNk" + }, + "outputs": [], + "source": [ + "%cat {dataset.location}/data.yaml" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9-_bFyUB3aNk" + }, + "source": [ + "#### This is needed to fix the location of the various training directories, as it might not be accurate. You can rerun the code block above to double check." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "kFPrZzb83aNk" + }, + "outputs": [], + "source": [ + "%%replaceAllInFile {dataset.location}/data.yaml\n", + "\n", + "[\n", + " ('test: ..', 'test: ' + dataset.location),\n", + " ('train: ..', 'train: ' + dataset.location),\n", + " ('val: ..', 'val: ' + dataset.location),\n", + "]\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "mD4_Tt_d3aNk" + }, + "outputs": [], + "source": [ + "# define number of classes based on YAML\n", + "import yaml\n", + "with open(dataset.location + \"/data.yaml\", 'r') as stream:\n", + " num_classes = str(yaml.safe_load(stream)['nc'])\n", + "print(f\"num_classes: {num_classes}\")\n", + "%cd {root_path}/ultralytics" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AI5SyHiH7e2j" + }, + "source": [ + "#### This is the custom template that we need to train on, it lets us eventually convert to rknn." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "xbipZKKv3aNk" + }, + "outputs": [], + "source": [ + "%%writetemplate custom_yolov8.yaml\n", + "# Ultralytics YOLO ๐Ÿš€, AGPL-3.0 license\n", + "# YOLOv8 object detection model with P3-P5 outputs. For Usage examples see https://docs.ultralytics.com/tasks/detect\n", + "\n", + "# Parameters\n", + "nc: {num_classes} # number of classes\n", + "scales: # model compound scaling constants, i.e. 'model=yolov8n.yaml' will call yolov8.yaml with scale 'n'\n", + " # [depth, width, max_channels]\n", + " n: [0.33, 0.25, 1024] # YOLOv8n summary: 225 layers, 3157200 parameters, 3157184 gradients, 8.9 GFLOPs\n", + " s: [0.33, 0.50, 1024] # YOLOv8s summary: 225 layers, 11166560 parameters, 11166544 gradients, 28.8 GFLOPs\n", + " m: [0.67, 0.75, 768] # YOLOv8m summary: 295 layers, 25902640 parameters, 25902624 gradients, 79.3 GFLOPs\n", + " l: [1.00, 1.00, 512] # YOLOv8l summary: 365 layers, 43691520 parameters, 43691504 gradients, 165.7 GFLOPs\n", + " x: [1.00, 1.25, 512] # YOLOv8x summary: 365 layers, 68229648 parameters, 68229632 gradients, 258.5 GFLOPs\n", + "activation: nn.ReLU()\n", + "# YOLOv8.0n backbone\n", + "backbone:\n", + " # [from, repeats, module, args]\n", + " - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2\n", + " - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4\n", + " - [-1, 3, C2f, [128, True]]\n", + " - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8\n", + " - [-1, 6, C2f, [256, True]]\n", + " - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16\n", + " - [-1, 6, C2f, [512, True]]\n", + " - [-1, 1, Conv, [1024, 3, 2]] # 7-P5/32\n", + " - [-1, 3, C2f, [1024, True]]\n", + " - [-1, 1, SPPF, [1024, 5]] # 9\n", + "\n", + "# YOLOv8.0n head\n", + "head:\n", + " - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\n", + " - [[-1, 6], 1, Concat, [1]] # cat backbone P4\n", + " - [-1, 3, C2f, [512]] # 12\n", + "\n", + " - [-1, 1, nn.Upsample, [None, 2, 'nearest']]\n", + " - [[-1, 4], 1, Concat, [1]] # cat backbone P3\n", + " - [-1, 3, C2f, [256]] # 15 (P3/8-small)\n", + "\n", + " - [-1, 1, Conv, [256, 3, 2]]\n", + " - [[-1, 12], 1, Concat, [1]] # cat head P4\n", + " - [-1, 3, C2f, [512]] # 18 (P4/16-medium)\n", + "\n", + " - [-1, 1, Conv, [512, 3, 2]]\n", + " - [[-1, 9], 1, Concat, [1]] # cat head P5\n", + " - [-1, 3, C2f, [1024]] # 21 (P5/32-large)\n", + "\n", + " - [[15, 18, 21], 1, Detect, [nc]] # Detect(P3, P4, P5)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-X2A1DKQ3aNl" + }, + "source": [ + "## Training\n", + "You can adjust the following settings:\n", + "- model: one of [yolov8n, yolov8s, yolov8m, yolov8l, yolov8x]\n", + "- epochs: How many iterations to train for\n", + "- image_size: The input size of the images fed to the model. Should be a multiple of 32.\n", + "- batch: Number of samples per epoch. You should set this to the highest number possible without the training taking too much memory (it would crash if that happens, which is ok, just lower the number and try again)\n", + "- cache: Enables caching of dataset images in memory (True/ram), on disk (disk), or disables it (False)\n", + "- patience: After how many epochs without improvement to stop the training" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "FBdAUThb3aNl" + }, + "outputs": [], + "source": [ + "%cd {root_path}/ultralytics\n", + "model = \"yolov8s\"\n", + "epochs = 100\n", + "image_size = 640\n", + "batch=48\n", + "cache=True\n", + "patience=50\n", + "!yolo epochs={epochs} task=detect mode=train model=./custom_{model}.yaml data={dataset.location}/data.yaml imgsz={image_size} cache={cache} batch={batch} patience={patience}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DBg0mr943aNl" + }, + "outputs": [], + "source": [ + "latest_modified_time = 0\n", + "latest = None\n", + "\n", + "for foldername, subfolders, filenames in os.walk(root_path):\n", + " for filename in filenames:\n", + " if filename == \"best.pt\":\n", + " file_path = os.path.join(foldername, filename)\n", + " modified_time = os.path.getmtime(file_path)\n", + " if modified_time > latest_modified_time:\n", + " latest_modified_time = modified_time\n", + " latest = file_path\n", + "print(latest)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aCc2dQtQ3aNl" + }, + "source": [ + "### Installing RKNN Toolkit 2" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6ElrnSYH3aNl" + }, + "outputs": [], + "source": [ + "!wget https://github.com/rockchip-linux/rknn-toolkit2/raw/2c2d03def0c0908c86985b8190e973976ecec74c/rknn-toolkit2/packages/rknn_toolkit2-1.6.0+81f21f4d-cp310-cp310-linux_x86_64.whl\n", + "!pip install ./rknn_toolkit2-1.6.0+81f21f4d-cp310-cp310-linux_x86_64.whl" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qEYZWCr43aNl" + }, + "outputs": [], + "source": [ + "%cd {root_path}\n", + "!git clone https://github.com/airockchip/rknn_model_zoo/\n", + "%cd rknn_model_zoo\n", + "!git checkout eaa94d6f57ca553d493bf3bd7399a070452d2774\n", + "%cd examples/yolov8/python" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IdlTZWbp70Jp" + }, + "source": [ + "## Sample Images For Quantization" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "b5uDYGTI3aNl" + }, + "outputs": [], + "source": [ + "%%writefile imgs.txt\n", + "imgs/1.jpg\n", + "imgs/2.jpg\n", + "imgs/3.jpg\n", + "imgs/4.jpg\n", + "imgs/5.jpg\n", + "imgs/6.jpg\n", + "imgs/7.jpg\n", + "imgs/8.jpg\n", + "imgs/9.jpg\n", + "imgs/10.jpg\n", + "imgs/11.jpg\n", + "imgs/12.jpg\n", + "imgs/13.jpg\n", + "imgs/14.jpg\n", + "imgs/15.jpg\n", + "imgs/16.jpg\n", + "imgs/17.jpg\n", + "imgs/18.jpg\n", + "imgs/19.jpg\n", + "imgs/20.jpg" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Iun6wm-c3aNl" + }, + "outputs": [], + "source": [ + "import os\n", + "import shutil\n", + "import random\n", + "import glob\n", + "\n", + "def copy_and_rename_images(source_folder, destination_folder, n):\n", + " if not os.path.exists(source_folder):\n", + " print(f\"Source folder '{source_folder}' does not exist.\")\n", + " return\n", + " if not os.path.exists(destination_folder):\n", + " os.makedirs(destination_folder)\n", + " image_files = glob.glob(os.path.join(source_folder, '*.jpg'))\n", + " selected_images = random.sample(image_files, min(n, len(image_files)))\n", + " for i, image_path in enumerate(selected_images, start=1):\n", + " destination_path = os.path.join(destination_folder, f'{i}.jpg')\n", + " shutil.copy(image_path, destination_path)\n", + " print(f\"{min(n, len(image_files))} random images copied from '{source_folder}' to '{destination_folder}' and renamed.\")\n", + "copy_and_rename_images(dataset.location+\"/test/images\" , \"imgs\", 20)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "LINVU8YC3aNl" + }, + "outputs": [], + "source": [ + "%%replaceAllInFile {root_path}/rknn_model_zoo/examples/yolov8/python/convert.py\n", + "[\n", + " ('../../../datasets/COCO/coco_subset_20.txt', 'imgs.txt'),\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5kjkeL-k3aNl" + }, + "source": [ + "## Exporting to ONNX\n", + "This is an intermediate step between the PyTorch model and the RKNN model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "3S6xT1vJ3aNl" + }, + "outputs": [], + "source": [ + "%cd {root_path}/ultralytics\n", + "!yolo mode=export format=rknn model={latest}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "umZNREa-3aNl" + }, + "outputs": [], + "source": [ + "ex_path = '.'.join(latest.split('.')[:-1]) + '.onnx'\n", + "print(ex_path)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NSN5gRPS3aNl" + }, + "source": [ + "## Quantization\n", + "Here you choose whether to perform quantization, which makes the model lighter and faster, by converting all 32/16 bit floates in the model into 8 bit ints, which costs performance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "EVoWRad03aNl" + }, + "outputs": [], + "source": [ + "to_quantize = True # @param {type: \"boolean\"}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2Xo3XJaB3aNl" + }, + "source": [ + "## Exporting to RKNN - Final step๐ŸŽ‰" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "H13l4tVO3aNl" + }, + "outputs": [], + "source": [ + "%cd {root_path}/rknn_model_zoo/examples/yolov8/python\n", + "quant_code = \"i8\" if to_quantize else \"fp\"\n", + "output_model = f\"{root_path}/{dataset.name}-{model}-{image_size}-{quant_code}.rknn\"\n", + "!python convert.py {ex_path} rk3588 {quant_code} {output_model}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NyaqquzKLTY2" + }, + "source": [ + "This notebook was forked from [laviRZ's](https://github.com/lavirz/photonvision/blob/master/devTools/yolov8-to-rknn.ipynb)" + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "gpuType": "T4", + "provenance": [] + }, + "kaggle": { + "accelerator": "nvidiaTeslaT4", + "dataSources": [], + "isGpuEnabled": true, + "isInternetEnabled": true, + "language": "python", + "sourceType": "notebook" + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.12" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} From d39f992482467f9e2e06a9bdafd833402429ebd2 Mon Sep 17 00:00:00 2001 From: Sam Freund Date: Sun, 12 Jan 2025 19:50:29 -0600 Subject: [PATCH 3/6] update retribution to reflect a specific commit --- scripts/yolov8_to_rknn.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/yolov8_to_rknn.ipynb b/scripts/yolov8_to_rknn.ipynb index 13bc69cdaf..c83b876ad5 100644 --- a/scripts/yolov8_to_rknn.ipynb +++ b/scripts/yolov8_to_rknn.ipynb @@ -473,7 +473,7 @@ "id": "NyaqquzKLTY2" }, "source": [ - "This notebook was forked from [laviRZ's](https://github.com/lavirz/photonvision/blob/master/devTools/yolov8-to-rknn.ipynb)" + "This notebook was forked from [laviRZ's](https://github.com/laviRZ/photonvision/blob/76f67483b284cbba043e9ee865200d951f7de538/devTools/yolov5-to-rknn.ipynb)" ] } ], From 54778b4e1c6ae0f6ccd3d663384ca250b0b119a5 Mon Sep 17 00:00:00 2001 From: Sam Freund Date: Sun, 12 Jan 2025 20:42:55 -0600 Subject: [PATCH 4/6] Fun Fact! --- scripts/yolov8_to_rknn.ipynb | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/scripts/yolov8_to_rknn.ipynb b/scripts/yolov8_to_rknn.ipynb index c83b876ad5..4e9bc28f3d 100644 --- a/scripts/yolov8_to_rknn.ipynb +++ b/scripts/yolov8_to_rknn.ipynb @@ -473,7 +473,7 @@ "id": "NyaqquzKLTY2" }, "source": [ - "This notebook was forked from [laviRZ's](https://github.com/laviRZ/photonvision/blob/76f67483b284cbba043e9ee865200d951f7de538/devTools/yolov5-to-rknn.ipynb)" + "This notebook was forked from [laviRZ's](https://github.com/photonvision/photonvision/blob/76f67483b284cbba043e9ee865200d951f7de538/devTools/yolov5-to-rknn.ipynb)" ] } ], @@ -483,14 +483,6 @@ "gpuType": "T4", "provenance": [] }, - "kaggle": { - "accelerator": "nvidiaTeslaT4", - "dataSources": [], - "isGpuEnabled": true, - "isInternetEnabled": true, - "language": "python", - "sourceType": "notebook" - }, "kernelspec": { "display_name": "Python 3", "name": "python3" From 9b8af16aeaa98f1943bfa60ce1b3e13baa7f94da Mon Sep 17 00:00:00 2001 From: Sam Freund Date: Sun, 12 Jan 2025 21:22:15 -0600 Subject: [PATCH 5/6] oooh scary warning --- docs/source/docs/objectDetection/about-object-detection.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/source/docs/objectDetection/about-object-detection.md b/docs/source/docs/objectDetection/about-object-detection.md index 78216f44d8..ca31e1dbf7 100644 --- a/docs/source/docs/objectDetection/about-object-detection.md +++ b/docs/source/docs/objectDetection/about-object-detection.md @@ -35,6 +35,9 @@ Photonvision will letterbox your camera frame to 640x640. This means that if you ## Training Custom Models +:::{warning} +Power users only. This requires some setup, such as obtaining a Roboflow API token and dataset. It's additionally advised to have a general knowledge of ML before attempting to train your own model. + There is a [Jupyter Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/yolov8_to_rknn.ipynb) that contains all of the code necessary to train a yolov8 model for upload. ## Uploading Custom Models From 09cbf2cb9d236bebc0728e6d35c034e8eff3a544 Mon Sep 17 00:00:00 2001 From: Sam948-byte Date: Sun, 12 Jan 2025 21:42:17 -0600 Subject: [PATCH 6/6] fix warning --- docs/source/docs/objectDetection/about-object-detection.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/docs/objectDetection/about-object-detection.md b/docs/source/docs/objectDetection/about-object-detection.md index ca31e1dbf7..f87667a87e 100644 --- a/docs/source/docs/objectDetection/about-object-detection.md +++ b/docs/source/docs/objectDetection/about-object-detection.md @@ -37,6 +37,7 @@ Photonvision will letterbox your camera frame to 640x640. This means that if you :::{warning} Power users only. This requires some setup, such as obtaining a Roboflow API token and dataset. It's additionally advised to have a general knowledge of ML before attempting to train your own model. +::: There is a [Jupyter Notebook](https://github.com/PhotonVision/photonvision/blob/main/scripts/yolov8_to_rknn.ipynb) that contains all of the code necessary to train a yolov8 model for upload.