ee046746_tut_13_deep_object_tracking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"https://img.icons8.com/bubbles/100/000000/3d-glasses.png\" style=\"height:50px;display:inline\"> EE 046746 - Technion - Computer Vision\n",
    "---\n",
    "#### Tal Daniel\n",
    "\n",
    "## Tutorial 13 - Deep Object Tracking\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut_track_anim.gif\" style=\"height:200px\">\n",
    "\n",
    "* <a href=\"https://www.imageannotation.ai/blog/object-tracking-in-videos\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "\n",
    "* [What is the Object Tracking Task?](#-What-is-Object-Tracking?)\n",
    "    * [Object Tracking Vs. Object Detection](#-Object-Tracking-Vs.-Object-Detection)\n",
    "* [Under the Hood of Object Tracking](#-Under-the-Hood-of-Object-Tracking)\n",
    "    * Motion Model \n",
    "    * Visual Appearance Model\n",
    "* [Object Tracking Procedure](#-Object-Tracking-Procedure)\n",
    "* [Types of Tracking Algorithms](#-Types-of-Tracking-Algorithms)\n",
    "* [From Classic to Deep Learning Algorithms](#-From-Classic-to-Deep-Learning-Algorithms)\n",
    "* [Deep Object Tracking Algorithms](#-Deep-Object-Tracking-Algorithms)\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# imports for the tutorial\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/50/000000/accuracy.png\" style=\"height:50px;display:inline\"> What is Object Tracking?\n",
    "---\n",
    "* Object tracking in videos, or video object tracking, is the process of **detecting an object as it moves through space** in a video.\n",
    "* Object tracking has a wide range of applications in Computer Vision, such as surveillance, human-computer interaction, and medical imaging, traffic flow monitoring, human activity recognition, etc. \n",
    "    * For example, if the FBI wants to track a criminal running away in a car using citywide surveillance cameras.\n",
    "    * Or analyze a soccer game and the performance of the players."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Another example would be tracking shoppers path in a mall and anlayzing the number of people that entered/exited the mall.\n",
    "<img src=\"./assets/tut_track_mall.gif\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/96/000000/fight-pokemon.png\" style=\"height:50px;display:inline\"> Object Tracking Vs. Object Detection\n",
    "---\n",
    "* Why can’t we use object detection in each frame in the whole video and track the object?\n",
    "    * If the image has multiple objects, then we have no way of connecting the objects in the current frame to the previous frames.\n",
    "    * If the object you were tracking goes out of the camera view for a few frames and another one appears, we have no way of knowing if it’s the same object.\n",
    "    * Essentially, during detection, we work with **one image at a time** and we have no idea about the *motion* and past movement of the object, so **we can’t uniquely track objects in a video**.\n",
    "* Whenever there is a moving object in the videos, there are certain cases when the visual appearance of the object is not clear. \n",
    "* In all such cases, *detection* would fail while *tracking* succeeds as it also has the motion model and history of the object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    " #### Detection Failure Cases\n",
    " ---\n",
    " * **Occlusion**-  the object in question is partially or completely occluded.\n",
    " * **Identity Switches** - after two objects cross each other, can't tell the correct identity.\n",
    " * **Motion Blur** - object is blurred due to the motion of the object or camera. Hence, visually, the object doesn’t look the same anymore.\n",
    " * **Viewpoint Variation** - different viewpoint of an object may look very different visually and without the context. It becomes very difficult to identify the object using only visual detection.\n",
    " * **Scale Change** - huge changes in object scale may cause a failure in detection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Background Clutters** - background near object has similar color or texture as the target object. Hence, it may become harder to separate the object from the background.\n",
    "* **Illumination Variation** - illumination near the target object is significantly changed. Hence, it may become harder to visually identify it.\n",
    "* **Low Resolution** - when the number of pixels inside the ground truth bounding box is low, it may be too hard to detect the objects visually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/car-service.png\" style=\"height:50px;display:inline\"> Under the Hood of Object Tracking\n",
    "---\n",
    "* There are various techniques and algorithms which try to solve the tracking problem in many different ways.\n",
    "* A good object tracker has two basic models: **Motion Model** and **Visual Appearance Model**.\n",
    "\n",
    "<img src=\"./assets/tut_track_optical_flow.gif\" style=\"height:250px\">\n",
    "\n",
    "* <a href=\"https://nanonets.com/blog/optical-flow/\">Image Source - Introduction to Motion Estimation with Optical Flow</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Motion Model** - the ability to understand and model the motion of the object.\n",
    "    * A good motion model captures the dynamic behavior of an object. \n",
    "    * It predicts the potential position of objects in the future frames, hence, reducing the search space.\n",
    "    * However, the motion model alone can fail in scenarios where motion is caused by things that are not in a video or abrupt direction and speed change.\n",
    "    * Some of the classic methods understand the motion pattern of the object and try to predict that. \n",
    "        * However, the problem with such approaches is that they can’t predict the abrupt motion and direction changes.\n",
    "        * Examples of such techniques are Optical Flow, Kalman Filtering, Kanade-Lucas-Tomashi (KLT) feature tracker, mean shift tracking."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Visual Appearance Model** - the ability to understand the appearance of the object that is tracked.\n",
    "    * Trackers need to learn to discriminate the object from the background. \n",
    "    * In single object trackers (one object), visual appearance alone could be enough to track the object across frames, while In multiple-object trackers, visual appearance alone is not enough."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/workflow.png\" style=\"height:50px;display:inline\"> Object Tracking Procedure\n",
    "---\n",
    "In general, the object tracking procedure is composed of **4 main modules**:\n",
    "* **Target initialization/object detection**: an initial set of object detections is created. This is typically done by taking a set of bounding box coordinates and using them as inputs for the network. \n",
    "    * The idea is to draw bounding box of the target in the initial frame of the video and tracker has to estimate the target position in the remaining frames in the video.\n",
    "* **Appearance modeling**: learning the visual appearance of the object by using (deep) learning techniques. In this phase, the model learns the visual features of the object while in motion, various view-points, scale, illuminations etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Motion estimation**: the objective of motion estimation is learning to predict a zone where the target is most likely to be present in the subsequent frames.\n",
    "* **Target positioning**: motion estimation predicts the possible region where the target could be present, thus, yielding an area to search to lock down the exact location of the target.\n",
    "\n",
    "It is usually the case that tracking algorithms don’t try to learn all the variations of the object. Hence, most of the tracking algorithms are much faster than regular object detection.\n",
    "\n",
    "<img src=\"./assets/tut_track_cars.gif\" style=\"height:200px\">\n",
    "\n",
    "* <a href=\"https://www.move-lab.com/blog/tracking-things-in-object-detection-videos\">Image Source - Tracking Things In Object Detection Videos</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/nolan/64/categorize.png\" style=\"height:50px;display:inline\"> Types of Tracking Algorithms\n",
    "---\n",
    "We can classifiy object trackers according to whether they are based on automatic object detection or manual, whether they track a single object or capable of tracking multi objects and whether they operate *online* or *offline*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/rectangle.png\" style=\"height:50px;display:inline\"> Detection-Based Vs. Detection-Free\n",
    "---\n",
    "* **Detection-based**: the consecutive video frames are given to a pretrained object detector that forms a detection hypothesis which in turn is used to form tracking trajectories. \n",
    "    * It is more popular because new objects are detected and disappearing objects are terminated automatically.\n",
    "    * In these approaches, the tracker is used for the failure cases of object detection. \n",
    "    * In an another approach, object detector is run every $n$ frames and the remaining predictions are done using the tracker.\n",
    "    * Suitable approach for tracking for a long time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Detection-free**: requires manual initialization of a fixed number of objects in the first frame. It then localizes these objects in the subsequent frames. \n",
    "    * Cannot deal with the case where new objects appear in the middle frames.\n",
    "    \n",
    "<img src=\"./assets/tut_track_manual.jpg\" style=\"height:250px\">\n",
    "\n",
    "\n",
    "* <a href=\"https://www.freecodecamp.org/news/object-detection-in-colab-with-fizyr-retinanet-efed36ac4af3/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/group-objects.png\" style=\"height:50px;display:inline\"> Single Object Vs. Multi Object\n",
    "---\n",
    "* **Single Object Tracking**: only a single object is tracked even if the environment has multiple objects in it. The object to be tracked is determined by the initialization in the first frame.\n",
    "* **Multi Object Tracking**: all the objects present in the environment are tracked over time. If a detection based tracker is used it can even track new objects that emerge in the middle of the video."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/96/000000/wi-fi-connected.png\" style=\"height:50px;display:inline\"> Offline Vs. Online\n",
    "---\n",
    "* **Offline Trackers** - used when you have to track an object in a recorded stream.\n",
    "    * For example if you have recorded videos of a soccer game of an opponent team which needs to be analyzed for strategic analysis. In such case, you can not only use the past frames but also future frames to make more accurate tracking predictions.\n",
    "* **Offline Learning Trackers** - the training of these trackers only happen offline.\n",
    "    * As opposed to online learning trackers, these trackers don’t learn anything during run time.\n",
    "    * We can train a tracker to identify persons and then these trackers can be used to continuously track all the persons in a video stream. Pre-trained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Online Trackers** - online trackers are used where predictions are available immediately and hence, they can’t use future frames to improve the results.\n",
    "* **Online Learning Trackers** - typically learn about the object to track using the initialization frame and few subsequent frames, making these trackers more general because you can just draw a bounding box around any object and track it.\n",
    "    * For example, if you want to track a person with red shirt in the airport, you can just draw a bounding box around that person in 1 or few frames. The tracker would learn about the object using these frames and would continue to track that person."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* In online learning trackers, Center Red box is specified by the user, it is taken as the positive example and all the boxes surrounding the object are taken as negative class and a classifier is trained which learns to distinguish the object from the background.\n",
    "\n",
    "<img src=\"./assets/tut_track_online.jpeg\" style=\"height:250px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/cute-clipart/64/000000/horizontal-settings-mixer.png\" style=\"height:50px;display:inline\"> From Classic to Deep Learning Algorithms\n",
    "---\n",
    "* Most of the classic trackers are not very accurate due to the limitations mentioned in the beginning.\n",
    "    * However, some times they can be useful to run in a resource constraint environment like an embedded system.\n",
    "* **Kernelized Correlation Filters (KCF)** tracker is a very fast, well-performing tracker. Read more: <a href=\"https://arxiv.org/abs/1404.7584\">High-Speed Tracking with Kernelized Correlation Filters</a>\n",
    "* A lot of classic tracking algorithms are integrated in <a href=\"https://www.pyimagesearch.com/2018/07/30/opencv-object-tracking/\">OpenCV’s tracking API</a>.\n",
    "    * A very simple tracking technique is based on finding centroids and can be easily implemented with OpenCV.\n",
    "        * <a href=\"https://www.pyimagesearch.com/2018/07/23/simple-object-tracking-with-opencv/\">Simple object tracking with OpenCV</a> (Code is available).\n",
    "* Deep learning based trackers are now miles ahead of traditional trackers in terms of accuracy.\n",
    "\n",
    "<a href=\"https://www.pyimagesearch.com/2018/07/23/simple-object-tracking-with-opencv/\"><img src=\"./assets/tut_track_centeroids_opencv.gif\" style=\"height:250px\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/cotton/64/000000/magic-crystal-ball.png\" style=\"height:50px;display:inline\"> Deep Object Tracking Algorithms\n",
    "---\n",
    "* We will present 3 popular deep learning based trackers, but there are more.\n",
    "* Only one of them is implemented in OpenCV, but we will provide PyTorch code for the rest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/network.png\" style=\"height:50px;display:inline\"> Generic Object Tracking Using Regression Networks (GOTURN)\n",
    "---\n",
    "* GOTURN was introduced by David Held, Sebastian Thrun, Silvio Savarese in their paper titled <a href=\"http://davheld.github.io/GOTURN/GOTURN.pdf\">“Learning to Track at 100 FPS with Deep Regression Networks”</a>.\n",
    "* GOTURN uses deep neural networks to track objects in an **offline** fashion. \n",
    "    * Most tracking algorithms train *online*, which is to say the algorithm learns how the object appears *only at runtime*. \n",
    "    * In contrast, GOTURN is trained on thousands of chunks of video before runtime, and as a result, it doesn’t need to train at all during runtime.\n",
    "* GOTURN is trained using a pair of *cropped* frames from thousands of videos and outputs the bounding box around the object in the second frame.\n",
    "\n",
    "<img src=\"./assets/tut_track_goturn.jpg\" style=\"height:250px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* In the first frame (also referred to as the previous frame), the location of the object is known, and the frame is cropped to two times the size of the bounding box around the object. \n",
    "* The object in the first cropped frame is *always centered*.\n",
    "* The location of the object in the second frame (also referred to as the current frame) needs to be predicted.\n",
    "* The bounding box used to crop the first frame is also used to crop the second frame. Because the object might have moved, the object is not centered in the second frame.\n",
    "* A **Convolutional Neural Network (CNN)** is trained to predict the location of the bounding box in the second frame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_track_goturn_arch.jpg\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* GOTURN is the only deep-learning algorithm implemented in OpenCV.\n",
    "* GOTURN can run very fast i.e. 100fps on a GPU powered machine.\n",
    "* We will now see how it works!\n",
    "* If you prefer to work with PyTorch, here are 2 repositories:\n",
    "    * <a href=\"https://github.com/amoudgl/pygoturn\">Source 1</a>\n",
    "    * <a href=\"https://github.com/nrupatunga/PY-GOTURN/\">Source 2</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# ------------------------- <img src=\"https://img.icons8.com/color/96/000000/code.png\" style=\"height:50px;display:inline\"> CODE TIME -------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The first step is downloading the pre-trained model files. There are 2 ways to download it:\n",
    "1. Downloading the files separately and merging them locally, instructions can be found here: <a href=\"https://github.com/spmallick/goturn-files\">LINK</a>\n",
    "2. Download the merged files from Dropbox: <a href=\"https://www.dropbox.com/sh/77frbrkmf9ojfm6/AACgY7-wSfj-LIyYcOgUSZ0Ua?dl=0\">LINK</a>\n",
    "\n",
    "You should end up with 2 files: `goturn.caffemodel` and `goturn.prototxt` and they should be placed in the current working directory (`./`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# initialize tracker\n",
    "tracker = cv2.TrackerGOTURN_create()\n",
    "\n",
    "# Read video\n",
    "video = cv2.VideoCapture(\"./datasets/biker.mp4\")\n",
    "\n",
    "# Exit if video not opened\n",
    "if not video.isOpened():\n",
    "    print(\"Could not open video\")\n",
    "    raise SystemError\n",
    "\n",
    "# Read first frame\n",
    "ok, frame = video.read()\n",
    "if not ok:\n",
    "    print(\"Cannot read video file\")\n",
    "    raise SystemError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Define a bounding box\n",
    "# bbox = (276, 23, 86, 320)\n",
    "# boundind box parameters: (y1, y2, x1, x2)\n",
    " \n",
    "# Uncomment the line below to select a different bounding box\n",
    "bbox = cv2.selectROI(frame, False)\n",
    "# False is to say that we want to draw rectangle from top left (and not from the center)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Initialize tracker with first frame and bounding box\n",
    "ok = tracker.init(frame, bbox)  \n",
    "\n",
    "# `ok` is a boolean which is False until the frame is tracked."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Predict the bounding box in a new frame \n",
    "\n",
    "We loop over all frames in the video and find the bounding box for new frames using `tracker.update`. The rest of the code is simply for timing and displaying."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "while True:\n",
    "    # this loop will stop once we reached the final frame\n",
    "    # Read a new frame\n",
    "    ok, frame = video.read()\n",
    "    if not ok:\n",
    "        break\n",
    "\n",
    "    # Start timer\n",
    "    timer = cv2.getTickCount()\n",
    "\n",
    "    # Update tracker\n",
    "    ok, bbox = tracker.update(frame)\n",
    "\n",
    "    # bbox is the returned bounding box, you can do all sorts of stuff with it\n",
    "\n",
    "    # Calculate Frames per second (FPS)\n",
    "    fps = cv2.getTickFrequency() / (cv2.getTickCount() - timer);\n",
    "\n",
    "    # Draw bounding box\n",
    "    if ok:\n",
    "        # Tracking success\n",
    "        p1 = (int(bbox[0]), int(bbox[1]))\n",
    "        p2 = (int(bbox[0] + bbox[2]), int(bbox[1] + bbox[3]))\n",
    "        cv2.rectangle(frame, p1, p2, (255, 0, 0), 2, 1)\n",
    "        # mark the object\n",
    "        # cv2.putText(frame, \"Object\", (int(bbox[0]), int(bbox[1])), cv2.FONT_HERSHEY_SIMPLEX, 0.75, (0, 0, 255),2)\n",
    "    else:\n",
    "        # Tracking failure\n",
    "        cv2.putText(frame, \"Tracking failure detected\", (100, 80), cv2.FONT_HERSHEY_SIMPLEX, 0.75, (0, 0, 255),2)\n",
    "\n",
    "    # Display tracker type on frame\n",
    "    cv2.putText(frame, \"GOTURN Tracker\", (100, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.75, (50, 170, 50),2);\n",
    "\n",
    "    # Display FPS on frame\n",
    "    cv2.putText(frame, \"FPS : \" + str(int(fps)), (100, 50), cv2.FONT_HERSHEY_SIMPLEX, 0.75, (50, 170, 50), 2);\n",
    "\n",
    "    # Display result\n",
    "    cv2.imshow(\"Tracking\", frame)\n",
    " \n",
    "    # Exit if ESC pressed\n",
    "    k = cv2.waitKey(1) & 0xff\n",
    "    if k == 27:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# close all windows\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/cotton/64/000000/network.png\" style=\"height:50px;display:inline\"> Multi-Domain Convolutional Neural Network Tracker (MDNet)\n",
    "---\n",
    "* MDNet, an **online** video object tracker, was introduced by Hyeonseob Nam and Bohyung Han in their paper titled <a href=\"https://arxiv.org/abs/1510.07945\">“Learning Multi-Domain Convolutional Neural Networks for Visual Tracking”</a>.\n",
    "    * Winner of VOT2015 challenge.\n",
    "* Because training a deep neural network is computationally expensive, small networks are used for training around deployment time. \n",
    "* The drawback to small networks is that they lack the classification/discrimination power of larger networks.\n",
    "* In order to deal with the fact that networks which train at runtime have lower discriminatory power, the training of the network can be split into different steps. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* For instance, the entire network can be trained *before* runtime, but during runtime, the first few layers of the network are used as feature extractors and only the last few layers of the network have their weights adjusted.\n",
    "* Essentially, the **CNNs are trained beforehand** and used to extract features, while the **last layers can quickly be trained online**. \n",
    "* Theoretically, this creates a multi-domain CNN that can be used in many different scenarios, capable of discriminating between background and target."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* In practice, the *background* of one video could be the *target* of a different video, and so the CNN must have some method of discriminating between these two situations. \n",
    "* MDNet handles possible confusion from similar targets and backgrounds by dividing the network into two portions, a shared portion and a portion that remains independent for every domain.\n",
    "* **Every domain has its own training video**, and the network is trained for the total number of different domains.\n",
    "    * The network is first trained over K-domains iteratively where each domain classifies between its target and background. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_track_mdnet.PNG\" style=\"height:300px\">\n",
    "\n",
    "* In the $k^{th}$ iteration, the network is updated based on a minibatch that consists of the training samples from the $(k mod K)^{th}$ sequence, where only a single branch `fc6(k mod K)` is enabled. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* After training is complete, the layers specific to the different domains are removed and as a result, a feature extractor capable of interpreting any given background/object pairs is created.\n",
    "* During the process of inference, a binary classification layer (a single fully-connected layer, `fc6`) is created by removing the domain-specific layers and adding a binary classifier.\n",
    "    * To estimate the target state in each frame, $N$ target candidates $x_1, . . . , x_N$ sampled around the previous target state are evaluated using the network, and we obtain their positive scores $f^{+}(x^{i})$ and negative scores $f^{−}(x^{i})$ from the network. The optimal target state $x^{∗}$ is given by finding the example with the maximum positive score as $$ x^{*} = arg \\max_{x^{i}} f^{+}(x^{i}) $$\n",
    "\n",
    "* MDNet is one of the most accurate deep learning based online training, detection free, single object tracker. \n",
    "* <a href=\"https://github.com/HyeonseobNam/py-MDNet\">PyTorch Code</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# ------------------------- <img src=\"https://img.icons8.com/color/96/000000/code.png\" style=\"height:50px;display:inline\"> CODE TIME -------------------------\n",
    "\n",
    "* Run from Terminal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"./models/pyMDNet/tracking\")\n",
    "from models.pyMDNet.tracking.run_tracker import track_mdnet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "vid_path = './datasets/ironman.mp4'\n",
    "img_path = './datasets/ironman_frames'\n",
    "seq_name = \"iron_man_custom\"\n",
    "track_mdnet(vid_path, img_path, display=False, seq_name=seq_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/50/000000/network.png\" style=\"height:50px;display:inline\"> Deep Simple Online and Realtime Tracking (Deep SORT)\n",
    "---\n",
    "* Deep SORT, an **online** multiple object tracker which is an extenstion of the SORT algorithm, was introduced by Nicolai Wojke, Alex Bewley and Dietrich Paulus in their paper titled <a href=\"https://arxiv.org/abs/1703.07402\">“Simple Online and Realtime Tracking with a Deep Association Metric”</a>.\n",
    "* Deep SORT integrates appearance information to improve the performance of SORT. \n",
    "    * Due to this extension it is possible to track objects through longer periods of occlusions, effectively reducing the number of *identity switches*. \n",
    "* Much of the computational complexity is placed into an offline pre-training stage where a deep association metric is learned on a large-scale person re-identification dataset. \n",
    "* During online application, measurement-to-track associations is established using nearest neighbor queries in visual appearance space. \n",
    "* Experimental evaluation shows that these extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_track_deepsort.gif\" style=\"height:250px\">\n",
    "\n",
    "* <a href=\"https://github.com/kimyoon-young/centerNet-deep-sort\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* The original SORT algorithm (<a href=\"https://arxiv.org/abs/1602.00763\">Paper</a>, <a href=\"https://github.com/abewley/sort\">Code</a>) proposed the following steps to perform tracking:\n",
    "    * **Detection**: using a CNN-based detection architecture (VGG16) to extract regions.\n",
    "    * **Estimation Model**: the representation and the motion model used to propagate a target’s identity into the next frame.\n",
    "        * Inter-frame displacements of each object are estimated with a linear constant velocity model which is independent of other objects and camera motion. The state of each target is modeled as: $$x=[u, v, s, r, \\dot{u}, \\dot{v}, \\dot{s}]$$\n",
    "        * $u$ and $v$ represent the horizontal and vertical pixel location of the centre of the target, while the scale $s$ and $r$ represent the scale (area) and the aspect ratio of the target’s bounding box respectively.\n",
    "        * When a detection is associated to a target, the detected bounding box is used to update the target state where the velocity components are solved optimally via a <a href=\"https://www.codeproject.com/articles/865935/object-tracking-kalman-filter-with-ease\">**Kalman filter**</a> framework."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Steps continued:\n",
    "    * **Data Association**: In assigning detections to existing targets, each target’s bounding box geometry is estimated by predicting its new location in the current frame.\n",
    "        * The assignment cost matrix is then computed as the intersection-over-union (IOU) distance between each detection and all predicted bounding boxes from the existing targets. \n",
    "    * **Creation and Deletion of Track Identities**: When objects enter and leave the image, unique identities need to be created or destroyed accordingly.\n",
    "        * If an object reappears, tracking will implicitly resume under a new identity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* So where is the *deep learning* in all of that?\n",
    "* Despite the effectiveness of Kalman filter, it fails in many of the real world scenarios, like occlusions, different view points etc.\n",
    "* **Deep SORT** replaced the CNNs with YOLO (You Only Look Once) deep object detector and introduced another distance metric based on the *“appearance”* of the object.\n",
    "* The idea is to obtain a vector that can describe all the features of a given image (here, the crop of the object is used).\n",
    "    * First, build a classifier over the dataset, train it till it achieves a reasonably good accuracy, and then strip the final classification layer. \n",
    "    * Assuming a classical architecture, we will be left with a dense layer producing a single feature vector, waiting to be classified.\n",
    "    * That feature vector becomes the “appearance descriptor” of the object.\n",
    "    * The loss of appearance vectors is added to the loss of the original SORT.\n",
    "* <a href=\"https://github.com/ZQPei/deep_sort_pytorch\">PyTorch Code (with YOLO)</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "* Deep Video Object Tracking - <a href=\"https://www.youtube.com/watch?v=CYW6T2Q24z0\">Deep Video Object Tracking - Xavier Giro - UPC Barcelona 2019</a>\n",
    "* GOTURN - <a href=\"https://www.youtube.com/watch?v=SygkiWNSkWk\">GOTURN : Deep Learning based Object Tracker </a>\n",
    "    * GOTURN - <a href=\"https://www.youtube.com/watch?v=kMhwXnLgT_I\"> GOTURN - a neural network tracker</a>\n",
    "* MDNet - <a href=\"https://www.youtube.com/watch?v=zYM7G5qd090\">Learning Multi-Domain Convolutional Neural Networks for Visual Tracking (MDNet)</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "---\n",
    "\n",
    "* <a href=\"https://cv-tricks.com/object-tracking/quick-guide-mdnet-goturn-rolo/\">Zero to Hero: A Quick Guide to Object Tracking: MDNET, GOTURN, ROLO - Ankit Sachan</a>\n",
    "* <a href=\"https://www.imageannotation.ai/blog/object-tracking-in-videos\">Object Tracking In Videos</a>\n",
    "* <a href=\"https://www.learnopencv.com/goturn-deep-learning-based-object-tracking/\">GOTURN : Deep Learning based Object Tracking</a>\n",
    "* <a href=\"https://nanonets.com/blog/object-tracking-deepsort/#deep-sort\">DeepSORT: Deep Learning to Track Custom Objects in a Video</a>\n",
    "* Icons from <a href=\"https://icons8.com/\">Icon8.com</a> - https://icons8.com"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}