YoloV3/Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autonomous driving - Car detection\n",
    "\n",
    "The ideas in this notebook are described in the two YOLO papers: YOLOv3: An Incremental Improvement ( https://arxiv.org/abs/1804.02767 ). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Kiko-PC\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\h5py\\__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.pyplot import imshow\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from keras import backend as K\n",
    "from keras.models import load_model\n",
    "from yad2k.utils.yolo_utils import read_classes, read_anchors, generate_colors, preprocess_image, letterbox_image, draw_boxes, scale_boxes\n",
    "from yad2k.keras_yolo import yolo_boxes_and_scores, yolo_head\n",
    "from lane_utils import *\n",
    "import time\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "Bounding box exmple :\n",
    "\n",
    "<img src=\"notebook_images/rectangle.png\" style=\"width:70%;height:70%;\">\n",
    "\n",
    "\n",
    "In this prject, I use pretrained weights, where we have 80 trained yolo classes, for recognition.  \n",
    "The class label is represented as $c$ and it's integer from 1 to 80, each number represent the class label accordingly.  \n",
    "If $c=3$, then the classified object is a $car$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - YOLO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "YOLO stands for You Only Look Once. It's an object detector that uses features learned by a deep convolutional neural network to detect an object. Before we get out hands dirty with code, we must understand how YOLO works.\n",
    "\n",
    "### 2.1 - Architercutre details\n",
    "\n",
    "Neural network io:\n",
    "-  **input** : (m, 416, 416, 3)\n",
    "-  **output** : confidece of an object being present in the rectangle, list of rectangles position and sizes and classes of the objects begin detected. Each bounding box is represented by 6 numbers $(Pc, Rx, Ry, Rh, Rw, C1..Cn)$ as explained above. In this case n=80, which means we have $c$ as 80-dimensional vector, and the final size of representing the bounding box is 85 \n",
    "\n",
    "The first detection is made by the 82nd layer. For the first 81 layers, the image is down sampled by the network, such that the 81st layer has a stride of 32. If we have an image of 416 x 416, the resultant feature map would be of size 13 x 13. One detection is made here using the 1 x 1 detection kernel, giving us a detection feature map of 13 x 13 x 3 x 85.\n",
    "\n",
    "Then, the feature map from layer 79 is subjected to a few convolutional layers before being up sampled by 2x to dimensions of 26 x 26. This feature map is then depth concatenated with the feature map from layer 61. Then the combined feature maps is again subjected a few 1 x 1 convolutional layers to fuse the features from the earlier layer (61). Then, the second detection is made by the 94th layer, yielding a detection feature map of 26 x 26 x 3 x 85.\n",
    "\n",
    "A similar procedure is followed again, where the feature map from layer 91 is subjected to few convolutional layers before being depth concatenated with a feature map from layer 36. Like before, a few 1 x 1 convolutional layers follow to fuse the information from the previous layer (36). We make the final of the 3 at 106th layer, yielding feature map of size 52 x 52 x 3 x 85.\n",
    "\n",
    "YOLO v3, in total uses 9 anchor boxes. Three for each scale. \n",
    "\n",
    "<img src=\"notebook_images/architecture.png\" style=\"width:95%;height:95%;\">\n",
    "\n",
    "<caption> Note : The last three terms of the loss function in YOLO v2 are the squared errors, whereas in YOLO v3, they’ve been replaced by cross-entropy error terms. In other words, object confidence and class predictions in YOLO v3 are now predicted through logistic regression.  \n",
    "YOLO v3, in total uses 9 anchor boxes. Three for each scale</caption>  \n",
    "\n",
    "---------------------------------------\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For keeping the things simple we will flatten the last two dimensions, from (19, 19, 5, 85) the output of our CNN is flatten to (19, 19, 425).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, for each box (of each cell) we will compute the following elementwise product and extract a probability that the box contains a certain class.\n",
    "\n",
    "<img src=\"notebook_images/probability_extraction.png\" style=\"width:700px;height:400;\">\n",
    "<caption><center>**Find the class detected by each box**<br> </center></caption>\n",
    "\n",
    "Here's one example visualizing what YOLO can predict on an image:\n",
    "- For each of the $SxS$ grid cells, find the maximum of the confidence scores (taking a max across both the 5 anchor boxes and across different classes).  \n",
    "A confidence score is: probability(containing an object) x IoU(pred, truth).\n",
    "If the cell contains an object, it predicts a probability of this object belonging to one class Ci, i=1,2,…, K: probability(the object belongs to the class Ci | containing an object). At this stage, the model only predicts one set of class probabilities per cell, regardless of the number of boxes B. \n",
    "- Color that grid cell according to what object that grid cell considers the most likely.\n",
    "- Draw rectangle \n",
    "\n",
    "<img src=\"notebook_images/yolo.png\" style=\"width:100%;height:100%;\">\n",
    "<caption> Each of the $SxS$ grid cells colored according the class that has the largest predicted probability in specified cell.</caption>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1 - Example Filtering with a threshold on class scores\n",
    "\n",
    "The output result may contain several rectangles that are false positives or overlap, so we need to find a way to reduce them.\n",
    "The first attempt to reduce these rectangles is to filter them by threshold.\n",
    "\n",
    "\n",
    "**Input arguments**: \n",
    "- `box_confidence`: tensor of shape $(19 \\times 19, 5, 1)$ containing $p_c$ (confidence probability that there's some object) for each of the 5 boxes predicted in each of the 19x19 cells.\n",
    "- `boxes`: tensor of shape $(19 \\times 19, 5, 4)$ containing $(b_x, b_y, b_h, b_w)$ for each of the 5 boxes per cell.\n",
    "- `box_class_probs`: tensor of shape $(19 \\times 19, 5, 80)$ containing the detection probabilities $(c_1, c_2, ... c_{80})$ for each of the 80 classes for each of the 5 boxes per cell.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def yolo_filter_boxes(box_confidence, boxes, box_class_probs, threshold = .6):\n",
    "    \"\"\"Filters YOLO boxes by thresholding on object and class confidence.\n",
    "    \n",
    "    Arguments:\n",
    "    box_confidence -- tensor of shape (19, 19, 5, 1)\n",
    "    boxes -- tensor of shape (19, 19, 5, 4)\n",
    "    box_class_probs -- tensor of shape (19, 19, 5, 80)\n",
    "    threshold -- real value, if [ highest class probability score < threshold], then get rid of the corresponding box\n",
    "    \n",
    "    Returns:\n",
    "    scores -- tensor of shape (None,), containing the class probability score for selected boxes\n",
    "    boxes -- tensor of shape (None, 4), containing (b_x, b_y, b_h, b_w) coordinates of selected boxes\n",
    "    classes -- tensor of shape (None,), containing the index of the class detected by the selected boxes\n",
    "    \n",
    "    Note: \"None\" is here because you don't know the exact number of selected boxes, as it depends on the threshold. \n",
    "    For example, the actual output size of scores would be (10,) if there are 10 boxes.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Step 1: Compute box scores\n",
    "    box_scores = box_confidence * box_class_probs #multiply box probability(p) with class probability\n",
    "    \n",
    "    # Step 2: Find the box_classes thanks to the max box_scores, keep track of the corresponding score\n",
    "    box_classes = K.argmax(box_scores, axis=-1) #max class index\n",
    "    box_class_scores = K.max(box_scores, axis=-1) #max class probability\n",
    "    \n",
    "    # Step 3: Create a filtering mask based on \"box_class_scores\" by using \"threshold\". The mask should have the\n",
    "    # same dimension as box_class_scores, and be True for the boxes you want to keep (with probability >= threshold)\n",
    "    filtering_mask = box_class_scores>=threshold\n",
    "    \n",
    "    # Step 4: Apply the mask to scores, boxes and classes\n",
    "    scores = tf.boolean_mask(tensor=box_class_scores, mask=filtering_mask, name='score_mask')\n",
    "    boxes  = tf.boolean_mask(tensor=boxes, mask=filtering_mask, name='box_mask')\n",
    "    classes = tf.boolean_mask(tensor=box_classes, mask=filtering_mask, name='classes_mask')\n",
    "    \n",
    "    return scores, boxes, classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2 : Non-max suppression ###\n",
    "\n",
    "Even after yolo filtering by thresholding over, we still have a lot of overlapping boxes. Second approach and filtering is Non-Max suppression algorithm.\n",
    "\n",
    "* Discard all boxes with $Pc <= 0.6$  \n",
    "* While tehre are any remaining boxes : \n",
    "    * Pick the box with the largest $Pc$\n",
    "    * Output that as a prediction\n",
    "    * Discard any remaining boxes with $IOU>=0.5$ with the box output in the previous step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<img src=\"notebook_images/nms_algo.jpg\" style=\"width:25%;height:25%;\">\n",
    "<caption>Example of non max suppression algorithm, on input the aglorithm receive 4 overlapping bounding boxes, and the output returns only one</caption>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Non-max suppression uses the very important function called **\"Intersection over Union\"**, or IoU.\n",
    "<img src=\"notebook_images/iou.png\" style=\"width:500px;height:400;\">\n",
    "\n",
    "**Below is implementation of iou**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def iou(box1, box2):\n",
    "    \"\"\"Implement the intersection over union (IoU) between box1 and box2\n",
    "    \n",
    "    Arguments:\n",
    "    box1 -- first box, list object with coordinates (x1, y1, x2, y2)\n",
    "    box2 -- second box, list object with coordinates (x1, y1, x2, y2)\n",
    "    \"\"\"\n",
    "\n",
    "    # Calculate the (y1, x1, y2, x2) coordinates of the intersection of box1 and box2. Calculate its Area.\n",
    "    xi1 = max(box1[0],box2[0])\n",
    "    yi1 = max(box1[1],box2[1])\n",
    "    xi2 = min(box1[2], box2[2])\n",
    "    yi2 = min(box1[3], box2[3])\n",
    "    inter_area = abs(xi1-xi2) * abs(yi1-yi2)\n",
    "\n",
    "    # Calculate the Union area by using Formula: Union(A,B) = A + B - Inter(A,B)\n",
    "    box1_area = abs(box1[0]-box1[2]) * abs(box1[1]-box1[3])\n",
    "    box2_area = abs(box2[0]-box2[2]) * abs(box2[1]-box2[3])\n",
    "    union_area = box1_area + box2_area -inter_area\n",
    "    \n",
    "    # compute the IoU\n",
    "    iou = inter_area/union_area\n",
    "\n",
    "    return iou"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iou = 0.14285714285714285\n"
     ]
    }
   ],
   "source": [
    "box1 = (2, 1, 4, 3)\n",
    "box2 = (1, 2, 3, 4) \n",
    "print(\"iou = \" + str(iou(box1, box2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With computing iou, we co mposed the third part of non max supression algorithm, now, lets combine everything into one :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Wrapping up the filtering\n",
    "\n",
    "Let's calculate the whole algorithm : \n",
    "* Get Yolo CNN output\n",
    "* Discard all boxes with $Pc <= 0.6$  \n",
    "* While tehre are any remaining boxes : \n",
    "    * Pick the box with the largest $Pc$\n",
    "    * Output that as a prediction\n",
    "    * Discard any remaining boxes with $IOU>=0.5$ with the box output in the previous step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def yolo_non_max_suppression(scores, boxes, classes, max_boxes = 10, iou_threshold = 0.5):\n",
    "    \"\"\"\n",
    "    Applies Non-max suppression (NMS) to set of boxes\n",
    "    \n",
    "    Arguments:\n",
    "    scores -- tensor of shape (None,), output of yolo_filter_boxes()\n",
    "    boxes -- tensor of shape (None, 4), output of yolo_filter_boxes() that have been scaled to the image size (see later)\n",
    "    classes -- tensor of shape (None,), output of yolo_filter_boxes()\n",
    "    max_boxes -- integer, maximum number of predicted boxes you'd like\n",
    "    iou_threshold -- real value, \"intersection over union\" threshold used for NMS filtering\n",
    "    \n",
    "    Returns:\n",
    "    scores -- tensor of shape (, None), predicted score for each box\n",
    "    boxes -- tensor of shape (4, None), predicted box coordinates\n",
    "    classes -- tensor of shape (, None), predicted class for each box\n",
    "    \n",
    "    Note: The \"None\" dimension of the output tensors has obviously to be less than max_boxes. Note also that this\n",
    "    function will transpose the shapes of scores, boxes, classes. This is made for convenience.\n",
    "    \"\"\"\n",
    "    \n",
    "    max_boxes_tensor = K.variable(max_boxes, dtype='int32')     # tensor to be used in tf.image.non_max_suppression()\n",
    "    K.get_session().run(tf.variables_initializer([max_boxes_tensor])) # initialize variable max_boxes_tensor\n",
    "    \n",
    "    # Use tf.image.non_max_suppression() to get the list of indices corresponding to boxes you keep\n",
    "    nms_indices = tf.image.non_max_suppression(boxes=boxes, \n",
    "                                               scores=scores,\n",
    "                                               max_output_size=max_boxes, \n",
    "                                               iou_threshold=iou_threshold, name='nms_indices')\n",
    "    \n",
    "    scores = K.gather(scores, nms_indices)\n",
    "    boxes = K.gather(boxes, nms_indices)\n",
    "    classes = K.gather(classes, nms_indices)\n",
    "    \n",
    "    return scores, boxes, classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def yolo_eval(yolo_outputs,\n",
    "              anchors,\n",
    "              num_classes,\n",
    "              image_shape,\n",
    "              max_boxes=20,\n",
    "              score_threshold=.6,\n",
    "              iou_threshold=.5):\n",
    "    # 1. Evaluate YOLO model on given input and return filtered boxes.\n",
    "    # 2. Scale boxes back to original image shape.\n",
    "    # 3. Filter boxes\n",
    "    # 4. Apply non max suppression algorithm\n",
    "    \n",
    "    num_layers = len(yolo_outputs)\n",
    "    anchor_mask = [[6,7,8], [3,4,5], [0,1,2]] if num_layers==3 else [[3,4,5], [1,2,3]] # default setting\n",
    "    input_shape = K.shape(yolo_outputs[0])[1:3] * 32\n",
    "    boxes = []\n",
    "    box_scores = []\n",
    "    for l in range(num_layers):\n",
    "        _boxes, _box_scores = yolo_boxes_and_scores (yolo_outputs[l],\n",
    "            anchors[anchor_mask[l]], num_classes, input_shape, image_shape)\n",
    "        boxes.append(_boxes)\n",
    "        box_scores.append(_box_scores)\n",
    "    boxes = K.concatenate(boxes, axis=0)\n",
    "    box_scores = K.concatenate(box_scores, axis=0)\n",
    "\n",
    "    mask = box_scores >= score_threshold\n",
    "    max_boxes_tensor = K.constant(max_boxes, dtype='int32')\n",
    "    boxes_ = []\n",
    "    scores_ = []\n",
    "    classes_ = []\n",
    "    for c in range(num_classes):\n",
    "        # TODO: use keras backend instead of tf.\n",
    "        class_boxes = tf.boolean_mask(boxes, mask[:, c])\n",
    "        class_box_scores = tf.boolean_mask(box_scores[:, c], mask[:, c])\n",
    "        nms_index = tf.image.non_max_suppression(\n",
    "            class_boxes, class_box_scores, max_boxes_tensor, iou_threshold=iou_threshold)\n",
    "        class_boxes = K.gather(class_boxes, nms_index)\n",
    "        class_box_scores = K.gather(class_box_scores, nms_index)\n",
    "        classes = K.ones_like(class_box_scores, 'int32') * c\n",
    "        boxes_.append(class_boxes)\n",
    "        scores_.append(class_box_scores)\n",
    "        classes_.append(classes)\n",
    "    boxes_ = K.concatenate(boxes_, axis=0)\n",
    "    scores_ = K.concatenate(scores_, axis=0)\n",
    "    classes_ = K.concatenate(classes_, axis=0)\n",
    "\n",
    "\n",
    "    return boxes_, scores_, classes_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3 :  YOLO V3. Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the purpose of this project, I'm using a pretrained weights for Yolo V3.\n",
    "You can download the weights <a href = \"https://pjreddie.com/media/files/yolov3.weights\"> here</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = K.get_session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 - Defining classes, anchors and image shape."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have gathered the information about the 80 classes and 5 boxes in two files \"coco_classes.txt\" and \"yolo_anchors.txt\". Let's load these quantities into the model by running the next cell. \n",
    "\n",
    "The car detection dataset has 720x1280 images, which we've pre-processed into 608x608 images. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anchor Boxes\n",
    "\n",
    "Generating anchor boxes using K-means clustering\n",
    "\n",
    "There are many ways to compute bounding boxes for detection tasks. One approach is to directly predict the bounding box values, however this approach is susceptible to errors as it tends to favor bounding boxes with large dimensions. Further, the training process is unstable because the range of values to predict can vary significantly.From YOLOv2, the first step is to compute good candidate anchor boxes. This is achieved using K-means clustering. However, using direct Euler distance metric for K-means minizers error for larger bounding boxes, but not for smaller boxes. Therefore, from YOLOv2, intersection over union (IOU) is used as a distance metric. The IOU calculations are made assuming all the bounding boxes are located at one point, i.e. only width and height are used as features. Figure below shows the height and width plotted against each other. Fixed slopes indicate that most bounding boxes have specific predefined aspect ratios, and size. This is not surprising given the fact that a person and vehicle are expected to have certain fixed dimensions.\n",
    "\n",
    "\n",
    "<img src=\"notebook_images/clustering.png\" style=\"width:500px;height:400;\">\n",
    "\n",
    "\n",
    "In Yolo V2, this specialization is ‘assisted’ with predefined anchors as in Faster-RCNN. The predefined anchors are chosen to be as representative as possible of the ground truth boxes, with the following K-means clustering algorithm to define them:\n",
    "\n",
    "* all ground-truth bounding boxes are centered on (0,0)\n",
    "\n",
    "* the algorithm initiates 5 centroïds by drawing randomly 5 of the ground-truth bounding boxes\n",
    "\n",
    "* then, the following two steps are alternated:\n",
    "\n",
    "    * each ground truth box is assigned to one of the centroïd, using as distance measure the IOU, in order to get 5 clusters or groups of ground-truth bouding boxes\n",
    "\n",
    "    * new centroids are computed by taking the box inside each cluster that minimizes the mean IOU with all other boxes inside the cluster\n",
    "\n",
    "\n",
    "\n",
    "Yet another clarification : \n",
    "We make predictions on the offsets to the anchors. Nevertheless, if it is unconstrained, our guesses will be randomized again. YOLO predicts 5 parameters (tx, ty, tw, th, and to) and applies the sigma function to constraint its possible offset range.\n",
    "\n",
    "<img src=\"notebook_images/formula.png\" style=\"width:50%;height:50%;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading class/anchors data\n",
    "Bellow we are reading the class labels and pre-computed anchors points with K-means algorithm\n",
    "\n",
    "To determine the priors, YOLOv3 applies k-means cluster. Then it pre-select 9 clusters. For COCO, the width and height of the anchors are (10×13),(16×30),(33×23),(30×61),(62×45),(59× 119),(116 × 90),(156 × 198),(373 × 326). These 9 priors are grouped into 3 different groups according to their scale. Each group is assigned to a specific feature map above in detecting objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 10.  13.]\n",
      " [ 16.  30.]\n",
      " [ 33.  23.]\n",
      " [ 30.  61.]\n",
      " [ 62.  45.]\n",
      " [ 59. 119.]\n",
      " [116.  90.]\n",
      " [156. 198.]\n",
      " [373. 326.]]\n"
     ]
    }
   ],
   "source": [
    "anchors  = read_anchors('model_data/yolo_anchors.txt')\n",
    "class_names = read_classes(\"model_data/coco_classes.txt\")\n",
    "\n",
    "print(anchors)\n",
    "image_shape = (720., 1280.)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 - Loading a pretrained model\n",
    "\n",
    "For the purpose of this project, I'm using a pretrained weights for Yolo V3.\n",
    "You can download the weights <a href = \"http://pjreddie.com/media/files/yolo.weights\"> here</a>\n",
    "\n",
    "These weights are generated with using pure numpy arrays, but luckily the authors of Yolo provided configuration files containing the full model information, which means we can convert the weights in any high level program language.\n",
    "Allan Zelener wrote a function for weights conversion, which works well for Yolo V2, and I provided small modifications for Yolo V3 References are at the end of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Kiko-PC\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\keras\\models.py:255: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
      "  warnings.warn('No training configuration found in save file: '\n"
     ]
    }
   ],
   "source": [
    "yolo_model = load_model(\"yolo.h5\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This loads the weights of a trained YOLO model. Here's a summary of the layers your model contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            (None, None, None, 3 0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_1 (Conv2D)               (None, None, None, 3 864         input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, None, None, 3 128         conv2d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_1 (LeakyReLU)       (None, None, None, 3 0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "zero_padding2d_1 (ZeroPadding2D (None, None, None, 3 0           leaky_re_lu_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_2 (Conv2D)               (None, None, None, 6 18432       zero_padding2d_1[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, None, None, 6 256         conv2d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_2 (LeakyReLU)       (None, None, None, 6 0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_3 (Conv2D)               (None, None, None, 3 2048        leaky_re_lu_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, None, None, 3 128         conv2d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_3 (LeakyReLU)       (None, None, None, 3 0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_4 (Conv2D)               (None, None, None, 6 18432       leaky_re_lu_3[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, None, None, 6 256         conv2d_4[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_4 (LeakyReLU)       (None, None, None, 6 0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add_1 (Add)                     (None, None, None, 6 0           leaky_re_lu_2[0][0]              \n",
      "                                                                 leaky_re_lu_4[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "zero_padding2d_2 (ZeroPadding2D (None, None, None, 6 0           add_1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_5 (Conv2D)               (None, None, None, 1 73728       zero_padding2d_2[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, None, None, 1 512         conv2d_5[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_5 (LeakyReLU)       (None, None, None, 1 0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_6 (Conv2D)               (None, None, None, 6 8192        leaky_re_lu_5[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, None, None, 6 256         conv2d_6[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_6 (LeakyReLU)       (None, None, None, 6 0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_7 (Conv2D)               (None, None, None, 1 73728       leaky_re_lu_6[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_7 (BatchNor (None, None, None, 1 512         conv2d_7[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_7 (LeakyReLU)       (None, None, None, 1 0           batch_normalization_7[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add_2 (Add)                     (None, None, None, 1 0           leaky_re_lu_5[0][0]              \n",
      "                                                                 leaky_re_lu_7[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_8 (Conv2D)               (None, None, None, 6 8192        add_2[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_8 (BatchNor (None, None, None, 6 256         conv2d_8[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_8 (LeakyReLU)       (None, None, None, 6 0           batch_normalization_8[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_9 (Conv2D)               (None, None, None, 1 73728       leaky_re_lu_8[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, None, None, 1 512         conv2d_9[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_9 (LeakyReLU)       (None, None, None, 1 0           batch_normalization_9[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add_3 (Add)                     (None, None, None, 1 0           add_2[0][0]                      \n",
      "                                                                 leaky_re_lu_9[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "zero_padding2d_3 (ZeroPadding2D (None, None, None, 1 0           add_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_10 (Conv2D)              (None, None, None, 2 294912      zero_padding2d_3[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_10 (BatchNo (None, None, None, 2 1024        conv2d_10[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_10 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_10[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_11 (Conv2D)              (None, None, None, 1 32768       leaky_re_lu_10[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_11 (BatchNo (None, None, None, 1 512         conv2d_11[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_11 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_11[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_12 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_11[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_12 (BatchNo (None, None, None, 2 1024        conv2d_12[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_12 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_12[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_4 (Add)                     (None, None, None, 2 0           leaky_re_lu_10[0][0]             \n",
      "                                                                 leaky_re_lu_12[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_13 (Conv2D)              (None, None, None, 1 32768       add_4[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_13 (BatchNo (None, None, None, 1 512         conv2d_13[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_13 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_13[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_14 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_13[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_14 (BatchNo (None, None, None, 2 1024        conv2d_14[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_14 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_14[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_5 (Add)                     (None, None, None, 2 0           add_4[0][0]                      \n",
      "                                                                 leaky_re_lu_14[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_15 (Conv2D)              (None, None, None, 1 32768       add_5[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_15 (BatchNo (None, None, None, 1 512         conv2d_15[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_15 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_15[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_16 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_15[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_16 (BatchNo (None, None, None, 2 1024        conv2d_16[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_16 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_16[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_6 (Add)                     (None, None, None, 2 0           add_5[0][0]                      \n",
      "                                                                 leaky_re_lu_16[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_17 (Conv2D)              (None, None, None, 1 32768       add_6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_17 (BatchNo (None, None, None, 1 512         conv2d_17[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_17 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_17[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_18 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_17[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_18 (BatchNo (None, None, None, 2 1024        conv2d_18[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_18 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_18[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_7 (Add)                     (None, None, None, 2 0           add_6[0][0]                      \n",
      "                                                                 leaky_re_lu_18[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_19 (Conv2D)              (None, None, None, 1 32768       add_7[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_19 (BatchNo (None, None, None, 1 512         conv2d_19[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_19 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_19[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_20 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_19[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_20 (BatchNo (None, None, None, 2 1024        conv2d_20[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_20 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_20[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_8 (Add)                     (None, None, None, 2 0           add_7[0][0]                      \n",
      "                                                                 leaky_re_lu_20[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_21 (Conv2D)              (None, None, None, 1 32768       add_8[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_21 (BatchNo (None, None, None, 1 512         conv2d_21[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_21 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_21[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_22 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_21[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_22 (BatchNo (None, None, None, 2 1024        conv2d_22[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_22 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_22[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_9 (Add)                     (None, None, None, 2 0           add_8[0][0]                      \n",
      "                                                                 leaky_re_lu_22[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_23 (Conv2D)              (None, None, None, 1 32768       add_9[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_23 (BatchNo (None, None, None, 1 512         conv2d_23[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_23 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_23[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_24 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_23[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_24 (BatchNo (None, None, None, 2 1024        conv2d_24[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_24 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_24[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_10 (Add)                    (None, None, None, 2 0           add_9[0][0]                      \n",
      "                                                                 leaky_re_lu_24[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_25 (Conv2D)              (None, None, None, 1 32768       add_10[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_25 (BatchNo (None, None, None, 1 512         conv2d_25[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_25 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_25[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_26 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_25[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_26 (BatchNo (None, None, None, 2 1024        conv2d_26[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_26 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_26[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_11 (Add)                    (None, None, None, 2 0           add_10[0][0]                     \n",
      "                                                                 leaky_re_lu_26[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "zero_padding2d_4 (ZeroPadding2D (None, None, None, 2 0           add_11[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_27 (Conv2D)              (None, None, None, 5 1179648     zero_padding2d_4[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_27 (BatchNo (None, None, None, 5 2048        conv2d_27[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_27 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_27[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_28 (Conv2D)              (None, None, None, 2 131072      leaky_re_lu_27[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_28 (BatchNo (None, None, None, 2 1024        conv2d_28[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_28 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_28[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_29 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_28[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_29 (BatchNo (None, None, None, 5 2048        conv2d_29[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_29 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_29[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_12 (Add)                    (None, None, None, 5 0           leaky_re_lu_27[0][0]             \n",
      "                                                                 leaky_re_lu_29[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_30 (Conv2D)              (None, None, None, 2 131072      add_12[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_30 (BatchNo (None, None, None, 2 1024        conv2d_30[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_30 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_30[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_31 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_30[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_31 (BatchNo (None, None, None, 5 2048        conv2d_31[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_31 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_31[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_13 (Add)                    (None, None, None, 5 0           add_12[0][0]                     \n",
      "                                                                 leaky_re_lu_31[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_32 (Conv2D)              (None, None, None, 2 131072      add_13[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_32 (BatchNo (None, None, None, 2 1024        conv2d_32[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_32 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_32[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_33 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_32[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_33 (BatchNo (None, None, None, 5 2048        conv2d_33[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_33 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_33[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_14 (Add)                    (None, None, None, 5 0           add_13[0][0]                     \n",
      "                                                                 leaky_re_lu_33[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_34 (Conv2D)              (None, None, None, 2 131072      add_14[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_34 (BatchNo (None, None, None, 2 1024        conv2d_34[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_34 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_34[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_35 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_34[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_35 (BatchNo (None, None, None, 5 2048        conv2d_35[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_35 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_35[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_15 (Add)                    (None, None, None, 5 0           add_14[0][0]                     \n",
      "                                                                 leaky_re_lu_35[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_36 (Conv2D)              (None, None, None, 2 131072      add_15[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_36 (BatchNo (None, None, None, 2 1024        conv2d_36[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_36 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_36[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_37 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_36[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_37 (BatchNo (None, None, None, 5 2048        conv2d_37[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_37 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_37[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_16 (Add)                    (None, None, None, 5 0           add_15[0][0]                     \n",
      "                                                                 leaky_re_lu_37[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_38 (Conv2D)              (None, None, None, 2 131072      add_16[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_38 (BatchNo (None, None, None, 2 1024        conv2d_38[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_38 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_38[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_39 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_38[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_39 (BatchNo (None, None, None, 5 2048        conv2d_39[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_39 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_39[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_17 (Add)                    (None, None, None, 5 0           add_16[0][0]                     \n",
      "                                                                 leaky_re_lu_39[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_40 (Conv2D)              (None, None, None, 2 131072      add_17[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_40 (BatchNo (None, None, None, 2 1024        conv2d_40[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_40 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_40[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_41 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_40[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_41 (BatchNo (None, None, None, 5 2048        conv2d_41[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_41 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_41[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_18 (Add)                    (None, None, None, 5 0           add_17[0][0]                     \n",
      "                                                                 leaky_re_lu_41[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_42 (Conv2D)              (None, None, None, 2 131072      add_18[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_42 (BatchNo (None, None, None, 2 1024        conv2d_42[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_42 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_42[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_43 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_42[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_43 (BatchNo (None, None, None, 5 2048        conv2d_43[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_43 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_43[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_19 (Add)                    (None, None, None, 5 0           add_18[0][0]                     \n",
      "                                                                 leaky_re_lu_43[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "zero_padding2d_5 (ZeroPadding2D (None, None, None, 5 0           add_19[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_44 (Conv2D)              (None, None, None, 1 4718592     zero_padding2d_5[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_44 (BatchNo (None, None, None, 1 4096        conv2d_44[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_44 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_44[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_45 (Conv2D)              (None, None, None, 5 524288      leaky_re_lu_44[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_45 (BatchNo (None, None, None, 5 2048        conv2d_45[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_45 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_45[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_46 (Conv2D)              (None, None, None, 1 4718592     leaky_re_lu_45[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_46 (BatchNo (None, None, None, 1 4096        conv2d_46[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_46 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_46[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_20 (Add)                    (None, None, None, 1 0           leaky_re_lu_44[0][0]             \n",
      "                                                                 leaky_re_lu_46[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_47 (Conv2D)              (None, None, None, 5 524288      add_20[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_47 (BatchNo (None, None, None, 5 2048        conv2d_47[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_47 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_47[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_48 (Conv2D)              (None, None, None, 1 4718592     leaky_re_lu_47[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_48 (BatchNo (None, None, None, 1 4096        conv2d_48[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_48 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_48[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_21 (Add)                    (None, None, None, 1 0           add_20[0][0]                     \n",
      "                                                                 leaky_re_lu_48[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_49 (Conv2D)              (None, None, None, 5 524288      add_21[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_49 (BatchNo (None, None, None, 5 2048        conv2d_49[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_49 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_49[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_50 (Conv2D)              (None, None, None, 1 4718592     leaky_re_lu_49[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_50 (BatchNo (None, None, None, 1 4096        conv2d_50[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_50 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_50[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_22 (Add)                    (None, None, None, 1 0           add_21[0][0]                     \n",
      "                                                                 leaky_re_lu_50[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_51 (Conv2D)              (None, None, None, 5 524288      add_22[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_51 (BatchNo (None, None, None, 5 2048        conv2d_51[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_51 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_51[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_52 (Conv2D)              (None, None, None, 1 4718592     leaky_re_lu_51[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_52 (BatchNo (None, None, None, 1 4096        conv2d_52[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_52 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_52[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_23 (Add)                    (None, None, None, 1 0           add_22[0][0]                     \n",
      "                                                                 leaky_re_lu_52[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_53 (Conv2D)              (None, None, None, 5 524288      add_23[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_53 (BatchNo (None, None, None, 5 2048        conv2d_53[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_53 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_53[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_54 (Conv2D)              (None, None, None, 1 4718592     leaky_re_lu_53[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_54 (BatchNo (None, None, None, 1 4096        conv2d_54[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_54 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_54[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_55 (Conv2D)              (None, None, None, 5 524288      leaky_re_lu_54[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_55 (BatchNo (None, None, None, 5 2048        conv2d_55[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_55 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_55[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_56 (Conv2D)              (None, None, None, 1 4718592     leaky_re_lu_55[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_56 (BatchNo (None, None, None, 1 4096        conv2d_56[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_56 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_56[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_57 (Conv2D)              (None, None, None, 5 524288      leaky_re_lu_56[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_57 (BatchNo (None, None, None, 5 2048        conv2d_57[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_57 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_57[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_60 (Conv2D)              (None, None, None, 2 131072      leaky_re_lu_57[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_59 (BatchNo (None, None, None, 2 1024        conv2d_60[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_59 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_59[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "up_sampling2d_1 (UpSampling2D)  (None, None, None, 2 0           leaky_re_lu_59[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, None, None, 7 0           up_sampling2d_1[0][0]            \n",
      "                                                                 add_19[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_61 (Conv2D)              (None, None, None, 2 196608      concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_60 (BatchNo (None, None, None, 2 1024        conv2d_61[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_60 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_60[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_62 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_60[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_61 (BatchNo (None, None, None, 5 2048        conv2d_62[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_61 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_61[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_63 (Conv2D)              (None, None, None, 2 131072      leaky_re_lu_61[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_62 (BatchNo (None, None, None, 2 1024        conv2d_63[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_62 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_62[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_64 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_62[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_63 (BatchNo (None, None, None, 5 2048        conv2d_64[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_63 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_63[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_65 (Conv2D)              (None, None, None, 2 131072      leaky_re_lu_63[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_64 (BatchNo (None, None, None, 2 1024        conv2d_65[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_64 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_64[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_68 (Conv2D)              (None, None, None, 1 32768       leaky_re_lu_64[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_66 (BatchNo (None, None, None, 1 512         conv2d_68[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_66 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_66[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "up_sampling2d_2 (UpSampling2D)  (None, None, None, 1 0           leaky_re_lu_66[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_2 (Concatenate)     (None, None, None, 3 0           up_sampling2d_2[0][0]            \n",
      "                                                                 add_11[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_69 (Conv2D)              (None, None, None, 1 49152       concatenate_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_67 (BatchNo (None, None, None, 1 512         conv2d_69[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_67 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_67[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_70 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_67[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_68 (BatchNo (None, None, None, 2 1024        conv2d_70[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_68 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_68[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_71 (Conv2D)              (None, None, None, 1 32768       leaky_re_lu_68[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_69 (BatchNo (None, None, None, 1 512         conv2d_71[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_69 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_69[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_72 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_69[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_70 (BatchNo (None, None, None, 2 1024        conv2d_72[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_70 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_70[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_73 (Conv2D)              (None, None, None, 1 32768       leaky_re_lu_70[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_71 (BatchNo (None, None, None, 1 512         conv2d_73[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_71 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_71[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_58 (Conv2D)              (None, None, None, 1 4718592     leaky_re_lu_57[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_66 (Conv2D)              (None, None, None, 5 1179648     leaky_re_lu_64[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_74 (Conv2D)              (None, None, None, 2 294912      leaky_re_lu_71[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_58 (BatchNo (None, None, None, 1 4096        conv2d_58[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_65 (BatchNo (None, None, None, 5 2048        conv2d_66[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_72 (BatchNo (None, None, None, 2 1024        conv2d_74[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_58 (LeakyReLU)      (None, None, None, 1 0           batch_normalization_58[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_65 (LeakyReLU)      (None, None, None, 5 0           batch_normalization_65[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_72 (LeakyReLU)      (None, None, None, 2 0           batch_normalization_72[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_59 (Conv2D)              (None, None, None, 2 261375      leaky_re_lu_58[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_67 (Conv2D)              (None, None, None, 2 130815      leaky_re_lu_65[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_75 (Conv2D)              (None, None, None, 2 65535       leaky_re_lu_72[0][0]             \n",
      "==================================================================================================\n",
      "Total params: 62,001,757\n",
      "Trainable params: 61,949,149\n",
      "Non-trainable params: 52,608\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "yolo_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Before we go further, I’d like to point out that stride of the network, or a layer is defined as the ratio by which it downsamples the input. In the following examples, I will assume we have an input image of size 416 x 416.\n",
    "\n",
    "YOLO v3 makes prediction at three scales, which are precisely given by downsampling the dimensions of the input image by 32, 16 and 8 respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 - Convert output of the model to usable bounding box tensors\n",
    "\n",
    "yolo_outputs gave you all the predicted boxes of yolo_model in the correct format. You're now ready to perform filtering and select only the best boxes. Lets now call yolo_eval, which you had previously implemented, to do this.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"concat_11:0\", shape=(?, 4), dtype=float32)\n",
      "Tensor(\"concat_12:0\", shape=(?,), dtype=float32)\n",
      "Tensor(\"concat_13:0\", shape=(?,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "from PIL import Image, ImageFont, ImageDraw\n",
    "import os\n",
    "\n",
    "\n",
    "image = Image.open('test/straight_lines1.jpg')\n",
    "image_data = preprocess_image(image)\n",
    "\n",
    "input_image_shape = K.placeholder(shape=(2, ))\n",
    "\n",
    "score = 0.6\n",
    "iou = 0.5\n",
    "\n",
    "boxes, scores, classes = yolo_eval(yolo_model.output, anchors,\n",
    "                len(class_names), input_image_shape,\n",
    "                score_threshold=score, iou_threshold=0.45)\n",
    "\n",
    "print(boxes) \n",
    "print(scores)\n",
    "print(classes)\n",
    "\n",
    "out_boxes, out_scores, out_classes = sess.run(\n",
    "    [boxes, scores, classes],\n",
    "    feed_dict={\n",
    "        yolo_model.input: image_data,\n",
    "        input_image_shape: [image.size[1], image.size[0]],\n",
    "        K.learning_phase(): 0\n",
    "    })\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You added `yolo_outputs` to your graph. This set of 4 tensors is ready to be used as input by your `yolo_eval` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 - Run the graph on an image\n",
    "\n",
    "Let the fun begin. You have created a (`sess`) graph that can be summarized as follows:\n",
    "\n",
    "1. <font color='purple'> yolo_model.input </font> is given to `yolo_model`. The model is used to compute the output <font color='purple'> yolo_model.output </font>\n",
    "2. <font color='purple'> yolo_model.output </font> is processed by `yolo_head`. It gives you <font color='purple'> yolo_outputs </font>\n",
    "3. <font color='purple'> yolo_outputs </font> goes through a filtering function, `yolo_eval`. It outputs your predictions: <font color='purple'> scores, boxes, classes </font>\n",
    "\n",
    "**Exercise**: Implement predict() which runs the graph to test YOLO on an image.\n",
    "You will need to run a TensorFlow session, to have it compute `scores, boxes, classes`.\n",
    "\n",
    "The code below also uses the following function:\n",
    "```python\n",
    "image, image_data = preprocess_image(\"images/\" + image_file, model_image_size = (608, 608))\n",
    "```\n",
    "which outputs:\n",
    "- image: a python (PIL) representation of your image used for drawing boxes. You won't need to use it.\n",
    "- image_data: a numpy-array representing the image. This will be the input to the CNN.\n",
    "\n",
    "**Important note**: when a model uses BatchNorm (as is the case in YOLO), you will need to pass an additional placeholder in the feed_dict {K.learning_phase(): 0}."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def predict(sess, image):\n",
    "    \"\"\"\n",
    "    Runs the graph stored in \"sess\" to predict boxes for \"image_file\". Prints and plots the preditions.\n",
    "    \n",
    "    Arguments:\n",
    "    sess -- your tensorflow/Keras session containing the YOLO graph\n",
    "    image_file -- name of an image stored in the \"images\" folder.\n",
    "    \n",
    "    Returns:\n",
    "    out_scores -- tensor of shape (None, ), scores of the predicted boxes\n",
    "    out_boxes -- tensor of shape (None, 4), coordinates of the predicted boxes\n",
    "    out_classes -- tensor of shape (None, ), class index of the predicted boxes\n",
    "    \n",
    "    Note: \"None\" actually represents the number of predicted boxes, it varies between 0 and max_boxes. \n",
    "    \"\"\"\n",
    "    # Preprocess your image\n",
    "    image = Image.fromarray(image.astype('uint8'), 'RGB')\n",
    "\n",
    "    image_data = preprocess_image(image, model_image_size = (416, 416))\n",
    "\n",
    "    out_boxes, out_scores, out_classes = sess.run(\n",
    "        [boxes, scores, classes],\n",
    "        feed_dict={\n",
    "            yolo_model.input: image_data,\n",
    "            input_image_shape: [image.size[1], image.size[0]],\n",
    "            K.learning_phase(): 0\n",
    "        })\n",
    "    ### END CODE HERE ###\n",
    "\n",
    "    # Generate colors for drawing bounding boxes.\n",
    "    colors = generate_colors(class_names)\n",
    "    # Draw bounding boxes on the image file\n",
    "    drawer=draw_boxes(image, out_scores, out_boxes, out_classes, class_names, colors)\n",
    "  \n",
    "\n",
    "    return out_scores, out_boxes, out_classes, drawer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell on the \"test.jpg\" image to verify that your function is correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def pipeline_final(img):\n",
    "\n",
    "    start_cars = time.time()\n",
    "    out_scores, out_boxes, out_classes, image = predict(sess, img)\n",
    "    end_cars = time.time()\n",
    "    \n",
    "    start_lanes = time.time()\n",
    "    initial_lane,left_fit, right_fit, left_fitx, right_fitx, ploty, filtered_image, warped, deep_search, out_img = pipeline_lanes(np.copy(img))\n",
    "    result=cv2.addWeighted(image, 1, initial_lane, 0.3, 1)\n",
    "    \n",
    "    \n",
    "    result  = add_diagnostic_image(result, filtered_image, 0)\n",
    "    result  = add_diagnostic_image(result, warped, 1)\n",
    "    result  = add_diagnostic_image(result, deep_search, 2)\n",
    "    result  = add_diagnostic_image(result, out_img, 3)\n",
    "    \n",
    "    end_lanes = time.time()\n",
    "    fps_lanes = 1/(end_lanes-start_lanes)\n",
    "    fps_cars = 1/(end_cars - start_cars)\n",
    "    fps_text_cars = \"FPS cars : %.2f\" % fps_cars\n",
    "    fps_text_lanes = \"FPS lanes : %.2f\" % fps_lanes\n",
    "\n",
    "    result  = add_diagnostic_text(result, fps_text_lanes, 4, offset=50)\n",
    "    result  = add_diagnostic_text(result, fps_text_cars, 4, offset=90)\n",
    "\n",
    "\n",
    "    return result\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pipeline_lanes(img):\n",
    "\n",
    "    global caches \n",
    "    \n",
    "    is_blind = False\n",
    "    left_fit, right_fit = None, None\n",
    "\n",
    "    if caches:\n",
    "        left_fit, right_fit, is_blind  = caches\n",
    "        \n",
    "    filtered_image = filter_image(img, is_blind)\n",
    "       \n",
    "    warped, filled = perspective_transform_with_filled_area(img, filtered_image)\n",
    "    \n",
    "    #The line below is added only for the first initial frame and for diagnostic purposes \n",
    "    ploty, left_fitx, right_fitx, left_fit, right_fit, deep_search = get_lane_rectangles(warped, left_fit, right_fit, is_blind)\n",
    "    if not caches : \n",
    "        caches = [left_fit, right_fit, is_blind]\n",
    "\n",
    "    if caches : \n",
    "        left_fit, right_fit, is_blind  = caches\n",
    "        out_img, ploty, left_fitx, right_fitx, left_fit, right_fit, is_blind = get_next_frame_lines(warped, left_fit, right_fit, is_blind)\n",
    "        caches = [left_fit, right_fit, is_blind]\n",
    "\n",
    "            \n",
    "\n",
    "    result = inverse_perspective_transform(img, warped, left_fitx, right_fitx, ploty)\n",
    "    \n",
    "\n",
    "    return result, left_fit, right_fit, left_fitx, right_fitx, ploty, filtered_image, warped, deep_search,out_img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[MoviePy] >>>> Building video custom_output.mp4\n",
      "[MoviePy] Writing video custom_output.mp4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  1%|▏         | 11/744 [00:01<01:57,  6.22it/s]"
     ]
    }
   ],
   "source": [
    "from moviepy.editor import VideoFileClip\n",
    "\n",
    "caches=[]\n",
    "output = 'custom_output.mp4'\n",
    "clip1 = VideoFileClip(\"project_video.mp4\")\n",
    "white_clip = clip1.fl_image(pipeline_final) \n",
    "%time white_clip.write_videofile(output, audio=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References**: \n",
    "- Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi - [You Only Look Once: Unified, Real-Time Object Detection](https://arxiv.org/abs/1506.02640) (2015)\n",
    "- Joseph Redmon, Ali Farhadi - [YOLO9000: Better, Faster, Stronger](https://arxiv.org/abs/1612.08242) (2016)\n",
    "- Allan Zelener - [YAD2K: Yet Another Darknet 2 Keras](https://github.com/allanzelener/YAD2K)\n",
    "- The official YOLO website (https://pjreddie.com/darknet/yolo/) \n",
    "- Vivek Yadav [<a href=\"https://medium.com/@vivek.yadav/part-1-generating-anchor-boxes-for-yolo-like-network-for-vehicle-detection-using-kitti-dataset-b2fe033e5807\">Generating Anchor boxes for Yolo-like network for vehicle detection using KITTI dataset]</a>\n",
    "- Christopher Bourez's blog [<a href=\"http://christopher5106.github.io/object/detectors/2017/08/10/bounding-box-object-detectors-understanding-yolo.html\">Bounding box object detectors: understanding YOLO, You Look Only Once</a>]\n",
    "- Andrew NG Coursera : [<a href=\"https://www.coursera.org/learn/convolutional-neural-networks\">Convolutional Neural Networks\n",
    "</a>]\n",
    "- Jonathan Hui : [<a href=\"https://medium.com/@jonathan_hui/real-time-object-detection-with-yolo-yolov2-28b1b93e2088\">Real-time Object Detection with YOLO, YOLOv2 and now YOLOv3\n",
    "</a>]\n"
   ]
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "convolutional-neural-networks",
   "graded_item_id": "OMdut",
   "launcher_item_id": "bbBOL"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}