self_supervised_learning_for_computer_vision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Levan Tsinadze"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Head of AI - MaxinAI\n",
    "<br>\n",
    "\n",
    "Head of School of AI / MaxinAI Education\n",
    "<br>\n",
    "\n",
    "Machine learning researcher - \n",
    "eyeo GmbH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We will focus on contrastive learning not on generative models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a href=\"https://www.youtube.com/watch?v=0KeR6i1_56g&feature=youtu.be\">Week 10 – Lecture: Self-supervised learning (SSL) in computer vision (CV)</a> from  NYU CENTER FOR DATA SCIENCE\n",
    "<br>\n",
    "\n",
    "<a href=\"https://www.youtube.com/watch?v=dMUes74-nYY\">Lecture 7 Self-Supervised Learning -- UC Berkeley Spring 2020 - CS294-158 Deep Unsupervised Learning</a> from UC Berkeley"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Self-supervised learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Self supervised learning is ML learning when labels are simple generated by the modification of the original data, without human-in-the loop:\n",
    "- Generative models:\n",
    " - Auto-encoders when labels are the original images\n",
    "- Discriminative models:\n",
    " - When labels are the meta information about original data modification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Labeling data is expensive:\n",
    "- Labeling images for classification\n",
    "- Tend to be biased and has an errors\n",
    "- ImageNet is one of the\n",
    "- Labeling images for detection and segmentation is at least twice expensive and biased"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Pretext tasks and downstream tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Pretext tasks task on which model is trained for better representation:\n",
    "- Auto encoders\n",
    "- Context encoders\n",
    "- One view from another\n",
    "- Contrastive learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Downstream task on which features learned from downstream task is used as input or fine-tuned entire model:\n",
    "- Classification\n",
    "- Localization\n",
    "- Backbone for object detection or semantic segmentation models\n",
    "- Feature extractor for image retrieval, image captioning etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ImageNet vs custom data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ImageNet - over the one million labeled images\n",
    "- Size range and three channels\n",
    "- Object are in central area\n",
    "- Limited brightness, contours, etc (augmentation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Our custom dataset might be completely different:\n",
    "- Grayscale images (X-ray, old images, etc)\n",
    "- Multi-channel images (MRI with eight channels, etc)\n",
    "- Objects located in different corners\n",
    "- Satellite images\n",
    "- Images from drones\n",
    "- Images from microscope\n",
    "- etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Center crop\n",
    "<img src=\"images/ssl/mic_1.jpeg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/mic_2.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/sat_1.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/sat_2.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/xr_1.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/xr_2.jpeg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Pluses of self-supervised learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Labeling is expensive and time-consuming, mistakes are almost inevitable.\n",
    "- Labeled data for classification contains small amount of information in comparison with images which has other different objects\n",
    "- Multi-task learning adds more difficulties and hyper-parameters\n",
    "- Multi object classifiers need more labeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ImageNet pre-trained models perform pretty well on even different domains, but if we have an alternative:\n",
    "- Pre-trained images for the similar domain\n",
    "- Train images on custom domain \n",
    "- and put it in public\n",
    "- Different model architectures (All of them are trained on ImageNet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For deep learning:\n",
    "<img src=\"images/ssl/dl_cacke_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Autoecoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Auto-encoders learn from the original images, they encode and reconstruct data\n",
    "<img src=\"images/ssl/ae_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Denoising auto-encoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we add augmentation and make the model extract features from images:\n",
    "<img src=\"images/ssl/dae_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Context encoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Deepak Pathak Philipp Krahenbuhl Jeff Donahue Trevor Darrell Alexei A. Efros\n",
    "\n",
    "Context Encoders: Feature Learning by Inpainting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/ce_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/ce_4.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/ce_5.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/ce_6.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Colorization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "DeOldify\n",
    "<img src=\"images/ssl/pv_10.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Relative positional image patches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Carl Doersch1, Abhinav Gupta, Alexei A. Efros\n",
    "\n",
    "Unsupervised Visual Representation Learning by Context Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/rp_3.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Solving jigsaw puzzles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Mehdi Noroozi, Paolo Favaro\n",
    "\n",
    "Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/jp_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/jp_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Rotation classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Spyros Gidaris, Praveer Singh, Nikos Komodakis\n",
    "\n",
    "UNSUPERVISED REPRESENTATION LEARNING BY PREDICTING IMAGE ROTATIONS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Learn many features to understand position\n",
    "<img src=\"images/ssl/rc_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/rc_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/rc_3.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/rc_4.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/rc_5.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/rc_6.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Instance discrimination"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/id_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/id_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Softmax on pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Softmax:\n",
    "$$\n",
    "q(x) = \\frac{exp(f(x_p))}{\\sum_{x \\in \\mathcal{X}}{exp(f(x))}}\n",
    "$$\n",
    "where $x_p$ is our positive example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Softmax on pairs: $f(x_p) = f(x_a, x_p)$ and $f(x) = f(x_p, x)$ all the other examples:\n",
    "$$\n",
    "H(p, q) = \\frac{exp(f(x_a, x_p))}{\\sum_{x \\in X}{exp(f(x_a, x))}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of all the negatives $\\sum_{x \\in X}$ we can take \"big enough\" amount of negative samples $X_0 \\subset X$ and $x_a, x_p \\in X_0$:\n",
    "$$\n",
    "H(p, q) = \\frac{exp(f(x_a, x_p))}{\\sum_{x \\in X_0}{exp(f(x_a, x))}}\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "f(x_a, x) = m(x_a)^T \\cdot m(x) = z_p^T \\cdot z\n",
    "$$\n",
    "or\n",
    "$$\n",
    "exp(f(x_a, x)) = e^{m(x_a)^T \\cdot m(x)} = e^{z_p^T \\cdot z}\n",
    "$$\n",
    "<br>\n",
    "\n",
    "or\n",
    "$\\{x_a, x_p\\}$ vs $\\{x_a\\} \\times X = \\{(x_a,x)|x \\in X\\}$\n",
    "<br>\n",
    "\n",
    "Or from some \"big enough\" subset of $X_0 \\subset X$\n",
    "<br>\n",
    "\n",
    "$\\{x_a, x_p\\}$ vs $\\{x_a\\} \\times X_0 = \\{(x_a,x)|x \\in X_0\\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Even with \"big enough\" samples it is hard to train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Run fast forward for anchor, positive and \"big enough\" negatives\n",
    "- Calculate loss\n",
    "- Backpropagate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Pre-compute negative samples:\n",
    "- Save vectors\n",
    "- use them as negative samples\n",
    "\n",
    "But after weights change negative vectors should be different\n",
    "<br>\n",
    "\n",
    "Make them as close as possible with features extracted by the current model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Momentum contrast (MoCo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross Girshick\n",
    "\n",
    "Momentum Contrast for Unsupervised Visual Representation Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/moco1_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/moco1_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Initialize model $f_q$ and make a copy - $f_k$, and fixed size queue $L$\n",
    "<br>\n",
    "than take image $x$ and two augmentation functions $a_q$ and $a_k$ (dynamically for each iteration)\n",
    "<br>\n",
    "Augment the source image:\n",
    "$$\n",
    "x_q = a_q(x)\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "x_k = a_k(x)\n",
    "$$\n",
    "Calculate representation vectors\n",
    "$$\n",
    "q = f_q(x_q)\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "k = f_k(x_k)\n",
    "$$\n",
    "<br>\n",
    "Calculate the cost:\n",
    "$$\n",
    "c = \\frac{f(q, k)}{\\sum_{l \\in L}f(q, l)}\n",
    "$$\n",
    "<br>\n",
    "Backpropagate only $f_q$ (so the weights $W_q$ are updated)\n",
    "<br>\n",
    "Update $f_k$ models weighs $W_k$ with exponential moving average:\n",
    "$$\n",
    "W_k = \\alpha \\cdot W_k + (1 - \\alpha) \\cdot W_q\n",
    "$$\n",
    "<br>\n",
    "and save representation $k$ in the queue $L = (l_1, l_2, \\dots, l_n)$ (remove oldest and thus less influenced by the $f_q$ models weighs vectors)\n",
    "$$\n",
    "k \\to l_1, \\\\\n",
    "l_1 \\to l_2 \\\\\n",
    "\\cdots \\\\\n",
    "l_{n-1} \\to l_n \\\\\n",
    "$$\n",
    "and $l_n$ goes out\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So in queue\n",
    "$$\n",
    "L = (l_1, l_2, \\dots, l_n)\n",
    "$$\n",
    "are representation vectors generated by the $f_k$\n",
    "<br>\n",
    "\n",
    "Lower the index $i$ is, $l_i$ is updated with $f_k = f(x, W_k)$ with closer weights with $f_q = f_q(x, W_q)$ which was updated by the backpropagation\n",
    "<br>\n",
    "\n",
    "The less influenced vectors from the last updated $f_q = f_q(x, W_q)$ is $l_n$ which is removed and queue is updated after each iteration\n",
    "<br>\n",
    "\n",
    "In this manner algorithm maintains the \"better\" representation vectors for negative sampling\n",
    "<br>\n",
    "\n",
    "In my opinion this works plus because $k = f_k(x_k)$ in positive sampling is also calculated with $f_k = f_k(x, W_k)$ which weights was updated in previous iteration with exponentially moving (closest) average of $f_q = f_q(x, W_q)$ model\n",
    "<br>\n",
    "\n",
    "Closest to backpropagated weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now translate everything in batches:\n",
    "<br>\n",
    "Just consider each call as batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/moco1_3.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/moco1_4.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/moco1_5.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/moco1_6.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SimCLR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton\n",
    "\n",
    "A Simple Framework for Contrastive Learning of Visual Representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/simclr_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/simclr_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/simclr_3.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/simclr_4.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MoCo2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Xinlei Chen, Haoqi Fan, Ross Girshick, Kaiming He\n",
    "\n",
    "Improved Baselines with Momentum Contrastive Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/moco2_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/moco2_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ssl/moco2_3.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Worth to mention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Using pre-trained backbone ResNet50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We trained Faster-RCNN with ResNeXt101 backbone pre-trained on ImageNet\n",
    "We used MoCov2 pre-trained ResNet50 for Faster-RCNN on our tasks on the 50K screenshots and 150k screenshots\n",
    "<br>\n",
    "\n",
    "Faster-RCNN with ResNeXt101\n",
    "- Bigger backbone\n",
    "- More layers for feature pyramid (four layers)\n",
    "- Higher dimensional features\n",
    "- Wider model (32 parallel convolutional block in each residual block)\n",
    "- Deeper model (101 layers)\n",
    "<br>\n",
    "\n",
    "Faster-RCNN with ResNet50\n",
    "- Smaller models\n",
    "- Less layers for feature pyramid (one layer)\n",
    "- Lower dimensional features\n",
    "- Narrower model (one convolutional block in each residual connection) \n",
    "- Shallower model (50 layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <th>Model</th>\n",
    "        <th>Data</th>\n",
    "        <th>TP</th>\n",
    "        <th>FN</th>\n",
    "        <th>FP</th>\n",
    "        <th>Recall</th>\n",
    "        <th>Precision</th>\n",
    "        <th>Confidence threshold</th>\n",
    "        <th>IoU threshold</th>\n",
    "        <th>mAP</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>Faster-RCNN RX101 ImageNet</td>\n",
    "        <td>50k</td>\n",
    "        <td>4619</td>\n",
    "        <td>4193</td>\n",
    "        <td>1534</td>\n",
    "        <td>52.41</td>\n",
    "        <td>75.06</td>\n",
    "        <td>0.5</td>\n",
    "        <td>0.4</td>\n",
    "        <td>63.62</td>\n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>Faster-RCNN R50 MoCov2</td>\n",
    "        <td>50k</td>\n",
    "        <td>4921</td>\n",
    "        <td>2126</td>\n",
    "        <td>2126</td>\n",
    "        <td>55.84</td>\n",
    "        <td>69.83</td>\n",
    "        <td>0.5</td>\n",
    "        <td>0.4</td>\n",
    "        <td>62.05</td>\n",
    "    <tr>\n",
    "        <td>Faster-RCNN RX101 ImageNet</td>\n",
    "        <td>50k</td>\n",
    "        <td>3882</td>\n",
    "        <td>4930</td>\n",
    "        <td>1022</td>\n",
    "        <td>44.05</td>\n",
    "        <td>79.15</td>\n",
    "        <td>0.98</td>\n",
    "        <td>0.4</td>\n",
    "        <td>61.46</td>\n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>Faster-RCNN R50 MoCov2</td>\n",
    "        <td>50k</td>\n",
    "        <td>2683</td>\n",
    "        <td>6129</td>\n",
    "        <td>369</td>\n",
    "        <td>30.45</td>\n",
    "        <td>87.91</td>\n",
    "        <td>0.98</td>\n",
    "        <td>0.4</td>\n",
    "        <td>45.23</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Conclusion\n",
    "\n",
    "- Performs less accurate on true-positives\n",
    "- Performs better on false-negatives\n",
    "- 0.5 confidence threshold: higher recall lower precision\n",
    "- 0.98 confidence threshold lower recall higher precision\n",
    "\n",
    "Better fits for our demands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <th>Model</th>\n",
    "        <th>Data</th>\n",
    "        <th>TP</th>\n",
    "        <th>FN</th>\n",
    "        <th>FP</th>\n",
    "        <th>Recall</th>\n",
    "        <th>Precision</th>\n",
    "        <th>Confidence threshold</th>\n",
    "        <th>IoU threshold</th>\n",
    "        <th>mAP</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>Faster-RCNN R50 MoCov2</td>\n",
    "        <td>150k</td>\n",
    "        <td>7797</td>\n",
    "        <td>1015</td>\n",
    "        <td>2838</td>\n",
    "        <td>88.48</td>\n",
    "        <td>73.31</td>\n",
    "        <td>0.5</td>\n",
    "        <td>0.4</td>\n",
    "        <td>87.84</td>\n",
    "    <tr>\n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>Faster-RCNN R50 MoCov2</td>\n",
    "        <td>50k</td>\n",
    "        <td>5508</td>\n",
    "        <td>3304</td>\n",
    "        <td>311</td>\n",
    "        <td>62.50</td>\n",
    "        <td>94.65</td>\n",
    "        <td>0.98</td>\n",
    "        <td>0.4</td>\n",
    "        <td>78.94</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The model outperforms human level annotators\n",
    "<br>\n",
    "We choose non-professional annotators and gave them task to detect ads on 200 screenshots (one of the our benchmarks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <th>Model</th>\n",
    "        <th>Data</th>\n",
    "        <th>TP</th>\n",
    "        <th>FN</th>\n",
    "        <th>FP</th>\n",
    "        <th>Recall</th>\n",
    "        <th>Precision</th>\n",
    "        <th>Confidence threshold</th>\n",
    "        <th>IoU threshold</th>\n",
    "        <th>mAP</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>Human</td>\n",
    "        <td>Non-professional</td>\n",
    "        <td>194</td>\n",
    "        <td>71</td>\n",
    "        <td>27</td>\n",
    "        <td>73.21</td>\n",
    "        <td>87.78</td>\n",
    "        <td>0.5</td>\n",
    "        <td>0.4</td>\n",
    "        <td>80.50</td>\n",
    "    <tr>\n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>Faster-RCNN RX101</td>\n",
    "        <td>50k</td>\n",
    "        <td>251</td>\n",
    "        <td>14</td>\n",
    "        <td>39</td>\n",
    "        <td>94.71</td>\n",
    "        <td>86.55</td>\n",
    "        <td>0.98</td>\n",
    "        <td>0.4</td>\n",
    "        <td>94.84</td>\n",
    "    </tr>        \n",
    "    </tr>\n",
    "        <tr>\n",
    "        <td>Faster-RCNN R50 MoCov2</td>\n",
    "        <td>50k</td>\n",
    "        <td>177</td>\n",
    "        <td>88</td>\n",
    "        <td>35</td>\n",
    "        <td>66.79</td>\n",
    "        <td>83.49</td>\n",
    "        <td>0.98</td>\n",
    "        <td>0.4</td>\n",
    "        <td>76.92</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Future experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Use MoCo-v2 pre-trained weights for DETR model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "DETR - Transformer based object detection model\n",
    "<img src=\"images/ssl/detr_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We used DERT model with ImageNet pre-trained ResNet50, ResNeXt101 backbones but they did not give us sufficient performance in comparison with Faster-RCNN models \n",
    "<br>\n",
    "\n",
    "Maybe MoCo-V2 pre-trained backbone will give us performance gain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Fine-tune MoCo-V2 on our custom dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Take pre-trained MoCo-V2 model and fine-tune it with the same method on our custom dataset\n",
    "- Less computational resources\n",
    "- Less data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Use different models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Train different models with self-supervised learning on custom dataset\n",
    "- ResNet34 for smaller images\n",
    "- WideResNet -maybe fine-tune"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Find proper hyperparameters for different domains"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Different augmentation for different datasets\n",
    "- Different LR policy\n",
    "- Fine-tune with discriminative learning rate: $lr = (lr / 1000, lr / 100, lr)$\n",
    "- Progressive growing approach\n",
    "- One-cycle learning\n",
    "- Maybe different moment or different temperature etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions\n",
    "\n",
    "<img src=\"images/od/questions_2.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thank you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}