4b - Sentinel-1 GRD Batch - Timeseries.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "P2SVmQcoWFBW"
   },
   "source": [
    "<font face=\"Calibri\" size=\"2\"> <i>Open SAR Toolkit - Tutorial 4b, version 1.2, June 2020. Andreas Vollrath, ESA/ESRIN phi-lab</i>\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y_9lvtkkWFBX"
   },
   "source": [
    "![title](auxiliary/header_image.PNG)\n",
    "\n",
    "--------\n",
    "\n",
    "# OST Tutorial IV-B\n",
    "## How to create a timeseries animation of Iceberg A-68. Introduction to GRD Batch Processing part II.\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ESA-PhiLab/OST_Notebooks/blob/master/4b%20-%20Sentinel-1%20GRD%20Batch%20-%20Timeseries.ipynb)\n",
    "\n",
    "--------\n",
    "\n",
    "**Short description**\n",
    "\n",
    "This notebook continues to introduce you to the general workflow of OST for the batch processing of GRD data using the *Sentinel1Batch* class. In this example:\n",
    "\n",
    "1. across-track mosaicking based on a refined inventory and\n",
    "2. processing in polar regions is shown.\n",
    "\n",
    "--------\n",
    "\n",
    "**Requirements**\n",
    "\n",
    "- a PC/Mac with at least 16GB of RAM\n",
    "- about 75 GB of free disk space\n",
    "- a Copernicus Open Data Hub user account, valid for at least 7 days (https://scihub.copernicus.eu)\n",
    "--------\n",
    "\n",
    "**NOTE:** all cells that have an * after its number can be executed without changing any code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0\\* - Install OST and dependencies \n",
    "\n",
    "**NOTE:** Applies only if you haven't fully installed OST yet, e.g. on Google Colab, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 129719,
     "status": "ok",
     "timestamp": 1594827783444,
     "user": {
      "displayName": "Andreas Vollrath",
      "photoUrl": "",
      "userId": "16529617641432137889"
     },
     "user_tz": -120
    },
    "id": "JdI1jtExWkWH",
    "outputId": "2c0c26ee-4137-41d2-8c4c-b389833401e6"
   },
   "outputs": [],
   "source": [
    "# !apt-get -y install wget\n",
    "# !wget https://raw.githubusercontent.com/ESA-PhiLab/OST_Notebooks/master/install_ost.sh\n",
    "# !bash install_ost.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3cZq5gYiWFBY"
   },
   "source": [
    "### 1* - Import of libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1205,
     "status": "ok",
     "timestamp": 1594828366359,
     "user": {
      "displayName": "Andreas Vollrath",
      "photoUrl": "",
      "userId": "16529617641432137889"
     },
     "user_tz": -120
    },
    "id": "6LgvmzL0WFBZ"
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from pprint import pprint\n",
    "\n",
    "from ost import Sentinel1Batch\n",
    "from ost.helpers import vector, raster\n",
    "\n",
    "#---------------------------------------------------\n",
    "# for plotting purposes we use this iPython magic\n",
    "%matplotlib inline\n",
    "%pylab inline\n",
    "pylab.rcParams['figure.figsize'] = (19, 19)\n",
    "#---------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q8fB1i_6WFBd"
   },
   "source": [
    "### 2 - Set up the project \n",
    "\n",
    "This follows the logic of the prior OST Tutorial notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 275
    },
    "colab_type": "code",
    "id": "FyKptPZJWFBe",
    "outputId": "47e445ea-a4b9-4270-ebca-6c93f11f9f95"
   },
   "outputs": [],
   "source": [
    "# define a project directory\n",
    "home = Path.home()\n",
    "# create a processing directory\n",
    "project_dir = home.joinpath('OST_Tutorials', '4b')\n",
    "\n",
    "# define aoi with helper function, i.e. get a buffered area around point coordinates\n",
    "lat, lon = '-67', '-61'\n",
    "aoi = vector.latlon_to_wkt(lat, lon, buffer_degree=1.5, envelope=True)\n",
    "\n",
    "# define the start and end date\n",
    "start = '2017-06-30'\n",
    "end = '2017-08-31'\n",
    "\n",
    "# initialize the class to s1_grd instance\n",
    "s1_grd = Sentinel1Batch(\n",
    "    project_dir=project_dir,\n",
    "    aoi = aoi,\n",
    "    start = start,\n",
    "    end = end,\n",
    "    product_type='GRD'\n",
    ")\n",
    "\n",
    "# trigger the search\n",
    "s1_grd.search()\n",
    "s1_grd.plot_inventory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mr8nkbgRWFBi"
   },
   "source": [
    "### 3 - Search Refinement\n",
    "\n",
    "In order to create a time-series of multiple tracks, a pre-condition is that all tracks feature the same amount of acquistions within our Time of Interest.\n",
    "Let's use some pandas syntax to see if this is the case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = s1_grd.inventory.pivot_table(index=['relativeorbit', 'acquisitiondate'], aggfunc='size').reset_index()\n",
    "df.pivot_table(index='relativeorbit', aggfunc='size').reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in most cases, we do not fulfill this pre-condition. By considering all tracks, our time-series would need to be reduced to 5 acquisitions. However, images taken over track 9 are not necessary, since our AOI is fully covered by the other 2 tracks.\n",
    "\n",
    "As already mentioned in OST Tutorial 2, the *refine_inventory* method takes care of those issues and prepares the inventory in a way that it is suitable for across-track mosaic time-series. This includes the splitting of images by orbit direction and polarzation mode in the first place. In addition, it checks if some tracks can be excluded because all the others fully overlap the AOI. In this way we reduce the amount of images to process, while optimising for our later time-series processing. See OST Tutorial 2 for full explanation and arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do the refinement\n",
    "s1_grd.refine_inventory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the refinement procedure gives some infos, e.g. the exclusion of track 9. At the very end it summarises the information. Since in our case we only have imagery acquired in descending orbit and HH polarization, we see that 10 mosaics in time can be created. Another **important** infomation is the **key** defiend by orbit direction and polarisation, i.e. **DESCENDING_HH**. We will need this to select the refined inventory stored in the *refined_inventory_dict* attribute of our class instance as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1461,
     "status": "ok",
     "timestamp": 1594828707720,
     "user": {
      "displayName": "Andreas Vollrath",
      "photoUrl": "",
      "userId": "16529617641432137889"
     },
     "user_tz": -120
    },
    "id": "FnlnX_l8WFBj",
    "outputId": "836e7ec2-eaf1-4ec1-a5db-d4fefa41ef28"
   },
   "outputs": [],
   "source": [
    "# select the key from output of refinement command\n",
    "key = 'DESCENDING_HH'\n",
    "\n",
    "# we wrap the information of the length of our refined inventory in a print statement\n",
    "print(f'The refined inventory holds {len(s1_grd.refined_inventory_dict[key])} acquisitions to process.')\n",
    "\n",
    "# we plot the full Inventory on a map\n",
    "s1_grd.plot_inventory(s1_grd.refined_inventory_dict[key], transparency=.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 - Data download\n",
    "\n",
    "Here we download the data. It is best to use the ASF data mirror. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WV9i2KwRWFBs"
   },
   "outputs": [],
   "source": [
    "s1_grd.download(s1_grd.refined_inventory_dict[key], concurrent=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5* - Customise ARD parameters\n",
    "\n",
    "1. For data reduction we lower the resolution to 100 meters. \n",
    "2. This will also reduce speckle, so we do not need it neither. \n",
    "3. We do not care about Layover and Shadow for this example, since there is anyway no high-resolution DEM for Antarctica that could provide sufficient inforamtion.\n",
    "4. Our time-series output will be scaled to dB scale for better stretching and visualisation\n",
    "5. We further reduce the amount of data by converting from 32-bit float to unsigned 8-bit integer data type\n",
    "6. Our AOI is only a rough seletion for the Area of Interest. We do not want to cut it to the boundaries to see the full data provided by the selected imagery."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Eu5GYUasWFBw"
   },
   "outputs": [],
   "source": [
    "s1_grd.ard_parameters['single_ARD']['resolution'] = 100\n",
    "s1_grd.ard_parameters['single_ARD']['remove_mt_speckle'] = False\n",
    "s1_grd.ard_parameters['single_ARD']['create_ls_mask'] = False\n",
    "s1_grd.ard_parameters['time-series_ARD']['to_db'] = True\n",
    "s1_grd.ard_parameters['time-series_ARD']['dtype_output'] = 'uint8'\n",
    "s1_grd.ard_parameters['mosaic']['cut_to_aoi'] = False\n",
    "\n",
    "pprint(s1_grd.ard_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6* - Produce Timeseries Mosaics\n",
    "\n",
    "Now the creation of our mosaic time-series is just a single command away."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GYgdavVDWFBz"
   },
   "outputs": [],
   "source": [
    "s1_grd.grds_to_ards(\n",
    "    inventory_df=s1_grd.refined_inventory_dict[key], \n",
    "    timeseries=True, \n",
    "    timescan=False, \n",
    "    mosaic=True, \n",
    "    overwrite=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Iat7lGGTWFB2"
   },
   "source": [
    "### 7* - Creation of an animated GIF of a timeseries\n",
    "\n",
    "**Note:** This needs imagemagick installed, which is not a default requirement by OST.\n",
    "You can install it on e.g. Ubuntu by typing:\n",
    "sudo apt-get install magick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6OF9NCoXWFB2"
   },
   "outputs": [],
   "source": [
    "from ost.helpers import raster\n",
    "\n",
    "# define the timeseries folder for which the animation should be created\n",
    "ts_folder = s1_grd.processing_dir.joinpath('Mosaic/Timeseries')\n",
    "\n",
    "raster.create_timeseries_animation(\n",
    "    ts_folder, \n",
    "    ['bs.HH'], \n",
    "    s1_grd.processing_dir,\n",
    "    shrink_factor=15,\n",
    "    add_dates=True,\n",
    "    duration=0.25\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "4b - Sentinel-1 GRD Batch processing -  Timeseries.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}