1 - The first S1 scene.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1PV8FN2-x2wj"
   },
   "source": [
    "<font face=\"Calibri\" size=\"2\"> <i>Open SAR Toolkit - Tutorial 1, version 1.3,  July 2020. Andreas Vollrath, ESA/ESRIN phi-lab</i>\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kOn6Dorgx2wk"
   },
   "source": [
    "![title](auxiliary/header_image.PNG)\n",
    "\n",
    "--------\n",
    "\n",
    "# OST Tutorial I \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Pre-processing your first Sentinel-1 image with OST\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ESA-PhiLab/OST_Notebooks/blob/master/1%20-%20The%20first%20S1%20scene.ipynb)\n",
    "\n",
    "--------\n",
    "\n",
    "**Short description**\n",
    "\n",
    "This notebook introduces you to the *Sentinel1Scene* class of the Open SAR Toolkit and demonstrates how it can be used to download, extract metadata and pre-process a single Sentinel-1 scene.\n",
    "\n",
    "--------\n",
    "\n",
    "**Requirements**\n",
    "\n",
    "- a PC/Mac with at least 16GB of RAM\n",
    "- about 4 GB of free disk space (or more if you want to process more scenes)\n",
    "- a NASA Earthdata account with signed EULA for use of https://search.asf.alaska.edu (just register directly there)\n",
    "--------\n",
    "\n",
    "**NOTE:** all cells that have an * after its number can be executed without changing any code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o4gN2FtIx2wl"
   },
   "source": [
    "### 0\\* - Install OST and dependencies \n",
    "\n",
    "**NOTE:** Applies only if you haven't fully installed OST and its dependencies yet, e.g. on Google Colab, so uncomment the lines in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KqtRYFnhx2wm"
   },
   "outputs": [],
   "source": [
    "# !apt-get -y install wget\n",
    "# !wget https://raw.githubusercontent.com/ESA-PhiLab/OST_Notebooks/master/install_ost.sh\n",
    "# !bash install_ost.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "keC_14irx2wq"
   },
   "source": [
    "### 1\\* - Import the OST *Sentinel1Scene* class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "l1WpZ6RSx2wr"
   },
   "outputs": [],
   "source": [
    "# these imports we need to handle the folders, independent of the OS\n",
    "from pathlib import Path\n",
    "from pprint import pprint\n",
    "\n",
    "# this is the Sentinel1Scene class, that basically handles all the workflow from beginning to the end\n",
    "from ost import Sentinel1Scene\n",
    "\n",
    "#from ost.helpers.settings import set_log_level\n",
    "#import logging\n",
    "#set_log_level(logging.DEBUG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hpm4Ii9gx2wu"
   },
   "source": [
    "### 2* - Create a folder for our outputs\n",
    "\n",
    "By executing this cell, a new folder will be created and the path will be written to the *output_dir* variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KDfHogBVx2wv"
   },
   "outputs": [],
   "source": [
    "# get home folder\n",
    "home = Path.home()\n",
    "\n",
    "# create a processing directory\n",
    "output_dir = home.joinpath('OST_Tutorials', 'Tutorial_1')\n",
    "output_dir.mkdir(parents=True, exist_ok=True)\n",
    "print(str(output_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9eR0Uy4nx2wy"
   },
   "source": [
    "### 3* - Choose scene ID and display some metadata\n",
    "\n",
    "In order to initialize an instance of the *Sentinel1Scene* class, all we need is a valid scene id of a Sentinel-1 product. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "km3z40Kjx2wz"
   },
   "outputs": [],
   "source": [
    "# create a S1Scene class instance based on the scene identifier of the first ever Dual-Pol Sentinel-1 IW product\n",
    "\n",
    "#---------------------------------------------------\n",
    "# Some scenes to choose from\n",
    "\n",
    "# very first IW (VV/VH) S1 image available over Istanbul/Turkey \n",
    "# NOTE:only available via ASF data mirror\n",
    "scene_id = 'S1A_IW_GRDH_1SDV_20141003T040550_20141003T040619_002660_002F64_EC04' \n",
    "\n",
    "# other scenes with different scene types to process (uncomment)\n",
    "# IW scene (dual-polarised HH/HV) over Norway/Spitzbergen\n",
    "# scene_id = 'S1B_IW_GRDH_1SDH_20200325T150411_20200325T150436_020850_02789D_2B85'  \n",
    "\n",
    "# IW scene (single-polarised VV) over Ecuadorian Amazon \n",
    "# scene_id = 'S1A_IW_GRDH_1SSV_20150205T232009_20150205T232034_004494_00583A_1C80' \n",
    "\n",
    "# EW scene (dual-polarised VV/VH) over Azores (needs a different DEM, see cell of ARD parameters below)\n",
    "# scene_id = 'S1B_EW_GRDM_1SDV_20200303T193150_20200303T193250_020532_026E82_5CE9' \n",
    "\n",
    "# EW scene (dual-polarised HH/HV) over Greenland \n",
    "# scene_id = 'S1B_EW_GRDM_1SDH_20200511T205319_20200511T205419_021539_028E4E_697E' \n",
    "\n",
    "# Stripmap mode S5 scene (dual-polarised VV/VH) over Germany \n",
    "# scene_id = 'S1B_S5_GRDH_1SDV_20170104T052519_20170104T052548_003694_006587_86AB' \n",
    "#---------------------------------------------------\n",
    "\n",
    "# create an S1Scene instance\n",
    "s1 = Sentinel1Scene(scene_id)\n",
    "\n",
    "# print summarising infos about the scene\n",
    "s1.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4KaMauPDx2w3"
   },
   "source": [
    "### 4* Download scene\n",
    "\n",
    "The first step is to download the selected scene. In our case we chose the first regular Sentinel-1 IW product acquired in the dual-polarisation VV/VH acquired the 3rd of October 2014. \n",
    "OST provides download from 3 different mirrors, ESA's scihub, CNES PEPS and Alaska Satellite Facility (NASA Earthdata). Since ESA's official scihub became a rolling archive, and so is PEPS, best is to download from the fantastic **Alaska Satellite Facility** mirror (selection 2), which holds the full Sentinel-1 archive online. \n",
    "\n",
    "**Note I:** You can interrupt the download and restart it. The download will continue from where it stopped.\n",
    "\n",
    "**Note II:** After the actual download, the file is checked for inconsistency. This assures that the download went fine and we can use it for processing. In addition, OST will magically remember that this file has been successfully downloaded (just run the cell again to see the behaviour)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pcdvGAS5x2w3"
   },
   "outputs": [],
   "source": [
    "s1.download(output_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cs_Ps0Agx2w7"
   },
   "source": [
    "### 5* - Define our ARD product\n",
    "\n",
    "ARD stands for Analysis Ready Data and is interpreted differently by different persons. OST provides various pre-defined flavours which can be used instantly.\n",
    "\n",
    "The following table shows the ARD types and corresponding processing steps applied for GRD data. \n",
    "\n",
    "|               ARD Types  | OST-GTC      | OST-RTC         | CEOS           | Earth-Engine |\n",
    "| :-------------           | :----------: | :-----------:   | :----------:   | -----------: |\n",
    "| **Scene processing steps**                                                                |        \n",
    "|  Update Orbit File       |       x      |       x         |       x        |      x       |\n",
    "|  Thermal Noise Removal   |       x      |       x         |       x        |      x       |\n",
    "|  GRD Border Noise        |       x      |       x         |       x        |      -       |\n",
    "|  Calibration             | $\\gamma^0$   |   $\\beta^0$     |  $\\beta^0$     |  $\\sigma^0$  |\n",
    "|  Multi-look              |       x      |       x         |       -        |      -       |\n",
    "|  Speckle-Filter          |       -      |       -         |       -        |      -       |\n",
    "|  Terrain Flattening      |       -      | x ($\\gamma^0_f$)|x ($\\gamma^0_f$)|      -       |\n",
    "|  Layover-Shadow Mask     |       -      |       x         |       x        |      -       |\n",
    "|  dB conversion           |       -      |       -         |       -        |      x       |\n",
    "|  Terrain Correction      |       x      |       x         |       x        |      x       |\n",
    "|  Resolution (in m)       |       20     |       20        |       10       |      10      |\n",
    "\n",
    "The default ARD type is *'OST-GTC'*, referring to a Geometrically Terrain Corrected product which is calibrated to the ellipsoid correceted $\\gamma^0$ backscatter coefficient at 20m resolution. Other pre-defined ARD types are available, but it is also possible to customise single ARD parameters via a dictionary where all parameters are stored (as demonstrated in the cell below). Note how the resolution and the resampling of the image during terrain correction is changed at the bottom. In this way, actually all relevant parameters for processing are customisable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HZfdAYg3x2w7"
   },
   "outputs": [],
   "source": [
    "# Default ARD parameter\n",
    "\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "print('Our ARD parameters dictionary contains 4 keys. For the moment, only single_ARD is relevant.')\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "pprint(s1.ard_parameters.keys())\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "print('')\n",
    "\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "print('Dictionary of our default OST ARD parameters for single scene processing:')\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "pprint(s1.ard_parameters['single_ARD'])\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "print('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HZfdAYg3x2w7"
   },
   "outputs": [],
   "source": [
    "# Template ARD parameters\n",
    "\n",
    "# we change ARD type\n",
    "# possible choices are:\n",
    "# 'OST_GTC', 'OST-RTC', 'CEOS', 'Earth Engine'\n",
    "s1.update_ard_parameters('Earth-Engine')\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "print('Dictionary of Earth Engine ARD parameters:')\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "pprint(s1.ard_parameters['single_ARD'])\n",
    "print('-----------------------------------------------------------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HZfdAYg3x2w7"
   },
   "outputs": [],
   "source": [
    "# Customised ARD parameters\n",
    "\n",
    "# we cusomize the resolution and image resampling\n",
    "s1.ard_parameters['single_ARD']['resolution'] = 100  # set output resolution to 100m\n",
    "s1.ard_parameters['single_ARD']['remove_speckle'] = False # apply a speckle filter\n",
    "s1.ard_parameters['single_ARD']['dem']['image_resampling'] = 'BILINEAR_INTERPOLATION' # BICUBIC_INTERPOLATION is default\n",
    "\n",
    "# s1.ard_parameters['single_ARD']['product_type'] = 'RTC-gamma0' \n",
    "\n",
    "# uncomment this for the Azores EW scene\n",
    "# s1.ard_parameters['single_ARD']['dem']['dem_name'] = 'GETASSE30'\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "print('Dictionary of our customised ARD parameters for the final scene processing:')\n",
    "print('-----------------------------------------------------------------------------------------------------------')\n",
    "pprint(s1.ard_parameters['single_ARD'])\n",
    "print('-----------------------------------------------------------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IavR8qQmx2w_"
   },
   "source": [
    "### 6* - Create an ARD product\n",
    "\n",
    "Our *Sentinel1Scene* class comes with the build-in method *create_ard* to produce a standardised ARD product based on the ARD dictionary above. \n",
    "\n",
    "To run the command we just have to provide: \n",
    "- the path to the downloaded file. We can use the *get_path* method in conjunction with the download directory provided\n",
    "- a directory where the output files will be written to\n",
    "- a filename prefix (the output format will be the standard SNAP Dimap format, consisting of the dim-file and the data directory)\n",
    "- and a directory for storing temporary files (can not be the same as the output folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BsSOUvVUx2w_"
   },
   "outputs": [],
   "source": [
    "s1.create_ard(\n",
    "    infile=s1.get_path(output_dir),\n",
    "    out_dir=output_dir, \n",
    "    overwrite=True\n",
    ") \n",
    "\n",
    "print(' The path to our newly created ARD product can be obtained the following way:')\n",
    "s1.ard_dimap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Umkg9vgux2xC"
   },
   "source": [
    "### 6* - Create a RGB color composite\n",
    "\n",
    "Sentinel-1 scenes usually consist of two polarisation bands. In order to create a 3 channel RGB composite a ratio between the Co- (VV or HH) and the Cross-polarised (VH or HV) band is added. The *create_rgb* method takes the *ard_dimap* file and converts it to a 3-channel GeoTiff."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bT5tNXs1x2xD"
   },
   "outputs": [],
   "source": [
    "s1.create_rgb(outfile = output_dir.joinpath(f'{s1.start_date}.tif'))\n",
    "\n",
    "print(' The path to our newly created RGB product can be obtained the following way:')\n",
    "s1.ard_rgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y_FbFahox2xF"
   },
   "source": [
    "### 7* - Visualise the RGB composite\n",
    "\n",
    "We can plot the newly created RGB image with the *visualise_rgb* method. A *shrink_factor* can be set, which reduces resolution in favour of memory requirements for plotting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PQzl1epox2xG"
   },
   "outputs": [],
   "source": [
    "#---------------------------------------------------\n",
    "# for plotting purposes we use this iPython magic\n",
    "%matplotlib inline\n",
    "%pylab inline\n",
    "pylab.rcParams['figure.figsize'] = (23, 23)\n",
    "#---------------------------------------------------\n",
    "s1.visualise_rgb(shrink_factor=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4QaGAsf8x2xJ"
   },
   "source": [
    "### 8* - Create thumbnail image\n",
    "\n",
    "For some it might be interesting to create a small thumbnail image in Jpeg format. The *create_rgb_thumbnail* method allows for this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bj6RWSpox2xK"
   },
   "outputs": [],
   "source": [
    "# define a filename for our thumbnail image\n",
    "path_to_thumbnail = output_dir.joinpath(f'{s1.start_date}.TN.jpg')\n",
    "\n",
    "# create the thumbnail image\n",
    "s1.create_rgb_thumbnail(outfile=str(path_to_thumbnail))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "efxTV4Zox2xN"
   },
   "outputs": [],
   "source": [
    "import imageio\n",
    "img = imageio.imread(path_to_thumbnail)\n",
    "!ls -sh {path_to_thumbnail}\n",
    "plt.imshow(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9* - Play around\n",
    "\n",
    "You can try out the different images and also check the difference in backscatter for RTC products (uncomment the line of product type in the customisable ARD parameters)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "1 - The first S1 scene.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}