6 Sentinel-1 SLC Parallel Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Process S1 SLC data using parallel processing**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First import all necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ost \n",
    "import ost.helpers as h\n",
    "from ost.helpers import onda, asf_wget, vector\n",
    "from ost import Sentinel1_SLCBatch\n",
    "import os\n",
    "from os.path import join\n",
    "from pathlib import Path\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ingest shapefile data and set start and end dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# create a processing directory\n",
    "project_dir = '/home/ost/Data/jwheeler/Sydney_Fires'\n",
    "\n",
    "# apply function with buffer in meters\n",
    "from ost.helpers import vector\n",
    "input_shp = \"/home/ost/Data/jwheeler/Shapefiles/Sydney_fires.shp\"\n",
    "aoi = vector.shp_to_wkt(input_shp)\n",
    "#----------------------------\n",
    "# Time of interest\n",
    "#----------------------------\n",
    "# we set only the start date to today - 30 days\n",
    "start = '2019-11-30'\n",
    "end = '2019-12-12'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initiate class with above attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create s1Project class instance\n",
    "s1_batch = Sentinel1_SLCBatch(\n",
    "    project_dir=project_dir, \n",
    "    aoi=aoi, \n",
    "    start=start, \n",
    "    end=end, \n",
    "    product_type='SLC',\n",
    "    ard_type='OST Plus')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Search for images on scihub and plot footprints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#---------------------------------------------------\n",
    "# for plotting purposes we use this iPython magic\n",
    "%matplotlib inline\n",
    "%pylab inline\n",
    "pylab.rcParams['figure.figsize'] = (19, 19)\n",
    "#---------------------------------------------------\n",
    "\n",
    "# search command\n",
    "s1_batch.search()\n",
    "# we plot the full Inventory on a map\n",
    "s1_batch.plot_inventory(transparency=.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refine image search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1_batch.refine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select appropriate key and plot filtered images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pylab.rcParams['figure.figsize'] = (13, 13)\n",
    "\n",
    "\n",
    "key = 'DESCENDING_VVVH'\n",
    "s1_batch.refined_inventory_dict[key]\n",
    "s1_batch.plot_inventory(s1_batch.refined_inventory_dict[key], 0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download using a selected S-1 mirror - ideally ASF (2 using request or 5 using wget) or onda (4) if accounts are set up correctly for fast, parallel downloading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1_batch.download(s1_batch.refined_inventory_dict[key],concurrent=8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create inventory of bursts in downloaded images, plot them and print information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1_batch.create_burst_inventory(key=key, refine=True)\n",
    "pylab.rcParams['figure.figsize'] = (13, 13)\n",
    "s1_batch.plot_inventory(s1_batch.burst_inventory, transparency=0.1)\n",
    "print('Our burst inventory holds {} bursts to process.'.format(len(s1_batch.burst_inventory)))\n",
    "print('------------------------------------------')\n",
    "print(s1_batch.burst_inventory.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the below command to view the current ard parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pprint(s1_batch.ard_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the s1SLCbatch class function bursts to ard to generate parameter text files for each step from burst to ard, ard to timeseries, timeseries to timescan and mosaic.\n",
    "\n",
    "**NB Use a base name for the exec file without a extension AND make sure to choose the number of cores that each process will use for parallel processing. ncores in this function x multiproc in the multiprocess function should not exceed the number of cores on your machine**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1_batch.bursts_to_ard(timeseries=True, timescan=True, mosaic=True, overwrite=False, exec_file='/home/ost/Data/jwheeler/Sydney_Fires/test', ncores=2)\n",
    "#print(s1_batch.temp_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the s1SLCbatch class function multiprocessing to run, sequentially, the parameters in the previously generated text files for each step from burst to ard, ard to timeseries, timeseries to timescan and mosaic.\n",
    "\n",
    "**NB Use the same base name for the exec file without a extension as before AND make sure to choose the number of cores that each process will use for parallel processing as well as the number of concurrent processes. ncores in the previous function x multiproc in this function should not exceed the number of cores on your machine. You should also include the ncores again as the text file generation is reiterated during this process**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1_batch.multiprocess(timeseries=True, timescan=True, mosaic=True, overwrite=False, exec_file='/home/ost/Data/jwheeler/Sydney_Fires/test', ncores=2,multiproc=4)\n",
    "#burst_to_ard_batch(s1_batch.burst_inventory,s1_batch.download_dir,s1_batch.processing_dir,s1_batch.temp_dir,s1_batch.proc_file,exec_file='/home/ost/Data/jwheeler/Sydney_Fires/test.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}