submit_job.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Submit Job\n",
    "\n",
    "Once your job-type has been registered and built (see: [pge_create.ipynb](pge_create.ipynb)), jobs can be submitted from python using the steps laid out in this notebook.\n",
    "\n",
    "While this notebook only shows submission of a single job/parameter-set, you can map or iterate over a collection of input parameter sets to efficiently submit large batches of jobs.\n",
    "\n",
    "### It also assumes you have a valid account on the cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize\n",
    "\n",
    "When prompted for the HySDS host, include the protocol, e.g. https://soamc-mozart.jpl.nasa.gov\n",
    "\n",
    "When prompted for \"HySDS cluster authenticated\", enter 'y' if the cluster requires a password to access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import otello\n",
    "import re\n",
    "import shutil\n",
    "\n",
    "from pathlib import Path\n",
    "from pprint import pprint\n",
    "\n",
    "if not os.path.exists(f\"{Path.home()}/.config/otello/config.yml\"):\n",
    "    otello.client.initialize()\n",
    "\n",
    "m = otello.mozart.Mozart()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instantiate a `JobType` object\n",
    "\n",
    "You will need to ensure that the the tag (i.e. the portion following the colon) matches the branch name used by your repository.\n",
    "\n",
    "You will also need to set *job_type_name* to the name of your job's notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_type_name = 'job-xing-topsstack_hamsar_pge:devel'  # This will need to be customised using the relevant action name (ie job/hysds-spec suffix) and branch name.\n",
    "\n",
    "job_type = m.get_job_types()[job_type_name]\n",
    "job_type.initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Obtain useful information about the job-type\n",
    "\n",
    "The line below lists:\n",
    "- the input schema of the job-type\n",
    "- type and default value of parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_type.get_queues()\n",
    "print(job_type.describe())\n",
    "#pprint(job_type.get_input_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify values to parameters for your job\n",
    "\n",
    "Here is where user-defined parameter values are provided"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for Goma\n",
    "custom_parameters = {\n",
    "    \"min_lat\": -1.77066085473858,\n",
    "    \"max_lat\": -1.462637710194377,\n",
    "    \"min_lon\": 29.13133428070789,\n",
    "    \"max_lon\": 29.38055714034091,\n",
    "    \"bbox\": [],\n",
    "    \"start_date\": \"2021-01-01\",\n",
    "    \"end_date\": \"2021-01-15\",\n",
    "    \"track_number\": 174\n",
    "}\n",
    "\n",
    "job_type.set_input_params(custom_parameters)\n",
    "print(\"Your job will be run with these parameters:\")\n",
    "pprint(job_type.get_input_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit the job\n",
    "\n",
    "A job tag (useful for finding the job later) and job queue are specified.\n",
    "\n",
    "Both are optional. Job submission is asynchronous, so this call will return almost immediately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "sample_job_tag = f'{datetime.strftime(datetime.now(), \"%Y%m%d\")}_topsstack_hamsar_test_66'\n",
    "print(sample_job_tag)\n",
    "\n",
    "#job_run = job_type.submit_job(tag=sample_job_tag, queue=\"factotum-job_worker-small\")\n",
    "job_run = job_type.submit_job(tag=sample_job_tag, queue=\"factotum-2-job_worker-large\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor job state\n",
    "\n",
    "Information about the job state will print periodically, until the job is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "job_run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}