Skip to content

Creating and Configuring a Mission

Jump to bottom
Thomas Bassler edited this page Feb 19, 2022 · 18 revisions

Creating and Configuring a Mission

This page aims to lead you through the steps required to set up the static configuration of a mission in prosEO. The following sequence of steps is recommended:

The documentation below is based on a configuration using the prosEO Command Line Interface (CLI). At the time of this writing this is the only way to configure prosEO missions. A future version of prosEO may have configuration functionality in the Graphical User Interface (GUI).

The mission configuration presented below is the sample configuration provided in the prosEO source code in the directory <project root>/samples/testdata. There convenience scripts are provided, which will generate the files used in this tutorial.

Create a Mission

In order to create a mission, a few attributes must be set. Basic attributes are:

  • code: The short code for the mission (e.g. S1B for the Sentinel-1A mission)
  • name: A longer, descriptive name (e.g. Sentinel-1A)
  • processingCentre: An identifying string for the processing centre (organisation) operating this prosEO instance (optional)

General options for file naming include:

  • fileClasses: A list of product file classifications (e.g. OPER, TEST etc.)
  • productFileTemplate: A template for generating output product file names in Spring Expression Language (SPEL), see below for examples

The following attributes influence processing and data storage:

  • processingModes: A list of operational modes for data processors (e.g. NRT for near-realtime processing mode vs. NTC for non-time critical processing, which may lead to different algorithms being applied)
  • productRetentionPeriod: The default period of time for keeping products after their generation in seconds (optional, if not set here or in a specific processing order, no automated product eviction will take place)
  • orderRetentionPeriod: The default period of time for keeping orders after their completion in seconds (optional; if set, processing orders in SYSTEMATIC production mode will be deleted automatically the given number of seconds after closing)

For the mission, its spacecraft(s) and payload(s) on the spacecraft(s) can be configured. This is required if production planning by orbits is intended. For each spacecraft the following attributes are available:

  • code: The short code for the spacecraft (e. g. TDX1 for the first satellite of the TanDEM-X mission)
  • name: The spacecraft name (e. g. TanDEM-X 1)
  • payloads: A list of payloads flown on the satellite (may be empty). Each payload is specified with:
    • name: The payload name (instrument short name), e. g. "TROPOMI" (on the Sentinel-5P satellite)
    • description: A descriptive text (optional)

The following subsections discuss in more detail:

Configuring the Product File Template

The product file template is a mandatory element in prosEO, which is needed when the Production Planner creates output product entries for the Job Order file, which is passed to the data processors. In order to be able to address the wildly different naming conventions for product files across (and even within) satellite missions, this attribute must be specified in the form of a template in Spring Expression Language, which allows for a significant degree of freedom in the specification of the template, but also incurs a certain unavoidable complexity in the definition.

Let's start with the simplest workable example:

product_${id}.nc

This will generate product files with the prefix product_, the database id of the product entry and the extension .nc, which is common for product files in the widely used NetCDF format. The variable part ${id} indicates that for the file name generation the id attribute of the Product class shall be used. For available attributes please see the Data Model Reference.

Since such a file name has not much information for the human reader, let us consider a more extended template (line breaks for readability inserted):

PTM_${fileClass}
  _${productClass.productType}
  _${T(java.time.format.DateTimeFormatter).ofPattern("uuuuMMdd'T'HHmmss").withZone(T(java.time.ZoneId).of("UTC")).format(sensingStartTime)}
  _${T(java.time.format.DateTimeFormatter).ofPattern("uuuuMMdd'T'HHmmss").withZone(T(java.time.ZoneId).of("UTC")).format(sensingStopTime)}
  _${(new java.text.DecimalFormat("00000")).format(null == orbit ? 0 : orbit.orbitNumber)}
  _${parameters.get("revision").getParameterValue()}
  _${configuredProcessor.processor.processorVersion}.nc

This template will generate product file names looking something like:

PTM_OPER_L2_SAR_20220218T132201_20220218T133201_01234_01_1.0.0.nc

where:

  • PTM is a fixed prefix used throughout the mission (PTM might be the mission code for some prosEO Test Mission)
  • OPER is one of the mission file classes as defined above
  • L2_SAR is the name of a product class (apparently some Synthetic Aperture Radar product at processing level L2)
  • 20220218T132201 and 20220218T133201 are the sensing start and stop times for the data acquisition on the satellite
  • 01234 is the orbit number
  • 1.0.0 is the version of the data processor used
  • .nc is a fixed extension, assuming that all output products of this mission are single files in NetCDF format

The remaining underscore characters are included literally in the template and are inserted to separate the individual, meaning-bearing parts from each other.

The template contains some idioms for selecting product attributes and formatting them according to the mission needs. Some idioms we have come across so far are listed below (line breaks inserted for readability):

Idiom Parameters Explanation
${X} X Include attribute X from class Product
${X.Y} X, Y Include attribute Y from the object referenced as X in the class Product; such references may also be used over several link levels, see examples for product type and processor version above; to avoid null pointer exceptions, the expression "X.Y" can be extended to "(null == X ? Z : X.Y)", with Z being some suitable replacement value
${T(java.time.format.DateTimeFormatter)
  .ofPattern("uuuuMMdd'T'HHmmss")
  .withZone(T(java.time.ZoneId).of("UTC"))
  .format(X)}		 X Include attribute X from class Product, but format it as a time stamp in the format "yyyymmddThhmmss", using the UTC representation of the timestamp stored for the product
${(new java.text.DecimalFormat("00000"))
  .format(X}		 X Include attribute X from class Product, formatted as a five-digit number with leading zeroes
${X.replaceAll("\.", "")} X Include attribute X from class Product, removing all periods (".")

For a full description of the Spring Expression Language see Spring Framework Reference – Core Technologies.

For the full description of prosEO's data model see Data Model Reference.

Creating the Mission in prosEO

After this theoretical introduction it is time to actually set up the mission in prosEO. We will use the Command Line Interface (CLI) for this – currently this is the only way to do this. Since prosEO requires a login before any action can be executed in the CLI, and a login requires a mission to login to, we would be stuck right at the beginning – were there not an exception to this rule: A user with ROOT privilege may login without a mission, and such a user is created in prosEO at the time of the first startup. The username is sysadm and its password initially is sysadm, too. Definitely do change this password after the first login!

The CLI is a Java client program that runs on any platform supporting Java (Windows, Linux, macOS, ...). If you built prosEO from source you will find the CLI as an executable JAR file at <project root>/ui/cli/target/proseo-ui-cli.jar. Otherwise it can be downloaded from https://proseo-registry.eoc.dlr.de/artifactory/proseo/proseo-ui-cli-<version>.jar (no login required; the version you need can be taken from the first lines of the startup log of any prosEO component). As a Java program in an executable JAR file, the CLI is started with the command (assuming that the JAR file is in the current directory):

java -jar proseo-ui-cli.jar

Hint: If you are on a Unix-like environment, you may like to create a shell alias to simplify the access to the CLI. Add something like the following to your .profile or .bashrc or whatever the name of your shell's startup script is:

alias proseo="java -jar </path/to>/proseo-ui-cli.jar"

Before the CLI can be started, it needs to know where to find the prosEO components it interacts with. It will take this information from a configuration file named application.yml, which must reside in the directory, from where the CLI is started (i.e. your current working directory). Again, a sample file can be found either at <project root>/ui/cli/src/main/resources/application.yml or at https://proseo-registry.eoc.dlr.de/artifactory/proseo/application.yml-<version>.sample. The sample file assumes a single-node installation running under Docker Desktop on your local machine. If that is not the case, replace localhost:<port> by the hostname(s) and ports for your actual deployment (for deployment options see the Deployment Guide).

Once the configuration is done and you start up the CLI, it will greet you with a startup message and a prompt:

user@localhost cli % java -jar proseo-ui-cli.jar

                            /-------\ /-------\
                            |       | |       |
                            |   /---/ |  /-\  |
                            |   \---\ |  | |  |   prosEO - The Processing System for Earth Observation Data
/----\ /----\ /----\ /----\ |       | |  | |  |
|    | |    | |    | | ---+ |   /---/ |  | |  |   Command Line Interface  (v0.8.2)
| {} | |  /-/ | {} | \    \ |   \---\ |  \-/  |
|    | |  |   |    | +--- | |       | |       |   :: Spring Boot ::       (v2.1.6.RELEASE)
|  /-/ \--/   \----/ \----/ \-------/ \-------/
|  |
\--/

(Type "help" for help.)

prosEO (no mission)>

The prompt indicates that your are not logged in to any mission. A discussion of the available CLI commands is beyond the scope of this document, please refer to the Command Line Interface (CLI) Guide. But one word of

Caution: Do not use Control-C (^C) to cancel a command. This will immediately stop the complete CLI!

We will login as sysadm and immediately change its password (blank lines inserted for readability):

prosEO (no mission)> login
Username (empty field cancels): sysadm
Password for user sysadm: 
(I2821) User sysadm logged in

prosEO (no mission)> password
Password (unencrypted; empty field cancels): 
Repeat Password: 
(I2730) Password changed for user sysadm

prosEO (no mission)>

Note that you are still not logged in to any mission – given that your are probably just going to configure your first mission at all, this is the only way it can be. And it will be for all future mission creations, or whenever you log in as sysadm (or whatever other user you created with the ROOT privilege).

The best way to create a mission is to prepare a JSON file with all the mission details. We will call the file PTM.json:

{
    "code": "PTM",
    "name": "prosEO Test Mission",
    "fileClasses": [ "TEST", "OPER" ],
    "processingModes": [ "OPER" ],
    "productFileTemplate": "PTM_${fileClass}_${productClass.productType}_${T(java.time.format.DateTimeFormatter).ofPattern(\"uuuuMMdd'T'HHmmss\").withZone(T(java.time.ZoneId).of(\"UTC\")).format(sensingStartTime)}_${T(java.time.format.DateTimeFormatter).ofPattern(\"uuuuMMdd'T'HHmmss\").withZone(T(java.time.ZoneId).of(\"UTC\")).format(sensingStopTime)}_${(new java.text.DecimalFormat(\"00000\")).format(null == orbit ? 0 : orbit.orbitNumber)}_${parameters.get(\"revision\").getParameterValue()}_${configuredProcessor.processor.processorVersion.replaceAll(\"\\\\.\", \"\")}_${T(java.time.format.DateTimeFormatter).ofPattern(\"uuuuMMdd'T'HHmmss\").withZone(T(java.time.ZoneId).of(\"UTC\")).format(generationTime)}.nc",
    "processingCentre": "PTM-PDGS",
    "productRetentionPeriod": 2592000,
    "spacecrafts": [ {
        "code": "PTS",
        "name": "prosEO Test Satellite",
        "payloads": [ {
            "name": "PTS-SENSOR",
            "description": "Super sensor for PTM"
        } ]
    } ]
}

With this file, creating the mission is just one command in the prosEO CLI:

prosEO (no mission)> mission create --file=PTM.json
(I2859) Mission PTM created (database ID 1)
prosEO (no mission)>

Now you have created your first mission in prosEO.

Setup Users and User Groups

TODO

Configure Product Classes and Selection Rules

TODO

Configure Processor Classes, Versions and Configurations

TODO

Reference Information

Tutorials

Clone this wiki locally