-
Notifications
You must be signed in to change notification settings - Fork 1
Creating and Configuring a Mission
This page aims to lead you through the steps required to set up the static configuration of a mission in prosEO. The following sequence of steps is recommended:
- Create the mission
- Setup users and user groups
- Configure product classes and selection rules
- Configure processor classes, versions and configurations
The documentation below is based on a configuration using the prosEO Command Line Interface (CLI). At the time of this writing this is the only way to configure prosEO missions. A future version of prosEO may have configuration functionality in the Graphical User Interface (GUI).
The mission configuration presented below is the sample configuration provided in the prosEO source code in the directory
<project root>/samples/testdata
. There convenience scripts are provided, which will generate the files used in this tutorial.
In order to create a mission, a few attributes must be set. Basic attributes are:
-
code
: The short code for the mission (e.g.S1B
for the Sentinel-1A mission) -
name
: A longer, descriptive name (e.g.Sentinel-1A
) -
processingCentre
: An identifying string for the processing centre (organisation) operating this prosEO instance (optional)
General options for file naming include:
-
fileClasses
: A list of product file classifications (e.g.OPER
,TEST
etc.) -
productFileTemplate
: A template for generating output product file names in Spring Expression Language (SPEL), see below for examples
The following attributes influence processing and data storage:
-
processingModes
: A list of operational modes for data processors (e.g.NRT
for near-realtime processing mode vs.NTC
for non-time critical processing, which may lead to different algorithms being applied) -
productRetentionPeriod
: The default period of time for keeping products after their generation in seconds (optional, if not set here or in a specific processing order, no automated product eviction will take place) -
orderRetentionPeriod
: The default period of time for keeping orders after their completion in seconds (optional; if set, processing orders inSYSTEMATIC
production mode will be deleted automatically the given number of seconds after closing)
For the mission, its spacecraft(s) and payload(s) on the spacecraft(s) can be configured. This is required if production planning by orbits is intended. For each spacecraft the following attributes are available:
-
code
: The short code for the spacecraft (e. g.TDX1
for the first satellite of the TanDEM-X mission) -
name
: The spacecraft name (e. g.TanDEM-X 1
) -
payloads
: A list of payloads flown on the satellite (may be empty). Each payload is specified with:-
name
: The payload name (instrument short name), e. g. "TROPOMI" (on the Sentinel-5P satellite) -
description
: A descriptive text (optional)
-
The following subsections discuss in more detail:
The product file template is a mandatory element in prosEO, which is needed when the Production Planner creates output product entries for the Job Order file, which is passed to the data processors. In order to be able to address the wildly different naming conventions for product files across (and even within) satellite missions, this attribute must be specified in the form of a template in Spring Expression Language, which allows for a significant degree of freedom in the specification of the template, but also incurs a certain unavoidable complexity in the definition.
Let's start with the simplest workable example:
product_${id}.nc
This will generate product files with the prefix product_
, the database id of the product entry and the extension .nc
, which
is common for product files in the widely used NetCDF format. The variable part ${id}
indicates that for the file name generation
the id
attribute of the Product
class shall be used. For available attributes please see the
Data Model Reference.
Since such a file name has not much information for the human reader, let us consider a more extended template (line breaks for readability inserted):
PTM_${fileClass}
_${productClass.productType}
_${T(java.time.format.DateTimeFormatter).ofPattern("uuuuMMdd'T'HHmmss").withZone(T(java.time.ZoneId).of("UTC")).format(sensingStartTime)}
_${T(java.time.format.DateTimeFormatter).ofPattern("uuuuMMdd'T'HHmmss").withZone(T(java.time.ZoneId).of("UTC")).format(sensingStopTime)}
_${(new java.text.DecimalFormat("00000")).format(null == orbit ? 0 : orbit.orbitNumber)}
_${parameters.get("revision").getParameterValue()}
_${configuredProcessor.processor.processorVersion}.nc
This template will generate product file names looking something like:
PTM_OPER_L2_SAR_20220218T132201_20220218T133201_01234_01_1.0.0.nc
where:
-
PTM
is a fixed prefix used throughout the mission (PTM
might be the mission code for some prosEO Test Mission) -
OPER
is one of the mission file classes as defined above -
L2_SAR
is the name of a product class (apparently some Synthetic Aperture Radar product at processing level L2) -
20220218T132201
and20220218T133201
are the sensing start and stop times for the data acquisition on the satellite -
01234
is the orbit number -
1.0.0
is the version of the data processor used -
.nc
is a fixed extension, assuming that all output products of this mission are single files in NetCDF format
The remaining underscore characters are included literally in the template and are inserted to separate the individual, meaning-bearing parts from each other.
The template contains some idioms for selecting product attributes and formatting them according to the mission needs. Some idioms we have come across so far are listed below (line breaks inserted for readability):
Idiom | Parameters | Explanation |
---|---|---|
${X} | X | Include attribute X from class Product |
${X.Y} | X, Y | Include attribute Y from the object referenced as X in the class Product; such references may also be used over several link levels, see examples for product type and processor version above; to avoid null pointer exceptions, the expression "X.Y" can be extended to "(null == X ? Z : X.Y)", with Z being some suitable replacement value |
${T(java.time.format.DateTimeFormatter) .ofPattern("uuuuMMdd'T'HHmmss") .withZone(T(java.time.ZoneId).of("UTC")) .format(X)} |
X | Include attribute X from class Product, but format it as a time stamp in the format "yyyymmddThhmmss", using the UTC representation of the timestamp stored for the product |
${(new java.text.DecimalFormat("00000")) .format(X} |
X | Include attribute X from class Product, formatted as a five-digit number with leading zeroes |
${X.replaceAll("\.", "")} | X | Include attribute X from class Product, removing all periods (".") |
For a full description of the Spring Expression Language see Spring Framework Reference – Core Technologies.
For the full description of prosEO's data model see Data Model Reference.
After this theoretical introduction it is time to actually set up the mission in prosEO. We will use the Command Line Interface
(CLI) for this – currently this is the only way to do this. Since prosEO requires a login before any action can be executed in the
CLI, and a login requires a mission to login to, we would be stuck right at the beginning – were there not an exception to this
rule: A user with ROOT
privilege may login without a mission, and such a user is created in prosEO at the time of the first
startup. The username is sysadm
and its password initially is sysadm
, too. Definitely do change this password after the
first login!
The CLI is a Java client program that runs on any platform supporting Java (Windows, Linux, macOS, ...). If you built prosEO from
source you will find the CLI as an executable JAR file at <project root>/ui/cli/target/proseo-ui-cli.jar
. Otherwise it can be
downloaded from https://proseo-registry.eoc.dlr.de/artifactory/proseo/proseo-ui-cli-<version>.jar
(no login required; the
version you need can be taken from the first lines of the startup log of any prosEO component). As a Java program in an executable
JAR file, the CLI is started with the command (assuming that the JAR file is in the current directory):
java -jar proseo-ui-cli.jar
Hint: If you are on a Unix-like environment, you may like to create a shell alias to simplify the access to the CLI. Add something
like the following to your .profile
or .bashrc
or whatever the name of your shell's startup script is:
alias proseo="java -jar </path/to>/proseo-ui-cli.jar"
Before the CLI can be started, it needs to know where to find the prosEO components it interacts with. It will take this information
from a configuration file named application.yml
, which must reside in the directory, from where the CLI is started (i.e. your
current working directory). Again, a sample file can be found either at <project root>/ui/cli/src/main/resources/application.yml
or at https://proseo-registry.eoc.dlr.de/artifactory/proseo/application.yml-<version>.sample
. The sample file assumes a
single-node installation running under Docker Desktop on your local machine. If that is not the case, replace localhost:<port>
by the hostname(s) and ports for your actual deployment (for deployment options see the Deployment Guide).
Once the configuration is done and you start up the CLI, it will greet you with a startup message and a prompt:
user@localhost cli % java -jar proseo-ui-cli.jar
/-------\ /-------\
| | | |
| /---/ | /-\ |
| \---\ | | | | prosEO - The Processing System for Earth Observation Data
/----\ /----\ /----\ /----\ | | | | | |
| | | | | | | ---+ | /---/ | | | | Command Line Interface (v0.8.2)
| {} | | /-/ | {} | \ \ | \---\ | \-/ |
| | | | | | +--- | | | | | :: Spring Boot :: (v2.1.6.RELEASE)
| /-/ \--/ \----/ \----/ \-------/ \-------/
| |
\--/
(Type "help" for help.)
prosEO (no mission)>
The prompt indicates that your are not logged in to any mission. A discussion of the available CLI commands is beyond the scope of this document, please refer to the Command Line Interface (CLI) Guide. But one word of
Caution: Do not use Control-C (^C) to cancel a command. This will immediately stop the complete CLI!
We will login as sysadm
and immediately change its password (blank lines inserted for readability):
prosEO (no mission)> login
Username (empty field cancels): sysadm
Password for user sysadm:
(I2821) User sysadm logged in
prosEO (no mission)> password
Password (unencrypted; empty field cancels):
Repeat Password:
(I2730) Password changed for user sysadm
prosEO (no mission)>
Passwords must conform to a password policy, which is currently set to be a minimum length of 10 characters and containing at least elements of three of the four character groups lowercase letters, uppercase letters, digits and special characters.
Note that you are still not logged in to any mission – given that your are probably just going to configure your first mission
at all, this is the only way it can be. And it will be for all future mission creations, or whenever you log in as sysadm
(or
whatever other user you created with the ROOT
privilege).
The best way to create a mission is to prepare a JSON file with all the mission details. We will call the file PTM.json
:
{
"code": "PTM",
"name": "prosEO Test Mission",
"fileClasses": [ "TEST", "OPER" ],
"processingModes": [ "OPER" ],
"productFileTemplate": "PTM_${fileClass}_${productClass.productType}_${T(java.time.format.DateTimeFormatter).ofPattern(\"uuuuMMdd'T'HHmmss\").withZone(T(java.time.ZoneId).of(\"UTC\")).format(sensingStartTime)}_${T(java.time.format.DateTimeFormatter).ofPattern(\"uuuuMMdd'T'HHmmss\").withZone(T(java.time.ZoneId).of(\"UTC\")).format(sensingStopTime)}_${(new java.text.DecimalFormat(\"00000\")).format(null == orbit ? 0 : orbit.orbitNumber)}_${parameters.get(\"revision\").getParameterValue()}_${configuredProcessor.processor.processorVersion.replaceAll(\"\\\\.\", \"\")}_${T(java.time.format.DateTimeFormatter).ofPattern(\"uuuuMMdd'T'HHmmss\").withZone(T(java.time.ZoneId).of(\"UTC\")).format(generationTime)}.nc",
"processingCentre": "PTM-PDGS",
"productRetentionPeriod": 2592000,
"spacecrafts": [ {
"code": "PTS",
"name": "prosEO Test Satellite",
"payloads": [ {
"name": "PTS-SENSOR",
"description": "Super sensor for PTM"
} ]
} ]
}
With this file, creating the mission is just one command in the prosEO CLI:
prosEO (no mission)> mission create --file=PTM.json
(I2859) Mission PTM created (database ID 1)
prosEO (no mission)>
Now you have created your first mission in prosEO.
Since we are still working with the sysadm
user, and by definition this user has very restricted privileges (most notably
it is not able to log in to any mission), the next step is to populate the newly created mission with user groups and users.
For a detailed discussion of the role-based access control model of prosEO please refer to
Using the Role-based Access Model. For the purpose of this tutorial it may suffice to say
that:
- Users are always created per mission. So the same username in a different mission technically speaking is a different user in prosEO. This implies that the passwords will be different, unless the users themselves keep them synchronized, and that privileges granted for one mission do not apply to any other mission.
- Users should not be granted privileges directly. Rather, create user groups representing certain roles in the project/mission, assign privileges to the group and make the users members of their appropriate groups.
- Low-level privileges are not overlapping (at least that is what we strive for). So having "manager" privileges (i.e. being able to create, update or delete a database object) does not entail having "reader" privileges on the same object. So as a general rule a user group with manager privileges for some database object class should also be assigned reader privileges for it.
- A general decision should be taken beforehand on whether personalized users shall be used, or functional user names. While the former is recommended from an information security perspective, the latter is often done to avoid problems regarding the storage and processing of personalized user data (with all its implications due to the EU's General Data Protection Regulation [GDPR] and other legal requirements), since user activities are logged to some extent in prosEO.
First a user to manage all other users needs to be created for the mission. Since the sysadm
user only has a very limited (!)
set of privileges, we need to create the user with all necessary permissions in one step:
prosEO (no mission)> user create --mission=PTM --identFile=usermgr.cred usermgr authorities=ROLE_MISSION_READER,ROLE_USERMGR,ROLE_CLI_USER,ROLE_GUI_USER
(I2701) User account usermgr created
prosEO (no mission)>
The option --identFile=<path to file>
allows us to specify the user credentials beforehand without resorting to the interactive
prompt of username and password. The credentials file is a two-line file containing the username in the first and the password
in the second line, like so:
usermgr
userMgr.456
Note: On Unix-like systems the permissions on the credentials file must be set to readable for the file owner only.
The authorities we grant to the user manager account give it a limited range of action: It is a single-purpose account, which only serves to create other users and user groups and assign the necessary privileges to them.
Now the remaining users for the prosEO Test Mission
can be created. The users are grouped into the following roles:
-
operator
: Creates and manages processing orders and monitors system performance -
archivist
: Manages product data, e.g. by uploading additional products and deleting outdated products -
engineer
: Performs configuration tasks on the system -
approver
: Special role to ensure the four-eyes principle when releasing large (and possibly costly) reprocessing orders into production -
prippublic
: Users (e.g. from the scientific community or external archives) who retrieve publicly available products from the PRIP API -
externalprocessor
: Accounts for systems performing additional processing tasks externally by downloading intermediate products withRESTRICTED
visibility and uploading the results to the system -
internalprocessor
: Accounts for the data processors controlled by prosEO, which may update product data and upload their processing results to the system
The users and groups are created with a CLI script, which can be fed into the CLI from the system command line by way of input
redirection. To simplify matters for the usage of the test mission, a multi-purpose user proseo
is created, which effectively
can do anything on the system except managing users:
login --identFile=usermgr.cred PTM
user create --identFile=proseo.cred proseo authorities=ROLE_CLI_USER,ROLE_GUI_USER,ROLE_PRIP_USER
user create --identFile=sciuserprip.cred sciuserprip
user create --identFile=cfidevprip.cred cfidevprip
user create --identFile=wrapper.cred wrapper
group create operator
group grant operator ROLE_MISSION_READER
group grant operator ROLE_PRODUCTCLASS_READER
group grant operator ROLE_PRODUCT_READER_ALL
group grant operator ROLE_PROCESSOR_READER
group grant operator ROLE_ORDER_READER
group grant operator ROLE_ORDER_MGR
group grant operator ROLE_ORDER_PLANNER
group grant operator ROLE_ORDER_MONITOR
group grant operator ROLE_FACILITY_READER
group grant operator ROLE_FACILITY_MONITOR
group create archivist
group grant archivist ROLE_MISSION_READER
group grant archivist ROLE_PRODUCTCLASS_READER
group grant archivist ROLE_PRODUCT_READER_ALL
group grant archivist ROLE_PRODUCT_INGESTOR
group grant archivist ROLE_PRODUCT_MGR
group grant archivist ROLE_FACILITY_READER
group grant archivist ROLE_FACILITY_MONITOR
group create engineer
group grant engineer ROLE_MISSION_READER
group grant engineer ROLE_MISSION_MGR
group grant engineer ROLE_PRODUCTCLASS_READER
group grant engineer ROLE_PRODUCTCLASS_MGR
group grant engineer ROLE_PRODUCT_READER_ALL
group grant engineer ROLE_PROCESSOR_READER
group grant engineer ROLE_PROCESSORCLASS_MGR
group grant engineer ROLE_CONFIGURATION_MGR
group grant engineer ROLE_FACILITY_READER
group grant engineer ROLE_FACILITY_MGR
group grant engineer ROLE_FACILITY_MONITOR
group grant engineer ROLE_ORDER_READER
group grant engineer ROLE_ORDER_MONITOR
group create approver
group grant approver ROLE_ORDER_READER
group grant approver ROLE_ORDER_APPROVER
group create prippublic
group grant prippublic ROLE_PRIP_USER
group grant prippublic ROLE_PRODUCT_READER
group create externalprocessor
group grant externalprocessor ROLE_PRIP_USER
group grant externalprocessor ROLE_PRODUCT_READER_RESTRICTED
group grant externalprocessor ROLE_PRODUCT_INGESTOR
group create internalprocessor
group grant internalprocessor ROLE_PRODUCT_GENERATOR
group grant internalprocessor ROLE_PRODUCT_INGESTOR
group grant internalprocessor ROLE_JOBSTEP_PROCESSOR
group add operator proseo
group add archivist proseo
group add engineer proseo
group add approver proseo
group add prippublic esaprip
group add externalprocessor knmiprip
group add internalprocessor wrapper
For the exact meaning of the privileges assigned to each user group please refer to Using the Role-based Access Model and the Data Model Reference.
Product classes and their dependencies, expressed as selection rules, are at the core of prosEO's flexible workflow engine, which creates loosely coupled workflows depending on the requirements of each single processing step instead of relying on explicitly modelled processing workflows (or "processing chains").
While in the field of earth observation the notion of different processing levels (ranging from L0 to L3 or sometimes L4) is common, where the processing of products for one level requires as input products of the next lower level, this model is at best of illustrative value for prosEO. Product classes can be assigned a certain processing level, but this is for informational purposes only and does not influence product processing control.
Let's consider a simple model, where products of different levels depend on one another for processing:
Figure 1: Product classes and selection rules
Here, four processing levels are present (L0 to L3), and additionally one type of auxiliary data is used (data from external sources, which is not derived from data collected by the mission’s satellite sensor):
-
PTM_L0
: Unprocessed instrument and payload data at full resolution. In our model there are no selection rules for products of this class, and no processor will be defined, so products of this class must be ingested into the system from some external source. In another configuration there may even be a L0 processor, which generates L0 products from raw satellite data. In our example Level 0 data comes in chunks of fixed length (20 min of sensing time). -
PTM_L1B
: Data derived from Level 0 at the instrument’s full time/space resolution, processed to sensor units. A product of this class consists of two component products of the classesPTM_L1B_P1
andPTM_L1B_P2
(think of P1 and P2 meaning "part 1" and "part 2", respectively). PTM_L1B is therefore called the enclosing class, PTM_L1B_P1 and PTM_L1B_P2 are its component classes. Level 1B data is generated per orbit in this example. -
PTM_L2_A
andPTM_L2_B
: Data derived from Level 1 data that have been processed to geophysical quantities of interest, say carbon monoxide and formaldehyde content of the atmosphere by geographic coordinate; for this example Level 2 data is kept at the same resolution as Level 1B data, i.e. products are generated per orbit. Note that PTM_L2_A and PTM_L2_B have different requirements as to their input data: PTM_L2_A requires the full Level 1B product (i.e. both parts), whereas for the generation of PTM_L2_B it is sufficient to provide PTM_L1B_P1. Selecting the needed component of a multi-component product only will reduce network transfer time and disk storage requirements and may even be compulsory for some data processors. -
PTM_L3
: Variables mapped on uniform space-time grid scales (resampling); in our example we use uniform product sizes of four hours of calendar time (it may be more common to use calendar days here, but restricting the size to four hours reduces the amount of test data we need to process).
As fig. 1 shows, the dependencies between the different product classes are expressed as selection rules. In this example we use three types of selection policies:
-
ValIntersect
: Select all input products with a sensing time period intersecting the sensing/validity time period of the product to generate. In the example this applies to the selection of L0 products for L1B processing, and of L2 products for L3 processing. This policy may be appropriate, if input products and intended output product differ in their validity times (usually because the input products cover a smaller proportion of sensing time than the output products). -
LatestValIntersect
: As above, but only select the newest input product (i.e. the one most recently generated); a selection policy common for selecting auxiliary data. -
LatestValCover
: Of all possible input products covering the full sensing/validity time period of the output product, select the one most recently generated. This policy is common when input and output product cover the (exactly) same time period, or when the input products cover a large time period compared to the time period of the output product (e.g. weather forecasts).
The full list of implemented selection policies can be found in the Data Model Reference, the authoritative source for selection rules is ESA’s Generic IPF Interface Specification (doc. ID MMFI-GSEG-EOPG-TN-07-0003, issue 1.8, 2009-08-03).
Selection policies may have parameters, which extend the selection interval to both sides of the sensing/validity time interval
of the output product. This is especially of interest for auxiliary products, whose validity periods may not be continuous (e.g.
the IERS-B bulletins, which are only released once per month, leading to an extension of the search interval for the ValIntersect
policy by 60 days to both sides the output product sensing time interval in our example).
Creation of a product class requires only a few attributes to be set:
-
productType
: The name of the product class (e.g.PTM_L1B
) -
processingLevel
: The processing level (optional, but recommended for output product classes of the mission; usually not set for auxiliary product classes) -
visibility
: Degree of visibility for products of this class in either the GUI or the PRIP (valid values arePUBLIC
,RESTRICTED
orINTERNAL
; see also the discussion of product class access privileges in section Setup Users and User Groups above)
The configuration of a product class in a JSON file is given below as an example (note that the additional attribute missionCode
is required to associate the product class with the correct mission):
{
"missionCode": "PTM",
"productType": "PTM_L1B",
"processingLevel": "L1B",
"visibility": "PUBLIC"
}
The specification of selection rules is best done in simple text files using Rule Language, e.g. for the PTM_L1B product class in our example (white space and line breaks increase readability):
FOR PTM_L0 SELECT ValIntersect(0, 0);
FOR AUX_IERS_B SELECT LatestValIntersect(60 D, 60 D)
A full specification of the language syntax can be found in the Javadoc for the SelectionRule class.
Given a file named PTM_L1B.json
for the product class definition and a file PTM_L1B_rule.txt
for the selection rules with
the content cited above, the PTM_L1B product class can be created by issuing the following commands in the CLI:
productclass create --file=testfiles/PTM_L1B.json
productclass rule create --file=testfiles/PTM_L1B_rule.txt --format=PLAIN PTM_L1B
Hint: When actually configuring a mission, it will be easier to create the processor classes (see below) first and the product classes later, because then the relationship between product class and processor class can also be established at the same time, thereby avoiding a separate update step later.
TODO