From 0de6500f25e812f130bf9a192e1d70ca0ed3e34e Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 15 Nov 2022 14:45:42 +0100 Subject: [PATCH 1/6] Jobs and services: Added `create_parameters` property in responses to define the parameters that can be submitted suring creation #276 --- CHANGELOG.md | 1 + openapi.yaml | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d922c7c..33975166 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) +- `GET /jobs` and `GET /services`: Added `create_parameters` property in responses to define the parameters that can be submitted when creating jobs or services. [#276](https://github.com/Open-EO/openeo-api/issues/276) - `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. [#453](https://github.com/Open-EO/openeo-api/issues/453) - `GET /jobs/{job_id}/results`: - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) diff --git a/openapi.yaml b/openapi.yaml index 7c7ac79d..28c5b17c 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2580,6 +2580,8 @@ paths: $ref: '#/components/schemas/service' links: $ref: '#/components/schemas/links_pagination' + create_parameters: + $ref: '#/components/schemas/processing_create_parameters' 4XX: $ref: '#/components/responses/client_error_auth' 5XX: @@ -2824,6 +2826,8 @@ paths: $ref: '#/components/schemas/batch_job' links: $ref: '#/components/schemas/links_pagination' + create_parameters: + $ref: '#/components/schemas/processing_create_parameters' 4XX: $ref: '#/components/responses/client_error_auth' 5XX: @@ -5687,6 +5691,22 @@ components: "foo" but cannot be 1. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.10.2). allOf: - $ref: '#/components/schemas/json_schema' + processing_create_parameters: + title: Creation Parameters + description: |- + List of additional custom parameters that a back-end offers during the creation + of batch jobs (`POST /jobs`) and secondary web services (`POST /services`) respectively. + type: array + items: + $ref: '#/components/schemas/parameter' + example: + - name: memory + description: Maximum amount of memory that will be allocated for processing, in gigabytes. + optional: true + default: 32 + schema: + type: integer + minimum: 1 error: title: General Error description: >- From e3265f775e7c0ce13649ca1293ec827e3e754e27 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 16 Nov 2022 11:40:20 +0100 Subject: [PATCH 2/6] Add processing options to a separate endpoint #276 --- CHANGELOG.md | 2 +- openapi.yaml | 39 +++++++++++++++++++++++++++++++++++---- 2 files changed, 36 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 33975166..19d9a009 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) -- `GET /jobs` and `GET /services`: Added `create_parameters` property in responses to define the parameters that can be submitted when creating jobs or services. [#276](https://github.com/Open-EO/openeo-api/issues/276) +- Added `GET /processing_options` to define the parameters that can be submitted when creating jobs or services. [#276](https://github.com/Open-EO/openeo-api/issues/276) - `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. [#453](https://github.com/Open-EO/openeo-api/issues/453) - `GET /jobs/{job_id}/results`: - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) diff --git a/openapi.yaml b/openapi.yaml index e2779d58..acf43aa6 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2386,6 +2386,41 @@ paths: $ref: '#/components/responses/client_error_auth' 5XX: $ref: '#/components/responses/server_error' + /processing_options: + get: + summary: Additional processing options + operationId: list-processing-options + description: |- + Lists additional custom processing options that a back-end offers for the + different processing modes (synchronous processing, batch jobs, secondary web services). + + The options specified here can be added to the corresponding `POST` requests + at the top-level of the object sent in the payload. + tags: + - Capabilities + - Data Processing + - Batch Jobs + - Secondary Services + security: + - {} + - Bearer: [] + responses: + '200': + description: >- + An object with a list of parameters per processing mode. + content: + application/json: + schema: + description: Processing parameters per processing mode. + type: object + properties: + create_job_parameters: + $ref: '#/components/schemas/processing_create_parameters' + create_service_parameters: + $ref: '#/components/schemas/processing_create_parameters' + create_synchronous_parameters: + $ref: '#/components/schemas/processing_create_parameters' + /service_types: get: summary: Supported secondary web service protocols @@ -2582,8 +2617,6 @@ paths: $ref: '#/components/schemas/service' links: $ref: '#/components/schemas/links_pagination' - create_parameters: - $ref: '#/components/schemas/processing_create_parameters' 4XX: $ref: '#/components/responses/client_error_auth' 5XX: @@ -2830,8 +2863,6 @@ paths: $ref: '#/components/schemas/batch_job' links: $ref: '#/components/schemas/links_pagination' - create_parameters: - $ref: '#/components/schemas/processing_create_parameters' 4XX: $ref: '#/components/responses/client_error_auth' 5XX: From 514b35b739bfb3e4b5953a4df6b870e75dd75320 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 28 Nov 2022 15:41:34 +0100 Subject: [PATCH 3/6] Options should be optional --- openapi.yaml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index 5b8865a1..a23d9d2d 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2478,7 +2478,9 @@ paths: different processing modes (synchronous processing, batch jobs, secondary web services). The options specified here can be added to the corresponding `POST` requests - at the top-level of the object sent in the payload. + at the top-level of the object that is sent as the payload. + All options SHOULD explicitly be made optional with reasonable defaults + as otherwise the interoperability between the implementations decreases. tags: - Capabilities - Data Processing From 830f9c2fcaa9a6d75ccb190cccba630d4a294260 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 18 Dec 2024 16:40:35 +0100 Subject: [PATCH 4/6] Move to an extension --- CHANGELOG.md | 2 +- README.md | 1 + .../processing-parameters/.spectral.yml | 12 +++ extensions/processing-parameters/README.md | 11 +++ extensions/processing-parameters/openapi.yaml | 98 +++++++++++++++++++ extensions/processing-parameters/package.json | 24 +++++ openapi.yaml | 53 ---------- 7 files changed, 147 insertions(+), 54 deletions(-) create mode 100644 extensions/processing-parameters/.spectral.yml create mode 100644 extensions/processing-parameters/README.md create mode 100644 extensions/processing-parameters/openapi.yaml create mode 100644 extensions/processing-parameters/package.json diff --git a/CHANGELOG.md b/CHANGELOG.md index db2cb135..7e1204f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **New extensions:** - [Remote Process Definition Extension](./extensions/remote-process-definition/README.md) + - [Processing Parameters Extension](./extensions/processing-parameters/README.md) - Added `version` property to `GET /processes` [#517](https://github.com/Open-EO/openeo-api/issues/517) ### Fixed @@ -33,7 +34,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /file_formats`: Add `pointcloud` to the `gis_data_types`. [#475](https://github.com/Open-EO/openeo-api/issues/475) - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) -- Added `GET /processing_options` to define the parameters that can be submitted when creating jobs or services. [#276](https://github.com/Open-EO/openeo-api/issues/276) - `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. [#453](https://github.com/Open-EO/openeo-api/issues/453) - `GET /jobs/{job_id}/results`: - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) diff --git a/README.md b/README.md index 9917cdfa..f677fee1 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/O | ------------------------------------------------------------------ | ------- | ------------ | ----------- | | [Commercial Data](extensions/commercial-data/) | 0.1.0 | experimental | Provides an interface for discovering, ordering and using commercial data. | | [Federation](extensions/federation/) | 0.1.0 | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. | +| [Processing Parameters](extensions/processing-parameters/) | 0.1.0 | experimental | Explore and handle additional processing parameters that a back-end can offer for the processing modes (sync. processing, batch jobs, web services). | | [Remote Process Definition](extensions/remote-process-definition/) | 0.1.0 | experimental | Load user-defined processes that are hosted externally through the process namespace into process graphs. | ## Repository diff --git a/extensions/processing-parameters/.spectral.yml b/extensions/processing-parameters/.spectral.yml new file mode 100644 index 00000000..ad10553b --- /dev/null +++ b/extensions/processing-parameters/.spectral.yml @@ -0,0 +1,12 @@ +extends: "spectral:oas" +rules: + contact-properties: true + tag-description: true + oas3-parameter-description: true + oas3-unused-component: true + operation-id-kebab-case: + given: "$..operationId" + then: + function: pattern + functionOptions: + match: ^[a-z][a-z0-9\-]*$ \ No newline at end of file diff --git a/extensions/processing-parameters/README.md b/extensions/processing-parameters/README.md new file mode 100644 index 00000000..68acc6af --- /dev/null +++ b/extensions/processing-parameters/README.md @@ -0,0 +1,11 @@ +# Processing Parameters Extension + +The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing options that a back-end can offer for the three processing modes (synchronous processing, batch jobs, secondary web services). + +- Version: **0.1.0** +- Stability: **experimental** +- [OpenAPI document](openapi.yaml) +- Conformance class: `https://api.openeo.org/extensions/processing-parameters/0.1.0` + +**Note:** This document only documents the additions to the specification. +Extensions can not change or break existing behavior of the openEO API. diff --git a/extensions/processing-parameters/openapi.yaml b/extensions/processing-parameters/openapi.yaml new file mode 100644 index 00000000..b19b6c7a --- /dev/null +++ b/extensions/processing-parameters/openapi.yaml @@ -0,0 +1,98 @@ +openapi: 3.0.2 +info: + title: openEO API - Processing Parameters Extension + version: 0.1.0 + description: |- + The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing options that a back-end can offer for the three processing modes (synchronous processing, batch jobs, secondary web services). + contact: + name: openEO Consortium + url: 'https://openeo.org' + email: openeo.psc@uni-muenster.de + license: + name: Apache 2.0 + url: 'http://www.apache.org/licenses/LICENSE-2.0.html' +externalDocs: + url: https://github.com/Open-EO/openeo-api/blob/draft/extensions/workspaces/README.md +tags: + - name: Capabilities + description: General information about the API implementation and other supported capabilities provided by the back-end. + - name: Data Processing + description: Organizes and manages data processing on the back-end, either as synchronous on-demand computation or batch jobs. + - name: Batch Jobs + description: Management of batch processing tasks (jobs) and their results. + - name: Secondary Services + description: On-demand access to data using other web service protocols. +servers: + - url: 'https://openeo.example/api/{version}' + description: >- + The URL of the API MAY freely be chosen by the back-end providers. The + path, including API versioning, is a *recommendation* only. Nevertheless, + all servers MUST support HTTPS as the authentication methods are not + secure with HTTP only! + variables: + version: + default: v1 + description: >- + API versioning is RECOMMENDED. As the openEO API is following + [SemVer](https://semver.org/) only the MAJOR part of the stable + version numbers (i.e. versions >= 1.0.0) SHOULD be used for API + versioning in the URL. The reason is that backward-incompatible + changes are usually introduced by major changes. Therefore, the + version number in the URL MUST not be used by the clients to detect + the version number of the API. Use the version number returned from + `GET /` instead. +paths: + /processing_options: + get: + summary: Additional processing options + operationId: list-processing-options + description: |- + Lists additional custom processing options ("additional back-end specific properties" in the API specification) + that a back-end offers for the different processing modes (synchronous processing, batch jobs, secondary web services). + The options specified here can be added to the corresponding `POST` requests at the top-level of the object that is sent as the payload. + All options SHOULD explicitly be made optional with reasonable defaults as otherwise the interoperability between the implementations decreases. + tags: + - Capabilities + - Data Processing + - Batch Jobs + - Secondary Services + security: + - {} + - Bearer: [] + responses: + '200': + description: >- + An object with a list of parameters per processing mode. + content: + application/json: + schema: + description: Processing parameters per processing mode. + type: object + properties: + create_job_parameters: + $ref: '#/components/schemas/processing_create_parameters' + create_service_parameters: + $ref: '#/components/schemas/processing_create_parameters' + create_synchronous_parameters: + $ref: '#/components/schemas/processing_create_parameters' +components: + schemas: + processing_create_parameters: + title: Creation Parameters + description: |- + List of additional custom parameters that a back-end offers during the creation + of batch jobs (`POST /jobs`) and secondary web services (`POST /services`) respectively. + type: array + items: + $ref: '../../openapi.yaml#/components/schemas/parameter' + example: + - name: memory + description: Maximum amount of memory that will be allocated for processing, in gigabytes. + optional: true + default: 32 + schema: + type: integer + minimum: 1 + securitySchemes: + Bearer: + $ref: '../../openapi.yaml#/components/securitySchemes/Bearer' diff --git a/extensions/processing-parameters/package.json b/extensions/processing-parameters/package.json new file mode 100644 index 00000000..841560a5 --- /dev/null +++ b/extensions/processing-parameters/package.json @@ -0,0 +1,24 @@ +{ + "name": "@openeo/api-extension-processing-parameters", + "version": "0.1.0", + "author": "openEO Consortium", + "license": "Apache-2.0", + "description": "The openEO API specification.", + "homepage": "https://openeo.org", + "bugs": { + "url": "https://github.com/Open-EO/openeo-api/issues" + }, + "repository": { + "type": "git", + "url": "git+https://github.com/Open-EO/openeo-api.git" + }, + "devDependencies": { + "@stoplight/spectral-cli": "^6.6.0", + "redoc-cli": "^0.13.21" + }, + "scripts": { + "start": "redoc-cli serve openapi.yaml --watch --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", + "build": "redoc-cli bundle openapi.yaml -o redoc.html --title \"openEO API - Processing Parameters Extension\" --cdn --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", + "test": "spectral lint openapi.yaml" + } +} diff --git a/openapi.yaml b/openapi.yaml index a23d9d2d..52313981 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2469,43 +2469,6 @@ paths: $ref: '#/components/responses/client_error_auth' 5XX: $ref: '#/components/responses/server_error' - /processing_options: - get: - summary: Additional processing options - operationId: list-processing-options - description: |- - Lists additional custom processing options that a back-end offers for the - different processing modes (synchronous processing, batch jobs, secondary web services). - - The options specified here can be added to the corresponding `POST` requests - at the top-level of the object that is sent as the payload. - All options SHOULD explicitly be made optional with reasonable defaults - as otherwise the interoperability between the implementations decreases. - tags: - - Capabilities - - Data Processing - - Batch Jobs - - Secondary Services - security: - - {} - - Bearer: [] - responses: - '200': - description: >- - An object with a list of parameters per processing mode. - content: - application/json: - schema: - description: Processing parameters per processing mode. - type: object - properties: - create_job_parameters: - $ref: '#/components/schemas/processing_create_parameters' - create_service_parameters: - $ref: '#/components/schemas/processing_create_parameters' - create_synchronous_parameters: - $ref: '#/components/schemas/processing_create_parameters' - /service_types: get: summary: Supported secondary web service protocols @@ -5883,22 +5846,6 @@ components: "foo" but cannot be 1. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.10.2). allOf: - $ref: '#/components/schemas/json_schema' - processing_create_parameters: - title: Creation Parameters - description: |- - List of additional custom parameters that a back-end offers during the creation - of batch jobs (`POST /jobs`) and secondary web services (`POST /services`) respectively. - type: array - items: - $ref: '#/components/schemas/parameter' - example: - - name: memory - description: Maximum amount of memory that will be allocated for processing, in gigabytes. - optional: true - default: 32 - schema: - type: integer - minimum: 1 error: title: General Error description: >- From da2eefee834f7e56165331c65440e6e90f2bac29 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 18 Dec 2024 18:17:38 +0100 Subject: [PATCH 5/6] Add UDP extension --- extensions/processing-parameters/README.md | 45 +++++++++++++++++++ extensions/processing-parameters/openapi.yaml | 8 ++-- 2 files changed, 50 insertions(+), 3 deletions(-) diff --git a/extensions/processing-parameters/README.md b/extensions/processing-parameters/README.md index 68acc6af..ca18becd 100644 --- a/extensions/processing-parameters/README.md +++ b/extensions/processing-parameters/README.md @@ -9,3 +9,48 @@ The Processing Parameters Extension to the openEO API provides an interface to e **Note:** This document only documents the additions to the specification. Extensions can not change or break existing behavior of the openEO API. + +This extension adds a new endpoint (`GET /processing_options`, see [OpenAPI document](openapi.yaml)) +to discover the additional processing options that a back-end offers. + +Additionally, this extension allows to provide specific default values for user-defined processes (UDPs, see below). + +In both cases, the parameters and its values are provided separately for each processing mode. + +## Processing Parameters in UDPs + +UDPs can provide default values for specific processing parameters. + +The values for each parameter are provided separately for each processing mode. +The following properties are added to the top-level of a UDP for the respective processing modes: + +- `default_synchronous_parameters` for synchronous processing +- `default_job_parameters` for batch jobs +- `default_service_parameters` for secondary web services + +The schema for each of these properties is: + +```yaml +type: object +additionalProperties: + description: Any type +``` + +The keys of the object are the respecive parameter names. +The values of the object are the (default) values for the parameters. +Schematic restrictions are not defined for the object, but the schemas for the parameters as defined in `GET /processing_options` apply to the given values. +These values provide the defaults unless a user overrides them in the actual data processing request (e.g. `POST /jobs`, see below). + +## Resolving parameters + +Due to the variety of places where processing parameters can be provided, the following +list defines how the parameters must be resolved. The prioritization is as follows: + +1. If present, use the parameter specified in the processing request directly (e.g. in `POST /jobs` as a top-level property) +2. If present, use the default parameter specified in the UDP +3. Otherwise, use the default value for the parameter as specified in `GET /processing_options` + +"Present" means that the property is present in the JSON representation regardless of the value given, i.e. +properties are present if an empty string, an empty array, an empty object, `false`, `0`, or `null` are provided. + +Unrecognized/unknown parameters that are provided through UDPs must be ignored by backends. diff --git a/extensions/processing-parameters/openapi.yaml b/extensions/processing-parameters/openapi.yaml index b19b6c7a..450166ea 100644 --- a/extensions/processing-parameters/openapi.yaml +++ b/extensions/processing-parameters/openapi.yaml @@ -2,8 +2,10 @@ openapi: 3.0.2 info: title: openEO API - Processing Parameters Extension version: 0.1.0 - description: |- - The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing options that a back-end can offer for the three processing modes (synchronous processing, batch jobs, secondary web services). + description: >- + The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing options that a back-end can offer for the three processing modes (synchronous processing, batch jobs, and secondary web services). + The openEO API specification allows back-ends to accept "additional back-end specific properties" for each of the processing modes. + As the openEO API specification does not define the name and schemas of these properties, this extension provides a standardized way to define and document these properties. contact: name: openEO Consortium url: 'https://openeo.org' @@ -47,7 +49,7 @@ paths: summary: Additional processing options operationId: list-processing-options description: |- - Lists additional custom processing options ("additional back-end specific properties" in the API specification) + Lists additional custom processing options that a back-end offers for the different processing modes (synchronous processing, batch jobs, secondary web services). The options specified here can be added to the corresponding `POST` requests at the top-level of the object that is sent as the payload. All options SHOULD explicitly be made optional with reasonable defaults as otherwise the interoperability between the implementations decreases. From 917a3fdb28987bbfcd1169f402fb41a8ab828eda Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 6 Jan 2025 16:51:38 +0100 Subject: [PATCH 6/6] Apply suggestions from code review Co-authored-by: Stefaan Lippens --- extensions/processing-parameters/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/extensions/processing-parameters/README.md b/extensions/processing-parameters/README.md index ca18becd..8b822e5f 100644 --- a/extensions/processing-parameters/README.md +++ b/extensions/processing-parameters/README.md @@ -36,8 +36,8 @@ additionalProperties: description: Any type ``` -The keys of the object are the respecive parameter names. -The values of the object are the (default) values for the parameters. +The keys of the object are the respective parameter names. +The values of the object are the default values for the parameters. Schematic restrictions are not defined for the object, but the schemas for the parameters as defined in `GET /processing_options` apply to the given values. These values provide the defaults unless a user overrides them in the actual data processing request (e.g. `POST /jobs`, see below).