diff --git a/propagation/enhancement.md b/propagation/enhancement.md index 13ff752..126c299 100644 --- a/propagation/enhancement.md +++ b/propagation/enhancement.md @@ -1,12 +1,74 @@ # Algorithm Enhancement Service -The Algorithm Enhancement Service starts with an assessment of your existing work, leveraging automated benchmarks and expert review -to enhance efficiency while maintaining the same quality within acceptable error bounds for your application. +The algorithm enhancement service helps scientists and algorithm developers to transform their research code into a +performant and stable piece of software that can be run in an operational setting. +These enhancement activities ensure code quality, maintainability and performance of your algorithm. +To ensure performant execution on a NoR processing platform which offers openEO or Application Package algorithms, +additional optimization dedicated for the platform capabilities may be needed. +The exact requirements, steps, and effort involved in this process can vary heavily from case to case, due to the +variations in initial implementations and operational requirements. -This service always require an approval from the project officer to start an initial assessment of your work and the -feasibility of enhancement within a reasonable timeframe. Therefore, the more detailed input you provide in terms of code and -algorithm description, the more accurate the assessment will be, and the more likely it is that the service can be offered. -Applications that cannot be sufficiently understood within the assessment period, may be considered too risky for enhancement. +The code of the enhanced version of the algorithm will be published in a dedicated APEx GitHub repository and algorithm +providers are encouraged to use it as a basis for further development of the algorithm. +This way, the version hosted on APEx can be kept up to date. -It's important to note that this service focuses on improving already good and promising algorithms. Therefore, provide -a compelling motivation for why your algorithm deserves the additional attention and effort. +The algorithm enhancement service has two phases. In the first phase, the algorithm provider submits their existing +implementation, including source code, input data, and instructions to build, install and execute their code, together +with details of any external dependencies to the enhancement team. The enhancement team will then analyze the algorithm +and estimate the effort for the service, which results in an offer. In the second phase, on acceptance of the offer by +the provider and by ESA, the enhancement is performed. +As a result, the service user receives an optimized version of their algorithm code with high code quality. +Optionally, the enhancement procedure includes a cloudification step, where the algorithm is implemented as an openEO +User Defined Process (UDP) or Application Package. This makes it possible to host the algorithm on an NoR platform and +provide it as an on-demand service. + +Complementary to this service, the [algorithm hosting service](./hosting.md) is responsible for hosting the algorithms +prepared for hosting by the algorithm enhancement service. On the other side, +the [toolbox cloudification service](./toolboxcloud.md) offers a similar set of features aimed at algorithms that are +already available as part of standalone software packages (toolboxes) and makes them available as on-demand +APEx-compatible algorithms. + +#### Features of the Algorithm Enhancement Service + +The algorithm enhancement service offers the following features: + +- Analysis of current algorithm implementation and performance. +- Optimization of the algorithm with respect to computational efficiency. + This may include switching to a distributed execution model and improving code quality. +- Optimization considering the capabilities of target processing platforms. +- Exploitation of algorithms available on APEx where appropriate, e.g. replacing custom functions with equivalent + pre-optimized openEO functions. +- Delivery of an enhanced implementation of the algorithm that is ready to be listed in the APEx algorithm catalogue. + +#### How to use the Algorithm Enhancement Service + +To use the service, the first step is to deliver the current algorithm implementation, together with the complementary +information that is required to make use of it. The necessary components are: + +- the source code +- documentation of dependencies +- build, installation and execution instructions +- input data +- expected outputs + +Once this information has been provided, the enhancement team will perform a two-phase analysis. The analysis consists +of a briefer, standardized analysis phase, followed by a more thorough cost and effort estimate. On acceptance, the +enhancement team will begin work to enhance and integrate the algorithm on APEx. The exact process of enhancement +depends on the nature of the initial algorithm and its implementation. Depending on these conditions, the enhanced +algorithm is integrated into APEx either as an openEO UDP or as an OGC Application Package. Application Packages are +based upon one or more container images implementing the algorithm in a custom environment. + +If the algorithm was originally implemented in accordance with the APEx compliance guidelines, the process can be more +straightforward. In this case there will be more overlap between the libraries and design patterns used in the initial +implementation and those required for an optimized APEx version, and the enhancement process will be different and fewer +changes have to be made to the code base. + + +#### Intellectual Property Rights + +The intellectual property rights to their algorithm implementation remain with the algorithm authors. +If the algorithm is an implementation of published research, the APEx team reserves itself the non-exclusive +right to run and further develop the enhanced implementation. The algorithm provider has the right and is encouraged +to use the enhanced code as a basis for further algorithm development. +For proprietary algorithms, the rights to publish the source code and run the algorithm remain with the original +property rights holder. \ No newline at end of file diff --git a/propagation/toolboxcloud.md b/propagation/toolboxcloud.md index 5f23c88..631bf6c 100644 --- a/propagation/toolboxcloud.md +++ b/propagation/toolboxcloud.md @@ -1,10 +1,55 @@ # Toolbox Cloudification Service -The Toolbox Cloudification Service focuses on porting functionality from existing (open source) toolboxes into the EO platform -targeted by your project. This service is designed to enhance the scalability and integration of your toolbox's features -within an APEx compatible cloud environment. +The toolbox cloudification service makes individual data processing procedures from existing toolboxes available as +cloud-based algorithms, compliant with APEx. +This makes it possible to freely combine functionalities from multiple toolboxes as well as other algorithms registered +through APEx. +The service focuses solely on data-processing capabilties of toolboxes and does +not include visualization features. +Instead of a file-centric and local processing scheme of a traditional toolbox, key operations can be performed in the +cloud where EO data archives are directly available as data cubes, where arbitrary extends in time and space can be +extracted. Furthermore, functions from various toolboxes can be combined in a single workflow through APEx. -The process begins with an assessment of your request to estimate the effort required to cloudify the toolbox. The main -advantage of this approach is that the specific functionality will also become available as a modular building block for -other projects. Therefore, it is crucial to provide a compelling justification for why this feature deserves to be 'cloudified'. +Service users may select functionalities from a toolbox that they would like to have access to as APEx-compliant +algorithms. The proposed functionalities and their toolbox will then be reviewed by ESA and APEx experts. An effort +estimation and an offer will be made following the review. On acceptance of the offer, the requested functionalities +will be provided as an Application Package according to OGC best practice, or as an openEO User Defined Process (UDP). +To ensure interoperability between procedures from different toolboxes, it is necessary to share data using standard +data formats, writing intermediary files to disk. Users should consider if the flexibility of this approach outweighs +the performance drawbacks and request deployment as a single, integrated package instead, if performance is critical. + +Functionalities made available through the toolbox cloudification service are hosted via the +APEx [algorithm hosting service](./hosting.md). A similar set of features is offered by +the [algorithm enhancement service](./enhancement.md), which focuses on improving maturity and performance of algorithms +and preparing them for hosting on APEx. + +#### Features of the Toolbox Cloudification Service + +The toolbox cloudification service offers an analysis of the implementation of the toolbox and the requested processing +capabilities. On the basis of this analysis, an implementation strategy is proposed and executed to provide the +functionality outside the context of the toolbox as a fully cloud-integrated APEx compliant algorithm. + +Projects may benefit from the toolbox cloudification service by bringing key functions from traditional desktop EO +toolboxes into the cloud, where they can be integrated into large-scale processing workflows. +Toolboxes evolve over time. If a toolbox migrated to APEx, it shall be kept up to date. This requires maintenance. +The procedure to support the evolution of toolboxes is yet to be defined. + +#### How to use the Toolbox Cloudification Service + +In order to use the toolbox cloudification service, a proposal specifies a candidate toolbox and selects key +functionality for cloudification. This proposal is then submitted to the APEx team. It should contain a short +justification explaining the need for the selected functionality. ESA reviews the proposal and submits selected +proposals to the APEx team for analysis and price offer. + +In a first phase, the APEx team makes an initial analysis of the candidate toolbox and an assessment of the associated +proposal. On this basis, the team provides an assessment report to ESA. The report includes a final recommendation of +the key functionality to be cloudified, a description of the specific cloudification approach, and an effort and cost +estimation. ESA decides whether to accept the offer. + +On acceptance, a team of suitably qualified APEx experts undertakes the cloudification process itself, recasting or +reimplementing the required functionality in cloud-hosted toolbox modules. In order for the cloudified modules to be +hosted through APEx, they will be implemented on one of the APEx compliant technologies, such as openEO UDPs and/or +Application Packages. The more suitable variant depends largely on the original implementation of the toolbox +functionality. The newly created modules will be deployed through the APEx [algorithm hosting service](./hosting.md) and +thus become available for direct use and integration into larger cloud-hosted processing workflows.