Skip to content

Deployment Guide_Cluster

Jump to bottom
Katharina Bassler edited this page Jun 28, 2023 · 3 revisions

Cluster Deployment Guide

Purpose of the deployment guide

This guide aims to assist in setting up and deploying prosEO, an Earth observation data processing software. It provides a high-level outline of the steps necessary for deployment, from system requirements to troubleshooting.

prosEO is a modular, scalable, and distributed processing software specifically designed to handle large volumes of Earth observation data. Deploying prosEO requires the setup and configuration of various microservices and system components.

System Requirements

The (virtual) machines that will be set up during the deployment process need to at least fulfill the following requirements:

  • Linux installed (e.g. CentOS 7)
  • Ability to run Docker and Kubernetes (i. e. standard CentOS kernel, no provider-specific builds) Note: Docker and Kubernetes do not need to be installed manually!

Suggested sizing for a small to medium installation:

  • CPUs: Bastion host and Kubernetes master 4 CPUs each, worker nodes 8-16 CPUs each (depending on the processors to be run)
  • RAM: Bastion host and Kubernetes master 8-16 GB each, worker nodes 32 GB or more (again depending on the processors)
  • Disk: Bastion host 40 GB system; NFS server 10 TB data (or more, we are talking earth observation satellite data here!); Kubernetes master 40 GB system; worker nodes 40 GB system, 80 GB Docker library and scratch working area (depending on processor needs)
  • Optional: Access to some object storage conforming to the AWS S3 protocol (if not, more disk space will be needed for the NFS server)

Network Configuration: A reliable, high-speed internet connection is required. Proper network configurations, including firewall and security settings, should be ensured.

Pre-Deployment Checklist

Before deploying prosEO:

  • Set up the infrastructure as described in deploy/infrastructure/README.md. As of now, instructions are available for IONOS and OpenStack (to be uploaded) environments. For other cloud flavors, the provided scripts and configuration files need to be adapted accordingly.
  • Verify that you have SSH access to all created virtual machines.

Installation and Configuration Instructions for prosEO Microservices

Follow the steps below to deploy and configure the various prosEO microservices:

  1. Create the Docker images with appropriate configurations for the prosEO microservices (see deploy/proseo-images/README.md).
  2. Configure the bastion host for the prosEO Control Instance, (see deploy/bastion-control/README.md).
  3. Configure and start the Kubernetes cluster (the "hands", see deploy/hands/README.md).
  4. Configure the prosEO Control Instance (the "brain", see deploy/brain/README.md).
  5. Configure the bastion host for the prosEO PRIP (see deploy/bastion-prip.README.md).
  6. Configure the NFS server (see deploy/nfs-server/README.md).
  7. Configure the logging and monitoring host (see deploy/loghost/README.md).
  8. Start the brain (see deploy/README.md).
  9. Start the Storage Manager (see deploy/README.md).
  10. Start the InfluxDB and the Grafana server (see deploy/README.md).

Administrative access usually is through the bastion host for the control instance, since all inner nodes are reachable from there. User access is through the bastion host for the prosEO PRIP.

Deployment Testing

Ensure the successful deployment by conducting appropriate tests. A minimum recommendation are:

Troubleshooting and Support

For any issues encountered during the deployment, refer to the deployment FAQ. For further technical support, contact [email protected].

Reference Information

Tutorials

Clone this wiki locally