Merge pull request #735 from MetaCell/feature/CH-100

CH-100 Add preliminary docker-compose support
MetaCell · Sep 17, 2024 · 3def2a6 · 3def2a6
2 parents 7761fe5 + 8bb9a1b
commit 3def2a6
Show file tree

Hide file tree

Showing 371 changed files with 17,340 additions and 34,784 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -19,4 +19,5 @@ __pycache__
 /application-templates
 /deployment-configuration
 /cloud-harness
-.openapi-generator
+.openapi-generator
+docker-compose.yaml
diff --git a/.gitignore b/.gitignore
@@ -7,6 +7,8 @@ node_modules
 .coverage
 *.DS_Store
 deployment/helm
+deployment/compose
+docker-compose.yaml
 *.egg-info
 *.idea
 /build

diff --git a/README.md b/README.md
@@ -2,12 +2,12 @@
 <img src="https://github.com/MetaCell/cloud-harness/blob/develop/cloudharness.png?raw=true" alt="drawing" width="200"/>
 </p>
 
-CloudHarness is a base infrastructure facilitator for microservice based applications deployed on Kubernetes.
+CloudHarness is a base infrastructure facilitator for microservice based applications deployed primarily on Kubernetes.
 Can scaffold and maintain your cloud solution on top of Cloudharness without writing
 Kubernetes templates, with in place common utilities and applications already configured for you.
 
 What building your cloud solution with CloudHarness gives to you:
-- Common framework and utilities to develop and deploy micro-service application 
+- Common framework and utilities to develop and deploy micro-service application
   - Helm chart automatic generation
     - deployments
     - services
@@ -17,6 +17,12 @@ What building your cloud solution with CloudHarness gives to you:
     - access gatekeepers configuration
     - secrets
     - templated config maps from files
+  - Docker compose configuration generation
+    - services
+    - traefik configuration
+    - databases (postgreql)
+    - access gatekeepers configuration
+    - secrets and configmaps
   * Automatic build and push of images
   * REST-API scaffolding building based on OpenApi
   * Continuous deployment script generation
@@ -46,17 +52,18 @@ In particular, these questions may rise:
  - How to manage databases without being locked to a specific vendor solution?
  - How to perform database backups?
  - How to manage secret data?
- - What about having a precounfigured account management application? 
- - Sooner rather than later I'll need an orchestration queue. Why not have that just ready to use? 
+ - What about having a precounfigured account management application?
+ - Sooner rather than later I'll need an orchestration queue. Why not have that just ready to use?
 
 # Command line tools
 
 CloudHarness provides the following command line tools to help application scaffolding and deployment.
 
-* `harness-deployment` - generate the helm chart to deploy on Kubernetes. 
+* `harness-deployment` - generate the helm chart to deploy on Kubernetes.
 * `harness-application` - create a new CloudHarness REST application.
 * `harness-generate` - generates server and client code for all CloudHarness REST applications.
 * `harness-test` - run end to end tests
+
 # Get started
 
 ## Prerequisites
@@ -67,13 +74,13 @@ Cloudharness can be used on all major operative systems.
 - Linux: supported and tested
 - MacOS: supported and tested
 - Windows/WSL2: supported and tested
-- Windows native: mostly working, unsupported  
+- Windows native: mostly working, unsupported
 
 ### Python
 Python 3.9 must be installed.
 
 It is recommended to setup a virtual environment.
-With conda: 
+With conda:
 ```bash
 conda create --name ch python=3.12
 conda activate ch
@@ -94,6 +101,10 @@ conda activate ch
 
 [Skaffold](https://skaffold.dev/docs/install/) is the way to go to build and debug your application in your local development environment.
 
+### Docker compose
+
+[Docker Compose](https://docs.docker.com/compose/) is required if the docker compose system is the target (instead of Kubernetes).
+
 ### Node environment
 
 A node environment with npm is required for developing web applications and to run end to end tests.
@@ -126,24 +137,25 @@ To (re)generate the code for your applications, run `harness-generate` from the
 The script will look for all openapi applications, and regenerate the Flask server code and documentation.
 Note: the script will eventually override any manually modified file. To avoid that, define a file openapi-generator-ignore.
 
-# Extend CloudHarness to build your solution
-CloudHarness is born to be extended. In order to extend CloudHarness you just need to mirror the folder structure:
-* **applications**: place here your custom applications, or override default ones
-* **deployment-configuration**: override the helm chart default values and templates
-* **infrastructure**: define base images to use in your application
+# Extend CloudHarness to build your project
+
+CloudHarness is born to be extended. 
+
+The quickest way to start is to install Cloud Harness, copy the *blueprint* folder and build from that with the cli tools, such as
+`harness-application`, `harness-generate`, `harness-deployment`.
 
-or simply copy the *blueprint* folder.
+See the [developers documentation](docs/dev.md#start-your-project) for more information.
 
 # Build and deploy
 
 The script `harness-deployment` scans your applications and configurations to create the build and deploy artifacts.
 Created artifacts include:
- - Helm chart
+ - Helm chart (or docker compose configuration file)
  - Skaffold build and run configuration
  - Visual Studio Code debug and run configuration
  - Codefresh pipeline yaml specification (optional)
 
-With your solution folder structure looking like
+With your project folder structure looking like
 
 ```
 applications
@@ -152,7 +164,7 @@ infrastructure
 cloud-harness
 ```
 
-run 
+run
 
 ```
 harness-deployment cloud-harness . [PARAMS]

diff --git a/application-templates/flask-server/backend/requirements.txt b/application-templates/flask-server/backend/requirements.txt
@@ -1,5 +1,7 @@
 connexion[swagger-ui]==2.14.2
-swagger-ui-bundle >= 0.0.2
+Flask == 2.2.5
+swagger-ui-bundle==0.0.9
 python_dateutil >= 2.6.0
 setuptools >= 21.0.0
-Flask<3.0.0
+
+
diff --git a/application-templates/flask-server/backend/setup.py b/application-templates/flask-server/backend/setup.py
@@ -14,11 +14,12 @@
 # http://pypi.python.org/pypi/setuptools
 
 REQUIRES = [
-    "connexion>=2.0.2",
-    "swagger-ui-bundle>=0.0.2",
+    "connexion[swagger-ui]<3.0.0",
+    "Flask>=2.2.5",
     "python_dateutil>=2.6.0",
     "pyjwt>=2.6.0",
-    "cloudharness"
+    "swagger-ui-bundle>=0.0.2",
+    "cloudharness",
 ]
 
 setup(

diff --git a/applications/events/deploy/templates-compose/deployments.yaml b/applications/events/deploy/templates-compose/deployments.yaml
@@ -0,0 +1,97 @@
+{{- define "events.deployment" }}
+events:
+  networks:
+    - ch
+  image: solsson/kafka:2.3.0@sha256:b59603a8c0645f792fb54e9571500e975206352a021d6a116b110945ca6c3a1d
+  ports:
+    - "9094:9092"
+  expose:
+    - 5555
+    - 9094
+    - 9092
+  environment:
+    - CLASSPATH=/opt/kafka/libs/extensions/*
+    - KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/etc/kafka/log4j.properties
+    - JMX_PORT=5555
+  command:
+    - "./bin/kafka-server-start.sh"
+    - "/etc/kafka/server.properties"
+    - "--override"
+    - "default.replication.factor=1"
+    - "--override"
+    - "min.insync.replicas=1"
+    - "--override"
+    - "offsets.topic.replication.factor=1"
+    - "--override"
+    - "offsets.topic.num.partitions=1"
+  depends_on:
+    events-kafka-init:
+      condition: service_completed_successfully
+
+events-kafka-init:
+  networks:
+    - ch
+  image: solsson/kafka-initutils@sha256:f6d9850c6c3ad5ecc35e717308fddb47daffbde18eb93e98e031128fe8b899ef
+  command:
+    - "/bin/bash"
+    - "/etc/kafka-configmap/init.sh"
+  environment:
+
+pzoo:
+  networks:
+    - ch
+  expose:
+    - 2181
+    - 2888
+    - 3888
+  image: solsson/kafka:2.3.0@sha256:b59603a8c0645f792fb54e9571500e975206352a021d6a116b110945ca6c3a1d
+  command:
+    - "./bin/zookeeper-server-start.sh"
+    - "/etc/kafka/zookeeper.properties"
+  environment:
+    - KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/etc/kafka/log4j.properties
+  depends_on:
+    events-pzoo-init:
+      condition: service_completed_successfully
+
+events-pzoo-init:
+  networks:
+    - ch
+  expose:
+  image: solsson/kafka-initutils@sha256:f6d9850c6c3ad5ecc35e717308fddb47daffbde18eb93e98e031128fe8b899ef
+  command:
+    - "/bin/bash"
+    - "/etc/kafka-configmap/init.sh"
+  environment:
+    - PZOO_REPLICAS=1
+    - ZOO_REPLICAS=0
+
+zoo:
+  networks:
+    - ch
+  expose:
+    - 2181
+    - 2888
+    - 3888
+  image: solsson/kafka:2.3.0@sha256:b59603a8c0645f792fb54e9571500e975206352a021d6a116b110945ca6c3a1d
+  command:
+    - "./bin/zookeeper-server-start.sh"
+    - "/etc/kafka/zookeeper.properties"
+  environment:
+    - KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/etc/kafka/log4j.properties
+  depends_on:
+    events-zoo-init:
+      condition: service_completed_successfully
+
+events-zoo-init:
+  networks:
+    - ch
+  image: solsson/kafka-initutils@sha256:f6d9850c6c3ad5ecc35e717308fddb47daffbde18eb93e98e031128fe8b899ef
+  command:
+    - "/bin/bash"
+    - "/etc/kafka-configmap/init.sh"
+  environment:
+    - PZOO_REPLICAS=1
+    - ZOO_REPLICAS=0
+    - ID_OFFSET=2
+{{- end }}
diff --git a/applications/jupyterhub/deploy/resources/hub/jupyterhub_config.py b/applications/jupyterhub/deploy/resources/hub/jupyterhub_config.py
@@ -1,11 +1,5 @@
 # load the config object (satisfies linters)
-from z2jh import (
-    get_config,
-    get_name,
-    get_name_env,
-    get_secret_value,
-    set_config_if_not_none,
-)
+
 c = get_config()  # noqa
 
 import glob
@@ -31,6 +25,14 @@
 configuration_directory = os.path.dirname(os.path.realpath(__file__))
 sys.path.insert(0, configuration_directory)
 
+from z2jh import (  # noqa
+    get_config,
+    get_name,
+    get_name_env,
+    get_secret_value,
+    set_config_if_not_none,
+)
+
 
 def camelCaseify(s):
     """convert snake_case to camelCase

diff --git a/applications/nfsserver/deploy/templates-compose/nfsserver-deployment.yaml b/applications/nfsserver/deploy/templates-compose/nfsserver-deployment.yaml
@@ -0,0 +1,19 @@
+{{- define "nfsserver.deployment" }}
+{{- with .apps.nfsserver}}
+
+{{ .name }}:
+  image: {{ .harness.deployment.image }}
+  environment:
+    # NFS useDNS? {{ .nfs.useDNS }}
+    {{- if .nfs.useDNS }}
+    - NFS_SERVER={{ printf "nfs-server.%s.svc.cluster.local" .namespace }}
+    {{- end }}
+    - NFS_PATH={{ .nfs.path }}
+    - PROVISIONER_NAME={{ printf "%s-nfs-provisioner" .namespace }}
+
+  volumes:
+    - type: volume
+      source: {{ .nfs.volumeName }}
+      target: {{ .nfs.path }}
+{{- end }}
+{{- end }}
diff --git a/applications/samples/api/openapi.yaml b/applications/samples/api/openapi.yaml
@@ -23,7 +23,6 @@ paths:
           description: This won't happen
         "500":
           description: Sentry entry should come!
-      deprecated: true
       operationId: error
       summary: test sentry is working
       x-openapi-router-controller: samples.controllers.test_controller
@@ -231,13 +230,11 @@ paths:
               schema:
                 type: object
           description: Operation result
-      deprecated: true
       operationId: submit_sync
       summary: Send a synchronous operation
       x-openapi-router-controller: samples.controllers.workflows_controller
   /operation_sync_results:
     get:
-      deprecated: true
       tags:
         - workflows
       parameters:

diff --git a/applications/samples/backend/requirements.txt b/applications/samples/backend/requirements.txt
@@ -1,11 +1,3 @@
-connexion[swagger-ui] >= 2.6.0; python_version>="3.6"
-# 2.3 is the last version that supports python 3.4-3.5
-connexion[swagger-ui] <= 2.3.0; python_version=="3.5" or python_version=="3.4"
-# connexion requires werkzeug but connexion < 2.4.0 does not install werkzeug
-# we must peg werkzeug versions below to fix connexion
-# https://github.com/zalando/connexion/pull/1044
-werkzeug == 0.16.1; python_version=="3.5" or python_version=="3.4"
-swagger-ui-bundle >= 0.0.2
-python_dateutil >= 2.6.0
-setuptools >= 21.0.0
-Flask == 2.1.1
+connexion[swagger-ui]==2.14.2
+Flask == 2.2.5
+swagger-ui-bundle==0.0.9
diff --git a/applications/samples/backend/samples/controllers/auth_controller.py b/applications/samples/backend/samples/controllers/auth_controller.py
@@ -13,6 +13,8 @@ def valid_token():  # noqa: E501
 
     :rtype: List[Valid]
     """
+    from cloudharness.middleware import get_authentication_token
+    token = get_authentication_token()
     return 'OK!'
 
 
@@ -24,4 +26,8 @@ def valid_cookie():  # noqa: E501
 
     :rtype: List[Valid]
     """
-    return 'OK!'
+    from cloudharness.middleware import get_authentication_token
+    from cloudharness.auth import decode_token
+    token = get_authentication_token()
+    assert decode_token(token)
+    return 'OK'
diff --git a/applications/samples/backend/setup.py b/applications/samples/backend/setup.py
@@ -14,10 +14,13 @@
 # http://pypi.python.org/pypi/setuptools
 
 REQUIRES = [
-    "connexion>=2.0.2",
+    "connexion[swagger-ui]==2.14.2",
+    "Flask >= 2.2.5",
     "python_dateutil>=2.6.0",
     "pyjwt>=2.6.0",
-    "cloudharness"
+    "swagger-ui-bundle>=0.0.2",
+    "cloudharness",
+
 ]
 
 setup(

diff --git a/applications/samples/deploy/values-test.yaml b/applications/samples/deploy/values-test.yaml
@@ -3,9 +3,7 @@ harness:
     soft:
       - workflows
       - events
-      - accounts 
       - common
-      - nfsserver
       - jupyterhub
   accounts:
     roles: