Add integration test framework (#492)

## Overview

This update adds a framework for testing the deployed stack using
pytest. This will allow developers to check
that their changes are consistent with the existing stack and to add
additionally tests when new functionality
is introduced. 

Changes to implement this include:

- existing unit tests are moved to the `tests/unit/` directory
- new integration tests are written in the `tests/integration/`
directory. More tests will be added in the
  future!
- `conftest.py` scripts updated to bring the stack up/down in a
consistent way for the integration tests.
- unit tests updated to accomodate new testing infrastructure as needed.
- unit tests updated to test logging outputs better
- `birdhouse` interface script updated to support testing infrastructure
(this should not change anything for
  other end-users).
- additional documentation added to `birdhouse` interface to improve
user experience.
- docker healthchecks added to more components so that the readiness of
the stack can be determined with or
  without the use of the `canarie-api` component.
- deprecates the `optional-components/wps-healthchecks` component since
the healthchecks are now added by default
    to all WPS components.

Next steps:

- add more integration tests as needed
- add a framework for testing migrating the stack from one version to
another

## Changes

**Non-breaking changes**
- Adds integration tests and healthchecks

**Breaking changes**
- None

## Related Issue / Discussion

- Introduces a mechanism for testing the stack locally so that we don't
need to rely on integration tests run by notebooks through Jenkins.
Since the Jenkins CI tests have been broken lately.
- This also gives developers a mechanism to tests their work locally and
establishes a simple way to write integration tests (without the need to
wrap them in jupyter notebooks).
- This PR only introduces the framework for creating these tests. Once
this is approved, the actual work of writing the tests can start.
- Further discussion/planning is needed if we want to eventually run
these tests in the CI pipeline

## Additional Information

## CI Operations

<!--
The test suite can be run using a different DACCS config with
``birdhouse_daccs_configs_branch: branch_name`` in the PR description.
To globally skip the test suite regardless of the commit message use
``birdhouse_skip_ci`` set to ``true`` in the PR description.

Using ``[<cmd>]`` (with the brackets) where ``<cmd> = skip ci`` in the
commit message will override ``birdhouse_skip_ci`` from the PR
description.
Such commit command can be used to override the PR description behavior
for a specific commit update.
However, a commit message cannot 'force run' a PR which the description
turns off the CI.
To run the CI, the PR should instead be updated with a ``true`` value,
and a running message can be posted in following PR comments to trigger
tests once again.
-->

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 2.7.2
+current_version = 2.7.3
 commit = True
 tag = False
 tag_name = {new_version}
@@ -30,11 +30,11 @@ search = {current_version}
 replace = {new_version}
 
 [bumpversion:file:RELEASE.txt]
-search = {current_version} 2025-01-16T18:35:17Z
+search = {current_version} 2025-01-17T18:54:34Z
 replace = {new_version} {utcnow:%Y-%m-%dT%H:%M:%SZ}
 
 [bumpversion:part:releaseTime]
-values = 2025-01-16T18:35:17Z
+values = 2025-01-17T18:54:34Z
 
 [bumpversion:file(version):birdhouse/components/canarie-api/docker_configuration.py.template]
 search = 'version': '{current_version}'

.github/workflows/tests.yml

@@ -18,4 +18,4 @@ jobs:
           pip install -r ./tests/requirements.txt
       - name: Test with pytest
         run: |
-          pytest ./tests/
+          pytest ./tests/unit

CHANGES.md

@@ -17,6 +17,38 @@
 
 [//]: # (list changes here, using '-' for each new entry, remove this when items are added)
 
+[2.7.3](https://github.com/bird-house/birdhouse-deploy/tree/2.7.3) (2025-01-17)
+------------------------------------------------------------------------------------------------------------------
+
+## Changes
+
+- Add integration test framework
+
+  This update adds a framework for testing the deployed stack using pytest. This will allow developers to check
+  that their changes are consistent with the existing stack and to add additionally tests when new functionality
+  is introduced. 
+
+  Changes to implement this include:
+
+  - existing unit tests are moved to the `tests/unit/` directory
+  - new integration tests are written in the `tests/integration/` directory. More tests will be added in the
+    future!
+  - `conftest.py` scripts updated to bring the stack up/down in a consistent way for the integration tests.
+  - unit tests updated to accomodate new testing infrastructure as needed.
+  - unit tests updated to test logging outputs better
+  - `birdhouse` interface script updated to support testing infrastructure (this should not change anything for
+    other end-users).
+  - additional documentation added to `birdhouse` interface to improve user experience.
+  - docker healthchecks added to more components so that the readiness of the stack can be determined with or
+    without the use of the `canarie-api` component.
+  - deprecates the `optional-components/wps-healthchecks` component since the healthchecks are now added by default
+    to all WPS components.
+
+  Next steps:
+
+  - add more integration tests as needed
+  - add a framework for testing migrating the stack from one version to another
+
 [2.7.2](https://github.com/bird-house/birdhouse-deploy/tree/2.7.2) (2025-01-16)
 ------------------------------------------------------------------------------------------------------------------
 

Makefile

@@ -1,7 +1,7 @@
 # Generic variables
 override SHELL       := bash
 override APP_NAME    := birdhouse-deploy
-override APP_VERSION := 2.7.2
+override APP_VERSION := 2.7.3
 
 # utility to remove comments after value of an option variable
 override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")

README.rst

@@ -18,13 +18,13 @@ for a full-fledged production platform.
     * - citation
       - | |citation|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.7.2.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.7.3.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.7.2...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.7.3...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-2.7.2-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-2.7.3-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.7.2
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.7.3
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

RELEASE.txt

@@ -1 +1 @@
-2.7.2 2025-01-16T18:35:17Z
+2.7.3 2025-01-17T18:54:34Z

bin/birdhouse

@@ -10,12 +10,11 @@ export __BIRDHOUSE_SUPPORTED_INTERFACE=True
 
 USAGE="USAGE: $THIS_BASENAME [-h|--help] 
                              [-b|--backwards-compatible]
-                             [-e|--env-file local-env-file]
-                             [-q|--quiet] [-q|--quiet {DEBUG|INFO|WARN|ERROR|CRITICAL}]
-                             [-s|--log-stdout] [-s|--log-stdout {DEBUG|INFO|WARN|ERROR|CRITICAL}]
-                             [-l|--log-file log-file-path]
-                             [-l|--log-file {DEBUG|INFO|WARN|ERROR|CRITICAL} log-file-path]
-                             [-L|--log-level {DEBUG|INFO|WARN|ERROR|CRITICAL}]
+                             [-e|--env-file path]
+                             [-q|--quiet [level]]
+                             [-s|--log-stdout [level]]
+                             [-l|--log-file [level] path]
+                             [-L|--log-level level]
                              {info|compose|configs}"
 USAGE=$(echo $USAGE | tr "\n" " ")
 
@@ -25,13 +24,13 @@ Manage the Birdhouse software stack.
 
 Commands:
   info      Print build information
-  compose   Run a \"docker compose\" command for the Birdhouse project
+  compose   Run a \"docker compose\" command for the Birdhouse project (run '${THIS_BASENAME} compose --help' for more options)
   configs   Load or execute commands in the Birdhouse configuration environment
 
 Options:
   -h, --help                                              Print this message and exit
   -b, --backwards-compatible                              Run in backwards compatible mode
-  -e, --env-file string                                   Override the local environment file, default is ${COMPOSE_DIR}/env.local
+  -e, --env-file path                                     Override the local environment file, default is ${COMPOSE_DIR}/env.local
   -s, --log-stdout                                        Write logs to stdout for all log levels, default is to write to stderr
   -s, --log-stdout {DEBUG|INFO|WARN|ERROR|CRITICAL}       Write logs to stdout for the given log level only (this option can be repeated)
   -l, --log-file path                                     Write logs to this file path for all log levels
@@ -41,12 +40,13 @@ Options:
   -L, --log-level {DEBUG|INFO|WARN|ERROR}                 Set log level, default is INFO
 "
 
-CONFIGS_USAGE="USAGE: $THIS_BASENAME configs [-h|--help] [-d|--default] {[-p|--print-config-command] | [-c|--command command]}"
+CONFIGS_USAGE="USAGE: $THIS_BASENAME [${THIS_BASENAME} options] configs [-h|--help] [-d|--default] {[-p|--print-config-command] | [-c|--command command]}"
 CONFIGS_HELP="$CONFIGS_USAGE
 
 Load or execute commands in the Birdhouse configuration environment.
 
 Options:
+  -h, --help                   Print this message and exit
   -d, --default                Only load/print a command for the default configuration settings, not those specified by the local environment file
   -p, --print-config-command   Print a command that can be used to load configuration settings as environment variables
   -c, --command string         Execute the given command after loading configuration settings
@@ -78,6 +78,7 @@ READ_CONFIGS_CMD=read_configs
 #   $ echo $BIRDHOUSE_BACKWARD_COMPATIBLE_ALLOWED
 #   True
 print_config_command() {
+  option="$1"
   configs_cmd_prefix="export __BIRDHOUSE_SUPPORTED_INTERFACE=True ;"
   configs_cmd_suffix="unset __BIRDHOUSE_SUPPORTED_INTERFACE ;"
   if [ "${BIRDHOUSE_BACKWARD_COMPATIBLE_ALLOWED+set}" = 'set' ]; then
@@ -98,6 +99,9 @@ print_config_command() {
     configs_cmd_suffix="${configs_cmd_suffix} unset BIRDHOUSE_LOCAL_ENV ;"
   fi
 
+  if [ "${option}" = "no-suffix" ]; then
+    configs_cmd_suffix=
+  fi
   echo "${configs_cmd_prefix} . ${COMPOSE_DIR}/read-configs.include.sh; ${READ_CONFIGS_CMD} ; ${configs_cmd_suffix}"
 }
 
@@ -167,9 +171,9 @@ parse_configs_args() {
         print_config_command
       elif [ "${CONFIGS_CMD+set}" = 'set' ]; then
         if [ "${CONFIGS_QUIET}" = "True" ]; then
-          eval "$(print_config_command)" 2> /dev/null
+          eval "$(print_config_command no-suffix)" 2> /dev/null
         else
-          eval "$(print_config_command)"
+          eval "$(print_config_command no-suffix)"
         fi
         exec /usr/bin/env sh -c "${CONFIGS_CMD}"
       else
@@ -285,7 +289,7 @@ parse_args() {
       parse_configs_args "$@"
       ;;
     -h|--help)
-      echo "$HELP"
+      echo "$HELP" | more
       ;;
     -??*)
       parse_multiple_short_flags parse_args "$@"

birdhouse/birdhouse-compose.sh

@@ -91,12 +91,14 @@ done
 export BIRDHOUSE_AUTODEPLOY_EXTRA_REPOS_AS_DOCKER_VOLUMES
 
 # we apply all the templates
-find ${ALL_CONF_DIRS} -name '*.template' 2>/dev/null |
-  while read FILE
-  do
-    DEST=${FILE%.template}
-    cat "${FILE}" | envsubst "$VARS" | envsubst "$OPTIONAL_VARS" > "${DEST}"
-  done
+if [ x"$1" = x"up" ]; then
+  find ${ALL_CONF_DIRS} -name '*.template' 2>/dev/null |
+    while read FILE
+    do
+      DEST=${FILE%.template}
+      cat "${FILE}" | envsubst "$VARS" | envsubst "$OPTIONAL_VARS" > "${DEST}"
+    done
+fi
 
 SHELL_EXEC_FLAGS=
 if [ "${BIRDHOUSE_LOG_LEVEL}" = "DEBUG" ]; then
@@ -115,7 +117,7 @@ fi
 
 COMPOSE_EXTRA_OPTS=""
 
-if [ x"$1" = x"up" ]; then
+if [ x"$1" = x"up" ] || [ x"$1" = x"restart" ]; then
   COMPOSE_EXTRA_OPTS="${BIRDHOUSE_COMPOSE_UP_EXTRA_OPTS}"
   for adir in $ALL_CONF_DIRS; do
     COMPONENT_PRE_COMPOSE_UP="$adir/pre-docker-compose-up"

birdhouse/components/canarie-api/docker-compose-extra.yml

@@ -21,6 +21,13 @@ services:
     restart: always
     entrypoint: /entrypoint
     logging: *default-logging
+    healthcheck:
+      test: ["CMD", "curl", "--fail", "http://localhost:2000"]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+      start_interval: 5s
 
   canarie-api-cron:
     image: ${CANARIE_IMAGE}

birdhouse/components/canarie-api/docker_configuration.py.template

@@ -108,8 +108,8 @@ SERVICES = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.7.2',
-            'releaseTime': '2025-01-16T18:35:17Z',
+            'version': '2.7.3',
+            'releaseTime': '2025-01-17T18:54:34Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',
@@ -141,8 +141,8 @@ PLATFORMS = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.7.2',
-            'releaseTime': '2025-01-16T18:35:17Z',
+            'version': '2.7.3',
+            'releaseTime': '2025-01-17T18:54:34Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',

birdhouse/components/cowbird/docker-compose-extra.yml

@@ -45,6 +45,13 @@ services:
       - "${BIRDHOUSE_DATA_PERSIST_SHARED_ROOT}:${BIRDHOUSE_DATA_PERSIST_SHARED_ROOT}"
     restart: always
     logging: *default-logging
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://0.0.0.0:7000"]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+      start_interval: 5s
 
   cowbird-worker:
     image: pavics/cowbird:${COWBIRD_VERSION}-worker
@@ -74,6 +81,7 @@ services:
       - "${BIRDHOUSE_DATA_PERSIST_SHARED_ROOT}/${USER_WORKSPACES}:${BIRDHOUSE_DATA_PERSIST_SHARED_ROOT}/${USER_WORKSPACES}"
     restart: always
     logging: *default-logging
+    # TODO: create a stable healthcheck for this service (`inspect ping` or `status` is not reliable)
 
   # Dedicated database for Cowbird, since other 'mongodb' image does not employ the same version.
   cowbird-mongodb:
@@ -85,6 +93,13 @@ services:
       - ${COWBIRD_MONGODB_DATA_DIR}:/data/db
     restart: always
     logging: *default-logging
+    healthcheck:
+      test: ["CMD", "mongo", "--eval", "print('connected')"]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+      start_interval: 5s
 
 networks:
   cowbird-mongodb: {}

birdhouse/components/finch/docker-compose-extra.yml

@@ -21,3 +21,16 @@ services:
       - /tmp
     restart: always
     logging: *default-logging
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "curl",
+          "--fail",
+          "http://localhost:5000?service=WPS&request=GetCapabilities",
+        ]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+      start_interval: 5s

birdhouse/components/geoserver/docker-compose-extra.yml

@@ -31,10 +31,12 @@ services:
     restart: always
     logging: *default-logging
     healthcheck:
-      test: curl --fail --silent http://localhost:8080/ || exit 1
+      test: curl --fail --silent http://localhost:8080/geoserver || exit 1
       interval: 1m30s
       timeout: 10s
       retries: 3
+      start_period: 30s
+      start_interval: 5s
 
   postgis:
     image: pavics/postgis:2.2

birdhouse/components/hummingbird/docker-compose-extra.yml

@@ -19,3 +19,16 @@ services:
       - postgres
     restart: always
     logging: *default-logging
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "curl",
+          "--fail",
+          "http://localhost:8080/wps?service=WPS&request=GetCapabilities",
+        ]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+      start_interval: 5s

birdhouse/components/jupyterhub/docker-compose-extra.yml

@@ -46,6 +46,13 @@ services:
     command: bash -c "jupyterhub upgrade-db && jupyterhub"
     restart: always
     logging: *default-logging
+    healthcheck:
+      test: ["CMD", "curl", "--fail", "http://localhost:8000/jupyter"]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+      start_interval: 5s
 
 # need external network so the folder name is not prefixed to network name
 networks:

birdhouse/components/magpie/docker-compose-extra.yml

@@ -35,6 +35,13 @@ services:
       - ./components/magpie/magpie.ini:/opt/local/src/magpie/config/magpie.ini
     restart: always
     logging: *default-logging
+    healthcheck:
+      test: ["CMD", "wget", "-qO-", "http://0.0.0.0:2001"]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+      start_interval: 5s
 
   postgres-magpie:
     image: postgres:9.6
@@ -49,3 +56,10 @@ services:
       - ./components/magpie/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
     restart: always
     logging: *default-logging
+    healthcheck:
+      test: ["CMD", "pg_isready"]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+      start_interval: 5s

birdhouse/components/mongodb/docker-compose-extra.yml

@@ -19,6 +19,13 @@ services:
     command: bash -c 'chown -R mongodb:mongodb /data && chmod -R 755 /data && mongod'
     restart: always
     logging: *default-logging
-
+    healthcheck:
+      test: ["CMD", "mongo", "--eval", "print('connected')"]
+      interval: 60s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+      start_interval: 5s
+
 networks:
   mongodb: {}