development ← feature/docker-compose-tes: Add initial Docker Compose + TES support

[lbeckman314]

Overview ⚙️

This PR adds initial support for running a minimal TES Server and MinIO for S3.

Quick Start ⚡

Start Funnel + MinIO Services

➜ docker compose -f docker-compose-tes.yaml up

Attaching to funnel-1, minio-1
funnel-1  | {"httpPort":"8000","level":"info","msg":"Server listening","ns":"server","rpcAddress":":9090","time":"2025-02-25T03:36:33Z"}
minio-1   | INFO: Formatting 1st pool, 1 set(s), 1 drives per set.
minio-1   | INFO: WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
minio-1   | MinIO Object Storage Server
minio-1   | Copyright: 2015-2025 MinIO, Inc.
minio-1   | License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
minio-1   | Version: RELEASE.2025-02-18T16-25-55Z (go1.23.6 linux/arm64)
minio-1   |
minio-1   | API: http://172.19.0.2:9000  http://127.0.0.1:9000
minio-1   | WebUI: http://172.19.0.2:9001 http://127.0.0.1:9001
minio-1   |
minio-1   | Docs: https://docs.min.io
minio-1   | WARN: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables

Connect with Funnel:

➜ curl localhost:8000/service-info

{
  "contactUrl":  "https://ohsu-comp-bio.github.io/funnel/",
  "createdAt":  "2016-03-21T16:27:49-07:00",
  "description":  "Funnel is a toolkit for distributed task execution via a simple, standard API.",
  "documentationUrl":  "https://ohsu-comp-bio.github.io/funnel/",
  "environment":  "development",
  "id":  "org.ga4gh.funnel",
  "name":  "Funnel",
  "organization":  {
    "name":  "OHSU Computational Biology",
    "url":  "https://github.com/ohsu-comp-bio"
  },
  "storage":  [
    "file:///path/to/local/funnel-storage",
    "s3://ohsu-compbio-funnel/storage"
  ],
  "tesResources_backend_parameters":  [
    ""
  ],
  "type":  {
    "artifact":  "tes",
    "group":  "org.ga4gh",
    "version":  "git commit:  0c1885ed\ngit branch: fix/k8s-tests\ngit upstream: \nbuild date: Tue Feb 11 00:45:05 UTC 2025\nversion: unknown"
  },
  "updatedAt":  "2025-02-25T03:41:42Z",
  "version":  "git commit:  0c1885ed\ngit branch: fix/k8s-tests\ngit upstream: \nbuild date: Tue Feb 11 00:45:05 UTC 2025\nversion: unknown"
}%

Next Steps 🌀

Liam: update this PR with the following items ✍️

• Add simple "Hello, World!" DAG that uses Funnel and MinIO
• Add MinIO S3 config to Funnel Server
• Add steps for integrating with existing Airflow Workflows and docker-compose.yml
• Add steps for creating and monitoring TES tasks
• Add steps for using S3 for inputs/outputs

[bwalsh]

Suggest adding section on how to configure airflow connection to both funnel and minio.

[lbeckman314]

Good call — adding now!