[minor] add Dockerfile healthchecks to some core services

[joecorall]

Inspired by seeing cantaloupe and solr having mysteriously died (though rarely) in my local instance over the past year, added some healthchecks so docker compose can report on container health.

I tested this locally by setting the CMD's in this PR in my docker compose YML's healthcheck.

$ docker ps --format "{{.Names}}\t{{.Status}}"
lehigh-d10-fcrepo-prod-1        Up 22 hours (healthy)
lehigh-d10-activemq-prod-1      Up 22 hours (healthy)
lehigh-d10-solr-prod-1  Up 22 hours (healthy)
lehigh-d10-drupal-prod-1        Up 22 hours (healthy)
lehigh-d10-cantaloupe-prod-1    Up 22 hours (healthy)
lehigh-d10-mariadb-prod-1       Up 22 hours (healthy)
lehigh-d10-memcached-prod-1     Up 22 hours (healthy)
lehigh-d10-traefik-prod-1       Up 22 hours (healthy)

If we instead define the healthcheck in the dockerfile, docker compose will automatically pick up on the command and we won't need to define a healthcheck section in the YML. Docker compose will use the default values for interval/timeout/start_period/start_interval which could be overriden in the docker compose yml.

Minor changes

I see this as a new feature on the buildkit images, so marking this PR as a minor version bump.

The only change that might effect folks is this PR does enable the API on cantaloupe, since that project requires that API to be enabled to access its health check route. Though I did add some protection since this will turn on the API to automatically set the API secret if it's a blank string (and it will get rotated every time the container restarts) since most people probably are just using the defaults.

Out of scope

Locally I also run a bash script on a systemd timer to check the health on the containers and attempt to restart automatically. I think a script like that might belong better in islandora documentation or in site template, so leaving it out of this PR. I am turning that script into a go utility that can be run on the host or within a docker compose service if the docker socket is mounted in the container: https://github.com/lehigh-university-libraries/docker-autoheal

[joecorall]

@seth-shaw-asu

Testing instructions

Create a new isle-site-template or isle-dc install using the ISLANDORA_TAG=healthchecks (instead of main). Let it run for a minute or two then run

docker ps --format "{{.Names}}\t{{.Status}}"

then ensure activemq, fcrepo, drupal, mariadb, solr, tomcat containers are marked as healthy

Though the tests that run here in CI are already doing basically this. SO maybe it's already tested 🤷

[nigelgbanks]

This is great work! I've started to dig into this for a few hours today, and I'm working on a pull request to your branch, to address some issues I've seen, it's not ready yet. I just wanted to highlight it, though, so it doesn't go through before you have a chance to see my feedback.

[seth-shaw-asu]

I'll defer to @nigelgbanks.

[joecorall]

@nigelgbanks - I've had these healthchecks running in our environment for several weeks and haven't seen any issues. What issues were you seeing?

[nigelgbanks]

@joecorall sorry for the delay, when I started digging into this I got lost down a rabbit hole of many improvements and bugs that have been in the repo for like 4 years. The set of changes I've made has become well, frankly, huge. So I'll send up a pull with the minimum recommendations to unblock this, and I'll raise a separate pull for the work I've been doing.

[nigelgbanks]

@joecorall sorry for the delay, when I started digging into this I got lost down a rabbit hole of many improvements and bugs that have been in the repo for like 4 years. The set of changes I've made has become well, frankly, huge. So I'll send up a pull with the minimum recommendations to unblock this, and I'll raise a separate pull for the work I've been doing.

The changes are here: #375