Remove Secrets and use Docker Volumes

[fxdgear]

There've been some issues relating to how stack-docker starts up stuff using the docker-compose image.

• changing project names
• volume mounts not working
• some other issues regaring containers not starting.

I've moved everything around and more use of docker volumes:

• created certs volume, for storing the certs. Now all containers can mount that volume and have access to the certs.
• created a config volume for each service
  • this mounts the config.yml for each service and also stores the keystore.

These changes help process a lot of the permissions issues that were happening.
This makes it a lot easer to clean up using a 'docker-compose down -v' and restart.

known issues:

• the setup script doesn't work as cleanly as I'd like and when trying execute it via a
  'docker-compose -f setup/setup.yml run setup' the script exits early.

closes: #73 #67 #65

[LeeDr]

Here are the steps I tried (not completely working at this point);

curl https://staging.elastic.co/6.6.0-52f15a2e/docker/elasticsearch-6.6.0.tar.gz | docker load
curl https://staging.elastic.co/6.6.0-52f15a2e/docker/kibana-6.6.0.tar.gz | docker load
curl https://staging.elastic.co/6.6.0-52f15a2e/docker/logstash-6.6.0.tar.gz | docker load
curl https://staging.elastic.co/6.6.0-52f15a2e/docker/metricbeat-6.6.0.tar.gz | docker load
curl https://staging.elastic.co/6.6.0-52f15a2e/docker/filebeat-6.6.0.tar.gz | docker load
curl https://staging.elastic.co/6.6.0-52f15a2e/docker/packetbeat-6.6.0.tar.gz | docker load
curl https://staging.elastic.co/6.6.0-52f15a2e/docker/heartbeat-6.6.0.tar.gz | docker load
curl https://staging.elastic.co/6.6.0-52f15a2e/docker/auditbeat-6.6.0.tar.gz | docker load
curl https://staging.elastic.co/6.6.0-52f15a2e/docker/apm-server-6.6.0.tar.gz | docker load

 TAG=6.6.0 ELASTIC_PASSWORD=changeit ELASTIC_VERSION=6.6.0 docker-compose -f setup/setup.yml run --rm setup

That appears to run successfully and ends with;

Successfully updated the keystore
ELASTIC_PASSWORD
Setting up dashboards...
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards


****************************

Setup completed successfully. To start the stack please run:
         docker-compose up -d

Your 'elastic' user password is: changeit

I check what's running (elasticsearch and kibana are);

docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                   PORTS                              NAMES
65298c5cc94a        docker.elastic.co/kibana/kibana:6.6.0                 "/usr/local/bin/kiba…"   3 minutes ago       Up 3 minutes (healthy)   0.0.0.0:5601->5601/tcp             kibana
4e8ca5ccfb73        docker.elastic.co/elasticsearch/elasticsearch:6.6.0   "/usr/local/bin/dock…"   3 minutes ago       Up 3 minutes (healthy)   0.0.0.0:9200->9200/tcp, 9300/tcp   elasticsearch

Then try to bring up the rest of the stack with;

TAG=6.6.0 ELASTIC_PASSWORD=changeit ELASTIC_VERSION=6.6.0 docker-compose up -d
Creating network "stackpr_stack" with the default driver
Creating volume "stackpr_ls_config" with default driver
Creating volume "stackpr_mb_config" with default driver
Creating volume "stackpr_kb_data" with default driver
Creating volume "stackpr_es_config" with default driver
Creating volume "stackpr_fb_config" with default driver
Creating volume "stackpr_pb_config" with default driver
Creating volume "stackpr_kb_config" with default driver
Creating volume "stackpr_ab_config" with default driver
Creating volume "stackpr_hb_config" with default driver
Creating volume "stackpr_apm_config" with default driver
Creating volume "stackpr_certs" with default driver
Creating volume "stackpr_es_data" with default driver
Creating elasticsearch ... error

ERROR: for elasticsearch  Cannot create container for service elasticsearch: Conflict. The container name "/elasticsearch" is already in use by container "4e8ca5ccfb73508e3bf8d19624e955542dddb99c4183bbe3f020a5825241dfdf". You have to remove (or rename) that container to be able to reuse that name.

ERROR: for elasticsearch  Cannot create container for service elasticsearch: Conflict. The container name "/elasticsearch" is already in use by container "4e8ca5ccfb73508e3bf8d19624e955542dddb99c4183bbe3f020a5825241dfdf". You have to remove (or rename) that container to be able to reuse that name.
Encountered errors while bringing up the project.

[fxdgear]

@LeeDr what version of compose do you have?

edit: fix name

[fxdgear]

@LeeDr ALSO I also noticed that your project name is stackpr

The stack-docker repository makes the assumption that you're going to clone the repo into the same name as the repository is itself.

If you're going to clone this project into a different directory name (such as stackpr you'll have to adjust your docker-compose command:

TAG=6.6.0 ELASTIC_PASSWORD=changeit ELASTIC_VERSION=6.6.0 docker-compose -p stack-docker up -d

I'll update the readme to account for this.

[MikeSourceCode]

I think this change, caused the password not to be picked up.

stack-docker/scripts/setup-logstash.sh

Line 24 in 69b32c7

-d '{"password": "${ELASTIC_PASSWORD}"}'

-d '{"password": "${ELASTIC_PASSWORD}"}'

setup-kibana.sh has it like this and does not produce a warning; image attached.
-d "{"password": "${ELASTIC_PASSWORD}"}"

[LeeDr]

@fxdgear I just ran this on 6.7.0 (the latest BC build) and it seemed to work fine.
But when I ran it on 7.0.0-rc1 it stopped here;

Determining if x-pack is installed...
Starting Elasticsearch....
Creating elasticsearch ... done
Running 'setup-users.sh'
No cert found at /usr/share/elasticsearch/config/certs/ssl/ca/ca.crt
Running 'setup-kibana.sh'
Starting elasticsearch ... done

docker logs elasticsearch shows;

{"type": "server", "timestamp": "2019-03-25T18:51:49,914+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "docker-cluster", "node.name": "7c401e061861",  "message": "uncaught exception in thread [main]" ,
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]",
...
"Caused by: java.lang.reflect.InvocationTargetException",
...
"Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory",
...
"Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/certs/ssl/ca/ca.crt",

It looks like that error caused elasticsearch to exit.

[LeeDr]

I hacked the setup-elasticsearch.sh script to add some more debug logging like this;

#!/bin/bash
configdir=/usr/share/elasticsearch/config
# Determine if x-pack is enabled
echo "Determining if x-pack is installed..."
echo "`ls /usr/share/elasticsearch/bin`"
if [[ -d /usr/share/elasticsearch/bin/x-pack ]]; then
    echo "=== Yes we found bin/x-pack ==="
    echo $ELASTIC_PASSWORD
    if [[ -n "$ELASTIC_PASSWORD" ]]; then

And that ls output did NOT contain x-pack. It does contain;

x-pack-env
x-pack-security-env
x-pack-watcher-env

I don't know if that means something is missing in the docker image, or if something changed in 7.0. I'll install a non-docker 7.0 stack and see.