Logstash container fails to start

[mohanss08]

My Elasticsearch, Logstash and Kibana stack setup is based on Docker Compose and the version is - 8.15.2.

Logstash container always fails to start and throws the following error.

logstash  | ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, exception=LogStash::ConfigurationError, message=>Cannot evaluate `${xpack.monitoring.elasticsearch.password}`. Replacement variable `xpack.monitoring.elasticsearch.password` is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
logstash  | [FATAL] 2024-10-09 05:30:54.028 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
logstash  | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
logstash  |     at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
logstash  | 2024/10/09 05:30:54 Setting 'xpack.monitoring.elasticsearch.password' from environment.
logstash  | 2024/10/09 05:30:54 Setting 'xpack.monitoring.enabled' from environment.
logstash  | 2024/10/09 05:30:54 Setting 'xpack.monitoring.elasticsearch.username' from environment.
logstash  | Using bundled JDK: /usr/share/logstash/jdk

It seems that Logstash cannot find or evaluate the variable ${xpack.monitoring.elasticsearch.password} which was working fine with v8.6.2 before the upgrade and it is working fine until with Logstash v8.12.2.

In the following release notes page - Logstash 8.15.2 Release Notes | Logstash Reference [8.15] | Elastic I could see that docker env variables related issues are fixed withv8.15.2 but it is not working as expected like old v8.12.2 and getting same above errors, when i start the container.

Please let me know whether I should wait for the next release?

[jsvd]

logstash  | 2024/10/09 05:30:54 Setting 'xpack.monitoring.elasticsearch.password' from environment.

the password seems to be detected correctly from the environment variables, can you show how the docker image is being launched?

[mohanss08]

@jsvd My docker-compose file's logstash content given below for your reference.

  logstash:
    image: docker.elastic.co/logstash/logstash:8.15.2
    container_name: logstash
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    command: sh -c "logstash -f /usr/share/logstash/pipeline/file.conf"
    ports:
      - "5044:5044"
      - "9600:9600" 
    environment:
      - xpack.monitoring.elasticsearch.username=$ELASTIC_USERNAME
      - xpack.monitoring.elasticsearch.password=$ELASTIC_PASSWORD
      - xpack.monitoring.enabled=true
      - 'pipeline.ecs_compatibility:disabled'
    networks:
      - stack_network
    depends_on:
      - elasticsearch
networks:
  stack_network:
    driver: bridge

Please take a look and let me know if any additional information is required.

[mashhurs]

I don't have a historical context how we allowed setting dot style env vars (example: foo.bar.hi). Indeed working well with 8.12.0. However, our recommendation is to follow the guide we have: https://www.elastic.co/guide/en/logstash/current/docker-config.html#docker-env-config
With 8.15.2, I had test coverages and multiple scenarios, including CONFIG_STRING has env vars which validate the underscore (example: FOO.BAR.HI) enr vars.

Any thoughts @jsvd?

[mohanss08]

@mashhurs - We have made changes to our docker-compose file. By replacing below respectively, we were able to successfully start the Logstash container without any issues.

- XPACK_MONITORING_ENABLED=true
- XPACK_MONITORING_ELASTICSEARCH_USERNAME=$ELASTIC_USERNAME
- XPACK_MONITORING_ELASTICSEARCH_PASSWORD=$ELASTIC_PASSWORD

It seems that starting from v8.13.0, the environment variables in Docker require using underscores instead of dots.