permission issue on /usr/share/elasticsearch/.aws/config

[gquintard]

Hi!

It looks like the elastisearch container reboots in a loop when I run

git checkout v3.2.2
make setup
sudo sysctl -w vm.max_map_count=262144
docker compose up

I'm attaching the beginning of the logs, but from what I saw elasticsearch-1 keeps getting restarted and crashing the same way.
Am I doing something wrong?

[+] Building 0.0s (0/0)                                                                                                                                                                                                                                                                                        docker:default
[+] Running 4/4
 ✔ Container elastic-kibana-1         Recreated                                                                                                                                                                                                                                                                          0.1s 
 ✔ Container elastic-elasticsearch-1  Created                                                                                                                                                                                                                                                                            0.0s 
 ✔ Container elastic-logstash-1       Created                                                                                                                                                                                                                                                                            0.0s 
 ✔ Container elastic-apm-server-1     Created                                                                                                                                                                                                                                                                            0.0s 
Attaching to apm-server-1, elasticsearch-1, kibana-1, logstash-1
logstash-1       | Using bundled JDK: /usr/share/logstash/jdk
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.257Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":139},"message":"Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.257Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":146},"message":"Beat ID: 3b34c166-e3b4-42da-8982-dcf45199a25c","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"warn","@timestamp":"2023-11-22T20:02:08.258Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.258Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://elasticsearch:9200","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.259Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":576},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/apm-server","data":"/usr/share/apm-server/data","home":"/usr/share/apm-server","logs":"/usr/share/apm-server/logs"},"type":"apm-server","uuid":"3b34c166-e3b4-42da-8982-dcf45199a25c"},"ecs.version":"1.6.0"}}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.259Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":584},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"825865682816d18b5e66e94949bce8d2f0172044","time":"2023-09-18T17:00:04.000Z","version":"8.10.2"},"ecs.version":"1.6.0"}}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.259Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":587},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":12,"version":"go1.20.7"},"ecs.version":"1.6.0"}}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.259Z","log.origin":{"file.name":"beatcmd/maxprocs.go","file.line":68},"message":"maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.318Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":591},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-11-22T19:38:31Z","containerized":false,"name":"9283fb790c7b","ip":["127.0.0.1/8","172.19.0.3/16"],"kernel_version":"6.6.2-arch1-1","mac":["02:42:ac:13:00:03"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","major":20,"minor":4,"patch":6,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.319Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":620},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/apm-server","exe":"/usr/share/apm-server/apm-server","name":"apm-server","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":false},"start_time":"2023-11-22T20:02:07.820Z"},"ecs.version":"1.6.0"}}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.346Z","log.logger":"config","log.origin":{"file.name":"config/agentconfig.go","file.line":70},"message":"using output.elasticsearch for fetching agent config","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.346Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":142},"message":"Listening on: [::]:8200","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.346Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":394},"message":"apm-server started.","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.348Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":201},"message":"no cgroups detected, falling back to total system memory","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.348Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":225},"message":"MaxConcurrentDecoders set to 1984 based on 80 percent of 19.4gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.348Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":232},"message":"Aggregation.MaxServices set to 19376 based on 19.4gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.348Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":239},"message":"Aggregation.Transactions.MaxGroups set to 96884 based on 19.4gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.348Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":246},"message":"Aggregation.ServiceTransactions.MaxGroups for service aggregation set to 19376 based on 19.4gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.351Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":787},"message":"docappender.DocumentBufferSize set to 15873 based on 19.4gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.351Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":801},"message":"docappender.MaxRequests set to 39 based on 19.4gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.351Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.355Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":138},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.355Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":138},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.355Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":138},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.355Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":138},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.355Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":138},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.355Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":138},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.355Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":138},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.355Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":138},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.355Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":138},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.356Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":215},"message":"Starting apm-server [825865682816d18b5e66e94949bce8d2f0172044 built 2023-09-18 13:00:04 -0400 -0400]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.356Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":94},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:08.356Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":98},"message":"SSL enabled.","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"error","@timestamp":"2023-11-22T20:02:09.281Z","log.logger":"esclientleg","log.origin":{"file.name":"transport/logging.go","file.line":38},"message":"Error dialing dial tcp 172.19.0.5:9200: connect: connection refused","service.name":"apm-server","network":"tcp","address":"elasticsearch:9200","ecs.version":"1.6.0"}
apm-server-1     | {"log.level":"info","@timestamp":"2023-11-22T20:02:09.281Z","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":224},"message":"Failed to connect to Elastic X-Pack Monitoring. Either Elasticsearch X-Pack monitoring is not enabled or Elasticsearch is not available. Will keep retrying. Error: cannot connect underlying Elasticsearch client: Get \"https://elasticsearch:9200\": dial tcp 172.19.0.5:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}
kibana-1         | {"log.level":"info","@timestamp":"2023-11-22T20:02:09.797Z","log":{"logger":"elastic-apm-node"},"agentVersion":"3.49.1","env":{"pid":7,"proctitle":"/usr/share/kibana/bin/../node/bin/node","os":"linux 6.6.2-arch1-1","arch":"x64","host":"fdbdd8a638fa","timezone":"UTC+00","runtime":"Node.js v18.17.1"},"config":{"serviceName":{"source":"start","value":"kibana","commonName":"service_name"},"serviceVersion":{"source":"start","value":"8.10.2","commonName":"service_version"},"serverUrl":{"source":"start","value":"https://kibana-cloud-apm.apm.us-east-1.aws.found.io/","commonName":"server_url"},"logLevel":{"source":"default","value":"info","commonName":"log_level"},"active":{"source":"start","value":true},"contextPropagationOnly":{"source":"start","value":true},"environment":{"source":"start","value":"production"},"logUncaughtExceptions":{"source":"start","value":true},"globalLabels":{"source":"start","value":[["git_rev","29555604d9c1721a91c9b948fd3ae1193e944ce4"]],"sourceValue":{"git_rev":"29555604d9c1721a91c9b948fd3ae1193e944ce4"}},"secretToken":{"source":"start","value":"[REDACTED]","commonName":"secret_token"},"breakdownMetrics":{"source":"start","value":false},"captureSpanStackTraces":{"source":"start","sourceValue":false},"centralConfig":{"source":"start","value":false},"metricsInterval":{"source":"start","value":120,"sourceValue":"120s"},"propagateTracestate":{"source":"start","value":true},"transactionSampleRate":{"source":"start","value":0.1,"commonName":"transaction_sample_rate"},"captureBody":{"source":"start","value":"off","commonName":"capture_body"},"captureHeaders":{"source":"start","value":false}},"activationMethod":"require","ecs":{"version":"1.6.0"},"message":"Elastic APM Node.js Agent v3.49.1"}
kibana-1         | [2023-11-22T20:02:11.306+00:00][INFO ][root] Kibana is starting
kibana-1         | [2023-11-22T20:02:11.384+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
elasticsearch-1  | {"type": "server", "timestamp": "2023-11-22T20:02:12,493Z", "level": "INFO", "component": "o.a.l.u.VectorUtilPanamaProvider", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "Java vector incubator API enabled; uses preferredBitSize=256" }
elasticsearch-1  | {"type": "server", "timestamp": "2023-11-22T20:02:13,084Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "version[8.10.2], pid[121], build[docker/6d20dd8ce62365be9b1aca96427de4622e970e9e/2023-09-19T08:16:24.564900370Z], OS[Linux/6.6.2-arch1-1/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/20.0.2/20.0.2+9-78]" }
elasticsearch-1  | {"type": "server", "timestamp": "2023-11-22T20:02:13,085Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
elasticsearch-1  | {"type": "server", "timestamp": "2023-11-22T20:02:13,085Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11417910431799224951, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xmx1024m, -Xms1024m, -Des.enforce.bootstrap.checks=true, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, --add-modules=org.elasticsearch.preallocate, -Djdk.module.main=org.elasticsearch.server]" }
apm-server-1     | {"log.level":"error","@timestamp":"2023-11-22T20:02:13.355Z","log.logger":"agentcfg","log.origin":{"file.name":"agentcfg/elasticsearch.go","file.line":150},"message":"refresh cache error: context deadline exceeded","service.name":"apm-server","ecs.version":"1.6.0"}
elasticsearch-1  | {"type": "server", "timestamp": "2023-11-22T20:02:13,767Z", "level": "WARN", "component": "c.a.a.p.i.BasicProfileConfigFileLoader", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "Unable to load config file null", 
elasticsearch-1  | "stacktrace": ["java.security.AccessControlException: access denied (\"java.io.FilePermission\" \"/usr/share/elasticsearch/.aws/config\" \"read\")",
elasticsearch-1  | "at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]",
elasticsearch-1  | "at java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]",
elasticsearch-1  | "at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]",
elasticsearch-1  | "at java.lang.SecurityManager.checkRead(SecurityManager.java:742) ~[?:?]",
elasticsearch-1  | "at java.io.File.exists(File.java:831) ~[?:?]",
elasticsearch-1  | "at com.amazonaws.profile.path.config.SharedConfigDefaultLocationProvider.getLocation(SharedConfigDefaultLocationProvider.java:36) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.profile.path.AwsProfileFileLocationProviderChain.getLocation(AwsProfileFileLocationProviderChain.java:41) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfilesConfigFile(BasicProfileConfigFileLoader.java:69) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfile(BasicProfileConfigFileLoader.java:55) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.retry.internal.RetryModeResolver.profile(RetryModeResolver.java:103) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.retry.internal.RetryModeResolver.resolveRetryMode(RetryModeResolver.java:89) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.retry.internal.RetryModeResolver.<init>(RetryModeResolver.java:55) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.retry.internal.RetryModeResolver.<init>(RetryModeResolver.java:48) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.retry.RetryPolicy.<clinit>(RetryPolicy.java:35) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.retry.PredefinedRetryPolicies.<clinit>(PredefinedRetryPolicies.java:32) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:89) ~[aws-java-sdk-core-1.12.270.jar:?]",
elasticsearch-1  | "at java.lang.Class.forName0(Native Method) ~[?:?]",
elasticsearch-1  | "at java.lang.Class.forName(Class.java:391) ~[?:?]",
elasticsearch-1  | "at java.lang.Class.forName(Class.java:382) ~[?:?]",
elasticsearch-1  | "at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$static$0(S3RepositoryPlugin.java:62) ~[repository-s3-8.10.2.jar:8.10.2]",
elasticsearch-1  | "at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]",
elasticsearch-1  | "at org.elasticsearch.repositories.s3.S3RepositoryPlugin.<clinit>(S3RepositoryPlugin.java:56) ~[repository-s3-8.10.2.jar:8.10.2]",
elasticsearch-1  | "at jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) ~[?:?]",
elasticsearch-1  | "at jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160) ~[?:?]",
elasticsearch-1  | "at jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300) ~[?:?]",
elasticsearch-1  | "at jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103) ~[?:?]",
elasticsearch-1  | "at jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:200) ~[?:?]",
elasticsearch-1  | "at java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:547) ~[?:?]",
elasticsearch-1  | "at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:497) ~[?:?]",
elasticsearch-1  | "at java.lang.reflect.Constructor.newInstance(Constructor.java:484) ~[?:?]",
elasticsearch-1  | "at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:668) ~[elasticsearch-8.10.2.jar:?]",
elasticsearch-1  | "at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:518) ~[elasticsearch-8.10.2.jar:?]",
elasticsearch-1  | "at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:303) ~[elasticsearch-8.10.2.jar:?]",
elasticsearch-1  | "at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:173) ~[elasticsearch-8.10.2.jar:?]",
elasticsearch-1  | "at org.elasticsearch.plugins.PluginsService.lambda$getPluginsServiceCtor$16(PluginsService.java:702) ~[elasticsearch-8.10.2.jar:?]",
elasticsearch-1  | "at org.elasticsearch.node.Node.<init>(Node.java:431) ~[elasticsearch-8.10.2.jar:?]",
elasticsearch-1  | "at org.elasticsearch.node.Node.<init>(Node.java:338) ~[elasticsearch-8.10.2.jar:?]",
elasticsearch-1  | "at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:234) ~[elasticsearch-8.10.2.jar:?]",
elasticsearch-1  | "at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:234) ~[elasticsearch-8.10.2.jar:?]",
elasticsearch-1  | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:72) ~[elasticsearch-8.10.2.jar:?]"] }

[MLB-BIOMETRIC]

@gquintard Did you figure this out? I get the same error.

[gquintard]

@MLB-BIOMETRIC , not yet, but I'll be sure to update this if I have a way around

[sherifabdlnaby]

Hey folks, I am not able to reproduce this on my setup.

[gquintard]

@sherifabdlnaby , thanks for looking at this. I've try a fresh install an still getting this. Could you start from scratch, notably by either wiping all container images first, or at least pulling all the images?

[egahmad]

I'm having the same error

[nachten]

also same problem here

[samuellvicente]

same here

[zuizuihao]

not investigate the root cause, I just change to single-node discovery type, and also remove log4j2.properties, it works. this issue should related with cluster mode and logs.

[DemiBSel]

@gquintard
The logs you have shared don't seem to show the actual reason for the crash.
The stacktrace related to aws config can be ignored according to this issue : elastic/elasticsearch#62493
I also got sidetracked by it, upon looking at the logs up to the elasticsearch container crashing, I noticed another error.

{"type": "server", "timestamp": "2024-07-17T13:55:06,065Z", "level": "ERROR", "component": "o.e.b.Elasticsearch", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "node validation exception\n[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.\nbootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]" }
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elastdocker-cluster.log
{"type": "server", "timestamp": "2024-07-17T13:55:06,070Z", "level": "WARN", "component": "o.e.n.Node", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "unexpected exception while waiting for http server to close", 
"stacktrace": ["java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Can't move to stopped state when not started",
"at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]",
"at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]",
"at org.elasticsearch.node.Node.prepareForClose(Node.java:1754) ~[elasticsearch-8.10.2.jar:?]",
"at org.elasticsearch.bootstrap.Elasticsearch.shutdown(Elasticsearch.java:466) ~[elasticsearch-8.10.2.jar:?]",
"at java.lang.Thread.run(Thread.java:1623) ~[?:?]",
"Caused by: java.lang.IllegalStateException: Can't move to stopped state when not started",
"at org.elasticsearch.common.component.Lifecycle.canMoveToStopped(Lifecycle.java:128) ~[elasticsearch-8.10.2.jar:?]",
"at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:73) ~[elasticsearch-8.10.2.jar:?]",
"at org.elasticsearch.node.Node.lambda$prepareForClose$58(Node.java:1746) ~[elasticsearch-8.10.2.jar:?]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]",
"... 1 more"] }
{"type": "server", "timestamp": "2024-07-17T13:55:06,071Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "stopping ..." }
{"type": "server", "timestamp": "2024-07-17T13:55:06,084Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "stopped" }
{"type": "server", "timestamp": "2024-07-17T13:55:06,084Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "closing ..." }
{"type": "server", "timestamp": "2024-07-17T13:55:06,091Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "closed" }
{"type": "server", "timestamp": "2024-07-17T13:55:06,093Z", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "elastdocker-cluster", "node.name": "elastdocker-node-0", "message": "Native controller process has stopped - no new native processes can be started" }

ERROR: Elasticsearch exited unexpectedly, with exit code 78

🧐 node validation exception\n[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.\nbootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

According to README.md :

• For Linux Users it's recommended to set the following configuration (run as root)

  sysctl -w vm.max_map_count=262144
  

  By default, Virtual Memory is not enough.

I did just that, and the crashes stopped !

[gquintard]

hi, thank you for your answer, I won't be able to check if the solution works though as I have move to other tools and will lack the time to set up an environment again.

[j-perez]

Bumped up against this issue running WSL 2 on Windows 10.0.19045. In PS/cmd shell use the 'wsl' command with no args to access the docker vm and do the command above sysctl -w vm.max_map_count=262144.