Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Development environment for Amazon Security Lake integration fails to start #664

Closed
AlexRuiz7 opened this issue Feb 3, 2025 · 0 comments · Fixed by #665 or #666
Closed

Development environment for Amazon Security Lake integration fails to start #664

AlexRuiz7 opened this issue Feb 3, 2025 · 0 comments · Fixed by #665 or #666
Labels
level/task Task issue request/operational Operational requests type/bug Bug issue

Comments

@AlexRuiz7
Copy link
Member

AlexRuiz7 commented Feb 3, 2025

Describe the bug
The Docker Compose development environment for the Amazon Security Lake integration does not start. The Logstash containers exits due to errors.

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalArgumentException", :message=>"org.logstash.config.ir.InvalidIRException: Config has duplicate Ids: \nID: output.stdout P[output-stdout{\"id\"=>\"output.stdout\", \"codec\"=>\"json_lines\"}|[file]/usr/share/logstash/pipeline/indexer-to-file.conf:24:4:```\nstdout { \n      id => \"output.stdout\"\n      codec => json_lines\n   }\n```]\nP[output-stdout{\"id\"=>\"output.stdout\", \"codec\"=>\"json_lines\"}|[file]/usr/share/logstash/pipeline/indexer-to-s3.conf:23:4:```\nstdout { \n      id => \"output.stdout\"\n      codec => json_lines\n   }\n```]\nID: output.file P[output-file{\"id\"=>\"output.file\", \"path\"=>\"/var/log/logstash/indexer-to-file-%{+YYYY-MM-dd-HH}.log\", \"file_mode\"=>420, \"codec\"=>\"json_lines\"}|[file]/usr/share/logstash/pipeline/indexer-to-file.conf:28:4:```\nfile {\n      id => \"output.file\"\n      path => \"/var/log/logstash/indexer-to-file-%{+YYYY-MM-dd-HH}.log\"\n      file_mode => 0644\n      codec => json_lines\n   }\n```]\nP[output-file{\"id\"=>\"output.file\", \"path\"=>\"/usr/share/logstash/logs/indexer-to-file-%{+YYYY-MM-dd-HH}.log\", \"file_mode\"=>420, \"codec\"=>\"json_lines\", \"flush_interval\"=>30}|[file]/usr/share/logstash/pipeline/indexer-to-s3.conf:46:4:```\nfile {\n      id => \"output.file\"\n      path => \"/usr/share/logstash/logs/indexer-to-file-%{+YYYY-MM-dd-HH}.log\"\n      file_mode => 0644\n      codec => json_lines\n      flush_interval => 30\n   }\n```]", :backtrace=>["org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:241)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:173)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:846)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1229)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:131)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:226)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:393)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:206)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:309)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base/java.lang.Thread.run(Thread.java:833)"]}
[2025-02-03T13:34:42,459][TRACE][logstash.agent           ] Converge results {:success=>false, :failed_actions=>["id: main, action_type: LogStash::PipelineAction::Create, message: org.logstash.config.ir.InvalidIRException: Config has duplicate Ids: \nID: output.stdout P[output-stdout{\"id\"=>\"output.stdout\", \"codec\"=>\"json_lines\"}|[file]/usr/share/logstash/pipeline/indexer-to-file.conf:24:4:```\nstdout { \n      id => \"output.stdout\"\n      codec => json_lines\n   }\n```]\nP[output-stdout{\"id\"=>\"output.stdout\", \"codec\"=>\"json_lines\"}|[file]/usr/share/logstash/pipeline/indexer-to-s3.conf:23:4:```\nstdout { \n      id => \"output.stdout\"\n      codec => json_lines\n   }\n```]\nID: output.file P[output-file{\"id\"=>\"output.file\", \"path\"=>\"/var/log/logstash/indexer-to-file-%{+YYYY-MM-dd-HH}.log\", \"file_mode\"=>420, \"codec\"=>\"json_lines\"}|[file]/usr/share/logstash/pipeline/indexer-to-file.conf:28:4:```\nfile {\n      id => \"output.file\"\n      path => \"/var/log/logstash/indexer-to-file-%{+YYYY-MM-dd-HH}.log\"\n      file_mode => 0644\n      codec => json_lines\n   }\n```]\nP[output-file{\"id\"=>\"output.file\", \"path\"=>\"/usr/share/logstash/logs/indexer-to-file-%{+YYYY-MM-dd-HH}.log\", \"file_mode\"=>420, \"codec\"=>\"json_lines\", \"flush_interval\"=>30}|[file]/usr/share/logstash/pipeline/indexer-to-s3.conf:46:4:```\nfile {\n      id => \"output.file\"\n      path => \"/usr/share/logstash/logs/indexer-to-file-%{+YYYY-MM-dd-HH}.log\"\n      file_mode => 0644\n      codec => json_lines\n      flush_interval => 30\n   }\n```]"], :successful_actions=>[]}

Moreover, the wazuh-indexer/integrations/docker/amazon-security-lake.yml file in the master branch was renamed to wazuh-indexer/integrations/docker/compose.amazon-security-lake.yml, but the file is still there. It needs to be removed.

It also happens for wazuh-indexer/integrations/amazon-security-lake/Dockerfile and wazuh-indexer/integrations/amazon-security-lake/aws-lambda.dockerfile.

To Reproduce
Steps to reproduce the behavior:

  1. Follow the README to start the environment.
  2. Check the logstash container
  3. See error
@AlexRuiz7 AlexRuiz7 added level/task Task issue request/operational Operational requests type/bug Bug issue labels Feb 3, 2025
@AlexRuiz7 AlexRuiz7 changed the title [BUG] Development environment for Amazon Security Lake integration fails to start [BUG] Development environment for Amazon Security Lake integration fails to start Feb 3, 2025
@AlexRuiz7 AlexRuiz7 changed the title [BUG] Development environment for Amazon Security Lake integration fails to start Development environment for Amazon Security Lake integration fails to start Feb 3, 2025
@AlexRuiz7 AlexRuiz7 linked a pull request Feb 3, 2025 that will close this issue
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
level/task Task issue request/operational Operational requests type/bug Bug issue
Projects
None yet
1 participant