[Bug]: OpenSearch Distribution builds are timing out

[gaiksaya]

Describe the bug

See below builds:

• https://build.ci.opensearch.org/view/Build/job/distribution-build-opensearch/10369/
• https://build.ci.opensearch.org/view/Build/job/distribution-build-opensearch/10368/
• https://build.ci.opensearch.org/view/Build/job/distribution-build-opensearch/10367/
  Looks like builds are timing out for all versions.

To reproduce

See above builds

Expected behavior

No response

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[gaiksaya]

Looks like bad agent node as all timeouts happened on i-037cd602ff5c46164 . The agent node does not exist anymore.
Triggered a new build to see if it goes through https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-build-opensearch/detail/distribution-build-opensearch/10370/pipeline

[gaiksaya]

# sudo journalctl -r | grep -i 'out of memory\|killed process'
Oct 09 02:14:07 ip-10-0-125-24.ec2.internal kernel: Out of memory: Killed process 16434 (java) total-vm:13441012kB, anon-rss:3964380kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:9280kB oom_score_adj:0
Oct 09 02:14:07 ip-10-0-125-24.ec2.internal kernel: Out of memory: Killed process 9652 (dbus-broker) total-vm:5208kB, anon-rss:140kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:52kB oom_score_adj:200
Oct 09 02:14:07 ip-10-0-125-24.ec2.internal kernel: Out of memory: Killed process 9643 (dbus-broker-lau) total-vm:9360kB, anon-rss:292kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:52kB oom_score_adj:200

[gaiksaya]

@peterzhuamazon has raised a PR to increase the memory #5086
Will check if this makes an difference. If not need to debug more.

[gaiksaya]

Closing this issue as builds are not timing out anymore.