Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flaky Test]: Failed to prepare an instance – could not run "unzip agent-repo.zip -d agent" though SSH: wait: remote command exited without exit status or exit signal #4810

Closed
rdner opened this issue May 24, 2024 · 6 comments
Closed

[Flaky Test]: Failed to prepare an instance – could not run "unzip agent-repo.zip -d agent" though SSH: wait: remote command exited without exit status or exit signal #4810

rdner opened this issue May 24, 2024 · 6 comments
Assignees
@rdner
Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team

Comments

@rdner
Copy link
Member

rdner commented May 24, 2024 •
edited
Loading

Failing test case

Multiple upgrade tests

Error message

could not run "unzip agent-repo.zip -d agent" though SSH: wait: remote command exited without exit status or exit signal

Build

OS

Linux

Stacktrace and notes

>>> (linux-amd64-ubuntu-2204-upgrade) Failed for instance linux-amd64-ubuntu-2204-upgrade (@ 34.16.95.179): failed to execute tests on instance ogc-linux-amd64-ubuntu-2204-upgrade-9d60: error running sudo tests: failed to fetched test output at $HOME/agent/build/TEST-go-remote-linux-amd64-ubuntu-2204-upgrade-sudo.integration.out
Error: error running test: failed to copy files to instance ogc-linux-arm64-ubuntu-2204-container-b24e: failed to unzip agent-repo.zip to agent directory: could not run "unzip agent-repo.zip -d agent" though SSH: wait: remote command exited without exit status or exit signal (stdout: Archive:  agent-repo.zip
  inflating: agent/.agent-versions.json
  inflating: agent/.buildkite/hooks/post-checkout
  inflating: agent/.buildkite/hooks/post-checkout.ps1
  inflating: agent/.buildkite/hooks/pre-command
  inflating: agent/.buildkite/hooks/pre-command.ps1
  inflating: agent/.buildkite/hooks/pre-exit
  inflating: agent/.buildkite/integration.pipeline.yml
  inflating: agent/.buildkite/misc/gce-cleanup.yml
  inflating: agent/.buildkite/pipeline.elastic-agent-binary-dra.yml
  inflating: agent/.buildkite/pipeline.elastic-agent-gce-cleanup.yml
  inflating: agent/.buildkite/pipeline.elastic-agent-package.yml
  inflating: agent/.buildkite/pipeline.yml
  inflating: agent/.buildkite/pull-requests.json
  inflating: agent/.buildkite/scripts/common.sh
  inflating: agent/.buildkite/scripts/install-gh.sh
  inflating: agent/.buildkite/scripts/install-kind.sh
  inflating: agent/.buildkite/scripts/install-kubectl.sh
  inflating: agent/.buildkite/scripts/steps/beats_tests.sh
  inflating: agent/.buildkite/scripts/steps/build-agent-core.sh
  inflating: agent/.buildkite/scripts/steps/check-ci.sh
  inflating: agent/.buildkite/scripts/steps/dra-publish.sh
  inflating: agent/.buildkite/scripts/steps/gce-cleanup.sh
  inflating: agent/.buildkite/scripts/steps/integration_tests.sh
  inflating: agent/.buildkite/scripts/steps/ironbank-cp-workaround.sh
  inflating: agent/.buildkite/scripts/steps/k8s-tests.sh
  inflating: agent/.buildkite/scripts/steps/merge.sh
  inflating: agent/.buildkite/scripts/steps/package.sh
  inflating: agent/.buildkite/scripts/steps/sync-k8s.sh
  inflating: agent/.buildkite/scripts/steps/unit-tests.ps1
  inflating: agent/.buildkite/scripts/steps/unit-tests.sh
  inflating: agent/.buildkite/scripts/unset-secrets.sh
  inflating: agent/.ci/bump-golang.yml
  inflating: agent/.ci/scripts/install-go.bat
  inflating: agent/.ci/scripts/install-go.sh
  inflating: agent/.dockerignore
  inflating: agent/.editorconfig
  inflating: agent/.gitattributes
  inflating: agent/.github/CODEOWNERS
  inflating: agent/.github/ISSUE_TEMPLATE/bug-report.md
  inflating: agent/.github/ISSUE_TEMPLATE/feature-request.md
  inflating: agent/.github/ISSUE_TEMPLATE/flaky-test.yml
  inflating: agent/.github/ISSUE_TEMPLATE/question.md
  inflating: agent/.github/PULL_REQUEST_TEMPLATE.md
  inflating: agent/.github/dependabot.yml
  inflating: agent/.github/stale.yml
  inflating: agent/.github/updatecli-bump-golang.yml
  inflating: agent/.github/workflows/add-issues-to-ingest-board.yml
  inflating: agent/.github/workflows/bump-agent-versions.sh
  inflating: agent/.github/workflows/bump-agent-versions.yml
  inflating: agent/.github/workflows/bump-golang.yml
  inflating: agent/.github/workflows/fragment-in-pr.yml
  inflating: agent/.github/workflows/golangci-lint.yml
  inflating: agent/.github/workflows/post-dependabot.yml
  inflating: agent/.gitignore
  inflating: agent/.go-version
  inflating: agent/.golangci.yml
  inflating: agent/.mergify.yml
  inflating: agent/.mockery.yaml
  inflating: agent/.package-version
  inflating: agent/.pre-commit-config.yaml
  inflating: agent/CHANGELOG.asciidoc
  inflating: agent/CONTRIBUTING.md
  inflating: agent/Dockerfile
  inflating: agent/Dockerfile.skaffold
  inflating: agent/GPG-KEY-elasticsearch
  inflating: agent/GUIDELINES.md
  inflating: agent/LICENSE.txt
  inflating: agent/Makefile
  inflating: agent/NOTICE.txt
  inflating: agent/README.md
  inflating: agent/Vagrantfile
  inflating: agent/_meta/config/common.p1.yml.tmpl
  inflating: agent/_meta/config/common.p2.yml.tmpl
  inflating: agent/_meta/config/common.reference.p1.yml.tmpl
  inflating: agent/_meta/config/common.reference.p2.yml.tmpl
  inflating: agent/_meta/config/elastic-agent.docker.yml.tmpl
  inflating: agent/_meta/config/elastic-agent.reference.yml.tmpl
  inflating: agent/_meta/config/elastic-agent.yml.tmpl
  inflating: agent/_meta/config/providers.yml.tmpl
  inflating: agent/_meta/elastic-agent.fleet.yml
  inflating: agent/_meta/elastic-agent.yml
  inflating: agent/catalog-info.yaml
  inflating: agent/changelog/8.10.0.asciidoc
  inflating: agent/changelog/8.10.0.yaml
  inflating: agent/changelog/8.10.1.asciidoc
  inflating: agent/changelog/8.10.1.yaml
  inflating: agent/changelog/8.10.2.asciidoc
  inflating: agent/changelog/8.10.2.yaml
  inflating: agent/changelog/8.10.3.asciidoc
  inflating: agent/changelog/8.10.3.yaml
  inflating: agent/changelog/8.10.4.asciidoc
  inflating: agent/changelog/8.10.4.yaml
  inflating: agent/changelog/8.11.0.asciidoc
  inflating: agent/changelog/8.11.0.yaml
  inflating: agent/changelog/8.11.1.asciidoc
  inflating: agent/changelog/8.11.1.yaml
  inflating: agent/changelog/8.11.2.asciidoc
  inflating: agent/changelog/8.11.2.yaml
  inflating: agent/changelog/8.11.3.asciidoc
  inflating: agent/changelog/8.11.3.yaml
  inflating: agent/changelog/8.11.4.asciidoc
  inflating: agent/changelog/8.11.4.yaml
  inflating: agent/changelog/8.5.0.yaml
  inflating: agent/changelog/8.5.1.yaml
  inflating: agent/changelog/8.5.2.yaml
  inflating: agent/changelog/8.6.0.yaml
  inflating: agent/changelog/8.6.1.yaml
  inflating: agent/changelog/8.6.2.yaml
  inflating: agent/changelog/8.7.0.yaml
  inflating: agent/changelog/8.7.1.yaml
  inflating: agent/changelog/8.8.0.asciidoc
  inflating: agent/changelog/8.8.0.yaml
  inflating: agent/changelog/8.8.1.asciidoc
  inflating: agent/changelog/8.8.1.yaml
  inflating: agent/changelog/8.8.2.asciidoc
  inflating: agent/changelog/8.8.2.yaml
  inflating: agent/changelog/8.9.0.asciidoc
  inflating: agent/changelog/8.9.0.yaml
  inflating: agent/changelog/8.9.1.asciidoc
  inflating: agent/changelog/8.9.1.yaml
  inflating: agent/changelog/8.9.2.asciidoc
  inflating: agent/changelog/8.9.2.yaml
  inflating: agent/changelog/fragments/1689328899-Elastic-Agent-container-runs-on-Azure-Container-Instances-.yaml
  inflating: agent/changelog/fragments/1693592658-concurrent-install.yaml
  inflating: agent/changelog/fragments/1694692246-hintspercontainer.yaml
  inflating: agent/changelog/fragments/1695389490-Support-flattened-data_stream.-fields.yaml
  inflating: agent/changelog/fragments/1696249276-Start-stop-monitoring-server-based-on-monitoring-config.yaml
  inflating: agent/changelog/fragments/1696935756-add-default-host.yaml
  inflating: agent/changelog/fragments/1697508156-upgrade-details-in-status.yaml
  inflating: agent/changelog/fragments/1697554456-Improve-upgrade-watcher-to-no-longer-need-root-access.yaml
  inflating: agent/changelog/fragments/1697565877-progress-race-fixes.yaml
  inflating: agent/changelog/fragments/1697588586-upgrade-details-in-diagnostics.yaml
  inflating: agent/changelog/fragments/1697662209-duplicate-tags.yaml
  inflating: agent/changelog/fragments/1698851635-Only-try-to-download-upgrade-locally-when-file---prefix-is-used-for-source-URI.yaml
  inflating: agent/changelog/fragments/1699016628-send-upgrade-details-to-fleet.yaml
  inflating: agent/changelog/fragments/1700520158-dynamic-timeout-for-diags.yaml
  inflating: agent/changelog/fragments/1700851577-Surface-errors-during-Agent_s-enroll-process-failing-if-any-happens.yaml
  inflating: agent/changelog/fragments/1701091034-add-cache-for-secrets.yaml
  inflating: agent/changelog/fragments/1701293846-upgrade-details-downloading.yaml
  inflating: agent/changelog/fragments/1701445320-preserve-build-metadata-in-upgrade-version-strings.yaml
  inflating: agent/changelog/fragments/1701882851-performance-presets.yaml
  inflating: agent/changelog/fragments/1701975889-Fix-component-control-protocol-to-allow-checkin-to-be-chunked-across-multiple-messages.yaml
  inflating: agent/changelog/fragments/1702463797-When-a-scheduled-upgrade-expires,-set-the-upgrade-state-to-failed.yaml
  inflating: agent/changelog/fragments/1702501610-Adjust-control-socket-path-into-the-installation-directory.yaml
  inflating: agent/changelog/fragments/1703157372-a-changeset-filename.yaml
  inflating: agent/changelog/fragments/1704382549-Upgrade-elastic-agent-libs-to-v0.7.3.yaml
  inflating: agent/changelog/fragments/1704397113-Add-mTLS-flags-to-install-enroll.yaml
  inflating: agent/changelog/fragments/1704824761-fleet-config-change-logging.yaml
  inflating: agent/changelog/fragments/1704994055-add-error-statements.yaml
  inflating: agent/changelog/fragments/1705605734-fix-creation-of-directories-when-unpacking-tar.gz-packages.yaml
  inflating: agent/changelog/fragments/1706123313-on-Windows-prevent-uninstall-from-within-installed-directory.yaml
  inflating: agent/changelog/fragments/1706213166-fqdn-ctx.yaml
  inflating: agent/changelog/fragments/1706703918-Add-the-full-version-number-to-the-installation-directory-name.yaml
  inflating: agent/changelog/fragments/1707070032-providers-default-disable.yaml
  inflating: agent/changelog/fragments/1707405322-newpodupdater.yaml
  inflating: agent/changelog/fragments/1707481157-etw-filebeat-spec.yaml
  inflating: agent/changelog/fragments/1707754331-websocket-filebeat-spec.yaml
  inflating: agent/changelog/fragments/1707857612-windows_stop_before_uninstall.yaml
  inflating: agent/changelog/fragments/1707935415-Add-mTLS-env-var-settings-for-containers.yaml
  inflating: agent/changelog/fragments/1707951532-change-install-marker-creation.yaml
  inflating: agent/changelog/fragments/1707999059-Fixes-an-issue-where-the-Elastic-Agent-did-not-utilize-the-download-settings-when-downloading-the-artifact-signature-file..yaml
  inflating: agent/changelog/fragments/1708594284-Increase-timeout-for-file-removal-during-uninstall.yaml
  inflating: agent/changelog/fragments/1708628530-add-monitoring-beats-to-metrics.yaml
  inflating: agent/changelog/fragments/1708732907-add-postrm-script-to-deb-and-rpm-packages.yaml
  inflating: agent/changelog/fragments/1709656001-logshints.yaml
  inflating: agent/changelog/fragments/1709824109-k8s-secret-provider-trigger-signal.yaml
  inflating: agent/changelog/fragments/1710153529-Add-CPU-profile-collection-to-diagnostics-action-handler.yaml
  inflating: agent/changelog/fragments/1710179479-Enable---unprivileged-on-Mac-OS.yaml
  inflating: agent/changelog/fragments/1710868294-ucfg-parser-fix.yaml
  inflating: agent/changelog/fragments/1711355973-Introduce-non-grouping-for-input-spec.yaml
  inflating: agent/changelog/fragments/1711474205-Reduce-false-positives-in-logging-an-API-switch-request-from-Fleet-server.yaml
  inflating: agent/changelog/fragments/1711653910-add-liveness-endpoint.yaml
  inflating: agent/changelog/fragments/1712176768-Upgrade-to-Go-1.21.10.yaml
  inflating: agent/changelog/fragments/1712305270-fix-upgrade-cmd-fail-when-disconnected.yaml
  inflating: agent/changelog/fragments/1712324173-improve-enroll-stability.yaml
  inflating: agent/changelog/fragments/1712583231-leader-election-issue.yaml
  inflating: agent/changelog/fragments/1712737337-always-launch-more-recent-watcher.yaml
  inflating: agent/changelog/fragments/1712949156-Switch-to-agentbeat.yaml
  inflating: agent/changelog/fragments/1713157536-upgrade-otel-collector-contrib-package.yaml
  inflating: agent/changelog/fragments/1713257367-Remove-beat-state-metricset-from-elastic-agent-monitoring.yaml
  inflating: agent/changelog/fragments/1713289288-Removed-cloud-defend-from-.tar.gz-archives.yaml
  inflating: agent/changelog/fragments/1713946896-remove-mandatory-ns-node-permissions.yaml
  inflating: agent/changelog/fragments/1714717717-Reduce-agent-logs-by-default.yaml
  inflating: agent/changelog/fragments/1715158488-use-snapshot-api.yaml
  inflating: agent/changelog/fragments/1715178093-otel-mode-add-filter-processor.yaml
  inflating: agent/changelog/fragments/1715178161-otel-mode-add-elasticsearch-exporter.yaml
  inflating: agent/changelog/fragments/1715257644-Set-log-level-from-fleet-policy.yaml
  inflating: agent/changelog/fragments/1715259944-Add-agent-info.yaml-to-diagnostics-bundle.yaml
  inflating: agent/changelog/fragments/1715266989-Make-delayed-enrollment-try-indefinitely.yaml
  inflating: agent/changelog/fragments/1716227441-Fix-delay-enrollment-to-work-in-unprivileged-mode-on-Windows.yaml
  inflating: agent/control_v1.proto
  inflating: agent/control_v2.proto
  inflating: agent/deploy/kubernetes/Makefile
  inflating: agent/deploy/kubernetes/README.md
  inflating: agent/deploy/kubernetes/creator_k8s_manifest.sh
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed/base/elastic-agent-managed-daemonset.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed/base/elastic-agent-managed-role-binding.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed/base/elastic-agent-managed-role.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed/base/elastic-agent-managed-service-account.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed/base/kustomization.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-managed/kustomization.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/elastic-agent-standalone-daemonset-configmap.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/elastic-agent-standalone-daemonset.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/elastic-agent-standalone-ksm-daemonset-configmap.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/elastic-agent-standalone-ksm-statefulset-configmap.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/elastic-agent-standalone-role-binding.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/elastic-agent-standalone-role.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/elastic-agent-standalone-service-account.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/base/kustomization.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/default/elastic-agent-standalone/kustomization.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-managed/base/elastic-agent-managed-daemonset.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-managed/base/elastic-agent-managed-role-binding.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-managed/base/elastic-agent-managed-role.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-managed/base/elastic-agent-managed-service-account.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-managed/base/kustomization.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-managed/extra/elastic-agent-managed-statefulset.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-managed/kustomization.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/base/elastic-agent-standalone-daemonset.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/base/elastic-agent-standalone-ksm-daemonset-configmap.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/base/elastic-agent-standalone-ksm-statefulset-configmap.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/base/elastic-agent-standalone-role-binding.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/base/elastic-agent-standalone-role.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/base/elastic-agent-standalone-service-account.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/base/kustomization.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/extra/elastic-agent-standalone-statefulset.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-kustomize/ksm-autosharding/elastic-agent-standalone/kustomization.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-managed-kubernetes.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-managed/elastic-agent-managed-daemonset.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-managed/elastic-agent-managed-role-binding.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-managed/elastic-agent-managed-role.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-managed/elastic-agent-managed-service-account.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-daemonset-configmap.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-daemonset.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-ksm-daemonset-configmap.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-ksm-statefulset-configmap.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-role-binding.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-role.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/elastic-agent-standalone-service-account.yaml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/activemq.yml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/apache.yml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/cassandra.yml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/cef.yml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/checkpoint.yml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/cockroachdb.yml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/container_logs.yml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/crowdstrike.yml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/cyberarkpas.yml
  inflating: agent/deploy/kubernetes/elastic-agent-standalone/templates.d/elasticsearch.yml
, stderr: )
~/builds/bk-agent-prod-gcp-1716448940794043269/elastic/elastic-agent-extended-testing
Cannot generate HTML test report: build/TEST-go-integration.xml not found
@rdner rdner added Team:Elastic-Agent Label for the Agent team flaky-test Unstable or unreliable test cases. labels May 24, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

@rdner
Copy link
Member Author

rdner commented Jun 3, 2024

Should be fixed by addressing #4794

@cmacknz
Copy link
Member

cmacknz commented Jun 4, 2024 •
edited
Loading

This is an interesting one for trying to solve with retries, the actual failing command is an unzip which should never fail. It is potentially resource intensive on the host machine, and could take longer than other commands.

I went looking for patterns and I see that every single failure here is on a linux-arm64-ubuntu-2204 machine. So the question we need to start asking is what is different about these machines? It also seems we fail at the exact same point every time in trying to unzip the agent repository after successfully doing the rest of the machine preparation (apt-get install)

Are they in a different region? Are they a weaker instance type? Can changing these things help?

Here are some snippets before I grabbed showing the instance type and timestamp showing this pattern:

2024-05-21 20:08:21 UTC | >>> (linux-arm64-ubuntu-2204-upgrade) ssh exec error: "could not run \"sudo apt-get install -y build-essential unzip\" though SSH: wait: remote command exited without exit status or exit signal", will try again in 5s
-- | --

2024-05-22 15:16:15 UTC | >>> (linux-arm64-ubuntu-2204-deb) Failed to copy files instance: failed to unzip agent-repo.zip to agent directory: could not run "unzip agent-repo.zip -d agent" though SSH: wait: remote command exited without exit status or exit signal (stdout: Archive:  agent-repo.zip
-- | --


2024-05-22 15:31:11 UTC | >>> (linux-arm64-ubuntu-2204-upgrade) Failed to copy files instance: failed to unzip agent-repo.zip to agent directory: could not run "unzip agent-repo.zip -d agent" though SSH: wait: remote command exited without exit status or exit signal (stdout: Archive:  agent-rep
-- | --

2024-05-23 07:52:31 UTC | >>> (linux-arm64-ubuntu-2204-container) ssh exec error: "could not run \"sudo apt-get install -y build-essential unzip\" though SSH: wait: remote command exited without exit status or exit signal", will try again in 5s
-- | --

2024-05-24 11:32:22 UTC | >>> (linux-arm64-ubuntu-2204-upgrade) Failed to copy files instance: failed to unzip agent-repo.zip to agent directory: could not run "unzip agent-repo.zip -d agent" though SSH: wait: remote command exited without exit status or exit signal (stdout: Archive:  agent-repo.zip
-- | --

2024-05-29 11:00:50 UTC | >>> (linux-arm64-ubuntu-2204-deb) ssh exec error: "could not run \"sudo apt-get install -y build-essential unzip\" though SSH: wait: remote command exited without exit status or exit signal", will try again in 5s
-- | --

@cmacknz
Copy link
Member

cmacknz commented Jun 4, 2024

The amd64 machines are e2-standard-2

elastic-agent/pkg/testing/ogc/supported.go

Lines 23 to 35 in 0b99ccb

{
OS: define.OS{
Type: define.Linux,
Arch: define.AMD64,
Distro: runner.Ubuntu,
Version: "22.04",
},
Provider: Google,
InstanceSize: "e2-standard-2", // 2 amd64 cpus
RunsOn: "ubuntu-2204-lts",
Username: "ubuntu",
RemotePath: "/home/ubuntu/agent",
},

The arm64 machines are t2a-standard-2

elastic-agent/pkg/testing/ogc/supported.go

Lines 49 to 61 in 0b99ccb

{
OS: define.OS{
Type: define.Linux,
Arch: define.ARM64,
Distro: runner.Ubuntu,
Version: "22.04",
},
Provider: Google,
InstanceSize: "t2a-standard-2", // 2 arm64 cpus
RunsOn: "ubuntu-2204-lts-arm64",
Username: "ubuntu",
RemotePath: "/home/ubuntu/agent",
},

@cmacknz
Copy link
Member

cmacknz commented Jun 4, 2024

Reading https://cloud.google.com/compute/docs/general-purpose-machines#t2a_machines this jumped out:

The Tau T2A machine series doesn't support simultaneous multithreading (SMT); each vCPU is equivalent to an entire core.

The T2A's have half the effective cores that the E2's do (which presumably support SMT/hyperthreading).

I wonder if bumping up to t2a-standard-4 would fix this so there is an equal amount of compute between the Linux machines we use.

@cmacknz cmacknz mentioned this issue Jun 4, 2024
Merged
@rdner rdner self-assigned this Jun 4, 2024
@rdner
Copy link
Member Author

rdner commented Jun 5, 2024

Should be fixed by #4852 for now following with the full resolution in https://github.com/elastic/ingest-dev/issues/3439 later.

@rdner rdner closed this as completed Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rdner rdner

Labels
flaky-test Unstable or unreliable test cases. Team:Elastic-Agent Label for the Agent team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rdner @cmacknz @elasticmachine