changes to serverless pipelines

[juliaElastic]

What is the problem this PR solves?

// Please do not just reference an issue. Explain WHAT the problem this PR solves here.

How does this PR solve the problem?

// Explain HOW you solved the problem in your code. It is possible that during PR reviews this changes and then this section should be updated.

Changes based on https://github.com/elastic/hello-world-service/blob/main/.buildkite/pipeline.hello-world-service-tests.yaml

tests-qa: before qa promotion, this will trigger observability-perf tests in a new dev env
tests-staging: before staging promotion, this will trigger observability-perf tests in qa env

How to test this PR locally

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

Relates https://github.com/elastic/ingest-dev/issues/2201

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-08-24T16:47:59.738+0000

• Duration: 33 min 25 sec

Test stats 🧪

Test	Results
Failed	0
Passed	753
Skipped	1
Total	754

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[jlind23]

Can't we merge tests-qa and tests-staging into a single pipeline and use just pass down the environment variable?

[juliaElastic]

Can't we merge tests-qa and tests-staging into a single pipeline and use just pass down the environment variable?

There will be differences between qa and staging later (like different quality gates, creating dev env before qa deploy), so we should keep them separate.

[juliaElastic]

E2E tests are failing with this error:

Dockerfile:1
--
  | --------------------
  | 1 \| >>> ARG ELASTIC_AGENT_IMAGE # e.g. docker.elastic.co/cloud-release/elastic-agent-cloud:8.8.0-4671daa2-SNAPSHOT
  | 2 \|
  | 3 \|     FROM --platform=linux/amd64 ${ELASTIC_AGENT_IMAGE} as elastic_agent_amd64
  | --------------------
  | ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref ab9d21b5-6579-4544-b659-dc46c0d0a913::ilaucf5ghgrcdcklesdghfsrq: failed to walk /var/lib/docker/tmp/buildkit-mount480826891/cover/fleet-server-8.10.0-SNAPSHOT-linux-x86_64: lstat /var/lib/docker/tmp/buildkit-mount480826891/cover/fleet-server-8.10.0-SNAPSHOT-linux-x86_64: no such file or directory
  | make: *** [Makefile:344: build-e2e-agent-image] Error 1
  | 🚨 Error: The command exited with status 2
  | user command error: exit status 2

[juliaElastic]

/test

[juliaElastic]

/test

[jlind23]

buildkite test it