Beats DRA pipeline #39126

pazone · 2024-04-22T14:49:26Z

Proposed commit message

Beats Packaging pipeline

Migrates packaging.groovy to buildkite.
Buildkite packaging pipeline is used to build and publish the packages for the Beats. The pipeline is triggered by a commit to the main or release branches.
The pipeline definition is located in the .buildkite/packaging.pipeline.yml

Triggers

Staging packaging DRA is triggered for the main and release branches.
Snapshot can be triggered for any branch by the /package comment in the PR. The release-manager dry-run will be used for PR builds.

Pipeline steps

Beats dashboards

Generates build/distributions/dependencies.csv and adds it to the beats-dashboards artifact. The dependencies.csv is required by the release-manager configuration

Packaging snapshot/staging

Builds the Beats packages for all supported platforms and architectures (mage package, mage ironbank)
Copies artifacts build/distributions/<beat>/ directory and adds it as an artifact. Where <beat> is the name of the beat
x-pack artifacts a also copied to build/distributions/<beat>/ directory, where <beat> is the name of the beat. For example, auditbeat, not x-pack/auditbeat

DRA publish

Downloads the artifacts from the packaging snapshot/staging step and publishes them to the Elastic DRA registry.

Snapshot publishing example (Dry run)

Beats-tester and e2e tests are not triggered

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

All Snapshot and staging artifacts are built, present, and properly transferred to the corresponding DRA steps
DRA dry run passes
All artifacts have the same size and permissions as on Jenkins. (Need to check one more time)

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify · 2024-04-22T14:50:19Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @pazone? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-04-22T14:58:47Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 14 min 28 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Co-authored-by: Dimitrios Liappis <[email protected]>

.buildkite/pull-requests.json

.buildkite/scripts/dra.sh

dliappis · 2024-04-24T15:20:03Z

.buildkite/scripts/dra.sh

+set -euo pipefail
+
+## TODO: Set to empty string when Jenkins is disabled
+DRY_RUN="--dry-run"


I recommend we change it to

if [[ "$DRY_RUN" == "false" ]]; then echo "--- Running in publish mode"; DRY_RUN=""; else echo "--- Running in dry-run mode"; DRY_RUN="--dry-run"; fi

to facilitate easy testing.

IMPORTANT because DRY_RUN maybe unset, we should also move set -euo pipefail from line 2 right after the suggested conditional.

dliappis

LGTM!

* Packaging pipieline resource * Revetred agentbeat pipeline deletion * Cleanup * Test auditbeat packaging pipeline * Fix steps keys * Fix steps keys * Fix env vars * Fix env vars * Fix env vars * Unified artifacts dir * Implemented DRA steps * Test filebeat * Test matrix * Aligned artifacts directories * Aligned artifacts directories * Aligned artifacts directories * Aligned artifacts directories * Debug * Debug * Package all beats * Test DRA snapshot * Test DRA snapshot * Rename artifacts * fix dashboards artifacts * Cleanup * cleanup * No need to install msi tools * Apply suggestions from code review Co-authored-by: Dimitrios Liappis <[email protected]> * Extraced platforms variables * Cleanup * Added RUN_SNAPSHOT condition * Cleanup * Boolean dry-run * Boolean dry-run --------- Co-authored-by: Dimitrios Liappis <[email protected]> (cherry picked from commit c6444db) # Conflicts: # .buildkite/packaging.pipeline.yml

This PR fixes a number of bugs from #39126

* Beats DRA pipeline (#39126) * Packaging pipieline resource * Revetred agentbeat pipeline deletion * Cleanup * Test auditbeat packaging pipeline * Fix steps keys * Fix steps keys * Fix env vars * Fix env vars * Fix env vars * Unified artifacts dir * Implemented DRA steps * Test filebeat * Test matrix * Aligned artifacts directories * Aligned artifacts directories * Aligned artifacts directories * Aligned artifacts directories * Debug * Debug * Package all beats * Test DRA snapshot * Test DRA snapshot * Rename artifacts * fix dashboards artifacts * Cleanup * cleanup * No need to install msi tools * Apply suggestions from code review Co-authored-by: Dimitrios Liappis <[email protected]> * Extraced platforms variables * Cleanup * Added RUN_SNAPSHOT condition * Cleanup * Boolean dry-run * Boolean dry-run --------- Co-authored-by: Dimitrios Liappis <[email protected]> (cherry picked from commit c6444db) # Conflicts: # .buildkite/packaging.pipeline.yml * Fixes for Buildkite packaging pipeline (#39207) This PR fixes a number of bugs from #39126 * Fix BK DRA pipeline (#39212) This commit fixes the error "interpolating remaining fields: Expected identifier to start with a letter, got /" * Fix permissions for packaging (#39218) This commit fixes the release-manager error > # > java.io.FileNotFoundException: /artifacts/build/distributions/... (Permission denied) by adjusting the permissions of parent directories. Additionally there are a few logging enhancements to improve debugging, as well as the introduction of a new optional `DRA_BRANCH` environment variable, to help running and debugging via PRs. Relates https://github.com/elastic/ingest-dev/issues/3095 --------- Signed-off-by: Alexandros Sapranidis <[email protected]> Co-authored-by: Dimitrios Liappis <[email protected]> * Added DEV flag to snapshot. prepare-release-manager.sh staging param (#39217) Added DEV flag to snapshot builds. Specified the workflow for prepare-release-manager.sh Relates: https://github.com/elastic/ingest-dev/issues/3095 --------- Signed-off-by: Alexandros Sapranidis <[email protected]> Co-authored-by: Pavel Zorin <[email protected]> Co-authored-by: Dimitrios Liappis <[email protected]> Co-authored-by: Alexandros Sapranidis <[email protected]>

Add DRA Buildkite packaging pipelines for 8.13 --------- Signed-off-by: Alexandros Sapranidis <[email protected]> Co-authored-by: Dimitrios Liappis <[email protected]> Co-authored-by: Alexandros Sapranidis <[email protected]>(cherry picked from commit c6444db)

Add DRA Buildkite packaging pipelines for 7.17 --------- Signed-off-by: Alexandros Sapranidis <[email protected]> Co-authored-by: Pavel Zorin <[email protected]> Co-authored-by: Dimitrios Liappis <[email protected]> Co-authored-by: Alexandros Sapranidis <[email protected]>

pazone added 4 commits April 22, 2024 16:48

Packaging pipieline resource

e2eccd1

Revetred agentbeat pipeline deletion

cbaba3e

Cleanup

7dc47b3

Test auditbeat packaging pipeline

ed599da

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 22, 2024

mergify bot assigned pazone Apr 22, 2024

pazone added 2 commits April 22, 2024 16:53

Fix steps keys

881fd97

Fix steps keys

75b4db9

pazone added 14 commits April 22, 2024 17:32

Fix env vars

720e129

Fix env vars

75e445b

Fix env vars

5790718

Unified artifacts dir

fefe674

Implemented DRA steps

3e312b6

Test filebeat

20b724b

Test matrix

1065e14

Aligned artifacts directories

f35efd3

Aligned artifacts directories

af251f7

Aligned artifacts directories

660decd

Aligned artifacts directories

cd8a604

Debug

7fd5c7a

Debug

0d1da08

Package all beats

5671e37

pazone added enhancement backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.13.0 Automated backport with mergify backport-v8.14.0 Automated backport with mergify labels Apr 23, 2024

pazone added 2 commits April 24, 2024 01:17

Test DRA snapshot

e583f3f

Test DRA snapshot

1377e5b

Apply suggestions from code review

dd3ecb5

Co-authored-by: Dimitrios Liappis <[email protected]>

alexsapran reviewed Apr 24, 2024

View reviewed changes

.buildkite/pull-requests.json Show resolved Hide resolved

pazone added 3 commits April 24, 2024 15:55

Extraced platforms variables

3e47581

Cleanup

e850d1d

Added RUN_SNAPSHOT condition

21793d9

dliappis reviewed Apr 24, 2024

View reviewed changes

.buildkite/scripts/dra.sh Outdated Show resolved Hide resolved

Cleanup

cc37121

dliappis reviewed Apr 24, 2024

View reviewed changes

pazone added 2 commits April 24, 2024 17:24

Boolean dry-run

15a04ca

Boolean dry-run

d13c0c5

dliappis approved these changes Apr 24, 2024

View reviewed changes

dliappis added the Team:Ingest-EngProd label Apr 24, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 24, 2024

pazone enabled auto-merge (squash) April 24, 2024 15:30

pazone merged commit c6444db into main Apr 24, 2024
85 of 107 checks passed

pazone deleted the ci_packaging_pipieline branch April 24, 2024 15:48

This was referenced Apr 24, 2024

[7.17](backport #39126) Beats DRA pipeline #39197

Merged

[8.13](backport #39126) Beats DRA pipeline #39198

Merged

[8.14](backport #39126) Beats DRA pipeline #39199

Merged

dliappis mentioned this pull request Apr 25, 2024

Fixes for packaging pipeline #39207

Merged

dliappis added a commit that referenced this pull request Apr 25, 2024

Fixes for Buildkite packaging pipeline (#39207)

9457f64

This PR fixes a number of bugs from #39126

pazone pushed a commit that referenced this pull request Apr 26, 2024

Fixes for Buildkite packaging pipeline (#39207)

634a2a6

This PR fixes a number of bugs from #39126

pazone pushed a commit that referenced this pull request Apr 26, 2024

Fixes for Buildkite packaging pipeline (#39207)

8616c4c

This PR fixes a number of bugs from #39126

pazone pushed a commit that referenced this pull request Apr 26, 2024

Fixes for Buildkite packaging pipeline (#39207)

a7192de

This PR fixes a number of bugs from #39126

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Beats DRA pipeline #39126

Beats DRA pipeline #39126

pazone commented Apr 22, 2024 •

edited by dliappis

Loading

mergify bot commented Apr 22, 2024

elasticmachine commented Apr 22, 2024 •

edited

Loading

Build stats

dliappis Apr 24, 2024 •

edited

Loading

dliappis left a comment

Beats DRA pipeline #39126

Beats DRA pipeline #39126

Conversation

pazone commented Apr 22, 2024 • edited by dliappis Loading

Proposed commit message

Beats Packaging pipeline

Triggers

Pipeline steps

Beats dashboards

Packaging snapshot/staging

DRA publish

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Apr 22, 2024

elasticmachine commented Apr 22, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

dliappis Apr 24, 2024 • edited Loading

Choose a reason for hiding this comment

dliappis left a comment

Choose a reason for hiding this comment

pazone commented Apr 22, 2024 •

edited by dliappis

Loading

elasticmachine commented Apr 22, 2024 •

edited

Loading

dliappis Apr 24, 2024 •

edited

Loading