Migrate from fleet-ci bucket to Ingest bucket

[jlind23]

In our Buildkite pipelines we are still relying on the fleet-ci buckets for DRA stage.
We need to stop using this one and rather start pushing artifacts to the ingest specific pipelines.
This change should be then back ported to the following branches:

• 7.17
• 8.12

https://github.com/elastic/fleet-server/blob/5e8a1e9f9f3cf9d7b97e8c41a7c6611b414eca55/.buildkite/hooks/pre-command#L11C1-L11C27

[jlind23]

@mrodm Looking at this it should be as easy as updating the vault value used to retrieved the gcs bucket name right?

Bucket name is retrieved here:

fleet-server/.buildkite/hooks/pre-command

Line 65 in 5e8a1e9

export JOB_GCS_BUCKET=$(retry 5 vault kv get -field plaintext ${JOB_GCS_BUCKET_SECRET_PATH})

And bucket is used here:

fleet-server/.buildkite/scripts/common.sh

Line 105 in 5e8a1e9

upload_packages_to_gcp_bucket() {

[mrodm]

It's also used in get_bucket_uri():

fleet-server/.buildkite/scripts/common.sh

Line 121 in 5e8a1e9

local baseUri="gs://${JOB_GCS_BUCKET}/jobs" #TODO: needs to add the "/buildkite" for rollback

The problem of updating the value directly in that vault path is that it could not be tested if that works or not in advance, and other builds (of any branch or PR) could fail.

It is needed to ensure that the current service account has the required permissions to upload/write files into the new bucket. The service account used here is retrieved from this vault path:

fleet-server/.buildkite/hooks/pre-command

Line 9 in 5e8a1e9

PRIVATE_CI_GCS_CREDENTIALS_PATH="kv/ci-shared/platform-ingest/private_ci_artifacts_gcs_credentials"

NOTE: Just realized that get_bucket_uri() function is duplicated in buildkite/scripts/common.sh, and it could be removed one of them:

fleet-server/.buildkite/scripts/common.sh

Lines 119 to 139 in 5e8a1e9

	get_bucket_uri() {
	local type=${1}
	local baseUri="gs://${JOB_GCS_BUCKET}/jobs" #TODO: needs to add the "/buildkite" for rollback
	if [[ ${type} == "snapshot" ]]; then
	local folder="commits"
	else
	local folder="${type}"
	fi
	bucketUri="${baseUri}/${folder}/${BUILDKITE_COMMIT}"
	}
	get_bucket_uri() {
	local type=${1}
	local baseUri="gs://${JOB_GCS_BUCKET}/jobs" #TODO: needs to add the "/buildkite" for rollback
	if [[ ${type} == "snapshot" ]]; then
	local folder="commits"
	else
	local folder="${type}"
	fi
	bucketUri="${baseUri}/${folder}/${BUILDKITE_COMMIT}"
	}