[zuul] Add 4.16 jobs #609

elfiesmelfie · 2024-07-05T16:25:25Z

Depends-On: #623

softwarefactory-project-zuul · 2024-07-05T17:22:36Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/868a2d18bb6148c38078874c2d4c9898

❌ stf-crc-ocp_416-local_build-index_deploy FAILURE in 52m 59s

softwarefactory-project-zuul · 2024-07-08T15:55:53Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/1d5777a265bc4937bc76a5ececfba005

❌ stf-crc-ocp_416-local_build RETRY_LIMIT in 6m 49s
❌ stf-crc-ocp_416-local_build-index_deploy RETRY_LIMIT in 6m 51s

elfiesmelfie · 2024-07-08T16:30:56Z

The local_build-index_deploy job is failing on building sg-core due to a resource limitation.

Log Tail:	The container could not be located when the pod was terminated
Events:
  Type		Reason		Age	From			Message
  ----		------		----	----			-------
  Normal	Scheduled	98s	default-scheduler	Successfully assigned service-telemetry/sg-core-5-build to crc
  Normal	AddedInterface	97s	multus			Add eth0 [10.217.0.70/23] from ovn-kubernetes
  Normal	Pulled		97s	kubelet			Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cee9993b6fc030e73c326a8561c9cc53b357e626552ecde2bca239d92963c136" already present on machine
  Normal	Started		96s	kubelet			Started container git-clone
  Normal	Created		96s	kubelet			Created container git-clone
  Normal	BuildStarted	96s	build-controller	Build service-telemetry/sg-core-5 is now running
  Normal	Pulled		95s	kubelet			Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cee9993b6fc030e73c326a8561c9cc53b357e626552ecde2bca239d92963c136" already present on machine
  Normal	Created		94s	kubelet			Created container manage-dockerfile
  Normal	Started		94s	kubelet			Started container manage-dockerfile
  Normal	Pulled		94s	kubelet			Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cee9993b6fc030e73c326a8561c9cc53b357e626552ecde2bca239d92963c136" already present on machine
  Normal	Created		93s	kubelet			Created container docker-build
  Normal	Started		93s	kubelet			Started container docker-build
  Warning	Evicted		12s	kubelet			The node was low on resource: ephemeral-storage. Threshold quantity: 4902142351, available: 4698036Ki. Container docker-build was using 200Ki, request is 0, has larger consumption of ephemeral-storage.
  Normal	Killing		12s	kubelet			Stopping container docker-build
  Normal	BuildFailed	11s	build-controller	Build service-telemetry/sg-core-5 failed

https://logserver.rdoproject.org/09/609/aad0abff5a8803ba01e07242023520e1a43014ef/github-check/stf-crc-ocp_416-local_build-index_deploy/b188fab/controller/controller/post_oc_describe_build_sg-core-5.log

danpawlik · 2024-07-09T11:09:25Z

recheck

softwarefactory-project-zuul · 2024-07-09T14:56:05Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/de47a2af1ba4415d831b981d8d5bcb0b

❌ stf-crc-ocp_416-local_build FAILURE in 1h 00m 27s
❌ stf-crc-ocp_416-local_build-index_deploy FAILURE in 57m 33s
❌ stf-crc-ocp_416-nightly_bundles FAILURE in 52m 30s

elfiesmelfie · 2024-07-10T18:53:52Z

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/de47a2af1ba4415d831b981d8d5bcb0b

❌ stf-crc-ocp_416-local_build FAILURE in 1h 00m 27s ❌ stf-crc-ocp_416-local_build-index_deploy FAILURE in 57m 33s ❌ stf-crc-ocp_416-nightly_bundles FAILURE in 52m 30s

Failure in local_build and local_build-index_deploy was due to not enough ephemeral storage available to build sg-core.
@danpawlik Are there some larger flavors for the OCP4.16 images?

https://logserver.rdoproject.org/09/609/5e9b58537a257df6c41ffec3ccf5a4c5b7f12e67/github-check/stf-crc-ocp_416-local_build-index_deploy/40126b0/controller/controller/post_oc_describe_build_sg-core-4.log

softwarefactory-project-zuul · 2024-07-11T13:02:26Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/672f3f3942474d56a4de13021f54460b

❌ stf-crc-ocp_416-local_build RETRY_LIMIT in 13m 16s
❌ stf-crc-ocp_416-local_build-index_deploy FAILURE in 53m 24s
❌ stf-crc-ocp_416-nightly_bundles FAILURE in 52m 47s

softwarefactory-project-zuul · 2024-07-11T15:30:08Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/428546996b2d4a7589b0f8dad745c538

❌ stf-crc-ocp_416-local_build FAILURE in 59m 51s
❌ stf-crc-ocp_416-local_build-index_deploy RETRY_LIMIT in 1h 02m 19s
❌ stf-crc-ocp_414-nightly_bundles RETRY_LIMIT in 1h 02m 01s
❌ stf-crc-ocp_416-nightly_bundles RETRY_LIMIT in 16m 13s

softwarefactory-project-zuul · 2024-07-11T20:50:47Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/0fe2beec16c549dca1047c8781e4bcc7

❌ stf-crc-ocp_416-local_build FAILURE in 57m 49s
❌ stf-crc-ocp_416-local_build-index_deploy RETRY_LIMIT in 7m 06s
❌ stf-crc-ocp_414-nightly_bundles RETRY_LIMIT in 1h 01m 52s
❌ stf-crc-ocp_416-nightly_bundles RETRY_LIMIT in 1h 02m 27s

softwarefactory-project-zuul · 2024-07-12T12:35:11Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/79dfdc2e86244ceea6958fc3629aae7a

❌ stf-crc-ocp_416-local_build ERROR Failed to update project infrawatch/service-telemetry-operator in 3m 00s
❌ stf-crc-ocp_416-local_build-index_deploy MERGE_CONFLICT in 1m 05s
❌ stf-crc-ocp_416-nightly_bundles-index_deploy ERROR Failed to update project infrawatch/service-telemetry-operator in 3m 40s
❌ stf-crc-ocp_414-nightly_bundles MERGE_CONFLICT in 1m 04s
❌ stf-crc-ocp_416-nightly_bundles ERROR Failed to update project infrawatch/service-telemetry-operator in 1m 01s

softwarefactory-project-zuul · 2024-07-16T13:39:55Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/ci-framework for 2019,c43a3f650748ed20e0da8fd5a75d6e51000d363e

elfiesmelfie · 2024-07-16T15:20:21Z

recheck

softwarefactory-project-zuul · 2024-07-16T15:20:29Z

Zuul encountered a syntax error while parsing its
configuration in the repo infrawatch/service-telemetry-operator on branch master. The
problem was:

Job stf-crc-ocp_416-nightly_bundles-index_deploy not defined

The problem appears in the the "stf-crc-jobs" project-template stanza:

project-template:
name: stf-crc-jobs
description: |
STF CRC jobs that build and deploy STF
github-check:
jobs:
#- stf-crc-ocp_412-local_build
#- stf-crc-ocp_414-local_build
- stf-crc-ocp_416-local_build
...

in "infrawatch/service-telemetry-operator/.zuul.yaml@master", line 221

softwarefactory-project-zuul · 2024-07-16T17:29:12Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/dbc2079ad7bf4116b677fd00f6e71eed

❌ stf-crc-ocp_416-local_build FAILURE in 52m 03s
❌ stf-crc-ocp_416-local_build-index_deploy FAILURE in 54m 54s
❌ stf-crc-ocp_416-nightly_bundles-index_deploy FAILURE in 45m 30s
❌ stf-crc-ocp_416-nightly_bundles FAILURE in 44m 31s

softwarefactory-project-zuul · 2024-07-17T18:26:55Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/9c3e469d90fa4b51b6a7554a21ebc88e

❌ stf-crc-ocp_416-nightly_bundles POST_FAILURE in 44m 36s

softwarefactory-project-zuul · 2024-07-18T13:40:28Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/738c16ed17da4efc8bd7e96fa73abd09

❌ stf-crc-ocp_416-nightly_bundles RETRY_LIMIT in 2m 29s

softwarefactory-project-zuul · 2024-07-18T16:44:07Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/527617ad5dde4af1bf45079ba0d063cb

❌ stf-crc-ocp_416-nightly_bundles FAILURE in 44m 23s

softwarefactory-project-zuul · 2024-07-19T13:04:29Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/5c12a5bce3b74584be37291a3b944de1

✔️ stf-crc-ocp_414-nightly_bundles SUCCESS in 26m 03s
❌ stf-crc-ocp_416-nightly_bundles FAILURE in 44m 44s

elfiesmelfie · 2024-07-23T17:26:21Z

recheck

softwarefactory-project-zuul · 2024-07-23T18:11:29Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/cba59c26860043ee8d2fec15089e255d

✔️ stf-crc-ocp_414-nightly_bundles SUCCESS in 27m 46s
❌ stf-crc-ocp_416-nightly_bundles FAILURE in 44m 03s

softwarefactory-project-zuul · 2024-07-24T18:36:10Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/4a1a7cabb1f94fe0b3ac484587f8ccbc

✔️ stf-crc-ocp_414-local_build SUCCESS in 33m 10s
❌ stf-crc-ocp_416-local_build FAILURE in 52m 22s

csibbitt · 2024-07-24T18:46:11Z

recheck

This is to see if sg-core just needs more resource to build. Eventually, the size will be reduced again to not-the-largest-available.

All the jobs are currently failing for the same reason, disable all but the fastest one to save resources

Switching from nightly bundles to local builds in order to test 4.16 fixes

vkmc

Minor nit, otherwise looks good. Thanks Emma!

vkmc · 2024-07-29T08:53:42Z

ci/README.md

@@ -15,7 +15,6 @@ Two scenarios run:

 Each of these scenarios run across the following OCP versions:
 - 4.12
- 4.13
 - 4.14


Shall we add 4.16 here?

elfiesmelfie force-pushed the efoley-zuul-ocp-4.16 branch from 44be5a2 to 84c8b96 Compare July 16, 2024 13:39

csibbitt mentioned this pull request Jul 24, 2024

Create our own token for prometheus-stf SA #623

Merged

csibbitt force-pushed the efoley-zuul-ocp-4.16 branch from 33ca26e to 32ec6e1 Compare July 24, 2024 15:25

elfiesmelfie added 5 commits July 26, 2024 19:25

[zuul] Add 4.16 jobs

895c21c

Add local_build variation

a52d3ba

[zuul] Update OCP 4.16 nodeset to use biggest flavor

f9479aa

This is to see if sg-core just needs more resource to build. Eventually, the size will be reduced again to not-the-largest-available.

Update the nodeset label for CRC 2.39

1a0f20a

Update supported openshift versions to 4.16

e0945e7

elfiesmelfie and others added 11 commits July 26, 2024 19:25

fix typo

5a9d440

[stf-collect-logs] Add oc get deployment to logs

9fdd16a

[zuul] Update 4.16 node to use any cloud provider

a3285b5

[zuul] Only run the nightly bundles job

506c26b

All the jobs are currently failing for the same reason, disable all but the fastest one to save resources

fix typo on stf-collect-logs

35f4263

Re-add vexxhost

4b50640

[tmp] Check whether failures are due to the new OCP version

4f9cdd5

Update .zuul.yaml

945b2ab

Switching from nightly bundles to local builds in order to test 4.16 fixes

test all 4.16 jobs

e379233

Re-enable all 4.16 jobs and remove 4.12

7535760

Remove commented lines

21b7a46

elfiesmelfie force-pushed the efoley-zuul-ocp-4.16 branch from a1db1d5 to 21b7a46 Compare July 26, 2024 18:26

elfiesmelfie marked this pull request as ready for review July 26, 2024 18:26

vkmc approved these changes Jul 29, 2024

View reviewed changes

Un-update stf-collectd-logs and update ci/README

f565915

vkmc merged commit 417131b into master Jul 30, 2024
10 checks passed

vkmc deleted the efoley-zuul-ocp-4.16 branch July 30, 2024 06:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[zuul] Add 4.16 jobs #609

[zuul] Add 4.16 jobs #609

elfiesmelfie commented Jul 5, 2024 •

edited by csibbitt

Loading

softwarefactory-project-zuul bot commented Jul 5, 2024

softwarefactory-project-zuul bot commented Jul 8, 2024

elfiesmelfie commented Jul 8, 2024

danpawlik commented Jul 9, 2024

softwarefactory-project-zuul bot commented Jul 9, 2024

elfiesmelfie commented Jul 10, 2024 •

edited

Loading

softwarefactory-project-zuul bot commented Jul 11, 2024

softwarefactory-project-zuul bot commented Jul 11, 2024

softwarefactory-project-zuul bot commented Jul 11, 2024

softwarefactory-project-zuul bot commented Jul 12, 2024

softwarefactory-project-zuul bot commented Jul 16, 2024

elfiesmelfie commented Jul 16, 2024

softwarefactory-project-zuul bot commented Jul 16, 2024

softwarefactory-project-zuul bot commented Jul 16, 2024

softwarefactory-project-zuul bot commented Jul 17, 2024

softwarefactory-project-zuul bot commented Jul 18, 2024

softwarefactory-project-zuul bot commented Jul 18, 2024

softwarefactory-project-zuul bot commented Jul 19, 2024

elfiesmelfie commented Jul 23, 2024

softwarefactory-project-zuul bot commented Jul 23, 2024

softwarefactory-project-zuul bot commented Jul 24, 2024

csibbitt commented Jul 24, 2024

vkmc left a comment

vkmc Jul 29, 2024

elfiesmelfie Jul 29, 2024

[zuul] Add 4.16 jobs #609

[zuul] Add 4.16 jobs #609

Conversation

elfiesmelfie commented Jul 5, 2024 • edited by csibbitt Loading

softwarefactory-project-zuul bot commented Jul 5, 2024

softwarefactory-project-zuul bot commented Jul 8, 2024

elfiesmelfie commented Jul 8, 2024

danpawlik commented Jul 9, 2024

softwarefactory-project-zuul bot commented Jul 9, 2024

elfiesmelfie commented Jul 10, 2024 • edited Loading

softwarefactory-project-zuul bot commented Jul 11, 2024

softwarefactory-project-zuul bot commented Jul 11, 2024

softwarefactory-project-zuul bot commented Jul 11, 2024

softwarefactory-project-zuul bot commented Jul 12, 2024

softwarefactory-project-zuul bot commented Jul 16, 2024

elfiesmelfie commented Jul 16, 2024

softwarefactory-project-zuul bot commented Jul 16, 2024

softwarefactory-project-zuul bot commented Jul 16, 2024

softwarefactory-project-zuul bot commented Jul 17, 2024

softwarefactory-project-zuul bot commented Jul 18, 2024

softwarefactory-project-zuul bot commented Jul 18, 2024

softwarefactory-project-zuul bot commented Jul 19, 2024

elfiesmelfie commented Jul 23, 2024

softwarefactory-project-zuul bot commented Jul 23, 2024

softwarefactory-project-zuul bot commented Jul 24, 2024

csibbitt commented Jul 24, 2024

vkmc left a comment

Choose a reason for hiding this comment

vkmc Jul 29, 2024

Choose a reason for hiding this comment

elfiesmelfie Jul 29, 2024

Choose a reason for hiding this comment

elfiesmelfie commented Jul 5, 2024 •

edited by csibbitt

Loading

elfiesmelfie commented Jul 10, 2024 •

edited

Loading