jenkins/jenkins:lts-jdk11 missing arm64 and s390x

[ewilkins-csi]

Service(s)

Docker Hub

Summary

The Jenkins LTS images were pushed around an hour ago and only have the amd image -- no arm or s390x.

This seems to be identical to #2890 from a year ago

Reproduction steps

1. Open https://hub.docker.com/r/jenkins/jenkins/tags?page=1&name=lts-jdk11
2. Compare the available platforms for lts-jdk11 and 2.332.2-lts-jdk11, note that s390x and arm64 are available for one and not the other

[MarkEWaite]

Thanks for reporting the issue @ewilkins-csi . Much appreciated.

The container images that were built two hours ago are based on Jenkins 2.426.2 (which is good, since they are not inadvertently packaging an outdated version), but they do not appear to have been generated by the standard build process that creates multiple architectures.

Circumstances looks similar to the earlier case.

[dduportal]

Thanks for opening this issue!. I confirm this is the exact same issue as #2890 .

The job on the trusted.ci (private) controller went through a branch indexing. It is unexpected but it detected a change on the tag jenkins-docker-packaging-2.164.2 which triggered not only 1 but 4 builds 😭

Currently searching for a solution 🤔

[dduportal]

Fixed this particular tag with the proper credentials and:

skopeo copy --all docker://docker.io/jenkins/jenkins:2.426.2-lts-jdk11 docker://docker.io/jenkins/jenkins:lts-jdk11

However, other images have been pushed by this rogue build:

• lts (skopeo copy --all docker://docker.io/jenkins/jenkins:2.426.2-lts-jdk17 docker://docker.io/jenkins/jenkins:lts`)
• WiP ...

[dduportal]

Update:

• I ran the following command to "restore" as much as posible, based on the datetime of the last changes. I've also included weekly releases even if the 2.440 will be deployed in a few hours

skopeo copy --all docker://docker.io/jenkins/jenkins:2.426.2-lts-jdk11 docker://docker.io/jenkins/jenkins:lts-jdk11

skopeo copy --all docker://docker.io/jenkins/jenkins:2.426.2-lts-jdk17 docker://docker.io/jenkins/jenkins:lts

skopeo copy --all docker://docker.io/jenkins/jenkins:2.426.2-slim-jdk17 docker://docker.io/jenkins/jenkins:lts-slim

skopeo copy --all docker://docker.io/jenkins/jenkins:lts-alpine-jdk17 docker://docker.io/jenkins/jenkins:lts-alpine

skopeo copy --all docker://docker.io/jenkins/jenkins:2.439-jdk17 docker://docker.io/jenkins/jenkins:latest

skopeo copy --all docker://docker.io/jenkins/jenkins:2.439-jdk11 docker://docker.io/jenkins/jenkins:jdk11

skopeo copy --all docker://docker.io/jenkins/jenkins:2.439-slim-jdk17 docker://docker.io/jenkins/jenkins:slim

skopeo copy --all docker://docker.io/jenkins/jenkins:2.439-alpine-jdk17 docker://docker.io/jenkins/jenkins:alpine

• I don't see any other "rogue" images

=> note this was necessary to avoid failing controller (as it changed the default JDK to 11 in the rogue images).

[dduportal]

Ping @daniel-beck @timja @MarkEWaite @lemeurherve @basil @NotMyFault giben this weird (and alas regular behavior) of Jenkins building tag when it should not, I would be encline to propose a deletion of the old tags, (only on the Docker repository) to avoid further mayhem.
This is not a light decision to make but this start to be more than an annoyance: uses building images from the latest images (e.g. not pinning the LTS version) for sure had their controller crachec due to JDK change, not mentioning the risk of running older JDKs accidentally.

Any thoughts?

[daniel-beck]

How about reverting the change to go with tag-based builds rather than querying what exists and filling the blanks we had before? (Might be a stupid question due to lack of awareness what was wrong with that process.)

[dduportal]

How about reverting the change to go with tag-based builds rather than querying what exists and filling the blanks we had before? (Might be a stupid question due to lack of awareness what was wrong with that process.)

No problem : this is what we did a few month ago for the reason you mentioned. But the old tag here stil’ had the « big publish script » which is the problem. This tag should NOT be considered to be built (older than 3 days and already in a success state), but it was.

[MarkEWaite]

This is not a light decision to make but this start to be more than an annoyance: uses building images from the latest images (e.g. not pinning the LTS version) for sure had their controller crachec due to JDK change, not mentioning the risk of running older JDKs accidentally.

Any thoughts?

I support deleting the old tags from this repository. We can archive a private copy of the repository before the tag deletion in case we need to recover it. We could even place the archived private copy of the repository inside this repository as a tar.gz file if we want to keep things "close".

[timja]

Fine to delete old tags from me, we don't need them.

[NotMyFault]

No objections from me to delete the old tags.

[dduportal]

FYI, without any formal objection, I'll proceed to remove the tags in 72h (as I already got 3 approvals).

[lemeurherve]

No objection from me either.

[dduportal]

All tags deleted with the following command (the GH releases have been kept):

for t in $(git tag | grep packaging); do echo "== Deleting tag $t..."; git push upstream "$t" --delete; echo 'Deletion done!';done

Ref. https://github.com/jenkinsci/docker/tags?after=2.409