-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Build] Define Docker images build using Jenkin's Job-DSL #2271
[Build] Define Docker images build using Jenkin's Job-DSL #2271
Conversation
d284d5c
to
497a5a7
Compare
I have now added two more commits to clean-up and simplify the Docker image build:
@sravanlakkimsetti do you want to have a look at this? In general it would of course be nice to make the entire pipeline dynamic to build all images that are in a specific folder, but I don't know an elegant way to define the image id, besides adding an extra file for it? As said before, ideally this would all be done as part of the main Jenkinsfile of the repo. Additionally we could also think about either to merge the swt specific images into their 'main' image or to define the SWT images at the SWT git-rep so that it can be maintained with SWT build. If it is possible to build docker images from the 'main' Jenkinsfile, we could maybe even build the SWT specific image in the same run and just pull it later. Or is it possible to publish images 'locally'? Similar to a local Maven install? |
I think having an own job makes sense here, as building an publishing the images takes considerable time and space also it looks like the images might be used by other parties and on the other hand they do not change very often. Docker images can of course also be used local, but then they must be local to the agent that creates the docker image (there are likely Jenkins plugins for such task available so shell scripts are not directly needed). |
} | ||
stage('Get Docker Files') { | ||
steps { | ||
sh \''' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't we use usual Jenkins clone tasks?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because this primarily a job definition (using the Job DSL), we have to define the repository URL somewhere. Either in the job configuration itself (and then we can use the usual checkout scm
) or at this location.
But since we have to checkout multiple repositories and I would like to have this at one location defined, I think it makes sense to keep it at this location.
stage('Build Docker Images') { | ||
steps { | ||
dir('eclipse.platform.releng.aggregator/cje-production/dockerfiles') { | ||
sh \''' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we probabbly amake it seperate steps instead of one big?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have not reworked this to use a scripted parallel pipeline that uses a method to avoid duplicated definition of the executed action.
This way there is an individual stage for each image.
I have also enabled all the default logging, so that one can check or download the logs from the Jenkins UI as usual.
Therefore I have disabled explicit writing and archiving of log files.
Adding or removing images is then basically a one-line change in the job configuration.
One can see the results for example in
https://ci.eclipse.org/releng/job/Build-Docker-images/745/pipeline-graph/
or in the sub-elements of
https://ci.eclipse.org/releng/job/Build-Docker-images/745/pipeline-console/?selected-node=16
497a5a7
to
8a87c3e
Compare
8a87c3e
to
3bf4f61
Compare
As described above I have now reworked the pipeline to build the images in parallel at Jenkins level. For testing I have applied the new pipeline in the existing job and it successfully built, see: Besides what has been mention in #2271 (comment) I have now also simplified the cron trigger to run only once a week instead of every two or three days (at different times). The images only change seldomly and rebuilds are not necessary that often. With that this is ready for submission for me. |
And since this will create a new job, I will delete the existing job once the new run ran successfully: |
Why do we need a scripted pipeline can't we just use a declarative one here? it looks a little bit strange and I can't see a direct reason for this... The idea seems to create the job itself by a script but why don't we simply reference a Jenkinsfile directly from the repository? |
I missed that comment but see above, why do we need a job that creates a job, I would simply create a regular jenkins file and then add a Job that reference it (either manually or with a script but I don't see a good reason for having it created by a script as it only complicates the setup?!? |
This just preserves the existing and currently manually maintained 'Build-Docker-images' job in a Jenkins Job-DSL script under version control.
- Update to jdt.core git repository at Github - Don't send mails on success only on failure and only to the mailing-list and not specific people - Build only once per week (if a recent change is needed a build is triggered manually any ways) - Delete complete workspace content after the build
- Inline bash-scripts to build and push docker-images - Just delete those scripts that are not used - Remove push of images that are not built (anymore)
3bf4f61
to
b1c8003
Compare
We can use a declarative pipeline here, but we would then need to specified the docker-image folder for each parallel stage, because in a declarative pipeline you cannot have a parallel step/stage within a
I contemplated initially to use a dedicated jenkinsfile and define the job that it references it in SCM, but discarded the idea because I wanted to avoid to have two files for one job. And since only very few string had to be escaped I was fine with it. And I think we should definitively generate the job using the job-dsl, ideally all jobs would be generated (then we could configure the seed job to also delete removed elements automatically, but having all jobs in the JIPP defined in SCM would be a longer tearm goal). However, after think about it more, the definition of the job itself is very small if we have a regular Jenkinsfile, so we can just add it to the |
No further objections raised. So I'm submitting and applying this. |
The new job has been successfully generated and succeeds: I also pushed a follow-up with fixes that I missed in this change All looks good and I verified at docker-hub that the images are successfully pushed:
And since all looks good I have also deleted the old, manually maintained job at: |
This just preserves the existing and currently manually maintained 'Build-Docker-images' job in a Jenkins Job-DSL script under version control.
Ideally the Docker images would be built as part of the master branch build of this repo. With that the images would be built as soon as one submits a change. And because I builds usually push a change each night, the images would updated regularly even without explicit change.
But this would probably require the corresponding credentials available in this repo. I also don't know if docker is available by default.