Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sign node label #467

Draft
wants to merge 9 commits into
base: master
Choose a base branch
from
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ You can contribute to this project in many different ways. **We appreciate all

## Project description

ci-jenkins-pipelines is a project that contains the groovy code that powers our jenkins ci build system.
ci-jenkins-pipelines is a project that contains the groovy code that powers our jenkins ci build tool.

* <https://github.com/adoptium/ci-jenkins-pipelines>

Expand Down Expand Up @@ -39,7 +39,7 @@ Contact the Eclipse Foundation Webdev team via [email protected].

Please let us know via our [issue tracker](https://github.com/adoptium/temurin-build/issues) if you find a problem, even if you don't have a fix for it. The ideal issue report should be descriptive, and where possible include the steps we can take to reproduce the problem for ourselves.

If you have a proposed fix for an issue, or an enhancement you would like to make to the code please describe it in an issue, then send us the code as a [Github pull request](https://help.github.com/articles/about-pull-requests) as described below.
If you have a proposed fix for an issue, or an enhancement you would like to make to the code please describe it in an issue, then send us the code as a [GitHub pull request](https://help.github.com/articles/about-pull-requests) as described below.

## Pull requests

Expand All @@ -56,7 +56,7 @@ Use a descriptive title, and if it relates to an issue in our tracker please ref

### Pull request comment

The PR comment should be formed by a one-line subject, followed by one line of white space, and one or more descriptive paragraphs, each separated by one line of white space. All of them should be finished by a dot.
The PR comment should be formed by a one-line subject, followed by one line of whitespace, and one or more descriptive paragraphs, each separated by one line of whitespace. All of them should be finished by a dot.

Where your PR fixes an issue, it should include a reference to the issue's identifier in the first line of the commit comment. The comment should provide enough information for a reviewer to understand the changes and their relation to the rest of the code.

Expand Down
7 changes: 5 additions & 2 deletions pipelines/build/common/openjdk_build_pipeline.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -485,6 +485,7 @@ class Build {
Run the Sign downstream job. We run this job on windows and jdk8 hotspot & jdk13 mac builds.
The job code signs and notarizes the binaries so they can run on these operating systems without encountering issues.
*/
def SIGN_NODE_LABEL = DEFAULTS_JSON['signNodeLabel']
Copy link
Contributor

@sophia-guo sophia-guo Oct 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is better to move this definition to the function build() to be visible to sign(), signInstaller() and probably other functions.

def sign(VersionInfo versionInfo) {
// Sign and archive jobs if needed
if (
Expand All @@ -493,7 +494,8 @@ class Build {
context.stage('sign') {
def filter = ''

def nodeFilter = 'eclipse-codesign'
def nodeFilter = "${SIGN_NODE_LABEL}"


if (buildConfig.TARGET_OS == 'windows') {
filter = '**/OpenJDK*_windows_*.zip'
Expand Down Expand Up @@ -701,7 +703,8 @@ class Build {
default: break
}

def nodeFilter = 'eclipse-codesign'
/* groovylint-disable-next-line VariableTypeRequired */
def nodeFilter = "${SIGN_NODE_LABEL}"
Comment on lines +706 to +707
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/* groovylint-disable-next-line VariableTypeRequired */
def nodeFilter = "${SIGN_NODE_LABEL}"
String nodeFilter = "${SIGN_NODE_LABEL}"


// Execute sign installer job
def installerJob = context.build job: 'build-scripts/release/sign_installer',
Expand Down
12 changes: 6 additions & 6 deletions pipelines/build/regeneration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ To enable concurrent pipeline builds (i.e. submitting two pipelines in parallel)
All of our [pipelines](https://ci.adoptopenjdk.net/job/build-scripts/) make use of [downstream jobs](https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/) to build Adopts JDK's. In short, the jobs are created in the pipelines with a set of configurations passed down to them.
To create these jobs, we utilise a plugin called [job dsl](https://github.com/jenkinsci/job-dsl-plugin) to create a `dsl` file for each downstream job, containing the configurations, node label, etc.

In the past, we created these dsl's in the [pipeline files](pipelines/build). So each time we wanted to create a downstream job, we would create all of the job dsl's that were possible for the pipeline and pick the one that we needed.
In the past, we created these dsl's in the [pipeline files](/pipelines/build). So each time we wanted to create a downstream job, we would create all of the job dsl's that were possible for the pipeline and pick the one that we needed.
Not only was this resource intensive and slow, but it also meant that concurrent builds were impossible due to the risk of one builds dsl's overwriting another's. This is why we have created pipeline job generators to create the dsl's for the pipelines to use, instead of creating them in the pipeline jobs.

The job regenerators are essentially downstream job makers. They pull in the [targetConfigurations](pipelines/jobs/configurations) and build the job DSL's for each possible downstream job.
The job regenerators are essentially downstream job makers. They pull in the [targetConfigurations](/pipelines/jobs/configurations) and build the job DSL's for each possible downstream job.
The pipelines can use these job dsl's to create their downstream jobs since they are created in the same node as them (the master one). This way, each of the pipelines has a fresh dsl each time, no matter how many builds are running at once.

## Where they are
Expand All @@ -24,12 +24,12 @@ NOTE: When the JDK HEAD updates, these jobs will need to be updated too (see [RE
There are three stages for each job regenerator.

- Execute the top level job:
- The jobs themselves are executed by GitHub Push on this repository. Each time there is a commit, all the pipeline regenerators are kicked off. This is so any potential changes to the [buildConfigurations](pipelines/jobs/configurations/jdk8u_pipeline_config.groovy) and [targetConfigurations](pipelines/jobs/configurations/jdk8u.groovy) are taken into account when creating a job dsl for each downstream job.
- Each of the jobs executes it's corresponding [regeneration](pipelines/build/regeneration) file, passing down it's version, targeted OS/ARCH/VARIANT and specific build configurations to the main [config_regeneration](pipelines/build/common/config_regeneration.groovy) file.
- The jobs themselves are executed by GitHub Push on this repository. Each time there is a commit, all the pipeline regenerators are kicked off. This is so any potential changes to the [buildConfigurations](/pipelines/jobs/configurations/jdk8u_pipeline_config.groovy) and [targetConfigurations](/pipelines/jobs/configurations/jdk8u.groovy) are taken into account when creating a job dsl for each downstream job.
- Each of the jobs executes it's corresponding [regeneration](/pipelines/build/regeneration) file, passing down it's version, targeted OS/ARCH/VARIANT and specific build configurations to the main [config_regeneration](/pipelines/build/common/config_regeneration.groovy) file.
- Check if the corresponding pipeline is in progress:
- Since we want to potentially avoid overwriting the job dsl's of any pipelines in progress, we use the [jenkins API](https://ci.adoptopenjdk.net/api/) to verify that there are no pipelines of that version queued or running. If there are, the job regenerator sleeps for 15mins and checks again afterwards. If not, it moves onto the next step.
- Regenerate the downstream jobs, one at a time:
- The regenerator then iterates through the keys in the `targetConfigurations` (e.g. [jdk11u.groovy](pipelines/jobs/configurations/jdk11u.groovy)), which are the same keys used in the `buildConfiguration` file.
- The regenerator then iterates through the keys in the `targetConfigurations` (e.g. [jdk11u.groovy](/pipelines/jobs/configurations/jdk11u.groovy)), which are the same keys used in the `buildConfiguration` file.
After parsing each variant in them and going through various error handling stages, the job name and folder path is constructed which is the bare minimum that the job dsl needs to be created. We only need the bare minimum as the pipelines will overwrite most the configs when they run.
- The job dsl for that downstream job is constructed and that job is then, successfully regenerated. The result is somewhat similar to this:

Expand Down Expand Up @@ -90,7 +90,7 @@ Unreferenced items:
## Build Pipeline Generator

This generator generates the [top level](https://ci.adoptopenjdk.net/job/build-scripts/) pipeline jobs. It works by iterating through the config files, defining a job dsl configuration for each version that has a version config file.
It then calls [pipeline_job_template.groovy](pipelines/jobs/pipeline_job_template.groovy) to finalise the dsl. By default, the [job that runs this file](https://ci.adoptopenjdk.net/job/build-scripts/job/utils/job/build-pipeline-generator/) has restricted read access so you will likely need to contact a jenkins admin to see the results of the job.
It then calls [pipeline_job_template.groovy](/pipelines/jobs/pipeline_job_template.groovy) to finalise the dsl. By default, the [job that runs this file](https://ci.adoptopenjdk.net/job/build-scripts/job/utils/job/build-pipeline-generator/) has restricted read access so you will likely need to contact a jenkins admin to see the results of the job.

## Downstream Test Jobs

Expand Down
3 changes: 2 additions & 1 deletion pipelines/defaults.json
Original file line number Diff line number Diff line change
Expand Up @@ -53,5 +53,6 @@
"numMachines" : ["3"]
}
},
"defaultsUrl" : "https://raw.githubusercontent.com/adoptium/ci-jenkins-pipelines/master/pipelines/defaults.json"
"defaultsUrl" : "https://raw.githubusercontent.com/adoptium/ci-jenkins-pipelines/master/pipelines/defaults.json",
"signNodeLabel" : "eclipse-codesign"
}