"Skipped parameter" warnings in jenkins server log

[sxa]

These aren't new but were discovered while working on #2108

There are a lot of warnings in the jenkins log which we should look at clearing up in the interests of avoid the risk of being unable to see the wood for the trees. All are related to NODE_LABEL's use in various places.

Presumably the parameters are being passed in from the upstream job somewhere but they are not defined on the callee so are superfluous. Can we wasily stop 'rogue' parameters being passed in or should we set the -D mentioned in the error?

90 WARNING hudson.model.ParametersAction#filter: Skipped parameter NODE_LABEL as it is undefined on build-scripts/release/create_installer_mac. Set -Dhudson.model.ParametersAction.keepUndefinedParameters=true to allow undefined parameters to be injected as environment variables or -Dhudson.model.ParametersAction.safeParameters=[comma-separated list] to whitelist specific parameter names, even though it represents a security breach or -Dhudson.model.ParametersAction.keepUndefinedParameters=false to no longer show this message.

[sxa]

FYI @andrew-m-leonard @smlambert (Shelley I sent you a log from this a while back and I think there may be fewer such messages now but unclear if that's my memory or if it was a slightly different error!

[sxa]

Also tagging @sophia-guo since she was involved in the discussions on the community call earlier today :-)

[sophia-guo]

According to the discussion if jenkins jobs are triggered with undefined parameters there will be those warnings in jenkins server log? If that is the case those warnings can be avoided by re-generating aqa-tests or smoketests jobs triggered by upstream.

Initially when new features was added there was a period that aqa-tests or smoke tests jobs were triggered with undefined parameters. For example, USE_TESTENV_PROPERTIES, the reason is testJobTemplate was not updated accordingly as it didn't affect the functionalities. I have double checked all parameters of aqa-tests and smoke tests, all of them have been updated in testJobTemplate. That means if there is still those warning message related with aqa-tests or smoketests regenerating those jobs should fix it. Note autoGen is enabled with aqa-tests and disabled with smoketests.

[sxa]

Thanks for checking @sophia-guo! It looks like the remaining ones are mostly on the build pipelines so the test jobs may now be all sorted. I've just done a check on the recent logs and these are the remaining warnings - interestingly only the (new) jdk19u pipelines are showing up here @andrew-m-leonard:

jenkins@jenkins-hetzner-ubuntu2004:~$ gzip -cd /var/log/jenkins/jenkins.log.?.gz | grep Skipped.*undefined /var/log/jenkins/jenkins.log | sed -e 's/.*Skipped/Skipped/g' -e 's/. Set.*/./g' | sort | uniq -c
    282 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-aix-ppc64-temurin`.
    291 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-alpine-linux-aarch64-temurin`.
    306 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-alpine-linux-x64-temurin`.
    291 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-aarch64-temurin`.
    286 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-arm-temurin`.
    278 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-ppc64le-temurin`.
    270 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-s390x-temurin`.
    294 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-x64-temurin`.
    485 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-mac-aarch64-temurin`.
    488 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-mac-x64-temurin`.
    432 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-windows-x64-temurin`.
    447 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-windows-x86-32-temurin`.
     36 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/release/create_installer_mac`.
     72 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/release/sign_installer`.
    240 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/release/sign_temurin_gpg`.
jenkins@jenkins-hetzner-ubuntu2004:~$ 

[andrew-m-leonard]

@sxa I don't understand the above, none of the build jobs have a Parameter called NODE_LABEL, they only have 4 paramaters:

• BUILD_CONFIGURATION
• USER_REMOTE_CONFIGS
• DEFAULTS_JSON
• ADOPT_DEFAULTS_JSON

[sxa]

That sounds consistent with the warning message - "it is undefined on " but presumably something is trying to start them using those extra, unusable, parameters.

[sxa]

Had a look at the log for something else today. We've still got:

• USE_TESTENV_PROPERTIES on most of the test jobs giving the warning
• RELEASE_TAG showing ONLY on Test_openjdk11_j9_sanity.openjdk_x86-64_linux in the most recenr log. Not sure why that might be special.
• NODE_LABEL on most of the build jobs
• GENERATE_JOBS on most of the test jobs
• DYNAMIC_COMPILE on most of the test jobs.

[smlambert]

Regenerated the following test jobs:

JDK 11,17,19 hotspot/platforms
https://ci.adoptopenjdk.net/view/Test_grinder/job/Test_Job_Auto_Gen/864/ (functional - all levels)
https://ci.adoptopenjdk.net/view/Test_grinder/job/Test_Job_Auto_Gen/865/ (perf,system,openjdk - sanity,extended)

JDK 8 hotspot/platforms
https://ci.adoptopenjdk.net/view/Test_grinder/job/Test_Job_Auto_Gen/866/ (functional - all levels)
https://ci.adoptopenjdk.net/view/Test_grinder/job/Test_Job_Auto_Gen/867/ (perf,system,openjdk - sanity,extended)

[andrew-m-leonard]

@sxa this look like job instances ? eg.build-scripts/jobs/jdk/jdk-mac-aarch64-hotspot` (#38)
Can't we just delete those particular job numbers?

[sxa]

@sxa this look like job instances ? eg.build-scripts/jobs/jdk/jdk-mac-aarch64-hotspot` (#38) Can't we just delete those particular job numbers?

A fair comment - I hadn't actually spotted that it was on non-current ones. If you confirm you're ok with us just deleting them I'm happy to go through and do that as a background task.

That would likely just leave things like this on the PR tester jobs:
2024-05-02 08:36:36.633+0000 [id=4239618] WARNING hudson.model.ParametersAction#filter: Skipped parameter NODE_LABEL as it is undefined on build-scripts-pr-tester/build-test/jobs/jdk/jdk-alpine-linux-x64-temurin (#9). Set -Dhudson.model.ParametersAction.keepUndefinedParameters=true to allow undefined parameters to be injected as environment variables or -Dhudson.model.ParametersAction.safeParameters=[comma-separated list] to whitelist specific parameter names, even though it represents a security breach or -Dhudson.model.ParametersAction.keepUndefinedParameters=false to no longer show this message.

[sxa]

Test jobs (28) J9(16)/Dragonwell(9)/Bisheng(3)

From a comment from Shelley in a meeting earlier today it seems likely that running https://ci.eclipse.org/temurin-compliance/job/AQA_Test_Pipeline with a VARIANT set appropriately and AUTO_AQA_GEN checked

(Although I'm not sure if this will regen the _quarkus_openshift, _xl and _rerun jobs so it would be good to have a way of doing that)

Similarly I'm unclear if we have a way to regen the various Grinder variants (although almost all of those in the twisty earlier were grinder_sandbox_iteration_2

[sxa]

Runs - noting that the AQA_Test_pipeline job does not have an option for SDK_RESOURCE=upstream so I'm having to locate a suitable customized URL for these (Although worst case they just fail but the regen will still have occured by that point!)

• Corretto 11 Linux/x64
• Bisheng 11 Linux/x64
• Bisheng 11 Linux/riscv64
• Dragonwell 11 Linux/x64
• J9 11 Linux on x64, ppc64le aarch64, s390x and Win64_xl
• J9 17 Linux on x64, ppc64le, aarch64, s390x and Win64_xl

This will cover most of the outstanding test jobs other than Grinders, _rerun and the specialist external ones.

[sxa]

Noting also that we're getting a couple of the parameters giving warnings on Test_Job_Auto_Gen as called from AQA_Test_Pipeline which will likely require some sort of manual remediation:

• 2024-05-07 16:43:03.074+0000 [id=85459] WARNING hudson.model.ParametersAction#filter: Skipped parameter LIGHT_WEIGHT_CHECKOUT as it is undefined on Test_Job_Auto_Gen (#5,239)...
• 2024-05-07 16:43:03.076+0000 [id=85459] WARNING hudson.model.ParametersAction#filter: Skipped parameter TEST_JOB_NAME as it is undefined on Test_Job_Auto_Gen (#5,239)...

(I've had 3100 of those lines from Test_Job_Auto_Gen in the last two hours since I kicked off those jobs in the previous comment)

[smlambert]

FYI, TEST_JOB_NAME only applies to AQA_Test_Pipeline, there is no parameter of that name used in the generated jobs (nor do we want their to be).

[smlambert]

Similarly I'm unclear if we have a way to regen the various Grinder variants (although almost all of those in the twisty earlier were grinder_sandbox_iteration_2

I have been slowly removing grinder variants especially if they were last run 1 yr or more ago

[sxa]

FYI, TEST_JOB_NAME only applies to AQA_Test_Pipeline, there is no parameter of that name used in the generated jobs (nor do we want their to be).

Makes sense, although the warning would suggest that AQA_Test_pipeline is trying to pass that parametr down to Test_Job_Auto_Gen somwhere.

[smlambert]

Makes sense, although the warning would suggest that AQA_Test_pipeline is trying to pass that parametr down to Test_Job_Auto_Gen somwhere.

The code cycles through all parameters and passes them down (with some exceptions at https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/aqaTestPipeline.groovy#L110-L114). It can be added to the set of exceptions that are not added to childParams so that when the downstreamJob is created (at https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/aqaTestPipeline.groovy#L147), the content of childParams is without it.

[sxa]

Deep dive into one of them (Corretto JDK8 sanity/system) for reference. There are only 13 entries relating to test skipped parameters today so far, and 8 of them are from different parameters on this job so yesterday's regens appear to have made quite a big difference.

• Job 480 is giving the skipped message with JDK_REPO JDK_BRANCH PARALLEL NUM_MACHINES USE_TESTENV_PROPERTIES GENERATE_JOBS ACTIVE_NODE_TIMEOUT and DYNAMIC_COMPILE
• Regenerating at https://ci.adoptium.net/job/Test_Job_Auto_Gen/5362/
• There is no difference in the individual builds (despite the reference to build 480 in the log). Diff between old and new config.xml for the job is here in case it's useful for future historic reference (it's quite big since it has reference to adoptopenjdk repository - the last run of that job was April 14th 2022):
  2774.Test_openjdk8_corretto_sanity.system_x86-64_linux.diff.txt

[sxa]

I have been slowly removing grinder variants especially if they were last run 1 yr or more ago

@smlambert Can "suffixed" jobs like https://ci.adoptium.net/job/Test_openjdk11_hs_sanity.external_s390x_linux_system-test/ be removed too? The ones I've looked at today seem to be ones that haven't been run in nearly 3 years so I guess they were likely generated as some sort of experiment and are no longer required.

I don't want to be deleting them myself though - would rather someone from the test side with knowledge of the particular jobs that should be in scope was able to handle it.

[smlambert]

I have a script that will look for jobs that have not been run in XX number of months, and optionally delete them if 'deleteJobs' parameter equals true. I think we should run such a job occasionally on the server to cull old, not-used jobs. I will do a pass in the coming weeks.

[sxa]

There are a lot of new instances of this showing up relating to the new RERUN_FAILURE parameter. I think we're going to need some sort of process for ensuring that all the pipelines are suitable updated when we make changes to the parameters.

[sxa]

Also @steelhead31 There are a few relating to create_installer_linux:

• UPSTREAM_JOB_NAME
• UPSTREAM_JOB_NUMBER
• UPLOAD_TESTRESULTS_ONLY
• TIMESTAMP
• GITHUB_TOKEN
• ARTIFACTS_TO_SKIP

[steelhead31]

Also @steelhead31 There are a few relating to create_installer_linux:

• UPSTREAM_JOB_NAME
• UPSTREAM_JOB_NUMBER
• UPLOAD_TESTRESULTS_ONLY
• TIMESTAMP
• GITHUB_TOKEN
• ARTIFACTS_TO_SKIP

I'll have a look at this, to see if I can stop it reporting this.. :)

Investigated, due to the pass parameters plugin not having the ability to be selective when triggering a downstream job, it will require a code change to the new installer jenkinsfile to handle the parameters that are surplus to requirement. I have code ready, so I'll look to get it merged once the release is completed.

This should be resolved now, let me know if not.