Disable resigning of already signed jars #2134

netomi · 2024-06-13T12:57:44Z

As you are aware, signing of jars using the Eclipse Foundation signing service is currently quite slow as the service is in degraded mode and only support 1 concurrent signing atm.

While trying to mitigate that for now till we have a more scalable solution, I have noticed in the build at https://ci.eclipse.org/platform/job/eclipse.platform.releng.aggregator/job/master/ that there are lots of jars (> 800) that are signed for each build. Many of these jars are already signed, and you will find entries like that in the log file:

14:50:16  [INFO] Signing jar: /home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.equinox.device/target/org.eclipse.equinox.device-1.2.100-SNAPSHOT.jar
14:50:19  [INFO] Jar '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.equinox.device/target/org.eclipse.equinox.device-1.2.100-SNAPSHOT-sources.jar' is already signed and will be resigned.

I was wondering if it would not make sense to change the resigningStrategy of the eclipse-jarsigner plugin to something like DO_NOT_RESIGN instead of keeping at the default which is RESIGN.

More information about the configuration can be found at: https://eclipse-cbi.github.io/org.eclipse.cbi/eclipse-jarsigner-plugin/sign-mojo.html

Disable RESIGNING would reduce the number of signing operations considerably imho.

cc @merks

The text was updated successfully, but these errors were encountered:

akurtakov · 2024-06-13T13:03:29Z

This is might be doable but has to be checked deeper by someone as resigning is done before the decision to replace or not with baseline version is taken

15:56:41  [INFO] --- eclipse-jarsigner:1.4.3:sign (sign) @ org.eclipse.osgi.tests ---
15:56:41  [INFO] Signing of nested jars of '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT.jar' is disabled.
15:56:41  [INFO] Jar '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT.jar' is already signed and will be resigned.
15:56:41  [INFO] Signing jar: /home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT.jar
15:56:46  [INFO] Signing of nested jars of '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT-sources.jar' is disabled.
15:56:46  [INFO] Jar '/home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT-sources.jar' is already signed and will be resigned.
15:56:46  [INFO] Signing jar: /home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/target/org.eclipse.osgi.tests-3.19.100-SNAPSHOT-sources.jar
15:56:50  [INFO] 
15:56:50  [INFO] --- tycho-p2:4.0.8:p2-metadata (p2-metadata) @ org.eclipse.osgi.tests ---
15:56:50  [INFO] MavenProject: org.eclipse.osgi:org.eclipse.osgi.tests:3.19.100-SNAPSHOT @ /home/jenkins/agent/workspace/latform.releng.aggregator_master/equinox/bundles/org.eclipse.osgi.tests/pom.xml
15:56:50      The main artifact has been replaced with the baseline version.
15:56:50      The following attached artifacts have been replaced with the baseline version: [sources]

Thus if the bundle is changed and not resigned - a not signed version will end up in the build output.
On the other side, this verification build most likely can have signing disabled entirely as it's not supposed to publish anything to downloads.

netomi · 2024-06-13T13:17:17Z

if builds on master would not be signed at all, that would also be quite some relief for other projects imho.

netomi · 2024-06-13T13:25:08Z

the eclipse-jarsigner could also be extended to have another resigningStrategy, something like ONLY_RESIGN_IF_INVALID, where the jar is only resigned, if it does not have a valid signature. That would avoid the risk that the jar is not resigned if a signature is found, but the jar was modified and the signature is not valid anymore.

HannesWell · 2024-06-13T20:29:21Z

This is might be doable but has to be checked deeper by someone as resigning is done before the decision to replace or not with baseline version is taken

One can exclude certain files from the comparator as it is for example done in M2E:
https://github.com/eclipse-m2e/m2e-core/blob/master/m2e-parent/pom.xml#L118-L119

IIRC for MANIFEST.MF only the main section is checked any ways so all signatures are ignored.

On the other side, this verification build most likely can have signing disabled entirely as it's not supposed to publish anything to downloads.

Thinking about it I wonder why artifacts signing was activated at all on the master. These artifacts are not published, only archived as Jenkins build artifacts. But I don't think they are used for anything else than checking build results.

laeubi · 2024-06-14T06:54:50Z

Signatures are (as far as I know) already excluded, anyways the log seem to suggest that the artifact is already signed, so skip the signing seems obvious, at least recently @merks needed to bump versions manually if signing changes anyways....

Reduce the number of actual signing by not signing artifacts on master branch builds in this repository (the built artifacts are not published) and not re-signing artifacts in the I-build that are baseline replaced and therefore already signed. Fixes eclipse-platform#2134

Reduce the number of actual signing by not signing artifacts on master branch builds in this repository (the built artifacts are not published anywhere). Fixes eclipse-platform#2134

Reduce the number of actual signing operations by not re-signing artifacts (in the I-build) are already signed (usually because they have been 'baseline-replaced'). Fixes eclipse-platform#2134

Reduce the number of actual signing operations by not re-signing artifacts (in the I-build) are already signed, usually because they have been 'baseline-replaced'. Fixes eclipse-platform#2134

HannesWell · 2024-06-19T18:41:50Z

With #2140 signing will be disabled entirely for the master/maintenance branch builds of this repository.
Assuming that there is one I-build and therefore at least one master build (submodule-updates) of this repo per day, in reality there is probably more, this should already reduce the number of signings of this repo to less than the half than it used to be.

Currently I find the String Signing jar: 973 times in the build logs of a single build. Assuming that each one means an actual sign operation and on average two builds (one I-build and one master build) ran per day, this means 2*973*365=710.290 sign operations per year. In reality it is probably higher due to more I-builds in milestone/RC weeks and 'manual' changes to the master.
Btw. the number of times I find is already signed and will be resigned. is 798 in the last master build. But as said elsewhere, many of them are tests that are not published.

As a follow-up I created #2143 to reduce the signings even further by not re-signing the actual jars.

merks · 2024-06-19T18:58:37Z

You’re awesome 😎

netomi · 2024-06-19T19:01:00Z

tyvm, that should reduce the number of signing operations considerably and give us more room for implementing a more scalable signing service (they are usually charged by signing operations / year and our number is really high).

Reduce the number of actual signing operations by not re-signing artifacts (in the I-build) are already signed, usually because they have been 'baseline-replaced'. In order to ensure that signatures of baseline-replaced artifacts can be considered, the p2-baseline-replacement is moved before the eclipse-jarsigner-plugin. Furthermore set 'defaultP2Metadata' to false to skip the default baseline replace. Fixes eclipse-platform#2134

akurtakov · 2024-06-19T19:09:59Z

tyvm, that should reduce the number of signing operations considerably and give us more room for implementing a more scalable signing service (they are usually charged by signing operations / year and our number is really high).

Why doesn't EF recommend going PGP signing wherever possible (everything but native content more or less) in this case?

netomi · 2024-06-20T05:49:15Z

We discussed this internally. Adding jar signing support to the Eclipse IDE was apparently a difficult process as I have been told and we do not want to open another can of worms atm that was the conclusion.

Hopefully we have a more scalable solution soon.

Reduce the number of actual signing operations by not re-signing artifacts (in the I-build) are already signed, usually because they have been 'baseline-replaced'. The baseline replacement is performed first the 'p2-metadata-default' goal executed by default and before the jar-signing. The subsequent explicit invocation of the p2-metadata (non-default) goal is again necessary to update the p2 metadata (like artifact size) after the jar has been altered by the signing operation. Fixes eclipse-platform#2134

Reduce the number of actual signing operations by not re-signing artifacts (in the I-build) are already signed, usually because they have been 'baseline-replaced'. The baseline replacement is performed first the 'p2-metadata-default' goal executed by default and before the jar-signing. The subsequent explicit invocation of the p2-metadata (non-default) goal is again necessary to update the p2 metadata (like artifact size) after the jar has been altered by the signing operation. Fixes #2134

HannesWell · 2024-06-20T20:19:04Z

FYI I just submitted #2143, which should again (usually) significantly reduce the number of signing operations in I-builds.
If one is interested, tonight's I-build can be checked.

That should also reduce the overall runtime of I-builds.

netomi added the enhancement New feature or request label Jun 13, 2024

HannesWell mentioned this issue Jun 17, 2024

[Build] Build don't resign signed jars and skip signing in master build #2140

Merged

HannesWell mentioned this issue Jun 19, 2024

[Build] Don't resign already signed jars #2143

Merged

HannesWell closed this as completed in #2140 Jun 19, 2024

HannesWell closed this as completed in 2c17a6d Jun 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable resigning of already signed jars #2134

Disable resigning of already signed jars #2134

netomi commented Jun 13, 2024 •

edited

Loading

akurtakov commented Jun 13, 2024

netomi commented Jun 13, 2024

netomi commented Jun 13, 2024 •

edited

Loading

HannesWell commented Jun 13, 2024 •

edited

Loading

laeubi commented Jun 14, 2024

HannesWell commented Jun 19, 2024

merks commented Jun 19, 2024

netomi commented Jun 19, 2024

akurtakov commented Jun 19, 2024

netomi commented Jun 20, 2024

HannesWell commented Jun 20, 2024

Disable resigning of already signed jars #2134

Disable resigning of already signed jars #2134

Comments

netomi commented Jun 13, 2024 • edited Loading

akurtakov commented Jun 13, 2024

netomi commented Jun 13, 2024

netomi commented Jun 13, 2024 • edited Loading

HannesWell commented Jun 13, 2024 • edited Loading

laeubi commented Jun 14, 2024

HannesWell commented Jun 19, 2024

merks commented Jun 19, 2024

netomi commented Jun 19, 2024

akurtakov commented Jun 19, 2024

netomi commented Jun 20, 2024

HannesWell commented Jun 20, 2024

netomi commented Jun 13, 2024 •

edited

Loading

netomi commented Jun 13, 2024 •

edited

Loading

HannesWell commented Jun 13, 2024 •

edited

Loading