Initial tuf-on-ci migration signing event

[jku]

Description

As documented in #1250, the plan is to migrate from the in-house scripts and workflows to tuf-on-ci. This was done in root-signing-staging already with decent results. That said the old "staging" was so different that the migration here is necessarily going to involve a lot of things we have not done before.

• We (@jku, @kommendorkapten) are going to build the initial signing event PR this week so it should be ready for signers on August 26th
• Signers can prepare by doing the one-time setup in signer playbook
• There is no tight deadline as originally suspected (online signing can continue while we work on the signing event)
• Because the signing event is a bit special and will contain other changes as well (the PR will be large, sorry about that). we will try to document every change here
• Full migration plan is in https://github.com/sigstore/root-signing/tree/main/docs/migration, pre-signing event work is tracked in tuf-on-ci migration preparation #1247

[jku]

#1321 makes the current metadata available for the signing event -- this is required for the signing event workflow to work correctly. We can update the "current metadata" with new PRs as legacy online signing happens as usual.

[jku]

Maintainer tasks in the signing event before any signing is needed:

• run the tuf-on-ci-import-repo with the configuration file from docs/migration/. This sets signer names, signing periods and expire periods. The meaningful changes are:
  • targets expiry is made very long (the point is that while root and targets have the same signers, there is no need to force both to expire frequently. This should make the signing events slightly simpler without affecting security posture)
  • there will be a few placeholders at first (snapshot online-url and registry.npmjs.org online-url are fixed in followup steps)
  • all keyids will change: this is an unfortunate result of TUF requiring keyids to be calculated from key json contents (which now contain an additional field)
• configure the online signing correctly with tuf-on-ci-delegate
  • both online roles now use the same key
  • timestamp expires in 7 days, signing starts 4 days before
  • snapshot expires in 3650 days, signing starts 4 days before: this is a change from current short expiry. there should be no advantage to snapshot expiring often
• configure registry.npmjs.org online signing correctly with tuf-on-ci-delegate (@kommendorkapten)
• Actual artifact changes if any -- I suggest these are done with PRs into the signing event so relevant folks can review them separately. As a general rule I suggest we do not include changes that have not been tested in staging. (@kommendorkapten) (we decided to skip these see next comment)
  • [targets v11] Add client signing configuration #1194 ?
  • [targets v11] What to do with the GitHub TSA in trusted_root.json #1268 ?
  • [targets v11] Add checkpoint key ID #1195 ?

After that the following steps are

• keyholders sign
• maintainers add the workflow changes from DO NOT MERGE: Enable tuf-on-ci #1313
• final PR review
• main is updated with docs/migration/prep-import.py once more (metadata/ dir must be up-to-date with current "legacy" timestamp before signing event is merged)
• merge, wait for smoke to clear

[jku]

Actual artifact changes if any

We've found a potential client compatibility issue in staging with that, preliminary plan is to avoid the artifact changes in this production signing event. We're hoping the whole signing event experience is going to be easier now, so having another signing event later on to actually handle the new artifacts should be less of a hassle -- as added bonus it gives clients a little more time to test those artifacts in staging.

Please comment here if you have opinions: sigstore/root-signing-staging#161 (see summary in comment)

[jku]

Documenting the remaining steps (with approximate schedule that aims to take over the repo after timestamp 216 on Friday Aug 30):

• keyholders sign
• signing-event PR review (although see next point)
• maintainers add the workflow changes from DO NOT MERGE: Enable tuf-on-ci #1313 into the singing event PR (so that we enable the tuf-on-ci workflows in the same PR)
• (Fri Aug 30) Make PR to update metadata/ dir with docs/migration/prep-import.py once more (metadata/ dir must be up-to-date with current production repository before signing event is merged)
• merge signing-event, wait for smoke to clear: the workflows are now enabled so online-sign and the preprod publish (to https://sigstore.github.io/root-signing/) should work ... but this is when we find any issues with KMS or project settings. This step disables legacy online-signing
• (Mon Sep 2) Resolve any KMS/permission issues
• Manual testing of the preprod repository -- e.g. maybe we want to test older cosign versions that are not currently part of client test matrix?
• Start using new preprod in sigstore-probers Switch the preprod TUF url sigstore-probers#270
• (Tue Sep 3) Make PR to enable publishing to production repository (uncomment code in publish.yml) switch tuf-on-ci "publish-to-production" on #1340
• manually dispatch online-sign so the production publishing happens. This step makes makes new metadata available to clients

As long as we complete the list by Wed Sep 4, on-call should not get any alerts. If we have issues at any point before the last item, we can revert the signing-event merge:

• this revert will make legacy online-signing work again
• we can still redo the steps on this list (without keyholders resigning) at a later date when the issues have been fixed

[haydentherapper]

preprod repository

Remind me, are we going to publish to the preprod GCS bucket, or is preprod the GitHub pages? I think the latter?

[kommendorkapten]

Yeah, GitHub Pages.

[jku]

Remind me, are we going to publish to the preprod GCS bucket, or is preprod the GitHub pages?

what kommendorkapten said.

I'll make sure we have something ready for sigstore-probers workflows

[jku]

reopen, we tried merging but had to revert:

• the GCP kms keyid is incorrect (I have missed the version from the end).
• this could not happen in normal operation (since the keyid is used with gcloud API when keys is created)... but the import that was done here does no such checks
• the keyid is encoded in metadata: I still think that's the right thing to do but the downside is that now metadata needs to be resigned

[jku]

Current status:

• we have merged the signing event
• online-signing with GCP KMS seems to work
• but pushing the signature fails because of required status checks (root-signing: Remove status checks from branch protection community#484)

[jku]

After various infra fixes, the "preprod" repo is now published at https://sigstore.github.io/root-signing/

[jku]

All steps are done, production repo has been published. Looks smooth so far