Upgrade to OpenSSH v9.7

[geedo0]

This is a first pass at resolving all of the merge conflicts between the current tip of OQS-v8 and the V_9_7_P1 tag in upstream OpenSSH (Issue #135).

The merge strategy here differs a bit from previous upstream merges (e.g. PR #106 and PR #121) where all of the changes were squashed and incorporated into a single commit and applied to the trunk. This is a more typical git merge in that we retain both parents and their commit histories. This will make future merges more straightforward by allowing git to notice the shared history and avoid marking these merged commits as conflicting changes. For the maintainers, make sure to merge using the default "Merge Pull Request" button. I tested this on my own fork and seems to be the only one of Github's options that preserves the history.

Here's the git-foo used to script the merge and handle the false positives from the "squash merges".

oqs_tip=OQS-v8
openssh_release=V_9_7_P1
git merge ${openssh_release}
base=`git merge-base ${oqs_tip} ${openssh_release}`
for f in `git diff --name-only --diff-filter=U`; do
  # This fetches all of the commits which touched the file since the merge base
  # Filter out the two commits for the 8.6 and 8.9 merges since they are technically already incorporated
  conflicts=$(git log --oneline ${base}..${oqs_tip} -- $f | ggrep -v -P '(1f58edd59|f058d3168)')
  # Check if we have no OQS-OpenSSH conflicts specific
  if [[ -z ${conflicts} ]]; then
    echo "$f has no conflicts"
    # Resolve the conflict by taking the upstream version of the file
    git checkout --theirs -- $f
    git add $f
  else
    echo "$f has conflicts"
    echo ${conflicts}
    # Send all of the OQS diffs to a file to help resolve the merge conflicts
    for c in `echo ${conflicts} | cut -d ' ' -f1`; do
      git show $c -- $f >> ~/conflicting_diffs.t
    done
  fi
done

For the remaining conflicts, I went through each file one-by-one with this pseudo-algorithm:

1. Incorporate all changes from both sides that have no direct conflicts.
2. Look for OQS specific changes with conflicts and apply them as-appropriate.
3. Take the upstream version for any remaining conflicts.

Callouts from this process:

• sshkey.c and sshkey.h experienced a major refactor upstream that impacted how OQS modified these files. I simply took the upstream versions for now and plan to address the conflict properly in a separate PR.
• Kept README.md as-is from OQS and applied changes to README.original.md.
• Took .depend from upstream, will update in a subsequent commit.
• version.h retained the 2022-01 datestamp from OQS, will update this when we're ready to stage a release.
• In ssh-keygen.c the OQS_TEMPLATE_FRAGMENT_PRINT_RESOURCE_RECORDS_START template changed to accept two additional arguments opts and nopts. I added these in manually for now.

To self-check I did the following:

• Test build by running build_openssh.sh and finding compiler errors.
• Run git diff HEAD V_9_7_P1 to highlight all the changes and assert that all changes were introduced by OQS alone.

This last process flagged a handful of issues. Mostly around duplicated code blocks from taking them from previous upstream merges and this current merge and git not noticing it. With that out of the way, I'm reasonably confident that this PR is pretty close to upstream v9.7 with only the changes from OQS applied to it.

So after all that, what's working so far? build_openssh.sh will build the project but fail to install with some error about unknown key types. The failure I had locally is the same one reported by the CI job.

What's next?

• Properly handle the merge conflicts in sshkey.(c|h).
• Regenerate .depend.
• Fix the impacted OQS templates and regenerate the source.
• Cut a new OQS-v9 branch and update version.h.

[baentsch]

Thanks very much for your work and particularly the explanations, @geedo0 ! So do you indeed prefer to merge this first to the new branch before all CI checks turn green? That feels not quite right, but would surely create a clean git log (upstream merge in this PR, getting OQS integration to work again in the next one), so fine with me unless someone objects.

[geedo0]

So do you indeed prefer to merge this first to the new branch before all CI checks turn green?

Yes, I think it's okay since it's going into the new OQS-v9 branch instead of the stable/working OQS-v8 branch. This way, we can have more targeted PRs that folks can actually scrutinize instead of this 30k LOC behemoth of a PR.

[dstebila]

I've read through the description of your process, and also skimmed the diff between geedo0:merge-ossh97 and openssh:V_9_7_P1. I'm okay with the idea of merging this into OQS-v9 and then continuing with the next phase of work on that branch.

My only comment would be about whether we should omit the .get_allowed_signers* files from our repository; I'm not sure what the purpose of these files is, but if it's about who's authorized to sign certain things, then we may nto intend to authorize the OpenSSH team to sign things related to our fork.

[geedo0]

My only comment would be about whether we should omit the .get_allowed_signers* files from our repository; I'm not sure what the purpose of these files is, but if it's about who's authorized to sign certain things, then we may nto intend to authorize the OpenSSH team to sign things related to our fork.

Good catch, you got me curious and I looked into it. It looks like you're allowed to use SSH keys to sign commits (instead of GPG keys). This file gives you granular control over the SSH keys authorized to do that. This fork doesn't enforce anything related to commit signing, but it's certainly good hygeine to not pull-in such enforcement from upstream.

[geedo0]

I removed the .git_allowed_signers file. As long as you the default "Merge Pull Request" button, that should do what we want and preserve the Git history. This is important as the latest upstream is actually 9.8 now and I need to follow this PR up with yet another upstream merge to keep us up to date.