Skip to content

Commit

Permalink
Upgrade to OpenSSH v9.7
Browse files Browse the repository at this point in the history
This is a first pass at resolving all of the merge conflicts between the current tip of `OQS-v8` and the `V_9_7_P1` tag in upstream OpenSSH.

The merge strategy here differs a bit from previous upstream merges (e.g. PR #106 and PR #121) where all of the changes were squashed and incorporated into a single commit and applied to the trunk. This is a more typical `git merge` in that we retain both parents and their commit histories. This will make future merges more straightforward by allowing git to notice the shared history and avoid marking these merged commits as conflicting changes.

Here's the git-foo used to script the merge and handle the false positives from the "squash merges".

```
oqs_tip=OQS-v8
openssh_release=V_9_7_P1
git merge ${openssh_release}
base=`git merge-base ${oqs_tip} ${openssh_release}`
for f in `git diff --name-only --diff-filter=U`; do
  # This fetches all of the commits which touched the file since the merge base
  # Filter out the two commits for the 8.6 and 8.9 merges since they are technically already incorporated
  conflicts=$(git log --oneline ${base}..${oqs_tip} -- $f | ggrep -v -P '(1f58edd|f058d3168)')
  # Check if we have no OQS-OpenSSH conflicts specific
  if [[ -z ${conflicts} ]]; then
    echo "$f has no conflicts"
    # Resolve the conflict by taking the upstream version of the file
    git checkout --theirs -- $f
    git add $f
  else
    echo "$f has conflicts"
    echo ${conflicts}
    # Send all of the OQS diffs to a file to help resolve the merge conflicts
    for c in `echo ${conflicts} | cut -d ' ' -f1`; do
      git show $c -- $f >> ~/conflicting_diffs.t
    done
  fi
done
```

For the remaining conflicts, I went through each file one-by-one with this pseudo-algorithm:
1. Incorporate all changes from both sides that have no direct conflicts.
2. Look for OQS specific changes with conflicts and apply them as-appropriate.
3. Take the upstream version for any remaining conflicts.

Callouts from this process:
- `sshkey.c` and `sshkey.h` experienced a major refactor upstream that impacted how OQS modified these files. I simply took the upstream versions for now and plan to address the conflict properly in a separate PR.
- Kept `README.md` as-is from OQS and applied changes to `README.original.md`.
- Took `.depend` from upstream, will update in a subsequent commit.
- `version.h` retained the 2022-01 datestamp from OQS, will update this when we're ready to stage a release.
- In `ssh-keygen.c` the `OQS_TEMPLATE_FRAGMENT_PRINT_RESOURCE_RECORDS_START` template changed to accept two additional arguments `opts` and `nopts`. I added these in manually for now.

To self-check I did the following:
- Test build by running `build_openssh.sh` and finding compiler errors.
- Run `git diff HEAD V_9_7_P1` to highlight all the changes and assert that all changes were introduced by OQS alone.

This last process flagged a handful of issues. Mostly around duplicated code blocks from taking them from previous upstream merges and this current merge and git not noticing it. With that out of the way, I'm reasonably confident that this PR is pretty close to upstream v9.7 with only the changes from OQS applied to it.

So after all that, what's working so far? `build_openssh.sh` will build the project but fail to install with some error about unknown key types.

What's next?
- Properly handle the merge conflicts in `sshkey.(c|h)`.
- Regenerate `.depend`.
- Fix the impacted OQS templates and regenerate the source.
- Cut a new `OQS-v9` branch and update `version.h`.
  • Loading branch information
geedo0 committed Jul 9, 2024
2 parents e452df2 + 86bdd38 commit 490ff89
Show file tree
Hide file tree
Showing 279 changed files with 18,225 additions and 11,728 deletions.
73 changes: 33 additions & 40 deletions .depend

Large diffs are not rendered by default.

17 changes: 14 additions & 3 deletions .github/ci-status.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,15 @@
[![C/C++ CI](https://github.com/openssh/openssh-portable/actions/workflows/c-cpp.yml/badge.svg)](https://github.com/openssh/openssh-portable/actions/workflows/c-cpp.yml)
[![C/C++ CI self-hosted](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/selfhosted.yml/badge.svg)](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/selfhosted.yml)
[![Upstream self-hosted](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/upstream.yml/badge.svg)](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/upstream.yml)
master :
[![C/C++ CI](https://github.com/openssh/openssh-portable/actions/workflows/c-cpp.yml/badge.svg)](https://github.com/openssh/openssh-portable/actions/workflows/c-cpp.yml?query=branch:master)
[![C/C++ CI self-hosted](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/selfhosted.yml/badge.svg)](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/selfhosted.yml?query=branch:master)
[![Upstream self-hosted](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/upstream.yml/badge.svg)](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/upstream.yml?query=branch:master)
[![CIFuzz](https://github.com/openssh/openssh-portable/actions/workflows/cifuzz.yml/badge.svg)](https://github.com/openssh/openssh-portable/actions/workflows/cifuzz.yml)
[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/openssh.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:openssh)
[![Coverity Status](https://scan.coverity.com/projects/21341/badge.svg)](https://scan.coverity.com/projects/openssh-portable)

9.4 :
[![C/C++ CI](https://github.com/openssh/openssh-portable/actions/workflows/c-cpp.yml/badge.svg?branch=V_9_4)](https://github.com/openssh/openssh-portable/actions/workflows/c-cpp.yml?query=branch:V_9_4)
[![C/C++ CI self-hosted](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/selfhosted.yml/badge.svg?branch=V_9_4)](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/selfhosted.yml?query=branch:V_9_4)

9.3 :
[![C/C++ CI](https://github.com/openssh/openssh-portable/actions/workflows/c-cpp.yml/badge.svg?branch=V_9_3)](https://github.com/openssh/openssh-portable/actions/workflows/c-cpp.yml?query=branch:V_9_3)
[![C/C++ CI self-hosted](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/selfhosted.yml/badge.svg?branch=V_9_3)](https://github.com/openssh/openssh-portable-selfhosted/actions/workflows/selfhosted.yml?query=branch:V_9_3)
210 changes: 184 additions & 26 deletions .github/configs
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,13 @@
# LTESTS

config=$1
if [ "$config" = "" ]; then
config="default"
fi

unset CC CFLAGS CPPFLAGS LDFLAGS LTESTS SUDO

TEST_TARGET="tests"
TEST_TARGET="tests compat-tests"
LTESTS=""
SKIP_LTESTS=""
SUDO=sudo # run with sudo by default
Expand All @@ -25,26 +30,97 @@ case "$config" in
default|sol64)
;;
c89)
# If we don't have LLONG_MAX, configure will figure out that it can
# get it by setting -std=gnu99, at which point we won't be testing
# C89 any more. To avoid this, feed it in via CFLAGS.
llong_max=`gcc -E -dM - </dev/null | \
awk '$2=="__LONG_LONG_MAX__"{print $3}'`
CPPFLAGS="-DLLONG_MAX=${llong_max}"

CC="gcc"
CFLAGS="-Wall -std=c89 -pedantic -Werror=vla"
CONFIGFLAGS="--without-zlib"
LIBCRYPTOFLAGS="--without-openssl"
TEST_TARGET=t-exec
;;
cygwin-release)
CONFIGFLAGS="--with-libedit --with-xauth=/usr/bin/xauth --disable-strip --with-security-key-builtin"
# See https://cygwin.com/git/?p=git/cygwin-packages/openssh.git;a=blob;f=openssh.cygport;hb=HEAD
CONFIGFLAGS="--with-xauth=/usr/bin/xauth --with-security-key-builtin"
CONFIGFLAGS="$CONFIGFLAGS --with-kerberos5=/usr --with-libedit --disable-strip"
;;
clang-12-Werror)
CC="clang-12"
# clang's implicit-fallthrough requires that the code be annotated with
# __attribute__((fallthrough)) and does not understand /* FALLTHROUGH */
CFLAGS="-Wall -Wextra -O2 -Wno-error=implicit-fallthrough"
CFLAGS="-Wall -Wextra -O2 -Wno-error=implicit-fallthrough -Wno-error=unused-parameter"
CONFIGFLAGS="--with-pam --with-Werror"
;;
*-sanitize-*)
case "$config" in
gcc-*)
CC=gcc
;;
clang-*)
# Find the newest available version of clang
for i in `seq 10 99`; do
clang="`which clang-$i 2>/dev/null`"
[ -x "$clang" ] && CC="$clang"
done
;;
esac
# Put Sanitizer logs in regress dir.
SANLOGS=`pwd`/regress
# - We replace chroot with chdir so that the sanitizer in the preauth
# privsep process can read /proc.
# - clang does not recognizes explicit_bzero so we use bzero
# (see https://github.com/google/sanitizers/issues/1507
# - openssl and zlib trip ASAN.
# - sp_pwdp returned by getspnam trips ASAN, hence disabling shadow.
case "$config" in
*-sanitize-address)
CFLAGS="-fsanitize=address -fno-omit-frame-pointer"
LDFLAGS="-fsanitize=address"
CPPFLAGS='-Dchroot=chdir -Dexplicit_bzero=bzero -D_FORTIFY_SOURCE=0 -DASAN_OPTIONS=\"detect_leaks=0:log_path='$SANLOGS'/asan.log\"'
CONFIGFLAGS=""
TEST_TARGET="t-exec"
;;
clang-sanitize-memory)
CFLAGS="-fsanitize=memory -fsanitize-memory-track-origins -fno-omit-frame-pointer"
LDFLAGS="-fsanitize=memory"
CPPFLAGS='-Dchroot=chdir -Dexplicit_bzero=bzero -DMSAN_OPTIONS=\"log_path='$SANLOGS'/msan.log\"'
CONFIGFLAGS="--without-zlib --without-shadow"
LIBCRYPTOFLAGS="--without-openssl"
TEST_TARGET="t-exec"
;;
*-sanitize-undefined)
CFLAGS="-fsanitize=undefined"
LDFLAGS="-fsanitize=undefined"
;;
*)
echo unknown sanitize option;
exit 1;;
esac
features="--disable-security-key --disable-pkcs11"
hardening="--without-sandbox --without-hardening --without-stackprotect"
privsep="--with-privsep-user=root"
CONFIGFLAGS="$CONFIGFLAGS $features $hardening $privsep"
# Because we hobble chroot we can't test it.
SKIP_LTESTS=sftp-chroot
;;
gcc-11-Werror)
CC="gcc"
CC="gcc-11"
# -Wnoformat-truncation in gcc 7.3.1 20180130 fails on fmt_scaled
# -Wunused-result ignores (void) so is not useful. See
# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66425
CFLAGS="-O2 -Wno-format-truncation -Wimplicit-fallthrough=4 -Wno-unused-parameter -Wno-unused-result"
CONFIGFLAGS="--with-pam --with-Werror"
;;
gcc-12-Werror)
CC="gcc-12"
# -Wnoformat-truncation in gcc 7.3.1 20180130 fails on fmt_scaled
CFLAGS="-Wall -Wextra -Wno-format-truncation -O2 -Wimplicit-fallthrough=4"
# -Wunused-result ignores (void) so is not useful. See
# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66425
CFLAGS="-O2 -Wno-format-truncation -Wimplicit-fallthrough=4 -Wno-unused-parameter -Wno-unused-result"
CONFIGFLAGS="--with-pam --with-Werror"
;;
clang*|gcc*)
Expand All @@ -53,7 +129,7 @@ case "$config" in
kitchensink)
CONFIGFLAGS="--with-kerberos5 --with-libedit --with-pam"
CONFIGFLAGS="${CONFIGFLAGS} --with-security-key-builtin --with-selinux"
CONFIGFLAGS="${CONFIGFLAGS} --with-cflags=-DSK_DEBUG"
CFLAGS="-DSK_DEBUG -DSANDBOX_SECCOMP_FILTER_DEBUG"
;;
hardenedmalloc)
CONFIGFLAGS="--with-ldflags=-lhardened_malloc"
Expand Down Expand Up @@ -81,11 +157,25 @@ case "$config" in
CONFIGFLAGS="--with-pam"
SSHD_CONFOPTS="UsePam yes"
;;
boringssl)
CONFIGFLAGS="--disable-pkcs11"
LIBCRYPTOFLAGS="--with-ssl-dir=/opt/boringssl --with-rpath=-Wl,-rpath,"
;;
libressl-*)
LIBCRYPTOFLAGS="--with-ssl-dir=/opt/libressl --with-rpath=-Wl,-rpath,"
;;
putty-*)
CONFIGFLAGS="--with-plink=/usr/local/bin/plink --with-puttygen=/usr/local/bin/puttygen"
# We don't need to rerun the regular tests, just the interop ones.
TEST_TARGET=interop-tests
;;
openssl-*)
LIBCRYPTOFLAGS="--with-ssl-dir=/opt/openssl --with-rpath=-Wl,-rpath,"
# OpenSSL 1.1.1 specifically has a bug in its RNG that breaks reexec
# fallback. See https://bugzilla.mindrot.org/show_bug.cgi?id=3483
if [ "$config" = "openssl-1.1.1" ]; then
SKIP_LTESTS="reexec"
fi
;;
selinux)
CONFIGFLAGS="--with-selinux"
Expand All @@ -97,7 +187,7 @@ case "$config" in
LIBCRYPTOFLAGS="--without-openssl"
TEST_TARGET=t-exec
;;
valgrind-[1-4]|valgrind-unit)
valgrind-[1-5]|valgrind-unit)
# rlimit sandbox and FORTIFY_SOURCE confuse Valgrind.
CONFIGFLAGS="--without-sandbox --without-hardening"
CONFIGFLAGS="$CONFIGFLAGS --with-cppflags=-D_FORTIFY_SOURCE=0"
Expand All @@ -106,15 +196,19 @@ case "$config" in
export TEST_SSH_ELAPSED_TIMES
# Valgrind slows things down enough that the agent timeout test
# won't reliably pass, and the unit tests run longer than allowed
# by github so split into three separate tests.
tests2="rekey integrity try-ciphers sftp"
tests3="krl forward-control sshsig agent-restrict kextype"
# by github so split into separate tests.
tests2="integrity try-ciphers"
tests3="krl forward-control sshsig agent-restrict kextype sftp"
tests4="cert-userkey cert-hostkey kextype sftp-perm keygen-comment percent"
tests5="rekey"
case "$config" in
valgrind-1)
# All tests except agent-timeout (which is flaky under valgrind)
#) and slow ones that run separately to increase parallelism.
SKIP_LTESTS="agent-timeout ${tests2} ${tests3} ${tests4}"
# All tests except agent-timeout (which is flaky under valgrind),
# connection-timeout (which doesn't work since it's so slow)
# and hostbased (since valgrind won't let ssh exec keysign).
# Slow ones are run separately to increase parallelism.
SKIP_LTESTS="agent-timeout connection-timeout hostbased"
SKIP_LTESTS="$SKIP_LTESTS ${tests2} ${tests3} ${tests4} ${tests5}"
;;
valgrind-2)
LTESTS="${tests2}"
Expand All @@ -125,11 +219,18 @@ case "$config" in
valgrind-4)
LTESTS="${tests4}"
;;
valgrind-5)
LTESTS="${tests5}"
;;
valgrind-unit)
TEST_TARGET="unit USE_VALGRIND=1"
;;
esac
;;
zlib-develop)
INSTALL_ZLIB=develop
CONFIGFLAGS="--with-zlib=/opt/zlib --with-rpath=-Wl,-rpath,"
;;
*)
echo "Unknown configuration $config"
exit 1
Expand All @@ -139,28 +240,53 @@ esac
# The Solaris 64bit targets are special since they need a non-flag arg.
case "$config" in
sol64*)
CONFIGFLAGS="x86_64 --with-cflags=-m64 --with-ldflags=-m64 ${CONFIGFLAGS}"
LIBCRYPTOFLAGS="--with-ssl-dir=/usr/local/ssl64"
CONFIGFLAGS="--target=x86_64 --with-cflags=-m64 --with-ldflags=-m64 ${CONFIGFLAGS}"
LIBCRYPTOFLAGS="--with-ssl-dir=/usr/local/ssl64 --with-rpath=-Wl,-rpath,"
;;
esac

case "${TARGET_HOST}" in
aix*)
CONFIGFLAGS="--disable-security-key"
LIBCRYPTOFLAGS="--without-openssl"
# These are slow real or virtual machines so skip the slowest tests
# (which tend to be thw ones that transfer lots of data) so that the
# test run does not time out.
# The agent-restrict test fails due to some quoting issue when run
# with sh or ksh so specify bash for now.
TEST_TARGET="t-exec unit TEST_SHELL=bash"
SKIP_LTESTS="rekey sftp"
;;
debian-riscv64)
# This machine is fairly slow, so skip the unit tests.
TEST_TARGET="t-exec"
;;
dfly58*|dfly60*)
# scp 3-way connection hangs on these so skip until sorted.
SKIP_LTESTS=scp3
;;
fbsd6)
# Native linker is not great with PIC so OpenSSL is built w/out.
CONFIGFLAGS="${CONFIGFLAGS} --disable-security-key"
;;
hurd)
SKIP_LTESTS="forwarding multiplex proxy-connect hostkey-agent agent-ptrace"
;;
minix3)
LIBCRYPTOFLAGS="--without-openssl --disable-security-key"
CONFIGFLAGS="${CONFIGFLAGS} --disable-security-key"
# Unix domain sockets don't work quite like we expect, so also
# disable FD passing (and thus multiplexing).
CONFIGFLAGS="${CONFIGFLAGS} --disable-fd-passing"
LIBCRYPTOFLAGS="--without-openssl"

# Minix does not have a loopback interface so we have to skip any
# test that relies on one.
# Also, Minix seems to be very limited in the number of select()
# calls that can be operating concurrently, so prune additional tests for that.
T="addrmatch agent-restrict brokenkeys cfgmatch cfgmatchlisten cfgparse connect
connect-uri exit-status forward-control forwarding hostkey-agent
key-options keyscan knownhosts-command login-timeout multiplex
T="addrmatch agent-restrict brokenkeys cfgmatch cfgmatchlisten cfgparse
connect connect-uri dynamic-forward exit-status forwarding
forward-control
hostkey-agent key-options keyscan knownhosts-command login-timeout
reconfigure reexec rekey scp scp-uri scp3 sftp sftp-badcmds
sftp-batch sftp-cmds sftp-glob sftp-perm sftp-uri stderr-data
transfer"
Expand All @@ -173,6 +299,11 @@ case "${TARGET_HOST}" in
# SHA256 functions in sha2.h conflict with OpenSSL's breaking sk-dummy
CONFIGFLAGS="${CONFIGFLAGS} --without-hardening --disable-security-key"
;;
openwrt-*)
CONFIGFLAGS="${CONFIGFLAGS} --without-zlib"
LIBCRYPTOFLAGS="--without-openssl"
TEST_TARGET="t-exec"
;;
sol10|sol11)
# sol10 VM is 32bit and the unit tests are slow.
# sol11 has 4 test configs so skip unit tests to speed up.
Expand All @@ -184,23 +315,50 @@ case "${TARGET_HOST}" in
;;
esac

# Unless specified otherwise, build without OpenSSL on Mac OS since
# modern versions don't ship with libcrypto.
case "`./config.guess`" in
host=`./config.guess`
case "$host" in
*cygwin)
SUDO=""
# Don't run compat tests on cygwin as they don't currently compile.
TEST_TARGET="tests"
;;
*-darwin*)
# Unless specified otherwise, build without OpenSSL on Mac OS since
# modern versions don't ship with libcrypto.
LIBCRYPTOFLAGS="--without-openssl"
TEST_TARGET=t-exec

# On some OS X runners we can't write to /var/empty.
CONFIGFLAGS="${CONFIGFLAGS} --with-privsep-path=/usr/local/empty"

case "$host" in
*-darwin22.*)
# sudo -S nobody doesn't work on macos 13 for some reason.
SKIP_LTESTS="agent-getpeereid" ;;
esac
;;
esac

# If we have a local openssl/libressl, use that.
# Unless specifically configured, search for a suitable version of OpenSSL,
# otherwise build without it.
if [ -z "${LIBCRYPTOFLAGS}" ]; then
LIBCRYPTOFLAGS="--without-openssl"
# last-match
for i in /usr/local /usr/local/ssl /usr/local/opt/openssl; do
for i in /usr /usr/local /usr/local/ssl /usr/local/opt/openssl; do
ver="none"
if [ -x ${i}/bin/openssl ]; then
LIBCRYPTOFLAGS="--with-ssl-dir=${i}"
ver="$(${i}/bin/openssl version)"
fi
case "$ver" in
none) ;;
"OpenSSL 0."*|"OpenSSL 1.0."*|"OpenSSL 1.1.0"*) ;;
"LibreSSL 2."*|"LibreSSL 3.0."*) ;;
*) LIBCRYPTOFLAGS="--with-ssl-dir=${i}" ;;
esac
done
if [ "${LIBCRYPTOFLAGS}" = "--without-openssl" ]; then
TEST_TARGET="t-exec"
fi
fi

CONFIGFLAGS="${CONFIGFLAGS} ${LIBCRYPTOFLAGS}"
Expand All @@ -210,5 +368,5 @@ if [ -x "$(which plink 2>/dev/null)" ]; then
export REGRESS_INTEROP_PUTTY
fi

export CC CFLAGS LTESTS SUDO
export CC CFLAGS CPPFLAGS LDFLAGS LTESTS SUDO
export TEST_TARGET TEST_SSH_UNSAFE_PERMISSIONS TEST_SSH_FAIL_FATAL
2 changes: 1 addition & 1 deletion .github/configure.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@ if [ "x$LDFLAGS" != "x" ]; then
fi

echo ./configure ${CONFIGFLAGS}
./configure ${CONFIGFLAGS}
./configure ${CONFIGFLAGS} 2>&1
16 changes: 15 additions & 1 deletion .github/run_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,22 @@

set -ex

# If we want to test hostbased auth, set up the host for it.
if [ ! -z "$SUDO" ] && [ ! -z "$TEST_SSH_HOSTBASED_AUTH" ]; then
sshconf=/usr/local/etc
hostname | $SUDO tee $sshconf/shosts.equiv >/dev/null
echo "EnableSSHKeysign yes" | $SUDO tee $sshconf/ssh_config >/dev/null
$SUDO mkdir -p $sshconf
$SUDO cp -p /etc/ssh/ssh_host*key* $sshconf
$SUDO make install
for key in $sshconf/ssh_host*key*.pub; do
echo `hostname` `cat $key` | \
$SUDO tee -a $sshconf/ssh_known_hosts >/dev/null
done
fi

output_failed_logs() {
for i in regress/failed*; do
for i in regress/failed*.log; do
if [ -f "$i" ]; then
echo -------------------------------------------------------------------------
echo LOGFILE $i
Expand Down
Loading

0 comments on commit 490ff89

Please sign in to comment.