run_test fails

[Yamin88]

Several algorithms didn't work properly

didn't learn ssh-rsa3072-falcon512 correctly
learn additional hostkeys, type=ssh-rsa3072-dilithium2aes
learn hostkeys ssh-rsa3072-dilithium2aes (got 1 wanted 30)
didn't learn ssh-rsa3072-dilithium2aes correctly
learn additional hostkeys, type=ssh-rsa3072-picnicL1full
learn hostkeys ssh-rsa3072-picnicL1full (got 1 wanted 30)
didn't learn ssh-rsa3072-picnicL1full correctly
learn additional hostkeys, type=ssh-rsa3072-sphincsharaka128fsimple
learn hostkeys ssh-rsa3072-sphincsharaka128fsimple (got 1 wanted 30)
didn't learn ssh-rsa3072-sphincsharaka128fsimple correctly
learn additional hostkeys, type=ssh-ecdsa-nistp256-falcon512
learn hostkeys ssh-ecdsa-nistp256-falcon512 (got 1 wanted 30)
didn't learn ssh-ecdsa-nistp256-falcon512 correctly
learn additional hostkeys, type=ssh-ecdsa-nistp521-falcon1024
learn hostkeys ssh-ecdsa-nistp521-falcon1024 (got 1 wanted 30)
didn't learn ssh-ecdsa-nistp521-falcon1024 correctly
learn additional hostkeys, type=ssh-ecdsa-nistp384-dilithium3
learn hostkeys ssh-ecdsa-nistp384-dilithium3 (got 1 wanted 30)
.
.
.
Another issue at the end of executing this command, see below

old key present
didn't learn changed key
check rotate primary hostkey
ssh -oStrictHostKeyChecking=yes -oHostKeyAlgorithms=ssh-ed25519 failed
didn't learn changed key
failed hostkey rotate
make[1]: *** [Makefile:224: t-exec] Error 1
make[1]: Leaving directory '/home/yaser/openssh/regress'
make: *** [Makefile:730: t-exec] Error 2

[christianpaquin]

Can you give a bit more details, about your environment (OS, version, etc.), and how you invoked the tests?

[Yamin88]

Hi christian,
Please refer below OS, Version
OS Version is Ubuntu 20.04.1 LTS.
OpenSSL 1.1.1f 31 Mar 2020

I follow the steps according the instruction

1. Step 0 - install successfully
2. Step 1 - liboqs build successfully
3. Step 2 - ./oqs-scripts/build_openssh.sh - Build Successful
   When I run oqs-test/run_tests.sh , I got 2 error according to the above.

Please kindly advise.

[baentsch]

@Yamin88 We happen to have an AWS Development VM with exactly those OS specs -- All steps you list above are running just fine -- however the tests seem to take hours -- but still the errors you list above didn't occur (yet -- tests still running...).

Can you pinpoint the specific test that fails so we can try to reproduce? Can you also confirm the test passes on a "stock" (non-OQS-) openssh (v8.9)? Particularly the last error message makes me wonder whether this really is an OQS error as a non-OQS algorithm (ed25519) is found failing in an openssh test...

[baentsch]

PS: After a long time, the errors reported above also appeared in my setup. However, they were preceded by

learn additional hostkeys, type=ssh-ed25519
learn hostkeys ssh-ed25519 (got 1 wanted 30)
learn additional hostkeys, [email protected]
ssh -oStrictHostKeyChecking=yes [email protected],ssh-ed25519,[email protected],rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],ssh-falcon512,ssh-falcon1024,ssh-dilithium3,ssh-dilithium2aes,ssh-dilithium5aes,ssh-picnicL1full,ssh-picnicL3FS,ssh-sphincsharaka128fsimple,ssh-sphincsharaka192frobust,ssh-rsa3072-falcon512,ssh-rsa3072-dilithium2aes,ssh-rsa3072-picnicL1full,ssh-rsa3072-sphincsharaka128fsimple,ssh-ecdsa-nistp256-falcon512,ssh-ecdsa-nistp521-falcon1024,ssh-ecdsa-nistp384-dilithium3,ssh-ecdsa-nistp256-dilithium2aes,ssh-ecdsa-nistp521-dilithium5aes,ssh-ecdsa-nistp256-picnicL1full,ssh-ecdsa-nistp384-picnicL3FS,ssh-ecdsa-nistp256-sphincsharaka128fsimple,ssh-ecdsa-nistp384-sphincsharaka192frobust failed
learn hostkeys [email protected] (got 1 wanted 30)
didn't learn [email protected] correctly

which again points to a more general error in the test case: ed25519 is not an OQS algorithm. But then again, I'm no expert on openssh (incl. its testing)., Maybe @xvzcf or @christianpaquin could comment more?

FYI (answering my own question above): The error gets triggered right away by running PATH=pwd/..:$PATH:. TEST_SHELL=/bin/sh sh test-exec.sh pwd hostkey-rotate.sh in regress.

[christianpaquin]

I'm no expert either on the OpenSSH internals, but indeed checking if the non-OQS version of OpenSSH fails would be very instructive.

[baentsch]

checking if the non-OQS version of OpenSSH fails would be very instructive.

Done. It doesn't (fail). So this is an OQS-issue.

But then again, further tests fail beyond the ones labelled "investigate further". I'm in the process of extending that list, checking it in and adding the test to CI such that this a) doesn't happen again and b) we can decide how/when to tackle #89.

[Yamin88]

Hi Everyone,
Thanks for giving advise.
It solved because I install classic OpenSSH and generate the keys for testing.
I uninstall openssh server , client and re-run again everything from scratch then it works.

Best Regards,
Yamin