[backport 2.3] Allow testing with different daemons #5650
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ericson2314
force-pushed
the
2.3-alt-daemon-test-suite
branch
from
d95f60b
to
406add7
Compare
|
Couldn’t we rather use master’s testsuite with an old client (adding guards like the existing |
|
@regnat the command line has changed so much, I expect that to be a big plain. Think of all those Also remember this is a 1-time cost. Going forward we already have the support we we need to e.g. test 2.4 with master daemon. |
Ericson2314
force-pushed
the
2.3-alt-daemon-test-suite
branch
from
406add7
to
730b079
Compare
|
🎉 All dependencies have been resolved ! |
Ericson2314
force-pushed
the
2.3-alt-daemon-test-suite
branch
2 times, most recently
from
6d6502c
to
c4c448c
Compare
|
I marked this as stale due to inactivity. → More info |
Ericson2314
force-pushed
the
2.3-alt-daemon-test-suite
branch
from
c4c448c
to
ab6af57
Compare
|
🎉 All dependencies have been resolved ! |
Ericson2314
force-pushed
the
2.3-alt-daemon-test-suite
branch
from
ab6af57
to
30cb189
Compare
The `remote-store` test loads the `user-env` one to test nix-env when using the daemon, but actually does it incorrectly because every test starts (in `common.sh`) by resetting the value of `NIX_REMOTE`, meaning that the `user-env` test will never use the daemon. Fix this by setting `NIX_REMOTE_` before sourcing `user-env.sh` in the `remote-store` test, so that `NIX_REMOTE` is correctly set inside the test (cherry picked from commit f6ac888)
This requires adding `nix` to its own closure which is a bit unfortunate, but as it is optional (the test will be disabled if `OUTER_NIX` is unset) it shouldn't be too much of an issue. (Ideally this should go in another derivation so that we can build Nix and run the test independently, but as the tests are running in the same derivation as the build it's a bit complicated to do so). (cherry picked from commit 5716345)
That way we can run them without rebuilding Nix (cherry picked from commit a0866c8)
When `NIX_DAEMON_PACKAGE` is set, make all the tests use the Nix daemon. That way we can test every piece of Nix functionality both with and without the daemon. Tests for which using the daemon isn’t possible or doesn’t make sens can selectively be disabled with `needLocalStore` (cherry picked from commit addacfc)
For some reason, an old socket occasionally stays here on OSX, causing the subsequent tests to fail (cherry picked from commit c2c0dba)
(cherry picked from commit 1e6faa7)
- Don’t hardcode the “newer” version - Remove an ill-placed `return` (cherry picked from commit 3a2fc9c)
Having the `post-build-hook` use `nix` from the client package can lead to a deadlock in case there’s a db migration to do between both, as a `nix` command running inside the hook will run as root (and as such will bypass the daemon), so might trigger a db migration, which will get stuck trying to get a global lock on the DB (as the daemon that ran the hook already has a lock on it). (cherry picked from commit 93eadd5)
The eventual PATH entry needs the `.../bin` or we will not use the right daemon. (cherry picked from commit 06fb6ae)
We want the old behavior, since this is Nix 2.3.
(cherry picked from commit 9813e54)
Add an `$` at the end of the `grep` regex. Without it, `checkRef foo` would always imply `checkRef foo.drv`. We want to tell these situations apart to more precisely test what is going on. (cherry picked from commit f587598)
(cherry picked from commit 5dbbf23)
Use `set -u` and `set -o pipefail` to catch accidental mistakes and failures more strongly. - `set -u` catches the use of undefined variables - `set -o pipefail` catches failures (like `set -e`) earlier in the pipeline. This makes the tests a bit more robust. It is nice to read code not worrying about these spurious success paths (via uncaught) errors undermining the tests. Indeed, I caught some bugs doing this. There are a few tests where we run a command that should fail, and then search its output to make sure the failure message is one that we expect. Before, since the `grep` was the last command in the pipeline the exit code of those failing programs was silently ignored. Now with `set -o pipefail` it won't be, and we have to do something so the expected failure doesn't accidentally fail the test. To do that we use `expect` and a new `expectStderr` to check for the exact failing exit code. See the comments on each for why. `grep -q` is replaced with `grepQuiet`, see the comments on that function for why. `grep -v` when we just want the exit code is replaced with `grepInverse, see the comments on that function for why. `grep -q -v` together is, surprise surprise, replaced with `grepQuietInverse`, which is both combined. (cherry picked from commit c118361)
Ericson2314
force-pushed
the
2.3-alt-daemon-test-suite
branch
from
30cb189
to
0435ae9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal is to make something like
testNixVersionson master which can run the install checks against two pre-built nix versions. The goal is to then use that function in a later version of Nix (to avoid cycle dependencies!) to check 2.3's (modernized) test suite against a 2.3 client but newer daemon.Concretely what his PR is doing is cherry-picking a number of test suite improves to 2.3 from later versions that generally harden the testsuite and in particular allow for this cross-version testing.
For later:
flake.nixto thisrelease.nix.depends on #5649depends on #5753depends on #5754Depends on #9495