From b84ab239247eb800bbe09a9cc60f746cecbaa1c0 Mon Sep 17 00:00:00 2001 From: Anderson Torres Date: Thu, 31 Oct 2024 22:19:26 -0300 Subject: [PATCH] doc/hooks/versionCheckHook.section.md: compare `testers.testVersion` --- doc/hooks/versionCheckHook.section.md | 32 +++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/doc/hooks/versionCheckHook.section.md b/doc/hooks/versionCheckHook.section.md index 1ede6a175c39..03c3f8fdea3e 100644 --- a/doc/hooks/versionCheckHook.section.md +++ b/doc/hooks/versionCheckHook.section.md @@ -2,7 +2,9 @@ This hook adds a `versionCheckPhase` to the [`preInstallCheckHooks`](#ssec-installCheck-phase) that runs the main program of the derivation with a `--help` or `--version` argument, and verifies if the `${version}` string is found in the output. -Here is a typical usage: +It does so in a clean environment (using `env --ignore-environment --chdir=/`), and it checks for the `${version}` string in both the `stdout` and the `stderr` of the command. It will report to you in the build log the output it received and it will fail the build if it failed to find `${version}`. + +Here is a typical usage example: :::{.example #ex-versioncheckhook-phantom} @@ -27,7 +29,7 @@ stdenv.mkDerivation (finalAttrs: { }) ``` -It does so in a clean environment (using `env --ignore-environment`), and it checks for the `${version}` string in both the `stdout` and the `stderr` of the command. It will report to you in the build log the output it received and it will fail the build if it failed to find `${version}`. +::: Note that `doInstallCheck` is enabled by default for [`buildPythonPackage`](#buildpythonpackage-function) and [`buildPythonApplication`](#buildpythonapplication-function). @@ -58,3 +60,29 @@ The following attributes control `versionCheckHook`: - `postVersionCheck`: Hook that runs after the check is done. Defaults to an empty string. Note: see also [`testers.testVersion`](#tester-testVersion). + +## Comparison between `versionCheckHook` and `testers.testVersion` {#versioncheckhook-comparison-testversion} + +The most notable difference between `versionCheckHook` and `testers.testVersion` is that `versionCheckHook` runs as a phase during the build time of the package, while `passthru.tests.versions` executes a whole derivation that depends on the package. + +Given the current limitations of Nixpkgs's tooling and CI, `versionCheckHook` fits better in most of typical situations. + +Below we tabulate the differences between `versionCheckHook` and `testers.testVersion`. + + +| Item | `versionCheckHook` | `testers.testVersion` | +|:---------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| +| Customization | Besides the attributes described above, the hook provides no other methods for controlling it. | The tester has access to all feature set of Nix and Nixpkgs, like generating multiple version test derivations from a list of strings, and accessing programs / tools outside the inputs of the derivation. | +| Execution environment | The hook uses `env --ignore-environment --chdir=/` to run the executables in an environment as clean as possible, but there is no way to change its behavior. | The tester executes in an identical environment of a consumer, independent from the building environment. | +| Rebuild after modification | The hook rebuilds the package when the hook is modified, since it is a phase running during the build time of the package. | The tester does not require rebuilding the package, since it is a derivation. | +| Overhead during package building | Negligible. Although running during the build time of the package, the hook is lean and executes few commands. | Zero, since the tester is a derivation that runs after package building. | +| Content-addressed-derivation awareness | Since it runs during the building, the hook does not deal with failures happening after building, like rewritings that happen post installation. | Since it runs after the building, the tester detects failures at this time. | +| Execution by OfBorg CI tool | Yes, since the hook is a phase running during the build time of the package. | OfBorg does run the tester for _directly affected packages only_; transitive dependencies are ignored by default, requiring extra commands like `@ofborg build dependency1.tests dependency2.tests ...`. | +| Execution by `nixpkgs-review` tool | Yes, since the hook is a phase running during the build time of the package. | No, since the tool has no support for executing `passthru.tests`. [^1] | +| Package breakage awareness | Loud and clear as soon as the hook is reached, since it is a phase running during the build time of the package. | Requires specific command to be noticed, e.g. `nix-build -A pkg.tests.version`, since it is a derivation dependent on the package | +| Transitive package breakage | Never ever goes unnoticed by `nixpkgs-review`, since the tool executes the transitive dependencies. | Since human beings are prone to forget the duty of running passthru tests and `nixpkgs-review` has no support for running it, it certainly goes unnoticed. | + + +[^1]: There is a [pull request proposal](https://github.com/Mic92/nixpkgs-review/pull/397) against [`nixpkgs-review`](https://github.com/Mic92/nixpkgs-review) for an extra flag to run `passthru.tests`. + +As said before, given the current limitations of Nixpkgs's tooling and CI demonstrated above, `versionCheckHook` fits better in most of typical situations.