diff --git a/doc/hooks/versionCheckHook.section.md b/doc/hooks/versionCheckHook.section.md index 03c3f8fdea3e1..e909539e301a8 100644 --- a/doc/hooks/versionCheckHook.section.md +++ b/doc/hooks/versionCheckHook.section.md @@ -70,19 +70,21 @@ Given the current limitations of Nixpkgs's tooling and CI, `versionCheckHook` fi Below we tabulate the differences between `versionCheckHook` and `testers.testVersion`. -| Item | `versionCheckHook` | `testers.testVersion` | -|:---------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| -| Customization | Besides the attributes described above, the hook provides no other methods for controlling it. | The tester has access to all feature set of Nix and Nixpkgs, like generating multiple version test derivations from a list of strings, and accessing programs / tools outside the inputs of the derivation. | -| Execution environment | The hook uses `env --ignore-environment --chdir=/` to run the executables in an environment as clean as possible, but there is no way to change its behavior. | The tester executes in an identical environment of a consumer, independent from the building environment. | -| Rebuild after modification | The hook rebuilds the package when the hook is modified, since it is a phase running during the build time of the package. | The tester does not require rebuilding the package, since it is a derivation. | -| Overhead during package building | Negligible. Although running during the build time of the package, the hook is lean and executes few commands. | Zero, since the tester is a derivation that runs after package building. | -| Content-addressed-derivation awareness | Since it runs during the building, the hook does not deal with failures happening after building, like rewritings that happen post installation. | Since it runs after the building, the tester detects failures at this time. | -| Execution by OfBorg CI tool | Yes, since the hook is a phase running during the build time of the package. | OfBorg does run the tester for _directly affected packages only_; transitive dependencies are ignored by default, requiring extra commands like `@ofborg build dependency1.tests dependency2.tests ...`. | -| Execution by `nixpkgs-review` tool | Yes, since the hook is a phase running during the build time of the package. | No, since the tool has no support for executing `passthru.tests`. [^1] | -| Package breakage awareness | Loud and clear as soon as the hook is reached, since it is a phase running during the build time of the package. | Requires specific command to be noticed, e.g. `nix-build -A pkg.tests.version`, since it is a derivation dependent on the package | -| Transitive package breakage | Never ever goes unnoticed by `nixpkgs-review`, since the tool executes the transitive dependencies. | Since human beings are prone to forget the duty of running passthru tests and `nixpkgs-review` has no support for running it, it certainly goes unnoticed. | - - -[^1]: There is a [pull request proposal](https://github.com/Mic92/nixpkgs-review/pull/397) against [`nixpkgs-review`](https://github.com/Mic92/nixpkgs-review) for an extra flag to run `passthru.tests`. +|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| +| Item | `versionCheckHook` | `testers.testVersion` via `passthru.tests.version` | +|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| +| Customization | Besides the attributes described above, `versionCheckHook` provides no other methods for controlling it. | `tests.version` has access to the whole feature set of Nixpkgs, like generating multiple test derivations and accessing tools outside the package inputs. | +| Execution environment | `versionCheckHook` uses `env --ignore-environment --chdir=/` to run the executables in an environment as clean as possible, but there is no way to change its behavior. | `tests.version` executes in an environment identical to that of a consumer, independent from the building environment. | +| Rebuild after modification | Modifying `versionCheckHook` triggers the package rebuild, since it is a phase executed during the the package build time. | `tests.version` does not require rebuilding the package, since it is a derivation that runs after package build time. | +| Overhead during package build time | Negligible. Although executed during the the package build time, `versionCheckHook` is lean and executes few commands. | Zero, since `tests.version` is a derivation that runs after package build time. | +| Failure detection time | `versionCheckHook` runs during the package build time, therefore it does not deal with failures happening after that. | `tests.version` detects failures after package build time, like rewritings and relocations. | +| OfBorg CI support | `versionCheckHook` will be executed by OfBorg, since it is a phase executed during the the package build time. | OfBorg supports automatic execution of `passthru.tests` for _directly affected packages_; transitive dependencies are ignored by default. | +| `nixpkgs-review` support | `versionCheckHook` will be executed by `nixpkgs-review`, since it is a phase executed during the the package build time. | `nixpkgs-review` has no support for executing `passthru.tests` at all, whether for the package or its transitive dependencies. [^1] | +| Package breakage awareness | Loud and clear as soon as `versionCheckHook` is reached during the package build time. | Requires specific commands to be noticed, e.g. `nix-build -A pkg.tests.version`. | +| Transitive dependencies awareness | Since `nixpkgs-review` executes the transitive dependencies, the ones containing `versionCheckHook` will never ever be ignored by this tool. | Since `nixpkgs-review` does not run `tests.version` and ofBorg does not execute transitive dependencies, extra measures are needed to find them. | +|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| + + +[^1]: There is a [pull request proposal](https://github.com/Mic92/nixpkgs-review/pull/397) against [`nixpkgs-review`](https://github.com/Mic92/nixpkgs-review) for adding support to `passthru.tests` execution. As said before, given the current limitations of Nixpkgs's tooling and CI demonstrated above, `versionCheckHook` fits better in most of typical situations.