From 62719bc8083aad2fa6d6363f75638ac30cf30c13 Mon Sep 17 00:00:00 2001 From: Yueh-Shun Li Date: Sun, 28 May 2023 11:26:14 +0000 Subject: [PATCH 1/3] lib.extendMkDerivation, lib.adaptMkDerivation: init Add functions to lib.customisation: - extendMkDerivation - adaptMkDerivation Co-authored-by: Robert Hensing Co-authored-by: Valentin Gagarin Co-authored-by: Lin Jian --- lib/customisation.nix | 193 +++++++++++++++++++++++++++++++++++++++ lib/default.nix | 4 +- pkgs/test/overriding.nix | 105 +++++++++++++++++++++ 3 files changed, 301 insertions(+), 1 deletion(-) diff --git a/lib/customisation.nix b/lib/customisation.nix index bcdc94f3c4c30..317fe754886f3 100644 --- a/lib/customisation.nix +++ b/lib/customisation.nix @@ -8,6 +8,7 @@ let optionalAttrs attrNames filter elemAt concatStringsSep sortOn take length filterAttrs optionalString flip pathIsDirectory head pipe isDerivation listToAttrs mapAttrs seq flatten deepSeq warnIf isInOldestRelease extends + toFunction id ; inherit (lib.strings) levenshtein levenshteinAtMost; @@ -661,4 +662,196 @@ rec { }; in self; + /** + Define a `mkDerivation`-like function based on another `mkDerivation`-like function. + + [`stdenv.mkDerivation`](#part-stdenv) gives access to + its final set of derivation attributes when it is passed a function, + or when it is passed an overlay-style function in `overrideAttrs`. + + Instead of composing new `stdenv.mkDerivation`-like build helpers + using normal function composition, + `extendMkDerivation` makes sure that the returned build helper + supports such first class recursion like `mkDerivation` does. + + `extendMkDerivation` takes an extra attribute set to confgure its behaviour. + One can optionally specify *modify* to modify the result derivation, + or `inheritFunctionArgs` to decide + whether to inherit the `__functionArgs` from the base build helper. + + # Inputs + + `extendMkDerivation`-specific configurations + : `inheritFunctionArgs`: Whether to inherit `__functionArgs` from the base build helper (default to `true`) + : `modify`: Function to modify the result derivation (default to `lib.id`) + + `mkDerivationBase` + : Base build helper, the `mkDerivation`-like build helper to extend + + `attrsOverlay` + : An overlay of attribute set, like the one taken by [overrideAttrs](#sec-pkg-overrideAttrs). + : It is the implementation detail of the result build helper. + + # Type + + ``` + extendMkDerivation :: + { + inheritFunctionArgs :: Bool, + modify :: a -> a, + } + -> ((FixedPointArgs | AttrSet) -> a) + -> (AttrSet -> AttrSet -> AttrSet) + -> (FixedPointArgs | AttrSet) -> a + + FixedPointArgs = AttrSet -> AttrSet + a = Derivation when defining a build helper + ``` + + # Examples + + :::{.example} + ## `lib.customisation.extendMkDerivation` usage example + ```nix-shell + mkLocalDerivation = lib.extendMkDerivation { } pkgs.stdenv.mkDerivation (finalAttrs: + args@{ preferLocalBuild ? true, allowSubstitute ? false, ... }: + { inherit preferLocalBuild allowSubstitute; }) + + mkLocalDerivation.__functionArgs + => { allowSubstitute = true; preferLocalBuild = true; } + + mkLocalDerivation { inherit (pkgs.hello) pname version src; } + => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv» + + mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; }) + => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv» + + (mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; passthru = { foo = "a"; bar = "${finalAttrs.passthru.foo}b"; } })).bar + => "ab" + ``` + ::: + + :::{.note} + If *modify* is specified, + it should take care of existing attributes that perform overriding + (e.g., [`overrideAttrs`](#sec-pkg-overrideAttrs)) + to ensure that the overriding functionality of the result derivation + work as expected. + Modifications that breaks the overriding include + direct [attribute set update](https://nixos.org/manual/nix/stable/language/operators#update) + and [`lib.extendDerivation`](#function-library-lib.customisation.extendDerivation). + ::: + */ + extendMkDerivation = + { + modify ? id, + inheritFunctionArgs ? true, + }: + mkDerivationBase: attrsOverlay: + setFunctionArgs + # Adds the fixed-point style support. + (fpargs: modify ((mkDerivationBase fpargs).overrideAttrs attrsOverlay)) + # Add __functionArgs + ( + # Inherit the __functionArgs from the base build helper + optionalAttrs inheritFunctionArgs (functionArgs mkDerivationBase) + # Recover the __functionArgs from the derived build helper + // functionArgs (attrsOverlay { }) + ) + // { + inherit + # Expose to the result build helper. + attrsOverlay + mkDerivationBase + modify; + }; + + /** + Like [`extendMkDerivation`](#function-library-lib.customisation.extendMkDerivation), + but accept an argument set adapter instead of an attribute overlay. + + The argument set adapter is a function in the form `finalAttrs: args: { ... }` + that returns the fixed attribute set to pass to the base build helper, + instead of a subet of attributes to update. + This allows removing arguments that we don't want to pass to the base build helper. + + In case the `args` [set pattern](https://nix.dev/manual/nix/stable/language/constructs#functions) + doesn't have an ellipsis (`{ , ... }@args:`), set `inheritFunctionArgs` as `false`. + + # Inputs + + `adaptMkDerivation`-specific configurations + : `inheritFunctionArgs`: Whether to inherit `__functionArgs` from the base build helper (default to `true`) + : `modify`: Function to modify the result derivation (default to `lib.id`) + + `mkDerivationBase` + : Base build helper, the `mkDerivation`-like build helper to extend. + + `adaptArgs` + : Argument set adapter, a function in the form `finalAttrs: args: { ... }`, transforming the taken argument set before passing down the base build helper. + : It is as the implementation detail of the result build helper. + + # Type + + ``` + adaptMkDerivation :: + { + inheritFunctionArgs :: Bool, + modify :: a -> a, + } + -> ((FixedPointArgs | AttrSet) -> a) + -> (AttrSet -> AttrSet -> AttrSet) + -> (FixedPointArgs | AttrSet) -> a + + FixedPointArgs = AttrSet -> AttrSet + a = Derivation when defining a build helper + ``` + + # Examples + + :::{.example} + ## `lib.customisation.adaptMkDerivation` usage example + + ```nix-repl + mkLocalDerivation = lib.adaptMkDerivation { } pkgs.stdenv.mkDerivation (finalAttrs: + args@{ preferLocalBuild ? true, allowSubstitute ? false, specialArg ? (_: false), ... }: + removeAttrs args [ "specialArg" ] // { inherit preferLocalBuild allowSubstitute; }) + + mkLocalDerivation.__functionArgs + => { allowSubstitute = true; specialArg = true; preferLocalBuild = true; } + + mkLocalDerivation { inherit (pkgs.hello) pname version src; specialArg = _: false; } + => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv» + + mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; specialArg = _: false; }) + => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv» + + (mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; passthru = { foo = "a"; bar = "${finalAttrs.passthru.foo}b"; } })).bar + => "ab" + ``` + ::: + */ + adaptMkDerivation = + { + modify ? id, + inheritFunctionArgs ? true, + }: + mkDerivationBase: adaptArgs: + setFunctionArgs + # Adds the fixed-point style support + (fpargs: modify (mkDerivationBase (finalAttrs: adaptArgs finalAttrs (toFunction fpargs finalAttrs)))) + # Add __functionArgs + ( + # Inherit the __functionArgs from the base build helper + optionalAttrs inheritFunctionArgs (functionArgs mkDerivationBase) + # Recover the __functionArgs from the derived build helper + // functionArgs (adaptArgs { }) + ) + // { + inherit + # Expose to the result build helper. + adaptArgs + mkDerivationBase + modify; + }; } diff --git a/lib/default.nix b/lib/default.nix index aff1df150f13b..43c20c8673a14 100644 --- a/lib/default.nix +++ b/lib/default.nix @@ -120,7 +120,9 @@ let noDepEntry fullDepEntry packEntry stringAfter; inherit (self.customisation) overrideDerivation makeOverridable callPackageWith callPackagesWith extendDerivation hydraJob - makeScope makeScopeWithSplicing makeScopeWithSplicing'; + makeScope makeScopeWithSplicing makeScopeWithSplicing' + extendMkDerivation extendMkDerivationModified adaptMkDerivation adaptMkDerivationModified + ; inherit (self.derivations) lazyDerivation optionalDrvAttr; inherit (self.meta) addMetaAttrs dontDistribute setName updateName appendToName mapDerivationAttrset setPrio lowPrio lowPrioSet hiPrio diff --git a/pkgs/test/overriding.nix b/pkgs/test/overriding.nix index a9fa482e4e588..fed545a61c951 100644 --- a/pkgs/test/overriding.nix +++ b/pkgs/test/overriding.nix @@ -59,6 +59,44 @@ let expr = lib.isString pet-vendored.drvPath; expected = true; }; + + ## Tests for lib.extendMkDerivation and lib.adaptMkDerivation + extendMkDerivation-helloLocal-imp-arguments = { + expr = helloLocal.preferLocalBuild; + expected = true; + }; + adaptMkDerivation-helloLocal-imp-arguments = { + expr = helloLocalWithSpecialArg.preferLocalBuild; + expected = true; + }; + extendMkDerivation-helloLocal-plain-equivalence = { + expr = helloLocal.drvPath == helloLocalPlain.drvPath; + expected = true; + }; + adaptMkDerivation-helloLocal-plain-equivalence = { + expr = helloLocalWithSpecialArg.drvPath == helloLocalPlainWithSpecialArg.drvPath; + expected = true; + }; + extendMkDerivation-helloLocal-finalAttrs = { + expr = helloLocal.bar == "ab"; + expected = true; + }; + adaptMkDerivation-helloLocal-finalAttrs = { + expr = helloLocalWithSpecialArg.bar == "ab"; + expected = true; + }; + extendMkDerivation-helloLocal-finalPackage = { + expr = lib.stringLength helloLocal.passthru.tests.run.outPath > 0; + expected = true; + }; + adaptMkDerivation-helloLocal-finalPackage = { + expr = lib.stringLength helloLocalWithSpecialArg.passthru.tests.run.outPath > 0; + expected = true; + }; + extendMkDerivation-adaptMkDerivation-drv-equivalence = { + expr = helloLocal.drvPath == helloLocalWithSpecialArg.drvPath; + expected = true; + }; }; addEntangled = origOverrideAttrs: f: @@ -151,6 +189,73 @@ let ); pet-vendored = pet-foo.overrideAttrs { vendorHash = null; }; + + mkLocalDerivation = + lib.extendMkDerivation { } pkgs.stdenv.mkDerivation (finalAttrs: + { preferLocalBuild ? true + , allowSubstitute ? false + , ... + }@args: + { + inherit preferLocalBuild allowSubstitute; + }); + + mkLocalDerivationWithSpecialArg = + lib.adaptMkDerivation { } pkgs.stdenv.mkDerivation (finalAttrs: + { preferLocalBuild ? true + , allowSubstitute ? false + , specialArg ? (_: false) + , ... + }@args: + removeAttrs args [ "specialArg" ] // { + inherit preferLocalBuild allowSubstitute; + passthru = args.passthru or { } // { + greeting = if specialArg "Hi!" then "Hi!" else "Hello!"; + }; + }); + + helloLocalPlainAttrs = { + inherit (pkgs.hello) pname version src; + }; + + helloLocalPlainAttrsWithSpecialArg = helloLocalPlainAttrs // { + specialArg = throw "impassiblePredicate: not implemented"; + }; + + helloLocalPlain = mkLocalDerivation helloLocalPlainAttrs; + helloLocalPlainWithSpecialArg = mkLocalDerivationWithSpecialArg helloLocalPlainAttrsWithSpecialArg; + + helloLocal = mkLocalDerivation (finalAttrs: helloLocalPlainAttrs // { + passthru = pkgs.hello.passthru or { } // { + foo = "a"; + bar = "${finalAttrs.passthru.foo}b"; + tests = pkgs.hello.passthru.tests or { } // { + run = pkgs.runCommandLocal "test-hello-run" { + nativeBuildInputs = [ finalAttrs.finalPackage ]; + } '' + set -eu -o pipefail + RESULT="$(hello | tee "$out")" + [[ "$RESULT" == "Hello, world!" ]] + ''; + }; + }; + }); + + helloLocalWithSpecialArg = mkLocalDerivation (finalAttrs: helloLocalPlainAttrs // { + passthru = pkgs.hello.passthru or { } // { + foo = "a"; + bar = "${finalAttrs.passthru.foo}b"; + tests = pkgs.hello.passthru.tests or { } // { + run = pkgs.runCommandLocal "test-hello-run" { + nativeBuildInputs = [ finalAttrs.finalPackage ]; + } '' + set -eu -o pipefail + RESULT="$(hello | tee "$out")" + [[ "$RESULT" == "Hello, world!" ]] + ''; + }; + }; + }); in stdenvNoCC.mkDerivation { From 14cb730dbdb4c97b1870123d7161b07b71ef9b54 Mon Sep 17 00:00:00 2001 From: Yueh-Shun Li Date: Sun, 28 May 2023 21:55:43 +0800 Subject: [PATCH 2/3] doc: add chapter Fixed-point arguments of build helpers Add "Fixed-point arguments of build helpers" chapter in "Builde helpers" part. Co-authored-by: nicoo Co-authored-by: Silvan Mosberger Co-authored-by: Valentin Gagarin Co-authored-by: Lin Jian --- doc/build-helpers.md | 1 + .../fixed-point-arguments.chapter.md | 185 ++++++++++++++++++ 2 files changed, 186 insertions(+) create mode 100644 doc/build-helpers/fixed-point-arguments.chapter.md diff --git a/doc/build-helpers.md b/doc/build-helpers.md index 010665484cfda..a9df144bacbf1 100644 --- a/doc/build-helpers.md +++ b/doc/build-helpers.md @@ -17,6 +17,7 @@ There is no uniform interface for build helpers. [Language- or framework-specific build helpers](#chap-language-support) usually follow the style of `stdenv.mkDerivation`, which accepts an attribute set or a fixed-point function taking an attribute set. ```{=include=} chapters +build-helpers/fixed-point-arguments.chapter.md build-helpers/fetchers.chapter.md build-helpers/trivial-build-helpers.chapter.md build-helpers/testers.chapter.md diff --git a/doc/build-helpers/fixed-point-arguments.chapter.md b/doc/build-helpers/fixed-point-arguments.chapter.md new file mode 100644 index 0000000000000..9f2266737fad3 --- /dev/null +++ b/doc/build-helpers/fixed-point-arguments.chapter.md @@ -0,0 +1,185 @@ +# Fixed-point arguments of build helpers {#chap-build-helpers-finalAttrs} + +As mentioned in the beginning of this part, `stdenv.mkDerivation` could alternatively accept a fixed-point function. The input of such function, typically named `finalAttrs`, is expected to be the final state of the attribute set. +A build helper like this is said to accept **fixed-point arguments**. + +Build helpers don't always support fixed-point arguments yet, as support in [`stdenv.mkDerivation`](#mkderivation-recursive-attributes) was first included in Nixpkgs 22.05. + +## Defining a build helper with `lib.extendMkDerivation` {#sec-build-helper-extendMkDerivation} + +Developers can use the Nixpkgs library function [`lib.customisation.extendMkDerivation`](#function-library-lib.customisation.extendMkDerivation) to define a build helper supporting fixed-point arguments from an existing one with such support, with an attribute overlay similar to the one taken by [`.overrideAttrs`](#sec-pkg-overrideAttrs). + +:::{.example #ex-build-helpers-mkLocalDerivation-extendMkDerivation} + +# Example definition of `mkLocalDerivation` extended from `stdenv.mkDerivation` with `lib.extendMkDerivation` + +We want to define a build helper named `mkLocalDerivation` that builds locally without using substitutes by default. + +Instead of taking a plain attribute set, + +```nix +{ + preferLocalBuild ? true, + allowSubstitute ? false, + ... +}@args: + +stdenv.mkDerivation ( + args + // { + # Attributes to update + inherit preferLocalBuild allowSubstitute; + } +) +``` + +we could define with `lib.extendMkDerivation` an attribute overlay to make the result build helper also accepts the the attribute set's fixed point passing to the underlying `stdenv.mkDerivation`, named `finalAttrs` here: + +```nix +lib.extendMkDerivation { } stdenv.mkDerivation ( + finalAttrs: + { + preferLocalBuild ? true, + allowSubstitute ? false, + ... + }@args: + + # No need of `args //` here. + # The whole set of input arguments is passed in advance. + { + # Attributes to update + inherit preferLocalBuild allowSubstitute; + } +) +``` +::: + +Should there be a need to modify the result derivation, pass the derivation modifying function to `lib.extendMkDerivation` as `lib.customisation.extendMkDerivation { modify = drv: ...; }`. + +## Wrapping existing build helper definition with `lib.adaptMkDerivation` {#sec-build-helper-adaptMkDerivation} + +Many existing build helpers only pass part of their arguments down to their base build helper, leading to the use of custom overriders (such as `overridePythonPackage`) and extra complexity when overriding. + +As a wrapper around [`overrideAttrs`](#sec-pkg-overrideAttrs), `lib.extendMkDerivation` passes the whole set of arguments directly to the base build helper before extending them, making it incompatible with build helpers that only pass part of their input arguments down the base build helper. + +To workaround this issue, [`lib.customisation.adaptMkDerivation`](#function-library-lib.customisation.adaptMkDerivation) is introduced. Instead of taking an attribute overlay that returns a subset of attributes to update, it takes an argument set adapter that returns the whole set of arguments to pass to the base build helper, allowing the removal of some arguments. The expression change needed to adopt `lib.adaptMkDerivation` is also smaller, enabling a smooth transition toward fixed-point arguments. + +:::{.example #ex-build-helpers-mkLocalDerivation-adaptMkDerivation} + +# Example definition of `mkLocalDerivation` extended from `stdenv.mkDerivation` with `lib.adaptMkDerivation` + +Should the original definition of build helper `mkLocalDerivation` take an argument `specialArg` that cannot be passed to `sdenv.mkDerivation`, + +```nix +{ + preferLocalBuild ? true, + allowSubstitute ? false, + specialArg ? (_: false), + ... +}@args: + +stdenv.mkDerivation ( + removeAttrs [ + # Don't pass specialArg into mkDerivation. + "specialArg" + ] args + // { + # Arguments to pass + inherit preferLocalBuild allowSubstitute; + # Some expressions involving specialArg + greeting = if specialArg "hi" then "hi" else "hello"; + } +) +``` + +wrap around the original definition with `lib.adaptMkDerivation` to make the result build helper accept fixed-point arguments. + +```nix +lib.adaptMkDerivation { } stdenv.mkDerivation ( + finalAttrs: + { + preferLocalBuild ? true, + allowSubstitute ? false, + specialArg ? (_: false), + ... + }@args: + + removeAttrs [ + # Don't pass specialArg into mkDerivation. + "specialArg" + ] args + // { + # Arguments to pass + inherit preferLocalBuild allowSubstitute; + # Some expressions involving specialArg + greeting = if specialArg "hi" then "hi" else "hello"; + } +) +``` +::: + +In the long run, we would like to refactor build helpers to pass every argument down to `stdenv.mkDerivation`, so that they can all be overridden by [`overrideAttrs`](#sec-pkg-overrideAttrs), eliminating the use of custom overriders (e.g., `overridePythonAttrs`). + +The following example shows a smooth migration from `lib.adaptMkDerivation` to `lib.extendMkDerivation`: + +:::{.example #ex-build-helpers-mkLocalDerivation-migration} + +# Migrating `mkLocalDerivation` from `lib.adaptMkDerivation` to `lib.extendMkDerivation` + +Refactor the definition to pass `specialArg` properly while keeping some backward compatibility. + +```nix +lib.adaptMkDerivation { } stdenv.mkDerivation ( + finalAttrs: + { + preferLocalBuild ? true, + allowSubstitute ? false, + specialArg ? (_: false), + ... + }@args: + + # The arguments to pass + { + inherit preferLocalBuild allowSubstitute; + passthru = { + specialArg = + if (args ? specialArg) then + # For backward compatibility only. + # TODO: Convert to throw after XX.XX branch-off. + lib.warn "mkLocalDerivation: Expect specialArg under passthru." args.specialArg + else + (_: false); + } // args.passthru or { }; + # Some expressions involving specialArg + greeting = if finalAttrs.passthru.specialArg "hi" then "hi" else "hello"; + } + // removeAttrs [ "specialArg" ] args +) +``` + +Convert to `lib.extendMkDerivation` after deprecating `specialArg`. + +```nix +lib.extendMkDerivation { } stdenv.mkDerivation ( + finalAttrs: + + { + preferLocalBuild ? true, + allowSubstitute ? false, + ... + }@args: + + # The arguments to pass + { + inherit preferLocalBuild allowSubstitute; + passthru = { + specialArg = + (lib.throwIf (args ? specialArg) "mkLocalDerivation: Expect specialArg under passthru.") + (_: false); + } // args.passthru or { }; + # Some expressions involving specialArg + greeting = if finalAttrs.passthru.specialArg "hi" then "hi" else "hello"; + } +) +``` +::: From 506a1be1eff64c6f0b2e6a38cf3916f500e0d7e9 Mon Sep 17 00:00:00 2001 From: Yueh-Shun Li Date: Fri, 1 Dec 2023 16:32:48 +0800 Subject: [PATCH 3/3] lib.extendMkDerivation: add a release note entry --- nixos/doc/manual/release-notes/rl-2411.section.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/nixos/doc/manual/release-notes/rl-2411.section.md b/nixos/doc/manual/release-notes/rl-2411.section.md index dd01ea9c9c5e0..e340ca3c6a506 100644 --- a/nixos/doc/manual/release-notes/rl-2411.section.md +++ b/nixos/doc/manual/release-notes/rl-2411.section.md @@ -44,6 +44,12 @@ Users can use it by `services.displayManager.ly.enable` and config it by `services.displayManager.ly.settings` to generate `/etc/ly/config.ini` +- [`lib.customisation.extendMkDerivation`](https://nixos.org/manual/nixpkgs/unstable#function-library-lib.customisation.extendMkDerivation) and [`lib.customisation.adaptMkDerivation`](https://nixos.org/manual/nixpkgs/unstable#function-library-lib.customisation.adaptMkDerivation) are introduced to bring [fixed-point arguments](https://nixos.org/manual/nixpkgs/unstable#chap-build-helpers-finalAttrs) support (capability to take `(finalAttrs: { ... })`) to arguments aside from `stdenv.mkDerivation`. + + `lib.extendMkDerivation` helps define a new build helper supporting (`(finalAttrs: { ... })`) based on another build helper with such support, while `lib.adaptMkDerivation` helps existing build helpers with arguments unable to pass to the base build helper (usually `stdenv.mkDerivation`) to adopt fixed-point arguments support with slight expression changes. + + See [Nixpkgs Manual chapter *Fixed-point arguments of build helpers*](https://nixos.org/manual/nixpkgs/unstable#chap-build-helpers-finalAttrs) for details. + - `srcOnly` was rewritten to be more readable, have additional warnings in the event that something is probably wrong, use the `stdenv` provided by the derivation, and Noogle-compatible documentation was added. - The default sound server for most graphical sessions has been switched from PulseAudio to PipeWire.