lib.extendMkDerivation, lib.adaptMkDerivation: init

doc/build-helpers.md

@@ -17,6 +17,7 @@ There is no uniform interface for build helpers.
 [Language- or framework-specific build helpers](#chap-language-support) usually follow the style of `stdenv.mkDerivation`, which accepts an attribute set or a fixed-point function taking an attribute set.
 
 ```{=include=} chapters
+build-helpers/fixed-point-arguments.chapter.md
 build-helpers/fetchers.chapter.md
 build-helpers/trivial-build-helpers.chapter.md
 build-helpers/testers.chapter.md

doc/build-helpers/fixed-point-arguments.chapter.md

@@ -0,0 +1,185 @@
+# Fixed-point arguments of build helpers {#chap-build-helpers-finalAttrs}
+
+As mentioned in the beginning of this part, `stdenv.mkDerivation` could alternatively accept a fixed-point function. The input of such function, typically named `finalAttrs`, is expected to be the final state of the attribute set.
+A build helper like this is said to accept **fixed-point arguments**.
+
+Build helpers don't always support fixed-point arguments yet, as support in [`stdenv.mkDerivation`](#mkderivation-recursive-attributes) was first included in Nixpkgs 22.05.
+
+## Defining a build helper with `lib.extendMkDerivation` {#sec-build-helper-extendMkDerivation}
+
+Developers can use the Nixpkgs library function [`lib.customisation.extendMkDerivation`](#function-library-lib.customisation.extendMkDerivation) to define a build helper supporting fixed-point arguments from an existing one with such support, with an attribute overlay similar to the one taken by [`<pkg>.overrideAttrs`](#sec-pkg-overrideAttrs).
+
+:::{.example #ex-build-helpers-mkLocalDerivation-extendMkDerivation}
+
+# Example definition of `mkLocalDerivation` extended from `stdenv.mkDerivation` with `lib.extendMkDerivation`
+
+We want to define a build helper named `mkLocalDerivation` that builds locally without using substitutes by default.
+
+Instead of taking a plain attribute set,
+
+```nix
+{
+  preferLocalBuild ? true,
+  allowSubstitute ? false,
+  ...
+}@args:
+
+stdenv.mkDerivation (
+  args
+  // {
+    # Attributes to update
+    inherit preferLocalBuild allowSubstitute;
+  }
+)
+```
+
+we could define with `lib.extendMkDerivation` an attribute overlay to make the result build helper also accepts the the attribute set's fixed point passing to the underlying `stdenv.mkDerivation`, named `finalAttrs` here:
+
+```nix
+lib.extendMkDerivation { } stdenv.mkDerivation (
+  finalAttrs:
+  {
+    preferLocalBuild ? true,
+    allowSubstitute ? false,
+    ...
+  }@args:
+
+  # No need of `args //` here.
+  # The whole set of input arguments is passed in advance.
+  {
+    # Attributes to update
+    inherit preferLocalBuild allowSubstitute;
+  }
+)
+```
+:::
+
+Should there be a need to modify the result derivation, pass the derivation modifying function to `lib.extendMkDerivation` as `lib.customisation.extendMkDerivation { modify = drv: ...; }`.
+
+## Wrapping existing build helper definition with `lib.adaptMkDerivation` {#sec-build-helper-adaptMkDerivation}
+
+Many existing build helpers only pass part of their arguments down to their base build helper, leading to the use of custom overriders (such as `overridePythonPackage`) and extra complexity when overriding.
+
+As a wrapper around [`overrideAttrs`](#sec-pkg-overrideAttrs), `lib.extendMkDerivation` passes the whole set of arguments directly to the base build helper before extending them, making it incompatible with build helpers that only pass part of their input arguments down the base build helper.
+
+To workaround this issue, [`lib.customisation.adaptMkDerivation`](#function-library-lib.customisation.adaptMkDerivation) is introduced. Instead of taking an attribute overlay that returns a subset of attributes to update, it takes an argument set adapter that returns the whole set of arguments to pass to the base build helper, allowing the removal of some arguments. The expression change needed to adopt `lib.adaptMkDerivation` is also smaller, enabling a smooth transition toward fixed-point arguments.
+
+:::{.example #ex-build-helpers-mkLocalDerivation-adaptMkDerivation}
+
+# Example definition of `mkLocalDerivation` extended from `stdenv.mkDerivation` with `lib.adaptMkDerivation`
+
+Should the original definition of build helper `mkLocalDerivation` take an argument `specialArg` that cannot be passed to `sdenv.mkDerivation`,
+
+```nix
+{
+  preferLocalBuild ? true,
+  allowSubstitute ? false,
+  specialArg ? (_: false),
+  ...
+}@args:
+
+stdenv.mkDerivation (
+  removeAttrs [
+    # Don't pass specialArg into mkDerivation.
+    "specialArg"
+  ] args
+  // {
+    # Arguments to pass
+    inherit preferLocalBuild allowSubstitute;
+    # Some expressions involving specialArg
+    greeting = if specialArg "hi" then "hi" else "hello";
+  }
+)
+```
+
+wrap around the original definition with `lib.adaptMkDerivation` to make the result build helper accept fixed-point arguments.
+
+```nix
+lib.adaptMkDerivation { } stdenv.mkDerivation (
+  finalAttrs:
+  {
+    preferLocalBuild ? true,
+    allowSubstitute ? false,
+    specialArg ? (_: false),
+    ...
+  }@args:
+
+  removeAttrs [
+    # Don't pass specialArg into mkDerivation.
+    "specialArg"
+  ] args
+  // {
+    # Arguments to pass
+    inherit preferLocalBuild allowSubstitute;
+    # Some expressions involving specialArg
+    greeting = if specialArg "hi" then "hi" else "hello";
+  }
+)
+```
+:::
+
+In the long run, we would like to refactor build helpers to pass every argument down to `stdenv.mkDerivation`, so that they can all be overridden by [`overrideAttrs`](#sec-pkg-overrideAttrs), eliminating the use of custom overriders (e.g., `overridePythonAttrs`).
+
+The following example shows a smooth migration from `lib.adaptMkDerivation` to `lib.extendMkDerivation`:
+
+:::{.example #ex-build-helpers-mkLocalDerivation-migration}
+
+# Migrating `mkLocalDerivation` from `lib.adaptMkDerivation` to `lib.extendMkDerivation`
+
+Refactor the definition to pass `specialArg` properly while keeping some backward compatibility.
+
+```nix
+lib.adaptMkDerivation { } stdenv.mkDerivation (
+  finalAttrs:
+  {
+    preferLocalBuild ? true,
+    allowSubstitute ? false,
+    specialArg ? (_: false),
+    ...
+  }@args:
+
+  # The arguments to pass
+  {
+    inherit preferLocalBuild allowSubstitute;
+    passthru = {
+      specialArg =
+        if (args ? specialArg) then
+          # For backward compatibility only.
+          # TODO: Convert to throw after XX.XX branch-off.
+          lib.warn "mkLocalDerivation: Expect specialArg under passthru." args.specialArg
+        else
+          (_: false);
+    } // args.passthru or { };
+    # Some expressions involving specialArg
+    greeting = if finalAttrs.passthru.specialArg "hi" then "hi" else "hello";
+  }
+  // removeAttrs [ "specialArg" ] args
+)
+```
+
+Convert to `lib.extendMkDerivation` after deprecating `specialArg`.
+
+```nix
+lib.extendMkDerivation { } stdenv.mkDerivation (
+  finalAttrs:
+
+  {
+    preferLocalBuild ? true,
+    allowSubstitute ? false,
+    ...
+  }@args:
+
+  # The arguments to pass
+  {
+    inherit preferLocalBuild allowSubstitute;
+    passthru = {
+      specialArg =
+        (lib.throwIf (args ? specialArg) "mkLocalDerivation: Expect specialArg under passthru.")
+          (_: false);
+    } // args.passthru or { };
+    # Some expressions involving specialArg
+    greeting = if finalAttrs.passthru.specialArg "hi" then "hi" else "hello";
+  }
+)
+```
+:::

lib/customisation.nix

@@ -8,6 +8,7 @@ let
     optionalAttrs attrNames filter elemAt concatStringsSep sortOn take length
     filterAttrs optionalString flip pathIsDirectory head pipe isDerivation listToAttrs
     mapAttrs seq flatten deepSeq warnIf isInOldestRelease extends
+    toFunction id
     ;
   inherit (lib.strings) levenshtein levenshteinAtMost;
 
@@ -661,4 +662,196 @@ rec {
       };
     in self;
 
+  /**
+    Define a `mkDerivation`-like function based on another `mkDerivation`-like function.
+
+    [`stdenv.mkDerivation`](#part-stdenv) gives access to
+    its final set of derivation attributes when it is passed a function,
+    or when it is passed an overlay-style function in `overrideAttrs`.
+
+    Instead of composing new `stdenv.mkDerivation`-like build helpers
+    using normal function composition,
+    `extendMkDerivation` makes sure that the returned build helper
+    supports such first class recursion like `mkDerivation` does.
+
+    `extendMkDerivation` takes an extra attribute set to confgure its behaviour.
+    One can optionally specify *modify* to modify the result derivation,
+    or `inheritFunctionArgs` to decide
+    whether to inherit the `__functionArgs` from the base build helper.
+
+    # Inputs
+
+    `extendMkDerivation`-specific configurations
+    : `inheritFunctionArgs`: Whether to inherit `__functionArgs` from the base build helper (default to `true`)
+    : `modify`: Function to modify the result derivation (default to `lib.id`)
+
+    `mkDerivationBase`
+    : Base build helper, the `mkDerivation`-like build helper to extend
+
+    `attrsOverlay`
+    : An overlay of attribute set, like the one taken by [overrideAttrs](#sec-pkg-overrideAttrs).
+    : It is the implementation detail of the result build helper.
+
+    # Type
+
+    ```
+    extendMkDerivation ::
+      {
+        inheritFunctionArgs :: Bool,
+        modify :: a -> a,
+      }
+      -> ((FixedPointArgs | AttrSet) -> a)
+      -> (AttrSet -> AttrSet -> AttrSet)
+      -> (FixedPointArgs | AttrSet) -> a
+
+    FixedPointArgs = AttrSet -> AttrSet
+    a = Derivation when defining a build helper
+    ```
+
+    # Examples
+
+    :::{.example}
+    ## `lib.customisation.extendMkDerivation` usage example
+    ```nix-shell
+    mkLocalDerivation = lib.extendMkDerivation { } pkgs.stdenv.mkDerivation (finalAttrs:
+      args@{ preferLocalBuild ? true, allowSubstitute ? false, ... }:
+      { inherit preferLocalBuild allowSubstitute; })
+
+    mkLocalDerivation.__functionArgs
+    => { allowSubstitute = true; preferLocalBuild = true; }
+
+    mkLocalDerivation { inherit (pkgs.hello) pname version src; }
+    => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv»
+
+    mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; })
+    => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv»
+
+    (mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; passthru = { foo = "a"; bar = "${finalAttrs.passthru.foo}b"; } })).bar
+    => "ab"
+    ```
+    :::
+
+    :::{.note}
+    If *modify* is specified,
+    it should take care of existing attributes that perform overriding
+    (e.g., [`overrideAttrs`](#sec-pkg-overrideAttrs))
+    to ensure that the overriding functionality of the result derivation
+    work as expected.
+    Modifications that breaks the overriding include
+    direct [attribute set update](https://nixos.org/manual/nix/stable/language/operators#update)
+    and [`lib.extendDerivation`](#function-library-lib.customisation.extendDerivation).
+    :::
+  */
+  extendMkDerivation =
+    {
+      modify ? id,
+      inheritFunctionArgs ? true,
+    }:
+    mkDerivationBase: attrsOverlay:
+    setFunctionArgs
+      # Adds the fixed-point style support.
+      (fpargs: modify ((mkDerivationBase fpargs).overrideAttrs attrsOverlay))
+      # Add __functionArgs
+      (
+        # Inherit the __functionArgs from the base build helper
+        optionalAttrs inheritFunctionArgs (functionArgs mkDerivationBase)
+        # Recover the __functionArgs from the derived build helper
+        // functionArgs (attrsOverlay { })
+      )
+    // {
+      inherit
+        # Expose to the result build helper.
+        attrsOverlay
+        mkDerivationBase
+        modify;
+    };
+
+  /**
+    Like [`extendMkDerivation`](#function-library-lib.customisation.extendMkDerivation),
+    but accept an argument set adapter instead of an attribute overlay.
+
+    The argument set adapter is a function in the form `finalAttrs: args: { ... }`
+    that returns the fixed attribute set to pass to the base build helper,
+    instead of a subet of attributes to update.
+    This allows removing arguments that we don't want to pass to the base build helper.
+
+    In case the `args` [set pattern](https://nix.dev/manual/nix/stable/language/constructs#functions)
+    doesn't have an ellipsis (`{ , ... }@args:`), set `inheritFunctionArgs` as `false`.
+
+    # Inputs
+
+    `adaptMkDerivation`-specific configurations
+    : `inheritFunctionArgs`: Whether to inherit `__functionArgs` from the base build helper (default to `true`)
+    : `modify`: Function to modify the result derivation (default to `lib.id`)
+
+    `mkDerivationBase`
+    : Base build helper, the `mkDerivation`-like build helper to extend.
+
+    `adaptArgs`
+    : Argument set adapter, a function in the form `finalAttrs: args: { ... }`, transforming the taken argument set before passing down the base build helper.
+    : It is as the implementation detail of the result build helper.
+
+    # Type
+
+    ```
+    adaptMkDerivation ::
+      {
+        inheritFunctionArgs :: Bool,
+        modify :: a -> a,
+      }
+      -> ((FixedPointArgs | AttrSet) -> a)
+      -> (AttrSet -> AttrSet -> AttrSet)
+      -> (FixedPointArgs | AttrSet) -> a
+
+    FixedPointArgs = AttrSet -> AttrSet
+    a = Derivation when defining a build helper
+    ```
+
+    # Examples
+
+    :::{.example}
+    ## `lib.customisation.adaptMkDerivation` usage example
+
+    ```nix-repl
+    mkLocalDerivation = lib.adaptMkDerivation { } pkgs.stdenv.mkDerivation (finalAttrs:
+      args@{ preferLocalBuild ? true, allowSubstitute ? false, specialArg ? (_: false), ... }:
+      removeAttrs args [ "specialArg" ] // { inherit preferLocalBuild allowSubstitute; })
+
+    mkLocalDerivation.__functionArgs
+    => { allowSubstitute = true; specialArg = true; preferLocalBuild = true; }
+
+    mkLocalDerivation { inherit (pkgs.hello) pname version src; specialArg = _: false; }
+    => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv»
+
+    mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; specialArg = _: false; })
+    => «derivation /nix/store/xirl67m60ahg6jmzicx43a81g635g8z8-hello-2.12.1.drv»
+
+    (mkLocalDerivation (finalAttrs: { inherit (pkgs.hello) pname version src; passthru = { foo = "a"; bar = "${finalAttrs.passthru.foo}b"; } })).bar
+    => "ab"
+    ```
+    :::
+  */
+  adaptMkDerivation =
+    {
+      modify ? id,
+      inheritFunctionArgs ? true,
+    }:
+    mkDerivationBase: adaptArgs:
+    setFunctionArgs
+      # Adds the fixed-point style support
+      (fpargs: modify (mkDerivationBase (finalAttrs: adaptArgs finalAttrs (toFunction fpargs finalAttrs))))
+      # Add __functionArgs
+      (
+        # Inherit the __functionArgs from the base build helper
+        optionalAttrs inheritFunctionArgs (functionArgs mkDerivationBase)
+        # Recover the __functionArgs from the derived build helper
+        // functionArgs (adaptArgs { })
+      )
+    // {
+      inherit
+        # Expose to the result build helper.
+        adaptArgs
+        mkDerivationBase
+        modify;
+    };
 }