Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runInLinuxVM: add hostPkgs #352946

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

runInLinuxVM: add hostPkgs #352946

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 20 additions & 7 deletions pkgs/build-support/vm/default.nix
View file
Open in desktop
Copy link
Member

@Artturin Artturin Nov 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would make sense to me: I want to run aarch64-multiplatform hello in an aarch64-linux VM but build, i.e. run the VM on a Darwin build-system.

But this fails very early. Evaluating the following:

(import <nixpkgs> { }).pkgsCross.aarch64-multiplatform.buildPackages.qemu_kvm'

There's no qemu-user for darwin

nixpkgs/lib/systems/default.nix

Lines 289 to 290 in 068b0aa

else if final.isLinux && pkgs.stdenv.hostPlatform.isLinux && final.qemuArch != null
then "${pkgs.qemu-user}/bin/qemu-${final.qemuArch}"

Package ‘qemu-user-9.1.1’ in ./pkgs/applications/virtualization/qemu/default.nix:318 is not available on the requested hostPlatform
which we need for cross-compiling gobject-introspection

Maybe a conditional wrong somewhere

nix-repl> pkgsCross.aarch64-multiplatform.buildPackages.graphene
error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'graphene-1.10.8'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'postPatch' of derivation 'graphene-1.10.8'
         at /home/artturin/nixgits/my-nixpkgs/pkgs/development/libraries/graphene/default.nix:95:3:
           94|
           95|   postPatch = ''
             |   ^
           96|     patchShebangs tests/gen-installed-test.py

       … from call site
         at /home/artturin/nixgits/my-nixpkgs/pkgs/development/libraries/graphene/default.nix:97:8:
           96|     patchShebangs tests/gen-installed-test.py
           97|   '' + lib.optionalString withIntrospection ''
             |        ^
           98|     PATH=${python3.withPackages (pp: [ pp.pygobject3 pp.tappy ])}/bin:$PATH patchShebangs tests/introspection.py

       … while calling 'optionalString'
         at /home/artturin/nixgits/my-nixpkgs/lib/strings.nix:678:5:
          677|     cond:
          678|     string: if cond then string else "";
             |     ^
          679|

       … while calling the 'getAttr' builtin
         at <nix/derivation-internal.nix>:19:19:
           18|       value = commonAttrs // {
           19|         outPath = builtins.getAttr outputName strict;
             |                   ^
           20|         drvPath = strict.drvPath;

       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'python3-3.12.7-env'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'passAsFile' of derivation 'python3-3.12.7-env'
         at /home/artturin/nixgits/my-nixpkgs/pkgs/build-support/trivial-builders/default.nix:60:9:
           59|         inherit buildCommand name;
           60|         passAsFile = [ "buildCommand" ]
             |         ^
           61|           ++ (derivationArgs.passAsFile or [ ]);

       … while evaluating the attribute 'passAsFile'
         at /home/artturin/nixgits/my-nixpkgs/pkgs/build-support/buildenv/default.nix:87:5:
           86|     # XXX: The size is somewhat arbitrary
           87|     passAsFile = if builtins.stringLength pkgs >= 128*1024 then [ "pkgs" ] else [ ];
             |     ^
           88|   }

       … while evaluating a branch condition
         at /home/artturin/nixgits/my-nixpkgs/pkgs/build-support/buildenv/default.nix:87:18:
           86|     # XXX: The size is somewhat arbitrary
           87|     passAsFile = if builtins.stringLength pkgs >= 128*1024 then [ "pkgs" ] else [ ];
             |                  ^
           88|   }

       … in the argument of the not operator
         at /home/artturin/nixgits/my-nixpkgs/pkgs/build-support/buildenv/default.nix:87:48:
           86|     # XXX: The size is somewhat arbitrary
           87|     passAsFile = if builtins.stringLength pkgs >= 128*1024 then [ "pkgs" ] else [ ];
             |                                                ^
           88|   }

       … while calling the 'lessThan' builtin
         at /home/artturin/nixgits/my-nixpkgs/pkgs/build-support/buildenv/default.nix:87:48:
           86|     # XXX: The size is somewhat arbitrary
           87|     passAsFile = if builtins.stringLength pkgs >= 128*1024 then [ "pkgs" ] else [ ];
             |                                                ^
           88|   }

       … while calling the 'stringLength' builtin
         at /home/artturin/nixgits/my-nixpkgs/pkgs/build-support/buildenv/default.nix:87:21:
           86|     # XXX: The size is somewhat arbitrary
           87|     passAsFile = if builtins.stringLength pkgs >= 128*1024 then [ "pkgs" ] else [ ];
             |                     ^
           88|   }

       … while calling the 'toJSON' builtin
         at /home/artturin/nixgits/my-nixpkgs/pkgs/build-support/buildenv/default.nix:82:12:
           81|             nativeBuildInputs buildInputs;
           82|     pkgs = builtins.toJSON chosenOutputs;
             |            ^
           83|     extraPathsFrom = lib.optional includeClosures (writeClosure pathsForClosure);

       … while calling the 'map' builtin
         at /home/artturin/nixgits/my-nixpkgs/pkgs/build-support/buildenv/default.nix:56:19:
           55| let
           56|   chosenOutputs = map (drv: {
             |                   ^
           57|     paths =

       … from call site
         at /home/artturin/nixgits/my-nixpkgs/pkgs/development/interpreters/python/wrapper.nix:20:13:
           19|   env = let
           20|     paths = requiredPythonModules (extraLibs ++ [ python ] ) ;
             |             ^
           21|     pythonPath = "${placeholder "out"}/${python.sitePackages}";

       … while calling 'requiredPythonModules'
         at /home/artturin/nixgits/my-nixpkgs/pkgs/development/interpreters/python/python-packages-base.nix:54:27:
           53|   # Get list of required Python modules given a list of derivations.
           54|   requiredPythonModules = drvs: let
             |                           ^
           55|     modules = lib.filter hasPythonModule drvs;

       … while calling the 'foldl'' builtin
         at /home/artturin/nixgits/my-nixpkgs/pkgs/development/interpreters/python/python-packages-base.nix:56:6:
           55|     modules = lib.filter hasPythonModule drvs;
           56|   in lib.unique ([python] ++ modules ++ lib.concatLists (lib.catAttrs "requiredPythonModules" modules));
             |      ^
           57|

       … while calling anonymous lambda
         at /home/artturin/nixgits/my-nixpkgs/lib/lists.nix:1793:25:
         1792|   */
         1793|   unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
             |                         ^
         1794|

       … while evaluating a branch condition
         at /home/artturin/nixgits/my-nixpkgs/lib/lists.nix:1793:28:
         1792|   */
         1793|   unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
             |                            ^
         1794|

       … while calling the 'elem' builtin
         at /home/artturin/nixgits/my-nixpkgs/lib/lists.nix:1793:31:
         1792|   */
         1793|   unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
             |                               ^
         1794|

       … while evaluating the attribute 'outPath'
         at /home/artturin/nixgits/my-nixpkgs/lib/customisation.nix:366:7:
          365|       drvPath = assert condition; drv.drvPath;
          366|       outPath = assert condition; drv.outPath;
             |       ^
          367|     };

       … while evaluating the attribute 'outPath'
         at /home/artturin/nixgits/my-nixpkgs/lib/customisation.nix:366:7:
          365|       drvPath = assert condition; drv.drvPath;
          366|       outPath = assert condition; drv.outPath;
             |       ^
          367|     };

       … while calling the 'getAttr' builtin
         at <nix/derivation-internal.nix>:19:19:
           18|       value = commonAttrs // {
           19|         outPath = builtins.getAttr outputName strict;
             |                   ^
           20|         drvPath = strict.drvPath;

       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'python3.12-pygobject-3.50.0'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'propagatedBuildInputs' of derivation 'python3.12-pygobject-3.50.0'
         at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:390:7:
          389|       depsHostHostPropagated      = elemAt (elemAt propagatedDependencies 1) 0;
          390|       propagatedBuildInputs       = elemAt (elemAt propagatedDependencies 1) 1;
             |       ^
          391|       depsTargetTargetPropagated  = elemAt (elemAt propagatedDependencies 2) 0;

       … while calling the 'getAttr' builtin
         at <nix/derivation-internal.nix>:19:19:
           18|       value = commonAttrs // {
           19|         outPath = builtins.getAttr outputName strict;
             |                   ^
           20|         drvPath = strict.drvPath;

       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'gobject-introspection-wrapped-1.82.0'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'depsTargetTargetPropagated' of derivation 'gobject-introspection-wrapped-1.82.0'
         at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:391:7:
          390|       propagatedBuildInputs       = elemAt (elemAt propagatedDependencies 1) 1;
          391|       depsTargetTargetPropagated  = elemAt (elemAt propagatedDependencies 2) 0;
             |       ^
          392|

       … while calling the 'getAttr' builtin
         at <nix/derivation-internal.nix>:19:19:
           18|       value = commonAttrs // {
           19|         outPath = builtins.getAttr outputName strict;
             |                   ^
           20|         drvPath = strict.drvPath;

       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'gobject-introspection-aarch64-unknown-linux-gnu-1.82.0'
         whose name attribute is located at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'mesonFlags' of derivation 'gobject-introspection-aarch64-unknown-linux-gnu-1.82.0'
         at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:562:7:
          561|       cmakeFlags = makeCMakeFlags attrs;
          562|       mesonFlags = makeMesonFlags attrs;
             |       ^
          563|     });

       … from call site
         at /home/artturin/nixgits/my-nixpkgs/pkgs/development/libraries/gobject-introspection/default.nix:114:34:
          113|     }}"
          114|     "-Dgi_cross_binary_wrapper=${stdenv.hostPlatform.emulator buildPackages}"
             |                                  ^
          115|     # can't use canExecute, we need prebuilt when cross

       … while calling 'emulator'
         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:303:20:
          302|
          303|         emulator = pkgs:
             |                    ^
          304|           if (final.emulatorAvailable pkgs)

       … while calling the 'throw' builtin
         at /home/artturin/nixgits/my-nixpkgs/lib/systems/default.nix:306:16:
          305|           then selectEmulator pkgs
          306|           else throw "Don't know how to run ${final.config} executables.";
             |                ^
          307|

       error: Don't know how to run aarch64-unknown-linux-gnu executables.

I think we need to improve the conditional for this or shift offsets in the python withPackages tree (if they're wrong)

nixpkgs/pkgs/development/libraries/graphene/default.nix

Lines 97 to 98 in b084db8

'' + lib.optionalString withIntrospection ''
PATH=${python3.withPackages (pp: [ pp.pygobject3 pp.tappy ])}/bin:$PATH patchShebangs tests/introspection.py

There's ++ lib.optionals (stdenv.hostPlatform == stdenv.buildPlatform) [ "installedTests" ]; above so for the installed test stuff we probably need that conditional too

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#353237

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wow, @Artturin thank you so much! What wizardry is that 😄 On your branch I can evaluate qemu_kvm but the build fails. I'm investigating that.

Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
{ lib

# pkgs used inside the VM
, pkgs

# pkgs used outside the VM
# (similar to NixOS's `virtualisation.host.pkgs` option)
# using over `buildPackages` as currently `nix-build -E '(import <nixpkgs> { }).pkgsCross.aarch64-multiplatform.buildPackages.qemu_kvm'`
# fails on aarch64-darwin with `error: Don't know how to run aarch64-unknown-linux-gnu executables.`
, hostPkgs ? pkgs.buildPackages

Copy link
Contributor Author

@felixscheinost felixscheinost Nov 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall there are 4 packages which need to comre from hostPkgs:

  • qemu_kvm
  • bash for builder
  • coreutils for run-vm
  • writeText for run-vm

, customQemu ? null
, kernel ? pkgs.linux
, img ? pkgs.stdenv.hostPlatform.linux-kernel.target
Expand All @@ -12,12 +21,15 @@
let
inherit (pkgs) bash bashInteractive busybox cpio coreutils e2fsprogs fetchurl kmod rpm
stdenv util-linux
buildPackages writeScript writeText runCommand;
buildPackages writeScript runCommand;
in
rec {
qemu-common = import ../../../nixos/lib/qemu-common.nix { inherit lib pkgs; };

qemu = buildPackages.qemu_kvm;
qemu = hostPkgs.qemu_kvm;

# export hostPkgs so that downstream tools that wrap around e.g. runInLinuxVM can use it
inherit hostPkgs;

modulesClosure = pkgs.makeModulesClosure {
inherit kernel rootModules;
Expand Down Expand Up @@ -232,10 +244,10 @@ rec {
'';


vmRunCommand = qemuCommand: writeText "vm-run" ''
vmRunCommand = qemuCommand: hostPkgs.writeText "vm-run" ''
export > saved-env

PATH=${coreutils}/bin
PATH=${hostPkgs.coreutils}/bin
mkdir xchg
mv saved-env xchg/

Expand All @@ -253,7 +265,7 @@ rec {
# debug inside the VM if the build fails (when Nix is called with
# the -K option to preserve the temporary build directory).
cat > ./run-vm <<EOF
#! ${bash}/bin/sh
#! ${hostPkgs.bash}/bin/sh
''${diskImage:+diskImage=$diskImage}
TMPDIR=$TMPDIR
cd $TMPDIR
Expand Down Expand Up @@ -334,8 +346,9 @@ rec {
that allows you to boot into the VM and debug it interactively. */

runInLinuxVM = drv: lib.overrideDerivation drv ({ memSize ? 512, QEMU_OPTS ? "", args, builder, ... }: {
requiredSystemFeatures = [ "kvm" ];
builder = "${bash}/bin/sh";
system = hostPkgs.system;
requiredSystemFeatures = if !hostPkgs.stdenv.isDarwin then [ "kvm" ] else [];
builder = "${hostPkgs.bash}/bin/sh";
args = ["-e" (vmRunCommand qemuCommandLinux)];
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing system here is a hack. This would probably go away when using pkgsCross properly which currently doesn't work due to qemu_kvm (see error in PR).

origArgs = args;
origBuilder = builder;
Expand Down