How can I investigate a nix build that hangs?

[theotheryuval]

At work we have a python project set up with poetry2nix and nix2container, a package for a our project, some flake checks, and a development shell.

Since yesterday, running nix flake update succeeds, but subsequent nix build of the default package (mkPoetryApplication based) hangs indefinitely during evaluating derivation, with what seems to be a memory leak or infinite loop, CPU usage is high until swap thrashing starts.

In nix repl, ":lf ." terminates but evaluating the default package hangs. Interrupting it prints:

❯ nix repl
Welcome to Nix 2.17.0. Type :? for help.

nix-repl> :lf .
warning: Git tree '/home/user/matan_aporia' is dirty
Added 16 variables.

nix-repl> packages.x86_64-linux.default
^Cerror: interrupted by the user
«derivation
nix-repl>

and trying to use tab completion to explore this value as an attr set showed also hung when trying to evaluate buildInputs but not many other fields, such as the name or version.

nix-repl> packages.x86_64-linux.default.python
«derivation /nix/store/8zacqzz62px96vz3705kawrkhk3najhp-python3-3.10.12.drv»

nix-repl> packages.x86_64-linux.default.buildInputs
[ ^Cerror: interrupted by the user
«derivation

Edit: specifically it seems to be the outPath attribute of both the top level package and those of its buildInputs that hang.

Searching the web for information about either was difficult, what little information I did find was mainly theoretical or regarding nix's memory usage and how to optimize it, but I couldn't find any practical advice for how to deal with this:

1. how to narrow down the cause
   1. which expression is causing this?
   2. which flake does it come from?
   3. what's a sensible strategy for reproducing it more minimally?
2. how to mitigate/work around this issue, apart from simply not updating see next comment

[theotheryuval]

In this particular case, this was easy enough to diagnose and work around, but not debug, exactly

1. nix build manpage -> --debug, --verbose, confirmed infinite loop
   1. using :lf github:nix-community/poetry2nix and legacyPackages.x86_64-linux.mkPoetryApplication from the repl did work, and nix flake update only mentioned nixpkgs updating 2023-08-{27,30}, presumably poetry2nix nixpkgs input was pinned. i tried commenting out our follows decl, but that still didn't work
2. bisected local nixpkgs from our repo with: nix eval --override-input nixpkgs ~/nixpkgs --verbose .#default.outPath --debug. some of these errored, and one of them introduced the hang:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
6b6390885543ee52d0569128648ffefb3d2439b7
253a291274c2790008fe57c97983d3581a2c7798
3cd71e0ae67cc48f1135e55bf78cb0d67b53ff86
5a9dda28aa00dd88de3329c29bcdae40591d4634
We cannot bisect more!

3. found https://discourse.nixos.org/t/psa-poetry2nix-is-currently-broken-with-nixpkgs-unstable/32281 thanks to nixos-discourse bot from merging PR
4. revisted decision to use nixpkgs-unstable as flake input ;-)