Don't ignore unstaged files in local flakes

[SuperSandro2000]

The current behavior to just ignore unstaged files is unintuitive and super annoying. Also the warning that the worktree is dirty makes you expect that unstaged files would be included.

I've been using this patch since a month and it made my development workflow so much easier and I no longer commit nix files by accident because flakes made me do a git add.

[ElvishJerricco]

Maybe this could be a config or command line option or something. I think the current behavior is the right default. It's far too easy to add tons of superfluous files like nixos.qcow2 images and the like if untracked files are included, which really slows down copying the flake to the nix store and wastes a ton of space.

[edolstra]

Yeah this behaviour should be configurable. Note that in this mode, if any untracked but unignored file exists in the worktree, then the repo should be considered dirty. So the dirtiness check (i.e. where we call git diff) needs to be modified as well.

[edolstra]

Note that in the lazy-trees branch (#6530), you get a warning if you try to access an untracked file, e.g.

error: access to path '/home/eelco/Dev/nix/.version2' is forbidden because it is not under Git control; maybe you should 'git add' it to the repository '/home/eelco/Dev/nix'?

I no longer commit nix files by accident because flakes made me do a git add.

I mean, you're supposed to add Nix files and anything else used by the evaluator to ensure reproducible evaluation. The alternative is that evaluation works for you but fails for somebody who tries to build the same Git revision.

[SuperSandro2000]

Note that in the lazy-trees branch (#6530), you get a warning if you try to access an untracked file, e.g.

Its good that it is no longer silent.

you're supposed to add Nix files and anything else used by the evaluator to ensure reproducible evaluation

Completely on your side but when you are still developing stuff locally and maybe even work on two changes at once then adding files is not an intuitive workflow. At that point it also does not need to be reproducible and you are still getting the dirty tree warning which also indicates that the tree is not reproducible.

Yeah this behaviour should be configurable.

thats above my c++ knowledge

[SuperSandro2000]

I am not sure what is wrong with the tests. I wouldn't expect my changes to break flake.lock's.

[ilkecan]

... I no longer commit nix files by accident because flakes made me do a git add.

-N (--intent-to-add) git option can be used to avoid this.

[mightyiam]

Sorry for going against the grain, here. I feel against ignoring untracked files by default. I may not be the most experienced Nixer, but relying on whether a file is tracked or not seems like an anti-pattern to me.
Git is for managing code. It shouldn't be used for the behavior of it. I realize that Nix is in the domains of building software and therefore figuring out "what is included in this software?" is part of the domain, but relying on whether a file is tracked or not seems bonkers to me. I shouldn't have to git add a file in order to make it part of the software. It's part of the software because it exists.

End of thoughts from an inexperienced Nixer who doesn't have all the context.

[ElvishJerricco]

It's part of the software because it exists.

This seems like a bad assumption. My example above was nixos.qcow2; a large file created by doing VM based testing with nixpkgs. There are plenty of files that you wouldn't want to have copied to the world readable store just because you're messing with them in you flake cwd.

[mightyiam]

This seems like a bad assumption. My example above was nixos.qcow2; a large file created by doing VM based testing with nixpkgs. There are plenty of files that you wouldn't want to have copied to the world readable store just because you're messing with them in you flake cwd.

Sorry. I don't know the details. This "feature" (no offence) is to prevent some otherwise implicit copying to the store?

I don't remember what error brought me here, but is that error necessarily one that occurs after that copying to the store?

[Ericson2314]

Triaged in the Nix Team meeting this morning (n.b. I could not make it):

• We should wait for lazy trees before changing the behavior.

• Improvements to the error message would be great though in the meantime if it's not too costly to implement (@thufschmitt had a PR doing something like that)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-08-04-nix-team-meeting-minutes-77/31487/1

[nrdxp]

Aren't there security implications to this change? What if you have untracked secrets in repo that are not world readable, but now Nix just copies them to a world readable location in the /nix/store? I suppose you could alleviate that by respecting .gitignore rules, but just thought I'd mention it since I didn't see it brought up here yet.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-cant-nixos-rebuild-see-files-that-arent-in-the-git-repository/49053/5

[tomberek]

@Ericson2314 Presumably with the new SourcePath and Accessor infrastructure, this should be done as a new GitSourcePath?