Don't allow IFD in flakes in some commands

[edolstra]

Import-from-derivation is a controversial feature because it can cause arbitrary amounts of building when running commands like nix flake show. It's better to disallow it now and re-enable it in a future version than the other way around.

This PR disables IFD for the folliowing operations:

• nix search (this was already implied by read-only mode)
• nix flake show
• nix flake check, but only on the hydraJobs output

[Kha]

Just to be safe, this means I can pass --allow-import-from-derivation to e.g. nix build (without --impure and losing the eval cache) to re-enable it?

[edolstra]

@Kha Yes.

[roberth]

IFD makes various language integrations much more pleasant in day to day development with Haskell, yarn, etc. It does not cause a performance problem in these use cases. I would also expect the imported derivation outputs to be in the cache, making it ok for distribution via the "well known" flake outputs.

Of course in Nixpkgs it's still not acceptable to rely on IFD, but Nix usage outside Nixpkgs can be different and that's a good thing.

Will we be able to enable IFD per flake and/or in nix.conf?

[edolstra]

Just to give another example of why IFD is a problem: it would require search.nixos.org to build arbitrary amounts of stuff just to find out what the outputs of a flake are. So it's better to avoid a flake ecosystem that heavily relies on IFD.

Will we be able to enable IFD per flake and/or in nix.conf?

In principle allow-import-from-derivation is a configuration setting, but this PR makes nix override it. We could do this:

-    evalSettings.enableImportFromDerivation = false;
+    evalSettings.enableImportFromDerivation.setDefault("false");

but allowing a global override seems dangerous.

[shlevy]

I have long thought the arguments against IFD do not justify losing its functionality, and that we can even work around the legitimate issues in principled ways, but if this is going to happen can we at least have some metadata (not a command line flag, not a nix conf override) in our flakes that lets us say "yes, this flake uses IFD, I'm OK with it being excluded from search.nixos.org etc."? This would obviously have to infect downstream flakes as well.

Every single real world usage of flakes I've worked with uses IFD in a fundamental way.

[disassembler]

This would break haskell.nix and many of our other build tools. There has to be a way to configure this right? We can't just drop IFD and say there's no solution for people using it.

[domenkozar]

I agree about IFD being extremely useful with language integrations.

What if there was an enableImportFromDerivation = true; top-level attribute on each flake that would default to false?

Then search can skip those and report only limited info for now.

EDIT: haven't seen that @shlevy proposed the same alternative

[blaggacao]

There is an update from the 1st Shepherd Meeting of RFC92 which precisely aims at solving problems related to language integrations:

• [RFC 0092] Computed derivations rfcs#92 (comment)

This is more a relevant outlook & by no means a solution

[roberth]

Regardless of RFC 92, IFD could be avoided by search.nixos.org as long as it doesn't evaluate drvPath, outPath and such. That should work, as long as the packages are lazy enough. Flake authors who want their package listed could make sure to provide just the required metadata and only perform IFD when an actual derivation is required.

This makes the problem a Nixpkgs problem, as it doesn't provide a separate mkPackage concept as would seem necessary also in the case of RFC 92. See how I imagined a mkPackage. The role of RFC 92 itself is mostly just to make it so that drvPath evaluates to something without performing the actual IFD. I don't believe this is very relevant for the purpose of search.nixos.org.

I would prefer for IFD to be on by default, which is very useful for private and "pre-release open source", and have search.nixos.org report an error referring to how the author can use mkPackage to provide the metadata without IFD.

[hamishmack]

I worry that requiring users to manually enumerate the outputs of their haskell.nix project in the flake.nix file will simply discourage them from using flakes at all.

I don't think --allow-import-from-derivation as a command line option is a good solution. Having to tell users of your haskell.nix project to pass that (or set something in nix.conf) would lead to confusion.

Having an enableImportFromDerivation flag in the flake.nix seems needlessly onerous. If the author of the flake.nix file has used an IFD in outputs in what situation would they not set this flag? If it is always set when IFDs are present why not just assume IFDs are enabled? Would the flake.nix author remember to remove it if they changed the outputs to not use an IFD?

Could we?

• Allow IFD by default (as is the case now).
• Print warning when it is used by nix flake show. Something like Warning: The outputs of this flake depend on an IFD (some tools and services like 'search.nixos.org' will not list them).
• Provide mechanism to disable IFD (for things like search.nixos.org to use). Instead of the outputs for the flake you would see an error something like Error: The outputs of this flake depend on an IFD and IFDs are disabled.

[blaggacao]

Starting from first principles, there must exist better solutions to the underlying issues. (e.g. RFC92?)

I want to add, that terraform has had similar discussions about multi stage apply (read "derivation") and the consistent answer & conclusion from the core team a.f.a.i.k. has been "no".

In my interpretation the general sense has been that terraform is not a (DAG) orchestrator, but would expect an external orchestrator / task runner to take on that role.

I think much of the argument is portable to nix, at least in certain aspects such as: is nix meant to be a (hidden) DAG task runner?

I understand that the thin line here "of what consitutes a task" is the distinction of shelling out to an external tool instead of having a builtin nix-native implemtation, which in itself is not the best definition of task.

What is the remote api for terraform/nomia, is the file system api for nix.

That file system api has stronger consistency guarantees than an eventually consistent remote state, which might justify a different stance & conclusion.

If we add in a networked filesystem, though, that property starts to blur. (hint: ipfs?)

Edit (rough analogy for the next comment):

• task = brick
• orchestrator = mortar

[Ericson2314]

In a way, banning IFD feels silly, just a needless straight-jacket.

In another way, however, I see why @edolstra wants it: Flakes come from a view where different repos provide different packages, and we build up the package ecosystem brick by brick, with the flakes being the bricks. After really using IFD consistently, however, one sees different repos not as providing different packages, but by providing helper code to work with different regimes of packages --- to wit, a version of Haskell.nix can be used with many different snapshots of Hackage, in which case it provides different packages. In that regime, the Nix code is the mortar around the bricks that upstream provides.

As to RFC 92, see what I wrote in NixOS/rfcs#92 (comment). I regret tying that RFC to IFD. That is still interesting and possible to me, but it is too much work for now. But in its stead, I hope to write a much simpler RFC for IFD in Nixpkgs. It won't address the fundamental issues here --- no magic to make search.nixos.org complete without any building --- but it would I hope allow IFD to go in Nixpkgs by doing 2 separate evals instead of longer evals.

If such a thing works out, I think we'll all be better positioned to judge the merits of IFD, the merits of the "mortar, not bricks" view, etc.

[roberth]

@blaggacao I'll reply to the first two lines I got in my inbox. I'm having trouble to see the relevance of your edits (the rest of the essay) about orchestration and consistency.

Terraform is a vastly different beast than Nix. Terraform deals with cloud resources; some of the most side-effectful things in anyone's infrastructure, whereas Nix only deals with pure computation and a necessary amount of I/O. This leads to different requirements regarding static inspectability, which is sufficient to explain the difference in the chosen trade-off. Furthermore, this is about disabling rather than enabling a feature.

@Ericson2314

Flakes come from a view where different repos provide different packages, and we build up the package ecosystem brick by brick, with the flakes being the bricks. [...] but by providing helper code to work with different regimes of packages

Isn't this what Nix is all about? It's the functional package manager. Without those we'd be stuck with something resembling Dockerfiles. Of course Nixpkgs is so core to our daily use of Nix that we can't see it anymore, like a fish would not have a concept of water. It is full of functions and generic code. By discouraging functions/glue/"mortar" in flakes other than Nixpkgs, you're doing a disservice to the original goal of allowing expressions to be distributed in a decentralized manner (a package manager for Nix expressions).

Any design where Nixpkgs is "special" in any way, shape or form is flawed because it means that something that can be done in Nixpkgs can't be done elsewhere and thus can't be independently innovated upon.

After really using IFD consistently, however, one sees different repos not as providing different packages, but by providing helper code to work with different regimes of packages --- to wit, a version of Haskell.nix can be used with many different snapshots of Hackage, in which case it provides different packages. In that regime, the Nix code is the mortar around the bricks that upstream provides.

No, instead what you're seeing is a taste of what a distributed expression ecosystem can become. haskell.nix could not be part of Nixpkgs because it disallows IFD. As such, it had to be developed outside the cosy confines of Nixpkgs where we don't care about whether something is bricks or mortar. Suppose IFD was allowed in Nixpkgs, the mortar would be added there because it is convenient and we could continue to believe the flakes can be built out of bricks only for a while because we ignore Nixpkgs.

to wit, a version of Haskell.nix can be used with many different snapshots of Hackage

This is no more true than "Nixpkgs can be used with many different snapshots of GNU hello" because you can overrideAttrs src. Haskell.nix does pin Hackage, so it does provide the bricks. Sure you can use its mortar, but that's a distinct feature that you don't have to use. Similarly, nix (as a flake) is mortar because you can use it with different nixpkgs via inputs.nix.inputs.nixpkgs.follows resulting in different packages.

In another way, however, I see why @edolstra wants it

What commercial users need is a standardized glue between repos and more/better language integrations. They're putting a lot of faith in Flakes for the glue part. Forming Flakes around the restrictions of Nixpkgs/NixOS is a good way to limit adoption.

I'm serious. Flakes do not leave a lot of space for other glue to exist, because such projects will be either 95% duplicated effort or a Nix fork, neither of which will be well received.

Please implement @hamishmack's suggestion instead of disallowing IFD.

[roberth]

I worry that requiring users to manually enumerate the outputs of their haskell.nix project in the flake.nix file will simply discourage them from using flakes at all.

This does not apply to private flakes and flakes that provide only their own packages. Those would presumably have to be augmented with some metadata anyway.

Of course this is not much help for haskell.nix, but that's an outlier. Maybe it's possible to generate the metadata and store it in a file?

Ideally, the evaluator would be capable of performing IFD from an arbitrary substituter specified in the flake. This can be done safely within the current implementation architecture if we accept the constraint that IFD must produce a single store path instead of a closure. This can be done by making the evaluator perform its IFD directly from a (somewhat short lived) Store object that is tied to the flake, so it can all happen outside the real local store which is where the security problem came from.
For 2.4 it's ok not to list flakes that use IFD.

• Allow IFD by default (as is the case now).
• Print warning when it is used by nix flake show. Something like Warning: The outputs of this flake depend on an IFD (some tools and services like 'search.nixos.org' will not list them).
• Provide mechanism to disable IFD (for things like search.nixos.org to use). Instead of the outputs for the flake you would see an error something like Error: The outputs of this flake depend on an IFD and IFDs are disabled.

💯

I would like to add that the warning will not trigger when the metadata is lazy enough and it doesn't record paths, allowing some/most IFD flakes to be listed.

Provide mechanism to disable IFD

This is covered by --option allow-import-from-derivation false iiuc.

[Mic92]

With this the NUR flake would be rendered useless.

[Ericson2314]

@roberth

Isn't this what Nix is all about? It's the functional package manager. Without those we'd be stuck with something resembling Dockerfiles. Of course Nixpkgs is so core to our daily use of Nix that we can't see it anymore, like a fish would not have a concept of water. It is full of functions and generic code. By discouraging functions/glue/"mortar" in flakes other than Nixpkgs, you're doing a disservice to the original goal of allowing expressions to be distributed in a decentralized manner (a package manager for Nix expressions).

I think I am agreeing with you. Unless I am out of date / misunderstanding something, the "output" side of flakes somewhat assumes the main thing you are providing is a bunch of packages, not this other stuff.

I'd say that the TOML flakes are really what's disagreeing with you. Those preclude custom glue altogether.

No, instead what you're seeing is a taste of what a distributed expression ecosystem can become. haskell.nix could not be part of Nixpkgs because it disallows IFD. As such, it had to be developed outside the cosy confines of Nixpkgs where we don't care about whether something is bricks or mortar. Suppose IFD was allowed in Nixpkgs, the mortar would be added there because it is convenient and we could continue to believe the flakes can be built out of bricks only for a while because we ignore Nixpkgs.

I see what you mean --- I also certainly wouldn't want us to miss out on requirements / aspirations because we took Nixpkgs for granted. But I suppose after making many cross cutting refactors I am skeptical of the benefits of distributing the mortar. Even if haskell.nix did stay in a separate repo, I think it would best upstream of Nixpkgs -- Nixpkgs to me is sort of the "terminal repo" that directly includes or references everything else to make sure it works together.

This is no more true than "Nixpkgs can be used with many different snapshots of GNU hello" because you can overrideAttrs src. Haskell.nix does pin Hackage, so it does provide the bricks. Sure you can use its mortar, but that's a distinct feature that you don't have to use.

Well what I mean is there is an actual specification around how Hackage works, that is true across many versions (though not all of them), so there is "morally" a type one could give Hackage were it a parameter to Haskell.nix. On the other hand, GNU Hello doesn't really have any spec except for "no one is touching this code, it's probably going to continue to use Autotools", so a parameter type would be a bit more of an empirical guess. The distinction is slight, but matters to me as to what ought to be parameterized and what ought to be a pin that's adjusted.

Similarly, nix (as a flake) is mortar because you can use it with different nixpkgs via inputs.nix.inputs.nixpkgs.follows resulting in different packages.

I guess I was also thinking haskell.nix could return a function taking Hackage rather than having it as an flake input. The currying is meaningful to me in that Hackage isn't a Nix repo. If every flake with inputs is "morter", "mortar" doesn't mean to much. To make it's about what the flake-ish thing returns, package or something else.

What commercial users need is a standardized glue between repos and more/better language integrations. They're putting a lot of faith in Flakes for the glue part. Forming Flakes around the restrictions of Nixpkgs/NixOS is a good way to limit adoption.

I'm serious. Flakes do not leave a lot of space for other glue to exist, because such projects will be either 95% duplicated effort or a Nix fork, neither of which will be well received.

I don't deny we need some conventions, but those conventions need to be good too. I am still very wary of flakes because of many design decisions, but more broadly because we are baking something into Nix so there is no opportunity to revist our abstractions in a fluid manner on an on going basis.

Nixpkgs at least has as the abstractions and packages together, and we've still failed as a community to reorganize it into something more sane. I think we need types, so we can actually make interesting refactors, and also IFD allowed in Nixpkgs (or repos Nixpkgs includes!), and only then be allowed to bake some design patterns into Nix after much experimentation a la flakes.

If all the effort for flakes instead went into, say Nickle, and we he started Nickling Nixpkgs, I would be much, much, more confident and less nervous.

[roberth]

I'd say that the TOML flakes are really what's disagreeing with you. Those preclude custom glue altogether.

Unless the TOML flakes are backed by, say, the module system and they only need to point at a flake to have their behavior imported and accessible via mere "data" options. But now I'm trying to design that while grossly unaware of the state of the feature. This would make it entirely possible to develop the feature outside of Nix and only put a tiny bit of invocation logic in C++ in the end. Same for flakes by the way, but that path wasn't chosen. (It's CLI could have remained separate as well, using Nix's libs.)

the abstractions and packages together, and we've still failed as a community to reorganize it into something more sane.

Perhaps the decentralized approach is an opportunity: don't fix but replace. That works for projects dedicated to Nix, but induces fatigue in merely Nix-ified projects like those that serve end-user goals instead of Nix user goals.

So this is an argument to flake-ify the mortar but not the bricks.

If all the effort for flakes instead went into, say Nickle, and we he started Nickling Nixpkgs, I would be much, much, more confident and less nervous.

Type errors do help a little bit, but you can only ask a regular (thus semi-disinterested) project maintainer to make changes so many times.
This is not a problem when expression maintenance happens in github.com/NixOS repos and similar, but I don't see a future where flakes are all in the NixOS, nix-community, etc orgs.

And then there was talk at some point about removing eval fetchers and FOD for flakes, so flake.nix in the actual application repo seems to be the goal. So we'd have to fork the world? That can't be right. I must have missed something.

[edolstra]

Updated the PR to only disable IFD on a few operations (see the updated description).

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-18/15300/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-nix3-0-issues/15626/10

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/another-simple-flake-for-haskell-development/18164/6

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pkgs-runcommand-not-being-run-in-home-manager-module/59400/5