Design and enable creation of remote gc roots

[layus]

Is your feature request related to a problem? Please describe.

It is impossible to get an atomic gc root on remote store paths.
The creation of gc roots is explicitly disabled in nix build if we do not have a LocalFSStore:

nix/src/nix/build.cc

Lines 141 to 143 in 26c3fc1

	if (outLink != "")
	if (auto store2 = store.dynamic_pointer_cast<LocalFSStore>())
	createOutLinks(outLink, buildables, *store2);

Describe the solution you'd like

Ideally, there should be a way to create remote gc roots for paths that are build remotely using --store ssh-ng:... and other remote stores and operations. Otherwise there is no way to enforce atomicity.

The biggest challenge is to create roots that are owned by the right user, and pick a place to put them.

I suggest we might discover the user from the ssh connection, and put the indirect root in a well-known location in side it's home. I need to check if there is an obvious place to do that. If there is a user process on the remote machine that proxies the requests to the local nix daemon (so the remote nix daemon from the point of view of the user) then it is the right place to do it.
If connection are forwarded to the remote nix daemon directly, it may become tricky.

As for the location, I assume .local/nix/gcroots would be good enough. .cache is probably not good.

And for the name, it is really tricky. I do not think we should trust the name provided on the command line, as there is a high potential for conflicts and race conditions. Using the nix store path name itself might work. Alternatively, trusting the name passed by the user provides lots of flexibility. They can decide on their own how to organize their remote roots.

As a note, I think these roots should be updated on each access, so as to allow a basic LRU gc to happen if needed.

Describe alternatives you've considered

• Direct gc roots are not good, as they are owned by root. The user who created them cannot remove them.
• If we had /nix/var/nix/gcroots/per-user/ around we could craft something with it, but it has been recently removed.

Additional context

Related issues:

#4002
#11505 (comment)
#11506
#7138 (comment)

Some hints in the code:

nix/src/libstore/gc-store.hh

Lines 88 to 92 in 26c3fc1

	* @todo actually this is not so good because stores are *views*.
	* Some views have only a no-op temp roots even though others to the
	* same store allow triggering GC. For instance one can't add a root
	* over ssh, but that doesn't prevent someone from gc-ing that store
	* accesed via SSH locally).

nix/src/libstore/indirect-root-store.hh

Lines 24 to 29 in 26c3fc1

	* | | `addPermRoot()` | `addIndirectRoot()` |
	* |-------------------|-----------------|---------------------|
	* | `LocalStore` | local | local |
	* | `UDSRemoteStore` | local | remote |
	* | `SSHStore` | doesn't have | doesn't have |
	* | `MountedSSHStore` | remote | doesn't have |

/cc @kjeremy @kevincox for you have showed interest in this topic before.

Priorities

Add 👍 to issues you find important.