Use multipleOutputPaths as optimization

[aameen-tulip]

In my case populating the cache using fetch-one is painfully slow. ~6,000 dependencies, each takes ~0.3 ms to build with yarn, but ~2-4 seconds each spinning up build areas.

I think that by using multiple outputs in your cache builder you could drop a massive chunk of time spent spinning up and tearing down build areas.

Because the current cache builder processes these one at a time it takes ages.
By using flat hash you won't poison the hashes by changing to multiple outputs AFAIK.

Possibly an even faster way since the yarn plugin runs externally of nix: when populating the nix store from an existing cache, use nix-store --add-fixed sha512 .yarn/cache/* and modify the tarball name or something to match the expected store path name.

[adrian-gierakowski]

@aameen-tulip thank you for your suggestions. I’d be interested in improving this, however I’m not sure what you mean by multipleOutputPaths?

[stephank]

In my case populating the cache using fetch-one is painfully slow. ~6,000 dependencies, each takes ~0.3 ms to build with yarn, but ~2-4 seconds each spinning up build areas.

I think that by using multiple outputs in your cache builder you could drop a massive chunk of time spent spinning up and tearing down build areas.

Because the current cache builder processes these one at a time it takes ages. By using flat hash you won't poison the hashes by changing to multiple outputs AFAIK.

I agree this is a pain, and I've been thinking about adding a mode that simply builds the cache as a whole, instead of per dependency. (Like how makeRustPackage does it, if that sounds familiar. I believe other language solutions in Nixpkgs do similar stuff.)

I'm not sure how multiple outputs fit in, though. I'm not that familiar beyond how Nixpkgs uses them to split bin/lib/dev/doc. I guess the benefit is, even on rebuild, an intermediate Nix cache would only have to serve what has changed? Does that still require fixed output derivations, though, and how does that work with multiple outputs? (Or, I guess, the content-addressable stuff will solve that for us in the future.)

I also wonder if Nix handles 6000 outputs, or if there is some limit on it. 🙂

Aside from multiple outputs, I think the main implementation obstacle to doing a whole cache build is that we have to implement NAR and directory hashing on the JavaScript side, in order to create a fixed output derivation for the cache build. It's not super complicated, just a bunch of work.

Possibly an even faster way since the yarn plugin runs externally of nix: when populating the nix store from an existing cache, use nix-store --add-fixed sha512 .yarn/cache/* and modify the tarball name or something to match the expected store path name.

I believe this is what preloading does? Or maybe I don't follow. 🙂

yarn-plugin-nixify/src/generate.ts

Lines 296 to 353 in 9d73564

	// Preload the cache entries into the Nix store.
	if (
	configuration.get(`enableNixPreload`) &&
	xfs.existsSync(npath.toPortablePath(`/nix/store`))
	) {
	await xfs.mktempPromise(async (tempDir) => {
	const toPreload: PortablePath[] = [];
	for (const [locator, { filename, sha512 }] of cacheEntries.entries()) {
	const name = sanitizeDerivationName(locator);
	// Check to see if the Nix store entry already exists.
	const hash = Buffer.from(sha512, "hex");
	const storePath = computeFixedOutputStorePath(name, `sha512`, hash);
	if (!xfs.existsSync(storePath)) {
	// The nix-store command requires a correct filename on disk, so we
	// prepare a temporary directory containing all the files to preload.
	//
	// Because some names may conflict (e.g. 'typescript-npm-xyz' and
	// 'typescript-patch-xyz' both have the same derivation name), we
	// create subdirectories based on hash.
	const subdir = ppath.join(tempDir, sha512.slice(0, 7) as Filename);
	await xfs.mkdirPromise(subdir);
	const src = ppath.join(cache.cwd, filename);
	const dst = ppath.join(subdir, name as Filename);
	await xfs.copyFilePromise(src, dst);
	toPreload.push(dst);
	}
	}
	try {
	// Preload in batches, to keep the exec arguments reasonable.
	const numToPreload = toPreload.length;
	while (toPreload.length !== 0) {
	const batch = toPreload.splice(0, 100);
	await execUtils.execvp(
	"nix-store",
	["--add-fixed", "sha512", ...batch],
	{
	cwd: project.cwd,
	strict: true,
	}
	);
	}
	if (numToPreload !== 0) {
	report.reportInfo(
	0,
	`Preloaded ${numToPreload} packages into the Nix store`
	);
	}
	} catch (err: any) {
	// Don't break if there appears to be no Nix installation after all.
	if (err.code !== "ENOENT") {
	throw err;
	}
	}
	});
	}