- About this document
- Problems addressed by Nix
- Nix at a high level
- Frustrations acknowledged
- Encouraging development with flakes
- Helping non-flakes users
- Documenting an end-user experience
This document introduces the Nix package manager and highlights some motivations to use Nix. It also covers the tradeoffs of using Nix and experimental features in Nix, such as flakes.
This document tries to capture enthusiasm while being honest about frustrations. Nix is a clear pioneer of an architectural approach that users will demand in the future. However, users need clear information up front where they are likely to face challenges.
The following sections cover various problems that Nix's architecture addresses.
When dealing with a new software project, wrangling dependencies can be a chore. Modern build systems for specific programming languages often don't manage system dependencies. For example, Python's pip install will download and install needed Python dependencies but may fail if the system doesn't provide shared libraries required for foreign function calls. Adding complexity, different operating systems have differing names for these system packages and install them with various commands (apt, dnf, etc.). This variation makes automation difficult. Consequently, many software projects only provide documentation as a surrogate for automation, which creates even more room for error.
Some projects might have all the automation necessary for building, but due to subtle differences among systems, what builds on one system might not build on another.
For example, environment variables often can influence the behavior of the commands called by scripts. It's hard to lock down these variables on every system where something might be built.
Once we've built some software and are ready to deploy it, it's not always obvious how to copy this built software to another system. For example, if the software dynamically links to system libraries, we need to know whether those libraries are on our target system.
Another complication we face is when an operating system only allows one version of a system library to be installed at a time. When this happens, we may be forced to make difficult choices if we need two programs requiring different system dependency versions.
It can be tedious to synthesize libraries and programs from different language ecosystems to make a new program for a unified user experience. For example, the world of machine learning often requires the mixing of C/C++, Python, and even basic shell scripts. These hybrid applications tend to be fragile.
Various build systems provide repositories for pre-built packages, which helps users save time by downloading packages instead of building them. We want this experience unified across all programming language ecosystems and system dependencies.
Note this is what traditional package managers like DNF and APT accomplish. However, there's an ergonomic difficulty in turning all software into standard Linux packages. Firstly, there are too many Linux distributions with too many package managers. Secondly, most package managers must adhere to policies for everything to work well together. For example, many distributions respect the Filesystem Hierarchy Standard (FHS). Confusion around policies has led many developers to steer away from package managers and toward container-based technologies like Docker despite the overhead and drawbacks of containers.
Nix addresses all the problems discussed above.
To build or install any project, we should be able to start with only the Nix package manager installed. No other library or system dependency should be required to be installed or configured.
Even if we have a library or system dependency installed, it shouldn't interfere with any build or installation we want to do. Nix builds and installs in its own directories.
Our build should get everything we need, all the way down to the system-level dependencies, irrespective of which programming language the dependencies have been authored in. If anything has been pre-built, we should download a cached result.
Above and beyond the problems discussed above, Nix has a precisely deterministic build, broadly guaranteeing reproducibility. If the package builds on one system, it should build on all systems, regardless of what's installed. Furthermore, multiple systems independently building the same package will often produce bit-for-bit identical builds.
Nix is also able to copy the transitive closure of a package's dependencies ergonomically from one system to another.
In broad strokes, Nix is a technology that falls into two categories:
- package manager
- build tool.
As a package manager, Nix does what most package managers do. Nix provides a suite of command-line tools to search registries of known packages, as well as install and uninstall them.
Packages can provide both executables and plain files alike. Installation entails putting these files into a good location for the package manager and the user. Nix has an elegant way of storing everything under /nix/store, discussed more below.
Significantly, the Nix package manager doesn't differentiate between system- and user-level installations. All builds and installations are hermetic and can't conflict with one another.
As a convenience, Nix has tools to help users put the executables provided by packages on their environment's PATH. This way, users don't have to find executables installed in /nix/store.
Nix combines the features of a package manager with those of a build tool. If a package or any of its dependencies (including low-level system dependencies) aren't found in a Nix substituter, Nix builds them locally. Otherwise, Nix downloads pre-built package and dependencies cached in the substituter. We only need the Nix package manager and a network connection to build or download any package.
Every Nix package is specified by a Nix expression, written in a small programming language also called Nix. This expression specifies everything needed to build the package down to the system-level. These expressions are saved in files with a ".nix" extension.
Some software provides these Nix expressions as part of their source. If some software doesn't offer a Nix expression, you can always use an externally authored expression.
What makes Nix unique is that these expressions specify a way to build that's
- precise
- repeatable
- guaranteed not to conflict with anything already installed
For some, it's easy to miss the degree to which Nix-built packages are precise and repeatable. If you build a package from a Nix expression on one system and then build the same expression on a system of the same architecture, you should get the same result. In most cases, the built artifacts will be identical bit-for-bit.
A system of thorough hashing accomplishes this degree of precision. In Nix, the dependencies needed to build packages are also themselves Nix packages. Every Nix expression has an associated hash calculated from the hashes of the package's dependencies and build instructions. When we change this dependency (even if only by a single bit), the hash for the Nix expression changes. This new hash cascades to a different calculated hash for any package relying on this dependency. But if nothing changes, all systems will calculate identical hashes.
The repeatability and precision of Nix form the basis of how substituters are trusted as caching services across the world. It also allows us to trust remote builds more easily without worrying about deviations in environment configuration.
Nix has a central substituter at https://cache.nixos.org, but there are third-party ones as well, like Cachix. Before building a package, the hash for the package is calculated. If any configured substituter has a build for the hash, it's pulled down as a substitute. A certificate-based protocol is used to establish the trust of substituters. Between this protocol and the algorithm for calculating hashes in Nix, you can have confidence that a package pulled from a substituter will be identical to what you would have built locally.
Finally, all packages are stored in /nix/store by their hash. This simple scheme allows us to install multiple versions of the same package without conflicts. References to dependencies all point back to the desired version in /nix/store they need. Though Nix has not eliminated the risk of concurrently running different versions of the same program, at least the flexibility to do so is in the user's hands.
Nix expressions help us create highly controlled environments to build packages precisely. However, Nix still calls the conventional build tools of various programming language ecosystems. Under the cover, Nix is ultimately a strictly controlled execution of Bash scripts orchestrating these tools.
To keep the Nix expressions for each package concise, the Nix community curates a Git repository of Nix expressions called Nixpkgs. Most Nix expressions for packages will start with a snapshot of Nixpkgs as a dependency, which provides library support to help keep Nix expressions compact.
This way, the complexity of shell scripting and calls to language-specific tooling can be mostly kept hidden away from general Nix authors. The Nix language lets us work with packages from any language ecosystem in a uniform way.
Having covered so many of Nix's strengths, it's good to be aware of some problems the Nix community is still working through.
There are parts of Nix that are notably simple. For example, there's an elegance to the hashing calculation and how /nix/store is used. Furthermore, the Nix language has a small footprint, making learning Nix easier.
However, because of the complexity of all the programming language ecosystems, there are a lot of supporting libraries in Nixpkgs to understand. There are over two million lines of Nix in Nixpkgs, some auto-generated, increasing the odds of getting lost.
The official Nixpkgs manual only seems to cover a fraction of what package authors need to know. Invariably, people seem to master Nix by exploring the source code of Nixpkgs, supplemented by example projects for reference. You can get surprisingly far mimicking code you find in Nixpkgs that packages something similar to what you have in front of you. But understanding what's going on so you avoid simple mistakes can take some time.
Various people have attempted to fill the gap with documentation and tutorials. Even this document you're reading now is one such attempt. However, we're missing a searchable index of all the critical functions in Nixpkgs for people to explore. Something as simple as parsed docstrings as an extension of the Nix language would go a long way, which would be far easier to implement than something more involved, like a type system for the Nix language.
The Nix community seems divided into the following camps:
- those who want new features and fixes to known grievances
- those who want stable systems based on Nix in industrial settings.
These groups don't need to be at odds. Unfortunately, Nix has released experimental features in a way that has created confusion about how to build stable systems with Nix.
An early complaint of Nix was the non-intuitiveness of Nix's original assortment of command-line tools. To address this, Nix 2.0 introduced a unifying tool called nix. Despite appreciable improvements in user experience, the newer nix command has taken some time to get enough functionality to replace the older tools (nix-build, nix-shell, nix-store, etc.). For a while, it's ended up yet another tool to learn.
If you look at the manpage for the latest release of nix, there's a clear warning at the top:
Warning: This program is experimental, and its interface is subject to change.
This warning has been there since 2018, when Nix 2.0 was released.
However, nix repl is the only way to get to a REPL session in Nix, which is an important tool for any programming language. The previous tool providing a REPL (nix-repl) has been removed from Nixpkgs.
Because something as basic as the REPL is only available with an experimental feature, the Nix community is confusing guidance on using Nix with some stability.
Eventually, with the release of Nix 2.4, experimental features were turned into flags that needed to be explicitly enabled by users. One of these flags was nix-command, which now gates users from any subcommand of nix beyond nix repl. However, because so many users already use the new nix command, the experimental nix-command feature is enabled by default if no experimental features have been configured explicitly.
In other words, Nix ships with an experimental feature enabled by default.
Enabling the new nix command by default almost indicates it isn't too unstable. However, Nix 2.4 did indeed change the API of nix subcommands. Industrial users scripting against nix had to figure out the appropriate changes.
In practice, the nix subcommands are relatively reliable. They are well-written and functionally robust. However, the core maintainers reserve the right to change input parameterization and output formatting without bumping a major version number.
They communicate this risk only with the warning atop the manpage, which most users have been training one another to ignore.
Though Nix expressions have an incredible potential to be precise and reproducible, there have always been some backdoors to break the reliability of builds. For example, Nix expressions have the potential to evaluate differently depending on the setting of some environment variables like NIX_PATH.
The motivation for these relaxations of determinism has been a quick way to let personal computing users have a convenient way to manage their environments. Some people are careful to avoid accidentally having non-deterministic builds. Still, accidents have occurred frequently enough for the community to want better. It's frustrating to have a broken build because someone else set an environment variable incorrectly.
Nix 2.4 corrected this by introducing an experimental feature called flakes. Users still have a primarily ergonomic way to manage their environments, but builds are more strictly deterministic. Determinism is a significant reason many turn to Nix in the first place. A nice benefit of strictly enforced determinism is the ability to cache evaluations of Nix expressions, which can be expensive to compute.
All this is generally good news. Flakes address problems that industrial users of Nix have long had to deal with.
However, flakes are an experimental feature that users need to enable explicitly. Similar to the nix command, the inputs and outputs of flake-related subcommands might change slightly. Such changes have already happened.
On top of this, because flakes are experimental, documentation of flakes is fractured in the official documentation. It almost seems like the Nix developers are delaying proper documentation until there's a declaration of stability. A preferred alternative would be developing documentation concurrently with the implementation, using the documentation's comprehensibility to inform the software's design. Good opportunities for redesign can be found in features that prove difficult to explain.
All this puts industrial Nix users in an annoying place. Not using flakes and instead of coaching coworkers and customers on how to use Nix safely
- increases the likelihood of defects as people make honest mistakes
- reduces the likelihood of adoption because people get frustrated with poor ergonomics and difficulty understanding nuances and corner cases.
However, if industrial users move to flakes to address these problems, we have the following problems:
- we have to be ready for the flakes API to change, as it's technically experimental
- we have to accept some added training hurdles since the documentation of flakes is tucked behind documentation of non-flake usage.
This project encourages the development of Nix projects using flakes. The benefits seem to outweigh the risks of instability. This choice is not made lightly, and this document is an exercise of due diligence to inform users of compromises.
Flakes are the future in Nix. They significantly address prior pains. Furthermore, enough people worldwide are using them that we have some confidence that the Nix commands are reliable as implemented. APIs have not changed too frequently, and changes don't invalidate anything fundamental, like data stored in /nix/store.
There might be documentation and training hurdles with flakes, but it's not much better not to use flakes.
Usage of both the new nix command and flakes has almost no risk beyond slight breakage of some scripts. By calling nix with a few extra arguments --extra-experimental-features 'nix-command flakes' we can access flakes commands for single invocations without enabling flakes globally. You can even make an alias for your shell that might look like the following:
alias nix-flakes = nix --extra-experimental-features 'nix-command flakes'This way, there's less to type interactively. Just don't script against this command, and there's no worry of scripts breaking if the flakes API changes.
A few users work in organizations or contribute to projects that disallow experimental features such as flakes.
For these users, this project uses and encourages the use of the flake-compat project, which enables an end user who has opted not to enable flakes to at least access the flake's contents, packages or otherwise.
With flake-compat, end users will have a regular (non-flake) Nix expression they can evaluate. However, since dependencies are managed with flakes, the project maintainer must have flakes enabled to manage dependencies (for example, updating to the latest dependencies with nix flake update).
To deal with the transition of the Nix community to flake, this project provides two user guides:
Links generally steer users to the recommended guide, which then links users to the non-flakes guide if they have the interest or need.
The non-flakes guide intentionally avoids commands like nix-shell and nix-channel. These commands lead users to set the NIX_PATH environment variable, which can lead to unreliable builds. These are the pitfalls that motivated the design of flakes.
Though this non-flakes guide avoids the flakes experimental feature, it still invites end users to use the experimental nix-command to get the following subcommands:
nix searchnix shellnix run
In general, the non-flakes guide only explains the usage of experimental nix subcommands when there exist no other alternatives or when the alternatives are considered worse for new users.
For example, nix search has no alternative within the set of non-experimental Nix tools, and it's too helpful not to tell users about it. Again, this is an example of the Nix community leading users to experimental features.
Additionally, nix shell and nix run are shown as improved alternatives to nix-shell. nix-shell is a complicated tool that has been historically used for a lot of different purposes:
- debugging the build environments of packages
- creating a developer environment for a package (
nix developdoes this better, but only for flakes) - entering a shell with Nix-build executables on the path (
nix shelldoes this better) - running arbitrary commands with Nix-build executables on the path (
nix rundoes this better)
To cover all of these scenarios, nix-shell became so complex it is hard to explain to new users. nix-shell is only best for debugging builds, which is beyond the scope of the documentation provided by this project.