From 02e3126a26fd72cdf11bc727f157643afc12c5f4 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Mon, 12 Dec 2022 17:54:11 +0100 Subject: [PATCH 01/48] RFC 137: Set Nix language version in magic comment --- rfcs/0137-nix-language-version.md | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 rfcs/0137-nix-language-version.md diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md new file mode 100644 index 000000000..de57e6988 --- /dev/null +++ b/rfcs/0137-nix-language-version.md @@ -0,0 +1,85 @@ +--- +feature: nix-language-version +start-date: 2022-12-12 +author: @fricklerhandwerk +co-authors: @thufschmitt @Ericson2314 +shepherd-team: +shepherd-leader: +related-issues: https://github.com/NixOS/nix/issues/7255 +--- + +# Summary +[summary]: #summary + +Add a mechanism to determine which version of the Nix language grammar to use for parsing Nix files. + +# Motivation +[motivation]: #motivation + +Currently it's impossible to introduce backwards-incompatible changes to the Nix language without breaking existing setups. +This proposal is an attempt to overcome that limitation. + +# Detailed design +[design]: #detailed-design + +Introduce a magic comment in the first line of a Nix file: + + #? Nix + +where `` is a released version of Nix the given file is known to work with. + +It is left to the implementation how to use this information. + +# Examples and Interactions +[examples-and-interactions]: #examples-and-interactions + +```nix +#? Nix 2.12 +"nothing" +``` + +# Advantages +[advantages]: #advantages + +- Backwards compatible +- Visually unintrusive +- Self-describing and human-readable +- Opt-in until the feature is implemented in Nix *and* the first backwards-incompatible change to the language is introduced. +- Follows a well-known convention of using [magic numbers in files](https://en.m.wikipedia.org/wiki/Magic_number_(programming)#In_files) +- Encourages, but does not require the Nix evaluator to deal with backwards-incompatible changes in a principled manner. + +# Drawbacks +[drawbacks]: #drawbacks + +- The syntax of the magic comment is arbitrary. +- At least one form of comment is forever bound to begin with `#` to maintain compatibility. +- There is a chance of abusing the magic comment for more metadata in the future. + Let's avoid that. +- It requires effort to implement an appropriate system to make use of the version information. + +# Alternatives +[alternatives]: #alternatives + +- Never introduce backwards-incompatible changes to the language. + + This is what has been happening so far, and required additions to be made very carefully. + +- Use a different versioning scheme for the Nix language that is decoupled from the rest of Nix. + + Although this can be done at any point in the future, because it's the evaluator that will read this information. + +- Use a magic string that is incompatible with evaluators prior to the feature, e.g. `%? Nix `. + + This would make clear to users that the file is not intended to be used without explicit handling of compatibility. + Such a breaking change could be reserved for later iterations of how Nix encodes language version information. + +# Unresolved questions +[unresolved]: #unresolved-questions + +- Is the proposed magic number already in use in [other file formats](https://en.m.wikipedia.org/wiki/Magic_number_(programming)#In_files)? +- Should we allow multiple known-good versions in one line? + +# Future work +[future]: #future-work + +Determine what exaclty to do with the information given in the magic comment. From bb73166eb1a7767c88a0e27969717657635a0dcd Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Dec 2022 09:35:07 +0100 Subject: [PATCH 02/48] rewrite arguments to be more precise --- rfcs/0137-nix-language-version.md | 61 +++++++++++++++++-------------- 1 file changed, 34 insertions(+), 27 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index de57e6988..ad022d8ae 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -11,13 +11,13 @@ related-issues: https://github.com/NixOS/nix/issues/7255 # Summary [summary]: #summary -Add a mechanism to determine which version of the Nix language grammar to use for parsing Nix files. +Introduce a convention to determine which version of the Nix language grammar to use for parsing a Nix file. # Motivation [motivation]: #motivation Currently it's impossible to introduce backwards-incompatible changes to the Nix language without breaking existing setups. -This proposal is an attempt to overcome that limitation. +This proposal is first step towards overcoming that limitation. # Detailed design [design]: #detailed-design @@ -26,9 +26,7 @@ Introduce a magic comment in the first line of a Nix file: #? Nix -where `` is a released version of Nix the given file is known to work with. - -It is left to the implementation how to use this information. +where `` is a released version of Nix the given file is intended to work with. # Examples and Interactions [examples-and-interactions]: #examples-and-interactions @@ -38,40 +36,49 @@ It is left to the implementation how to use this information. "nothing" ``` -# Advantages +# Arguments [advantages]: #advantages -- Backwards compatible -- Visually unintrusive -- Self-describing and human-readable -- Opt-in until the feature is implemented in Nix *and* the first backwards-incompatible change to the language is introduced. -- Follows a well-known convention of using [magic numbers in files](https://en.m.wikipedia.org/wiki/Magic_number_(programming)#In_files) -- Encourages, but does not require the Nix evaluator to deal with backwards-incompatible changes in a principled manner. - -# Drawbacks -[drawbacks]: #drawbacks - -- The syntax of the magic comment is arbitrary. -- At least one form of comment is forever bound to begin with `#` to maintain compatibility. -- There is a chance of abusing the magic comment for more metadata in the future. - Let's avoid that. -- It requires effort to implement an appropriate system to make use of the version information. +* (+) Encourages developing Nix to deal with changes to the Nix language in a principled manner. +* (+) Backwards-compatible + * (+) Allows for gradual adoption: opt-in until semantics is implemented in Nix *and* the first backwards-incompatible change to the language is introduced. +* (+) Visually unintrusive +* (+) Self-describing and human-readable +* (+) Follows a well-known convention of using [magic numbers in files](https://en.m.wikipedia.org/wiki/Magic_number_(programming)#In_files) +* (-) The syntax of the magic comment is arbitrary. +* (-) There is a chance of abusing the magic comment for more metadata in the future. Let's avoid that. +* (-) At least one form of comment is forever bound to begin with `#` to maintain compatibility. +* (-) It requires significant additional effort to implement and maintain an appropriate system to make use of the version information. # Alternatives [alternatives]: #alternatives - Never introduce backwards-incompatible changes to the language. - This is what has been happening so far, and required additions to be made very carefully. + * (+) No additional effort required. + * (-) Requires additions to be made very carefully. + * (-) Makes solving some well-known problems impossible. + +- Use the output of [`builtins.langVersion`] for specifying the version of the Nix language. -- Use a different versioning scheme for the Nix language that is decoupled from the rest of Nix. + * (-) `builtins.langVersion` is currently only internal and undocumented. + * (-) Requires adding another built-in or command line option to the public API. + * (-) Using a language feature requires an additional steps from users to determine the current version. + * (+) We can add a command line option such that it is not more effort than `nix --version`. + * (+) The Nix language version is decoupled Nix version numbering. + * (+) It changes less often than the Nix version. + * (-) That was probably due to making changes being so hard. + * (-) There are two version numbers to keep track of. + * (-) The magic comment should reflect that it's specifying the *Nix language* version, which would make it longer. + * (-) Renaming the Nix language will be impossible once the mechanism is part of stable Nix. - Although this can be done at any point in the future, because it's the evaluator that will read this information. +[`builtins.langVersion`]: https://github.com/NixOS/nix/blob/26c7602c390f8c511f326785b570918b2f468892/src/libexpr/primops.cc#L3952-L3957 - Use a magic string that is incompatible with evaluators prior to the feature, e.g. `%? Nix `. - This would make clear to users that the file is not intended to be used without explicit handling of compatibility. - Such a breaking change could be reserved for later iterations of how Nix encodes language version information. + * (+) Makes clear that the file is not intended to be used without explicit handling of compatibility. + * (-) Cannot be introduced gradually. + * (+) Such a breaking change could also be reserved for later iterations of the Nix language. # Unresolved questions [unresolved]: #unresolved-questions @@ -82,4 +89,4 @@ It is left to the implementation how to use this information. # Future work [future]: #future-work -Determine what exaclty to do with the information given in the magic comment. +Define semantics, that is, what exactly to do with the information given in the magic comment. From c6d692c6ebd94b010ce7b20e70bdfd9b1275664d Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Dec 2022 11:36:51 +0100 Subject: [PATCH 03/48] using Nix language version istead would help third-party evaluators --- rfcs/0137-nix-language-version.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index ad022d8ae..45b16c4c1 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -61,7 +61,9 @@ where `` is a released version of Nix the given file is intended to wor - Use the output of [`builtins.langVersion`] for specifying the version of the Nix language. + * (+) This would serve other Nix language evaluators which are not and should not be tied to the rest of Nix. * (-) `builtins.langVersion` is currently only internal and undocumented. + * (+) Documentation is easy to add. * (-) Requires adding another built-in or command line option to the public API. * (-) Using a language feature requires an additional steps from users to determine the current version. * (+) We can add a command line option such that it is not more effort than `nix --version`. From 4265b085ab0aba6e6f55ce9b20882e7bd474f56f Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Dec 2022 11:46:07 +0100 Subject: [PATCH 04/48] remove non-argument about encouraging a principled approach the convention indeed does not encourage discipline, as @andir points out[1], and at the worst would encourage haphazard changes because it makes them appear harmless, as @tazjin adds [2]. [1]: https://github.com/NixOS/rfcs/pull/137/files#r1050542977 [2]: https://github.com/NixOS/rfcs/pull/137/files#r1050580213 --- rfcs/0137-nix-language-version.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 45b16c4c1..c49cb7f1a 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -39,12 +39,15 @@ where `` is a released version of Nix the given file is intended to wor # Arguments [advantages]: #advantages -* (+) Encourages developing Nix to deal with changes to the Nix language in a principled manner. +* (+) Makes explicit what can be expected to work. +* (+) Enables communicating language changes systematically. * (+) Backwards-compatible * (+) Allows for gradual adoption: opt-in until semantics is implemented in Nix *and* the first backwards-incompatible change to the language is introduced. * (+) Visually unintrusive * (+) Self-describing and human-readable * (+) Follows a well-known convention of using [magic numbers in files](https://en.m.wikipedia.org/wiki/Magic_number_(programming)#In_files) +* (-) May make the appearance that changing the language is harmless. + * (+) The convention itself is harmless and independent of the development culture around the language. * (-) The syntax of the magic comment is arbitrary. * (-) There is a chance of abusing the magic comment for more metadata in the future. Let's avoid that. * (-) At least one form of comment is forever bound to begin with `#` to maintain compatibility. @@ -91,4 +94,5 @@ where `` is a released version of Nix the given file is intended to wor # Future work [future]: #future-work -Define semantics, that is, what exactly to do with the information given in the magic comment. +- Define semantics, that is, what exactly to do with the information given in the magic comment. +- Define rules deciding when a change to the language is appropriate to avoid proliferation and limit complexity of implementation. From 288c7f1fa969a5050229813b92d05be063e83198 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Dec 2022 12:52:15 +0100 Subject: [PATCH 05/48] add prior art as suggested by @piegamesde --- rfcs/0137-nix-language-version.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index c49cb7f1a..42b79e92c 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -85,6 +85,20 @@ where `` is a released version of Nix the given file is intended to wor * (-) Cannot be introduced gradually. * (+) Such a breaking change could also be reserved for later iterations of the Nix language. +# Prior art + +- [Rust `edition` field] + + Rust has an easier problem to solve. Cargo files are written in TOML, so the `edition` information does not have to be part of Rust itself. + +- [Flakes `edition` field] + + There had been an attempt to include an `edition` field into the Flakes schema. + It did not solve the problem of having to evaluate the Nix expression using *some* version of the grammar. + +[Rust `edition` field]: https://doc.rust-lang.org/cargo/reference/manifest.html#the-edition-field +[Flakes `edition` field]: https://discourse.nixos.org/t/nix-2-8-0-released/18714/6 + # Unresolved questions [unresolved]: #unresolved-questions From 9afffbefff8d44779afbcddb949e871d1526520d Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Dec 2022 12:53:22 +0100 Subject: [PATCH 06/48] reorder arguments to better capture their relation --- rfcs/0137-nix-language-version.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 42b79e92c..1dad1251d 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -67,9 +67,10 @@ where `` is a released version of Nix the given file is intended to wor * (+) This would serve other Nix language evaluators which are not and should not be tied to the rest of Nix. * (-) `builtins.langVersion` is currently only internal and undocumented. * (+) Documentation is easy to add. - * (-) Requires adding another built-in or command line option to the public API. + * (-) Requires adding another built-in to the public API. * (-) Using a language feature requires an additional steps from users to determine the current version. * (+) We can add a command line option such that it is not more effort than `nix --version`. + * (-) Requires adding another command line option to the public API. * (+) The Nix language version is decoupled Nix version numbering. * (+) It changes less often than the Nix version. * (-) That was probably due to making changes being so hard. From 32436e5c313f914aeee5da89a757d3f9e40d3c16 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Dec 2022 15:38:35 +0100 Subject: [PATCH 07/48] Update rfcs/0137-nix-language-version.md --- rfcs/0137-nix-language-version.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 1dad1251d..eb1bfc07d 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -28,6 +28,8 @@ Introduce a magic comment in the first line of a Nix file: where `` is a released version of Nix the given file is intended to work with. +Add an appropriate parameter to commands that allow encoding the same information where where files are not involved. + # Examples and Interactions [examples-and-interactions]: #examples-and-interactions From 37cd0bb585987cd056c9fea92377939ddd086452 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 16 Dec 2022 16:52:04 +0100 Subject: [PATCH 08/48] assume a fixed version when none is specified as suggested by @infinisil and @kevincox --- rfcs/0137-nix-language-version.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index eb1bfc07d..2d15b4972 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -28,6 +28,8 @@ Introduce a magic comment in the first line of a Nix file: where `` is a released version of Nix the given file is intended to work with. +If no magic comment is present, Nix 2.12 – the stable release at the time this RFC was accepted – is assumed. + Add an appropriate parameter to commands that allow encoding the same information where where files are not involved. # Examples and Interactions From e1ed3be6831b7c1fa350a86ae5f2692202d9c79f Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 4 May 2023 14:30:47 +0200 Subject: [PATCH 09/48] answer questions and incorporate suggestions reworked the proposal to add semantics and combine it with related efforts to provide a systematic way of issuing deprecation warnings. --- rfcs/0137-nix-language-version.md | 672 ++++++++++++++++++++++++++---- 1 file changed, 594 insertions(+), 78 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 2d15b4972..55de65651 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -1,117 +1,633 @@ --- feature: nix-language-version start-date: 2022-12-12 -author: @fricklerhandwerk -co-authors: @thufschmitt @Ericson2314 -shepherd-team: -shepherd-leader: +author: @fricklerhandwerk @yorickvp +co-authors: @thufschmitt @Ericson2314 @infinisil +shepherd-team: +shepherd-leader: related-issues: https://github.com/NixOS/nix/issues/7255 --- +# RFC 137 – Nix language versioning + # Summary [summary]: #summary -Introduce a convention to determine which version of the Nix language grammar to use for parsing a Nix file. +Introduce a convention to determine which version of the Nix language grammar to use for parsing and evaluating Nix expressions. + +Add parameters to the Nix language evaluator, controlling the behavior of deprecation warnings and errors. # Motivation [motivation]: #motivation -Currently it's impossible to introduce backwards-incompatible changes to the Nix language without breaking existing setups. -This proposal is first step towards overcoming that limitation. +The stability of Nix language has been praised on multiple occasions, e.g. [Nix and legacy enterprise software development: an unlikely match made in heaven](https://talks.nixcon.org/nixcon-2022/talk/QQPBFW/). +Yet, as with any software system, in order to accommodate new insights, we want to allow the Nix language to evolve. +This sometimes involves backward-incompatible ("breaking") changes that currently cannot be made without significant downstream disruption. + +Therefore we propose a mechanims and policies to introduce changes to the Nix language in a controlled and deliberate manner. +It aims to avoid breaking existing setups, and to minimise maintenance burden for implementors and users. +The goal is for new versions of the Nix language evaluator to stay backward compatible with existing Nix expressions, but not necessarily for new Nix expressions to stay forward compatible with existing evaluators. + +Regardless, changes to the language, especially breaking changes, should remain a rare exception. + +## Motivating examples + +Incompatible changes from the past: + +- [A changelog of Nix (language) versions](https://code.tvl.fyi/about/tvix/docs/lang-version.md), as reflected in `builtins.langVersion` +- There have been other, sometimes breaking changes to the language that have not resulted in an increment of the language version (e.g. the recent `fetchGit` changes). +- The `builtins.toJSON 1.000001` output [changed in Nix 2.12](https://github.com/NixOS/nix/issues/8259). + +Possbile future changes that are in discussion: + +- Remove URL literals (currently implemented via experimental-features) +- Remove the old `let { body = ... }` syntax +- Disallow leading zeroes in integer literals (such as `umask = 0022`) +- Disallow `a.x or y` if `a` is not an attribute set +- Simplifying semantics of `builtins.toString` and string interpolation +- Remove the `let { body }` syntax +- `__functor` and `__toString`, probably +- Remove `__overrides` +- [Make `builtins` more consistent](https://github.com/NixOS/nix/issues/7290), e.g. not exposing `map`, `removeAttrs`, `fetchGit` and and others in the global scope +- Fix [imprecision in string representation of floating point numbers](https://github.com/NixOS/nix/pull/6238) +- Make the `@`-pattern consistent with destructuring +- A syntax to index into lists, e.g. `[ 1 2 3 ].0 == 1` +- Use `,` to delimit elements of a list expression, like `[ 1, 2, 3 ]` +- Do something about `?` meaning two different things depending on where it occurs (`{ x ? "" }:` vs `x ? ""`) +- Better support for static analysis +- [Syntax for hexadecimal numbers](https://github.com/NixOS/nix/pull/7695) + +Other discussions around language changes: + +- [Make path semantics less surprsing](https://github.com/NixOS/nix/issues/7338) +- [RFC 110](https://github.com/NixOS/rfcs/pull/110) proposing a alternatives to the `with` expression +- [Nix 2 – a hypothetical syntax](https://md.darmstadt.ccc.de/s/nix2) +- [Nix language changes](https://gist.github.com/edolstra/29ce9d8ea399b703a7023073b0dbc00d) + +## Alternatives + +- Keep the language as implemented by Nix compatible, but socially restrict the usage of undesirable features. + - (+) Roughly matches the current practice, no technical change needed + - (+) Maintains usability of old Nixpkgs versions (up to availability of fixed-output artifacts) + - (+) Does not break third-party codebases before making a decision, keeping Nix a dependable upstream + - (-) This proposal does not allow for breakages unless there is some eventual phase-out of support + - (-) Strict enforcement requires extra tooling that this proposal would obviate + - (-) The implementation of the features that are no longer desirable still incur complexity and maintenance cost + - (-) It's still not really possible to make changes to the language + +- Introduce changes to the language with language extensions or feature flags + - (-) Combinatorial explosion + - See [Haskell language extensions] for real-world experience + - (-) Even more maintenance overhead + - (+) Allows gradual adoption of features + - (-) We already have experimental feature flags as an orthogonal mechanism, with the added benefit that they don't incur support costs and can be dropped without loss + +- Never make breaking changes to the language + - (+) No additional maintenance effort required + - (-) Blocks improvements + - (-) Requires additions to be made very carefully + - (-) Even incremental changes are really expensive that way + - (-) Makes solving some well-known problems impossible + +- Continue current practice + - (-) There is no process for breaking changes + - (-) Breaking changes are not always announced + - (-) There are no means of determining compatibility between expressions and evaluator versions + +# Detailed Design + +## Language versioning + +1. The language version is a natural number. + +
Arguments + + - (+) Formally decouples the Nix language version from the Nix version + - (+) The Nix language is supposed to change much less often than the rest of Nix + - (-) There are two version numbers to keep track of + - (-) Makes more evident that the Nix language is a distinct architectural component of the Nix ecosystem + - (+) It's currently handled that way, no change needed apart from documentation + - See [`builtins.langVersion`] (currently undocumented) + - (+) Simple and unambiguous + - (+) Concise, even in the long term, since the language is supposed to change very rarely + + [`builtins.langVersion`]: https://github.com/NixOS/nix/blob/26c7602c390f8c511f326785b570918b2f468892/src/libexpr/primops.cc#L3952-L3957 + +
+ +
Alternatives + + - [Calendar Versioning](https://calver.org/) + - (+) Provides information on when changes happened + - (-) This is not needed because only compatibility information is needed + - (-) Requires a minimum amount of characters + - This may be relevant depending on where it has to be encoded + - (+) Restricting to only the year would force language changes to be rare + - (+) This would allow obvious synchronisation points with Nixpkgs releases + - (-) It may be too much policy encoded in a mechanism + - [Semantic Versioning](https://semver.org/) + - (+) Can distinguish additions from other changes + - (-) This is not needed for our use case, since any addition to an expression will break for older evaluators even if the major version matches + - (-) Requires more characters to account for the added expressiveness + - This may be relevant depending on where it has to be encoded + - Use version numbers of Nix stable releases for specifying the version of the Nix language + - (+) More obvious to see for users what the current Nix version is rather than `builtins.langVersion` + - (-) Would tie alternative Nix language evaluators to the rest of Nix + - (-) One can add a command line option such that it is not more effort than `nix --version` + - (+) That requires adding another built-in to the public API + - (-) Using a language feature requires an additional steps from users to determine the current version + - (-) Requires adding another command line option to the public API + - (+) The Nix language version is decoupled Nix version numbering + - (+) It changes less often than the Nix version + - (-) That was probably due to making changes being so hard + - (+) The language changing slowly is a desirable property for wider adoption + - (-) There are two version numbers to keep track of + +
+ +1. The language version for Nix files is denoted in the file extension. + +
Arguments + + - (+) Sidesteps misinterpretation keeping metadata out of the actual data + - (-) Will introduce a proliferation of syntax highlighters and other tooling + - (+) this is correct though, because a syntax highligher has to know the syntax if it changes with the language version + - (+) Makes accidental mixing of versions impossible + - Have to specify the file extension when importing a file + - (-) have to rename all files in a project to change the version + - (-) Makes filenames longer, introduces visual noise + - This is the cost of being explicit + - (-) `default.nix` resolving needs specification: + - If for `import ./foo`, all of `./foo/default.nix{6,7,8}` exist, pick the one matching the version used by the evaluator, otherwise fail + - Then you'd have to specify a file using a different version explicitly + +
+ +
Alternatives + + - Use a magic comment at the beginning of the file + - (+) Makes explicit what can be expected to work + - (+) Enables communicating language changes systematically + - (+) Backwards-compatible + - (+) Allows for gradual adoption: opt-in until semantics is implemented in Nix *and* the first backwards-incompatible change to the language is introduced + - (+) Visually unintrusive + - (+) Self-describing and human-readable + - (+) Follows a well-known convention of using [magic numbers in files](https://en.m.wikipedia.org/wiki/Magic_number_(programming)#In_files) + - (-) May make the appearance that changing the language is harmless + - (+) The convention itself is harmless and independent of the development culture around the language + - (-) The syntax of the magic comment is arbitrary + - (-) There is a chance of abusing the magic comment for more metadata in the future. Let's avoid that. + - (-) At least one form of comment is forever bound to begin with `#` to maintain compatibility + - (-) Editor support is made harder, since it requires switching the language based on file contents + - (-) Requires significant additional effort to implement and maintain an appropriate system to make use of the version information + - (-) Raises the question if the new syntax should be a breaking change to the language + - Use a magic string that is incompatible with evaluators prior to the feature, e.g. `%? Nix ` + - (+) Makes clear that the file is not intended to be used without explicit handling of compatibility + - (-) Cannot be introduced gradually + - (+) Such a breaking change could also be reserved for later iterations of the Nix language + + - Language versioning per "project" + - (+) This would easily allow inheriting the language version across imports (solving a lot of issues in this proposal) + - (-) There is currently no notion of "project" + - (-) Attempting to establish one would be a large undertaking and not immediately help solving the problem at hand + - (-) It would require a separate language to encode project metadata such as the language version + - The `edition` field was [removed from the flakes schema](https://github.com/NixOS/nix/commit/e5ea01c1a8bbd328dcc576928bf3e4271cb55399) for that reason, as it not not allow distinguishing data from metadata + - (+) Other languages do the same (Python, Haskell, Rust, JavaScript, ...) + - (-) Recursive (albeit smaller) problem of managing the additional langauge for project metadata + +
+ +1. The file extension consists of the language version followed by `.nix`. + Example: + + ``` + default.7.nix + ``` + +
Arguments + + - (+) Consistent with the convention of file extensions carrying soft meaning as metadata + – (-) A conflicting convention is for denoting nested file types, such as `tar.gz` + - (+) Visually least intrusive + +
+ +
Alternatives + + - No separator + – (-) Hard to discern visually + - `^` + – (-) Overlaps with derivation output syntax + - `-` + - (+) Visually not intrusive + - `_` + – (-) Visually more intrusive + - All of the following characters will interfere with some tooling: + - `!` - shells + - `"` - shells + - `#` - URLs + - `$` - shells + - `%` - URLs + - `&` - shells, URLs + - `+` - URLs + - `,` - natural language + - `/` - paths, URLs + - `:` - URLs + - `;` - shells + - `=` - URLs, Nix language + - `?` - URLs + - `@` - URLs, Nix language + - `\` - Windows paths + +
+ +1. The language version for bare Nix expressions is specified with a parameter to the evaluator. + +1. If no language version is specified in the file name, assume version 6. + This is the version implemented in the stable release of Nix at the time of writing this RFC. + +
Arguments + + - (+) Backwards compatible with existing evaluators + - (+) Does not require changing any existing code + +
+ +
Alternatives + + - Assume the evaluator's current version + - (-) When the evaluator advances in language version, evaluation may fail on existing code + - (-) Defeats the purpose of explicit versioning: Which evaluator to use for a given file is left unspecified + - (-) Following the latest evaluator version may inadvertently break the code for older evaluators + - (+) Don't have to look up the latest version of the Nix language when writing code + - (+) Does not clutter the file names for what is supposed to be the latest version of the code + +
+ +1. If no version is specified when invoking the evaluator on a bare Nix expression, assume the most recent language version supported by the evaluator. + +1. Language versions prior to 6 are not supported. + +
Arguments + + - (+) Does not require additional development effort + - (+) Prior langauge versions are not fully supported by current code already, and the rest of this proposal argues to deprecate old versions in the future in order to keep the implementation manageable + - (-) Legacy code will not get support for managing compatibility + +
+ +1. The Nix language evaluator provides a command to output the latest Nix language version. + This command provides options to list all Nix language versions supported by the evaluator. + +
Arguments + + - (+) This is for convenience to determine which features are available + +
+ +1. `builtins.langVersion` returns the language version used for evaluating the given expression. + +
Arguments + + - (+) Could make use of it for generating Nix expressions programmatically and annotating them with the correct version + - (+) `builtins.langVersion` is already part of the stable interface, this way we can make it pure + - (-) Requires maintaining more API surface without a clear use case + +
+ +
Alternatives + + - Return the latest language version instead + + - (-) Doesn't help to determine what is used for evaluating the current expression + - (+) Might provide opportunities for forward-compatible Nix code + - (-) Brittle and defeats the purpose of this RFC + - (-) Impure, the value depends on the environment + + - Don't expose `builtins.langVersion` at all + + - (+) No need to have this if the Nix files are versioned + - (+) `builtins.langVersion` is not documented and not widely used + - (-) Requires a Nix language version bump to implement this RFC + +
+ +1. Each time the language specification (currently as embodied by the Nix language evaluator) is changed, the language version must be incremented. + +
Arguments + + - (+) The principled solution: guarantees reproducibility given a fixed language version + - (-) Implies additional overhead in development effort: + - Either for Nix maintainers to accommodate that practice in the release lifecycle + - For example, one would have to batch language changes for a version bump to limit the number of increments + - (+) This would be beneficial for alternative implementations in terms of churn and effort to keep up + - Or implementors of alternative evaluators catching up with changes + - (+) Specifying the language precisely via the version actually offers alternative implementations an alternative to catching up: only support a given language version + - (+) This essentially nudges one to organise Nix language (specification or evaluator implementation ) development to be more independent of the rest of Nix + This is good, since it in turn forces stronger separation of concerns and more architectural clarity + - (-) Prohibits best-effort attempts at evaluating expressions with possibly incompatible evaluators + - (+) With the proposed level of strictness, one doesn't have to rely on best effort but can instead be explicit + - (-) Fixing evaluator bugs (i.e., clearly unintended behavior) after releases would technically require a version bump and therefore (theoretically) cooperation by expression authors + - This could be communicated with an increment in the Nix patch-level version, as is already practice + +
+ +
Alternatives + + - Bump version only when evaluation result on prior version evaluators would be *substantially* different + - (+) Leaves room for judgement by developers + - (+) Allows controlling progression of versions to some degree + - (-) Conversely, leaves room for sneaking in breaking changes unannounced + - (-) This loses compatibility guarantees we'd get from a stricter paradigm + - (-) Deprives expression authors of ability to be selective with evaluator versions + - (-) Due to hashing this is often not much different from taking *any* change into account + - Bump version when prior evaluators would fail + - (-) Derivation hashes may change between evaluator versions if different evaluation results were allowed within one version + - This amounts to giving up on specifying the language exhaustively using a version label + - (+) Fewer version increments or precautions required given current development practice + +
-# Detailed design -[design]: #detailed-design +1. Whenever Nix drops support for evaluating prior language versions, a major version bump is required. -Introduce a magic comment in the first line of a Nix file: + Example: Assuming the current language version is 8, the Nix release version is 2.20, and support for language version 6 is dropped, the next Nix release must be version 3.0 - #? Nix +
Arguments -where `` is a released version of Nix the given file is intended to work with. + - (+) This enforces that existing code that works will not break inadvertently when upgrading Nix -If no magic comment is present, Nix 2.12 – the stable release at the time this RFC was accepted – is assumed. +
-Add an appropriate parameter to commands that allow encoding the same information where where files are not involved. +1. Semantics are preserved across file boundaries for past language versions. + + Example: [Best-effort interoperability](#best-effort-interoperability) + +
Arguments + + - (+) This allow for incremental evolution of code bases without having to change existing code at all. + - (+) Expressions that are valid today will still be valid as long as a given evaluator supports the language version they are written in + - (+) The root of evaluation is always at a language version determined by the user, and authors of expressions in newer language versions are responsible (and made aware of the fact by the file name signaling proposed here) to interoperate with existing code + - (-) Passing around incompatible values (e.g. builtins or data types) between language versions can lead to surprising errors + - (+) We can postpone dealing with particular issues as they arise, but the general setup should support most cases on a best(-and-minimal)-effort basis + - (+) In any case, such breakages will only happen when adding new code, and will never break existing code + +
+ +
Alternatives + + - Do not support version interoperability at all + - (+) Avoids any unforseen issues at no cost + - (-) Prohibits incremental changes, as any language update will require updating all files involved + - Do not support passing values to functions from older language versions + - (-) Calling functions is the most common use case, and you can hardly do anything without it + +
+ +1. It is not possible to import expressions written in newer versions. + + Example: [Expressions are not forward compatible](#expressions-are-not-forward-compatible) + +1. The Nix evaluator provides options to issue deprecation warnings and errors against a language version newer than the one under evaluation. + A detailed design is provided in the next section. + +
Arguments + + - (+) This allows systematic migration of existing code written for prior evaluators to the most recent language version + - (+) It will notify users about what's going on instead of just breaking +
+ +## Deprecation warnings and errors + +1. Each language construct to deprecate relative to a prior version is given a symbolic name. + There is a way to refer to all language constructs. + + Examples: `url-literal`, `let-body`, `int-leading-zeros` + + These symbolic names must not be reused in future versions. + Names for experimental language constructs of prior versions can be reused. + +
Arguments + + - (+) Disallowing reuse precludes dealing with change of meaning across versions + - (+) Not considering experimental features simplifies their handling and is not required by them being exempt from compatibility guarantees. + +
+ +1. The following options for issuing warning and errors are supported: + + - Default: + - Issue warnings on deprecated language constructs without considering imported expressions written in prior versions + - Don't warn (selection): + - Do not issue warnings for selected language constructs + - Errors instead of warnings (selection): + - Throw an error instead of a warning for selected language constructs + - The error setting overrides the warning setting + - Recursive (version bound): + - Issue warnings or errors on imported expressions written in prior versions (higher or equal than the version bound) + - Verbose mode. During evaluation: + - In non-verbose mode, issue a message once for each deprecated construct + - In verbose mode, issue a message for each occurrence + + For example, this can be exposed as the following flags: + + ``` + --lang-no-warn=all + --lang-no-warn="url-literal let-body int-leading-zeroes non-attr-select" + --lang-error=all + --lang-error="url-literal let-body int-leading-zeroes non-attr-select" + --lang-warn-recursive=6 + --lang-warn-verbose + ``` + + The naming may need some bikeshedding. For example, one could use the same syntax as with C-compilers (probably not though): + + - `-Wall` + - `-Wno-` + - `-Werror=` + +
Arguments + + - (+) Disallowing reuse avoids dealing with change of meaning across versions + - (+) Not considering experimental features simplifies their handling and is not required by them being exempt from compatibility guarantees. + - (-) Maintenance burden for everyone using these old constructs, or evaluating old revisions of Nixpkgs. + +
+ +
Alternatives + + - Opt-in warnings + - (-) Can't really make breaking changes as people won't be warned ahead of time + - No warnings, just errors + - (-) Doesn't offer a transition window to users + - (+) Easier to implement + +
+ +1. A the end of the evaluation, print statistics and explanations. + The specifics of displaying warnings and errors is up to implementation, but should include the symbolic name of the langauge construct in question. # Examples and Interactions [examples-and-interactions]: #examples-and-interactions +## Show the current Nix language version + +```console +nix --language-version +7 +``` + +```console +nix --supported-language-versions +6 +7 +``` + +## Version interoperability + +### Versioned built-ins can be passed across file boundaries + +```nix +# a.6.nix +builtins.langVersion +``` + +```nix +# b.7.nix +[ (import ./a.6.nix) builtins.langVersion ] +``` + +```console +$ nix-instantiate --eval b.7.nix +[ 6 7 ] +``` + +### `default.nix` handling is version-sensitive + +```nix +# default.6.nix +builtins.langVersion +``` + ```nix -#? Nix 2.12 -"nothing" -``` - -# Arguments -[advantages]: #advantages - -* (+) Makes explicit what can be expected to work. -* (+) Enables communicating language changes systematically. -* (+) Backwards-compatible - * (+) Allows for gradual adoption: opt-in until semantics is implemented in Nix *and* the first backwards-incompatible change to the language is introduced. -* (+) Visually unintrusive -* (+) Self-describing and human-readable -* (+) Follows a well-known convention of using [magic numbers in files](https://en.m.wikipedia.org/wiki/Magic_number_(programming)#In_files) -* (-) May make the appearance that changing the language is harmless. - * (+) The convention itself is harmless and independent of the development culture around the language. -* (-) The syntax of the magic comment is arbitrary. -* (-) There is a chance of abusing the magic comment for more metadata in the future. Let's avoid that. -* (-) At least one form of comment is forever bound to begin with `#` to maintain compatibility. -* (-) It requires significant additional effort to implement and maintain an appropriate system to make use of the version information. - -# Alternatives -[alternatives]: #alternatives - -- Never introduce backwards-incompatible changes to the language. - - * (+) No additional effort required. - * (-) Requires additions to be made very carefully. - * (-) Makes solving some well-known problems impossible. - -- Use the output of [`builtins.langVersion`] for specifying the version of the Nix language. - - * (+) This would serve other Nix language evaluators which are not and should not be tied to the rest of Nix. - * (-) `builtins.langVersion` is currently only internal and undocumented. - * (+) Documentation is easy to add. - * (-) Requires adding another built-in to the public API. - * (-) Using a language feature requires an additional steps from users to determine the current version. - * (+) We can add a command line option such that it is not more effort than `nix --version`. - * (-) Requires adding another command line option to the public API. - * (+) The Nix language version is decoupled Nix version numbering. - * (+) It changes less often than the Nix version. - * (-) That was probably due to making changes being so hard. - * (-) There are two version numbers to keep track of. - * (-) The magic comment should reflect that it's specifying the *Nix language* version, which would make it longer. - * (-) Renaming the Nix language will be impossible once the mechanism is part of stable Nix. - -[`builtins.langVersion`]: https://github.com/NixOS/nix/blob/26c7602c390f8c511f326785b570918b2f468892/src/libexpr/primops.cc#L3952-L3957 - -- Use a magic string that is incompatible with evaluators prior to the feature, e.g. `%? Nix `. - - * (+) Makes clear that the file is not intended to be used without explicit handling of compatibility. - * (-) Cannot be introduced gradually. - * (+) Such a breaking change could also be reserved for later iterations of the Nix language. +# foo.7.nix +import ./. +``` + +```console +$ nix-instantiate --eval foo.7.nix +error: getting status of '/home/$USER/default.7.nix': No such file or directory +``` + +### Expressions are not forward compatible + +```nix +# a.7.nix +builtins.langVersion +``` + +```nix +# b.6.nix +import ./a.7.nix +``` + +```console +$ nix-instantiate --eval foo.7.nix +error: unsupported Nix language version 7 +``` + + +### Best-effort interoperability + +```nix +# a.6.nix +{ increment }: +increment 1 +``` + +```nix +# b.7.nix +import ./a.6.nix { increment = x: x + 1 } +``` + +Since `increment` written in version 7 carries its own implementation with it, forcing it within an expression written in version 6 just works: + +```console +$ nix-instantiate --eval b.7.nix +2 +``` + +### Pathological example + +Usually existing code will be interacted with by calling functions. +When passing values from newer versions to functions from older versions of the language, interoperatbility can only be supported on a best-effort basis. + +```nix +# a.6.nix +{ value }: +value + 1 +``` + +Here we pretend that langauge version 7 introduced a new value type and syntax for complex numbers: + +```nix +# b.7.nix +import ./a.6.nix { value = %5 + 7i%; } +``` + +```console +$ nix-instantiate --eval b.7.nix +error: unsupported value type `complex` at built-in operator `+` +``` + +## Deprecation warnings + +``` +nix eval --json ./test.nix +warning: URL literals are deprecated (url-literal) + please replace this with a string: "https://nixos.org" + + at test.nix:1:1: + + 1| https://nixos.org + | ^ + +"https://nixos.org" + +warning: The following deprecated features were used: + - url-literal (httsp://nixos.org), 1 time + + Add `--lang-warn-verbose` to show all occurrences + Use `--lang-no-warn=url-literal` to disable this warning. + Use `--lang-error=url-literal` to issue errors instead of warnings. +``` # Prior art - [Rust `edition` field] - Rust has an easier problem to solve. Cargo files are written in TOML, so the `edition` information does not have to be part of Rust itself. + Rust has an easier problem to solve. + Cargo files are written in TOML, so the `edition` information does not have to be part of Rust itself. + + [Rust `edition` field]: https://doc.rust-lang.org/cargo/reference/manifest.html#the-edition-field + +- [Haskell language extensions] + + Haskell allows enabling separate language features per file. + + [Haskell language extensions]: https://downloads.haskell.org/ghc/latest/docs/users_guide/exts/intro.html + +- JaveScript modules + - .cjs and .mjs extensions for commonjs/es-modules syntax variants + - `function() { "use strict"; return 10 }` - [Flakes `edition` field] There had been an attempt to include an `edition` field into the Flakes schema. It did not solve the problem of having to evaluate the Nix expression using *some* version of the grammar. -[Rust `edition` field]: https://doc.rust-lang.org/cargo/reference/manifest.html#the-edition-field -[Flakes `edition` field]: https://discourse.nixos.org/t/nix-2-8-0-released/18714/6 - -# Unresolved questions -[unresolved]: #unresolved-questions - -- Is the proposed magic number already in use in [other file formats](https://en.m.wikipedia.org/wiki/Magic_number_(programming)#In_files)? -- Should we allow multiple known-good versions in one line? + [Flakes `edition` field]: https://discourse.nixos.org/t/nix-2-8-0-released/18714/6 # Future work [future]: #future-work -- Define semantics, that is, what exactly to do with the information given in the magic comment. -- Define rules deciding when a change to the language is appropriate to avoid proliferation and limit complexity of implementation. +- Define rules deciding when a change to the language is appropriate, to avoid version proliferation and limit complexity of implementations. From abcda5a3873f1cbb3856fd9ab6419d05b68b69a9 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 1 Jun 2023 19:44:52 +0200 Subject: [PATCH 10/48] incorporate feedback and re-evaluate prefer the special syntax approach, based on the arguments that all other alternatives would defeat the purpose of guaranteeing predictable results, which this proposal values higher than forward compatibility. --- rfcs/0137-nix-language-version.md | 230 ++++++++++++++++-------------- 1 file changed, 120 insertions(+), 110 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 55de65651..4db5aa206 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -26,7 +26,7 @@ This sometimes involves backward-incompatible ("breaking") changes that currentl Therefore we propose a mechanims and policies to introduce changes to the Nix language in a controlled and deliberate manner. It aims to avoid breaking existing setups, and to minimise maintenance burden for implementors and users. -The goal is for new versions of the Nix language evaluator to stay backward compatible with existing Nix expressions, but not necessarily for new Nix expressions to stay forward compatible with existing evaluators. +The goal is for new versions of the Nix language evaluator to stay backward compatible with existing Nix expressions, and for new Nix expressions to be deliberately incompatible with existing evaluators. Regardless, changes to the language, especially breaking changes, should remain a rare exception. @@ -64,6 +64,11 @@ Other discussions around language changes: - [Nix 2 – a hypothetical syntax](https://md.darmstadt.ccc.de/s/nix2) - [Nix language changes](https://gist.github.com/edolstra/29ce9d8ea399b703a7023073b0dbc00d) +## Drawbacks + +Allowing multiple language versions to coexist will over time introduce a proliferation of syntax highlighters and other tooling. +Once the language version is accessible though, tooling can at least be adapted in a systematic way. + ## Alternatives - Keep the language as implemented by Nix compatible, but socially restrict the usage of undesirable features. @@ -145,116 +150,132 @@ Other discussions around language changes: -1. The language version for Nix files is denoted in the file extension. +1. The language version for Nix expressions is denoted in special syntax at the beginning of parse unit.
Arguments - - (+) Sidesteps misinterpretation keeping metadata out of the actual data - - (-) Will introduce a proliferation of syntax highlighters and other tooling - - (+) this is correct though, because a syntax highligher has to know the syntax if it changes with the language version - - (+) Makes accidental mixing of versions impossible - - Have to specify the file extension when importing a file - - (-) have to rename all files in a project to change the version - - (-) Makes filenames longer, introduces visual noise - - This is the cost of being explicit - - (-) `default.nix` resolving needs specification: - - If for `import ./foo`, all of `./foo/default.nix{6,7,8}` exist, pick the one matching the version used by the evaluator, otherwise fail - - Then you'd have to specify a file using a different version explicitly + - (+) Will prevent older evaluators from evaluating expressions written in a newer language version following this proposal (no forward compatibility) + - (-) The errors on older evaluators will be opaque + - (+) Syntax can be made self-describing and human-readable to alleviate that to some extent + - (-) The syntax has to be fixed forever if one wanted to provide meaningful errors on language upgrades + - This has the same trade-offs as when introducing the new syntax to begin with + - (-) Editor support is made harder, since it requires switching the language based on file contents + - (+) Making the language version accessible at all will probably outweigh the costs
Alternatives - Use a magic comment at the beginning of the file - - (+) Makes explicit what can be expected to work - - (+) Enables communicating language changes systematically - - (+) Backwards-compatible - - (+) Allows for gradual adoption: opt-in until semantics is implemented in Nix *and* the first backwards-incompatible change to the language is introduced - - (+) Visually unintrusive - - (+) Self-describing and human-readable + - (+) Allows for gradual adoption: opt-in until semantics is implemented in Nix *and* the first backwards-incompatible change to the language is introduced + - (-) This will produce surprising results if the next language version preserves syntax but changes semantics (forward compatibility) + - (-) Requires the first language version following this proposal to be syntactically incompatible with the current language to avoid forward compatibility + - (+) Can be made self-describing and human-readable - (+) Follows a well-known convention of using [magic numbers in files](https://en.m.wikipedia.org/wiki/Magic_number_(programming)#In_files) - (-) May make the appearance that changing the language is harmless - (+) The convention itself is harmless and independent of the development culture around the language - - (-) The syntax of the magic comment is arbitrary - - (-) There is a chance of abusing the magic comment for more metadata in the future. Let's avoid that. + - (-) There is a chance of abusing the magic comment for more metadata in the future - (-) At least one form of comment is forever bound to begin with `#` to maintain compatibility - - (-) Editor support is made harder, since it requires switching the language based on file contents - - (-) Requires significant additional effort to implement and maintain an appropriate system to make use of the version information - - (-) Raises the question if the new syntax should be a breaking change to the language - - Use a magic string that is incompatible with evaluators prior to the feature, e.g. `%? Nix ` - - (+) Makes clear that the file is not intended to be used without explicit handling of compatibility - - (-) Cannot be introduced gradually - - (+) Such a breaking change could also be reserved for later iterations of the Nix language - - - Language versioning per "project" - - (+) This would easily allow inheriting the language version across imports (solving a lot of issues in this proposal) - - (-) There is currently no notion of "project" + - Forward compatibility is undesirable anyway + - (-) Requires support by all tooling, lose semantics otherwise + + - Use `assert builtins.languageVersion` in the first line of the file + - (+) Produces more telling error messages in existing evaluators + - (+) Future evaluators could be augmented to treat this as specially for better errors + - (-) Special treatment may confuse users: why does `assert` at the beginning of a file work differently than somewhere else? + - (-) Bulky expression that can only be replaced by the magic string solution or kept forever + + - Denote the language version in the file extension + - (+) Sidesteps misinterpretation by keeping metadata out of the actual data + - (-) In general it does not prevent forward compatibility with current evaluators. + - (+) Makes accidental mixing of versions impossible at the syntax level + - Have to specify the file extension when importing a file + - (-) Have to rename all files in a project to change the version + - This is somewhat worse than replacing a magic string which is fixed to the beginning of the file + - (-) Makes filenames longer, introduces visual noise + - This is the cost of being explicit + - (-) Enforces narrow restrictions on what information can be encoded and how + - The only reasonable alternative is `-`, e.g. `default.7-nix` + - `.` + - (-) Nixpkgs has been packaging Linux kernels as `linux-${major}.${minor}.nix` + - This may break backwards compatibility of newer evaluators with existing code in surprising ways + - No separator + – (-) Hard to discern visually + - `-` + - (+) Visually not intrusive + - `_` + – (-) Visually more intrusive + - `^` + – (-) Overlaps with derivation output syntax + - All of the following characters will interfere with some tooling: + - `!` - shells + - `"` - shells + - `#` - URLs + - `$` - shells + - `%` - URLs + - `&` - shells, URLs + - `+` - URLs + - `,` - natural language + - `/` - paths, URLs + - `:` - URLs + - `;` - shells + - `=` - URLs, Nix language + - `?` - URLs + - `@` - URLs, Nix language + - `\` - Windows paths + - (-) `default.nix` resolving needs specification: + - If for `import ./foo`, all of `./foo/default.nix{6,7,8}` exist, pick the one matching the version used by the evaluator, otherwise fail + - Then you'd have to specify a file using a different version explicitly + + - Language versioning per "project" in a sidecar file + - (+) This would easily allow inheriting the language version across imports (obviating many specifications in this proposal) + - (-) There is currently no notion of "project" in the Nix language - (-) Attempting to establish one would be a large undertaking and not immediately help solving the problem at hand + - (-) Cannot be introduced gradually, particularly relevant for a large codebase like Nixpkgs - (-) It would require a separate language to encode project metadata such as the language version - The `edition` field was [removed from the flakes schema](https://github.com/NixOS/nix/commit/e5ea01c1a8bbd328dcc576928bf3e4271cb55399) for that reason, as it not not allow distinguishing data from metadata - (+) Other languages do the same (Python, Haskell, Rust, JavaScript, ...) - - (-) Recursive (albeit smaller) problem of managing the additional langauge for project metadata + - (-) Recursive (albeit smaller) problem of managing the additional language for project metadata
-1. The file extension consists of the language version followed by `.nix`. - Example: +1. The following syntax is used for declaring the language version of a Nix expression: ``` - default.7.nix + version \d*; ``` + This implies that if no language version is specified in a Nix file, it is written in version 6 (the version implemented in the stable release of Nix at the time of writing this RFC). + + The syntax is open for bikeshedding. + Alternatives should be very short and self-describing. +
Arguments - - (+) Consistent with the convention of file extensions carrying soft meaning as metadata - – (-) A conflicting convention is for denoting nested file types, such as `tar.gz` - - (+) Visually least intrusive + - (+) Short and fairly self-descriptive + - (+) Not an invasive change
Alternatives - - No separator - – (-) Hard to discern visually - - `^` - – (-) Overlaps with derivation output syntax - - `-` - - (+) Visually not intrusive - - `_` - – (-) Visually more intrusive - - All of the following characters will interfere with some tooling: - - `!` - shells - - `"` - shells - - `#` - URLs - - `$` - shells - - `%` - URLs - - `&` - shells, URLs - - `+` - URLs - - `,` - natural language - - `/` - paths, URLs - - `:` - URLs - - `;` - shells - - `=` - URLs, Nix language - - `?` - URLs - - `@` - URLs, Nix language - - `\` - Windows paths + - `use v\d*;` + - (+) Shorter + - (-) Does not explain much + - `Nix language version \d*` + - (+) Says it all + - (-) Very long
-1. The language version for bare Nix expressions is specified with a parameter to the evaluator. - -1. If no language version is specified in the file name, assume version 6. - This is the version implemented in the stable release of Nix at the time of writing this RFC. - -
Arguments - - - (+) Backwards compatible with existing evaluators - - (+) Does not require changing any existing code - -
+1. The language version declaration is optional for bare Nix expressions, and can be specified with a parameter to the evaluator. + If no version is specified in a bare Nix expression, assume the most recent language version supported by the evaluator.
Alternatives + - Make it mandatory + - (-) This will be very inconvenient to use in the REPL + - Assume the evaluator's current version - (-) When the evaluator advances in language version, evaluation may fail on existing code - (-) Defeats the purpose of explicit versioning: Which evaluator to use for a given file is left unspecified @@ -264,8 +285,6 @@ Other discussions around language changes:
-1. If no version is specified when invoking the evaluator on a bare Nix expression, assume the most recent language version supported by the evaluator. - 1. Language versions prior to 6 are not supported.
Arguments @@ -276,7 +295,7 @@ Other discussions around language changes:
-1. The Nix language evaluator provides a command to output the latest Nix language version. +1. The Nix language evaluator provides a command to output the most recent Nix language version. This command provides options to list all Nix language versions supported by the evaluator.
Arguments @@ -487,51 +506,38 @@ nix --supported-language-versions ### Versioned built-ins can be passed across file boundaries ```nix -# a.6.nix +# a.nix +use 6; builtins.langVersion ``` ```nix -# b.7.nix +# b.nix +use 7; [ (import ./a.6.nix) builtins.langVersion ] ``` ```console -$ nix-instantiate --eval b.7.nix +$ nix-instantiate --eval b.nix [ 6 7 ] ``` -### `default.nix` handling is version-sensitive - -```nix -# default.6.nix -builtins.langVersion -``` - -```nix -# foo.7.nix -import ./. -``` - -```console -$ nix-instantiate --eval foo.7.nix -error: getting status of '/home/$USER/default.7.nix': No such file or directory -``` - ### Expressions are not forward compatible ```nix -# a.7.nix -builtins.langVersion +# a.nix +use 6; +import ./b.nix ``` ```nix -# b.6.nix -import ./a.7.nix +# b.nix +use 7; +builtins.langVersion ``` ```console -$ nix-instantiate --eval foo.7.nix +$ nix-instantiate --eval a.nix error: unsupported Nix language version 7 ``` @@ -539,20 +545,22 @@ error: unsupported Nix language version 7 ### Best-effort interoperability ```nix -# a.6.nix +# a.nix +use 6; { increment }: increment 1 ``` ```nix -# b.7.nix -import ./a.6.nix { increment = x: x + 1 } +# b.nix +use 7; +import ./a.nix { increment = x: x + 1 } ``` Since `increment` written in version 7 carries its own implementation with it, forcing it within an expression written in version 6 just works: ```console -$ nix-instantiate --eval b.7.nix +$ nix-instantiate --eval b.nix 2 ``` @@ -562,7 +570,8 @@ Usually existing code will be interacted with by calling functions. When passing values from newer versions to functions from older versions of the language, interoperatbility can only be supported on a best-effort basis. ```nix -# a.6.nix +# a.nix +use 6; { value }: value + 1 ``` @@ -570,12 +579,13 @@ value + 1 Here we pretend that langauge version 7 introduced a new value type and syntax for complex numbers: ```nix -# b.7.nix -import ./a.6.nix { value = %5 + 7i%; } +# b.nix +use 7; +import ./a.nix { value = %5 + 7i%; } ``` ```console -$ nix-instantiate --eval b.7.nix +$ nix-instantiate --eval bnix error: unsupported value type `complex` at built-in operator `+` ``` From 4c99631c1ebcb285ba42c14e9d917b500600d2cd Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 1 Jun 2023 19:51:39 +0200 Subject: [PATCH 11/48] add Perl `use` as prior art --- rfcs/0137-nix-language-version.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 4db5aa206..8ad6a9244 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -155,6 +155,7 @@ Once the language version is accessible though, tooling can at least be adapted
Arguments - (+) Will prevent older evaluators from evaluating expressions written in a newer language version following this proposal (no forward compatibility) + - (+) Precedent: [Perl `use VERSION`] - (-) The errors on older evaluators will be opaque - (+) Syntax can be made self-describing and human-readable to alleviate that to some extent - (-) The syntax has to be fixed forever if one wanted to provide meaningful errors on language upgrades @@ -613,6 +614,12 @@ warning: The following deprecated features were used: # Prior art +- [Perl `use VERSION`] + + [Perl `use VERSION`]: https://perldoc.perl.org/functions/use#use-VERSION + + Many similarities, with versions declared per file and having to deal with interoperability. + - [Rust `edition` field] Rust has an easier problem to solve. From 6ee9ecd95040f6e21373b7224934f53b4d2e5c5c Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 1 Jun 2023 22:31:49 +0200 Subject: [PATCH 12/48] fix typo --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 8ad6a9244..9b14a8220 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -150,7 +150,7 @@ Once the language version is accessible though, tooling can at least be adapted
-1. The language version for Nix expressions is denoted in special syntax at the beginning of parse unit. +1. The language version for Nix expressions is denoted in special syntax, at the beginning of a parse unit.
Arguments From 6e081bbf5b6c4408abd3b077b4a6ae6ab5ba28c2 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 1 Jun 2023 22:32:42 +0200 Subject: [PATCH 13/48] explain exception for bare expressions --- rfcs/0137-nix-language-version.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 9b14a8220..507cf31b2 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -272,6 +272,8 @@ Once the language version is accessible though, tooling can at least be adapted 1. The language version declaration is optional for bare Nix expressions, and can be specified with a parameter to the evaluator. If no version is specified in a bare Nix expression, assume the most recent language version supported by the evaluator. + This is to ease use of the REPL and commands taking bare expressions. +
Alternatives - Make it mandatory From 5d125cd32ff5a0e18c336de7060e1e83147f64c2 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 1 Jun 2023 22:35:42 +0200 Subject: [PATCH 14/48] add brain worm syntax --- rfcs/0137-nix-language-version.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 507cf31b2..a91b296eb 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -266,6 +266,9 @@ Once the language version is accessible though, tooling can at least be adapted - `Nix language version \d*` - (+) Says it all - (-) Very long + - `with \d*;` + - (+) Allows for forward compatibilty hacks such as better error messages + - (🔥) Because why not
From 4776e4e2e05c9448a397b5098938fa7b6a2d3ed9 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 2 Jun 2023 10:58:09 +0200 Subject: [PATCH 15/48] add Go as prior art --- rfcs/0137-nix-language-version.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index a91b296eb..c3bcfbd33 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -642,6 +642,14 @@ warning: The following deprecated features were used: - .cjs and .mjs extensions for commonjs/es-modules syntax variants - `function() { "use strict"; return 10 }` +- [Go language changes proposal](https://github.com/golang/proposal/blob/master/design/28221-go2-transitions.md#language-changes) + + The document features a discussion of multiple approaches to evolve the language without breaking existing code, and comes to the similar conclusions as our proposal, mainly: + - One needs a way to specify the language version + - Interestingly, Go developers decided to equate the language version with the compiler release version, despite admitting that this may be confusing + - Keep support for code written in older language versions + - Breaking down into features that can be enabled separately is not practical + - [Flakes `edition` field] There had been an attempt to include an `edition` field into the Flakes schema. From 33af9134723ddabb4e41e9444b8511baa88e2f24 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 2 Jun 2023 14:38:15 +0200 Subject: [PATCH 16/48] fix typo --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index c3bcfbd33..8bd6cdea7 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -266,7 +266,7 @@ Once the language version is accessible though, tooling can at least be adapted - `Nix language version \d*` - (+) Says it all - (-) Very long - - `with \d*;` + - `with import \d*;` - (+) Allows for forward compatibilty hacks such as better error messages - (🔥) Because why not From 38793cf4d543f1b194071b1f72160be1d6b2488d Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 2 Jun 2023 17:37:57 +0200 Subject: [PATCH 17/48] add roadmap as future work, with example --- rfcs/0137-nix-language-version.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 8bd6cdea7..810fc454f 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -661,3 +661,12 @@ warning: The following deprecated features were used: [future]: #future-work - Define rules deciding when a change to the language is appropriate, to avoid version proliferation and limit complexity of implementations. +- Define a roadmap to introduce the next language versions, for example: + - Version 7 commits to changes that do not require additional work on Nixpkgs: + - Introduce the version declaration, required to distinguish versions 6 and 7 + - Deprecate URL literals + - Deprecate the `let-body` syntax + - Drop support for leading zeroes on integers + - Formalise change in float representation + - Version 8 is released at least a year after version 7, and only when version annotations according to version 7 are fully supported in Nixpkgs + From cc5b1b41fae3190f49d89c97747cbb31326d512e Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 8 Jun 2023 14:46:16 +0200 Subject: [PATCH 18/48] use the proposed `version` syntax in examples --- rfcs/0137-nix-language-version.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 810fc454f..0d7e0445d 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -513,13 +513,13 @@ nix --supported-language-versions ```nix # a.nix -use 6; +version 6; builtins.langVersion ``` ```nix # b.nix -use 7; +version 7; [ (import ./a.6.nix) builtins.langVersion ] ``` @@ -532,13 +532,13 @@ $ nix-instantiate --eval b.nix ```nix # a.nix -use 6; +version 6; import ./b.nix ``` ```nix # b.nix -use 7; +version 7; builtins.langVersion ``` @@ -552,14 +552,14 @@ error: unsupported Nix language version 7 ```nix # a.nix -use 6; +version 6; { increment }: increment 1 ``` ```nix # b.nix -use 7; +version 7; import ./a.nix { increment = x: x + 1 } ``` @@ -577,7 +577,7 @@ When passing values from newer versions to functions from older versions of the ```nix # a.nix -use 6; +version 6; { value }: value + 1 ``` @@ -586,7 +586,7 @@ Here we pretend that langauge version 7 introduced a new value type and syntax f ```nix # b.nix -use 7; +version 7; import ./a.nix { value = %5 + 7i%; } ``` From 3611cd2eaea196d2e39e5b26ce756bee41375099 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 8 Jun 2023 14:47:00 +0200 Subject: [PATCH 19/48] add argument against `with import ...;` syntax --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 0d7e0445d..19a614f7b 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -268,7 +268,7 @@ Once the language version is accessible though, tooling can at least be adapted - (-) Very long - `with import \d*;` - (+) Allows for forward compatibilty hacks such as better error messages - - (🔥) Because why not + - (-) Will likely mislead beginners to think this is has the same semantics as the original `with import ...;`
From a17e28f41bb3693ce842f59ec3dfcbd892494c39 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Fri, 9 Jun 2023 00:18:19 +0200 Subject: [PATCH 20/48] highlight design goals --- rfcs/0137-nix-language-version.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 19a614f7b..a6018943f 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -25,10 +25,7 @@ Yet, as with any software system, in order to accommodate new insights, we want This sometimes involves backward-incompatible ("breaking") changes that currently cannot be made without significant downstream disruption. Therefore we propose a mechanims and policies to introduce changes to the Nix language in a controlled and deliberate manner. -It aims to avoid breaking existing setups, and to minimise maintenance burden for implementors and users. -The goal is for new versions of the Nix language evaluator to stay backward compatible with existing Nix expressions, and for new Nix expressions to be deliberately incompatible with existing evaluators. - -Regardless, changes to the language, especially breaking changes, should remain a rare exception. +It aims to avoid breaking existing code, to prevent inadvertently breaking reproducibility, and to minimise maintenance burden for implementors and users. ## Motivating examples @@ -101,6 +98,14 @@ Once the language version is accessible though, tooling can at least be adapted # Detailed Design +## Design goals + +1. New versions of the Nix language evaluator should to stay backward compatible with existing Nix expressions. + +1. New Nix expressions should not work with existing evaluators at all. + +1. Changes to the language, especially backward-incompatible changes, should remain a rare exception. + ## Language versioning 1. The language version is a natural number. @@ -168,7 +173,7 @@ Once the language version is accessible though, tooling can at least be adapted
Alternatives - Use a magic comment at the beginning of the file - - (+) Allows for gradual adoption: opt-in until semantics is implemented in Nix *and* the first backwards-incompatible change to the language is introduced + - (+) Allows for gradual adoption: opt-in until semantics is implemented in Nix *and* the first backward-incompatible change to the language is introduced - (-) This will produce surprising results if the next language version preserves syntax but changes semantics (forward compatibility) - (-) Requires the first language version following this proposal to be syntactically incompatible with the current language to avoid forward compatibility - (+) Can be made self-describing and human-readable From 29e24243bdd743cf264d5c291b735af635b89fea Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Sat, 10 Jun 2023 11:31:05 +0200 Subject: [PATCH 21/48] Update rfcs/0137-nix-language-version.md Co-authored-by: Ryan Lahfa --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index a6018943f..5c670899d 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -35,7 +35,7 @@ Incompatible changes from the past: - There have been other, sometimes breaking changes to the language that have not resulted in an increment of the language version (e.g. the recent `fetchGit` changes). - The `builtins.toJSON 1.000001` output [changed in Nix 2.12](https://github.com/NixOS/nix/issues/8259). -Possbile future changes that are in discussion: +Possible future changes that are in discussion: - Remove URL literals (currently implemented via experimental-features) - Remove the old `let { body = ... }` syntax From f6d519ee09119041fc23d943d31a5aded2e028ee Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Sun, 11 Jun 2023 02:19:34 +0200 Subject: [PATCH 22/48] Update rfcs/0137-nix-language-version.md --- rfcs/0137-nix-language-version.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 5c670899d..de4592332 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -156,6 +156,7 @@ Once the language version is accessible though, tooling can at least be adapted
1. The language version for Nix expressions is denoted in special syntax, at the beginning of a parse unit. + A parse unit is any text stream, e.g. a file or string.
Arguments From 3364b1cbb139913d9f1a09e6929f8c1be059e86e Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Sun, 11 Jun 2023 17:38:27 +0200 Subject: [PATCH 23/48] Update rfcs/0137-nix-language-version.md --- rfcs/0137-nix-language-version.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index de4592332..544f9234e 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -278,10 +278,10 @@ Once the language version is accessible though, tooling can at least be adapted
-1. The language version declaration is optional for bare Nix expressions, and can be specified with a parameter to the evaluator. +1. As an exception, the language version declaration is optional for Nix expressions passed directly as an argument to the evaluator and on the REPL, and can be specified with a separate parameter to the evaluator. If no version is specified in a bare Nix expression, assume the most recent language version supported by the evaluator. - This is to ease use of the REPL and commands taking bare expressions. + This is to ease use of the REPL and evaluating ad hoc expressions.
Alternatives From 71ae7e770a3a3e1cf1ef5b0ddea2049acbd114fa Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Mon, 12 Jun 2023 11:10:04 +0200 Subject: [PATCH 24/48] fix wording --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 544f9234e..1115eb6bf 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -24,7 +24,7 @@ The stability of Nix language has been praised on multiple occasions, e.g. [Nix Yet, as with any software system, in order to accommodate new insights, we want to allow the Nix language to evolve. This sometimes involves backward-incompatible ("breaking") changes that currently cannot be made without significant downstream disruption. -Therefore we propose a mechanims and policies to introduce changes to the Nix language in a controlled and deliberate manner. +Therefore we propose mechanisms and policies for introducing changes to the Nix language in a controlled and deliberate manner. It aims to avoid breaking existing code, to prevent inadvertently breaking reproducibility, and to minimise maintenance burden for implementors and users. ## Motivating examples From 0583d77e44ca442e612ceda3b03af55149c73de4 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Mon, 12 Jun 2023 12:37:01 +0200 Subject: [PATCH 25/48] remove duplicate example Co-authored-by: sternenseemann --- rfcs/0137-nix-language-version.md | 1 - 1 file changed, 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 1115eb6bf..77925d6aa 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -42,7 +42,6 @@ Possible future changes that are in discussion: - Disallow leading zeroes in integer literals (such as `umask = 0022`) - Disallow `a.x or y` if `a` is not an attribute set - Simplifying semantics of `builtins.toString` and string interpolation -- Remove the `let { body }` syntax - `__functor` and `__toString`, probably - Remove `__overrides` - [Make `builtins` more consistent](https://github.com/NixOS/nix/issues/7290), e.g. not exposing `map`, `removeAttrs`, `fetchGit` and and others in the global scope From a7def97e9a48592101fb71a9fe28f2c8ce944948 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Mon, 12 Jun 2023 12:45:53 +0200 Subject: [PATCH 26/48] reword drawbacks, move drawbacks and alternatives to end of document --- rfcs/0137-nix-language-version.md | 69 +++++++++++++++---------------- 1 file changed, 34 insertions(+), 35 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 77925d6aa..030c60f04 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -60,41 +60,6 @@ Other discussions around language changes: - [Nix 2 – a hypothetical syntax](https://md.darmstadt.ccc.de/s/nix2) - [Nix language changes](https://gist.github.com/edolstra/29ce9d8ea399b703a7023073b0dbc00d) -## Drawbacks - -Allowing multiple language versions to coexist will over time introduce a proliferation of syntax highlighters and other tooling. -Once the language version is accessible though, tooling can at least be adapted in a systematic way. - -## Alternatives - -- Keep the language as implemented by Nix compatible, but socially restrict the usage of undesirable features. - - (+) Roughly matches the current practice, no technical change needed - - (+) Maintains usability of old Nixpkgs versions (up to availability of fixed-output artifacts) - - (+) Does not break third-party codebases before making a decision, keeping Nix a dependable upstream - - (-) This proposal does not allow for breakages unless there is some eventual phase-out of support - - (-) Strict enforcement requires extra tooling that this proposal would obviate - - (-) The implementation of the features that are no longer desirable still incur complexity and maintenance cost - - (-) It's still not really possible to make changes to the language - -- Introduce changes to the language with language extensions or feature flags - - (-) Combinatorial explosion - - See [Haskell language extensions] for real-world experience - - (-) Even more maintenance overhead - - (+) Allows gradual adoption of features - - (-) We already have experimental feature flags as an orthogonal mechanism, with the added benefit that they don't incur support costs and can be dropped without loss - -- Never make breaking changes to the language - - (+) No additional maintenance effort required - - (-) Blocks improvements - - (-) Requires additions to be made very carefully - - (-) Even incremental changes are really expensive that way - - (-) Makes solving some well-known problems impossible - -- Continue current practice - - (-) There is no process for breaking changes - - (-) Breaking changes are not always announced - - (-) There are no means of determining compatibility between expressions and evaluator versions - # Detailed Design ## Design goals @@ -622,6 +587,40 @@ warning: The following deprecated features were used: Use `--lang-error=url-literal` to issue errors instead of warnings. ``` +## Drawbacks + +Allowing multiple language versions to coexist complicates implementation of valuators and support tooling, and makes comprehensive test coverage harder. + +# Alternatives + +- Keep the language as implemented by Nix compatible, but socially restrict the usage of undesirable features. + - (+) Roughly matches the current practice, no technical change needed + - (+) Maintains usability of old Nixpkgs versions (up to availability of fixed-output artifacts) + - (+) Does not break third-party codebases before making a decision, keeping Nix a dependable upstream + - (-) This proposal does not allow for breakages unless there is some eventual phase-out of support + - (-) Strict enforcement requires extra tooling that this proposal would obviate + - (-) The implementation of the features that are no longer desirable still incur complexity and maintenance cost + - (-) It's still not really possible to make changes to the language + +- Introduce changes to the language with language extensions or feature flags + - (-) Combinatorial explosion + - See [Haskell language extensions] for real-world experience + - (-) Even more maintenance overhead + - (+) Allows gradual adoption of features + - (-) We already have experimental feature flags as an orthogonal mechanism, with the added benefit that they don't incur support costs and can be dropped without loss + +- Never make breaking changes to the language + - (+) No additional maintenance effort required + - (-) Blocks improvements + - (-) Requires additions to be made very carefully + - (-) Even incremental changes are really expensive that way + - (-) Makes solving some well-known problems impossible + +- Continue current practice + - (-) There is no process for breaking changes + - (-) Breaking changes are not always announced + - (-) There are no means of determining compatibility between expressions and evaluator versions + # Prior art - [Perl `use VERSION`] From 8413407247cbcc1f64bd8e5dcd3973a9f4a78d61 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 14 Jun 2023 00:40:06 +0200 Subject: [PATCH 27/48] fix typo Co-authored-by: sternenseemann --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 030c60f04..5c0ab9697 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -150,7 +150,7 @@ Other discussions around language changes: - Forward compatibility is undesirable anyway - (-) Requires support by all tooling, lose semantics otherwise - - Use `assert builtins.languageVersion` in the first line of the file + - Use `assert builtins.langVersion` in the first line of the file - (+) Produces more telling error messages in existing evaluators - (+) Future evaluators could be augmented to treat this as specially for better errors - (-) Special treatment may confuse users: why does `assert` at the beginning of a file work differently than somewhere else? From e41b750cede1d6b63fc2ae50a4d84fcc2b01972a Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 14 Jun 2023 01:00:44 +0200 Subject: [PATCH 28/48] remove irrelevant argument --- rfcs/0137-nix-language-version.md | 1 - 1 file changed, 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 5c0ab9697..6d8eb021d 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -303,7 +303,6 @@ Other discussions around language changes: - (+) No need to have this if the Nix files are versioned - (+) `builtins.langVersion` is not documented and not widely used - - (-) Requires a Nix language version bump to implement this RFC
From e97309a97f99e747b24bde8148225c44487fd61a Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 14 Jun 2023 01:31:34 +0200 Subject: [PATCH 29/48] remove allusion to timelines --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 6d8eb021d..db7abe7f7 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -671,5 +671,5 @@ Allowing multiple language versions to coexist complicates implementation of val - Deprecate the `let-body` syntax - Drop support for leading zeroes on integers - Formalise change in float representation - - Version 8 is released at least a year after version 7, and only when version annotations according to version 7 are fully supported in Nixpkgs + - Version 8 is released only when version annotations according to version 7 are fully supported in Nixpkgs From 80255aa2be6e1feed289989af6419104d51beaab Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 14 Jun 2023 01:57:32 +0200 Subject: [PATCH 30/48] fix typo Co-authored-by: sternenseemann --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index db7abe7f7..cf5030e36 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -551,7 +551,7 @@ version 6; value + 1 ``` -Here we pretend that langauge version 7 introduced a new value type and syntax for complex numbers: +Here we pretend that language version 7 introduced a new value type and syntax for complex numbers: ```nix # b.nix From 567d2d4ccfd498d31fa27182138c725d9f49cbc5 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 14 Jun 2023 02:01:24 +0200 Subject: [PATCH 31/48] add syntax constraint for shebang lines --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index cf5030e36..5184013da 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -215,7 +215,7 @@ Other discussions around language changes: ``` version \d*; ``` - + The evaluator must ignore a shebang line (starting with `#!) at the start of files, to leave room for additional tooling. This implies that if no language version is specified in a Nix file, it is written in version 6 (the version implemented in the stable release of Nix at the time of writing this RFC). The syntax is open for bikeshedding. From a2f921fb18b32868eabbca205d4e646b7a662fac Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 14 Jun 2023 02:05:58 +0200 Subject: [PATCH 32/48] clarify design goal of deliberate forward non-compatibility --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 5184013da..d2c0ea047 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -66,7 +66,7 @@ Other discussions around language changes: 1. New versions of the Nix language evaluator should to stay backward compatible with existing Nix expressions. -1. New Nix expressions should not work with existing evaluators at all. +1. A Nix expression written in a newer version of the language should never work with older evaluators not supporting that version. 1. Changes to the language, especially backward-incompatible changes, should remain a rare exception. From 371ddd5dc867122e0b3a647833efeb5f3f6f2f79 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 14 Jun 2023 12:45:02 +0200 Subject: [PATCH 33/48] fix typo --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index d2c0ea047..cbd186506 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -317,7 +317,7 @@ Other discussions around language changes: - (+) This would be beneficial for alternative implementations in terms of churn and effort to keep up - Or implementors of alternative evaluators catching up with changes - (+) Specifying the language precisely via the version actually offers alternative implementations an alternative to catching up: only support a given language version - - (+) This essentially nudges one to organise Nix language (specification or evaluator implementation ) development to be more independent of the rest of Nix + - (+) This essentially nudges one to organise Nix language (specification or evaluator implementation) development to be more independent of the rest of Nix This is good, since it in turn forces stronger separation of concerns and more architectural clarity - (-) Prohibits best-effort attempts at evaluating expressions with possibly incompatible evaluators - (+) With the proposed level of strictness, one doesn't have to rely on best effort but can instead be explicit From 76a3d1fe016829c8dec8944cc8b5a83ddf1b7632 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 15 Jun 2023 14:15:55 +0200 Subject: [PATCH 34/48] add risk of increased maintenance burden --- rfcs/0137-nix-language-version.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index cbd186506..ffc286a80 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -588,7 +588,8 @@ warning: The following deprecated features were used: ## Drawbacks -Allowing multiple language versions to coexist complicates implementation of valuators and support tooling, and makes comprehensive test coverage harder. +Allowing multiple language versions to coexist complicates implementation of evaluators and support tooling, and makes comprehensive test coverage harder. +All else being equal, it may increase maintenance burden and the likelihood of introducing bugs. # Alternatives From f193f543ba57ee5f6adbc27f0fcb8025e9234cb3 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 15 Jun 2023 14:39:46 +0200 Subject: [PATCH 35/48] add example of best effort interop --- rfcs/0137-nix-language-version.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index ffc286a80..70cd0c794 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -564,6 +564,30 @@ $ nix-instantiate --eval bnix error: unsupported value type `complex` at built-in operator `+` ``` +In the following example, version 7 *removed* the `null` primitive, such that it can no longer be used. + +```nix +# a.nix +null +``` + +```nix +# b.nix +version 7; +import ./a.nix +``` + +```console +$ nix-instantiate --eval b.nix +error: unsupported value `null` +``` + +This is consistent with best-effort interoperability: +Old code keeps working on its own, and new code has to be adapted because it was not there before the breaking change. + +While this requires additional effort to adopt the new language version, expression authors can always recourse to writing new code in older versions while using newer evaluators. +This in fact allows for creating compatibility wrappers as needed. + ## Deprecation warnings ``` From 571c807923ac690a78040a36282aeb46f75d9b4d Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 15 Jun 2023 14:40:29 +0200 Subject: [PATCH 36/48] add argument on version proliferation risk --- rfcs/0137-nix-language-version.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 70cd0c794..99b01d458 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -615,6 +615,10 @@ warning: The following deprecated features were used: Allowing multiple language versions to coexist complicates implementation of evaluators and support tooling, and makes comprehensive test coverage harder. All else being equal, it may increase maintenance burden and the likelihood of introducing bugs. +Providing a pathway for introducing breaking changes bears the risk of version proliferation. +We argue though that the implementation overhead incurred by the strict compatibility requirements will by itself balance that out. +At least such a trade-off now could then be made to begin with, as currently breaking changes cannot be made at all. + # Alternatives - Keep the language as implemented by Nix compatible, but socially restrict the usage of undesirable features. From d9d6a1000ca4e8cc60cdd49f7b1cd6c3f8b8d862 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 15 Jun 2023 14:48:08 +0200 Subject: [PATCH 37/48] fix typo --- rfcs/0137-nix-language-version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 99b01d458..71e8bb680 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -237,7 +237,7 @@ Other discussions around language changes: - (+) Says it all - (-) Very long - `with import \d*;` - - (+) Allows for forward compatibilty hacks such as better error messages + - (+) Allows for forward compatibility hacks such as better error messages - (-) Will likely mislead beginners to think this is has the same semantics as the original `with import ...;`
From d97603e26db64406a169e48804f5f9ed9feb0874 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 15 Jun 2023 15:01:37 +0200 Subject: [PATCH 38/48] add hint at what implementation would look like --- rfcs/0137-nix-language-version.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 71e8bb680..e880341de 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -354,6 +354,9 @@ Other discussions around language changes: 1. Semantics are preserved across file boundaries for past language versions. + This should be fairly straightforward to implement since values passed around in the evaluator carry all the information needed to force them. + Newer parts of the evaluator can always wrap their values in interfaces that are accepted by older parts, as far as possible. + Example: [Best-effort interoperability](#best-effort-interoperability)
Arguments From d54d6bc67055d46db46f79814198b06c574e8de5 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 15 Jun 2023 15:02:19 +0200 Subject: [PATCH 39/48] add argument in favor of semver --- rfcs/0137-nix-language-version.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index e880341de..e9439c682 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -102,6 +102,7 @@ Other discussions around language changes: - [Semantic Versioning](https://semver.org/) - (+) Can distinguish additions from other changes - (-) This is not needed for our use case, since any addition to an expression will break for older evaluators even if the major version matches + - (+) Simplifies evaluator implementation: pure additions can be guarded by the minor version cleanly within a major version's evaluator code - (-) Requires more characters to account for the added expressiveness - This may be relevant depending on where it has to be encoded - Use version numbers of Nix stable releases for specifying the version of the Nix language From a64de1bfb2f0ee65c07fc614a36796c56274689d Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 22 Jun 2023 14:09:52 +0200 Subject: [PATCH 40/48] remove reference to `builtins.langVersion` this is unrelated to the proposal, as the version declaration is only relevant for parsing, and would cover all use cases of `langVersion` if we have minor version numbers --- rfcs/0137-nix-language-version.md | 26 -------------------------- 1 file changed, 26 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index e9439c682..50721c6ae 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -281,32 +281,6 @@ Other discussions around language changes:
-1. `builtins.langVersion` returns the language version used for evaluating the given expression. - -
Arguments - - - (+) Could make use of it for generating Nix expressions programmatically and annotating them with the correct version - - (+) `builtins.langVersion` is already part of the stable interface, this way we can make it pure - - (-) Requires maintaining more API surface without a clear use case - -
- -
Alternatives - - - Return the latest language version instead - - - (-) Doesn't help to determine what is used for evaluating the current expression - - (+) Might provide opportunities for forward-compatible Nix code - - (-) Brittle and defeats the purpose of this RFC - - (-) Impure, the value depends on the environment - - - Don't expose `builtins.langVersion` at all - - - (+) No need to have this if the Nix files are versioned - - (+) `builtins.langVersion` is not documented and not widely used - -
- 1. Each time the language specification (currently as embodied by the Nix language evaluator) is changed, the language version must be incremented.
Arguments From c3d5411fa4a0ea42cdbec8c0df6f0043a168c8de Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 22 Jun 2023 14:53:48 +0200 Subject: [PATCH 41/48] promote a major-minor versioning scheme --- rfcs/0137-nix-language-version.md | 196 +++++++++++++++++------------- 1 file changed, 112 insertions(+), 84 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 50721c6ae..60403cbab 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -72,18 +72,25 @@ Other discussions around language changes: ## Language versioning -1. The language version is a natural number. +1. The language version consists of two integers, denoting a major and a minor component.
Arguments - - (+) Formally decouples the Nix language version from the Nix version - - (+) The Nix language is supposed to change much less often than the rest of Nix - - (-) There are two version numbers to keep track of - - (-) Makes more evident that the Nix language is a distinct architectural component of the Nix ecosystem - - (+) It's currently handled that way, no change needed apart from documentation - - See [`builtins.langVersion`] (currently undocumented) - - (+) Simple and unambiguous - - (+) Concise, even in the long term, since the language is supposed to change very rarely + - (+) Can distinguish additions from breaking changes + - (-) This may not be needed for our use case, since any addition to an expression will break for older evaluators even if the major version matches + - (+) Weaker forms of this are already in use: + + - `builtins.foo ? workaround` + - `builtins.langVersion < 6` + - `builtins.nixVersion` + + An explicit version declaration would would make obvious to readers what to expect and allow for better error messages. + + - (+) Simplifies evaluator implementation: pure additions can be guarded by the minor version cleanly within a major version's evaluator code + - (-) Requires more characters to account for the added expressiveness + - This may be relevant depending on where it has to be encoded + - (+) Follows well-known convention of [Semantic Versioning](https://semver.org/) + - (-) May be confused with the version number of the Nix release [`builtins.langVersion`]: https://github.com/NixOS/nix/blob/26c7602c390f8c511f326785b570918b2f468892/src/libexpr/primops.cc#L3952-L3957 @@ -91,6 +98,15 @@ Other discussions around language changes:
Alternatives + - The language version is a natural number. + - (+) Formally decouples the Nix language version from the Nix version + - (+) The Nix language is supposed to change much less often than the rest of Nix + - (-) There are two version numbers to keep track of + - (-) Makes more evident that the Nix language is a distinct architectural component of the Nix ecosystem + - (+) It's currently handled that way, no change needed apart from documentation + - See [`builtins.langVersion`] (currently undocumented) + - (+) Simple and unambiguous + - (+) Concise, even in the long term, since the language is supposed to change very rarely - [Calendar Versioning](https://calver.org/) - (+) Provides information on when changes happened - (-) This is not needed because only compatibility information is needed @@ -99,12 +115,6 @@ Other discussions around language changes: - (+) Restricting to only the year would force language changes to be rare - (+) This would allow obvious synchronisation points with Nixpkgs releases - (-) It may be too much policy encoded in a mechanism - - [Semantic Versioning](https://semver.org/) - - (+) Can distinguish additions from other changes - - (-) This is not needed for our use case, since any addition to an expression will break for older evaluators even if the major version matches - - (+) Simplifies evaluator implementation: pure additions can be guarded by the minor version cleanly within a major version's evaluator code - - (-) Requires more characters to account for the added expressiveness - - This may be relevant depending on where it has to be encoded - Use version numbers of Nix stable releases for specifying the version of the Nix language - (+) More obvious to see for users what the current Nix version is rather than `builtins.langVersion` - (-) Would tie alternative Nix language evaluators to the rest of Nix @@ -214,8 +224,9 @@ Other discussions around language changes: 1. The following syntax is used for declaring the language version of a Nix expression: ``` - version \d*; + version \d*.\d*; ``` + The evaluator must ignore a shebang line (starting with `#!) at the start of files, to leave room for additional tooling. This implies that if no language version is specified in a Nix file, it is written in version 6 (the version implemented in the stable release of Nix at the time of writing this RFC). @@ -231,13 +242,13 @@ Other discussions around language changes:
Alternatives - - `use v\d*;` + - `use v\d*\.\d*;` - (+) Shorter - (-) Does not explain much - - `Nix language version \d*` + - `Nix language version \d*\.\d*` - (+) Says it all - (-) Very long - - `with import \d*;` + - `with import \d*\.\d*;` - (+) Allows for forward compatibility hacks such as better error messages - (-) Will likely mislead beginners to think this is has the same semantics as the original `with import ...;` @@ -262,68 +273,45 @@ Other discussions around language changes:
-1. Language versions prior to 6 are not supported. - -
Arguments - - - (+) Does not require additional development effort - - (+) Prior langauge versions are not fully supported by current code already, and the rest of this proposal argues to deprecate old versions in the future in order to keep the implementation manageable - - (-) Legacy code will not get support for managing compatibility - -
- -1. The Nix language evaluator provides a command to output the most recent Nix language version. - This command provides options to list all Nix language versions supported by the evaluator. +1. Each time the language specification (currently as embodied by the Nix language evaluator) is changed: -
Arguments + - When backward compatibility is preserved, the minor version number is incremented. - - (+) This is for convenience to determine which features are available + A backward compatible change to the language means that all existing expressions written for the prior version will still evaluate to the same result. + Examples include additions of `builtins`, operators, or syntactic constructs. -
+ - When breaking changes are introduced, the major version number is incremented. -1. Each time the language specification (currently as embodied by the Nix language evaluator) is changed, the language version must be incremented. + Examples include semantic changes or removal of `builtins`, operators, or syntactic constructs.
Arguments - - (+) The principled solution: guarantees reproducibility given a fixed language version - - (-) Implies additional overhead in development effort: - - Either for Nix maintainers to accommodate that practice in the release lifecycle - - For example, one would have to batch language changes for a version bump to limit the number of increments - - (+) This would be beneficial for alternative implementations in terms of churn and effort to keep up - - Or implementors of alternative evaluators catching up with changes - - (+) Specifying the language precisely via the version actually offers alternative implementations an alternative to catching up: only support a given language version - - (+) This essentially nudges one to organise Nix language (specification or evaluator implementation) development to be more independent of the rest of Nix - This is good, since it in turn forces stronger separation of concerns and more architectural clarity - - (-) Prohibits best-effort attempts at evaluating expressions with possibly incompatible evaluators - - (+) With the proposed level of strictness, one doesn't have to rely on best effort but can instead be explicit - - (-) Fixing evaluator bugs (i.e., clearly unintended behavior) after releases would technically require a version bump and therefore (theoretically) cooperation by expression authors - - This could be communicated with an increment in the Nix patch-level version, as is already practice + - (+) The principled solution: guarantees reproducibility given a fixed language version + - (+) Makes explicit current practice of adding features to the language, and allows introducing breaking changes in a controlled fashion, which is currently not possible at all + - (-) Implies additional overhead in development effort: + - Either for Nix maintainers to accommodate that practice in the release lifecycle + - For example, one would have to batch language changes for a major version bump to limit the number of increments + - (+) This would be beneficial for alternative implementations in terms of churn and effort to keep up + - Or implementors of alternative evaluators catching up with changes + - (+) Specifying the language precisely via the version actually offers alternative implementations an alternative to catching up: only support a given language version + - (+) This essentially nudges one to organise Nix language (specification or evaluator implementation) development to be more independent of the rest of Nix + This is good, since it in turn forces stronger separation of concerns and more architectural clarity + - (-) Prohibits best-effort attempts at evaluating expressions with possibly incompatible evaluators + - (+) With the proposed level of strictness, one doesn't have to rely on best effort but can instead be explicit + - (-) Fixing evaluator bugs (i.e., clearly unintended behavior) after releases would technically require a version bump and therefore (theoretically) cooperation by expression authors + - This could be communicated with an increment in the Nix patch-level version, as is already practice
Alternatives - - Bump version only when evaluation result on prior version evaluators would be *substantially* different + - Bump major version only when evaluation result on prior version evaluators would be *substantially* different - (+) Leaves room for judgement by developers - (+) Allows controlling progression of versions to some degree - (-) Conversely, leaves room for sneaking in breaking changes unannounced - (-) This loses compatibility guarantees we'd get from a stricter paradigm - (-) Deprives expression authors of ability to be selective with evaluator versions - (-) Due to hashing this is often not much different from taking *any* change into account - - Bump version when prior evaluators would fail - - (-) Derivation hashes may change between evaluator versions if different evaluation results were allowed within one version - - This amounts to giving up on specifying the language exhaustively using a version label - - (+) Fewer version increments or precautions required given current development practice - -
- -1. Whenever Nix drops support for evaluating prior language versions, a major version bump is required. - - Example: Assuming the current language version is 8, the Nix release version is 2.20, and support for language version 6 is dropped, the next Nix release must be version 3.0 - -
Arguments - - - (+) This enforces that existing code that works will not break inadvertently when upgrading Nix
@@ -368,6 +356,43 @@ Other discussions around language changes: - (+) It will notify users about what's going on instead of just breaking
+1. Language versions prior to 6 are not supported. + +
Arguments + + - (+) Does not require additional development effort + - (+) Prior langauge versions are not fully supported by current code already, and the rest of this proposal argues to deprecate old versions in the future in order to keep the implementation manageable + - (-) Legacy code will not get support for managing compatibility + +
+ +1. The Nix language evaluator provides a command to output the most recent Nix language version. + This command provides options to list all Nix language versions supported by the evaluator. + +
Arguments + + - (+) This is for convenience to determine which features are available + +
+ +1. Whenever Nix drops support for evaluating prior language versions, a major version bump is required. + + Example: Assuming the current language version is 8.0, the Nix release version is 2.20, and support for language version 6 is dropped, the next Nix release must be version 3.0 + +
Arguments + + - (+) This enforces that existing code that works will not break inadvertently when upgrading Nix + +
+ +
Alternatives + + - Separate Nix development from the Nix language entirely and keep it outside of the scope of this proposal + - (-) Currently the upstream Nix language evaluator and compatibility of expressions in Nixpkgs is closely tied to the rest of Nix, and have to take that into account + - Further separation of concerns is out of scope for this proposal + +
+ ## Deprecation warnings and errors 1. Each language construct to deprecate relative to a prior version is given a symbolic name. @@ -445,13 +470,15 @@ Other discussions around language changes: ```console nix --language-version -7 +7.0 ``` ```console nix --supported-language-versions -6 -7 +6.1 +6.2 +6.3 +7.0 ``` ## Version interoperability @@ -460,38 +487,38 @@ nix --supported-language-versions ```nix # a.nix -version 6; +version 6.1; builtins.langVersion ``` ```nix # b.nix -version 7; -[ (import ./a.6.nix) builtins.langVersion ] +version 7.0; +[ (import ./a.nix) builtins.null ] ``` ```console $ nix-instantiate --eval b.nix -[ 6 7 ] +[ 6 null ] ``` ### Expressions are not forward compatible ```nix # a.nix -version 6; +version 6.1; import ./b.nix ``` ```nix # b.nix -version 7; -builtins.langVersion +version 7.0; +builtins.null ``` ```console $ nix-instantiate --eval a.nix -error: unsupported Nix language version 7 +error: unsupported Nix language version 7.0 ``` @@ -499,18 +526,18 @@ error: unsupported Nix language version 7 ```nix # a.nix -version 6; +version 6.1; { increment }: increment 1 ``` ```nix # b.nix -version 7; +version 7.0; import ./a.nix { increment = x: x + 1 } ``` -Since `increment` written in version 7 carries its own implementation with it, forcing it within an expression written in version 6 just works: +Since `increment` written in version 7.0 carries its own implementation with it, forcing it within an expression written in version 6.1 just works: ```console $ nix-instantiate --eval b.nix @@ -524,7 +551,7 @@ When passing values from newer versions to functions from older versions of the ```nix # a.nix -version 6; +version 6.1; { value }: value + 1 ``` @@ -533,7 +560,7 @@ Here we pretend that language version 7 introduced a new value type and syntax f ```nix # b.nix -version 7; +version 7.0; import ./a.nix { value = %5 + 7i%; } ``` @@ -542,22 +569,23 @@ $ nix-instantiate --eval bnix error: unsupported value type `complex` at built-in operator `+` ``` -In the following example, version 7 *removed* the `null` primitive, such that it can no longer be used. +In the following example, assume version 7.0 *removed* floating point numbers, such that they can no longer be used. ```nix # a.nix -null +version 6.1; +1.1 ``` ```nix # b.nix -version 7; -import ./a.nix +version 7.0; +(import ./a.nix) * 2 ``` ```console $ nix-instantiate --eval b.nix -error: unsupported value `null` +error: unsupported type `float` for multiplication ``` This is consistent with best-effort interoperability: From 8f2cf113c2d08b70d188c48791349864524c3d59 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 22 Jun 2023 14:55:54 +0200 Subject: [PATCH 42/48] add some details to summary --- rfcs/0137-nix-language-version.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 60403cbab..d709a9c9f 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -13,9 +13,9 @@ related-issues: https://github.com/NixOS/nix/issues/7255 # Summary [summary]: #summary -Introduce a convention to determine which version of the Nix language grammar to use for parsing and evaluating Nix expressions. - -Add parameters to the Nix language evaluator, controlling the behavior of deprecation warnings and errors. +- Introduce a convention to determine which version of the Nix language grammar to use for parsing and evaluating Nix expressions. +- Add parameters to the Nix language evaluator, controlling the behavior of deprecation warnings and errors. +- Codify a versioning policy for the Nix language specification # Motivation [motivation]: #motivation From 8ae831e085e88631d99761f32e77f291b9624ad7 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 22 Jun 2023 14:57:35 +0200 Subject: [PATCH 43/48] add item to the sample change roadmap --- rfcs/0137-nix-language-version.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index d709a9c9f..1431514f6 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -702,6 +702,7 @@ At least such a trade-off now could then be made to begin with, as currently bre - Define a roadmap to introduce the next language versions, for example: - Version 7 commits to changes that do not require additional work on Nixpkgs: - Introduce the version declaration, required to distinguish versions 6 and 7 + - Remove `builtins.langVersion`, as it's not needed any more - Deprecate URL literals - Deprecate the `let-body` syntax - Drop support for leading zeroes on integers From 7bb92d4e7d5868b91dee9456499285f699226eb2 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 22 Jun 2023 14:58:37 +0200 Subject: [PATCH 44/48] remove unrelated future work item --- rfcs/0137-nix-language-version.md | 1 - 1 file changed, 1 deletion(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 1431514f6..780ba9aae 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -698,7 +698,6 @@ At least such a trade-off now could then be made to begin with, as currently bre # Future work [future]: #future-work -- Define rules deciding when a change to the language is appropriate, to avoid version proliferation and limit complexity of implementations. - Define a roadmap to introduce the next language versions, for example: - Version 7 commits to changes that do not require additional work on Nixpkgs: - Introduce the version declaration, required to distinguish versions 6 and 7 From 9711fb0e52f790b88954732fa42cea75456fa370 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 22 Jun 2023 14:59:34 +0200 Subject: [PATCH 45/48] make clear example roadmap talks about major versions --- rfcs/0137-nix-language-version.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 780ba9aae..2f05e39ca 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -699,12 +699,12 @@ At least such a trade-off now could then be made to begin with, as currently bre [future]: #future-work - Define a roadmap to introduce the next language versions, for example: - - Version 7 commits to changes that do not require additional work on Nixpkgs: - - Introduce the version declaration, required to distinguish versions 6 and 7 + - Major version 7 commits to changes that do not require additional work on Nixpkgs: + - Introduce the version declaration, required to distinguish major versions 6 and 7 - Remove `builtins.langVersion`, as it's not needed any more - Deprecate URL literals - Deprecate the `let-body` syntax - Drop support for leading zeroes on integers - Formalise change in float representation - - Version 8 is released only when version annotations according to version 7 are fully supported in Nixpkgs + - Major version 8 is released only when version annotations according to version 7 are fully supported in Nixpkgs From a461ef9d0ef1cac0c44cf6e12256fa205e08fb7e Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 6 Sep 2023 14:08:56 +0200 Subject: [PATCH 46/48] add another item to motivation section Co-authored-by: piegames --- rfcs/0137-nix-language-version.md | 1 + 1 file changed, 1 insertion(+) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 2f05e39ca..1d0fb85f8 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -52,6 +52,7 @@ Possible future changes that are in discussion: - Do something about `?` meaning two different things depending on where it occurs (`{ x ? "" }:` vs `x ? ""`) - Better support for static analysis - [Syntax for hexadecimal numbers](https://github.com/NixOS/nix/pull/7695) +- [Fix multiline strings](https://github.com/NixOS/nix/issues/3759) Other discussions around language changes: From 3e27f3c8fefdbd7c04814700d2f141fa696f706b Mon Sep 17 00:00:00 2001 From: fricklerhandwerk Date: Wed, 4 Oct 2023 12:52:01 +0200 Subject: [PATCH 47/48] update details on version declaration and backward compatibility MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit as discussed with @​roberth and @​piegamesde Co-authored-by: Yorick van Pelt --- rfcs/0137-nix-language-version.md | 54 +++++++++++++++++++++++++++++-- 1 file changed, 51 insertions(+), 3 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index 1d0fb85f8..bd7bd5cee 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -133,6 +133,10 @@ Other discussions around language changes: 1. The language version for Nix expressions is denoted in special syntax, at the beginning of a parse unit. A parse unit is any text stream, e.g. a file or string. + The evaluator must ignore a shebang line (starting with `#!) at the start of files, to leave room for additional tooling. + + The language version declaration is optional if it is instead made in an external per-project file. + The details of the per-project file syntax are out of scope for this proposal.
Arguments @@ -144,6 +148,7 @@ Other discussions around language changes: - This has the same trade-offs as when introducing the new syntax to begin with - (-) Editor support is made harder, since it requires switching the language based on file contents - (+) Making the language version accessible at all will probably outweigh the costs + - (+) Using a per-project file avoids littering every file with version declarations.
@@ -228,7 +233,6 @@ Other discussions around language changes: version \d*.\d*; ``` - The evaluator must ignore a shebang line (starting with `#!) at the start of files, to leave room for additional tooling. This implies that if no language version is specified in a Nix file, it is written in version 6 (the version implemented in the stable release of Nix at the time of writing this RFC). The syntax is open for bikeshedding. @@ -285,6 +289,10 @@ Other discussions around language changes: Examples include semantic changes or removal of `builtins`, operators, or syntactic constructs. + Values are decidedly not covered by versioning, but must instead stay the same indefinitely. + In particular, there must not be, e.g., string values internally tagged with different language versions. + This constraint can be loosened with a follow-up RFC. +
Arguments - (+) The principled solution: guarantees reproducibility given a fixed language version @@ -317,11 +325,20 @@ Other discussions around language changes:
1. Semantics are preserved across file boundaries for past language versions. + In other words, code written in an old language version evaluates to the same result when used from the new versions for all input values which are legal on the old version. - This should be fairly straightforward to implement since values passed around in the evaluator carry all the information needed to force them. + This should be fairly straightforward to implement since values passed around in the evaluator can carry all the information needed to force them. Newer parts of the evaluator can always wrap their values in interfaces that are accepted by older parts, as far as possible. - Example: [Best-effort interoperability](#best-effort-interoperability) + Examples: + - [Best-effort interoperability](#best-effort-interoperability) + - [Preserving semantics across version boundaries](#preserving-semantics-across-version-boundaries) + + When new value types are added to the language: + + - Passing new values to functions is allowed, as it cannot be prevented anyways due to laziness and composite types (like lists). + - Any values of unknown type to code from an older Nix version are treated as the opaque "external" type (which already exists for things like plugins). + Attempts at using them other than passing them around will thus cause type errors.
Arguments @@ -344,6 +361,14 @@ Other discussions around language changes:
+1. The backward compatibility mechanism must be "zero cost" when not used, meaning that no performance overhead must be paid when no legacy Nix files are imported. + +
Arguments + + - (+) This additional implementation constraint encourages being conservative with substantial changes. + +
+ 1. It is not possible to import expressions written in newer versions. Example: [Expressions are not forward compatible](#expressions-are-not-forward-compatible) @@ -545,6 +570,29 @@ $ nix-instantiate --eval b.nix 2 ``` +### Preserving semantics across version boundaries + +```nix +# a.nix +version 6.1; +{ float }; +toString float +``` + +```nix +# b.nix +version 7.0; +{ + old = import ./a.nix 1.1; + new = toString 1.1; +} +``` + +```console +$ nix-instantiate --eval b.nix --strict +{ old = "1.100000"; new = "1.1"; } +``` + ### Pathological example Usually existing code will be interacted with by calling functions. From 2fed67cf26848a03012f6807c4d703e468cebf80 Mon Sep 17 00:00:00 2001 From: fricklerhandwerk Date: Wed, 4 Oct 2023 12:59:37 +0200 Subject: [PATCH 48/48] add shepherd team --- rfcs/0137-nix-language-version.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/0137-nix-language-version.md b/rfcs/0137-nix-language-version.md index bd7bd5cee..602435061 100644 --- a/rfcs/0137-nix-language-version.md +++ b/rfcs/0137-nix-language-version.md @@ -3,8 +3,8 @@ feature: nix-language-version start-date: 2022-12-12 author: @fricklerhandwerk @yorickvp co-authors: @thufschmitt @Ericson2314 @infinisil -shepherd-team: -shepherd-leader: +shepherd-team: @piegamesde @sternenseemann @gabriel-doriath-dohler +shepherd-leader: @sternenseemann related-issues: https://github.com/NixOS/nix/issues/7255 ---