`nix profile`: Allow referring to elements by human-readable name #8678

iFreilicht · 2023-07-09T21:02:54Z

Motivation

This PR replaces the index as the main handle for user-interaction with packages with a heuristically determined, human-readable name.

This enables commands nix profile upgrade and nix profile remove to refer to packages by their expected name like so:

$ nix profile install nixpkgs#vim
$ nix profile upgrade vim
$ nix profile remove vim

This is desirable as the index is a surprising interaction handle and the better attribute path requires matching with regex to use properly, otherwise the user gets a surprising error message. Using a name that closely matches what the user would expect is more intuitive and in line with the interface of other CLI package managers.

Context

This PR is a starting point for improving the UX of nix profile in general, see #7966.
In particular, it implements the feature requested in #7967 to fix #7960, #7961 and #7962.

The old index is still displayed, but using it to upgrade or remove a package prints a deprecation warning. Actually removing the index will be done in #9171.

Implementation strategy

A heuristic approach is used that generates the names when the profile manifest is loaded. The names are not written to the manifest, and so its structure does not change, making this change 100% backward- and forward-compatible. I deem this a good approach as it allows easy additions to these heuristics should we run into edge-cases in the future.

Duplicate packages are handled by suffixing the name with a number. This is simple and effective, and I assume we don't hit that case too often in reality.

Manually overriding the auto-generated name is not supported yet, as this would only work when installing a single package at once with the current evaluation of command line arguments, which I deemed out-of-scope for this PR.

This change is compatible with profiles managed by nix-env as well as profiles managed by nix profile that still have old packages installed by nix-env in them.

To avoid confusing "does not match any packages" warnings, this PR also adds additional warning messages when packages are found, but can't be upgraded due to having been installed via nix-env, a store-path or a locked flake ref:

warning: Found package 'vim', but it was not installed from a flake, so it can't be checked for upgrades!
warning: Found package 'github:NixOS/nixpkgs/842e90934a352f517d23963df3ec0474612e483c#legacyPackages.x86_64-linux.vim', but it was installed from a locked flake reference so it can't be upgraded!

this does not fix silently skipping upgrades for pinned flakes, though, see #8704.

Alternatives

I also considered storing the generated names in the manifest.json but decided against it to ensure potential edge-cases in the heuristics would not be permanently stored in the profile. This will us to change the heuristics quickly without any concerns about incompatibility and users will immediately benefit from upgrades without us having to implement additional mechanisms for overriding names manually, upgrading the generated names or anything like that.

We could add the names to the output of nix profile list --json, and that might actually be useful, but I decided against it for now as I'm not sure we want that command's output to differ from manifest.json.

For deduplication of names, we might want a smarter strategy or for the names to stay consistent, which would mean storing them in the manifest, but the semantics for that interacting with remove are unclear, so I decided against that for now as well.

Examples for `nix list` output:

New profile managed entirely by `nix profile`:

$ outputs/out/bin/nix profile list --profile ~/repos/experiments/test-profile
Name:               vim
Index:              0
Flake attribute:    legacyPackages.x86_64-linux.vim
Original flake URL: flake:nixpkgs
Locked flake URL:   github:NixOS/nixpkgs/842e90934a352f517d23963df3ec0474612e483c
Store paths:        /nix/store/6j38m8vm8gp9a8qpw3b7dj9g50x1w95n-vim-9.0.1562

Name:               autojump
Index:              1
Flake attribute:    legacyPackages.x86_64-linux.autojump
Original flake URL: flake:nixpkgs
Locked flake URL:   github:NixOS/nixpkgs/842e90934a352f517d23963df3ec0474612e483c
Store paths:        /nix/store/aj1hslpfsw2a6vx83wmc2q8pjsafzr70-autojump-22.5.3

Old profile generated by `nix-env`:

$ outputs/out/bin/nix profile list --profile /nix/var/nix/profiles/per-user/felix/profile-38-link
Name:               autojump
Index:              0
Store paths:        /nix/store/y7xy1cvnf1ad0by318ny6hr13dqaq6rm-autojump-22.5.3

Name:               bat
Index:              1
Store paths:        /nix/store/822la7lq7sc36m1d9l30xjl33j0iyyf1-bat-0.18.3

Name:               direnv
Index:              2
Store paths:        /nix/store/67lwbcd8wcfn0npi94dhwb16hq5mf9ab-direnv-2.28.0

Name:               fd
Index:              3
Store paths:        /nix/store/0a1xw59mzy0hwn1q7ncrmn83aamq7mkw-fd-8.2.1

In a profile managed by `nix profile` with old packages installed by `nix-env` with duplicate packages:

outputs/out/bin/nix profile list --profile ~/repos/experiments/old-test-profile
Name:               vim
Index:              0
Store paths:        /nix/store/r37bkiiin3xfmkjm98lgpybchr9gmm15-vim-9.0.1562

Name:               asdf-vm
Index:              1
Store paths:        /nix/store/ciyg3pml5msmvq8y2m24lflkkfldq1z0-asdf-vm-0.11.3

Name:               ripgrep
Index:              2
Flake attribute:    legacyPackages.x86_64-linux.ripgrep
Original flake URL: flake:nixpkgs
Locked flake URL:   github:NixOS/nixpkgs/842e90934a352f517d23963df3ec0474612e483c
Store paths:        /nix/store/rsyhr2viynf9smk0ipma9h7lx7v1hxs3-ripgrep-13.0.0

Name:               vim1
Index:              3
Flake attribute:    legacyPackages.x86_64-linux.vim
Original flake URL: flake:nixpkgs
Locked flake URL:   github:NixOS/nixpkgs/842e90934a352f517d23963df3ec0474612e483c
Store paths:        /nix/store/6j38m8vm8gp9a8qpw3b7dj9g50x1w95n-vim-9.0.1562

My own profile, managed declaratively inside a single, local flake:

$ outputs/out/bin/nix profile list
Name:               .dotfiles
Index:              0
Flake attribute:    packages.x86_64-linux.default
Original flake URL: path:/home/felix/.dotfiles
Locked flake URL:   path:/home/felix/.dotfiles?lastModified=1688827211&narHash=sha256-1IfyfZq3U8KDHMX2hvnPIIsHQqUUgoFcGWAw0HFgw74%3D
Store paths:        /nix/store/6rkydkpb56iavshjm2cafb341am4gcvd-ifreilicht-default-packages

To-Do:

Fix tests
Potentially fix formatting
Actually remove Index functionality
Fix nix profile remove not matching packages installed by nix-env

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

agreed on idea
agreed on implementation strategy
tests, as appropriate
- functional tests - tests/**.sh
- unit tests - src/*/tests
- integration tests - tests/nixos/*
documentation in the manual
documentation in the internal API docs
code and comments are self-explanatory
commit message explains why the change was made
new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

nixos-discourse · 2023-07-09T22:02:18Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-as-a-unified-format-for-profiles/29476/16

NixOS/nix#8678

nixos-discourse · 2023-07-15T17:12:25Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-07-14-nix-team-meeting-minutes-71/30472/1

blaggacao · 2023-07-15T20:48:27Z

a heuristic approach was used that generates the names when the profile manifest is loaded.

This heuristic, whatever it ends up being, is independently useful, i.e. for ecosystem commands such as treefmt.

bobvanderlinden

This works really well!

I was a bit confused at first why name wasn't incorporated into manifest.json, but I see what you're saying. It's not clear yet if all bases are covered correctly already in terms of naming. I don't see problems with your approach atm compared to what we already have.

I caught myself attempting to make 'this could be better for UX'-type suggestions I realized this PR is a steppingstone. It does the minimal to replace indexes with names for Nix profile. This is already a step in the right direction for nix profile UX 👍 Increments on this can be done in separate PRs I think.

Thanks for the work so far already!

src/nix/profile-upgrade.md

src/nix/profile-list.md

src/nix/profile.cc

nixos-discourse · 2023-09-15T20:35:25Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/flakes-as-a-unified-format-for-profiles/29476/17

nixos-discourse · 2023-10-09T20:07:07Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-flakes-is-an-experiment-that-did-too-much-at-once/32707/11

src/libutil/tests/url-name.cc

iFreilicht · 2023-10-17T22:24:23Z

@bobvanderlinden if you could have another look, that would be great! I updated the documentation a bit as well. The only thing missing are the release notes now.

TLATER

I have one small painpoint to raise, and that is that I think users actually expect this interface:

$ nix profile install nixpkgs#vim
$ nix profile upgrade nixpkgs#vim
$ nix profile remove nixpkgs#vim

The feasibility of that is a different question (because unfortunately the meaning of nixpkgs is neither unique nor clear in all contexts), and either way this change is probably the first step to that.

@bobvanderlinden also mentions potential UX improvements they intended to suggest, and this is probably one of them, I agree this point should not block the PR at all, but I think it's worth mentioning for reference.

Otherwise, this is great, working with names rather than numbers is obviously more human-friendly. Uniqueness is meanwhile guaranteed with numeric postfixes, so as far as I'm concerned this is a clean improvement over the current state.

It's also a beautiful diff, by the way, this is a very clean PR :)

src/nix/profile.cc

src/libutil/url-name.cc

src/nix/profile.cc

tests/functional/nix-profile.sh

iFreilicht · 2023-10-18T19:24:36Z

It's also a beautiful diff, by the way, this is a very clean PR :)

Thank you, I make extensive use of rebase -i master and it feels amazing.

PR is ready for another review, I now added the the release notes as well and would say this is final and ready to merge in my book.

The one remaining discussion would have to be decided by a maintainer.

Users may select specific outputs using the ^output syntax or selecting any output using ^*. URL parsing currently doesn't support these kinds of output references: parsing will fail. Currently `queryRegex` was reused for URL fragments, which didn't include support for ^. Now queryRegex has been split from fragmentRegex, where only the fragmentRegex supports ^.

bobvanderlinden · 2023-12-15T06:20:17Z

@edolstra @fricklerhandwerk could this get another look? What does this need to get approval/rejection? Or what does this need to speed up the reviewing process?

szlend · 2023-12-16T15:07:35Z

It kind of feels like nobody in the Nix team actually uses nix profile. I've been trying to advocate Nix in our company and I'm frankly embarrassed every time I have to explain to someone how to upgrade or remove a package. This PR is a small backwards compatible change and a massive UX improvement. A lot of thought and effort has been put into this and it would be a shame if it was abandoned because of lack of interest from the maintainers.

fricklerhandwerk · 2023-12-18T15:47:43Z

@edolstra said to take a look this week

nixos-discourse · 2023-12-19T02:59:13Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-12-18-nix-team-meeting-minutes-113/37050/1

…ad-of-index

bobvanderlinden · 2023-12-21T16:18:16Z

🥳 thanks for the review + merge!

Ericson2314 · 2023-12-21T19:02:48Z

src/libutil/url-name.hh

+#include "url.hh"
+#include "url-parts.hh"
+#include "util.hh"
+#include "split.hh"
+
+namespace nix {
+
+/**
+ * Try to extract a reasonably unique and meaningful, human-readable
+ * name of a flake output from a parsed URL.
+ * When nullopt is returned, the callsite should use information available
+ * to it outside of the URL to determine a useful name.
+ * This is a heuristic approach intended for user interfaces.
+ * @return nullopt if the extracted name is not useful to identify a
+ * flake output, for example because it is empty or "default".
+ * Otherwise returns the extracted name.
+ */
+std::optional<std::string> getNameFromURL(const ParsedURL & url);
+
+}


@iFreilicht I would appreciate if you could move this elsewhere. libutil should be just utilities and not define Nix-specific concepts like store paths or flake things. (In the "Domain-Driven Design" parlance, this is a "application layer" or "domain layer" concept, but libutil is supposed to be the "infrastructure layer".)

Sure. Could you maybe give me a hint where "elsewhere" could be?

Sure. I would put in src/libexpr/flake/ for now. There are other flakes things in there that don't strictly relate to evaluation already.

Done, see #9655

## Summary The latest nix version (2.20) changed how the nix profile output is represented: From the [release notes](https://nixos.org/manual/nix/stable/release-notes/rl-2.20): > nix profile now allows referring to elements by human-readable names NixOS/nix#8678 > [nix profile](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-profile) now uses names to refer to installed packages when running [list](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-profile-list), [remove](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-profile-remove) or [upgrade](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-profile-upgrade) as opposed to indices. Profile element names are generated when a package is installed and remain the same until the package is removed. > Warning: The manifest.nix file used to record the contents of profiles has changed. Nix will automatically upgrade profiles to the new version when you modify the profile. After that, the profile can no longer be used by older versions of Nix. and for `nix search`: > Disallow empty search regex in nix search [#9481](NixOS/nix#9481) > [nix search](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-search) now requires a search regex to be passed. To show all packages, use ^. TODOs: - [x] update `nix.readManifest` to handle the new format - [x] `nix search` requires a regex to be passed - [x] manually test on nix < 2.20 on devbox.sh to verify the older nix still works Fixes #1767 ## How was it tested? CICD should pass Installed nix 2.20.1 locally and am using Devbox with it to add, remove packages and run scripts and shell. verified flake updating works: 1. `examples/flakes/remote`. Did `devbox shell`, dropped the `v0.43.1` from process-compose flake, did `devbox update`, and verified that `process-compose` now had the latest version (IIRC `0.80+`) 2. `examples/flakes/php`. Did `devbox shell`, edited the flake to drop `ds` and did `devbox update`. Verified no `ds` in `php -m | grep ds`

Based off of commit 257b768 Upstream-PR: NixOS/nix#8678 Co-authored-by: Felix Uhl <[email protected]> Change-Id: Idcb7f6191ca3310ef9dc854197f7798260c3f71d

These names are parsed from the URL provided for that package Based off of commit 257b768 Upstream-PR: NixOS/nix#8678 Co-authored-by: Felix Uhl <[email protected]> Change-Id: I76d5f9cfb11d3d2915b3dd1db21d7bb49e91f4fb

Based off of commit 257b768 Upstream-PR: NixOS/nix#8678 Co-authored-by: Felix Uhl <[email protected]> Change-Id: Idcb7f6191ca3310ef9dc854197f7798260c3f71d

These names are parsed from the URL provided for that package Based off of commit 257b768 Upstream-PR: NixOS/nix#8678 Co-authored-by: Felix Uhl <[email protected]> Change-Id: I76d5f9cfb11d3d2915b3dd1db21d7bb49e91f4fb

iFreilicht requested review from fricklerhandwerk and edolstra as code owners July 9, 2023 21:02

github-actions bot added documentation new-cli Relating to the "nix" command labels Jul 9, 2023

iFreilicht marked this pull request as draft July 9, 2023 21:03

iFreilicht force-pushed the profile-names-instead-of-index branch from 8658be2 to 3a40929 Compare July 9, 2023 21:43

github-actions bot added the with-tests Issues related to testing. PRs with tests have some priority label Jul 9, 2023

max-privatevoid added a commit to privatevoid-net/nix-super that referenced this pull request Jul 10, 2023

Merge remote-tracking branch 'iFreilicht/profile-names-instead-of-index'

a4a636c

NixOS/nix#8678

edolstra self-assigned this Jul 14, 2023

iFreilicht force-pushed the profile-names-instead-of-index branch 2 times, most recently from 0539674 to 0c4c043 Compare July 16, 2023 12:07

iFreilicht marked this pull request as ready for review July 16, 2023 12:13

bobvanderlinden mentioned this pull request Aug 8, 2023

Nix recommends to remove the wrong installation name #8728

Closed

iFreilicht mentioned this pull request Aug 10, 2023

nix profiles are hard to reproduce #7965

Open

bobvanderlinden reviewed Sep 10, 2023

View reviewed changes

src/nix/profile-upgrade.md Outdated Show resolved Hide resolved

src/nix/profile-list.md Show resolved Hide resolved

src/nix/profile.cc Outdated Show resolved Hide resolved

iFreilicht mentioned this pull request Oct 16, 2023

Remove Index for nix profile remove and nix profile upgrade #9171

Closed

iFreilicht force-pushed the profile-names-instead-of-index branch from 0c4c043 to e1a74d6 Compare October 16, 2023 20:47

bobvanderlinden reviewed Oct 17, 2023

View reviewed changes

src/libutil/tests/url-name.cc Outdated Show resolved Hide resolved

iFreilicht force-pushed the profile-names-instead-of-index branch from e1a74d6 to 01ea317 Compare October 17, 2023 22:21

TLATER reviewed Oct 18, 2023

View reviewed changes

src/nix/profile.cc Show resolved Hide resolved

src/libutil/url-name.cc Outdated Show resolved Hide resolved

src/nix/profile.cc Outdated Show resolved Hide resolved

tests/functional/nix-profile.sh Show resolved Hide resolved

iFreilicht force-pushed the profile-names-instead-of-index branch 2 times, most recently from 7fbb9c4 to c1f6b77 Compare October 18, 2023 19:22

TLATER approved these changes Oct 19, 2023

View reviewed changes

iFreilicht and others added 2 commits November 6, 2023 21:21

Enable using human-readable name in nix profile

257b768

iFreilicht force-pushed the profile-names-instead-of-index branch from cfb52e6 to 9c0a09f Compare November 6, 2023 20:21

bobvanderlinden mentioned this pull request Dec 14, 2023

'nix profile' has terrible experience upgrading packages #7530

Closed

edolstra added 3 commits December 21, 2023 16:21

Merge remote-tracking branch 'origin/master' into profile-names-inste…

9d9d9ff

…ad-of-index

Fix indent

d77a39a

Typo

14508ad

edolstra enabled auto-merge December 21, 2023 15:26

Fix release notes

942d635

edolstra merged commit 17c202b into NixOS:master Dec 21, 2023
8 checks passed

iFreilicht deleted the profile-names-instead-of-index branch December 21, 2023 17:54

iFreilicht mentioned this pull request Dec 21, 2023

nix profile has bad UX #7966

Open

16 tasks

Ericson2314 reviewed Dec 21, 2023

View reviewed changes

bobvanderlinden mentioned this pull request Dec 21, 2023

nix profile subcommands rely on index identifier #7961

Closed

iFreilicht mentioned this pull request Dec 21, 2023

Move url-name utility to libexpr/flake #9655

Merged

edolstra mentioned this pull request Dec 22, 2023

Make profile element names stable #9656

Merged

This was referenced Jan 31, 2024

Failing to add packages due to Go unmarshalling error jetify-com/devbox#1767

Closed

[nix profile] Changes to support format changes from nix 2.20 jetify-com/devbox#1770

Merged

bobvanderlinden mentioned this pull request Feb 22, 2024

"nix profile" allows installing duplicate packages #5587

Closed

SandaruKasa mentioned this pull request Feb 29, 2024

Bad user-readable name for defaultPackage.* in nix profile #10118

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`nix profile`: Allow referring to elements by human-readable name #8678

`nix profile`: Allow referring to elements by human-readable name #8678

iFreilicht commented Jul 9, 2023 •

edited

Loading

nixos-discourse commented Jul 9, 2023

nixos-discourse commented Jul 15, 2023

blaggacao commented Jul 15, 2023

bobvanderlinden left a comment

nixos-discourse commented Sep 15, 2023

nixos-discourse commented Oct 9, 2023

iFreilicht commented Oct 17, 2023

TLATER left a comment •

edited

Loading

iFreilicht commented Oct 18, 2023

bobvanderlinden commented Dec 15, 2023

szlend commented Dec 16, 2023 •

edited

Loading

fricklerhandwerk commented Dec 18, 2023

nixos-discourse commented Dec 19, 2023

bobvanderlinden commented Dec 21, 2023

Ericson2314 Dec 21, 2023

iFreilicht Dec 21, 2023

Ericson2314 Dec 21, 2023

iFreilicht Dec 21, 2023

nix profile: Allow referring to elements by human-readable name #8678

nix profile: Allow referring to elements by human-readable name #8678

Conversation

iFreilicht commented Jul 9, 2023 • edited Loading

Motivation

Context

Implementation strategy

Alternatives

Examples for nix list output:

New profile managed entirely by nix profile:

Old profile generated by nix-env:

In a profile managed by nix profile with old packages installed by nix-env with duplicate packages:

My own profile, managed declaratively inside a single, local flake:

To-Do:

Checklist for maintainers

Priorities

nixos-discourse commented Jul 9, 2023

nixos-discourse commented Jul 15, 2023

blaggacao commented Jul 15, 2023

bobvanderlinden left a comment

Choose a reason for hiding this comment

nixos-discourse commented Sep 15, 2023

nixos-discourse commented Oct 9, 2023

iFreilicht commented Oct 17, 2023

TLATER left a comment • edited Loading

Choose a reason for hiding this comment

iFreilicht commented Oct 18, 2023

bobvanderlinden commented Dec 15, 2023

szlend commented Dec 16, 2023 • edited Loading

fricklerhandwerk commented Dec 18, 2023

nixos-discourse commented Dec 19, 2023

bobvanderlinden commented Dec 21, 2023

Ericson2314 Dec 21, 2023

Choose a reason for hiding this comment

iFreilicht Dec 21, 2023

Choose a reason for hiding this comment

Ericson2314 Dec 21, 2023

Choose a reason for hiding this comment

iFreilicht Dec 21, 2023

Choose a reason for hiding this comment

`nix profile`: Allow referring to elements by human-readable name #8678

`nix profile`: Allow referring to elements by human-readable name #8678

iFreilicht commented Jul 9, 2023 •

edited

Loading

Examples for `nix list` output:

New profile managed entirely by `nix profile`:

Old profile generated by `nix-env`:

In a profile managed by `nix profile` with old packages installed by `nix-env` with duplicate packages:

TLATER left a comment •

edited

Loading

szlend commented Dec 16, 2023 •

edited

Loading