Build a minimized Nix with MinGW #8901

Ericson2314 · 2023-09-02T21:39:20Z

Motivation

Chasing my white whale of Nix on Windows again. Based off of @volth's great work.

At this point many features are stripped out, but this works:

Can run libnix{util,store,expr} unit tests
Can run some Nix commands

Context

~~Depends on #8887~~
~~Depends on #8920~~
~~Depends on #8886~~
~~Depends on #9518~~
~~Depends on #9519~~
~~Depends on #9736~~
~~Depends on #9737~~
~~Depends on #9757~~
~~Depends on #9767~~
~~Depends on #10329~~
~~Depends on #10361~~
~~Depends on #10362~~
~~Depends on #10363~~
~~Depends on #10364~~
~~Depends on #10399~~
~~Depends on #10400~~
~~Depends on #10401~~

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

agreed on idea
agreed on implementation strategy
tests, as appropriate
- functional tests - tests/**.sh
- unit tests - src/*/tests
- integration tests - tests/nixos/*
documentation in the manual
documentation in the internal API docs
code and comments are self-explanatory
commit message explains why the change was made
new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

Artturin · 2023-09-04T20:28:07Z

I guess this overlaps with cygwin stuff CC @corngood #4953

Ericson2314 · 2023-09-04T20:29:11Z

Oh yes it does! If we could get a cygwin (and msys2) build in CI that would be a great first step.

corngood · 2023-09-05T00:03:55Z

I've never had much trouble building nix on cygwin. What I did have a lot of trouble with is fibers/coroutines, which are used by nix via boost. They are sort of fundamentally broken in cygwin, which causes nix to crash (e.g. on network operations).

I actually proposed a patch to cygwin which allows them to work, but it hasn't been merged:

https://cygwin.com/pipermail/cygwin-developers/2020-September/011970.html

Last time I did any work on this, it was mostly on the nixpkgs side. The idea was to be able to cross compile a patched cygwin1.dll, and use that to bootstrap, and build up from there. Unfortunately I haven't had time to work on it for a while.

I gave up on bootstrapping using unmodified cygwin.

dpulls · 2023-09-25T17:46:58Z

🎉 All dependencies have been resolved !

dpulls · 2023-11-05T20:53:14Z

🎉 All dependencies have been resolved !

Ericson2314 · 2023-11-06T07:34:02Z

I think we'll want to install the unit tests into a separate output so we can run them on windows.

nixos-discourse · 2023-11-20T14:47:43Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-11-17-nix-team-meeting-minutes-104/35753/1

2xsaiko · 2024-04-08T01:12:01Z

Just curious, what is the oldest version of Windows this supports?

(Also I haven't looked too closely at the code but related: if you're touching winapi you might want to enable UTF-8 mode so you don't have to touch wchar_t which I think is only supported on Windows 10 and up)

Ericson2314 · 2024-04-08T01:19:33Z

Just curious, what is the oldest version of Windows this supports?

Don't know!

(Also I haven't looked too closely at the code but related: if you're touching winapi you might want to enable UTF-8 mode so you don't have to touch wchar_t which I think is only supported on Windows 10 and up)

Does that support ill-formed UTF-16 the way https://simonsapin.github.io/wtf-8/ does? In any event the plan for paths (main place this comes up) is using std::filesystem::path (#9205) which I suppose makes working with the native 16-bit characters less bad.

2xsaiko · 2024-04-08T17:19:36Z

Does that support ill-formed UTF-16 the way https://simonsapin.github.io/wtf-8/ does? In any event the plan for paths (main place this comes up) is using std::filesystem::path (#9205) which I suppose makes working with the native 16-bit characters less bad.

No, it seems to be actual UTF-8. What it does when you explicitly call the WideCharToMultiByte function it depends on whether the MB_ERR_INVALID_CHARS is set:

MB_ERR_INVALID_CHARS Fail if an invalid input character is encountered.

Starting with Windows Vista, the function does not drop illegal code points if the application does not set this flag, but instead replaces illegal sequences with U+FFFD (encoded as appropriate for the specified codepage).

Windows 2000 with SP4 and later, Windows XP: If this flag is not set, the function silently drops illegal code points. A call to GetLastError returns ERROR_NO_UNICODE_TRANSLATION.

I assume the default with conversions in Windows's own functions is that it isn't set, no idea though.

For paths I would also just use std::filesystem::path so you don't have to worry about that though, yeah.

src/libcmd/repl.cc

src/libexpr/eval.hh

edolstra · 2024-04-17T09:26:34Z

src/libfetchers/git-utils.cc

@@ -151,11 +153,11 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
    {
        initLibGit2();

-        if (pathExists(path.native())) {
-            if (git_repository_open(Setter(repo), path.c_str()))
+        if (pathExists(path.string())) {


I'm not entirely clear on the difference between native() and string(), but the docs for string() say:

Otherwise, if path::value_type is wchar_t, conversion, if any, is unspecified. This is the case on Windows, where wchar_t is 16 bit and the native encoding is UTF-16.

which seems worrying.

I think the more we use std::filesystem::path the more it will go away --- for now I am fine with it because it has no effect on Unix and it gets Windows at least compiling.

src/libmain/shared.cc

src/libstore/store-api.cc

src/libutil/file-descriptor.hh

edolstra · 2024-04-17T10:03:01Z

src/libutil/file-system.cc

+    AutoCloseFD fd = toDescriptor(open(path.c_str(), O_RDONLY
+// TODO
+#ifndef _WIN32
+       | O_CLOEXEC


Maybe we can define O_CLOEXEC as 0 on Windows in a header? Are filehandles created by open() inherited by default on Windows? We could use CreateFile() which returns a handle that isn't inherited by default.

Yeah I hope to go to CreateFile and side-step this issue longer term. (As the original Windows port did.) Just didn't do that for now to minimize CPP / windows-specific code for the first version.

src/libutil/unix/environment-variables.cc

edolstra · 2024-04-17T10:11:59Z

src/nix-env/nix-env.cc


    auto [gens, curGen] = findGenerations(globals.profile);

    RunPager pager;

    for (auto & i : gens) {
+#ifdef _WIN32 // TODO portable wrapper in libutil


This could probably be made portable using std::chrono, though the time zone stuff is C++20.

I'll make this a good-first-issue follow-up task, since it is localized.

This keeps the call sites simple, eventually this should be filled in.

At this point many features are stripped out, but this works: - Can run libnix{util,store,expr} unit tests - Can run some Nix commands Co-Authored-By volth <[email protected]> Co-Authored-By Brian McKenna <[email protected]>

We don't need it now that our (minimized) Windows build of Nix succeeds!

This is a hacky solution, but it will do for now.

Ericson2314 · 2024-04-17T16:30:27Z

Yay this is finally approved! Fixed what things I could from the review. The rest I will create issues for.

nixos-discourse · 2024-04-17T17:04:38Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-on-windows/1113/109

github-actions bot added new-cli Relating to the "nix" command with-tests Issues related to testing. PRs with tests have some priority fetching Networking with the outside (non-Nix) world, input locking labels Sep 2, 2023

This was referenced Sep 4, 2023

Split utils.hh #8913

Closed

Split up util.{hh,cc} #8920

Merged

corngood mentioned this pull request Sep 5, 2023

fixes for cygwin #4953

Open

Ericson2314 force-pushed the mingw branch from 61c8df0 to 3906cdd Compare September 7, 2023 18:45

Ericson2314 force-pushed the mingw branch from 3906cdd to f553c4c Compare September 20, 2023 13:08

Ericson2314 changed the title ~~Build Nix with MinGW~~ Build libnixutil with MinGW Oct 21, 2023

Ericson2314 force-pushed the mingw branch from f553c4c to e17e095 Compare October 21, 2023 17:29

github-actions bot added store Issues and pull requests concerning the Nix store repl The Read Eval Print Loop, "nix repl" command and debugger labels Oct 21, 2023

Ericson2314 force-pushed the mingw branch 3 times, most recently from 21abf88 to 1aaf36b Compare October 22, 2023 16:50

Ericson2314 added the windows label Oct 22, 2023

Ericson2314 force-pushed the mingw branch from 1aaf36b to 034e938 Compare October 30, 2023 20:40

Ericson2314 force-pushed the mingw branch from 034e938 to 73d212a Compare November 6, 2023 07:27

github-actions bot removed store Issues and pull requests concerning the Nix store repl The Read Eval Print Loop, "nix repl" command and debugger labels Nov 6, 2023

Ericson2314 force-pushed the mingw branch 2 times, most recently from 1ff607e to c9f5d8a Compare November 15, 2023 00:09

github-actions bot added the store Issues and pull requests concerning the Nix store label Nov 15, 2023

Ericson2314 force-pushed the mingw branch from a6fac58 to 319d77c Compare April 8, 2024 00:56

Ericson2314 force-pushed the mingw branch from 319d77c to ab8da72 Compare April 8, 2024 13:12

Ericson2314 force-pushed the mingw branch 3 times, most recently from 61b74e5 to 78ff3a1 Compare April 15, 2024 15:28

edolstra approved these changes Apr 17, 2024

View reviewed changes

Create no-op Window pathlocks implementation

2248a3f

This keeps the call sites simple, eventually this should be filled in.

Ericson2314 force-pushed the mingw branch from 78ff3a1 to 408038c Compare April 17, 2024 15:48

Ericson2314 added 4 commits April 17, 2024 12:26

Build a minimized Nix with MinGW

8433027

At this point many features are stripped out, but this works: - Can run libnix{util,store,expr} unit tests - Can run some Nix commands Co-Authored-By volth <[email protected]> Co-Authored-By Brian McKenna <[email protected]>

Get rid of shellCrossSystems

98691b4

We don't need it now that our (minimized) Windows build of Nix succeeds!

Prefix -DNIX_ paths to be windows-complient for windows

b529d91

This is a hacky solution, but it will do for now.

Relax store path canonicalization

1063aa5

Ericson2314 force-pushed the mingw branch from 408038c to 1063aa5 Compare April 17, 2024 16:26

Ericson2314 enabled auto-merge April 17, 2024 16:29

This was referenced Apr 17, 2024

Rename SearchPath #10539

Closed

Implement setStackSize on Windows #10540

Closed

Use portable C++ Pseudorandom number generator #10541

Open

Abstract over getrusage #10542

Open

Implement updateWindowSize on windows #10543

Closed

Ericson2314 merged commit ac253fb into NixOS:master Apr 17, 2024
9 checks passed

Ericson2314 deleted the mingw branch April 17, 2024 16:58

This was referenced Apr 17, 2024

Implement process spawning on Windows #10544

Closed

Implement PathLocks on windows #10545

Closed

Ericson2314 mentioned this pull request Apr 17, 2024

Run unit tests in Wine in CI #10547

Closed

steve-chavez mentioned this pull request Apr 25, 2024

Windows release 12.0.2 is linked against external DLLs, but doesn't bundle them or provide instructions on how to get them PostgREST/postgrest#3136

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build a minimized Nix with MinGW #8901

Build a minimized Nix with MinGW #8901

Ericson2314 commented Sep 2, 2023 •

edited

Loading

Artturin commented Sep 4, 2023

Ericson2314 commented Sep 4, 2023

corngood commented Sep 5, 2023

dpulls bot commented Sep 25, 2023

dpulls bot commented Nov 5, 2023

Ericson2314 commented Nov 6, 2023

nixos-discourse commented Nov 20, 2023

2xsaiko commented Apr 8, 2024

Ericson2314 commented Apr 8, 2024

2xsaiko commented Apr 8, 2024

edolstra Apr 17, 2024

Ericson2314 Apr 17, 2024

edolstra Apr 17, 2024

Ericson2314 Apr 17, 2024

edolstra Apr 17, 2024

Ericson2314 Apr 17, 2024

Ericson2314 commented Apr 17, 2024

nixos-discourse commented Apr 17, 2024

Build a minimized Nix with MinGW #8901

Build a minimized Nix with MinGW #8901

Conversation

Ericson2314 commented Sep 2, 2023 • edited Loading

Motivation

Context

Checklist for maintainers

Priorities

Artturin commented Sep 4, 2023

Ericson2314 commented Sep 4, 2023

corngood commented Sep 5, 2023

dpulls bot commented Sep 25, 2023

dpulls bot commented Nov 5, 2023

Ericson2314 commented Nov 6, 2023

nixos-discourse commented Nov 20, 2023

2xsaiko commented Apr 8, 2024

Ericson2314 commented Apr 8, 2024

2xsaiko commented Apr 8, 2024

edolstra Apr 17, 2024

Choose a reason for hiding this comment

Ericson2314 Apr 17, 2024

Choose a reason for hiding this comment

edolstra Apr 17, 2024

Choose a reason for hiding this comment

Ericson2314 Apr 17, 2024

Choose a reason for hiding this comment

edolstra Apr 17, 2024

Choose a reason for hiding this comment

Ericson2314 Apr 17, 2024

Choose a reason for hiding this comment

Ericson2314 commented Apr 17, 2024

nixos-discourse commented Apr 17, 2024

Ericson2314 commented Sep 2, 2023 •

edited

Loading