Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0132] Meson Builds Nix #132

Draft
wants to merge 61 commits into
base: master
Choose a base branch
from
Draft
Changes from 13 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
9714f41
Use Meson to build Nix
AndersonTorres Aug 25, 2022
85759af
Modify feature attribute
AndersonTorres Aug 26, 2022
003393e
Small clarification
AndersonTorres Aug 26, 2022
3e0be88
Reformulates the alternatives section
AndersonTorres Aug 26, 2022
d4497fd
Answer new drawback
AndersonTorres Aug 26, 2022
18c732d
Typo
AndersonTorres Aug 26, 2022
a843fb1
Rewording
AndersonTorres Aug 26, 2022
0bf382d
Unfill the paragraphs
AndersonTorres Aug 26, 2022
6c3f5d2
Rename RFC file
AndersonTorres Aug 26, 2022
bd4bfe9
Small punctuation fixes
AndersonTorres Aug 27, 2022
e1a0620
Rewording
AndersonTorres Aug 27, 2022
0060937
Add usage example
AndersonTorres Aug 27, 2022
265020c
Add references
AndersonTorres Aug 27, 2022
b7b859b
Improvements on the wording
AndersonTorres Aug 27, 2022
3eea629
Reword the reference to Nix
AndersonTorres Aug 31, 2022
8650a36
Typo
AndersonTorres Aug 31, 2022
ab2dec1
Typo
AndersonTorres Aug 31, 2022
6980048
Reword a paragraph
AndersonTorres Aug 31, 2022
b58c8b6
Reword the detailed design
AndersonTorres Aug 31, 2022
17069b6
Reword the examples
AndersonTorres Aug 31, 2022
9821ad2
Add a paragraph about the current (quasi-) autotools
AndersonTorres Aug 31, 2022
ab42c2f
Reword commentary about transition
AndersonTorres Aug 31, 2022
3111e49
Reword alternatives
AndersonTorres Aug 31, 2022
9d89692
Reword commentary about Cmake
AndersonTorres Aug 31, 2022
dca9276
Remove commentary about backport
AndersonTorres Aug 31, 2022
38f3c44
Bring more items to future work
AndersonTorres Aug 31, 2022
714457b
Add references
AndersonTorres Aug 31, 2022
894946b
Reorder references
AndersonTorres Sep 1, 2022
72dec20
Reword motivation section
AndersonTorres Sep 1, 2022
6d8f6bd
Reword paragraph
AndersonTorres Sep 1, 2022
94eb5c7
Rework examples and interactions
AndersonTorres Sep 1, 2022
934555d
Rework drawbacks
AndersonTorres Sep 1, 2022
16e2033
Rework alternatives
AndersonTorres Sep 1, 2022
8dadf6a
Some fixups
AndersonTorres Sep 5, 2022
587d349
Include some more examples of build systems
AndersonTorres Sep 9, 2022
86bc4bc
Reorder references
AndersonTorres Sep 9, 2022
9ad0874
Reword cmake subsection
AndersonTorres Sep 9, 2022
5e621af
Move paragraphs about Meson and Ninja to detailed design section
AndersonTorres Sep 9, 2022
3de1372
Remove unneeded paragraph
AndersonTorres Sep 10, 2022
c5a9e48
Include Bazel as alternative
AndersonTorres Sep 10, 2022
3f1182c
Reword paragraph
AndersonTorres Sep 10, 2022
e361407
Punctuation
AndersonTorres Sep 10, 2022
a228c6f
More references
AndersonTorres Sep 10, 2022
aef0827
Reword
AndersonTorres Sep 11, 2022
212bf63
Typo
AndersonTorres Sep 11, 2022
4df4944
Reword
AndersonTorres Sep 11, 2022
0a41323
A bit more about Bazel
AndersonTorres Sep 11, 2022
e071944
add feature: out-of-source build
AndersonTorres Oct 6, 2022
d8904e9
Typos, typos and more typos
AndersonTorres Nov 11, 2022
680e6b8
Link to the now free-as-in-beer book from the creator of Meson
AndersonTorres Nov 11, 2022
0430b71
Another typo
AndersonTorres Nov 11, 2022
61ad7c4
Add shepherds
edolstra Nov 17, 2022
afca581
add cross-compilation support to the criteria of evaluation of altern…
AndersonTorres Aug 12, 2023
9bb63e0
Typo
AndersonTorres Feb 25, 2024
e6b4e05
Reword examples
AndersonTorres Feb 25, 2024
719562c
Add some more advantages for Bazel
AndersonTorres Feb 25, 2024
e4715c4
Reword and add more disadvantages for Bazel
AndersonTorres Feb 25, 2024
f37cedd
More references
AndersonTorres Feb 25, 2024
5b0a734
Reword Google company name
AndersonTorres Feb 25, 2024
56c86fa
Add Prior Art section
AndersonTorres Feb 25, 2024
b03e5b6
Rewords and comparisons
AndersonTorres Feb 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 124 additions & 0 deletions rfcs/0132-meson-builds-nix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
---
feature: meson_builds_nix
start-date: 2022-08-25
author: Anderson Torres
co-authors: @p01arst0rm
shepherd-team:
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
shepherd-leader:
related-issues:
---

# Summary
[summary]: #summary

Use meson as an alternative build system for the reference implementation of Nix.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

# Motivation
[motivation]: #motivation

Currently, the reference implementation of Nix evaluator and its companion toolset generated from the Nix source tree are built using the typical `./configure` shell script that relies on autoconf and the standard GNU Make utility.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

This build system became clunky and plastered, and consequently hard to understand, modify, improve and port to other systems besides Linux.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

Such state of things hinders development, specially outside the Linux and NixOS niches.

In light of this, we propose a novel, from-scratch alternative build infrastructure.

We expect to accomplish, among other goals,

- better code structuring;
- improved cross-platform support, especially in other programming environments, including but not limited to Unix-like operating systems;
- shorter build times;
- an overall improved user experience.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

# Detailed design
[design]: #detailed-design

A carefully crafted set of files written in Meson should be included in the Nix repository, in order to describe how to deploy the Nix repository, generating all the expected artifacts (command line tools, libraries, configuration files etc.)

This novel build infrastructure should be able to provide at least all the features already present on the current quasi-autotools implementation, possibly with a different user interface.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

# Examples and Interactions
[examples-and-interactions]: #examples-and-interactions

Currently @p01arst0rm is writing an implementation from scratch.

Some typical expected interactions are:

- Configuring: `meson setup build_dir`
- Building: `ninja -C build_dir` (or the backend-agnostic `meson -C build_dir compile`)
- Installing: `ninja -C build_dir install` (or the backend-agnostic `meson -C build_dir install`)
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

Usually, commandline options assume the format `-Dname=value`. This small change in user interface when compared to the typical `--name=value` is mostly harmless.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

# Drawbacks
[drawbacks]: #drawbacks

Some possible drawbacks:

- A new build system would require changes on the code
+ On the other hand, such changes are likely to improve the code base.

- A new build system requires the developers become familiarized with it
- Specially when this build system uses its own description language.
+ However, Meson is well documented, and its language is easy to grasp, specially for those familiarized with Python.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

- A new build system indirectly brings its own dependencies to the Nix project
- In particular, the reference implementation of Meson is written in Python.
- Further, this reference implementation generates script files meant to be consumed by Ninja, a tool written in C++ that acts like a Make replacement.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
- This particular setting brings concerns about complexifying the bootstrap route.
+ Given that Nix is currently written in C++, we can assume a C++ compiler as part of such a bootstrap route.
+ There are full-featured alternative tools that replace Meson and Ninja.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
+ Namely, Muon and Samurai are implementations of Meson and Ninja that require only a C compiler and a set of POSIX standard tools.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would definitely keep this drawback in mind and explore alternatives that don't expand the bootstrap closure.

Maybe you could also mention what the current bootstrap closure is?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asymmetric please read the prior PR to read discussions; meson does not increase bootstrap dependencies, it dramatically decreases them. available tools are provided that only require a working c compiler; c --> muon --> samurai --> nix. the benefit & end goal of this is removing gnu userland tools as a requirement of building, dramatically decreasing the amount of tools needed for bootstrapping.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please read the prior PR to read discussions

You mean NixOS/nix#3160?

meson does not increase bootstrap dependencies

Didn't you say in the RFC that Meson is in Python? What I wanted to say with my comment is to emphasize the idea that we shouldn't increase the bootstrap closure, and so I approve that you mentioned it and provided alternatives that don't increase it.

But AFAIU Python is not part of the current Nix bootstrap closure, so if we use straight Meson, rather than muon and/or Ninja, we would indeed increase it. Is that correct?

If that's the case, why wouldn't we choose muon directly?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who said that muon isn't "directly" chosen?

Maybe the confusion here is that the RFC is to use "Meson" the buildsystem description DSL, but not specifically to bootstrap Nix with "Meson" the reference implementation of "Meson".

What's the goal of this review comment? To rename this RFC to

[RFC 0132] Muon Builds Nix

???

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the confusion here is that the RFC is to use "Meson" the buildsystem description DSL, but not specifically to bootstrap Nix with "Meson" the reference implementation of "Meson".

That was indeed the thing that confused me

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we need a line in the RFC that says "meson the dsl language != meson.py the tool"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you could also mention what the current bootstrap closure is?

As far as I know, M4 (the somewhat obscure macro language) and Bash, and whatever autoconf brings.

Copy link

@p01arst0rm p01arst0rm Aug 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some required build programs include

  • flex
  • bison
  • sed
  • tar
  • dot
  • jq
  • bash
  • tar
  • patch
  • gcc (last time i attempted, clang compilation is very sketch)
  • autotools
  • m4


- A new build system would require new strategies from the end users
- In particular, package managers that deploy Nix for their respective platforms.
+ However, Meson and Ninja are nowadays a widespread toolset.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
+ Many open source projects use it, from mpv and dosbox-staging to Xorg and GNOME
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
+ According to Repology, Meson is present in 53 package manager's families

- The transition between between the old and new build systems should be smooth
- A wrapper script, maybe?
+ Meson is not an obscure project; a mere documentation update should suffice

# Alternatives
[alternatives]: #alternatives
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the sake of completeness there should probably also be an item here about bazel, buck & friends

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it is well summarized on the examples of waf and premake.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not exactly. The argument against them is that they are a language-specific DSL, but that's not the case of Bazel or Buck.

They do have a bootstrap problem ofc, but it's a different one (single implem, lots of deps)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A different one, and considerably worse than Meson. Cmake would be way better than them.

I do not believe they deserve more than a mention. If anyone is interested on picking Buck or Bazel or Imake, kudos to them!

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well now, let's be fair, there's definitely more than one implementation of Bazel.

The original design of it is Google's internal closed-source thing designed specifically to meet Google's distributed cloud computing needs, after all. ;) Bazel itself is just an open-source attempt to make BUILD / *.bzl files look like an acceptable thing for other projects to include to make Google's life easier. ;)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My list of complaints:

  1. I know nothing about them

  2. They are built in Java

  3. There is no alternative implementation in another language like C or C++ or even Python

  4. I do not like Java

  5. Java support on Nixpkgs is not the best in this planet

  6. Bootstrap Java is a hell:

    Unfortunately, most of the software needed for the bootstrap has been abandoned. To ensure that the JDK can be built from sources without the need for an existing installation of the OpenJDK we propose to continue maintaining the links of this bootstrap chain.

  7. Java is heavier than Python

  8. Have I said I do not like Java?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep sounds about right 🤣

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe too late. Has Zig been considered? It claims being an alternative:

but you can use Zig as a replacement for autotools, cmake, make, scons, ninja, etc.

Zig is offering to replace a project’s build system with a reasonable language using a declarative API for building projects, that also provides package management, and thus the ability to actually depend on other C libraries. The ability to have dependencies enables higher level abstractions, and thus the proliferation of reusable high-level code.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the entire point of meson is it offers the ability to compile all projects regardless of source language, build environment, linker, or compiler configurations. it offers flexibility and simplicity which allows easy maintainability into the future. the zig buildsystem is designed for c/c++ and is not nearly as well documented or flexible.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also Zig is oriented to Zig itself, while Meson is general-purpose.
Using Zig as Autotools replacement would be as worse as - or even worser than - using Bazel or Waf.


The alternatives are

- Doing nothing

It would keep the current code confusing and harder to work with.
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved

- Other building systems (cmake, waf, premake etc.)
- Their strenghts and weaknesses should be evaluated.
- Tools strongly tied to other programming languages are strongly discouraged, because they further complexifies the bootstrap route as discussed above.
- Namely, waf is basically a Python library, whereas premake is a Lua library.
- Cmake has many noteworthy advantages:
+ Can generates Make- and Ninja-compatible scripts;
+ Supports Windows NT;
+ Supports typical high level idiomatic constructions;
- On the other hand, the language is arguably more complex.

# Unresolved questions
[unresolved]: #unresolved-questions

Questions that deserve further inquiry:

- Unexpected interactions with Meson and Ninja
- Specially, vendoring and reproducibility.

# Future work
[future]: #future-work

- Deprecate the current build scripts
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
- Backport the new build system to Nix 2.3
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
- It was the latest release without Flakes support; it is important to bring such a deep modification to it.

AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved
# References
[references]: #references

- [Current work in progress from @p01arst0rm](https://github.com/NixOS/nix/pull/3160)
- [Muon](https://muon.build/), a C99 implementation of [Meson](https://meson.build/)
- [Samurai](https://github.com/michaelforney/samurai), a C99 implementation of [Ninja](https://ninja-build.org/)
- [Cmake](https://cmake.org/)
AndersonTorres marked this conversation as resolved.
Show resolved Hide resolved