[experiment] Migrate to bzlmod #24216

jwnrt · 2024-08-05T11:58:25Z

Expecting tonnes of failures here but wanted to see what breaks in CI while it's quiet. I know of a few breakages already around airgapped builds, the bitstream cache, and the C toolchain.

This will be broken into a few PRs later, but it includes:

Updating Bazel to 7.1.1.
Updating most dependencies.
Changing how airgapped builds work (due to changes in 7.1.1).
Changing how Rust dependencies work (due to changes in rules_rust).
Changing how Python dependencies work (due to changes in rules_python).
Migrating WORKSPACE to MODULE.bazel.

The lowrisc toolchain and closed-source manufacturing hooks are not yet migrated.

Problems found so far by this PR:

rules_python now refuses to run as root, we may need to use rootless containers in CI.
- For now I've added the ignore_root_user_error = True param which disables .pyc caching.
Bazel cannot find serde_annotate_derive for some reason. This was working before I rebased so I must have broken something.
- Nevermind, this is a rules_rust regression Dependency vendoring broken in 0.49.1 bazelbuild/rules_rust#2777.
Airgapped build takes way too long (2 hours before failure).
- Turned out to be a problem with the airgapped build itself, see [ci] Fix false-positive Bazel airgapped build test #24372.
Bitstream rules cannot load python dependencies.
- Worked around in Update bazel dependencies #24361 by making the jsonschema dependency optional.
License checker rule is racing internally, sometimes the python executable is there and sometimes it's not.
- Observed again in [bazel] Update Bazel to version 7.3.1 #24392 which indicates it's something to do with Bazel 7 and not Bzlmod.
- Fixed by Fix licence checker paths in Bazel 7 onwards misc-linters#39.

Previously we were on a very old commit (not a release) from 2020. Signed-off-by: James Wainwright <[email protected]>

We use this in OpenTitan so we should explicitly depend on it. Signed-off-by: James Wainwright <[email protected]>

There is a complicated dependency cycle here if we don't separate them. Signed-off-by: James Wainwright <[email protected]>

These dependencies are mostly important toolchain rules: * `rules_python` * `rules_cc` * `rules_foreign_cc` We need to load these right at the start of `WORKSPACE` to avoid our other dependencies (e.g. CRT) from installing different versions first. Bazel will stick with the first version installed even if we explicitly ask for another later. Signed-off-by: James Wainwright <[email protected]>

Signed-off-by: James Wainwright <[email protected]>

Our patch has been upstreamed here: bazelbuild/buildtools@762712d Signed-off-by: James Wainwright <[email protected]>

Signed-off-by: James Wainwright <[email protected]>

We were previously taking these from the Bazel repository at the version we were using. This has broken, but Bazel has a documented way of doing this now independent of version: https://bazel.build/run/build#repository_cache_with_bazel_7_or_later Signed-off-by: James Wainwright <[email protected]>

Signed-off-by: James Wainwright <[email protected]>

Drops the `system_includes` feature of `bindgen_toolchain`. Need to check for breakages. Signed-off-by: James Wainwright <[email protected]>

This change has been upstreamed, just waiting for it to merge. Signed-off-by: James Wainwright <[email protected]>

`rules_rust` has changed the layout of the Bazel files it outputs when repinning dependencies. The actual dependencies have not changed. Signed-off-by: James Wainwright <[email protected]>

Signed-off-by: James Wainwright <[email protected]>

jwnrt · 2025-01-08T16:56:22Z

Everything experimented on here has been merged!

jwnrt force-pushed the bzlmod branch 5 times, most recently from 2e61bc2 to 936f9e5 Compare August 9, 2024 08:57

jwnrt force-pushed the bzlmod branch from 936f9e5 to 086addd Compare August 13, 2024 08:55

jwnrt force-pushed the bzlmod branch 2 times, most recently from 55bb362 to 2caa217 Compare August 20, 2024 09:30

pamaury mentioned this pull request Aug 20, 2024

[python] Add support for python 3.12 #24360

Closed

jwnrt mentioned this pull request Aug 20, 2024

[bazel] Migrating to Bzlmod #24365

Closed

17 tasks

jwnrt added 20 commits August 21, 2024 11:40

[bazel] Update rules_cc to version 0.9.0

691cd39

Previously we were on a very old commit (not a release) from 2020. Signed-off-by: James Wainwright <[email protected]>

[bazel] Explicitly import platforms dependency

8ae7fb9

We use this in OpenTitan so we should explicitly depend on it. Signed-off-by: James Wainwright <[email protected]>

[bazel] Separate rules_fuzzing from Google deps

5fe1c93

There is a complicated dependency cycle here if we don't separate them. Signed-off-by: James Wainwright <[email protected]>

[bazel] Update tockloader to 1.13

5a5e039

Signed-off-by: James Wainwright <[email protected]>

[bazel] Freeze tockloader pip dependencies

0a46ec3

Signed-off-by: James Wainwright <[email protected]>

[bazel] Remove jsonschema from bitstream cache script

e310ee6

Signed-off-by: James Wainwright <[email protected]>

[bazel] Update rules_python to upstream 0.34.0

08d1bd1

Signed-off-by: James Wainwright <[email protected]>

[bazel] Update rules_fuzzing to 0.5.2

f6a904e

Signed-off-by: James Wainwright <[email protected]>

[bazel] Update buildifier to 6.4.0

89a2864

Our patch has been upstreamed here: bazelbuild/buildtools@762712d Signed-off-by: James Wainwright <[email protected]>

[bazel] Apply new buildifier fixes

b530e83

Signed-off-by: James Wainwright <[email protected]>

[bazel] Rename abseil-cpp repo to match bzlmod

57ebea5

Signed-off-by: James Wainwright <[email protected]>

[bazel] Update bazel to 7.3.1

c1ff4f7

Signed-off-by: James Wainwright <[email protected]>

[DELETEME] Create PR "Update Bazel to v7.3.1"

fe6fca1

Signed-off-by: James Wainwright <[email protected]>

[bazel] Update to upstream rules_rust

3f4cdab

Drops the `system_includes` feature of `bindgen_toolchain`. Need to check for breakages. Signed-off-by: James Wainwright <[email protected]>

[bazel] Add static lib patch to rules_rust

d976db3

This change has been upstreamed, just waiting for it to merge. Signed-off-by: James Wainwright <[email protected]>

[bazel] Repin Rust dependencies

c77d792

`rules_rust` has changed the layout of the Bazel files it outputs when repinning dependencies. The actual dependencies have not changed. Signed-off-by: James Wainwright <[email protected]>

[bazel] Remove support for local repositories in Tock

4dae584

Signed-off-by: James Wainwright <[email protected]>

[DELETEME] Create PR "Update rules_rust to upstream"

786e94f

Signed-off-by: James Wainwright <[email protected]>

jwnrt added 26 commits August 21, 2024 16:50

[bazel] Remove special-casing for serde-annotate dependency

aa45bb2

Signed-off-by: James Wainwright <[email protected]>

[bazel] Fix paths to Cargo dependencies

6f7ca5d

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move Tock dependencies to main Cargo.toml

9f38f68

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move rules_rust to bzlmod

5427f87

Signed-off-by: James Wainwright <[email protected]>

[bazel] Remove old crates

3374c25

Signed-off-by: James Wainwright <[email protected]>

[bazel] Separate tock repos from deps

e2b4db7

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move Tock repository to bzlmod

6178c31

Signed-off-by: James Wainwright <[email protected]>

[bazel] Add bzlmod extension for rules repos

737d88d

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move rules_python to bzlmod

610ad86

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move bitstreams repo to bzlmod

e47852c

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move nonhermetic rule to bzlmod

f5558cc

Signed-off-by: James Wainwright <[email protected]>

[bazel] Update bzlmod dependencies

19c2397

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move release rule to bzlmod

e71d747

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move lychee to bzlmod

3ddb22a

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move OpenOCD to bzlmod

29d68f6

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move FreeRTOS to bzlmod

a1c4edd

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move llvm_compiler_rt to bzlmod

2997d67

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move riscv-compliance to bzlmod

457dc10

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move coremark to bzlmod

c874f15

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move xkcp to bzlmod

6c0a4a9

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move hsm extras to bzlmod

1a47e56

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move nist cavp testvectors to bzlmod

6c78b28

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move wycheproof to bzlmod

2afbb45

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move sphincsplus to bzlmod

4bafa9c

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move HyperDebug firmware to bzlmod

f9528fb

Signed-off-by: James Wainwright <[email protected]>

[bazel] Move localtools registration to bzlmod

00a8158

Signed-off-by: James Wainwright <[email protected]>

jwnrt force-pushed the bzlmod branch from 2caa217 to 00a8158 Compare August 21, 2024 15:51

jwnrt mentioned this pull request Aug 23, 2024

[bazel] Update Bazel to version 7.3.1 #24392

Merged

jwnrt closed this Jan 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[experiment] Migrate to bzlmod #24216

[experiment] Migrate to bzlmod #24216

jwnrt commented Aug 5, 2024 •

edited

Loading

jwnrt commented Jan 8, 2025

[experiment] Migrate to bzlmod #24216

[experiment] Migrate to bzlmod #24216

Conversation

jwnrt commented Aug 5, 2024 • edited Loading

jwnrt commented Jan 8, 2025

jwnrt commented Aug 5, 2024 •

edited

Loading