Adapt scheduler to work with dynamic derivations #8829
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
with-tests
4 tasks
Ericson2314
changed the title
Ericson2314
changed the title
Ericson2314
force-pushed
the
build-dynamic-derivations
branch
2 times, most recently
from
4e3db84
to
a95779d
Compare
github-actions
bot
removed
the
store
Ericson2314
force-pushed
the
build-dynamic-derivations
branch
from
a95779d
to
f396d8c
Compare
tomberek
reviewed
Aug 16, 2023
tomberek
reviewed
Aug 16, 2023
| @@ -18,4 +18,6 @@ clearStore | |||
|
|
|||
| drvDep=$(nix-instantiate ./text-hashed-output.nix -A producingDrv) | |||
|
|
|||
| expectStderr 1 nix build "${drvDep}^out^out" --no-link | grepQuiet "Building dynamic derivations in one shot is not yet implemented" | |||
| out2=$(nix build "${drvDep}^out^out" --no-link) | |||
4 tasks
Ericson2314
force-pushed
the
build-dynamic-derivations
branch
2 times, most recently
from
c355ead
to
060e9c1
Compare
Ericson2314
added a commit
to obsidiansystems/nix
that referenced
this pull request
Aug 16, 2023
See that issue for details on the general goal. The `DerivationGoal::derivationType` field had a bogus initialization, now caught, so I made it `std::optional`. I think NixOS#8829 can make it non-optional again because it will ensure we always have the derivation when we construct a `DerivationGoal`.
8 tasks
Ericson2314
added a commit
to obsidiansystems/nix
that referenced
this pull request
Aug 18, 2023
Types converted: - `NixStringContextElem` - `OutputsSpec` - `ExtendedOutputsSpec` - `DerivationOutput` - `DerivationType` The `DerivationGoal::derivationType` field had a bogus initialization, now caught, so I made it `std::optional`. I think NixOS#8829 can make it non-optional again because it will ensure we always have the derivation when we construct a `DerivationGoal`. See that issue (NixOS#7479) for details on the general goal. `git grep 'Raw::Raw'` indicates the two types I didn't yet convert `DerivedPath` and `BuiltPath` (and their `Single` variants) . This is because @roberth and I (can't find issue right now...) plan on reworking them somewhat, so I didn't want to churn them more just yet.
Ericson2314
added a commit
to obsidiansystems/nix
that referenced
this pull request
Aug 18, 2023
Types converted: - `NixStringContextElem` - `OutputsSpec` - `ExtendedOutputsSpec` - `DerivationOutput` - `DerivationType` The `DerivationGoal::derivationType` field had a bogus initialization, now caught, so I made it `std::optional`. I think NixOS#8829 can make it non-optional again because it will ensure we always have the derivation when we construct a `DerivationGoal`. See that issue (NixOS#7479) for details on the general goal. `git grep 'Raw::Raw'` indicates the two types I didn't yet convert `DerivedPath` and `BuiltPath` (and their `Single` variants) . This is because @roberth and I (can't find issue right now...) plan on reworking them somewhat, so I didn't want to churn them more just yet. Co-authored-by: Eelco Dolstra <[email protected]>
Ericson2314
added a commit
to obsidiansystems/nix
that referenced
this pull request
Aug 18, 2023
Types converted: - `NixStringContextElem` - `OutputsSpec` - `ExtendedOutputsSpec` - `DerivationOutput` - `DerivationType` Existing ones mostly conforming the pattern cleaned up: - `ContentAddressMethod` - `ContentAddressWithReferences` The `DerivationGoal::derivationType` field had a bogus initialization, now caught, so I made it `std::optional`. I think NixOS#8829 can make it non-optional again because it will ensure we always have the derivation when we construct a `DerivationGoal`. See that issue (NixOS#7479) for details on the general goal. `git grep 'Raw::Raw'` indicates the two types I didn't yet convert `DerivedPath` and `BuiltPath` (and their `Single` variants) . This is because @roberth and I (can't find issue right now...) plan on reworking them somewhat, so I didn't want to churn them more just yet. Co-authored-by: Eelco Dolstra <[email protected]>
Ericson2314
force-pushed
the
build-dynamic-derivations
branch
2 times, most recently
from
7b7416c
to
d77a2c2
Compare
roberth
reviewed
Aug 19, 2023
Ericson2314
force-pushed
the
build-dynamic-derivations
branch
3 times, most recently
from
ccdca62
to
b174ef8
Compare
Ericson2314
force-pushed
the
build-dynamic-derivations
branch
from
b174ef8
to
2e0f461
Compare
|
OK, I think all review items are addressed! |
roberth
reviewed
Aug 25, 2023
Hopefully they make the code easier to understand!
…ath` Now we are consistent with the other `resolveDerivedPath`, and other such functions.
We're about to split up `DerivationGoal` a bit. At that point `makeDerivationGoal` will mean something more specific than it does today. (Perhaps a future rename will make this clearer.) On the other hand, the more public `Worker::makeGoal` function will continue to work exactly as before. So by moving some call sites to use that instead, we preemptively avoid issues in the next step.
Ericson2314
force-pushed
the
build-dynamic-derivations
branch
from
bf4361f
to
5c5e332
Compare
roberth
reviewed
Aug 25, 2023
To avoid dealing with an optional `drvPath` (because we might not know it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal just builds/substitutes the derivation file, and then kicks of a build for that obtained derivation; in other words it does the chaining of goals when the drv file is missing (as can already be the case) or computed (new case). This also means the `getDerivation` state can be removed from `DerivationGoal`, which makes the `BasicDerivation` / in memory case and `Derivation` / drv file file case closer together. The map type is factored out for clarity, and because we will soon hvae a second use for it (`Derivation` itself). Co-authored-by: Robert Hensing <[email protected]>
Ericson2314
force-pushed
the
build-dynamic-derivations
branch
from
6ce18c5
to
5e3986f
Compare
roberth
approved these changes
Aug 25, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
To avoid dealing with an optional
drvPath(because we might not know it yet) everywhere, make anCreateDerivationAndRealiseGoal. This goal just builds/substitutes the derivation file, and then kicks of a build for that obtained derivation; in other words it does the chaining of goals when the drv file is missing (as can already be the case) or computed (new case).This also means the
getDerivationstate can be removed fromDerivationGoal, which makes theBasicDerivation/ in memory case andDerivation/ drv file case closer together.Context
The description is taken from the final commit. There are two preparatory ones before it.
Part of RFC 92: dynamic derivations.
Checklist for maintainers
Maintainers: tick if completed or explain if not relevant
tests/**.shsrc/*/teststests/nixos/*Priorities
Add 👍 to pull requests you find important.