Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Parse Apple SDK versions #131478

Draft
wants to merge 10 commits into
base: master
Choose a base branch
from
Draft

Conversation

madsmtm
Copy link
Contributor

@madsmtm madsmtm commented Oct 10, 2024

High-level goals of this PR:

  1. Easy cross compile on macOS host to iOS, tvOS, watchOS and visionOS.
    • If SDKROOT is specified for such a target, we need to find a macOS SDK to compile build scripts etc. for the host.
  2. Support cross compile to older macOS targets.
    • Similar to above, we are potentially given an SDK that is not suitable for the host machine (e.g. Aarch64 to x86 with the macOS 10.13 SDK that does not support Aarch64).
  3. Support cross compile on non-macOS host.
    • Give a warning if no SDK root is set, and guide the user towards setting it.
    • Should still support zig cc (though that maybe only works on macOS? I don't think Zig provides the .tdbs and headers to build and link iOS?)

Previously, we did a simplistic check against the SDKROOT path to try to figure out whether we should use the environment variable, or try to find a better matching SDK instead.

This is brittle since the user might want their SDK to be in a different place in the file system (that's kinda the whole idea of the SDKROOT variable), and also didn't work properly with SDKs in the Xcode Command Line Tools.

Instead, we now:

  1. Parse the SDKSettings.json file under the SDKROOT, to determine whether it is suitable for the target we're compiling for (which it might not be when e.g. using Cargo, and targetting iOS but compiling build scripts on macOS).
  2. If it's not, attempt to find another SDK.

This allows us to give much more detailed error messages, to better support cross-compile scenarios, and allows using parameters from the SDK, such as the SDK version, which we need to pass to ld64 for correctness (this fixes the remaining difference between linking with cc and ld64).

Final part of #129432.

Additionally, this fixes #80120 by introducing better error messages, so that it's possible to more clearly see when /Library/Developer/CommandLineTools is invalid.

Test

Tested with:

./x test tests/run-make/apple-* --target="aarch64-apple-darwin,aarch64-apple-ios,aarch64-apple-ios-macabi,aarch64-apple-ios-sim,aarch64-apple-tvos,aarch64-apple-tvos-sim,aarch64-apple-visionos,aarch64-apple-visionos-sim,aarch64-apple-watchos,aarch64-apple-watchos-sim,arm64_32-apple-watchos,armv7k-apple-watchos,armv7s-apple-ios,x86_64-apple-darwin,x86_64-apple-ios,x86_64-apple-ios-macabi,x86_64-apple-tvos,x86_64-apple-watchos-sim,x86_64h-apple-darwin"
IPHONEOS_DEPLOYMENT_TARGET=10.0 ./x test tests/run-make/apple-* --target=i386-apple-ios
MACOSX_DEPLOYMENT_TARGET=10.12 SDKROOT=$(pwd)/MacOSX10.13.sdk ./x test tests/run-make/apple-* --target=i686-apple-darwin

Which includes every Apple target except for arm64e.

Note that using the MacOSX10.13.sdk now works (and should work even more reliably once this change hits beta).

TODO

  • Manually test a few more scenarios:
    • Using cargo +stage1 inside Xcode for a project with both build scripts and proc macros (namely Bevy's).
    • Use zig cc without any SDKs available.
    • Cross compile with zig cc from a non macOS host.
    • Set SDKROOT to different crazy values.
    • Test running xcrun --sdk XYZ rustc +stage1.
  • Clean up diagnostic setup (and use subdiagnostics for certain things?).
  • Fix bootstrap so that it doesn't pass SDKROOT by default
    • Not sure if this is possible, since we also want the user to be able to override SDKROOT when building with bootstrap?
  • Test properly with i686

CC @BlackHoleFox @thomcc

Currently an unholy git combination of #131037, #131433 and #131477, so:
@rustbot blocked

The OS version depends on the deployment target environment variables,
the access of which we want to move to later in the compilation pipeline
that has access to more information, for example `env_depinfo`.
Also make the SDK name be the same casing as used in the file system.
The exact reasoning why we do not always pass the SDK root with
`-isysroot` to `cc` when linking on macOS eludes me (the git history
dead ends in rust-lang#100286), but I suspect it's because we want to support
`cc`s which do not support this option.

So instead, we pass the SDK root via the environment variable SDKROOT.
This way, compiler drivers that support setting the SDK root (such as
Clang and GCC) can use it, while compiler drivers that don't (presumably
because they figure out the SDK root in some other way) can just ignore
it.

This fixes rust-lang#80817 (by always
passing the SDK root, even when linking with cc on macOS).
Instead of being a type-alias to a tuple. This allows us to implement
Deserialize for OSVersion, which will be useful when parsing SDK
settings.
@rustbot
Copy link
Collaborator

rustbot commented Oct 10, 2024

r? @jieyouxu

rustbot has assigned @jieyouxu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

@rustbot rustbot added A-run-make Area: port run-make Makefiles to rmake.rs O-apple Operating system: Apple (macOS, iOS, tvOS, visionOS, watchOS) S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. labels Oct 10, 2024
@rust-log-analyzer

This comment has been minimized.

Previously, we did a simplistic check against the SDKROOT path to try
to figure out whether we should use the environment variable, or try to
find a better matching SDK instead.

This is brittle since the user might want their SDK to be in a
different place in the file system (that's kinda the whole idea of the
SDKROOT variable), and also didn't work properly with SDKs in the Xcode
Command Line Tools.

Instead, we now:
1. Parse the SDKSettings.json file under the SDKROOT, to determine
   whether it is suitable for the target we're compiling for (which it
   might not be when using Cargo, and targetting iOS but compiling build
   scripts on macOS).
2. If it's not, attempt to find another SDK.

This allows us to give much more detailed error messages, to better
support cross-compile scenarios, and allows using parameters from the
SDK (such as the SDK version) elsewhere in the future.
This is an important property of the binary that we were previously
setting incorrectly, and is a big step towards making binaries linked
with cc and ld64 equal.
@rust-log-analyzer
Copy link
Collaborator

The job mingw-check-tidy failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/

# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
           --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---
#13 2.789 Building wheels for collected packages: reuse
#13 2.790   Building wheel for reuse (pyproject.toml): started
#13 3.036   Building wheel for reuse (pyproject.toml): finished with status 'done'
#13 3.037   Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132715 sha256=dfa09868353292d98f811d3efdb0d54d07389e808efc71d68e3b93c514bf8bec
#13 3.037   Stored in directory: /tmp/pip-ephem-wheel-cache-z57sj4k1/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#13 3.040 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#13 3.429 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#13 3.430 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#13 3.943 Collecting virtualenv
#13 3.943 Collecting virtualenv
#13 4.003   Downloading virtualenv-20.26.6-py3-none-any.whl (6.0 MB)
#13 4.294      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 20.8 MB/s eta 0:00:00
#13 4.352 Collecting filelock<4,>=3.12.2
#13 4.362   Downloading filelock-3.16.1-py3-none-any.whl (16 kB)
#13 4.394 Collecting platformdirs<5,>=3.9.1
#13 4.403   Downloading platformdirs-4.3.6-py3-none-any.whl (18 kB)
#13 4.425 Collecting distlib<1,>=0.3.7
#13 4.447      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.0/469.0 KB 45.4 MB/s eta 0:00:00
#13 4.447      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.0/469.0 KB 45.4 MB/s eta 0:00:00
#13 4.525 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#13 4.712 Successfully installed distlib-0.3.9 filelock-3.16.1 platformdirs-4.3.6 virtualenv-20.26.6
#13 DONE 4.8s

#14 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#14 DONE 0.0s
---
DirectMap4k:      235456 kB
DirectMap2M:    11298816 kB
DirectMap1G:     7340032 kB
##[endgroup]
Executing TIDY_PRINT_DIFF=1 python2.7 ../x.py test            --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
+ TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
    Finished `dev` profile [unoptimized] target(s) in 0.05s
##[endgroup]
downloading https://static.rust-lang.org/dist/2024-09-22/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz
extracting /checkout/obj/build/cache/2024-09-22/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz to /checkout/obj/build/x86_64-unknown-linux-gnu/rustfmt
---
fmt check
Diff in /checkout/compiler/rustc_codegen_ssa/src/apple.rs:384:
     /// This is needed by the linker.
     ///
     /// settings["Version"] or settings["VersionMap"]["macOS_iOSMac"][settings["Version"]].
-    pub(crate) fn sdk_version(&self, target: &Target, sdkroot: &Path) -> Result<AppleOSVersion, AppleSdkError> {
+    pub(crate) fn sdk_version(
+        target: &Target,
+        sdkroot: &Path,
+        sdkroot: &Path,
+    ) -> Result<AppleOSVersion, AppleSdkError> {
         if target.abi == "macabi" {
                 .version_map
                 .version_map
fmt error: Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/rustfmt/bin/rustfmt" "--config-path" "/checkout" "--edition" "2021" "--unstable-features" "--skip-children" "--check" "/checkout/compiler/rustc_infer/src/infer/region_constraints/mod.rs" "/checkout/compiler/rustc_infer/src/infer/region_constraints/leak_check.rs" "/checkout/compiler/rustc_infer/src/infer/lexical_region_resolve/mod.rs" "/checkout/compiler/rustc_infer/src/infer/type_variable.rs" "/checkout/compiler/rustc_codegen_ssa/src/base.rs" "/checkout/compiler/rustc_codegen_ssa/src/assert_module_sources.rs" "/checkout/compiler/rustc_codegen_ssa/src/debuginfo/mod.rs" "/checkout/compiler/rustc_codegen_ssa/src/debuginfo/type_names.rs" "/checkout/compiler/rustc_infer/src/infer/canonical/instantiate.rs" "/checkout/compiler/rustc_infer/src/infer/canonical/mod.rs" "/checkout/compiler/rustc_infer/src/infer/canonical/query_response.rs" "/checkout/compiler/rustc_infer/src/infer/canonical/canonicalizer.rs" "/checkout/compiler/rustc_infer/src/infer/opaque_types/mod.rs" "/checkout/compiler/rustc_infer/src/infer/opaque_types/table.rs" "/checkout/compiler/rustc_infer/src/lib.rs" "/checkout/compiler/rustc_codegen_ssa/src/apple/tests.rs" "/checkout/compiler/rustc_codegen_ssa/src/common.rs" "/checkout/compiler/rustc_codegen_ssa/src/mono_item.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/coverageinfo.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/locals.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/mod.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/rvalue.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/statement.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/analyze.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/intrinsic.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/debuginfo.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/operand.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/place.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/constant.rs" "/checkout/compiler/rustc_codegen_ssa/src/mir/block.rs" "/checkout/compiler/rustc_codegen_ssa/src/codegen_attrs.rs" "/checkout/compiler/rustc_codegen_ssa/src/size_of_val.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/coverageinfo.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/misc.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/abi.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/mod.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/type_.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/backend.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/builder.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/declare.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/asm.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/intrinsic.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/debuginfo.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/statics.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/consts.rs" "/checkout/compiler/rustc_codegen_ssa/src/traits/write.rs" "/checkout/compiler/rustc_codegen_ssa/src/errors.rs" "/checkout/compiler/rustc_codegen_ssa/src/apple.rs" "/checkout/compiler/rustc_codegen_ssa/src/meth.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/link.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/archive.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/mod.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/rpath/tests.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/metadata.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/rpath.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/command.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/symbol_export.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/lto.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/write.rs" "/checkout/compiler/rustc_codegen_ssa/src/back/linker.rs" "/checkout/compiler/rustc_codegen_ssa/src/target_features.rs" "/checkout/compiler/rustc_codegen_ssa/src/lib.rs" "/checkout/compiler/rustc_mir_transform/src/mentioned_items.rs" "/checkout/compiler/rustc_infer/src/infer/resolve.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
  local time: Thu Oct 10 01:38:15 UTC 2024
  network time: Thu, 10 Oct 2024 01:38:15 GMT
##[error]Process completed with exit code 1.
Post job cleanup.

@jieyouxu jieyouxu removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 10, 2024
@jieyouxu
Copy link
Member

jieyouxu commented Oct 12, 2024

  • Fix bootstrap so that it doesn't pass SDKROOT by default
    • Not sure if this is possible, since we also want the user to be able to override SDKROOT when building with bootstrap?

bootstrap can possibly handle that as some kind of target specific config option, depending on the specifics of what you want.

@bors
Copy link
Contributor

bors commented Nov 2, 2024

☔ The latest upstream changes (presumably #132497) made this pull request unmergeable. Please resolve the merge conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-run-make Area: port run-make Makefiles to rmake.rs O-apple Operating system: Apple (macOS, iOS, tvOS, visionOS, watchOS) S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

macOS bug with newest update
5 participants