Xcode 15.2 on macos-14 image can not run the Apple Vision Pro simulator for testing, while other Apple platform (like iOS/tvOS/macOS) works

[dreampiggy]

Description

You can see actions/runner#3169 as well, which use markdown and has a better page render

Background

I recently switch from macos-13 Intel based runner image into the arm64 macos-14 one, and found a strange issue.

For macos-13, since it's Intel based, and does not has visionOS SDK and simulator installed, so I use the tricks from community:

      - name: Prepare VisionOS
        run: |
          defaults write com.apple.dt.Xcode AllowUnsupportedVisionOSHost -bool YES
          defaults write com.apple.CoreSimulator AllowUnsupportedVisionOSHost -bool YES
          xcodebuild -downloadPlatform visionOS

It used to work.

Current status

However, when switch into the macos-14 arm based machine. I remove the tricks above ⬆️ because the visionOS toolchain is already installed in Xcode 15.2 on arm based machine.

So, I choose to just run the test case, it will run the following command on GitHub runner

xcodebuild test-without-building -workspace "SDWebImage.xcworkspace" -scheme "Tests Vision" -configuration Debug -destination "platform=visionOS Simulator,name=Apple Vision Pro" CODE_SIGNING_ALLOWED=NO

However, the test seems never booted, it just suck and wait for 30 minutes then timeout. I have to cancel the job by myself.

From the link I provided https://github.com/SDWebImage/SDWebImage/actions/runs/8029051513, You can see, other jobs (iOS/macOS/tvOS) run the similar command and just work.

Only the Apple Vision Simulator will suck and no output at all. It will suck for 30 minutes and I have to cancel itself.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

The action link: https://github.com/SDWebImage/SDWebImage/actions/runs/8029051513

Current runner version: '2.313.0'
Operating System
  macOS
  14.[2](https://github.com/SDWebImage/SDWebImage/actions/runs/8029051513/job/21934904715#step:1:2).1
  2[3](https://github.com/SDWebImage/SDWebImage/actions/runs/8029051513/job/21934904715#step:1:3)C71
Runner Image
  Image: macos-1[4](https://github.com/SDWebImage/SDWebImage/actions/runs/8029051513/job/21934904715#step:1:4)-arm64
  Version: 20240116.1
  Included Software: https://github.com/actions/runner-images/blob/macOS-14/20240116.1/images/macos/macos-14-arm64-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-14%2F20240116.1
Runner Image Provisioner
  2.0.341.1+ea3907f30f87ddf37e44fa4[5](https://github.com/SDWebImage/SDWebImage/actions/runs/8029051513/job/21934904715#step:1:5)b[7](https://github.com/SDWebImage/SDWebImage/actions/runs/8029051513/job/21934904715#step:1:8)53abddb03[8](https://github.com/SDWebImage/SDWebImage/actions/runs/8029051513/job/21934904715#step:1:9)[11](https://github.com/SDWebImage/SDWebImage/actions/runs/8029051513/job/21934904715#step:1:13)b4
GITHUB_TOKEN Permissions
  Contents: read
  Metadata: read

Is it regression?

Maybe, but this is my first time to use macos-14 image and found this issue (previouslly macos-13 use Intel Mac, which seems does not has issue)

Expected behavior

The Github Runner image macos-14 should works when running xcodebuild test command for Apple Vision Pro simulator. Just like local Mac environment.

Actual behavior

See above in issue descrption, the Github Runner suck at xcodebuild test on Apple Vision Pro simulator only, not other platforms.

My Local test

I have a macOS 13.5 running on Apple MacBook Pro 13 (M1 Pro chip), and running the command, it just run succfully.

Note that Testing started in stdout.

However, only GitHub runner will suck without any extra output.

Possible Reason

Because I can not log-on the GitHub runner VM Mac machine, so I can not understand what's actually happended. Is this something like a GUI alert poped-up which breaks the xcodebuild test command to continue ?

Or, maybe something about the compatible issue Xcode 15's Apple Vision Pro Simulator on GitHub action. Which can not been booted at all. (But however, I try to use xcrun simctl boot "Apple Vision Pro" on GitHub action, it returns 0 and success.)

Repro steps

To Reproduce
Steps to reproduce the behavior:

Goto https://github.com/dreampiggy/SDWebImage/tree/ci/vision (this is fork repo)
Create a PR to the https://github.com/dreampiggy/SDWebImage, target branch is master (Or simple, just click https://github.com/SDWebImage/SDWebImage/compare/master...dreampiggy:SDWebImage:ci/vision?expand=1)
See the GitHub runner running status, it will suck as Unit Test (visionOS) job, but all other jobs will pass.

[MaksimZhukov]

Hello @dreampiggy!
Thank you for reporting this!
We will investigate the issue and come back to you as soon as we have any updates!

[PSchmiedmayer]

@MaksimZhukov @sergei-pyshnoi I can echo this report; we run into a similar issue in
StanfordBDHG/SwiftPackageTemplate#17

Testing failed:
	Simulator device returned an error for the requested operation.
	TestAppUITests-Runner encountered an error (Failed to install or launch the test runner. (Underlying Error: Simulator device returned an error for the requested operation. The system shell probably crashed. (Underlying Error: The request to terminate "edu.stanford.templatepackage.testapp.uitests.xctrunner" failed. The system shell probably crashed.)))

[sergei-pyshnoi]

Hello @dreampiggy . Can you try to switch to Xcode 15.4 ? Looks like your behavior is fixed with new version.

[PSchmiedmayer]

Thank you for working on this @sergei-pyshnoi!
I can't speak for others, but we have tried to validate if this is fixed with Xcode 15.4 but we still see that watchOS as well as visionOS builds don't run on the GitHub hosted runners while they work fine on our self-hosted macOS runners.

This can be replicated by forking https://github.com/StanfordBDHG/SwiftPackageTemplate and removing the runsonlabels: '["macOS", "self-hosted"]' inputs to our reusable workflow for the watchOS and visionOS builds running a minimal application with no logic and a UI test that only checks two texts on screen.

[sureshe456]

@dreampiggy - Does it help you out switching to Xcode 15.4?

[sureshe456]

@dreampiggy, We hope you try to switch to Xcode 15.4 then your code is fixed. So, we are closing the issue.

[PSchmiedmayer]

@sureshe456 @sergei-pyshnoi Should we create a separate issue that tracks the same problem and description as it doesn't seem to be resolved as noted in #9511 (comment) or would you like to re-open this issue?

Happy to provide some additional context if you need any additional elements to resolve the problem. We just recently also noticed it in techprimate/TPPDF#384 so it is not constrained to our setup and Actions.