Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[windows][toolchain] Run Swift Runtime execution tests in x86_64 Android emulator #79081

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

weliveindetail
Copy link
Member

First version for early feedback

@weliveindetail
Copy link
Member Author

I expect to see the same issues with the Android Emulator in CI as in #78762. I will get back to that next week. Results I get locally:

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
********************
Failed Tests (2):
  Swift(android-x86_64) :: AutoDiff/validation-test/cross_module_derivative_attr.swift
  Swift(android-x86_64) :: AutoDiff/validation-test/cross_module_differentiation.swift

Testing Time: 12.00s

Total Discovered Tests: 11193
  Excluded   : 11159 (99.70%)
  Unsupported:     3 (0.03%)
  Passed     :    29 (0.26%)
  Failed     :     2 (0.02%)

8 warning(s) in tests

@weliveindetail weliveindetail marked this pull request as draft January 31, 2025 18:53
@finagolfin
Copy link
Member

Nice, any reason you only ran about 34 tests from the compiler validation suite locally? It would be great if the Windows CI ran more of those, alongside the new proposed Android community CI that runs everything but the executable tests on linux, swiftlang/swift-community-hosted-continuous-integration#55, ie those tests that don't require the emulator.

@weliveindetail
Copy link
Member Author

Thanks for your note. This is an early version for a minimal subset of tests. Idea was to get the emulator approach evaluated early, because it would take longer to stabilize in CI. Non-execution tests will follow. This will add a substantial amount of test runtime. I guess we will need to figure out how much of it we run in CI regularly.

@weliveindetail
Copy link
Member Author

Most non-executable tests failed because SWIFT_NATIVE_CLANG/LLVM_TOOLS_PATH wasn't set. Changing that brings us to a pass rate of almost 60%

Total Discovered Tests: 11193
  Unsupported        : 3588 (32.06%)
  Passed             : 6532 (58.36%)
  Expectedly Failed  :   35 (0.31%)
  Failed             : 1035 (9.25%)
  Unexpectedly Passed:    3 (0.03%)

@weliveindetail weliveindetail changed the title [windows][toolchain] Run Swift Runtime tests in x86_64 Android Emulator [windows][toolchain] Run Swift Runtime tests for x86_64 Android Feb 4, 2025
@weliveindetail
Copy link
Member Author

Let's see if we can reproduce this in CI
@swift-ci please test Windows

@weliveindetail
Copy link
Member Author

@swift-ci please test Windows

@weliveindetail
Copy link
Member Author

The build failed for an unrelated reason (flake in LLDB). Enabled AndroidSDK x86_64 now (and disable unrelated tests). Let's try again.

@swift-ci please test Windows

@weliveindetail
Copy link
Member Author

Pass rate on CI machines is much lower:

Total Discovered Tests: 18516
  Unsupported      : 5308 (28.67%)
  Passed           : 2284 (12.34%)
  Expectedly Failed:   30 (0.16%)
  Unresolved       : 6518 (35.20%)
  Failed           : 4376 (23.63%)

I will check the logs and investigate.

@weliveindetail
Copy link
Member Author

Actually, not a good idea to mix these up. I will do the non-executable tests in #79185 and keep this PR focussed on execution tests.

@weliveindetail weliveindetail changed the title [windows][toolchain] Run Swift Runtime tests for x86_64 Android [windows][toolchain] Run Swift Runtime execution tests in x86_64 Android emulator Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants