Proposal: Allow Bluepill to run non-app hosted tests

[dostrander]

Proposal

Apple has provided a few different ways to run tests. Logic tests without an application, logic tests within an application, and a UI Test target. Currently Bluepill doesn't support tests without an app bundle path associated with them, actively erroring out when one isn't found. This proposal is to add the ability to run logic tests without an associated application to Bluepill.

Benefits

There are 2 main benefits for allowing non-app hosted tests.

Faster Signal

Tests that run without an application start more quickly and run on a much smaller process compared to an app, usually the xctest process. This allows them to run before a simulator is fully booted (aka when the simulator hits the home screen) and doesn't require a full app install. Fully booting and installing these apps can take a large portion of time and aren't needed when performing tests outside the application context. Removing this need to be hosted will allow them to finish much quicker.

Booting and installing an application uses lot of resources on the accompanying machine (CPU and memory) slowing down other simulators that are running simultaneously. This could also alleviate some of that burden.

One thing to note, on a small scale this faster signal doesn't have a huge impact but as you grow your codebase to many hours and/or many modules worth of tests this can be a significant boost

Single Test Execution Path
While I understand Bluepill was originally intended to be used for UI tests this creates a bifurcation of how tests are done and information about those tests are gathered. One for app hosted tests and one for non-hosted tests. Adding the ability to run non-hosted tests allows for developers to have a single way to run and gather information about their tests.

[ob]

I think this is a good idea and am fully supportive of doing it. I'm not sure how exactly we'd plug it in though... the existing architecture is sort of tied to running simulators. Are there any details of how the non-app hosted tests are run? Is it through the xctest binary? One idea would be to have a different mode for bp that executes the xctest binary directly and outputs the test result fragment that the bluepill binary knows how to merge.

[dostrander]

Are there any details of how the non-app hosted tests are run? Is it through the xctest binary?
The best explanation I've gotten on how these tests are run is through the fbidb documentation.

Logic tests are typically injected into a small host process like the xctest commandline. macOS has a xctest commandline at /usr/bin/xctest/, the iOS Simulator xctest executable is located at $DEVELOPER_DIR/Platforms/iPhoneSimulator.platform/Developer/Library/Xcode/Agents/xctest.

The xctest commandline will load the xctest bundle provided as an argument and run tests according to the arguments, or from an XCTestConfigurationFilePath provided as an environment variable passed to the process.

However they are saying they bypass xcodebuild. We use xcodebuild our implementation and we still get a simulator to boot however it never FULLY boots (never hits the home screen) before the tests finish and it sends back a test passed signal. This is something I think that needs exploring how xcodebuild exactly runs non-app hosted logic tests.

[ob]

I think the first step would be to start a WIP PR to add a few non-app hosted tests and a script to run them via xcodebuild (or however you typically run them). We can then start experimenting with calling xctest directly without needing to call xcodebuild.