Reusing the main output base for Xcode builds

[brentleyjones]

Currently rules_xcodeproj makes use of two additional output bases, nested inside of the output base that you run the generation from; one for project generation and Xcode builds, and one for Index Build.

For people that also build on the command-line (and aren’t using rules_xcodeproj’s command-line API), this means outputs end up in up to three output bases and up to three Bazel servers retain analysis cache memory.

Currently there isn’t a way around the Index Build output base, otherwise you can run into weird delays in starting builds. So let’s ignore that one for now. But the build output base, why does that one exist?

There are a couple reasons for the different build output base:

1. Builds in Xcode can’t be delayed by, and can’t delay, bazel commands on the command-line
2. Bazel commands on the command-line can’t invalidate the analysis cache of Xcode builds, and vice versa

The first reason is something that people can choose to deal with, so let’s focus on the second one. When build-affecting command-line flags change between bazel commands, the analysis cache is discarded. This can result in a substantial delay before your command even starts to do meaningful work. rules_xcodeproj sets various command-line flags for performance and correctness reasons, which means it would be incredibly easy to invalidate the analysis cache if it shared an output base with normal bazel commands.

But let’s say that you find a way to ensure the same flags are set when you run bazel commands (e.g. with a properly configured tools/bazel wrapper script). Then you would theoretically not need one of the additional output bases. There are some additional problems that would need to be tackled, like getting cache hits on building targets (because of the transitions that rules_xcodeproj uses), but otherwise this would “work”.

Which brings me to the questions I have for everyone:

1. Does this additional output base affect you/your team? If so, in which way, and to what extent?
2. Would having an option to reuse the normal output base be beneficial to you?
3. What, if anything, would rules_xcodeproj need to do to assist you in using the same output base (e.g. to help ensure the right flags are set to prevent analysis cache discarding, or ways to build targets with the same transitions that rules_xcodeproj applies)? And how important of a change would this be (e.g. would you/your organization be willing to sponsor the change if it’s non-trivial to implement and/or support?)

[luispadron]

From our experience at Cash App.

1. Yes. This affects us in the sense that it takes more memory, CPU, and probably the biggest reason: disk space when paired with a disk cache. Potentially Bazel 7.2 fixes much of the disk space concerns as it'll be able to prune it as needed.

2. Yes for the reasons in 1. Assuming the drawbacks aren't too bad and we can ensure analysis cache doesn't invalidate (I believe we do this correctly already)

3. It could print a diff of the flags so we can determine what is missing from which builds.

[brentleyjones]

Are you saying that devs regularly build on the command-line in addition to with Xcode?

[luispadron]

Yeah our developers use both as we have tooling that wraps Bazel CLI to perform tasks like updating snapshot images. For the most part though I believe our engineers use Xcode + Bazel

[BalestraPatrick]

From our experience at Spotify:

1. No, we don't see the current set up being a problem for us. We don't specifically profile memory and CPU during builds, so it's hard to know if this is affecting us in a meaningful way. We have received reports in the past about a Bazel instance taking an absurd amount of CPU and memory resources, but from what we've observed it wasn't caused by having multiple instances of Bazel running. It was more likely a one-off Bazel performance issue (especially during heavy indexing of the project).
2. I don't think we would adopt such a feature. Even with support from rules_xcodeproj, the overhead of trying to keep all builds in sync with the various flags to retain the analysis cache might make it not worth it for us. We have engineers that semi-regularly build on the command-line in addition to Xcode. This was prompted by the regressions in Xcode 15 we have been experiencing (slow builds to start, frequent crashes of the IDE, etc.), so running builds on the command-line allows at least builds to continue if Xcode crashes randomly.

[thiagohmcruz]

From our experience at Square:

1. Does this additional output base affect you/your team? If so, in which way, and to what extent?

Yes. We're in the process rolling out rules_xcodeproj to devs and I've received reports of high disk usage a few times and high memory/CPU usage a good amount of times so far. Our devs build from the command line often, some switch between Xcode and command line builds during the day and they expect both to be "equivalent" since "as a dev" they're just worried about building "target X" regardless if they're in Xcode or not.

To be clear, a lot of that is likely from the size/complexity of our graph which causes everything to be more expensive (memory, CPU, disk usage). The analysis phase is very expensive for us when it has to run and it's easy to see the impact of that in index build times, memory used by the Bazel servers for each output base (15GB+ if we don't cap it using JVM settings), many cores being used (e.g. java processes with 700%+ usage for a long time), etc.

Regardless, any Bazel server we can stop running or any output base we can stop creating would improve DX for our devs significantly and also reduce our burden to maintain the infra.

2. Would having an option to reuse the normal output base be beneficial to you?

Yes.

3. What, if anything, would rules_xcodeproj need to do to assist you in using the same output base (e.g. to help ensure the right flags are set to prevent analysis cache discarding, or ways to build targets with the same transitions that rules_xcodeproj applies)? And how important of a change would this be (e.g. would you/your organization be willing to sponsor the change if it’s non-trivial to implement and/or support?)

I believe detailed documentation and examples would help a lot. If there's anything that rules_xcodeproj can assert or "flag" to consumers would be awesome too (e.g. automatically detect (directly or indirectly?) that your tools/bazel is not following best practices).

Anything that is specifically built to address the "big/complex codebase" use case is welcomed too IMO. Last, I honestly have to idea if my org would be willing to sponsor this (but it's not gonna hurt to ask worst case).

[justinseanmartin]

Coming from Square/Block, so Thiago's responses are similar to what I've seen from reports.

Two questions for you in reply @brentleyjones:

1. Would it be a reasonable strategy for eliminating the second analysis cache by implicitly wrapping commands with the rules_xcodeproj CLI in our bazel wrapper? Would that even work? Seems like maybe an easier way to achieve less redundant output base, but I'm not sure if the CLI works for all cases.
2. I know you said to ignore the extra index output base, but it does seem like there are good reasons to strive to land on a unified output base here too. On top of reducing resource utilization, it seems like it should also reduce the time spent doing redundant work to populate/update them both. By unifying that with the build cache, assuming the indexing build commands match those of user initiated builds and will cache hit, it seems like it could also lead to faster user initiated incremental builds (populated by the indexer in the background). From a brainstorming perpesctive - how bad is the delay for halting the index builds? Is that something that Xcode does explicitly (e.g. by sending a SIGABORT) or would the user initiated build just wait for the index build invocation to complete? Would it help if user initiated builds sent an explicit abort to the running bazel process (if Xcode isn't already doing this) in a script phase before the build starts? I'm not certain how much of a delay that would lead to and if that could cause the analysis cache to get flushed.

Take these questions/suggestions with a grain of salt. I'm just trying to propose other angles that might address the pain without the pain of potential coordination to avoid analysis cache invalidation.

[brentleyjones]

1. The CLI depends on the fact that it uses a second output base. Otherwise the outer invocation would invalidate the inner one, mainly because things like --xcode_version wouldn't be set the same. I believe any solution that would allow reusing the same output base for project generation and command-line would require adjusting or modifying a bazel wrapper, maybe by running a script that rules_xcodeproj can generate. So similar to what you suggested, but not quite.
2. There are a couple of bugs that would be solved by reusing the Build and Index Build output bases (e.g Bug: Xcode beachballs when opening a generated file in certain situations #2480). Indexing would also complete a lot quicker if it could have a hot analysis cache after project generation. The delay is currently bad because it doesn't seem like Xcode sends cancellations, or it does it after a long delay. I haven't tested it recently though. A cancelled build should keep the analysis cache hot (SIGINT), but not if killed (SIGKILL). If we can make the coordination between these two sets of command work, then yes, this would be a great change.

[erikkerber]

I'll keep mine short since most of it is redundant:

Does this additional output base affect you/your team? If so, in which way, and to what extent?

Similar to Thiago, but mostly disk space. I feel we could optimize some things so our output bases aren't always filled with hundreds of 300MB test binaries, but I also would rather not have to think about disk space any more than we already do.

Would having an option to reuse the normal output base be beneficial to you?

Yes. While I do wish developers leveraged Bazel directly more than they do (the vast majority only do via Xcode and project gen), the reality is the benefit of separate output bases is rarely felt by our non-tooling-team developers.

What, if anything, would rules_xcodeproj need to do to assist you in using the same output base (e.g. to help ensure the right flags are set to prevent analysis cache discarding, or ways to build targets with the same transitions that rules_xcodeproj applies)? And how important of a change would this be (e.g. would you/your organization be willing to sponsor the change if it’s non-trivial to implement and/or support?)

It's hard for me to anticipate how much we'd be annoyed by the downsides of moving to a shared output base. Both what we'd want rules_xcodeproj to do for us, and our own investment, would probably be informed by just observing it for a while.

[karim-alweheshy]

Similar to Thiago, but mostly disk space. I feel we could optimize some things so our output bases aren't always filled with hundreds of 300MB test binaries, but I also would rather not have to think about disk space any more than we already do.

At Reddit, we've added a background daemon that runs from the bazel wrapper when running test/coverage. This deamon streams and parses the proto BEP and finishes with EOF.

From the parsed testResults, we look for these binaries and all intermediate folder/files and delete them all. This helped us maintain a smaller disk usage by bazel test/coverage runs locally and on CI.

That of course has the tradeoff of linking these binaries again in case of an identical command run with no changes

[andre-alves]

Does this additional output base affect you/your team? If so, in which way, and to what extent?

Yes.. memory is the top concern for us because we use 16GB macbooks, and Bazel takes 4GB~ of ram by itself (the 3 output bases combined).

Would having an option to reuse the normal output base be beneficial to you?

Yes, I would try it.

What, if anything, would rules_xcodeproj need to do to assist you in using the same output base (e.g. to help ensure the right flags are set to prevent analysis cache discarding, or ways to build targets with the same transitions that rules_xcodeproj applies)? And how important of a change would this be (e.g. would you/your organization be willing to sponsor the change if it’s non-trivial to implement and/or support?)

I think it would be a great optimization, but we are very happy with rules_xcodeproj as-is.

[yongjincho92]

Does this additional output base affect you/your team? If so, in which way, and to what extent?

Yes, it consumes CPU and RAM a lot for two output bases. For our main application project, it takes about 20GB of RAM for running one bazel server. And having another one is impactful to local environments. Same with disk space. We have a choppy disk clean up logic that we hope to remove after Bazel 7.2 release with in flight disk garbage collection feature.

Would having an option to reuse the normal output base be beneficial to you?

Yes.

What, if anything, would rules_xcodeproj need to do to assist you in using the same output base (e.g. to help ensure the right flags are set to prevent analysis cache discarding, or ways to build targets with the same transitions that rules_xcodeproj applies)?

Like others commented, having a clear way to view the flag difference between CLI bazel build vs XCode bazel build would be helpful.

And how important of a change would this be (e.g. would you/your organization be willing to sponsor the change if it’s non-trivial to implement and/or support?)

Yes, as this is direly exhausting our local developer machines.

[brentleyjones]

I've merged the normal Xcode build output base and the Index build output base with #3074. Longer term I think I'll be able to merge the command-line output base as well (by de-nesting the Xcode one), with an optional flag similar to the VS Code extension.