materialized: strip debuginfo #31207
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The materialized binary generates massive debuginfo: 8GB with full debuginfo and 3.5GB with limited debuginfo. (This is not the compressed size of the debuginfo on disk, but the amount of memory required to load the debuginfo into memory so that backtraces can be symbolized.) We've historically shipped the materialized binary with full debuginfo, so that we get rich backtraces for any crashes users run into when running the emulator. Unfortunately, the full debuginfo is so large that it is itself the *cause* of OOMs [0], which is unacceptable. (When a console query encounters certain routine errors, like a connection failing to validate, the adapter attempts to log a backtrace, which requires loading the debuginfo.) Even the limited debuginfo size (3.5GB) is unacceptable for a Docker image that's meant to be run on developer laptops. So, this commit adjusts the materialized image to strip all debuginfo from the binary. If a user reports a crash with an unsymbolized backtrace, it's still *possible* (just painful) to manually symbolize the backtrace as long as they give us the exact version of Materialize they were running. We'll just need to manually run `addr2line` on each address reported in the backtrace. (We do irrevocably lose access to frames for inlined functions, but that seems tolerable.) One silver lining here is that the Docker image will get much smaller. It's currently about 1GB. I expect this change to shave off a huge chunk of that. [0]: https://materializeinc.slack.com/archives/C07FX1W1Y03/p1737414021061139
ParkMyCar
approved these changes
Jan 27, 2025
benesch
added a commit
to benesch/materialize
that referenced
this pull request
Jan 27, 2025
These fix an issue with memory usage: see MaterializeInc#31207.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The materialized binary generates massive debuginfo: 8GB with full debuginfo and 3.5GB with limited debuginfo. (This is not the compressed size of the debuginfo on disk, but the amount of memory required to load the debuginfo into memory so that backtraces can be symbolized.)
We've historically shipped the materialized binary with full debuginfo, so that we get rich backtraces for any crashes users run into when running the emulator. Unfortunately, the full debuginfo is so large that it is itself the cause of OOMs 0, which is unacceptable. (When a console query encounters certain routine errors, like a connection failing to validate, the adapter attempts to log a backtrace, which requires loading the debuginfo.) Even the limited debuginfo size (3.5GB) is unacceptable for a Docker image that's meant to be run on developer laptops.
So, this commit adjusts the materialized image to strip all debuginfo from the binary.
If a user reports a crash with an unsymbolized backtrace, it's still possible (just painful) to manually symbolize the backtrace as long as they give us the exact version of Materialize they were running. We'll just need to manually run
addr2lineon each address reported in the backtrace. (We do irrevocably lose access to frames for inlined functions, but that seems tolerable.)One silver lining here is that the Docker image will get much smaller. It's currently about 1GB. I expect this change to shave off a huge chunk of that.
Motivation
Tips for reviewer
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.