Skip compaction for datasources with partial-eternity segments #15542
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cryptoe
approved these changes
Dec 12, 2023
.../src/test/java/org/apache/druid/server/coordinator/compact/NewestSegmentFirstPolicyTest.java
Show resolved
Hide resolved
| // This is to prevent the coordinator from crashing as raised in https://github.com/apache/druid/issues/13208 | ||
| log.warn("Cannot compact datasource[%s] with ALL granularity", dataSource); | ||
| log.warn("Cannot compact datasource[%s] containing segments with partial-ETERNITY intervals", dataSource); |
There was a problem hiding this comment.
While at it, could you add the segment id as well? It'd make it easy to take any action in the metadata store or via the console, if needed.
There was a problem hiding this comment.
There could be several segments and it could be noisy to log all of them
| log.warn("Cannot compact datasource[%s] since interval is ETERNITY.", dataSourceName); | ||
| if (Intervals.ETERNITY.getStart().equals(lookupInterval.getStart()) | ||
| || Intervals.ETERNITY.getEnd().equals(lookupInterval.getEnd())) { | ||
| log.warn("Cannot compact datasource[%s] since interval start or end coincides with ETERNITY.", dataSourceName); |
There was a problem hiding this comment.
Suggested change
| log.warn("Cannot compact datasource[%s] since interval start or end coincides with ETERNITY.", dataSourceName); | |
| log.warn("Cannot compact datasource[%s] since lookupInterval[%s] coincides with ETERNITY.", dataSourceName, lookupInterval); |
abhishekrb19
approved these changes
Dec 12, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR builds on #13304 to skip compaction for datasources with segments that have their interval start or end coinciding with Eternity interval end-points.
This is needed in order to prevent an issue similar to #13208 as the Coordinator tries to iterate over a large number of intervals when trying to compact an interval with infinite start or end.
This PR has: