Supports Force Committing Segments in Batches #14811

noob-se7en · 2025-01-14T15:37:39Z

Problem Statement
The Force Commit API can cause high ingestion lag and slower queries because it triggers the simultaneous segment commit for all consuming segments. This happens because:

If N is the number of partition groups a server is consuming from, The API will cause all of the N consuming segments to commit. Hence N consumer threads will rush to acquire segment build semaphore. If the Semaphore allows only M permits, Only M consuming segments are in the segment build stage and the remaining (N - M) consumer threads are waiting on the Semaphore. Since (N - M) consumer threads are waiting, the consumption lag can become substantial.
Since M consuming segments are built in parallel, queries can become slower on poorly sized servers due to high memory consumption.

Solution
Adds an additional optional Integer parameter: batchSize to the forceCommit API (Default Value = Integer.MAX_VALUE i.e. commit as many segments as possible at once if no batchSize is provided) .

codecov-commenter · 2025-01-14T16:51:46Z

Codecov Report

Attention: Patch coverage is 52.57732% with 46 lines in your changes missing coverage. Please review.

Project coverage is 63.75%. Comparing base (59551e4) to head (b95a2f6).
Report is 1593 commits behind head on master.

Files with missing lines	Patch %	Lines
.../core/realtime/PinotLLCRealtimeSegmentManager.java	64.55%	26 Missing and 2 partials ⚠️
...ller/api/resources/PinotRealtimeTableResource.java	0.00%	18 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14811      +/-   ##
============================================
+ Coverage     61.75%   63.75%   +2.00%     
- Complexity      207     1612    +1405     
============================================
  Files          2436     2708     +272     
  Lines        133233   151313   +18080     
  Branches      20636    23364    +2728     
============================================
+ Hits          82274    96475   +14201     
- Misses        44911    47596    +2685     
- Partials       6048     7242    +1194

Flag	Coverage Δ
custom-integration1	`100.00% <ø> (+99.99%)`	⬆️
integration	`100.00% <ø> (+99.99%)`	⬆️
integration1	`100.00% <ø> (+99.99%)`	⬆️
integration2	`0.00% <ø> (ø)`
java-11	`63.71% <52.57%> (+2.00%)`	⬆️
java-21	`63.65% <52.57%> (+2.02%)`	⬆️
skip-bytebuffers-false	`63.73% <52.57%> (+1.98%)`	⬆️
skip-bytebuffers-true	`63.63% <52.57%> (+35.90%)`	⬆️
temurin	`63.75% <52.57%> (+2.00%)`	⬆️
unittests	`63.75% <52.57%> (+2.00%)`	⬆️
unittests1	`56.32% <ø> (+9.43%)`	⬆️
unittests2	`34.07% <52.57%> (+6.34%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Jackie-Jiang · 2025-01-15T03:45:01Z

This solves #11950

Jackie-Jiang

Seems you are pushing the batch throttling to the server side. What will happen if some replicas decide to commit, and others get throttled? Even worse, could this cause deadlock?

siddharthteotia · 2025-01-15T07:25:35Z

Is it not possible to solve the problem on controller / coordinate from the controller ? Pushing this down to the individual server will likely lead to error-prone situations

…ce_commit

noob-se7en · 2025-01-15T18:48:26Z

@Jackie-Jiang I don't quite get what is meant by

and others get throttled?

Regarding Deadlock or any edge case - Server will use the same logic which is used /tables/forceCommitStatus/{jobId} to check the status of the batch, so there should be no deadlock.

… into add_batch_force_commit

…_commit

…mit_status

… into add_batch_force_commit

KKcorps · 2025-01-17T11:25:08Z

...ain/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java

-    sendForceCommitMessageToServers(tableNameWithType, targetConsumingSegments);
+
+    List<Set<String>> segmentBatchList = getSegmentBatchList(idealState, targetConsumingSegments, batchSize);
+    ExecutorService executorService = Executors.newFixedThreadPool(1);


Do not create new thread pool on every request

ohh right!
Fixed (created a separate bounded ExecutorService for forceCommitAPI)

KKcorps · 2025-01-17T11:25:36Z

...ain/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java

+    sendForceCommitMessageToServers(tableNameWithType, segmentBatchToCommit);
+
+    try {
+      Thread.sleep(FORCE_COMMIT_STATUS_CHECK_INTERVAL_MS);


Why sleep at start?

This sleep is to save zookeeper calls, as at this point of code it's almost certain that all segments are yet to be committed.
(we need a new retryPolicy with initial start delay)

KKcorps · 2025-01-17T11:43:04Z

...ain/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java

+        String state = instanceToStateMap.get(instance);
+        if (state.equals(SegmentStateModel.CONSUMING)) {
+          instanceToConsumingSegments.putIfAbsent(instance, new LinkedList<>());
+          instanceToConsumingSegments.get(instance).add(segmentName);


You can simply this by using compute instead of putIfAbsent and get

KKcorps · 2025-01-17T11:45:42Z

...java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManagerTest.java

+    List<Set<String>> segmentBatchList =
+        realtimeSegmentManager.getSegmentBatchList(idealState, targetConsumingSegment, 2);
+
+    // i1 = [seg0, seg4, seg5, seg6]


remove these comments

KKcorps · 2025-01-17T11:47:25Z

...ller/src/main/java/org/apache/pinot/controller/api/resources/PinotRealtimeTableResource.java

@@ -170,19 +169,28 @@ public Map<String, String> forceCommit(
      @ApiParam(value = "Comma separated list of partition group IDs to be committed") @QueryParam("partitions")
      String partitionGroupIds,
      @ApiParam(value = "Comma separated list of consuming segments to be committed") @QueryParam("segments")
-      String consumingSegments, @Context HttpHeaders headers) {
+      String consumingSegments,
+      @ApiParam(value = "Max number of consuming segments to commit at once") @QueryParam("batchSize")


mention that default is all consuming segments at once in description

KKcorps · 2025-01-17T11:49:17Z

Please update PR description to reflect the change in API signature

noob-se7en added 13 commits January 13, 2025 01:07

Supports batching in ForceCommit API

eeb5be1

nit

5f5a554

Refactoring

ca5104a

nit

434e8a3

nit

504f3c9

nit

987bb00

nit

ff25c5f

nit

e28ff47

nit

99a7cee

lint

255bc34

nit

3a9e41a

fixes lint

b2eeb85

nit

1782207

Jackie-Jiang added feature ingestion rest-api labels Jan 15, 2025

Jackie-Jiang reviewed Jan 15, 2025

View reviewed changes

noob-se7en added 8 commits January 15, 2025 21:43

Merge branch 'master' of github.com:Harnoor7/pinot into add_batch_for…

90db3b8

…ce_commit

refactoring

fa418b9

refactoring

470c6eb

fixes bug

8de7bfc

nit

4f2d4fc

nit

50af02e

nit

09d557e

nit

32b7fd5

noob-se7en requested a review from Jackie-Jiang January 15, 2025 18:49

noob-se7en added 5 commits January 16, 2025 18:03

adds tests

8c8d8d3

saves 1 zk call

1be0316

Merge branch 'update_forceCommit_status' of github.com:Harnoor7/pinot…

55fa6e2

… into add_batch_force_commit

updates log

3297ddd

Adds tests

748d0d3

noob-se7en marked this pull request as ready for review January 16, 2025 21:24

noob-se7en requested a review from Jackie-Jiang January 16, 2025 21:28

noob-se7en added 13 commits January 17, 2025 15:33

Addresses PR comments

095acc0

nit

36360b8

Merge branch 'update_forceCommit_status' of github.com:Harnoor7/pinot…

d3d42ca

… into add_batch_force_commit

pulls latest changes for pauseless

262bee0

adds unit test

68cdc26

Merge branch 'update_forceCommit_status' of github.com:Harnoor7/pinot…

9f833c6

… into add_batch_force_commit

addresses comment

f5d68ae

Merge branch 'master' of github.com:apache/pinot into add_batch_force…

bffab6d

…_commit

Addresses Pr comment

a1079c2

Merge branch 'master' of github.com:apache/pinot into update_forceCom…

b8a2e7f

…mit_status

addresses PR comments

5730a06

nit

c8565d6

Merge branch 'update_forceCommit_status' of github.com:Harnoor7/pinot…

165e7ab

… into add_batch_force_commit

KKcorps reviewed Jan 17, 2025

View reviewed changes

noob-se7en added 3 commits January 18, 2025 00:51

refactoring

0cab772

Addresses PR comments

de04824

nit

b95a2f6

noob-se7en requested a review from KKcorps January 17, 2025 20:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Supports Force Committing Segments in Batches #14811

Supports Force Committing Segments in Batches #14811

noob-se7en commented Jan 14, 2025 •

edited

Loading

codecov-commenter commented Jan 14, 2025 •

edited

Loading

Jackie-Jiang commented Jan 15, 2025

Jackie-Jiang left a comment

siddharthteotia commented Jan 15, 2025

noob-se7en commented Jan 15, 2025

KKcorps Jan 17, 2025

noob-se7en Jan 17, 2025

KKcorps Jan 17, 2025

noob-se7en Jan 17, 2025

KKcorps Jan 17, 2025 •

edited

Loading

KKcorps Jan 17, 2025

KKcorps Jan 17, 2025

KKcorps commented Jan 17, 2025 •

edited

Loading

Supports Force Committing Segments in Batches #14811

Are you sure you want to change the base?

Supports Force Committing Segments in Batches #14811

Conversation

noob-se7en commented Jan 14, 2025 • edited Loading

codecov-commenter commented Jan 14, 2025 • edited Loading

Codecov Report

Jackie-Jiang commented Jan 15, 2025

Jackie-Jiang left a comment

Choose a reason for hiding this comment

siddharthteotia commented Jan 15, 2025

noob-se7en commented Jan 15, 2025

KKcorps Jan 17, 2025

Choose a reason for hiding this comment

noob-se7en Jan 17, 2025

Choose a reason for hiding this comment

KKcorps Jan 17, 2025

Choose a reason for hiding this comment

noob-se7en Jan 17, 2025

Choose a reason for hiding this comment

KKcorps Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

KKcorps Jan 17, 2025

Choose a reason for hiding this comment

KKcorps Jan 17, 2025

Choose a reason for hiding this comment

KKcorps commented Jan 17, 2025 • edited Loading

noob-se7en commented Jan 14, 2025 •

edited

Loading

codecov-commenter commented Jan 14, 2025 •

edited

Loading

KKcorps Jan 17, 2025 •

edited

Loading

KKcorps commented Jan 17, 2025 •

edited

Loading