Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

supervisor: make rejection periods work with stopTasksCount #17442

Merged
merged 13 commits into from
Nov 18, 2024
Merged

supervisor: make rejection periods work with stopTasksCount #17442

merged 13 commits into from
Nov 18, 2024

Conversation

adithyachakilam
Copy link
Contributor

@adithyachakilam adithyachakilam commented Nov 3, 2024 •
edited
Loading

Description

In cases where stopTasksCount is configured which makes streaming tasks run much longer (in some cases) than the configured task duration, early/late rejection periods are not correctly implemented since they are configured during the task startup time in IOConfig. Instead, this PR propagates updates the min/max times once every task duration.

Key changed/added classes in this PR
  • SeekableStreamIndexTaskRunner
  • SeekableStreamIndexTaskIOConfig

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

suneet-s
suneet-s reviewed Nov 3, 2024
View reviewed changes
Copy link
Contributor

@suneet-s suneet-s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implementation of refreshing the min / max times every 10 min is a change in behavior, so I think the person submitting the supervisor spec needs to opt in to this new behavior which doesn't appear possible in the current approach.

An alternative approach to consider is to update the IOConfig to include a "refresh interval" field that defaults to infinity that would refresh the min / max times in the IOConfig.

The tasks could then update their min / max times every duration just by adding the refresh interval so we don't need to worry about clock skew across the peons.

Since this patch is meant to just deal with stopTaskCount - have you considered refreshing the min / max times every task duration to mimic the existing behavior if stopTaskCount was not set (aka every task duration the min/max times across all peons are updated)

...vice/src/main/java/org/apache/druid/indexing/rabbitstream/RabbitStreamIndexTaskIOConfig.java Outdated
Comment on lines 53 to 54
@JsonProperty("lateMessageRejectionPeriod") Duration lateMessageRejectionPeriod,
@JsonProperty("earlyMessageRejectionPeriod") Duration earlyMessageRejectionPeriod,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These new fields need to be in addition to the old fields to ensure a newer version of the overlord works with older versions of the indexing tasks. But more importantly, in the case of a rollback - an older overlord needs to work with a newer version of indexing services

@adithyachakilam
Copy link
Contributor Author

@suneet-s Thanks for the review, I've changed the patch to just update the min/max times once every task duration. So this also avoids all those version problems.

@adithyachakilam adithyachakilam changed the title supervisor: propagate rejection periods to tasks supervisor: make rejection periods work with stopTasksCount Nov 5, 2024
suneet-s
suneet-s reviewed Nov 5, 2024
View reviewed changes
Copy link
Contributor

@suneet-s suneet-s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not reviewed the tests yet.

For this change, we should also include docs for what this new parameter is in the IOConfig

...fka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaConsumerMonitor.java Outdated Show resolved Hide resolved
.../test/java/org/apache/druid/indexing/rabbitstream/supervisor/RabbitStreamSupervisorTest.java Outdated Show resolved Hide resolved
.../src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskIOConfig.java Outdated Show resolved Hide resolved
.../src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskIOConfig.java Outdated Show resolved Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java Outdated Show resolved Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java Outdated Show resolved Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java Outdated Show resolved Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java Outdated Show resolved Hide resolved
suneet-s
suneet-s reviewed Nov 14, 2024
View reviewed changes
Copy link
Contributor

@suneet-s suneet-s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this change. I have one last small ask to add some logging. Also, the docs in https://github.com/apache/druid/blob/master/docs/ingestion/supervisor.md#io-configuration mention that there are problems if the message rejection period is set too low - can that caveat be removed now?

The CI failure seems like it's ok to overrule since it is complaining about branch coverage about moved code and the code that is missing tests is debug logging.

indexing-service/src/test/resources/log4j2.xml Outdated Show resolved Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java Show resolved Hide resolved
@suneet-s
Copy link
Contributor

Overruling CI. I had to re-trigger a bunch of failed ITs, and now they have all passed. The only remaining failures are code coverage.

@suneet-s suneet-s merged commit 6f43630 into apache:master Nov 18, 2024
84 of 89 checks passed
cryptoe
cryptoe reviewed Nov 19, 2024
View reviewed changes
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
maxMessageTime = ioConfig.getMaximumMessageTime().or(DateTimes.MAX);

if (ioConfig.getRefreshRejectionPeriodsInMinutes() != null) {
Execs.scheduledSingleThreaded("RejectionPeriodUpdater-Exec--%d")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who closes this exec service ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be creating on thread for each supervisor. Seems a bit much no ? Can we use a common exec service for this ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @adithyachakilam

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#17490

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be creating on thread for each supervisor. Seems a bit much no ? Can we use a common exec service for this ?

+1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is creating thread on task not on supervisor ?

@adithyachakilam adithyachakilam mentioned this pull request Nov 19, 2024
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@suneet-s suneet-s suneet-s left review comments

@kfaraz kfaraz kfaraz left review comments

@cryptoe cryptoe cryptoe left review comments

Assignees
No one assigned
Labels
Area - Documentation Area - Ingestion Area - Streaming Ingestion
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@adithyachakilam @suneet-s @cryptoe @kfaraz