Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect handling of OperationCancelledException can causes hanging ingestion, unnoticed failing tasks, and no logging of timeouts #4780

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

ramonsmits
Copy link
Member

@ramonsmits ramonsmits commented Feb 6, 2025

The following line in the logs can indicate that the instance eventually hangs:

2025-02-07 16:25:29.6882|17|Warn|ServiceControl.Audit.Auditing.AuditPersister|Bulk insertion dispose failed|Raven.Client.Exceptions.Documents.BulkInsert.BulkInsertAbortedException: Failed to execute bulk insert

@ramonsmits ramonsmits self-assigned this Feb 6, 2025
@ramonsmits ramonsmits added the Bug label Feb 6, 2025
…ich can cause the ingestion to never used incoming context tasks and hang.
… - only ignore cancellations set by caller
@ramonsmits ramonsmits marked this pull request as ready for review February 7, 2025 14:48
@ramonsmits
Copy link
Member Author

@andreasohlund Load tested this build for 12+ hours with multiple instances on this version and 6.3.0. All 6.3.0 instances eventually halted processing.

@ramonsmits ramonsmits marked this pull request as draft February 8, 2025 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants