You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using aws-sdk-cpp to subscribe to a Kinesis stream with enhanced fanout via the SubscribeToShard() method, the SetEventStreamHandler() callback is invoked with noticeable delays, and data is received in batches instead of real-time. This causes inefficient processing of the stream data, leading to potential performance issues, especially in low-latency or real-time systems.stream data processing
Expected Behavior
The SetEventStreamHandler() should be invoked in near real-time as soon as new records are available in the Kinesis stream.
Records should be delivered individually or as soon as they are available without being unnecessarily batched or delayed.
Current Behavior
The SetEventStreamHandler() callback is delayed and not invoked immediately when new records arrive in the Kinesis shard.
Data is accumulated in batches and delivered with noticeable delays rather than being processed as soon as records are produced in the stream.
Reproduction Steps
Use aws-sdk-cpp to create a Kinesis enhanced fanout consumer.
Set up the SubscribeToShard() method with enhanced fanout and set the event handler using SetEventStreamHandler() to process incoming records.
Write a simple record producer that sends data to the Kinesis stream.
Observe that the SetEventStreamHandler() callback is invoked with delays and receives records in batches rather than as they become available
Possible Solution
No response
Additional Information/Context
The problem persists regardless of the producer's rate of sending records to the stream.
Slower the producer, higher the delays are (155 seconds delay if producer sends a single record)
Can you provide a minimal reproduction sample that follows the steps you listed above?
jmklix
added
response-requested
Waiting on additional info and feedback. Will move to "closing-soon" in 10 days.
p3
This is a minor priority issue
and removed
needs-triage
This issue or PR still needs to be triaged.
labels
Sep 24, 2024
I've attached the consumer code. This code does not throw any exception or set any error in subscribeOutcome. Its just that the handler is invoked with significant delay which is explained above.
Producer is pretty straightforward which works fine with other consumers.
Describe the bug
When using aws-sdk-cpp to subscribe to a Kinesis stream with enhanced fanout via the SubscribeToShard() method, the SetEventStreamHandler() callback is invoked with noticeable delays, and data is received in batches instead of real-time. This causes inefficient processing of the stream data, leading to potential performance issues, especially in low-latency or real-time systems.stream data processing
Expected Behavior
Current Behavior
Reproduction Steps
Possible Solution
No response
Additional Information/Context
AWS CPP SDK version used
1.11.389
Compiler and Version used
gcc 11.4.0
Operating System and version
Ubuntu 22.04
The text was updated successfully, but these errors were encountered: