Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pulsar backpressure mechanism ignores client's memory limit setting #2573

Open
marekczajkowski opened this issue Apr 10, 2024 · 0 comments
Open
Labels

Comments

@marekczajkowski
Copy link

marekczajkowski commented Apr 10, 2024

In reference to the discussion from this issue #2548 (comment)

The backpressure mechanism in the SmallRye Pulsar connector does not consider memory limit settings. While Pulsar clients can be configured with memoryLimitBytes, which is shared among all consumers and producers used by the client, the backpressure mechanism solely relies on the maxPendingMessages set for the producer. Consequently, it will refrain from sending more messages to the producer queue. However, disregarding the memory limit may result in a MemoryBufferIsFullError if a certain number of messages (lower than maxPendingMessages) with large payload sizes exceed memoryLimitBytes.

In order to address this, backpressure should also verify whether the client can reserve memory for the message to be sent via the producer. This could be achieved by introducing a new processor, such as the MemoryLimitAwareSendProcessor, which would check memory availability before sending the message to the producer.

@marekczajkowski marekczajkowski changed the title Pulsar backpressure mechanism ignores clientmemory limit Pulsar backpressure mechanism ignores client's memory limit setting Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants