You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The chainHead_v1_follow is using unbounded channels to send out messages on the JSON-RPC connection which may use lots of memory if the client is slow and can't really keep up with server i.e, substrate may keep lots of message in memory
Close the subscription/connections if that client can keep up with the server
Replace older items in queue.
For chainHead_follow replacing older items is not a good idea but one may loose important state and only 1) is option.
Thus, I propose that we create a dedicated fixed size buffer, poll items for the combined stream, push them to the buffer and if the buffer limit is exceeded then we throw a stop event and close down the subscription.
/cc @paritytech/subxt-team
The text was updated successfully, but these errors were encountered:
Thus, I propose that we create a dedicated buffer and if it's exceeded then we throw a stop event and close down the subscription.
I thought we were basically doing this already with the pipe_from_stream thing; is that not correct?
I'm basically in favour of having some buffer that drains as fast as the user is accepting messages, and we send a stop notification if the buffer fills up with events (ie the user isn't consuming them quickly enough)
I thought we were basically doing this already with the pipe_from_stream thing; is that not correct?
I think we were but it has been changed to this, where it takes one item at the time and it could block on send.await and buffer up with items in memory.
I think we could just use pipe_from_stream but we need to tweak it to able to re-use the sink to send the out the stop event.
I'm basically in favour of having some buffer that drains as fast as the user is accepting messages, and we send a stop notification if the buffer fills up with events (ie the user isn't consuming them quickly enough)
The
chainHead_v1_follow
is using unbounded channels to send out messages on the JSON-RPC connection which may use lots of memory if the client is slow and can't really keep up with server i.e, substrate may keep lots of message in memoryCurrently three parts are unbounded:
internal chainHead_v1 follow channelAs I see it we have to options either:
For
chainHead_follow
replacing older items is not a good idea but one may loose important state and only 1) is option.Thus, I propose that we create a dedicated fixed size buffer, poll items for the combined stream, push them to the buffer and if the buffer limit is exceeded then we throw a stop event and close down the subscription.
/cc @paritytech/subxt-team
The text was updated successfully, but these errors were encountered: