Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a mempool #180

Open
sanjayprabhu opened this issue Dec 21, 2024 · 3 comments
Open

Implement a mempool #180

sanjayprabhu opened this issue Dec 21, 2024 · 3 comments
Labels
help wanted Extra attention is needed

Comments

@sanjayprabhu
Copy link
Contributor

sanjayprabhu commented Dec 21, 2024

Right now messages are only accepted by a single node andw e have to wait for that node's turn to be a proposer to have them be accepted. So, fix this and to allow eventual read nodes to submit messages, we should implement a gossip based mempool. It should:

  • On receiving a message by submitMessage, the node should add it to a local hashmap and then gossip it out in a separate topic
  • Other nodes should listen to the topic and add it to their local hashmap
  • When messages are successfully included in the block, they should be evicted from the local hashmaps on all nodes
  • The mempool should also enforce rate limits by fid based on the last n blocks (say, 5-10% of fid storage limit per 1000 blocks)
  • The mempool should be persisted to disk to messages not lost on restart or crash
@sanjayprabhu sanjayprabhu added the help wanted Extra attention is needed label Dec 21, 2024
@FrederikBolding
Copy link
Contributor

FrederikBolding commented Jan 2, 2025

@sanjayprabhu I have been looking at this a bit. How should the proposed hashmap mempool work with the existing "mempool" (meaning the message queue)? Should this be effectively a replacement or something else?

Additionally, do you prefer using an entirely separate topic instead of just a new message for the mempool gossip? If so, why?

@sanjayprabhu
Copy link
Contributor Author

Should effectively be a replacement.

I think we should create a separate topic for separation of concerns. Consensus messages are critical and we shouldn't have any other messages sharing that topic. Eventually I think we'd also want to enforce nodes that aren't active validators cannot publish to the consensus topic (if they do, it's treated as an invalid message and is not forwarded to the mesh).

Read nodes, when they are implemented, will want to listen and publish to the mempool topic but don't need to know about the consensus topic.

@FrederikBolding
Copy link
Contributor

Sounds good! I have started working on this, will let you know once I have something to share for feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants