Horizontal scaling of the nodes

[dsmello]

Hello everyone,

That discussion is more to try to aggregate all information we had about: How to horizontal scaling the chain nodes?, and if that wasn't possible yet, what we need to do to accomplish that.

I search the web for more information about how to scale cosmos nodes, how to run at Kubernetes, etc. And I don't have much success.

Did you know something?

I've one idea about how we can address that problem. If you find a problem or don't agree with leave a comment and explain why.

1. create a "load balancer" for cosmos nodes that will be a proxy for all connections. With that, we'll have the capability to make routing rules or strategies, multiple aggregate nodes under a unique address, and scale nodes as the usage or throughput.

Example:

The load balancer will receive all transactions. So if it was a new block or something will pass it to nodes as usual. But if it was a new transaction, the load balancer will verify the sender address and check if that address was already attributed to a node and used in the last three times of the block time (that will help to avoid double sign). Then, the new block will usually be routed to the other nodes—just that.

The problem I saw was that I couldn't decouple the data from the node itself, forcing me to have the disk previously prepared for usage. But I'll spend more time reading the existing database the cosmos are using and see if they have some way to allow multi-instance usage of the same files or a server.

[alexanderbez]

Hey @dsmello I'm not really following for what you're asking for here? Traditional "horizontal scaling" doesn't really apply here in the same sense. Load balancers aren't needed and in fact, can be harmful because it requires all nodes behind the LB to be perfectly in sync.

Nodes in cosmos really operate under two contexts: (1) full nodes and (2) validator nodes, both of which run the exact same software but where one is signing blocks whereas the other isn't. So there really isn't anything to "scale" here. If you want to run more full nodes or IBC relayers, you can do this with k8s, but there isn't anything particular or special about scaling cosmos nodes.

[dsmello]

Hi @alexanderbez, how're you?

So my goal is to find a way to have high availability at nodes, to run using spot instances and Kubernetes, which will allow me to save 60%~80% on the costs. But as you said, "load balancer" wasn't a good word for that; sorry for the missing lead.

And see if we could run the nodes in a distributed way instead of a giant machine. If it isn't possible will be fine, I opened this discussion more to see more information about that.

I'll start a relayer this week. Did you know if possible to have HA at relayers? Or any documentation to refer to?
I'm reading about the Cosmos and tendermint database to understand better. And did you know anyone who's using remote DB in their nodes?

[alexanderbez]

What do you mean by HA of nodes? If you want redundancy and fault tolerance in terms of always having node uptime, just launch more full-nodes in k8s or something.

And see if we could run the nodes in a distributed way instead of a giant machine. If it isn't possible will be fine, I opened this discussion more to see more information about that.

You can run full-nodes in a distributed fashion, yes. Many validators and operators run nodes within a region and across multiple regions for redundancy. There isn't a need for HA in this case though.

I'll start a relayer this week. Did you know if possible to have HA at relayers? Or any documentation to refer to?

What does HA for relayers mean? Relayers are stateless processes that only require access to a node to query against and a hot wallet to pay for funds -- you can run as many as you want.