Enabling/disabling NATS clustering without losing Jetstream data

[SimonSchmid]

Hey all,

We have NATS running in non-clustering mode in our Kubernetes cluster and use its Jetstream feature to persist events. I am currently investigating the clustering mode NATS offers and have a few questions:

1. It seems there is no easy migration path from non-clustering to clustering mode (and back) as the stream data seems to get lost/unrecoverable. I found this issue which proposes to do a manual backup and restore to avoid data loss. Is this really the way to go? It would be great to be able to just turn clustering on and off without having to care about this.
2. Turning clustering on does not seem be enough to make Jetstream "highly available" as streams are not replicated across the different pods/pvcs automatically. One needs to edit the config of a stream and increase the number of its replicas to achieve this. Is there an option to make the number of stream replicas automatically adapt to the number of nats replicas configured? E.g. if I run nats clustering with 3 replicas, having also 3 replicas of my streams?

In general, it would be great to be flexible with how many nats replicas one runs and have Jetstream being able to automatically adjust to it.

Best,
Simon

[caleblloyd]

1. It seems there is no easy migration path from non-clustering to clustering mode (and back) as the stream data seems to get lost/unrecoverable. I found this Migration of a standalone instance to a cluster nats-server#3267 which proposes to do a manual backup and restore to avoid data loss. Is this really the way to go? It would be great to be able to just turn clustering on and off without having to care about this.

This is fundamentally how nats-server works right now. When JetStream is in Cluster mode, it requires more than 1 server to form a cluster, and each server stores information about its own cluster/server name. In standalone mode, there is no such consideration. I would recommend running a Cluster of 3 for JetStream if you think you may ever need clustering.

2. Turning clustering on does not seem be enough to make Jetstream "highly available" as streams are not replicated across the different pods/pvcs automatically. One needs to edit the config of a stream and increase the number of its replicas to achieve this. Is there an option to make the number of stream replicas automatically adapt to the number of nats replicas configured? E.g. if I run nats clustering with 3 replicas, having also 3 replicas of my streams?

True, the level of replication must be chosen on a per-stream (and even per-consumer) basis. Consumers will default to the same number of replicas as their Stream if no Consumer replicas are specified. But there may be cases where you want to trade of availability for speed. Any multi-replica stream forms a RAFT group and will perform slower than a single-replica stream. So Replicas are editable on a per-stream (and per-consumer) basis for this reason.

It is common for teams to centralize the configuration of Streams with tools like NACK or terraform-provider-jetstream to ensure that settings such as replication can be configured in a repeatable manner.

[SimonSchmid]

Thank you for your quick reply!

When JetStream is in Cluster mode, it requires more than 1 server to form a cluster, and each server stores information about its own cluster/server name. In standalone mode, there is no such consideration.

So the data loss when changing standalone to cluster mode is only related to the server information that is stored? Couldn't the stream data itself stay persisted (perhaps as a future feature)? Right now, it seems all the data is just wiped.
We are running it in standalone mode right now to save resources. If we want to change to cluster mode, it sounds like there is no other way than doing a manual backup, changing then to cluster mode, and restoring the backup. Are there any issues to expect with that?

I would recommend running a Cluster of 3 for JetStream if you think you may ever need clustering.

It is sometimes hard to foresee whether clustering mode will be required at some point. It seems a waste of resources if one runs a cluster of 3 just to be able to potentially use clustering without some manual data backup.

True, the level of replication must be chosen on a per-stream (and even per-consumer) basis. Consumers will default to the same number of replicas as their Stream if no Consumer replicas are specified. But there may be cases where you want to trade of availability for speed. Any multi-replica stream forms a RAFT group and will perform slower than a single-replica stream. So Replicas are editable on a per-stream (and per-consumer) basis for this reason.

Not sure if I got this right. Why would a consumer care about the number of replicas? I would expect that the availability increases if there are more stream replicas but the write speed, i.e. producer speed, decreases. Why is the "read speed" by consumers affected?

It is common for teams to centralize the configuration of Streams with tools like NACK or terraform-provider-jetstream to ensure that settings such as replication can be configured in a repeatable manner.

Thank you for this pointer!

[pbennett]

Not sure if I got this right. Why would a consumer care about the number of replicas? I would expect that the availability increases if there are more stream replicas but the write speed, i.e. producer speed, decreases. Why is the "read speed" by consumers affected?

Because the 'state' of the consumers is persisted on the nats server itself - so the # of replicas for a consumer is the # of replicas the read state is replicated to. Kafka for eg makes the consumers read state the consumers problem to deal with.