Is there a way to avoid interruptions during redeployment when using the ruler in remote mode?

[changliu-wk]

Recently we want to switch ruler from internal mode to remote mode, plus using memcached for data caching, it will reduce the query duration, but I found when I redeploy the service, the latency will increase a lot, after 10 mins it will back to normal. I think that is because the caching pods have been new created, we lost all caching data. I want to come up a strategy:

1. Just create a new pod with the caches, keep one old replica working,
2. After the new pod created 5 mins after(regenerating the new caching), the old one can be killed. But I had been thinking it for a whole day, and didn't find a way to implement it.

I thought Redis would be the right way for this problem, but it was deprecated lately, do you have a better solution for this problem, thanks!