SwitchWrites: improve performance of sequence initialisation steps

[rohit-nayak-ps]

Feature Description

While switching primary write traffic, for sharded keyspaces, if require, we update the sequence backing table with the largest value found for existing data, after stopping writes.

Currently we do multiple tabletmanager calls, one set each per table to

• get the current max auto-increment value from the data tables.
• create sequence tables, if required.
• update the sequence in the backing tables. We always call the update regardless of whether the sequence needs to be updated or not.

Depending on the number of tables and shards this can take very long since there can be several thousands of queries / rpc calls. This can cause timeouts preventing the workflow from switching traffic.

Even if the process is successful, since these are happening after we stop writes, it increases the downtime while switching traffic.

We should try and minimise this by

• using the intermediate step introduced in VReplication SwitchTraffic: for a dry run check if sequences need to be updated  #17842 only update required sequences
• potentially creating RPCs on the tablets which process all tables at once so that we have one RPC per