Fallback

Connection pool partitioning

Connection pool partitioning is a broad topic covering consistent hashing, sharding and fallback policies of connection pools with support for shared hosts. Each of these scenarios can easily be implemented using several connection pools that are linked together for an operation using standard RxJava functions. Note that the connection pools will likely share a single HostClientConnector so that Client may be shared.

Fallback (ex. VIP cascading)

The fallback use case is used when it is desirable to try one set of hosts but switch to a completely different set if there are no hosts in the first set. This can be used for datacenter or rack fallback scenarios where preference should be given to hosts in the same data center but fallback to another datacenter on failures.

Observable<Client> rackA = ...;
Observable<Client> rackB = ...;
Observable<Client> rackC = ...;

rackA()                   // Prefer hosts from rack A
  .concatWith(rackB)      // Switch to rack B if rack A has no hosts
  .single()               // Makes sure that only one host is selected or an error is thrown if no hosts are available
  .concatMap(operation)   // Perform the operation 
  .subscribe(responseHandler);    // Handle the response

A slightly modified use case attempts rackA on the first try but immediately fails over to rackB on any failure

Observable.defer(rackA, rackB, rackC)
  .single()
  .concatMap(operation)
  .retry(2)
  .subscribe(responseHandler);

Consistent Hashing

Consistent hashing is normally used by distributed data stores to partition data across multiple nodes. With consistent hashing each node 'owns' a range of tokens in a finite token space. Furthermore, replication strategies are applied to ensure that data is replicated on neighboring nodes. Cassandra, for example, arranges nodes in a ring where each node also contains the data for the next 2 adjacent nodes (assuming replication factor of 3).

Observable<Client> tokenRange = ClientHasher.clientsForRange();
tokenRange
   .concatMap(operation)
   .subscribe(handler);

Sharding