Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shard-centric Interface #14

Open
adammck opened this issue Jun 1, 2022 · 0 comments
Open

Shard-centric Interface #14

adammck opened this issue Jun 1, 2022 · 0 comments
Labels

Comments

@adammck
Copy link
Owner

adammck commented Jun 1, 2022

One of the key insights of the Shard Manager paper is that many services do not need dynamic range-based sharding in order to balance their load. They can simply pre-split the keyspace into a fixed (though large) number of shards, and balance load by moving shards around. Not having to deal with key ranges can make implementation/integration of consumers much simpler, and is also a more natural fit when wrapping systems which are already partitioned statically and just want balancing.

This is unlike the Slicer model, which is key-range-based like Ranger. (Though the paper claims "Slicer will support unhashed application-defined keys and implement range sharding to preserve locality among adjacent application-defined keys"; maybe that future is here already.)


I believe we can implement this as a subset of the range-based approach as follows:

  • Allow controller to operate in "pre-sharded" mode, where:
    • All ranges are created at Keyspace creation, where we currently create the genesis range.
    • Range start and end keys are always the same; they have a single key. (One would assume that this key would be a hash, but that's up to the consumer.)
    • Ranges are never split nor joined. Only placed and moved.
  • Provide an alternative shard-centric Rangelet interface, which is basically the same but uses RangeKey (or ShardKey? something else?) parameters instead of RangeMeta and RangeID. This would be a helpful wrapper around the existing interface, for keyspaces where the controller is in pre-sharded mode. Same underlying RPCs etc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant