Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persist the BlobStore with two-layer modeling #9

Open
toru opened this issue Oct 23, 2024 · 0 comments
Open

Persist the BlobStore with two-layer modeling #9

toru opened this issue Oct 23, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@toru
Copy link
Member

toru commented Oct 23, 2024

The original plan for blob storage was to keep things simple, and persist the blobs in a sequence of:

* Blob ID / Value Checksum (32-bytes)
* Reference Count (4-bytes)
* Value Length (8-bytes)
* Value (Length determined by Value Length)
* Metadata Checksum (4-bytes)

While simple, this format does not support lazy-loading efficiently. Instead, we will decouple the blob metadata as a fixed-length record from the unstructured dynamic value data. The new fixed-length metadata format is:

* Blob ID / Value Checksum (32-bytes)
* Reference Count (4-bytes)
* Value Length (8-bytes)
* Value Offset (8-bytes)
* Metadata Checksum (4-bytes)

With this change, value data is stored separately, after the fixed-length regions of the file. By adopting this revised format, we can sort the fixed-length metadata records by their Blob ID, and provide binary search capability for efficient lazy-loading.

@toru toru added the enhancement New feature or request label Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant