Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement file blockstore #660

Merged
merged 20 commits into from
Jan 9, 2025
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ axum = "0.7.5"
base64 = "0.22.1"
bitflags = "2.5.0"
blake2b_simd = { version = "1.0.2", default-features = false }
blockstore = "0.7.1"
bls12_381 = "0.8"
bs58 = "0.5.1"
byteorder = "1.5.0"
Expand Down
6 changes: 6 additions & 0 deletions mater/lib/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ name = "mater" # name WIP
repository.workspace = true
version = "0.1.0"

[features]
blockstore = ["dep:blockstore"]

[dependencies]
async-stream.workspace = true
bitflags.workspace = true
Expand All @@ -27,6 +30,9 @@ tokio = { workspace = true, features = ["fs", "macros", "rt-multi-thread"] }
tokio-stream.workspace = true
tokio-util = { workspace = true, features = ["io"] }

# Optional dependencies
blockstore = { workspace = true, optional = true }

[dev-dependencies]
criterion = { workspace = true, features = ["async_tokio", "html_reports"] }
rand = { workspace = true, default_features = true }
Expand Down
2 changes: 1 addition & 1 deletion mater/lib/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ mod v2;

// We need to re-expose this because `read_block` returns `(Cid, Vec<u8>)`.
pub use ipld_core::cid::Cid;
pub use stores::{create_filestore, Blockstore, Config};
pub use stores::{create_filestore, Blockstore, Config, FileBlockstore};
pub use v1::{Header as CarV1Header, Reader as CarV1Reader, Writer as CarV1Writer};
pub use v2::{
verify_cid, Characteristics, Header as CarV2Header, Index, IndexEntry, IndexSorted,
Expand Down
9 changes: 3 additions & 6 deletions mater/lib/src/stores/blockstore.rs
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ impl Blockstore {

/// Get the [`CarV2Header`] that will be written out.
fn header_v2(&self) -> CarV2Header {
let data_offset = CarV2Header::SIZE as u64;
let data_offset = CarV2Header::SIZE;
let data_size: u64 = self
.blocks
.iter()
Expand Down Expand Up @@ -298,16 +298,13 @@ mod tests {
car_reader.read_pragma().await.unwrap();

let car_v2_header = car_reader.read_header().await.unwrap();
assert_eq!(car_v2_header.data_offset, CarV2Header::SIZE as u64);
assert_eq!(car_v2_header.data_offset, CarV2Header::SIZE);
// Extracted with go-car and validated with an hex viewer
// to extract the values, run the following commands:
// $ car inspect <output of this process>
// The dump is necessary because go-car does not support parametrization
assert_eq!(car_v2_header.data_size, 1358);
assert_eq!(
car_v2_header.index_offset,
(CarV2Header::SIZE as u64) + 1358
);
assert_eq!(car_v2_header.index_offset, CarV2Header::SIZE + 1358);

let car_v1_header = car_reader.read_v1_header().await.unwrap();
assert_eq!(car_v1_header.roots.len(), 1);
Expand Down
Loading