Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming bulk load API #93

Open
marvin-j97 opened this issue Dec 13, 2024 · 0 comments
Open

Streaming bulk load API #93

marvin-j97 opened this issue Dec 13, 2024 · 0 comments
Labels
api enhancement New feature or request epic good first issue Good for newcomers help wanted Extra attention is needed test

Comments

@marvin-j97
Copy link
Contributor

marvin-j97 commented Dec 13, 2024

Possible API

// tree needs to be empty
//
// writer wraps MultiWriter
let mut writer = tree.ingest()?;

// items needs to be sorted ascending
for item in items {
  writer.write(item.k, item.v)?;
}

// commit needs to register all segments into the last (?) level
writer.commit()?;

Should also work for blob tree.

Possible API consideration: hash all inputs and allow verifying import was 100% correct (paranoid integrity check).

@marvin-j97 marvin-j97 added enhancement New feature or request help wanted Extra attention is needed good first issue Good for newcomers test api labels Dec 13, 2024
@marvin-j97 marvin-j97 transferred this issue from fjall-rs/fjall Dec 14, 2024
@marvin-j97 marvin-j97 changed the title Bulk Ingestion API Streaming bulk load API Dec 14, 2024
@marvin-j97 marvin-j97 pinned this issue Dec 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api enhancement New feature or request epic good first issue Good for newcomers help wanted Extra attention is needed test
Projects
None yet
Development

No branches or pull requests

1 participant