Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Turbopack] add persisting to the new backend #69668

Merged
merged 46 commits into from
Oct 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
63e47a2
add lmdb persisting and restoring
sokra Aug 9, 2024
c9f303a
pass test name to test_config to construct db name
sokra Aug 9, 2024
a08b822
continue uncompleted operations
sokra Aug 9, 2024
77d5258
create dir and logging
sokra Aug 12, 2024
9719916
improve error messages
sokra Aug 13, 2024
df96007
handle keys larger than 511 bytes
sokra Aug 13, 2024
7c092e7
avoid storing transient tasks
sokra Aug 14, 2024
3e9d86f
show lookup error
sokra Aug 14, 2024
f190d6f
handle state serialization
sokra Aug 14, 2024
c4a2414
validate serialization and improve errors
sokra Aug 14, 2024
f76a16e
add turbo_tasks_backend to tracing, add tracing for restore
sokra Aug 16, 2024
e0fd036
disable TLS
sokra Aug 16, 2024
bf6e895
print lookup error
sokra Aug 16, 2024
b77626b
verify serialization
sokra Aug 16, 2024
6fa201d
fix lookup deserialization
sokra Aug 16, 2024
de99d8b
replace bincode with pot
sokra Aug 16, 2024
31d3f07
fix restore data trace
sokra Aug 17, 2024
0e016fb
more tracing in db
sokra Aug 17, 2024
d26c6c8
remove verify_serialization
sokra Aug 19, 2024
02eed51
fix race condition
sokra Aug 17, 2024
0ddd690
do not interrupt persisting while there is data
sokra Aug 19, 2024
ebab310
add persist trace
sokra Aug 19, 2024
d980550
improve task aggregation
sokra Sep 3, 2024
23ebb6e
restore task_pair
sokra Sep 4, 2024
2c51283
verify persistent function only calls persistent functions
sokra Aug 19, 2024
8fa030f
clippy
sokra Sep 6, 2024
6cc4c27
add more details to save_snapshot tracing
sokra Sep 20, 2024
ca084fc
store task data and aggregation separately
sokra Sep 25, 2024
a50a274
compare with old value before storing to db
sokra Sep 24, 2024
bdd8f8f
Early stop snapshot interval when stopping
sokra Sep 24, 2024
9ef72f7
parallel save_snapshot
sokra Sep 24, 2024
9ec165c
run snapshot in spawn_blocking
sokra Sep 25, 2024
4a9e70c
save snapshot when starting idle
sokra Sep 25, 2024
1624294
wait until idle for a second before persisting
sokra Sep 25, 2024
2f3b5fb
increase max DB size
sokra Sep 26, 2024
e492741
remove db lookup tracing
sokra Sep 26, 2024
8024141
remove database println
sokra Sep 26, 2024
361c1cf
sharded transaction log
sokra Sep 26, 2024
451c135
share read transition
sokra Oct 4, 2024
bbc443b
remove aggregation update queue tracing
sokra Sep 27, 2024
dee5f7b
improve messaging
sokra Sep 27, 2024
be67250
handle idle end event for idle detection in persisting
sokra Sep 27, 2024
077eec9
improve imports
sokra Oct 4, 2024
041751d
clippy
sokra Oct 4, 2024
594b0a8
apply review comments
sokra Oct 8, 2024
bdd1c62
clippy
sokra Oct 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 53 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions turbopack/crates/turbo-tasks-backend/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,31 @@ bench = false
[lints]
workspace = true

[features]
default = []
verify_serialization = []

[dependencies]
anyhow = { workspace = true }
async-trait = { workspace = true }
auto-hash-map = { workspace = true }
byteorder = "1.5.0"
dashmap = { workspace = true, features = ["raw-api"]}
either = { workspace = true }
hashbrown = { workspace = true }
indexmap = { workspace = true }
lmdb-rkv = "0.14.0"
once_cell = { workspace = true }
parking_lot = { workspace = true }
pot = "3.0.0"
rand = { workspace = true }
rayon = { workspace = true }
rustc-hash = { workspace = true }
serde = { workspace = true }
serde_path_to_error = { workspace = true }
smallvec = { workspace = true }
tokio = { workspace = true }
tokio-scoped = "0.2.0"
tracing = { workspace = true }
turbo-prehash = { workspace = true }
turbo-tasks = { workspace = true }
Expand Down
Loading
Loading