Implement `suggest_scan_ranges` and updated chain validation. #872

nuttycom · 2023-07-07T00:07:08Z

No description provided.

codecov · 2023-07-07T19:41:27Z

Codecov Report

Patch coverage: 72.45% and project coverage change: +0.27 🎉

Comparison is base (4d5dc28) 69.85% compared to head (6bddc46) 70.12%.

❗ Current head 6bddc46 differs from pull request most recent head c7b308b. Consider uploading reports for the commit c7b308b to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #872      +/-   ##
==========================================
+ Coverage   69.85%   70.12%   +0.27%     
==========================================
  Files         127      129       +2     
  Lines       12006    12480     +474     
==========================================
+ Hits         8387     8752     +365     
- Misses       3619     3728     +109

Impacted Files	Coverage Δ
zcash_client_backend/src/data_api.rs	`40.00% <0.00%> (-0.68%)`	⬇️
zcash_client_backend/src/data_api/chain.rs	`56.62% <ø> (ø)`
zcash_client_backend/src/proto/service.rs	`0.00% <0.00%> (ø)`
zcash_client_backend/src/scanning.rs	`42.40% <0.00%> (-2.85%)`	⬇️
zcash_client_sqlite/src/wallet/init.rs	`53.57% <0.00%> (-0.65%)`	⬇️
zcash_primitives/src/sapling/tree.rs	`89.13% <0.00%> (-6.22%)`	⬇️
zcash_client_sqlite/src/wallet/commitment_tree.rs	`93.68% <37.50%> (-1.30%)`	⬇️
...te/src/wallet/init/migrations/shardtree_support.rs	`55.20% <48.71%> (+1.98%)`	⬆️
zcash_client_sqlite/src/chain.rs	`52.42% <50.00%> (-0.10%)`	⬇️
zcash_client_backend/src/data_api/scanning.rs	`82.85% <82.85%> (ø)`
... and 5 more

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

zcash_client_backend/src/data_api/chain.rs

str4d · 2023-07-08T01:30:17Z

zcash_client_sqlite/src/wallet/scanning.rs

+        );
+        insert_into_cache(&db_cache, &cb);
+
+        for i in 1..=10 {


This should be an exclusive end, or the 11th block should either be scanned, or expected not to be scanned (and noted as such somewhere).

zcash_client_sqlite/src/wallet/scanning.rs

This implements a priority queue backed by the wallet database for scan range ordering. The scan queue is updated on each call to `put_blocks` or to `update_chain_tip`.

zcash_client_backend/CHANGELOG.md

zcash_client_backend/proto/service.proto

str4d

Flushing my review comments.

str4d · 2023-07-10T20:29:34Z

zcash_client_backend/proto/service.proto

@@ -118,6 +118,22 @@ message TreeState {
    string orchardTree = 6; // orchard commitment tree state
 }

+enum ShieldedProtocol {


I confirmed that this matches the canonical state of service.proto as of zcash/lightwalletd@ab90099.

zcash_client_backend/src/data_api/scanning.rs

zcash_client_sqlite/src/wallet/scanning.rs

daira · 2023-07-18T17:57:52Z

zcash_client_sqlite/src/wallet/scanning.rs

+        use super::RangeOrdering::*;
+        // Equal
+        assert_eq!(RangeOrdering::cmp(&(0..1), &(0..1)), Equal);
+


Suggested change

assert_eq!(RangeOrdering::cmp(&(1..1), &(1..1)), Equal);

This is technically needed to cover all the cases, but it's non-blocking.

daira

Reviewed with blocking comments (the questions about ordering of inputs to ScanRange::from_parts).

daira · 2023-07-18T18:25:23Z

zcash_client_sqlite/src/wallet/scanning.rs

+                    LeftFirstOverlap => {
+                        let split_point = left.span().end;
+                        if split_point > to_insert.block_range().start {
+                            Self::from_split(*left, *right, to_insert, split_point)


I think we should pay attention to the check warnings saying that some of these cases are not covered by tests. If you disagree, I won't block on it.

zcash_client_sqlite/src/wallet/scanning.rs

Co-authored-by: Daira Hopwood <[email protected]>

Previously it would panic due to constructing invalid `ScanRange`s.

zcash_client_sqlite/src/wallet/scanning.rs

daira

utACK

zcash_client_sqlite/src/wallet.rs

zcash_client_sqlite/src/lib.rs

Co-authored-by: Daira Emma Hopwood <[email protected]>

ebfull

ACK and tested. I haven't thoroughly reviewed some parts but those appear to have been given plenty of attention by others.

daira

utACK; remaining suggestions are non-blocking.

nuttycom force-pushed the feature/pre_dag_sync-suggest_scan_ranges branch 5 times, most recently from 12e182d to 86a4af7 Compare July 7, 2023 19:22

nuttycom changed the title ~~WIP: Implement suggest_scan_ranges and updated chain validation.~~ Implement suggest_scan_ranges and updated chain validation. Jul 7, 2023

nuttycom marked this pull request as ready for review July 7, 2023 19:22

nuttycom force-pushed the feature/pre_dag_sync-suggest_scan_ranges branch from 86a4af7 to ea5faa0 Compare July 7, 2023 19:40

nuttycom force-pushed the feature/pre_dag_sync-suggest_scan_ranges branch 7 times, most recently from 4d775aa to c5a8609 Compare July 7, 2023 23:51

str4d reviewed Jul 8, 2023

View reviewed changes

zcash_client_backend/src/data_api/chain.rs Outdated Show resolved Hide resolved

zcash_client_backend/src/data_api/chain.rs Show resolved Hide resolved

nuttycom force-pushed the feature/pre_dag_sync-suggest_scan_ranges branch 2 times, most recently from a7b59bb to caea5da Compare July 8, 2023 01:09

str4d reviewed Jul 8, 2023

View reviewed changes

zcash_client_sqlite/src/wallet/scanning.rs Outdated Show resolved Hide resolved

nuttycom added 3 commits July 7, 2023 20:13

Implement suggest_scan_ranges and update_chain_tip

6fa0b46

This implements a priority queue backed by the wallet database for scan range ordering. The scan queue is updated on each call to `put_blocks` or to `update_chain_tip`.

Fix an error in dominance calculation.

6db4355

Add a test for scan_complete and update_chain_tip

352e1c7

nuttycom force-pushed the feature/pre_dag_sync-suggest_scan_ranges branch from caea5da to 352e1c7 Compare July 8, 2023 02:13

daira reviewed Jul 10, 2023

View reviewed changes

zcash_client_backend/CHANGELOG.md Show resolved Hide resolved

daira reviewed Jul 10, 2023

View reviewed changes

zcash_client_backend/proto/service.proto Show resolved Hide resolved

str4d reviewed Jul 10, 2023

View reviewed changes

daira reviewed Jul 10, 2023

View reviewed changes

zcash_client_backend/src/data_api/scanning.rs Outdated Show resolved Hide resolved

daira reviewed Jul 10, 2023

View reviewed changes

zcash_client_backend/src/data_api/scanning.rs Outdated Show resolved Hide resolved

daira reviewed Jul 18, 2023

View reviewed changes

zcash_client_sqlite/src/wallet/scanning.rs Outdated Show resolved Hide resolved

daira reviewed Jul 18, 2023

View reviewed changes

zcash_client_sqlite/src/wallet/scanning.rs Show resolved Hide resolved

daira reviewed Jul 18, 2023

View reviewed changes

zcash_client_sqlite/src/wallet/scanning.rs Outdated Show resolved Hide resolved

daira reviewed Jul 18, 2023

View reviewed changes

zcash_client_sqlite/src/wallet/scanning.rs Outdated Show resolved Hide resolved

str4d force-pushed the feature/pre_dag_sync-suggest_scan_ranges branch from 89d5661 to fa8b135 Compare July 18, 2023 18:54

daira reviewed Jul 18, 2023

View reviewed changes

zcash_client_sqlite/src/wallet/scanning.rs Show resolved Hide resolved

str4d force-pushed the feature/pre_dag_sync-suggest_scan_ranges branch from fa8b135 to 1a5bc86 Compare July 18, 2023 19:04

str4d and others added 4 commits July 19, 2023 14:54

Documentation updates, fixes, and cleanups

a87dca0

Co-authored-by: Daira Hopwood <[email protected]>

Migrate to latest shardtree to improve performance

98ea08e

Do nothing if update_chain_tip is given a tip lower than we can use

359ff55

Previously it would panic due to constructing invalid `ScanRange`s.

Add test for join_nonoverlapping.

e16aa41

str4d force-pushed the feature/pre_dag_sync-suggest_scan_ranges branch from 2cb284f to e16aa41 Compare July 19, 2023 14:54

daira reviewed Jul 19, 2023

View reviewed changes

zcash_client_sqlite/src/wallet/scanning.rs Show resolved Hide resolved

daira previously approved these changes Jul 19, 2023

View reviewed changes

daira mentioned this pull request Jul 19, 2023

Update comment in protobuf file zcash/lightwalletd#446

Closed

daira reviewed Jul 19, 2023

View reviewed changes

zcash_client_sqlite/src/wallet.rs Outdated Show resolved Hide resolved

daira reviewed Jul 19, 2023

View reviewed changes

zcash_client_sqlite/src/lib.rs Outdated Show resolved Hide resolved

str4d dismissed daira’s stale review via f679e56 July 19, 2023 16:02

Rename VALIDATION_DEPTH constant to VERIFY_LOOKAHEAD

c7b308b

Co-authored-by: Daira Emma Hopwood <[email protected]>

str4d force-pushed the feature/pre_dag_sync-suggest_scan_ranges branch from f679e56 to c7b308b Compare July 19, 2023 16:03

ebfull approved these changes Jul 19, 2023

View reviewed changes

daira approved these changes Jul 19, 2023

View reviewed changes

str4d merged commit f422188 into zcash:main Jul 19, 2023
10 checks passed

This was referenced Jul 19, 2023

Fix spent nullifier detection for non-linear scanning #876

Closed

zcash_client_sqlite: Improve performance of shardtree database migration #877

Closed

nuttycom deleted the feature/pre_dag_sync-suggest_scan_ranges branch July 31, 2023 22:08

nuttycom mentioned this pull request Aug 16, 2023

Extract "scan arbitrary block range" function from scan_cached_blocks #728

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `suggest_scan_ranges` and updated chain validation. #872

Implement `suggest_scan_ranges` and updated chain validation. #872

nuttycom commented Jul 7, 2023

codecov bot commented Jul 7, 2023 •

edited

Loading

str4d Jul 8, 2023

str4d left a comment

str4d Jul 10, 2023

daira Jul 18, 2023 •

edited

Loading

daira left a comment

daira Jul 18, 2023 •

edited

Loading

daira left a comment

ebfull left a comment

daira left a comment

Implement suggest_scan_ranges and updated chain validation. #872

Implement suggest_scan_ranges and updated chain validation. #872

Conversation

nuttycom commented Jul 7, 2023

codecov bot commented Jul 7, 2023 • edited Loading

Codecov Report

str4d Jul 8, 2023

Choose a reason for hiding this comment

str4d left a comment

Choose a reason for hiding this comment

str4d Jul 10, 2023

Choose a reason for hiding this comment

daira Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

daira left a comment

Choose a reason for hiding this comment

daira Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

daira left a comment

Choose a reason for hiding this comment

ebfull left a comment

Choose a reason for hiding this comment

daira left a comment

Choose a reason for hiding this comment

Implement `suggest_scan_ranges` and updated chain validation. #872

Implement `suggest_scan_ranges` and updated chain validation. #872

codecov bot commented Jul 7, 2023 •

edited

Loading

daira Jul 18, 2023 •

edited

Loading

daira Jul 18, 2023 •

edited

Loading