feat(sync): add state diff sync to p2p sync

[ShahakShama]

This change is 

[ShahakShama]

Reviewable status: 0 of 8 files reviewed, 1 unresolved discussion (waiting on @nagmo-starkware and @ShahakShama)

---

Cargo.toml line 128 at r1 (raw file):

[patch.crates-io]
starknet_api = { git = "https://github.com/starkware-libs/starknet-api", branch = "shahak/ThinStateDiff_is_empty" }

Note to self: fix this once https://reviewable.io/reviews/starkware-libs/starknet-api/192 is merged

[codecov]

Codecov Report

Attention: Patch coverage is 73.78049% with 43 lines in your changes are missing coverage. Please review.

Project coverage is 72.02%. Comparing base (ad8e8f6) to head (e78b88b).

Files	Patch %	Lines
crates/papyrus_p2p_sync/src/state_diff.rs	76.37%	17 Missing and 13 partials ⚠️
crates/papyrus_p2p_sync/src/stream_factory.rs	55.55%	8 Missing and 4 partials ⚠️
crates/papyrus_p2p_sync/src/header.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1812      +/-   ##
==========================================
+ Coverage   71.99%   72.02%   +0.02%     
==========================================
  Files         142      143       +1     
  Lines       20369    20522     +153     
  Branches    20369    20522     +153     
==========================================
+ Hits        14664    14780     +116     
- Misses       3531     3552      +21     
- Partials     2174     2190      +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nagmo-starkware]

Reviewed 3 of 8 files at r1, 5 of 7 files at r2, all commit messages.
Reviewable status: 8 of 10 files reviewed, 7 unresolved discussions (waiting on @ShahakShama)

---

crates/papyrus_p2p_sync/src/header.rs line 43 at r2 (raw file):

    const DATA_TYPE: DataType = DataType::SignedBlockHeader;
    const BLOCK_NUMBER_LIMIT: BlockNumberLimit = BlockNumberLimit::Unlimited;
    const SHOULD_LOG_ADDED_BLOCK: bool = false;

I think it should be set to true

---

crates/papyrus_p2p_sync/src/lib.rs line 69 at r2 (raw file):

            num_headers_per_query: 10000,
            // State diffs are split into multiple messages, so big queries can lead to a lot of
            // messages in the network buffers.

did you test it to be the case? or is it your estimation?
the network buffer is configurable so normally the application side is doing what makes most sense applicatively and if needed changes the default network buffer size. (usually it's not needed since it is dynamically sized)

---

crates/papyrus_p2p_sync/src/lib.rs line 102 at r2 (raw file):

        "Received an empty state diff part from the network (this is a potential DDoS vector)."
    )]
    EmptyStateDiffPart,

maybe we should just fail to convert it in the converter?

---

crates/papyrus_p2p_sync/src/lib.rs line 110 at r2 (raw file):

         {missing_field}"
    )]
    OldHeaderInStorage { block_number: BlockNumber, missing_field: &'static str },

that means we need to re-sync from genesis? please add it to the error message to make it more helpful

---

crates/papyrus_p2p_sync/src/state_diff.rs line 25 at r2 (raw file):

    ) -> Result<(), StorageError> {
        // TODO(shahak): Add a way to write ThinStateDiff to the storage.
        let state_diff = StateDiff {

why not use a conversion function? (From/ TryFrom)

---

crates/papyrus_p2p_sync/src/state_diff.rs line 74 at r2 (raw file):

            let mut prev_result_len = 0;
            let mut result_len = 0;
            let target_len = storage_reader

Suggestion:

num_expected_state_diffs

---

crates/papyrus_p2p_sync/src/state_diff.rs line 96 at r2 (raw file):

                        return Ok(None);
                    } else {
                        return Err(P2PSyncError::WrongStateDiffLength {

I don't understand this error. what's possible length?

[ShahakShama]

Reviewable status: 8 of 10 files reviewed, 7 unresolved discussions (waiting on @nagmo-starkware and @ShahakShama)

---

crates/papyrus_p2p_sync/src/header.rs line 43 at r2 (raw file):

Previously, nagmo-starkware wrote…

I think it should be set to true

I've set it to true everywhere and removed this variable

---

crates/papyrus_p2p_sync/src/lib.rs line 69 at r2 (raw file):

Previously, nagmo-starkware wrote…

did you test it to be the case? or is it your estimation?
the network buffer is configurable so normally the application side is doing what makes most sense applicatively and if needed changes the default network buffer size. (usually it's not needed since it is dynamically sized)

Yes, I initially set it to 1000 and the node failed so I tuned it down to 100. I can instead increase the header buffer limit from 100,000 to 1,000,000 if you prefer

---

crates/papyrus_p2p_sync/src/lib.rs line 102 at r2 (raw file):

Previously, nagmo-starkware wrote…

maybe we should just fail to convert it in the converter?

I'm not sure which is better so I've added a TODO to consider

---

crates/papyrus_p2p_sync/src/lib.rs line 110 at r2 (raw file):

Previously, nagmo-starkware wrote…

that means we need to re-sync from genesis? please add it to the error message to make it more helpful

It means we need to re-sync from the given block number. Added to the error message

---

crates/papyrus_p2p_sync/src/state_diff.rs line 25 at r2 (raw file):

Previously, nagmo-starkware wrote…

why not use a conversion function? (From/ TryFrom)

It's a hack. I don't want other people to accidentally use it. In general you can't convert ThinStateDiff to StateDiff

---

crates/papyrus_p2p_sync/src/state_diff.rs line 74 at r2 (raw file):

            let mut prev_result_len = 0;
            let mut result_len = 0;
            let target_len = storage_reader

Done

---

crates/papyrus_p2p_sync/src/state_diff.rs line 96 at r2 (raw file):

Previously, nagmo-starkware wrote…

I don't understand this error. what's possible length?

The length that the block's state diff could be

[ShahakShama]

Reviewable status: 8 of 10 files reviewed, 7 unresolved discussions (waiting on @nagmo-starkware)

---

crates/papyrus_p2p_sync/src/lib.rs line 110 at r2 (raw file):

Previously, ShahakShama wrote…

It means we need to re-sync from the given block number. Added to the error message

Done.

[nagmo-starkware]

Reviewable status: 3 of 10 files reviewed, 1 unresolved discussion (waiting on @ShahakShama)

---

crates/papyrus_p2p_sync/src/lib.rs line 69 at r2 (raw file):

Previously, ShahakShama wrote…

Yes, I initially set it to 1000 and the node failed so I tuned it down to 100. I can instead increase the header buffer limit from 100,000 to 1,000,000 if you prefer

no I think we should change the socket buffer. resolving for now

---

crates/papyrus_p2p_sync/src/state_diff.rs line 96 at r2 (raw file):

Previously, ShahakShama wrote…

The length that the block's state diff could be

f2f

[ShahakShama]

Reviewable status: 3 of 10 files reviewed, 1 unresolved discussion (waiting on @nagmo-starkware)

---

crates/papyrus_p2p_sync/src/state_diff.rs line 96 at r2 (raw file):

Previously, nagmo-starkware wrote…

f2f

I still think it's useful information. Improved the message

[nagmo-starkware]

Reviewed 1 of 7 files at r2, 3 of 6 files at r3, 3 of 3 files at r4, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ShahakShama)