Enabling tracing by usage of opentelemetry #99

- Avoid unnecessary copying of prev outpoints - When looking for both mempool and on-chain txos, accumulate the set of outpoints that remain to be looked up to avoid re-checking for them later again in the found set - Refactored lookup_txos() to use lookup_txo() internally rather than the other way around, which was less efficient - Lookup txos in mempool first, then on-chain - ChainQuery::lookup_txos() now returns a Result instead of panicking when outpoints are missing - Removed ChainQuery::lookup_avail_txos() and allow_missing, which are no longer neceesary

add binLiquid to the inherit list to enable specific builds

Using criterion so that we don't need nightly to run benchmark. The "bench" features is also introduced as a way to bench private methods via public methods feature gated via this feature.

on my machine: add_blocks time: [4.7562 ms 4.7754 ms 4.7972 ms] change: [-15.059% -14.650% -14.169%] (p = 0.00 < 0.05) Performance has improved.

add_blocks time: [3.9458 ms 3.9586 ms 3.9717 ms] change: [-17.564% -17.103% -16.660%] (p = 0.00 < 0.05) Performance has improved.

add_blocks time: [2.9788 ms 2.9938 ms 3.0101 ms] change: [-24.810% -24.373% -23.897%] (p = 0.00 < 0.05) Performance has improved.

Now on nix env we provide the env vars of the executables needed for integration testing, so we can enable tests. To be coherent with the electrum nix version used, upgrade also the autodownloaded one. Note we have to change a test assertion, it seems electrum behaviour changed, upgrading balances before confirmation.

to instead have logs like before use `RUST_LOG=debug cargo test`

By doing so simply `cargo run` launch the server, while now it is required `cargo run --bin electrs`

While compaction during initial sync makes thing much slower, it may be preferred to not require to size the disk the double of the final required space.

it is less interesting to see how many rows are written in the db and more interesting knowing the last height indexed

@TheBlueMatt

This actually hurts performance because the batched response has to be bueffered on the bitcoind side, as @TheBlueMatt explains at romanz#373 (comment) Instead, send multiple individual RPC requests in parallel using a thread pool, with a separate RPC TCP connection for each thread. Also see romanz#374

The indexing process was adding transactions into the store so that prevouts funded & spent within the same batch could be looked up via Mempool::lookup_txos(). If the indexing process later failed for any reason, these transactions would remain in the store. With this change, we instead explicitly look for prevouts funded within the same batch, then look for the rest in the chain/mempool indexes and fail if any are missing, without keeping the transactions in the store.

Previously, if any of the mempool transactions were not available because they were evicted between getting the mempool txids and txs themselves, the mempool syncing operation would be aborted and tried again from scratch. With this change, we instead keep whatever transactions we were able to fetch, then get the updated list of mempool txids and re-try fetching missing ones continuously until we're able to get a full snapshot.

- Reduce logging level for bitcoind's JSONRPC response errors These can happen pretty often for missing mempool txs and do not warrant warn-level logging. Unexpected RPC errors will bubble up and be handled appropriately. - Add more verbose logging for mempool syncing

Keep RPC TCP connections open between sync runs and reuse them, to reduce TCP connection initialization overhead.

Following Blockstream#89 (comment) and Blockstream#89 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabling tracing by usage of opentelemetry #99

Enabling tracing by usage of opentelemetry #99

Commits on Nov 6, 2024

Enabling tracing by usage of opentelemetry #99

Are you sure you want to change the base?

Enabling tracing by usage of opentelemetry #99

Commits on Nov 6, 2024