Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hyperchain da removal #259

Merged
merged 30 commits into from
Sep 13, 2024
Merged

Conversation

gianbelinche
Copy link

@gianbelinche gianbelinche commented Sep 12, 2024

What ❔

This PR removes the need to use hyperchain-da repository, most of the changes involve merging main, which makes the diff a bit big, to see the actual changes involved, check the last commit

It is not necessary to do this export anymore export EIGEN_DA_CLIENT_API_NODE_URL=http://127.0.0.1:3100
However this one is needed now: export DA_CLIENT=EigenDA

You can now just run zk server

Why ❔

Checklist

  • PR title corresponds to the body of PR (we generate changelog entries from PRs).
  • Tests for the changes have been added / updated.
  • Documentation comments have been added / updated.
  • Code has been formatted via zk fmt and zk lint.

pbeza and others added 29 commits September 7, 2024 07:38
…#2824)

## What ❔

Passthrough env vars to the SGX enclave.

Relevant logs showcasing the issue:
https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1

## Why ❔

To fix the bug.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
## What ❔
Add setup keys step to prover init
…s#2800)

## What ❔

Add `--docker` flag for `zk_inception prover run` which will allow
prover components to run from docker images.

## Why ❔

To decrease setup time/improve UX

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
## What ❔

Adds README for verified sources fetcher

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
## What ❔

Limits rayon threadpool size to 2 and tokio threadpool size to 4 in
zk_toolbox CI.
I have checked locally, and with this configuration time to run
integration tests is pretty close to the default configuration.

## Why ❔

By default, both tokio and rayon will try to use all the CPUs. When we
run multiple Rust binaries at the same time (3 servers and 3 ENs in our
case), it causes a lot of conflict for resources, regardless of the
number of CPUs.
Extraction of the attester committee from consensus registry state. If
consensus registry address is not specified, we fall back to attester
committee from genesis. This pr does NOT enable the dynamic attestation,
as the registry address needs to be added to the main node config first.

---------

Co-authored-by: Moshe Shababo <[email protected]>
Co-authored-by: Igor Aleksanov <[email protected]>
## What ❔

- Removes a data race from the revert integration test (a single node
one); a batch gets executed after it is verified that not all batches
are executed.
- Removes copy-paste between single-node and EN tests.
- Structures revert tests into smaller steps.

## Why ❔

Makes the test less flaky and easier to maintain.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
…atter-labs#2804)

## What ❔

- Fixes the "missing revert data" error by updating the used reth Docker
image. The error is probably caused by [this
issue](paradigmxyz/reth#7381) fixed in the new
reth versions.
- Removes "web3 API compatibility tests › Should check API returns error
when there are too many logs in eth_getLogs" test as fundamentally flaky
and able to poison other tests.
- Adds logging for upgrade test to investigate L1 "nonce too low"
errors.

## Why ❔

Flaky CI bad.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
…labs#2806)

## What ❔

Extracts oneshot VM executor to the executor crate.

## Why ❔

To make executor logic more reusable and maintainable.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
…abs#2836)

## What ❔

- Fixes DB connection starvation during snapshot recovery. Caused by the
insufficient number of connections in the DB pool provided to recovery
logic.
- Additionally, fixes max concurrency of recovery not being set.

## Why ❔

Connection starvation errors degrade UX.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.
When a method is missing:
proxyd returns HTTP 403:  methodnotfound
while api server returns HTTP 200: methodnotfound

we need to handle both.
)

"{err}" is not useful, because it truncates the stack trace of anyhow
errors.
`zks t i` now accepts a `-t` flag that can can be used to specify a
pattern. Only matching tests are run.
## What ❔

Resolves matter-labs#2783

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
…r-labs#2830)

## What

- Removes `locate_workspace` and `workspace_dir_or_current_dir` methods.
- Instead, introduces `Workspace` type that is aware of different Cargo
workspaces in the codebase.

## Why

The approach with a single `locate_workspace` doesn't work well for our
codebase, since we have multiple workspaces. It resulted in some very
implicit and convoluted code (see the removed `get_base_dir` in prover
workspace).

New approach handles all 3 workspaces _plus_ the lack of a workspace.
## What ❔

Add prover subcommand for `zk_supervisor`. Add the following subcommand:
* `zk_supervisor prover info` - Prints information about current prover
setup.
* `zk_supervisor prover insert-version` - Insert new protocol version in
prover database(integration with `prover_cli`).
* `zk_supervisor prover insert-batch` - Insert new batch in prover
database(integration with `prover_cli`).

Add automatic creation of `prover/artifacts/witness_inputs` dirs if the
storage is file backed on init.

## Why ❔

To improve UX of working with provers.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

Signed-off-by: Danil <[email protected]>
## What ❔

* Reduced uncompressed size from 7.5GB to 4.81 GB.
* Changes to make zk_environment image smaller
* compile solc with release mode
* remove unnecessary components from google cli
* build things first, and then only copy the artifacts.

## Why ❔

* This image is used on ALL of our CI builds - so reducing its size will
speed up the CIs.
## What ❔

Fix getting chain config in zk_supervisor prover
Fix artifacts path when initializing.
Setup data path for provers

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
## What ❔

`cargo install --list` doesn't work properly in CI because we copy only
binaries to zk-environment
also increase reth block-time and fixes upgrade-test which improves
tests stability

## Why ❔

we rely on cargo-nextest in tests

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
## What ❔

This PR changes the approach to managing 3rd party DA clients. It was
assumed before that they will be stored in a separate repository
(hyperchain-da), but to simplify the processes and solve a recursive
dependency problem, we decided to manage those within `zksync-era`.

The config now defines which DA client will be used, for proto-based
configuration it requires adding these lines to general.yaml:
```
da_client:
  avail:
    api_node_url: wss://turing-rpc.avail.so/ws
    bridge_api_url: undefined
    seed: SEED_PHRASE
    app_id: 82
    timeout: 3
    max_retries: 5
``` 
for env-based:
```
    DA_CLIENT="Avail"
    DA_API_NODE_URL="localhost:12345"
    DA_BRIDGE_API_URL="localhost:54321"
    DA_SEED="SEED_PHRASE"
    DA_APP_ID=1
    DA_TIMEOUT=2
    DA_MAX_RETRIES=3
```
If no config is provided - the default behavior is to use NoDA client
(same as now).

The `da_client` config might be merged with `da_dispatcher` at some
point as the second depends on the first one, so their separation does
not make much sense (apart from simplification of the configs). But I'd
prefer to do it as a separate PR in case we decide to merge them.

The client was reimplemented using only lightweight libraries from
crates.io, so it doesn't have any visible impact on build time.

## Why ❔

To enable seamless integration with 3rd party DA clients in
`zksync-era`.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
## What ❔

Support legacy bridges in zk toolbox 

## Why ❔

We still have some legacy bridges in production, so we have to have an
opportunity to test it

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

---------

Signed-off-by: Danil <[email protected]>
## What ❔
Add external_node consensus support

---------

Signed-off-by: Danil <[email protected]>
Co-authored-by: Danil <[email protected]>
## What ❔

* A small tool to generate the selector hashes based on the ABI from
json files

## Why ❔

* The output json can be useful for humans to better understand some of
the errors (and calldata)
* It can also be read by our tools, to make the debugging easier.

In the future, we could call this tool regularly on each contracts
version change, but for now it can stay as manual.
## What ❔

- Loads setup keys to memory in parallel (for GPU prover only).
- Refactors a bunch of related code for simplicity.

## Why ❔

- Locally I've observed load time going from ~30s to ~12s, so ~60%
improvement for prover start time.
- Readability & maintainability.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
## What ❔

- Use latest versions of crypto, protocol, gpu, and consensus crates.
- Remove solved cargo deny advisories from the allowlist.

## Why ❔

- A bunch of fixes/improvements were done.
- Optimization of dependency graph.
- Solving cargo deny advisories.
## What ❔

Introduce new structure for witness generators.
Introduce `ArtifactsManager` trait responsible for operations with
object store and artifacts.

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.
Copy link

github-actions bot commented Sep 12, 2024

Hey there! 👋🏼

We require pull request titles to follow the Conventional Commits specification and it looks like your proposed title needs to be adjusted.
Examples of valid PR titles:

  • feat(eth_sender): Support new transaction type
  • fix(state_keeper): Correctly handle edge case
  • ci: Add new workflow for linting

Details:

No release type found in pull request title "Hyperchain da removal". Add a prefix to indicate what kind of release this pull request corresponds to. For reference, see https://www.conventionalcommits.org/

Available types:
 - feat: A new feature
 - fix: A bug fix
 - docs: Documentation only changes
 - style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
 - refactor: A code change that neither fixes a bug nor adds a feature
 - perf: A code change that improves performance
 - test: Adding missing tests or correcting existing tests
 - build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
 - ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
 - chore: Other changes that don't modify src or test files
 - revert: Reverts a previous commit

@gianbelinche gianbelinche marked this pull request as ready for review September 12, 2024 20:49
@gianbelinche gianbelinche merged commit 824fa98 into eigen-implementation Sep 13, 2024
7 of 50 checks passed
@gianbelinche gianbelinche deleted the hyperchain-da-removal branch September 13, 2024 14:32
juan518munoz pushed a commit that referenced this pull request Sep 17, 2024
* fix(tee-prover): passthrough env vars to the SGX enclave (matter-labs#2824)

## What ❔

Passthrough env vars to the SGX enclave.

Relevant logs showcasing the issue:
https://grafana.matterlabs.dev/goto/1iFHMIeIg?orgId=1

## Why ❔

To fix the bug.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* feat(zk_toolbox): Add setup keys step to prover init (matter-labs#2811)

## What ❔
Add setup keys step to prover init

* feat(zk_toolbox): Allow running docker images for provers (matter-labs#2800)

## What ❔

Add `--docker` flag for `zk_inception prover run` which will allow
prover components to run from docker images.

## Why ❔

To decrease setup time/improve UX

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* chore: Add README for verified sources fetcher (matter-labs#2829)

## What ❔

Adds README for verified sources fetcher

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* chore(ci): Limit tokio/rayon pools for zk_toolbox CI (matter-labs#2828)

## What ❔

Limits rayon threadpool size to 2 and tokio threadpool size to 4 in
zk_toolbox CI.
I have checked locally, and with this configuration time to run
integration tests is pretty close to the default configuration.

## Why ❔

By default, both tokio and rayon will try to use all the CPUs. When we
run multiple Rust binaries at the same time (3 servers and 3 ENs in our
case), it causes a lot of conflict for resources, regardless of the
number of CPUs.

* feat: attester committees data extractor (BFT-434) (matter-labs#2684)

Extraction of the attester committee from consensus registry state. If
consensus registry address is not specified, we fall back to attester
committee from genesis. This pr does NOT enable the dynamic attestation,
as the registry address needs to be added to the main node config first.

---------

Co-authored-by: Moshe Shababo <[email protected]>
Co-authored-by: Igor Aleksanov <[email protected]>

* test: Improve revert integration test (matter-labs#2822)

## What ❔

- Removes a data race from the revert integration test (a single node
one); a batch gets executed after it is verified that not all batches
are executed.
- Removes copy-paste between single-node and EN tests.
- Structures revert tests into smaller steps.

## Why ❔

Makes the test less flaky and easier to maintain.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* test: Fix "missing revert data" error; fix / debug integration tests (matter-labs#2804)

## What ❔

- Fixes the "missing revert data" error by updating the used reth Docker
image. The error is probably caused by [this
issue](paradigmxyz/reth#7381) fixed in the new
reth versions.
- Removes "web3 API compatibility tests › Should check API returns error
when there are too many logs in eth_getLogs" test as fundamentally flaky
and able to poison other tests.
- Adds logging for upgrade test to investigate L1 "nonce too low"
errors.

## Why ❔

Flaky CI bad.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* refactor(api): Extract oneshot VM executor to executor crate (matter-labs#2806)

## What ❔

Extracts oneshot VM executor to the executor crate.

## Why ❔

To make executor logic more reusable and maintainable.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* fix(en): Fix connection starvation during snapshot recovery (matter-labs#2836)

## What ❔

- Fixes DB connection starvation during snapshot recovery. Caused by the
insufficient number of connections in the DB pool provided to recovery
logic.
- Additionally, fixes max concurrency of recovery not being set.

## Why ❔

Connection starvation errors degrade UX.

## Checklist

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [x] Tests for the changes have been added / updated.
- [x] Documentation comments have been added / updated.
- [x] Code has been formatted via `zk fmt` and `zk lint`.

* fix: handling of HTTP 403 thrown by proxyd (matter-labs#2835)

When a method is missing:
proxyd returns HTTP 403:  methodnotfound
while api server returns HTTP 200: methodnotfound

we need to handle both.

* chore: log the whole error message when the task fails (matter-labs#2834)

"{err}" is not useful, because it truncates the stack trace of anyhow
errors.

* feat: allow specifying what tests to run with zks (matter-labs#2841)

`zks t i` now accepts a `-t` flag that can can be used to specify a
pattern. Only matching tests are run.

* chore: clean up dependencies (matter-labs#2839)

## What ❔

Resolves matter-labs#2783

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* feat(utils): Rework locate_workspace, introduce Workspace type (matter-labs#2830)

## What

- Removes `locate_workspace` and `workspace_dir_or_current_dir` methods.
- Instead, introduces `Workspace` type that is aware of different Cargo
workspaces in the codebase.

## Why

The approach with a single `locate_workspace` doesn't work well for our
codebase, since we have multiple workspaces. It resulted in some very
implicit and convoluted code (see the removed `get_base_dir` in prover
workspace).

New approach handles all 3 workspaces _plus_ the lack of a workspace.

* feat(zk_toolbox):  `zk_supervisor prover` subcommand (matter-labs#2820)

## What ❔

Add prover subcommand for `zk_supervisor`. Add the following subcommand:
* `zk_supervisor prover info` - Prints information about current prover
setup.
* `zk_supervisor prover insert-version` - Insert new protocol version in
prover database(integration with `prover_cli`).
* `zk_supervisor prover insert-batch` - Insert new batch in prover
database(integration with `prover_cli`).

Add automatic creation of `prover/artifacts/witness_inputs` dirs if the
storage is file backed on init.

## Why ❔

To improve UX of working with provers.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* fix(zk-toolbox): Make token multiplier optional (matter-labs#2843)

## What ❔

<!-- What are the changes this PR brings about? -->
<!-- Example: This PR adds a PR template to the repo. -->
<!-- (For bigger PRs adding more context is appreciated) -->

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

Signed-off-by: Danil <[email protected]>

* feat: Smaller zk_environment image (matter-labs#1920)

## What ❔

* Reduced uncompressed size from 7.5GB to 4.81 GB.
* Changes to make zk_environment image smaller
* compile solc with release mode
* remove unnecessary components from google cli
* build things first, and then only copy the artifacts.

## Why ❔

* This image is used on ALL of our CI builds - so reducing its size will
speed up the CIs.

* fix(zk_toolbox): secrets path, artifacts path (matter-labs#2850)

## What ❔

Fix getting chain config in zk_supervisor prover
Fix artifacts path when initializing.
Setup data path for provers

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* ci: fix using cargo nextest (matter-labs#2855)

## What ❔

`cargo install --list` doesn't work properly in CI because we copy only
binaries to zk-environment
also increase reth block-time and fixes upgrade-test which improves
tests stability

## Why ❔

we rely on cargo-nextest in tests

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* feat: add da clients (matter-labs#2743)

## What ❔

This PR changes the approach to managing 3rd party DA clients. It was
assumed before that they will be stored in a separate repository
(hyperchain-da), but to simplify the processes and solve a recursive
dependency problem, we decided to manage those within `zksync-era`.

The config now defines which DA client will be used, for proto-based
configuration it requires adding these lines to general.yaml:
```
da_client:
  avail:
    api_node_url: wss://turing-rpc.avail.so/ws
    bridge_api_url: undefined
    seed: SEED_PHRASE
    app_id: 82
    timeout: 3
    max_retries: 5
``` 
for env-based:
```
    DA_CLIENT="Avail"
    DA_API_NODE_URL="localhost:12345"
    DA_BRIDGE_API_URL="localhost:54321"
    DA_SEED="SEED_PHRASE"
    DA_APP_ID=1
    DA_TIMEOUT=2
    DA_MAX_RETRIES=3
```
If no config is provided - the default behavior is to use NoDA client
(same as now).

The `da_client` config might be merged with `da_dispatcher` at some
point as the second depends on the first one, so their separation does
not make much sense (apart from simplification of the configs). But I'd
prefer to do it as a separate PR in case we decide to merge them.

The client was reimplemented using only lightweight libraries from
crates.io, so it doesn't have any visible impact on build time.

## Why ❔

To enable seamless integration with 3rd party DA clients in
`zksync-era`.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* feat(zk_toolbox): deploy legacy bridge (matter-labs#2837)

## What ❔

Support legacy bridges in zk toolbox 

## Why ❔

We still have some legacy bridges in production, so we have to have an
opportunity to test it

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

---------

Signed-off-by: Danil <[email protected]>

* feat(zk_toolbox): Add external_node consensus support (matter-labs#2821)

## What ❔
Add external_node consensus support

---------

Signed-off-by: Danil <[email protected]>
Co-authored-by: Danil <[email protected]>

* feat: Selector generator tool (matter-labs#2844)

## What ❔

* A small tool to generate the selector hashes based on the ABI from
json files

## Why ❔

* The output json can be useful for humans to better understand some of
the errors (and calldata)
* It can also be read by our tools, to make the debugging easier.

In the future, we could call this tool regularly on each contracts
version change, but for now it can stay as manual.

* feat(prover): Optimize setup keys loading (matter-labs#2847)

## What ❔

- Loads setup keys to memory in parallel (for GPU prover only).
- Refactors a bunch of related code for simplicity.

## Why ❔

- Locally I've observed load time going from ~30s to ~12s, so ~60%
improvement for prover start time.
- Readability & maintainability.

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* feat: Bump crypto and protocol deps (matter-labs#2825)

## What ❔

- Use latest versions of crypto, protocol, gpu, and consensus crates.
- Remove solved cargo deny advisories from the allowlist.

## Why ❔

- A bunch of fixes/improvements were done.
- Optimization of dependency graph.
- Solving cargo deny advisories.

* feat(prover): Refactor WitnessGenerator (matter-labs#2845)

## What ❔

Introduce new structure for witness generators.
Introduce `ArtifactsManager` trait responsible for operations with
object store and artifacts.

## Why ❔

<!-- Why are these changes done? What goal do they contribute to? What
are the principles behind them? -->
<!-- Example: PR templates ensure PR reviewers, observers, and future
iterators are in context about the evolution of repos. -->

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [ ] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zk fmt` and `zk lint`.

* Move hyperchain-da to zksync-era

* Uncomment validium config checks

---------

Signed-off-by: Danil <[email protected]>
Co-authored-by: Patrick <[email protected]>
Co-authored-by: Matías Ignacio González <[email protected]>
Co-authored-by: Artem Fomiuk <[email protected]>
Co-authored-by: perekopskiy <[email protected]>
Co-authored-by: Igor Aleksanov <[email protected]>
Co-authored-by: Grzegorz Prusak <[email protected]>
Co-authored-by: Moshe Shababo <[email protected]>
Co-authored-by: Alex Ostrovski <[email protected]>
Co-authored-by: Joonatan Saarhelo <[email protected]>
Co-authored-by: Danil <[email protected]>
Co-authored-by: Marcin M <[email protected]>
Co-authored-by: Dima Zhornyk <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.