-
-
Notifications
You must be signed in to change notification settings - Fork 328
Lodestar Weekly Standups 2025
Phil Ngo edited this page Feb 7, 2025
·
5 revisions
The Lodestar team hosts planning/standup meetings weekly on Tuesdays at 3:00pm Universal Standard Time. These meetings allow the team to conduct release planning, prioritise tasks, sync on current issues, implementations and provide status updates on the Lodestar roadmap.
Note that these notes are transcribed and summarized by AI language models and may not accurately reflect the context discussed.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7422
- RC1 was pushed yesterday with a change from Nico to increase block production timeouts
- No issues reported so far on beta deployment and Nico's mainnet validators
- Team will do a final review after the call before releasing
- Discussion on optimizing testing infrastructure and server configurations
- Focus on covering both Docker and binary setups due to performance differences
- Proposal to have ARM64 servers in unstable, stable, and beta groups for sanity checking, not required in feature groups.
- We should make sure that all servers of a similar type are actually using similar hardware.
- Redunancies to the core use case/workload that we measure against is important
- Consider that subscribe all subnets helps spot regressions more than it is to be performant while subscribing to all subnets
- We should have holesky validators attached with subscribe all subnets, same hardware to do better apples to apples comparisons
- Debate on the usefulness and performance impact of subscribe all subnets feature
- Consensus that it's not beneficial for validator effectiveness but useful for testing
- We generally see performance of validators suffering under it and is not really beneficial at all.
- Head votes are worse
- Block imports are delayed
- A worse peer overall
- Might be worth putting in our documentation that subscribe all subnets is not recommended for normal use in any validator staking configuration
- We've come to the conclusion that some changes improve validator performance but may regress metrics on subscribe all subnets. There is no correlation that shows better performance on subscribe all subnets means better validator effectiveness for normal users.
- Decision to use it as a tool for spotting regressions and optimizations, but not to cater resources specifically to being performant with subscribe all subnets
- Agreement to not advertise or advocate for subscribe all subnets as a primary feature
- Snappy-wasm + async aggregate will be rebased and tested on a feature server for inclusion
- JS-libp2p will also be rebased and tested on a feature server for inclusion
- Matt will also re-test Multi-Scalar Multiplication (MSM) functionality for blst-ts as an independent feature on unstable
- PeerDAS devnets are turning into Fulu devnets, additional PRs going in to ensure readiness for the next devnet, including Tuyen's optimizations:
- Expected to be ready for review soon
- This is a breaking change but will be hidden under a flag for merging to unstable and disabled by default
- It is not backwards compatible once upgraded
- Once we test everything and the feature is matured, we can then think about writing a migration to completely deprecate and remove the old format which is more important for historical nodes rather than validators
- It is also possible to just use a different bucket name to prevent it from being a breaking change
- Consideration for making it default in Lodestar 2.0 after thorough testing where we can see the reduction of disk size usage
- Note that computation will likely be higher for generating binary diffs
- Discussed having a pruning history flag to get rid of historical blocks and states (we currently store one state every 4096 slots by default) for better UX.
- Some validator type users can have up to 500GB of worthless storage that can be cleaned up via startup using this flag to purge history
- https://github.com/ChainSafe/lodestar/pull/7427
- Discussion on managing long-lived fork branches
- Agreement to merge sooner into unstable branch to avoid conflicts
- Emphasis on merging code that's already in consensus spec into unstable
- Working on FOCIL implementation
- Encountering issues with Engine API and syncing
- Planning to focus efforts on resolving these issues before the next FOCIL call
- Binary diff PR nearly ready for review
- Will deploy to feature group for testing after review
- Completed work on PeerDAS
- Planning to look into MSM implementation
- Working on refactoring PeerDAS code and implementing Twiyn's suggestions
- Discussion on deprecating deposit snapshot after Pectra goes live
- Consideration of pruning old blocks for improved storage management
- Ongoing work on various PRs including flattening fork logic and deprecating eth1 data poll
Agenda: https://github.com/ChainSafe/lodestar/discussions/7395
- Version 1.26 is scheduled for RC release on Friday.
- DevNet 6 is expected to happen this week, but the timeline is uncertain as teams are still fixing DevNet 5 bugs.
- The team aims to include some hard fork numbers by Thursday's ACD call.
- SSZ 1.0.2 release is facing CI issues, but the team plans to swap the token and try again.
- Most remaining Electra tasks are optimizations and not urgent for the 1.26 release.
- The team wants to include the Electra attester slashing handling in Friday's release.
- The team discussed the challenges faced during the Petra upgrade and how to improve for future hard forks like Fulu.
- Key issues identified:
- Scope creep and difficulty in maintaining timelines
- Challenges in prioritizing features and EIPs
- The need for better communication and consensus-building mechanisms
- Suggestions for improvement:
- Consider implementing a more structured approach to EIP inclusion and hard fork planning
- Potentially increase the frequency of hard forks (e.g., two per year) with smaller scopes
- Improve tooling and processes for identifying consensus and prioritizing features
- Balance between decentralized decision-making and the need for clear direction
- The team discussed the difficulty of removing features from a hard fork once implementation has begun, as they are often intertwined in the codebase.
- There was a discussion about the potential benefits of a more modular architecture that would allow for easier feature toggling, similar to what some other clients have implemented.
- The importance of maintaining Ethereum's high reliability standards was emphasized, with the team agreeing that shortening hard fork cycles should not compromise security or stability.
- The team is working on fixing CI issues for the SSZ 1.0.2 release.
- Work continues on optimizing and finalizing Electra features for the upcoming 1.26 release.
- The team is closely monitoring DevNet 5 and preparing for DevNet 6.
- There was a discussion about recent developments in the L2 ecosystem, particularly regarding the Fabric initiative and its potential impact on Ethereum's development process.
The team will continue to work on the 1.26 release and prepare for upcoming discussions at the ACD call and town hall meeting.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7360
- The team discussed the roadmap for Lodestar V2, reviewing a pinned issue with the planned milestones.
- There was a debate about whether to support old forks in V2:
- Consensus was to keep old forks for now unless they become blockers, as removing them would require significant effort.
- The team agreed to remove the sim merge test, as pre-merge networks are no longer relevant.
- Some regressions were observed after merging DevNet 5 to Unstable:
- Increased job wait times
- Issues with mesh peers
- Higher GC on the main thread
- Increased attestation reprocessing (from 4 to 24 per slot on average)
- Gajinder reviewed the new EIP for delayed state root computation.
- Concerns were raised about potential complications for CL validations:
- Without the state root, it's harder to verify if a block was properly executed.
- Suggested adding a commitment or hash of state diff changes to address this issue.
- The scheduled RC release this week may be pushed to next week, depending on the progress in fixing the regressions on unstable and to accommodate potential hard fork release epochs for public testnets.
Matthew:
- Merged non-SIMD changes
- Worked on PeerDAS integration with DevNet 5
- Reviewing PeerDAS specs for discussion with Gajinder
Nazar:
- Fixed memory leak in benchmark setup caused by Vitest runner
- Working on migrating Lodestar repo benchmarks to the upgraded version
Cayman:
- Worked on a PR for Tuyen's ZIG SSZ library (paused for now)
- Analyzed metrics for libP2P 2.0 branch (feat-4), looking promising
- Started familiarizing with sync code for potential refactoring or backfill sync work
- Planning to revisit the Bun runtime effort for running Lodestar
Gajinder:
- Assisted with Kurtosis configs debugging
- Investigated NAPRS and TypeScript type interactions
- Reviewed and commented on EIP for delayed state root (EIP 7732)
Nazar:
- Fixed memory leak in benchmark utility related to Vitest runner
- Working on migrating Lodestar repo benchmarks
Andrew:
- Reaching out to node operators for feedback
- Potential new node operator interested in testing Lodestar
Nico:
- Discovered issues with some testing servers not running binaries as expected
- Planning to rejig testnet infrastructure, consolidating smaller validator nodes
- Participated in Twitter spaces event bridging builders with core devs and researchers
The team will continue to debug the Unstable branch issues asynchronously and reassess the RC release schedule based on progress.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7337
- Metrics review for 1.25 release showed no concerning issues. Release planned for later today.
- Next RC due date discussed for 1.26:
- Agreed to cut RC on January 24th, aiming for release on January 31st
- Acknowledged shorter timeframe but aligns with goal of more frequent releases
- Plans include merging work to switch to Snappy Wasm for decompression once it is rebased and redeployed.
- They also want to revisit the libp2p 2.0 and GossipSub updates, testing them separately from the “IDONTWANT” feature to address prior latency issues.
- SIMD support for SSZ was merged and tested, aiming for inclusion in 1.26
- Continued benchmarks for different JavaScript runtimes (e.g., Bun, Node) are underway to handle garbage collection impacts on benchmarking consistency
The discussion around the Devnet-5 branch cleanup and integration plan back into unstable covered a few key points:
- There is a need to clean up the DevNet-5 branch.
- The branch contains changes that are not only related to Electra but also affect how forks are handled.
- The DevNet-5 branch includes changes to the NEP (Network Upgrade Process), which requires caution during merging.
- There are preset value updates in the minimal preset that affect CI and sim tests.
- Current CI is failing due to the preset changes.
- The team needs to wait for a Lighthouse image with updated presets to fix the CI issues.
- As a temporary measure, they might consider running tests up to Deneb instead of Electra until the Lighthouse image is available.
Action Items
- Review and fix any issues in the DevNet-5 branch before merging.
- Wait for the Lighthouse image update to resolve CI problems.
- Plan the integration carefully to avoid disrupting the main development flow.
The team emphasized the importance of not rushing the integration process and ensuring thorough testing before merging DevNet-5 changes into the unstable branch.
- Team to review and tag remaining Electra-related todos
- Plan to integrate DevNet-5 changes into Unstable after 1.26 RC
Gajinder
- Reviewed latest PeerDAS spec changes
- Investigated TypeScript and Rust interop for potential light client snarkification
Nazar
- Working on moving runtime setup to SSZ repo
- Debugging benchmark issues related to garbage collection
Nico
- Completed gas limit check PR
- Finished remaining DevNet-5 tasks
- Started looking into FOSSIL implementation
Matt
- Finalized remaining work on the SSZ SIMD PR.
- Focused on PeerDAS groups PR and a thorough review of the overall PeerDAS code.
- Rebasing changes on Electra and merging updates for Fulu, aiming to test them in DevNet soon.
Cayman
- Made PRs to discv5 for library improvements and removing costly crypto dependencies.
- Reviewed the SSZ ZIG implementation and has a PR coming for enhancements.
- Continues to review PRs such as Peer Discovery for subnet requests and related code[2].
Tuyen
- Continued work on BLST binding for Zig and Bun testing signature, secret key, and public key classes.
- Working on more advanced features like verifying multiple aggregate signatures.
- Raised small PRs for PeerDAS improvements, including minimum subnet checks and batch download fixes.
NC
- Focused on FOCIL implementation, estimated to be halfway complete.
- Contributing to the Beacon API aspects for FOCIL and coordinating on design discussions in the open spec documents.
Phil
- Upcoming call with Alex Stokes and Peter for quarterly feedback
- Coordinating team attendance for upcoming interops
Andrew
- Followed up on RPC endpoints discussion with infrastructure team
- Reconnecting with operators for feedback, including questions on new release stability and validator effectiveness data
Peter
- Highlighted an upcoming blog post draft for a new “Fabric” standard for rollups, aiming to unify L2 solutions without introducing new networks or tokens
- Mentioned synergy with potential L2 API expansions in Lodestar and continuing research on universal rollup definitions
Agenda: https://github.com/ChainSafe/lodestar/discussions/7326
- Implement stricter cadence for releases to prevent performance regressions from large accumulations of unreleased commits.
- Proposed cadence: Close door to PRs on specific day (Friday), cut RC, deploy to beta group, review metrics and release after standup the following week.
- Goal is to avoid situations like the recent accumulation of 97 PRs.
-
Various PRs discussed. The following aimed for this specific release:
- Snappy Wasm to be redeployed, observed and merged to unstable
- refactor: update snappy frame decompress to be included
- feat: use subnet request mechanism for subnet sampling strategy to be reviewed with Tuyen and Matt
- refactor: use SubnetID type from CL spec instead of number has merge conflicts. To be resolved by Nico
- feat: implement EIP-7691 increase blob throughput needs one more round of reviews. Needs to be merged for devnet-5 launch
- Confirmed two-week cadence for releases; aim to deploy RC to beta group by end of week.
- FOCIL project gaining momentum; need to assign lead and break down spec for implementation.
- NC and Nico will take charge on this implementation
- Spec itself is small. Changes to beacon node, validator, P2P and fork choice. Due for next breakout call January 15.
- PeerDAS is high priority; need to get it working to avoid max blob count issues. Continue implementation and testing.
- Matt and Gajinder will take charge on this implementation. Tuyen in support role.
- Kurtosis configs and interop testing scheduled for two weeks from now.
- Local testing via Kurtosis is ongoing, but no public spin-up yet.
- Builder flow will start being tested.
- Focus on fixing bugs and ensuring stability before public release.
- Closing role, shifting BlockOps role back towards software engineering focus.
-
Cayman:
- Completed ENR encoding, decoding, signing, and verifying implementation in Zig.
- Will share work and findings with team soon.
-
Tuyen:
- Making progress on getting Zig BLS library operational.
- Expects to have good news to share next week.