Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: parallelize registry builds #86

Merged
merged 13 commits into from
Nov 14, 2023
Merged

Conversation

marshacb
Copy link
Collaborator

@marshacb marshacb commented Nov 14, 2023

  • Parallelizes querys to nodes to improve registry build times
  • Skips processing certain RPC endpoints throwing abnormal closures/errors which caused registry builds to fail
  • Removes XcmV3MultiLocation type construction and replaces with custom XCM MultiLocation type

closes: #81

parallelize calls to nodes in order to speed up building of the ATA registry
fix issue where existing poolAssets, foreignAsssets and assets Info were not being preserved when updating the registry
update fetChainInfo inline doc
@marshacb marshacb requested review from TarikGul and bee344 November 14, 2023 17:36
src/createRegistry.ts Outdated Show resolved Hide resolved
update logged message relay chain message
Copy link
Member

@TarikGul TarikGul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I think this is a great first step to parallelizing the calls. But we should continue to increase the performance by isolating the connections into parallel connections. So currently we just add each chains process as a single clump into a Promise.all, but if we want to make it even faster we can take each rpc connection under a single chain and run them all at the same time.

Here's how we do it in sidecar for /blocks?range=: https://github.com/paritytech/substrate-api-sidecar/blob/master/src/util/PromiseQueue.ts

We can also use generators in JS to achieve this but the above Promise queue is a little faster. We can also set a max size so we don't run to many connection at parallel.

For example: Under Kusama: we can take every parachain, and run all the rpc connections, and calls in clumps of 5-10. and so on

@TarikGul
Copy link
Member

Let's merge this in as its a big upgrade, then we can focus on increasing performance even more. It might take a bit of refactoring to get each rpc connection isolated into a Promise Queue.

@marshacb marshacb merged commit 9a9b3c6 into main Nov 14, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parralelize all calls to the node
2 participants