Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make all inventory collection operations concurrent #6818

Open
andrewjstone opened this issue Oct 10, 2024 · 2 comments
Open

Make all inventory collection operations concurrent #6818

andrewjstone opened this issue Oct 10, 2024 · 2 comments
Labels
Update System Replacing old bits with newer, cooler bits

Comments

@andrewjstone
Copy link
Contributor

We have a lot more inventory collected since #4753 and all of it is collected serially. We should make it all concurrent.

@andrewjstone andrewjstone added the Update System Replacing old bits with newer, cooler bits label Oct 10, 2024
@hawkw
Copy link
Member

hawkw commented Oct 10, 2024

If nobody else is planning on working on this, I'd be happy to take a crack at it, I could use a little break from RFD 520. :)

@jgallagher
Copy link
Contributor

Porting over @andrewjstone's comment on a PR:

There is actually a comment regarding keeping things serial:

// We're about to do a bunch of asynchronous operations. With a
// combination of async, futures, and some cleverness, we could do much
// of this in parallel. But this code path is not remotely
// latency-sensitive. And there's real risk of overloading our
// downstream services. So we just do one step at a time. This also
// keeps the code simpler.

I think instead we should probably do it in parallel and limit the concurrency with buffer_unordered or something.

This comment seems like it still applies even though we're collecting more stuff, I think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Update System Replacing old bits with newer, cooler bits
Projects
None yet
Development

No branches or pull requests

3 participants