Decoupled Async Execute #350
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kthui
force-pushed
the
jacky-py-aio
branch
2 times, most recently
from
37c8524
to
5333613
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
GuanLuo
reviewed
Apr 5, 2024
Tabrizian
reviewed
Apr 5, 2024
Tabrizian
reviewed
Apr 10, 2024
Tabrizian
approved these changes
Apr 10, 2024
mc-nv
pushed a commit
that referenced
this pull request
Apr 11, 2024
* Add async decoupled execute * Enable decoupled bls async exec * Improve handling for async execute future object * Add docs for async execute for decoupled model * Fix link on docs * Improve docs wording * Improve destruction steps for async execute future object * Piggy back on GIL for protection * Document model should not modify event loop * Use Python add_done_callback * Protect infer_payload_ * Use traceback API that supports Python 3.8 and 3.9 * Update docs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related PR: triton-inference-server/server#7062
Add support for decoupled async execute function on
model.pyand enable overlapping between different calls into the execute function by having them execute on the same event loop. Also, enable async exec BLS function for decoupled async execute function.Closes: triton-inference-server/server#3482