Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Expose TRITONSERVER_Server instance in Python In-Process API #385

Merged
merged 32 commits into from
Aug 30, 2024

Conversation

KrishnanPrash
Copy link
Contributor

Goal: To be able to start frontends in python, given that the server instance is being created with the Python In-Process API (tritonserver).

By exposing the c++ pointer to a running TRITONSERVER_Server instance, we can retrieve the memory address of the instance in python and pass it to the tritonfrontend python package. This package will then use reinterepet_cast<...> and recover the TRITONSERVER_Server object from the memory address given.

@rmccorm4 rmccorm4 changed the title Exposing TRITONSERVER_Server instance in Python In-Process API chore: Expose TRITONSERVER_Server instance in Python In-Process API Aug 6, 2024
GuanLuo
GuanLuo previously approved these changes Aug 8, 2024
@nnshah1
Copy link
Contributor

nnshah1 commented Aug 15, 2024

@KrishnanPrash , @GuanLuo , @rmccorm4 -

Two points:

  1. Instead of passing the underlying TRITONSERVER object between modules - could we pass the PyServer object? Reasoning: Can avoid directly exposing the TRITONSERVER ptr and leverage PyBind / ctypes to access the pointer to PyServer. If this is not possible / advisable due to ABI compat - would just like confirmation to understand.

  2. If we need to expose the ptr, let's make it private with a leading underscore and also remove the get_c - as we don't use get / set generally.

Co-authored-by: Neelay Shah <[email protected]>
@KrishnanPrash
Copy link
Contributor Author

@nnshah1

  1. To my understanding, passing the PyServer object would not be sufficient, because tritonfrontend (and by extension server repo) does not have access to the PyServer implementation and therefore cannot recover the underlying pointer to the TRITONSERVER_SERVER object.
  2. Thank you for the suggestion, the new function name should be just _ptr().

python/tritonserver/_c/tritonserver_pybind.cc Outdated Show resolved Hide resolved
python/tritonserver/_c/tritonserver_pybind.cc Outdated Show resolved Hide resolved
.gitignore Outdated Show resolved Hide resolved
@KrishnanPrash KrishnanPrash merged commit 623d0a5 into main Aug 30, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants