Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable global thread pool in python #23495

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

alex-halpin
Copy link

@alex-halpin alex-halpin commented Jan 26, 2025

Description

allows users to create a global thread pool via the python interface

  • exposes the use_per_session_threads attribute to the python binding
  • adds functionality to EnvInitializer singleton that allows setting of global thread pool sizes before instantiation

Motivation and Context

#23523

my particular use case requires the instantiation of many models in memory at once (thousands). this results in the spinning up of an unmanageable number of threads and halting the program. the global thread pool allows me to effectively limit the number of threads created by the inference sessions.

the global thread pool obviously already existed as a feature in C api, just was not exposed to python.

@alex-halpin
Copy link
Author

@microsoft-github-policy-service agree company="PrizePicks"

@alex-halpin alex-halpin marked this pull request as ready for review January 28, 2025 23:15
@snnn snnn added core runtime issues related to core runtime python Pull requests that update Python code labels Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core runtime issues related to core runtime python Pull requests that update Python code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants