New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bugfix]fix the thread lock when user input same id #585

Open

GGBond8488 wants to merge 1 commit into triton-inference-server:main from GGBond8488:Bugfix]fix_the_thread_lock_when_receive_same_id

GGBond8488 commented Aug 27, 2024 •

edited

Loading

deploy with the IFB, when user input the paylod as follow:

    {
        "text_input": str(question),
        "max_tokens": 512,
        "bad_words": "",
        "stop_words": stop_words,
        "pad_id": pad_id,
        "end_id": end_id,
        "top_p": 1,
        "id": "ggbond_test",
        "temperature": 0.0000001
    }

if every pyload‘s id is same, will cause the error:

Exception in thread Thread-1 (awaiter_loop):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
self._target(*self._args, **self._kwargs)
File "/code/model_pipeline_name/tensorrt_llm/1/model.py", line 920, in awaiter_loop
del self.triton_user_id_to_req_ids[
KeyError: 'ggbond_test'


          fix the thread lock when user input same id

9cdbd31

GGBond8488 changed the title ~~fix the thread lock when user input same id~~ [Bugfix]fix the thread lock when user input same id

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet