Error in azureml/evaluate/mlflow/hftransformers/_task_based_predictors.py - from transformers import pipeline, Conversation

[jakeatmsft]

Operating System

Linux

Version Information

Error when deploying a packaged MLFlow model from HF [huggingface.co/meta-llama/Meta-Llama-Guard-2-8B, huggingface.co/Qwen/Qwen1.5-4B] to Managed online endpoint, error seems to be occurring in mlflow scoring script, in this file: azureml/evaluate/mlflow/hftransformers/_task_based_predictors.py

Steps to reproduce

1. Download HF model - meta-llama/Meta-Llama-Guard-2-8B
2. Convert HF model to MLFlow - using aml component "convert_to_mlflow_model" https://ml.azure.com/registries/azureml/components/convert_model_to_mlflow
3. Package MLFlow model using python sdk v2 (modelpackage).
4. Deploy packaged model env to Managed Online Endpoint.

Expected behavior

Deployed MOE is successful, MOE should respond to JSON request with generated text.

Actual behavior

Error: 2024-07-18 18:08:37,534 E [89] azmlinfsrv - Encountered Exception: Traceback (most recent call last):
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/azureml_inference_server_http/server/user_script.py", line 132, in invoke_run
run_output = self._wrapped_user_run(**run_parameters, request_headers=dict(request.headers))
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/azureml_inference_server_http/server/user_script.py", line 156, in
self._wrapped_user_run = lambda request_headers, **kwargs: self._user_run(**kwargs)
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/inference_schema/schema_decorators.py", line 68, in decorator_input
return user_run(*args, **kwargs)
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/inference_schema/schema_decorators.py", line 68, in decorator_input
return user_run(*args, **kwargs)
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/inference_schema/schema_decorators.py", line 96, in decorator_input
return user_run(*args, **kwargs)
File "/var/mlflow_resources/mlflow_score_script.py", line 448, in run
result = model.predict(input, params=params)
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/mlflow/pyfunc/init.py", line 719, in predict
return self._predict_fn(data)
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/azureml/evaluate/mlflow/hftransformers/init.py", line 1002, in predict
from azureml.evaluate.mlflow.hftransformers._task_based_predictors import get_predictor
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/azureml/evaluate/mlflow/hftransformers/_task_based_predictors.py", line 18, in
from transformers import pipeline, Conversation
ImportError: cannot import name 'Conversation' from 'transformers' (/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/transformers/init.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/azureml_inference_server_http/server/routes.py", line 222, in handle_score
timed_result = main_blueprint.user_script.invoke_run(request, timeout_ms=config.scoring_timeout)
File "/opt/miniconda/envs/azureml_env94d7f4560f6648c58f9156957403aeee/lib/python3.9/site-packages/azureml_inference_server_http/server/user_script.py", line 139, in invoke_run
raise UserScriptException(ex) from ex
azureml_inference_server_http.server.user_script.UserScriptException: Caught an unhandled exception from the user script

Addition information

No response