md/06.E2ESamples/translations/zh-cn/E2E_Phi-3-vision-image-text-to-text-online-endpoint.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用在线终结点进行(视觉)聊天补全推理\n",
    "\n",
    "本示例展示了如何将 `Phi-3-vision-128k-instruct` 部署到在线总结点进行推理。\n",
    "\n",
    "## 大纲\n",
    "* 设置先决条件\n",
    "* 选择要部署的模型\n",
    "* 下载并准备推理数据\n",
    "* 部署模型以实现实时推理\n",
    "* 测试终结点\n",
    "* 使用 Azure OpenAI 风格的负载测试终结点\n",
    "* 清理资源"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 设置先决条件\n",
    "* 安装依赖项\n",
    "* 连接到 AzureML 工作区。了解更多信息请参阅[设置 SDK 认证](https://learn.microsoft.com/azure/machine-learning/how-to-setup-authentication?tabs=sdk)。替换下文代码中的 `<WORKSPACE_NAME>`、`<RESOURCE_GROUP>` 和 `<SUBSCRIPTION_ID>`。\n",
    "* 连接到 `azureml` 系统注册表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# I导入所需的库\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import (\n",
    "    DefaultAzureCredential,\n",
    "    InteractiveBrowserCredential,\n",
    ")\n",
    "\n",
    "try:\n",
    "    # 尝试获得默认凭据\n",
    "    credential = DefaultAzureCredential()\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    # 如果默认凭据不可用，尝试使用交互式浏览器凭据\n",
    "    credential = InteractiveBrowserCredential()\n",
    "\n",
    "try:\n",
    "    # 尝试使用凭据创建MLClient实例\n",
    "    workspace_ml_client = MLClient.from_config(credential)\n",
    "    subscription_id = workspace_ml_client.subscription_id\n",
    "    resource_group = workspace_ml_client.resource_group_name\n",
    "    workspace_name = workspace_ml_client.workspace_name\n",
    "except Exception as ex:\n",
    "    print(ex)\n",
    "    # 如果创建MLClient实例失败，使用手动提供的信息\n",
    "    subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "    resource_group = \"<RESOURCE_GROUP>\"\n",
    "    workspace_name = \"<WORKSPACE_NAME>\"\n",
    "\n",
    "# 用之前提供的凭据和工作区信息创建MLClient实例\n",
    "workspace_ml_client = MLClient(\n",
    "    credential, subscription_id, resource_group, workspace_name\n",
    ")\n",
    "\n",
    "# 模型、微调管道和环境可在 AzureML 系统注册表 \"azureml\" 中找到\n",
    "registry_ml_client = MLClient(credential, registry_name=\"azureml\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 部署模型到在线终结点\n",
    "\n",
    "在线端点提供一个持久的REST API，可以用于与需要使用该模型的应用程序进行集成。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 这段代码检查注册表中是否存在指定名称的模型。\n",
    "# 如果模型存在，它会检索模型的第一个版本并打印其详细信息。\n",
    "# 如果模型不存在，它会打印一条消息，表明该模型未找到。\n",
    "\n",
    "# model_name: 要检查的注册表中的模型名称\n",
    "model_name = \"Phi-3-vision-128k-instruct\"\n",
    "\n",
    "# 获取指定型号名称的版本列表\n",
    "version_list = list(registry_ml_client.models.list(model_name))\n",
    "\n",
    "# 检查注册表中是否存在模型的任何版本\n",
    "if len(version_list) == 0:\n",
    "    print(\"Model not found in registry\")\n",
    "else:\n",
    "    # 获取模型的第一个版本\n",
    "    model_version = version_list[0].version\n",
    "    foundation_model = registry_ml_client.models.get(model_name, model_version)\n",
    "    \n",
    "    # 打印模型的详细信息\n",
    "    print(\n",
    "        \"\\n\\nUsing model name: {0}, version: {1}, id: {2} for inferencing\".format(\n",
    "            foundation_model.name, foundation_model.version, foundation_model.id\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 导入所需的库\n",
    "import time\n",
    "from azure.ai.ml.entities import ManagedOnlineEndpoint, ManagedOnlineDeployment\n",
    "\n",
    "# 端点名称在一个区域内必须是唯一的，因此使用时间戳来创建唯一的端点名称\n",
    "timestamp = int(time.time())\n",
    "online_endpoint_name = model_name[:13] + str(timestamp)\n",
    "print(f\"Creating online endpoint with name: {online_endpoint_name}\")\n",
    "\n",
    "# 创建在线终结点\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    description=f\"Online endpoint for {foundation_model.name}, for visual chat-completion task\",\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "workspace_ml_client.begin_create_or_update(endpoint).wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 此代码将部署在线终结点。\n",
    "# 它将设置部署名称、端点名称、模型、实例类型、实例数量和请求设置。\n",
    "# 还会设置有效性探针和就绪性探针。\n",
    "# 最后，更新端点的流量分布。\n",
    "\n",
    "from azure.ai.ml.entities import OnlineRequestSettings, ProbeSettings\n",
    "\n",
    "# create a deployment\n",
    "deployment_name = \"phi-3-vision\"\n",
    "demo_deployment = ManagedOnlineDeployment(\n",
    "    name=deployment_name,\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    model=foundation_model.id,\n",
    "    instance_type=\"Standard_NC48ads_A100_v4\",\n",
    "    instance_count=1,\n",
    "    request_settings=OnlineRequestSettings(\n",
    "        request_timeout_ms=180000,\n",
    "        max_queue_wait_ms=500,\n",
    "    ),\n",
    "    liveness_probe=ProbeSettings(\n",
    "        failure_threshold=49,\n",
    "        success_threshold=1,\n",
    "        timeout=299,\n",
    "        period=180,\n",
    "        initial_delay=180,\n",
    "    ),\n",
    "    readiness_probe=ProbeSettings(\n",
    "        failure_threshold=10,\n",
    "        success_threshold=1,\n",
    "        timeout=10,\n",
    "        period=10,\n",
    "        initial_delay=10,\n",
    "    ),\n",
    ")\n",
    "workspace_ml_client.online_deployments.begin_create_or_update(demo_deployment).wait()\n",
    "endpoint.traffic = {deployment_name: 100}\n",
    "workspace_ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 使用样本数据测试终结点\n",
    "\n",
    "我们将使用下面创建的 json 向模型发送一个示例请求。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 导入所需的库\n",
    "import json\n",
    "import os\n",
    "\n",
    "# 定义测试 JSON\n",
    "test_json = {\n",
    "    \"input_data\": {\n",
    "        \"input_string\": [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": [\n",
    "                    {\n",
    "                        \"type\": \"image_url\",\n",
    "                        \"image_url\": {\n",
    "                            \"url\": \"https://www.ilankelman.org/stopsigns/australia.jpg\"\n",
    "                        },\n",
    "                    },\n",
    "                    {\n",
    "                        \"type\": \"text\",\n",
    "                        \"text\": \"What is shown in this image? Be extremely detailed and specific.\",\n",
    "                    },\n",
    "                ],\n",
    "            },\n",
    "        ],\n",
    "        \"parameters\": {\"temperature\": 0.7, \"max_new_tokens\": 2048},\n",
    "    }\n",
    "}\n",
    "\n",
    "# 将 JSON 对象写入文件\n",
    "sample_score_file_path = os.path.join(\".\", \"sample_chat_completions_score.json\")\n",
    "with open(sample_score_file_path, \"w\") as f:\n",
    "    json.dump(test_json, f, indent=4)\n",
    "\n",
    "# 打印输入负载\n",
    "print(\"Input payload:\\n\")\n",
    "print(test_json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 导入所需的库\n",
    "import pandas as pd\n",
    "\n",
    "# 使用在线终结点和 azureml 终结点的 invoke 方法为 sample_chat_completions_score.json 文件评分\n",
    "response = workspace_ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    deployment_name=deployment_name,\n",
    "    request_file=sample_score_file_path,\n",
    ")\n",
    "print(\"Raw JSON Response: \\n\", response, \"\\n\")\n",
    "\n",
    "# Parse the JSON string\n",
    "json_data = json.loads(response)\n",
    "\n",
    "# Convert the parsed JSON to a DataFrame\n",
    "response_df = pd.DataFrame([json_data])\n",
    "print(\"Generated Text:\\n\", response_df[\"output\"].iloc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 用 Azure OpenAI 风格的负载测试终结点\n",
    "\n",
    "我们将向模型发送一条 Azure OpenAI 风格的示例请求。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 此代码定义了一个 JSON 载荷，用于使用 Azure OpenAI 风格的载荷测试在线端点。\n",
    "# 它包括模型名称、带有用户角色和内容（图片 URL 和文本）的消息列表、\n",
    "# 温度和最大生成令牌数。\n",
    "\n",
    "aoai_test_json = {\n",
    "    \"model\": foundation_model.name,\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": [\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": {\n",
    "                        \"url\": \"https://www.ilankelman.org/stopsigns/australia.jpg\"\n",
    "                    },\n",
    "                },\n",
    "                {\n",
    "                    \"type\": \"text\",\n",
    "                    \"text\": \"What is shown in this image? Be extremely detailed and specific.\",\n",
    "                },\n",
    "            ],\n",
    "        }\n",
    "    ],\n",
    "    \"temperature\": 0.7,\n",
    "    \"max_new_tokens\": 2048,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取得分 URL\n",
    "scoring_uri = workspace_ml_client.online_endpoints.get(\n",
    "    name=online_endpoint_name\n",
    ").scoring_uri\n",
    "# 将得分 URL 转换为 AOAI 格式\n",
    "aoai_format_scoring_uri = scoring_uri.replace(\"/score\", \"/v1/chat/completions\")\n",
    "\n",
    "# 获取数据平面操作的密钥\n",
    "data_plane_token = workspace_ml_client.online_endpoints.get_keys(\n",
    "    name=online_endpoint_name\n",
    ").primary_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "import json\n",
    "\n",
    "# 准备请求\n",
    "body = str.encode(json.dumps(aoai_test_json))\n",
    "url = aoai_format_scoring_uri\n",
    "api_key = data_plane_token\n",
    "\n",
    "headers = {\"Content-Type\": \"application/json\", \"Authorization\": (\"Bearer \" + api_key)}\n",
    "req = urllib.request.Request(url, body, headers)\n",
    "\n",
    "# 发生请求并获得响应\n",
    "try:\n",
    "    response = urllib.request.urlopen(req)\n",
    "    result = response.read().decode(\"utf-8\")\n",
    "    print(result)\n",
    "except urllib.error.HTTPError as error:\n",
    "    print(\"The request failed with status code: \" + str(error.code))\n",
    "    # 打印标头--其中包括请求 ID 和时间戳，用于调试\n",
    "    print(error.info())\n",
    "    print(error.read().decode(\"utf8\", \"ignore\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 删除在线终结点\n",
    "\n",
    "不要忘记删除在线终结点，否则该端点会一直产生费用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 删除工作空间\n",
    "workspace_ml_client.online_endpoints.begin_delete(name=online_endpoint_name).wait()"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}