python_sdk2/exercise08_publish_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise08 : Publish as a Web Service\n",
    "\n",
    "Finally we publish our model as a web service.\n",
    "\n",
    "Before running this code, **complete the model registration in \"[Exercise04 : Train on Remote GPU Virtual Machine](./exercise04_train_remote.ipynb)\"**.\n",
    "\n",
    "*back to [index](https://github.com/tsmatz/azureml-tutorial/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize MLClient\n",
    "\n",
    "Replace below's branket's string with your subscription id, resource group name, and AML workspace name.<br>\n",
    "(I note that creating ```MLClient``` will not connect to AML workspace, and the client initialization is lazy.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DeviceCodeCredential, TokenCachePersistenceOptions\n",
    "\n",
    "# When you run on remote\n",
    "cache_opt = TokenCachePersistenceOptions(allow_unencrypted_storage=True)\n",
    "cred = DeviceCodeCredential(cache_persistence_options=cache_opt)\n",
    "\n",
    "# # When you run on Azure ML Notebook\n",
    "# from azure.identity import DefaultAzureCredential\n",
    "# cred = DefaultAzureCredential()\n",
    "\n",
    "# Get a handle to the workspace\n",
    "ml_client = MLClient(\n",
    "    credential=cred,\n",
    "    subscription_id=\"{SUBSCRIPTION ID}\",\n",
    "    resource_group_name=\"{RESOURCE GROUP NAME}\",\n",
    "    workspace_name=\"{AML WORKSPACE NAME}\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create entry script (.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to deploy as web service, first we generate the following scoring code.<br>\n",
    "This entry script in AML should include both ```init()``` and ```run()```.\n",
    "\n",
    "> Note : The serving compute (VM) provides [managed identity endpoint](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-use-vm-token). (Your script can use both system-assigned identity and user-assigned identity.) Your script can then get the access permissions for Azure resources without providing secure information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "script_folder = './inference_script'\n",
    "os.makedirs(script_folder, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing inference_script/score.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile inference_script/score.py\n",
    "import os\n",
    "import json\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "def init():\n",
    "    global loaded_model\n",
    "    ## model_path = azureml.core.Model.get_model_path(model_name='mnist_model_test')\n",
    "    model_path = os.path.join(\n",
    "        os.getenv(\"AZUREML_MODEL_DIR\"), \"mnist_tf_model\"\n",
    "    )\n",
    "    loaded_model = tf.keras.models.load_model(model_path)\n",
    "\n",
    "def run(raw_data):\n",
    "    try:\n",
    "        data = json.loads(raw_data)[\"data\"]\n",
    "        pred_output = loaded_model(np.array(data))\n",
    "        pred_list = tf.math.argmax(pred_output, axis=-1).numpy().tolist()\n",
    "        return pred_list\n",
    "    except Exception as e:\n",
    "       result = str(e)\n",
    "       return 'Internal Exception : ' + result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a managed endpoint\n",
    "\n",
    "There exist **endpoint** and **deployment** in deployment topology in managed online endpoint.<br>\n",
    "You can run multiple deployments in a single endpoint, and allocate appropriate traffic for these multiple deployments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, create a managed endpoint for deployment target.<br>\n",
    "I note that **```name``` should be unique and then specify arbitrary unique name**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name = \"{UNIQUE_ENDPOINT_NAME}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace the following ```UNIQUE_ENDPOINT_NAME```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AFS9GTWSZ to authenticate.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "ManagedOnlineEndpoint({'public_network_access': 'Enabled', 'provisioning_state': 'Succeeded', 'scoring_uri': 'https://my-mnist-test123.eastus.inference.ml.azure.com/score', 'swagger_uri': 'https://my-mnist-test123.eastus.inference.ml.azure.com/swagger.json', 'name': 'my-mnist-test123', 'description': None, 'tags': {}, 'properties': {'azureml.onlineendpointid': '/subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/resourcegroups/aml-rg/providers/microsoft.machinelearningservices/workspaces/ws01/onlineendpoints/my-mnist-test123', 'AzureAsyncOperationUri': 'https://management.azure.com/subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/providers/Microsoft.MachineLearningServices/locations/eastus/mfeOperationsStatus/oe:030b8dcf-1ffc-4245-9b2d-faabb1c4e31a:4c422361-d6f7-4ef8-8d19-b9025468e4dc?api-version=2022-02-01-preview'}, 'id': '/subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/resourceGroups/AML-rg/providers/Microsoft.MachineLearningServices/workspaces/ws01/onlineEndpoints/my-mnist-test123', 'Resource__source_path': None, 'base_path': '/home/tsmatsuz/python_sdk2', 'creation_context': None, 'serialize': <msrest.serialization.Serializer object at 0x7fa9b76c4eb0>, 'auth_mode': 'key', 'location': 'eastus', 'identity': <azure.ai.ml._restclient.v2022_02_01_preview.models._models_py3.ManagedServiceIdentity object at 0x7fa9b76c49d0>, 'traffic': {}, 'mirror_traffic': {}, 'kind': 'Managed'})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from azure.ai.ml.entities import ManagedOnlineEndpoint\n",
    "\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=endpoint_name,\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "ml_client.begin_create_or_update(endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go to [AML Studio UI](https://ml.azure.com/), click \"Endpoints\", and select the above endpoint.<br>\n",
    "Please wait until the provisioning state is succeeded.\n",
    "\n",
    "![Endpoint status](https://tsmatz.github.io/images/github/azure-ml-tensorflow-complete-sample/20221220_Endpoint_Status.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy as web service\n",
    "\n",
    "Next we deploy the serving code (```score.py```) as a web service in the previous endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before deployment, create conda configuration for serving environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing 08_conda_env.yml\n"
     ]
    }
   ],
   "source": [
    "%%writefile 08_conda_env.yml\n",
    "name: serving_example\n",
    "dependencies:\n",
    "- python=3.8\n",
    "- pip:\n",
    "  - azureml-defaults\n",
    "  - tensorflow==2.10.0\n",
    "  - numpy\n",
    "channels:\n",
    "- anaconda\n",
    "- conda-forge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we deploy a web service with yaml configuration for deployment.\n",
    "\n",
    "When you change the model (or code) in managed endpoint, you can submit multiple deployments and transfer the traffic allocation without causing any disruption.<br>\n",
    "With the following ```--all-traffic``` option, all traffic (100% traffic) will be allocated to this single deployment.\n",
    "\n",
    "In this example, I use the trained model in Exercise04, and **run \"[Exercise04 : Train on Remote GPU Virtual Machine](./exercise04_train_remote.ipynb)\", before running this code.**\n",
    "\n",
    "Replace the following ```UNIQUE_ENDPOINT_NAME```.\n",
    "\n",
    "> Note : You can scale computes by increasing the following ```instance_count```. (You can also define auto-scale settings.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Check: endpoint my-mnist-test123 exists\n",
      "\u001b[32mUploading inference_script (0.0 MBs): 100%|█████████████████████████████████| 659/659 [00:00<00:00, 143833.39it/s]\u001b[0m\n",
      "\u001b[39m\n",
      "\n",
      "data_collector is not a known attribute of class <class 'azure.ai.ml._restclient.v2022_02_01_preview.models._models_py3.ManagedOnlineDeployment'> and will be ignored\n",
      "Creating/updating online deployment my-mnist-deployment-v1 "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".........................................................................................................................................................................."
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Done (15m 18s)\n"
     ]
    }
   ],
   "source": [
    "from azure.ai.ml.entities import ManagedOnlineDeployment, Environment, CodeConfiguration\n",
    "\n",
    "model = ml_client.models.get(\"mnist_model_test\", version=1)\n",
    "\n",
    "deployment = ManagedOnlineDeployment(\n",
    "    name=\"my-mnist-deployment-v1\",\n",
    "    endpoint_name=endpoint_name,\n",
    "    model=model,\n",
    "    environment=Environment(\n",
    "        conda_file=\"08_conda_env.yml\",\n",
    "        image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04\",\n",
    "    ),\n",
    "    code_configuration=CodeConfiguration(\n",
    "        code=\"./inference_script\", scoring_script=\"score.py\"\n",
    "    ),\n",
    "    instance_type=\"Standard_DS2_v2\",\n",
    "    instance_count=1,\n",
    ")\n",
    "\n",
    "ml_client.begin_create_or_update(deployment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please wait until the deployment state is succeeded.\n",
    "\n",
    "![Deployment status](https://tsmatz.github.io/images/github/azure-ml-tensorflow-complete-sample/20221220_Deployment_Status.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assign all traffic (100%) to this deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ManagedOnlineEndpoint({'public_network_access': 'Enabled', 'provisioning_state': 'Succeeded', 'scoring_uri': 'https://my-mnist-test123.eastus.inference.ml.azure.com/score', 'swagger_uri': 'https://my-mnist-test123.eastus.inference.ml.azure.com/swagger.json', 'name': 'my-mnist-test123', 'description': None, 'tags': {}, 'properties': {'azureml.onlineendpointid': '/subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/resourcegroups/aml-rg/providers/microsoft.machinelearningservices/workspaces/ws01/onlineendpoints/my-mnist-test123', 'AzureAsyncOperationUri': 'https://management.azure.com/subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/providers/Microsoft.MachineLearningServices/locations/eastus/mfeOperationsStatus/oe:030b8dcf-1ffc-4245-9b2d-faabb1c4e31a:c00fd922-c45a-424c-8c24-87a7ce87f072?api-version=2022-02-01-preview'}, 'id': '/subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/resourceGroups/AML-rg/providers/Microsoft.MachineLearningServices/workspaces/ws01/onlineEndpoints/my-mnist-test123', 'Resource__source_path': None, 'base_path': '/home/tsmatsuz/python_sdk2', 'creation_context': None, 'serialize': <msrest.serialization.Serializer object at 0x7fa9b7687e80>, 'auth_mode': 'key', 'location': 'eastus', 'identity': <azure.ai.ml._restclient.v2022_02_01_preview.models._models_py3.ManagedServiceIdentity object at 0x7fa9b7687520>, 'traffic': {'my-mnist-deployment-v1': 100}, 'mirror_traffic': {}, 'kind': 'Managed'})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "endpoint.traffic = {\"my-mnist-deployment-v1\": 100}\n",
    "ml_client.begin_create_or_update(endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please wait until the traffic becomes 100%.\n",
    "\n",
    "![Deployment traffic](https://tsmatz.github.io/images/github/azure-ml-tensorflow-complete-sample/20221220_Deployment_Traffic.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For debugging purpose, you can also submit your deployment on local docker runtime. (See [here](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-managed-online-endpoints).)<br>\n",
    "With Visual Studio Code, you can also attach debugger on local deployment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test your web service\n",
    "\n",
    "Let's invoke your deployed web service and check the returned results in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get endpoint URI (address) as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://my-mnist-test123.eastus.inference.ml.azure.com/score'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml_client.online_endpoints.get(endpoint_name).scoring_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also extract key credential in your endpoint as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'IQlK8NXVaqyjPfqdIgNf5tLtmjQcJ6tm'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml_client.online_endpoints.get_keys(endpoint_name).primary_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's invoke scoring web service in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-10-05 03:09:28.146497: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-10-05 03:09:28.337236: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2022-10-05 03:09:28.337283: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2022-10-05 03:09:28.377910: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2022-10-05 03:09:29.517958: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
      "2022-10-05 03:09:29.518100: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
      "2022-10-05 03:09:29.518117: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "2022-10-05 03:09:31.203051: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2022-10-05 03:09:31.203105: W tensorflow/stream_executor/cuda/cuda_driver.cc:263] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2022-10-05 03:09:31.203169: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (client1004): /proc/driver/nvidia/version does not exist\n",
      "2022-10-05 03:09:31.203440: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted :  [7, 2, 1]\n",
      "Actual    :  [7, 2, 1]\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "SERVING_URI = ml_client.online_endpoints.get(endpoint_name).scoring_uri\n",
    "API_KEY = ml_client.online_endpoints.get_keys(endpoint_name).primary_key\n",
    "\n",
    "# Read data by tensor\n",
    "test_data = tf.data.Dataset.load(\"./data/test\")\n",
    "\n",
    "# Generate data\n",
    "image_arr = []\n",
    "label_arr = []\n",
    "for image, label in test_data.take(3):\n",
    "    image_arr.append(image.numpy().tolist())\n",
    "    label_arr.append(label.numpy().item())\n",
    "\n",
    "# Invoke web service !\n",
    "headers = {\n",
    "    'Content-Type':'application/json',\n",
    "    'Authorization':('Bearer '+ API_KEY)\n",
    "} \n",
    "values = json.dumps(image_arr)\n",
    "input_data = \"{\\\"data\\\": \" + values + \"}\"\n",
    "http_res = requests.post(\n",
    "    SERVING_URI,\n",
    "    input_data,\n",
    "    headers = headers)\n",
    "print('Predicted : ', http_res.text)\n",
    "print('Actual    : ', label_arr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also invoke web service by using AML Python SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"sample-request.json\", 'w') as f:\n",
    "    f.write(input_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[7, 2, 1]'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=endpoint_name,\n",
    "    deployment_name=\"my-mnist-deployment-v1\",\n",
    "    request_file=\"./sample-request.json\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Deleting endpoint my-mnist-test123 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".........................................................................................................."
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Done (9m 5s)\n"
     ]
    }
   ],
   "source": [
    "ml_client.online_endpoints.begin_delete(name=endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}