python_sdk2/exercise02_prepare_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 02 : Prepare Data\n",
    "\n",
    "Here we uploda data as data asset in Azure Machine Learning.<br>\n",
    "The subsequent all exercises (Exercise 04 -) will use data provisioned in this exercise, and you should then run this exercise beforehand.\n",
    "\n",
    "Here we use hand-writing digit's dataset ([MNIST](http://yann.lecun.com/exdb/mnist/)) to train in this tutorial.\n",
    "\n",
    "*back to [index](https://github.com/tsmatz/azureml-tutorial/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize MLClient\n",
    "\n",
    "Before running code, you need to connect to your Azure ML workspace.<br>\n",
    "Replace below's branket's string with your subscription id, resource group name, and AML workspace name.\n",
    "\n",
    "I note that creating ```MLClient``` will not connect to AML workspace, and the client initialization is lazy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DeviceCodeCredential, TokenCachePersistenceOptions\n",
    "\n",
    "# When you run on remote\n",
    "cache_opt = TokenCachePersistenceOptions(allow_unencrypted_storage=True)\n",
    "cred = DeviceCodeCredential(cache_persistence_options=cache_opt)\n",
    "\n",
    "# # When you run on Azure ML Notebook\n",
    "# from azure.identity import DefaultAzureCredential\n",
    "# cred = DefaultAzureCredential()\n",
    "\n",
    "# Get a handle to the workspace\n",
    "ml_client = MLClient(\n",
    "    credential=cred,\n",
    "    subscription_id=\"{SUBSCRIPTION ID}\",\n",
    "    resource_group_name=\"{RESOURCE GROUP NAME}\",\n",
    "    workspace_name=\"{AML WORKSPACE NAME}\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download data in local folder\n",
    "\n",
    "Download MNIST (hand-writing digits) dataset in ```./data``` folder.<br>\n",
    "The generated train data has 60,000 records and test data has 10,000 records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-10-04 23:28:44.201455: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-10-04 23:28:44.417613: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2022-10-04 23:28:44.417654: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2022-10-04 23:28:44.461965: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2022-10-04 23:28:45.607578: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
      "2022-10-04 23:28:45.607809: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
      "2022-10-04 23:28:45.607831: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "2022-10-04 23:28:46.821697: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2022-10-04 23:28:46.821755: W tensorflow/stream_executor/cuda/cuda_driver.cc:263] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2022-10-04 23:28:46.821842: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (client1004): /proc/driver/nvidia/version does not exist\n",
      "2022-10-04 23:28:46.822272: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(train_images, train_labels),(test_images, test_labels) = mnist.load_data()\n",
    "\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))\n",
    "test_dataset = tf.data.Dataset.from_tensor_slices((test_images, test_labels))\n",
    "\n",
    "train_dataset.save(\"./data/train\")\n",
    "test_dataset.save(\"./data/test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload local files to AML default datastore\n",
    "\n",
    "Azure Machine Learning (AML) workspace has its own default datastore. When you create an AML workspace, a storage account (default datastore) is automatically generated in the same resource group.<br>\n",
    "Now we upload files in local ```data``` folder to Azure ML workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ARR7SANTZ to authenticate.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32mUploading data (58.66 MBs): 100%|███████████████████████████████| 58660145/58660145 [00:00<00:00, 64947296.86it/s]\u001b[0m\n",
      "\u001b[39m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from azure.ai.ml.entities import Data\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "\n",
    "mnist_data = Data(\n",
    "    name=\"mnist_data\",\n",
    "    path=\"data\",\n",
    "    type=AssetTypes.URI_FOLDER,\n",
    ")\n",
    "credit_data = ml_client.data.create_or_update(mnist_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show registered data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can extract the registered data with ```get()``` or ```list()``` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Data({'skip_validation': False, 'mltable_schema_url': None, 'referenced_uris': None, 'type': 'uri_folder', 'is_anonymous': False, 'auto_increment_version': False, 'name': 'mnist_data', 'description': None, 'tags': {}, 'properties': {}, 'id': '/subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/resourceGroups/AML-rg/providers/Microsoft.MachineLearningServices/workspaces/ws01/data/mnist_data/versions/1', 'Resource__source_path': None, 'base_path': '/home/tsmatsuz/python_sdk2', 'creation_context': <azure.ai.ml.entities._system_data.SystemData object at 0x7f48ea1d2b20>, 'serialize': <msrest.serialization.Serializer object at 0x7f48e8104ac0>, 'version': '1', 'latest_version': None, 'path': 'azureml://subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/resourcegroups/AML-rg/workspaces/ws01/datastores/workspaceblobstore/paths/LocalUpload/b6a4ea84008bfe761be7adab68271677/data/', 'datastore': None})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = ml_client.data.get(\"mnist_data\", version=1)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}