cli_yaml/exercise02_prepare_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 02 : Prepare Data\n",
    "\n",
    "Here we uploda data as data asset in Azure Machine Learning.<br>\n",
    "The subsequent all exercises (Exercise 04 -) will use data provisioned in this exercise, and you should then run this exercise beforehand.\n",
    "\n",
    "Here we use hand-writing digit's dataset ([MNIST](http://yann.lecun.com/exdb/mnist/)) to train in this tutorial.\n",
    "\n",
    "*back to [index](https://github.com/tsmatz/azureml-tutorial/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variable's Setting\n",
    "\n",
    "Replace below's branket's string and set the required variables.\n",
    "\n",
    "> Note : By the following ```az configure --defaults```, you can skip setting for ```--resource-group``` and ```--workspace-name``` options in each ```az ml``` command.<br>\n",
    "> ```az configure --defaults group=$resource_group workspace=$aml_workspace```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_resource_group = \"{AML-RESOURCE-GROUP-NAME}\"\n",
    "my_workspace = \"{AML-WORSPACE-NAME}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download data in local folder\n",
    "\n",
    "Download MNIST (hand-writing digits) dataset in ```./data``` folder.<br>\n",
    "The generated train data has 60,000 records and test data has 10,000 records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-10-04 00:19:51.356954: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-10-04 00:19:51.543509: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2022-10-04 00:19:51.543566: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2022-10-04 00:19:51.586882: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2022-10-04 00:19:52.630716: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory\n",
      "2022-10-04 00:19:52.630900: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory\n",
      "2022-10-04 00:19:52.630917: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n",
      "2022-10-04 00:19:53.943781: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2022-10-04 00:19:53.943853: W tensorflow/stream_executor/cuda/cuda_driver.cc:263] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2022-10-04 00:19:53.943909: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (client1004): /proc/driver/nvidia/version does not exist\n",
      "2022-10-04 00:19:53.944292: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(train_images, train_labels),(test_images, test_labels) = mnist.load_data()\n",
    "\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))\n",
    "test_dataset = tf.data.Dataset.from_tensor_slices((test_images, test_labels))\n",
    "\n",
    "train_dataset.save(\"./data/train\")\n",
    "test_dataset.save(\"./data/test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload local files to AML default datastore\n",
    "\n",
    "Azure Machine Learning (AML) workspace has its own default datastore. When you create an AML workspace, a storage account (default datastore) is automatically generated in the same resource group.<br>\n",
    "Now, we create yaml and upload files (in ```data``` folder) to AML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we create yaml for data asset registration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing 02_file_upload.yml\n"
     ]
    }
   ],
   "source": [
    "%%writefile 02_file_upload.yml\n",
    "$schema: https://azuremlschemas.azureedge.net/latest/data.schema.json\n",
    "name: mnist_data\n",
    "description: This is example.\n",
    "type: uri_folder\n",
    "path: data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we register data asset (upload local data) with AML CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mUploading data (58.66 MBs): 100%|█| 58660143/58660143 [00:00<00:00, 69704595.74i\u001b[0m\n",
      "\u001b[39m\n",
      "\n",
      "{\n",
      "  \"creation_context\": {\n",
      "    \"created_at\": \"2022-10-04T00:20:15.859121+00:00\",\n",
      "    \"created_by\": \"Tsuyoshi Matsuzaki\",\n",
      "    \"created_by_type\": \"User\",\n",
      "    \"last_modified_at\": \"2022-10-04T00:20:15.868007+00:00\"\n",
      "  },\n",
      "  \"description\": \"This is example.\",\n",
      "  \"id\": \"/subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/resourceGroups/AML-rg/providers/Microsoft.MachineLearningServices/workspaces/ws01/data/mnist_data/versions/1\",\n",
      "  \"name\": \"mnist_data\",\n",
      "  \"path\": \"azureml://subscriptions/b3ae1c15-4fef-4362-8c3a-5d804cdeb18d/resourcegroups/AML-rg/workspaces/ws01/datastores/workspaceblobstore/paths/LocalUpload/92ee0eb89708f2423c1e3240cb34b8b3/data/\",\n",
      "  \"resourceGroup\": \"AML-rg\",\n",
      "  \"tags\": {},\n",
      "  \"type\": \"uri_folder\",\n",
      "  \"version\": \"1\"\n",
      "}\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!az ml data create --file 02_file_upload.yml \\\n",
    "  --resource-group $my_resource_group \\\n",
    "  --workspace-name $my_workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}