notebooks/scikit-learn/HyperparameterTuningWithScikitLearnInCMLE.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Copyright 2018 Google LLC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# scikit-learn HP Tuning on AI Platform\n",
    "This notebook trains a model on Ai Platform using Hyperparameter Tuning to predict a car's Miles Per Gallon. It uses Auto MPG Data Set from [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Auto+MPG).\n",
    "\n",
    "Citation: Dua, D. and Karra Taniskidou, E. (2017). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# How to train your model on AI Platform with HP tuning.\n",
    "Using HP Tuning for training can be done in a few steps:\n",
    "1. Create your python model file\n",
    "    1. Add argument parsing for the hyperparameter values. (These values are chosen for you in this notebook)\n",
    "    1. Add code to download your data from [Google Cloud Storage](https://cloud.google.com/storage) so that AI Platform can use it\n",
    "    1. Add code to track the performance of your hyperparameter values.\n",
    "    1. Add code to export and save the model to [Google Cloud Storage](https://cloud.google.com/storage) once AI Platform finishes training the model\n",
    "1. Prepare a package\n",
    "1. Submit the training job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Prerequisites\n",
    "Before you jump in, let’s cover some of the different tools you’ll be using to get HP tuning up and running on AI Platform. \n",
    "\n",
    "[Google Cloud Platform](https://cloud.google.com/) lets you build and host applications and websites, store data, and analyze data on Google's scalable infrastructure.\n",
    "\n",
    "[AI Platform](https://cloud.google.com/ml-engine/) is a managed service that enables you to easily build machine learning models that work on any type of data, of any size.\n",
    "\n",
    "[Google Cloud Storage](https://cloud.google.com/storage/) (GCS) is a unified object storage for developers and enterprises, from live data serving to data analytics/ML to data archiving.\n",
    "\n",
    "[Cloud SDK](https://cloud.google.com/sdk/) is a command line tool which allows you to interact with Google Cloud products. In order to run this notebook, make sure that Cloud SDK is [installed](https://cloud.google.com/sdk/downloads) in the same environment as your Jupyter kernel.\n",
    "\n",
    "[Overview of Hyperparameter Tuning](https://cloud.google.com/ml-engine/docs/tensorflow/hyperparameter-tuning-overview) - Hyperparameter tuning takes advantage of the processing infrastructure of Google Cloud Platform to test different hyperparameter configurations when training your model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Part 0: Setup\n",
    "* [Create a project on GCP](https://cloud.google.com/resource-manager/docs/creating-managing-projects)\n",
    "* [Create a Google Cloud Storage Bucket](https://cloud.google.com/storage/docs/quickstart-console)\n",
    "* [Enable AI Platform Training and Prediction and Compute Engine APIs](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component&_ga=2.217405014.1312742076.1516128282-1417583630.1516128282)\n",
    "* [Install Cloud SDK](https://cloud.google.com/sdk/downloads)\n",
    "* [Install scikit-learn](http://scikit-learn.org/stable/install.html) [Optional: used if running locally]\n",
    "* [Install pandas](https://pandas.pydata.org/pandas-docs/stable/install.html) [Optional: used if running locally]\n",
    "* [Install cloudml-hypertune](https://pypi.org/project/cloudml-hypertune/) [Optional: used if running locally]\n",
    "\n",
    "These variables will be needed for the following steps.\n",
    "* `TRAINER_PACKAGE_PATH <./auto_mpg_hp_tuning>` - A packaged training application that will be staged in a Google Cloud Storage location. The model file created below is placed inside this package path.\n",
    "* `MAIN_TRAINER_MODULE <auto_mpg_hp_tuning.train>` - Tells AI Platform which file to execute. This is formatted as follows <folder_name.python_file_name>\n",
    "* `JOB_DIR <gs://$BUCKET_ID/scikit_learn_job_dir>` - The path to a Google Cloud Storage location to use for job output.\n",
    "* `RUNTIME_VERSION <1.9>` - The version of AI Platform to use for the job. If you don't specify a runtime version, the training service uses the default AI Platform runtime version 1.0. [See the list of runtime versions for more information](https://cloud.google.com/ml-engine/docs/tensorflow/runtime-version-list).\n",
    "* `PYTHON_VERSION <3.5>` - The Python version to use for the job. Python 3.5 is available with runtime version 1.4 or greater. If you don't specify a Python version, the training service uses Python 2.7.\n",
    "* `HPTUNING_CONFIG <hptuning_config.yaml>` - Path to the job configuration file.\n",
    "\n",
    "** Replace: **\n",
    "* `PROJECT_ID <YOUR_PROJECT_ID>` - with your project's id. Use the PROJECT_ID that matches your Google Cloud Platform project.\n",
    "* `BUCKET_ID <YOUR_BUCKET_ID>` - with the bucket id you created above.\n",
    "* `JOB_DIR <gs://YOUR_BUCKET_ID/scikit_learn_job_dir>` - with the bucket id you created above.\n",
    "* `REGION <REGION>` - select a region from [here](https://cloud.google.com/ml-engine/docs/regions) or use the default '`us-central1`'. The region is where the model will be deployed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%env PROJECT_ID PROJECT_ID\n",
    "%env BUCKET_ID BUCKET_ID\n",
    "%env JOB_DIR gs://BUCKET_ID/scikit_learn_job_dir\n",
    "%env REGION us-central1\n",
    "\n",
    "%env TRAINER_PACKAGE_PATH ./auto_mpg_hp_tuning\n",
    "%env MAIN_TRAINER_MODULE auto_mpg_hp_tuning.train\n",
    "%env RUNTIME_VERSION 1.9\n",
    "%env PYTHON_VERSION 3.5\n",
    "%env HPTUNING_CONFIG hptuning_config.yaml\n",
    "! mkdir auto_mpg_hp_tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "##  The data\n",
    "The [Auto MPG Data Set](https://archive.ics.uci.edu/ml/datasets/Auto+MPG) that this sample\n",
    "uses for training is provided by the [UC Irvine Machine Learning\n",
    "Repository](https://archive.ics.uci.edu/ml/datasets/). We have hosted the data on a public GCS bucket `gs://cloud-samples-data/ml-engine/auto_mpg/`. The data has been pre-processed to remove rows with incomplete data so as not to create additional steps for this notebook.\n",
    "\n",
    " * Training file is `auto-mpg.data`\n",
    "\n",
    "Note: Your typical development process with your own data would require you to upload your data to GCS so that AI Platform can access that data. However, in this case, we have put the data on GCS to avoid the steps of having you download the data from UC Irvine and then upload the data to GCS.\n",
    "\n",
    "Citation: Dua, D. and Karra Taniskidou, E. (2017). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Disclaimer\n",
    "This dataset is provided by a third party. Google provides no representation,\n",
    "warranty, or other guarantees about the validity or any other aspects of this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Part 1: Create your python model file\n",
    "\n",
    "First, we'll create the python model file (provided below) that we'll upload to AI Platform. This is similar to your normal process for creating a scikit-learn model. However, there are a few key differences:\n",
    "1. Downloading the data from GCS at the start of your file, so that AI Platform can access the data.\n",
    "1. Exporting/saving the model to GCS at the end of your file, so that you can use it for predictions.\n",
    "1. Define a command-line argument in your main training module for each tuned hyperparameter.\n",
    "1. Use the value passed in those arguments to set the corresponding hyperparameter in your application's scikit-learn code.\n",
    "1. Use `cloudml-hypertune` to track your training jobs metrics.\n",
    "\n",
    "The code in this file first handles the hyperparameters passed to the file from AI Platform. Then it loads the data into a pandas DataFrame that can be used by scikit-learn. Then the model is fit against the training data and the metrics for that data are shared with AI Platform. Lastly, sklearn's built in version of joblib is used to save the model to a file that can be uploaded to [AI Platform's prediction service](https://cloud.google.com/ml-engine/docs/scikit/getting-predictions#deploy_models_and_versions).\n",
    "\n",
    "Note: In normal practice you would want to test your model locally on a small dataset to ensure that it works, before using it with your larger dataset on AI Platform. This avoids wasted time and costs.\n",
    "\n",
    "### Setup the imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%writefile ./auto_mpg_hp_tuning/train.py\n",
    "#!/usr/bin/env python\n",
    " # Copyright 2018 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "\n",
    "import argparse\n",
    "import datetime\n",
    "import os\n",
    "import pandas as pd\n",
    "import subprocess\n",
    "\n",
    "from google.cloud import storage\n",
    "import hypertune\n",
    "\n",
    "from sklearn.externals import joblib\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.model_selection import train_test_split\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Load the hyperparameter values that are passed to the model during training.\n",
    "\n",
    "In this tutorial, the Lasso regressor is used, because it has several parameters that can be used to help demonstrate how to choose HP tuning values. (The range of values are set below in the configuration file for the HP tuning values.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%writefile -a ./auto_mpg_hp_tuning/train.py\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument(\n",
    "    '--job-dir',  # handled automatically by AI Platform\n",
    "    help='GCS location to write checkpoints and export models',\n",
    "    required=True\n",
    ")\n",
    "parser.add_argument(\n",
    "    '--alpha',  # Specified in the config file\n",
    "    help='Constant that multiplies the L1 term.',\n",
    "    default=1.0,\n",
    "    type=float\n",
    ")\n",
    "parser.add_argument(\n",
    "    '--max_iter',  # Specified in the config file\n",
    "    help='The maximum number of iterations.',\n",
    "    default=1000,\n",
    "    type=int\n",
    ")\n",
    "parser.add_argument(\n",
    "    '--tol',  # Specified in the config file\n",
    "    help='The tolerance for the optimization: if the updates are smaller than tol, '\n",
    "         'the optimization code checks the dual gap for optimality and continues '\n",
    "         'until it is smaller than tol.',\n",
    "    default=0.0001,\n",
    "    type=float\n",
    ")\n",
    "parser.add_argument(\n",
    "    '--selection',  # Specified in the config file\n",
    "    help='Supported criteria are “cyclic” loop over features sequentially and '\n",
    "         '“random” a random coefficient is updated every iteration ',\n",
    "    default='cyclic'\n",
    ")\n",
    "\n",
    "args = parser.parse_args()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Add code to download the data from GCS\n",
    "In this case, using the publicly hosted data,AI Platform will then be able to use the data when training your model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%writefile -a ./auto_mpg_hp_tuning/train.py\n",
    "# Public bucket holding the auto mpg data\n",
    "bucket = storage.Client().bucket('cloud-samples-data')\n",
    "# Path to the data inside the public bucket\n",
    "blob = bucket.blob('ml-engine/auto_mpg/auto-mpg.data')\n",
    "# Download the data\n",
    "blob.download_to_filename('auto-mpg.data')\n",
    "\n",
    "\n",
    "# ---------------------------------------\n",
    "# This is where your model code would go. Below is an example model using the auto mpg dataset.\n",
    "# ---------------------------------------\n",
    "# Define the format of your input data including unused columns\n",
    "# (These are the columns from the auto-mpg data files)\n",
    "COLUMNS = (\n",
    "    'mpg',\n",
    "    'cylinders',\n",
    "    'displacement',\n",
    "    'horsepower',\n",
    "    'weight',\n",
    "    'acceleration',\n",
    "    'model-year',\n",
    "    'origin',\n",
    "    'car-name'\n",
    ")\n",
    "\n",
    "# Load the training auto mpg dataset\n",
    "with open('./auto-mpg.data', 'r') as train_data:\n",
    "    raw_training_data = pd.read_csv(train_data, header=None, names=COLUMNS, delim_whitespace=True)\n",
    "\n",
    "# Remove the column we are trying to predict ('mpg') from our features list\n",
    "# Convert the Dataframe to a lists of lists\n",
    "features = raw_training_data.drop('mpg', axis=1).drop('car-name', axis=1).values.tolist()\n",
    "\n",
    "# Create our training labels list, convert the Dataframe to a lists of lists\n",
    "labels = raw_training_data['mpg'].values.tolist()\n",
    "\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size=0.15)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Use the Hyperparameters\n",
    "Use the Hyperparameter values passed in those arguments to set the corresponding hyperparameters in your application's scikit-learn code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%writefile -a ./auto_mpg_hp_tuning/train.py\n",
    "# Create the regressor, here we will use a Lasso Regressor to demonstrate the use of HP Tuning.\n",
    "# Here is where we set the variables used during HP Tuning from\n",
    "# the parameters passed into the python script\n",
    "regressor = Lasso(\n",
    "    alpha=args.alpha,\n",
    "    max_iter=args.max_iter,\n",
    "    tol=args.tol,\n",
    "    selection=args.selection)\n",
    "\n",
    "# Transform the features and fit them to the regressor\n",
    "regressor.fit(train_features, train_labels)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Report the mean accuracy as hyperparameter tuning objective metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%writefile -a ./auto_mpg_hp_tuning/train.py\n",
    "# Calculate the mean accuracy on the given test data and labels.\n",
    "score = regressor.score(test_features, test_labels)\n",
    "\n",
    "# The default name of the metric is training/hptuning/metric. \n",
    "# We recommend that you assign a custom name. The only functional difference is that \n",
    "# if you use a custom name, you must set the hyperparameterMetricTag value in the \n",
    "# HyperparameterSpec object in your job request to match your chosen name.\n",
    "# https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs#HyperparameterSpec\n",
    "hpt = hypertune.HyperTune()\n",
    "hpt.report_hyperparameter_tuning_metric(\n",
    "    hyperparameter_metric_tag='my_metric_tag',\n",
    "    metric_value=score,\n",
    "    global_step=1000)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Export and save the model to GCS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%writefile -a ./auto_mpg_hp_tuning/train.py\n",
    "# Export the model to a file\n",
    "model_filename = 'model.joblib'\n",
    "joblib.dump(regressor, model_filename)\n",
    "\n",
    "# Example: job_dir = 'gs://BUCKET_ID/scikit_learn_job_dir/1'\n",
    "job_dir =  args.job_dir.replace('gs://', '')  # Remove the 'gs://'\n",
    "# Get the Bucket Id\n",
    "bucket_id = job_dir.split('/')[0]\n",
    "# Get the path\n",
    "bucket_path = job_dir.lstrip('{}/'.format(bucket_id))  # Example: 'scikit_learn_job_dir/1'\n",
    "\n",
    "# Upload the model to GCS\n",
    "bucket = storage.Client().bucket(bucket_id)\n",
    "blob = bucket.blob('{}/{}'.format(\n",
    "    bucket_path,\n",
    "    model_filename))\n",
    "blob.upload_from_filename(model_filename)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Part 2: Create Trainer Package with Hyperparameter Tuning\n",
    "Next we need to build the Trainer Package, which holds all your code and dependencies need to train your model on AI Platform. \n",
    "\n",
    "First, we create an empty `__init__.py` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%writefile ./auto_mpg_hp_tuning/__init__.py\n",
    "#!/usr/bin/env python\n",
    "\n",
    "# Copyright 2018 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "\n",
    "# Note that __init__.py can be an empty file.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Next, we need to set the hp tuning values used to train our model. Check [HyperparameterSpec](https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs#HyperparameterSpec) for more info. \n",
    "\n",
    "In this config file several key things are set:\n",
    "* `maxTrials` - How many training trials should be attempted to optimize the specified hyperparameters.\n",
    "* `maxParallelTrials: 5` - The number of training trials to run concurrently. \n",
    "* `params` - The set of parameters to tune.. These are the different parameters to pass into your model and the specified ranges you wish to try.\n",
    " * `parameterName` - The parameter name must be unique amongst all ParameterConfigs\n",
    " * `type` - The type of the parameter. [INTEGER, DOUBLE, ...]\n",
    " * `minValue` & `maxValue` - The range of values that this parameter could be. \n",
    " * `scaleType` - How the parameter should be scaled to the hypercube. Leave unset for categorical parameters. Some kind of scaling is strongly recommended for real or integral parameters (e.g., UNIT_LINEAR_SCALE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%writefile ./hptuning_config.yaml\n",
    "#!/usr/bin/env python\n",
    "\n",
    "# Copyright 2018 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "\n",
    "# hyperparam.yaml\n",
    "trainingInput:\n",
    "  hyperparameters:\n",
    "    goal: MAXIMIZE\n",
    "    maxTrials: 30\n",
    "    maxParallelTrials: 5\n",
    "    hyperparameterMetricTag: my_metric_tag\n",
    "    enableTrialEarlyStopping: TRUE \n",
    "    params:\n",
    "    - parameterName: alpha\n",
    "      type: DOUBLE\n",
    "      minValue: 0.0\n",
    "      maxValue: 10.0\n",
    "      scaleType: UNIT_LINEAR_SCALE\n",
    "    - parameterName: max_iter\n",
    "      type: INTEGER\n",
    "      minValue: 1000\n",
    "      maxValue: 5000\n",
    "      scaleType: UNIT_LINEAR_SCALE   \n",
    "    - parameterName: tol\n",
    "      type: DOUBLE\n",
    "      minValue: 0.0001\n",
    "      maxValue: 0.1\n",
    "      scaleType: UNIT_LINEAR_SCALE\n",
    "    - parameterName: selection\n",
    "      type: CATEGORICAL\n",
    "      categoricalValues: [\n",
    "          \"cyclic\",\n",
    "          \"random\"\n",
    "      ]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Lastly, we need to install the dependencies used in our model. Check [adding_standard_pypi_dependencies](https://cloud.google.com/ml-engine/docs/tensorflow/packaging-trainer#adding_standard_pypi_dependencies) for more info.\n",
    "\n",
    "To do this, AI Platform uses a setup.py file to install your dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "%%writefile ./setup.py\n",
    "#!/usr/bin/env python\n",
    "\n",
    "# Copyright 2018 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "\n",
    "from setuptools import find_packages\n",
    "from setuptools import setup\n",
    "\n",
    "REQUIRED_PACKAGES = ['cloudml-hypertune']\n",
    "\n",
    "setup(\n",
    "    name='auto_mpg_hp_tuning',\n",
    "    version='0.1',\n",
    "    install_requires=REQUIRED_PACKAGES,\n",
    "    packages=find_packages(),\n",
    "    include_package_data=True,\n",
    "    description='Auto MPG sklearn HP tuning training application'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Part 3: Submit Training Job\n",
    "Next we need to submit the job for training on AI Platform. We'll use gcloud to submit the job which has the following flags:\n",
    "\n",
    "* `job-name` - A name to use for the job (mixed-case letters, numbers, and underscores only, starting with a letter). In this case: `auto_mpg_hp_tuning_$(date +\"%Y%m%d_%H%M%S\")`\n",
    "* `job-dir` - The path to a Google Cloud Storage location to use for job output.\n",
    "* `package-path` - A packaged training application that is staged in a Google Cloud Storage location. If you are using the gcloud command-line tool, this step is largely automated.\n",
    "* `module-name` - The name of the main module in your trainer package. The main module is the Python file you call to start the application. If you use the gcloud command to submit your job, specify the main module name in the --module-name argument. Refer to Python Packages to figure out the module name.\n",
    "* `region` - The Google Cloud Compute region where you want your job to run. You should run your training job in the same region as the Cloud Storage bucket that stores your training data. Select a region from [here](https://cloud.google.com/ml-engine/docs/regions) or use the default '`us-central1`'.\n",
    "* `runtime-version` - The version of AI Platform to use for the job. If you don't specify a runtime version, the training service uses the default AI Platform runtime version 1.0. See the list of runtime versions for more information.\n",
    "* `python-version` - The Python version to use for the job. Python 3.5 is available with runtime version 1.4 or greater. If you don't specify a Python version, the training service uses Python 2.7.\n",
    "* `scale-tier` - A scale tier specifying the type of processing cluster to run your job on. This can be the CUSTOM scale tier, in which case you also explicitly specify the number and type of machines to use.\n",
    "* `config` - Path to the job configuration file. This file should be a YAML document (JSON also accepted) containing a Job resource as defined in the API\n",
    "\n",
    "Note: Check to make sure gcloud is set to the current PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Submit the training job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "! gcloud ml-engine jobs submit training auto_mpg_hp_tuning_$(date +\"%Y%m%d_%H%M%S\") \\\n",
    "  --job-dir $JOB_DIR \\\n",
    "  --package-path $TRAINER_PACKAGE_PATH \\\n",
    "  --module-name $MAIN_TRAINER_MODULE \\\n",
    "  --region $REGION \\\n",
    "  --runtime-version=$RUNTIME_VERSION \\\n",
    "  --python-version=$PYTHON_VERSION \\\n",
    "  --scale-tier BASIC \\\n",
    "  --config $HPTUNING_CONFIG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# [Optional] StackDriver Logging\n",
    "You can view the logs for your training job:\n",
    "1. Go to https://console.cloud.google.com/\n",
    "1. Select \"Logging\" in left-hand pane\n",
    "1. In left-hand pane, go to \"AI Platform\" and select Jobs\n",
    "1. In filter by prefix, use the value of $JOB_NAME to view the logs\n",
    "\n",
    "On the logging page of your model, you can view the different results for each HP tuning job. \n",
    "\n",
    "Example:\n",
    "```\n",
    "{\n",
    "  \"trialId\": \"2\",\n",
    "  \"hyperparameters\": {\n",
    "    \"selection\": \"random\",\n",
    "    \"max_iter\": \"1892\",\n",
    "    \"tol\": \"0.0609819896050862\",\n",
    "    \"alpha\": \"4.3704164028167725\"\n",
    "  },\n",
    "  \"finalMetric\": {\n",
    "    \"trainingStep\": \"1000\",\n",
    "    \"objectiveValue\": 0.8658283435394591\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# [Optional] Verify Model File in GCS\n",
    "View the contents of the destination model folder to verify that all 30 model files have indeed been uploaded to GCS.\n",
    "\n",
    "Note: The model can take a few minutes to train and show up in GCS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "! gsutil ls $JOB_DIR/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Next Steps:\n",
    "The AI Platform online prediction service manages computing resources in the cloud to run your models. Check out the [documentation pages](https://cloud.google.com/ml-engine/docs/scikit/) that describe the process to get online predictions from these exported models using AI Platform."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}