000_Getting_Started_with_TensorFlow2-Keras.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M2mBgOsBJk1L"
      },
      "source": [
        "# Getting Started with TensorFlow2-Keras\n",
        "\n",
        "**Author:** [FarrokhKarimi](https://github.com/farrokhkarimi)<br>\n",
        "**Description:** In this tutorial we will learn how to getting started with TensorFlow2-Keras. You can run this notebook via Google Colab Cloud System."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nv2gjwKNJk1Q"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "3Bbn0C0iJk1U"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CSyPYeLRJk1w"
      },
      "source": [
        "## Data loading & preprocessing\n",
        "\n",
        "Neural networks don't process raw data, like text files, encoded JPEG image files, or\n",
        " CSV files. They process **vectorized** & **standardized** representations.\n",
        "\n",
        "- Text files need to be read into string tensors, then split into words. Finally, the\n",
        " words need to be indexed & turned into integer tensors.\n",
        "- Images need to be read and decoded into integer tensors, then converted to floating\n",
        " point and normalized to small values (usually between 0 and 1).\n",
        "- CSV data needs to be parsed, with numerical features converted to floating point\n",
        " tensors and categorical features indexed and converted to integer tensors.\n",
        "Then each feature typically needs to be normalized to zero-mean and unit-variance.\n",
        "- Etc.\n",
        "\n",
        "Let's start with data loading.\n",
        "\n",
        "## Data loading\n",
        "\n",
        "Keras models accept three types of inputs:\n",
        "\n",
        "- **NumPy arrays**, just like Scikit-Learn and many other Python-based libraries. This\n",
        " is a good option if your data fits in memory.\n",
        "- **TensorFlow `Dataset` objects**. This is a\n",
        "high-performance option that is more suitable for datasets that do not fit in memory\n",
        " and that are streamed from disk or from a distributed filesystem.\n",
        "- **Python generators** that yield batches of data (such as custom subclasses of\n",
        "the `keras.utils.Sequence` class).\n",
        "\n",
        "Before you start training a model, you will need to make your data available as one of\n",
        "these formats. If you have a large dataset and you are training on GPU(s), consider\n",
        "using `Dataset` objects, since they will take care of performance-critical details,\n",
        " such as:\n",
        "\n",
        "- Asynchronously preprocessing your data on CPU while your GPU is busy, and bufferring\n",
        " it into a queue.\n",
        "- Prefetching data on GPU memory so it's immediately available when the GPU has\n",
        " finished processing the previous batch, so you can reach full GPU utilization.\n",
        "\n",
        "Keras features a range of utilities to help you turn raw data on disk into a `Dataset`:\n",
        "\n",
        "- `tf.keras.preprocessing.image_dataset_from_directory` turns image files sorted into\n",
        " class-specific folders into a labeled dataset of image tensors.\n",
        "- `tf.keras.preprocessing.text_dataset_from_directory` does the same for text files.\n",
        "\n",
        "In addition, the TensorFlow `tf.data` includes other similar utilities, such as\n",
        "`tf.data.experimental.make_csv_dataset` to load structured data from CSV files.\n",
        "\n",
        "**Example: obtaining a labeled dataset from image files on disk**\n",
        "\n",
        "Supposed you have image files sorted by class in different folders, like this:\n",
        "\n",
        "```\n",
        "main_directory/\n",
        "...class_a/\n",
        "......a_image_1.jpg\n",
        "......a_image_2.jpg\n",
        "...class_b/\n",
        "......b_image_1.jpg\n",
        "......b_image_2.jpg\n",
        "```\n",
        "\n",
        "Then you can do:\n",
        "\n",
        "```python\n",
        "# Create a dataset.\n",
        "dataset = keras.preprocessing.image_dataset_from_directory(\n",
        "  'path/to/main_directory', batch_size=64, image_size=(200, 200))\n",
        "\n",
        "# For demonstration, iterate over the batches yielded by the dataset.\n",
        "for data, labels in dataset:\n",
        "   print(data.shape)  # (64, 200, 200, 3)\n",
        "   print(data.dtype)  # float32\n",
        "   print(labels.shape)  # (64,)\n",
        "   print(labels.dtype)  # int32\n",
        "```\n",
        "\n",
        "The label of a sample is the rank of its folder in alphanumeric order. Naturally, this\n",
        " can also be configured explicitly by passing, e.g.\n",
        "`class_names=['class_a', 'class_b']`, in which cases label `0` will be `class_a` and\n",
        " `1` will be `class_b`.\n",
        "\n",
        "**Example: obtaining a labeled dataset from text files on disk**\n",
        "\n",
        "Likewise for text: if you have `.txt` documents sorted by class in different folders,\n",
        " you can do:\n",
        "\n",
        "```python\n",
        "dataset = keras.preprocessing.text_dataset_from_directory(\n",
        "  'path/to/main_directory', batch_size=64)\n",
        "\n",
        "# For demonstration, iterate over the batches yielded by the dataset.\n",
        "for data, labels in dataset:\n",
        "   print(data.shape)  # (64,)\n",
        "   print(data.dtype)  # string\n",
        "   print(labels.shape)  # (64,)\n",
        "   print(labels.dtype)  # int32\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yYkWm86IJk2G"
      },
      "source": [
        "## Data preprocessing with Keras\n",
        "\n",
        "Once your data is in the form of string,int,float NumpPy arrays, or a `Dataset` object\n",
        " (or Python generator) that yields batches of string,int,float tensors,\n",
        "it is time to **preprocess** the data. This can mean:\n",
        "\n",
        "- Tokenization of string data, followed by token indexing.\n",
        "- Feature normalization.\n",
        "- Rescaling the data to small values (in general, input values to a neural network\n",
        "should be close to zero -- typically we expect either data with zero-mean and\n",
        " unit-variance, or data in the `[0, 1]` range.\n",
        "\n",
        "### Using Keras preprocessing layers\n",
        "\n",
        "In Keras, you do in-model data preprocessing via **preprocessing layers**. This\n",
        " includes:\n",
        "\n",
        "- Vectorizing raw strings of text via the `TextVectorization` layer\n",
        "- Feature normalization via the `Normalization` layer\n",
        "- Image rescaling, cropping, or image data augmentation\n",
        "\n",
        "The key advantage of using Keras preprocessing layers is that **they can be included\n",
        " directly into your model**, either during training or after training,\n",
        "which makes your models portable.\n",
        "\n",
        "Some preprocessing layers have a state:\n",
        "\n",
        "- `TextVectorization` holds an index mapping words or tokens to integer indices\n",
        "- `Normalization` holds the mean and variance of your features\n",
        "\n",
        "The state of a preprocessing layer is obtained by calling `layer.adapt(data)` on a\n",
        " sample of the training data (or all of it).\n",
        "\n",
        "\n",
        "**Example: turning strings into sequences of integer word indices**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VsIrrREDJk2H",
        "outputId": "86bb2d0d-ebcb-462b-ee0f-a73cfd32ffd4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[[4 5 2 9 3]\n",
            " [7 6 2 8 3]], shape=(2, 5), dtype=int64)\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "\n",
        "# Example training data, of dtype `string`.\n",
        "training_data = np.array([[\"This is the 1st sample.\"], [\"And here's the 2nd sample.\"]])\n",
        "\n",
        "# Create a TextVectorization layer instance. It can be configured to either\n",
        "# return integer token indices, or a dense token representation (e.g. multi-hot\n",
        "# or TF-IDF). The text standardization and text splitting algorithms are fully\n",
        "# configurable.\n",
        "vectorizer = TextVectorization(output_mode=\"int\")\n",
        "\n",
        "# Calling `adapt` on an array or dataset makes the layer generate a vocabulary\n",
        "# index for the data, which can then be reused when seeing new data.\n",
        "vectorizer.adapt(training_data)\n",
        "\n",
        "# After calling adapt, the layer is able to encode any n-gram it has seen before\n",
        "# in the `adapt()` data. Unknown n-grams are encoded via an \"out-of-vocabulary\"\n",
        "# token.\n",
        "integer_data = vectorizer(training_data)\n",
        "print(integer_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KeGd6x-KJk2W"
      },
      "source": [
        "**Example: turning strings into sequences of one-hot encoded bigrams**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OP0WHJJeJk2Y",
        "outputId": "efd06d45-d0a9-4361-bf89-6dd7b9a1b9ff"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[[0. 1. 1. 1. 1. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 1. 1.]\n",
            " [0. 1. 1. 0. 0. 1. 0. 0. 0. 1. 1. 1. 1. 1. 1. 0. 0.]], shape=(2, 17), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "\n",
        "# Example training data, of dtype `string`.\n",
        "training_data = np.array([[\"This is the 1st sample.\"], [\"And here's the 2nd sample.\"]])\n",
        "\n",
        "# Create a TextVectorization layer instance. It can be configured to either\n",
        "# return integer token indices, or a dense token representation (e.g. multi-hot\n",
        "# or TF-IDF). The text standardization and text splitting algorithms are fully\n",
        "# configurable.\n",
        "vectorizer = TextVectorization(output_mode=\"binary\", ngrams=2)\n",
        "\n",
        "# Calling `adapt` on an array or dataset makes the layer generate a vocabulary\n",
        "# index for the data, which can then be reused when seeing new data.\n",
        "vectorizer.adapt(training_data)\n",
        "\n",
        "# After calling adapt, the layer is able to encode any n-gram it has seen before\n",
        "# in the `adapt()` data. Unknown n-grams are encoded via an \"out-of-vocabulary\"\n",
        "# token.\n",
        "integer_data = vectorizer(training_data)\n",
        "print(integer_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9vjbjndkJk2j"
      },
      "source": [
        "**Example: normalizing features**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y7_Vc-xEJk2k",
        "outputId": "473f636b-6644-4434-ed52-c31db6c2b589"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "var: 1.0000\n",
            "mean: -0.0000\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.layers.experimental.preprocessing import Normalization\n",
        "\n",
        "# Example image data, with values in the [0, 255] range\n",
        "training_data = np.random.randint(0, 256, size=(64, 200, 200, 3)).astype(\"float32\")\n",
        "\n",
        "normalizer = Normalization(axis=-1)\n",
        "normalizer.adapt(training_data)\n",
        "\n",
        "normalized_data = normalizer(training_data)\n",
        "print(\"var: %.4f\" % np.var(normalized_data))\n",
        "print(\"mean: %.4f\" % np.mean(normalized_data))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mmrfy-CTJk2u"
      },
      "source": [
        "**Example: rescaling & center-cropping images**\n",
        "\n",
        "Both the `Rescaling` layer and the `CenterCrop` layer are stateless, so it isn't\n",
        " necessary to call `adapt()` in this case."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mFyE2szbJk2y",
        "outputId": "d3b57a11-8a6e-4d90-b05d-ce10d63b2961"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "shape: (64, 150, 150, 3)\n",
            "min: 0.0\n",
            "max: 1.0\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.layers.experimental.preprocessing import CenterCrop\n",
        "from tensorflow.keras.layers.experimental.preprocessing import Rescaling\n",
        "\n",
        "# Example image data, with values in the [0, 255] range\n",
        "training_data = np.random.randint(0, 256, size=(64, 200, 200, 3)).astype(\"float32\")\n",
        "\n",
        "cropper = CenterCrop(height=150, width=150)\n",
        "scaler = Rescaling(scale=1.0 / 255)\n",
        "\n",
        "output_data = scaler(cropper(training_data))\n",
        "print(\"shape:\", output_data.shape)\n",
        "print(\"min:\", np.min(output_data))\n",
        "print(\"max:\", np.max(output_data))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvvGx_30Jk27"
      },
      "source": [
        "## Building models with the Keras Functional API\n",
        "\n",
        "A \"layer\" is a simple input-output transformation (such as the scaling &\n",
        "center-cropping transformations above). For instance, here's a linear projection layer\n",
        " that maps its inputs to a 16-dimensional feature space:\n",
        "\n",
        "```python\n",
        "dense = keras.layers.Dense(units=16)\n",
        "```\n",
        "\n",
        "A \"model\" is a directed acyclic graph of layers. You can think of a model as a\n",
        "\"bigger layer\" that encompasses multiple sublayers and that can be trained via exposure\n",
        " to data.\n",
        "\n",
        "The most common and most powerful way to build Keras models is the Functional API. To\n",
        "build models with the Functional API, you start by specifying the shape (and\n",
        "optionally the dtype) of your inputs. If any dimension of your input can vary, you can\n",
        "specify it as `None`. For instance, an input for 200x200 RGB image would have shape\n",
        "`(200, 200, 3)`, but an input for RGB images of any size would have shape `(None,\n",
        " None, 3)`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "6nhgKhIxJk29"
      },
      "outputs": [],
      "source": [
        "# Let's say we expect our inputs to be RGB images of arbitrary size\n",
        "inputs = keras.Input(shape=(None, None, 3))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "34tqLhnwJk3E"
      },
      "source": [
        "After defining your input(s), you chain layer transformations on top of your inputs,\n",
        " until your final output:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "raIswIGNJk3F"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras import layers\n",
        "\n",
        "# Center-crop images to 150x150\n",
        "x = CenterCrop(height=150, width=150)(inputs)\n",
        "# Rescale images to [0, 1]\n",
        "x = Rescaling(scale=1.0 / 255)(x)\n",
        "\n",
        "# Apply some convolution and pooling layers\n",
        "x = layers.Conv2D(filters=32, kernel_size=(3, 3), activation=\"relu\")(x)\n",
        "x = layers.MaxPooling2D(pool_size=(3, 3))(x)\n",
        "x = layers.Conv2D(filters=32, kernel_size=(3, 3), activation=\"relu\")(x)\n",
        "x = layers.MaxPooling2D(pool_size=(3, 3))(x)\n",
        "x = layers.Conv2D(filters=32, kernel_size=(3, 3), activation=\"relu\")(x)\n",
        "\n",
        "# Apply global average pooling to get flat feature vectors\n",
        "x = layers.GlobalAveragePooling2D()(x)\n",
        "\n",
        "# Add a dense classifier on top\n",
        "num_classes = 10\n",
        "outputs = layers.Dense(num_classes, activation=\"softmax\")(x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YH_61AJUJk3Q"
      },
      "source": [
        "Once you have defined the directed acyclic graph of layers that turns your input(s) into\n",
        " your outputs, instantiate a `Model` object:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "Rq-XsDJxJk3R"
      },
      "outputs": [],
      "source": [
        "model = keras.Model(inputs=inputs, outputs=outputs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SuOhYRw1Jk3b"
      },
      "source": [
        "This model behaves basically like a bigger layer. You can call it on batches of data, like\n",
        " this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BHCWfXhDJk3c",
        "outputId": "c47d8f99-8669-49a1-cbf9-fd4604c3dba2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(64, 10)\n"
          ]
        }
      ],
      "source": [
        "data = np.random.randint(0, 256, size=(64, 200, 200, 3)).astype(\"float32\")\n",
        "processed_data = model(data)\n",
        "print(processed_data.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "diHnzhPLJk3t"
      },
      "source": [
        "You can print a summary of how your data gets transformed at each stage of the model.\n",
        " This is useful for debugging.\n",
        "\n",
        "Note that the output shape displayed for each layer includes the **batch size**. Here\n",
        " the batch size is None, which indicates our model can process batches of any size."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EiBlExl_Jk3v",
        "outputId": "ddb591d8-e962-424c-d453-75eb4f0c9511"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, None, None, 3)]   0         \n",
            "                                                                 \n",
            " center_crop_1 (CenterCrop)  (None, 150, 150, 3)       0         \n",
            "                                                                 \n",
            " rescaling_1 (Rescaling)     (None, 150, 150, 3)       0         \n",
            "                                                                 \n",
            " conv2d (Conv2D)             (None, 148, 148, 32)      896       \n",
            "                                                                 \n",
            " max_pooling2d (MaxPooling2D  (None, 49, 49, 32)       0         \n",
            " )                                                               \n",
            "                                                                 \n",
            " conv2d_1 (Conv2D)           (None, 47, 47, 32)        9248      \n",
            "                                                                 \n",
            " max_pooling2d_1 (MaxPooling  (None, 15, 15, 32)       0         \n",
            " 2D)                                                             \n",
            "                                                                 \n",
            " conv2d_2 (Conv2D)           (None, 13, 13, 32)        9248      \n",
            "                                                                 \n",
            " global_average_pooling2d (G  (None, 32)               0         \n",
            " lobalAveragePooling2D)                                          \n",
            "                                                                 \n",
            " dense (Dense)               (None, 10)                330       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 19,722\n",
            "Trainable params: 19,722\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jqSsQihwJk4A"
      },
      "source": [
        "The Functional API also makes it easy to build models that have multiple inputs (for\n",
        "instance, an image *and* its metadata) or multiple outputs (for instance, predicting\n",
        "the class of the image *and* the likelihood that a user will click on it)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cGsCr_RWJk4B"
      },
      "source": [
        "## Training models with `fit()`\n",
        "\n",
        "At this point, you know:\n",
        "\n",
        "- How to prepare your data (e.g. as a NumPy array or a `tf.data.Dataset` object)\n",
        "- How to build a model that will process your data\n",
        "\n",
        "The next step is to train your model on your data. The `Model` class features a\n",
        "built-in training loop, the `fit()` method. It accepts `Dataset` objects, Python\n",
        " generators that yield batches of data, or NumPy arrays.\n",
        "\n",
        "Before you can call `fit()`, you need to specify an optimizer and a loss function (we\n",
        " assume you are already familiar with these concepts). This is the `compile()` step:\n",
        "\n",
        "```python\n",
        "model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),\n",
        "              loss=keras.losses.CategoricalCrossentropy())\n",
        "```\n",
        "\n",
        "Loss and optimizer can be specified via their string identifiers (in this case\n",
        "their default constructor argument values are used):\n",
        "\n",
        "\n",
        "```python\n",
        "model.compile(optimizer='rmsprop', loss='categorical_crossentropy')\n",
        "```\n",
        "\n",
        "Once your model is compiled, you can start \"fitting\" the model to the data.\n",
        "Here's what fitting a model looks like with NumPy data:\n",
        "\n",
        "```python\n",
        "model.fit(numpy_array_of_samples, numpy_array_of_labels,\n",
        "          batch_size=32, epochs=10)\n",
        "```\n",
        "\n",
        "Besides the data, you have to specify two key parameters: the `batch_size` and\n",
        "the number of epochs (iterations on the data). Here our data will get sliced on batches\n",
        " of 32 samples, and the model will iterate 10 times over the data during training.\n",
        "\n",
        "Here's what fitting a model looks like with a dataset:\n",
        "\n",
        "```python\n",
        "model.fit(dataset_of_samples_and_labels, epochs=10)\n",
        "```\n",
        "\n",
        "Since the data yielded by a dataset is expect to be already batched, you don't need to\n",
        " specify the batch size here.\n",
        "\n",
        "Let's look at it in practice with a toy example model that learns to classify MNIST\n",
        " digits:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R8k7NKEHJk4D",
        "outputId": "1cb32d79-8cac-465b-f79e-d7dc269e0001"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n",
            "11501568/11490434 [==============================] - 0s 0us/step\n",
            "Model: \"model_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_2 (InputLayer)        [(None, 28, 28)]          0         \n",
            "                                                                 \n",
            " rescaling_2 (Rescaling)     (None, 28, 28)            0         \n",
            "                                                                 \n",
            " flatten (Flatten)           (None, 784)               0         \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 128)               100480    \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 128)               16512     \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 10)                1290      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 118,282\n",
            "Trainable params: 118,282\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Fit on NumPy data\n",
            "938/938 [==============================] - 3s 3ms/step - loss: 0.2676\n",
            "Fit on Dataset\n",
            "938/938 [==============================] - 2s 2ms/step - loss: 0.1178\n"
          ]
        }
      ],
      "source": [
        "# Get the data as Numpy arrays\n",
        "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "\n",
        "# Build a simple model\n",
        "inputs = keras.Input(shape=(28, 28))\n",
        "x = layers.experimental.preprocessing.Rescaling(1.0 / 255)(inputs)\n",
        "x = layers.Flatten()(x)\n",
        "x = layers.Dense(128, activation=\"relu\")(x)\n",
        "x = layers.Dense(128, activation=\"relu\")(x)\n",
        "outputs = layers.Dense(10, activation=\"softmax\")(x)\n",
        "model = keras.Model(inputs, outputs)\n",
        "model.summary()\n",
        "\n",
        "# Compile the model\n",
        "model.compile(optimizer=\"adam\", loss=\"sparse_categorical_crossentropy\")\n",
        "\n",
        "# Train the model for 1 epoch from Numpy data\n",
        "batch_size = 64\n",
        "print(\"Fit on NumPy data\")\n",
        "history = model.fit(x_train, y_train, batch_size=batch_size, epochs=1)\n",
        "\n",
        "# Train the model for 1 epoch using a dataset\n",
        "dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(batch_size)\n",
        "print(\"Fit on Dataset\")\n",
        "history = model.fit(dataset, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sEnQgPELJk4K"
      },
      "source": [
        "The `fit()` call returns a \"history\" object which records what happened over the course\n",
        "of training. The `history.history` dict contains per-epoch timeseries of metrics\n",
        "values (here we have only one metric, the loss, and one epoch, so we only get a single\n",
        " scalar):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h7y5bNqzJk4M",
        "outputId": "c4d88fe9-db25-46dc-9b64-a9c985b8ff10"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'loss': [0.11784098297357559]}\n"
          ]
        }
      ],
      "source": [
        "print(history.history)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kePs5N2WJk4e"
      },
      "source": [
        "### Keeping track of performance metrics\n",
        "\n",
        "As you're training a model, you want to keep track of metrics such as classification\n",
        "accuracy, precision, recall, AUC, etc. Besides, you want to monitor these metrics not\n",
        " only on the training data, but also on a validation set.\n",
        "\n",
        "**Monitoring metrics**\n",
        "\n",
        "You can pass a list of metric objects to `compile()`, like this:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v3RHXa6yJk4f",
        "outputId": "35e4b75a-3b13-4e6e-f3c0-a3fe297c8a63"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "938/938 [==============================] - 3s 3ms/step - loss: 0.0815 - acc: 0.9752\n"
          ]
        }
      ],
      "source": [
        "model.compile(\n",
        "    optimizer=\"adam\",\n",
        "    loss=\"sparse_categorical_crossentropy\",\n",
        "    metrics=[keras.metrics.SparseCategoricalAccuracy(name=\"acc\")],\n",
        ")\n",
        "history = model.fit(dataset, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XEmsPabPJk4q"
      },
      "source": [
        "**Passing validation data to `fit()`**\n",
        "\n",
        "You can pass validation data to `fit()` to monitor your validation loss & validation\n",
        " metrics. Validation metrics get reported at the end of each epoch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T6MfwAcuJk4q",
        "outputId": "30567eff-cbbb-433a-e5fe-10d49995a1b9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "938/938 [==============================] - 5s 5ms/step - loss: 0.0554 - acc: 0.9832 - val_loss: 0.0896 - val_acc: 0.9732\n"
          ]
        }
      ],
      "source": [
        "val_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(batch_size)\n",
        "history = model.fit(dataset, epochs=1, validation_data=val_dataset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dp2qlGYjJk43"
      },
      "source": [
        "### Using callbacks for checkpointing (and more)\n",
        "\n",
        "If training goes on for more than a few minutes, it's important to save your model at\n",
        " regular intervals during training. You can then use your saved models\n",
        "to restart training in case your training process crashes (this is important for\n",
        "multi-worker distributed training, since with many workers at least one of them is\n",
        " bound to fail at some point).\n",
        "\n",
        "An important feature of Keras is **callbacks**, configured in `fit()`. Callbacks are\n",
        " objects that get called by the model at different point during training, in particular:\n",
        "\n",
        "- At the beginning and end of each batch\n",
        "- At the beginning and end of each epoch\n",
        "\n",
        "Callbacks are a way to make model trainable entirely scriptable.\n",
        "\n",
        "You can use callbacks to periodically save your model. Here's a simple example: a\n",
        " `ModelCheckpoint` callback\n",
        "configured to save the model at the end of every epoch. The filename will include the\n",
        " current epoch.\n",
        "\n",
        "```python\n",
        "callbacks = [\n",
        "    keras.callbacks.ModelCheckpoint(\n",
        "        filepath='path/to/my/model_{epoch}',\n",
        "        save_freq='epoch')\n",
        "]\n",
        "model.fit(dataset, epochs=2, callbacks=callbacks)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0aqWsGxKJk45"
      },
      "source": [
        "You can also use callbacks to do things like periodically changing the learning of your\n",
        "optimizer, streaming metrics to a Slack bot, sending yourself an email notification\n",
        " when training is complete, etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wXq23y8QJk4-"
      },
      "source": [
        "### Monitoring training progress with TensorBoard\n",
        "\n",
        "Staring at the Keras progress bar isn't the most ergonomic way to monitor how your loss\n",
        " and metrics are evolving over time. There's a better solution:\n",
        "[TensorBoard](https://www.tensorflow.org/tensorboard),\n",
        "a web application that can display real-time graphs of your metrics (and more).\n",
        "\n",
        "To use TensorBoard with `fit()`, simply pass a `keras.callbacks.TensorBoard` callback\n",
        " specifying the directory where to store TensorBoard logs:\n",
        "\n",
        "\n",
        "```python\n",
        "callbacks = [\n",
        "    keras.callbacks.TensorBoard(log_dir='./logs')\n",
        "]\n",
        "model.fit(dataset, epochs=2, callbacks=callbacks)\n",
        "```\n",
        "\n",
        "You can then launch a TensorBoard instance that you can open in your browser to monitor\n",
        " the logs getting written to this location:\n",
        "\n",
        "```\n",
        "tensorboard --logdir=./logs\n",
        "```\n",
        "\n",
        "What's more, you can launch an in-line TensorBoard tab when training models in Jupyter\n",
        " / Colab notebooks.\n",
        "[Here's more information](https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "563iZMW-Jk5A"
      },
      "source": [
        "### After `fit()`: evaluating test performance & generating predictions on new data\n",
        "\n",
        "Once you have a trained model, you can evaluate its loss and metrics on new data via\n",
        " `evaluate()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bGGDQNLOJk5B",
        "outputId": "4abe4abd-e660-4e6f-ba47-51d6e221ee47"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "157/157 [==============================] - 0s 2ms/step - loss: 0.0896 - acc: 0.9732\n",
            "loss: 0.09\n",
            "acc: 0.97\n"
          ]
        }
      ],
      "source": [
        "loss, acc = model.evaluate(val_dataset)  # returns loss and metrics\n",
        "print(\"loss: %.2f\" % loss)\n",
        "print(\"acc: %.2f\" % acc)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W5DEMl0YJk5G"
      },
      "source": [
        "You can also generate NumPy arrays of predictions (the activations of the output\n",
        " layer(s) in the model) via `predict()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zR-tzOaCJk5J",
        "outputId": "4b7d6751-4799-4e78-e082-abc919d05e2d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(10000, 10)\n"
          ]
        }
      ],
      "source": [
        "predictions = model.predict(val_dataset)\n",
        "print(predictions.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L3aEHR_xJk5s"
      },
      "source": [
        "## Finding the best model configuration with hyperparameter tuning\n",
        "\n",
        "Once you have a working model, you're going to want to optimize its configuration --\n",
        "architecture choices, layer sizes, etc. Human intuition can only go so far, so you'll\n",
        " want to leverage a systematic approach: hyperparameter search.\n",
        "\n",
        "You can use\n",
        "[Keras Tuner](https://keras-team.github.io/keras-tuner/documentation/tuners/) to find\n",
        " the best hyperparameter for your Keras models. It's as easy as calling `fit()`.\n",
        "\n",
        "Here how it works.\n",
        "\n",
        "First, place your model definition in a function, that takes a single `hp` argument.\n",
        "Inside this function, replace any value you want to tune with a call to hyperparameter\n",
        " sampling methods, e.g. `hp.Int()` or `hp.Choice()`:\n",
        "\n",
        "```python\n",
        "def build_model(hp):\n",
        "    inputs = keras.Input(shape=(784,))\n",
        "    x = layers.Dense(\n",
        "        units=hp.Int('units', min_value=32, max_value=512, step=32),\n",
        "        activation='relu'))(inputs)\n",
        "    outputs = layers.Dense(10, activation='softmax')(x)\n",
        "    model = keras.Model(inputs, outputs)\n",
        "    model.compile(\n",
        "        optimizer=keras.optimizers.Adam(\n",
        "            hp.Choice('learning_rate',\n",
        "                      values=[1e-2, 1e-3, 1e-4])),\n",
        "        loss='sparse_categorical_crossentropy',\n",
        "        metrics=['accuracy'])\n",
        "    return model\n",
        "```\n",
        "\n",
        "The function should return a compiled model.\n",
        "\n",
        "Next, instantiate a tuner object specifying your optimiation objective and other search\n",
        " parameters:\n",
        "\n",
        "\n",
        "```python\n",
        "import kerastuner\n",
        "\n",
        "tuner = kerastuner.tuners.Hyperband(\n",
        "  build_model,\n",
        "  objective='val_loss',\n",
        "  max_epochs=100,\n",
        "  max_trials=200,\n",
        "  executions_per_trial=2,\n",
        "  directory='my_dir')\n",
        "```\n",
        "\n",
        "Finally, start the search with the `search()` method, which takes the same arguments as\n",
        " `Model.fit()`:\n",
        "\n",
        "```python\n",
        "tuner.search(dataset, validation_data=val_dataset)\n",
        "```\n",
        "\n",
        "When search is over, you can retrieve the best model(s):\n",
        "\n",
        "```python\n",
        "models = tuner.get_best_models(num_models=2)\n",
        "```\n",
        "\n",
        "Or print a summary of the results:\n",
        "\n",
        "```python\n",
        "tuner.results_summary()\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cHsYiiMnTXEa"
      },
      "source": [
        "Contact : [FarrokhKarimi](https://zil.ink/farrokhkarimi)  \n",
        "Reference : [tensorflow.com](https://www.tensorflow.org/)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "intro_to_keras_for_engineers",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.2"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}