Ex6.AutoEncoder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "91c76ea3",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float:right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9505dc0",
   "metadata": {},
   "source": [
    "# 6th exercise: <font color=\"#C70039\">Work with Autoencoders for anomaly detection</font>\n",
    "* Course: AML\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date:   08.08.2023\n",
    "\n",
    "<img src=\"https://blog.keras.io/img/ae/autoencoder_schema.jpg\" style=\"float: center;\" width=\"700\">\n",
    "\n",
    "---------------------------------\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information on your tasks (e.g. regarding the set of certain paramaters or a specific computational trick), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation please use English language, too.\n",
    "* This applies to all exercises throughout this course.\n",
    "\n",
    "---------------------------------\n",
    "\n",
    "### <font color=\"ce33ff\">DESCRIPTION</font>:\n",
    "Autoencoder is an unsupervised artificial neural network (ANN) that learns how to efficiently compress and encode data and then learns how to reconstruct the data back from the reduced encoded representation to a representation that is as close to the original input as possible.\n",
    "\n",
    "An Autoencoder reduces data dimensions by learning how to ignore the noise in the data and thus outliers.\n",
    "In the section above, you can seen an example of the input/output image from the MNIST dataset to an Autoencoder.\n",
    "\n",
    "#### Autoencoder Components:\n",
    "An Autoencoder consists of four main parts:\n",
    "\n",
    "1. Encoder: In which the model learns how to reduce the input dimensions and compress the input data into an encoded representation.\n",
    "\n",
    "2. Bottleneck: which is the layer that contains the compressed representation of the input data. This is the lowest possible dimensions of the input data. The bottlneck is also called latent vector. The concept of the latent space and latent vectors becomes important later on as we move forward to understand Generative Models. \n",
    "\n",
    "3. Decoder: In which the model learns how to reconstruct the data from the encoded representation to be as close to the original input as possible.\n",
    "\n",
    "4. Reconstruction Loss: This is the method that measures how well the decoder is performing and how close the output is to the original input.\n",
    "\n",
    "As always in ANNs, the training itself involves back propagation in order to minimize the network’s reconstruction loss.\n",
    "\n",
    "Due to this features of an Autoencoder the use cases are manyfold. One of the obviously is anomaly detection. \n",
    "\n",
    "#### Autoencoder Architecture:\n",
    "\n",
    "The network architecture for Autoencoders can vary between simple Feed Forward networks, Recurrent Neural Networks (LSTM) or Convolutional Neural Networks (CNN) depending on the use case. \n",
    "\n",
    "---------------------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "The tasks that you need to work on within this notebook are always indicated below as bullet points. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done by using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook to Google Colab or use your local machine.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date. \n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully \n",
    "    * add comments whereever you feel it necessary for better understanding\n",
    "    * run the notebook for the first time.\n",
    "    * the example below shows how to use an autoencoder for anomaly detection\n",
    "\n",
    "4. <font color=green>Develop an Autoencoder for Domain Adaptation (Me -> Walter White ). You can of course also take own data, e.g. a photo of yours and someone else.</font>\n",
    "5. Set at least the following hyperparameters for training (epochs=100000, shuffle=True).\n",
    "6. Implement a CNN for working out important features for the adaptation. If you feel lost in the exercise, please visit the sample solution.\n",
    "7. There is also an implementation of data augmentation that helps you building up your data set from one single \"original\" image. \n",
    "---------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e8ff998",
   "metadata": {},
   "source": [
    "### Auto-Encoding\n",
    "If you have correlated input data, the auto-encoder method will work very well because the encoding operation relies on the correlated features to compress the data.\n",
    "Let’s consider that an auto-encoder is trained on the MNIST dataset. \n",
    "As you know already, using a simple FeedForward neural network, this can be done by building a simple 6 layers network as below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f2c0a59",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import keras\n",
    "from   keras.datasets import mnist\n",
    "from   keras.models import Sequential, Model\n",
    "from   keras.layers import Dense, Input\n",
    "from   keras import optimizers\n",
    "from   keras.optimizers import Adam\n",
    "\n",
    "# load the inbuild mnist data set (8bit grayscale digits)\n",
    "# https://en.wikipedia.org/wiki/MNIST_database\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "# normalize the training and the validation data set\n",
    "train_x = x_train.reshape(60000, 784) / 255\n",
    "val_x = x_test.reshape(10000, 784) / 255\n",
    "\n",
    "# build the auto-encoding layers\n",
    "autoencoder = Sequential()\n",
    "autoencoder.add(Dense(512,  activation='elu', input_shape=(784,)))\n",
    "autoencoder.add(Dense(128,  activation='elu'))\n",
    "autoencoder.add(Dense(10,   activation='linear', name=\"bottleneck\"))\n",
    "autoencoder.add(Dense(128,  activation='elu'))\n",
    "autoencoder.add(Dense(512,  activation='elu'))\n",
    "autoencoder.add(Dense(784,  activation='sigmoid'))\n",
    "autoencoder.compile(loss='mean_squared_error', optimizer = Adam())\n",
    "\n",
    "'''\n",
    "NOTE:\n",
    "-----\n",
    "The Exponential Linear Unit (ELU) is an activation function for neural networks. \n",
    "In contrast to ReLUs (which you know), ELUs have negative values which allows them to push mean unit \n",
    "activations closer to zero like batch normalization but with lower computational complexity.\n",
    "'''    \n",
    "\n",
    "# train the model and finally assign the encoding to the decoder\n",
    "'''\n",
    "NOTE:\n",
    "-----\n",
    "make sure you understand, that you are training on train_x and not on train_y but train_x again for the reconstruction\n",
    "the same for the validation (val_x, val_x)\n",
    "'''\n",
    "trained_model = autoencoder.fit(train_x, train_x, batch_size=1024, epochs=10, verbose=1, validation_data=(val_x, val_x))\n",
    "encoder = Model(autoencoder.input, autoencoder.get_layer('bottleneck').output)\n",
    "\n",
    "encoded_data = encoder.predict(train_x)  # bottleneck representation\n",
    "\n",
    "decoded_output = autoencoder.predict(train_x)        # reconstruction\n",
    "encoding_dim = 10\n",
    "\n",
    "# return the decoder\n",
    "encoded_input = Input(shape=(encoding_dim,))\n",
    "decoder = autoencoder.layers[-3](encoded_input)\n",
    "decoder = autoencoder.layers[-2](decoder)\n",
    "decoder = autoencoder.layers[-1](decoder)\n",
    "decoder = Model(encoded_input, decoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e8b39fb",
   "metadata": {},
   "source": [
    "As you can see in the output, the last reconstruction loss/error for the validation set is approx. 0.0197, which is great. \n",
    "Now, if we pass any normal image from the MNIST dataset, the reconstruction loss will be very low (< 0.02) BUT if we tried to pass any other different image (outlier / anomaly), we will get a high reconstruction loss value because the network failed to reconstruct the image/input that is considered an anomaly.\n",
    "\n",
    "Notice in the code above, you can use only the encoder part to compress some data or images and you can also only use the decoder part to decompress the data by loading the decoder layers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adf39053",
   "metadata": {},
   "source": [
    "#### Auto-Encoders for Anomaly Detection\n",
    "Now, let’s do some anomaly detection. The code below uses two different images to predict the anomaly score (reconstruction error) using the autoencoder network we trained above. \n",
    "\n",
    "The first image is from the MNIST and the result is error=2.46241018. This means that the image is not an anomaly. The second image (yoda.png) obviously does not belong to the training dataset and the result is: error=2727.0718. This high error means that the image is an anomaly. Even the third image. The same concept applies to any type of dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da9e8232",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# If you are using a newer version of keras than '2.4.3', read this article below.\n",
    "# It describes a versioning issue in the keras libs\n",
    "'''https://stackoverflow.com/questions/72383347/how-to-fix-it-attributeerror-module-keras-preprocessing-image-has-no-attribu'''\n",
    "from keras.preprocessing import image\n",
    "\n",
    "# take an image from the validation data set or the training data set, respectively\n",
    "input_img = val_x[50] \n",
    "input_img_flat = input_img.reshape(1,784)\n",
    "\n",
    "target_data = autoencoder.predict(input_img_flat)\n",
    "\n",
    "dist = np.linalg.norm(input_img_flat - target_data, axis=-1)\n",
    "\n",
    "print(dist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccafc75d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Now take Master Yoda as the test image. The error score will be very high (error=2727.0718)\n",
    "img = load_img(\"./data/yoda.png\", target_size=(28, 28), color_mode = \"grayscale\")\n",
    "input_img = img_to_array(img)\n",
    "\n",
    "input_img_flat = input_img.reshape(1,784)\n",
    "target_data = autoencoder.predict(input_img_flat)\n",
    "dist = np.linalg.norm(input_img_flat - target_data, axis=-1)\n",
    "print(dist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b553f742",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "''' \n",
    "Now take a Mnist image which is taken from the google image search and although it is super similar to the training data\n",
    "it does not belong to the same data distribution the auto-encoder was trained on. \n",
    "It produces an error almost as high as yoda.png (approx error=2551.99)\n",
    "This makes autoencoders being a very robust technique for anomaly detection.\n",
    "'''\n",
    "img = load_img(\"./data/similarMnistNumber.jpg\", target_size=(28, 28), color_mode = \"grayscale\")\n",
    "input_img = img_to_array(img)\n",
    "\n",
    "input_img_flat = input_img.reshape(1,784)\n",
    "target_data = autoencoder.predict(input_img_flat)\n",
    "dist = np.linalg.norm(input_img_flat - target_data, axis=-1)\n",
    "\n",
    "print(dist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91d2447b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08ca3a4d-b4f5-492a-8d40-b80a8e748d36",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}