01a-neuron-from-scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The basic building blocks\n",
    "In this notebook we will take a closer look at elements used in computation in most machine learning systems, the **tensor**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tensor is a generalization of vectors and matrices to an arbitrary number of **axes**. What does that mean? A scalar, or a single value, is a 0D tensor. A vector is a 1D tensor. A matrix is a 2D tensor and so on. When a new axis is added we increase the dimension of the tensor. The usage of the word **dimension** can be confusing as it is used to refer to the number of axes as well as the number of elements along an axis. We can eliminate this ambiguity by talking about the **rank** of a tensor as the number of axes it has. Similarly we also refer to the **shape** of the tensor as vector which describes the number of dimensions along each axis of a tensor. The picture below will show you how to think of tensors and their rank visually:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Tensors](images/tensor.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "R uses the internal data type `vector` to manipulate 0D and 1D tensors and the [`matrix`](https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/matrix) data type to manipulate 2D tensors. For higher dimensional data the [`array`](https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/array) data type is used. To better understand these concepts and how they relate to R, you will make the following excercise:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Excercise 1\n",
    "In the following cell complete the following tasks:\n",
    "1. Using R's [`c`](https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/c) function, define a 0D tensor called 'x' with the value 4:\n",
    "$$\n",
    "x =\n",
    "\\left(\n",
    "\\begin{array}{c}\n",
    "4 \\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "$$\n",
    "2. Again using [`c`](https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/c), define a 1D tensor called 'y':\n",
    "$$\n",
    "y =\n",
    "\\left(\n",
    "\\begin{array}{c}\n",
    "1\\\\\n",
    "2\\\\\n",
    "3\\\\\n",
    "4\\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "$$\n",
    "3. With the [`matrix`](https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/matrix) function, define a 2D tensor called 'Z':\n",
    "$$\n",
    "Z =\n",
    "\\left(\n",
    "\\begin{array}{c}\n",
    "1 & 2 & 3 & 4\\\\\n",
    "5 & 6 & 7 & 8\\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "$$\n",
    "4. Print the dimensions of each tensor using [`dim`](https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/dim) (use the `as.array()` function to get meaningful results for 1. and 2.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In deep learning you will encounter data in various tensor dimensions\n",
    "- Vector data, 2D (samples, features)\n",
    "- Time-series, 3D (samples, timesteps, features)\n",
    "- Images, 4D (samples, height, width, channel) or (samples, channel, height, width)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manipulating tensors\n",
    "Deep learning involves usually involves a lot of preprocessing of data to get it into the right shape for a model to train on. Often, you will need to manipulate a tensor's dimensions and axes, and it is crucial to understand how this works in R.\n",
    "\n",
    "During the course you will have plenty of opportunity to explore the different techniques with which R provides us. \n",
    "\n",
    "### Slicing\n",
    "The first technique we will discuss is **slicing**. With the bracket operators `[` and `]` we can slice data from a tensor by specifying the index or indices for axes we want to select:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z <- matrix(c(1, 2, 3, 4, 5, 6),  ncol=2)\n",
    "z # This will show z\n",
    "\n",
    "slice <- z[1,] # This will slice the first row\n",
    "slice # This will show the slice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z[,1] # This will slice the first column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z[1:2, 1] # This will extract the first two elements of the first column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2\n",
    "In the next cell we have defined a two-dimensional tensor called `Z`. It is a matrix of size 4 by 4. Extract the 'bottom right' section of the matrix, with size 2 by 2, pictured as the numbers in red in the following diagram:\n",
    "\n",
    "$$\n",
    "Z =\n",
    "\\left(\n",
    "\\begin{array}{cccc}\n",
    "1 & 2 & 3 & 4\\\\\n",
    "5 & 6 & 7 & 8\\\\\n",
    "9 & 10 & \\color{red}{11} & \\color{red}{12} \\\\\n",
    "13 & 14 & \\color{red}{15} & \\color{red}{16} \\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Z <- matrix(c(1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15, 4, 8, 12, 16), ncol = 4)\n",
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3\n",
    "In the next cell we have defined a three-dimensional tensor, also called `Z`. It is of size 4 by 4 by 4, and you can think of it as a cube. Extract the middle of the cube, with dimensions 2 by 2 by 2. The result (in R) should be:\n",
    "\n",
    "```\n",
    "22 23 26 27 38 39 42 43 \n",
    "```\n",
    "\n",
    "Please note that because we cannot easily visualise a three-dimensional tensor, R will give a list of numbers as its output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z <- array(1:64, dim = c(4, 4, 4))  # 1:64 will generate a vector with elements 1, 2 ... 64\n",
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor arithmetic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a number of operations which can be performed on tensors, the first one is an **element-wise** operation, which applies a function to each element of the tensor independently. By default `+` and `*` in R are element-wise operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z <- matrix(c(1, 2, 3, 4), nrow=2, ncol=2) \n",
    "z\n",
    "a <- z * z\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z + z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiplication\n",
    "The **multiplication** of two tensors returns a new tensor and is denoted by `%*%` in R. The multiplication will calculate the product for each row and column in the two tensors as follows:\n",
    "\n",
    "![Dot product](./images/Matrix_multiplication_diagram_2.svg)\n",
    "[[source]](https://en.wikipedia.org/wiki/Matrix_multiplication)\n",
    "\n",
    "In the following example the output tensor is of the same dimensions as the two input tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A <- matrix(c(1, 2, 3, 4), ncol = 2)\n",
    "A\n",
    "A %*% A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A <- matrix(1:4, ncol = 2)\n",
    "A\n",
    "B <- matrix(5:8, ncol = 2)\n",
    "B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A %*% B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4\n",
    "Calculate the following matrix multiplication:\n",
    "\n",
    "$$\n",
    "\\left(\n",
    "\\begin{array}{cc}\n",
    "1 & 2 \\\\\n",
    "3 & 4 \\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "\\left(\n",
    "\\begin{array}{cc}\n",
    "5 & 6 \\\\\n",
    "7 & 8 \\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "=\n",
    "\\left(\n",
    "\\begin{array}{cc}\n",
    "19 & 22 \\\\\n",
    "43 & 50 \\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5\n",
    "Calculate the following multiplication of a matrix with a vector:\n",
    "\n",
    "$$\n",
    "\\left(\n",
    "\\begin{array}{cc}\n",
    "1 & 2 \\\\\n",
    "3 & 4 \\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "\\left(\n",
    "\\begin{array}{c}\n",
    "1\\\\\n",
    "2\\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "=\n",
    "\\left(\n",
    "\\begin{array}{c}\n",
    "5\\\\\n",
    "11\\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transposition\n",
    "Transposition will flip a matrix over its main diagonal, like so:\n",
    "\n",
    "$$\n",
    "\\left(\n",
    "\\begin{array}{cc}\n",
    "1 & 2\\\\\n",
    "3 & 4\\\\\n",
    "\\end{array}\n",
    "\\right)^T\n",
    "=\n",
    "\\left(\n",
    "\\begin{array}{cc}\n",
    "1 & 3\\\\\n",
    "2 & 4\\\\\n",
    "\\end{array}\n",
    "\\right)\n",
    "$$\n",
    "\n",
    "R has a [`t`](https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/t) function to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a <- matrix(c(1, 3, 2, 4), ncol = 2)\n",
    "a\n",
    "t(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6\n",
    "Create a matrix with 1 column and 1 row and transpose it. Inspect the dimensions with the [`dim`](https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/dim) function. Try the same with a vector. What happens?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see that any vector that is transposed is automatically converted into a matrix. R has no concept of a row vector or column vector. Instead, it uses a matrix with one dimension set to 1, as you saw above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a single dense layer\n",
    "In the remainder of this notebook you will solve the problem introduced in the slides. Let's load the data first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source(\"01-helpers.R\")  # this will load helper functions for this session\n",
    "\n",
    "data <- dataset_linear()\n",
    "X <- data$X\n",
    "y <- data$y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We inspect the input variables `X` and the labels `y` using the [`head`](https://www.rdocumentation.org/packages/utils/versions/3.5.1/topics/head) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 7\n",
    "Inspect the dimensions of `X` and `y`. How many instances do we have? How many input variables?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the data set using one of the helper functions you just loaded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_dataset(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Solving the problem - building a neuron\n",
    "During the remainder of this notebook you will build a single neuron that will perfectly classify all instances.\n",
    "\n",
    "Let's look at a diagram of this neuron you are going to build:\n",
    "\n",
    "![The neuron we are going to build](images/neuron.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will build this neuron step by step, going from left to right through the diagram. Let's begin by defining the neuron's weights:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 8\n",
    "The neuron will have two weights, one for each input, $w_1$ and $w_2$. Create a weight vector `w` with the values `0.5` and `0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 9\n",
    "Create the bias variable `b`. Set it to `0` for now. This means that we will ignore its effect in subsequent calculations. You will modify it later in order to solve the problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 10\n",
    "Create the sigmoid activation function `sigmoid`, which takes a vector or number `x` and returns as output the following:\n",
    "$$\n",
    "\\sigma(x) = sigmoid(x) = \\frac{1}{1 + e^{-x}}\n",
    "$$\n",
    "\n",
    "We have provided you with a skeleton below:\n",
    "\n",
    "**Hint**: you will need R's [`exp`](https://www.rdocumentation.org/packages/base/versions/3.5.1/topics/log) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sigmoid <- function(x) {\n",
    "    <FILL IN>\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 11\n",
    "Perform the complete calculation of the neuron in R. You can either use the diagram as a reference above, or use the following equation:\n",
    "\n",
    "$$\n",
    "\\hat{y} = \\sigma(\\mathbf{w} \\mathbf{X}^\\top + b)\n",
    "$$\n",
    "\n",
    "The result should be a matrix with **1 row and 50 columns**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 12\n",
    "Put the calculation above in an R function called `neuron` that takes as its input an input matrix `X` and returns the neuron's output. We will use this function for easily recalculating the neuron's output when modifying the weights and bias of the neuron.\n",
    "\n",
    "We have provided you with a skeleton to fill in below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "neuron <- function(X) {\n",
    "    <FILL IN>\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 13\n",
    "Apply the function you defined above to the input matrix `X`.\n",
    "\n",
    "The results will be the probability of the instance belonging to the positive class (the blue one). For classification problems we usually threshold the this probability, with a value larger than $0.5$ meaning the instance is classified as positive (the 'blue' class in our case). Given the probabilities and the labels in `y`, how do you think your classifier is performing?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the results of your prediction with a helper function that we have created for you, called `plot_predictions`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_predictions(X, y,neuron)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, our accuracy is 50% simply because all instances are positive (blue). We need to tweak the decision boundary in order to get 100% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 14\n",
    "Modify the weights `w` and the bias variables `b` such that you classify the instances with 100% accuracy. Plot the probabilities and decision boundary based on your new weights and bias.\n",
    "\n",
    "How do the bias and weights affect the decision boundary?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}