hw 1NumPy Basics and Logistic Regression-Lei Wu 221020509.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before you begin, learn to write equations under Jupyter Markdown mode\n",
    "\n",
    "First, under \"help\" tab, click \"Keyboard Shortcuts\" to view shortcut operations, or click \"Markdown\" to view how to edit markdown cells.\n",
    "\n",
    "To switch between \"Markdown\" mode or \"code\" mode, click a cell you want to change, then under the tool tab (which is right above this cell) click the dropdown option which is to the left of the \"keyboard\" icon. \n",
    "\n",
    "Click Ctrl + Enter to run any cell in any mode.\n",
    "\n",
    "In jupyter you can write equations with Latex format. Simply type equations between two signs \"$\".\n",
    "\n",
    "Double click the cell to view the source code. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part I Basics: Broadcasting and sigmoid function\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Broadcasting\n",
    "\n",
    "Broadcasting in NumPy follows a strict set of rules to determine the interaction between the two arrays:\n",
    "\n",
    "* Rule 1: If the two arrays differ in their number of dimensions, the shape of the one with fewer dimensions is padded with ones on its leading (left) side.\n",
    "* Rule 2: If the shape of the two arrays does not match in any dimension, the array with shape equal to 1 in that dimension is stretched to match the other shape.\n",
    "* Rule 3: If in any dimension the sizes disagree and neither is equal to 1, an error is raised."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([0, 1, 2])\n",
    "b = np.array([5, 5, 5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 6, 7])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* If arrays are of same size, element to element operation takes place\n",
    "* Because of broadcasting, scaler value 5 is converted into vector & ops takes place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5, 6, 7])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a + 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([ [1,2,3], [4,5,6] ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6,  7,  8],\n",
       "       [ 9, 10, 11]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a + 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/awantik/PythonDataScienceHandbook/raw/f2c4a8af3f6e7e5f455469839e31b09ab6c22868/notebooks/figures/02.05-broadcasting.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(3).reshape(3,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0],\n",
       "       [1],\n",
       "       [2]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = np.arange(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 2],\n",
       "       [1, 2, 3],\n",
       "       [2, 3, 4]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets initiate a matrix M and a vector a. Notice that the length of the vector a has to match the last dimension of M (or the dimension 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "M = np.ones((2, 3))\n",
    "a = np.arange(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((2, 3),\n",
       " array([[1., 1., 1.],\n",
       "        [1., 1., 1.]]))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M.shape, M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3,)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2., 3.],\n",
       "       [1., 2., 3.]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M + a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2., 3.],\n",
       "       [1., 2., 3.]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a + M"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([[1,2,3],[4,5,6]])\n",
    "b = np.array([[1,2,3],[4,5,6],[7,8,9]])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "operands could not be broadcast together with shapes (2,3) (3,3) ",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/l2/08sytppn5c5_fqw0ww0zx7740000gn/T/ipykernel_28043/3378120176.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# will this work? why?\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0ma\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m: operands could not be broadcast together with shapes (2,3) (3,3) "
     ]
    }
   ],
   "source": [
    "# will this work? why?\n",
    "a + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = a.reshape(1,2,3)\n",
    "b = b.reshape(3,1,3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[[1, 2, 3],\n",
       "         [4, 5, 6]]]),\n",
       " array([[[1, 2, 3]],\n",
       " \n",
       "        [[4, 5, 6]],\n",
       " \n",
       "        [[7, 8, 9]]]))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a, b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 2,  4,  6],\n",
       "        [ 5,  7,  9]],\n",
       "\n",
       "       [[ 5,  7,  9],\n",
       "        [ 8, 10, 12]],\n",
       "\n",
       "       [[ 8, 10, 12],\n",
       "        [11, 13, 15]]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# will this work? why?\n",
    "\n",
    "a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Practice time !!**\n",
    "\n",
    "First let M be a zero matrix of size 4 * 3, $a$ be an array of size (3, ) with elements ranging from 0 to 2, $b$ be an array of size (4, ) with elements ranging from 1 to 4. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question 1\n",
    "M = np.zeros((4,3))\n",
    "\n",
    "a = np.arange(3)\n",
    "b = np.arange(1,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0. 0.]\n",
      " [0. 0. 0.]\n",
      " [0. 0. 0.]\n",
      " [0. 0. 0.]]\n",
      "-------\n",
      "[0 1 2]\n",
      "-------\n",
      "[1 2 3 4]\n"
     ]
    }
   ],
   "source": [
    "print(M)\n",
    "print(\"-------\")\n",
    "print(a)\n",
    "print(\"-------\")\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's add $a$ to each row of M and assign the resulting matrix to N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 1., 2.],\n",
       "       [0., 1., 2.],\n",
       "       [0., 1., 2.],\n",
       "       [0., 1., 2.]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Question 2\n",
    "N = M + a\n",
    "N"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, add $b$ to each column of N using broadcast. There are multiple ways to do it. You can try np.newaxis or reshape. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question 3\n",
    "N = N+b.reshape((4,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2., 3.],\n",
       "       [2., 3., 4.],\n",
       "       [3., 4., 5.],\n",
       "       [4., 5., 6.]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sigmoid function and np.exp() function\n",
    "\n",
    "\n",
    "**Reminder**:\n",
    "$sigmoid(x) = \\frac{1}{1+e^{-x}}$ is sometimes also known as the logistic function. It is a non-linear function used not only in Machine Learning (Logistic Regression), but also in Deep Learning.\n",
    "\n",
    "<img src=\"images/Sigmoid.png\" style=\"width:500px;height:228px;\">\n",
    "\n",
    "To refer to a function belonging to a specific package you could call it using package_name.function(). Run the code below to see an example with math.exp()."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Implement the sigmoid function and its derivative using numpy. \n",
    "\n",
    "**Instructions**: When you call a function, say exponential on an array, numpy allows for element-wise operation over the whole array.\n",
    "$$ \\text{For } x \\in \\mathbb{R}^n \\text{,     } sigmoid(x) = sigmoid\\begin{pmatrix}\n",
    "    x_1  \\\\\n",
    "    x_2  \\\\\n",
    "    ...  \\\\\n",
    "    x_n  \\\\\n",
    "\\end{pmatrix} = \\begin{pmatrix}\n",
    "    \\frac{1}{1+e^{-x_1}}  \\\\\n",
    "    \\frac{1}{1+e^{-x_2}}  \\\\\n",
    "    ...  \\\\\n",
    "    \\frac{1}{1+e^{-x_n}}  \\\\\n",
    "\\end{pmatrix}\\tag{1} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run this\n",
    "\n",
    "a = np.random.rand(3) # This initiate an random arry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.74149521, 0.54266982, 0.56783204])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = np.exp(-a)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wrtie your code for sigmoid here\n",
    "\n",
    "\n",
    "\n",
    "def sigmoid(x):\n",
    "    \"\"\"\n",
    "    Compute the sigmoid of x\n",
    "\n",
    "    Arguments:\n",
    "    x -- A scalar or numpy array of any size\n",
    "\n",
    "    Return:\n",
    "    s -- sigmoid(x)\n",
    "    \"\"\"\n",
    "    s = 1/(1+ np.exp(-x))\n",
    "    \n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.57421921, 0.64822685, 0.63782342])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Excercise** Define a class named My_Sigmoid Within the class, define two methods forward and backward. The forward method will take an input array $x$ and calculate its sigmoid transformation. The backward will calculate the derivative of sigmoid function at $x$\n",
    "\n",
    "**Instruction** The formula is: $$sigmoid\\_derivative(x) = \\sigma'(x) = \\sigma(x) (1 - \\sigma(x))\\tag{2}$$\n",
    "You often code this function in two steps:\n",
    "1. Set s to be the sigmoid of x. You might find your sigmoid(x) function useful.\n",
    "2. Compute $\\sigma'(x) = s(1-s)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "class My_Sigmoid:\n",
    "    '''\n",
    "    forward: compute the sigmoid(x) given x, x can be a list or a numpy array or a value, if x is a list, then perform element \n",
    "    wise transformation.\n",
    "    \n",
    "    backward: compute the derivative of sigmoid(x), using the fomula above\n",
    "    \n",
    "    '''\n",
    "    def __init__(self, x):\n",
    "        if x is None:\n",
    "            self.y= 0\n",
    "            self.dy = 0\n",
    "        else:\n",
    "            x = np.array(x)\n",
    "            self.y = 1/(np.exp(-x) + 1)\n",
    "            self.dy = self.y*(1-self.y)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = np.array(x)\n",
    "        #######one line code here#######\n",
    "        self.y = 1/(1 + np.exp(-x))\n",
    "        return self.y\n",
    "    def backward(self, x):\n",
    "        x = np.array(x)\n",
    "        #######one line code here#######\n",
    "        self.y = mysigmoid.forward(x)\n",
    "        #######one line code here#######\n",
    "        self.dy = self.y * (1 - self.y)\n",
    "        \n",
    "        \n",
    "        return self.dy\n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9.99954602e-01 4.53978687e-05 5.00000000e-01 8.80797078e-01]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([4.53958077e-05, 4.53958077e-05, 2.50000000e-01, 1.04993585e-01])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test:\n",
    "mysigmoid= My_Sigmoid(x = None)\n",
    "\n",
    "mysigmoid.forward([10, -10, 0, 2])\n",
    "print(mysigmoid.y)\n",
    "mysigmoid.backward([10, -10, 0, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reshaping arrays\n",
    "\n",
    "Two common numpy functions used in deep learning are [np.shape](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.shape.html) and [np.reshape()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html). \n",
    "- X.shape is used to get the shape (dimension) of a matrix/vector X. \n",
    "- X.reshape(...) is used to reshape X into some other dimension. \n",
    "\n",
    "For example, in computer science, an image is represented by a 3D array of shape $(length, height, depth = 3)$. However, when you read an image as the input of an algorithm you convert it to a vector of shape $(length*height*3, 1)$. In other words, you \"unroll\", or reshape, the 3D array into a 1D vector.\n",
    "\n",
    "*Credit for the picture: Andrew Ng Deep Learning*\n",
    "\n",
    "<img src=\"images/image2vector_kiank.png\" style=\"width:500px;height:300;\">\n",
    "\n",
    "**Exercise**: Implement `image2vector()` that takes an input of shape (length, height, 3) and returns a vector of shape (length\\*height\\*3, 1). For example, if you would like to reshape an array v of shape (a, b, c) into a vector of shape (a*b,c) you would do:\n",
    "``` python\n",
    "v = v.reshape((v.shape[0]*v.shape[1], v.shape[2])) # v.shape[0] = a ; v.shape[1] = b ; v.shape[2] = c\n",
    "```\n",
    "- Please don't hardcode the dimensions of image as a constant. Instead look up the quantities you need with `image.shape[0]`, etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: image2vector\n",
    "def image2vector(image):\n",
    "    \"\"\"\n",
    "    Argument:\n",
    "    image -- a numpy array of shape (length, height, depth)\n",
    "    \n",
    "    Returns:\n",
    "    v -- a vector of shape (length*height*depth, 1)\n",
    "    \"\"\"\n",
    "    v = image.reshape((image.shape[0] * image.shape[1] * image.shape[2], 1))\n",
    "    \n",
    "    return v\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "image2vector(image) = [[0.67826139]\n",
      " [0.29380381]\n",
      " [0.90714982]\n",
      " [0.52835647]\n",
      " [0.4215251 ]\n",
      " [0.45017551]\n",
      " [0.92814219]\n",
      " [0.96677647]\n",
      " [0.85304703]\n",
      " [0.52351845]\n",
      " [0.19981397]\n",
      " [0.27417313]\n",
      " [0.60659855]\n",
      " [0.00533165]\n",
      " [0.10820313]\n",
      " [0.49978937]\n",
      " [0.34144279]\n",
      " [0.94630077]]\n"
     ]
    }
   ],
   "source": [
    "# This is a 3 by 3 by 2 array, typically images will be (num_px_x, num_px_y,3) where 3 represents the RGB values\n",
    "image = np.array([[[ 0.67826139,  0.29380381],\n",
    "        [ 0.90714982,  0.52835647],\n",
    "        [ 0.4215251 ,  0.45017551]],\n",
    "\n",
    "       [[ 0.92814219,  0.96677647],\n",
    "        [ 0.85304703,  0.52351845],\n",
    "        [ 0.19981397,  0.27417313]],\n",
    "\n",
    "       [[ 0.60659855,  0.00533165],\n",
    "        [ 0.10820313,  0.49978937],\n",
    "        [ 0.34144279,  0.94630077]]])\n",
    "\n",
    "vectorized_image = image2vector(image)\n",
    "\n",
    "print (\"image2vector(image) = \" + str(vectorized_image))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**: Implement `vector2image()` that takes the above vector and reshape it back to an image, but this time of the shape (length\\*height ,depth)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vector2image(vec, shape):\n",
    "    \"\"\"\n",
    "    Argument:\n",
    "    vec -- a flattened image vector of shape (length*height*depth, 1)\n",
    "    shape -- a array indicating desired shape\n",
    "    \n",
    "    Returns:\n",
    "    v -- a vector of shape (length*height*depth, 1)\n",
    "    \"\"\"\n",
    "    image = vec.reshape(shape[0], shape[1])\n",
    "    \n",
    "    return image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vector2image(vectorized_image ) = [[0.67826139 0.29380381]\n",
      " [0.90714982 0.52835647]\n",
      " [0.4215251  0.45017551]\n",
      " [0.92814219 0.96677647]\n",
      " [0.85304703 0.52351845]\n",
      " [0.19981397 0.27417313]\n",
      " [0.60659855 0.00533165]\n",
      " [0.10820313 0.49978937]\n",
      " [0.34144279 0.94630077]]\n"
     ]
    }
   ],
   "source": [
    "# call the function, fill in the right shape\n",
    "\n",
    "image_new = vector2image(vectorized_image, shape =(image.shape[0] * image.shape[1] , image.shape[2]) )\n",
    "print (\"vector2image(vectorized_image ) = \" + str(image_new))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part II Logistic Regression with a Neural Network mindset\n",
    "\n",
    "This assignment will step you through how to do this with a Neural Network mindset, and so will also hone your intuitions about deep learning.\n",
    "\n",
    "**Instructions:**\n",
    "- Do not use loops (for/while) in your code, unless the instructions explicitly ask you to do so.\n",
    "\n",
    "\n",
    "Implement  a gradient descent algorithm for logistic regression .This data are taken from a larger dataset, described in a South African Medical Journal.\n",
    "\n",
    "**Dataset: Use the following dataset for the implementation.**\n",
    "\n",
    "The full data is available at: http://statweb.stanford.edu/~tibs/ElemStatLearn/datasets/ For this project, you can use those located under in the folder. \n",
    "\n",
    "\n",
    "#### Description\n",
    "A retrospective sample of males in a heart-disease high-risk region of South Africa. There are roughly two controls per case of CHD. Many of the CHD positive men have undergone blood pressure reduction treatment and other programs to reduce their risk factors after their CHD event. In some cases, the measurements were made after these treatments.\n",
    " \n",
    "### Steps to implement\n",
    "\n",
    "*\tEncode the categorical variables.\n",
    "*\tNormalize the numerical variables\n",
    "*\tRandomly initialize beta values\n",
    "*\tDefine a sigmoid function to predict Y\n",
    "* Define a function for calculating binary cross entropy loss function \n",
    "* Define a function for updating beta values. The derivative term is same as derivative term for the linear regression as discussed in the class.\n",
    "* Write the code for gradient descent iterations.\n",
    "* Plot the cost function for different alpha (learning parameters) values.\n",
    "* Use sklearn logistic regression API and compare the estimation of beta values.\n",
    "\n",
    "\n",
    "### Important! \n",
    "\n",
    "Read the codes and comments carefully. You may need to fill in the code when you see \"################finish the code below##################\" in the comment line. Do not miss them.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Packages ##\n",
    "\n",
    "\n",
    "If you don't have those packages, try pip/conda install *-insert package name-*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load required libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Overview of the Problem set ##\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load SAHeart csv file\n",
    "SAHeart_df = pd.read_csv('SAHeart.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row.names</th>\n",
       "      <th>sbp</th>\n",
       "      <th>tobacco</th>\n",
       "      <th>ldl</th>\n",
       "      <th>adiposity</th>\n",
       "      <th>famhist</th>\n",
       "      <th>typea</th>\n",
       "      <th>obesity</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>age</th>\n",
       "      <th>chd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>160</td>\n",
       "      <td>12.00</td>\n",
       "      <td>5.73</td>\n",
       "      <td>23.11</td>\n",
       "      <td>Present</td>\n",
       "      <td>49</td>\n",
       "      <td>25.30</td>\n",
       "      <td>97.20</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>144</td>\n",
       "      <td>0.01</td>\n",
       "      <td>4.41</td>\n",
       "      <td>28.61</td>\n",
       "      <td>Absent</td>\n",
       "      <td>55</td>\n",
       "      <td>28.87</td>\n",
       "      <td>2.06</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>118</td>\n",
       "      <td>0.08</td>\n",
       "      <td>3.48</td>\n",
       "      <td>32.28</td>\n",
       "      <td>Present</td>\n",
       "      <td>52</td>\n",
       "      <td>29.14</td>\n",
       "      <td>3.81</td>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>170</td>\n",
       "      <td>7.50</td>\n",
       "      <td>6.41</td>\n",
       "      <td>38.03</td>\n",
       "      <td>Present</td>\n",
       "      <td>51</td>\n",
       "      <td>31.99</td>\n",
       "      <td>24.26</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>134</td>\n",
       "      <td>13.60</td>\n",
       "      <td>3.50</td>\n",
       "      <td>27.78</td>\n",
       "      <td>Present</td>\n",
       "      <td>60</td>\n",
       "      <td>25.99</td>\n",
       "      <td>57.34</td>\n",
       "      <td>49</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>132</td>\n",
       "      <td>6.20</td>\n",
       "      <td>6.47</td>\n",
       "      <td>36.21</td>\n",
       "      <td>Present</td>\n",
       "      <td>62</td>\n",
       "      <td>30.77</td>\n",
       "      <td>14.14</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>142</td>\n",
       "      <td>4.05</td>\n",
       "      <td>3.38</td>\n",
       "      <td>16.20</td>\n",
       "      <td>Absent</td>\n",
       "      <td>59</td>\n",
       "      <td>20.81</td>\n",
       "      <td>2.62</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>114</td>\n",
       "      <td>4.08</td>\n",
       "      <td>4.59</td>\n",
       "      <td>14.60</td>\n",
       "      <td>Present</td>\n",
       "      <td>62</td>\n",
       "      <td>23.11</td>\n",
       "      <td>6.72</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>114</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.83</td>\n",
       "      <td>19.40</td>\n",
       "      <td>Present</td>\n",
       "      <td>49</td>\n",
       "      <td>24.86</td>\n",
       "      <td>2.49</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>132</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.80</td>\n",
       "      <td>30.96</td>\n",
       "      <td>Present</td>\n",
       "      <td>69</td>\n",
       "      <td>30.11</td>\n",
       "      <td>0.00</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   row.names  sbp  tobacco   ldl  adiposity  famhist  typea  obesity  alcohol  \\\n",
       "0          1  160    12.00  5.73      23.11  Present     49    25.30    97.20   \n",
       "1          2  144     0.01  4.41      28.61   Absent     55    28.87     2.06   \n",
       "2          3  118     0.08  3.48      32.28  Present     52    29.14     3.81   \n",
       "3          4  170     7.50  6.41      38.03  Present     51    31.99    24.26   \n",
       "4          5  134    13.60  3.50      27.78  Present     60    25.99    57.34   \n",
       "5          6  132     6.20  6.47      36.21  Present     62    30.77    14.14   \n",
       "6          7  142     4.05  3.38      16.20   Absent     59    20.81     2.62   \n",
       "7          8  114     4.08  4.59      14.60  Present     62    23.11     6.72   \n",
       "8          9  114     0.00  3.83      19.40  Present     49    24.86     2.49   \n",
       "9         10  132     0.00  5.80      30.96  Present     69    30.11     0.00   \n",
       "\n",
       "   age  chd  \n",
       "0   52    1  \n",
       "1   63    1  \n",
       "2   46    0  \n",
       "3   58    1  \n",
       "4   49    1  \n",
       "5   45    0  \n",
       "6   38    0  \n",
       "7   58    1  \n",
       "8   29    0  \n",
       "9   53    1  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SAHeart_df.head(10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row.names</th>\n",
       "      <th>sbp</th>\n",
       "      <th>tobacco</th>\n",
       "      <th>ldl</th>\n",
       "      <th>adiposity</th>\n",
       "      <th>famhist</th>\n",
       "      <th>typea</th>\n",
       "      <th>obesity</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>age</th>\n",
       "      <th>chd</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>452</th>\n",
       "      <td>454</td>\n",
       "      <td>154</td>\n",
       "      <td>5.53</td>\n",
       "      <td>3.20</td>\n",
       "      <td>28.81</td>\n",
       "      <td>Present</td>\n",
       "      <td>61</td>\n",
       "      <td>26.15</td>\n",
       "      <td>42.79</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>453</th>\n",
       "      <td>455</td>\n",
       "      <td>124</td>\n",
       "      <td>1.60</td>\n",
       "      <td>7.22</td>\n",
       "      <td>39.68</td>\n",
       "      <td>Present</td>\n",
       "      <td>36</td>\n",
       "      <td>31.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>51</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>454</th>\n",
       "      <td>456</td>\n",
       "      <td>146</td>\n",
       "      <td>0.64</td>\n",
       "      <td>4.82</td>\n",
       "      <td>28.02</td>\n",
       "      <td>Absent</td>\n",
       "      <td>60</td>\n",
       "      <td>28.11</td>\n",
       "      <td>8.23</td>\n",
       "      <td>39</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>455</th>\n",
       "      <td>457</td>\n",
       "      <td>128</td>\n",
       "      <td>2.24</td>\n",
       "      <td>2.83</td>\n",
       "      <td>26.48</td>\n",
       "      <td>Absent</td>\n",
       "      <td>48</td>\n",
       "      <td>23.96</td>\n",
       "      <td>47.42</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>456</th>\n",
       "      <td>458</td>\n",
       "      <td>170</td>\n",
       "      <td>0.40</td>\n",
       "      <td>4.11</td>\n",
       "      <td>42.06</td>\n",
       "      <td>Present</td>\n",
       "      <td>56</td>\n",
       "      <td>33.10</td>\n",
       "      <td>2.06</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>457</th>\n",
       "      <td>459</td>\n",
       "      <td>214</td>\n",
       "      <td>0.40</td>\n",
       "      <td>5.98</td>\n",
       "      <td>31.72</td>\n",
       "      <td>Absent</td>\n",
       "      <td>64</td>\n",
       "      <td>28.45</td>\n",
       "      <td>0.00</td>\n",
       "      <td>58</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>458</th>\n",
       "      <td>460</td>\n",
       "      <td>182</td>\n",
       "      <td>4.20</td>\n",
       "      <td>4.41</td>\n",
       "      <td>32.10</td>\n",
       "      <td>Absent</td>\n",
       "      <td>52</td>\n",
       "      <td>28.61</td>\n",
       "      <td>18.72</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>461</td>\n",
       "      <td>108</td>\n",
       "      <td>3.00</td>\n",
       "      <td>1.59</td>\n",
       "      <td>15.23</td>\n",
       "      <td>Absent</td>\n",
       "      <td>40</td>\n",
       "      <td>20.09</td>\n",
       "      <td>26.64</td>\n",
       "      <td>55</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>460</th>\n",
       "      <td>462</td>\n",
       "      <td>118</td>\n",
       "      <td>5.40</td>\n",
       "      <td>11.61</td>\n",
       "      <td>30.79</td>\n",
       "      <td>Absent</td>\n",
       "      <td>64</td>\n",
       "      <td>27.35</td>\n",
       "      <td>23.97</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>461</th>\n",
       "      <td>463</td>\n",
       "      <td>132</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.82</td>\n",
       "      <td>33.41</td>\n",
       "      <td>Present</td>\n",
       "      <td>62</td>\n",
       "      <td>14.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>46</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     row.names  sbp  tobacco    ldl  adiposity  famhist  typea  obesity  \\\n",
       "452        454  154     5.53   3.20      28.81  Present     61    26.15   \n",
       "453        455  124     1.60   7.22      39.68  Present     36    31.50   \n",
       "454        456  146     0.64   4.82      28.02   Absent     60    28.11   \n",
       "455        457  128     2.24   2.83      26.48   Absent     48    23.96   \n",
       "456        458  170     0.40   4.11      42.06  Present     56    33.10   \n",
       "457        459  214     0.40   5.98      31.72   Absent     64    28.45   \n",
       "458        460  182     4.20   4.41      32.10   Absent     52    28.61   \n",
       "459        461  108     3.00   1.59      15.23   Absent     40    20.09   \n",
       "460        462  118     5.40  11.61      30.79   Absent     64    27.35   \n",
       "461        463  132     0.00   4.82      33.41  Present     62    14.70   \n",
       "\n",
       "     alcohol  age  chd  \n",
       "452    42.79   42    0  \n",
       "453     0.00   51    1  \n",
       "454     8.23   39    1  \n",
       "455    47.42   27    1  \n",
       "456     2.06   57    0  \n",
       "457     0.00   58    0  \n",
       "458    18.72   52    1  \n",
       "459    26.64   55    0  \n",
       "460    23.97   40    0  \n",
       "461     0.00   46    1  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SAHeart_df.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Process variables ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert 'famhis' to dummies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "SAHeart_df= SAHeart_df.join(pd.get_dummies(SAHeart_df[\"famhist\"]))\n",
    "SAHeart_df = SAHeart_df.drop(labels=['famhist', 'Present'],axis=1)\n",
    "# If famhist = 'Absent', then famhist = 1, otherwise famhist = 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row.names</th>\n",
       "      <th>sbp</th>\n",
       "      <th>tobacco</th>\n",
       "      <th>ldl</th>\n",
       "      <th>adiposity</th>\n",
       "      <th>typea</th>\n",
       "      <th>obesity</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>age</th>\n",
       "      <th>chd</th>\n",
       "      <th>Absent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>457</th>\n",
       "      <td>459</td>\n",
       "      <td>214</td>\n",
       "      <td>0.4</td>\n",
       "      <td>5.98</td>\n",
       "      <td>31.72</td>\n",
       "      <td>64</td>\n",
       "      <td>28.45</td>\n",
       "      <td>0.00</td>\n",
       "      <td>58</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>458</th>\n",
       "      <td>460</td>\n",
       "      <td>182</td>\n",
       "      <td>4.2</td>\n",
       "      <td>4.41</td>\n",
       "      <td>32.10</td>\n",
       "      <td>52</td>\n",
       "      <td>28.61</td>\n",
       "      <td>18.72</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>461</td>\n",
       "      <td>108</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.59</td>\n",
       "      <td>15.23</td>\n",
       "      <td>40</td>\n",
       "      <td>20.09</td>\n",
       "      <td>26.64</td>\n",
       "      <td>55</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>460</th>\n",
       "      <td>462</td>\n",
       "      <td>118</td>\n",
       "      <td>5.4</td>\n",
       "      <td>11.61</td>\n",
       "      <td>30.79</td>\n",
       "      <td>64</td>\n",
       "      <td>27.35</td>\n",
       "      <td>23.97</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>461</th>\n",
       "      <td>463</td>\n",
       "      <td>132</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.82</td>\n",
       "      <td>33.41</td>\n",
       "      <td>62</td>\n",
       "      <td>14.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>46</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     row.names  sbp  tobacco    ldl  adiposity  typea  obesity  alcohol  age  \\\n",
       "457        459  214      0.4   5.98      31.72     64    28.45     0.00   58   \n",
       "458        460  182      4.2   4.41      32.10     52    28.61    18.72   52   \n",
       "459        461  108      3.0   1.59      15.23     40    20.09    26.64   55   \n",
       "460        462  118      5.4  11.61      30.79     64    27.35    23.97   40   \n",
       "461        463  132      0.0   4.82      33.41     62    14.70     0.00   46   \n",
       "\n",
       "     chd  Absent  \n",
       "457    0       1  \n",
       "458    1       1  \n",
       "459    0       1  \n",
       "460    0       1  \n",
       "461    1       0  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SAHeart_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename the columns label. Change 'Absent' to 'famhist' dummy variable.\n",
    "SAHeart_df.rename(columns = {'Absent' : 'famhist'}, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>row.names</th>\n",
       "      <th>sbp</th>\n",
       "      <th>tobacco</th>\n",
       "      <th>ldl</th>\n",
       "      <th>adiposity</th>\n",
       "      <th>typea</th>\n",
       "      <th>obesity</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>age</th>\n",
       "      <th>chd</th>\n",
       "      <th>famhist</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>160</td>\n",
       "      <td>12.00</td>\n",
       "      <td>5.73</td>\n",
       "      <td>23.11</td>\n",
       "      <td>49</td>\n",
       "      <td>25.30</td>\n",
       "      <td>97.20</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>144</td>\n",
       "      <td>0.01</td>\n",
       "      <td>4.41</td>\n",
       "      <td>28.61</td>\n",
       "      <td>55</td>\n",
       "      <td>28.87</td>\n",
       "      <td>2.06</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>118</td>\n",
       "      <td>0.08</td>\n",
       "      <td>3.48</td>\n",
       "      <td>32.28</td>\n",
       "      <td>52</td>\n",
       "      <td>29.14</td>\n",
       "      <td>3.81</td>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>170</td>\n",
       "      <td>7.50</td>\n",
       "      <td>6.41</td>\n",
       "      <td>38.03</td>\n",
       "      <td>51</td>\n",
       "      <td>31.99</td>\n",
       "      <td>24.26</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>134</td>\n",
       "      <td>13.60</td>\n",
       "      <td>3.50</td>\n",
       "      <td>27.78</td>\n",
       "      <td>60</td>\n",
       "      <td>25.99</td>\n",
       "      <td>57.34</td>\n",
       "      <td>49</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>132</td>\n",
       "      <td>6.20</td>\n",
       "      <td>6.47</td>\n",
       "      <td>36.21</td>\n",
       "      <td>62</td>\n",
       "      <td>30.77</td>\n",
       "      <td>14.14</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>142</td>\n",
       "      <td>4.05</td>\n",
       "      <td>3.38</td>\n",
       "      <td>16.20</td>\n",
       "      <td>59</td>\n",
       "      <td>20.81</td>\n",
       "      <td>2.62</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>114</td>\n",
       "      <td>4.08</td>\n",
       "      <td>4.59</td>\n",
       "      <td>14.60</td>\n",
       "      <td>62</td>\n",
       "      <td>23.11</td>\n",
       "      <td>6.72</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>114</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.83</td>\n",
       "      <td>19.40</td>\n",
       "      <td>49</td>\n",
       "      <td>24.86</td>\n",
       "      <td>2.49</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>132</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.80</td>\n",
       "      <td>30.96</td>\n",
       "      <td>69</td>\n",
       "      <td>30.11</td>\n",
       "      <td>0.00</td>\n",
       "      <td>53</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   row.names  sbp  tobacco   ldl  adiposity  typea  obesity  alcohol  age  \\\n",
       "0          1  160    12.00  5.73      23.11     49    25.30    97.20   52   \n",
       "1          2  144     0.01  4.41      28.61     55    28.87     2.06   63   \n",
       "2          3  118     0.08  3.48      32.28     52    29.14     3.81   46   \n",
       "3          4  170     7.50  6.41      38.03     51    31.99    24.26   58   \n",
       "4          5  134    13.60  3.50      27.78     60    25.99    57.34   49   \n",
       "5          6  132     6.20  6.47      36.21     62    30.77    14.14   45   \n",
       "6          7  142     4.05  3.38      16.20     59    20.81     2.62   38   \n",
       "7          8  114     4.08  4.59      14.60     62    23.11     6.72   58   \n",
       "8          9  114     0.00  3.83      19.40     49    24.86     2.49   29   \n",
       "9         10  132     0.00  5.80      30.96     69    30.11     0.00   53   \n",
       "\n",
       "   chd  famhist  \n",
       "0    1        0  \n",
       "1    1        1  \n",
       "2    0        0  \n",
       "3    1        0  \n",
       "4    1        0  \n",
       "5    0        0  \n",
       "6    0        1  \n",
       "7    1        0  \n",
       "8    0        0  \n",
       "9    1        0  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SAHeart_df.head(10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = (SAHeart_df[['famhist','sbp','tobacco','ldl','adiposity','typea','obesity','alcohol','age']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>famhist</th>\n",
       "      <th>sbp</th>\n",
       "      <th>tobacco</th>\n",
       "      <th>ldl</th>\n",
       "      <th>adiposity</th>\n",
       "      <th>typea</th>\n",
       "      <th>obesity</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>160</td>\n",
       "      <td>12.00</td>\n",
       "      <td>5.73</td>\n",
       "      <td>23.11</td>\n",
       "      <td>49</td>\n",
       "      <td>25.30</td>\n",
       "      <td>97.20</td>\n",
       "      <td>52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>144</td>\n",
       "      <td>0.01</td>\n",
       "      <td>4.41</td>\n",
       "      <td>28.61</td>\n",
       "      <td>55</td>\n",
       "      <td>28.87</td>\n",
       "      <td>2.06</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>118</td>\n",
       "      <td>0.08</td>\n",
       "      <td>3.48</td>\n",
       "      <td>32.28</td>\n",
       "      <td>52</td>\n",
       "      <td>29.14</td>\n",
       "      <td>3.81</td>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>170</td>\n",
       "      <td>7.50</td>\n",
       "      <td>6.41</td>\n",
       "      <td>38.03</td>\n",
       "      <td>51</td>\n",
       "      <td>31.99</td>\n",
       "      <td>24.26</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>134</td>\n",
       "      <td>13.60</td>\n",
       "      <td>3.50</td>\n",
       "      <td>27.78</td>\n",
       "      <td>60</td>\n",
       "      <td>25.99</td>\n",
       "      <td>57.34</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   famhist  sbp  tobacco   ldl  adiposity  typea  obesity  alcohol  age\n",
       "0        0  160    12.00  5.73      23.11     49    25.30    97.20   52\n",
       "1        1  144     0.01  4.41      28.61     55    28.87     2.06   63\n",
       "2        0  118     0.08  3.48      32.28     52    29.14     3.81   46\n",
       "3        0  170     7.50  6.41      38.03     51    31.99    24.26   58\n",
       "4        0  134    13.60  3.50      27.78     60    25.99    57.34   49"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0:5]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = np.array(SAHeart_df[['chd']])\n",
    "#Y = (SAHeart_df[['chd']])\n",
    "#print(len(Y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1]])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sbp</th>\n",
       "      <th>tobacco</th>\n",
       "      <th>ldl</th>\n",
       "      <th>adiposity</th>\n",
       "      <th>typea</th>\n",
       "      <th>obesity</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>160</td>\n",
       "      <td>12.00</td>\n",
       "      <td>5.73</td>\n",
       "      <td>23.11</td>\n",
       "      <td>49</td>\n",
       "      <td>25.30</td>\n",
       "      <td>97.20</td>\n",
       "      <td>52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>144</td>\n",
       "      <td>0.01</td>\n",
       "      <td>4.41</td>\n",
       "      <td>28.61</td>\n",
       "      <td>55</td>\n",
       "      <td>28.87</td>\n",
       "      <td>2.06</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>118</td>\n",
       "      <td>0.08</td>\n",
       "      <td>3.48</td>\n",
       "      <td>32.28</td>\n",
       "      <td>52</td>\n",
       "      <td>29.14</td>\n",
       "      <td>3.81</td>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>170</td>\n",
       "      <td>7.50</td>\n",
       "      <td>6.41</td>\n",
       "      <td>38.03</td>\n",
       "      <td>51</td>\n",
       "      <td>31.99</td>\n",
       "      <td>24.26</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>134</td>\n",
       "      <td>13.60</td>\n",
       "      <td>3.50</td>\n",
       "      <td>27.78</td>\n",
       "      <td>60</td>\n",
       "      <td>25.99</td>\n",
       "      <td>57.34</td>\n",
       "      <td>49</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>132</td>\n",
       "      <td>6.20</td>\n",
       "      <td>6.47</td>\n",
       "      <td>36.21</td>\n",
       "      <td>62</td>\n",
       "      <td>30.77</td>\n",
       "      <td>14.14</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>142</td>\n",
       "      <td>4.05</td>\n",
       "      <td>3.38</td>\n",
       "      <td>16.20</td>\n",
       "      <td>59</td>\n",
       "      <td>20.81</td>\n",
       "      <td>2.62</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>114</td>\n",
       "      <td>4.08</td>\n",
       "      <td>4.59</td>\n",
       "      <td>14.60</td>\n",
       "      <td>62</td>\n",
       "      <td>23.11</td>\n",
       "      <td>6.72</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>114</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.83</td>\n",
       "      <td>19.40</td>\n",
       "      <td>49</td>\n",
       "      <td>24.86</td>\n",
       "      <td>2.49</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>132</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.80</td>\n",
       "      <td>30.96</td>\n",
       "      <td>69</td>\n",
       "      <td>30.11</td>\n",
       "      <td>0.00</td>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sbp  tobacco   ldl  adiposity  typea  obesity  alcohol  age\n",
       "0  160    12.00  5.73      23.11     49    25.30    97.20   52\n",
       "1  144     0.01  4.41      28.61     55    28.87     2.06   63\n",
       "2  118     0.08  3.48      32.28     52    29.14     3.81   46\n",
       "3  170     7.50  6.41      38.03     51    31.99    24.26   58\n",
       "4  134    13.60  3.50      27.78     60    25.99    57.34   49\n",
       "5  132     6.20  6.47      36.21     62    30.77    14.14   45\n",
       "6  142     4.05  3.38      16.20     59    20.81     2.62   38\n",
       "7  114     4.08  4.59      14.60     62    23.11     6.72   58\n",
       "8  114     0.00  3.83      19.40     49    24.86     2.49   29\n",
       "9  132     0.00  5.80      30.96     69    30.11     0.00   53"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols_to_Normal = ['sbp','tobacco','ldl','adiposity','typea','obesity','alcohol','age']\n",
    "\n",
    "X_new= X[cols_to_Normal]\n",
    "X_new.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Normalise the dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = X_new[cols_to_Normal].apply( lambda rec:(rec - rec.mean())/rec.std(), axis =0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "type(X_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new['famhist'] = SAHeart_df['famhist']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['sbp',\n",
       " 'tobacco',\n",
       " 'ldl',\n",
       " 'adiposity',\n",
       " 'typea',\n",
       " 'obesity',\n",
       " 'alcohol',\n",
       " 'age',\n",
       " 'famhist']"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new.head(10)\n",
    "list(X_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.05741729,  1.8210988 ,  0.47789413, -0.29518321, -0.41801699,\n",
       "        -0.17659445,  3.27418871,  0.62865426,  0.        ],\n",
       "       [ 0.27678925, -0.78938174, -0.15950708,  0.41169419,  0.19313443,\n",
       "         0.67064592, -0.61208112,  1.38161701,  1.        ],\n",
       "       [-0.99173133, -0.77414124, -0.6085852 ,  0.8833742 , -0.11244128,\n",
       "         0.73472292, -0.54059729,  0.2179473 ,  0.        ],\n",
       "       [ 1.54530982,  0.84135214,  0.80625232,  1.62238239, -0.21429985,\n",
       "         1.41109128,  0.2947424 ,  1.03936121,  0.        ],\n",
       "       [-0.21110328,  2.16945317, -0.59892761,  0.30501996,  0.70242729,\n",
       "        -0.01284211,  1.64599115,  0.42330078,  0.        ]])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new = np.array(X_new)\n",
    "X_new[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(462, 9)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4- Randomly Initialising values of beta coefficient ## "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialize_betas(dim):\n",
    "    b = random.random()\n",
    "    w = np.random.rand(dim)\n",
    "    return b,w \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5685537051940374 [0.68286358 0.3776644  0.69823257 0.23433112 0.61337348 0.81984558\n",
      " 0.77348627 0.10434849 0.8914872 ]\n"
     ]
    }
   ],
   "source": [
    "b,w = initialize_betas(X_new.shape[1])\n",
    "print(b,w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9,)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "w.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5- Define a sigmoid function to predict Y ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(b, w ,X_new):\n",
    "    '''\n",
    "    Z= wx+b\n",
    "    '''\n",
    "    \n",
    "    ##############finish the code here#####################\n",
    "    Z = np.dot(X_new, w ) + b\n",
    "    Sigmoid_Z = 1 / (1 + np.exp(-Z))\n",
    "    return  Sigmoid_Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.98833996, 0.84224815, 0.38202256, 0.98588819, 0.93305877])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat = sigmoid(b,w,X_new)\n",
    "\n",
    "y_hat[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "Y[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1],\n",
       "       [1],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1]])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6- Define a function for calculating binary cross entropy loss function ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cost( y, y_hat):\n",
    "    \n",
    "    '''\n",
    "    Input: y: a vector of true labels, y_hat: a vector of predicted outcome from logistic regression\n",
    "    Return: the value of cost or loss function\n",
    "    Loss function L = -1/m\\Sigma (y log y_hat + (1-y)log 1-y_hat)\n",
    "    \n",
    "    \n",
    "    '''\n",
    "    \n",
    "    ### finish the code here\n",
    "    loss = - np.sum(y * np.log(y_hat) + (1-y) * np.log(1-y_hat))/y.shape[0]\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((462,), (462,))"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reshape Y\n",
    "Y=Y.reshape(len(Y), )\n",
    "current_cost= get_cost(Y,y_hat)\n",
    "#print(current_cost)\n",
    "Y.shape, y_hat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(462, 9)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7- Define a function for updating w and b using backpropagation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write backpropagation function here\n",
    "\n",
    "def backprop (b_0, w_0 , y , y_hat, X_new, alpha = 0.1):\n",
    "    '''\n",
    "    Input: weight to be updated b_0, and w_0, as well as y and y_hat to calculate dz (defined as dL/dz)\n",
    "    output: new updated b_0 and w_0\n",
    "    \n",
    "    \n",
    "    '''\n",
    "    dz = y_hat-y\n",
    "    ########finish the code here, to calculate db defined as dL/db\n",
    "    db = np.sum(dz)/y.shape[0]\n",
    "    \n",
    "    # update b_0\n",
    "    b_0 = b_0 - alpha * db\n",
    "    \n",
    "    \n",
    "    ########finish the code here, , to calculate db defined as dL/db\n",
    "    dw = np.dot(dz.T, X_new)/y.shape[0]\n",
    "    \n",
    "    #update w_0\n",
    "    w_0 = w_0 - alpha * dw\n",
    "   \n",
    "   \n",
    "    return b_0,w_0\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(462, 9)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alpha = 0.1\n",
    "\n",
    "type(y_hat)\n",
    "#(X_new).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "()"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b,w = backprop (b,w, Y, y_hat,X_new, alpha)\n",
    "\n",
    "b.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5392337331886381"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.67689848, 0.37935624, 0.69444912, 0.22433213, 0.60933634,\n",
       "       0.80401551, 0.76421198, 0.10613973, 0.8681401 ])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#print(b,w)\n",
    "w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  8- Finally, let's try the gradient descent!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initial guess of b and w:  0.8135413952865866 [0.67467834 0.44799528 0.75524346 0.8193404  0.87323671 0.92205084\n",
      " 0.90368817 0.20712673 0.08040002]\n",
      "Iteration:  0 Cost:  1.1235819658179953\n",
      "Iteration:  10 Cost:  0.589786959902852\n",
      "Iteration:  20 Cost:  0.5223673728628698\n",
      "Iteration:  30 Cost:  0.5140342593297114\n",
      "Iteration:  40 Cost:  0.5121161120787974\n",
      "Iteration:  50 Cost:  0.5114695780462635\n",
      "Iteration:  60 Cost:  0.5112051071115653\n",
      "Iteration:  70 Cost:  0.5110870479240975\n",
      "Iteration:  80 Cost:  0.511031875208611\n",
      "Iteration:  90 Cost:  0.5110051615718628\n",
      "Iteration:  100 Cost:  0.5109917310585275\n",
      "Iteration:  110 Cost:  0.510984671812462\n",
      "Iteration:  120 Cost:  0.5109807659628048\n",
      "Iteration:  130 Cost:  0.5109784828573557\n",
      "Iteration:  140 Cost:  0.5109770751502772\n",
      "Iteration:  150 Cost:  0.5109761654535522\n",
      "Iteration:  160 Cost:  0.5109755549327676\n",
      "Iteration:  170 Cost:  0.5109751334401028\n",
      "Iteration:  180 Cost:  0.5109748365605642\n",
      "Iteration:  190 Cost:  0.5109746245780742\n",
      "Iteration:  200 Cost:  0.5109744718347239\n",
      "Iteration:  210 Cost:  0.5109743611185386\n",
      "Iteration:  220 Cost:  0.5109742805528407\n",
      "Iteration:  230 Cost:  0.5109742217768698\n",
      "Iteration:  240 Cost:  0.510974178824144\n",
      "Iteration:  250 Cost:  0.5109741473980441\n",
      "Iteration:  260 Cost:  0.510974124386037\n",
      "Iteration:  270 Cost:  0.5109741075246164\n",
      "Iteration:  280 Cost:  0.5109740951635379\n",
      "Iteration:  290 Cost:  0.5109740860976472\n",
      "Iteration:  300 Cost:  0.5109740794458333\n",
      "Iteration:  310 Cost:  0.5109740745633824\n",
      "Iteration:  320 Cost:  0.5109740709782742\n",
      "Iteration:  330 Cost:  0.5109740683447616\n",
      "Iteration:  340 Cost:  0.5109740664094901\n",
      "Iteration:  350 Cost:  0.5109740649867422\n",
      "Iteration:  360 Cost:  0.5109740639403346\n",
      "Iteration:  370 Cost:  0.5109740631703735\n",
      "Iteration:  380 Cost:  0.5109740626035596\n",
      "Iteration:  390 Cost:  0.51097406218609\n",
      "Iteration:  400 Cost:  0.510974061878458\n",
      "Iteration:  410 Cost:  0.5109740616516435\n",
      "Iteration:  420 Cost:  0.5109740614843221\n",
      "Iteration:  430 Cost:  0.510974061360817\n",
      "Iteration:  440 Cost:  0.510974061269599\n",
      "Iteration:  450 Cost:  0.5109740612021851\n",
      "Iteration:  460 Cost:  0.5109740611523309\n",
      "Iteration:  470 Cost:  0.5109740611154376\n",
      "Iteration:  480 Cost:  0.5109740610881165\n",
      "Iteration:  490 Cost:  0.5109740610678694\n",
      "Final estimates of b and q are:  -0.3377722761339356 [ 0.13330635  0.36457426  0.36018352  0.14460336  0.38872598 -0.26507213\n",
      "  0.00298149  0.66070866 -0.92532871]\n"
     ]
    }
   ],
   "source": [
    "num_iterations = 500\n",
    "alpha = 0.5\n",
    "\n",
    "all_costs = []\n",
    "b,w = initialize_betas(X_new.shape[1])\n",
    "print(\"initial guess of b and w: \" , b ,w)\n",
    "\n",
    "for each_iter in range (num_iterations ):\n",
    "    ###### finish the code below\n",
    "    y_hat = sigmoid(b,w,X_new)\n",
    "    current_cost = get_cost(Y,y_hat)\n",
    "    prev_b = b\n",
    "    prev_w = w\n",
    "    b, w = backprop (prev_b, prev_w, Y, y_hat,X_new, alpha)\n",
    "    all_costs.append(current_cost)\n",
    "    if each_iter % 10 == 0:\n",
    "        print('Iteration: ', each_iter, 'Cost: ', current_cost)\n",
    "        each_iter += 1\n",
    "    \n",
    "#print('b_0:', b_0, 'b_1:',b_1,'b_2:',b_2,'b_3:',b_3,'b_4:', b_4, 'b_5:',b_5,'b_6:',b_6,'b_7:',b_7,'b_8:',b_8,'b_9:',b_9)\n",
    "print(\"Final estimates of b and q are: \", b,w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9- Let's validate our result against sklearn logistic regression API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "lreg = LogisticRegression()\n",
    "lmodel = lreg.fit(X_new, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.35563668])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lmodel.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.13250588,  0.36074464,  0.3561588 ,  0.14243272,  0.37963858,\n",
       "        -0.25523773,  0.00444212,  0.64962484, -0.88057699]])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lmodel.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10- Visualization using matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sn\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f7efe2c8a60>"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAApzUlEQVR4nO3df3TU1Z3/8dfMkGQoTQYDkkwkwEiRJaSyTWgwodil1kjUtJzu2dK6+Kvab2Hrj5jac4xsm4b1nLi7XQ52lVgr1FL8Hjhb1K98xRyz2/LDsj0pCflKjKVaokGcNA3RSfyRBCf3+wdmliEJZIbJ3CTzfJzzOadz534y77l6nFc/93Pvx2GMMQIAALDEabsAAACQ2AgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKyaYruA0RgYGNA777yj1NRUORwO2+UAAIBRMMaop6dHWVlZcjpHvv4xIcLIO++8o+zsbNtlAACAKJw4cUKzZ88e8f0JEUZSU1MlnfkyaWlplqsBAACj0d3drezs7NDv+EgmRBgZnJpJS0sjjAAAMMFc6BYLbmAFAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWDUhNj0bC8EBo/rWLnX09GpWqlsFvnS5nDz3BgCAeEvIMFLb7FfVnhb5A72hNq/HrcrSHK3K9VqsDACAxJNw0zS1zX6t39EYFkQkqT3Qq/U7GlXb7LdUGQAAiSmhwkhwwKhqT4vMMO8NtlXtaVFwYLgeAABgLCRUGKlv7RpyReRsRpI/0Kv61q74FQUAQIJLqDDS0TNyEImmHwAAuHgJFUZmpbpj2g8AAFy8hAojBb50eT1ujbSA16Ezq2oKfOnxLAsAgISWUGHE5XSosjRHkoYEksHXlaU57DcCAEAcJVQYkaRVuV7VrM1Tpid8KibT41bN2jz2GQEAIM4SctOzVbleXZuTyQ6sAACMAwkZRqQzUzaF82fYLgMAgISXcNM0AABgfCGMAAAAqyIOIwcOHFBpaamysrLkcDj03HPPnbe/3+/XTTfdpIULF8rpdKqsrCzKUgEAwGQUcRj54IMPtGTJEj366KOj6t/X16dLL71UGzZs0JIlSyIuEAAATG4R38BaUlKikpKSUfefN2+eHnnkEUnStm3bIv04AAAwyY3L1TR9fX3q6+sLve7u7rZYDQAAGEvj8gbW6upqeTye0JGdnW27JAAAMEbGZRipqKhQIBAIHSdOnLBdEgAAGCPjcpomJSVFKSkptssAAABxMC6vjAAAgMQR8ZWR999/X2+88UbodWtrq5qampSenq45c+aooqJCJ0+e1Pbt20N9mpqaQuf+5S9/UVNTk5KTk5WTk3Px3wAAAExoDmOMieSEffv2aeXKlUPab731Vj311FO67bbb9Oabb2rfvn3/8yGOoQ+gmzt3rt58881RfWZ3d7c8Ho8CgYDS0tIiKRcAAFgy2t/viMOIDYQRAAAmntH+fnPPCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsGqK7QJsCg4Y1bd2qaOnV7NS3SrwpcvldNguCwCAhJKwYaS22a+qPS3yB3pDbV6PW5WlOVqV67VYGQAAiSUhp2lqm/1av6MxLIhIUnugV+t3NKq22W+pMgAAEk/ChZHggFHVnhaZYd4bbKva06LgwHA9AABArCVcGKlv7RpyReRsRpI/0Kv61q74FQUAQAJLuDDS0TNyEImmHwAAuDgJF0Zmpbpj2g8AAFychAsjBb50eT1ujbSA16Ezq2oKfOnxLAsAgISVcGHE5XSosjRHkoYEksHXlaU57DcCAECcJFwYkaRVuV7VrM1Tpid8KibT41bN2jz2GQEAII4SdtOzVbleXZuTyQ6sAABYlrBhRDozZVM4f4btMgAASGgJOU0DAADGD8IIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKyKOIwcOHBApaWlysrKksPh0HPPPXfBc/bv36/8/Hy53W5dfvnlevzxx6OpFQAATEIRh5EPPvhAS5Ys0aOPPjqq/q2trbr++uu1YsUKHTlyRA8++KDuuece7d69O+JiAQDA5BPxDqwlJSUqKSkZdf/HH39cc+bM0ebNmyVJixYt0uHDh/XjH/9Yf/u3fxvpxwMAgElmzO8Z+e///m8VFxeHtV133XU6fPiwTp8+Pew5fX196u7uDjsAAMDkNOZhpL29XRkZGWFtGRkZ+vjjj9XZ2TnsOdXV1fJ4PKEjOzt7rMsEAACWxGU1jcMR/iRcY8yw7YMqKioUCARCx4kTJ8a8RgAAYMeYP7U3MzNT7e3tYW0dHR2aMmWKZswY/om5KSkpSklJGevSAADAODDmV0YKCwtVV1cX1vbSSy9p6dKlSkpKGuuPBwAA41zEYeT9999XU1OTmpqaJJ1ZutvU1KS2tjZJZ6ZYbrnlllD/devW6a233lJ5eblee+01bdu2TVu3btX9998fm28AAAAmtIinaQ4fPqyVK1eGXpeXl0uSbr31Vj311FPy+/2hYCJJPp9Pe/fu1X333afHHntMWVlZ+slPfsKyXgAAIElymMG7Scex7u5ueTweBQIBpaWl2S4HAACMwmh/v3k2DQAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAq6bYLsC24IBRfWuXOnp6NSvVrQJfulxOh+2yAABIGAkdRmqb/ara0yJ/oDfU5vW4VVmao1W5XouVAQCQOBJ2mqa22a/1OxrDgogktQd6tX5Ho2qb/ZYqAwAgsSRkGAkOGFXtaZEZ5r3Btqo9LQoODNcDAADEUkKGkfrWriFXRM5mJPkDvapv7YpfUQAAJKiEDCMdPSMHkWj6AQCA6CVkGJmV6o5pPwAAEL2EDCMFvnR5PW6NtIDXoTOragp86fEsCwCAhJSQYcTldKiyNEeShgSSwdeVpTnsNwIAQBwkZBiRpFW5XtWszVOmJ3wqJtPjVs3aPPYZAQAgThJ607NVuV5dm5PJDqwAAFiU0GFEOjNlUzh/hu0yAABIWAk7TQMAAMYHwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwKqowsiWLVvk8/nkdruVn5+vgwcPnrf/Y489pkWLFmnq1KlauHChtm/fHlWxAABg8pkS6Qm7du1SWVmZtmzZouXLl+unP/2pSkpK1NLSojlz5gzpX1NTo4qKCv3sZz/T5z//edXX1+vb3/62LrnkEpWWlsbkSwAAgInLYYwxkZywbNky5eXlqaamJtS2aNEirV69WtXV1UP6FxUVafny5frXf/3XUFtZWZkOHz6sl19+eVSf2d3dLY/Ho0AgoLS0tEjKBQAAloz29zuiaZr+/n41NDSouLg4rL24uFiHDh0a9py+vj653e6wtqlTp6q+vl6nT58e8Zzu7u6wAwAATE4RhZHOzk4Fg0FlZGSEtWdkZKi9vX3Yc6677jo9+eSTamhokDFGhw8f1rZt23T69Gl1dnYOe051dbU8Hk/oyM7OjqRMAAAwgUR1A6vD4Qh7bYwZ0jboBz/4gUpKSnTVVVcpKSlJX/3qV3XbbbdJklwu17DnVFRUKBAIhI4TJ05EUyYAAJgAIgojM2fOlMvlGnIVpKOjY8jVkkFTp07Vtm3b9OGHH+rNN99UW1ub5s2bp9TUVM2cOXPYc1JSUpSWlhZ2AACAySmiMJKcnKz8/HzV1dWFtdfV1amoqOi85yYlJWn27NlyuVzauXOnbrzxRjmdbHMCAECii3hpb3l5uW6++WYtXbpUhYWFeuKJJ9TW1qZ169ZJOjPFcvLkydBeIn/84x9VX1+vZcuW6d1339WmTZvU3NysX/ziF7H9JgAAYEKKOIysWbNGp06d0saNG+X3+5Wbm6u9e/dq7ty5kiS/36+2trZQ/2AwqH/7t3/TsWPHlJSUpJUrV+rQoUOaN29ezL4EAACYuCLeZ8QG9hkBAGDiGZN9RgAAAGIt4mmaySg4YFTf2qWOnl7NSnWrwJcul3P4pcoAACC2Ej6M1Db7VbWnRf5Ab6jN63GrsjRHq3K9FisDACAxJPQ0TW2zX+t3NIYFEUlqD/Rq/Y5G1Tb7LVUGAEDiSNgwEhwwqtrTouHu3h1sq9rTouDAuL+/FwCACS1hw0h9a9eQKyJnM5L8gV7Vt3bFrygAABJQwoaRjp6Rg0g0/QAAQHQSNozMSnXHtB8AAIhOwoaRAl+6vB63RlrA69CZVTUFvvR4lgUAQMJJ2DDicjpUWZojSUMCyeDrytIc9hsBAGCMJWwYkaRVuV7VrM1Tpid8KibT41bN2jz2GQEAIA4SftOzVbleXZuTyQ6sAABYkvBhRDozZVM4f4btMgAASEgJPU0DAADsI4wAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwaortAsaL4IBRfWuXOnp6NSvVrQJfulxOh+2yAACY9Agjkmqb/ara0yJ/oDfU5vW4VVmao1W5XouVAQAw+SX8NE1ts1/rdzSGBRFJag/0av2ORtU2+y1VBgBAYkjoMBIcMKra0yIzzHuDbVV7WhQcGK4HAACIhYQOI/WtXUOuiJzNSPIHelXf2hW/ogAASDAJHUY6ekYOItH0AwAAkUvoMDIr1R3TfgAAIHJRhZEtW7bI5/PJ7XYrPz9fBw8ePG//p59+WkuWLNGnPvUpeb1e3X777Tp16lRUBcdSgS9dXo9bIy3gdejMqpoCX3o8ywIAIKFEHEZ27dqlsrIybdiwQUeOHNGKFStUUlKitra2Yfu//PLLuuWWW3THHXfo1Vdf1X/8x3/o97//ve68886LLv5iuZwOVZbmSNKQQDL4urI0h/1GAAAYQxGHkU2bNumOO+7QnXfeqUWLFmnz5s3Kzs5WTU3NsP1/97vfad68ebrnnnvk8/n0hS98Qd/5znd0+PDhiy4+FlblelWzNk+ZnvCpmEyPWzVr89hnBACAMRbRpmf9/f1qaGjQAw88ENZeXFysQ4cODXtOUVGRNmzYoL1796qkpEQdHR361a9+pRtuuGHEz+nr61NfX1/odXd3dyRlRmxVrlfX5mSyAysAABZEdGWks7NTwWBQGRkZYe0ZGRlqb28f9pyioiI9/fTTWrNmjZKTk5WZmanp06fr3//930f8nOrqank8ntCRnZ0dSZlRcTkdKpw/Q1/968tUOH8GQQQAgDiJ6gZWhyP8h9oYM6RtUEtLi+655x798Ic/VENDg2pra9Xa2qp169aN+PcrKioUCARCx4kTJ6IpEwAATAARTdPMnDlTLpdryFWQjo6OIVdLBlVXV2v58uX6/ve/L0m68sorNW3aNK1YsUIPPfSQvN6h92SkpKQoJSUlktIAAMAEFdGVkeTkZOXn56uuri6sva6uTkVFRcOe8+GHH8rpDP8Yl8sl6cwVFQAAkNginqYpLy/Xk08+qW3btum1117Tfffdp7a2ttC0S0VFhW655ZZQ/9LSUj3zzDOqqanR8ePH9dvf/lb33HOPCgoKlJWVFbtvAgAAJqSIpmkkac2aNTp16pQ2btwov9+v3Nxc7d27V3PnzpUk+f3+sD1HbrvtNvX09OjRRx/V9773PU2fPl1f+tKX9M///M+x+xYAAGDCcpgJMFfS3d0tj8ejQCCgtLQ02+UAAIBRGO3vd0I/mwYAANhHGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGBVxE/tncyCA0b1rV3q6OnVrFS3CnzpcjkdtssCAGBSI4x8orbZr6o9LfIHekNtXo9blaU5WpXrtVgZAACTG9M0OhNE1u9oDAsiktQe6NX6HY2qbfZbqgwAgMkv4cNIcMCoak+LzDDvDbZV7WlRcGC4HgAA4GIlfBipb+0ackXkbEaSP9Cr+tau+BUFAEACSfgw0tEzchCJph8AAIhMwoeRWanumPYDAACRSfgwUuBLl9fj1kgLeB06s6qmwJcez7IAAEgYCR9GXE6HKktzJGlIIBl8XVmaw34jAACMkYQPI5K0KtermrV5yvSET8VketyqWZvHPiMAAIwhNj37xKpcr67NyWQHVgAA4owwchaX06HC+TNslwEAQEJhmgYAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFVTbBcw3gQHjOpbu9TR06tZqW4V+NLlcjpslwUAwKRFGDlLbbNfVXta5A/0htq8HrcqS3O0KtdrsTIAACYvpmk+Udvs1/odjWFBRJLaA71av6NRtc1+S5UBADC5EUZ0Zmqmak+LzDDvDbZV7WlRcGC4HgAA4GIQRiTVt3YNuSJyNiPJH+hVfWtX/IoCACBBRBVGtmzZIp/PJ7fbrfz8fB08eHDEvrfddpscDseQY/HixVEXHWsdPSMHkWj6AQCA0Ys4jOzatUtlZWXasGGDjhw5ohUrVqikpERtbW3D9n/kkUfk9/tDx4kTJ5Senq6/+7u/u+jiY2VWqjum/QAAwOhFHEY2bdqkO+64Q3feeacWLVqkzZs3Kzs7WzU1NcP293g8yszMDB2HDx/Wu+++q9tvv/2ii4+VAl+6vB63RlrA69CZVTUFvvR4lgUAQEKIKIz09/eroaFBxcXFYe3FxcU6dOjQqP7G1q1b9eUvf1lz584dsU9fX5+6u7vDjrHkcjpUWZojSUMCyeDrytIc9hsBAGAMRBRGOjs7FQwGlZGREdaekZGh9vb2C57v9/v14osv6s477zxvv+rqank8ntCRnZ0dSZlRWZXrVc3aPGV6wqdiMj1u1azNY58RAADGSFSbnjkc4VcIjDFD2obz1FNPafr06Vq9evV5+1VUVKi8vDz0uru7O26B5NqcTHZgBQAgjiIKIzNnzpTL5RpyFaSjo2PI1ZJzGWO0bds23XzzzUpOTj5v35SUFKWkpERSWsy4nA4Vzp9h5bMBAEhEEU3TJCcnKz8/X3V1dWHtdXV1KioqOu+5+/fv1xtvvKE77rgj8ioBAMCkFfE0TXl5uW6++WYtXbpUhYWFeuKJJ9TW1qZ169ZJOjPFcvLkSW3fvj3svK1bt2rZsmXKzc2NTeUAAGBSiDiMrFmzRqdOndLGjRvl9/uVm5urvXv3hlbH+P3+IXuOBAIB7d69W4888khsqgYAAJOGwxgz7h+40t3dLY/Ho0AgoLS0NNvlAACAURjt7zfPpgEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVkW1HfxkFxwwbAkPAECcEEbOUdvsV9WeFvkDvaE2r8etytIcHpYHAMAYYJrmLLXNfq3f0RgWRCSpPdCr9TsaVdvst1QZAACTF2HkE8EBo6o9LRpuB7jBtqo9LQoOjPs94gAAmFAII5+ob+0ackXkbEaSP9Cr+tau+BUFAEACIIx8oqNn5CASTT8AADA6hJFPzEp1x7QfAAAYHcLIJwp86fJ63BppAa9DZ1bVFPjS41kWAACTHmHkEy6nQ5WlOZI0JJAMvq4szWG/EQAAYowwcpZVuV7VrM1Tpid8KibT41bN2jz2GQEAYAyw6dk5VuV6dW1OJjuwAgAQJ4SRYbicDhXOn2G7DAAAEgLTNAAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwiqW9IwgOGPYaAQAgDggjw6ht9qtqT4v8gf95Qq/X41ZlaQ67sAIAEGNM05yjttmv9Tsaw4KIJLUHerV+R6Nqm/2WKgMAYHIijJwlOGBUtadFZpj3Btuq9rQoODBcDwAAEA3CyFnqW7uGXBE5m5HkD/SqvrUrfkUBADDJEUbO0tEzchCJph8AALgwwshZZqW6Y9oPAABcGGHkLAW+dHk9bo20gNehM6tqCnzp8SwLAIBJjTByFpfTocrSHEkaEkgGX1eW5rDfCAAAMUQYOceqXK9q1uYp0xM+FZPpcatmbR77jAAAEGNsejaMVbleXZuTyQ6sAADEAWFkBC6nQ4XzZ9guAwCASY9pGgAAYBVhBAAAWMU0zXnw5F4AAMYeYWQEPLkXAID4YJpmGDy5FwCA+CGMnIMn9wIAEF+EkXPw5F4AAOKLMHIOntwLAEB8RRVGtmzZIp/PJ7fbrfz8fB08ePC8/fv6+rRhwwbNnTtXKSkpmj9/vrZt2xZVwWONJ/cCABBfEa+m2bVrl8rKyrRlyxYtX75cP/3pT1VSUqKWlhbNmTNn2HO+/vWv689//rO2bt2qz3zmM+ro6NDHH3980cWPhcEn97YHeoe9b8ShM8+p4cm9AADEhsMYE9GdmMuWLVNeXp5qampCbYsWLdLq1atVXV09pH9tba2+8Y1v6Pjx40pPj+4HvLu7Wx6PR4FAQGlpaVH9jUgMrqaRFBZIBncY4YF5AABc2Gh/vyOapunv71dDQ4OKi4vD2ouLi3Xo0KFhz3n++ee1dOlS/cu//Isuu+wyXXHFFbr//vv10UcfRfLRccWTewEAiJ+Ipmk6OzsVDAaVkZER1p6RkaH29vZhzzl+/Lhefvllud1uPfvss+rs7NQ//MM/qKura8T7Rvr6+tTX1xd63d3dHUmZMcGTewEAiI+obmB1OMJ/kI0xQ9oGDQwMyOFw6Omnn1ZBQYGuv/56bdq0SU899dSIV0eqq6vl8XhCR3Z2djRlAgCACSCiKyMzZ86Uy+UachWko6NjyNWSQV6vV5dddpk8Hk+obdGiRTLG6O2339aCBQuGnFNRUaHy8vLQ6+7u7rgHEraDBwAgPiK6MpKcnKz8/HzV1dWFtdfV1amoqGjYc5YvX6533nlH77//fqjtj3/8o5xOp2bPnj3sOSkpKUpLSws74ont4AEAiJ+Ip2nKy8v15JNPatu2bXrttdd03333qa2tTevWrZN05qrGLbfcEup/0003acaMGbr99tvV0tKiAwcO6Pvf/76+9a1vaerUqbH7JjHCdvAAAMRXxPuMrFmzRqdOndLGjRvl9/uVm5urvXv3au7cuZIkv9+vtra2UP9Pf/rTqqur0913362lS5dqxowZ+vrXv66HHnoodt8ihiLZDr5w/oz4FQYAwCQV8T4jNsRzn5H/03RS9+5sumC/R77x1/rqX182prUAADCRjck+I4mA7eABAIgvwsg5BreDH2k3EYfOrKphO3gAAGKDMHIOl9OhytIcSRoSSAZfV5bmsPkZAAAxQhgZxkjbwV8yLUmP3fQ59hkBACCGCCMjWJXr1Q9uyFH6tORQW9cHp/VPL7zGPiMAAMQQYWQEtc1+ffd/N6rrg/6wdjY+AwAgtggjw2DjMwAA4ocwMoxINj4DAAAXhzAyjI6ekYNINP0AAMDICCPDYOMzAADihzAyjAttfCax8RkAALFCGBnG2RufjeQrS7xsfAYAQAwQRkawKter/3W1b8T3nzjQyvJeAABigDAyguCA0fP/7/xhg+W9AABcPMLICFjeCwBAfBBGRsDyXgAA4oMwMgKW9wIAEB+EkREMLu+9kHfPeXYNAACIDGFkBC6nQz+4YdEF+/3TC9zECgDAxSCMnMcl01Iu2IebWAEAuDiEkfPgJlYAAMYeYeQ8Rntz6pudH45xJQAATF6EkfMo8KUrM+3CUzU7f9/GfSMAAESJMHIeLqdD3yyYc8F+3DcCAED0CCMXMG/mtFH1474RAACiQxi5AO4bAQBgbBFGLmC09438/FAr940AABAFwsgFjPa+kfc+PK1Hf/1GHCoCAGByIYyMwmjvG/npgT9xdQQAgAgRRkZhtPeNfNgf5OoIAAARIoyMQoEvXdOnJo2q7+P73+DqCAAAESCMjILL6dDty+eNqu9Hpwd0784jY1sQAACTCGFklO760gJ9Ktk1qr7/9xW/9r7iH+OKAACYHAgjo+RyOvSdqy8fdf97dzaq/+OBMawIAIDJgTASgUiujpwekBb94EVtrjvGPSQAAJwHYSQCkV4dCRpp83+9oSs27NW6Xx7Wb9/oJJgAAHAOhzFm3P86dnd3y+PxKBAIKC0tzWotwQGjnB++qL6Poxs2h6QrZn1KCzM9cjoduuySqSqaP1NXXT5DLqcjtsUCAGDRaH+/p8SxpknB5XRo/Rfna/N/RbefiJF0rONDHev4n2fZPPabP2mKQ1qS7VHKFJd6Pw7KPcWlmZ9OkeOcfGKMUef7/eftM9p+seozXv/WZP+8iVw7YzV+P28i185YRddnatIULZk9XcsX2Ps/xlwZiUJwwGjxD2vVyw2qAIBJZPqnkvTw1z6rVbnemPy90f5+c89IFFxOhzZ9fYntMgAAiKn3PjytdTsaVdsc3+0pCCNRuv7KLH17xTzbZQAAEHNVe1riuuCCMHIRNtywWN9e4bNdBgAAMeUP9Kq+tStun0cYuUgbbsjRlpvy5J7CUAIAJo+Ont64fRa/oDFw/ZVevbpxlcquWaAklucCACaB0T6xPhaiCiNbtmyRz+eT2+1Wfn6+Dh48OGLfffv2yeFwDDn+8Ic/RF30eORyOlR27RX6w0MlKrtmAVdKAAATltfjVoEvPW6fF/Ev5q5du1RWVqYNGzboyJEjWrFihUpKStTW1nbe844dOya/3x86FixYEHXR49lgKHl14yo9fccyXbc4Qy5yCQBgAqkszYnrfiMR7zOybNky5eXlqaamJtS2aNEirV69WtXV1UP679u3TytXrtS7776r6dOnR1XkeNtnJFLBAaNDr3fqV40n9Pa7HyllilP9wQH9v7cDOh0c99u8AAASxCWfSlK1hX1GItqBtb+/Xw0NDXrggQfC2ouLi3Xo0KHznvu5z31Ovb29ysnJ0T/+4z9q5cqVI/bt6+tTX19f6HV3d3ckZY47LqdDKxZeqhULLw1rDw4Y/e5Pp/TbP/1FJ9/9aNzv0seOhuPr8yZy7YzV+P28iVw7YxVdn/GwA2tEYaSzs1PBYFAZGRlh7RkZGWpvbx/2HK/XqyeeeEL5+fnq6+vTL3/5S11zzTXat2+frr766mHPqa6uVlVVVSSlTUgup0PLF8zU8gUzbZcCAIA1UT2bxnFOxDLGDGkbtHDhQi1cuDD0urCwUCdOnNCPf/zjEcNIRUWFysvLQ6+7u7uVnZ0dTakAAGCci+jWypkzZ8rlcg25CtLR0THkasn5XHXVVXr99ddHfD8lJUVpaWlhBwAAmJwiCiPJycnKz89XXV1dWHtdXZ2KiopG/XeOHDkirzc2N8cAAICJLeJpmvLyct18881aunSpCgsL9cQTT6itrU3r1q2TdGaK5eTJk9q+fbskafPmzZo3b54WL16s/v5+7dixQ7t379bu3btj+00AAMCEFHEYWbNmjU6dOqWNGzfK7/crNzdXe/fu1dy5cyVJfr8/bM+R/v5+3X///Tp58qSmTp2qxYsX64UXXtD1118fu28BAAAmrIj3GbFhou8zAgBAIhrt7zd7gwIAAKsIIwAAwCrCCAAAsCqqTc/ibfC2lom+LTwAAIlk8Hf7QrenTogw0tPTI0nswgoAwATU09Mjj8cz4vsTYjXNwMCA3nnnHaWmpo647Xw0BreZP3HiBKt0xhhjHR+Mc3wwzvHDWMfHWI2zMUY9PT3KysqS0znynSET4sqI0+nU7Nmzx+zvs+V8/DDW8cE4xwfjHD+MdXyMxTif74rIIG5gBQAAVhFGAACAVQkdRlJSUlRZWamUlBTbpUx6jHV8MM7xwTjHD2MdH7bHeULcwAoAACavhL4yAgAA7COMAAAAqwgjAADAKsIIAACwKqHDyJYtW+Tz+eR2u5Wfn6+DBw/aLmlCOXDggEpLS5WVlSWHw6Hnnnsu7H1jjH70ox8pKytLU6dO1d/8zd/o1VdfDevT19enu+++WzNnztS0adP0la98RW+//XYcv8X4V11drc9//vNKTU3VrFmztHr1ah07diysD2N98WpqanTllVeGNn0qLCzUiy++GHqfMR4b1dXVcjgcKisrC7Ux1rHxox/9SA6HI+zIzMwMvT+uxtkkqJ07d5qkpCTzs5/9zLS0tJh7773XTJs2zbz11lu2S5sw9u7dazZs2GB2795tJJlnn3027P2HH37YpKammt27d5ujR4+aNWvWGK/Xa7q7u0N91q1bZy677DJTV1dnGhsbzcqVK82SJUvMxx9/HOdvM35dd9115uc//7lpbm42TU1N5oYbbjBz5swx77//fqgPY33xnn/+efPCCy+YY8eOmWPHjpkHH3zQJCUlmebmZmMMYzwW6uvrzbx588yVV15p7r333lA7Yx0blZWVZvHixcbv94eOjo6O0PvjaZwTNowUFBSYdevWhbX91V/9lXnggQcsVTSxnRtGBgYGTGZmpnn44YdDbb29vcbj8ZjHH3/cGGPMe++9Z5KSkszOnTtDfU6ePGmcTqepra2NW+0TTUdHh5Fk9u/fb4xhrMfSJZdcYp588knGeAz09PSYBQsWmLq6OvPFL34xFEYY69iprKw0S5YsGfa98TbOCTlN09/fr4aGBhUXF4e1FxcX69ChQ5aqmlxaW1vV3t4eNsYpKSn64he/GBrjhoYGnT59OqxPVlaWcnNz+edwHoFAQJKUnp4uibEeC8FgUDt37tQHH3ygwsJCxngMfPe739UNN9ygL3/5y2HtjHVsvf7668rKypLP59M3vvENHT9+XNL4G+cJ8aC8WOvs7FQwGFRGRkZYe0ZGhtrb2y1VNbkMjuNwY/zWW2+F+iQnJ+uSSy4Z0od/DsMzxqi8vFxf+MIXlJubK4mxjqWjR4+qsLBQvb29+vSnP61nn31WOTk5of/wMsaxsXPnTjU2Nur3v//9kPf49zl2li1bpu3bt+uKK67Qn//8Zz300EMqKirSq6++Ou7GOSHDyCCHwxH22hgzpA0XJ5ox5p/DyO666y698sorevnll4e8x1hfvIULF6qpqUnvvfeedu/erVtvvVX79+8Pvc8YX7wTJ07o3nvv1UsvvSS32z1iP8b64pWUlIT+92c/+1kVFhZq/vz5+sUvfqGrrrpK0vgZ54Scppk5c6ZcLteQZNfR0TEkJSI6g3dsn2+MMzMz1d/fr3fffXfEPvgfd999t55//nn95je/0ezZs0PtjHXsJCcn6zOf+YyWLl2q6upqLVmyRI888ghjHEMNDQ3q6OhQfn6+pkyZoilTpmj//v36yU9+oilTpoTGirGOvWnTpumzn/2sXn/99XH373RChpHk5GTl5+errq4urL2urk5FRUWWqppcfD6fMjMzw8a4v79f+/fvD41xfn6+kpKSwvr4/X41Nzfzz+EsxhjdddddeuaZZ/TrX/9aPp8v7H3GeuwYY9TX18cYx9A111yjo0ePqqmpKXQsXbpUf//3f6+mpiZdfvnljPUY6evr02uvvSav1zv+/p2O6e2wE8jg0t6tW7ealpYWU1ZWZqZNm2befPNN26VNGD09PebIkSPmyJEjRpLZtGmTOXLkSGh59MMPP2w8Ho955plnzNGjR803v/nNYZeNzZ492/znf/6naWxsNF/60pdYnneO9evXG4/HY/bt2xe2RO/DDz8M9WGsL15FRYU5cOCAaW1tNa+88op58MEHjdPpNC+99JIxhjEeS2evpjGGsY6V733ve2bfvn3m+PHj5ne/+5258cYbTWpqauh3bjyNc8KGEWOMeeyxx8zcuXNNcnKyycvLCy2VxOj85je/MZKGHLfeeqsx5szSscrKSpOZmWlSUlLM1VdfbY4ePRr2Nz766CNz1113mfT0dDN16lRz4403mra2NgvfZvwabowlmZ///OehPoz1xfvWt74V+u/BpZdeaq655ppQEDGGMR5L54YRxjo2BvcNSUpKMllZWeZrX/uaefXVV0Pvj6dxdhhjTGyvtQAAAIxeQt4zAgAAxg/CCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKv+P6fL1o3gZAfKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter( range(num_iterations), all_costs)\n",
    "\n",
    "#len(range(each_iter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient descent is converging correctly for 500 iterations and with learning rate =0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11- Tuning learning rate alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "current learning rate is 0.01\n",
      "initial guess of b and w:  0.24602852419434018 [0.13328034 0.68833151 0.76672845 0.69324465 0.20385939 0.36738247\n",
      " 0.73938839 0.63844865 0.23711801]\n",
      "Iteration:  0 Cost:  0.8668244038369632\n",
      "Iteration:  30 Cost:  0.8248677055759897\n",
      "Iteration:  60 Cost:  0.7864862240317503\n",
      "Iteration:  90 Cost:  0.7516865307975718\n",
      "Iteration:  120 Cost:  0.7204203703744122\n",
      "Iteration:  150 Cost:  0.6925834853943287\n",
      "Iteration:  180 Cost:  0.6680187340787032\n",
      "Iteration:  210 Cost:  0.646523246285408\n",
      "Iteration:  240 Cost:  0.627858728327675\n",
      "Iteration:  270 Cost:  0.6117635960725168\n",
      "Iteration:  300 Cost:  0.5979654714756857\n",
      "Iteration:  330 Cost:  0.5861927168309452\n",
      "Iteration:  360 Cost:  0.5761840316621805\n",
      "Iteration:  390 Cost:  0.5676955881675981\n",
      "Iteration:  420 Cost:  0.5605056172901494\n",
      "Iteration:  450 Cost:  0.5544166926805183\n",
      "Iteration:  480 Cost:  0.5492561560731698\n",
      "Iteration:  510 Cost:  0.5448751930262856\n",
      "Iteration:  540 Cost:  0.54114703851658\n",
      "Iteration:  570 Cost:  0.5379647100776764\n",
      "Iteration:  600 Cost:  0.5352385670284193\n",
      "Iteration:  630 Cost:  0.5328939002857213\n",
      "Iteration:  660 Cost:  0.5308686792789412\n",
      "Iteration:  690 Cost:  0.5291115236341632\n",
      "Iteration:  720 Cost:  0.5275799262036164\n",
      "Iteration:  750 Cost:  0.5262387274198034\n",
      "Iteration:  780 Cost:  0.5250588252300549\n",
      "Iteration:  810 Cost:  0.5240160967697932\n",
      "Iteration:  840 Cost:  0.5230905048678367\n",
      "Iteration:  870 Cost:  0.5222653625202269\n",
      "Iteration:  900 Cost:  0.5215267302723245\n",
      "Iteration:  930 Cost:  0.5208629241148708\n",
      "Iteration:  960 Cost:  0.5202641144612707\n",
      "Iteration:  990 Cost:  0.5197219996938849\n",
      "Final estimates of b and q are:  -0.5124461091605034 [ 0.06741881  0.4012098   0.42562237  0.16058754  0.31402657 -0.19852709\n",
      "  0.12229079  0.51653687 -0.44755153]\n",
      "current learning rate is 0.001\n",
      "initial guess of b and w:  0.3541647584295976 [0.36676512 0.3432828  0.82286022 0.24373731 0.95512767 0.99860941\n",
      " 0.08982368 0.90607887 0.33110346]\n",
      "Iteration:  0 Cost:  0.9378389130425917\n",
      "Iteration:  30 Cost:  0.9331777867561599\n",
      "Iteration:  60 Cost:  0.9285484978604471\n",
      "Iteration:  90 Cost:  0.9239511183654562\n",
      "Iteration:  120 Cost:  0.9193857177605835\n",
      "Iteration:  150 Cost:  0.9148523629658273\n",
      "Iteration:  180 Cost:  0.9103511182833118\n",
      "Iteration:  210 Cost:  0.9058820453491727\n",
      "Iteration:  240 Cost:  0.9014452030858359\n",
      "Iteration:  270 Cost:  0.8970406476547284\n",
      "Iteration:  300 Cost:  0.892668432409464\n",
      "Iteration:  330 Cost:  0.8883286078495425\n",
      "Iteration:  360 Cost:  0.8840212215746008\n",
      "Iteration:  390 Cost:  0.8797463182392607\n",
      "Iteration:  420 Cost:  0.8755039395086127\n",
      "Iteration:  450 Cost:  0.8712941240143817\n",
      "Iteration:  480 Cost:  0.8671169073118132\n",
      "Iteration:  510 Cost:  0.862972321837328\n",
      "Iteration:  540 Cost:  0.858860396866987\n",
      "Iteration:  570 Cost:  0.8547811584758123\n",
      "Iteration:  600 Cost:  0.8507346294980112\n",
      "Iteration:  630 Cost:  0.8467208294881428\n",
      "Iteration:  660 Cost:  0.8427397746832813\n",
      "Iteration:  690 Cost:  0.8387914779662173\n",
      "Iteration:  720 Cost:  0.8348759488297446\n",
      "Iteration:  750 Cost:  0.8309931933420798\n",
      "Iteration:  780 Cost:  0.8271432141134606\n",
      "Iteration:  810 Cost:  0.8233260102639722\n",
      "Iteration:  840 Cost:  0.8195415773926467\n",
      "Iteration:  870 Cost:  0.8157899075478827\n",
      "Iteration:  900 Cost:  0.8120709891992336\n",
      "Iteration:  930 Cost:  0.8083848072106066\n",
      "Iteration:  960 Cost:  0.8047313428149266\n",
      "Iteration:  990 Cost:  0.8011105735903035\n",
      "Final estimates of b and q are:  0.14352642084622103 [0.31291605 0.3426509  0.7406018  0.1031066  0.89626939 0.81860643\n",
      " 0.06400629 0.85516289 0.17370006]\n",
      "current learning rate is 0.0001\n",
      "initial guess of b and w:  0.7890160374780241 [0.76707286 0.81603961 0.16581228 0.68775648 0.29745891 0.51821259\n",
      " 0.06755004 0.45217809 0.3194495 ]\n",
      "Iteration:  0 Cost:  0.9908883433994427\n",
      "Iteration:  30 Cost:  0.9903468625918582\n",
      "Iteration:  60 Cost:  0.9898057585311594\n",
      "Iteration:  90 Cost:  0.9892650313160201\n",
      "Iteration:  120 Cost:  0.9887246810448687\n",
      "Iteration:  150 Cost:  0.9881847078158905\n",
      "Iteration:  180 Cost:  0.9876451117270252\n",
      "Iteration:  210 Cost:  0.987105892875968\n",
      "Iteration:  240 Cost:  0.9865670513601665\n",
      "Iteration:  270 Cost:  0.9860285872768206\n",
      "Iteration:  300 Cost:  0.9854905007228844\n",
      "Iteration:  330 Cost:  0.9849527917950612\n",
      "Iteration:  360 Cost:  0.9844154605898054\n",
      "Iteration:  390 Cost:  0.9838785072033214\n",
      "Iteration:  420 Cost:  0.9833419317315624\n",
      "Iteration:  450 Cost:  0.9828057342702292\n",
      "Iteration:  480 Cost:  0.9822699149147709\n",
      "Iteration:  510 Cost:  0.9817344737603826\n",
      "Iteration:  540 Cost:  0.9811994109020051\n",
      "Iteration:  570 Cost:  0.9806647264343246\n",
      "Iteration:  600 Cost:  0.9801304204517717\n",
      "Iteration:  630 Cost:  0.9795964930485201\n",
      "Iteration:  660 Cost:  0.9790629443184864\n",
      "Iteration:  690 Cost:  0.9785297743553291\n",
      "Iteration:  720 Cost:  0.9779969832524471\n",
      "Iteration:  750 Cost:  0.9774645711029822\n",
      "Iteration:  780 Cost:  0.9769325379998126\n",
      "Iteration:  810 Cost:  0.9764008840355577\n",
      "Iteration:  840 Cost:  0.9758696093025737\n",
      "Iteration:  870 Cost:  0.9753387138929542\n",
      "Iteration:  900 Cost:  0.9748081978985296\n",
      "Iteration:  930 Cost:  0.9742780614108657\n",
      "Iteration:  960 Cost:  0.9737483045212636\n",
      "Iteration:  990 Cost:  0.9732189273207583\n",
      "Final estimates of b and q are:  0.7612138865249451 [0.75781582 0.81316522 0.16319892 0.67319877 0.29988384 0.5029468\n",
      " 0.06325917 0.44537402 0.29956828]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFp0lEQVR4nO3de1xUdf4/8NcwMAMoYIJyUUDthpddL7ir6JKWLWblZqZitqjtfvsuu79WgfWa23bdqDSztsDNsM1KI4V6aOuW5IbiQvmVwC2xqxcUQYIMMC4Dw+f3x2lGBgaYM7czl9fz8ZiHy+Fzznzm4Mqrz+V9VEIIASIiIiKF+CjdASIiIvJuDCNERESkKIYRIiIiUhTDCBERESmKYYSIiIgUxTBCREREimIYISIiIkUxjBAREZGifJXugCU6Oztx4cIFBAUFQaVSKd0dIiIisoAQAk1NTYiKioKPT+/jH24RRi5cuIDo6Gilu0FERERWOHfuHIYPH97r990ijAQFBQGQPkxwcLDCvSEiIiJLNDY2Ijo62vh7vDduEUYMUzPBwcEMI0RERG6mvyUWXMBKREREimIYISIiIkUxjBAREZGiZIeRw4cPY+7cuYiKioJKpcI777zT7zmHDh1CfHw8/P39MWrUKGzdutWavhIREZEHkh1GfvjhB4wfPx4vvPCCRe1Pnz6NW2+9FYmJiSgrK8MDDzyAFStWIC8vT3ZniYiIyPPI3k0zZ84czJkzx+L2W7duRUxMDLZs2QIAGD16NI4dO4ZNmzbhrrvukvv2RERE5GEcvmakpKQESUlJJsdmz56NY8eOob293ew5bW1taGxsNHkRERGRZ3J4GKmpqUF4eLjJsfDwcHR0dKCurs7sOZmZmQgJCTG+WH2ViIjIczml6Fn3YidCCLPHDdavX4+MjAzj14YKbvak1wOFhdILAGbOlF5qtV3fhoiIiPrh8DASERGBmpoak2O1tbXw9fVFaGio2XO0Wi20Wq3D+pSfD/zv/wL19VeOPf649OeAAUBgIODvL33d2ioFF7Ua0GqBtrYrX5trY8t5ajUQFQXceSewYgWg0TjsFhAREbkMh4eRhIQE7Nu3z+TYgQMHMHnyZPj5+Tn67XvIzwf6Wjf7ww/SSylnzgDFxcDq1VeCkT2Cjtzz/PyAUaOA+fMZjIiIyLFkh5HLly/j66+/Nn59+vRplJeXY/DgwYiJicH69etRVVWFHTt2AABSU1PxwgsvICMjA/fddx9KSkqQk5ODXbt22e9TWEivl36xugulg9HFi0BJiRSMNBogJMSxo0O9tdHpgOBgYNYsYPNmICDA+feCiIgcRyUMCzgsVFhYiBtvvLHH8WXLluEf//gHli9fjjNnzqDQsBgDUtGz9PR0nDhxAlFRUVi7di1SU1Mtfs/GxkaEhISgoaHBpgflFRYCZrpObsbPDxg0yHnTZ12P+foCYWHAhAnA8uXATTdxnRERUW8s/f0tO4wowV5hZNcuYMkSO3aMCMDgwdKUmiNHh8xdu61NCmZxcdLo1c03MxgRkWux9Pe3U3bTuIrISKV7QJ7ou++kl1KqqoCDB6X/3dcCbEevOfLxAQYOBMaP56gREcnjVSMjej0QEQH0Ut6EiOwsKEgavXH09Jm58wIDgWHDuDuNSEmcpunF7t3AokV26hgRuQ2tFhg6VPrfztqyb2gjBBAeDixdCqSlMRiR92AY6cOaNcDGjXboGBGRFfz9pZEbZ2/ZZ00jcjaGkX7s2QOkpEj/JyUi8mZK1jTiDjXPxjBiAb1eWvj3yivA8ePSWhIl/s+oVgNNTQxGRESAtBB60CDnbdnvel5np7TWKSEBuPdehiNbMYy4oZYWID1dCkgNDfbfCmrpeQxGRERXGEaOnFnTyHBee7t7F31kGCGb6HTA888DeXnAN984d7Ff1zaXLgGXLytzD4iIXE33oo/2+PfWkY//YBghj2EIRvn5Uk0NIZy72M9wrKWFwYiIPJtKBaxaBTz9tH2uxzBC5AB6PXDgALBpE/D559IQqjOez9P92s3Npk+dJiKyp9Wr7RNIGEaIPJzcBdiOXHPEUSMiz6JWS//RY+uUDcvBE3k4tRpISpJerqC/6TRnBqSWFulFRNbR64GsLKlInzMwjBCRXWg00lzzqlVK90Si0wFbtgCvvgpcvCg9O8fZW/a1Wmk6jcGI3NE33zjvvRhGiMgjaTRSteU1a5Tuiemo0fnzzn2AYfc23LpPlrr6aue9F9eMEBF5GVepacS1Rq6La0aIiMihAgKArVuV7oXEsBA7Jwf46CMpnDhzy37X8xobpRcBGRnOfWYRwwgRESnG1RZi97VLzdkBSYmRI3vXGbEUwwgREdGPXC0cWbNLzZUqsFqKa0aIiIjIISz9/e3jxD4RERER9cAwQkRERIpiGCEiIiJFMYwQERGRohhGiIiISFEMI0RERKQohhEiIiJSFMMIERERKYphhIiIiBTFMEJERESKYhghIiIiRTGMEBERkaIYRoiIiEhRvkp3QCn6Tj0KzxSi8EwhAGDmiJmYOWIm1D5qZTtGRETkZbwyjOSfzMf/7vtf1LfUG489XvQ4AGCw/2BMjJiI1dNW4+arb2Y4ISIicjCVEEIo3Yn+NDY2IiQkBA0NDQgODrbpWvkn83HXW3dZ3H5IwBBcG3ot7oy7EyumrIDGV2PT+xMREXkLS39/e1UY0XfqEbslFlVNVVZfI9gvGOPCxzGcEBER9YNhxIzCM4W48dUb7dgzhhMiIqLeWPr726vWjFQ3Vdv9mo3tjSg+X4zi88VY/cFqBPkGIWZQDMZHjMfyCctx08ibuO6EiIioD14VRiKDIh3+Hk0dTThRdwIn6k5g52c7AQDDg4bjhtgbGE6IiIjM8KppGn2nHhGbIlDXUmfH3snHcEJERN6Aa0Z6sfvEbizas8hOPbOPsMAw/HLkL3HvxHsZToiIyGMwjPRhTcEabCzeaIeeOQZrnRARkSdgGOnHnhN7kPJOClo7Wu1yPUdiOCEiInfEMGIBfaceB08dxCvlr6DobBGqLltff8SZWIiNiIjcAcOIFbqGkyNnj6DqchUEXP72IFAdiGEhwzBrxCxsnr0ZAZoApbtERETEMGIP7hpOtD5aXDP4GtY6ISIiRTGMOEDXcPLfmv/i1Pen0Kp3/TUnABAdFI3E2ESGEyIichqGESdp0bUg/UA6Dn5zELXNtWjUNSrdJYuw1gkRETkaw4hC9J16HPj6ADYVb0JZTRkutV1SuksWCQsIw3Wh13FRLBER2Q3DiItw13AS4BuACRETMD9uPsMJERFZhWHERRnWneSU5eDfp/6NulZlS9Nbyl/tj6uvuhpLxy9F2tQ0hhMiIuoXw4ibcNdaJ1ofLSIHRiIhOoFl7ImIyCyGETflruEEYKVYIiIyxTDiIdy11gkAhGhCMHTgUBZjIyLyUgwjHqp7rZPKhkpc7risdLcs4ufjhxGDRjCcEBF5CYYRL6Lr0OH5o88jvyIfn9V+hqb2JqW7ZBFWiiUi8mwMI16sazj5qv4rt9mxA/AhgEREnoRhhIzctdYJAAT7BWNc+DiGEyIiN8QwQr3qGk4+r/scl9ouoaWjReluWSTINwgxg2I4tUNE5AYYRkgWXYcOWz7eglfLXsWpS6fQ2ukeDwAEWMqeiMhVWfr728eai2dlZWHkyJHw9/dHfHw8ioqK+mz/4osvYvTo0QgICMD111+PHTt2WPO25EAaXw3WTF+DE/efQMuDLWjb0IaNv9yIqVFTEeDr2rte6lrqUHy+GKs/WA3tX7UIeSIE03OmY9N/NkHXoVO6e0RE1A/ZIyO5ublISUlBVlYWpk+fjr///e94+eWXUVFRgZiYmB7ts7OzsXbtWmzbtg0/+9nPcPToUdx3333YuXMn5s6da9F7cmREeYZFsXkn8lBRV+E2TycGpFL2o64ahQkREzi1Q0TkRA6bppkyZQomTZqE7Oxs47HRo0dj3rx5yMzM7NF+2rRpmD59OjZu3Gg8lpaWhmPHjuHIkSMWvSfDiOtx50qxABAWGIZfjvwlS9kTETmQpb+/feVcVKfTobS0FOvWrTM5npSUhOLiYrPntLW1wd/f3+RYQEAAjh49ivb2dvj5+Zk9p62tzeTDkGtR+6iRdE0Skq5JAuB+xdjqmuuw68Qu7DqxCwAQrA3G2CFj+ZRiIiIFyAojdXV10Ov1CA8PNzkeHh6Ompoas+fMnj0bL7/8MubNm4dJkyahtLQU27dvR3t7O+rq6hAZGdnjnMzMTDzyyCNyukYK6x5OANN6J5/WforL7a4bThrbGlFyvgQl50uktSd8ECARkdNYtYBVpVKZfC2E6HHM4MEHH8ScOXMwdepU+Pn54Y477sDy5csBAGq1+X/c169fj4aGBuPr3Llz1nSTFKbx1WDVtFUo/p9iND3QZFwUmzAsAUF+QUp3r09tnW0403gGu07sQtLrSfB9zBehT4Xi5ldvxvtfvQ99p17pLhIReQxZa0Z0Oh0CAwOxe/du3HnnncbjK1euRHl5OQ4dOtTrue3t7bh48SIiIyPx0ksvYe3atfj+++/h49N/HuKaEc/UvVJsfWu92zwEEACC/IIQExKDpeOXIm1qGqd2iIi6cegC1vj4eGRlZRmPjRkzBnfccYfZBazmzJgxA8OGDcPOnTstas8w4h26rzs59f0ptOrdp95JgE8ABg8YjLjQOKyetho3X30zp3aIyKs5LIwYtvZu3boVCQkJeOmll7Bt2zacOHECsbGxWL9+Paqqqoy1RL788kscPXoUU6ZMwaVLl7B582YUFBSgtLQUI0aMsOuHIc/TomtB+oF0HPzmIGqba91qSzHAZ+0QkXdzyG4aAEhOTkZ9fT0effRRVFdXY9y4cdi/fz9iY2MBANXV1aisrDS21+v1eOaZZ/DFF1/Az88PN954I4qLiy0OIuTdAjQB2Hr7VuPX3UvZ17XUQdfpuoXNvm35Ft+e/9ZYlC1QHYhhIcMwa8QsbJ69GQEa1y4oR0TkDCwHT26v6+hJ1eUqt3nODgBoVBpEBUUhKjiKoydE5HH4bBryWl2fs+Pq9U7M4egJEXkKhhGiHxkWxuaU5eDfp/6NutY6pbskC0dPiMhdMYwQ9aJrOPno3Eeo/aHWrZ5SDADBfsEYFz6O4YSIXBrDCJEM7vwgQAAI8g1CzKAYjI8Yz4cBEpHLYBghsoG7PwgQ4LZiIlIewwiRHXUNJ8erj+NMwxm06N1n1w7A0RMicj6GESIH61rO/sv6L1HfWq90l2QLCwjDdaHXcfSEiByCYYTIybqOnhw5ewRVl6vc6lk7AEdPiMi+GEaIFNb9WTvuWPME4OgJEVmPYYTIBbn7k4oBwF/tj1FXjcKEiAkcPSGiPjGMELkBjp4QkSdjGCFyUxw9ISJPwTDSH70eKCoCqquByEggMRFQ8x9Lcj3dR09OfX8KrXr3qhgLcPSEyBsxjPQlPx9YuRI4f/7KsbAwICsLWLjQ9usTOZg7P6nYQOujRcTACAwLHsaAQuShGEZ6k58PLFgA9PaxV68Gnn7atvcgcrKuUztVjVWoba51y9ETTu8QeRaGEXP0emDECNMREXN275YCC5Eb6zp6Uttc63bP2zHg9A6R+2IYMaewELjxxv7bBQQATU1cQ0IeRd+px4GvD2BT8SZ8Xvc56lrqoOvUKd0t2Th6QuQ+GEbM2bULWLLEsrYPPQQ8/LD170XkBjxl9CRYG4yxQ8Ziftx8jp4QuRCGEXMsHRkBAI0GaG7m6Ah5le6jJ5faLnFxLBFZjWHEHL0euOoqaQrGEhwdIYKuQ4ctH2/Bq2Wvum1RNoDTO0RKYBjpzcMPA488Yllbjo4Q9eApoycAp3eIHI1hpDd6vbRAtb3dsvYcHSHql6eMnnB6h8i+GEb6Imd0hDtriGTzpNETTu8QWY9hpC9yR0c++ACYNcv29yXyYl1HTy5evojmjma06N0zoARpghAdHM2AQtQPhpH+yBkdmT8fyMuzz/sSkZGnTO8ALM5GZA7DSH/0ekCrlf7sj68v0NrKqRoiB/Ok6R2uPyFiGLHMXXdJz6qxBBeyEinCk0ZPgnyDEDMoBuMjxnN6h7wCw4glDh4Ebr7Zsrbc5kvkEvSdehw8dRA5ZTn46NxHqP2hFq2d7vdQQINQ/1BEDIxgQCGPxDBiCb0eCAoCWiwcBuboCJFL8qTFsQDXn5DnYBixFIugEXkkT5reGageiLABYYgKjmJAIbfCMGIpFkEj8gqeNr3D9SfkDhhG5ODoCJFX8rTpHa4/IVfDMCIHR0eI6EeeNL0DAGH+YQgfGM6AQopgGJGLJeKJyIzu0zvftXzHgEJkIYYRuVginogs5EnF2QyGBAzBtaHXcoEs2RXDiDVYIp6IrNR9/UlTexN0nTqlu2U17uAhe2AYsQZLxBORHbXoWpB+IB0HvzmI2uZaNOoale6STQJ8AjB4wGDEhcZh9bTVuPnqmzm9Q31iGLEWS8QTkYN0n9653H7Z7QNKsF8wgvyDGFDILIYRa7FEPBE5UfeAUtdS59bTOwADCl3BMGItlognIoV1nd5paGtAk64JrXr3LdAGAAN9ByIsMAwJ0Qm4d+K93MHjJRhGbMEiaETkYjxt/QnAgOINGEZswSJoROTiPHF7McCA4mkYRmzFImhE5GZ0HTo8f/R55Ffko6qxCpdaL6GpvUnpbtksSBOE6OBoTIiYwCJtboZhxFYsgkZEHoABhZTEMGIPLIJGRB6oa0D5qv4r1LfWQ8DlfxX0iwHF9TCM2AOLoBGRFzA8f+eV8lfw35r/ovaHWo8JKHySsbIYRuyFRdCIyAt1DygXL19EfVu90t2yCwYU52EYsRcWQSMiAsCAQvIxjNgLi6AREfXKowOKNhRB2iA+LNAGDCP2JGchq1oNtLVxdISIvFbXgHK8+jiqmqrQ2O7+RdoAPixQLoYRe5K7zXfaNOA//3Fsn4iI3IgnBxQ+i6d3DCP2Jmd0BAByc4FFixzWHSIid+fJASXINwhaPy3CB4Rj6filSJua5pVTPAwj9iZ3dCQwEGhs5HQNEZEMhoCSU5aDj859hO9avsPljstKd8suNCoNrgq4CqMGj8L8uPlesQaFYcQR5I6OsCorEZHNPD2gDPIfhLABYR5ZrI1hxBHkjo6wKisRkUN4ckABPKeaLMOIo+TmAosXW9aWVVmJiJym+5OML7dfRqPOM9agAMBg7WAEa4Pdaqsxw4gjXXcd8NVXlrVl3REiIsV4ekAZqB6IsAFhLhtQGEYciVVZiYjcVveAcqntElo6LCxs6QYG+AzAQP+BCPYPxqwRs7B59mYEaAIU6QvDiCOxKisRkUfp+iTjqsYqNLc3e8zDAgHAD34YHDjY6QGFYcTR5Oys8fOTggtHR4iI3IYnl7oHADXUGOQ/CEGaICREJ+DeiffafaGspb+/fay5eFZWFkaOHAl/f3/Ex8ejqKioz/ZvvPEGxo8fj8DAQERGRuLee+9Ffb2b/0AffFAKGZZobweWLHFsf4iIyK7UPmokXZOEXQt24cT9J1C3rg4dD3bg/Xvex+KxizEmdAxCtaFKd9NqeuhR31qPM41nsOvELiS9noSQJ0OQf9LCJ9XbkeyRkdzcXKSkpCArKwvTp0/H3//+d7z88suoqKhATExMj/ZHjhzBjBkz8Oyzz2Lu3LmoqqpCamoqrr32Wrz99tsWvadLjowA8uuO7N4NLFjgsO4QEZHzdR9BaWptQl1rHVr07rsOJW9RHuaPnm/zdRw2TTNlyhRMmjQJ2dnZxmOjR4/GvHnzkJmZ2aP9pk2bkJ2djW+++cZ47G9/+xuefvppnDt3zqL3dNkwIrfuSHAw8N13nK4hIvIChnUoeSfy8M2lb9DU1oTWzlalu2WR0IBQXFx10eYpG4dM0+h0OpSWliIpKcnkeFJSEoqLi82eM23aNJw/fx779++HEAIXL17Enj17cNttt8l5a9ekVgMPPGB5+8ZGoLDQYd0hIiLXofHVYNW0VSi5rwS1a2rR8mAL2ja0YeMvN2Jq1FQMCRiCgb4Dle6mWfUt9Sg8U+i095MVRurq6qDX6xEeHm5yPDw8HDU1NWbPmTZtGt544w0kJydDo9EgIiICgwYNwt/+9rde36etrQ2NjY0mL5clZ+0IAGRlOa4vRETk0roHlKYNTcZ1KIvGLEJMUIzLBBSXDSMGKpXK5GshRI9jBhUVFVixYgX+8pe/oLS0FO+99x5Onz6N1NTUXq+fmZmJkJAQ4ys6OtqabjqH3NGRvXul6R0iIiJcWSibuzAXZzPOGgPK/rv346bYmxA1IApDAobAX+2vdFcdRtaaEZ1Oh8DAQOzevRt33nmn8fjKlStRXl6OQ4cO9TgnJSUFra2t2L17t/HYkSNHkJiYiAsXLiAyMrLHOW1tbWhrazN+3djYiOjoaNdbM2Igd+0I644QEZEVWnQtSD+QjoPfHERDWwOadE1o1TtmHcoHKR9g1ijbHvbqkDUjGo0G8fHxKCgoMDleUFCAadOmmT2nubkZPj6mb6P+cQFnbzlIq9UiODjY5OXS5I6OPPEER0eIiEi2AE0Att6+FV+t/Epah/LnFjSvb8bv4n+HawZdY7d1KKEBoZg5YqbtHbaQ1Vt7t27dioSEBLz00kvYtm0bTpw4gdjYWKxfvx5VVVXYsWMHAOAf//gH7rvvPjz//POYPXs2qqurkZaWBh8fH3z88ccWvafL7qbpSu7oyKJF0kP3iIiI7Kx7RVm5TzV29tZeX7kXTk5ORn19PR599FFUV1dj3Lhx2L9/P2JjYwEA1dXVqKysNLZfvnw5mpqa8MILL+BPf/oTBg0ahJtuuglPPfWUFR/LhRlGRyytO/LWW8DChaw7QkREdmdYKLtq2irjsa4B5XzDeTS0NeBy+2V0otPYZnjwcDx3y3N2CSJysBy8PbHuCBERuRF9px5FlUWobqpGZFAkEmMSFSkHL3tkhPogd3TEUHdklm0LhIiIiKyh9lE7dW1Ib6za2kt9YN0RIiIiWRhG7I11R4iIiGRhGHEEOaMjHR3AY485tj9EREQujGHEEeSOjjz+OEdHiIjIazGMOIqc0RG9HrjhBsf2h4iIyEUxjDiK3NGR4mKp9ggREZGXYZ0RR5Jbd0SjAZqbWXeEiIg8gkOeTUMyyR0d0em4mJWIiLwOR0YcjaMjRETkpTgy4irUauC11yxvz9ERIiLyMgwjzpCcDEybZnn7zExu9SUiIq/BMOIshw8DPhbebo6OEBGRF2EYcRa1Gpg3z/L2TzzB0REiIvIKDCPO9Ic/WN62vR1YssRxfSEiInIRDCPONHMmEBRkefu33gL27HFYd4iIiFwBw4gzqdVATo68c5Yu5XQNERF5NIYRZ1u4EFi0yPL2LS1czEpERB6NRc+UwEJoRETkBVj0zJWxTDwREZERR0aUotcDAwcCra2WtefoCBERuRmOjLg6tRrYscPy9hwdISIiD8UwoqSFC4GpUy1v//jj3FlDREQeh2FEaY8/bnlbvR644QbH9YWIiEgBDCNKk1sIrbhYKoZGRETkIRhGlGZNIbR77+V0DREReQyGEVcgtxBaczNQWOiw7hARETkTw4ir2LkT8POzvP3vf++4vhARETkRw4irUKuB116zvP1XX3HtCBEReQSGEVeSnAxce63l7Zcs4doRIiJyewwjriY72/K23OpLREQegGHE1cycCQQGWt6eW32JiMjNMYy4GrUa2L5d3jn33MPpGiIiclsMI64oORmYNs3y9h0d0voRIiIiN8Qw4qoOH5b3hN633pIepkdERORmGEZclVoNvPGGvHNmz3ZMX4iIiByIYcSVyZ2uKSwE9uxxWHeIiIgcgWHE1R0+DKhUlrdPSeFiViIicisMI65OrQb+/GfL27e2cjErERG5FYYRd/DQQ/IXs3K6hoiI3ATDiDuQOzoCAL/9LadriIjILTCMuIsHHwT8/S1v39goLWglIiJycQwj7kKtBnbskHdOaqpj+kJERGRHDCPuZOFCYNEiy9t//TXwpz85rj9ERER2wDDibnbuBPz8LG+/eTMXsxIRkUtjGHE3ajXwwAPyzmHtESIicmEMI+7owQfljY6w9ggREbkwhhF3pFYDr70m7xzWHiEiIhfFMOKu5D63BgCWLuV0DRERuRyGEXd2+LC86ZqWFuCxxxzXHyIiIiswjLgztRp44w155zz+OEdHiIjIpTCMuDu5tUf0eiAx0XH9ISIikolhxBPIrT1SUsJiaERE5DIYRjyBNbVHWAyNiIhchEoIIZTuRH8aGxsREhKChoYGBAcHK90d16TXAwMHSjVFLBUQADQ1SWGGiIjIziz9/c2REU9hzYP0uLuGiIhcAMOIJ1m4EEhPl3cOd9cQEZHCGEY8zebNQEKC5e31eqmAGhERkUIYRjxRUZG83TV5ecDq1Y7rDxERUR8YRjyRNcXQNm3i7hoiIlIEw4inWrgQmDlT3jn33MP1I0RE5HRWhZGsrCyMHDkS/v7+iI+PR1FRUa9tly9fDpVK1eM1duxYqztNFnr/fXntdTpgyRLH9IWIiKgXssNIbm4u0tLSsGHDBpSVlSExMRFz5sxBZWWl2fbPPfccqqurja9z585h8ODBWLhwoc2dp35oNPJKxQPAW29JoYSIiMhJZBc9mzJlCiZNmoTs7GzjsdGjR2PevHnIzMzs9/x33nkH8+fPx+nTpxEbG2vRe7LomQ30eqm4WXu75eeMGQOcOOG4PhERkVdwSNEznU6H0tJSJCUlmRxPSkpCcXGxRdfIycnBzTff3GcQaWtrQ2Njo8mLrGTNYtaKCj67hoiInEZWGKmrq4Ner0d4eLjJ8fDwcNTU1PR7fnV1Nf71r3/hf/7nf/psl5mZiZCQEOMrOjpaTjepu4UL5YcLPruGiIicxKoFrCqVyuRrIUSPY+b84x//wKBBgzBv3rw+261fvx4NDQ3G17lz56zpJnW1aRNw113yzrn7bu6uISIih/OV0zgsLAxqtbrHKEhtbW2P0ZLuhBDYvn07UlJSoNFo+myr1Wqh1WrldI0skZsLaLWWB4yODmDsWODzzx3bLyIi8mqyRkY0Gg3i4+NRUFBgcrygoADTpk3r89xDhw7h66+/xm9/+1v5vST7sGb9yBdfcP0IERE5lOxpmoyMDLz88svYvn07Tp48ifT0dFRWViI1NRWANMWydOnSHufl5ORgypQpGDdunO29JuslJwP9BMceNm/mdl8iInIYWdM0AJCcnIz6+no8+uijqK6uxrhx47B//37j7pjq6uoeNUcaGhqQl5eH5557zj69JtscPix/u+/VVwNcu0NERA4gu86IElhnxAF275ZfEO3224F9+xzTHyIi8jgOqTNCHmThQiA9Xd45774r/xwiIqJ+MIx4s82bgalT5Z2zZQuwerVDukNERN6JYcTbHTkC+Mj8a7BpEwuiERGR3TCMeDu1Gti5U/55KSksiEZERHbBMELSdt+5c+Wd09oKLF7smP4QEZFXYRghyd69wHXXyTtnzx6uHyEiIpsxjNAVFRWAn5+8c7h+hIiIbMQwQldYUy4eAJYs4foRIiKyGsMImVq4UH4xtPZ2IDHRMf0hIiKPxzBCPe3cCfj7yzunpIQP1CMiIqswjFBPajWwY4f88zZv5voRIiKSjWGEzFu40LqRjrvv5voRIiKShWGEerdpE5CRIe+cjg5g9GjH9IeIiDwSwwj17ZlngLvuknfOV18Bkyc7pj9ERORxGEaof7m58uuPlJbKr+pKREReiWGE+mdt/ZF33wXS0+3fHyIi8igMI2QZaxe0btnCkvFERNQnhhGy3KZNQFqadedxyy8REfWCYYTkefZZ4Pbb5Z/HLb9ERNQLhhGSb98+4Npr5Z3T0QGMGeOY/hARkVtjGCHrnDwJ+Mj86/Pll9xhQ0REPTCMkHXUauDNN+Wfxx02RETUDcMIWc+WHTZ8qB4REf2IYYRsY03JeEB6qB63/BIRERhGyB6eeYZbfomIyGoMI2Qfzz4LTJ0q/7xFiwCdzv79ISIit8EwQvZz5Ij8HTZCAAEBwO7djukTERG5PIYRsh9rd9h0dkojJKtW2b9PRETk8hhGyL6s3WEDSGtPuKiViMjrMIyQ/Vm7w8ZwLhe1EhF5FYYRcgxrd9gAQHIyn2NDRORFGEbIcax9qF5nJxAXZ//+EBGRS2IYIcfatw+YNEn+eV9/bd15RETkdhhGyPFKS4H4ePnnlZUBo0bZvz9ERORSGEbIOY4ds27K5vRpBhIiIg/HMELOs28fsGKF/PNOnwauvZaLWomIPBTDCDnXc89ZN0Ly9deARsNKrUREHohhhJzP2kWthkqta9bYv09ERKQYhhFSRmmp9btlNm4EcnPt2x8iIlIMwwgpp7QUmDjRunMXL2YgISLyEAwjpKxPPgFGjrTu3MWL+XA9IiIPwDBCyjt1yvpA8swzQHq6fftDREROxTBCruHUKeunbLZsAX71K7t2h4iInIdhhFzHJ59YV6kVkHbozJ1r3/4QEZFTMIyQazl2DLjtNuvOffddICGBxdGIiNwMwwi5nnfftT6QfPQRoNUC+fn27RMRETkMwwi5pnfftX7KRq8H7rqL1VqJiNwEwwi5LmsfrmewaBGwa5f9+kNERA7BMEKubd8+IC3N+vOXLOFOGyIiF8cwQq7v2WeBjAzrz7f2WThEROQUDCPkHp55xrZAUlYGREZypw0RkQtiGCH38cwztpV/r6kBfH35TBsiIhfDMELuZeNGaZeMSmX9NRYv5joSIiIXwjBC7mfBAqC9HbjmGuuvwXUkREQug2GE3JNaDXz1lfW1SACuIyEichEMI+Tejh2z7Zk0XEdCRKQ4hhFyf3v32h4mFi/mg/aIiBTCMEKeYdEioKMDCA+3/hrvvgsMHgzodPbrFxER9YthhDyHWi1Nu4waZf01Ll2SHrS3cCHXkhAROQnDCHmeb76x7Zk2ALBnD+Dnx7UkREROYFUYycrKwsiRI+Hv74/4+HgUFRX12b6trQ0bNmxAbGwstFotrr76amzfvt2qDhNZZN8+24OEEKxJQkTkBLLDSG5uLtLS0rBhwwaUlZUhMTERc+bMQWVlZa/nLFq0CAcPHkROTg6++OIL7Nq1C3FxcTZ1nKhf9lhHAkjBZvhwriUhInIQlRBCyDlhypQpmDRpErKzs43HRo8ejXnz5iEzM7NH+/feew+LFy/GqVOnMHjwYKs62djYiJCQEDQ0NCA4ONiqa5CXu/pq4NQp26+zYAHw5pvS+hQiIuqTpb+/feVcVKfTobS0FOvWrTM5npSUhOLiYrPn7N27F5MnT8bTTz+N1157DQMGDMCvfvUrPPbYYwgICDB7TltbG9ra2kw+jCX0ej3a29st/DRkT35+flC78i/ob76Rtu6++65t19mzB9BopECycKF9+kZE5OVkhZG6ujro9XqEdxv2Dg8PR01NjdlzTp06hSNHjsDf3x9vv/026urq8Ic//AHfffddr+tGMjMz8cgjj1jcLyEEampq8P3331t8DtnfoEGDEBERAZUtz41xpH37gLfeApKTbbtOZ6c0BcRREiIiu5AVRgy6/7IRQvT6C6izsxMqlQpvvPEGQkJCAACbN2/GggUL8OKLL5odHVm/fj0yujwuvrGxEdHR0b32xxBEhg4disDAQNf9ZeihhBBobm5GbW0tACAyMlLhHvVh0SLgrruA6Gigutq2a+3ZI20DfuMN2wMOEZEXkxVGwsLCoFare4yC1NbW9hgtMYiMjMSwYcOMQQSQ1pgIIXD+/Hlce+21Pc7RarXQarUW9Umv1xuDSGhoqIxPQ/ZkCJW1tbUYOnSoa0/ZqNXAhQtAejqwZYtt19LrpR03zz0HFBVxlISIyAqydtNoNBrEx8ejoKDA5HhBQQGmTZtm9pzp06fjwoULuHz5svHYl19+CR8fHwwfPtyKLpsyrBEJDAy0+VpkG8PPwG3W7Tz7LNDWBowebfu1SkqkZ9z85S8slkZEJJPsrb0ZGRl4+eWXsX37dpw8eRLp6emorKxEamoqAGmKZenSpcb2S5YsQWhoKO69915UVFTg8OHDWL16NX7zm9/0uoDVGpyaUZ5b/gw0GqCiAugyLWiTxx6Trrlrl32uR0TkBWSHkeTkZGzZsgWPPvooJkyYgMOHD2P//v2IjY0FAFRXV5vUHBk4cCAKCgrw/fffY/Lkybjnnnswd+5cPP/88/b7FES2euYZYPdu+0yzdHYCS5ZI61JYm4SIqF+y64wooa99yq2trTh9+rSxIiwpxyN+FoY1IHv22O+a3HVDRF7K0jojfDaNwuSW1j906BDi4+Ph7++PUaNGYevWrSbfP3HiBO666y6MGDECKpUKW2xdoOlt1GpphKStDRg2zD7X3LNHWk/y4INcT0JEZAbDiIFeDxQWSnP9hYVO+aUht7T+6dOnceuttyIxMRFlZWV44IEHsGLFCuTl5RnbNDc3Y9SoUXjyyScRERHh8M/gsTQa4Px5qVCavTz+OBe5EhGZI9xAQ0ODACAaGhp6fK+lpUVUVFSIlpYW698gL0+I4cOFkB6NJr2GD5eOO9DPf/5zkZqaanIsLi5OrFu3zmz7NWvWiLi4OJNjv/vd78TUqVPNto+NjRXPPvusXfpqCbv8LFxRbq4QarXp3w9bXz4+QuzcqfQnIyJyqL5+f3fFkZH8fGlO//x50+NVVdLx/HyHvK2htH5SUpLJ8b5K65eUlPRoP3v2bBw7dsx9ttO6o0WLpGmbBQvsd00uciUiMvLuMKLXAytXSv+t2p3hWFqaQ4bUrSmtX1NTY7Z9R0cH6urq7N5H6sKwlmT3bsDPz37XPX9equI6ejRQUMDpGyLySt4dRoqKeo6IdCUEcO6c1M5B5JTW7629uePkIAsWAC0t0mJUHzv+3+fzz4GkJCAgAMjNtd91iYjcgHeHEUufTWLrM0zMsKa0fkREhNn2vr6+LIXvTGo18Oij0vTKgw/a99rt7dLW4uHDOVJCRF7Du8OIpQ90c8CD36wprZ+QkNCj/YEDBzB58mT42XPqgCxjCCUdHdLD9+ypqkoaKdFqufuGiDyed4eRxETpv0B7m+JQqaQFhomJDnl7uaX1U1NTcfbsWWRkZODkyZPYvn07cnJysGrVKmMbnU6H8vJylJeXQ6fToaqqCuXl5fj6668d8hkIUijZs0da5DpmjH2vrddLJeZ9fYGlS7nYlYg8k1P29tjIoVt78/KEUKmkV9etl4ZjDt7e++KLL4rY2Fih0WjEpEmTxKFDh4zfW7ZsmZgxY4ZJ+8LCQjFx4kSh0WjEiBEjRHZ2tsn3T58+LQD0eHW/jiN47NZeuXJzpa279twK3PV1ww1CtLUp/SmJiPpl6dZeloMHpO27K1eaLmaNjpYeLz9/vvUd9zIeUQ7eXvR64KGHgCeeML9byx7i4oDnnwduuoml5onIJbEcvBzz5wNnzgAffgjs3Cn9efo0gwhZT62WKq62t9t/kauBYQeOnx9LzRORW2MYMVCrgZkzgbvvlv7kf2mSPThykauBEFdKzScmchcOEbkdhhEiZ+i6yHXGDMe9z5EjV0ZLuOCViNwEwwiRM2k00oMYHR1KhABee43VXYnILTCMECnBWaEEuLK2hNM4ROSiGEaIlOTMUAJcmcbx9ZX+bGlx/HsSEfWDYYTIFXQNJUuX2ve5N70pKAACA4HBg4FNm7i+hIgUwzBC5Eo0GuDVV6Vg8P77wLBhjn/PS5eA1aul9SXDhjGYEJHTMYwQuSK1WppGOX8eaG4GfvpT57zvhQtXgslVVwG//z2ncojI4RhGiFxdQABw/PiVKRxn+f57YOtWaSonKAi45x4ufiUih2AYUVhWVpaxfHp8fDyKior6bH/o0CHEx8fD398fo0aNwtatW3u0ycvLw5gxY6DVajFmzBi8/fbbJt8/fPgw5s6di6ioKKhUKrzzzjv2/EjkKIYpnI4OYMMG56wrMbh8WapObFj8ev31nM4hIrthGPmRvlOPwjOF2PXpLhSeKYS+0/H/9Zebm4u0tDRs2LABZWVlSExMxJw5c1BZWWm2/enTp3HrrbciMTERZWVleOCBB7BixQrk5eUZ25SUlCA5ORkpKSk4fvw4UlJSsGjRInz88cfGNj/88APGjx+PF154weGfkRzAUGresK4kLs75ffjyyyvTOcHBwJIlHDUhIqvxQXkA8k/mY+V7K3G+8cqD8oYHD8dztzyH+aMd93yaKVOmYNKkScjOzjYeGz16NObNm4fMzMwe7deuXYu9e/fi5MmTxmOpqak4fvw4SkpKAADJyclobGzEv/71L2ObW265BVdddRV27drV45oqlQpvv/025s2bZ/Pn4YPyFKTTSQ/N27wZqK5Wti+RkcAdd0h9CQhQti9EpCg+KM9C+SfzseCtBSZBBACqGquw4K0FyD+Z75D31el0KC0tRVJSksnxpKQkFBcXmz2npKSkR/vZs2fj2LFjaG9v77NNb9ckD6HRAKtWSQtQnVWzpDfV1VfWmmg0wHXXcSEsEfXJq8OIvlOPle+thEDPwSHDsbT30hwyZVNXVwe9Xo/w8HCT4+Hh4aipqTF7Tk1Njdn2HR0dqKur67NNb9ckD9S1ZsnGjVIYUKmU6Ut7O/DVV1fCib8/MG4cF8MSkQmvDiNFlUU9RkS6EhA413gORZV9Lyq1harbLwkhRI9j/bXvflzuNclDGUZLvvhCCgXvvy8tPFVSWxtw4oTpYtihQ4Hp07kglsiLeXUYqW6ybG7d0nZyhIWFQa1W9xixqK2t7TGyYRAREWG2va+vL0JDQ/ts09s1yUsY6pZ8/vmVEZNBg5TuleTbb4HiYtMFsRw9IfIqXh1GIoMi7dpODo1Gg/j4eBQUFJgcLygowLRp08yek5CQ0KP9gQMHMHnyZPj5+fXZprdrkhcyjJhcuiQVVPvd71wnmABAU1PP0ZMhQxhQiDyYV4eRxJhEDA8eDhXMT2GooEJ0cDQSYxId8v4ZGRl4+eWXsX37dpw8eRLp6emorKxEamoqAGD9+vVY2qXIVWpqKs6ePYuMjAycPHkS27dvR05ODlatWmVss3LlShw4cABPPfUUPv/8czz11FP44IMPkJaWZmxz+fJllJeXo7y8HIC0Zbi8vLzXLcXkwQICpPUcly5dGTGJtH/4tlldHQMKkScTbqChoUEAEA0NDT2+19LSIioqKkRLS4tV186ryBOqh1VC9bBK4GEYX4ZjeRV5tna/Ty+++KKIjY0VGo1GTJo0SRw6dMj4vWXLlokZM2aYtC8sLBQTJ04UGo1GjBgxQmRnZ/e45u7du8X1118v/Pz8RFxcnMjLM/0MH374oQDQ47Vs2TKbPoutPwtyIW1tQmzcKMR11wkBuM8rKEiIsDAhxo4V4qmnpM9BRIrp6/d3V6wzAvN1RqKDo7Hlli0OrTPiaVhnxEPp9cCBA9IC048/Bn74QekeyePvL+3k8fMDRo0C5s8HVqyQpquIyKEsrTPCMPIjfaceRZVFqG6qRmRQJBJjEqH2Udvada/CMOIldDpgyxbgb3+THuTnrjQaICRECirDhgF33smQQmRnDCPkdPxZeCG9Hjh4EHjlFWn05LvvlO6R7QYOlJ5Y3NYmjabExUk7fW6+WdqVREQWszSM+DqxT0TkaQxbhg1Vfw1l6fPzgU8/lR6w524uXzbtd1WVFLgA6enFfn7S5w4IkL4ePx5Yvhy46SaGFSIrMYwQkf0Ytg0bdnh1DSeffSZt23Vn5vpv2OUDAIMHS4FEr+caFSIZOE1DdsOfBfWrazipqgJqa4HWVqV75RxarVRtFpA+MwMLeQGuGSGn48+CrNLSAqSnS1MhVVXe/UA9w6JarVZas6LXSzVVwsKACRM4HURuh2tGiMg9GAqvGXQfPbl0yf2ndyyl00nl8burrQUqKq5MBxkW2QJXRlm4joXcGMMIEbmW7utOAO8OKOZ0X2TbXW/rWNRqqe4KIIWYzk4pvCQkAPfey/BCimEYISLXZ0lAEUIqG+/N0zzm9Lfdur4eOHMG2LVL+rq3UZeuU0dc60J2xjUjZDf8WZBL6BpSzp+XfqE2NXnPQlklGNa6dB11MRdiDCMzajUQFcVCc17A0jUjXv2gPFeQlZVl/OUdHx+PoqKiPtsfOnQI8fHx8Pf3x6hRo7C161z7j/Ly8jBmzBhotVqMGTMGb7/9tuz3zc/Px+zZsxEWFgaVSmV8qB6RyzOMohQXA5WV0nqLlpYrDwKcOlV6yN7gwfwlaC+GtS7nzkmvb7+VRmS+/VYKhF2/PndOGokpLpaKyWm10lRRTIy02yg0VPozOtr065iYnm1iY/mwRE/hwOfj2I0jH5Rn0NEhxIcfCrFzp/RnR4dNl7PIm2++Kfz8/MS2bdtERUWFWLlypRgwYIA4e/as2fanTp0SgYGBYuXKlaKiokJs27ZN+Pn5iT179hjbFBcXC7VaLZ544glx8uRJ8cQTTwhfX1/x0UcfyXrfHTt2iEceeURs27ZNABBlZWX9fh4+KI/cUnOzEL/7nRDXXCPEkCFCREcLERqq/EP/+LLuddVV0s9x8OArP8/o6P6PDR9+5euhQ4UYM0aIJUuEOHDAOb8QPBQflCdDfj6wcqXpYzaGDweee06aEnWUKVOmYNKkScjOzjYeGz16NObNm4fMzMwe7deuXYu9e/fi5MmTxmOpqak4fvw4SkpKAADJyclobGzEv/71L2ObW265BVdddRV2/TgnLOd9z5w5g5EjR6KsrAwTJkzo8/NwmoY8StdS9//975WpHr1eGmnh2hTvMnAgMGhQzyknwHRaytKpKkMbIYDwcGDpUiAtzeNG67i110L5+cCCBdLfh66qqqTje/Y4JpDodDqUlpZi3bp1JseTkpJQXFxs9pySkhIkGcpu/2j27NnIyclBe3s7/Pz8UFJSgvT09B5ttmzZYvX7Enml7qXuu+u+gBaQ/iHhGhXP1N8OJlvU1QFr10ovw1Om+wsx1oYfF12Q7NVhRK+XRkTMjQ0JAahUUlC94w7773arq6uDXq9HeHi4yfHw8HDU1NSYPaempsZs+46ODtTV1SEyMrLXNoZrWvO+RGSGuR0+XRmebvzqq8DFi4CPj+kvBQYWMqe1VZm/FxcvAiUlwJo10t/pp5926tt7dRgpKur7CehCSGutioqAmTMd0weVStXtPUWPY/21737ckmvKfV8ikkmjkf5hX7Om9zaG0ZW8POCbb3r+V2tLi3s+bJDclxDSQm/AqYHEq8NIdbV928kRFhYGtVrdYzSitra2x6iFQUREhNn2vr6+CA0N7bON4ZrWvC8ROUh/oyvAlbUrOTnARx9J4cTc0DvXsZA9bd4MPP6406ZsvHprb2SkfdvJodFoEB8fj4KCApPjBQUFmDZtmtlzEhISerQ/cOAAJk+eDD8/vz7bGK5pzfsSkYIMa1dyc4GzZ6UiZbW10rZlw9bl+nqgufnK9uWEhCtbYaOjr2xlHjJE+jo6GuhjMSER9HogK8t57+fwfT124KitvR0d0m4ulcr8DjGVStr95ahdXYYttjk5OaKiokKkpaWJAQMGiDNnzgghhFi3bp1ISUkxtjds7U1PTxcVFRUiJyenx9be//znP0KtVosnn3xSnDx5Ujz55JO9bu3t7X2FEKK+vl6UlZWJf/7znwKAePPNN0VZWZmorq7u9fNway+Rm+noEOL994VYtEiImBjLtr76+yu/fZcv57zuv9/mv2KWbu2Fze/kBI6sM5KXJ4WO7oHEcCwvz9be9+3FF18UsbGxQqPRiEmTJolDhw4Zv7ds2TIxY8YMk/aFhYVi4sSJQqPRiBEjRojs7Owe19y9e7e4/vrrhZ+fn4iLixN5Zj5EX+8rhBCvvPKKANDj9dBDD/X6WRhGiLxEW5sQGzcKMXWqvPodDDPu9Xr2WZv/qrDOiAzm6oxER0sL4R1ZZ8TTsM4IEVmkpQVIT5fWwjQ0WLc9taMDaGxU7jN4OrVamvqzcc0I64zIMH++tH23qEharBoZCSQm8uGVREQOERAAmHmUhWx6PXDgALBpE/D550B7u+21ObiDSZKR4dR6IwwjP1KrHbd9l4iIHECtBubMkV721NsOJnsXIXPFp0yrVKwzQkREpLj+qu/ak7mnTMuppMoKrERERGQTS2rNeAGvrjNCREREyvOYMNLZ2al0F7wefwZERGQNt5+m0Wg08PHxwYULFzBkyBBoNBo+Y8XJhBDQ6XT49ttv4ePjA42HPQKbiIgcy6owkpWVhY0bN6K6uhpjx47Fli1bkJiYaLZtYWEhbrzxxh7HT548ibi4OGve3oSPjw9GjhyJ6upqXLhwwebrkfUCAwMRExMDHx+PGXAjIiInkB1GcnNzkZaWhqysLEyfPh1///vfMWfOHFRUVCAmJqbX87744guTgidDhgyxrsdmaDQaxMTEoKOjA3q93m7XJcup1Wr4+vpyVIqIiGSTXYF1ypQpmDRpErKzs43HRo8ejXnz5iEzM7NHe8PIyKVLlzBo0CCrOmlpBTciIiJyHZb+/pY1nq7T6VBaWoqkbnuvk5KSUFxc3Oe5EydORGRkJGbNmoUPP/ywz7ZtbW1obGw0eREREZFnkhVG6urqoNfrER4ebnI8PDwcNTU1Zs+JjIzESy+9hLy8POTn5+P666/HrFmzcPjw4V7fJzMzEyEhIcZXdHS0nG4SERGRG7FqAWv3dQFCiF7XClx//fW4/vrrjV8nJCTg3Llz2LRpE2644Qaz56xfvx4ZGRnGrxsbGxlIiIiIPJSsMBIWFga1Wt1jFKS2trbHaElfpk6ditdff73X72u1Wmi1WuPXhmUtnK4hIiJyH4bf2/0tT5UVRjQaDeLj41FQUIA777zTeLygoAB33HGHxdcpKytDZGSkxe2bmpoAgKMjREREbqipqQkhISG9fl/2NE1GRgZSUlIwefJkJCQk4KWXXkJlZSVSU1MBSFMsVVVV2LFjBwBgy5YtGDFiBMaOHQudTofXX38deXl5yMvLs/g9o6KicO7cOQQFBdl166hh+ufcuXPcpeNgvNfOwfvsHLzPzsH77DyOutdCCDQ1NSEqKqrPdrLDSHJyMurr6/Hoo4+iuroa48aNw/79+xEbGwsAqK6uRmVlpbG9TqfDqlWrUFVVhYCAAIwdOxb//Oc/ceutt1r8nj4+Phg+fLjcrlosODiYf9GdhPfaOXifnYP32Tl4n53HEfe6rxERA9l1RjwJ65c4D++1c/A+Owfvs3PwPjuP0veadbuJiIhIUV4dRrRaLR566CGTnTvkGLzXzsH77By8z87B++w8St9rr56mISIiIuV59cgIERERKY9hhIiIiBTFMEJERESKYhghIiIiRXl1GMnKysLIkSPh7++P+Ph4FBUVKd0lt5GZmYmf/exnCAoKwtChQzFv3jx88cUXJm2EEHj44YcRFRWFgIAAzJw5EydOnDBp09bWhj/+8Y8ICwvDgAED8Ktf/Qrnz5935kdxK5mZmVCpVEhLSzMe4322n6qqKvz6179GaGgoAgMDMWHCBJSWlhq/z3ttu46ODvz5z3/GyJEjERAQgFGjRuHRRx9FZ2ensQ3vs3UOHz6MuXPnIioqCiqVCu+8847J9+11Xy9duoSUlBSEhIQgJCQEKSkp+P77723rvPBSb775pvDz8xPbtm0TFRUVYuXKlWLAgAHi7NmzSnfNLcyePVu88sor4rPPPhPl5eXitttuEzExMeLy5cvGNk8++aQICgoSeXl54tNPPxXJyckiMjJSNDY2GtukpqaKYcOGiYKCAvHJJ5+IG2+8UYwfP150dHQo8bFc2tGjR8WIESPET3/6U7Fy5Urjcd5n+/juu+9EbGysWL58ufj444/F6dOnxQcffCC+/vprYxvea9s9/vjjIjQ0VLz77rvi9OnTYvfu3WLgwIFiy5Ytxja8z9bZv3+/2LBhg8jLyxMAxNtvv23yfXvd11tuuUWMGzdOFBcXi+LiYjFu3Dhx++2329R3rw0jP//5z0VqaqrJsbi4OLFu3TqFeuTeamtrBQBx6NAhIYQQnZ2dIiIiQjz55JPGNq2trSIkJERs3bpVCCHE999/L/z8/MSbb75pbFNVVSV8fHzEe++959wP4OKamprEtddeKwoKCsSMGTOMYYT32X7Wrl0rfvGLX/T6fd5r+7jtttvEb37zG5Nj8+fPF7/+9a+FELzP9tI9jNjrvlZUVAgA4qOPPjK2KSkpEQDE559/bnV/vXKaRqfTobS0FElJSSbHk5KSUFxcrFCv3FtDQwMAYPDgwQCA06dPo6amxuQea7VazJgxw3iPS0tL0d7ebtImKioK48aN48+hm//3//4fbrvtNtx8880mx3mf7Wfv3r2YPHkyFi5ciKFDh2LixInYtm2b8fu81/bxi1/8AgcPHsSXX34JADh+/DiOHDlifF4Z77Nj2Ou+lpSUICQkBFOmTDG2mTp1KkJCQmy697IflOcJ6urqoNfrER4ebnI8PDwcNTU1CvXKfQkhkJGRgV/84hcYN24cABjvo7l7fPbsWWMbjUaDq666qkcb/hyuePPNN/HJJ5/g//7v/3p8j/fZfk6dOoXs7GxkZGTggQcewNGjR7FixQpotVosXbqU99pO1q5di4aGBsTFxUGtVkOv1+Ovf/0r7r77bgD8O+0o9rqvNTU1GDp0aI/rDx061KZ775VhxEClUpl8LYTocYz6d//99+O///0vjhw50uN71txj/hyuOHfuHFauXIkDBw7A39+/13a8z7br7OzE5MmT8cQTTwAAJk6ciBMnTiA7OxtLly41tuO9tk1ubi5ef/117Ny5E2PHjkV5eTnS0tIQFRWFZcuWGdvxPjuGPe6rufa23nuvnKYJCwuDWq3ukeJqa2t7pEbq2x//+Efs3bsXH374IYYPH248HhERAQB93uOIiAjodDpcunSp1zberrS0FLW1tYiPj4evry98fX1x6NAhPP/88/D19TXeJ95n20VGRmLMmDEmx0aPHo3KykoA/DttL6tXr8a6deuwePFi/OQnP0FKSgrS09ORmZkJgPfZUex1XyMiInDx4sUe1//2229tuvdeGUY0Gg3i4+NRUFBgcrygoADTpk1TqFfuRQiB+++/H/n5+fj3v/+NkSNHmnx/5MiRiIiIMLnHOp0Ohw4dMt7j+Ph4+Pn5mbSprq7GZ599xp/Dj2bNmoVPP/0U5eXlxtfkyZNxzz33oLy8HKNGjeJ9tpPp06f32J7+5ZdfIjY2FgD/TttLc3MzfHxMf/Wo1Wrj1l7eZ8ew131NSEhAQ0MDjh49amzz8ccfo6GhwbZ7b/XSVzdn2Nqbk5MjKioqRFpamhgwYIA4c+aM0l1zC7///e9FSEiIKCwsFNXV1cZXc3Ozsc2TTz4pQkJCRH5+vvj000/F3XffbXYb2fDhw8UHH3wgPvnkE3HTTTd5/fa8/nTdTSME77O9HD16VPj6+oq//vWv4quvvhJvvPGGCAwMFK+//rqxDe+17ZYtWyaGDRtm3Nqbn58vwsLCxJo1a4xteJ+t09TUJMrKykRZWZkAIDZv3izKysqMJSvsdV9vueUW8dOf/lSUlJSIkpIS8ZOf/IRbe23x4osvitjYWKHRaMSkSZOM21KpfwDMvl555RVjm87OTvHQQw+JiIgIodVqxQ033CA+/fRTk+u0tLSI+++/XwwePFgEBASI22+/XVRWVjr507iX7mGE99l+9u3bJ8aNGye0Wq2Ii4sTL730ksn3ea9t19jYKFauXCliYmKEv7+/GDVqlNiwYYNoa2sztuF9ts6HH35o9t/lZcuWCSHsd1/r6+vFPffcI4KCgkRQUJC45557xKVLl2zqu0oIIawfVyEiIiKyjVeuGSEiIiLXwTBCREREimIYISIiIkUxjBAREZGiGEaIiIhIUQwjREREpCiGESIiIlIUwwgREREpimGEiIiIFMUwQkRERIpiGCEiIiJFMYwQERGRov4/us+FIRVbMuIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "####plot the cost function for different alpha values, 0.01,0.001,0.0001\n",
    "\n",
    "# For alpha =0.01\n",
    "\n",
    "alphas = [0.01, 0.001, 0.0001]\n",
    "num_iterations = 1000\n",
    "colors = ['r', 'g', 'b']\n",
    "for i, alpha in enumerate(alphas):\n",
    "    print(f'current learning rate is {alpha}')\n",
    "\n",
    "    all_costs = []\n",
    "    b,w = initialize_betas(X_new.shape[1])\n",
    "    print(\"initial guess of b and w: \" , b ,w)\n",
    "\n",
    "    for each_iter in range (num_iterations ):\n",
    "        ################finish the code below##################\n",
    "        y_hat = sigmoid(b,w,X_new)\n",
    "        current_cost = get_cost(Y,y_hat)\n",
    "        prev_b = b\n",
    "        prev_w = w\n",
    "        \n",
    "        ################finish the code below##################\n",
    "        b, w = backprop (prev_b, prev_w, Y, y_hat,X_new, alpha)\n",
    "        all_costs.append(current_cost)\n",
    "        if each_iter % 30 == 0:\n",
    "            print('Iteration: ', each_iter, 'Cost: ', current_cost)\n",
    "            each_iter += 1\n",
    "\n",
    "    #print('b_0:', b_0, 'b_1:',b_1,'b_2:',b_2,'b_3:',b_3,'b_4:', b_4, 'b_5:',b_5,'b_6:',b_6,'b_7:',b_7,'b_8:',b_8,'b_9:',b_9)\n",
    "    print(\"Final estimates of b and q are: \", b,w)\n",
    "    plt.scatter( range(num_iterations), all_costs, c = colors[i], label = f'{alpha}')\n",
    "    plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the experiment above, which is the best choice for alpha? Why?\n",
    "\n",
    "**Answer Below**:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Answer: \n",
    "#### 0.01 is the best alpha.\n",
    "#### (1)when alpha is equal to 0.01, the model converges to best alpha quicker than the others alphas(0.001, 0.0001)\n",
    "#### (2)the model achieves smaller cost under given number of iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}