10_intro_rna.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "adolescent-theorem",
   "metadata": {},
   "source": [
    "# Introducción a redes neuronales artificiales con Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vertical-freedom",
   "metadata": {},
   "source": [
    "Las redes neuronales artificiales (RNAs) aparecieron en la investigación en Inteligencia Artificial en el año 1943 en un artículo escrito por un neurofisiólogo y un matemático (McCulloch & Pitts 1943).\n",
    "\n",
    "Había mucho interés al principio, con predicciones que iban a desarrollar \"máquinas inteligentes\". Pero la realidad no fue así (las redes neuronales en esa época todavía eran bastante básicas). La comunidad se perdió interés en ~1960s.\n",
    "\n",
    "~1980s inventaron nuevas arquitecturas y métodos de entrenamiento, pero el progreso fue lento y en los 90 otros métodos fueron inventados (e.g. *support vector machines*). Se fue el interés de nuevo...\n",
    "\n",
    "Hoy en día, hay mucho más interés en RNAs. Por qué?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pretty-celebrity",
   "metadata": {},
   "source": [
    "* Hay muchos datos para entrenar las redes neuronales, y frecuentemente superan los otros métodos en problemas grandes y complejos.\n",
    "* Hay mucho más poder computacional (GPUs).\n",
    "* Los algoritmos de entrenamiento han sido mejorados.\n",
    "* Algunas limitaciones de las RNAs no son tan graves (por ejemplo, es poco común encontrar un mínimo local en el entrenamiento, y si pasa, muchas veces están cercas al mínimo global).\n",
    "* Hoy en día es una tecnología **exponencial**: hay mucho interés, así que atraen investigadores y fondos, los resultados mejoran y genera más interés..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bearing-charlotte",
   "metadata": {},
   "source": [
    "##### Neuronas biológicas\n",
    "\n",
    "![](figures_intro_rna/fig10-1.png)\n",
    "\n",
    "Cada neurona es bastante simple, pero están organizadas en una red de billones de neuronas, cada una con conexiones a miles de otras.\n",
    "\n",
    "![](figures_intro_rna/fig10-2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "purple-customs",
   "metadata": {},
   "source": [
    "### Cálculos lógicos con neuronas\n",
    "\n",
    "El modelo de una neurona **artificial**, desarrollado por McCulloch y Pitts, es mucho más básico: tiene entradas binarias y una salida binaria. La neurona se activa cuando hay más que un cierto número de sus entradas activas.\n",
    "\n",
    "![](figures_intro_rna/fig10-3.png)\n",
    "\n",
    "* La primera red a la izquierda es la función de identidad: si neurona $A$ se activa, neurona $C$ se activa también.\n",
    "* La segunda red corresponde al operador lógico AND: neurona $C$ se activa solamente cuando ambas neuronas $A$ y $B$ se activan.\n",
    "* La tercera red corresponde al operador lógico OR: neurona $C$ se activa si $A$ o $B$ se activan.\n",
    "* Finalmente, si se puede tener conexiones que inhiben la actividad de una neurona, la cuarta red corresponde a $A$ AND NOT $B$: neurona $C$ se activa solamente si $A$ se activa y $B$ se apaga.\n",
    "\n",
    "En principio, podemos combinar redes así para calcular cualquier expresión lógica."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "first-float",
   "metadata": {},
   "source": [
    "### El Perceptrón\n",
    "\n",
    "Inventado en 1957 por Frank Rosenblatt, basado en un modelo para una neurona que se llama **unidad lógica de umbral** (*threshold logic unit*, TLU).\n",
    "\n",
    "Las entradas y salida ahora son números y cada conexión de entrada está asociada con un peso.\n",
    "\n",
    "La TLU calcula la suma ponderada de sus entradas:\n",
    "\n",
    "$$z = w_1x_1 + w_2x_2 + \\cdots + w_nx_n = \\boldsymbol{x}^T \\boldsymbol{w}$$\n",
    "\n",
    "Después se aplica una función escalón para determinar la salida de la neurona:\n",
    "\n",
    "$$h_w(\\boldsymbol{x}) = \\Theta(z)$$\n",
    "\n",
    "![](figures_intro_rna/fig10-4.png)\n",
    "\n",
    "Típicamente se usa la función escalón de Heaviside. A veces se usa la función de signo:\n",
    "\n",
    "$\\Theta(z) = \\begin{cases} 0 & z < 0 \\\\ 1 & z \\geq 0 \\end{cases}$\n",
    "\n",
    "$\\text{sgn}(z) = \\begin{cases} -1 & z < 0 \\\\ 0 & z = 0 \\\\ +1 & z > 0 \\end{cases}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "empty-innocent",
   "metadata": {},
   "source": [
    "Un **perceptrón** está compuesto de una capa de TLUs, con cada TLU conectada a todas las entradas.\n",
    "\n",
    "Cuando todas las neuronas en una capa están conectadas a todas las neuronas en la capa anterior, se llama una **capa totalmente conectada** o una **capa densa**.\n",
    "\n",
    "Para enfatizar que todas las entradas van a todas las TLU, es común dibujar una capa de neuronas de entrada, que solamente pasan sus entradas a la próxima capa (*passthrough neurons*). Todas las neuronas de entrada forman la **capa de entrada**.\n",
    "\n",
    "Típicamente un *bias feature* está agregado ($x_0 = 1$) usando una neurona de *bias* que siempre tiene salida igual a $1$.\n",
    "\n",
    "![](figures_intro_rna/fig10-5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wicked-rough",
   "metadata": {},
   "source": [
    "El perceptrón arriba puede clasificar instancias en $3$ clases binarias diferentes simultaneamente, así que es un clasificador de *multioutput*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fallen-citizenship",
   "metadata": {},
   "source": [
    "Se puede determinar las salidas de una capa densa (totalmente conectada) usando:\n",
    "\n",
    "$$h_{\\boldsymbol{W},\\boldsymbol{b}}(\\boldsymbol{X}) = \\phi(\\boldsymbol{X}\\boldsymbol{W}+\\boldsymbol{b})$$\n",
    "\n",
    "* $\\boldsymbol{X}$ es una matriz de *features* de entrada, una fila por instancia, una columna por *feature*.\n",
    "* $\\boldsymbol{W}$ es una matriz de pesos, que contiene todos los pesos de las conexiones aparte de las conexiones con la neurona de *bias*. Tiene una fila por neurona de entrada, y una columna por neurona TLU.\n",
    "* El vector de *bias* $\\boldsymbol{b}$ contiene todos los pesos de las conexiones entre la neurona de *bias* y las neuronas TLU. Hay un término de *bias* por neurona TLU.\n",
    "* La función $\\phi$ se llama la **función de activación**. En el caso de una TLU, es una función escalón."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "finished-timber",
   "metadata": {},
   "source": [
    "##### Entrenamiento de un perceptrón\n",
    "\n",
    "Rosenblatt propuso la **regla de Hebb** para entrenar un perceptrón. Motivada por el comportamiento de neuronas biológicas, la regla de Hebb aumenta el peso de una conexión entre dos neuronas cuando ambas neuronas tienen la misma salida.\n",
    "\n",
    "De hecho, un perceptrón ocupa una variante de esta regla que toma en cuenta el error de la red: hay refuerza de conexiones que ayudan en reducir el error.\n",
    "\n",
    "La red recibe las instancias de entrenamiento una a la vez y realiza una predicción (hay una salida). Para cada neurona de salida que produce una predicción incorrecta, hay refuerza de los pesos de conexiones de entradas que habrían contribuido a una predicción correcta.\n",
    "\n",
    "$$w_{i,j}^{(t+1)} = w_{i,j}^{(t)} + \\eta \\left( y_j - \\hat{y}_j^{(t)} \\right) x_i$$\n",
    "\n",
    "* $w_{i,j}$ es el peso de la conexión entre la $i$-esima neurona de entrada y la $j$-esima neurona de salida.\n",
    "* $x_i$ es el $i$-esima valor de entrada de la instancia de entrenamiento actual.\n",
    "* $\\hat{y}_j$ es la salida de la $j$-esima neurona de salida, para la instancia de entrenamiento actual.\n",
    "* $y_j$ es la salida objetivo de la $j$-esima neurona de salida, para la instancia de entrenamiento actual.\n",
    "* $\\eta$ es la taza de aprendizaje."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "personal-brief",
   "metadata": {},
   "source": [
    "El límite de decisión de cada neurona de salida es lineal, así que un perceptrón no puede aprender un patrón complejo.\n",
    "\n",
    "Si las instancias están linealmente separables, Rosenblatt demostró que el algoritmo converge a una solución (**teorema de convergencia del Perceptrón**)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "urban-midnight",
   "metadata": {},
   "source": [
    "Scikit-Learn tiene la clase `Perceptron` que implementa una red con una sola TLU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "religious-casting",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.linear_model import Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "pediatric-lawrence",
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = load_iris()\n",
    "X = iris.data[:, (2, 3)] #longitud y ancho de los pétalos\n",
    "y = (iris.target == 0).astype(np.int) #Iris setosa?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "pressed-robinson",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Perceptron()"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "per_clf = Perceptron()\n",
    "per_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "operational-clearance",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = per_clf.predict([[2, 0.5]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "joined-forty",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "divided-madagascar",
   "metadata": {},
   "source": [
    "La clase `Perceptron` es equivalente a usar `SGDClassifier` con los siguientes hiperparámetros: `loss=\"perceptron\", learning_rate=\"constant\", eta0=1, penalty=None`.\n",
    "\n",
    "Los perceptrones son limitados. Son incapaz de resolver algunos problemas triviales (clasificación de OR exclusivo, XOR)\n",
    "\n",
    "![](figures_intro_rna/fig10-6.png)\n",
    "\n",
    "Se puede superar estas limitaciones usando varias capas de perceptrones para formar un perceptrón de multicapa (*multilayer perceptron*, MLP).\n",
    "\n",
    "El siguiente MLP puede resolver el problema de XOR:\n",
    "\n",
    "![](figures_intro_rna/fig10-6b.png)\n",
    "\n",
    "Todos los pesos de las conexiones son igual a $1$, aparte de las conexiones indicadas en rojo. Con entradas $(0,0)$ o $(1,1)$ la red da $0$, pero con entradas $(0,1)$ o $(1,0)$ la salida es $1$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ultimate-queue",
   "metadata": {},
   "source": [
    "### Perceptrón Multicapa y *backpropagation*\n",
    "\n",
    "Un MLP está compuesto de una capa de entrada (*passthrough*), una o más capas de TLUs (llamadas **capas ocultas**), y una capa final de TLUs llamada la capa de salida.\n",
    "\n",
    "![](figures_intro_rna/fig10-7.png)\n",
    "\n",
    "Cada capa aparte de la capa de salida incluye una neurona de *bias* y está conectada completamente a la próxima capa.\n",
    "\n",
    "Una red neuronal con varias capas ocultas se llama una **red neuronal profunda** (*deep neural network*)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alpine-significance",
   "metadata": {},
   "source": [
    "El algoritmo usado para el entrenamiento de una red neuronal profunda se llama ***backpropagation*** (Rumelhart et al. 1985)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alert-cycle",
   "metadata": {},
   "source": [
    "### Propagación hacia atrás (*backpropagation*)\n",
    "\n",
    "Ahora definimos el algoritmo de *backpropagation* para una red *feedforward*.\n",
    "\n",
    "Tenemos los siguientes ingredientes:\n",
    "\n",
    "* Datos: $\\boldsymbol{X} = \\left\\{ (\\boldsymbol{x}_1,\\boldsymbol{y}_1),\\ldots,(\\boldsymbol{x}_N,\\boldsymbol{y}_N) \\right\\}$\n",
    "\n",
    "\n",
    "* Red neuronal *feedforward*: parámetros $\\theta$, que son los pesos $w_{ij}^k$ (peso entre el nodo $j$ en la capa $l_k$ u el nodo $i$ en la capa $l_{k-1}$) y $b_i^k$ (el *bias* del nodo $i$ en la capa $l_k$). No hay conexiones entre neuronas en la misma capa y todas las capas son totalmente conectadas.\n",
    "\n",
    "\n",
    "* Función de error (costo, perdida) $E(\\boldsymbol{X},\\theta)$ que define el error entre la salida deseada $\\boldsymbol{y}_i$ y la salida calculada por la red $\\hat{\\boldsymbol{y}}_i$ en la entrada $\\boldsymbol{x}_i$ para un conjunto de datos $\\boldsymbol{X}$ y un valor específico de los parámetros $\\theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "appropriate-wayne",
   "metadata": {},
   "source": [
    "El entrenamiento con descenso por gradiente corresponde a la actualización de los parámetros de la red en la siguiente manera:\n",
    "\n",
    "$$\\theta^{t+1} = \\theta^t - \\alpha \\frac{\\partial E(\\boldsymbol{X},\\theta^t)}{\\partial \\theta}$$\n",
    "\n",
    "Aquí vemos inmediatamente la razón por usar una función de activación diferenciable... Eso fue uno de las inovaciones del algoritmo de *backpropagation*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "specific-bronze",
   "metadata": {},
   "source": [
    "Para simplificar la presentación del algoritmo, vamos a considerar una red con una sola salida $y$. Es decir, NO es un vector."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "animal-schedule",
   "metadata": {},
   "source": [
    "Una opción fácil para la función de costo es el error cuadrado promedio:\n",
    "\n",
    "$$E(\\boldsymbol{X},\\theta) = \\frac{1}{2M}\\sum_{m=1}^M (\\hat{y}_m - y_m)^2$$\n",
    "\n",
    "Hay otras funciones usadas, dependiendo de la red y la tarea (clasificación multiclase, etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acquired-taste",
   "metadata": {},
   "source": [
    "Podemos incluir el *bias* como otro peso en la siguiente forma:\n",
    "\n",
    "$$w_{0i}^k = b_i^k$$\n",
    "\n",
    "y incluimos una salida de la neurona \"$0$\" (igual a $1$) en la capa $k-1$:\n",
    "\n",
    "$$s_0^{k-1} = 1$$\n",
    "\n",
    "Así que la \"activación\" de la neurona $i$ en la capa $k$ está dada por\n",
    "\n",
    "$$a_i^k = b_i^k + \\sum_{j=1}^{r_{k-1}}w_{ji}^ks_j^{k-1} = \\sum_{j=0}^{r_{k-1}}w_{ji}^ks_s^{k-1}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "funded-belgium",
   "metadata": {},
   "source": [
    "La derivada de la función de costo con respecto a los pesos (incluyendo el *bias*) se puede escribir como\n",
    "\n",
    "$$\\frac{\\partial E(\\boldsymbol{X},\\theta)}{\\partial w_{ij}^k} = \\frac{1}{M} \\sum_{m=1}^M \\frac{\\partial}{\\partial w_{ij}^k} \\left( \\frac{1}{2}(\\hat{y}_m - y_m)^2 \\right) = \\frac{1}{M} \\sum_{m=1}^M \\frac{\\partial E_m}{\\partial w_{ij}^k}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "minute-sport",
   "metadata": {},
   "source": [
    "Entonces, podemos calcular la derivada de la función de costo sumando las derivadas de $E_m$, la función de costo para cada par de entrada-salida. Entonces ahora escribimos simplemente $E$ (en vez de $E_m$) y consideramos solamente un par de entrada-salida:\n",
    "\n",
    "$$E = \\frac{1}{2}(\\hat{y}-y)^2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "traditional-headquarters",
   "metadata": {},
   "source": [
    "Comenzamos con la aplicación de la regla de cadena:\n",
    "\n",
    "$$\\frac{\\partial E}{\\partial w_{ij}^k} = \\frac{\\partial E}{\\partial a^k_j} \\frac{\\partial a^k_j}{\\partial w_{ij}^k}$$\n",
    "\n",
    "El primer factor típicamente se denomina el error, y se denota:\n",
    "\n",
    "$$\\delta_j^k \\equiv \\frac{\\partial E}{\\partial a^k_j}$$\n",
    "\n",
    "Podemos calcular el otro factor usando la definición de la \"activación\" dada arriba:\n",
    "\n",
    "$$\\frac{\\partial a^k_j}{\\partial w_{ij}^k} = \\frac{\\partial}{\\partial w_{ij}^k} \\left( \\sum_{l=0}^{r_{k-1}}w_{lj}^ks_l^{k-1} \\right) = s_i^{k-1}$$\n",
    "\n",
    "Entonces, queda\n",
    "\n",
    "$$\\frac{\\partial E}{\\partial w_{ij}^k} = \\delta_j^ks_i^{k-1}$$\n",
    "\n",
    "Para evaluar $\\delta_j^k$ necesitamos elegir una función $E$ y una función de activación $g$. El proceso de *backpropagation* corresponde a la propagación hacia atrás del error."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "requested-union",
   "metadata": {},
   "source": [
    "#### La capa de salida\n",
    "\n",
    "En este ejemplo tenemos una sola neurona en la última capa, así que hay que calcular $\\delta_1^f$ donde $f$ es la capa final.\n",
    "\n",
    "$$E = \\frac{1}{2}(\\hat{y} - y)^2 = \\frac{1}{2}(g_o(a_1^f)-y)^2$$\n",
    "\n",
    "donde $g_o(x)$ es la función de activación de la capa de salida. Entonces tenemos\n",
    "\n",
    "$$\\delta_1^f = (g_o(a_1^f)-y)g'_o(a_1^f) = (\\hat{y}-y)g'_o(a_1^f)$$\n",
    "\n",
    "y la derivada de la función de error en la última capa de la red queda\n",
    "\n",
    "$$\\frac{\\partial E}{\\partial w_{i1}^f} = \\delta_1^f s_i^{f-1} = (\\hat{y}-y)g'_o(a_1^f)s_i^{f-1}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "magnetic-advertiser",
   "metadata": {},
   "source": [
    "#### Las capas ocultas\n",
    "\n",
    "Aplicamos la regla de cadena de nuevo para calcular\n",
    "\n",
    "$$\\delta_j^k = \\frac{\\partial E}{\\partial a_j^k} = \\sum_{l=1}^{r^{k+1}} \\frac{\\partial E}{\\partial a_{l}^{k+1}} \\frac{\\partial a_{l}^{k+1}}{\\partial a_{j}^k}$$\n",
    "\n",
    "donde $l = 1,\\ldots,r^{k+1}$, el número de neuronas en la **próxima** capa. Podemos escribir la ecuación arriba como\n",
    "\n",
    "$$\\delta_j^k = \\sum_{l=1}^{r^{k+1}} \\delta_l^{k+1} \\frac{\\partial a_{l}^{k+1}}{\\partial a_{j}^k}$$\n",
    "\n",
    "La definición de $a_{l}^{k+1}$ es\n",
    "\n",
    "$$a_l^{k+1} = \\sum_{j=1}^{r^{k}} w_{jl}^{k+1} g(a_{j}^k)$$\n",
    "\n",
    "donde $g(x)$ es la función de activación para las capas ocultas. Entonces, tenemos\n",
    "\n",
    "$$\\frac{\\partial a_{l}^{k+1}}{\\partial a_{j}^k} = w_{jl}^{k+1} g'(a_{j}^k)$$\n",
    "\n",
    "y el error en las capas ocultas queda\n",
    "\n",
    "$$\\delta_j^k = \\sum_{l=1}^{r^{k+1}}\\delta_l^{k+1}w_{jl}^{k+1} g'(a_{j}^k) = g'(a_{j}^k)\\sum_{l=1}^{r^{k+1}}\\delta_l^{k+1}w_{jl}^{k+1}$$\n",
    "\n",
    "Aquí vemos de donde viene el nombre *backpropagation*. El error en la capa $k$ depende de los errores en la capa $k+1$. Es decir, los errores se propagan hacia atrás..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "color-platform",
   "metadata": {},
   "source": [
    "#### El algoritmo completo\n",
    "\n",
    "Ahora tenemos todos los ingredientes necesarios para implementar el algoritmo de *backpropagation* en una red *feedforward* con una sola neurona en la capa de salida.\n",
    "\n",
    "1. Elegir **aleatoriamente** los pesos $w_{ij}^k$ y la taza de aprendizaje $\\alpha$.\n",
    "\n",
    "\n",
    "2. Calcular la pase hacia adelante (*forward pass*) para cada par de entrada-salida $(\\boldsymbol{x}_m,y_m)$ y guardar los resultados $\\hat{y}_m$, $a_j^k$ y $s_j^k$ para cada neurona $j$ en capa $k$, procediendo de la capa $0$ (entrada) hasta la capa $f$ (salida).\n",
    "\n",
    "\n",
    "3. Calcular la pase hacia atrás (*backward pass*) para cada par de entrada-salida $(\\boldsymbol{x}_m,y_m)$ y guardar los resultados $\\partial E_m/\\partial w_{ij}^k$ para cada peso $w_{ij}^k$ conectando neurona $i$ en capa $k-1$ con neurona $j$ en capa $k$, procediendo de la capa $f$ (salida) hasta la capa $1$ (entrada).\n",
    "\n",
    "    1. Evaluar el error para la capa final $\\delta_1^f$.\n",
    "    2. Propagar hacia atrás los errores para las capas ocultas $\\delta_j^k$, comenzando en $k=f-1$.\n",
    "    3. Evaluar las derivadas parciales del error $E_m$ con respecto a $w_{ij}^k$\n",
    "\n",
    "\n",
    "4. Promediar los gradientes $\\partial E_m/\\partial w_{ij}^k$ para obtener la derivada $\\partial E(\\boldsymbol{X},\\theta)/\\partial w_{ij}^k$ para todo el conjunto de datos.\n",
    "\n",
    "\n",
    "5. Actualizar los pesos usando el negativo de los gradientes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "loving-scenario",
   "metadata": {},
   "source": [
    "## [Keras](https://keras.io/)\n",
    "\n",
    "![](figures_intro_rna/fig10-10.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "supreme-ukraine",
   "metadata": {},
   "source": [
    "## Implementando un MLP con Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broke-article",
   "metadata": {},
   "source": [
    "Vamos a implementar un modelo para clasificar imagenes, con la función *softmax* como función de activación en la capa de salida.\n",
    "\n",
    "Usaremos *Fashion MNIST*, imagenes de $28 \\times 28$ pixeles, con $10$ clases. Cada imagen representa ropa o zapatos, así que las imagenes son más diversas que en el caso de MNIST."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "proof-lithuania",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "subsequent-seventh",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.3.0'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "million-failure",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.4.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "broke-march",
   "metadata": {},
   "outputs": [],
   "source": [
    "fashion_mnist = keras.datasets.fashion_mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "massive-keyboard",
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "featured-quarter",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_full.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "powerful-activation",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('uint8')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_full.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rolled-import",
   "metadata": {},
   "source": [
    "Vamos a escalar los *features* para ayudar el método de descenso por gradiente:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "loaded-fabric",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_valid, X_train = X_train_full[:5000] / 255.0, X_train_full[5000:] / 255.0\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lesbian-range",
   "metadata": {},
   "source": [
    "Creamos una lista de nombres de las clases para estos datos:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "banned-wright",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_names = [\"T-shirt/top\", \"Trouser\", \"Pullover\", \"Dress\", \"Coat\",\n",
    "               \"Sandal\", \"Shirt\", \"Sneaker\", \"Bag\", \"Ankle boot\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "rough-diagram",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Coat'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class_names[y_train[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "signal-knock",
   "metadata": {},
   "source": [
    "![](figures_intro_rna/fig10-11.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "norman-headset",
   "metadata": {},
   "source": [
    "Ahora, implementamos el modelo usando el API \"secuencial\" de Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "assured-strength",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=[28, 28]))\n",
    "model.add(keras.layers.Dense(300, activation=\"relu\"))\n",
    "model.add(keras.layers.Dense(100, activation=\"relu\"))\n",
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "demanding-directive",
   "metadata": {},
   "source": [
    "* `Sequential` significa construir el modelo capa por capa, conectadas secuencialmente. Se llama el API secuencial.\n",
    "\n",
    "* La primera capa es `Flatten` que convierte cada imagene de entrada en un array unidimensional: aplica `X.reshape(-1, 1)`. Hay que especificar la forma de las instancias con `input_shape`. Un alternativo es usar una capa antes de `keras.layers.InputLayer`.\n",
    "\n",
    "* La segunda capa (oculta) es `Dense` con $300$ neuronas. Ocupa la función de activación \"ReLU\". Cada capa densa maneja su propia matriz de pesos, con todas las conexiones entre las neuronas y sus entradas. También maneja un vector de *bias* (uno por neurona).\n",
    "\n",
    "* La tercera capa (oculta) es `Dense` con $100$ neuronas, también con funciones de activación \"ReLU\".\n",
    "\n",
    "* La última capa es `Dense` con $10$ neuronas (una por clase), con la función de activación *softmax*.\n",
    "\n",
    "![](figures_intro_rna/relu.png)\n",
    "\n",
    "La lista completa de funciones de activación está [aquí](https://keras.io/activations/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coral-occupation",
   "metadata": {},
   "source": [
    "Otro método de armar el modelo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "allied-tuesday",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(300, activation=\"relu\"),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fluid-level",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten_1 (Flatten)          (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 266,610\n",
      "Trainable params: 266,610\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "phantom-channel",
   "metadata": {},
   "source": [
    "El modelo tiene **muchos** parámetros, así que hay mucho riesgo de *overfitting*, especialmente si tenemos muy pocos datos..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "violent-prediction",
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import plot_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "native-stability",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOkAAAHBCAIAAAAD3QtbAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3daVAU19oH8KdnUQGB3DisghrKBU2QkIRFRAtKQESQiKyCBBQhmliK0RBzLSWlEamLiblVWiquBFRAS9RCVCjIDQgErnFcEN+QEI0sylKAMAjMDP1+6JvJhGUccIbuM/X8Pk2fXuY59N/mdDtzoGiaBoQIxGO7AITGCLOLSIXZRaTC7CJSCZQXysvLv/nmG7ZKQUi1BQsWbN26VbH4t+vu06dPL1y4MO4lIfRqFRUV5eXlyi2CoRvl5OSMVz0IqSs4OHhQC453Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpMLuIVJhdRCrMLiIVZheRCrOLSDXG7NI0/e233+7fv3/WrFnh4eGlpaVJSUk3b97UbHGvr6OjY+fOnTt27FBz+9zcXGtr65qaGq1WpY7CwsLY2FiKoiiKWrp0aWZmprbf8cKFCy4uLsw7bt68WSwWa/sdXxetJCsra1DLSJKSkuLj42maLikpEYlE4eHhAHD8+PFX7tjY2KhiUbOuXLkSEhICAJ9++qmau9y8efO9996rq6vTXlWj6rKJiQkA1NfXj089zIe73333Xe293ZgFBQUFBQUpt4zxunv48OEZM2YAgJubW0tLy/bt29XZq729PTIycqRFjfP3909LSxvVLl5eXrdv337rrbe0VNJou2xkZAQAxsbG41PPG2+8odW306yxZLe3t7e5uZmiKEXLhAkTXrlXf3//6tWr6+rqhl3UkokTJ2r1+KMyhi4zP2TlH7VW69Hq22ncqLN75syZ9evXA0BOTs769etTUlKGbvP8+fO4uLg9e/asX79+5cqVbW1tzPbV1dWtra3r169PTU0dtAgANE0fOXJkw4YNzs7O3t7etbW1ACAWi7dv325jY9Pe3h4dHS0SiZycnLSU+Pb29hMnTnh5eeXm5qp+64cPH/7zn/+cN29eY2Pjhx9++Oabbzo5OVVUVADAuXPnjIyMrK2tAeDFixcHDx6cNGnSggULhv4EAODWrVvW1tb5+fnqlDcO9bzSsGf28uXLhoaGFEUdPHiwv78fAMrLyy0sLPbt2wcjnNZnz55988038+fPb2pq8vb2nj59OnOo0VEeQKg53m1tbQWAvXv3KloePHgASuNdd3f30NBQ5rW9vX1kZCTz2s/Pb8aMGYq9Bi0mJyefPn2apmmZTObi4mJubi6RSJqamjw9PQEgPj6+urq6oKDAyMgoLCxMzUFSb28vqD3effjwYUJCAgBcuHCBpmkVb/3FF1+88cYbfD4/ISGhuLj44sWLIpFIX1+fGTt6e3tbWVkpDvvBBx+4uLgM2+W8vDw9Pb3MzMyRSpo5cyYAdHd3j089jx49AgB3d/eR6hnpzH7xxRcAUFVVxSz29fU5Ozszr4c9rfn5+ba2tnw+Pykp6cSJE05OTg0NDSpODa3B8a5qFEXZ29szr99555179+69cpfGxsaDBw+uWbMGAPh8flBQ0LNnz65evWpubu7o6AgAX3/99bx58zw9PRctWnT79m1tlD137tyAgADFooq3Tk5O9vX15fF4KSkp7u7ugYGBR44c6enpOXLkCADo6+srH1YgGOYLrQxfX9+urq7Vq1erU9441PNKI53ZTz75RCAQHD16lFksKCjw8/ODkU+rj4/PwoUL5XJ5RETE2rVrf/rpJ0tLy9EWM/ZuqFBUVAQAEokkIyOjqqpqYGDglbuUlZVJpdL4+HhFS2xsrJ6eHgDw+XxQ+okbGhp2dXVpo2wYcl5VvLW+vj6fzxcKhcxiQEDAxIkT79+/P9p3ZN5iVBtrtR7VRjqzVlZWwcHBGRkZycnJIpEoOzt79+7doPK0CoVCgUDA/GIZG61kVy6Xp6Sk1NbWbt26tbS0lBl4qVZTU2NgYDDaxwLcIRAILC0tZTIZ24X8j5bqUXFmExISzp07d+zYsW3btrW2ttrY2ICWT6vmszswMODr62tqavr999+rv5e+vn59fX19fb2VlZWisbW1VSQSabxCLenp6bG1tWW7ir9otp5ff/3V0tJy5cqVI51ZR0fHhQsXHjp0yNbW1t/fn2nU6mkdy3iXVjllb2Vl5c2bN93d3ZlFqVSq2J7H40ml0r/eW2nRzs6OpunExETF2ubm5lOnTo2hPFY0NTW1tLQEBQUBgEAg6O7ulsvlzKru7m7F79ZBPwEAUD2gYn50qn/gGqxHxRt99tlnd+7cGenMMj7//PPGxsbPPvtMMQ+IVk/rWLIrkUgAoKenR9Hy4sULAGB+QzFPB8+cOXP//v2TJ09WV1c/f/783r17z58/t7S0fPbsmVgs/uGHH3p6epQXFy5c6OjoePbs2VWrVn3//fe7d++OiIiIiYkBAOaHq/j19/LlS+W3VqdU5mmDOl6+fAkAfX19zKLqt+7r67t79y7zeu/evR999JGTkxMA2NnZdXR0JCcn//LLL3v37u3r6/u///u/O3fuAMCgn0BhYeE//vEPFfNoMT/Yzs7O8amHeaOOjg7lGjo7Oz/66CNmeAojnFlmS39//3feecfe3n7KlClMi5eX10inVSaTyeXy1xrVKD90UOcZ2e3bt5nbxrfeeiszM7Ojo+POnTsrV64EgEWLFhUXF9M0/fHHHxsaGrq4uBQWFl67dk0kEgUFBXV3d9+9e9fa2nr27Nk5OTk0TQ9abGtri4iIMDU1NTExiYqKYh6aFBYWMsP5jRs3Njc3p6enM//rk5SUJJPJVJdaUlKybt06ADAzMzt37lxTU5Pq7cvLy319fQFg8eLFt27dUv3WsbGxEyZMSEhICA4OXrdu3Z49ewYGBpjjdHZ2+vv7T5482cXFpaqqKjo6OjIy8sqVK0O7XFRUZGFhkZubO7SY4uLijRs3MufIx8fn/Pnz2q4nNzfXzc2NeUcXF5elS5d6eXnZ2toy//F09OhRFWdWUfbmzZuZrikMe1ozMjIsLCwAYPPmzQ8ePFB9XhhDn5GN8fMMKDY2dtKkSWxX8ReO1LNkyZKXL19q48hDs6uV5wzjg/mcyrBOnjypuF3QyF5IHcXFxe+///6kSZPG5+0Izm5LS8u47TVUd3c3c7PCkf/9Z7Ge0tLS+Pj4t99++8GDBz/++OO4vS9+9nws0tPTCwoK5HL5Z599VllZyXY5LNczZcqU3t7en3/++ejRo+P5TJOilR5zZGdnM/9bPW5vj5CamOduypND43UXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGphvn87tA/mI0Q6yoqKlxcXJRb/nbdtba2Zr5ZijTiypUrjY2NbFehI1xcXJhp1BQo/LSu9lAUlZWVxcwBjDQOx7uIVJhdRCrMLiIVZheRCrOLSIXZRaTC7CJSYXYRqTC7iFSYXUQqzC4iFWYXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpMLuIVJhdRCrMLiIVZheRCrOLSIXZRaTC7CJSYXYRqTC7iFQ477kmrVmzRiwWKxYfP35sYmJiYGDALAqFwqtXr06dOpWl6nTNMH8rBY3ZnDlzMjIylFu6u7sVr21tbTG4GoRjBk0KDw+nKGrYVUKhMDo6enzL0XE4ZtCw999/XywWDwwMDGqnKKqurm7GjBlsFKWb8LqrYVFRUTze4J8qRVFOTk4YXM3C7GpYaGjo0Isuj8eLiopipR4dhtnVMHNz80WLFvH5/EHtq1atYqUeHYbZ1bw1a9YoL/J4PA8PDzMzM7bq0VWYXc0LDg4eNOQdlGakEZhdzTMyMvLx8REI/vfsnM/nBwQEsFuSTsLsakVkZKRcLgcAgUCwYsUKY2NjtivSQZhdrVixYoWenh4AyOXyiIgItsvRTZhdrZg0aVJgYCAA6OvrL1u2jO1ydBPLn2eor68vKytjtwYtsba2BgBHR8crV66wXYtWWFtbL1iwgM0KaFZlZWWx2Xn0GoKCgtgNDyc+R0br6GcqkpKSdu7cqXjgoEuCg4PZLgHHu9qkq8HlCMyuFmFwtQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpiPmwCE3TBw8e7OvrO3HixAcffPDJJ58UFha6urp6e3uzXdrfdHR0pKamyuXy5ORkdbbPycn517/+VVVVNWHChEWLFgmFQpqmX758+ejRo+bm5ocPH3Z2dl6/fp2DPWUfux8fZj57rs6WSUlJ8fHxNE2XlJSIRKLw8HAAOH78+Ct3bGxsVLGoWVeuXAkJCQGATz/9VP29bt26BQALFy5UbpRKpYsXLz5x4kRMTAwHexoUFMT6Z8+JGTMcPnyYmc/Lzc2tpaVl+/bt6uzV3t4eGRk50qLG+fv7p6WljXavN998EwCEQqFyo0Ag+Pjjj11dXTdt2qTOQca5p1xAxpiht7e3ublZeXrQCRMmvHKv/v7+1atX19XVDbuoJRMnThztLiNNe8r8bqmurn7lEVjpKesIuO6eOXNm/fr1AJCTk7N+/fqUlJSh2zx//jwuLm7Pnj3r169fuXJlW1sbs311dXVra+v69etTU1MHLQIATdNHjhzZsGGDs7Ozt7d3bW0tAIjF4u3bt9vY2LS3t0dHR4tEIicnp9fMwa1bt6ytrfPz89Xf5auvvhq2neM9HVfsDlnUHO+2trYCwN69exUtDx48AKVRoLu7e2hoKPPa3t4+MjKSee3n5zdjxgzFXoMWk5OTT58+TdO0TCZzcXExNzeXSCRNTU2enp4AEB8fX11dXVBQYGRkFBYWpmaPent7Ych4Ny8vT09PLzMzc9hdHj16BADu7u7Molwuf/jwoa2tLZd7yoXxLhljhleiKMre3p55/c4779y7d++VuzQ2Nh48eLCxsREA+Hx+UFDQtm3brl69Ghoa6ujoWFhY+PXXX0+ZMmXevHmLFi26ffv265Tn6+vb1dU1dHJIZT///DPzlXGZTPbkyROpVDrsZhzv6XjSkewWFRUBgEQiycjIqKqqGjoD7lBlZWVSqTQ+Pl7REhsby0xmw4RM8W0zQ0PDrq6u16xQdXAB4L333isuLmZeS6VSLy+vYTfjfk/HjY5kVy6Xp6Sk1NbWbt26tbS0tKKi4pW71NTUGBgYjOGxwDgQCoWff/75sKt0rKevQxeyOzAw4Ovra2pq+v3336u/l76+fn19fX19vZWVlaKxtbVVJBJpocZR8/X1Hdqokz0dMwKeM8CrJh+prKy8efOmu7s7syiVShXb83g85YGj8qKdnR1N04mJiYq1zc3Np06d0mzlCip+uTPVqu4jg4iejhsyrrsSiQQAenp6FC0vXrwAAJlMBn8+Hz1z5oyTk1NVVVV1dfXz58/v3btnZmZmaWmZl5cnFos7OjqcnJyUFxcuXOjo6Hj27Nne3t4PP/zw119/LSsrO3fuHAAwZ505OAC8fPlS+a3VKZV52qBQWFi4atWqEydOBAUFDd2lo6MD/v6X2JRxtqfsY/EZB63eM7Lbt28z84a/9dZbmZmZHR0dd+7cWblyJQAsWrSouLiYpumPP/7Y0NDQxcWlsLDw2rVrIpEoKCiou7v77t271tbWs2fPzsnJoWl60GJbW1tERISpqamJiUlUVFRDQwNN04WFhTNnzgSAjRs3Njc3p6enM7PnJiUlyWQy1aWWlJSsW7cOAMzMzM6dO9fU1MS0FxUVWVhY5ObmDt0lNzd38eLFAEBR1I4dO6qrq5XXcranXHhGxvLfV8vOzmaeVrJYAxoDZj6ynJwcFmsgY8zAESYmJiOtOnnypL+//3gWgzC7o9DS0sJ2CegvZDxnQGgozC4iFWYXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpOPH53ezsbLZLQKMz6FvHrOBEdkNDQ9kuAY3asN8bHU8sf19Nt1EUlZWVxczIizQOx7uIVJhdRCrMLiIVZheRCrOLSIXZRaTC7CJSYXYRqTC7iFSYXUQqzC4iFWYXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpMLuIVJhdRCrMLiIVZheRCrOLSIXZRaTC7CJSYXYRqTC7iFScmLNfZxw7dqy9vV255fLly7///rtiMTo62szMbNzr0k04Z78mxcfHHzt2bOLEicwiTdMURTGvZTKZsbHxs2fPhEIhewXqFBwzaFJ4eDgA9P2pv79f8ZrH44WHh2NwNQivu5o0MDBgYWHR3Nw87NrS0tKFCxeOc0k6DK+7msTj8SIjIydMmDB0lYWFhaur6/iXpMMwuxoWHh7e398/qFEoFEZFRSnGvkgjcMygeTY2NsrPFhhisdje3p6VenQVXnc1LyoqatA9mY2NDQZX4zC7mhcZGSmVShWLQqEwJiaGxXp0FY4ZtGL+/PkPHjxQ/Gx/+eWXWbNmsVuS7sHrrlZERUXx+XwAoCjKwcEBg6sNmF2tWL16tVwuBwA+n//RRx+xXY5uwuxqhaWlpaurK0VRAwMDwcHBbJejmzC72rJmzRqaphcvXmxpacl2LbqJc/dq+ACfs7KyskJCQtiu4i9c/Azkli1bFixYwHYVGnDgwIH4+PjJkyezXYgGhIaGsl3CYFzM7oIFCzj173vMXF1drays2K5CMziYXRzvapHOBJebMLuIVJhdRCrMLiIVZheRCrOLSIXZRaTC7CJSYXYRqTC7iFSYXUQqzC4iFWYXkUpHstvd3c12CWi8EZ/dtLQ0Ly+vuXPnsl0IAEBOTs577703efLk+fPnX758WZ1drl275u/vT1EURVGurq5ubm4ODg4uLi6JiYm//fabtgsmG80xAJCVlaX+9jKZzM3NzdzcXHslqen06dNxcXHFxcVFRUUODg5CofCXX35RZ8f6+noAmD59uqKlsrLSx8eHz+d/+eWXcrlcWxWPxmjPyzgg/rrL5/O58DFZqVTa2dl59OhRd3d3Dw+P48ePS6XSn376SZ19DQwMAEBPT0/R4ujomJeXFxYWtm/fvpSUFG0VTTjis8sRPB5v48aNisUpU6YAgKOjozr7DvsVPR6Pd+jQIVNT07179/7xxx+aqlOXkJrdy5cvx8XFJSYmbtq0qampSdFO0/SRI0c2bNjg7Ozs7e1dW1sLAGKxePv27TY2Nu3t7dHR0SKRyMnJqa6ujtlFLBbHxMSkpKQEBAR4eXmpOI4KfD5fIPjrC1SZmZlfffXVnDlzmMVbt25ZW1vn5+ePqo/GxsYhISE9PT3Z2dksdo272B2yDAVqjKsyMzOdnZ1fvnxJ03RLS4uJiYlivJucnHz69GmapmUymYuLi7m5uUQiaWpq8vT0BID4+Pjq6uqCggIjI6OwsDBmlzlz5pSWltI03dfX5+fnp+I46tTf1dX11VdfiUSi9PR0RWNeXp6enl5mZuawu3R0dACAra3t0FUZGRkAEBMTw3rX1Dkv44y87EokEgsLi7NnzypaAgMDmew2NDSYmZkpbm5SU1MB4Pz58zRN79ixAwBaW1uZVcuXL581axZN0/39/RRFfffdd0z79evXVR9Hte7u7qSkpKCgIB6PBwDHjx9XrJLJZCPtpSK7N27cAIAlS5aw3jUOZpeL3xNWraSkpKmpyc7OTtGimGe8rKxMKpXGx8crVsXGxjL3QMzsYIpf64aGhl1dXQAgFAq9vb23bNny4MGD/fv3L126VPVxVDMwMNi9ezcAFBYWhoSE7Nu3b926dcwqpoDR6uzsBIDZs2ez3jUOIi+7jx49AqW8KqupqTEwMEhLSxvVAc+fPx8eHp6Wlnbp0qXs7GwPD4+xHUeZp6dnQkLC7t27pVLp6/x9FKaz9vb23Okad5B3r8ak9smTJ0NX6evr19fXM49LFVpbW1UfUF9fPz8/PyMjQyAQ+Pj41NTUjO04g7z99ttWVlavE1yapnNycoyMjPz8/DjVNY4gL7vz588HgKysLEXLwMAAM+minZ0dTdOJiYmKVc3NzadOnVJxtL6+vmPHjgFARERERUUFTdPFxcVjOM5Qjx49WrFihXKRI21JjzCt1oEDB+7fv5+amjp16lROdY0r2BtqDw/UuCfw8PDg8/mHDx+WSCSVlZXMZHVnz57t7u5mHqkGBgamp6fv2rXL09OzpaWF/vNsKW5oAgICjIyMaJru7e11cHBgbqT6+/tFIlF5efnAwMBIxxlJe3v76tWrMzIyBgYGaJqura319vbu7u5m1jK3/zk5OcPu+/TpUwCYNm2aouXx48ebNm2iKGrz5s1Mi4qStN01hjrnZZwRmd3Ozs61a9eamZlNmzYtKSkpLi4uJiamsLBQLpe3tbVFRESYmpqamJhERUU1NDTQNF1YWDhz5kwA2LhxY3Nzc3p6urGxMQAkJSVJJBJHR8elS5fu378/Li5O8WRg2OOo0NXV5efnN2XKlMWLF+/ZsycjI0MqlSrWFhUVWVhY5ObmDt3xxo0b/v7+zHXEzc1tyZIlvr6+y5YtS0hIEIvFyluy1TUGB7PLxXkguTbfIAJOnhfynjOwyMTEZKRVJ0+eVFw+0fjA7I5CS0sL2yWgv5D3nAEhBmYXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpMLuIVFz8/G5oaCgH/2o44hrOZVf5C8CkCw0N3bJly4IFC9guRDNcXV3ZLuFvOPd9NV3Cwe946RIc7yJSYXYRqTC7iFSYXUQqzC4iFWYXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpMLuIVJhdRCrMLiIVZheRCrOLSIXZRaTC7CJSYXYRqTC7iFSYXUQqzC4iFWYXkQqzi0iF2UWk4ty850R78uSJXC5Xbnn+/HldXZ1i0cLCQk9Pb9zr0k0477kmLVu27Pr16yOtFQgEz549mzJlyniWpMNwzKBJYWFhFEUNu4rH43l5eWFwNQizq0mBgYFCoXCktWvWrBnPYnQeZleTDA0N/fz8ho2vUCj09/cf/5J0GGZXwyIiImQy2aBGgUCwcuXKyZMns1KSrsLsatjy5csNDAwGNcrl8oiICFbq0WGYXQ2bOHFiUFDQhAkTlBsnT57s7e3NVkm6CrOreatXr+7v71csCoXCsLCwQWlGrw+f72rewMCAmZlZa2uroqW4uNjd3Z29inQTXnc1j8fjrV69WnGhNTExWbRoEbsl6STMrlaEh4czw4YJEyZERUXx+Xy2K9JBOGbQCpqmp0+f/vTpUwCoqqr64IMP2K5IB+F1VysoioqKigKA6dOnY3C1hHOfIwsODma7BM148eIFABgYGOhMj7Zu3bpgwQK2q/gL5667Fy5cqK+vZ7sKDTAyMjI2NraysmK7EM24cOECMwTiDs5ddwEgISEhJCSE7So04MaNG0uXLmW7Cs0Y6fNxLOLcdVeX6ExwuQmzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpdCS73d3dbJeAxhvx2U1LS/Py8po7dy7bhfxNUVGRpaWlOlteu3bN39+foiiKolxdXd3c3BwcHFxcXBITE3/77Tdt10k2mmMAICsrS/3tZTKZm5ububm59koara6urhkzZqhfEvNZ++nTpytaKisrfXx8+Hz+l19+KZfLtVLlKI32vIwD4q+7fD6fa99N2LVr17x589TfnpkDSnlOaUdHx7y8vLCwsH379qWkpGi+RJ1AfHa55ocffjAzMxtVdof9SgKPxzt06JCpqenevXv/+OMPzRWoO0jN7uXLl+Pi4hITEzdt2tTU1KRop2n6yJEjGzZscHZ29vb2rq2tBQCxWLx9+3YbG5v29vbo6GiRSOTk5KSYSl8sFsfExKSkpAQEBHh5eak4zitJJJLDhw9v27ZtUPutW7esra3z8/NH1UdjY+OQkJCenp7s7GzWu8ZF7A5ZhgI1xlWZmZnOzs4vX76kabqlpcXExEQxuExOTj59+jRN0zKZzMXFxdzcXCKRNDU1eXp6AkB8fHx1dXVBQYGRkVFYWBizy5w5c0pLS2ma7uvr8/PzU3GcVxa/ZcuWe/fu0TS9bds25fFuXl6enp5eZmbmsHt1dHQAgK2t7dBVGRkZABATE8N619Q5L+OMvOxKJBILC4uzZ88qWgIDA5mgNDQ0mJmZKW5uUlNTAeD8+fM0Te/YsQMAWltbmVXLly+fNWsWTdP9/f0URX333XdM+/Xr11UfR4Uffvhhz549zOtB2aVpWiaTjbSjiuzeuHEDAJYsWcJu12hOZpeL3xNWraSkpKmpyc7OTtGimPmrrKxMKpXGx8crVsXGxjL3QMysSgLB//praGjY1dUFAEKh0Nvbe8uWLQ8ePNi/fz/z7UgVxxmJRCL597//ff78+ZE2GNu0Tp2dnQAwe/ZsFrvGWeRl99GjR6CUV2U1NTUGBgZpaWmjOuD58+fDw8PT0tIuXbqUnZ3t4eExhuPs3LnTz8/v4cOHzGJzc7NUKr17966ent7s2bNHVY8yprP29vYsdo2zyLtXY1L75MmToav09fXr6+sHTU2iPJfosPT19fPz8zMyMgQCgY+PT01NzRiOU1FRsXbt2nf/lJ6e3tbW9u6774aGhqrbsSFoms7JyTEyMvLz82Oxa5xFXnbnz58PAFlZWYqWgYEB5i/y2dnZ0TSdmJioWNXc3Hzq1CkVR+vr6zt27BgAREREVFRU0DRdXFw8huOUl5crD8W++OILZrx7584dRZEj7UuPMJ3hgQMH7t+/n5qaOnXqVBa7xl3jPL5+JVDjnsDDw4PP5x8+fFgikVRWVjL/+3r27Nnu7m5HR0cACAwMTE9P37Vrl6enZ0tLC/3n2VLc0AQEBBgZGdE03dvb6+DgwNxI9Uquef0AAAlYSURBVPf3i0Si8vLygYGBkY6jJkV2Gcztf05OzrAbM3MlTZs2TdHy+PHjTZs2URS1efNmpkVFSePTNXXOyzgjMrudnZ1r1641MzObNm1aUlJSXFxcTExMYWGhXC5va2uLiIgwNTU1MTGJiopqaGigabqwsHDmzJkAsHHjxubm5vT0dGNjYwBISkqSSCSOjo5Lly7dv39/XFzc8ePHmbcY9jjqG5TdoqIiCwuL3NzcoVveuHFD8ber3NzclixZ4uvru2zZsoSEBLFYrLwlu13jYHY5N/8uRVFZWVm6MR+ZLuHgeSHvOQOLTExMRlp18uRJ/NN/4wyzOwotLS1sl4D+Qt5zBoQYmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpMLuIVJhdRCrMLiIVZheRCrOLSIXZRaTC7CJScfHzu99++21OTg7bVSCu49x1NygoiGvzOo7ZlStXGhsb2a5CM4KCgqytrdmu4m849301XcLB73jpEs5ddxFSE2YXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJhdhGpMLuIVJhdRCrMLiIVZheRCrOLSIXZRaTC7CJSYXYRqTC7iFSYXUQqzC4iFWYXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhXOe65Ja9asEYvFisXHjx+bmJgYGBgwi0Kh8OrVq1OnTmWpOl3Dxb+VQq45c+ZkZGQot3R3dyte29raYnA1CMcMmhQeHk5R1LCrhEJhdHT0+Jaj43DMoGHvv/++WCweGBgY1E5RVF1d3YwZM9goSjfhdVfDoqKieLzBP1WKopycnDC4moXZ1bDQ0NChF10ejxcVFcVKPToMs6th5ubmixYt4vP5g9pXrVrFSj06DLOreWvWrFFe5PF4Hh4eZmZmbNWjqzC7mhccHDxoyDsozUgjMLuaZ2Rk5OPjIxD879k5n88PCAhgtySdhNnVisjISLlcDgACgWDFihXGxsZsV6SDMLtasWLFCj09PQCQy+URERFsl6ObMLtaMWnSpMDAQADQ19dftmwZ2+XoJs59niE7O5vtEjTD2toaABwdHa9cucJ2LZrh6upqZWXFdhV/4dz/CY/0eQDEuqysrJCQELar+AsXxwxZWVm0Tti9e7dUKmW7Cs1gOxTD4GJ2dcbOnTsVT8qQxmF2tQiDq1WYXUQqzC4iFWYXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5FKR7KrPO0Xd/z+++8SiYTtKnQW8dlNS0vz8vKaO3cu24UAAHR1db3xxhvUnwIDAxWTQI7k2rVr/v7+zPaurq5ubm4ODg4uLi6JiYm//fbb+JRNKOI/6LR27dr09HSZTMZ2IQAAJ06cWLVqlY2NDbPo7e39yl18fX3t7e2trKymT59eVlbGNFZVVe3atWvOnDmJiYl79uwZOkkUAh3ILp/Pt7Ky+vXXX9kuBORy+eXLlwsKCkb70Ufm2sx8N5Ph6OiYl5cXFRW1b9++yZMn79ixQ8O16gT8B60xFy9eFIvFYWFhx44de/Hihfo7Dvs1Jx6Pd+jQIVNT07179/7xxx+aK1N3kJrdy5cvx8XFJSYmbtq0qampSdFO0/SRI0c2bNjg7Ozs7e1dW1sLAGKxePv27TY2Nu3t7dHR0SKRyMnJqa6ujtlFLBbHxMSkpKQEBAR4eXmpOI5qxcXFPT09Fy9ejI+Pnzdv3s2bNxWrbt26ZW1tnZ+fP6o+Ghsbh4SE9PT0MN8/ZbFrHMXut6CGAjW+r5aZmens7Pzy5UuapltaWkxMTMzNzZlVycnJp0+fpmlaJpO5uLiYm5tLJJKmpiZPT08AiI+Pr66uLigoMDIyCgsLY3aZM2dOaWkpTdN9fX1+fn4qjvPK4qVS6X//+9/o6Ggejzdp0qSHDx8y7Xl5eXp6epmZmcPu1dHRAQC2trZDVzGzqMfExLDeNXXOyzgjL7sSicTCwuLs2bOKlsDAQCa7DQ0NZmZmcrmcaU9NTQWA8+fP0zTNDBlbW1uZVcuXL581axZN0/39/RRFfffdd0z79evXVR9HTRcvXqQoauXKlYoWmUw20sYqsnvjxg0AWLJkCetd42B2ybtXKykpaWpqsrOzU7RMmDCBeVFWViaVSuPj4xWrYmNjmXsgZlJRxV2UoaFhV1cXAAiFQm9v7y1btjx48GD//v1Lly5VfRw1BQYGBgcHV1VVKVqGzmqqjs7OTgCYPXs2d7rGHeRl99GjR6CUV2U1NTUGBgZpaWmjOuD58+fDw8PT0tIuXbqUnZ3t4eExtuMMsnjx4pKSktc5AvzZWXt7e051jSPIu1djUvvkyZOhq/T19evr6+vr65UbW1tbVR9QX18/Pz8/IyNDIBD4+PjU1NSM7ThD2drajnYXZTRN5+TkGBkZ+fn5ca1rXEBedufPnw8AWVlZipaBgQFm0kU7OzuaphMTExWrmpubT506peJofX19x44dA4CIiIiKigqapouLi8dwnKH+85//xMTEKBc50pb0CDN3HDhw4P79+6mpqVOnTuVU17iCvaH28ECNewIPDw8+n3/48GGJRFJZWWlpaQkAZ8+e7e7udnR0BIDAwMD09PRdu3Z5enq2tLTQf54txQ1NQECAkZERTdO9vb0ODg7MjVR/f79IJCovLx8YGBjpOCP58ccf7ezsvv32W+ZQly5dioqKUqxlbv9zcnKG3ffp06cAMG3aNEXL48ePN23aRFHU5s2bmRYVJWm7awx1zss4IzK7nZ2da9euNTMzmzZtWlJSUlxcXExMTGFhoVwub2tri4iIMDU1NTExiYqKamhooGm6sLBw5syZALBx48bm5ub09HRmQtykpCSJROLo6Lh06dL9+/fHxcUdP36ceYthj6PC48ePPT0933zzTQ8Pj3/+85+XLl1SXltUVGRhYZGbmzt0xxs3bvj7+zPXETc3tyVLlvj6+i5btiwhIUEsFitvyVbXGBzMLhfn0uPanG0IOHleyHvOwCITE5ORVp08eVJx+UTjA7M7Ci0tLWyXgP5C3nMGhBiYXUQqzC4iFWYXkQqzi0iF2UWkwuwiUmF2Eakwu4hUmF1EKswuIhVmF5EKs4tIhdlFpMLsIlJx8fO75eXlbJeACMDF7/ywXQIaHte+88O57CKkJhzvIlJhdhGpMLuIVJhdRKr/Bwz53w3KI3H/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "plot_model(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "verified-symbol",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tensorflow.python.keras.layers.core.Flatten at 0x7f265c066940>,\n",
       " <tensorflow.python.keras.layers.core.Dense at 0x7f265c068780>,\n",
       " <tensorflow.python.keras.layers.core.Dense at 0x7f26089cafd0>,\n",
       " <tensorflow.python.keras.layers.core.Dense at 0x7f25c07dfa90>]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "solar-challenge",
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden1 = model.layers[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "conditional-appeal",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'dense_3'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden1.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "removable-cargo",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_layer('dense_3') is hidden1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "based-stanford",
   "metadata": {},
   "outputs": [],
   "source": [
    "weights, biases = hidden1.get_weights()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "electrical-retro",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.05857375,  0.01345541,  0.02950101, ...,  0.03264638,\n",
       "         0.03299656, -0.01004117],\n",
       "       [-0.06238474, -0.01696922, -0.00365889, ..., -0.00986675,\n",
       "        -0.02094959, -0.0299071 ],\n",
       "       [-0.03669054, -0.00370631, -0.0731606 , ..., -0.07421935,\n",
       "        -0.05710587, -0.05018653],\n",
       "       ...,\n",
       "       [-0.06372381,  0.04984473, -0.00389662, ...,  0.03585874,\n",
       "         0.063226  , -0.02182372],\n",
       "       [-0.0501985 , -0.06869788,  0.01067672, ..., -0.02409806,\n",
       "         0.04619578, -0.07279514],\n",
       "       [-0.02049983,  0.02913231,  0.05829445, ..., -0.01215513,\n",
       "        -0.04034815, -0.03252812]], dtype=float32)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "collective-thriller",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(784, 300)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "liquid-spokesman",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "looking-spectacular",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300,)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "biases.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "moderate-chinese",
   "metadata": {},
   "source": [
    "Se puede inicializar los pesos y *bias* con `kernel_initializer` o `bias_initializer` en el momento de crear la capa (por defecto los pesos son aleatorios y los *bias* son ceros)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "satisfied-velvet",
   "metadata": {},
   "source": [
    "#### Compilar el modelo\n",
    "\n",
    "Después de definir el modelo, hay que usar `compile()` para especificar la función de costo y el optimizador. También se puede especificar una lista de métricas para calcular durante el entrenamiento y la evaluación."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "incorporated-brave",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=\"sgd\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "oriented-afternoon",
   "metadata": {},
   "source": [
    "Usamos `sparse_categorical_crossentropy` ya que tenemos etiquetas \"esparcidas\" (los *targets* son números enteros entre $0$ y $9$) y las clases son exclusivas.\n",
    "\n",
    "Si tuvieramos probabilidades por clase (con *one-hot vector* por ejemplo) usaríamos `categorical_crossentropy`.\n",
    "\n",
    "Para clasificación binaria, usaríamos como función de activación en la última capa `sigmoid` (en vez de `softmax`) con `binary_crossentropy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "hawaiian-seminar",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "1719/1719 [==============================] - 6s 4ms/step - loss: 0.7338 - accuracy: 0.7598 - val_loss: 0.5134 - val_accuracy: 0.8256\n",
      "Epoch 2/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.4910 - accuracy: 0.8285 - val_loss: 0.4545 - val_accuracy: 0.8432\n",
      "Epoch 3/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4441 - accuracy: 0.8442 - val_loss: 0.4205 - val_accuracy: 0.8574\n",
      "Epoch 4/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.4159 - accuracy: 0.8541 - val_loss: 0.3935 - val_accuracy: 0.8648\n",
      "Epoch 5/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3961 - accuracy: 0.8603 - val_loss: 0.3781 - val_accuracy: 0.8710\n",
      "Epoch 6/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3800 - accuracy: 0.8653 - val_loss: 0.3715 - val_accuracy: 0.8716\n",
      "Epoch 7/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3660 - accuracy: 0.8709 - val_loss: 0.4256 - val_accuracy: 0.8444\n",
      "Epoch 8/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3542 - accuracy: 0.8742 - val_loss: 0.3482 - val_accuracy: 0.8800\n",
      "Epoch 9/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3440 - accuracy: 0.8776 - val_loss: 0.3484 - val_accuracy: 0.8760\n",
      "Epoch 10/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.3341 - accuracy: 0.8799 - val_loss: 0.3544 - val_accuracy: 0.8720\n",
      "Epoch 11/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3250 - accuracy: 0.8835 - val_loss: 0.3368 - val_accuracy: 0.8824\n",
      "Epoch 12/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3168 - accuracy: 0.8853 - val_loss: 0.3336 - val_accuracy: 0.8826\n",
      "Epoch 13/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3099 - accuracy: 0.8878 - val_loss: 0.3360 - val_accuracy: 0.8778\n",
      "Epoch 14/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.3024 - accuracy: 0.8907 - val_loss: 0.3274 - val_accuracy: 0.8808\n",
      "Epoch 15/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2960 - accuracy: 0.8935 - val_loss: 0.3803 - val_accuracy: 0.8634\n",
      "Epoch 16/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2905 - accuracy: 0.8952 - val_loss: 0.3158 - val_accuracy: 0.8864\n",
      "Epoch 17/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2836 - accuracy: 0.8975 - val_loss: 0.3186 - val_accuracy: 0.8858\n",
      "Epoch 18/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2786 - accuracy: 0.8988 - val_loss: 0.3134 - val_accuracy: 0.8854\n",
      "Epoch 19/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2742 - accuracy: 0.9011 - val_loss: 0.3110 - val_accuracy: 0.8908\n",
      "Epoch 20/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2680 - accuracy: 0.9025 - val_loss: 0.3083 - val_accuracy: 0.8898\n",
      "Epoch 21/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2630 - accuracy: 0.9039 - val_loss: 0.3042 - val_accuracy: 0.8900\n",
      "Epoch 22/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2590 - accuracy: 0.9059 - val_loss: 0.3017 - val_accuracy: 0.8920\n",
      "Epoch 23/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2536 - accuracy: 0.9079 - val_loss: 0.2986 - val_accuracy: 0.8918\n",
      "Epoch 24/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2495 - accuracy: 0.9088 - val_loss: 0.3231 - val_accuracy: 0.8848\n",
      "Epoch 25/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2458 - accuracy: 0.9111 - val_loss: 0.3025 - val_accuracy: 0.8922\n",
      "Epoch 26/30\n",
      "1719/1719 [==============================] - 5s 3ms/step - loss: 0.2418 - accuracy: 0.9123 - val_loss: 0.2990 - val_accuracy: 0.8928\n",
      "Epoch 27/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2381 - accuracy: 0.9137 - val_loss: 0.3144 - val_accuracy: 0.8876\n",
      "Epoch 28/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2342 - accuracy: 0.9147 - val_loss: 0.2965 - val_accuracy: 0.8936\n",
      "Epoch 29/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2309 - accuracy: 0.9164 - val_loss: 0.3312 - val_accuracy: 0.8780\n",
      "Epoch 30/30\n",
      "1719/1719 [==============================] - 6s 3ms/step - loss: 0.2269 - accuracy: 0.9168 - val_loss: 0.3021 - val_accuracy: 0.8912\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=30,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "disciplinary-central",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "sustained-minute",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAEzCAYAAAACSWsXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABT1UlEQVR4nO3dd3xUVd7H8c+ZkkxmJr0RQuhNWkCagFKVsouiLvaCrOVxbavYHruPbV3brrquLmvvYl1XwQoBsSEgSEekJSSkt8kkmXafP+5kSJmEAAkTJr/36zWve+fOnTtnjiPfnHPPPVdpmoYQQgghQscQ6gIIIYQQnZ2EsRBCCBFiEsZCCCFEiEkYCyGEECEmYSyEEEKEmISxEEIIEWIHDWOl1ItKqQKl1MZmXldKqaeUUjuUUr8opY5v+2IKIYQQ4as1LeOXgZktvD4L6Od/XAE8e+TFEkIIITqPg4axpmkrgJIWdpkDvKrpfgDilFJpbVVAIYQQIty1xTnjdCC73vMc/zYhhBBCtIKpDY6hgmwLOsemUuoK9K5soqKiRmZkZLTBx+t8Ph8Gg4xHa0zqJTipl+CkXoKTeglO6iW4lupl+/btRZqmJTfe3hZhnAPUT9VuQG6wHTVNWwgsBBg1apS2evXqNvh4XVZWFpMnT26z44ULqZfgpF6Ck3oJTuolOKmX4FqqF6XUnmDb2+JPmo+Bi/2jqk8AyjVNy2uD4wohhBCdwkFbxkqpt4DJQJJSKge4BzADaJr2HLAY+B2wA3AC89ursEIIIUQ4OmgYa5p23kFe14Cr26xEQgghRCcjZ96FEEKIEJMwFkIIIUJMwlgIIYQIMQljIYQQIsQkjIUQQogQkzAWQgghQkzCWAghhAgxCWMhhBAixCSMhRBCiBCTMBZCCCFCTMJYCCGECDEJYyGEECLEJIyFEEKIEJMwFkIIIUJMwlgIIYQIMQljIYQQIsRMoS6AEEII0W40DXwe8HlB8/kfdetave31XvfVez2hNxjav90qYSyEEOLIaRp4asFbCx6Xf1kLXlejpf91n1sPSa9HX/f6n/s8/nW3/7VGr7ur9eN4auo9auttr/e62/862uF/r//dC5bYNqum5kgYCyFEOPF6wF0FripwOf3rzuDbXFX6urtaD0uv27/0r3tqm24LrLsYX1MF32l6wHpd7fedDGYwmvWl2QKmukckmKP0ZWRMw+em+vtFgDKCMoDBv1QG/zYVZLvhwP6mqPb7XvVIGAshRFvxuvVgc1eD26m3ylxOfd1dfSD46p4Hfc3/qGslBlqK3gOtyUCLsl7L0ec9EJSHwmTRA8wYqYeWse5hPrAeYQVjfMNtRjOF+UWkd++lPzdFHliaLE23BY4feeA4BpM/ZE0HwtZgPLBu9D/vBCSMhRDhwedt2A3qqWnUNVqv27RJ12nL2wbmZkP+8/6grKkXoP7ArVv3eQ693EZ/ay7Cpi/NUXprzOgPI7NVDyuDCYymA+uNH3WhFmHT3xNhBbNNfx5Yt/pfr9tmPaKw+zUri/TJkw/7/eIACWMhRNvz+fzn+ep3dwbrBvUHm8vfhXoo615/4NadnzycIGyOsV4L0RRJrNsHvgR/WFrBWm/dZNGXdc/NloavBUK2XtjWbTNF6QErOj35FQjRmfh8Bwa9uKvrrevL+JK1sLncfz7R0TQIA88dB84/uhz+c471zi8eaTAqQ8NWXV1rzhIDMWn+oKvr+oxs2g16sG1Gc9Mu1PrblGpQnB+zspgsLUDRjiSMheioNE0PvNrKeo+KA+suR8Pn9R91AVl/RGldt20LMgF+CfJCky5OG0TYwZ56oJVX/1ygqd564/OPDR5m/TiNu09NliaBGKgWnw+fwwEGI8psQplMKOPRPa+oeTz4amrQqqv1ZU0NvpoafNXV+np1DVpNNb7aWv0NSqGU8n8n/1Ip/2q95/7XVIQZc3o6ERkZGGNi2uU7eB0OXHv24N6zB/f+fDSPfvmP5vWC14fmO/gyOr+Awg0bMaWkYEpO1pcpyZgSEw/7v4nP6cSdl4c7bz/uvFw8eXm4c/Nw5+Wh1dZiGz8O+5QpWAYPRrXjJUe+2lpqt24lKjOz3T6jPgljIdqa1+NvOfpbj7UOcNULypq6AC1v9Lxe2NZU6O/RfAf/PGMkREY3fNi7+EedRjUcfRoYaRp8+9oNWzh+7En+cLT7u12tR+U6S9D/Ifbsy8edX4CnIB9Pvn89379eUICnsBA8jVreSqFMJjCb9XCu98BsQpnMgdDW0PQrXTSt4QMNTQv+WmJVFb8qpQduTQ243UelPgCMsbGYu3cnIiMDc/cMIjK6E9E9A3NGBqaUlBYDyVdVhWvPHlx79+LavUdf9z+8xcUtf7BSYDTqxzca9XCt/9xgwFLtpGjlSn/91WMwYEpK8oezP6D9YW1OScFgs+HOz9eDNm+/P3zz8OTm4i0vb3qslBTMaWkAFD33L4r++Sym5GTsU6ZgnzIZ27hxGCyWw6jdAzS3m+oNG3H++ANVP/xI9c8/o7nd9PvuW0zx8Ud07NaQMBadj7uaiNpiKNnZsNXY+DrF+s/r71frCBK29Z57alpXDpNFvxyjLkAtMWDrfWCbJUYPREtMw/0i7P71GIi06+F6pFWSm0vJa69j/uQTdkR+AEYDymCst2z4j3CzS/9DGeouF2l+HYMCtwdPYSHugnw8+QX4KiublM1gs2FKTcWUmoJtzBhMqakYE+LBp6F5PGgeN5rbDR4PmttzYJvHA4Hn+gOPp16rtFHrlOa3lxcVkdSjByrKgiHSoi8tURiiLKjAMsi2iAj9S9QLdj23tCaBH/hDAA1fdTXunH24c7Jx7c3Gnb2X6l9+oeLzz8HrDdSNiozE3K1bIKiNcXG49+07ELiFRQ1/csnJRPTogX3KZCJ69CCiew8ievbAnJaGMpsb/vdspmeivqysLCZNmICnuBhPYSGeggI8BQX6H00FBXgKCnHv20f1unV4S0qCHsMQE4O5SxfMaWlEDc/EnNYVc1oa5q5pmLt00f/gMJsD+3tKS6lasYLKpcuo+PRTyhYtQlks2MaPJ3rqFOyTJ2NKSjpo2TWvl5rNW3Cu+pGqH37EuWYNmtMJQORxxxF/3nlYTxiLwWo96LHagoSxaFOeoiJUZCQGu71V/zMfNq/bf87SAdWlDR/OkibbfJUl1O4rozrXSU0RdK0ws0tvDEHdQqtX3np/6B/4o18PE2OUAZPdhCk6EmOMBVNsNKb4dEzxMZgSEzAmJKCiYg505UbY9dAMBKg/WE0RBz7D48HncOB1OPBVVuKtrNS7Yms1rJljMEZHt0s1Vm/YQMlLL1Hx+Rd6tQ4aRFTPnuD1ovl8jZb1uil9mr7N5UKr28fnQ9N8+ms+fSYjrW7d59PDpvG6wYApOZnIXr2wjT0BU2oq5lR/ayo1FVNKKka7rV2++6H4NSuL44/yOeOowYObbNPcbtx5eYGAdu3NxpW9F3d2DlWrVqE5nRiTk4jo3gP7SRP1wO2hB25ERgYGW9vXpTKb9TDt0qXF/TSXC09REZ6CAryOKv2/c1oaRrv9kD7PFB9P7Jw5xM6Zg8/lwrnqJxzLllG5bCmOpUtBKSzDhhI9ZSr2KVOI7N8PpRSaz0ftrzv0lu+Pq3CuWhX44y+iTx/iTj8d6wljsY4efVRawk2+11H/RBGWvOXl7L//ASo++UTfYDJhjI/DFBePMd7/iI3BGB2FyRaB0WrCGGXAGOnDFOHBaHajVC0qMDjI2cJgIWeL5z59XqittFLjiKWmNJLqQqgtcoFPATaMdgveJDu2uDj9vKPB6G+pmfTLPPwPVffcaDwwIYDXg6eklOqiQrxbi/A5i4FG3X0GA8aEBL2bLjERU1ISxrhYfE4n3kp/2Doq8dWtV1UF/iIPRpnN2CZOJGbmTOxTphxxOGleL5VLl1Ly8itUr1mDwW4nYd48Ei68gG+3b2e4DFTqsJTZTET37kR07w5MaPCapmloLheGyCPvKWkPKiICc9eumLt2bbNjGiIisJ84AfuJE0i98w5qt23Tg3npMgr//ncK//53zOnpRPbvT/X69YHWubl7d2JmzsQ6dizWMaMxp6S0WZkOl4SxaD1N08Owphyqy/RlTRlVP/1M7j8/wlPuJHFqX4wWH95yB95KJ96q3Xj2/krtNi/eGvC6DA1boPUog4bRomG0GDBaDZisZoz2CIx2C0Z7AsaY7hhjozGlxmKMi8OYkIiy2KktrKV6byk1u/Ko+XUvNTt2Bs4pGmOjsAwZgv30wViGDCZqyBBMaWksX768TUbH+qqq9C66omI8RYV4iorwFhXhKSzSWwFFRdTu2omvrByDzYbBbscQHY3RbsfcJQ1DtB2jPVpfRkdjsNkPrNuj0WqqqfzqKyo++xzH11+jIiOxT5xIzO9mYZ806ZC60HxVVZR9+BElr76Ke+9ezF27knrb/xL7h7kHAn779iOuExEaSilUBw3io0EphWXgQCwDB5L0pz/hLijAkZWFY1kWrp07sZ90EtYTTsA2dkyb/kHQViSMOxjN59P/Ic/LxZ1b71FQoJ//atJ12NxS71JMcLsp3Z9P3B/ObHDeJcDjgso8qMiFin3+Za6+raasUfCW65Op+/k8ioL10ZT+aicixk3Pk8uISqsESxz0jwVLqj6na1Scvs0ShxYZi89nwes24qlReKt9eJ0evJU1eCsdeEpL8ZaW4S0tpaa0FO9vpXgripoOEKmjVOA1Q3Q0liGDSZw/EcvgIViGDMacnt6u3eUGm40Im83fUmkf1tGjSbn1Vqp//pmKxUuo+OJzKr/8EhUVhX3yJGJmzsI+aWKzA1jc+fmUvv4GpYsW4Ssvx5I5jJQFNxB98sn6ACchwpA5JYX4s88m/uyzQ12UVpH/E48yzeXCvX+/P2TzGgZurj6MX2s0UtMQE4M5NQVljmgyYEaZTKjICH93aqNBN/io3rKJ/ffeS/FzT5E8O5OYARaUY/+B4K0qaFpIs02/ljMqHqxJkNDHH6ix/lCNpTqnktyn38OVk0/8WaeRcsP1GOJSDjqbjwKM/kdEi3vWqzOvF29FBd7S0sCjLrR91U4s/fphGTIEc0ZG+56nDiFlMGAdORLryJGk3n4bztVrqPhsCZWff0Hlks8wWK3Yp04lZtZMbCeeiCEykpotWyh5+WXKP10MPh/RJ59MwvxLsI4YEeqvI4RoRMK4HWguF66cHP1Sgr17AtfyuXbvwZ2X16SVZ0pOxty1K1FDBmOefgqmtDT/uZV0zOldGw5w8PnAWQyOfHDsh8p8/3o+VO4HR4G+3VEALgfp48GRG0nhL25yn19GcbyPlJNisA3tjkrLhJh0iOmqh2/demRM89d4ut0U/WshRc8+hyk5me4vvYht3Lj2rE6U0YgpPj4kgyo6ImU0Yhs7BtvYMXS54w6cP/1ExeIlVH7xBRWffILBbieiZ09qNm5EWa3En38eCRddRERGRqiLLoRohoTxYdIDdx+uvf6g9Yeta+9e3Lm5emj6GWJiiOjRg6jjjye2e3fM6XrImtPSMKWlYah/+YOzBMr3QnkOlHwFu7KhbK/ekq3M11uywWY3iowBe4p+fWnXEfpkDPZUtuaUcNy8adhtXaj4bgOF/1xI9sfZRO2LImXBPKwjR7b6O9fu3EXurbdSs2EDMaedSpc772y3CQlE6yiTCdu4cdjGjaPL3XdR9cOPVHy2hNotW0m5+SbizjpL/hsJcQyQMG4lzeejZtNmqr79lqqVK3GuW9dg4gGD3a4H7rBhxJ52auCSAnOPHhjj4vTuU03TQ7VsL5TtgT0r4ZdsKMuG8mw9gN2NRtWabRCXAbHdIHWwP2S76MEb3SUQukQEH8iTn5XFcb0noYDYMwYQ8/vTKH3vPYqefZY9F1yIfdIkkm+4HsvAgS1+99I336LgsccwREaS/ve/EzNzRhvUqmhLymzGftKJ2E86MdRFEUIcIgnjFrjzCwLhW/Xdd3jLygD9gvCEiy8msl+/wDV8xvj4hucrfV4o+hWyv4RV6yFvPez/RR8EVZ81SQ/b5IHQb7oeurEZ/gDO0M/btuF5UBURQcL55xN3+umUvP4Gxc8/z64zziTmd78j+bpriejRo1Ed5JN32+1UffcdtoknkfbAAx3iMgAhhAgnEsb1+Gprca5eTdXKb6n69ltq/Zd5GJOSsE+aiG3CBGzjxzed3cVTC3nrIO+XA6Gbv+lAK9dk0Vu1g8+ELkMhvifEddeD13x0blzdmMFqJemKy4k/52yKn3+Bktdeo+Lzz4mb+weS/nQV5tQUyj/5lP333YfmdtPl3nuIO+ecsB0gJYQQodSpw1jTNFy//YZj5UqqVn6L86ef0GprUWYzUSNHknLTjdgmTCBywICG878W/wa/LYXcdXr4Fm45cB43Mga6DIORl0Bapr6e1L/D3ibNGBtLyo0LiL/oQoqfe47SRe9S/tF/iBoyBOfq1URlZtL1rw8T0bNnqIsqhBBhq2MmRDur/e03/XrNJUtw7dwJQETv3sSdczb2CROwjh7dcDIFnxf2/gjbFsO2JVC0Td9uTdIDt98pkDZMX4/redQm1W9L5pQUutx9NwmXXELh0/+g8ssvSb7+ehIvu1SuRRVCiHbWaf6VdWVn6wG8eDG127aBUlhHjybhIn0QU5MZWWodsHOZHr7bPwdnkT5dYs8TYfSl0H8GxPVo0/O5HUFE9+6kP/oIms/XrrcnE0IIcUBYh7E7L4+KJZ9RsWQJNRs2ABA1fDipt99O9IwZmFMbDUSqyIPtS/QA3rlcv1m6JVYfWDVgFvQ9WX/eCUgQCyHE0RN2YewpLKTi8y+oWLyY6rVrAbAMHkzKzTcRM3Mm5vT0hm/I3wRbP9W7oHN/1rfF99RbvwNmQfdx+g3QhRBCiHYSFmHsLSsj6ptv2PPyKzhXrQKfj8h+/Ui+/s/EzJrV5HKdgHVvwkd/AhR0GwXT7oYBv4fkAWHX/SyEEKLjCoswrt60iZg33sTTowdJV/4PMbNmEdmvX8tvKtgCnyyAnifBH16A6NSjU1ghhBCikbAIY9vYsRTffjsTLrqwddfBuqpg0Tz9hu8SxEIIIUIsLMJYmUx4uh/CHXs+vQmKtsNFH0oQCyGECLnON2T25zdg/Zsw6RboMyXUpRFCCCFaF8ZKqZlKqW1KqR1Kqf8N8nqsUuq/Sqn1SqlNSqn5bV/UNlCwBT69UT9PPOnWUJdGCCGEAFoRxkopI/AMMAsYBJynlBrUaLergc2apmUCk4HHlVKtvXf80eGqgncv8Z8nfh4MxlCXSAghhABa1zIeA+zQNG2npmku4G1gTqN9NCBa6Sdt7UAJEOSmuyG0+GYo3AZn/lu/9aAQQgjRQShN01reQam5wExN0y7zP78IGKtp2jX19okGPgYGAtHAOZqmfRrkWFcAVwCkpqaOfPvtt9vqe+BwOLDb7UFfS92/lOO2PsnuHmezu9cFbfaZx4KW6qUzk3oJTuolOKmX4KRegmupXqZMmbJG07RRjbe3ZjR1sCHKjRN8BrAOmAr0Ab5USn2jaVpFgzdp2kJgIcCoUaO0yZMnt+LjWycrK4ugxyvYCt/+G3qeRM+Ln6NnJ+uebrZeOjmpl+CkXoKTeglO6iW4w6mX1nRT5wAZ9Z53A3Ib7TMf+EDT7QB2obeSQ8tVBe/OgwibnCcWQgjRYbUmjH8C+imlevkHZZ2L3iVd315gGoBSKhUYAOxsy4IelsW3+M8TL5TzxEIIITqsg3ZTa5rmUUpdA3wOGIEXNU3bpJS60v/6c8D9wMtKqQ3o3dq3appW1I7lPrh1b8G612HizdBnakiLIoQQQrSkVTNwaZq2GFjcaNtz9dZzgeltW7QjULgNPl0APU6ESU0uixZCCCE6lPCbgcvl1OedNlv188TGsJjxUwghRBgLv6RacjMUboUL34eYtFCXRgghhDio8GoZr38bfn4dJt4EfaeFujRCCCFEq4RNy9halQ3f3gI9Jsh5YiGEEMeU8GgZu5wM2vyo/zzxC3KeWAghxDElPFJr9zdYnTlwwbtynlgIIcQxJzxaxv1n8OPY5+Q8sRBCiGNSeIQxUGtJCXURhBBCiMMSNmEshBBCHKskjIUQQogQkzAWQgghQiwswnh/eQ1f7XFT5nSFuihCCCHEIQuLMN5dXMXrW1z8vLcs1EURQgghDllYhPHQ9FgUsC67LNRFEUIIIQ5ZWISxLdJEul1JGAshhDgmhUUYA/SOM7I+pwxN00JdFCGEEOKQhE8Yxxooc7rZW+IMdVGEEEKIQxI2YdwrVv8q0lUthBDiWBM2YdzNbsBiNkgYCyGEOOaETRgbDYqh6bGslzAWQghxjAmbMAbI7BbHxtwK3F5fqIsihBBCtFp4hXFGHC6Pj237K0NdFCGEEKLVwiqMh2fEAfCzdFULIYQ4hoRVGHeLjyLRFiHnjYUQQhxTwiqMlVJkZsRJGAshhDimhFUYgz6Ia0ehg8oad6iLIoQQQrRK2IXx8O5xaBpsyCkPdVGEEEKIVgm7MM7sFgvAupyy0BZECCGEaKWwC+M4awQ9E61y3lgIIcQxI+zCGPTrjWVaTCGEEMeKsAzj4Rlx5FfUsr+8JtRFEUIIIQ4qLMM40z/5h7SOhRBCHAvCMowHpcVgMijWyyAuIYQQx4CwDGOL2chxaTGs21sW6qIIIYQQBxWWYQz6eeMN+8rx+rRQF0UIIYRoUdiGcWZGHI5aDzsLHaEuihBCCNGisA3j4Rn+yT9kEJcQQogOLmzDuHeSnehIk4SxEEKIDi9sw9hgUAzLiJUR1UIIITq8sA1j0O/gtDWvkhq3N9RFEUIIIZoV3mGcEYfHp7EpV+7gJIQQouMK6zAeEZiJS8JYCCFExxXWYZwSYyEt1iJ3cBJCCNGhhXUYg37eWAZxCSGE6MjCP4wz4thT7KSkyhXqogghhBBBtSqMlVIzlVLblFI7lFL/28w+k5VS65RSm5RSy9u2mIdvuP+8sbSOhRBCdFQHDWOllBF4BpgFDALOU0oNarRPHPBP4DRN0wYDZ7V9UQ/P0G6xKIWcNxZCCNFhtaZlPAbYoWnaTk3TXMDbwJxG+5wPfKBp2l4ATdMK2raYh88eaaJfil3CWAghRIfVmjBOB7LrPc/xb6uvPxCvlMpSSq1RSl3cVgVsC5nd4liXXYamyR2chBBCdDymVuyjgmxrnGomYCQwDYgCvldK/aBp2vYGB1LqCuAKgNTUVLKysg65wM1xOBzNHs9a7abU6ebdJctIsYb9mLUGWqqXzkzqJTipl+CkXoKTegnucOqlNWGcA2TUe94NyA2yT5GmaVVAlVJqBZAJNAhjTdMWAgsBRo0apU2ePPmQCtuSrKwsmjte0r5yXtm8Ekv6QCZndm2zzzwWtFQvnZnUS3BSL8FJvQQn9RLc4dRLa5qJPwH9lFK9lFIRwLnAx432+Q9wklLKpJSyAmOBLYdUknY0oEs0kSaDnDcWQgjRIR20ZaxpmkcpdQ3wOWAEXtQ0bZNS6kr/689pmrZFKfUZ8AvgA57XNG1jexb8UJiNBoamx8rtFIUQQnRIremmRtO0xcDiRtuea/T8UeDRtita28rMiOP1H/bg9vowGzvXeWMhhBAdW6dJpcyMOGo9Prbtrwx1UYQQQogGOk0YD+8WByBd1UIIITqcThPGGQlRJNgiZBCXEEKIDqfThLFSisxusTJHtRBCiA6n04Qx6OeNfy1w4Kj1hLooQgghRECnC2NNg1+kdSyEEKID6VRhXDeIa312eWgLIoQQQtTTqcI43hZBj0SrDOISQgjRoXSqMAb9Dk4yiEsIIURH0vnCOCOOvPIa8itqQl0UIYQQAuiEYTw8Iw6QyT+EEEJ0HJ0ujAd3jcFkUHLeWAghRIfR6cLYYjYyMC1azhsLIYToMDpdGIPeVf1Ldjk+nxbqogghhBCdM4wzu8VRWethZ5Ej1EURQgghOmcYHxjEJZN/CCGECL1OGca9k+3YI02syy4NdVGEEEKI8AjjSlcl75e8j9PtbNX+RoNiWLdYmRZTCCFEhxAWYbyhaAPLK5dz64pb8fq8rXpPZkYcW/IqqHG3bn8hhBCivYRFGI/vOp65CXPJysniL6v+gqYdfJR0Zrc4PD6NzXkVR6GEQgghRPPCIowBJkZP5JLBl/DOtnd4ZdMrB90/MIhrb1n7FkwIIYQ4CFOoC9CWbhh5A7mOXB5f8zhp9jRm9JzR7L5dYi10ibHI5B9CCCFCLqzC2KAMPHTSQxRWF3L7N7eTHJXM8anHN7t/ZkasTIsphBAi5MKmm7pOpDGSp6Y8RZo9jeuWXceu8l3N7puZEcfuYielVa6jWEIhhBCiobALY4A4SxzPTnsWozJy1VdXUVxdHHS/cb0TAbj+nXU4XZ6jWUQhhBAiICzDGCAjJoOnpz5NUXUR1y69lmpPdZN9RnSP5+Ezh/LNr4Wc/+8fpYUshBAiJMI2jAGGJQ/j4YkPs7FoY7PXIJ87pjvPXjiSzXkVzH3uO/aVNQ1tIYQQoj2FdRgDTOs+jVvH3Mqy7GU88tMjQa9BnjG4C6/9cQwFlbXMffY7fs2vDEFJhRBCdFZhH8YAFxx3ARcNuog3t77Ja5tfC7rP2N6JLPqfcXh8GnOf+541e2TeaiGEEEdHpwhjgJtG3cQpPU7hsdWP8eWeL4Puc1xaDB/8aTzxVjMXPP8DS7fmH+VSCiGE6Iw6TRgblIGHTnyIYcnDuO2b21hXsC7ofhkJVt7703j6pti5/NU1vL8m5+gWVAghRKfTacIYwGKy8NTUp0i1pnLt0mvZU7En6H5J9kjeuvwETuidwI3vrmfhit+OckmFEEJ0Jp0qjAESLAk8e/KzKBRXfXUVJTUlQfeLtph58ZLR/H5YGg8t3spDi7fg8x38BhRCCCHEoep0YQzQPaY7T019inxnPtcuvZby2uD3NY40GXnq3BFcPK4HC1fs5Kb31uP2+o5yaYUQQoS7ThnGAMNThvPwSQ+zqWgTZ/znDLKys4LuZzQo/u+0wSw4pT8frN3H/7y2hmqX3ANZCCFE2+m0YQxwco+TeeP3bxBniePapddy64pbKaspa7KfUorrpvXjwTOGkLWtgAue/4Eyp8zWJYQQom106jAGGJw4mHd+/w5/yvwTX+z+gjn/mdPspU8XjO3BPy84no37Kjjrue/ljk9CCCHaRKcPYwCz0cxVw6/i7dlvk2pNZUHWAm7MujHoDSZmDknjlT+OoaTKxZxnvuXat35mb7EzBKUWQggRLiSM6xmQMIA3fv8G1464lmXZyzjjP2ewZNeSJlNojuuTSNbNk7l2al++3LyfaU9kcd9/N8uNJoQQQhwWCeNGzAYzVwy7gkWzF5FuT+eWFbdw/bLrKaouarBftMXMjdMHkHXTFM4c0Y2Xv9vFxEeX8dzy36hxywAvIYQQrSdh3Iy+8X157XevsWDkAlbuW8mcj+bw39/+26SV3CXWwl/nDmPJnycyumcCDy/ZytTHsvhgbY5clyyEEKJVJIxbYDKYmD9kPu+d9h69Y3tz+8rbuWbpNeRXNZ2zekCXaF68ZDRvXj6WRHskCxatZ/bTK/nm18IQlPzI5DnyeGrtU1S65O5VQghxNEgYt0Kv2F68PPNlbhl9C6vyVnH6f07ng18/wKc1nQBkfJ8k/nP1BJ48dzgVNW4uemEVF7+4ii15FSEo+aErdBZy2ReX8e8N/+ax1Y+FujhCCNEpSBi3ktFg5KJBF/H+ae8zIGEA93x3D+d+ci7f5HzTpOvaYFDMGZ7O1zdO4s7fH8f67DJ+99Q33PTuevLKq0P0DQ6utKaUK768gsLqQk7pcQof/PoB3+77NtTFEkKIsCdhfIi6x3TnxRkv8tCJD1HpquSqr6/iks8uYU3+mib7RpqMXHZSb1bcPIUrTurNx+tzmfRoFn9++2e+21HUoc4pV7oqufKrK8muzOYfU//BX076C71je3PPd/dId7UQQrSzVoWxUmqmUmqbUmqHUup/W9hvtFLKq5Sa23ZF7HgMysCpfU7l49M/5q4T7iK7MptLPruEK7+6ks3Fm5vsH2s1c9vvjmPpjZM4Z1QGS7cWcP7zPzLx0WU8+dWv7CsLbWvZ6XZy9ddXs710O09MfoIxaWOINEZy/4T7Kawu5PHVj4e0fEIIEe4OGsZKKSPwDDALGAScp5Qa1Mx+fwU+b+tCdlRmo5mzB5zNp2d+yo0jb2Rj0UbO+eQcbsy6kZ3lO5vs3y3eyv2nD+GnO07myXOH0yPRyt++2s6Jf13KRS/8yH/X51LrObqXRdV6a7l+2fWsL1zPX0/6KxO7TQy8Nix5GPMGz+P9X9/nu33fHdVyCSFEZ9KalvEYYIemaTs1TXMBbwNzgux3LfA+UNCG5TsmRJmiuGTIJSw5cwlXZl7Jyn0rOeM/Z3DXt3eR68htsr/FbGTO8HTeuOwEvrllCtdO7cdvBQ6ufetnxj70Nfd+vIlNucHvJNWW3D43Ny2/ie/zvue+8fcxvef0JvtcPfxqesX24p7v78HhcrR7mYQQojNqTRinA9n1nuf4twUopdKBM4Dn2q5ox57oiGiuHn41S/6whAuPu5DFOxcz+8PZPLzq4SaThtTJSLCy4JT+fHPrVF67dAwn9k3izR/38vunVjL76W949fvdlDvdbV5Wr8/LHSvvICs7izvG3sGcvsH+viLQXV3gLODxNdJdLYQQ7UE1HgncZAelzgJmaJp2mf/5RcAYTdOurbfPu8Djmqb9oJR6GfhE07T3ghzrCuAKgNTU1JFvv/12m30Rh8OB3W5vs+O1hVJPKZ+Xf873ju8xKROToyczLWYaVqO1xfc5XBo/5HlYkeNhb6UPkwFGphgZ3cXE0CQjkSbV6jIEqxdN03ir5C2+d3zPnLg5nBx78kGP81HpR3xd8TVXp1zNwKiBrf78jqoj/l46AqmX4KRegpN6Ca6lepkyZcoaTdNGNd7emjAeB9yradoM//PbADRN+0u9fXYBdQmRBDiBKzRN+6i5444aNUpbvXp1i599KLKyspg8eXKbHa8t7anYwzPrnmHJriXYzXYmZUxicrfJTEifQHREdIvv3bivnHdXZ/Of9bmUOd1Emgyc2DeJ6YNTmXZcKkn2yBbf37heNE3jkZ8e4fUtr3PFsCu4dsS1zb+5nhpPDWf99yxqvbV8OOdDbGZbq97XUXXk30soSb0EJ/USnNRLcC3Vi1IqaBibWnHcn4B+SqlewD7gXOD8+jtomtar3ge9jN4y/qi1BQ93PWJ68MjER7h0yKW8uvlVvsn5hk93fopJmRiZOpKJ3SYyOWMy3WO6N3nvkPRYhqTHctfsQfy0u5QvNu/ni035fL21AKU2MKpHPNMHdeGUQan0TDp4QP5j3T94fcvrXHjchVwz/JpWfweLycL9E+5n3mfzeHz149w97u5DqgMhhBDNO2gYa5rmUUpdgz5K2gi8qGnaJqXUlf7XO/V54kMxIGEAD574IF6flw1FG8jKzmJ5znIeXf0oj65+lF6xvZjcbTITu01keMpwTIYD/3lMRgPj+iQyrk8id88exOa8Cr7cnM8Xm/J5cPEWHly8hQGp0ZwyKJXpg1MZmh6LUg27s1/Y8AILf1nIH/r9gVtG39Lk9YMZnjKciwddzMubXuaUHqcwruu4tqgWIYTo9FrTMkbTtMXA4kbbgoawpmmXHHmxwpvRYGR4ynCGpwzn+pHXk12ZzYqcFWRlZ/Haltd4adNLxEbGcmL6iUzqNokJ6ROIiYgJvF8pxeCusQzuGsv1J/cnu8SpB/Pm/fwzawf/WLaDtFgLpwxKJcXt5QS3lw9/W8Tf1/6dWb1mcdcJdx1yENe5evjVZGVnce939/LBnA+O+e7qcOT2utlWuo3BiYMP+7+zEOLoalUYi/aVEZ3BBcddwAXHXYDD5eC73O9YnrOcFTkrAt3ZmSmZ9I7tTbo9nfTodNJt+jI+Mp6MBCt/PLEXfzyxF6VVLpZuLeCLzftZtDqbGrePZ3Y+hCl1Eb2tY7ik320Y1OFPvFbXXX3xkov525q/cecJd7ZhTYgj5fa6uSHrBpbnLOf2sbdz3sDzQl0kIUQrSBh3MPYIO9N7Tmd6z+l4fV5+KfqF5dnLWbV/FV/u+ZKy2rIG+0eZoki3p9PN3o306HS62rqSnpzOgt9344Ez+/HAx/9mqetdzK4BrN96Kr9b8x2pMZGc2DeZif2TmNA36aCDwBqr665+ZfMrnNLjFMamjW3DGhCHy+1zc8uKW1ies5w+sX14ZNUj9Ivrx6guTcaKCCE6GAnjDsxoMDIiZQQjUkYEtlW5q8ipzCHXkcs+xz72OfaR48hhn2Mfq/avwulxNjnOiJQRPHfyc5RWKVb+Wsg3vxbx9dZ83l+bA8CgtBhO6p/ExH7JjOwRj8VsPGjZrhlxDctzlnPPd/fwwWkfYDW3fLmWaF9en5c7vrmDr/Z+xa2jb2VO3zmc/+n53Lj8Rt6Z/Q5dbF1CXUQhRAskjI8xNrONAQkDGJAwoMlrmqZRXlveIKC37djGndPuxGq2Yo2Dc0Z355zR3fH6NDbllvPNr0Ws2F7Iiyt38a/lO7GYDYzplchJfZMY3SuBQWkxRJiadmtbTBbum3Af85bM44k1T0h3dQj5NB93f3c3S3Yv4YaRN3DhoAsBeHLKk5z36XksyFrASzNfItJ4aD0gQoijR8I4jCiliLPEEWeJY3DSYACyirKCXstsNCiGdYtjWLc4rp7Sl6paDz/uKmbF9iK++bWQBxdvAcBiNjCsWxyjesQz0v+Is0YAeov7wkEX8trm15jeYzpj0sYcvS8rAD2I7/v+Pj7+7WOuGX4Nfxzyx8BrveN689CJD3F91vU8+MOD/N/4/5MBXUJ0UBLGAgBbpImpA1OZOjAVgP3lNazZU8rqPSWs3VPKwhU78fhv+dg3xc7I7vGM7BnPqRl/ZEX2Cu7+7m7prj7KNE3joR8f4v1f3+eKYVfwP5n/02SfaT2mccWwK1j4y0IGJw7mnIHnhKCkQoiDkTAWQXWJtfD7YWn8flgaAE6Xh/XZ5azdW8rq3SUs2ZjHO6v1KcvjEk7Fm/oMl/33/7jh+FsZ3DUGa0T7/rQqXZXsr9qPw+1gSOIQzEZzu35eR6NpGo+ufpR3tr3D/MHzW5zA5arMq9hSvIWHVz1Mv/h+HJ96/FEsqRCiNSSMRatYI0yBSUcAfD6N3wodrN5Typo93VhetJENLObcV9PRqvvQJ9kemD1saHosg7vGYIts3c/N7XWT78wnryqP/VX72V+1P7Bet3S4D9xBKjoimpO7n8zMnjMZkzamwWQp4UjTNJ5c+ySvbX6NC4+7kBtG3tBi97PRYOThiQ9z3if6+eN3Zr9Dqi31KJZYCHEw4f2vlmg3BoOiX2o0/VKjOW9Md6o9j3DGR3+gqt9HpJiGU1pVy7IiF4v3e2ANKDSiLSbirWZirSbirGZiokyYDAoNDY/PQ35VPvur9lNYXYhGwznT4yPj6WLrQvfo7ozpMoY0Wxpd7F0wKzNLs5fyxZ4v+HDHhyRYEvRg7jWT41OOx2g4+MjwY82z65/lhY0vcHb/s1s9k1pMRAxPTnmS8xefz4LlC3hpxktEGCOOQmmFEK0hYSzaRJQpiocnPsQdK++g2L0WZVUkWBU+DTxeDY8P3F6N/S4f+2qAEj1ATEYDkUYjFrOZVGsKo1JPoEdsuh62ti6k2dJItaUSZYpq9rOn9ZjG3d67WblvJZ/t+oz/7vwvi7YvIjkqmek9pzOz50wykzPDYvDS8xue59n1z3JG3zO444Q7Duk79Y3vy4MnPsiCrAX8ZdVfuGfcPe1YUiHEoZAwFm1meMpwPj3z04PuV1hZy8Z95WzwPzbuK2dveQ170e9Kkh4XxeCuMQzuGsugrlFEdAVLrNZi8EQaI5nWfRrTuk/D6XayImcFn+3+jHe3vcsbW94gzZbGzJ4zmdFrBoMSBrXdlz6KXtn0Ck+ufZLZvWdzz7h7DmsmtVN6nMJlQy/j+Q3PMyhxEGf1P6sdSiqEOFQSxuKoS46OZMrAFKYMTAlsK3bUsim3wv8oZ3NuBV9uyafuDp8JtggGpcUwuGsMg/xB3SvJhtHQNKCtZisze81kZq+ZOFwOlmUvY8muJby2WZ/3u3t0d/qpfuRsziE6IjrwiImICazbzLbDCjuvz0u1p5oqdxVOj1N/uJ14NS/do7uTZks7rBb6m1ve5LHVjzG9x3Tun3D/EXW/XzP8GrYUb+GhHx+iX1w/hqcMP+xjCSHahoSx6BAS7ZFM7J/MxP7JgW1VtR625OkBvTm3gk155bz07W5cXh8AUWYjx6VFM6BLDAO7RPsfMcRaD4ystkfYObXPqZza51TKa8v5eu/XLNm1hGV5y/j6p6+bLY9CYY+wNwjoaHM0UeYoajw1ON3OBmFb7anG6XZS461p8XtaTVb6xPWhT1wf+sb1DSxTranNhvS729/lL6v+wtSMqTw88eEjHqBmNBj568S/cu4n5wYGdCVbkw/+RiFEu5EwFh2WLdLEqJ4JjOqZENjm8vjYUeBgU265HtJ5FSzekMdbq/YG9ukSY2FgWjQD/AE9IDWGPik2YiNjObPfmZzZ70y+XvY1o8aPosJVQaWrssGjuW3Zjmyq3dVYTBZ9RjOTlURLYmDdZrYRZY7CarIGttWtA+wu382Osh38Vv4bK3JW8NGOjwJltpvt9I7rrQd07IGg/i73O+77/j5OSj+JRyc9itnQNpdwxUbG8uTUJ7lw8YUsyFrAizNe7HSXh4nw53Q7eXLtkyzPWc4fh/yRM/ud2WGvtuiYpRKiGREmA4P8XdV1Zzs1TWN/RQ1b91eyzf/YklfBtzuKcHv1fm6TQdE72RZoRbsKNfo6THSL74Yx+ugM7Gp8Q42SmhJ+K/uN38p+00O67DeW7V3GB7UfNNhvXNo4/jblb20++rl/fH/um3AfNy+/mb/+9FeZ0lSElbX5a7nr27vYW7mXPrF9uP+H+3lr61vcNOomJqRPCHXxmpAwFsc8pRRpsVGkxUYxZcCB89Bur49dRVVs3V/J1rwKtu2vZO2eUv67PheAJ9dmEWky0CvJRt8Ue4NHryQbkab2vSwqwZJAQpcERncZ3WB7cXVxIKCrPdWcf9z57Tav9MyeM9lcvJmXNr7EoMRBJJBw8DeJDsfpdvLe9vf4ueBnpnafyik9TsFisoS6WCFR46nh6Z+f5rXNr9HV3pUXZ7zIqNRRfL33ax5f/ThXfnUlJ6afyE2jbqJPXJ9QFzdAwliELbPRQP/UaPqnRnNaZtfA9ooaN+8sWUFMt37sKHCwo8DB+pwyPt2QFxgwZlDQI9FGn+SGId0n2Ua0pX27cxOjEkmMSjxqc33/ecSf2Vq8lQd+eIDrUq5r8JrX56XKU0WVqwqH20GVW186XI4Gz11eF4mWRFKsKSRFJQWWMj1q+yqvLefNrW/yxpY3KK8tJ8GSwFd7v+LhVQ9zWp/TOKv/WfSO6x3qYh41vxT+wh0r72B3xW7O7n82C0YtwGa2AXByj5OZ2G0ib219i3+t/xd/+PgPzO0/l6uGX0WCJfR/hEoYi04nxmKmX7yRyaO7N9he7fKys0gP598KHPzqD+rl2wsC3d0AibYIMhKsdK/3yEiw0j3RSpcYS9AR3h2Z0WDkkYmPcO6n5/LP/H/y/ofvB0K22lPdqmOYDCY8Pk+T7XaznWRrMilRKSRbk0mOStaX/vWkqCRsZhtWkxWLyXJYI9g7o0JnIa9tfo13tr2D0+NkcsZkLht6GUOThvLT/p94b/t7vL3tbV7f8jrHpxzP3P5zmd5zesjv3KVpGttLt7O3ci9j08YSExHTJsd1eV38c90/eWnTS6RYU/jXKf9ifNfxTfaLMEYwb/A8TutzGs+uf5ZF2xbx6c5PuWLYFVxw3AUhnQhHwlgIv6gII4O7xjK4a2yD7R6vjz0lTnYUONhZWMXeEifZJU7WZeutaa/vQFCbjYpu8f5wTohqENY9E22tnhL0aIuzxPH01Ke576v7SE1IxW62YzfbsUXYDqybbQ221T23mq0oFBWuCoqqiyhwFlBYXUiBs+DAc2chPxf8TKGzEJfP1Ww5okyNBsDVW9a9ZjPbsJr1Zf1HXZnqlyvcwj27MpuXN77MRzs+wqN5mNlzJpcOvZT+8f0D+4xNG8vYtLEUVxfz8W8f897297h95e0hay3XemtZlbeK5TnLWZGzgryqPADMBjMnpp/I73r9jkkZk1qc2Kclm4o3cefKO9lRtoMz+53JTaNuCnqnuvriLfHcPvZ2zh1wLo+veZwn1jzBO9veYcHIBZzS45SQTBDUMf9lEKIDMRkN9Em20yfZ3uQ1j9dHXnkNe0uc7Cl2BoJ6b4mT9dlllFe7G+yfEh1JryRb4NEzyUbvJBsZCVYs5tBO3dkvvh+XJl/K5EmTD+v9sZGxxEbGtngeTtM0KlwVgcAuri5ucplY/WW1u5oKVwX5zvwGr7UU6PXVhXf9PyTiI+OJi4wj3hKvPyLjibPENdje0aYK3VG6gxc2vsCSXUswKAOn9z2d+YPnkxGT0ex7EqMSmT9kPvMGz+On/T/x7vZ3G7SWzxpwFqf0OKVdWsuFzkJW5Kxgec5yfsj7gWpPNVGmKMaljeNPmX8iIzqDr/d+zee7P2dZ9jKiTFFMyZjC73r9jvFdx7dqZL/b6+Zfv/yL5zc8T6IlkX9O+ycndTvpkMrZO643z0x7hu9yv+Ox1Y9x4/IbOT7leG4efTNDkoYc7tc/LBLGQhwBk9FAhr/lO6Fv09fLnW6yS/Wg3l1cxc7CKnYXV/HF5nxKqg4EilL6zGO9kmz0TGwY1l3jLO0+mOxoUUoFQrtffL/DPo7b66bKXUWVpwqHy4HT48Th0s9h13Wx163Xf17pqmRr1VZKa0spry1v9vg2s00PZn9Q15bVsvKHlYFWel3Xel0rvXGL3WrSW/JH2sLaULiBf2/4dyCwLjzuQi4efDEp1pSDv9nPoAxBW8u3fXMbD696mFN7n8rsPrNJtaYSGxF7WJe4aZrG5pLNrMheQVZOFpuLNwOQZktjTp85TM6YzKguoxoE/6guo7hp1E2syV/Dkt1L+HLPlyzetZiYiBhO6XEKs3rNYlTqqKAT3Gwr2cYdK+9gW+k2TutzGreMvoXYyNgm+7XW+K7jeXf2u3y440Oe/vlpzvv0PGb3ns2fj/8zXWxdDvu4h0JpmnbwvdrBqFGjtNWrV7fZ8bKyspg8eXKbHS9cSL0E1xHqpbzaze6iKnbVe+wurmJXYRWVtQfOvyoFqdEWusVH+R9WusVHkZGgL9Nio4gwtU13bEeol6PF4/NQXltOWW0ZpTWl+rK2lNKa0gbPy2rK2F++H5/Rh9PjpNZb26rjKxRWsxW72R6YOKb+euPn9dfznfm8uPFFfsz7kZiIGC447gLOH3g+cZa4NvnuPs0XaC1/vffrBuf7o0xRxETEEBMZQ2xEbGA9JkJ/xEYe2LZm/RpK40r5JucbCqoLUCgykzOZlDGJSd0m0Teub6v/IHF73Xyf9z2Ldy1m6d6lVHuqSY5KZkbPGczqNYuhSUPxaB5e2PAC/1r/L2IjY7ln3D1M6T6lTeqkjsPl4IWNL/DqplcxKAMfzvmQbtHdDukYLf1/pJRao2naqMbbpWUsRIjERpnJzIgjMyOuwXZN0yiucunhXFRFTmm1/+Hkp92lfLw+l3qnqVFKn+ikflCnx0WRGmuhS4z+iLOaw+JGGW3JZDAFRq4fTP1/XN0+d2DWtbqWt9Pj1JduvRu9ylMVeF7pqsThdlDpqqSouohd5bsCI9I9WtNBb3WSo5K5adRNzO0/NzAiuK00bi2v2r+KstoyKmorqHDpj/La8sBkNxXF+rZgA/psZTYmdJ3ApIxJnJh+4mGPTDYbzUzsNpGJ3SZS7almec5yluxcwjvb3uH1La+Tbk/Harbya+mvzOo1i9vH3N5mf5zUZ4+w8+fj/8xZ/c/i892fH3IQHy4JYyE6GKUUSfZIkuyRjO7Z9B82t9fH/vKaQEDXD+tVu0r4eH1Ng0FlAJEmA6n+YNZDOlJ/7g/sVP9DHJzZYA50tR8JTdOo9lQHgrnCVRFYNygDkzImHZXRz4lRiczqNatV+7q9bspdekhX1Fawdu1aLjrlojafvS3KFMXMnjOZ2XMmla7KwDS2uY5cnpj8BKf0OKVNPy+YrvauzB8yv90/p46EsRDHGHO989TQtFXn8frYX1FDfkUN+8tr663XsL+ihl9yyviivIZaj6/Je2MioPemb+mR2PDSrR6JNlKiIzEcY5dtdWRK6d3YVrP1kM4Bh5LZaCYpKomkqCQAyixl7T6NanRENKf3PZ3T+57erp8TahLGQoQZk9Hg765ufsINTdOoqPawv0IP6Hx/UK/ZshNPpJG1e/WZyuo3sCNNhgbXV9cFdo9E/bNCPRpciGOZhLEQnZBSilirmVirmQFdDlyTmWXcx+TJJwB6d3huWTV7ip3s8V+ytae4ir0l1fy4s5gql7fBMWMsJlJiLKRER+qPGAvJ9khSYiJJjo4kJdpCcnQkMRaTnL8WohEJYyFEUGajgR6JNnokNh08pGkaJVWuQEhnlzgpqKylsLKWgspa1uwtpaCiNmhXeKTJQEqMP5ztkSTaI0i0R5JkjyDRpj+vW4+NMkvXuOgUJIyFEIdMKUWiPZJEeyTHd48Puo+maVTWeiioqKWgsobCemFdUFFDQWUtvxU6WLXbRanTRbCrLI0GRYItgkRbBEl1wW2LJCk6gq6xUaTXjRw/BqchFaI+CWMhRLtQShFjMRNjMdM3pensZfV5fRqlThfFDhfFjlqKqvRlscNFcVUtRf7t2dlOih0uHLUNLwkyGRRdYi2kxx0I6PrrXeOi5Jy26NAkjIUQIWc0HLicC1qeVxjA6fKQW6Zf0rWvrJp9/mVuWTU//FbM/ooaGl3dRZI9kvQ4C8nRFv08tv98dt257JRo/fPbagIVIQ6FhLEQ4phjjTDRNyWavinBg7vuWuz6Qb2vtJrccv167J/3llJcFXx+6wRbRIOBZ8nRkVTkuyn9OYd4awQJtojA0hphlMFook1IGAshwk7Da7GDc3t9FDn857Erail01DY4v11QWcvOwioKK2txeX28tXV9k2NEmAwkWCOIt0WQYDM3CesEW0RgspWUmEjpKhfNkjAWQnRKZqOBtFh9bu+WaJrG4q+yGDRiDCVVLkqrXJQ4Gy2r3JQ6XWzOraDE6aLM6Q56rDirOTDjmb6MDExbWjcjWoI1QkaQd0IdKozdbjc5OTnU1NQc8ntjY2PZsmVLO5Tq2HYk9WKxWOjWrRtmc/vOsCNER6aUwmZWgTtptYbH66O82k1xlSsw+1m+f4KV/eV663tLXgWFjtomo8jNRkWyPZIEewQJtkgS/S3tRPuB1nbdI9EWQYxFLv8KBx0qjHNycoiOjqZnz56HfB6msrKS6OiDD/zobA63XjRNo7i4mJycHHr16tUOJRMifJmMhsClX/1Tm///z+P1Ueio9Yd1bSCwCypqKamqpaTKxa4iByUOV5NJVuoYDYp4qznQPR5vjSDOaibOv4y3momNiiDevy3eP9lLuNyWM1x0qDCuqak5rCAWbU8pRWJiIoWFhaEuihBhy9TKrnKAGreXkipX0EdxlYuSqlpKq9zsLHJQ6nRT5nTh9jZ/i1xrhJG4qHqhbdODOsEaQZz/nHdc/ZC3RWCTAWvtpkOFMSD/oTsQ+W8hRMdhMRvp6r9mujU0TcPp8lJW7aa0ykV5tX5eu8wf1GVON6VON+XVLkqdbrbkVVBa5aKs2h10AhbQu9DrWt/xNjOeqho+L9ngb3X7gz3qQLDHRumBbjbK5WIH0+HCONTsdjsOhyPUxRBCiCOilMIWacIWaSK9lQEO+gQsFdVu/0C0A4PTSqv00NaX+iPX4WP35v2UOd14Gl/YXU90pIlYq7lBF3pC3dJW153ecL2zXTYmYSyEECLAaFB6y9YWcdB9s7KymDx5Mpqm4aj1+Fvd7kBYl1e7Ka1yU1btCmwvc7rZW+KktMpFRY2n2WNHmAzE+4O5riUeZ40gNkqf1S0myuRfmomxmPxLffuxeD5cwrgZmqZxyy23sGTJEpRS3HnnnZxzzjnk5eVxzjnnUFFRgcfj4dlnn2X8+PFceumlrF69GqUUf/zjH7nhhhtC/RWEEOKoUEoRbTETbTGTkdD699WNOtfDu36ru2lrfNv+Ssqcbipq3C2eCwf9ZiR1IR0bpQf2gevB664DN5NgiwxcHx5njQjp/OYdNoz/77+b2Jxb0er9vV4vRmPLfw0N6hrDPacObtXxPvjgA9atW8f69espKipi9OjRTJw4kTfffJMZM2Zwxx134PV6cTqdrFu3jn379rFx40YAysrKWl1uIYTorOqPOm8tTdOo9eghXlGth3NFtce/dFNR42myvdjhYkeBg5IqF85mRqUrBbH1QrsusO+aPYhoS/tf3tlhwzjUVq5cyXnnnYfRaCQ1NZVJkybx008/MXr0aP74xz/idrs5/fTTGT58OL1792bnzp1ce+21/P73v2f69OmhLr4QQoQlpRQWsxGL2UhqjOWQ31/j9lLqdPkncHHXm7jFdWC708W+smo27CvjvqM0+KzDhnFrW7B12vo6Y62Z4YQTJ05kxYoVfPrpp1x00UXcfPPNXHzxxaxfv57PP/+cZ555hkWLFvHiiy+2WVmEEEK0DYvZ2OrLyY4mGW/ejIkTJ/LOO+/g9XopLCxkxYoVjBkzhj179pCSksLll1/OpZdeytq1aykqKsLn8/GHP/yB+++/n7Vr14a6+EIIIY4hHbZlHGpnnHEG33//PZmZmSileOSRR+jSpQuvvPIKjz76KGazGbvdzquvvsq+ffuYP38+Pp8PgL/85S8hLr0QQohjSavCWCk1E3gSMALPa5r2cKPXLwBu9T91AH/SNK3pLU6OAXXXGCulePTRR3n00UcbvD5v3jzmzZvX5H3SGhZCCHG4DtpNrZQyAs8As4BBwHlKqUGNdtsFTNI0bRhwP7CwrQsqhBBChKvWnDMeA+zQNG2npmku4G1gTv0dNE37TtO0Uv/TH4BubVtMIYQQIny1pps6Hciu9zwHGNvC/pcCS4K9oJS6ArgCIDU1laysrAavx8bGUllZ2YoiNeX1eg/7veHsSOulpqamyX+ncOBwOMLyex0pqZfgpF6Ck3oJ7nDqpTVhHGxKkqDX/SilpqCH8YnBXtc0bSH+LuxRo0ZpkydPbvD6li1bDvvyJLmFYnBHWi8Wi4URI0a0YYk6hrpp/ERDUi/BSb0EJ/US3OHUS2vCOAfIqPe8G5DbeCel1DDgeWCWpmnFh1QKIYQQohNrzTnjn4B+SqleSqkI4Fzg4/o7KKW6Ax8AF2matr3tiymEEEKEr4O2jDVN8yilrgE+R7+06UVN0zYppa70v/4ccDeQCPzTf8srj6Zpo9qv2EIIIUT4aNV1xpqmLQYWN9r2XL31y4DL2rZo4c3j8WAyyZwrQgghZDrMoE4//XRGjhzJ4MGDWbhQv2T6s88+4/jjjyczM5Np06YB+oi5+fPnM3ToUIYNG8b7778PgN1uDxzrvffe45JLLgHgkksuYcGCBUyZMoVbb72VVatWMX78eEaMGMH48ePZtm0boI+AvummmwLHffrpp/n6668544wzAsf98ssvOfPMM49GdQghhGhnHbdptuR/Yf+GVu8e5fWA8SBfp8tQmPVwy/sAL774IgkJCVRXVzN69GjmzJnD5ZdfzooVK+jVqxclJSUA3H///cTGxrJhg17O0tLSlg4LwPbt2/nqq68wGo1UVFSwYsUKTCYTX331Fbfffjvvv/8+CxcuZNeuXfz888+YTCZKSkqIj4/n6quvprCwkOTkZF566SXmz59/8IoRQgjR4XXcMA6hp556ig8//BCA7OxsFi5cyMSJE+nVqxcACQn63bO/+uor3n777cD74uPjD3rss846K3Df5fLycubNm8evv/6KUgq32x047pVXXhnoxq77vIsuuojXX3+d+fPn8/333/Pqq6+20TcWQggRSh03jFvRgq2vuo2uM87KyuKrr77i+++/x2q1MnnyZDIzMwNdyPVpmoZ/wFoD9bfV1NQ0eM1mswXW77rrLqZMmcKHH37I7t27A9elNXfc+fPnc+qpp2KxWDjrrLPknLMQQoQJOWfcSHl5OfHx8VitVrZu3coPP/xAbW0ty5cvZ9euXQCBburp06fzj3/8I/Deum7q1NRUtmzZgs/nC7Swm/us9PR0AF5++eXA9unTp/Pcc8/h8XgafF7Xrl3p2rUrDzzwQOA8tBBCiGOfhHEjM2fOxOPxMGzYMO666y5OOOEEkpOTWbhwIWeeeSaZmZmcc845ANx5552UlpYyZMgQMjMzWbZsGQAPP/wws2fPZurUqaSlpTX7Wbfccgu33XYbEyZMwOv1BrZfdtlldO/enWHDhpGZmcmbb74ZeO2CCy4gIyODQYMa36tDCCHEsUr6ORuJjIxkyZKgU2sza9asBs/tdjuvvPJKk/3mzp3L3Llzm2yv3/oFGDduHNu3H5gj5f777wfAZDLxxBNP8MQTTzQ5xsqVK7n88ssP+j2EEEIcOySMjyEjR47EZrPx+OOPh7ooQggh2pCE8TFkzZo1oS6CEEKIdiDnjIUQQogQkzAWQgghQkzCWAghhAgxCWMhhBAixCSMhRBCiBCTMD4C9e/O1Nju3bsZMmTIUSyNEEKIY5WEsRBCCBFiHfY647+u+itbS7a2en+v1xu4G1JzBiYM5NYxtzb7+q233kqPHj246qqrALj33ntRSrFixQpKS0txu9088MADzJkzp9XlAv1mEX/6059YvXp1YHatKVOmsGnTJubPn4/L5cLn8/H+++/TtWtXzj77bHJycvB6vdx1112B6TeFEEKEpw4bxqFw7rnncv311wfCeNGiRXz22WfccMMNxMTEUFRUxAknnMBpp50W9K5KzXnmmWcA2LBhA1u3bmX69Ols376d5557jj//+c9ccMEFuFwuvF4vixcvpmvXrnz66aeAfjMJIYQQ4a3DhnFLLdhgKtvgFoojRoygoKCA3NxcCgsLiY+PJy0tjRtuuIEVK1ZgMBjYt28f+fn5dOnSpdXHXblyJddeey0AAwcOpEePHmzfvp1x48bx4IMPkpOTw5lnnkm/fv0YOnQoN910E7feeiuzZ8/mpJNOOqLvJIQQouOTc8aNzJ07l/fee4933nmHc889lzfeeIPCwkLWrFnDunXrSE1NbXKP4oPRNC3o9vPPP5+PP/6YqKgoZsyYwdKlS+nfvz9r1qxh6NCh3Hbbbdx3331t8bWEEEJ0YB22ZRwq5557LpdffjlFRUUsX76cRYsWkZKSgtlsZtmyZezZs+eQjzlx4kTeeOMNpk6dyvbt29m7dy8DBgxg586d9O7dm+uuu46dO3fyyy+/MHDgQBISErjwwgux2+1N7vQkhBAi/EgYNzJ48GAqKytJT08nLS2NCy64gFNPPZVRo0YxfPhwBg4ceMjHvOqqq7jyyisZOnQoJpOJl19+mcjISN555x1ef/11zGYzXbp04e677+ann37i5ptvxmAwYDabefbZZ9vhWwohhOhIJIyD2LBhQ2A9KSmJ77//Puh+Doej2WP07NmTjRs3AmCxWIK2cG+77TZuu+22BttmzJjBjBkzDqPUQgghjlVyzlgIIYQIMWkZH6ENGzZw0UUXNdgWGRnJjz/+GKISCSGEONZIGB+hoUOHsm7dulAXQwghxDFMuqmFEEKIEJMwFkIIIUJMwlgIIYQIMQljIYQQIsQkjI9AS/czFkIIIVpLwjgMeDyeUBdBCCHEEeiwlzbtf+ghare0/n7GHq+XkoPczzjyuIF0uf32Zl9vy/sZOxwO5syZE/R9r776Ko899hhKKYYNG8Zrr71Gfn4+V155JTt37gTg2WefpWvXrsyePTswk9djjz2Gw+Hg3nvvZfLkyYwfP55vv/2W0047jf79+/PAAw/gcrlITEzkjTfeIDU1FYfDwXXXXcfq1atRSnHPPfdQVlbGxo0b+dvf/gbAv//9b7Zs2cITTzxx8IoWQgjR5jpsGIdCW97P2GKx8OGHHzZ53+bNm3nwwQf59ttvSUpKoqSkBIDrrruOSZMm8eGHH+L1enE4HJSWlrb4GWVlZSxfvhyA0tJSfvjhB5RSPP/88zzyyCM8/vjjPPLII8TGxgam+CwtLSUiIoJhw4bxyCOPYDabeemll/jXv/51pNUnhBDiMHXYMG6pBRtMR7ufsaZp3H777U3et3TpUubOnUtSUhIACQkJACxdupRXX30VAKPRSGxs7EHD+Jxzzgms5+TkcM4555CXl4fL5aJXr14AZGVlsWjRosB+8fHxAEydOpVPPvmE4447DrfbzdChQw+xtoQQQrSVDhvGoVJ3P+P9+/c3uZ+x2WymZ8+erbqfcXPv0zTtoK3qOiaTCZ/PF3je+HNtNltg/dprr2XBggWcdtppZGVlce+99wI0+3mXXXYZDz30EAMHDmT+/PmtKo8QQoj2IQO4Gjn33HN5++23ee+995g7dy7l5eWHdT/j5t43bdo0Fi1aRHFxMUCgm3ratGmB2yV6vV4qKipITU2loKCA4uJiamtr+eSTT1r8vPT0dABeeeWVwPapU6fyj3/8I/C8rrU9duxYsrOzefPNNznvvPNaWz1CCCHagYRxI8HuZ7x69WpGjRrFG2+80er7GTf3vsGDB3PHHXcwadIkMjMzWbBgAQBPPvkky5YtY+jQoYwcOZJNmzZhNpu5++67GTt2LLNnz27xs++9917OOussTjrppEAXOMDNN99MaWkpQ4YMITMzk2XLlgVeO/vss5kwYUKg61oIIURoSDd1EG1xP+OW3jdv3jzmzZvXYFtqair/+c9/mux73XXXcd111zXZnpWV1eD5nDlzgo7yttvtDVrK9a1cuZIbbrihua8ghBDiKJGWcSdUVlZG//79iYqKYtq0aaEujhBCdHrSMj5Cx+L9jOPi4ti+fXuoiyGEEMJPwvgIyf2MhRBCHKkO102taVqoiyD85L+FEEIcHR0qjC0WC8XFxRICHYCmaRQXF2OxWEJdFCGECHsdqpu6W7du5OTkUFhYeMjvrampkeAI4kjqxWKx0K1btzYukRBCiMZaFcZKqZnAk4AReF7TtIcbva78r/8OcAKXaJq29lALYzabA9M4HqqsrCxGjBhxWO8NZ1IvQgjR8R20m1opZQSeAWYBg4DzlFKDGu02C+jnf1wBPNvG5RRCCCHCVmvOGY8BdmiatlPTNBfwNtB4dok5wKua7gcgTimV1sZlFUIIIcJSa8I4Hciu9zzHv+1Q9xFCCCFEEK05ZxzsFkONhzu3Zh+UUlegd2MDOJRS21rx+a2VBBS14fHChdRLcFIvwUm9BCf1EpzUS3At1UuPYBtbE8Y5QEa9592A3MPYB03TFgILW/GZh0wptVrTtFHtcexjmdRLcFIvwUm9BCf1EpzUS3CHUy+t6ab+CeinlOqllIoAzgU+brTPx8DFSncCUK5pWt6hFEQIIYTorA7aMtY0zaOUugb4HP3Sphc1TduklLrS//pzwGL0y5p2oF/aJHerF0IIIVqpVdcZa5q2GD1w6297rt66BlzdtkU7ZO3S/R0GpF6Ck3oJTuolOKmX4KRegjvkelEy9aQQQggRWh1qbmohhBCiMwqLMFZKzVRKbVNK7VBK/W+oy9NRKKV2K6U2KKXWKaVWh7o8oaKUelEpVaCU2lhvW4JS6kul1K/+ZXwoyxgKzdTLvUqpff7fzDql1O9CWcZQUEplKKWWKaW2KKU2KaX+7N/eqX8zLdRLp/7NKKUsSqlVSqn1/nr5P//2Q/q9HPPd1P7pOrcDp6BfYvUTcJ6maZtDWrAOQCm1GxilaVqnvg5QKTURcKDPEjfEv+0RoETTtIf9f8DFa5p2ayjLebQ1Uy/3Ag5N0x4LZdlCyT97YJqmaWuVUtHAGuB04BI68W+mhXo5m078m/Hfm8GmaZpDKWUGVgJ/Bs7kEH4v4dAybs10naIT0zRtBVDSaPMc4BX/+ivo/6h0Ks3US6enaVpe3Y1uNE2rBLagzyjYqX8zLdRLp+afBtrhf2r2PzQO8fcSDmEsU3E2TwO+UEqt8c9+Jg5IrbsW3r9MCXF5OpJrlFK/+LuxO1VXbGNKqZ7ACOBH5DcT0KheoJP/ZpRSRqXUOqAA+FLTtEP+vYRDGLdqKs5OaoKmacej31Xran+3pBAteRboAwwH8oDHQ1qaEFJK2YH3ges1TasIdXk6iiD10ul/M5qmeTVNG44+++QYpdSQQz1GOIRxq6bi7Iw0Tcv1LwuAD9G79IUuv+7OYv5lQYjL0yFompbv/4fFB/ybTvqb8Z/7ex94Q9O0D/ybO/1vJli9yG/mAE3TyoAsYCaH+HsJhzBuzXSdnY5SyuYfZIFSygZMBza2/K5O5WNgnn99HvCfEJalw2h069Mz6IS/Gf+AnBeALZqmPVHvpU79m2muXjr7b0YplayUivOvRwEnA1s5xN/LMT+aGsA/lP7vHJiu88HQlij0lFK90VvDoM+09mZnrRel1FvAZPQ7qeQD9wAfAYuA7sBe4CxN0zrVYKZm6mUyenejBuwG/qezzTOvlDoR+AbYAPj8m29HPz/aaX8zLdTLeXTi34xSahj6AC0jegN3kaZp9ymlEjmE30tYhLEQQghxLAuHbmohhBDimCZhLIQQQoSYhLEQQggRYhLGQgghRIhJGAshhBAhJmEshBBChJiEsRBCCBFiEsZCCCFEiP0/6Zzxd/hLlvEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.DataFrame(history.history).plot(figsize=(8, 5))\n",
    "plt.grid(True)\n",
    "plt.gca().set_ylim(0, 1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fixed-angel",
   "metadata": {},
   "source": [
    "Podemos evaluar el modelo en el conjunto de prueba:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "brazilian-angle",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - 1s 2ms/step - loss: 81.3090 - accuracy: 0.8149\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[81.30901336669922, 0.8148999810218811]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hazardous-happiness",
   "metadata": {},
   "source": [
    "También podemos realizar predicciones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "authorized-peninsula",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = X_test[:3] #usamos las primeras 3 imagenes \n",
    "                   #del conjunto de prueba como imagenes \"nuevas\"\n",
    "y_proba = model.predict(X_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "applicable-moisture",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],\n",
       "       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "stuck-upgrade",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-35-81ace37e545f>:1: Sequential.predict_classes (from tensorflow.python.keras.engine.sequential) is deprecated and will be removed after 2021-01-01.\n",
      "Instructions for updating:\n",
      "Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype(\"int32\")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([9, 2, 1])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict_classes(X_new)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "widespread-vegetarian",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib.pyplot import imshow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "moderate-matthew",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9, 2, 1])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(model.predict(X_new), axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "fresh-lunch",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Ankle boot', 'Pullover', 'Trouser'], dtype='<U11')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(class_names)[y_pred]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "approximate-captain",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9, 2, 1], dtype=uint8)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_new = y_test[:3]\n",
    "y_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "light-consideration",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f251d6229e8>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQPElEQVR4nO3dW4xd9XXH8d+amTPjYWxjD77UNQZsMAhaCdNOTVqqiog0JbyYSCGCh5RKSI5UkIKE1CL6ENQn2jSN+lBFchoUt0pBqRIEqlADsmholAgxXGIMJFwshwwePJjxZXyd2+rDbKoJzF57OPd0fT/S6MzsdfY+y2fOz/vM+e+9/+buAvD/X0+nGwDQHoQdSIKwA0kQdiAJwg4k0dfOB+u3AV+hoXY+JJDKOZ3WtJ+3pWoNhd3Mbpb0T5J6Jf2Luz8U3X+FhnS93dTIQwIIPOf7Smt1v403s15J/yzpc5KukXSHmV1T7/YAtFYjf7PvlPSWux9092lJj0ra1Zy2ADRbI2HfLOlXi34eK5b9GjPbbWajZjY6o/MNPByARjQS9qU+BPjYsbfuvsfdR9x9pKaBBh4OQCMaCfuYpC2Lfr5Y0uHG2gHQKo2E/XlJ281sq5n1S7pd0hPNaQtAs9U99Obus2Z2j6QfamHo7WF3f7VpnQFoqobG2d39SUlPNqkXAC3E4bJAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBqastnMDkmakjQnadbdR5rRFIDmayjshU+7+9EmbAdAC/E2Hkii0bC7pKfM7AUz273UHcxst5mNmtnojM43+HAA6tXo2/gb3P2wmW2Q9LSZ/dzdn118B3ffI2mPJK22YW/w8QDUqaE9u7sfLm4nJD0maWczmgLQfHWH3cyGzGzVh99L+qykA81qDEBzNfI2fqOkx8zsw+38u7v/V1O6AtB0dYfd3Q9KuraJvQBoIYbegCQIO5AEYQeSIOxAEoQdSKIZJ8IAHWF98cvX5+aCYmMHc/ZccEFYnz9zJqzbdb9TWvOXXq2rpyrs2YEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcbZs1s4RTmoV+wP5oOxbEm927eV1iZu3Biuu+E/Xgvrc8dPhPVWqhpHr3Lwi6tLa1tfamjTpdizA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLMjVjGOXuW9z5SPpR8bmQnXPb2p/JxvSbrkb39SV0/N0HfplrD+7q64XptqZjfLw54dSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnD0566uFdZ+ZDuszn/n9sH7iqvLrs9fejx/7/OXn4vpTl4X1946vKq1dsCL+dx0buzCs19aeD+sXrjoa1k8cjrffCpV7djN72MwmzOzAomXDZva0mb1Z3K5tbZsAGrWct/HfkXTzR5bdL2mfu2+XtK/4GUAXqwy7uz8rafIji3dJ2lt8v1fSrc1tC0Cz1fsB3UZ3H5ek4nZD2R3NbLeZjZrZ6Iziv3MAtE7LP4139z3uPuLuIzUNtPrhAJSoN+xHzGyTJBW3E81rCUAr1Bv2JyTdWXx/p6THm9MOgFapHGc3s0ck3ShpnZmNSfqqpIckfc/M7pL0jqTbWtkkGtDTG5arxtF718TjwW98Id6+BR/TzA3Ec6QProw/4zGL1+/pKa9XrXvFVeNh/eDhdWH92ImhsK6+xuaHr0dl2N39jpLSTU3uBUALcbgskARhB5Ig7EAShB1IgrADSXCK63JFUxt7xTBKxfCXfL6iHm/f+sp/jT47G2+7wtv3XRPWByoOp+o9V/68nbkk7u2CgfhS02Pvxydb9vSWP6/z8/F+bvLMYFifn45/pwOr4mHDWn/5v71quLPeqarZswNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEnnG2aNxcql6rLyqHmlw2uNoHF1qbCx94i//KKxPb4jHutfsjy8HPR+03rc6Pr128lh8mqgf64/rF5Vvv9YX/05qvY39zqLTayVp5WD5OPzMtdvibf/opfp6qmstAL9xCDuQBGEHkiDsQBKEHUiCsANJEHYgiTzj7I2Mk0vhOenWW3G55tl4rLqqt0bG0cfvi8fRp66It73i3YpplYfjx/fg8IYVg/E4+6nxlfHGV8Zj4dFlAk6djWcnGhyIe1PlYRsVdwj88uYVYX3rj+rbLnt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUjiN2ucver665Gqa7Nbxf97wTnp3uD56lV6r9ga1g/dvqm0NjdYcV712/FLYLZi5uGqaZenh8ufm/7p+LGtYqy6b7Di+IXA3Fz8+z43HR9foLm4t/NnKs7zny9f/9KdY/Fj16lyz25mD5vZhJkdWLTsQTN718xeLr5uaUl3AJpmOW/jvyPp5iWWf8PddxRfTza3LQDNVhl2d39W0mQbegHQQo18QHePme0v3uaXTrplZrvNbNTMRmcUz38FoHXqDfs3JV0uaYekcUlfL7uju+9x9xF3H6kpPvkAQOvUFXZ3P+Luc+4+L+lbknY2ty0AzVZX2M1s8VjP5yUdKLsvgO5QOc5uZo9IulHSOjMbk/RVSTea2Q5JLumQpC8v69GswbnEWzme7fVvu2/LxWH97FUbw/rk1fGfN2d/Kx7L7glOva5NxePB0xfG255dVXGufa3iOgH95cc3eDDWLEkXXhzPQz5Qi18vkyfKDxKYm624BkFFb6q4LryfrTh+obd8/aOn4oMb1v/hteXFn/2ktFQZdne/Y4nF365aD0B34XBZIAnCDiRB2IEkCDuQBGEHkmjvKa7e2GWR+y67pLR29soN4bozK+Ohlumh+P+92cHy2tRl4aqVp5n2zMT1vtPxMJAHrU+vjrc9tyKuW9Vo6GB86rCdLX/eZ6bj53y6P37w40dWhfXa6vLDs6suY336ePALl1Qbitdfv+ZUWD9xpnz7V687Eq47tmF7aW2+Vv5aYc8OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0l01aWkT912fVz/7fIx256K8eBz6+K6B6ccSpIFlw7uma1Y91Q8Tj47FK9/bmPF6bfR5oNTTCWp93j8EojG8CWpd2X8xPf0lD/+TMXlls+ejk/97T0ZHzsxsL7+YzqqzByPp1WemI+fuGicf03/2XDdw8FxGRa8lNizA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASbR1nn187pKk/+1RpffbPPwjXP/XmRaW1FUfi/7dq8enF8p54LDy6XLP3Vlx2uKJcqxiHn6/F/zYLhtJnKi4FXdVb1fnulTNh95WvP7zhZLju1RdNxBu/Ii6vrp0rrfVZxbELW+Lye+dWh/UNA/ELbnL6gtLa4TMXhusOHj5dWuuZLv+FsGcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSTaOs7eO3Vea/77YGn9jZ3bwvU3XPN+ae3SPzhWd1+SdG42Prf6yJmVpbWjx+Lrl88e7w/rtYrzsucrpkX2YKzch2fCdXdseyesr18RjxdvGzwa1ueCE+IfWPeLcN2/+6D8+uiS9NSRq8P61678z9LacG98rvycVxyfUOGMx8/7D8+Uz4Hw1rl4iu//WbO5tOZ95c935Z7dzLaY2TNm9rqZvWpmXymWD5vZ02b2ZnG7tmpbADpnOW/jZyXd5+5XS/qUpLvN7BpJ90va5+7bJe0rfgbQpSrD7u7j7v5i8f2UpNclbZa0S9Le4m57Jd3aoh4BNMEn+oDOzC6TdJ2k5yRtdPdxaeE/BElLTrZmZrvNbNTMRqfn42trAWidZYfdzFZK+r6ke909PoNhEXff4+4j7j7S3xNPlgegdZYVdjOraSHo33X3HxSLj5jZpqK+SVLFKUoAOsm8YojBzEwLf5NPuvu9i5Z/TdIH7v6Qmd0vadjd/yra1mob9uvtpsa7XkLv2ngw4ORNV4b1Y1fGw199O8uH9i4fjoefLhmKhwU3D8T1XlVMuxycpzozH4+uvnZqU1j/6cGtYX3tM/Elldc/ur+0Nn+6/FTNZpjfV36e6qfXvxGuu3+qfHhLkt47HZ/i+sHp8lNYJWl2NprKOv6dXXl3+fD1T08+rhOz7y/5gljOOPsNkr4k6RUze7lY9oCkhyR9z8zukvSOpNuWsS0AHVIZdnf/scovcdCa3TSApuNwWSAJwg4kQdiBJAg7kARhB5KoHGdvplaOswOQnvN9OumTS46esWcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkKsNuZlvM7Bkze93MXjWzrxTLHzSzd83s5eLrlta3C6Bey5mffVbSfe7+opmtkvSCmT1d1L7h7v/QuvYANMty5mcflzRefD9lZq9L2tzqxgA01yf6m93MLpN0naTnikX3mNl+M3vYzNaWrLPbzEbNbHRG5xvrFkDdlh12M1sp6fuS7nX3k5K+KelySTu0sOf/+lLrufsedx9x95GaBhrvGEBdlhV2M6tpIejfdfcfSJK7H3H3OXefl/QtSTtb1yaARi3n03iT9G1Jr7v7Py5avmnR3T4v6UDz2wPQLMv5NP4GSV+S9IqZvVwse0DSHWa2Q5JLOiTpyy3oD0CTLOfT+B9LWmq+5yeb3w6AVuEIOiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBLm7u17MLP3Jf1y0aJ1ko62rYFPplt769a+JHqrVzN7u9Td1y9VaGvYP/bgZqPuPtKxBgLd2lu39iXRW73a1Rtv44EkCDuQRKfDvqfDjx/p1t66tS+J3urVlt46+jc7gPbp9J4dQJsQdiCJjoTdzG42s1+Y2Vtmdn8neihjZofM7JViGurRDvfysJlNmNmBRcuGzexpM3uzuF1yjr0O9dYV03gH04x39Lnr9PTnbf+b3cx6Jb0h6U8ljUl6XtId7v5aWxspYWaHJI24e8cPwDCzP5F0StK/uvvvFsv+XtKkuz9U/Ee51t3/ukt6e1DSqU5P413MVrRp8TTjkm6V9Bfq4HMX9PVFteF568Sefaekt9z9oLtPS3pU0q4O9NH13P1ZSZMfWbxL0t7i+71aeLG0XUlvXcHdx939xeL7KUkfTjPe0ecu6KstOhH2zZJ+tejnMXXXfO8u6Skze8HMdne6mSVsdPdxaeHFI2lDh/v5qMppvNvpI9OMd81zV8/0543qRNiXmkqqm8b/bnD335P0OUl3F29XsTzLmsa7XZaYZrwr1Dv9eaM6EfYxSVsW/XyxpMMd6GNJ7n64uJ2Q9Ji6byrqIx/OoFvcTnS4n//TTdN4LzXNuLrguevk9OedCPvzkrab2VYz65d0u6QnOtDHx5jZUPHBicxsSNJn1X1TUT8h6c7i+zslPd7BXn5Nt0zjXTbNuDr83HV8+nN3b/uXpFu08In825L+phM9lPS1TdLPiq9XO92bpEe08LZuRgvviO6SdJGkfZLeLG6Hu6i3f5P0iqT9WgjWpg719sda+NNwv6SXi69bOv3cBX215XnjcFkgCY6gA5Ig7EAShB1IgrADSRB2IAnCDiRB2IEk/hc7XfypYQ/4nQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "imshow(X_test[0,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "alive-plate",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f25c01c9940>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAATw0lEQVR4nO3de3Bc5XkG8Ofd1UqyZRlZli8Ci4uNzSWEGKqaACkDoSUOnXLplHJpUujQ2O1AIU2mhSHpmH86pZ2QhDQNVFyC06H2ME0INGMo1JPWIW2MBTXGxuAbN1tGFhhfZHmt1e7bP3RgFNB5P3nP7p6F9/nNeCTvu2f309qPzkrv+b5PVBVE9MmXSXsARFQbDDuREww7kRMMO5ETDDuREw21fLJGadJmtNTyKT8RpCFr1outzbG1zHuHKj2co9M6Ob5WLNnHDuUrOxYH8jiEYT0i49UShV1EFgO4B0AWwAOqepd1/2a04By5OMlTupRtazfrBy6aH1tr+be1lR7OUSn+5tmxtYYDR8xj9flNlR7OJ95aXR1bK/ttvIhkAfwTgC8COB3AtSJyermPR0TVleRn9kUAtqnqDlUdBrASwOWVGRYRVVqSsB8H4K0xf98Z3fZrRGSJiPSKSG8B9ts2IqqeJGEf75cAH7n2VlV7VLVbVbtzaErwdESURJKw7wTQNebvcwD0JRsOEVVLkrCvAzBfRE4SkUYA1wB4ojLDIqJKK7v1pqojInIzgP/AaOvtIVV12SvJtNjXDmz/mzPN+o2/+59m/YxJr5j1c5r+PbbW9y27R39mY3yPvhLeKf4yttZftM81ebXHfsur15j10vKZsbWpK35lHvtJlKjPrqqrAKyq0FiIqIp4uSyREww7kRMMO5ETDDuREww7kRMMO5ETUsvVZadKu35cp7huuW9RbG3V4u+ax87N5cx6f9GeM/B20b7M+GApvlc+OztoHntMpmjWG2XcqdEf2BeYkt430hpby8mIeWx7xp7PPttuw6NJ4jvLt+66yDz2zXNSXgegTGt1NQ7o3nH/0XhmJ3KCYSdygmEncoJhJ3KCYSdygmEncqKmS0nXs123nWfWX7vsB7G1NXljuWQAbx22W28lTDHrGdj9ralGi2qgaE+/HbA7byiOuyDRmLra54uWTPlLkQ2U7Nf1jRG7JZnX+Nf9+3P+yzz2stVXmnVcvNOu1yGe2YmcYNiJnGDYiZxg2ImcYNiJnGDYiZxg2ImcYJ898sDSfzTr2wuHY2sFPcY8tjlTMOsXJFzNedPwcGxtuGTPAx0q2b3qroZ9Zn1G1r4GYP2Rtthao9hNfqtPDgDtgem72Y9uUPSBZ/OTzGN/cPJKs37LnKvN+sjOXWY9DTyzEznBsBM5wbATOcGwEznBsBM5wbATOcGwEznBPnvklJw973qv0U7OBfrFoT76vNV/Ytbn9tjH/2xl/B12BebSL55sf92vFeyv7aeDC8z6+ZO2x9b2BXr8F06ye/hPD9nz3QeKU2Nr8xvfNo+dlbWjcfj0TrOeq8M+e6Kwi8jrAA4CKAIYUdXuSgyKiCqvEmf2i1T1nQo8DhFVEX9mJ3IiadgVwNMi8ryILBnvDiKyRER6RaS3gPLXIyOiZJK+jT9fVftEZCaAZ0TkFVVdM/YOqtoDoAcY3est4fMRUZkSndlVtS/6uAfAYwDidz8kolSVHXYRaRGR1vc/B3AJgI2VGhgRVVaSt/GzADwmo1v6NgD4V1V9qiKjSsG0bKBnW4rfwjcbWNc99D31lK/Za5AXBwbMepPE99JnNxw0j/3jNy4x6/3nHjDrIYWX4+fT39T2lnnspZ/+vFnfetspdv1L98bWngv8+ign9joAfZ+zr1844Wn78dNQdthVdQeAz1RwLERURWy9ETnBsBM5wbATOcGwEznBsBM54WaKa6Y52XrNBWNr4nZjy+RRdlvvyAp7WeOG3w48vOHMRvvrDrXWtt7zWbOeO2hv6fzTpfGvzcoZjeaxkxbYr+u8FYG24JfiS42Bdmle7Xru0/vt565DPLMTOcGwEznBsBM5wbATOcGwEznBsBM5wbATOeGmzy7zTgjc41dm1eqzz8raWzKHnNvxmllfB3u6paV72Z+b9en4X7O+4GF7imzmUOAag4b4sWd+8X/2oXNPNOu6P9n02yQuPn6LWd9co3EcDZ7ZiZxg2ImcYNiJnGDYiZxg2ImcYNiJnGDYiZxw02fPd06p2mO3ZuyXcbBk96IvmfqSWV+X+Y2jHtP7Zj1lL9c8Ejj+hpWrzPo1re+Z9fVH4tds/trSm8xjH37gu2b97/ZcZNbfHBmMrYWWih4q2VtV/1ZrqM8+16yngWd2IicYdiInGHYiJxh2IicYdiInGHYiJxh2Iifc9NkPdtlrlIdkRMs+tq9o92wvCCxp/7eBnu8Xjl0YW5PuNvPYN+6eZtZ/aO+KjB/CXifgypfjt5t+9zT73+RPz7varL/6l11m/XvXroutbRi2r33YV7LPg1+YvMes93wc++wi8pCI7BGRjWNuaxeRZ0Rka/TR/h9DRKmbyNv4hwEs/tBttwNYrarzAayO/k5EdSwYdlVdA2Dvh26+HMDy6PPlAK6o7LCIqNLK/QXdLFXdDQDRx5lxdxSRJSLSKyK9BcRfJ01E1VX138arao+qdqtqdw5N1X46IopRbtj7RaQTAKKP9q8miSh15Yb9CQDXR59fD+DxygyHiKol2GcXkRUALgTQISI7ASwDcBeAR0XkRgBvAriqmoOshPwMex/xEGvd+KbA3OjJYs8at+ZdA8DW759j1rUh/hqAr5z33+axT3W8atb/6oWzzPqJze+Y9T9r2xVbO/WW+8xj//5+e2/4Y88o/9qJZrGvXbD+vQFgSiZwcUQdCoZdVa+NKV1c4bEQURXxclkiJxh2IicYdiInGHYiJxh2IifcTHE9PKuU6PiCxrfXQssSt4j9PfXVgn1l4Y7f/2ezbtlSOGTWf5mfZNb/ouMXZT83AKzJxy/hvajJnmb65Lb/SfTcRY3/N28OTFkulD+jGQAgDXa0dCS0iHfl8cxO5ATDTuQEw07kBMNO5ATDTuQEw07kBMNO5ISbPnupY7hqj72/dNis/9G2PzDr98171Kw/NTTdrOc1F1try9jfzydn7KXCdhSmmvWQ1kx8L/3ZfIt57PSsfY3A9sIMs74l3xlb+2bHK+ax1lbTEyGfmm/W9cXNiR6/HDyzEznBsBM5wbATOcGwEznBsBM5wbATOcGwEznhps8+5Ri7Fx5yQkP88U8esrcO7l9pb2t8/LL4Od8A0DcyZNYtucCSyVkEJm4H+vAhRcQv4d0SeOz2jH1txKGG/Wb9jqfjFkYGvnmd3WdPKj/bvoag8cWqPv24eGYncoJhJ3KCYSdygmEncoJhJ3KCYSdygmEncsJNn33OMXZP1lpjHAA6G+J74esGTzKPbX4v2SLkB0r29sBWvzpj9LlroWRsfdwc2Mo6tNJ/mzFXHgBmrjOK19mPbV0fAAB7ivZce82k+7qPJ3hmF5GHRGSPiGwcc9udIrJLRNZHfy6t7jCJKKmJvI1/GMDicW7/jqoujP6squywiKjSgmFX1TUA9tZgLERURUl+QXeziGyI3uZPi7uTiCwRkV4R6S0g2XXWRFS+csN+L4B5ABYC2A3g7rg7qmqPqnarancO9gaGRFQ9ZYVdVftVtaiqJQD3A1hU2WERUaWVFXYRGbtG75UANsbdl4jqQ7DPLiIrAFwIoENEdgJYBuBCEVkIQAG8DmBp9YZYGXOnvGvW3wus/d6RjZ+fvCvfZh6799Rk1y4Nqf3jz1TY/WZLqJ+cVEbiu+Wh5w7VT8vFr5cPAIEt2E2hef65wNgOz7CjlcYPtMGwq+p4KwA8WIWxEFEV8XJZIicYdiInGHYiJxh2IicYdiIn3ExxbcoUzHpoOqVl3Q57qejSSQmXYzamiQL2ctGh9lVwKemErOdvDixzvbdoT+1dkMua9cm7y3/dmwJjy0io9WbX2452QBXAMzuREww7kRMMO5ETDDuREww7kRMMO5ETDDuRE2767JOydp89r+X3mxu3TTLr0899u+zHBsJbG1tCffRQPekUWOvxc4GrGw5pY+DR7V54447+2NpTQ/Yk07Ob7KWiEXhdCvaOzangmZ3ICYadyAmGncgJhp3ICYadyAmGncgJhp3ICTd99r2Bxmdey+8nG6slAwCu7nrerA+W7KWgc2LP205TLvDFl4zXtRA41+TVXio61GcfOuPY2Nqag6eYx17Q3GvW95eGzXpxcnXXCSgHz+xETjDsRE4w7EROMOxETjDsRE4w7EROMOxETrjpsx8u2j3b5gT7+5Zy9rFnT3rNrPcV7X5xs9hz8aspNJ891Am3FALr4Sf9ut+4LP76hPzb881jl820r42w/8WAQlvoHrUXPLOLSJeI/FxENovIJhG5Nbq9XUSeEZGt0cdp1R8uEZVrIm/jRwB8XVVPA/BZADeJyOkAbgewWlXnA1gd/Z2I6lQw7Kq6W1VfiD4/CGAzgOMAXA5geXS35QCuqNIYiagCjuoXdCJyIoCzAKwFMEtVdwOj3xAAzIw5ZomI9IpIbwHJ9jwjovJNOOwiMgXAjwF8VVUPTPQ4Ve1R1W5V7c7BXuSPiKpnQmEXkRxGg/6Iqv4kurlfRDqjeieAPdUZIhFVQrD1JiIC4EEAm1X122NKTwC4HsBd0cfHqzLCCjlStL/Ujkxo2eJ4pflDZr0tsBR0aGvilkALatj4np10S+akS1GXEixFHW692eeqtq59sbWBTTPMY5s+YzcVS6EfSRuSbAJeHRPps58P4MsAXhKR9dFtd2A05I+KyI0A3gRwVVVGSEQVEQy7qj6L+BXxL67scIioWni5LJETDDuREww7kRMMO5ETDDuRE26muA6O2FfvZaX8fvD0tkGzPitr91z3lezntvroIQW1l6EOdbJDU1xD9ZIxjTUTWIY61MPfUrC3Vf7GqU/G1v56+3XmsSHFwOUL2UkfwymuRPTJwLATOcGwEznBsBM5wbATOcGwEznBsBM54abPfnjEnp/cX7TnJx/fEH980/fa7ce+1/6eOjtrz4fPB3rlpsDlA+E+uV3PhJbglvh+c7NRA8Jf97yGSWZ96ZaLYmsn/ixwhcHVdjkfWAa7ITdiP0AKeGYncoJhJ3KCYSdygmEncoJhJ3KCYSdygmEncsJNn316sz33OR/oJw+W8rG1UqN97Lr8CWb9hqn2/hqPHJxu1nNSvZ5u4nXnjTnrw4E++lDJXoPgzEb7ddv1Tlts7eS37TUIQo4Exr7wuF1m/b1Ez14entmJnGDYiZxg2ImcYNiJnGDYiZxg2ImcYNiJnJjI/uxdAH4EYDaAEoAeVb1HRO4E8BUAA9Fd71DVVdUaaFLP9S4w661ddj95oBjfy27d0G8eu+LUY+067DqNL/S6nYQXY2t65qnmsa8V7D58R2CJgbUvnmzWF+A5+wGqYCIX1YwA+LqqviAirQCeF5Fnotp3VPVb1RseEVXKRPZn3w1gd/T5QRHZDOC4ag+MiCrrqH5mF5ETAZwFYG10080iskFEHhKRaTHHLBGRXhHpLcBe+omIqmfCYReRKQB+DOCrqnoAwL0A5gFYiNEz/93jHaeqPararardOdjXOhNR9Uwo7CKSw2jQH1HVnwCAqvaralFVSwDuB7CoesMkoqSCYRcRAfAggM2q+u0xt3eOuduVADZWfnhEVCkT+W38+QC+DOAlEVkf3XYHgGtFZCEABfA6gKVVGF/FzOi1p6F2XjXFrO8vHY4vluyth6n+aKP9X789a/fWjsnYy1g3DCZY/rtKJvLb+Gcx/urjddtTJ6KP4hV0RE4w7EROMOxETjDsRE4w7EROMOxETrhZSrr1Lfu6/GUDnzLr7w7H9+F1/4GyxvQ+yTWadR0JbC8sPr9nS8a+dkJHjCW2179iHvt7m64z63Om7DPrs56rv2svfP4vIXKIYSdygmEncoJhJ3KCYSdygmEncoJhJ3JCVJNtyXtUTyYyAOCNMTd1AHinZgM4OvU6tnodF8CxlauSYztBVWeMV6hp2D/y5CK9qtqd2gAM9Tq2eh0XwLGVq1Zj49t4IicYdiIn0g57T8rPb6nXsdXruACOrVw1GVuqP7MTUe2kfWYnohph2ImcSCXsIrJYRF4VkW0icnsaY4gjIq+LyEsisl5EelMey0MiskdENo65rV1EnhGRrdHHcffYS2lsd4rIrui1Wy8il6Y0ti4R+bmIbBaRTSJya3R7qq+dMa6avG41/5ldRLIAtgD4HQA7AawDcK2qvlzTgcQQkdcBdKtq6hdgiMgFAAYB/EhVz4hu+wcAe1X1rugb5TRVva1OxnYngMG0t/GOdivqHLvNOIArANyAFF87Y1x/iBq8bmmc2RcB2KaqO1R1GMBKAJenMI66p6prAOz90M2XA1gefb4co/9Zai5mbHVBVXer6gvR5wcBvL/NeKqvnTGumkgj7McBeGvM33eivvZ7VwBPi8jzIrIk7cGMY5aq7gZG//MAmJnyeD4suI13LX1om/G6ee3K2f48qTTCPt7CYfXU/ztfVc8G8EUAN0VvV2liJrSNd62Ms814XSh3+/Ok0gj7TgBdY/4+B0BfCuMYl6r2RR/3AHgM9bcVdf/7O+hGH/ekPJ4P1NM23uNtM446eO3S3P48jbCvAzBfRE4SkUYA1wB4IoVxfISItES/OIGItAC4BPW3FfUTAK6PPr8ewOMpjuXX1Ms23nHbjCPl1y717c9VteZ/AFyK0d/IbwfwjTTGEDOuuQBejP5sSntsAFZg9G1dAaPviG4EMB3AagBbo4/tdTS2fwHwEoANGA1WZ0pj+xxGfzTcAGB99OfStF87Y1w1ed14uSyRE7yCjsgJhp3ICYadyAmGncgJhp3ICYadyAmGnciJ/weJz8Y26iF93AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "imshow(X_test[1,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "complicated-musical",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f24f77b6fd0>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQV0lEQVR4nO3dbYxc9XUG8OeZ2Te8trG9xmYxNlDiOLUaatqVobhBRDSI+IuhbSqslroSqtMKJJCiCkQrQdUvqGqSIrWN5BQrzgugpAnCrRyC69BSaEBeU+OXOInBtROzi20w4PXauzu7c/phL83a7D13du682ef5SavZvWfu3MPFz9yZ+c+9f5oZROTiV2h2AyLSGAq7SBAKu0gQCrtIEAq7SBBtjdxYBzutC92N3ORFoTzf32e8rJRaGzvb7j94W9l/7DH/eGBZh4uiM9qTMRDU0THu1nlwLGPj8YxgGGM2yulqucJO8nYAjwMoAvhnM3vMu38XunEDb82zyQsTp933v5Qx/Dn8Oze49c4/G0ytHd53hbtuYdGIX//fS9z6eLffu81LfyKykv9McdVVJ9x6522H3XpEr9qO1FrVL+NJFgH8I4DPAlgJYD3JldU+nojUV5737KsBvGFmh8xsDMDTANbVpi0RqbU8YV8C4BdT/j6aLDsHyY0k+0n2lzCaY3MikkeesE/3RvQjb+DMbJOZ9ZlZXzs6c2xORPLIE/ajAJZO+ftKAAP52hGReskT9p0AlpO8hmQHgLsAbK1NWyJSa1UPvZnZOMn7APwAk0Nvm81sf806u5gw4znVJtzydQ++7tb/ackr6cWc4yNvrjnt1nuLHW59ViG9Pjie8dhts936DXf/uVuf940fufVoco2zm9k2ANtq1IuI1JG+LisShMIuEoTCLhKEwi4ShMIuEoTCLhJEQ89nD6vsj6NneWjxv7v1PWPp/xt3nr3aXXdp+7tuvavgj3XvGr3UrZ8pp39FuoCF7rp/PPcdt/7+CreMeX45HB3ZRYJQ2EWCUNhFglDYRYJQ2EWCUNhFgtDQ2wVgWcapnidG0y+pvLzzbXfdDvjDgu+W/ctYdzH96rEA0NOefhrruxP+f1eWsSW6lPRM6MguEoTCLhKEwi4ShMIuEoTCLhKEwi4ShMIuEoTG2VtA29XLMu6x260OlbtSaxPTTtzzSx30x9mzxtGHzZ/lp2Tp/8TKGfM9v1nyLzW9YOGQW5dz6cguEoTCLhKEwi4ShMIuEoTCLhKEwi4ShMIuEoTG2VvAB329udY/5YyzX972gbvuiLXnqmeN0xdQTq11Ffwx/Hedy1ADwLXz/ctg+//l8eQKO8nDAIYATAAYN7O+WjQlIrVXiyP7p83Mv5q/iDSd3rOLBJE37AbgeZK7SG6c7g4kN5LsJ9lfwmjOzYlItfK+jF9jZgMkFwHYTvInZvbi1DuY2SYAmwBgLhdYzu2JSJVyHdnNbCC5PQ7gGQCra9GUiNRe1WEn2U1yzoe/A7gNwL5aNSYitZXnZfxiAM+Q/PBxnjSz52rSVTDvXOc/535QPuvWT4xfnlpb0va+u25PwX/s5W3+OeWvj/W49bJzPPHG4AGgp+B/xnPirH/d+Q744/DRVB12MzsE4Ndr2IuI1JGG3kSCUNhFglDYRYJQ2EWCUNhFgtApri2g+3p/iKhk/hDVkvb3UmvD1uGuu6J9xK0/cuxmt/5Xi15y63tLs1JrIxlTNvcW/d6PDPjDfstxxK1HoyO7SBAKu0gQCrtIEAq7SBAKu0gQCrtIEAq7SBAaZ28Bv3fV6259qOxf4GfMiqm1lRmnqP7w7CK3vu83/TH++QPp4+gA0FFKv9R0O8fddWcV/HF2vufX5Vw6sosEobCLBKGwiwShsIsEobCLBKGwiwShsIsEoXH2FrCia9Ctn3HG0QGgZOn/G5e1+eeMr+2/060vwX63nqXLGUsfKWeNk/vn2pc7/O8AyLl0ZBcJQmEXCUJhFwlCYRcJQmEXCUJhFwlCYRcJQuPsLeCmrgG3PjDhj0dPgFVve8535lS9LgC8N3HGrX+yoyu1tmvEPxceOOWXL0k/V14+KvPITnIzyeMk901ZtoDkdpIHk9v59W1TRPKq5GX81wDcft6yhwDsMLPlAHYkf4tIC8sMu5m9CODkeYvXAdiS/L4FwB21bUtEaq3aD+gWm9kgACS3qRcyI7mRZD/J/hJGq9yciORV90/jzWyTmfWZWV87Ouu9ORFJUW3Yj5HsBYDk9njtWhKReqg27FsBbEh+3wDg2dq0IyL1kjnOTvIpALcAWEjyKIBHADwG4Nsk7wHwcwCfq2eTF7vejHPOj4z748ndheo/C5n37B63nnXG+P1Hzx+oOdfjVz6XWusqlDIe3Vc82Z5r/Wgyw25m61NKt9a4FxGpI31dViQIhV0kCIVdJAiFXSQIhV0kCJ3iehGYU0i/5PKZ8pi7bvmMf4pqlv63lrn1zqXp/8SKmQN7vvZTOlbNhPaWSBAKu0gQCrtIEAq7SBAKu0gQCrtIEAq7SBAaZ78AZF0qei7TT3H95tA1tW7nHCMD3W69nenTTU/oWNNQ2tsiQSjsIkEo7CJBKOwiQSjsIkEo7CJBKOwiQWic/QIwXPZn0lnakX5O+pYjN7rrzsahqnr60LLv++ekn/nd9PPp2zmea9syMzqyiwShsIsEobCLBKGwiwShsIsEobCLBKGwiwShcfYLQAf9KZu9Z+yBIz3uuh/POc4+6+WfuvVLC5ek1uY617uvRFu+S96Hk3lkJ7mZ5HGS+6Yse5TkWyR3Jz9r69umiORVycv4rwG4fZrlXzazVcnPttq2JSK1lhl2M3sRwMkG9CIidZTnA7r7SO5JXubPT7sTyY0k+0n2l5B+rTQRqa9qw/4VANcCWAVgEMAX0+5oZpvMrM/M+trhn9AhIvVTVdjN7JiZTZhZGcBXAayubVsiUmtVhZ1k75Q/7wSwL+2+ItIaMsfZST4F4BYAC0keBfAIgFtIrgJgAA4D+Hz9Wrz4PXfGf3tzRdsHbr1k6bXOt9uraaliNubP/+7pYinXttuGc60eTmbYzWz9NIufqEMvIlJH+rqsSBAKu0gQCrtIEAq7SBAKu0gQOsW1Bbx0+uNu/Q/nverWu5wZncc/draalipWHqn+NNURyxoW9L9ePT6r6k2HpCO7SBAKu0gQCrtIEAq7SBAKu0gQCrtIEAq7SBAaZ28BT+/vc+v3fupHbv1kuZhaW7vCv9SAfyHo+lpQPJ1xD38cvqirnM2IjuwiQSjsIkEo7CJBKOwiQSjsIkEo7CJBKOwiQWicvQXMeTl9WmMA6LrZf04eKnek1v568X+6696Fm9x6XqOWfrnoroypqLPG2VmuoqHAdGQXCUJhFwlCYRcJQmEXCUJhFwlCYRcJQmEXCULj7C2g9z/ecesnHnTmZAYwbOnj7P892l1VT7VyqJQ+zl6Ec8H7CpgOVTOSubtILiX5AskDJPeTvD9ZvoDkdpIHk9v59W9XRKpVyXPjOIAvmNmvArgRwL0kVwJ4CMAOM1sOYEfyt4i0qMywm9mgmb2W/D4E4ACAJQDWAdiS3G0LgDvq1KOI1MCM3vWQvBrA9QBeBbDYzAaByScEAItS1tlIsp9kfylj7i4RqZ+Kw05yNoDvAnjAzE5Vup6ZbTKzPjPra0dnNT2KSA1UFHaS7ZgM+rfM7HvJ4mMke5N6L4Dj9WlRRGohc+iNJAE8AeCAmX1pSmkrgA0AHktun61LhwFM/Phnbv1gqcet9xSGU2uXFdNrAFC47hNuvbznJ249y5AzLXM3x3M9tqVfQVumUck4+xoAdwPYS3J3suxhTIb82yTvAfBzAJ+rS4ciUhOZYTezl4DUbz/cWtt2RKRe9B0kkSAUdpEgFHaRIBR2kSAUdpEgdIrrBcAbRweALme8ekHBH8s+teJStz57j1vO9MLplam135/7P+66e8ZG3LrG2WdGR3aRIBR2kSAUdpEgFHaRIBR2kSAUdpEgFHaRIDTO3gjMuGSy+ZeK/qNX7nHr29f8Q2otayj67Zv83j72nYwHyPDW6Lyq1y3C3y+d7/l1OZeO7CJBKOwiQSjsIkEo7CJBKOwiQSjsIkEo7CJBaJy9EZjxnGoTbvmyf+ty692fSh8rHyr7Y9H3fuZ5t/4DzHXrWS4ppk/ZPJExZXNWvTiqcfaZ0JFdJAiFXSQIhV0kCIVdJAiFXSQIhV0kCIVdJIhK5mdfCuDrAC4HUAawycweJ/kogD8FcCK568Nmtq1ejV7IWPTPKreyP84+98lX3Prev0kfC+8pnHHXLdX54utb3/hkau0vbnzZXffYhD+OPtzrH6v8K+LHU8mXasYBfMHMXiM5B8AuktuT2pfN7O/q156I1Eol87MPAhhMfh8ieQDAkno3JiK1NaP37CSvBnA9gFeTRfeR3ENyM8n5KetsJNlPsr+E0XzdikjVKg47ydkAvgvgATM7BeArAK4FsAqTR/4vTreemW0ysz4z62tHZ/6ORaQqFYWdZDsmg/4tM/seAJjZMTObMLMygK8CWF2/NkUkr8ywkySAJwAcMLMvTVneO+VudwLYV/v2RKRWKvk0fg2AuwHsJbk7WfYwgPUkVwEwAIcBfL4O/V0UbDz9NM9a+Nf3r0+t/X1vv7vulW273fr31z7g1ju37XTrxWI5tbaw2O2uO6fg77fRHp3iOhOVfBr/EjDticUaUxe5gOgbdCJBKOwiQSjsIkEo7CJBKOwiQSjsIkHoUtKNkDElc14/fDL9y4srf+sT7rrz/mW2W5+zzT+9NsulT6U//qfnrHPXPTk8y61f8V/jVfUUlY7sIkEo7CJBKOwiQSjsIkEo7CJBKOwiQSjsIkHQ6jwGfM7GyBMAjkxZtBDAOw1rYGZatbdW7QtQb9WqZW9Xmdll0xUaGvaPbJzsN7O+pjXgaNXeWrUvQL1Vq1G96WW8SBAKu0gQzQ77piZv39OqvbVqX4B6q1ZDemvqe3YRaZxmH9lFpEEUdpEgmhJ2kreT/CnJN0g+1Iwe0pA8THIvyd0k/Yuu17+XzSSPk9w3ZdkCkttJHkxup51jr0m9PUryrWTf7Sa5tkm9LSX5AskDJPeTvD9Z3tR95/TVkP3W8PfsJIsAfgbgMwCOAtgJYL2Z/bihjaQgeRhAn5k1/QsYJG8GcBrA183s15JlfwvgpJk9ljxRzjezB1ukt0cBnG72NN7JbEW9U6cZB3AHgD9BE/ed09cfoAH7rRlH9tUA3jCzQ2Y2BuBpAP4lS4IysxcBnDxv8ToAW5Lft2DyH0vDpfTWEsxs0MxeS34fAvDhNONN3XdOXw3RjLAvAfCLKX8fRWvN924Anie5i+TGZjczjcVmNghM/uMBsKjJ/ZwvcxrvRjpvmvGW2XfVTH+eVzPCPt1UUq00/rfGzH4DwGcB3Ju8XJXKVDSNd6NMM814S6h2+vO8mhH2owCWTvn7SgADTehjWmY2kNweB/AMWm8q6mMfzqCb3B5vcj//r5Wm8Z5umnG0wL5r5vTnzQj7TgDLSV5DsgPAXQC2NqGPjyDZnXxwApLdAG5D601FvRXAhuT3DQCebWIv52iVabzTphlHk/dd06c/N7OG/wBYi8lP5N8E8JfN6CGlr18B8Hrys7/ZvQF4CpMv60qYfEV0D4AeADsAHExuF7RQb98AsBfAHkwGq7dJvf02Jt8a7gGwO/lZ2+x95/TVkP2mr8uKBKFv0IkEobCLBKGwiwShsIsEobCLBKGwiwShsIsE8X/zBbX+MErCIAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "imshow(X_test[2,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baking-saskatchewan",
   "metadata": {},
   "source": [
    "## Regresión con el API secuencial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "absolute-assembly",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dated-matthew",
   "metadata": {},
   "outputs": [],
   "source": [
    "housing = fetch_california_housing()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "stupid-queens",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    housing.data, housing.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "electrical-picking",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train_full,\n",
    "                                                     y_train_full)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "endless-brush",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_valid = scaler.transform(X_valid)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broken-return",
   "metadata": {},
   "source": [
    "Para regresión necesitamos solamente una neurona en la capa de salida (queremos predecir un sólo valor) sin función de activación, y la función de costo es el error cuadrado promedio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "different-indie",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"relu\", input_shape=X_train.shape[1:]),\n",
    "    keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "suspended-liability",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"mean_squared_error\", optimizer=\"sgd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "developed-fellow",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 1.0776 - val_loss: 10.2714\n",
      "Epoch 2/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.8995 - val_loss: 0.4316\n",
      "Epoch 3/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3954 - val_loss: 0.4190\n",
      "Epoch 4/20\n",
      " 22/363 [>.............................] - ETA: 0s - loss: 0.3971"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-109-484eb4c531ed>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m history = model.fit(X_train, y_train, epochs=20,\n\u001b[0;32m----> 2\u001b[0;31m                     validation_data=(X_valid, y_valid))\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36m_method_wrapper\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    106\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_method_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    107\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_multi_worker_mode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 108\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    109\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    110\u001b[0m     \u001b[0;31m# Running inside `run_distribute_coordinator` already.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m   1100\u001b[0m                 \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1101\u001b[0m               \u001b[0mlogs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtmp_logs\u001b[0m  \u001b[0;31m# No error, now safe to assign to logs.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1102\u001b[0;31m               \u001b[0mend_step\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep_increment\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1103\u001b[0m               \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mend_step\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlogs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1104\u001b[0m         \u001b[0mepoch_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlogs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py\u001b[0m in \u001b[0;36mstep_increment\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1210\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_steps_per_execution\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0massign\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_steps_per_execution_value\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1211\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1212\u001b[0;31m   \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1213\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mstep_increment\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1214\u001b[0m     \u001b[0;34m\"\"\"The number to increment the step for `on_batch_end` methods.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=20,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "russian-roberts",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "162/162 [==============================] - 0s 1ms/step - loss: 0.4103\n"
     ]
    }
   ],
   "source": [
    "mse_test = model.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "fifty-automation",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = X_test[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "super-salmon",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "international-conservative",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.7080083 ],\n",
       "       [3.6439445 ],\n",
       "       [0.47018892]], dtype=float32)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "communist-cache",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_new = y_test[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "multiple-degree",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.312, 3.477, 1.625])"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_new"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sealed-johnston",
   "metadata": {},
   "source": [
    "## Modelos complejos con el API funcional\n",
    "\n",
    "Un ejemplo de un modelo **no secuencial** es una red neuronal \"ancho y profundo\" (*wide & deep*).\n",
    "\n",
    "Modelos así conectan una parte de las entradas directamente con las salidas. Así la red puede aprender patrones profundas y reglas simples.\n",
    "\n",
    "![](figures_intro_rna/fig10-14.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "orange-accessory",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_ = keras.layers.Input(shape=X_train.shape[1:])\n",
    "hidden1 = keras.layers.Dense(30, activation=\"relu\")(input_)\n",
    "hidden2 = keras.layers.Dense(30, activation=\"relu\")(hidden1)\n",
    "concat = keras.layers.Concatenate()([input_, hidden2])\n",
    "output = keras.layers.Dense(1)(concat)\n",
    "model = keras.Model(inputs=[input_], outputs=[output])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "direct-interview",
   "metadata": {},
   "source": [
    "* `Input`: especifica el tipo y forma de entrada. Se puede tener varios entradas.\n",
    "\n",
    "* Después creamos una capa densa con $30$ neuronas y función de activación \"ReLU\". Pasamos la capa de salida como argumento a una función para conectar ambas capas.\n",
    "\n",
    "* Creamos otra capa densa, y pasamos la primera como argumento a la \"función\" para conectar las capas.\n",
    "\n",
    "* `Concatenate` es para combinar las salidas de las capas de entrada y la segunda capa oculta.\n",
    "\n",
    "* La capa de salida es una sola neurona, y pasamos como argumento la capa de concatenación.\n",
    "\n",
    "* Finalmente creamos el modelo indicando las capas de entrada y salida.\n",
    "\n",
    "Después tenemos que usar `compile()`, entrenar, evaluar, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "smoking-flooring",
   "metadata": {},
   "source": [
    "¿Qué hacemos si queremos usar una parte de los datos en la rama \"profunda\" de la red y otra parte en la rama \"superficial\"?\n",
    "\n",
    "![](figures_intro_rna/fig10-15.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "collectible-perspective",
   "metadata": {},
   "source": [
    "Podemos usar los *features* $0$ a $4$ en la entrada A, y los *features* $2$ a $7$ en la entrada B:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "ultimate-cleaners",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_A = keras.layers.Input(shape=[5], name=\"wide_input\")\n",
    "input_B = keras.layers.Input(shape=[6], name=\"deep_input\")\n",
    "hidden1 = keras.layers.Dense(30, activation=\"relu\")(input_B)\n",
    "hidden2 = keras.layers.Dense(30, activation=\"relu\")(hidden1)\n",
    "concat = keras.layers.Concatenate()([input_A, hidden2])\n",
    "output = keras.layers.Dense(1, name=\"output\")(concat)\n",
    "model = keras.Model(inputs=[input_A, input_B], outputs=[output])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "casual-hampton",
   "metadata": {},
   "source": [
    "Cuando usamos `fit()` hay que pasar un par de matrices como entradas `(X_train_A, X_train_B)`. También para validación y predicción."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "imposed-technique",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"mse\", optimizer=keras.optimizers.SGD(lr=1e-3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "elementary-bernard",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_A, X_train_B = X_train[:, :5], X_train[:, 2:]\n",
    "X_valid_A, X_valid_B = X_valid[:, :5], X_valid[:, 2:]\n",
    "X_test_A, X_test_B = X_test[:, :5], X_test[:, 2:]\n",
    "X_new_A, X_new_B = X_test_A[:3], X_test_B[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "egyptian-wisdom",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 2.1934 - val_loss: 1.1015\n",
      "Epoch 2/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.7900 - val_loss: 0.7402\n",
      "Epoch 3/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.6610 - val_loss: 0.6796\n",
      "Epoch 4/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.6164 - val_loss: 0.6452\n",
      "Epoch 5/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5868 - val_loss: 0.6334\n",
      "Epoch 6/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5653 - val_loss: 0.6031\n",
      "Epoch 7/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5469 - val_loss: 0.5820\n",
      "Epoch 8/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5314 - val_loss: 0.5617\n",
      "Epoch 9/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5171 - val_loss: 0.5554\n",
      "Epoch 10/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5069 - val_loss: 0.5410\n",
      "Epoch 11/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4967 - val_loss: 0.5373\n",
      "Epoch 12/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4899 - val_loss: 0.5207\n",
      "Epoch 13/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4829 - val_loss: 0.5096\n",
      "Epoch 14/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4768 - val_loss: 0.4969\n",
      "Epoch 15/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4719 - val_loss: 0.4945\n",
      "Epoch 16/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4672 - val_loss: 0.4907\n",
      "Epoch 17/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4628 - val_loss: 0.4846\n",
      "Epoch 18/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4590 - val_loss: 0.4817\n",
      "Epoch 19/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4556 - val_loss: 0.4758\n",
      "Epoch 20/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4524 - val_loss: 0.4675\n"
     ]
    }
   ],
   "source": [
    "history = model.fit((X_train_A, X_train_B), y_train, epochs=20,\n",
    "                    validation_data=((X_valid_A, X_valid_B), y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "resident-finance",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "162/162 [==============================] - 0s 1ms/step - loss: 0.4497\n"
     ]
    }
   ],
   "source": [
    "mse_test = model.evaluate((X_test_A, X_test_B), y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "measured-apparatus",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict((X_new_A, X_new_B))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "surprised-ferry",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.5746727],\n",
       "       [3.1722462],\n",
       "       [0.113276 ]], dtype=float32)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "friendly-stocks",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.312, 3.477, 1.625])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accessible-portsmouth",
   "metadata": {},
   "source": [
    "Otra forma de pasar las entradas es con un diccionario, que puede ayudar para organizar bien los datos, e.g. `{\"wide_input\": X_train_A, \"deep_input\": X_train_B}`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "binding-atlas",
   "metadata": {},
   "source": [
    "También podemos tener multiples salidas, por ejemplo para regularización de una parte de la red.\n",
    "\n",
    "![](figures_intro_rna/fig10-16.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "attractive-efficiency",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_A = keras.layers.Input(shape=[5], name=\"wide_input\")\n",
    "input_B = keras.layers.Input(shape=[6], name=\"deep_input\")\n",
    "hidden1 = keras.layers.Dense(30, activation=\"relu\")(input_B)\n",
    "hidden2 = keras.layers.Dense(30, activation=\"relu\")(hidden1)\n",
    "concat = keras.layers.Concatenate()([input_A, hidden2])\n",
    "output = keras.layers.Dense(1, name=\"main_output\")(concat)\n",
    "aux_output = keras.layers.Dense(1, name=\"aux_output\")(hidden2)\n",
    "model = keras.Model(inputs=[input_A, input_B], outputs=[output, aux_output])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "talented-prairie",
   "metadata": {},
   "source": [
    "Cada salida requiere su propia función de perdida, así que hay que pasar una lista de tales funciones en el momento de compilar el modelo.\n",
    "\n",
    "Por defecto el valor final del error de la red será simplemente la suma de las funciones de costo, pero podemos asociar un peso a cada función:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "attractive-antique",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=[\"mse\", \"mse\"], loss_weights=[0.9, 0.1], optimizer=\"sgd\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "extraordinary-notification",
   "metadata": {},
   "source": [
    "Ahora tenemos que también dar las etiquetas para cada salida. En este caso queremos que cada parte de la red prediga la misma cosa, así que usamos `y_train` para cada salida:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "comparable-electric",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 1.0194 - main_output_loss: 0.9350 - aux_output_loss: 1.7786 - val_loss: 0.5931 - val_main_output_loss: 0.5219 - val_aux_output_loss: 1.2340\n",
      "Epoch 2/20\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.5163 - main_output_loss: 0.4666 - aux_output_loss: 0.9635 - val_loss: 0.5078 - val_main_output_loss: 0.4579 - val_aux_output_loss: 0.9570\n",
      "Epoch 3/20\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.4630 - main_output_loss: 0.4274 - aux_output_loss: 0.7834 - val_loss: 0.4718 - val_main_output_loss: 0.4351 - val_aux_output_loss: 0.8027\n",
      "Epoch 4/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4580 - main_output_loss: 0.4329 - aux_output_loss: 0.6833 - val_loss: 0.4518 - val_main_output_loss: 0.4235 - val_aux_output_loss: 0.7061\n",
      "Epoch 5/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4247 - main_output_loss: 0.4030 - aux_output_loss: 0.6203 - val_loss: 0.4360 - val_main_output_loss: 0.4127 - val_aux_output_loss: 0.6459\n",
      "Epoch 6/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4164 - main_output_loss: 0.3978 - aux_output_loss: 0.5836 - val_loss: 0.4267 - val_main_output_loss: 0.4066 - val_aux_output_loss: 0.6083\n",
      "Epoch 7/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4082 - main_output_loss: 0.3912 - aux_output_loss: 0.5613 - val_loss: 0.4237 - val_main_output_loss: 0.4062 - val_aux_output_loss: 0.5816\n",
      "Epoch 8/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4032 - main_output_loss: 0.3875 - aux_output_loss: 0.5442 - val_loss: 0.4130 - val_main_output_loss: 0.3966 - val_aux_output_loss: 0.5608\n",
      "Epoch 9/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3929 - main_output_loss: 0.3779 - aux_output_loss: 0.5277 - val_loss: 0.4012 - val_main_output_loss: 0.3856 - val_aux_output_loss: 0.5415\n",
      "Epoch 10/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3876 - main_output_loss: 0.3732 - aux_output_loss: 0.5175 - val_loss: 0.3962 - val_main_output_loss: 0.3818 - val_aux_output_loss: 0.5259\n",
      "Epoch 11/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3797 - main_output_loss: 0.3660 - aux_output_loss: 0.5033 - val_loss: 0.3929 - val_main_output_loss: 0.3792 - val_aux_output_loss: 0.5162\n",
      "Epoch 12/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3741 - main_output_loss: 0.3611 - aux_output_loss: 0.4910 - val_loss: 0.3861 - val_main_output_loss: 0.3728 - val_aux_output_loss: 0.5057\n",
      "Epoch 13/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3705 - main_output_loss: 0.3578 - aux_output_loss: 0.4856 - val_loss: 0.3862 - val_main_output_loss: 0.3738 - val_aux_output_loss: 0.4979\n",
      "Epoch 14/20\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.3669 - main_output_loss: 0.3550 - aux_output_loss: 0.4745 - val_loss: 0.3902 - val_main_output_loss: 0.3780 - val_aux_output_loss: 0.4995\n",
      "Epoch 15/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3644 - main_output_loss: 0.3529 - aux_output_loss: 0.4682 - val_loss: 0.3776 - val_main_output_loss: 0.3655 - val_aux_output_loss: 0.4867\n",
      "Epoch 16/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3574 - main_output_loss: 0.3460 - aux_output_loss: 0.4605 - val_loss: 0.3805 - val_main_output_loss: 0.3690 - val_aux_output_loss: 0.4841\n",
      "Epoch 17/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3573 - main_output_loss: 0.3464 - aux_output_loss: 0.4557 - val_loss: 0.3747 - val_main_output_loss: 0.3636 - val_aux_output_loss: 0.4747\n",
      "Epoch 18/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3510 - main_output_loss: 0.3402 - aux_output_loss: 0.4481 - val_loss: 0.3723 - val_main_output_loss: 0.3618 - val_aux_output_loss: 0.4667\n",
      "Epoch 19/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3507 - main_output_loss: 0.3403 - aux_output_loss: 0.4445 - val_loss: 0.3638 - val_main_output_loss: 0.3531 - val_aux_output_loss: 0.4609\n",
      "Epoch 20/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3547 - main_output_loss: 0.3447 - aux_output_loss: 0.4440 - val_loss: 0.3649 - val_main_output_loss: 0.3546 - val_aux_output_loss: 0.4569\n"
     ]
    }
   ],
   "source": [
    "history = model.fit([X_train_A, X_train_B],\n",
    "                    [y_train, y_train],\n",
    "                    epochs=20,\n",
    "                    validation_data=([X_valid_A, X_valid_B], [y_valid, y_valid]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "great-tackle",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "162/162 [==============================] - 0s 2ms/step - loss: 0.3642 - main_output_loss: 0.3542 - aux_output_loss: 0.4543\n"
     ]
    }
   ],
   "source": [
    "total_loss, main_loss, aux_loss = model.evaluate(\n",
    " [X_test_A, X_test_B], [y_test, y_test])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "induced-uniform",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_main, y_pred_aux = model.predict([X_new_A, X_new_B])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "equal-jewel",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.675177 ],\n",
       "       [3.494091 ],\n",
       "       [0.3852623]], dtype=float32)"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_main"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "remarkable-insertion",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.874715 ],\n",
       "       [3.653451 ],\n",
       "       [0.4929863]], dtype=float32)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_aux"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "respective-enclosure",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.312, 3.477, 1.625])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "purple-intranet",
   "metadata": {},
   "source": [
    "### Construyendo modelos dinámicos con el API de *subclassing*\n",
    "\n",
    "Los APIs funcional y secuencial construyen los modelos en una forma estática. Las ventajas son que se puede guardar, copiar y compartir los modelos; se puede visualizar y analizar su estructura; y el proceso de *debugging* es más fácil.\n",
    "\n",
    "Para construir un modelo más dinámico, con ciclos, variación de la forma de las matrices, condicionales, etc. hay que usar el API de *subclassing*.\n",
    "\n",
    "Para usar este API:\n",
    "\n",
    "* Crear una subclase de la clase `Model`.\n",
    "* Crear las capas que queremos en el constructor de la clase.\n",
    "* Los cálculos están implementados en el método `call()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "dried-display",
   "metadata": {},
   "outputs": [],
   "source": [
    "class WideAndDeepModel(keras.Model):\n",
    "    def __init__(self, units=30, activation=\"relu\", **kwargs):\n",
    "        super().__init__(**kwargs) # para argumentos como \"name\"\n",
    "        self.hidden1 = keras.layers.Dense(units, activation=activation)\n",
    "        self.hidden2 = keras.layers.Dense(units, activation=activation)\n",
    "        self.main_output = keras.layers.Dense(1)\n",
    "        self.aux_output = keras.layers.Dense(1)\n",
    "        \n",
    "    def call(self, inputs):\n",
    "        input_A, input_B = inputs\n",
    "        hidden1 = self.hidden1(input_B)\n",
    "        hidden2 = self.hidden2(hidden1)\n",
    "        concat = keras.layers.concatenate([input_A, hidden2])\n",
    "        main_output = self.main_output(concat)\n",
    "        aux_output = self.aux_output(hidden2)\n",
    "        return main_output, aux_output\n",
    "    \n",
    "model = WideAndDeepModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "entitled-plastic",
   "metadata": {},
   "source": [
    "Este método da más flexibilidad (podemos hacer cualquier cosa en `call()`) pero la desventaja es que Keras no puede inspeccionar mucho del modelo..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "massive-crime",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "artistic-absence",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=[\"mse\", \"mse\"], loss_weights=[0.9, 0.1], optimizer=\"sgd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "three-blast",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "363/363 [==============================] - 1s 4ms/step - loss: 0.9998 - output_1_loss: 0.8443 - output_2_loss: 2.3989 - val_loss: 0.9677 - val_output_1_loss: 0.9346 - val_output_2_loss: 1.2660\n",
      "Epoch 2/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.5242 - output_1_loss: 0.4665 - output_2_loss: 1.0435 - val_loss: 0.7031 - val_output_1_loss: 0.6614 - val_output_2_loss: 1.0778\n",
      "Epoch 3/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4826 - output_1_loss: 0.4384 - output_2_loss: 0.8798 - val_loss: 0.6140 - val_output_1_loss: 0.5856 - val_output_2_loss: 0.8698\n",
      "Epoch 4/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4650 - output_1_loss: 0.4303 - output_2_loss: 0.7773 - val_loss: 0.5870 - val_output_1_loss: 0.5629 - val_output_2_loss: 0.8032\n",
      "Epoch 5/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4532 - output_1_loss: 0.4255 - output_2_loss: 0.7024 - val_loss: 0.4835 - val_output_1_loss: 0.4594 - val_output_2_loss: 0.6997\n",
      "Epoch 6/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4376 - output_1_loss: 0.4142 - output_2_loss: 0.6473 - val_loss: 0.4475 - val_output_1_loss: 0.4245 - val_output_2_loss: 0.6548\n",
      "Epoch 7/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4280 - output_1_loss: 0.4070 - output_2_loss: 0.6175 - val_loss: 0.4349 - val_output_1_loss: 0.4138 - val_output_2_loss: 0.6253\n",
      "Epoch 8/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4236 - output_1_loss: 0.4044 - output_2_loss: 0.5970 - val_loss: 0.4391 - val_output_1_loss: 0.4200 - val_output_2_loss: 0.6109\n",
      "Epoch 9/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4236 - output_1_loss: 0.4061 - output_2_loss: 0.5812 - val_loss: 0.4242 - val_output_1_loss: 0.4061 - val_output_2_loss: 0.5874\n",
      "Epoch 10/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4125 - output_1_loss: 0.3952 - output_2_loss: 0.5680 - val_loss: 0.4229 - val_output_1_loss: 0.4059 - val_output_2_loss: 0.5756\n",
      "Epoch 11/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4068 - output_1_loss: 0.3903 - output_2_loss: 0.5556 - val_loss: 0.4134 - val_output_1_loss: 0.3968 - val_output_2_loss: 0.5629\n",
      "Epoch 12/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4027 - output_1_loss: 0.3867 - output_2_loss: 0.5471 - val_loss: 0.4089 - val_output_1_loss: 0.3929 - val_output_2_loss: 0.5523\n",
      "Epoch 13/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3965 - output_1_loss: 0.3813 - output_2_loss: 0.5341 - val_loss: 0.4134 - val_output_1_loss: 0.3983 - val_output_2_loss: 0.5497\n",
      "Epoch 14/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3966 - output_1_loss: 0.3819 - output_2_loss: 0.5288 - val_loss: 0.4007 - val_output_1_loss: 0.3853 - val_output_2_loss: 0.5396\n",
      "Epoch 15/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3878 - output_1_loss: 0.3732 - output_2_loss: 0.5187 - val_loss: 0.3974 - val_output_1_loss: 0.3828 - val_output_2_loss: 0.5291\n",
      "Epoch 16/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3819 - output_1_loss: 0.3677 - output_2_loss: 0.5099 - val_loss: 0.3920 - val_output_1_loss: 0.3784 - val_output_2_loss: 0.5151\n",
      "Epoch 17/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3762 - output_1_loss: 0.3624 - output_2_loss: 0.5003 - val_loss: 0.3912 - val_output_1_loss: 0.3782 - val_output_2_loss: 0.5082\n",
      "Epoch 18/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3723 - output_1_loss: 0.3589 - output_2_loss: 0.4926 - val_loss: 0.3914 - val_output_1_loss: 0.3785 - val_output_2_loss: 0.5068\n",
      "Epoch 19/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3735 - output_1_loss: 0.3608 - output_2_loss: 0.4885 - val_loss: 0.3791 - val_output_1_loss: 0.3665 - val_output_2_loss: 0.4922\n",
      "Epoch 20/20\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.3753 - output_1_loss: 0.3636 - output_2_loss: 0.4804 - val_loss: 0.3937 - val_output_1_loss: 0.3805 - val_output_2_loss: 0.5124\n"
     ]
    }
   ],
   "source": [
    "history = model.fit([X_train_A, X_train_B],\n",
    "                    [y_train, y_train],\n",
    "                    epochs=20,\n",
    "                    validation_data=([X_valid_A, X_valid_B], [y_valid, y_valid]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "unsigned-moral",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"wide_and_deep_model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_26 (Dense)             multiple                  210       \n",
      "_________________________________________________________________\n",
      "dense_27 (Dense)             multiple                  930       \n",
      "_________________________________________________________________\n",
      "dense_28 (Dense)             multiple                  36        \n",
      "_________________________________________________________________\n",
      "dense_29 (Dense)             multiple                  31        \n",
      "=================================================================\n",
      "Total params: 1,207\n",
      "Trainable params: 1,207\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "honey-discrimination",
   "metadata": {},
   "source": [
    "### Guardar y cargar un modelo\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "featured-pocket",
   "metadata": {},
   "source": [
    "En el caso de los modelos secuencial y funcional, podemos guardar el formato HDF5. Con esto podemos guardar todos los parámetros y hiperparámetros del modelo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "crude-exposure",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    housing.data, housing.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "recent-discipline",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train_full,\n",
    "                                                     y_train_full)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "pediatric-valley",
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_valid = scaler.transform(X_valid)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "occupied-secretary",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"relu\", input_shape=X_train.shape[1:]),\n",
    "    keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "opposite-scene",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"mean_squared_error\", optimizer=\"sgd\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "damaged-anchor",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 1.6556 - val_loss: 0.5754\n",
      "Epoch 2/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5006 - val_loss: 0.5023\n",
      "Epoch 3/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4583 - val_loss: 0.4683\n",
      "Epoch 4/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4386 - val_loss: 0.4784\n",
      "Epoch 5/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4272 - val_loss: 0.4421\n",
      "Epoch 6/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4194 - val_loss: 0.4342\n",
      "Epoch 7/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4109 - val_loss: 0.4239\n",
      "Epoch 8/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4073 - val_loss: 0.4170\n",
      "Epoch 9/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4062 - val_loss: 0.4783\n",
      "Epoch 10/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4009 - val_loss: 0.4166\n",
      "Epoch 11/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3968 - val_loss: 0.4070\n",
      "Epoch 12/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3924 - val_loss: 0.4045\n",
      "Epoch 13/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3883 - val_loss: 0.4181\n",
      "Epoch 14/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3915 - val_loss: 0.3972\n",
      "Epoch 15/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3826 - val_loss: 0.4031\n",
      "Epoch 16/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3823 - val_loss: 0.3919\n",
      "Epoch 17/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3785 - val_loss: 0.3973\n",
      "Epoch 18/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3758 - val_loss: 0.3880\n",
      "Epoch 19/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3761 - val_loss: 0.3910\n",
      "Epoch 20/20\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3733 - val_loss: 0.3859\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=20,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "decent-notion",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_keras_model.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "civic-webcam",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_keras_model.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pacific-penalty",
   "metadata": {},
   "source": [
    "#### Utilizando *callbacks*\n",
    "\n",
    "El método `fit()` tiene un argumento para `callbacks` donde se puede especificar funciones que Keras llamará o antes y después del entrenamiento, o antes y después de cada época, o antes y después del procesamiento de cada lote de instancias.\n",
    "\n",
    "Por ejemplo, un *callback* de tipo `ModelCheckpoint` guardar *checkpoints* del modelo en intervalos regulares durante el entrenamiento, por defecto al final de cada época:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "flying-swedish",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3725\n",
      "Epoch 2/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3688\n",
      "Epoch 3/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3779\n",
      "Epoch 4/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3647\n",
      "Epoch 5/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3657\n",
      "Epoch 6/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3635\n",
      "Epoch 7/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3604\n",
      "Epoch 8/10\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.3558\n",
      "Epoch 9/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3562\n",
      "Epoch 10/10\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.3533\n"
     ]
    }
   ],
   "source": [
    "checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_keras_model.h5\")\n",
    "history = model.fit(X_train, y_train, epochs=10, callbacks=[checkpoint_cb])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "verified-thirty",
   "metadata": {},
   "source": [
    "Con el uso de un conjunto de validación, podemos especificar que queremos guardar el *checkpoint* solamente si su rendimiento en el conjunto de validación es el mejor hasta ahora. Este es una forma simple de implementar *early stopping* que vimos antes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "identified-polls",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Guardar el mejor modelo\n",
    "checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_keras_model.h5\",\n",
    "                                                save_best_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "cutting-whale",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3454 - val_loss: 0.3598\n",
      "Epoch 2/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3590 - val_loss: 0.3602\n",
      "Epoch 3/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3474 - val_loss: 0.3639\n",
      "Epoch 4/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3617 - val_loss: 0.3598\n",
      "Epoch 5/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3426 - val_loss: 0.3685\n",
      "Epoch 6/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3515 - val_loss: 0.4229\n",
      "Epoch 7/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3401 - val_loss: 0.3536\n",
      "Epoch 8/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3413 - val_loss: 0.3547\n",
      "Epoch 9/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3415 - val_loss: 0.3766\n",
      "Epoch 10/10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3360 - val_loss: 0.3551\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=10, \n",
    "                    validation_data=(X_valid, y_valid),\n",
    "                    callbacks=[checkpoint_cb])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "trying-member",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Cargar el mejor modelo\n",
    "model = keras.models.load_model(\"my_keras_model.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "random-party",
   "metadata": {},
   "source": [
    "Otra manera de implementar *early stopping* es con un *callback* diseñado para eso! Terminará el entrenamiento si no hay progreso en el puntaje de validación por un número de épocas (definido por el parámetro `patience`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "alleged-compromise",
   "metadata": {},
   "outputs": [],
   "source": [
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=10,\n",
    "                                                  restore_best_weights=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "stock-advocate",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3301 - val_loss: 0.3488\n",
      "Epoch 2/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3359 - val_loss: 0.3485\n",
      "Epoch 3/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3285 - val_loss: 0.3512\n",
      "Epoch 4/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3361 - val_loss: 0.3432\n",
      "Epoch 5/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3454 - val_loss: 0.3847\n",
      "Epoch 6/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3344 - val_loss: 0.3581\n",
      "Epoch 7/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3335 - val_loss: 0.3537\n",
      "Epoch 8/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3420 - val_loss: 0.3537\n",
      "Epoch 9/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3293 - val_loss: 0.3652\n",
      "Epoch 10/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3276 - val_loss: 0.3536\n",
      "Epoch 11/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3253 - val_loss: 0.3453\n",
      "Epoch 12/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3239 - val_loss: 0.3659\n",
      "Epoch 13/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3237 - val_loss: 0.3432\n",
      "Epoch 14/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3212 - val_loss: 0.3462\n",
      "Epoch 15/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3221 - val_loss: 0.3990\n",
      "Epoch 16/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3386 - val_loss: 0.3453\n",
      "Epoch 17/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3316 - val_loss: 0.3420\n",
      "Epoch 18/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3213 - val_loss: 0.3436\n",
      "Epoch 19/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3280 - val_loss: 0.3581\n",
      "Epoch 20/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3250 - val_loss: 0.4872\n",
      "Epoch 21/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3275 - val_loss: 0.3467\n",
      "Epoch 22/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3317 - val_loss: 0.3790\n",
      "Epoch 23/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3206 - val_loss: 0.3497\n",
      "Epoch 24/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3935 - val_loss: 0.3628\n",
      "Epoch 25/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3285 - val_loss: 0.3451\n",
      "Epoch 26/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3208 - val_loss: 0.3541\n",
      "Epoch 27/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3183 - val_loss: 0.3443\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=100, \n",
    "                    validation_data=(X_valid, y_valid),\n",
    "                    callbacks=[checkpoint_cb, early_stopping_cb])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "automated-filter",
   "metadata": {},
   "source": [
    "También podemos definir nuestros propios *callbacks*, por ejemplo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "useful-planet",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PrintValTrainRatioCallback(keras.callbacks.Callback):\n",
    "    def on_epoch_end(self, epoch, logs):\n",
    "        print(\"\\nval/train: {:.2f}\".format(logs[\"val_loss\"] / logs[\"loss\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "convertible-factory",
   "metadata": {},
   "outputs": [],
   "source": [
    "printvaltrain_cb = PrintValTrainRatioCallback()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "auburn-village",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "353/363 [============================>.] - ETA: 0s - loss: 0.3280\n",
      "val/train: 1.13\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3290 - val_loss: 0.3720\n",
      "Epoch 2/100\n",
      "355/363 [============================>.] - ETA: 0s - loss: 0.3335\n",
      "val/train: 1.07\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3345 - val_loss: 0.3563\n",
      "Epoch 3/100\n",
      "346/363 [===========================>..] - ETA: 0s - loss: 0.3310\n",
      "val/train: 1.07\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3297 - val_loss: 0.3518\n",
      "Epoch 4/100\n",
      "362/363 [============================>.] - ETA: 0s - loss: 0.3261\n",
      "val/train: 1.05\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3260 - val_loss: 0.3436\n",
      "Epoch 5/100\n",
      "339/363 [===========================>..] - ETA: 0s - loss: 0.3208\n",
      "val/train: 1.07\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3192 - val_loss: 0.3425\n",
      "Epoch 6/100\n",
      "351/363 [============================>.] - ETA: 0s - loss: 0.3178\n",
      "val/train: 1.14\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3175 - val_loss: 0.3626\n",
      "Epoch 7/100\n",
      "354/363 [============================>.] - ETA: 0s - loss: 0.3179\n",
      "val/train: 1.09\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3191 - val_loss: 0.3489\n",
      "Epoch 8/100\n",
      "341/363 [===========================>..] - ETA: 0s - loss: 0.3170\n",
      "val/train: 1.07\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3179 - val_loss: 0.3414\n",
      "Epoch 9/100\n",
      "358/363 [============================>.] - ETA: 0s - loss: 0.3150\n",
      "val/train: 1.13\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3165 - val_loss: 0.3591\n",
      "Epoch 10/100\n",
      "329/363 [==========================>...] - ETA: 0s - loss: 0.3209\n",
      "val/train: 1.09\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3166 - val_loss: 0.3451\n",
      "Epoch 11/100\n",
      "351/363 [============================>.] - ETA: 0s - loss: 0.3149\n",
      "val/train: 1.08\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3155 - val_loss: 0.3396\n",
      "Epoch 12/100\n",
      "342/363 [===========================>..] - ETA: 0s - loss: 0.3145\n",
      "val/train: 1.12\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3161 - val_loss: 0.3535\n",
      "Epoch 13/100\n",
      "360/363 [============================>.] - ETA: 0s - loss: 0.3151\n",
      "val/train: 1.14\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3147 - val_loss: 0.3585\n",
      "Epoch 14/100\n",
      "341/363 [===========================>..] - ETA: 0s - loss: 0.3189\n",
      "val/train: 1.09\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3171 - val_loss: 0.3456\n",
      "Epoch 15/100\n",
      "363/363 [==============================] - ETA: 0s - loss: 0.3158\n",
      "val/train: 1.10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3158 - val_loss: 0.3487\n",
      "Epoch 16/100\n",
      "361/363 [============================>.] - ETA: 0s - loss: 0.3133\n",
      "val/train: 1.10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3149 - val_loss: 0.3462\n",
      "Epoch 17/100\n",
      "348/363 [===========================>..] - ETA: 0s - loss: 0.3134\n",
      "val/train: 1.32\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3155 - val_loss: 0.4177\n",
      "Epoch 18/100\n",
      "337/363 [==========================>...] - ETA: 0s - loss: 0.3260\n",
      "val/train: 1.06\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3238 - val_loss: 0.3448\n",
      "Epoch 19/100\n",
      "353/363 [============================>.] - ETA: 0s - loss: 0.3140\n",
      "val/train: 1.10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3142 - val_loss: 0.3449\n",
      "Epoch 20/100\n",
      "352/363 [============================>.] - ETA: 0s - loss: 0.3125\n",
      "val/train: 1.08\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3134 - val_loss: 0.3384\n",
      "Epoch 21/100\n",
      "351/363 [============================>.] - ETA: 0s - loss: 0.3163\n",
      "val/train: 1.11\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3168 - val_loss: 0.3524\n",
      "Epoch 22/100\n",
      "358/363 [============================>.] - ETA: 0s - loss: 0.3189\n",
      "val/train: 1.09\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3181 - val_loss: 0.3477\n",
      "Epoch 23/100\n",
      "353/363 [============================>.] - ETA: 0s - loss: 0.3129\n",
      "val/train: 1.12\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3132 - val_loss: 0.3524\n",
      "Epoch 24/100\n",
      "363/363 [==============================] - ETA: 0s - loss: 0.3112\n",
      "val/train: 1.12\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3112 - val_loss: 0.3474\n",
      "Epoch 25/100\n",
      "344/363 [===========================>..] - ETA: 0s - loss: 0.3128\n",
      "val/train: 1.09\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3113 - val_loss: 0.3388\n",
      "Epoch 26/100\n",
      "351/363 [============================>.] - ETA: 0s - loss: 0.3126\n",
      "val/train: 1.10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3097 - val_loss: 0.3417\n",
      "Epoch 27/100\n",
      "359/363 [============================>.] - ETA: 0s - loss: 0.3101\n",
      "val/train: 1.10\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3100 - val_loss: 0.3404\n",
      "Epoch 28/100\n",
      "352/363 [============================>.] - ETA: 0s - loss: 0.3098\n",
      "val/train: 1.11\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3112 - val_loss: 0.3452\n",
      "Epoch 29/100\n",
      "360/363 [============================>.] - ETA: 0s - loss: 0.3131\n",
      "val/train: 1.12\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3140 - val_loss: 0.3509\n",
      "Epoch 30/100\n",
      "344/363 [===========================>..] - ETA: 0s - loss: 0.3101\n",
      "val/train: 1.11\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3107 - val_loss: 0.3439\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=100, \n",
    "                    validation_data=(X_valid, y_valid),\n",
    "                    callbacks=[checkpoint_cb, early_stopping_cb, \n",
    "                               printvaltrain_cb])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "geological-reservation",
   "metadata": {},
   "source": [
    "Se puede implementar:\n",
    "\n",
    "* `on_train_begin()`\n",
    "* `on_train_end()`\n",
    "* `on_epoch_begin()`\n",
    "* `on_epoch_end()`\n",
    "* `on_batch_begin()`\n",
    "* `on_batch_end()`\n",
    "\n",
    "También se puede usar *callbacks* en la evaluación del modelo (cuando usamos `evaluate()`) y en las predicciones (cuando usamos `predict()`):\n",
    "\n",
    "* `on_test_begin()`\n",
    "* `on_test_end()`\n",
    "* `on_test_batch_begin()`\n",
    "* `on_test_batch_end()`\n",
    "\n",
    "\n",
    "* `on_predict_begin()`\n",
    "* `on_predict_end()`\n",
    "* `on_predict_batch_begin()`\n",
    "* `on_predict_batch_end()`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "expected-laundry",
   "metadata": {},
   "source": [
    "### TensorBoard\n",
    "\n",
    "*TensorBoard* es una herramienta interactiva que permite visualizar las curvas de aprendizaje del modelo durante el proceso de entrenamiento, comparar entre varias ejecuciones del modelo, visualizar el gráfo de cómputo, y mucho más...\n",
    "\n",
    "Para usarlo, hay que guardar los datos del modelo en archivos de binario que se llaman **archivos de eventos** (*event files*). Cada record de datos binarios se llama un **resumen** (*summary*).\n",
    "\n",
    "*TensorBoard* monitorea la carpeta de *logs*, y automaticamente puede detectar cambios y actualizar las visualizaciones. Así se puede visualizar en (casi) tiempo real el rendiemiento del modelo durante el entrenamiento.\n",
    "\n",
    "La mejor forma de organizar los *logs* es crear una carpeta principal de *logs*, y crear una sub-carpeta nueva cada vez que ejecutamos el modelo.\n",
    "\n",
    "Ahora vamos a definir la carpeta raiz de *logs*, y una función para generar las rutas a las sub-carpetas basadas en la fecha actual y la hora."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "unlimited-grounds",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "root_logdir = os.path.join(os.curdir, \"my_logs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "distinct-greek",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_run_logdir():\n",
    "    import time\n",
    "    run_id = time.strftime(\"run_%Y_%m_%d-%H_%M_%S\")\n",
    "    return os.path.join(root_logdir, run_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "efficient-uniform",
   "metadata": {},
   "outputs": [],
   "source": [
    "run_logdir = get_run_logdir()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "valuable-garlic",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./my_logs/run_2022_07_06-15_57_46'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_logdir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "supported-johns",
   "metadata": {},
   "outputs": [],
   "source": [
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "special-bennett",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"relu\", input_shape=X_train.shape[1:]),\n",
    "    keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "dense-david",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.05)\n",
    "model.compile(loss=\"mean_squared_error\", optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "wired-conditions",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "  1/363 [..............................] - ETA: 0s - loss: 4.4870WARNING:tensorflow:From /home/graeme/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager.profiler) is deprecated and will be removed after 2020-07-01.\n",
      "Instructions for updating:\n",
      "use `tf.profiler.experimental.stop` instead.\n",
      "  2/363 [..............................] - ETA: 28s - loss: 3.7443WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0027s vs `on_train_batch_end` time: 0.1567s). Check your callbacks.\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.8584 - val_loss: 0.5094\n",
      "Epoch 2/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4683 - val_loss: 0.4509\n",
      "Epoch 3/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4324 - val_loss: 0.3777\n",
      "Epoch 4/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4047 - val_loss: 0.3676\n",
      "Epoch 5/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3979 - val_loss: 0.3605\n",
      "Epoch 6/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3886 - val_loss: 0.3806\n",
      "Epoch 7/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4118 - val_loss: 0.3488\n",
      "Epoch 8/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4069 - val_loss: 0.4177\n",
      "Epoch 9/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3898 - val_loss: 0.3522\n",
      "Epoch 10/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3746 - val_loss: 0.3559\n",
      "Epoch 11/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3743 - val_loss: 0.3372\n",
      "Epoch 12/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3739 - val_loss: 0.3524\n",
      "Epoch 13/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3657 - val_loss: 0.3496\n",
      "Epoch 14/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.9142 - val_loss: 0.3831\n",
      "Epoch 15/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4708 - val_loss: 0.3906\n",
      "Epoch 16/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4180 - val_loss: 0.4031\n",
      "Epoch 17/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3984 - val_loss: 0.3851\n",
      "Epoch 18/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3867 - val_loss: 0.3425\n",
      "Epoch 19/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3954 - val_loss: 0.3370\n",
      "Epoch 20/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3824 - val_loss: 0.3325\n",
      "Epoch 21/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3743 - val_loss: 0.4751\n",
      "Epoch 22/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4004 - val_loss: 0.3368\n",
      "Epoch 23/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3799 - val_loss: 0.3414\n",
      "Epoch 24/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3740 - val_loss: 0.3358\n",
      "Epoch 25/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3657 - val_loss: 0.3261\n",
      "Epoch 26/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3698 - val_loss: 0.3246\n",
      "Epoch 27/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3578 - val_loss: 0.3590\n",
      "Epoch 28/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3565 - val_loss: 0.4111\n",
      "Epoch 29/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3796 - val_loss: 0.3630\n",
      "Epoch 30/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3840 - val_loss: 0.3270\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=30,\n",
    "                    validation_data=(X_valid, y_valid),\n",
    "                    callbacks=[tensorboard_cb])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "color-dublin",
   "metadata": {},
   "outputs": [],
   "source": [
    "run_logdir = get_run_logdir()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "false-stephen",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./my_logs/run_2022_07_06-15_59_50'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_logdir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "brief-noise",
   "metadata": {},
   "outputs": [],
   "source": [
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "ignored-worth",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD(lr=0.001)\n",
    "model.compile(loss=\"mean_squared_error\", optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "enhanced-initial",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "  1/363 [..............................] - ETA: 0s - loss: 0.2265WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0024s vs `on_train_batch_end` time: 0.0173s). Check your callbacks.\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3485 - val_loss: 0.3177\n",
      "Epoch 2/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3460 - val_loss: 0.3165\n",
      "Epoch 3/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3444 - val_loss: 0.3154\n",
      "Epoch 4/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3433 - val_loss: 0.3148\n",
      "Epoch 5/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3425 - val_loss: 0.3142\n",
      "Epoch 6/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3420 - val_loss: 0.3137\n",
      "Epoch 7/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3415 - val_loss: 0.3136\n",
      "Epoch 8/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3409 - val_loss: 0.3131\n",
      "Epoch 9/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3407 - val_loss: 0.3130\n",
      "Epoch 10/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3403 - val_loss: 0.3127\n",
      "Epoch 11/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3399 - val_loss: 0.3132\n",
      "Epoch 12/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3398 - val_loss: 0.3127\n",
      "Epoch 13/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3394 - val_loss: 0.3124\n",
      "Epoch 14/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3392 - val_loss: 0.3120\n",
      "Epoch 15/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3387 - val_loss: 0.3123\n",
      "Epoch 16/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3386 - val_loss: 0.3119\n",
      "Epoch 17/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3385 - val_loss: 0.3119\n",
      "Epoch 18/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3384 - val_loss: 0.3118\n",
      "Epoch 19/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3378 - val_loss: 0.3119\n",
      "Epoch 20/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3378 - val_loss: 0.3114\n",
      "Epoch 21/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3376 - val_loss: 0.3114\n",
      "Epoch 22/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3374 - val_loss: 0.3113\n",
      "Epoch 23/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3373 - val_loss: 0.3108\n",
      "Epoch 24/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3370 - val_loss: 0.3112\n",
      "Epoch 25/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3368 - val_loss: 0.3113\n",
      "Epoch 26/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3368 - val_loss: 0.3112\n",
      "Epoch 27/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3366 - val_loss: 0.3109\n",
      "Epoch 28/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3363 - val_loss: 0.3107\n",
      "Epoch 29/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3363 - val_loss: 0.3103\n",
      "Epoch 30/30\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3360 - val_loss: 0.3105\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train, y_train, epochs=30,\n",
    "                    validation_data=(X_valid, y_valid),\n",
    "                    callbacks=[tensorboard_cb])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "disciplinary-quantum",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run_2022_07_06-15_57_46  run_2022_07_06-15_59_50\r\n"
     ]
    }
   ],
   "source": [
    "!ls my_logs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "smart-wrist",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train  validation\r\n"
     ]
    }
   ],
   "source": [
    "!ls my_logs/run_2022_07_06-15_57_46/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "french-cause",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "events.out.tfevents.1657137567.graeme-GL752VW.4046.57983.v2  plugins\r\n",
      "events.out.tfevents.1657137567.graeme-GL752VW.profile-empty\r\n"
     ]
    }
   ],
   "source": [
    "!ls my_logs/run_2022_07_06-15_57_46/train/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "precious-contrary",
   "metadata": {},
   "source": [
    "Tenemos archivos de eventos, pero también tenemos información de *profiling* (cuanto tiempo demora la ejecución de cada parte del modelo). Esta información es muy útil para ubicar partes del modelo que corren muy lento y beneficiarían de optimización (paralelización, etc.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "marked-warren",
   "metadata": {},
   "source": [
    "Se puede inicializar *TensorBoard* en el terminal:\n",
    "\n",
    "`tensorboard --logdir=./my_logs --port=6006`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "controlling-mirror",
   "metadata": {},
   "source": [
    "También se puede usar *TensorBoard* directamente en el Notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "radio-parameter",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "increasing-video",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "      <iframe id=\"tensorboard-frame-95d2261da8d20d98\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
       "      </iframe>\n",
       "      <script>\n",
       "        (function() {\n",
       "          const frame = document.getElementById(\"tensorboard-frame-95d2261da8d20d98\");\n",
       "          const url = new URL(\"/\", window.location);\n",
       "          const port = 6006;\n",
       "          if (port) {\n",
       "            url.port = port;\n",
       "          }\n",
       "          frame.src = url;\n",
       "        })();\n",
       "      </script>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%tensorboard --logdir=./my_logs --port=6006"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "convenient-playlist",
   "metadata": {},
   "source": [
    "También se puede usar *TensorFlow* para guardar información que después se puede visualizar en *TensorBoard*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "weird-thickness",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "alternate-stake",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_logdir = get_run_logdir()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "english-minnesota",
   "metadata": {},
   "outputs": [],
   "source": [
    "writer = tf.summary.create_file_writer(test_logdir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "wrapped-jimmy",
   "metadata": {},
   "outputs": [],
   "source": [
    "with writer.as_default():\n",
    "    for step in range(1, 1000 + 1):\n",
    "        tf.summary.scalar(\"my_scalar\", np.sin(step / 10), step=step)\n",
    "        data = (np.random.randn(100) + 2) * step / 100\n",
    "        tf.summary.histogram(\"my_hist\", data, buckets=50, step=step)\n",
    "        images = np.random.rand(2, 32, 32, 3)\n",
    "        tf.summary.image(\"my_images\", images * step / 1000, step=step)\n",
    "        texts = [\"This step is \" + str(step), \"Its square is \" + str(step**2)]\n",
    "        tf.summary.text(\"my_text\", texts, step=step)\n",
    "        sine_wave = tf.math.sin(tf.range(12000) / 48000 * 2 * np.pi * step)\n",
    "        audio = tf.reshape(tf.cast(sine_wave, tf.float32), [1, -1, 1])\n",
    "        tf.summary.audio(\"my_audio\", audio, sample_rate=48000, step=step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "scenic-deadline",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run_2022_07_05-23_24_33  run_2022_07_06-12_19_27  run_2022_07_06-12_29_26\r\n"
     ]
    }
   ],
   "source": [
    "!ls my_logs/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "directed-collector",
   "metadata": {},
   "source": [
    "### Ajuste de los hiperparámetros\n",
    "\n",
    "Hay muchos hiperparámetros en una red neuronal: se puede cambiar la arquitectura de la red, el número de nueronas, las funciones de activación, las conexiones entre las nueronas, etc.\n",
    "\n",
    "Una opción es usar `GridSearchCV` o `RandomizedSearchCV` como lo que hemos visto antes. Para hacer eso, hay que poner la construcción del modelo dentro de una función:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "digital-escape",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(n_hidden=1, n_neurons=30, learning_rate=3e-3, input_shape=[8]):\n",
    "    model = keras.models.Sequential()\n",
    "    model.add(keras.layers.InputLayer(input_shape=input_shape))\n",
    "    for layer in range(n_hidden):\n",
    "        model.add(keras.layers.Dense(n_neurons, activation=\"relu\"))\n",
    "    model.add(keras.layers.Dense(1))\n",
    "    optimizer = keras.optimizers.SGD(lr=learning_rate)\n",
    "    model.compile(loss=\"mse\", optimizer=optimizer)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "computational-rolling",
   "metadata": {},
   "source": [
    "Ahora usamos `KerasRegressor` que es simplemente una clase básica para tener un interfaz con nuestro modelo como si fuera un regresor de Scikit-Learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "radio-variance",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras_reg = keras.wrappers.scikit_learn.KerasRegressor(build_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "progressive-madagascar",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 1.3077 - val_loss: 0.7346\n",
      "Epoch 2/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.7197 - val_loss: 0.6269\n",
      "Epoch 3/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.6219 - val_loss: 0.5691\n",
      "Epoch 4/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5747 - val_loss: 0.5303\n",
      "Epoch 5/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5444 - val_loss: 0.5111\n",
      "Epoch 6/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5238 - val_loss: 0.4927\n",
      "Epoch 7/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5158 - val_loss: 0.4826\n",
      "Epoch 8/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.5022 - val_loss: 0.4735\n",
      "Epoch 9/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4984 - val_loss: 0.4668\n",
      "Epoch 10/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4858 - val_loss: 0.4611\n",
      "Epoch 11/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4820 - val_loss: 0.4553\n",
      "Epoch 12/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4762 - val_loss: 0.4505\n",
      "Epoch 13/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4702 - val_loss: 0.4466\n",
      "Epoch 14/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4679 - val_loss: 0.4647\n",
      "Epoch 15/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4647 - val_loss: 0.4385\n",
      "Epoch 16/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4579 - val_loss: 0.4343\n",
      "Epoch 17/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4544 - val_loss: 0.4307\n",
      "Epoch 18/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4510 - val_loss: 0.4268\n",
      "Epoch 19/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4468 - val_loss: 0.4267\n",
      "Epoch 20/100\n",
      "363/363 [==============================] - 1s 3ms/step - loss: 0.4433 - val_loss: 0.4201\n",
      "Epoch 21/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4408 - val_loss: 0.4224\n",
      "Epoch 22/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4386 - val_loss: 0.4146\n",
      "Epoch 23/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4406 - val_loss: 0.4177\n",
      "Epoch 24/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4369 - val_loss: 0.4103\n",
      "Epoch 25/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4321 - val_loss: 0.4070\n",
      "Epoch 26/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4274 - val_loss: 0.4060\n",
      "Epoch 27/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4261 - val_loss: 0.4021\n",
      "Epoch 28/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4230 - val_loss: 0.4001\n",
      "Epoch 29/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4210 - val_loss: 0.4082\n",
      "Epoch 30/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4249 - val_loss: 0.3994\n",
      "Epoch 31/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4212 - val_loss: 0.3941\n",
      "Epoch 32/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4150 - val_loss: 0.3939\n",
      "Epoch 33/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4169 - val_loss: 0.3945\n",
      "Epoch 34/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4144 - val_loss: 0.4175\n",
      "Epoch 35/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4110 - val_loss: 0.3864\n",
      "Epoch 36/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4066 - val_loss: 0.3853\n",
      "Epoch 37/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4066 - val_loss: 0.3855\n",
      "Epoch 38/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4035 - val_loss: 0.3830\n",
      "Epoch 39/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4062 - val_loss: 0.3825\n",
      "Epoch 40/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3998 - val_loss: 0.3797\n",
      "Epoch 41/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3987 - val_loss: 0.3782\n",
      "Epoch 42/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3967 - val_loss: 0.3778\n",
      "Epoch 43/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3963 - val_loss: 0.3802\n",
      "Epoch 44/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3938 - val_loss: 0.3748\n",
      "Epoch 45/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3983 - val_loss: 0.3738\n",
      "Epoch 46/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3945 - val_loss: 0.3729\n",
      "Epoch 47/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3894 - val_loss: 0.3709\n",
      "Epoch 48/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3885 - val_loss: 0.3702\n",
      "Epoch 49/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3879 - val_loss: 0.3687\n",
      "Epoch 50/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3856 - val_loss: 0.3699\n",
      "Epoch 51/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3852 - val_loss: 0.3678\n",
      "Epoch 52/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3831 - val_loss: 0.3670\n",
      "Epoch 53/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3826 - val_loss: 0.3648\n",
      "Epoch 54/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3809 - val_loss: 0.3636\n",
      "Epoch 55/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3802 - val_loss: 0.3663\n",
      "Epoch 56/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3791 - val_loss: 0.3647\n",
      "Epoch 57/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3781 - val_loss: 0.3614\n",
      "Epoch 58/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3781 - val_loss: 0.3631\n",
      "Epoch 59/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3760 - val_loss: 0.3581\n",
      "Epoch 60/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3757 - val_loss: 0.3585\n",
      "Epoch 61/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3748 - val_loss: 0.3562\n",
      "Epoch 62/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3748 - val_loss: 0.3575\n",
      "Epoch 63/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3723 - val_loss: 0.3546\n",
      "Epoch 64/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3714 - val_loss: 0.3540\n",
      "Epoch 65/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3754 - val_loss: 0.3548\n",
      "Epoch 66/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3691 - val_loss: 0.3536\n",
      "Epoch 67/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3692 - val_loss: 0.3522\n",
      "Epoch 68/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3683 - val_loss: 0.3523\n",
      "Epoch 69/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3686 - val_loss: 0.3568\n",
      "Epoch 70/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3678 - val_loss: 0.3605\n",
      "Epoch 71/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3678 - val_loss: 0.4034\n",
      "Epoch 72/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3807 - val_loss: 0.3720\n",
      "Epoch 73/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3716 - val_loss: 0.3502\n",
      "Epoch 74/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3703 - val_loss: 0.3486\n",
      "Epoch 75/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3626 - val_loss: 0.3472\n",
      "Epoch 76/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3629 - val_loss: 0.3466\n",
      "Epoch 77/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3619 - val_loss: 0.3483\n",
      "Epoch 78/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3598 - val_loss: 0.3459\n",
      "Epoch 79/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3618 - val_loss: 0.3478\n",
      "Epoch 80/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3587 - val_loss: 0.3465\n",
      "Epoch 81/100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3585 - val_loss: 0.3458\n",
      "Epoch 82/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3569 - val_loss: 0.3448\n",
      "Epoch 83/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3583 - val_loss: 0.3410\n",
      "Epoch 84/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4083 - val_loss: 0.3415\n",
      "Epoch 85/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3648 - val_loss: 0.3461\n",
      "Epoch 86/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3571 - val_loss: 0.3408\n",
      "Epoch 87/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3664 - val_loss: 0.3453\n",
      "Epoch 88/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3691 - val_loss: 0.3491\n",
      "Epoch 89/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3550 - val_loss: 0.3400\n",
      "Epoch 90/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3660 - val_loss: 0.3393\n",
      "Epoch 91/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3544 - val_loss: 0.3369\n",
      "Epoch 92/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3612 - val_loss: 0.3387\n",
      "Epoch 93/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3519 - val_loss: 0.3378\n",
      "Epoch 94/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3711 - val_loss: 0.3383\n",
      "Epoch 95/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3519 - val_loss: 0.3445\n",
      "Epoch 96/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3501 - val_loss: 0.3390\n",
      "Epoch 97/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3500 - val_loss: 0.3437\n",
      "Epoch 98/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3488 - val_loss: 0.3383\n",
      "Epoch 99/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3507 - val_loss: 0.3341\n",
      "Epoch 100/100\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.3770 - val_loss: 0.3346\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f3fb86f7f28>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras_reg.fit(X_train, y_train, epochs=100,\n",
    "              validation_data=(X_valid, y_valid),\n",
    "              callbacks=[keras.callbacks.EarlyStopping(patience=10)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "billion-halloween",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "162/162 [==============================] - 0s 1ms/step - loss: 0.3523\n"
     ]
    }
   ],
   "source": [
    "mse_test = keras_reg.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "painted-emerald",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = X_test[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "later-hobby",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = keras_reg.predict(X_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "animated-layer",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.9301076, 2.7603898, 2.9088507], dtype=float32)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "julian-foundation",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.568, 1.253, 1.012])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "atmospheric-future",
   "metadata": {},
   "source": [
    "Ahora usamos `RandomizedSearchCV` (mejor que `GridSearchCV` cuando hay muchos parámetros) para encontrar el mejor modelo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "expensive-possession",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import reciprocal\n",
    "from sklearn.model_selection import RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "related-format",
   "metadata": {},
   "outputs": [],
   "source": [
    "param_distribs = {\n",
    "    \"n_hidden\": [0, 1, 2, 3],\n",
    "    \"n_neurons\": np.arange(1, 100),\n",
    "    \"learning_rate\": reciprocal(3e-4, 3e-2),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "departmental-consumption",
   "metadata": {},
   "outputs": [],
   "source": [
    "rnd_search_cv = RandomizedSearchCV(keras_reg, param_distribs, n_iter=10, cv=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "abroad-oasis",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "242/242 [==============================] - 1s 4ms/step - loss: 1.7169 - val_loss: 0.7215\n",
      "Epoch 2/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.7007 - val_loss: 0.6360\n",
      "Epoch 3/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.6359 - val_loss: 0.5876\n",
      "Epoch 4/100\n",
      "242/242 [==============================] - 1s 4ms/step - loss: 0.5922 - val_loss: 0.5501\n",
      "Epoch 5/100\n",
      "242/242 [==============================] - 1s 5ms/step - loss: 0.5552 - val_loss: 0.5159\n",
      "Epoch 6/100\n",
      "242/242 [==============================] - 1s 2ms/step - loss: 0.5267 - val_loss: 0.4899\n",
      "Epoch 7/100\n",
      "242/242 [==============================] - 1s 4ms/step - loss: 0.5019 - val_loss: 0.4691\n",
      "Epoch 8/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4844 - val_loss: 0.4531\n",
      "Epoch 9/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4699 - val_loss: 0.4405\n",
      "Epoch 10/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4584 - val_loss: 0.4316\n",
      "Epoch 11/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4478 - val_loss: 0.4230\n",
      "Epoch 12/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4417 - val_loss: 0.4163\n",
      "Epoch 13/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4345 - val_loss: 0.4206\n",
      "Epoch 14/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4297 - val_loss: 0.4065\n",
      "Epoch 15/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4236 - val_loss: 0.4028\n",
      "Epoch 16/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4196 - val_loss: 0.4022\n",
      "Epoch 17/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4152 - val_loss: 0.3991\n",
      "Epoch 18/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4115 - val_loss: 0.3910\n",
      "Epoch 19/100\n",
      "242/242 [==============================] - 1s 3ms/step - loss: 0.4077 - val_loss: 0.3891\n",
      "Epoch 20/100\n",
      " 92/242 [==========>...................] - ETA: 0s - loss: 0.4089"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-5cf6e6ebab80>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m rnd_search_cv.fit(X_train, y_train, epochs=100,\n\u001b[1;32m      2\u001b[0m                   \u001b[0mvalidation_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_valid\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_valid\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m                   callbacks=[keras.callbacks.EarlyStopping(patience=10)])\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36minner_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     70\u001b[0m                           FutureWarning)\n\u001b[1;32m     71\u001b[0m         \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 72\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     73\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0minner_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_search.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, groups, **fit_params)\u001b[0m\n\u001b[1;32m    734\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mresults\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    735\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 736\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_run_search\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mevaluate_candidates\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    737\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    738\u001b[0m         \u001b[0;31m# For multi-metric evaluation, store the best_index_, best_params_ and\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_search.py\u001b[0m in \u001b[0;36m_run_search\u001b[0;34m(self, evaluate_candidates)\u001b[0m\n\u001b[1;32m   1529\u001b[0m         evaluate_candidates(ParameterSampler(\n\u001b[1;32m   1530\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparam_distributions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_iter\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1531\u001b[0;31m             random_state=self.random_state))\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_search.py\u001b[0m in \u001b[0;36mevaluate_candidates\u001b[0;34m(candidate_params)\u001b[0m\n\u001b[1;32m    713\u001b[0m                                \u001b[0;32mfor\u001b[0m \u001b[0mparameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    714\u001b[0m                                in product(candidate_params,\n\u001b[0;32m--> 715\u001b[0;31m                                           cv.split(X, y, groups)))\n\u001b[0m\u001b[1;32m    716\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    717\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, iterable)\u001b[0m\n\u001b[1;32m   1039\u001b[0m             \u001b[0;31m# remaining jobs.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1040\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_iterating\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1041\u001b[0;31m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispatch_one_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1042\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_iterating\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_original_iterator\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1043\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36mdispatch_one_batch\u001b[0;34m(self, iterator)\u001b[0m\n\u001b[1;32m    857\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    858\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 859\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dispatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtasks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    860\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    861\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m_dispatch\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    775\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    776\u001b[0m             \u001b[0mjob_idx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 777\u001b[0;31m             \u001b[0mjob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    778\u001b[0m             \u001b[0;31m# A job can complete so quickly than its callback is\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    779\u001b[0m             \u001b[0;31m# called before we get here, causing self._jobs to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36mapply_async\u001b[0;34m(self, func, callback)\u001b[0m\n\u001b[1;32m    206\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    207\u001b[0m         \u001b[0;34m\"\"\"Schedule a func to be run\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 208\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mImmediateResult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    209\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    210\u001b[0m             \u001b[0mcallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    570\u001b[0m         \u001b[0;31m# Don't delay the application, to avoid keeping the input\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    571\u001b[0m         \u001b[0;31m# arguments in memory\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 572\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    573\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    574\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    261\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    262\u001b[0m             return [func(*args, **kwargs)\n\u001b[0;32m--> 263\u001b[0;31m                     for func, args, kwargs in self.items]\n\u001b[0m\u001b[1;32m    264\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    265\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__reduce__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m    261\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    262\u001b[0m             return [func(*args, **kwargs)\n\u001b[0;32m--> 263\u001b[0;31m                     for func, args, kwargs in self.items]\n\u001b[0m\u001b[1;32m    264\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    265\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__reduce__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_validation.py\u001b[0m in \u001b[0;36m_fit_and_score\u001b[0;34m(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, return_n_test_samples, return_times, return_estimator, error_score)\u001b[0m\n\u001b[1;32m    529\u001b[0m             \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    530\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 531\u001b[0;31m             \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    532\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    533\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/wrappers/scikit_learn.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, **kwargs)\u001b[0m\n\u001b[1;32m    164\u001b[0m     \u001b[0mfit_args\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    165\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 166\u001b[0;31m     \u001b[0mhistory\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mfit_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    167\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mhistory\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36m_method_wrapper\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    106\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_method_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    107\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_multi_worker_mode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 108\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    109\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    110\u001b[0m     \u001b[0;31m# Running inside `run_distribute_coordinator` already.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m   1096\u001b[0m                 batch_size=batch_size):\n\u001b[1;32m   1097\u001b[0m               \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1098\u001b[0;31m               \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1099\u001b[0m               \u001b[0;32mif\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_sync\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1100\u001b[0m                 \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    778\u001b[0m       \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    779\u001b[0m         \u001b[0mcompiler\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"nonXla\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 780\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    781\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    782\u001b[0m       \u001b[0mnew_tracing_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m    805\u001b[0m       \u001b[0;31m# In this case we have created variables on the first call, so we run the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    806\u001b[0m       \u001b[0;31m# defunned version which is guaranteed to never create variables.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 807\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateless_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# pylint: disable=not-callable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    808\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateful_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    809\u001b[0m       \u001b[0;31m# Release the lock early so that multiple threads can perform the call\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   2827\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2828\u001b[0m       \u001b[0mgraph_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_define_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2829\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_filtered_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2830\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2831\u001b[0m   \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_filtered_call\u001b[0;34m(self, args, kwargs, cancellation_manager)\u001b[0m\n\u001b[1;32m   1846\u001b[0m                            resource_variable_ops.BaseResourceVariable))],\n\u001b[1;32m   1847\u001b[0m         \u001b[0mcaptured_inputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcaptured_inputs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1848\u001b[0;31m         cancellation_manager=cancellation_manager)\n\u001b[0m\u001b[1;32m   1849\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1850\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_call_flat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcaptured_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcancellation_manager\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m   1922\u001b[0m       \u001b[0;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1923\u001b[0m       return self._build_call_outputs(self._inference_function.call(\n\u001b[0;32m-> 1924\u001b[0;31m           ctx, args, cancellation_manager=cancellation_manager))\n\u001b[0m\u001b[1;32m   1925\u001b[0m     forward_backward = self._select_forward_and_backward_functions(\n\u001b[1;32m   1926\u001b[0m         \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m    548\u001b[0m               \u001b[0minputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    549\u001b[0m               \u001b[0mattrs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 550\u001b[0;31m               ctx=ctx)\n\u001b[0m\u001b[1;32m    551\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    552\u001b[0m           outputs = execute.execute_with_cancellation(\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m     58\u001b[0m     \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     59\u001b[0m     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 60\u001b[0;31m                                         inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m     61\u001b[0m   \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     62\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "rnd_search_cv.fit(X_train, y_train, epochs=100,\n",
    "                  validation_data=(X_valid, y_valid),\n",
    "                  callbacks=[keras.callbacks.EarlyStopping(patience=10)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incredible-blend",
   "metadata": {},
   "source": [
    "`RandomizedSearchCV` ocupa validación cruzada con $k$ *folds*, no utiliza el conjunto de validación `(X_train, y_train)`. Este está usado solamente para *early stopping*.\n",
    "\n",
    "La búsqueda puede demorar horas, dependiendo de la complejidad del modelo, tamaño de conjunto de datos, etc.\n",
    "\n",
    "Se puede acceder a los mejores parámetros así:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "verbal-window",
   "metadata": {},
   "outputs": [],
   "source": [
    "rnd_search_cv.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "manual-willow",
   "metadata": {},
   "outputs": [],
   "source": [
    "rnd_search_cv.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "disturbed-september",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = rnd_search_cv.best_estimator_.model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "modular-summary",
   "metadata": {},
   "source": [
    "Hay otras librerias que ayudan en el proceso de buscar los mejores valores para los hiperparámetros. Todos funcionan en una manera similar: primero aplican una búsqueda amplia, pero no muy detallada. Cuando encuentran una región \"buena\" del espacio de parámetros, hacen un *zoom* y exploran esa región más detalladamente."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "assisted-vision",
   "metadata": {},
   "source": [
    "* [*Hyperopt*](http://hyperopt.github.io/hyperopt/)\n",
    "* [*Hyperas*](https://github.com/maxpumperla/hyperas), [*kopt*](https://github.com/Avsecz/kopt), [*Talos*](https://github.com/autonomio/talos)\n",
    "* [*Keras Tuner*](https://keras.io/keras_tuner/)\n",
    "* [*Scikit-Optimize* (skopt)](https://scikit-optimize.github.io/stable/)\n",
    "* [*Spearmint*](https://github.com/HIPS/Spearmint)\n",
    "* [*Hyperband*](https://github.com/zygmuntz/hyperband)\n",
    "* [*Sklearn-Deap*](https://github.com/rsteca/sklearn-deap)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "least-battle",
   "metadata": {},
   "source": [
    "El ajuste mejor de los hiperparámetros es un área de investigación activa, y algoritmos basados en la evolución biológica están llamando mucho interés hoy en día. El servicio **AutoML** de Google implementa un algoritmo así (más información [aquí](https://ai.googleblog.com/2018/03/using-evolutionary-automl-to-discover.html))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "optimum-hartford",
   "metadata": {},
   "source": [
    "### Algunas sugerencias para la construcción de redes nueronales\n",
    "\n",
    "**¿Cuántas capas ocultas?**\n",
    "\n",
    "Típicamente $1$ o $2$ capas ocultas es suficiente. Si el problema es más complejo, es mejor agregar más capas con un número limitado de neuronas que poner muchas nueronas en pocas capas. Una **red profunda** así puede aprender sobre los datos en una manera jerarquica.\n",
    "\n",
    "**¿Cuántas neuronas por capa?**\n",
    "\n",
    "Hoy en día es más común usar el mismo número de neuronas en cada capa oculta. Una forma de decidir el número es comenzar con pocas, y aumentar el número hasta que el modelo comienza a sufrir *overfitting*.\n",
    "\n",
    "Otra opción es poner un número \"grande\" de neuronas y restringir el modelo con regularización.\n",
    "\n",
    "**Taza de aprendizaje**\n",
    "\n",
    "Este hiperparámetro es probablemente lo más importante. Típicamente el valor óptimo es la mitad del máximo (el máximo corresponde al valor donde el algoritmo de entrenamiento diverge).\n",
    "\n",
    "Una forma de econtrar un buen valor de $\\eta$ es comenzar con un valor muy pequeño ($10^{-5}$) para algunas iteraciones y progresivamente aumentar el valor hasta un valor grande ($10$). Después se puede graficar la función de perdida $L$ contra $\\eta$. El mejor valor de $\\eta$ será alrededor de $10$ veces más pequeño que el punto donde $L$ tiene su mínimo en ese gráfico.\n",
    "\n",
    "**Optimizador**\n",
    "\n",
    "Veremos más sobre otros optimizadores (mejor que SGD) más tarde.\n",
    "\n",
    "**Tamaño de lote (*batch size*)**\n",
    "\n",
    "Los GPUs pueden acelerar el entrenamiento con lotes grandes, pero a veces sufre inestabilidades. Los expertos no están de acuerdo sobre el mejor valor...\n",
    "\n",
    "**Función de activación**\n",
    "\n",
    "Veremos más sobre las consecuencias de la elección de las funciones de activación más tarde. La función ReLU sirve en muchos casos. Para la capa de salida va a depender de la aplicación del modelo (sigmoide para regresión, *softmax* para clasificación, etc.)\n",
    "\n",
    "**Número de interaciones**\n",
    "\n",
    "Con *early stopping* no hace falta ajustar este hiperparámetro."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coastal-temple",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}