Preface.tex

\chapter{Preface}

\yinipar{\fontsize{60pt}{72pt}\usefont{U}{Kramer}{xl}{n}I} started learning about deep learning fundamentals in February 2017. At this time, I knew nothing about backpropagation, and was completely ignorant about the differences between a Feedforward, Convolutional and a Recurrent Neural Network.

\vspace{0.2cm}

As I navigated through the humongous amount of data available on deep learning online, I found myself quite frustrated when it came to really understand what deep learning is, and not just applying it with some available library.

\vspace{0.2cm}

In particular, the backpropagation update rules are seldom derived, and never in index form. Unfortunately for me, I have an "index" mind: seeing a 4 Dimensional convolution formula in matrix form does not do it for me. Since I am also stupid enough to like recoding the wheel in low level programming languages, the matrix form cannot be directly converted into working code either.


\vspace{0.2cm}

I therefore started some notes for my personal use, where I tried to rederive everything from scratch in index form.

\vspace{0.2cm}

I did so for the vanilla Feedforward network, then learned about L1 and L2 regularization , dropout\cite{Srivastava:2014:DSW:2627435.2670313}, batch normalization\cite{Ioffe2015}, several gradient descent optimization techniques... Then turned to convolutional networks, from conventional single digit number of layer conv-pool architectures\cite{Lecun98gradient-basedlearning} to recent VGG\cite{DBLP:journals/corr/SimonyanZ14a} ResNet\cite{He2015} ones, from local contrast normalization and rectification to bacthnorm... And finally I studied Recurrent Neural Network structures\cite{GravesA2016}, from the standard formulation to the most recent LSTM one\cite{Gers:2000:LFC:1121912.1121915}.

\vspace{0.2cm}

As my work progressed, my notes got bigger and bigger, until a point when I realized I might have enough material to help others starting their own deep learning journey.

\vspace{0.2cm}

This work is bottom-up at its core. If you are searching a working Neural Network in 10 lines of code and 5 minutes of your time, you have come to the wrong place. If you can mentally multiply and convolve 4D tensors, then I have nothing to convey to you either.

\vspace{0.2cm}

If on the other hand you like(d) to rederive every tiny calculation of every theorem of every class that you stepped into, then you might be interested by what follow!