Simple guide to Neural Network

What is Neural network?

Neural network is an algorithm which make input go through at least one hidden and output layers to output.
Graphically it is like below.

How it works?

To understand how neural network works, let’s just think about very simple model like this.

On this structure
Input: [1, 3] :$X=\left(\begin{matrix} x_{1}& x_{2}& x_{3}\end{matrix}\right)$
Weight: [3, 1] :$W=\left(\begin{matrix} w_{1}& w_{2}& w_{3}\end{matrix}\right)^{T}$
$\begin{align*} & d=XW\\ & =\left( \begin{matrix}x_{1}& x_{2}& x_{3}\end{matrix}\right) \left( \begin{matrix} w_{1}\\ w_{2}\\ w_{3}\end{matrix} \right) \end{align*}$
Let’s focus on $l_{2}$.

The input $d$ go through the function $f\left( x\right)$.
The purpose of training on neural network is to fix the weights $W$.
By choosing the function $f\left( x\right)$, you can do regression and classification.
The function $f\left( x\right)$ is called activation function.

Follow the calculation

To understand better, here, I follow the calculations.

The weights are assigned on all arrows, meaning by training you the model fixes the all arrow’s values.
By matrix expression, all the calculation can be simply expressed.
$\begin{align*} & X=\left( \begin{matrix} x_{1}& x_{2}&x_{3}\end{matrix} \right) \\ & W_{1}=\left( \begin{matrix} w_{11}& w_{12}\\ w_{21}& w_{22}\\ w_{31}& w_{32}\end{matrix} \right) \\ & w_{2}=\left( \begin{matrix} w_{1}^{\ast}\\ w_{2}^{\ast}\end{matrix} \right) \end{align*}$
Under the condition of the parameters above, I just follow the calculation and express the output by those.

$\begin{align*} & XW_{1}\\ & =\left( \begin{matrix} x_{1}&x_{2}& x_{3}\end{matrix} \right) \left( \begin{matrix} w_{1}& w_{12}\\ w_{21}& w_{22}\\ w_{31}& w_{32}\end{matrix} \right) \\ & =(x_{1}w_{11}+x_{2}w_{21}+x_{3}w_{31}&x_{1}w_{12}+x_{2}w_{22}+x_{3}w_{23})\end{align*}$
$\begin{align*} & A=x_{1}w_{11}+x_{2}w_{21}+x_{3}w_{31}\\ & B=x_{1}w_{12}+x_{2}w_{22}+x_{3}w_{32}\end{align*}$
plot_1’s A and B are like above.

The output of $l_{2}$ is $\begin{align*} & (f_{1}\left( x_{1}w_{11}+x_{2}w_{21}+x_{3}w_{31}\right) , &f_{2}\left( x_{1}w_{12}+x_{2}w_{22}+x_{3}w_{32})\right) \end{align*}$.
I take $D$ for this. So, the input of $l_{3}$ can be expressed as $DW_{2}$.

I take $C$ for $DW_{2}$.
$\begin{align*} & C=DW_{2}\\ & =(f_{1}\left( x_{1}w_{11}+x_{2}w_{21}+x_{3}w_{21}\right) & f_{2}\left( x_{1}w_{12}+x_{2}w_{22}+x_{3}w_{32}\right) )\left( \begin{matrix} w_{11}^{\ast }\\ w_{21}^{\ast }\end{matrix} \right) \end{align*}$
So, we can express the input of $l_{3}$ layer as below.
$\begin{align*} & C=\left( f_{1}\left( x_{1}w_{11}+x_{2}w_{21}+x_{3}w_{31}\right) \right) w_{11}^{\ast }+\left( f_{2}\left( x_{2}w_{12}+x_{2}w_{22}+x_{3}w_{32}\right) \right) w_{21}^{\ast }\end{align*}$

The output of $l_{3}$ layer is the value that the input goes through the activation function $f_{3}\left( x\right)$.
$\begin{align*} & f_{3}\left( \left( f_{1}\left( x_{1}w_{11}+x_{2}w_{21}+x_{3}w_{31}\right) \right) w_{11}^{\ast }+\left( f_{2}\left( x_{2}w_{12}+x_{2}w_{22}+x_{3}w_{32}\right) \right) w_{21}^{\ast } \right)\end{align*}$

Fix parameters

Getting parameters is as following.

• define loss function and find parameters which minimize that(to say precisely, not minimize)

example
$f\left( x\right) =\dfrac {1} {3}x^{3}-x$

In this case, the local minimum is the local optimal solution. It means the parameter is $x=1$.
But actually, neural network has so many parameters and the functions are nested. So it’s not easy to optimize.
The fundamental way to get parameters is to give initial values to parameters and by training update those.
On the example

• $x$: the parameter
• $f\left( x\right)$: loss function

The goal is to get parameter which locally minimizes $f\left( x\right)$.
Procedure

1. give initial values to parameter. This time give 0 to $x$.
2. update parameter

how to update parameter

move $x$ to $+$ or to $-$
$\rightarrow \begin{cases} w^{t+1}=W^{t}-\varepsilon \nabla E\\ \nabla E=\dfrac {\partial E} {\partial w}\end{cases}$