2.1-Introduction_to_ANN-Slides.qmd

---
title: "Advanced Machine Learning 4 Metabolomics"
subtitle: 'Neural Networks, Deep Learning and Artificial Intelligence'
format:
  revealjs: 
    incremental: false  
    transition: slide
    background-transition: fade
    transition-speed: slow
    scrollable: true
    menu:
      side: left
      width: half
      numbers: true
    slide-number: c/t
    show-slide-number: all
    progress: true
    css: "css4CU.css"
    theme: sky
knit:
  quarto:
    chunk_options:
      echo: true
      cache: false
      prompt: false
      tidy: true
      comment: NA
      message: false
      warning: false
    knit_options:
      width: 75
bibliography: "DeepLearning.bib"
editor_options: 
  chunk_output_type: console
---

# From Artificial Neural Networks to Artifial Intelligence

## Historical Background (1)

-   In the post-pandemic world, a lightning rise of AI, with a mess of realities and promises is impacting society.

-   Since ChatGPT entered the scene everybody has an experience, an opinion, or a fear on the topic.

![](https://bernardmarr.com/wp-content/uploads/2022/04/The-Dangers-Of-Not-Aligning-Artificial-Intelligence-With-Human-Values.jpg){fig-align="center" width="100%"}

## Is it just machine learning?

- Most tasks performed by AI can be described as Classification or  Prediction used in applications as:
   
   - Recommendation systems,
   - Image recognition, Image generation
   - Natural language processing

-   AI relies on machine learning algorithms, to make predictions based on large amounts of data.

-   AI has far-reaching implications beyond its predictive capabilities, including ethical, social or technological.

## AI, ANNs and Deep learning

- In many contexts, talking about AI means talking about 
*Deep Learning (DL)*.

- DL is a successful AI model which has powered many application such as *self-driving cars, voice assistants, and medical diagnosis systems*.

- DL originates in the field of *Artificial Neural Networks*

- But DL extends the basic principles of ANNs by:

    -   Adding complex architectures and algorithms and
    -   At the same time becoming more automatic


## The early history of AI (1)


```{r, out.width="90%", fig.cap= "[A Quick History of AI, ML and DL](https://nerdyelectronics.com/a-quick-history-of-ai-ml-and-dl/)"}
knitr::include_graphics("images/AIHistory1.jpg")
```


## Milestones in the history of DL

We can see several hints worth to account for:

-   The **Perceptron** and the first **Artificial Neural Network** where the basic building block was introduced.

-   The **Multilayered perceptron** and back-propagation where complex architectures were suggested to improve the capabilities.

-   **Deep Neural Networks**, with many hidden layers, and auto-tunability capabilities.

## From ANN to Deep learning

![Why Deep Learning Now?](images/WhyDLNow.png){fig-align="center" width="100%"} 

:::{.font90}
Source: [Alex Amini's MIT Introduction to Deep Learning' course](introtodeeplearning.com)
:::

## Success stories

Success stories such as

- the development of self-driving cars,

- the use of AI in medical diagnosis, and

- online shopping personalized recommendations

have also contributed to the widespread adoption of AI.

## Not to talk abou the fears

:::: {.columns}

::: {.column width='60%'}
- AI also comes with fears from multiple sources from science fiction to religion

  - Mass unemployment
  
  - Loss of privacity
  
  - AI bias
  
  - AI fakes
  
  - Or, simply, AI takeover

:::

::: {.column width='40%'}

<br>

![](2.1-Introduction_to_ANN-Slides_insertimage_2.png)

:::

::::


## Back to science

Where/How does it all fit?

```{r, fig.align='center', out.width="100%"}
knitr::include_graphics("images/AI-ML-DL-1.jpg")
```

## AI, ML, DL ...

-   **Artificial intelligence**: Ability of a computer to perform tasks commonly associated with intelligent beings.

-   **Machine learning**: study of algorithms that learn from examples and experience instead of relying on hard-coded rules and make predictions on new data

-   **Deep learning**: sub field of ML focusing on learning data representations as successive successive layers of increasingly meaningful representations.

## How does DL improve

```{r, fig.align='center', out.width="100%", fig.cap="[ML and DL Approaches for Brain Disease Diagnosis](https://ieeexplore.ieee.org/document/9363896)"}
knitr::include_graphics("images/ML_vs_DL-2.png")
```


::: {.notes}
-   DNN: feature extraction and classification without (or with much les) human intervention.
-   DNN improves with data availability, without seemingly reaching plateaus.
:::

## Size does matter!

![An illustration of the performance comparison between deep learning (DL) and other machine learning (ML) algorithms, where DL modeling from large amounts of data can increase the performance](images/PerformanceVsAmountOfData.png){fig-align="center" width="100%"}

## The impact of Deep learning {.smaller}

-   Near-human-level image classification

-   Near-human-level speech transcription

-   Near-human-level handwriting transcription

-   Dramatically improved machine translation

-   Dramatically improved text-to-speech conversion

-   Digital assistants such as Google Assistant and Amazon Alexa

-   Near-human-level autonomous driving

-   Improved ad targeting, as used by Google, Baidu, or Bing

-   Improved search results on the web

-   Ability to answer natural language questions

-   Superhuman Go playing

## Not all that glitters is gold ...

-   According to F. Chollet, the developer of Keras,

    -   "*we shouldn't believe the short-term hype, but should believe in the long-term vision*.
    -   *It may take a while for AI to be deployed to its true potential---a potential the full extent of which no one has yet dared to dream*
    -   *but AI is coming, and it will transform our world in a fantastic way*".

# The Artificial Neurone (AN)


## Emulating biological neurons


:::: {.columns}

::: {.column width='50%'}

```{r, fig.align='center', out.width="100%", fig.cap="[A biological Neuron]()"}
knitr::include_graphics("images/NaturalNeuron.png")
```

:::

::: {.column width='50%'}

```{r, fig.align='center', out.width="100%", fig.cap="[MuCulloch & Pitts proposal]()"}
knitr::include_graphics("images/MacCulloghPitts-Neuron.png")
```

:::

::::

- The first model of an artifial neurone was proposed by Mc Cullough & Pitts in 1943


## Mc Cullough's neuron

-   It may be divided into 2 parts.
    -   The first part, $g$,takes an input (as the dendrites of a neuron would do),
    -   It performs an aggregation and
    -   based on the aggregated value the second part, $f$, makes a decision.

See [the source of this picture](https://towardsdatascience.com/mcculloch-pitts-model-5fdf65ac5dd1) for an illustration on how this can be used to emulate logical operations such as AND, OR or NOT, but not XOR.

## Limitations

This first attempt to emulate neurons succeeded but with limitations:

-   What about non-Boolean (say, real) inputs?

-   What if all inputs are not equal?

-   What if we want to assign more importance to some inputs?

-   What about functions which are not linearly separable? Say XOR function

## Overcoming the limitations

-   To overcome these limitations Rosenblatt, proposed the perceptron model, or  *artificial neuron*, in 1958.

-   Generalizes McCullough-Pitts neuron in that *weights and thresholds can be learnt over time*.

    -   It takes a weighted sum of the inputs and
    -   It sets the output to iff the sum is more than an arbitrary threshold (**$\theta$**).

## Rosenblatt's perceptron

[![](images/RosenblattPerceptron1.png){width="100%"}](https://towardsdatascience.com/perceptron-the-artificial-neuron-4d8c70d5cc8d)

## Rosenblatt's perceptron

:::{.font80}
-   Instead of hand coding the thresholding parameter $\theta$,
-   It is added as one of the inputs, with the weight $w_0=-\theta$.
:::

[![](images/RosenblattPerceptron2.png){width="100%"}](https://towardsdatascience.com/perceptron-the-artificial-neuron-4d8c70d5cc8d)

## Comparison between the two

[![](images/McCullaughVSRosenblattPerceptron.png){width="100%"}](https://towardsdatascience.com/perceptron-the-artificial-neuron-4d8c70d5cc8d)

## Comparison between the two

-   This is an improvement because

    -   both, weights and threshold, can be learned and
    -   the inputs can be real values

-   But there is still a drawback in that a single perceptron can only be used to implement linearly separable functions.

-   Artificial Neural Networks improve on this by introducing *Activation Functions*

## Activation in biological neurons

-   Biological neurons are specialized cells that transmit signals to communicate with each other.
- Neuron's activation is based on releasing *neurotransmitters*, chemicals that transmit signals between nerve cells.
    -   When the signal reaching the neuron exceeds a certain threshold, it releases neurotransmitters to continue the communication process.

## Activation functions in AN

-   Analogously, *activation functions* in AN are functions to decide if the AN it is activated or not.
-   AN's activation function is a mathematical function applied to the neuron's input to produce an output.
    -   In practice it extends to complicated functions that can learn complex patterns in the data.
    -   Activation functions can incorporate non-linearity, improving over linear classifiers.

## Activation function

```{r, fig.align='center', out.width="100%"}
knitr::include_graphics("images/ActivationFunction0.png")
```

## Artificial Neuron

With all these ideas in mind we can now define an Artificial Neuron as a *computational unit* that :

-   takes as input $x=(x_0,x_1,x_2,x_3),\ (x_0 = +1 \equiv bias)$,

-   outputs $h_{\theta}(x) = f(\theta^\intercal x) = f(\sum_i \theta_ix_i)$,

-   where $f:\mathbb{R}\mapsto \mathbb{R}$ is called the **activation function**.

## Activation functions

:::{.font90}

-   Goal of activation function is to provide the neuron with *the capability of producing the required outputs*.

-   Flexible enough to produce

    -   Either linear or non-linear transformations.
    -   Output in the desired range (\[0,1\], {-1,1}, $\mathbb{R}^+$...)

-   Usually chosen from a (small) set of possibilities.

    -   Sigmoid function
    -   Hyperbolic tangent, or `tanh`, function
    -   ReLU

:::

## The sigmoid function {.smaller}

::: columns
::: {.column width="50%"}

$$
f(z)=\frac{1}{1+e^{-z}}
$$

-   Output real values $\in (0,1)$.

-   Natural interpretations as *probability*


```{r, fig.align='center', out.width="100%"}
knitr::include_graphics("images/sigmoidFunction.png")
```

:::

::: {.column width="50%"}

```{r, fig.align='center', out.width="100%"}
knitr::include_graphics("images/sigmoidFunctionDerivative.png")
```


:::
:::

## the hyperbolic tangent {.smaller}

::: columns
::: {.column width="50%"}
Also called `tanh`, function:

$$
f(z)=\frac{e^{z}-e^{-z}}{e^{z}+e^{-z}}
$$

-   outputs are zero-centered and bounded in −1,1

-   scaled and shifted Sigmoid

-   stronger gradient but still has vanishing gradient problem

-   Its derivative is $f'(z)=1-(f(z))^2$.
:::

::: {.column width="50%"}
```{r, fig.align='center', out.width="100%"}
knitr::include_graphics("images/TanhFunction.png")
```
:::
:::

## The ReLU {.smaller}

::: columns
::: {.column width="50%"}
-   *rectified linear unit*: $f(z)=\max\{0,z\}$.

-   Close to a linear: piece-wise *linear* function with two linear pieces.

-   Outputs are in %(0,\infty)\$ , thus not bounded

-   Half rectified: activation threshold at 0

-   No vanishing gradient problem
:::

::: {.column width="50%"}
```{r, fig.align='center', out.width="100%"}
knitr::include_graphics("images/ReLUFunction.png")
```
:::
:::

## More activation functions

![](images/ActivationFunctions.png){width="100%"}.

## Putting it all together

```{r, fig.align='center', out.width="100%"}
knitr::include_graphics("images/ArtificialNeuron.png")
```

## In words {.smaller}

- An ANN takes a vector of input values $x_{1}, \ldots, x_{d}$ and combines it with some weights that are local to the neuron $\left(w_{0}, w_{1}, . ., w_{d}\right)$ to compute a net input $w_{0}+\sum_{i=1}^{d} w_{i} \cdot x_{i}$. 

- To compute its output, it then passes the net input through a possibly non-linear univariate activation function $g(\cdot)$, usually vchosen from a set of options such as *Sigmoid*, *Tanh* or *ReLU* functions

- To deal with the  *bias*, we create an extra input variable $x_{0}$ with value always equal to 1 , and so the function computed by a single artificial neuron (parameterized by its weights $\mathbf{w}$ ) is:

$$
y(\mathbf{x})=g\left(w_{0}+\sum_{i=1}^{d} w_{i} x_{i}\right)=g\left(\sum_{i=0}^{d} w_{i} x_{i}\right)=g\left(\mathbf{w}^{\mathbf{T}} \mathbf{x}\right)
$$

# From neurons to neural networks

## The basic neural network {.smaller}

Following with the brain analogy one can combine (artificial) neurons to create better learners.

A simple artificial neural network is usually created by combining two types of modifications to the basic perceptron (AN).

- Stacking several neurons insteads of just one.
- Adding an additional layer of neurons, which is call a *hidden* layer, 

This yields a system where the output of a \emph{neuron} can be the input
of another in many different ways.


## An Artificial Neural network

```{r, fig.align='center', out.width="90%"}
knitr::include_graphics("images/nn.jpg")
```

## The architecture of ANN {.smaller}

In this figure, we have used circles to also denote the inputs to the
network. 

- Circles labeled +1 are *bias units*, and correspond to the intercept term. 

- The leftmost layer of the network is called the *input layer*.

- The rightmost layer of the network is called the *output* layer.

- The middle layer of nodes is called the *hidden layer*, because its values are not observed in the training set.


Bias nodes are not counted when stating the neuron size. 

With all this in mind our example neural network has three layers with:

 - 3 input units (not counting the bias unit), 
 - 3 hidden units,  
 - 1 output unit.

## How an ANN works 

An ANN is a predictive model (a *learner*) whose properties and behaviour  can be well characterized.

<!-- In practice this means: -->

- It operates through a process known as *forward propagation*, which encompasses the  information flow from the input layer to the output layer.

- Forward propagation is performed by composing a series of linear and non-linear (activation) functions.

- These are characterized (parametrized) by their *weights* and *biases*, that need to be *learnt*. 
  - This is done by *training the ANN*.

## Training the ANN {.smaller}

- In order for the ANN to perform well, the training process aims at finding the best possible  parameter values for the learning task defined by the fnctions. This is done by

  - Selecting an appropriate (convex) loss function,
  - Finding those weights that minimize a the total *cost* function (avg. loss).

- This is usually done using some iterative optimization procedure such as *gradient descent*.
  - This requires evaluating derivatives in a huge number of points.
  - Such high number may be reduced by *Stochastic Gradient Descent*.
  - The evaluation of derivatives is simplified thanks to *Backpropagation*.


## Forward propagation {.smaller}

As described above the process that encompasses the computations required to go from the input values to the final output is known as *forward propagation*.

The  weights are combined with the input to produce the final output.

Each node, $a_i^{(2)}$ of the hidden layer opperates on all nodes of the input values

\begin{eqnarray}
a_1^{(2)}&=&f(\theta_{10}^{(1)}+\theta_{11}^{(1)}x_1+\theta_{12}^{(1)}x_2+\theta_{13}^{(1)}x_3)\\
a_2^{(2)}&=&f(\theta_{20}^{(1)}+\theta_{21}^{(1)}x_1+\theta_{22}^{(1)}x_2+\theta_{23}^{(1)}x_3)\\
a_3^{(2)}&=&f(\theta_{30}^{(1)}+\theta_{31}^{(1)}x_1+\theta_{32}^{(1)}x_2+\theta_{33}^{(1)}x_3))
\end{eqnarray}

The output of the hidden layer is transformed through the activation function:

$$
h_{\Theta}(x)=a_1^{(3)}=f(\theta_{10}^{(2)}+\theta_{11}^{(2)}a_1^{(2)}+\theta_{12}^{(2)}a_2^{(2)}+\theta_{13}^{(2)}a_3^{(2)}
$$

## A compact notation (1) {.smaller}

Let  $z_i^{(l)}$ denote the total weighted sum of inputs to unit $i$ in layer $l$:

$$
z_i^{(2)}=\theta_{i0}^{(1)}+\theta_{i1}^{(1)}x_1+\theta_{i2}^{(1)}x_2+\theta_{i3}^{(1)}x_3,
$$ 
the output becomes: $a_i^{(l)}=f(z_i^{(l)})$.


Extending the activation function $f(\cdot)$ to apply elementwise to vectors: 

$$
    f([z_1,z_2,z_3]) = [f(z_1), f(z_2),f(z_3)],
$$
we can write the previous equations more compactly as:

```{=tex}
\begin{eqnarray}
z^{(2)}&=&\Theta^{(1)}x\nonumber\\
a^{(2)}&=&f(z^{(2)})\nonumber\\
z^{(3)}&=&\Theta^{(2)}a^{(2)}\nonumber\\
h_{\Theta}(x)&=&a^{(3)}=f(z^{(3)})\nonumber
\end{eqnarray}
```

## A compact notation (2) 

More generally, recalling that we also use $a^{(1)}=x$ to also  denote the values from the input layer,

Given layer $l$'s activations $a^{(l)}$, we can compute layer   $l+1$'s activations $a^{(l+1)}$ as:


\begin{equation}
z^{(l+1)}=\Theta^{(l)}a^{(l)}
\label{eqforZs}
\end{equation}

\begin{equation}
a^{(l+1)}=f(z^{(l+1)})
\label{eqforAs}
\end{equation}

## A compact notation (3) {.smaller}


This can be used to provide a matrix representation for the weighted sum
of inputs of all neurons:

$$
z^{(l+1)}=
\begin{bmatrix}
z_1^{(l+1)}\\
z_2^{(l+1)}\\
\vdots\\
z_{s_{l+1}}^{(l)}
\end{bmatrix}=
\begin{bmatrix}
\theta_{10}^{(l)}& \theta_{11}^{(l)}&\theta_{12}^{(l)}&...&\theta_{1s_{l}}^{(l)}&\\
\theta_{20}^{(l)}& \theta_{21}^{(l)}&\theta_{22}^{(l)}&...&\theta_{2s_{l}}^{(l)}&\\
\vdots & \vdots& \vdots & \vdots & \vdots\\
\theta_{s_{l+1}0}^{(l)}& \theta_{s_{l+1}1}^{(l)}&\theta_{s_{l+1}2}^{(l)}&...&\theta_{s_{l+1}s_{l}}^{(l)}&\\
\end{bmatrix}
\cdot\begin{bmatrix}
1\\
a_1^{(l)}\\
a_2^{(l)}\\
\vdots\\
a_{s_l}^{(l)}
\end{bmatrix}
$$

## A compact notation (4) {.smaller}

So that, the activation is then:

$$
a^{(l+1)}=
\begin{bmatrix}
a_1^{(l+1)}\\
a_2^{(l+1)}\\
\vdots\\
a_{s_{l+1}}^{(l)}
\end{bmatrix}=f(z^{(l+1)})=\begin{bmatrix}
f(z_1^{(l+1)})\\
f(z_2^{(l+1)})\\
\vdots\\
f(z_{s_{l+1}}^{(l)})
\end{bmatrix}
$$

## Eficient Forward propagation

- The way input data is transformed, through a series of weightings and transformations, until the ouput layer is called   *forward propagation*.

- By organizing parameters in matrices, and using matrix-vector operations, fast linear algebra routines can be used to perform the required calculations in a fast efficent way.


## Multiple architectures for ANN

-   We have so far focused on a single hidden layer neural network of the example. 

- One can. however build neural networks with many distinct   architectures (meaning patterns of connectivity between neurons),
    including ones with multiple hidden layers.

-   See [here the Neural Network
    Zoo](https://www.asimovinstitute.org/neural-network-zoo/).


## Multiple architectures for ANN {.smaller}

:::: {.columns}

::: {.column width='50%'}

- We have so far focused on a single hidden layer neural network of the example

- One can build neural networks with many distinct architectures (meaning patterns of connectivity between neurons), including ones with multiple hidden layers.


:::

::: {.column width='50%'}

![](2.1-Introduction_to_ANN-Slides_insertimage_3.png)
[The Neural Network Zoo](https://www.asimovinstitute.org/neural-network-zoo/)

:::

::::


## Multiple layer dense Networks

-   Most common choice is a $n_l$-layered network:
    -   layer 1 is the input layer,
    -   layer $n_l$ is the output layer,
    -   and each layer $l$ is densely connected to layer $l+1$.
-   In this setting, to compute the output of the network, we can compute all the activations in layer $L_2$, then layer $L_3$, and so on, up to layer $L_{nl}$, using equations seen previously.

## Feed Forward NNs

-   The type of NN described is called feed-forward *neural network (FFNN)*, since
    -   All computations are done by Forward propagation
    -   The connectivity graph does not have any directed loops or cycles.

# Training Neural Networks


## Training an ANN {.smaller}

- An ANN is a predictive model whose properties and behaviour  can be mathematically characterized.

- In practice this means:
  - The ANN acts by composing a series of linear and non-linear (activation) functions.
  - These are characterized by their *weights* and *biases*, that need to be *learnt*
  .
- *Training* the network is done by 
  - Selecting an appropriate (convex) loss function,
  - Finding those weights that minimize a the total *cost* function (avg loss).

## The tools for training

- Training an ANN is usually done using some iterative optimization procedure such as *Gradient Descent*.

- This requires evaluating derivatives in a huge number of points.
    - Such high number may be reduced by *Stochastic Gradient Descent*.
    - The evaluation of derivatives is simplified thanks to *Backpropagation*.

## A loss function for optimization

Depending on the form of the activation function we may decide to use one or another form of loss function.

-   At first an idea may be to use *squared error loss*:
$$
  l(h_\theta(x),y)=\left (y-\frac{1}{1+e^{-\theta^\intercal x}}\right )^2
$$

- However it happens to be that, given a sigmoid activation function, the resulting loss function [* is  not a convex   problem*](https://towardsdatascience.com/why-not-mse-as-a-loss-function-for-logistic-regression-589816b5e03c)  which means that MSE is not appropriate.

- Quadratic loss may be used, for instance, with ReLu activation.

## Illustrating non-convexity {.smaller}

```{r, eval=FALSE, echo=TRUE}
library(plot3D)

# Define the squared error loss function
squared_error <- function(y, y_hat) {
  return(0.5 * (y - y_hat)^2)
}

# Define the logistic activation function
logistic <- function(z) {
  return(1 / (1 + exp(-z)))
}

# Generate data
x <- seq(-10, 10, length.out = 200)
y <- seq(-10, 10, length.out = 200)
z <- outer(x, y, FUN = function(x, y) squared_error(1, logistic(x + y)))

# Plot the exaggerated loss surface
persp3D(x, y, z, theta = 30, phi = 30, expand = 0.5, col = "lightblue",
        ticktype = "detailed", xlab = "Weight 1", ylab = "Weight 2", zlab = "Loss")

```

## Illustrating non-convexity {.smaller}

```{r echo=FALSE}
library(plot3D)

# Define the squared error loss function
squared_error <- function(y, y_hat) {
  return(0.5 * (y - y_hat)^2)
}

# Define the logistic activation function
logistic <- function(z) {
  return(1 / (1 + exp(-z)))
}

# Generate data
x <- seq(-10, 10, length.out = 200)
y <- seq(-10, 10, length.out = 200)
z <- outer(x, y, FUN = function(x, y) squared_error(1, logistic(x + y)))


# Plot the exaggerated loss surface
persp3D(x, y, z, theta = 30, phi = 30, expand = 0.5, col = "lightblue", ticktype = "detailed", xlab = "Weight 1", ylab = "Weight 2", zlab = "Loss")

```


## Cross-entropy loss function {.smaller}

$$
    l(h_\theta(x),y)=\big{\{}\begin{array}{ll}
    -\log h_\theta(x) & \textrm{if }y=1\\
    -\log(1-h_\theta(x))& \textrm{if }y=0
    \end{array}
$$

This function can also be written as:

$$
l(h_\theta(x),y)=-y\log h_\theta(x) - (1-y)\log(1-h_\theta(x))
$$

Using cross-entropy loss, the cost function is of the form:

\begin{eqnarray*}
    J(\theta)=-\frac{1}{n}\big[\sum_{i=1}^n&&(y^{(i)}\log h_\theta(x^{(i)})+\\ &&(1-y^{(i)})\log(1-h_\theta(x^{(i)}))\big]
\end{eqnarray*}

Now, this is a convex optimization problem.

## Regularized cross entropy

In practice we often work with a *regularized version* of the cost function (we don't regularize the bias units)

```{=tex}
\begin{eqnarray*}
J(\Theta)&=&-\frac{1}{n}\big[\sum_{i=1}^n \sum_{k=1}^K y_k^{(i)}\log( h_\theta(x^{(i)}))_k\\
&+&(1-y_k^{(i)})\log(1-(h_\theta(x^{(i)}))_k)\big]\\
&+&\lambda\sum_{l=1}^{L-1}\sum_{i=1}^{s_l}\sum_{j=1}^{s_{l+1}}
(\theta_{ji}^{(l)})^2
\end{eqnarray*}
```


## Gradient Descent 

- Training a network corresponds to choosing the parameters, that is, the weights and biases, that minimize the cost function. 

- The weights and biases take the form of
matrices and vectors, but at this stage it is convenient to imagine them
stored as a single vector that we call $\theta$. 

- Generally, we will suppose $\theta\in\mathbb{R}^p$, and write the cost function as $J(\theta)$ to emphasize its dependence on the parameters. So Cost
$J: \mathbb{R}^p\rightarrow \mathbb{R}$.

## Gradient Descent {.smaller}

![Error hyper-surface](images/errorsurface.jpg){width="60%"}

## Gradient Descent {.smaller}

- *Gradient Descent* is a classical method in optimization that is often
referred to as *steepest descent*.

- Given a function $J(\theta)$ to be minimized, the method proceeds iteratively, computing a sequence of vectors $\theta^1, \theta^2, ..., \theta^n$ in $\mathbb{R}^p$ with the
aim of converging to a vector that minimizes the cost function. 

- Suppose that our current vector is $\theta$. 
*How should we choose a perturbation, $\Delta\theta$, so that the next vector, $\theta+\Delta\theta$, represents an improvement, that is: $J(\theta +\Delta\theta) < J(\theta)$?*


## Gradient Descent {.smaller}

- We proceed by linearization of the cost function using a Taylor approximation

- If $\Delta\theta$ is small, then ignoring terms of order $||\Delta\theta||^2$ or higher:

$$
J(\theta+\Delta\theta)\approx J(\theta)+\sum_{i=1}^p\frac{\partial J(\theta)}{\partial\theta_i}\Delta\theta_i
$$ 

  - where ${\displaystyle \frac{\partial J(\theta)}{\partial\theta_i}}$ denotes the
partial derivative of the cost function with respect to the $i$-th weight. 


## Gradient Descent {.smaller}

- Let $\nabla J(\theta)\in\mathbb{R}^p$
 denote the *gradient*, i.e. the vector of partial derivatives.

\begin{equation}\label{g1}
\nabla J(\theta)=\left(\frac{\partial J(\theta)}{\partial\theta_1},...,\frac{\partial J(\theta)}{\partial\theta_p}\right)^\intercal
\end{equation}

- Now the approximation can be written as:

\begin{equation}\label{g2}
J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^\intercal\Delta\theta
\end{equation}

## Gradient Descent 

- Recalling that our aim is to reduce the value of the cost function, 
- Taylor approximation above motivates the idea of choosing $\Delta\theta$ to *make $\nabla J(\theta)^\intercal\Delta\theta$ negative*, 
- because this will make the value of $J(\theta+\Delta\theta)$ smaller. 

- The bigger in absolute value we can make this negative expression, the smaller will be the value of the cost function.

## The Cauchy-Schwarz inequality

- The [Cauchy-Schwarz inequality](https://en.wikipedia.org/wiki/Cauchy%E2%80%93Schwarz_inequality), states that for any $f,g\in\mathbb{R}^p$, we have:
$$
|f^\intercal g|\leq ||f||\cdot ||g||.
$$ 
-  Moreover, the two sides are equal if and only if $f$ and $g$ are linearly dependent (meaning they are parallel).

## Back to gradient descent 

- How much can $\nabla J(\theta)^\intercal\Delta\theta$ decrease?

- By Cauchy-Schwarz,biggest possible value for $\nabla J(\theta)^\intercal\Delta\theta$ is the upper bound, $||\nabla J(\theta)||\cdot ||\Delta\theta||$.
  - The equality is only reached when $||\nabla J(\theta)||= ||\Delta\theta||$
  - The highest possible negative value will  come out when $-\nabla J(\theta)=\Delta\theta$

- That is, *we should choose $\Delta\theta$ to lie in the direction of $-\nabla J(\theta)$*. 


## Gradient Descent

- Keeping in mind that the Taylor linearization of $J(\theta)$ is an approximation that is relevant only for small $\Delta\theta$, 
- We will limit ourselves to a small step in that direction, defined by the *learning rate*. $\eta$.

- This leads to the update formula that defines the steepest descent method.

\begin{equation}\label{g3}
\theta \rightarrow \theta-\eta\nabla J(\theta)
\end{equation}

## Gradient Descent {.smaller}

In summary, givent a cost function $J(\theta)$ to be optimized the gradient descent optimization proceeds as follows:

1. **Initialize** $\theta_0$ randomly or with some predetermined values
2. **Repeat until convergence:**
    $$
    \theta_{t+1} = \theta_{t} - \eta \nabla J(\theta_{t})
    $$
3. **Stop when:** $|J(\theta_{t+1}) - J(\theta_{t})| < \epsilon$

- $\theta_0$ is the initial parameter vector,
- $\theta_t$ is the parameter vector at iteration $t$,
- $\eta$ is the learning rate,
- $\nabla J(\theta_{t})$ is the gradient of the loss function with respect to $\theta$ at iteration $t$,
- $\epsilon$ is a small positive value indicating the desired level of convergence.

## Computing gradients {.smaller}

- The gradient method provides a way to optimize weights and biases $(\theta =\{W, b\})$ by minimizing the cost function, $J(\theta)$.

- This minimization requires, of course, the computation of an important number of partial derivatives
$$ 
\frac{\partial}{\partial\theta_j}J(\theta)
$$
- The algorithm used to perform these computation is known as the *backpropagation algorithm*

## A short history

- The backpropagation algorithm was originally introduced in the 1970s in a MSc thesis.
- In 1986 a [paper by Rumelhart, Hinton, and Williams](https://www.nature.com/articles/323533a0) describes several neural networks where backpropagation works far faster than earlier approaches to learning.
- This improvement in efficiency made it possible to use neural nets to solve problems which had previously been insoluble. 


<!-- ## Backpropagation intuition {.smaller} -->

<!-- - A term originated in the expression "Error backpropagation -->

<!-- - Recall that the ANN computes its output through  *forward propagation*:  -->

<!--   - It takes the input values -->
<!--   - It applies a series of linear and non-linear  transformations, through the networks layers, to produce the predicted value. -->

<!-- - The output will be different from the known output value in the train set -->
<!--   - The difference (the "error") is used *backwards* to adjust the weights, -->
<!--   - So that the error decreases as we iterate the process. -->



<!-- ## The delta rule {.smaller} -->

<!-- - How can we use the error to update the weights using the gradient method? -->

<!-- - The idea is to use the difference between the *real output* and the predicted output ("$\delta$") to update the weights in such a way that in the next iteration the error decrease. -->

<!--   - This difference is computed first for the layer -->
<!--   - And then is backpropagated to the previous layers until all the weights are recomputed and the process can re-start. -->


## Error backpropagation {.smaller}


$$
\begin{equation*}
\Delta w=\eta(c-y) d \tag{3.21}
\end{equation*}
$$

- La regla delta es la base de la retropropagación:

  - $c$ es el valor que indica la clase en el ejemplo de entrenamiento, 
   
  - $y$ el valor de la función de salida para la instancia $d$, y 
    
  - $\eta$ la tasa o velocidad de aprendizaje (*learning rate*).

- El método utilizado para entrenar este tipo de redes, y basado en la regla delta, se muestra en el algoritmo siguiente (Algoritmo 1).

  - Para que el algoritmo converja es necesario que las clases de los datos sean linealmente separables.
  
## Algoritmo 1 (Gradiente) {.smaller}


- **Entrada**: $W$ (conjunto de vectores de pesos) y $D$ (conjunto de instancias de entrenamiento) 
  - mientras $\left(y \not \approx c_{i} \forall\left(d_{i}, c_{i}\right) \in D\right)$ hacer 
    - para todo $\left(\left(d_{i}, c_{i}\right) \in D\right)$ hacer

      - Calcular la salida $y$ de la red cuando la entrada es $d_{i}$

      - si $\left(y \not \approx c_{i}\right)$ entonces
        - Modificar el vector de pesos $w^{\prime}=w+\eta(c-y) d_{i}$

      - fin si

    - fin para

  - fin mientras

**Devolver**: El conjunto de vectores de pesos $W$


## Retropropagación del error {.smaller}

- En las redes multicapa no podemos aplicar el algoritmo de entrenamiento visto en la sección anterior. 

- El problema aparece con los nodos de las capas ocultas: no podemos saber a priori cuáles son los valores de salida correctos.

- En el caso de una neurona $j$ con función sigmoide, la regla delta es:

$$
\begin{equation*}
\Delta w_{i}^{j}=\eta \sigma^{\prime}\left(z^{j}\right)\left(c^{j}-y^{j}\right) x_{i}^{j}, \tag{3.24}
\end{equation*}
$$

donde:

- $\sigma^{\prime}\left(z^{j}\right)$ indica la pendiente (derivada) de la función sigmoide, que representa el factor con que el nodo $j$ puede afectar al error. 


## Ajustando los pesos {.smaller}

$$
\begin{equation*}
\Delta w_{i}^{j}=\eta \sigma^{\prime}\left(z^{j}\right)\left(c^{j}-y^{j}\right) x_{i}^{j}, \tag{3.24}
\end{equation*}
$$

- Si el valor de $\sigma^{\prime}\left(z^{j}\right)$ es pequeño, nos encontramos en los extremos de la función, donde los cambios no afectan demasiado a la salida. 
- Por el contrario, si el valor es grande, nos encontramos en el centro de la función, donde pequeñas variaciones pueden alterar considerablemente la salida.
  - $\left(c^{j}-y^{j}\right)$ representa la medida del error que se produce en la neurona $j$.
  - $x_{i}^{j}$ indica la *responsabilidad* de la entrada $i$ de la neurona $j$ en el error. 

- Cuando este valor es igual a cero, no se modifica el peso, mientras que si es superior a cero, se modifica proporcionalmente a este valor.

## Expresion general de $delta$ {.smaller}

- La $\delta$ que corresponde a la neurona $j$ puede expresarse de forma general para toda neurona, simplificando la notación anterior:

$$
\begin{gather*}
\delta^{j}=\sigma^{\prime}\left(z^{j}\right)\left(c^{j}-y^{j}\right)  \tag{3.25}\\
\Delta w_{i}^{j}=\eta \delta^{j} x_{i}^{j} \tag{3.26}
\end{gather*}
$$

- A partir de aquí debemos determinar qué parte del error total se asigna a cada una de las neuronas de las capas ocultas. 

- Es decir, debemos definir *cómo modificar los pesos y tasas de aprendizaje de las neuronas de las capas ocultas a partir del error observado en la capa de salida*.

## La retropropagación {.smaller}

- El método de retropropagación (backpropagation) se basa en un esquema general de dos pasos:

1. Propagación hacia adelante (*feedforward*), que consiste en introducir una instancia de entrenamiento y obtener la salida de la red neuronal.

2. Propagación hacia atrás (*backpropagation*), que consiste en calcular el error cometido en la capa de salida y propagarlo hacia atrás para calcular los valores delta de las neuronas de las capas ocultas.

- La idea que subyace a este método es relativamente sencilla, y se basa en *propagar el error de forma proporcional a la influencia que ha tenido cada nodo de las capas ocultas en el error final producido por cada una de las neuronas de la capa de salida*.


## El algoritmo (1) {.smaller}

1. El primer paso, que es la propagación hacia adelante, consiste en aplicar el ejemplo a la red y obtener los valores de salida (línea 3 ).

2. A continuación, el método inicia la propagación hacia atrás, empezando por la capa de salida. Para cada neurona $j$ de la capa de salida:

  2.1 Se calcula, en primera instancia, el valor $\delta^{j}$ basado en el valor de salida de la red para la neurona $j$ $\left(y^{j}\right)$, el valor de la clase de la instancia $\left(c^{j}\right)$ y la derivada de la función sigmoide $\left(\sigma^{\prime}\left(z^{j}\right)\right)$ (línea 5).

  2.2 A continuación, se modifica el vector de pesos de la neurona de la capa de salida, a partir de la tasa de aprendizaje $(\eta)$, el valor delta de la neurona calculado en el paso anterior $\left(\delta^{j}\right)$ y el factor $x_{i}^{j}$ que indica la responsabilidad de la entrada $i$ de la neurona $j$ en el error (línea 6 ).

## El algoritmo (2) {.smaller}

3. Finalmente, la propagación hacia atrás se aplica a las capas ocultas de la red. Para cada neurona $k$ de las capas ocultas:

  3.1 En primer lugar, se calcula el valor $\delta^{k}$ basado en la derivada de la función sigmoide $\left(\sigma^{\prime}\left(z^{k}\right)\right)$ y el sumatorio del producto de la delta calculada en el paso anterior $\left(\delta^{k}\right)$ por el valor $w_{k}^{j}$, que indica el peso de la conexión entre la neurona $k$ y la neurona $j$ (línea 9). 
  El conjunto $S_{k}$ está formado por todos los nodos de salida a los que se encuentra conectada la neurona $k$.

  3.2  En el último paso de la iteración, se modifica el vector de pesos de la neurona $k$, a partir de la tasa de aprendizaje $(\eta)$, el valor delta de la neurona calculado en el paso anterior $\left(\delta^{k}\right)$ y el factor $x_{i}^{k}$ que indica la responsabilidad de la entrada $k$ de la neurona $j$ en el error (línea 10).

## El algoritmo


```{r out.width="150%", fig.align='center'}
knitr::include_graphics("2.1-Introduction_to_ANN-Slides_insertimage_4.png")
```

## Forward propagation example

```{r echo=FALSE}
Top = 100
Left = 50
Width =800
Height = 400
```


::: {.r-stack}

![](images/img01.gif){.fragment .absolute top=100 left=50 width="800" height="400" }

![](images/img02.gif){.fragment .absolute top=100 left=50 width="800" height="400" }

![](images/img03.gif){.fragment .absolute top=100 left=50 width="800" height="400" }

![](images/img04.gif){.fragment .absolute top=100 left=50 width="800" height="400" }

![](images/img05.gif){.fragment .absolute top=100 left=50 width="800" height="400" }

![](images/img06.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

![](images/img07.gif){.fragment .absolute top=100 left=50 width="800" height="400" }

![](images/img08.gif){.fragment .absolute top=100 left=50 width="800" height="400" }
:::



## Backpropagation illustrated

![](images/img08.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

::: {.r-stack}

![](images/img09.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

![](images/img10.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

![](images/img11.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

![](images/img12.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

![](images/img13.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

![](images/img14.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

![](images/img15.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

![](images/img16.gif){.fragment .absolute top=100 left=50 width="800" height="400"}

![](images/img17.gif){.fragment .absolute top=100 left=50 width="900" height="500"}

![](images/img18.gif){.fragment .absolute top=100 left=50 width="900" height="500"}

![](images/img19.gif){.fragment .absolute top=100 left=50 width="900" height="500"}

:::



# Improving the learning process

## Learning optimization

- The learning porocess such as it has been derived may be improved in different ways.
  
  - Predictions can be be bad and require improvement.
  - Computations may be inefficent or slow.
  - The network may overfit and lack generalizability.

- This can be partially soved applying distinct approaches.

## Network architechture

- Network performance is affected by many hyperparameters

  - Network topology
  - Number of layers
  - Number of neurons per layer
  - Activation function(s)
  - Weights initialization procedure
  - etc.
  

## Hyperparameter tuning

- Hyperparameters selection and tuning may be hard, due simply to dimensionality.
- Standard approaches to search for best parameters combinations are used.

```{r fig.align='center'}
knitr::include_graphics("images/GridVsRandomSearch.png")
```

## How many (hidden) layers

- Traditionally considered that one layer may be enough

  - *Shallow Networks*
  
- Posterior research showed that adding more layers increases efficency
  - Number of neurons per layer decreases exponentially
- Although there is also risk of overfitting


## Epochs and iterations

- It has been shown that using the whole training set only once may not be enough for training an ANN.

- One iteration of the training set is known as an *epoch*.

- The number of epochs $N_E$, defines how many times we iterate along the whole training set.

- $N_E$ can be fixed, determined by cross-validation or left open and stop the training when it does not improve anymore.


## Iterations and batches

- A complementary strategy to increasing the number of epochs is decreasing the number of instances in each iteration.

- That is, the training set is broken in a number of *batches* that are trained separately.

  - Batch learning allows weights to be updated more frequently per epoch.
  
  - The advantage of batch learning is related to the gradient descent approach used.
  
## Training in batches

```{r out.width="150%", fig.align='center'}
knitr::include_graphics("2.1-Introduction_to_ANN-Slides_insertimage_5.png")
```


## Gradient descent drawbacks

-   With many parameters and many training points, computing the gradient vector at every iteration of the steepest descent method can be time consuming.
    - It is mainly due to that we have *to sum across all training points*.
    - With big data this becomes prohibitive.

- Several alternatives exist (many indeed).

## Stochastic Gradient {.smaller}

-   A much cheaper alternative is to replace the mean of the individual gradients over all training points *by the gradient at a single, randomly chosen, point*.

-   This leads to the simplest form of the *stochastic gradient method*:
  -   Choose an integer $i$ uniformly at random from $\{1,...,n\}$ and update \begin{equation}\label{g4}
    \theta_j=\theta_j-\eta\frac{\partial}{\partial\theta_j}J(\theta;x^{(i)})
    \end{equation}

-   $x^{(i)}$ incuded in the notation of $J(\theta;x^{(i)})$ to remark the dependence.

## Rationale for SGD  {.smaller}

- At each step, the SGD method uses one randomly chosen training point to represent the full training set.
-   As the iteration proceeds, the method sees more training points.
-   So *there is some hope* that this dramatic reduction in cost-per-iteration will be worthwhile overall.
-   Note that, even for very small $\eta$, the update is not guaranteed to reduce the overall cost function because we traded the mean for a single sample.
  -  Hence, although the phrase stochastic gradient descent is widely used, we prefer to use **stochastic gradient**.

## Batch Gradient Descent

- Batch Gradient Descent computes the error for each  example in the training dataset but the model is updated only after evaluating all examples.

- As a benefits there is computational efficiency, which produces a stable error gradient and a stable convergence. 

- As drawbacks, the stable error gradient can sometimes result in a state of convergence that isn’t the best the model can achieve. It also requires the entire training dataset to be in memory and available to the algorithm.


## Mini-Batch GD

- Mini-batch Gradient Descent (MBGD) divides the training dataset into small batches to compute error and update model coefficients.

- MBGD balances the robustness of stochastic gradient descent with the efficiency of batch gradient descent, making it a prevalent implementation in deep learning.

- A common batch size for MBGD is 32, although it can vary based on the specific problem and data.


## Types of Gradient Descent

```{r out.width="150%", fig.align='center'}
knitr::include_graphics("images/GradientDescentTypes.png")
```
[Source: Gradient Descent and its Types](https://www.analyticsvidhya.com/blog/2022/07/gradient-descent-and-its-types/)

## Alternative optimizers

- Gradient descent can be improved adopting different strategies

- Other optimizers exist that may make the training faster with strategies such as accumulating velocity in relevant directions or adjusting learning rates based on parameter frequency.

-  Overall these optimizers enable quicker convergence and better handling of various data characteristics, making them favorable choices over traditional gradient descent.

## Momentum

  - Accelerates SGD by accumulating momentum in relevant directions, akin to a rolling ball gaining speed downhill.
  - Increases acceleration for dimensions with consistent gradients, reducing updates for changing gradients, leading to faster convergence and reduced oscillation.

## Adagrad

  - Adapts learning rate based on parameter frequency, making smaller updates for frequent features and larger updates for rare ones, suitable for sparse data.
  - Efficient for handling sparse data due to its adaptive learning rates.

## Adadelta

  - An extension of Adagrad aiming to mitigate its aggressive, monotonically decreasing learning rate.
  - Restricts accumulated past gradients to a fixed-size window, avoiding over-accumulation.

## Adam

  - Combines momentum optimization with adaptive learning rates, tracking both past gradients' exponential decay and a moving average of gradients.
  - Recent studies suggest caution with adaptive optimization methods like Adam due to potential generalization issues, prompting exploration of alternatives such as Momentum optimization.

## Optimizing the training speed

- Training speed optimization can be based on

  - Weight inizialization
  
  - Changing learning rate
  
  - Using efficient cost functions
  
## Optimizing to avoid overfitting

In order to fight overfitting distinct strategies are common

- L2 regularization

- Early stopping

- Dropout

- Data augmentation

## Summarizing optimization

![Source: Introducción al Deep Learning (UOC)](2.1-Introduction_to_ANN-Slides_insertimage_18.png)

# An example using R

## A predictive ANN

We use the `neuralnet` package to build a simple neural network to predict if a type of stock pays dividends or not.

```{r echo=TRUE}
if (!require(neuralnet)) 
  install.packages("neuralnet", dep=TRUE)
```

## Data for the example

And use the `dividendinfo.csv` dataset from <https://github.com/MGCodesandStats/datasets>

```{r echo=TRUE}
mydata <- read.csv("https://raw.githubusercontent.com/MGCodesandStats/datasets/master/dividendinfo.csv")
str(mydata)
```

## Data pre-processing

```{r echo=TRUE}
normalize <- function(x) {
  return ((x - min(x)) / (max(x) - min(x)))
}
normData <- as.data.frame(lapply(mydata, normalize))
```

## Test and training sets

Finally we break our data in a test and a training set:

```{r echo=TRUE}
perc2Train <- 2/3
ssize <- nrow(normData)
set.seed(12345)
data_rows <- floor(perc2Train *ssize)
train_indices <- sample(c(1:ssize), data_rows)
trainset <- normData[train_indices,]
testset <- normData[-train_indices,]
```

## Training a neural network

We train a simple NN with two hidden layers, with 4 and 2 neurons respectively.

```{r echo=TRUE}
#Neural Network
library(neuralnet)
nn <- neuralnet(dividend ~ fcfps + earnings_growth + de + mcap + current_ratio, 
                data=trainset, 
                hidden=c(2,1), 
                linear.output=FALSE, 
                threshold=0.01)
```

## Network plot

The output of the procedure is a neural network with estimated weights

```{r echo=TRUE}
plot(nn, rep = "best")
```

## Predictions

```{r echo=TRUE}
temp_test <- subset(testset, select =
                      c("fcfps","earnings_growth", 
                        "de", "mcap", "current_ratio"))
nn.results <- compute(nn, temp_test)
results <- data.frame(actual = 
                  testset$dividend, 
                  prediction = nn.results$net.result)
head(results)
```

## Model evaluation

```{r echo=TRUE}
roundedresults<-sapply(results,round,digits=0)
roundedresultsdf=data.frame(roundedresults)
attach(roundedresultsdf)
table(actual,prediction)
```


# References and Resources