.. role:: python(code)
   :language: python

..
    ` <>`  


.. index::
   deep learning; gradient
   deep learning; fit
   gradient
   gradient descent
   stochastic gradient descent
   deep learning; stochastic gradient descent
   deep learning; test set
   deep learning; train set
   deep learning; validation set


.. _`sec:gradient_descent`:

Training
--------

Gradient descent
^^^^^^^^^^^^^^^^

Both the regression and classification problems lead to the minimization of the loss function as a function of the weights and biases. 
The gradient vector of the loss is orthogonal to the loss isovalue lines and points towards the larger values.  
On a topography map, the gradient at a given location points towards the steepest slope direction towards the top of the hill. 
The gradient vector norm is a measure of this slope.

The gradient descent method is a very simple iterative minimization method which enables, step by step (from :math:`(k)` to :math:`(k+1)`), 
to advance in the opposite direction of the gradient  in order to reach the minimum value of a function. 
The value of the learning rate :math:`\alpha`  is used to regulate the amplitude of the step.  
The algorithm stops when the advance between two steps is too small 
(which indicates the proximity of a minimum where :math:`\nabla_\theta J=0` ).

.. math::
    :label: eq:gradient_descent

     \begin{cases}
        \theta^{(0)} & \text{starting point} \\
        \theta^{(k+1)} = \theta^{(k)} - \alpha \nabla_{\theta} J(\theta^{(k)}) & \text{progression} \\
        \Vert \alpha \nabla_{\theta} J(\theta^{(k)})\Vert_2 < \epsilon & \text{stopping criteria}
    \end{cases}   



Figure :ref:`fig:gradient_example` illustrates how the gradient descent algorithm would progress 
if a loss function for 2 parameters was plotted as a function of the parameter values. 
From a starting point, the algorithm progresses to the next parameter vector value using the local maximum slope.  
In this example, only the isovalue lines of the loss function at the different steps were plotted. 

.. _`fig:gradient_example`: 

.. figure:: gradient_example.png 
    :width: 400
    :alt: gradient_example

    Gradient example for only two parameters



In practice, the gradients are efficiently computed using the back-propagation algorithm, 
which applies the chain rule through all layers of the network.


Stochastic gradient descent
^^^^^^^^^^^^^^^^^^^^^^^^^^^

The loss function is a measure of the proximity of the estimations and the labels for all the training examples. 
However, the gradient descent is an iterative algorithm, a good approximation of the gradient at each step is sufficient to converge.  
The idea of the **stochastic** gradient descent is, rather than using all the training examples every time the gradient is calculated, 
it would be much more efficient to only approximate the gradient by using a subset of the training examples. 
If the training example are randomly sampled in the training database, the calculated gradient will be a satisfying approximation. 
Usually, the training database is initially shuffled, and used chunk by chunk until all the examples have been consumed. 
Then, one loops over to the first chunk and so on. 
The term **epoch** designates how many times the entire database has been entirely scanned. 
The chunk size and maximum epoch are two hyper-parameters of the solver.


Data set used
^^^^^^^^^^^^^

The training set is used to optimize the network parameters. 
The gradients of the cost function are computed from these data, and the parameters are updated accordingly using the type of algorithm describe above. 
The validation set is used to evaluate different choices of hyper-parameters. 
The validation set must NOT be used to update the network parameters. 
It provides an unbiased estimate of the performance associated with each hyper-parameter configuration. 
The test set is used only after all model choices have been finalized. 
It provides an independent evaluation of the generalization capability of the trained network.