.. role:: python(code)
   :language: python


.. index::
    regularization
    over-fitting

..
    `  <>`

.. _`sec:regularization`:

Regularization
--------------

Neural networks tend to over-fit thanks to their large number of degrees of freedom. 
Over-fitting occurs when the network learns the details and noise of the training data instead of capturing the underlying relationships. 
In an over-fitting situation, when the depth or the number of nodes per layer is increased:

* the training error continues to decrease;
* the validation error eventually increases.

The model performs well on the training examples but poorly on unseen data.

Therefore, a regularization is usually used. 
The objective of regularization is to improve the generalization capability of the model and reduce over-fitting. 
Regularization discourages overly complex solutions and limits excessively large parameter values.

An hyperparameter, the regularization weight :math:`\lambda_r`, is adjusted to maximize the performance of the network on the **validation** data. 
The training data is not used to adjust this parameter because it would always lead to an optimal value of zero 
(which would not help to not over-fit and improve generalization).


The regular weights and biases are trained using a fixed value of the regularization weight to obtain an optimal prediction of the training data. 
Then, the network generalization capability is evaluated on the validation data. 
Imagine that the network dramatically over-fit the training data; it will lead to an unsatisfying prediction accuracy on the validation data. 
The idea is in this case to increase the regularization weight.  

The entire process is repeated until the ideal regularization value is found, i.e. the value which gives an optimal prediction on the validation data. 
At the end of this hyperparameter tuning process, the accuracy of the prediction on the **test** data is checked. 

The simplest regularization is done by penalizing the values of weights and biases and adding the regularization cost :math:`J_r` to the initial cost:

.. math::
    :label: eq:regularization

    J_r = \frac{\lambda_r}{2} \left( \sum _\textrm{all} w^2 + \sum _\textrm{all} b^2 \right)


The gradient components are for any weight or bias:

.. math::
    :label: eq:regularization_gradient

    \cfrac{\partial J}{\partial w} = \lambda_r w

    \cfrac{\partial J}{\partial b} = \lambda_r b