.. role:: python(code)
   :language: python


.. index::
    loss
    deep learning; loss
    mean square error
    deep learning; mean square error
    cross-entropy
    one-hot encoding

..
    `  <>`


.. _`sec:loss`:

Loss 
----

The purpose of a loss (also called a **cost**) is to quantify the discrepancy between the predictions of a neural network and the expected outputs. 
During training, the objective is to minimize this quantity by adjusting the model parameters.

Regression
^^^^^^^^^^

In a regression problem, the target variable is continuous. 
A common choice of cost function is the **sum of squared errors**, also called the **mean squared error (MSE)**:

.. math::
    :label: eq:loss_MLE

    J(\theta) = \frac{1}{n_t} \sum_{t=0}^{n_t-1} \cfrac{1}{2} \sum_{j=0}^{n_j-1} \left( y_{jt} - \hat{y}_{jt} \right)^2

where:

* :math:`n_t` is the number of training samples,
* :math:`y_{jt}` are the label component :math:`j` for the training example :math:`t`,
* :math:`\hat{y}_{jt}` are the prediction of the network,
* :math:`\theta` denotes all trainable parameters (weights and biases).

Binary classification
^^^^^^^^^^^^^^^^^^^^^

In binary classification, the single output represents the probability that an input belongs to one of two classes. 
The network output is therefore typically constrained to the interval :math:`[0,1]` by a logistic activation function in the output layer.

The appropriate cost function is the **binary cross-entropy**:

.. math::
    :label: eq:loss_binary

    J(\theta) = -\frac{1}{n_t}    \sum_{t=0}^{n_t-1}   \left[    y_{t} \ln(\hat{y}_{t})    +    (1-y_{t})\ln(1-\hat{y}_{t})    \right]

The motivation for **cross-entropy** comes from probability theory. 
The network output :math:`\hat{y}_{t}` represents the estimated probability of observing the correct class for the training example :math:`t`. 
Minimizing the cross-entropy is equivalent to maximizing the **log-likelihood** of the training data under the model. 
This cost function strongly penalizes confident but incorrect predictions. 
For example, assigning a probability close to zero to the correct class produces a very large penalty, 
encouraging the network to learn reliable probability estimates.



Multi-class classification
^^^^^^^^^^^^^^^^^^^^^^^^^^

In multi-class classification, each sample belongs to one among :math:`K` possible classes. 
The corresponding **categorical cross-entropy** is:

.. math::
    :label: eq:loss_multi

    J(\theta) =    -\frac{1}{n_t}    \sum_{t=0}^{n_t-1}    \sum_{j=0}^{K-1}    y_{jt} \ln(\hat{y}_{jt})

where :math:`y_{jt}` equals one if sample :math:`t` belongs to class :math:`j`, and zero otherwise. 
This strategy is called **one-hot encoding** where only one component equals 1:

.. math::
    :label: eq:hot_encoding

    y = (0,\ldots,0,1,0,\ldots,0).

One may wonder why there is no term :math:`(1-y_{jt}) \ln(1-\hat{y}_{jt})` in order to penalize false positives. 
The fact that all the output values are coupled through the softmax function, means that if you incentivize the true positive, 
you mechanically decrease the other incorrect answers (the sum of all outputs is always 1).


.. admonition:: Exercise (easy)

    Implement the method :python:`mean_accuracy` and  the two functions :python:`is_correct_class_single` (indicates if the most 
    probable class is the true one) and :python:`mean_square_loss_single`. 

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn_empty_2.py
        :pyobject: FNN.mean_accuracy
        :name: fnn_mean_accuracy_empty
        :linenos:       
        :caption: src      


    .. literalinclude:: ../../../../../../src/ai/fnn/fnn_empty_2.py
        :pyobject: is_correct_class_single
        :name: is_correct_class_single_empty
        :linenos:       
        :caption: src      


    .. literalinclude:: ../../../../../../src/ai/fnn/fnn_empty_2.py
        :pyobject: mean_square_loss_single
        :name: mean_square_loss_single_empty
        :linenos:       
        :caption: src                      


.. _`ex:mean_accuracy`:        

.. collapse:: Correction

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: FNN.mean_accuracy
        :name: fnn_mean_accuracy
        :linenos:       
        :caption: src      


    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: is_correct_class_single
        :name: is_correct_class_single
        :linenos:       
        :caption: src      


    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: mean_square_loss_single
        :name: mean_square_loss_single
        :linenos:       
        :caption: src