Loss

The purpose of a loss (also called a cost) is to quantify the discrepancy between the predictions of a neural network and the expected outputs. During training, the objective is to minimize this quantity by adjusting the model parameters.

Regression

In a regression problem, the target variable is continuous. A common choice of cost function is the sum of squared errors, also called the mean squared error (MSE):

(1)\[J(\theta) = \frac{1}{n_t} \sum_{t=0}^{n_t-1} \cfrac{1}{2} \sum_{j=0}^{n_j-1} \left( y_{jt} - \hat{y}_{jt} \right)^2\]

where:

  • \(n_t\) is the number of training samples,

  • \(y_{jt}\) are the label component \(j\) for the training example \(t\),

  • \(\hat{y}_{jt}\) are the prediction of the network,

  • \(\theta\) denotes all trainable parameters (weights and biases).

Binary classification

In binary classification, the single output represents the probability that an input belongs to one of two classes. The network output is therefore typically constrained to the interval \([0,1]\) by a logistic activation function in the output layer.

The appropriate cost function is the binary cross-entropy:

(2)\[J(\theta) = -\frac{1}{n_t} \sum_{t=0}^{n_t-1} \left[ y_{t} \ln(\hat{y}_{t}) + (1-y_{t})\ln(1-\hat{y}_{t}) \right]\]

The motivation for cross-entropy comes from probability theory. The network output \(\hat{y}_{t}\) represents the estimated probability of observing the correct class for the training example \(t\). Minimizing the cross-entropy is equivalent to maximizing the log-likelihood of the training data under the model. This cost function strongly penalizes confident but incorrect predictions. For example, assigning a probability close to zero to the correct class produces a very large penalty, encouraging the network to learn reliable probability estimates.

Multi-class classification

In multi-class classification, each sample belongs to one among \(K\) possible classes. The corresponding categorical cross-entropy is:

(3)\[J(\theta) = -\frac{1}{n_t} \sum_{t=0}^{n_t-1} \sum_{j=0}^{K-1} y_{jt} \ln(\hat{y}_{jt})\]

where \(y_{jt}\) equals one if sample \(t\) belongs to class \(j\), and zero otherwise. This strategy is called one-hot encoding where only one component equals 1:

(4)\[y = (0,\ldots,0,1,0,\ldots,0).\]

One may wonder why there is no term \((1-y_{jt}) \ln(1-\hat{y}_{jt})\) in order to penalize false positives. The fact that all the output values are coupled through the softmax function, means that if you incentivize the true positive, you mechanically decrease the other incorrect answers (the sum of all outputs is always 1).

Exercise (easy)

Implement the method mean_accuracy and the two functions is_correct_class_single (indicates if the most probable class is the true one) and mean_square_loss_single.

src
 1    def mean_accuracy(
 2        self,
 3        inputs: list[Vector],
 4        targets: list[Vector],
 5        metric_single_example: Callable[[Vector, Vector], float | bool],
 6    ) -> float:
 7        """with a given metric callable on every pair (prediction, target)
 8        which gives either a measure of their distance or a measure of exact match,
 9        this method computes a mean accuracy of the FNN over a train or test set."""
10        ...
src
1def is_correct_class_single(prediction: Vector, target: Vector) -> bool: ...
src
1def mean_square_loss_single(prediction: Vector, target: Vector) -> float: ...
Correction
src
 1    def mean_accuracy(
 2        self,
 3        inputs: list[Vector],
 4        targets: list[Vector],
 5        metric_single_example: Callable[[Vector, Vector], float | bool],
 6    ) -> float:
 7        """with a given metric callable on every pair (prediction, target)
 8        which gives either a measure of their distance or a measure of exact match,
 9        this method computes a mean accuracy of the FNN over a train or test set."""
10        residual: float = 0.0
11        for input_, target in zip(inputs, targets):
12            prediction: Vector = self.infer(input_)
13            residual += metric_single_example(prediction, target)
14        return residual / len(inputs)
src
1def is_correct_class_single(prediction: Vector, target: Vector) -> bool:
2    """is the most probable estimated class the correct one?"""
3    return np.argmax(prediction) == np.argmax(target)
src
1def mean_square_loss_single(prediction: Vector, target: Vector) -> float:
2    return float(np.linalg.norm(prediction - target) ** 2 / target.size)