Activation

An activation function introduces nonlinearity into the model. Without nonlinear activations, multiple layers would be equivalent to a single linear transformation. Common activation functions include:

Rectified linear

The rectified linear function is defined by:

(1)\[\begin{split}\textrm{ReLU}(x) = \begin{cases} 0 & \text{if } x \leq 0\\ x & \text{otherwise} \end{cases}\end{split}\]

It is widely used because of its simplicity and computational efficiency.

Derivative:

(2)\[\begin{split}\mathrm{ReLU}'(x)= \begin{cases} 1 &x>0 \\ 0 &x\leq 0 \end{cases}\end{split}\]

Logistic

(3)\[\sigma(x)=\frac{1}{1+e^{-x}}.\]

It maps values into the interval \([0,1]\) and is commonly used in binary classification outputs.

Derivative:

(4)\[\sigma'(x)=\sigma(x)(1-\sigma(x)).\]

Hyperbolic tangent

(5)\[\tanh(x)= \frac{e^x-e^{-x}}{e^x+e^{-x}}.\]

Its outputs lie in \([-1,1]\).

Derivative:

(6)\[\tanh'(x)=1-\tanh^2(x).\]

Softmax

The softmax activation function transforms a vector into a probability distribution and is typically used for multi-class classification. For multi-class classification, the last layer activation function converts these values into probability-like numbers (numbers between \(0\) and \(1\) which add up to \(1\)). The softmax function is used to this purpose.

(7)\[ a_j = \cfrac{e^{z_j}}{\sum _{i=0}^{n_j-1} e^{z_i}}\]

Derivative:

(8)\[\begin{split}\cfrac{\partial a_j}{\partial z_j} &= a_j \cdot (1 - a_j) \\ \cfrac{\partial a_j}{\partial z_i} &= -a_i \cdot a_j ~~ \textrm{for} ~ i \neq j\end{split}\]

Output layer

The choice of activation function in the output layer depends on the problem:

  • Regression: no activation (linear output), allowing arbitrary real values.

  • Binary classification: logistic activation.

  • Multi-class classification: softmax activation.

Exercise (easy)

  1. Implement the linear, Relu and softmax activation functions and their derivative (only the first for softmax).

  2. Implement the suggested methods of the Layer class (the input is not a layer; a layer is all the hidden layers and the output layer).

  3. Implement the suggested methods of the FNN class.

src
  1"""Forward Neural Network implementation from scratch using numpy.
  2Three sections:
  3- the vectorized activation functions and their derivative
  4- the Layer class to be used in the FNN class
  5- the FNN class"""
  6
  7from dataclasses import dataclass, field
  8from collections.abc import Callable, Sequence
  9
 10import numpy as np
 11import numpy.typing as npt
 12
 13FLOAT = np.float64
 14type Array2D = npt.NDArray[FLOAT]
 15type Vector = npt.NDArray[FLOAT]
 16type FuncVector = Callable[[Vector], Vector]
 17
 18
 19class IncompatibleDimensions(Exception): ...
 20
 21
 22### vectorized activation and their derivative
 23
 24
 25def relu(vector: Vector) -> Vector: ...
 26
 27
 28def relu_derivative(vector: Vector) -> Vector: ...
 29
 30
 31def linear(vector: Vector) -> Vector: ...
 32
 33
 34def linear_derivative(vector: Vector) -> Vector: ...
 35
 36
 37def softmax(vector: Vector) -> Vector: ...
 38
 39
 40def softmax_derivative(vector: Vector) -> Vector: ...
 41
 42
 43_activations_and_derivatives: dict[FuncVector, FuncVector] = {
 44    relu: relu_derivative,
 45    linear: linear_derivative,
 46    softmax: softmax_derivative,
 47}
 48
 49
 50@dataclass
 51class Layer:
 52    """A layer has a certain number of nodes on the left,
 53    and another on the right. Every node on the left is
 54    fully connected to every node on the right.
 55
 56    A layer has the ownership of the nodes on the right only.
 57
 58    The nodes on the right have two states: z then a
 59    - before activation z_i = sum_k w_ik a_k_previous_layer
 60    - affter activation a_i = sigma(z_i)
 61
 62    The weights and biases are initialized randomly,
 63    potentially using a seed for repeatability."""
 64
 65    nb_nodes_left: int
 66    nb_nodes_right: int
 67    seed: int | None = None
 68
 69    def __post_init__(self) -> None: ...
 70
 71    def init_parameters_random(self) -> None: ...
 72
 73
 74@dataclass
 75class FNN:
 76    """A FNN has a sequence of layers. It owns all the hidden layers
 77    and the output layer, not the input layer. However, to build
 78    the first layer, the number of nodes of the input layer must be known.
 79
 80    Every layer has an activation, and its derivative can be retrieved in
 81    the activation_derivatives dict.
 82
 83    The three main methods are:
 84    - infer: forward execution of the FNN from input to output
 85      with the current weights and biases
 86    - fit: learn all the weights and biases on a train set
 87    - mean_accuracy: assess the accuracy of the estimations
 88    """
 89
 90    nb_nodes_per_layer_including_input: Sequence[int]
 91    activations: Sequence[FuncVector]
 92    seed: int | None = None
 93
 94    activation_derivatives: dict[FuncVector, FuncVector] = field(
 95        default_factory=_activations_and_derivatives.copy
 96    )
 97
 98    @staticmethod
 99    def bad_input_or_target_size(expected_size: int, vectors: Sequence[Vector]) -> bool:
100        return any([vector.size != expected_size for vector in vectors])
101
102    def __post_init__(self) -> None:
103        """creates a sequence of layers."""
104        ...
105
106    def infer(self, input_: Vector) -> Vector:
107        """forward sweep across the FNN,
108        z_right = w . a_left
109        a_right = sigma(z_right)
110
111        changes the attributes z and a of every layer."""
112        ...
Correction
src
1def linear(vector: Vector) -> Vector:
2    return vector
src
1def linear_derivative(vector: Vector) -> Vector:
2    return np.ones(vector.shape)
src
1def relu(vector: Vector) -> Vector:
2    result: Vector = np.zeros(vector.shape)
3    is_pos = vector > 0.0
4    result[is_pos] = vector[is_pos]
5    return result
src
1def relu_derivative(vector: Vector) -> Vector:
2    result: Vector = np.zeros(vector.shape)
3    is_pos = vector > 0.0
4    result[is_pos] = 1.0
5    return result
src
1def softmax(vector: Vector) -> Vector:
2    num: Vector = np.exp(vector)
3    return num / sum(num)
src
1def softmax_derivative(vector: Vector) -> Vector:
2    return vector * (1.0 - vector)
src
 1@dataclass
 2class Layer:
 3    """A layer has a certain number of nodes on the left,
 4    and another on the right. Every node on the left is
 5    fully connected to every node on the right.
 6
 7    A layer has the ownership of the nodes on the right only.
 8
 9    The nodes on the right have two states: z then a
10    - before activation z_i = sum_k w_ik a_k_previous_layer
11    - affter activation a_i = sigma(z_i)
12
13    The weights and biases are initialized randomly,
14    potentially using a seed for repeatability."""
15
16    nb_nodes_left: int
17    nb_nodes_right: int
18    seed: int | None = None
19
20    def __post_init__(self) -> None:
21        self.init_parameters_random()
22        self.z: Vector = np.zeros(self.nb_nodes_right, dtype=FLOAT)
23        self.a: Vector = np.zeros(self.nb_nodes_right, dtype=FLOAT)
24
25    def init_parameters_random(self) -> None:
26        rng = np.random.default_rng(self.seed)
27        self.weights: Array2D = (
28            rng.random((self.nb_nodes_right, self.nb_nodes_left), dtype=FLOAT) - 0.5
29        )
30        self.biases: Vector = rng.random(self.nb_nodes_right, dtype=FLOAT) - 0.5
src
 1    def __post_init__(self) -> None:
 2        """creates a sequence of layers."""
 3        self.layers: list[Layer] = []
 4        self.nb_layers: int = len(self.nb_nodes_per_layer_including_input) - 1
 5        if self.nb_layers != len(self.activations):
 6            raise IncompatibleDimensions(
 7                "The number of activation functions is not compatible with the number of nodes."
 8            )
 9
10        for ind in range(self.nb_layers):
11            layer = Layer(
12                self.nb_nodes_per_layer_including_input[ind],
13                self.nb_nodes_per_layer_including_input[ind + 1],
14                self.seed + ind if self.seed is not None else None,
15                # to avoid using the same sequence of random numbers
16                # for every layer which could introduce unwanted symetries
17            )
18            self.layers.append(layer)
src
 1    def infer(self, input_: Vector) -> Vector:
 2        """forward sweep across the FNN,
 3        z_right = w . a_left
 4        a_right = sigma(z_right)
 5
 6        changes the attributes z and a of every layer."""
 7        if self.bad_input_or_target_size(
 8            self.nb_nodes_per_layer_including_input[0], [input_]
 9        ):
10            raise IncompatibleDimensions("The size of the input is incompatible.")
11
12        left: Vector = input_
13        for ind_layer, layer in enumerate(self.layers):
14            layer.z = layer.weights.dot(left) + layer.biases
15            layer.a = self.activations[ind_layer](layer.z)
16            left = layer.a
17        return left

Obviously, for the moment, parameters are random, as a result the inference has real meaning. One can still check that the operation does what it is supposed to.