.. role:: python(code) :language: python .. index:: deep learning neural network supervised learning deep learning label training validation test deep learning; test regression classification classification; binary classification; multi-class model parameters model parameters; weights model parameters; biases hyper-parameters activation function loss deep learning; cost deep learning; loss gradient gradient descent back-propagation gradient; back-propagation over-fitting generalization regularization .. `Examples regression and multi-class classification ` `Number of parameters in a FNN ` .. _`sec:context_dl`: Context ------- A **forward neural network** (FNN) is a parametrized function taking a vector and spitting another vector (potentially of different length). Its **parameters**, generally noted :math:`\theta`, are adjustable. This flexibility is convenient in the context of **supervised** machine learning, here **deep learning**. Supervised learning is when the estimation algorithm, here the FNN, is trained on **labeled** data. Each **training** example is a collection of **features** (numerical values in the **input** vector usually noted :math:`x`) associated with the **output target**: the label usually noted :math:`y`. Two tasks are commonly undertaken: either **regression** or **classification**. A regression means that the label is a vector of real numbers, whereas a classification means that the label is a discrete category. Two flavors are encountered for a classification, either a **binary** classification where the label is category 0 or category 1, or a **multi-class** classification where the label can take more than two values. The goal is to determine the best model parameters: the parameters which enable the model to obtain the best estimations possible on the training set. A model is just a function from the feature space to the label space. The features are combined in the model with the parameters in order to make a prediction of the label. .. admonition:: Exercise (easy) Give an example of a regression and a multi-class classification. .. _`ex:example_regression_classification`: .. collapse:: Correction Regression: for example, fitting a linear model through measurements of a tensile strength as a function of a deformation. The input is the deformation, the label is the stress, the parameter is the Young modulus. Multi-class classification: for example, categorizing a system in three class 'no action required', 'to be monitored closely', 'to be replaced' as a function of a vibration level. The input is the vibration level, the label is one of the three categories. To quantify the proximity between the estimation and the label, a **loss** or **cost** function is defined and usually noted :math:`J`. Its definition differs if the task is a regression or a classification. Then the 'best parameters' are found by minimizing the loss function. This **minimization** step is what the training / learning is. It is relatively straightforward to compute the **gradient** of the loss function with respect to the model parameters, :math:`\nabla_\theta J`, using the **back-propagation** algorithm. This gradient is then used in the minimization in the **gradient descent** algorithm. The overall goal is not just to find the best parameters on the training data, the goal is to make sure that the model also **generalizes** to examples which were not part of the training data. The other term which is used is to avoid **over-fitting**. This independent data set (not part of the training) is called the **test** data. The performance of the FNN on the test data are the only trustworthy metric of the prediction quality. The internal structure of a FNN is flexible especially the number of parameters. This number of parameters, controlled by the network depth and / or the number of nodes per layer, is a prime example of an **hyper-parameter** (the hyper-parameters are chosen by the user before training and are not directly optimized by the learning algorithm). To optimize the hyper-parameters, such as the FNN structure, a data set independent of both the training and the test data is required, it is called the **validation** data. Finally, by taking advantage of the derivability of the FNN, the loss function can be advantageously tweaked. First, a **regularization** can be introduced in order to limit over-fitting (the regularization coefficient is another example of a hyper-parameter). Second, since the partial derivative of the output coefficients with respect to the input components can be calculated, it opens the possibility to incentivize the FNN to respect differential equations. This is the foundation of **physics informed neural networks** (PINN). A very nice introduction to FNN can be found in :cite:`3blue1brown2017_NN`. .. _`sec:structure_dl`: Structure --------- The structure of a basic FNN consists of layers. Every layer is usually represented as a collection of nodes. Each node of a layer is connected to every node of the next layer. In the :ref:`figure ` below, there is a single **hidden layer** (between the input :math:`x` and the output layer). The value of each node is calculated in two steps, first, a linear combination of every node of the previous layer plus a constant, then, a non-linear function :math:`\sigma`: For example :math:`a_j`: .. math:: :label: eq:FNN_structure z_j &= b_j + \sum_k w_{jk} \cdot a_k a_j &= \sigma(z_j) The :math:`w` parameters are the **weights**; the :math:`b` parameters are the **biases**; the :math:`\sigma` functions are the **activation** functions. .. _`fig:FNN_structure`: .. figure:: fnn_structure.png :width: 400 :alt: FNN_structure FNN structure Examples of hyper-parameters associated with the structure include: * the number of hidden layers; * the number of neurons per layer; * the choice of activation functions; .. admonition:: Exercise (easy) Assuming the network graph above represents all the nodes and layers. How many weights and biases must be optimized? .. _`ex:nb_parameters`: .. collapse:: Correction 3 x 3 x 2 = 18 weights 2 x 3 = 6 biases