.. role:: python(code)
   :language: python


.. index::
    activation
    deep learning; non-linearity

..
    `Activation functions  <ex:activation>`


.. _`sec:activation_functions`:

Activation
----------

An **activation function** introduces nonlinearity into the model. Without nonlinear activations, multiple layers would be equivalent to a single linear transformation. 
Common activation functions include:


.. _`sec:relu`:

Rectified linear
^^^^^^^^^^^^^^^^

The rectified linear function is defined by: 

.. math::
    :label: eq:relu

    \textrm{ReLU}(x) = \begin{cases}
			0 & \text{if } x \leq 0\\
            x & \text{otherwise}
	\end{cases}

It is widely used because of its simplicity and computational efficiency.

Derivative:

.. math::
    :label: eq:relu_derivative

    \mathrm{ReLU}'(x)=
    \begin{cases}
    1 &x>0 \\
    0 &x\leq 0
    \end{cases}




.. _`sec:softmax`:


Logistic
^^^^^^^^

.. math::
    :label: eq:logistic

    \sigma(x)=\frac{1}{1+e^{-x}}.

It maps values into the interval :math:`[0,1]` and is commonly used in binary classification outputs.

Derivative:

.. math::
    :label: eq:logistic_derivative

    \sigma'(x)=\sigma(x)(1-\sigma(x)).    




Hyperbolic tangent
^^^^^^^^^^^^^^^^^^

.. math::
    :label: eq:tanh

    \tanh(x)=    \frac{e^x-e^{-x}}{e^x+e^{-x}}.

Its outputs lie in :math:`[-1,1]`.

Derivative:

.. math::
    :label: eq:tanh_derivative

    \tanh'(x)=1-\tanh^2(x).    




Softmax
^^^^^^^

The softmax activation function transforms a vector into a probability distribution and is typically used for multi-class classification.
For multi-class classification, the last layer activation function converts these values into probability-like numbers 
(numbers between :math:`0` and :math:`1` which add up to :math:`1`).  
The softmax function is used to this purpose. 

.. math::
   :label: eq:softmax

    a_j = \cfrac{e^{z_j}}{\sum _{i=0}^{n_j-1} e^{z_i}}


Derivative:


.. math::
    :label: eq:softmax_derivatives

    \cfrac{\partial a_j}{\partial z_j} &= a_j \cdot (1 - a_j) \\
    \cfrac{\partial a_j}{\partial z_i} &= -a_i \cdot a_j ~~ \textrm{for} ~ i \neq j


Output layer
^^^^^^^^^^^^

The choice of activation function in the output layer depends on the problem:

* Regression: no activation (linear output), allowing arbitrary real values.
* Binary classification: logistic activation.
* Multi-class classification: softmax activation.


.. admonition:: Exercise (easy)

   1. Implement the linear, Relu and softmax activation functions and their derivative (only the first for softmax).
   2. Implement the suggested methods of the :python:`Layer` class  (the input is not a layer; a layer is all the hidden layers and the output layer).
   3. Implement the suggested methods of the :python:`FNN` class. 

   .. literalinclude:: ../../../../../../src/ai/fnn/fnn_empty_1.py
      :name: fnn_empty_1
      :linenos:       
      :caption: src   

.. _`ex:activation`:

      
.. collapse:: Correction     

    1.

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: linear
        :name: linear
        :linenos:       
        :caption: src  

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: linear_derivative
        :name: linear_derivative
        :linenos:       
        :caption: src 

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: relu
        :name: relu
        :linenos:       
        :caption: src  

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: relu_derivative
        :name: relu_derivative
        :linenos:       
        :caption: src 

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: softmax
        :name: softmax
        :linenos:       
        :caption: src 

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: softmax_derivative
        :name: softmax_derivative
        :linenos:       
        :caption: src 

    2.

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: Layer
        :name: layer
        :linenos:       
        :caption: src      

    3.

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: FNN.__post_init__
        :name: fnn_post_init
        :linenos:       
        :caption: src  

    .. literalinclude:: ../../../../../../src/ai/fnn/fnn.py
        :pyobject: FNN.infer
        :name: infer
        :linenos:       
        :caption: src  

    Obviously, for the moment, parameters are random, as a result the inference has real meaning. 
    One can still check that the operation does what it is supposed to.