.. role:: python(code)
   :language: python

..
    `Output derivative <ex:output_derivative>`   


.. index::
   Forward Neural Network; layer
   Forward Neural Network; back-propagation
   layer
   gradient

.. _`sec:back-prop:simple case`:

Simple case
^^^^^^^^^^^

.. note::
    In this section, indices have no particular meaning, they are just part of the name of a parameter.
   
A single input and output case is first studied. 
A single hidden layer is present. 
The input is :math:`x` ; the ouput is :math:`a_j` ; the target prediction is :math:`y`. 
The figure :ref:`Simple case FNN<fig:simple_fnn>` shows this graph. 
The weights and biases are indicated over the edges. 
The dashed line symbolizes a comparison.

.. _`fig:simple_fnn`:
.. figure:: backprop_simple_case.dot.png 
    :width: 400
    :alt: Simple case FNN

    Simple case FNN

For regression, the cost :math:`J` in equation :eq:`eq:cost_simple_case` represents the agreement between prediction and a single target:

.. math::
   :label: eq:cost_simple_case
   
    J = \frac{1}{2} (a_j - y)^2

A value of the node :math:`a_j` is obtained in two steps from the value of the previous node :math:`a_k`. 
First one obtains the intermediate variable :math:`z_j` using the weight :math:`w_j` and bias :math:`b_j` of the edge leading to the node :math:`a_j`, then activation function :math:`\sigma(z_j)` is used, see equations :eq:`eq:steps_node_value_aj_simple_case`:


.. math::
   :label: eq:steps_node_value_aj_simple_case
   
   z_j &= w_j \cdot a_k + b_j \\
   a_j &= \sigma(z_j)


The derivation chain rule is applied to compute the gradient terms for the last layer, see equations :eq:`eq:chain_rule_simple_case_wj` 
and :eq:`eq:chain_rule_simple_case_bj`:


.. math::
   :label: eq:chain_rule_simple_case_wj

    \cfrac{\partial J}{\partial w_j} &= \left( \cfrac{\partial J}{\partial a_j}\cfrac{\partial a_j}{\partial z_j} \right) \cfrac{\partial z_j}{\partial w_j} \\
    &=   \underbrace{ \left( (a_j - y)\cdot\sigma'(z_j) \right) }_{  \cfrac{\partial J}{\partial z_j} }  \cdot a_k


.. math::
   :label: eq:chain_rule_simple_case_bj

    \cfrac{\partial J}{\partial b_j} &=  \cfrac{\partial J}{\partial z_j} \cfrac{\partial z_j}{\partial b_j} \\
    &= \cfrac{\partial J}{\partial z_j} \cdot 1

To obtain the node :math:`a_k` from node :math:`x`:
 
.. math::
   :label: eq:steps_node_value_ak_simple_case
   
   z_k &= w_k \cdot x + b_k \\
   a_k &= \sigma(z_k)

Similarly, the derivation chain rule is applied to compute the gradient terms for the second to last layer, see equations :eq:`eq:chain_rule_simple_case_wk` and :eq:`eq:chain_rule_simple_case_bk`:


.. math::
   :label: eq:chain_rule_simple_case_wk

    \cfrac{\partial J}{\partial w_k} &=  \cfrac{\partial J}{\partial z_j}  \left( \cfrac{\partial z_j}{\partial a_k}\cfrac{\partial a_k}{\partial z_k}  \right) \cfrac{\partial z_k}{\partial w_k} \\
    &= \underbrace{\cfrac{\partial J}{\partial z_j} \cdot \left( w_j \cdot\sigma'(z_k) \right) }_{  \cfrac{\partial J}{\partial z_k} } \cdot  x 


.. math::
   :label: eq:chain_rule_simple_case_bk

    \cfrac{\partial J}{\partial b_k} &=  \cfrac{\partial J}{\partial z_k} \cfrac{\partial z_k}{\partial b_k} \\
    &=  \cfrac{\partial J}{\partial z_k} \cdot 1

The method should seem straightforward. 
Start from the last node, compute :math:`\cfrac{\partial J}{\partial z_j}` and save it for the second to last node calculation. 
Repeat the process for :math:`\cfrac{\partial J}{\partial z_k}` and save it for the third to last node, and so on. 

.. note::

    The training is not performed using a single data :math:`(x, y)`, but many. 
    The gradient is obtained by averaging over the individual gradients.


.. admonition:: Exercise (easy)

    How would compute :math:`\cfrac{\partial a_j}{\partial x}` (the derivative of the output with respect to the input)?
    What happens when only linear activations are used?

.. _`ex:output_derivative`:

.. collapse:: Correction

    For a single hidden layer (but trivially generalizable):

    .. math::

        \cfrac{\partial a_j}{\partial x} &= \cfrac{\partial a_j}{\partial z_j} \cfrac{\partial z_j}{\partial a_k} 
                                            \cfrac{\partial a_k}{\partial z_k} \cfrac{\partial z_k}{\partial x} \\
                                        &= \sigma'(z_j) w_j \sigma'(z_k) w_k

    with all :math:`\sigma'=1`, the network is just a linear operator, the derivative is just the product of the weights.