Modified loss

Physics Informed Neural Network (PINN) are an alternative tool for solving differential problems (you already know finite differences and finite elements) with the advantage to be able to seamlessly incorporate some experimental data to also fit. As explained in the previous sections, a FNN can be seen as a adjustable function: output layer as a function of the input layer. The FNN is differentiable. Imagine that the input layer is constituted of the space variables: \(x, y, z\), the time \(t\) and potentially some physical parameters \(\phi\) (not to be confused with the FNN parameters and hyper-parameters); the output layer is the value of an unknown function \(u(x,y,z,t;\phi)\). If the function \(u\) is defined to be the solution of a differential problem \(\mathcal{N}\):

(1)\[\mathcal{N}[u]=0\]

An example of such problem is the 1d transient heat conduction equation:

(2)\[\cfrac{\partial u}{\partial t} - \phi \cfrac{\partial^2 u }{\partial x^2} = 0\]

Let us explicitly state that the unknown, which will be the function represented by the FNN, depends on the FNN parameters \(\theta\). To do so, let us use the notation \(u_\theta\).

As long as the \(\theta\) are not perfectly tuned, \(u_\theta\) is not perfectly solution of \(\mathcal{N}[u]=0\). The residual of the equation is:

(3)\[r = \mathcal{N}[u_\theta].\]

The basic idea behind PINNs is to tweak the loss function in order to combines several contributions:

(4)\[J= \lambda_\mathcal{N} J_{\mathrm{residual}} + \lambda_b J_{\mathrm{boundary}} + \lambda_c J_{\mathrm{initial conditions}} + J_{\mathrm{data}}\]

The \(\lambda\) parameters are hyper-parameters to weight the importance given to each objective. The \(J\) are:

  • residual: imbalance of \(\mathcal{N}[u_{\theta}]\) measured by \(r\). In order to avoid negative errors to compensate positive ones, the square of \(r\) is used. The points (in space and time) where one decides to evaluate \(J_{\mathrm{residual}}\) are called colocation points. The number of colocation points is \(n_c\).

  • boundary conditions: adequation of the values of \(u_{\theta}\) or its space derivatives and some imposed conditions on the frontier of the space domain, typically Dirichlet or Neumann boundary conditions. The points (in space and time) where one decides to evaluate \(J_{\mathrm{residual}}\) are also called colocation points. The number of colocation points on the boundaries is \(n_b\).

  • initial conditions: value of \(u_\theta\) or its time derivative at a given time usually at the start time.

  • data: information to fit from experimental source, similarly to basic FNN.

Training

The new loss function makes the training more complex. Indeed, the basic concept behind training is to use the gradient of the loss with respect to the FNN parameters \(\theta\): the weights and biases. However, now the loss is directly function of an arbitrary differential problem.

(5)\[\begin{split}\cfrac{\partial}{\partial \theta_j} J_{\mathrm{residual}} &= \cfrac{1}{2 n_c} \sum_i \cfrac{\partial (r_i)^2}{\partial \theta_j} \\ &= \cfrac{1}{ n_c} \sum_i r_i \cfrac{\partial r_i}{\partial \theta_j}\end{split}\]

Let us focus on the Fourier heat conduction to make things more concrete, but remember that the method should be able to adapt to any differential problem.

(6)\[\begin{split}\cfrac{\partial}{\partial \theta_j} J_{\mathrm{residual}} &= \cfrac{1}{ n_c} \sum_i r_i \cfrac{\partial }{\partial \theta_j} \left( \cfrac{\partial u_\theta}{\partial t} - \phi \cfrac{\partial^2 u_\theta }{\partial x^2} \right)_i \\ &=\cfrac{1}{ n_c} \sum_i \left( \cfrac{\partial u_\theta}{\partial t} - \phi \underbrace{\cfrac{\partial^2 u_\theta }{\partial x^2}}_{\textrm{higher order}} \right)_i \left(\underbrace{ \cfrac{\partial }{\partial t} \cfrac{\partial u_\theta}{\partial \theta_j} - \phi \cfrac{\partial^2 }{\partial x^2}\cfrac{\partial u_\theta}{\partial \theta_j}}_{\textrm{cross derivatives}} \right)_i \\\end{split}\]

The loss gradient is not only function of higher order derivative such as \(\cfrac{\partial^2 u_\theta }{\partial x^2}\), but even worse, it is now function of cross derivative between input parameters \(x, t\) and FNN parameters \(\theta_j\) with for example \(\cfrac{\partial }{\partial t} \cfrac{\partial u_\theta}{\partial \theta_j}\).

The differential problem is not fixed, the derivatives can not be hard coded like we did previously, the derivatives must be obtained automatically. The automatic differentiation is there to save the day.

Actually, thanks to automatic differentiation, the gradient will never be written, the loss function will automatically be differentiated.