Forward Neural NetworkΒΆ Context Structure Activation Rectified linear Logistic Hyperbolic tangent Softmax Output layer Loss Regression Binary classification Multi-class classification Training Gradient descent Stochastic gradient descent Data set used Back-propagation Simple case Real case Regression Multi-class classification Regularization Examples PINN Modified loss Training Automatic Differentiation First order derivative Unary operation Binary: addition Binary: multiplication Reverse order Implementation Higher order derivatives Example Implementation Vectorized differentiation Basic operations Dot product Back to PINNs