DGM: A deep learning algorithm for solving partial differential equations☆,☆☆
Section snippets
Deep learning and high-dimensional PDEs
High-dimensional partial differential equations (PDEs) are used in physics, engineering, and finance. Their numerical solution has been a longstanding challenge. Finite difference methods become infeasible in higher dimensions due to the explosion in the number of grid points and the demand for reduced time step size. If there are d space dimensions and 1 time dimension, the mesh is of size . This quickly becomes computationally intractable when the dimension d becomes even moderately
Algorithm
Consider a parabolic PDE with d spatial dimensions: where . The DGM algorithm approximates with a deep neural network where are the neural network's parameters. Note that the differential operators and can be calculated analytically. Construct the objective function: Here,
A Monte Carlo method for fast computation of second derivatives
This section describes a modified algorithm which may be more computationally efficient in some cases. The term contains second derivatives which may be expensive to compute in higher dimensions. For instance, 20,000 second derivatives must be calculated in dimensions.
The complicated architectures of neural networks can make it computationally costly to calculate the second derivatives (for example, see the neural network architecture (4.2)). The computational
Numerical analysis for a high-dimensional free boundary PDE
We test our algorithm on a class of high-dimensional free boundary PDEs. These free boundary PDEs are used in finance to price American options and are often referred to as “American option PDEs”. An American option is a financial derivative on a portfolio of stocks. The option owner may at any time choose to exercise the American option and receive a payoff which is determined by the underlying prices of the stocks in the portfolio. T is called the maturity date of the option and the
High-dimensional Hamilton–Jacobi–Bellman PDE
We also test the deep learning algorithm on a high-dimensional Hamilton–Jacobi–Bellman (HJB) equation corresponding to the optimal control of a stochastic heat equation. Specifically, we demonstrate that the deep learning algorithm accurately solves the high-dimensional PDE (5.5). The PDE (5.5) is motivated by the problem of optimally controlling the stochastic partial differential equation (SPDE):
Burgers' equation
It is often of interest to find the solution of a PDE over a range of problem setups (e.g., different physical conditions and boundary conditions). For example, this may be useful for the design of engineering systems or uncertainty quantification. The problem setup space may be high-dimensional and therefore may require solving many PDEs for many different problem setups, which can be computationally expensive.
Let the variable p represent the problem setup (i.e., physical conditions, boundary
Neural network approximation theorem for PDEs
Let the L2 error measure how well the neural network f satisfies the differential operator, boundary condition, and initial condition. Define as the class of neural networks with n hidden units and let be a neural network with n hidden units which minimizes . We prove that in the appropriate sense, for a class of quasilinear parabolic PDEs with the principle term in divergence form under certain growth and
Conclusion
We believe that deep learning could become a valuable approach for solving high-dimensional PDEs, which are important in physics, engineering, and finance. The PDE solution can be approximated with a deep neural network which is trained to satisfy the differential operator, initial condition, and boundary conditions. We prove that the neural network converges to the solution of the partial differential equation as the number of hidden units increases.
Our deep learning algorithm for solving PDEs
References (49)
- et al.
Nonlinear parabolic equations with measure data
J. Funct. Anal.
(1997) - et al.
Option pricing with a direct adaptive sparse grid approach
J. Comput. Appl. Math.
(2012) - et al.
Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks
Neural Netw.
(1990) Approximation capabilities of multilayer feedforward networks
Neural Netw.
(1991)Neural algorithm for solving differential equations
J. Comput. Phys.
(1990)Existence results for a Cauchy–Dirichlet parabolic problem with a repulsive gradient term
Nonlinear Anal.
(2018)- et al.
Numerical solution for high order differential equations using a hybrid neural network-optimization method
Appl. Math. Comput.
(2006) - et al.
Stochastic Simulation: Algorithms and Analysis
(2007) - et al.
Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
- et al.
Gradient convergence in gradient methods via errors
SIAM J. Optim.
(2000)
Summability and existence results for nonlinear parabolic equations
Nonlinear Anal., Theory Methods Appl.
Sparse grids
Acta Numer.
Feedforward sigmoidal networks – equicontinuity and fault-tolerance properties
IEEE Trans. Neural Netw.
Deep Relaxation: Partial Differential Equations for Optimizing Deep Neural Networks
Stationary Hamilton–Jacobi equations in Hilbert spaces and applications to a stochastic optimal control problem
SIAM J. Control Optim.
Approximation by superposition of a sigmoidal function
Math. Control Signals Syst.
Convergence of numerical schemes for the solution of parabolic stochastic partial differential equations
Math. Comput.
Large scale distributed deep networks
Optimal control of a stochastic heat equation with boundary-noise and boundary-control
ESAIM Control Optim. Calc. Var.
Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations
Asymptotic expansion as prior knowledge in deep learning method for high dimensional BSDEs
A real variable lemma and the continuity of paths of some Gaussian processes
Indiana Univ. Math. J.
Understanding the difficulty of training deep feedforward neural networks
Numerical Experiments with SPDEs
Cited by (1906)
Learning scattering waves via coupling physics-informed neural networks and their convergence analysis
2024, Journal of Computational and Applied MathematicsLocal randomized neural networks with discontinuous Galerkin methods for partial differential equations
2024, Journal of Computational and Applied MathematicsSolving PDEs on unknown manifolds with machine learning
2024, Applied and Computational Harmonic AnalysisData-driven and knowledge-guided denoising diffusion model for flood forecasting
2024, Expert Systems with ApplicationsCoupling parameter and particle dynamics for adaptive sampling in Neural Galerkin schemes
2024, Physica D: Nonlinear PhenomenaLearning based numerical methods for acoustic frequency-domain simulation with high frequency
2024, Engineering Analysis with Boundary Elements
- ☆
The authors thank seminar participants at the JP Morgan Machine Learning and AI Forum seminar, the Imperial College London Applied Mathematics and Mathematical Physics seminar, the Department of Applied Mathematics at the University of Colorado Boulder, Princeton University, and Northwestern University for their comments. The authors would also like to thank participants at the 2017 INFORMS Applied Probability Conference, the 2017 Greek Stochastics Conference, and the 2018 SIAM Annual Meeting for their comments.
- ☆☆
Research of K.S. supported in part by the National Science Foundation (DMS 1550918). Computations for this paper were performed using the Blue Waters supercomputer grant “Distributed Learning with Neural Networks”.