Abstract
A new parallel solver for the volumetric integral equations (IE) of electrodynamics is presented. The solver is based on the Galerkin method, which ensures convergent numerical solution. The main features include: (i) memory usage eight times lower compared with analogous IE-based algorithms, without additional restrictions on the background media; (ii) accurate and stable method to compute matrix coefficients corresponding to the IE; and (iii) high degree of parallelism. The solver’s computational efficiency is demonstrated on a problem of magnetotelluric sounding of media with large conductivity contrast, revealing good agreement with results obtained using the second-order finite-element method. Due to the effective approach to parallelization and distributed data storage, the program exhibits perfect scalability on different hardware platforms.
Similar content being viewed by others
References
Anderson W (1979) Numerical integration of related hankel transforms of order 0 and 1 by adaptive digital filtering. Geophysics 44:1287–1305
Avdeev D, Knizhnik S (2009) 3D integral equation modeling with a linear dependence on dimensions. Geophysics 74:89–94
Avdeev D, Kuvshinov A, Pankratov O, Newman G (2002) Three-dimensional induction logging problems, part I: an integral equation solution and model comparisons. Geophysics 67(2):413–426
Avdeev DB, Kuvshinov AV, Pankratov OV, Newman GA (1997) High-performance three-dimensional electromagnetic modelling using modified Neumann series. Wide-band numerical solution and examples. J Geomagn Geoelectr 49:1519–1539
Chave A, Jones A (2012) The magnetotelluric method: theory and practice. Cambridge University Press, Cambridge
Delves LM, Mohamed JL (1985) Computational methods for integral equations. Cambridge University Press, Cambridge
Dmitriev V, Silkin A, Farzan R (2002) Tensor Green function for the system of Maxwell’s equations in a layered medium. Comput Math Model 13(2):107–118
Egbert GD, Kelbert A (2012) Computational recipes for electromagnetic inverse problems. Geophys J Int 189(1):251–267. doi:10.1111/j.1365-246X.2011.05347.x
Ernst O, Gander M (2012) Why it is difficult to solve helmholtz problems with classical iterative methods. In: Graham IG, Hou TY, Lakkis O, Scheichl R (eds) Numerical analysis of multiscale problems, lecture notes in computational science and engineering, vol 83. Springer, Berlin, pp 325–363
Farquharson CG, Miensopust MP (2011) Three-dimensional finite-element modelling of magnetotelluric data with a divergence correction. J Appl Geophys 75:699–710
Fraysse V, Giraud L, Gratton S, Langou J (2003) A set of GMRES routines for real and complex arithmetics on high performance computers. http://www.cerfacs.fr/algor/reports/2003/TR_PA_03_03.pdf
Frigo M, Johnson SG (2005) The design and implementation of FFTW3. Proc IEEE 93(2):216–231 (special issue on “Program Generation, Optimization, and Platform Adaptation”)
Grayver A, Kolev T (2015) Large-scale 3D geo-electromagnetic modeling using parallel adaptive high-order finite element method. Geophysics 80(6):277–291
Haber E, Ascher UM (2001) Fast finite volume simulation of 3D electromagnetic problems with highly discontinuous coefficients. SIAM J Sci Comput 22(6):1943–1961
Hursan G, Zhdanov MS (2002) Contraction integral equation method in three-dimensional electromagnetic modeling. Radio Sci 37(6):1–13. doi:10.1029/2001RS002513
Jaysaval P, Shantsev DV, de la Kethulle de Ryhove S (2015) Efficient 3-D controlled-source electromagnetic modelling using an exponential finite-difference method. Geophys J Int 203(3):1541–1574. doi:10.1093/gji/ggv377
Kamm J, Pedersen LB (2014) Inversion of airborne tensor VLF data using integral equations. Geophys J Int 198(2):775–794. doi:10.1093/gji/ggu161
Koyama T, Utada H, Avdeev D (2008) Fast and memory-saved 3-D forward modeling code for MT by using integral equation method. In: Abstract book, 19th workshop on electromagnetic induction in the Earth, China
Kruglyakov M (2011) Modified integral current methods in electrodynamics of nonhomogeneous media. Comput Math Model 22(3):246–254
Kruglyakov M, Geraskin A, Kuvshinov A (2016) Novel accurate and scalable 3-D MT forward solver based on a contracting integral equation method. Comput Geosci 96:208–217. doi:10.1016/j.cageo.2016.08.017, http://www.sciencedirect.com/science/article/pii/S009830041630293X
Kruglyakov M, Geraskin A, Kuvshinov A (2016b) Novel scalable 3-D MT inverse solver (extrEMeI). In: Abstract GP51A-1357 presented at 2016 Fall meeting, AGU, San Francisco, Calif. 12–16 December, AGU, San Francisco
Mackie R, Smith J, Madden T (1994) 3-Dimensional electromagnetic modeling using finite-difference equation—the magnetotelluric example. Radio Sci 29(4):923–935
Mulder W (2006) A multigrid solver for 3D electromagnetic diffusion. Geophys Prospect 54:633–649
Newman G, Alumbaugh D (2002) Three-dimensional induction logging problems, part 2: a finite-difference solution. Geophysics 61:484–491
Pankratov O, Kuvshinov A (2016) Applied mathematics in EM studies with special emphasis on an uncertainty quantification and 3-D integral equation modelling. Surv Geophys 37(1):109–147. doi:10.1007/s10712-015-9340-4
Pankratov O, Avdeyev D, Kuvshinov A (1995) Electromagnetic-field scattering in a heterogeneous earth: a solution to the forward problem. Izv Phys Solid Earth 31(3):201–209
Puzyrev V, Koldan J, de la Puente J, Houzeaux G, Vazquez M, Cela JM (2013) A parallel finite-element method for three-dimensional controlled-source electromagnetic forward modelling. Geophys J Int 193:678–693
Raiche AP (1974) An integral equation approach to three-dimensional modelling. Geophys J Int 36(2):363–376
Ren Z, Kalscheuer T, Maurer Greenhalgh H S (2013) A goal-oriented adaptive finite-element approach for plane wave 3-D electromagnetic modelling. Geophys J Int 194:700–718
Saad Y (1993) A flexible inner-outer preconditioned GMRES algorithm. SIAM J Sci Comput 14(2):461–469
Sadovnichy V, Tikhonravov A, Voevodin V, Opanasenko V (2013) “Lomonosov”: Supercomputing at Moscow State University. In: Vetter JS (ed) Contemporary high performance computing: from petascale toward exascale. Chapman & Hall/CRC Computational Science, Boca Raton, pp 283–307
Schwarzbach C, Börner RU, Spitzer K (2011) Three-dimensional adaptive higher order finite element simulation for geo-electromagnetics—a marine CSEM example. Geophys J Int 187:63–74
Singer B (1995) Method for solution of Maxwell’s equations in non-uniform media. Geophys J Int 120:590–598
Singer B (2008) Electromagnetic integral equation approach based on contraction operator and solution optimization in Krylov subspace. Geophys J Int 175:857–884
Sun J, Kuvshinov A (2015) Accelerating EM integral equation forward solver for global geomagnetic induction using SVD based matrix compression method. Geophys J Int 200(2):1005–1011
Varentsov IM, Fomenko IY, Golubev NG, Mehanee S, Hursan G, Zhdanov MS (2000) Comparative study of 3-D finite difference and integral equation methods. In: Proceedings of 2000 consortium for electromagnetic modeling and inversion annual meeting. University of Utah, Salt Lake City, pp 35–74
Wang Q, Zhang X, Zhang Y, Yi Q (2013) AUGEM: automatically generate high performance dense linear algebra kernels on x86 CPUs. In: International conference for high performance computing, networking, storage and analysis, SC’13, Denver, CO, USA, pp 1–12. http://www.openblas.net
Wannmaker PE (1991) Advances in three-dimensional magnetotelluric modeling using integral equations. Geophysics 56:1716–1728
Ward SH, Hohmann GW (1988) 4. Electromagnetic theory for geophysical applications, SEG,USA, chap 4, pp 130–311
Weidelt P (1975) Electromagnetic induction in three-dimensional structures. J Geophys-Z Geophys 41:85–109
Acknowledgements
The research of the first author was supported by the Russian Foundation for Basic Research (grant no. 13-05-12018-OFI_M). As a visiting fellow in ETH Zurich, he was also partially supported by the Swiss National Science Foundation (grant no. IZK0Z2_163494) and ETH Zurich. Authors acknowledge the teams of HPC CMC Lomonosov MSU for access to Bluegene/P HPC, the Lomonosov MSU Research Computing Center for access to HPC Lomonosov (Sadovnichy et al. 2013), and the Swiss National Supercomputing Center (CSCS) grant (project ID s660). The authors would also like to thank Alexander Grayver, ETH Zurich, for providing data for comparison, and Alexey Kuvshinov, ETH Zurich, for suggestions and helpful discussions.
Author information
Authors and Affiliations
Corresponding author
Appendices
Appendix A: Green’s Tensor
Following the notations from (Dmitriev et al. 2002), the electrical \(\widehat{G}^E\) and magnetic \(\widehat{G}^H\) tensors of layered media can be written as
where
and
Here, \(J_0\) is a zero-order Bessel function of the first kind, and functions \(U_\gamma (\lambda ,z,z_0)\), \(\gamma =1,\sigma \) are the fundamental functions of the layered media (Appendix B).
From Eqs. (21) and (22), one gets
Note that \(G^E_{xz}(M,M_0)=-G^E_{zx}(M_0,M)\) and \(G^E_{yz}(M,M_0)=-G^E_{zy}(M_0,M)\), according to Lorentz reciprocity.
Let \(\mathrm {\Omega }_n=S_n \times [z_n^1,z_n^2]\), \(\mathrm {\Omega }_m=S_m \times [z_m^1,z_m^2]\), where \(S_n, S_m\) are horizontal rectangular domains, and let \({\sigma }_{\text {b}}={\sigma }_{\text {b}}^n\) inside \([z_n^1, z_n^2]\), \({\sigma }_{\text {b}}={\sigma }_{\text {b}}^m\) inside \([z_m^1, z_m^2]\); That is, the subdomains do not intersect the boundaries of the layers. Taking into account Eqs. (12), (21), and (24), one can see that \(\widehat{B}_n^m\) is expressed in terms of double volumetric integrals with weak integrable singularity, so the order of integration can be changed. Then, using Eqs. (23) and (24), one obtains
where
Here \(\alpha =x,y\), \(\beta =x,y\), and
Therefore, to obtain the coefficients of \(\hat{B}_n^m\), one needs computational methods to find “horizontal” integrals (26) and “vertical” integrals (27). These methods are presented in the following appendices.
Appendix B: Vertical Integration
The integrals in formulas (27) are expressed in terms of the so-called fundamental function of layered media (Dmitriev et al. 2002). Consider media with \(N_\mathrm{lay}-1\) homogeneous layers with complex conductivities \({\sigma }_n\), \(n=1\ldots N_\mathrm{lay}-1\), the upper halfspace (air, the zeroth layer) with complex conductivity \({\sigma }_0\), and the lower halfspace (the \(N_\mathrm{lay}\)-th layer) with conductivity \({\sigma }_{N_\mathrm{lay}}\). Note that, in EM sounding problems, typically \(\mathop {\mathbf {Re}}{{\sigma }_0} \le 10^{-9}\).
The function \(U_\gamma (z,z_*,\lambda )\) is defined as a unique solution of the problem
The following procedure is performed to obtain an explicit expression for \(U_\gamma \) that allows analytical integration. Let \(l_0=0\), \(l_{N_\mathrm{lay}}=0\), \(l_n=d_{n+1}-d_n\), \(n=1\ldots N_\mathrm{lay}-1\). Define \(p_{m}^\gamma \), \(q_{m}^\gamma \), \({m=0\ldots N_\mathrm{lay}}\) by the recurrent expressions
Let \(d_0=d_1\), \(d_{N_{lay+1}}=d_{N_\mathrm{lay}}\) and let points \(z_r\), \(z_s\) belong to layers r and s, respectively, \(0 \le r, s \le N_\mathrm{lay}\). Then, using expressions (29), one gets
where
To check Eq. (30), one can explicitly substitute Eq. (30) into Eq. (28), taking into account expressions (29) and (31).
In view of Eq. (30), one can see that integrals in formulas (27) (i.e., the integrals over \(U_\gamma (z,z_*,\lambda )\) and its partial derivatives) can be obtained analytically with respect to z, \(z_*\) over any domains that do not intersect the layer boundaries. However, rounding errors arising in addition and multiplication of very small or large quantities make the formula (30) impractical for \(\lambda \gg 1\). Instead, the following formula is used:
Formula (32) overcomes the aforementioned problem, since the real parts of all the exponents’ powers are negative. The consequent calculations provide accurate and robust results for any \(0< \lambda <\infty \).
Consider \(N_z\) subdomains in the discretization in vertical direction. To obtain the matrix \(\widehat{B}_n^m\) for the system (15), one needs to compute \(O\left( N_z^2\right) \) complex exponents in Eq. (32). An algorithm requiring only \(O\left( N_z\right) \) complex exponent calculations has been developed to speed up this integration procedure.
Let \(z_0<z_1< \cdots < z_{N_z}\). Suppose that the intervals \([z_l,z_{l+1}]\), \(l=0\ldots N_z-1\) do not intersect the layers’ boundaries. For \(i,j=0\ldots N_z-1\), \(0 \le \alpha +\beta \le 2\), one needs to calculate \(W_{i,j}^{\alpha ,\beta }(\gamma )=\int \nolimits _{z_i}^{z_{i+1}}\frac{\partial {^\alpha }}{\partial {z^\alpha }}\left( \int \nolimits _{z_j}^{z_{j+1}}\left[ \frac{\partial {^\beta }}{\partial {z_*^\beta }}\, U_\gamma (z,z_{*},\lambda )\right] \mathrm{d}z_{*}\right) \mathrm{d}z\).
Let \(r_l\) be an index of the layer containing \([z_l,z_{l+1}]\), \(l=0 \ldots N_{z}-1\). Then, using Eq. (30), one obtains for \(z_i < z_j\)
where
All the exponents in Eq. (34) vanish as \(\lambda \rightarrow \infty \), so the corresponding computations do not depend on round-off errors due to machine precision. The formulas for \(z_i>z_j\) are similar. The integrals \(W_{ii}^{\alpha ,\beta }\) are computed analytically using formulas (32). Since \(\Theta _l^\gamma \) depends only on \(l=1\ldots N_{z}\) and \(\gamma ={1,{\sigma }}\), one only needs to calculate \(O\left( N_z\right) \) complex exponents using factorization from expressions (33), (34).
Appendix C: Horizontal Integration
The integrals (26) are a particular case of the integral
where \(f(\lambda )\) is some easily computed function, and \(S_n=[x_n,x_n+h_x]\times [y_n,y_n+h_y]\), \(S_m=[x_m,x_m+h_x]\times [y_m,y_m+h_y]\) are rectangular domains with similar sizes.
The key feature of the proposed method is transformation of the integrals (35) to one-dimensional convolution integrals. Taking for simplicity \(\alpha =\beta =0\), one has
where
Let \(\lambda =\mathrm {e}^{-t}\), \(R=\mathrm {e}^{s}\)
For fixed \(p,q,\varphi \) the integral in Eq. (37) is the convolution integral with kernel \(\Phi \). Note that, for different values of \(\alpha \) and \(\beta \), the kernels can be obtained similarly.
The main advantage of using the convolution integrals is that their computation does not require explicit calculation of the kernel \(\Phi \). Consider the input function v(t) and the output function u(s) such that
For some \(N=2M\), l, \(0<\xi <0.5l\), \(k=-M\ldots M-1\), define
Then, for any g(t), one gets
The tradeoff between the accuracy of approximation (40) and the computational time is achieved by the particular selection of M, \(N_1\), \(N_2\), l, \(\xi \) and functions u and v.
From Eqs. (36) and (40), the approximation formulas for integral (35) can be obtained as
For the given input function \(v(t)=8\mathrm {e}^{-t^2}\left( t^5-4t^3+2t\right) \), the output functions for different kernels can be expressed analytically by Gaussian and error functions. Inspired by (Anderson 1979), the parameters used are \(l=0.2\), \(\xi =0.0964\), \(M=512\), \(N_1=-250\), \(N_2=200\). In computational experiments, these parameters provided appropriate accuracy in calculation of \(\widehat{B}_n^m\).
Rights and permissions
About this article
Cite this article
Kruglyakov, M., Bloshanskaya, L. High-Performance Parallel Solver for Integral Equations of Electromagnetics Based on Galerkin Method. Math Geosci 49, 751–776 (2017). https://doi.org/10.1007/s11004-017-9677-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11004-017-9677-y