Research paperPFLOTRAN-E4D: A parallel open source PFLOTRAN module for simulating time-lapse electrical resistivity data
Introduction
Electrical Resistivity Tomography (ERT) is a geophysical imaging application whereby the bulk electrical conductivity (i.e. the reciprocal of resistivity) distribution of the subsurface is remotely estimated. ERT has found wide application for understanding the near subsurface because bulk conductivity is governed by both the structural and geochemical properties that characterize subsurface interactions (e.g. Revil et al., 2012). More recently, time-lapse ERT developments have enabled subsurface processes to be monitored in terms of spatial and temporal changes in bulk conductivity (Binley et al., 2015, Kemna et al., 2006, Singha et al., 2015). Time-lapse imaging allows the static contributions of bulk conductivity to be removed, thereby revealing the temporal evolution of spatial changes in bulk conductivity. These changes can often be uniquely related to some subsurface process (e.g. fluid transport, geochemical alteration etc.), thereby providing the capability to remotely monitor that process in space and time (see previous references for examples).
Despite the growing popularity of time-lapse ERT imaging, multi-physical simulation codes designed specifically to simulate changes in subsurface bulk conductivity arising from some subsurface process, and then to simulate the corresponding time-lapse ERT data arising from that process, are lacking. Because of this, ERT practitioners have limited tools for investigating the feasibility or performance of a time-lapse ERT imaging campaign for a particular subsurface process prior to field trial. The resulting uncertainty raises the risk of failure and reduces the overall utility of ERT imaging. In addition, ERT survey design, which comprises both electrode layout and measurement sequence, is typically based on standardized applications, as opposed to being customized to a particular anticipated subsurface process. The capability to accurately simulate the ERT data arising from that process given multiple survey design scenarios would facilitate customized survey designs to improve imaging resolution in both space and time. It would also enable the discovery of critical electrical signatures associated with some processes of interest, and the exploitation of that signature for effective process monitoring. Such a simulator would also facilitate the use of time-lapse ERT data in hydrogeological parameter estimation (deterministic and stochastic) algorithms (Commer et al., 2014, Linde et al., 2006, Rings et al., 2010). Given the recent advancements in understanding the mechanistic relationships between bulk conductivity and subsurface structural and geochemical states (Revil, 2013, Slater, 2007 and references therein), the capability to accurately derive bulk conductivity from existing subsurface simulators is within reach.
In this work we describe a parallel ERT computing module for PFLOTRAN (Hammond et al., 2014), a massively parallel reactive flow and transport model for simulating surface and subsurface processes. Using Message Passing Interface (MPI) libraries (Gropp et al., 1999), a group of processors is created exclusively to accommodate ERT forward modeling. After each time step, PFLOTRAN passes relevant simulation results to this group of modules, which are then transformed to subsurface bulk conductivity, and used to compute a simulated ERT survey for that time step. We demonstrate the code using a groundwater/surface water interaction simulation based upon actual field data. The results show the complex nature of the changes in bulk conductivity arising from stage-driven changes in water table elevation and groundwater/surface water interaction. The results also demonstrate the sensitivity of the ERT data to these processes, thereby validating the capability to monitor them at the field scale. The code and user documentation are available with the PFLOTRAN distribution and associated open source license (https://bitbucket.org/pflotran/pflotran-dev). Users will find basic instructions and an example problem demonstrating the use of PFLOTRAN-E4D in the the tutorial section of the E4D developement repository located at https://bitbucket.org/john775/e4d_dev/ .
Section snippets
E4D (background, capabilities, parallel structure)
E4D is a parallel ERT forward modeling and inversion code available with a Berkeley Software Distribution open source license (Johnson and Wellman, 2015a, Johnson et al., 2010) (see https://e4d.pnnl.gov). The ERT method uses hardware and software to image the bulk electrical conductivity of the subsurface. Bulk electrical conductivity is influenced by physical, biological, and geochemical interactions, making time-lapse images of changes in bulk conductivity useful for monitoring subsurface
Groundwater/surface water interaction monitoring simulation
We simulated ERT monitoring of a groundwater/surface-water interaction system along the Columbia River at the Hanford Site 300 Area in south-central WA, USA (Fig. 6) Aquifer sediments bounding the Columbia River are generally coarse grained and highly permeable (Hammond and Lichtner, 2010, Williams et al., 2008). Coupled with dynamic stage variations, the resulting system is characterized by active stage driven intrusion and retreat of river water into the adjacent unconfined aquifer system.
Results
The simulation was executed with 128 processors allocated to PFLOTRAN and 353 processors allocated to the E4D module (1 master and 352 slaves, one for each electrode). With this allocation, E4D was able to complete a forward simulation (from mesh interpolation through output) within the time required for a single PFLOTRAN time-step, so there was no additional computation time required beyond that required for PFLOTRAN alone. The entire simulation required approximately 3 h. Given the high
Discussion
In the introduction we stated that a multi-physics simulator capable of modeling the bulk electrical conductivity evolution arising from a given subsurface processes would, among other things, 1) enable practitioners to investigate the feasibility of monitoring that process using ERT, and 2) enable the discovery of critical electrical signatures associated with the process, and the exploitation of that signature for effective process monitoring. The groundwater/surface water interaction
Conclusion
It is well recognized that geophysical data, particularly time-lapse geophysical data, have the potential to significantly reduce uncertainty in hydrogeologic parameter estimations (Dafflon et al., 2011, Jardani et al., 2013, Linde et al., 2006, Pollock and Cirpka, 2012). Progress toward informating hydrogeologic inversions with geophysical data is complicated by the general requirement for an accurate field scale petrophysical transform, and by the computational demands of simulating both the
Acknowledgments
This research was supported by the U.S. Department of Energy (DOE) grant number 54737, Office of Biological and Environmental Research (BER), as part of BER's Subsurface Biogeochemistry Research Program (SBR). This contribution originates from the SBR Scientific Focus Area (SFA) at the Pacific Northwest National Laboratory (PNNL).
References (27)
- et al.
MPiTOUGH2: a parallel parameter estimation framework for hydrological and hydrogeophysical applications
Comput. Geosci. UK
(2014) - et al.
Stochastic joint inversion of hydrogeophysical data for salt tracer test monitoring and hydraulic conductivity imaging
Adv. Water Resour.
(2013) The electrical resistivity log as an aid in determining some reservoir characteristics
Am. Min. Met Eng.
(1942)- et al.
PETSc Users Manual, Tech. Rep. ANL-95/11—Revision 3.5
(2015) - et al.
The emergence of hydrogeophysics for improved understanding of subsurface processes over multiple scales
Water Resour. Res.
(2015) Relative permeability calculations from pore size distribution data
Am. Min. Met Eng.
(1953)- et al.
Application of ensemble-based data assimilation techniques for aquifer characterization using tracer data at Hanford 300 area
Water Resour. Res.
(2013) - et al.
Hydrological parameter estimations from a conservative tracer test with variable-density effects at the boise hydrogeophysical research site
Water Resour. Res.
(2011) - et al.
Cross-borehole resistivity tomography
Geophysics
(1991) - et al.
Using MPI-2: Advanced Features of the Message-Passing Interface
(1999)
Field-scale model for the natural attenuation of uranium at the Hanford 300 Area using high-performance computing
Water Resour. Res.
Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN
Water Resour. Res.
Four-dimensional electrical conductivity monitoring of stage-driven river water intrusion: accounting for water table effects using a transient mesh boundary and conditional inversion constraints
Water Resour. Res.
Cited by (28)
Deep learning to estimate permeability using geophysical data
2022, Advances in Water ResourcesCitation Excerpt :PFLOTRAN (Hammond et al., 2012, 2014; Lichtner et al., 2015), an open-source multi-physics code is used to simulate subsurface flow and tracer transport. E4D (Johnson et al., 2010, 2017), a geophysics code is used to simulate time-lapse ERT. These two codes are loosely coupled using Jupyter notebooks.
Stratigraphic identification with airborne electromagnetic methods at the Hanford Site, Washington
2021, Journal of Applied GeophysicsCitation Excerpt :In this paper, we present the first inversion results of the AEM data at the Hanford Site obtained using a 3D EM modeling and inversion code. This code has recently been developed as a part of a capability extension to E4D, an open-source software widely used for 3D/4D ERT data modeling and inversion (Johnson et al., 2010, 2017). The forward modeling is based on solving frequency-domain Maxwell's equations using an unstructured-mesh finite-element method, and the inversion uses a Gauss-Newton optimization scheme.
Coupling surface flow with high-performance subsurface reactive flow and transport code PFLOTRAN
2021, Environmental Modelling and SoftwareAn overview of multimethod imaging approaches in environmental geophysics
2021, Advances in GeophysicsThree-dimensional direct current resistivity forward modeling based on the hp-adaptive finite element method
2020, Computers and GeosciencesCitation Excerpt :With the development of three-dimensional (3D) DC acquisition instruments and forward and inversion methods, 3D measurement and data processing have been gradually applied, and recently many scholars have provided us with open source codes (Gernez et al., 2020; Blanchy et al., 2020) for learning and use. At present, the most widely used numerical methods in forward modeling are the finite difference method (Spitzer, 1995; Wang and Fang, 2001), integral equation method (Hvoždara and Kaikkonen, 1998; Mirgalikyzy et al., 2015; Ren et al., 2017) and finite element method (Bibby, 1978; Li and Spitzer, 2005; Ren and Tang, 2009; Cai et al., 2017; Karaoulis et al., 2013; Johnson et al., 2017). In contrast to the former, the finite element method (FEM) can be combined with unstructured grids (Li and Pek, 2008; Ren and Tang, 2010; Tang et al., 2010; Wang et al., 2013), providing an important role in simulating topography models and complex geologic bodies.
Integrated hydrogeophysical modelling and data assimilation for geoelectrical leak detection
2020, Journal of Contaminant HydrologyCitation Excerpt :Also, the setup for the ERT experiment (e.g. mesh, electrode locations, measurement protocols, petrophysical transforms) need to be included. Once we have an ensemble of model proposals, they are fed to simulate the ERT response using PFLOTRAN-E4D (Johnson et al., 2017). The misfits between observed and simulated ERT responses are used to form data error covariance matrices, which in turn are used to update the model proposals.