Elsevier

Computers & Geosciences

Volume 99, February 2017, Pages 72-80
Computers & Geosciences

Research paper
PFLOTRAN-E4D: A parallel open source PFLOTRAN module for simulating time-lapse electrical resistivity data

https://doi.org/10.1016/j.cageo.2016.09.006Get rights and content

Highlights

  • Subsurface simulator PFLOTRAN is internally coupled with geophysical simulator E4D.

  • E4D runs as a module to PFLOTRAN, using a dedicated set of parallel compute nodes.

  • The coupled code enables the simulation of changes in bulk electrical conductivity.

  • The coupled code enables the simulation of time-lapse electrical resistivity surveys.

  • A simulation of geophysical monitoring of groundwater/river water interaction is demonstrated.

Abstract

Time-lapse electrical resistivity tomography (ERT) is finding increased application for remotely monitoring processes occurring in the near subsurface in three-dimensions (i.e. 4D monitoring). However, there are few codes capable of simulating the evolution of subsurface resistivity and corresponding tomographic measurements arising from a particular process, particularly in parallel and with an open source license. Herein we describe and demonstrate an electrical resistivity tomography module for the PFLOTRAN subsurface flow and reactive transport simulation code, named PFLOTRAN-E4D. The PFLOTRAN-E4D module operates in parallel using a dedicated set of compute cores in a master-slave configuration. At each time step, the master processes receives subsurface states from PFLOTRAN, converts those states to bulk electrical conductivity, and instructs the slave processes to simulate a tomographic data set. The resulting multi-physics simulation capability enables accurate feasibility studies for ERT imaging, the identification of the ERT signatures that are unique to a given process, and facilitates the joint inversion of ERT data with hydrogeological data for subsurface characterization. PFLOTRAN-E4D is demonstrated herein using a field study of stage-driven groundwater/river water interaction ERT monitoring along the Columbia River, Washington, USA. Results demonstrate the complex nature of subsurface electrical conductivity changes, in both the saturated and unsaturated zones, arising from river stage fluctuations and associated river water intrusion into the aquifer. The results also demonstrate the sensitivity of surface based ERT measurements to those changes over time. PFLOTRAN-E4D is available with the PFLOTRAN development version with an open-source license at https://bitbucket.org/pflotran/pflotran-dev.

Introduction

Electrical Resistivity Tomography (ERT) is a geophysical imaging application whereby the bulk electrical conductivity (i.e. the reciprocal of resistivity) distribution of the subsurface is remotely estimated. ERT has found wide application for understanding the near subsurface because bulk conductivity is governed by both the structural and geochemical properties that characterize subsurface interactions (e.g. Revil et al., 2012). More recently, time-lapse ERT developments have enabled subsurface processes to be monitored in terms of spatial and temporal changes in bulk conductivity (Binley et al., 2015, Kemna et al., 2006, Singha et al., 2015). Time-lapse imaging allows the static contributions of bulk conductivity to be removed, thereby revealing the temporal evolution of spatial changes in bulk conductivity. These changes can often be uniquely related to some subsurface process (e.g. fluid transport, geochemical alteration etc.), thereby providing the capability to remotely monitor that process in space and time (see previous references for examples).

Despite the growing popularity of time-lapse ERT imaging, multi-physical simulation codes designed specifically to simulate changes in subsurface bulk conductivity arising from some subsurface process, and then to simulate the corresponding time-lapse ERT data arising from that process, are lacking. Because of this, ERT practitioners have limited tools for investigating the feasibility or performance of a time-lapse ERT imaging campaign for a particular subsurface process prior to field trial. The resulting uncertainty raises the risk of failure and reduces the overall utility of ERT imaging. In addition, ERT survey design, which comprises both electrode layout and measurement sequence, is typically based on standardized applications, as opposed to being customized to a particular anticipated subsurface process. The capability to accurately simulate the ERT data arising from that process given multiple survey design scenarios would facilitate customized survey designs to improve imaging resolution in both space and time. It would also enable the discovery of critical electrical signatures associated with some processes of interest, and the exploitation of that signature for effective process monitoring. Such a simulator would also facilitate the use of time-lapse ERT data in hydrogeological parameter estimation (deterministic and stochastic) algorithms (Commer et al., 2014, Linde et al., 2006, Rings et al., 2010). Given the recent advancements in understanding the mechanistic relationships between bulk conductivity and subsurface structural and geochemical states (Revil, 2013, Slater, 2007 and references therein), the capability to accurately derive bulk conductivity from existing subsurface simulators is within reach.

In this work we describe a parallel ERT computing module for PFLOTRAN (Hammond et al., 2014), a massively parallel reactive flow and transport model for simulating surface and subsurface processes. Using Message Passing Interface (MPI) libraries (Gropp et al., 1999), a group of processors is created exclusively to accommodate ERT forward modeling. After each time step, PFLOTRAN passes relevant simulation results to this group of modules, which are then transformed to subsurface bulk conductivity, and used to compute a simulated ERT survey for that time step. We demonstrate the code using a groundwater/surface water interaction simulation based upon actual field data. The results show the complex nature of the changes in bulk conductivity arising from stage-driven changes in water table elevation and groundwater/surface water interaction. The results also demonstrate the sensitivity of the ERT data to these processes, thereby validating the capability to monitor them at the field scale. The code and user documentation are available with the PFLOTRAN distribution and associated open source license (https://bitbucket.org/pflotran/pflotran-dev). Users will find basic instructions and an example problem demonstrating the use of PFLOTRAN-E4D in the the tutorial section of the E4D developement repository located at https://bitbucket.org/john775/e4d_dev/ .

Section snippets

E4D (background, capabilities, parallel structure)

E4D is a parallel ERT forward modeling and inversion code available with a Berkeley Software Distribution open source license (Johnson and Wellman, 2015a, Johnson et al., 2010) (see https://e4d.pnnl.gov). The ERT method uses hardware and software to image the bulk electrical conductivity of the subsurface. Bulk electrical conductivity is influenced by physical, biological, and geochemical interactions, making time-lapse images of changes in bulk conductivity useful for monitoring subsurface

Groundwater/surface water interaction monitoring simulation

We simulated ERT monitoring of a groundwater/surface-water interaction system along the Columbia River at the Hanford Site 300 Area in south-central WA, USA (Fig. 6) Aquifer sediments bounding the Columbia River are generally coarse grained and highly permeable (Hammond and Lichtner, 2010, Williams et al., 2008). Coupled with dynamic stage variations, the resulting system is characterized by active stage driven intrusion and retreat of river water into the adjacent unconfined aquifer system.

Results

The simulation was executed with 128 processors allocated to PFLOTRAN and 353 processors allocated to the E4D module (1 master and 352 slaves, one for each electrode). With this allocation, E4D was able to complete a forward simulation (from mesh interpolation through output) within the time required for a single PFLOTRAN time-step, so there was no additional computation time required beyond that required for PFLOTRAN alone. The entire simulation required approximately 3 h. Given the high

Discussion

In the introduction we stated that a multi-physics simulator capable of modeling the bulk electrical conductivity evolution arising from a given subsurface processes would, among other things, 1) enable practitioners to investigate the feasibility of monitoring that process using ERT, and 2) enable the discovery of critical electrical signatures associated with the process, and the exploitation of that signature for effective process monitoring. The groundwater/surface water interaction

Conclusion

It is well recognized that geophysical data, particularly time-lapse geophysical data, have the potential to significantly reduce uncertainty in hydrogeologic parameter estimations (Dafflon et al., 2011, Jardani et al., 2013, Linde et al., 2006, Pollock and Cirpka, 2012). Progress toward informating hydrogeologic inversions with geophysical data is complicated by the general requirement for an accurate field scale petrophysical transform, and by the computational demands of simulating both the

Acknowledgments

This research was supported by the U.S. Department of Energy (DOE) grant number 54737, Office of Biological and Environmental Research (BER), as part of BER's Subsurface Biogeochemistry Research Program (SBR). This contribution originates from the SBR Scientific Focus Area (SFA) at the Pacific Northwest National Laboratory (PNNL).

References (27)

  • M. Commer et al.

    MPiTOUGH2: a parallel parameter estimation framework for hydrological and hydrogeophysical applications

    Comput. Geosci. UK

    (2014)
  • A. Jardani et al.

    Stochastic joint inversion of hydrogeophysical data for salt tracer test monitoring and hydraulic conductivity imaging

    Adv. Water Resour.

    (2013)
  • G.E. Archie

    The electrical resistivity log as an aid in determining some reservoir characteristics

    Am. Min. Met Eng.

    (1942)
  • S. Balay et al.

    PETSc Users Manual, Tech. Rep. ANL-95/11—Revision 3.5

    (2015)
  • A. Binley et al.

    The emergence of hydrogeophysics for improved understanding of subsurface processes over multiple scales

    Water Resour. Res.

    (2015)
  • N.T. Burdine

    Relative permeability calculations from pore size distribution data

    Am. Min. Met Eng.

    (1953)
  • X. Chen et al.

    Application of ensemble-based data assimilation techniques for aquifer characterization using tracer data at Hanford 300 area

    Water Resour. Res.

    (2013)
  • B. Dafflon et al.

    Hydrological parameter estimations from a conservative tracer test with variable-density effects at the boise hydrogeophysical research site

    Water Resour. Res.

    (2011)
  • W. Daily et al.

    Cross-borehole resistivity tomography

    Geophysics

    (1991)
  • W. Gropp et al.

    Using MPI-2: Advanced Features of the Message-Passing Interface

    (1999)
  • G.E. Hammond et al.

    Field-scale model for the natural attenuation of uranium at the Hanford 300 Area using high-performance computing

    Water Resour. Res.

    (2010)
  • G.E. Hammond et al.

    Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN

    Water Resour. Res.

    (2014)
  • T. Johnson et al.

    Four-dimensional electrical conductivity monitoring of stage-driven river water intrusion: accounting for water table effects using a transient mesh boundary and conditional inversion constraints

    Water Resour. Res.

    (2015)
  • Cited by (28)

    • Deep learning to estimate permeability using geophysical data

      2022, Advances in Water Resources
      Citation Excerpt :

      PFLOTRAN (Hammond et al., 2012, 2014; Lichtner et al., 2015), an open-source multi-physics code is used to simulate subsurface flow and tracer transport. E4D (Johnson et al., 2010, 2017), a geophysics code is used to simulate time-lapse ERT. These two codes are loosely coupled using Jupyter notebooks.

    • Stratigraphic identification with airborne electromagnetic methods at the Hanford Site, Washington

      2021, Journal of Applied Geophysics
      Citation Excerpt :

      In this paper, we present the first inversion results of the AEM data at the Hanford Site obtained using a 3D EM modeling and inversion code. This code has recently been developed as a part of a capability extension to E4D, an open-source software widely used for 3D/4D ERT data modeling and inversion (Johnson et al., 2010, 2017). The forward modeling is based on solving frequency-domain Maxwell's equations using an unstructured-mesh finite-element method, and the inversion uses a Gauss-Newton optimization scheme.

    • Three-dimensional direct current resistivity forward modeling based on the hp-adaptive finite element method

      2020, Computers and Geosciences
      Citation Excerpt :

      With the development of three-dimensional (3D) DC acquisition instruments and forward and inversion methods, 3D measurement and data processing have been gradually applied, and recently many scholars have provided us with open source codes (Gernez et al., 2020; Blanchy et al., 2020) for learning and use. At present, the most widely used numerical methods in forward modeling are the finite difference method (Spitzer, 1995; Wang and Fang, 2001), integral equation method (Hvoždara and Kaikkonen, 1998; Mirgalikyzy et al., 2015; Ren et al., 2017) and finite element method (Bibby, 1978; Li and Spitzer, 2005; Ren and Tang, 2009; Cai et al., 2017; Karaoulis et al., 2013; Johnson et al., 2017). In contrast to the former, the finite element method (FEM) can be combined with unstructured grids (Li and Pek, 2008; Ren and Tang, 2010; Tang et al., 2010; Wang et al., 2013), providing an important role in simulating topography models and complex geologic bodies.

    • Integrated hydrogeophysical modelling and data assimilation for geoelectrical leak detection

      2020, Journal of Contaminant Hydrology
      Citation Excerpt :

      Also, the setup for the ERT experiment (e.g. mesh, electrode locations, measurement protocols, petrophysical transforms) need to be included. Once we have an ensemble of model proposals, they are fed to simulate the ERT response using PFLOTRAN-E4D (Johnson et al., 2017). The misfits between observed and simulated ERT responses are used to form data error covariance matrices, which in turn are used to update the model proposals.

    View all citing articles on Scopus
    View full text