A survey of high level frameworks in block-structured adaptive mesh refinement packages

Highlights

• A survey of mature openly available state-of-the-art structured AMR libraries and codes.

• Discussion of their frameworks, challenges and design trade-offs.

• Directions being pursued by the codes to prepare for the future many-core and heterogeneous platforms.

Abstract

Over the last decade block-structured adaptive mesh refinement (SAMR) has found increasing use in large, publicly available codes and frameworks. SAMR frameworks have evolved along different paths. Some have stayed focused on specific domain areas, others have pursued a more general functionality, providing the building blocks for a larger variety of applications. In this survey paper we examine a representative set of SAMR packages and SAMR-based codes that have been in existence for half a decade or more, have a reasonably sized and active user base outside of their home institutions, and are publicly available. The set consists of a mix of SAMR packages and application codes that cover a broad range of scientific domains. We look at their high-level frameworks, their design trade-offs and their approach to dealing with the advent of radical changes in hardware architecture. The codes included in this survey are BoxLib, Cactus, Chombo, Enzo, FLASH, and Uintah.

Introduction

Block-structured adaptive mesh refinement (SAMR)  [8], [7] first appeared as a computational technique almost 30 years ago; since then it has been used in many individual research codes and, increasingly over the last decade, in large, publicly available code frameworks and application codes. The first uses of SAMR focused almost entirely on explicit methods for compressible hydrodynamics, and these types of problems motivated the building of many of the large code frameworks. SAMR frameworks have evolved along different paths. Some have stayed focused on specific domain areas, adding large amounts of functionality and problem-specific physics modules that are relevant to those applications. Examples of these include AstroBEAR  [26], CRASH  [91], Cactus  [42], [15], Enzo [14], [38], FLASH  [40], [32], Overture  [76], PLUTO  [68], [89], and Uintah  [77], [78]. Other frameworks have pursued a more general functionality, providing the building blocks for a larger variety of applications while enabling domain-specific codes to be built using that framework. As an example, while almost every SAMR framework can be used to solve systems of hyperbolic conservation laws explicitly, not all frameworks include the functionality to solve elliptic equations accurately on the entire hierarchy or a subset of levels. Examples of frameworks constructed specifically for solving hyperbolic conservation laws include AMROC  [29] and AMRClaw  [5], both based on the wave propagation algorithms of R. LeVeque. Extensions of AMRClaw include GeoClaw  [41], the widely used tsunami simulation tool. BoxLib  [13], Chombo  [23], Jasmine  [73] and SAMRAI  [17], [47] are more general in that they supply full functionality for solving equation sets containing hyperbolic, parabolic and elliptic equations, and facilitate the development of codes for simulating a wide variety of different applications. PARAMESH  [39] supplies only the mesh management capability and as such is equation-independent. A more comprehensive list of codes that use SAMR, and other useful adaptive mesh refinement (AMR) resources can be found at  [46].

SAMR codes all rely on the same fundamental concept, viz. that the solution can be computed in different regions of the domain with different spatial resolutions, where each region at a particular resolution has a logically rectangular structure. In some SAMR codes the data is organized by level, so that the description of the hierarchy is fundamentally defined by the union of blocks at each level; while others organize their data with unique parent–child relationships. Along with the spatial decomposition, different codes solving time-dependent equations make different assumptions about the time stepping, i.e., whether blocks at all levels advance at the same time step, or blocks advance with a time step unique to their level. Finally, even when frameworks are used to solve exactly the same equations with exactly the same algorithm, the performance can vary due to the fact that different frameworks are written in different languages, with different choices of data layout, and also differ in other implementation details. However, despite their differences in infrastructure and target domain applications, the codes have many aspects that are similar, and many of these codes follow a set of very similar software engineering practices.

In this survey paper we examine a representative set of SAMR packages and SAMR-based codes that: (1) have been in existence for half a decade or more, (2) have a reasonably sized and active user base outside of their home institutions, and most importantly, (3) are publicly available to any interested user. In selecting the codes we have taken care to include variations in spatial and temporal refinement practices, load distribution and meta-information management. Therefore, we have octree and patch based SAMR, no subcycling and subcycling done in different ways, load distribution on a level by level or all levels at once, and globally replicated meta-data or local view. Additionally, the set covers a broad range of scientific domains that use SAMR technology in different ways. We look at their high-level frameworks, consider the trade-offs between various approaches, and the challenges posed by the advent of radical changes in hardware architecture. The codes studied in detail in this survey are BoxLib, Cactus, Chombo, Enzo, FLASH, and Uintah. The application domains covered by a union of these codes include astrophysics, cosmology, general relativity, combustion, climate science, subsurface flow, turbulence, fluid–structure interactions, plasma physics, and particle accelerators.