Daoce Wang
Dingwen Tao
ABSTRACT
As supercomputers advance towards exascale capabilities, computational intensity increases significantly, and the volume of data requiring storage and transmission experiences exponential growth. Adaptive Mesh Refinement (AMR) has emerged as an effective solution to address these two challenges. Concurrently, error-bounded lossy compression is recognized as one of the most efficient approaches to tackle the latter issue. Despite their respective advantages, few attempts have been made to investigate how AMR and error-bounded lossy compression can function together. To this end, this study presents a novel in-situ lossy compression framework that employs the HDF5 filter to improve both I/O costs and boost compression quality for AMR applications. We implement our solution into the AMReX framework and evaluate on two real-world AMR applications, Nyx and WarpX, on the Summit supercomputer. Experiments with 4096 CPU cores demonstrate that AMRIC improves the compression ratio by up to 81× and the I/O performance by up to 39× over AMReX's original compression solution.
- 2023. HDF5 Filters. https://docs.hdfgroup.org/hdf5/develop/_f_i_l_t_e_r.html Online.Google Scholar
- Mark Ainsworth, Ozan Tugluk, Ben Whitney, and Scott Klasky. 2018. Multilevel techniques for compression and reduction of scientific data---the univariate case. Computing and Visualization in Science 19, 5--6 (2018), 65--76.Google ScholarDigital Library
- AMReX - HDF5 Plotfile Compression. 2023. https://amrex-codes.github.io/amrex/docs_html/IO.html#hdf5-plotfile-compression. Online.Google Scholar
- AMReX's documentation. 2023. https://amrex-codes.github.io/amrex/docs_html/Basics.html#boxarray. Online.Google Scholar
- Allison H Baker, Dorit M Hammerling, and Terece L Turton. 2019. Evaluating image quality measures to assess the impact of lossy data compression applied to climate simulation data. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 517--528.Google Scholar
- Kevin J Bowers, BJ Albright, L Yin, B Bergen, and TJT Kwan. 2008. Ultrahigh performance three-dimensional electromagnetic relativistic kinetic plasma simulation. Physics of Plasmas 15, 5 (2008), 055703.Google ScholarCross Ref
- Franck Cappello, Sheng Di, Sihuan Li, Xin Liang, Ali Murat Gok, Dingwen Tao, Chun Hong Yoon, Xin-Chuan Wu, Yuri Alexeev, and Frederic T Chong. 2019. Use cases of lossy compression for floating-point data in scientific data sets. The International Journal of High Performance Computing Applications (2019).Google Scholar
- cuZFP. 2023. https://github.com/LLNL/zfp/tree/develop/src/cuda_zfp. Online.Google Scholar
- Sheng Di. 2023. H5Z-SZ. https://github.com/disheng222/H5Z-SZ Online.Google Scholar
- Sheng Di and Franck Cappello. 2016. Fast error-bounded lossy HPC data compression with SZ. In 2016 IEEE International Parallel and Distributed Processing Symposium. IEEE, 730--739.Google ScholarCross Ref
- Sheng Di and Franck Cappello. 2016. Fast error-bounded lossy HPC data compression with SZ. In 2016 IEEE International Parallel and Distributed Processing Symposium. IEEE, IEEE, Chicago, IL, USA, 730--739.Google ScholarCross Ref
- Bo Fang, Daoce Wang, Sian Jin, Quincey Koziol, Zhao Zhang, Qiang Guan, Surendra Byna, Sriram Krishnamoorthy, and Dingwen Tao. 2021. Characterizing Impacts of Storage Faults on HPC Applications: A Methodology and Insights. 409--420. Google ScholarCross Ref
- L. Fedeli, A. Huebl, F. Boillod-Cerneux, T. Clark, K. Gott, C. Hillairet, S. Jaure, A. Leblanc, R. Lehe, A. Myers, C. Piechurski, M. Sato, N. Zaim, W. Zhang, J. Vay, and H. Vincenti. 2022. Pushing the Frontier in the Design of Laser-Based Electron Accelerators with Groundbreaking Mesh-Refined Particle-In-Cell Simulations on Exascale-Class Supercomputers. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, Los Alamitos, CA, USA, 1--12. Google ScholarCross Ref
- Mike Folk, Gerd Heber, Quincey Koziol, Elena Pourmal, and Dana Robinson. 2011. An overview of the HDF5 technology suite and its applications. In Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases. 36--47.Google ScholarDigital Library
- Pascal Grosset, Christopher Biwer, Jesus Pulido, Arvind Mohan, Ayan Biswas, John Patchett, Terece Turton, David Rogers, Daniel Livescu, and James Ahrens. 2020. Foresight: analysis that matters for data reduction. In 2020 SC20: International Conference for High Performance Computing, Networking, Storage and Analysis (SC). IEEE Computer Society, 1171--1185.Google ScholarCross Ref
- Guénolé Harel, Jacques-Bernard Lekien, and Philippe P Pébaÿ. 2017. Two new contributions to the visualization of AMR grids: I. interactive rendering of extreme-scale 2-dimensional grids ii. novel selection filters in arbitrary dimension. arXiv preprint arXiv:1703.00212 (2017).Google Scholar
- hipdac tac. 2023. https://github.com/hipdac-lab/HPDC22-TAC. Online.Google Scholar
- Sian Jin, Pascal Grosset, Christopher M Biwer, Jesus Pulido, Jiannan Tian, Dingwen Tao, and James Ahrens. 2020. Understanding GPU-Based Lossy Compression for Extreme-Scale Cosmological Simulations. arXiv preprint arXiv:2004.00224 (2020).Google Scholar
- Sian Jin, Jesus Pulido, Pascal Grosset, Jiannan Tian, Dingwen Tao, and James Ahrens. 2021. Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling. arXiv preprint arXiv:2104.00178 (2021).Google Scholar
- Sian Jin, Dingwen Tao, Houjun Tang, Sheng Di, Suren Byna, Zarija Lukic, and Franck Cappello. 2022. Accelerating parallel write via deeply integrating predictive lossy compression with HDF5. arXiv preprint arXiv:2206.14761 (2022).Google Scholar
- Xin Liang, Sheng Di, Dingwen Tao, Sihuan Li, Shaomeng Li, Hanqi Guo, Zizhong Chen, and Franck Cappello. 2018. Error-controlled lossy compression optimized for high compression ratios of scientific datasets. In 2018 IEEE International Conference on Big Data. IEEE, 438--447.Google ScholarCross Ref
- Xin Liang, Kai Zhao, Sheng Di, Sihuan Li, Robert Underwood, Ali M. Gok, Jiannan Tian, Junjing Deng, Jon C. Calhoun, Dingwen Tao, Zizhong Chen, and Franck Cappello. 2022. SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors. IEEE Transactions on Big Data (2022), 1--14. Google ScholarCross Ref
- Peter Lindstrom. 2014. Fixed-rate compressed floating-point arrays. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 2674--2683.Google ScholarCross Ref
- Peter Lindstrom. 2023. H5Z-ZFP. https://github.com/LLNL/H5Z-ZFP Online.Google Scholar
- Tao Lu, Qing Liu, Xubin He, Huizhang Luo, Eric Suchyta, Jong Choi, Norbert Podhorszki, Scott Klasky, Mathew Wolf, Tong Liu, et al. 2018. Understanding and modeling lossy compression schemes on HPC scientific data. In 2018 IEEE International Parallel and Distributed Processing Symposium. IEEE, 348--357.Google ScholarCross Ref
- Huizhang Luo, Dan Huang, Qing Liu, Zhenbo Qiao, Hong Jiang, Jing Bi, Haitao Yuan, Mengchu Zhou, Jinzhen Wang, and Zhenlu Qin. 2019. Identifying Latent Reduced Models to Precondition Lossy Compression. In 2019 IEEE International Parallel and Distributed Processing Symposium. IEEE.Google ScholarCross Ref
- Huizhang Luo, Junqi Wang, Qing Liu, Jieyang Chen, Scott Klasky, and Norbert Podhorszki. 2021. zMesh: Exploring Application Characteristics to Improve Lossy Compression Ratio for Adaptive Mesh Refinement. In 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 402--411.Google ScholarCross Ref
- NYX simulation. 2019. https://amrex-astro.github.io/Nyx/. Online.Google Scholar
- Oak Ridge Leadership Computing Facility. [n.d.]. Summit Supercomputer. https://www.olcf.ornl.gov/summit/Google Scholar
- Oak Ridge Leadership Computing Facility. 2023. WarpX, granted early access to the exascale supercomputer Frontier, receives the high-performance computing world's highest honor. https://www.olcf.ornl.gov/2022/11/17/plasma-simulation-code-wins-2022-acm-gordon-bell-prize/ Online.Google Scholar
- Russ Rew and Glenn Davis. 1990. NetCDF: an interface for scientific data access. IEEE computer graphics and applications 10, 4 (1990), 76--82.Google Scholar
- James M Stone, Kengo Tomida, Christopher J White, and Kyle G Felker. 2020. The Athena++ adaptive mesh Refinement framework: Design and magnetohy-drodynamic solvers. The Astrophysical Journal Supplement Series 249, 1 (2020), 4.Google ScholarCross Ref
- Dingwen Tao, Sheng Di, Zizhong Chen, and Franck Cappello. 2017. Significantly improving lossy compression for scientific data sets based on multidimensional prediction and error-controlled quantization. In 2017 IEEE International Parallel and Distributed Processing Symposium. IEEE, 1129--1139.Google ScholarCross Ref
- Dingwen Tao, Sheng Di, Zizhong Chen, and Franck Cappello. 2017. Significantly improving lossy compression for scientific data sets based on multidimensional prediction and error-controlled quantization. In 2017 IEEE International Parallel and Distributed Processing Symposium. IEEE, 1129--1139.Google ScholarCross Ref
- Dingwen Tao, Sheng Di, Xin Liang, Zizhong Chen, and Franck Cappello. 2019. Optimizing lossy compression rate-distortion from automatic online selection between SZ and ZFP. IEEE Transactions on Parallel and Distributed Systems 30, 8 (2019), 1857--1871.Google ScholarCross Ref
- The HDF Group. 2023. Hierarchical data format version 5. http://www.hdfgroup.org/HDF5 Online.Google Scholar
- Jiannan Tian, Sheng Di, Xiaodong Yu, Cody Rivera, Kai Zhao, Sian Jin, Yunhe Feng, Xin Liang, Dingwen Tao, and Franck Cappello. 2021. Optimizing error-bounded lossy compression for scientific data on GPUs. In 2021 IEEE International Conference on Cluster Computing (CLUSTER). IEEE, 283--293.Google ScholarCross Ref
- Jiannan Tian, Sheng Di, Kai Zhao, Cody Rivera, Megan Hickman Fulp, Robert Underwood, Sian Jin, Xin Liang, Jon Calhoun, Dingwen Tao, and Franck Cappello. 2020. cuSZ: An Efficient GPU-Based Error-Bounded Lossy Compression Framework for Scientific Data. (2020), 3--15.Google Scholar
- Marc-André Vef. 2016. Analyzing file create performance in IBM spectrum scale. Master's thesis, Johannes Gutenberg University Mainz (2016).Google Scholar
- Daoce Wang, Jesus Pulido, Pascal Grosset, Sian Jin, Jiannan Tian, James Ahrens, and Dingwen Tao. 2022. TAC: Optimizing Error-Bounded Lossy Compression for Three-Dimensional Adaptive Mesh Refinement Simulations. In Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing. 135--147.Google ScholarDigital Library
- Feng Wang, Nathan Marshak, Will Usher, Carsten Burstedde, Aaron Knoll, Timo Heister, and Chris R. Johnson. 2020. CPU Ray Tracing of Tree-Based Adaptive Mesh Refinement Data. Computer Graphics Forum 39, 3 (2020), 1--12.Google ScholarCross Ref
- Weiqun Zhang, Ann Almgren, Vince Beckner, John Bell, Johannes Blaschke, Cy Chan, Marcus Day, Brian Friesen, Kevin Gott, Daniel Graves, et al. 2019. AMReX: a framework for block-structured adaptive mesh refinement. Journal of Open Source Software 4, 37 (2019), 1370--1370.Google ScholarCross Ref
- Kai Zhao, Sheng Di, Maxim Dmitriev, Thierry-Laurent D Tonellot, Zizhong Chen, and Franck Cappello. 2021. Optimizing error-bounded lossy compression for scientific data by dynamic spline interpolation. In 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 1643--1654.Google ScholarCross Ref
Index Terms
- AMRIC: A Novel In Situ Lossy Compression Framework for Efficient I/O in Adaptive Mesh Refinement Applications
Recommendations
Temporal Lossless and Lossy Compression in Wireless Sensor Networks
Energy efficiency is one of the most critical issues in the design and deployment of Wireless Sensor Networks (WSNs). Data compression is an important approach to reducing energy consumption of data gathering in multihop sensor networks. Existing ...
Efficient Context-Based Entropy Coding Lossy Wavelet Image Compression
DCC '97: Proceedings of the Conference on Data CompressionWe present an adaptive image coding algorithm based on novel backward-adaptive quantization/classification techniques. We use a simple uniform scalar quantizer to quantize the image subbands. Our algorithm puts the coefficient into one of several ...
Comments