Skip to main content
SpringerLink
Log in
Book cover

Smoky Mountains Computational Sciences and Engineering Conference

SMC 2021: Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation pp 99–118Cite as

Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2

  • Conference paper
  • First Online:

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1512))

Abstract

This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment. By using the openPMP-api, traditional workflows limited by filesystem bottlenecks can be overcome and flexibly extended for in situ analysis. The openPMD-api is a library for the description of scientific data according to the Open Standard for Particle-Mesh Data (openPMD). Its approach towards recent challenges posed by hardware heterogeneity lies in the decoupling of data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory. This paper surveys two openPMD-based loosely-coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightly-coupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange. We determine new challenges in resource allocation and in the need of strategies for a flexible data distribution, demonstrating their influence on efficiency and scaling on the Summit compute system. The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://www.top500.org/.

  2. 2.

    https://github.com/openPMD/openPMD-projects.

  3. 3.

    https://ofiwg.github.io/libfabric/.

  4. 4.

    https://openpmd-api.readthedocs.io/en/0.13.3/usage/streaming.html.

  5. 5.

    "QueueFullPolicy" = "Discard" The alternative is to block. A queue of steps can be held for some additional leeway, but it requires additional memory.

  6. 6.

    After recent successful streaming setups at 1024 nodes, the likely cause for this were scalability issues in the metadata strategy used in the openPMD-api.

References

  1. Abbasi, H., Wolf, M., Eisenhauer, G., Klasky, S., Schwan, K., Zheng, F.: Datastager: scalable data staging services for petascale applications. Cluster Comput. 13(3), 277–290 (2010). https://doi.org/10.1145/1551609.1551618

    Article  Google Scholar 

  2. Ahn, D.H., Garlick, J., Grondona, M., Lipari, D., Springmeyer, B., Schulz, M.: Flux: a next-generation resource management framework for large HPC centers. In: 2014 43rd International Conference on Parallel Processing Workshops, pp. 9–17 (2014). https://doi.org/10.1109/ICPPW.2014.15

  3. Bussmann, M., et al.: Radiative signatures of the relativistic Kelvin-Helmholtz instability. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC 2013, pp. 5:1–5:12. ACM, New York (2013). ISBN 978-1-4503-2378-9, https://doi.org/10.1145/2503210.2504564

  4. Rocklin, M.: Dask: parallel computation with blocked algorithms and task scheduling. In: Huff, K., Bergstra, J. (eds.) Proceedings of the 14th Python in Science Conference, pp. 130–136 (2015). https://dask.org

  5. E, J.C., Wang, L., Chen, S., Zhang, Y.Y., Luo, S.N.: GAPD: a GPU-accelerated atom-based polychromatic diffraction simulation code. J. Synchrotron Radiat. 25(2), 604–611 (2018). https://doi.org/10.1107/S1600577517016733

  6. Fortmann-Grote, C., et al.: SIMEX: simulation of experiments at advanced light sources. In: 11th NOBUGS Conference, Copenhagen, pp. 29–34 (2016). https://doi.org/10.17199/NOBUGS2016.21

  7. Fortmann-Grote, C., et al.: Simulations of ultrafast X–ray laser experiments. In: Proceedings of SPIE, Advances in X-ray Free-Electron Lasers Instrumentation IV, vol. 10237, p. 102370S (2017). https://doi.org/10.1117/12.2270552

  8. Godoy, W.F., et al.: ADIOS 2: the adaptable input output system. A framework for high-performance data management. SoftwareX 12, 100561 (2020). https://doi.org/10.1016/j.softx.2020.100561

    Article  Google Scholar 

  9. Huebl, A.: PIConGPU: predictive simulations of laser-particle accelerators with manycore hardware. Ph.D. thesis, Technische Universität Dresden (2019). https://doi.org/10.5281/zenodo.3266820

  10. Huebl, A., et al.: On the scalability of data reduction techniques in current and upcoming HPC systems from an application perspective. In: Kunkel, J.M., Yokota, R., Taufer, M., Shalf, J. (eds.) ISC High Performance 2017. LNCS, vol. 10524, pp. 15–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67630-2_2

  11. Huebl, A., et al.: openPMD: a meta data standard for particle and mesh based data (2015). https://doi.org/10.5281/zenodo.591699, https://github.com/openPMD

  12. Johnson, D.: Near-optimal bin packing algorithms. Ph.D. thesis, Massachusetts Institute of Technology (1973)

    Google Scholar 

  13. Koller, F., Poeschel, F., Gu, J., Huebl, A.: openPMD-api: C++ & Python API for scientific I/O with openPMD (2018). https://github.com/openPMD/openPMD-api. https://doi.org/10.14278/rodare.27

  14. Kress, J., et al.: Comparing time-to-solution for in situ visualization paradigms at scale. In: 2020 IEEE 10th Symposium on Large Data Analysis and Visualization (LDAV), pp. 22–26 (2020). https://doi.org/10.1109/LDAV51489.2020.00009

  15. Larsen, M., et al.: The alpine in situ infrastructure: ascending from the ashes of strawman. In: Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization, pp. 42–46 (2017). https://doi.org/10.1145/3144769.3144778

  16. Liu, Q., et al.: Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurr. Comput.: Pract. Exp. 26(7), 1453–1473 (2014). https://doi.org/10.1002/cpe.3125

    Article  Google Scholar 

  17. Logan, J., et al.: Extending the publish/subscribe abstraction for high-performance I/O and data management at extreme scale. IEEE Data Eng. Bull. 43, 35–46 (2020)

    Google Scholar 

  18. Loring, B., et al.: Improving performance of M-to-N processing and data redistribution in In Transit analysis and visualization. In: Frey, S., Huang, J., Sadlo, F. (eds.) Eurographics Symposium on Parallel Graphics and Visualization. The Eurographics Association (2020). ISBN 978-3-03868-107-6, https://doi.org/10.2312/pgv.20201073

  19. Mayes, C.E., et al.: Lightsource unified modeling environment (LUME) - a start-to-end simulation ecosystem. In: IPAC (2021)

    Google Scholar 

  20. Poeschel, F., et al.: Supplementary material: transitioning from file-based HPC workflows to streaming data pipelines with openPMD and ADIOS2. https://doi.org/10.5281/zenodo.4906276

  21. Vazhkudai, S.S., et al.: The design, deployment, and evaluation of the coral pre-exascale systems. In: SC18: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 661–672 (2018). https://doi.org/10.1109/SC.2018.00055

  22. Wan, L., et al.: Improving I/O performance for exascale applications through online data layout reorganization. Under review (2021)

    Google Scholar 

  23. Wilkinson, M.D., et al.: The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3(1), 160018 (2016). ISSN 2052-4463, https://doi.org/10.1038/sdata.2016.18

  24. Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016). ISSN 0001-0782, https://doi.org/10.1145/2934664

Download references

Acknowledgements

This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of two U.S. Department of Energy organizations (Office of Science and the National Nuclear Security Administration). Supported by EC through Laserlab-Europe, H2020 EC-GA 871124. Supported by the Consortium for Advanced Modeling of Particles Accelerators (CAMPA), funded by the U.S. DOE Office of Science under Contract No. DE-AC02-05CH11231. This work was partially funded by the Center of Advanced Systems Understanding (CASUS), which is financed by Germany’s Federal Ministry of Education and Research (BMBF) and by the Saxon Ministry for Science, Culture and Tourism (SMWK) with tax funds on the basis of the budget approved by the Saxon State Parliament.

Author information

Authors and Affiliations

  1. Center for Advanced Systems Understanding (CASUS), 02826, Görlitz, Germany

    Franz Poeschel & Michael Bussmann

  2. Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA, 94720, USA

    Junmin Gu & Axel Huebl

  3. Oak Ridge National Laboratory (ORNL), Oak Ridge, TN, 37830, USA

    William F. Godoy, Norbert Podhorszki, Scott Klasky, Lipeng Wan & Ana Gainaru

  4. Helmholtz-Zentrum Dresden-Rossendorf (HZDR), 01328, Dresden, Germany

    Franz Poeschel, Fabian Koller, René Widera, Michael Bussmann & Axel Huebl

  5. European XFEL GmbH (EU XFEL), 22869, Schenefeld, Germany

    Juncheng E

  6. Georgia Institute of Technology (Georgia Tech), Atlanta, GA, 30332, USA

    Greg Eisenhauer

  7. Rutgers University (Rutgers), New Brunswick, NJ, 08901, USA

    Philip E. Davis

Authors
  1. Franz Poeschel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juncheng E
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. William F. Godoy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Norbert Podhorszki
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Scott Klasky
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Greg Eisenhauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Philip E. Davis
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Lipeng Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ana Gainaru
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Junmin Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Fabian Koller
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. René Widera
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Michael Bussmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Axel Huebl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Franz Poeschel or Axel Huebl .

Editor information

Editors and Affiliations

  1. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Jeffrey Nichols

  2. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Arthur ‘Barney’ Maccabe

  3. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    James Nutaro

  4. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Swaroop Pophale

  5. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Pravallika Devineni

  6. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Theresa Ahearn

  7. Oak Ridge National Laboratory, Oak Ridge, TN, USA

    Becky Verastegui

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Poeschel, F. et al. (2022). Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2. In: Nichols, J., et al. Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation. SMC 2021. Communications in Computer and Information Science, vol 1512. Springer, Cham. https://doi.org/10.1007/978-3-030-96498-6_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-96498-6_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-96497-9

  • Online ISBN: 978-3-030-96498-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation