Skip to main content
SpringerLink
Log in

Domain-Specific Type-Safe APIs for Hierarchical Scientific Data with Modern C++

  • Conference paper
  • First Online:
Responsible Data Science

Part of the book series: Lecture Notes in Electrical Engineering ((LNEE,volume 940))

  • 255 Accesses

Abstract

General-purpose library application programming interfaces (APIs) for self-describing hierarchical scientific data storage, such as the HDF5 and NetCDF libraries, are traditionally of runtime nature. Runtime errors for entry existence and data types are typically caught later in the development process of higher-level application-specific APIs. In this paper, we propose exploiting modern C++ metaprogramming features to add compile-time type-safety to improve the interaction with a well-defined metadata-rich scientific schema in domain-specific hierarchical datasets. We tackle two aspects of common use: (i) direct data access, (ii) flexible “in-memory” index models for efficient search and data processing. The proposed APIs use C++17’s template type auto deduction features, C++11’s enum class for type-safety and C-style preprocessor macros for generative templated code. We showcase the pros and cons of our initial work on the standard NeXus schema used for annotating and storing experimental neutron scattering data at several facilities around the world on top of HDF5. Extendable compile-time type-safe APIs are a desirable feature that could be indexed by any modern integrated development environment (IDE). Hence, such APIs can help ease the learning curve for domain scientists using a less error-prone software interaction to enhance the findability of their data without resorting to a domain-specific language (DSL).

The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). Work at Oak Ridge National Laboratory was sponsored by the Division of Scientific User Facilities, Office of Basic Energy Sciences, US Department of Energy, under Contract no. DE-AC05-00OR22725 with UT-Battelle, LLC.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/ORNL/ncio.

References

  1. Antcheva I, Ballintijn M, Bellenot B, Biskup M, Brun R, Buncic N, Canal P, Casadei D, Couet O, Fine V, Franco L, Ganis G, Gheata A, Maline DG, Goto M, Iwaszkiewicz J, Kreshuk A, Segura DM, Maunder R, Moneta L, Naumann A, Offermann E, Onuchin V, Panacek S, Rademakers F, Russo P, Tadel M (2009) ROOT—a C++ framework for petabyte data storage, statistical analysis and visualization. Comput Phys Commun 180(12):2499–2512. https://doi.org/10.1016/j.cpc.2009.08.005. 40 YEARS OF CPC: A celebratory issue focused on quality software for high performance, grid and novel computing architectures

  2. Byna S, Breitenfeld MS, Dong B, Koziol Q, Pourmal E, Robinson D, Soumagne J, Tang H, Vishwanath V, Warren R (2020) ExaHDF5: delivering efficient parallel I/O on exascale computing systems. J Comput Sci Technol 35(1). https://doi.org/10.1007/s11390-020-9822-9

  3. Eclipse Foundation: Eclipse CDT (C/C++ development tooling). https://www.eclipse.org/cdt/

  4. Folk M, Heber G, Koziol Q, Pourmal E, Robinson D (2011) An overview of the HDF5 technology suite and its applications. In: Proceedings of the EDBT/ICDT 2011 workshop on array databases, AD ’11. Association for Computing Machinery, New York, NY, USA, pp 36–47. https://doi.org/10.1145/1966895.1966900

  5. Gaur N, Mahammad S, Sharma V, Dhar D, Ramakrishnan R (2012) Template based HDF5 satellite digital data product generation software. In: 2012 international conference on data science and engineering (ICDSE), pp 160–165. https://doi.org/10.1109/ICDSE.2012.6281903

  6. Godoy WF, Peterson PF, Hahn SE, Billings JJ (2020) Efficient data management in neutron scattering data reduction workflows at ORNL. In: 2020 IEEE international conference on big data (big data), pp 2674–2680. https://doi.org/10.1109/BigData50022.2020.9377836

  7. Godoy WF, Peterson PF, Hahn SE, Hetrick J, Doucet M, Billings JJ (2020) Performance improvements on SNS and HFIR instrument data reduction workflows using Mantid. In: Nichols J, Verastegui B, Maccabe AB, Hernandez O, Parete-Koon S, Ahearn T (eds) Driving scientific and engineering discoveries through the convergence of HPC, big data and AI. Springer, Cham, pp 175–186

    Chapter  Google Scholar 

  8. Godoy WF, Podhorszki N, Wang R, Atkins C, Eisenhauer G, Gu J, Davis P, Choi J, Germaschewski K, Huck K, Huebl A, Kim M, Kress J, Kurc T, Liu Q, Logan J, Mehta K, Ostrouchov G, Parashar M, Poeschel F, Pugmire D, Suchyta E, Takahashi K, Thompson N, Tsutsumi S, Wan L, Wolf M, Wu K, Klasky S (2020) ADIOS 2: the adaptable input output system. A framework for high-performance data management. SoftwareX 12:100561. https://doi.org/10.1016/j.softx.2020.100561

  9. Godoy WF, Savici AT, Hahn SE, Peterson PF (2021) Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities. In: (2021) IEEE International Conference on Big Data (Big Data), 2021, pp. 2949–2955. https://doi.org/10.1109/BigData52589.2021.9671354

  10. Golasowski M, Bispo J, Martinovič J, Slaninová K, Cardoso JMP (2017) Expressing and applying C++ code transformations for the HDF5 API through a DSL. In: Saeed K, Homenda W, Chaki R (eds) Computer information systems and industrial management. Springer, Cham, pp 303–314

    Chapter  Google Scholar 

  11. Gosink L, Shalf J, Stockinger K, Wu K, Bethel W (2006) HDF5-FastQuery: accelerating complex queries on HDF datasets using fast bitmap indices. In: 18th international conference on scientific and statistical database management (SSDBM’06), pp 149–158. https://doi.org/10.1109/SSDBM.2006.27

  12. Gray J, Liu DT, Nieto-Santisteban M, Szalay A, DeWitt DJ, Heber G (2005) Scientific data management in the coming decade. SIGMOD Rec 34(4):34–41. https://doi.org/10.1145/1107499.1107503

    Article  Google Scholar 

  13. Guennebaud G, Jacob B et al (2010) Eigen v3. https://eigen.tuxfamily.org

  14. Hinsen K (2013) Software development for reproducible research. Comput Sci Eng 15(4):60–63. https://doi.org/10.1109/MCSE.2013.91

    Article  Google Scholar 

  15. Ishwarappa Anuradha J (2015) A brief introduction on big data 5Vs characteristics and Hadoop technology. Procedia Comput Sci 48:319–324. https://doi.org/10.1016/j.procs.2015.04.188

    Article  Google Scholar 

  16. Könnecke M, Akeroyd FA, Bernstein HJ, Brewster AS, Campbell SI, Clausen B, Cottrell S, Hoffmann JU, Jemian PR, Männicke D, Osborn R, Peterson PF, Richter T, Suzuki J, Watts B, Wintersberger E, Wuttke J (2015) The NeXus data format. J Appl Crystallogr 48(1):301–305. https://doi.org/10.1107/S1600576714027575

    Article  Google Scholar 

  17. Lofstead J, Ross R (2013) Insights for exascale IO APIs from building a petascale IO API. In: SC ’13: proceedings of the international conference on high performance computing, networking, storage and analysis, pp 1–12. https://doi.org/10.1145/2503210.2503238

  18. Nam B, Sussman A (2003) Improving access to multi-dimensional self-describing scientific datasets. In: CCGrid 2003. 3rd IEEE/ACM international symposium on cluster computing and the grid, 2003. Proceedings, pp 172–179. https://doi.org/10.1109/CCGRID.2003.1199366

  19. Rew R, Davis G (1990) NetCDF: an interface for scientific data access. IEEE Comput Graph Appl 10(4):76–82. https://doi.org/10.1109/38.56302

    Article  Google Scholar 

  20. Ritchie DM, Thompson K (1974) The UNIX time-sharing system. Commun ACM 17(7):365–375. https://doi.org/10.1145/361011.361061

    Article  MATH  Google Scholar 

  21. Sanderson C, Curtin R (2016) Armadillo: a template-based C++ library for linear algebra. J Open Source Soft 1(2):26. https://doi.org/10.21105/joss.00026

  22. Silvano C, Agosta G, Bartolini A, Beccari AR, Benini L, Besnard L, Bispo J, Cmar R, Cardoso JM, Cavazzoni C, Cesarini D, Cherubin S, Ficarelli F, Gadioli D, Golasowski M, Libri A, Martinovič J, Palermo G, Pinto P, Rohou E, Slaninová K, Vitali E (2019) The ANTAREX domain specific language for high performance computing. Microprocessors Microsyst 68:58–73. https://doi.org/10.1016/j.micpro.2019.05.005

    Article  Google Scholar 

  23. Spreckelsen F, Rüchardt B, Lebert J, Luther S, Parlitz U, Schlemmer A (2020) Guidelines for a standardized filesystem layout for scientific data. Data 5(2). https://doi.org/10.3390/data5020043

  24. Stroustrup B (2013) The C++ programming language, 4th edn. Addison-Wesley Professional

    Google Scholar 

  25. Sutton A, Holeman R, Maletic JI (2009) Abstracting the template instantiation relation in C++. In: 2009 IEEE international conference on software maintenance, pp 559–562. https://doi.org/10.1109/ICSM.2009.5306392

  26. The HDF Group. Hierarchical data format, version 5 (1997-NNNN). http://www.hdfgroup.org/HDF5/

  27. Touton J, Spertus M (2016) Declaring non-type template parameters with auto. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0127r2.html

  28. Tratt L (2008) Domain specific language implementation via compile-time meta-programming. ACM Trans Program Lang Syst 30(6). https://doi.org/10.1145/1391956.1391958

  29. Varga S: Easy to use HDF5 C++ templates for serial and parallel HDF5. https://github.com/steven-varga/h5cpp

  30. Wells D, Greisen E, Harten R (1981) FITS—a flexible image transport system. Astron Astrophys Suppl 44:363

    Google Scholar 

  31. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, Blomberg N, Boiten JW, da Silva Santos LB, Bourne PE et al (2016) The FAIR guiding principles for scientific data management and stewardship. Sci Data 3

    Google Scholar 

  32. Zhang W, Byna S, Niu C, Chen Y (2019) Exploring metadata search essentials for scientific data management. In: 2019 IEEE 26th international conference on high performance computing, data, and analytics (HiPC), pp 83–92

    Google Scholar 

  33. Zhang W, Byna S, Tang H, Williams B, Chen Y (2019) Miqs: metadata indexing and querying service for self-describing file formats. In: Proceedings of the international conference for high performance computing, networking, storage and analysis, SC ’19. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3295500.3356146

Download references

Acknowledgements

Work at Oak Ridge National Laboratory was sponsored by the Division of Scientific User Facilities, Office of Basic Energy Sciences, US Department of Energy, under Contract no. DE-AC05-00OR22725 with UT-Battelle, LLC.

Author information

Authors and Affiliations

  1. Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, 37830, USA

    William F. Godoy, Addi Malviya Thakur & Steven E. Hahn

Authors
  1. William F. Godoy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Addi Malviya Thakur
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Steven E. Hahn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William F. Godoy .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, India

    Jimson Mathew

  2. Department of Computer Science, Cochin University of Science and Technology, Cochin, Kerala, India

    G. Santhosh Kumar

  3. School of EEECS, Queens University Belfast, London, UK

    Deepak P.

  4. School of Computing Science, University of Glasgow, Glasgow, UK

    Joemon M. Jose

Rights and permissions

Reprints and permissions

Copyright information

© 2022 This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Godoy, W.F., Thakur, A.M., Hahn, S.E. (2022). Domain-Specific Type-Safe APIs for Hierarchical Scientific Data with Modern C++. In: Mathew, J., Santhosh Kumar, G., P., D., Jose, J.M. (eds) Responsible Data Science. Lecture Notes in Electrical Engineering, vol 940. Springer, Singapore. https://doi.org/10.1007/978-981-19-4453-6_14

Download citation

  • DOI: https://doi.org/10.1007/978-981-19-4453-6_14

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-19-4452-9

  • Online ISBN: 978-981-19-4453-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation