Salil Mahajan
ABSTRACT
Effective utilization of novel hybrid architectures of pre-exascale and exascale machines requires transformations to global climate modeling systems that may not reproduce the original model solution bit-for-bit. Round-off level differences grow rapidly in these non-linear and chaotic systems. This makes it difficult to isolate bugs/errors from innocuous growth expected from round-off level differences. Here, we apply two modern multivariate two sample equality of distribution tests to evaluate statistical reproducibility of global climate model simulations using standard monthly output of short (~ 1-year) simulation ensembles of a control model and a modified model of US Department of Energy's Energy Exascale Earth System Model (E3SM). Both the tests are able to identify changes induced by modifications to some model tuning parameters. We also conduct formal power analyses of the tests by applying them on designed suites of short simulation ensembles each with an increasingly different climate from the control ensemble. The results are compared against those from another such test. These power analyses provide a framework to quantify the degree of differences that can be detected confidently by the tests for a given ensemble size (sample size). This will allow model developers using the tests to make an informed decision when accepting/rejecting an unintentional non-bit-for-bit change to the model solution.
- A. H. Baker, D. M. Hammerling, M. N. Levy, H. Xu, J. M. Dennis, B. E. Eaton, J. Edwards, C. Hannay, S. A. Mickelson, R. B. Neale, D. Nychka, J. Shollenberger, J. Tribbia, M. Vertenstein, and D. Williamson. 2015. A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0). Geoscientific Model Development 8, 9 (2015), 2829--2840.Google Scholar
Cross Ref
- E3SM Project. 2018. Energy Exascale Earth System Model (E3SM). {Computer Software}Google Scholar
- Pierre Gentine, Mike Pritchard, Stephan Rasp, Gael Reinaudi, and Galen Yacalis. 2018. Could machine learning break the convection parameterization deadlock? Geophysical Research Letters 45, 11 (2018), 5742--5751.Google ScholarCross Ref
- Chris Golaz and co authors. 2019. The DOE E3SM coupled model version 1: Overview and evaluation at standard resolution. Journal of Advances in Modeling of Earth Systems (accepted) (2019).Google Scholar
- Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A Kernel Two-sample Test. J. Mach. Learn. Res. 13, 1 (March 2012), 723--773. http://dl.acm.org/citation.cfm?id=2503308.2188410 Google ScholarDigital Library
- Joseph H. Kennedy, Salil Mahajan, and Katherine J. Evans. 2018. LIVVkit/evv4esm: EVV 0.1.1.Google Scholar
- Salil Mahajan, Katherine J. Evans, Joe Kennedy, Min Xu, Matt R. Norman, and Marcia L. Branstetter. 2019. Ongoing solution reproducibility of earth system models as they progress toward exascale computing. International Journal of High Performance Computing Applications (in press) (2019).Google Scholar
- Salil Mahajan, Abigail L. Gaddis, Katherine J. Evans, and Matthew R. Norman. 2017. Exploring an Ensemble-Based Approach to Atmospheric Climate Modeling and Testing at Scale. Procedia Computer Science 108 (2017), 735--744. International Conference on Computational Science, ICCS 2017, 12-14 June 2017, Zurich, Switzerland.Google Scholar
- F. Massonnet, M. Ménégoz, M. C. Acosta, X. Yepes-Arbós, E. Exarchou, and F.J. Doblas-Reyes. 2018. Reproducibility of an Earth System Model under a change in computing environment. Technical Report. Barcelona Supercomputing Center.Google Scholar
- Nvidia 2018. PGI compiler and tools: User's guide for x86-64 CPUs. Nvidia.Google Scholar
- Yun Qian, Huiping Yan, Zhangshuan Hou, Gardar Johannesson, Stephen Klein, Donald Lucas, Richard Neale, Philip Rasch, Laura Swiler, John Tannahill, Hailong Wang, Minghuai Wang, and Chun Zhao. 2015. Parametric sensitivity analysis of precipitation at global and local scales in the Community Atmosphere Model CAM5. Journal of Advances in Modeling Earth Systems 7, 2 (2015), 382--411.Google ScholarCross Ref
- Paul R. Rosenbaum. 2005. An exact distribution-free test comparing two multivariate distributions based on adjacency. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67, 4 (2005), 515--530.Google ScholarCross Ref
- J. Rosinski and D. Williamson. 1997. The Accumulation of Rounding Errors and Port Validation for Global Atmospheric Models. SIAM Journal on Scientific Computing 18, 2 (1997), 552--564. Google ScholarDigital Library
- Gábor J Székely and Maria L Rizzo. 2004. Testing for equal distributions in high dimension. InterStat 5 (2004), 1249--1272.Google Scholar
- H. Wan, K. Zhang, P. J. Rasch, B. Singh, X. Chen, and J. Edwards. 2017. A new and inexpensive non-bit-for-bit solution reproducibility test based on time step convergence (TSC1.0). Geoscientific Model Development 10, 2 (2017), 537--552.Google ScholarCross Ref
Index Terms
- A Multivariate Approach to Ensure Statistical Reproducibility of Climate Model Simulations
Recommendations
Ensuring statistical reproducibility of ocean model simulations in the age of hybrid computing
PASC '21: Proceedings of the Platform for Advanced Scientific Computing ConferenceNovel high performance computing systems that feature hybrid architectures require large scale code refactoring to unravel underlying exploitable parallelism. Such redesign can often be accompanied with machine-precision changes as the order of ...
Asymptotics for testing hypothesis in some multivariate variance components model under non-normality
We consider the problem of deriving the asymptotic distribution of the three commonly used multivariate test statistics, namely likelihood ratio, Lawley-Hotelling and Bartlett-Nanda-Pillai statistics, for testing hypotheses on the various effects (main, ...
Rough Sets and Multivariate Statistical Classification: A Simulation Study
The classification of a set of objects into predefined homogenous groups is a problem with major practical interest in many fields. Over the past two decades several non-parametric approaches have been developed to address the classification problem, ...
Comments