Abstract
In the context of the contemporary push for “big data” in many fields, we review recent experiences building large databases for turbulence research. We consider data from direct numerical simulations (DNS) of various canonical flows and from experimental studies and related numerical simulations of wall-bounded turbulence, where the data storage needs are particularly challenging due to the very large range of length and time scales that exists in these flows at high Reynolds numbers. The focus is on a move from the traditional approach of data-handling and analysis where datasets are moved to individual computers, to one where much of the analysis is moved to the hosting system that stores these data. In this context we give a summary of a unique open numerical laboratory that archives over 200 Terabytes of DNS data, including full spatio-temporal flow fields of various canonical flows. Particular attention is given to the unique access requirements for large datasets to become open to the research community and the success the system has had in democratizing access to large datasets.
Keywords
- Particle Image Velocimetry
- Direct Numerical Simulation
- Simple Object Access Protocol
- Direct Numerical Simulation Data
- Virtual Sensor
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
A. Hey, S. Tansley, K. Tolle, et al., The Fourth Paradigm: Data-intensive scientific discovery, Microsoft Research Redmond, WA, 2009
D. Donzis, P. Yeung, K. Sreenivasan, Dissipation and enstrophy in isotropic turbulence: resolution effects and scaling in direct numerical simulations. Phys. Fluids 20, 045108 (2008)
T. Ishihara, T. Gotoh, Y. Kaneda, Study of high-Reynolds number isotropic turbulence by direct numerical simulation. Annu. Rev. Fluid Mech. 41, 165–180 (2009)
X. Wu, P. Moin, Direct numerical simulation of turbulence in a nominally zero-pressure-gradient flat-plate boundary layer. J. Fluid Mech. 630, 5–41 (2009)
Y. Khalighi, J. Nichols, S. Lele, F. Ham, P. Moin, Unstructured large eddy simulation for prediction of noise issued from turbulent jets in various configurations, AIAA paper 2886, 2011
M. Lee, R. Moser, Direct numerical simulation of turbulent channel flow up to Re τ = 5200. J. Fluid Mech. 774, 395–415 (2015)
A. Smits, I. Marusic, Wall-bounded turbulence. Phys. Today 66, 25–30 (2013)
I. Marusic, G.V. Candler, V. Interrante, P.K. Subbareddy, A. Moss, Real time feature extraction for the analysis of turbulent flows, in Data Mining for Scientific and Engineering Applications, ed. by R. Grossman, C. Kamath, P. Kegelmeyer, V. Kumar, R. Namburu (Springer, US, 2001), pp. 223–238
R.J. Adrian, J. Westerweel, Particle Image Velocimetry (Cambridge University Press, Cambridge, 2011)
A. Schröder, R. Geisler, K. Staack, G. Elsinga, F. Scarano, B. Wieneke, A. Henning, C. Poelma, J. Westerweel, Eulerian and Lagrangian views of a turbulent boundary layer flow using time-resolved tomographic PIV. Exp. Fluids 50, 1071–1091 (2011)
C.M. de Silva, E. Gnanamanickam, C. Atkinson, N.A. Buchmann, N. Hutchins, J. Soria, I. Marusic, High spatial range velocity measurements in a high Reynolds number turbulent boundary layer. Phys. Fluids 26, 025117 (2014)
A.S. Szalay, P. Kunszt, A. Thakar, J. Gray, D. Slutz, R.J. Brunner, Designing and mining multi-terabyte astronomy archives: the Sloan digital sky survey, in Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (2000), pp. 451–462
W. O’Mullane, J. Gray, N. Li, T. Budavari, M. Nieto-Santisteban, A. Szalay, Batch query system with interactive local storage for SDSS and the VO. in Proceedings of the ADASS XIII, ASP Conference Series, ed. by F. Ochsenbein, M. Allen, D. Egret (2004), pp. 314–372
G. Lemson, the Virgo Consortium, Halo and galaxy formation histories from the millennium simulation: public release of a VO-oriented and SQL-queryable database for studying the evolution of galaxies in the lambdaCDM cosmogony. arXiv:astro-ph/0608019 (2006)
R. Overzier, G. Lemson, R. Angulo, et al., The millennium run observatory: first light. MNRAS 428, 778 (2013)
R. Burns, J. Vogelstein, A. Szalay, From cosmos to connectomes: the evolution of data-intensive science. Neuron 83, 1249–1252 (2014)
Y. Li, E. Perlman, M. Wan, Y. Yang, C. Meneveau, R. Burns, S. Chen, A. Szalay, G. Eyink, A public turbulence database cluster and applications to study Lagrangian evolution of velocity increments in turbulence. J. Turbul. 9, N31 (2008)
K. Kanov, C. Burns, R. Lalescu, G. Eyink, The Johns Hopkins turbulence databases: an open simulation laboratory for turbulence research. Comput. Sci. Eng. 17 (5), 10–17 (2015)
Y. Gu, R. Grossman, A. Szalay, A. Thakar, Distributing the Sloan Digital Sky Survey using UDT and sector, in e-Science (2006), p. 56
H. Samet, Foundations of Multidimensional and Metric Data Structures (Morgan Kaufmann Publishers, Los Altos, CA, 2006)
E. Perlman, R. Burns, Y. Li, C. Meneveau, Data exploration of turbulence simulations using a database cluster, in SC07 07 (2007)
J. Graham, K. Kanov, E. Givelberg, R. Burns, G. Eyink, A. Szalay, C. Meneveau, M. Lee, N. Malaya, R. Moser, A web-services accessible database for channel flow turbulence at Re-tau = 1000. Bull. Am. Phys. Soc. 58, 236 (2013)
M. Lee, N. Malaya, R. Moser, Petascale direct numerical simulation of turbulent channel flow on up to 786k cores, in Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis (Association for Computing Machinery, New York, 2013), p. 61
J. Graham, K. Kanov, X. Yang, M. Lee, N. Malaya, C. Lalescu, R. Burns, G. Eyink, A. Szalay, R. Moser, C. Meneveau, A web services-accessible database of turbulent channel flow and its use for testing a new integral wall model for LES. J. Turbul. 17 (2), 181–215 (2016)
D. Livescu, J. Ristorcelli, Variable-density mixing in buoyancy-driven turbulence. J. Fluid Mech. 605, 145–180 (2008)
D. Livescu, Numerical simulations of two-fluid turbulent mixing at large density ratios and applications to the Rayleigh–Taylor instability. Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci. 371, 20120185 (2013)
K. Kanov, R. Burns, G. Eyink, C. Meneveau, A. Szalay, Data-intensive spatial filtering in large numerical simulation datasets, in International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (IEEE Computer Society Press Los Alamitos, CA, 2012)
K. Kanov, R. Burns, C. Lalescu, Efficient evaluation of threshold queries of derived fields in a numerical simulation database, in Proceedings of the 18th International Conference on Extending Database Technology (EDBT), 2015
G. Eyink, E. Vishniac, C. Lalescu, H. Aluie, K. Kanov, K. Bürger, R. Burns, C. Meneveau, A. Szalay, Flux-freezing breakdown in high-conductivity magnetohydrodynamic turbulence. Nature 497, 466–469 (2013)
F. Alexander, C. Meneveau, Foreword to the “open simulation laboratories” special issue. Comput. Sci. Eng. 17 (5), 7–9 (2015)
K. Kanov, Efficient evaluation of data-intensive batch-queries in open simulation laboratories, Ph.D. thesis, Johns Hopkins University, 2015
P. Johnson, C. Meneveau, Large-deviation joint statistics of the finite-time Lyapunov spectrum in isotropic turbulence. Phys. Fluids 27 (8), 085110 (2015)
G. Eyink, Stochastic flux freezing and magnetic dynamo. Phys. Rev. E 83, 056405 (2011)
H. Yu, C. Meneveau, Lagrangian refined Kolmogorov similarity hypothesis for gradient time evolution and correlation in turbulent flows. Phys. Rev. Lett. 104, 084502 (2010)
H. Yu, C. Meneveau, Scaling of conditional Lagrangian time correlation functions of velocity and pressure gradient magnitudes in isotropic turbulence. Flow Turbul. Combust. 85, 457–472 (2010)
Y. Li, L. Chevillard, C. Meneveau, G. Eyink, Matrix exponential-based closures for the turbulent subgrid-scale stress tensor. Phys. Rev. E 79, 016305 (2009)
K. Kanov, E. Perlman, R. Burns, Y. Ahmad, A. Szalay, I/O streaming evaluation of batch queries for data-intensive computational turbulence, in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (Association for Computing Machinery, New York, 2011), p. 29
X. Wang, E. Perlman, R. Burns, T. Malik, T. Budavári, C. Meneveau, A. Szalay, Jaws: job-aware workload scheduling for the exploration of turbulence simulations, in Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (IEEE Computer Society, Washington DC, 2010), pp. 1–11
H. Yu, K. Kanov, E. Perlman, J. Graham, E. Frederix, R. Burns, A. Szalay, G. Eyink, C. Meneveau, Studying Lagrangian dynamics of turbulence using on-demand fluid particle tracking in a public turbulence database. J. Turbul. 13, N12 (2012)
M. Treib, K. Burger, F. Reichl, C. Meneveau, A. Szalay, R. Westermann, Turbulence visualization at the Terascale on desktop PCs. IEEE Trans. Vis. Comput. Graph. 18, 2169–2177 (2012)
G. Eyink, D. Benveniste, Diffusion approximation in turbulent two-particle dispersion. Phys. Rev. E 88, 041001 (2013)
J. Graham, E. Givelberg, K. Kanov, Run-time creation of the turbulent channel flow database by an HPC simulation using MPI-DB, in Proceedings of the 20th European MPI Users’ Group Meeting (Association for Computing Machinery, New York, 2013), pp. 151–156
D. Benveniste, T. Drivas, Asymptotic results for backwards two-particle dispersion in a turbulent flow. Phys. Rev. E 89, 041003 (2014)
C. Meneveau, Lagrangian dynamics and models of the velocity gradient tensor in turbulent flows. Annu. Rev. Fluid Mech. 43, 219–245 (2011)
C. Meneveau, Germano identity-based subgrid-scale modeling: a brief survey of variations on a fertile theme. Phys. Fluids 24, 121301 (2012)
J. Boschung, P. Schaefer, N. Peters, C. Meneveau, The local topology of stream-and vortex lines in turbulent flows. Phys. Fluids 26, 045107 (2014)
Y. Ahmad, R. Burns, M. Kazhdan, C. Meneveau, A. Szalay, A. Terzis, Scientific data management at the Johns Hopkins Institute for Data Intensive Engineering and Science. ACM SIGMOD Rec. 39, 18–23 (2011)
H. Xu, A. Pumir, G. Falkovich, E. Bodenschatz, M. Shats, H. Xia, N. Francois, G. Boffetta, Flight–crash events in turbulence. Proc. Natl. Acad. Sci. 111, 7558–7563 (2014)
J. Jucha, H. Xu, A. Pumir, E. Bodenschatz, Time-reversal-symmetry breaking in turbulence. Phys. Rev. Lett. 113, 054501 (2014)
K. Gustavsson, J. Einarsson, B. Mehlig, Tumbling of small axisymmetric particles in random and turbulent flows. Phys. Rev. Lett. 112, 014501 (2014)
N. Buchmann, C. Willert, J. Soria, Pulsed, high-power LED illumination for tomographic particle image velocimetry. Exp. Fluids 53, 1545–1560 (2012)
X. Liu, J. Katz, Vortex-corner interactions in a cavity shear layer elucidated by time-resolved measurements of the pressure field. J. Fluid Mech. 728, 417–457 (2013)
D. Xu, J. Chen, Accurate estimate of turbulent dissipation rate using PIV data. Exp. Therm. Fluid Sci. 44, 662–672 (2013)
D. Fiscaletti, J. Westerweel, G. Elsinga, Long-range μPIV to resolve the small scales in a jet at high Reynolds number. Exp. Fluids 55, 1–15 (2014)
J.M. Lawson, J.R. Dawson, On velocity gradient dynamics and turbulent structure. J. Fluid Mech. 780, 60–98 (2015)
B. Luethi, M. Holzer, A. Tsinober, Expanding the Q–R space to three dimensions. J. Fluid Mech. 641, 497–507 (2010)
J. Cardesa, D. Mistry, L. Gan, J. Dawson, Invariants of the reduced velocity gradient tensor in turbulent flows. J. Fluid Mech. 716, 597–615 (2013)
H. Lu, Assessment of the modulated gradient model in decaying isotropic turbulence. Theor. Appl. Mech. Lett. 1, 041004 (2011)
A.G. Gungor, S. Menon, A new two-scale model for large eddy simulation of wall-bounded flows. Prog. Aerospace Sci. 46, 28–45 (2010)
F. Grinstein, A. Gowardhan, A. Wachtor, Simulations of Richtmyer–Meshkov instabilities in planar shock-tube experiments. Phys. Fluids 23, 034106 (2011)
W. Liu, E. Ribeiro, Scale and rotation invariant detection of singular patterns in vector flow fields, in Structural, Syntactic, and Statistical Pattern Recognition (Springer, Berlin, Heidelberg, 2010), pp. 522–531
P. Bhat, K. Subramanian, Fluctuation dynamos and their Faraday rotation signatures. Month. Not. R. Astron. Soc. 429, 2469–2481 (2013)
C. Keylock, T. Tokyay, G. Constantinescu, A method for characterising the sensitivity of turbulent flow fields to the structure of inlet turbulence. J. Turbul. 12, N45 (2011)
M. Holzner, M. Guala, B. Lüthi, A. Liberzon, N. Nikitin, W. Kinzelbach, A. Tsinober, Viscous tilting and production of vorticity in homogeneous turbulence. Phys. Fluids 22, 061701 (2010)
C. Wu, T. Chang, Rank-ordered multifractal analysis (ROMA) of probability distributions in fluid turbulence. Nonlinear Processes Geophys. 18, 261–268 (2011)
C. Keylock, K. Nishimura, J. Peinke, A classification scheme for turbulence based on the velocity-intermittency structure with an application to near-wall flow and with implications for bed load transport. J. Geophys. Res.: Earth Surf. (2003–2012) 117, F1 (2012)
W. Liu, E. Ribeiro, Detecting singular patterns in 2D vector fields using weighted Laurent polynomial. Pattern Recognit. 45, 3912–3925 (2012)
M. Mishra, X. Liu, M. Skote, C.-W. Fu, Kolmogorov spectrum consistent optimization for multi-scale flow decomposition. Phys. Fluids 26, 055106 (2014)
L. Moriconi, R. Pereira, Vorticity statistics and the time scales of turbulent strain. Phys. Rev. E 88, 013005 (2013)
F. Grinstein, A. Gowardhan, J. Ristorcelli, A. Wachtor, On coarse-grained simulations of turbulent material mixing. Phys. Scripta 86, 058203 (2012)
T. Chang, C. Wu, M. Echim, H. Lamy, M. Vogelsberger, L. Hernquist, D. Sijacki, Complexity phenomena and ROMA of the earth’s magnetospheric cusp, hydrodynamic turbulence, and the cosmic web. Pure Appl. Geophys. 172 (7), 2025–2043 (2015)
A. Pumir, H. Xu, G. Boffetta, G. Falkovich, E. Bodenschatz, Redistribution of kinetic energy in turbulent flows. Phys. Rev. X 4, 041006 (2014)
Acknowledgements
The authors congratulate Prof. William George, for whom this Festschrift is intended, on occasion of his 70th birthday and salute him for his many accomplishments in turbulence research. CM gratefully acknowledges the large collaborative and interdisciplinary effort that has made the JHTDB open numerical turbulence laboratory possible, specifically the following colleagues, scholars, students, and staff: A. Szalay, R. Burns, G. Eyink, S. Chen, E. Vishniac, T. Zaki, E. Perlman, Y. Li, T. Budavari, M. Wan, H. Yu, E. Frederix, K. Buerger, H. Aluie, J. Graham, C. Lalescu, K. Kanov, S. Hamilton, P. Johnson, D. Livescu, R.D. Moser, M. Lee and N. Malaya, K. Yang, J. Lee, J. Vandenberg, S. Werner, V. Paul, and G. Lemson. The funding from the National Science Foundation (grants CMMI-0941530 and CBET-1507469) is gratefully acknowledged. IM thanks the support of the Australian Research Council.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Meneveau, C., Marusic, I. (2017). Turbulence in the Era of Big Data: Recent Experiences with Sharing Large Datasets. In: Pollard, A., Castillo, L., Danaila, L., Glauser, M. (eds) Whither Turbulence and Big Data in the 21st Century?. Springer, Cham. https://doi.org/10.1007/978-3-319-41217-7_27
Download citation
DOI: https://doi.org/10.1007/978-3-319-41217-7_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-41215-3
Online ISBN: 978-3-319-41217-7
eBook Packages: EngineeringEngineering (R0)