Abstract
Cloud computing has empowered users to provision virtually unlimited computational resources and are accessible over the Internet on demand. This makes Cloud computing a compelling technology that tackles the issues rising with the growing size and complexity of scientific applications, which are characterized by high variance in usage, large volume of data and high compute load, flash crowds, unpredictable load, and varying compute and storage requirements. In order to provide users an automated and scalable platform for hosting scientific workflow applications, while hiding the complexity of the underlying Cloud infrastructure, we present the design and implementation of a PaaS middleware solution along with resource provisioning techniques. We apply our PaaS solution to the data analysis pipeline of a physics application, a gravitational wave search, utilizing public Clouds. The system architecture, a load-balancing approach, and the system’s behavior over varying loads are detailed. The performance evaluation on scalability and load-balancing characteristics of the automated PaaS middleware demonstrates the feasibility and advantages of the approach over existing monolithic approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Large scale computing and storage requirements for basic energy sciences research. Workshop Report LBNL-4809E, Lawrence Berkeley National Laboratory, USA, Jun. 2011.
B. P. Abbott et al. LIGO: the laser interferometer gravitational-wave observatory. Reports on Progress in Physics, 72(7):076901, Jul. 2009.
Advanced LIGO Team. Advanced ligo reference design. Technical Report LIGO M060056-08-M, LIGO Laboratory, USA, May 2007.
Lars Bildsten. Gravitational radiation and rotation of accreting neutron stars. The Astrophysical Journal Letters, 501(1):L89–L93, Jul. 1998.
E. Casalicchio and L. Silvestri. Architectures for autonomic service management in cloud-based systems. In Proceedings of the 2011 IEEE Symposium on Computers and Communications (ISCC’11), 2011.
Wei Chen, Junwei Cao, and Ziyang Li. Customized virtual machines for software provisioning in scientific clouds. In Proceedings of the 2nd International Conference on Networking and Distributed Computing (ICNDC’11), 2011.
Ewa Deelman et al. GriPhyN and LIGO, building a virtual data grid for gravitational wave scientists. In Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing (HPDC’02), 2002.
Ewa Deelman, Gurmeet Singh, Mei-Hui Su, James Blythe, Yolanda Gil, Carl Kesselman, Gaurang Mehta, Karan Vahi, G. Bruce Berriman, John Good, Anastasia Laity, Joseph C. Jacob, and Daniel S. Katz. Pegasus: A framework for mapping complex scientific workflows onto distributed systems. Scientific Programming, 13(3):219–237, Jul. 2005.
B. Dougherty, J. White, and D. C. Schmidt. Model-driven auto-scaling of green cloud computing infrastructure. Future Generation Computer Systems, 28(2):371–378, Feb. 2012.
Piotr Jaranowski, Andrzej Królak, and Bernard F. Schutz. Data analysis of gravitational-wave signals from spinning neutron stars: The signal and its detection. Physics Review D, 58(6), Aug. 1998.
Hyunjoo Kim, Yaakoub el Khamra, Ivan Rodero, Shantenu Jha, and Manish Parashar. Autonomic management of application workflows on hybrid computing infrastructure. Scientific Programming, 19(2–3):75–89, Jun. 2011.
Sifei Lu, Reuben Mingguang Li, William Chandra Tjhi, Long Wang, Xiaorong Li, Terence Hung, and Di Ma. A framework for cloud-based large-scale data analytics and visualization: Case study on multiscale climate data. In Proceedings of the 3rd International Conference on Cloud Computing Technology and Science (CloudCom’11), 2011.
Bertram Ludäscher, Ilkay Altintas, Chad Berkley, Dan Higgins, Efrat Jaeger, Matthew Jones, Edward A. Lee, Jing Tao, and Yang Zhao. Scientific workflow management and the Kepler system. Concurrency and Computation: Practice and Experience, 18(10):1039–1065, Aug. 2006.
M. Mao and M. Humphrey. Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In Proceedings of the 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC’11), 2011.
Ming Mao and Marty Humphrey. Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In Proceedings of the 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC’11), 2011.
A. Melatos and D. J. B. Payne. Gravitational radiation from an accreting millisecond pulsar with a magnetically confined mountain. The Astrophysical Journal, 623(2):1044–1050, Apr. 2005.
C. Messenger and G. Woan. A fast search strategy for gravitational waves from low-mass x-ray binaries. Classical and Quantum Gravity, 24(19):S469–S480, 2007.
Ashish Nagavaram, Gagan Agrawal, Michael A. Freitas, and Kelly H. Telu. A cloud-based dynamic workflow for mass spectrometry data analysis. In Proceedings of the 7th IEEE International Conference on eScience (eScience’11), 2011.
Tom Oinn, Matthew Addis, Justin Ferris, Darren Marvin, Martin Senger, Mark Greenwood, Tim Carver, Kevin Glover, Matthew R. Pocock, Anil Wipat, and Peter Li. Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics, 20(17):3045–3054, Nov. 2004.
Simon Ostermann, Radu Prodan, and Thomas Fahringer. Extending grids with cloud resource management for scientific computing. In Proceedings of the 10th IEEE/ACM International Conference on Grid Computing (GRID’09), 2009.
S. Pandey, D. Karunamoorthy, and R. Buyya. Workflow engine for clouds. In R. Buyya, J. Broberg, and A.Goscinski, editors, Cloud Computing: Principles and Paradigms, chapter 12, pages 321–344. Wiley, 2011.
Stuart L. Shapiro and Saul A. Teukolsky. Black holes, white dwarfs, and neutron stars: The physics of compact objects. Wiley-Interscience, New York, USA, 1983.
Luis M. Vaquero, Luis Rodero-Merino, and Rajkumar Buyya. Dynamically scaling applications in the cloud. SIGCOMM Computer Communication Review, 41(1):45–52, Jan. 2011.
Anna L. Watts, Badri Krishnan, Lars Bildsten, and Bernard F. Schutz. Detecting gravitational wave emission from the known accreting neutron stars. Monthly Notices of the Royal Astronomical Society, 389(2):839–868, 2008.
Fan Zhang, Junwei Cao, Kai Hwang, and Cheng Wu. Ordinal optimized scheduling of scientific workflows in elastic compute clouds. In Proceedings of the 3rd IEEE International Conference on Cloud Computing Technology and Science (CloudCom’11), 2011.
Acknowledgements
This project is partially supported by project grants from the University of Melbourne (Sustainable Research Excellence Implementation Fund and Melbourne School of Engineering) and the Australian Research Council (ARC). We thank Amazon for providing access to their Cloud infrastructure, the Australian and international LIGO communities for their guidance and support, and Dong Leng for his contribution towards extending the Workflow Engine for the LIGO experiment.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer Science+Business Media New York
About this chapter
Cite this chapter
Pandey, S., Sammut, L., Calheiros, R.N., Melatos, A., Buyya, R. (2014). Scalable Deployment of a LIGO Physics Application on Public Clouds: Workflow Engine and Resource Provisioning Techniques. In: Li, X., Qiu, J. (eds) Cloud Computing for Data-Intensive Applications. Springer, New York, NY. https://doi.org/10.1007/978-1-4939-1905-5_1
Download citation
DOI: https://doi.org/10.1007/978-1-4939-1905-5_1
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4939-1904-8
Online ISBN: 978-1-4939-1905-5
eBook Packages: Computer ScienceComputer Science (R0)