A computational economy for grid computing and its implementation in the Nimrod-G resource broker

https://doi.org/10.1016/S0167-739X(02)00085-7Get rights and content

Abstract

Computational grids that couple geographically distributed resources such as PCs, workstations, clusters, and scientific instruments, have emerged as a next generation computing platform for solving large-scale problems in science, engineering, and commerce. However, application development, resource management, and scheduling in these environments continue to be a complex undertaking. In this article, we discuss our efforts in developing a resource management system for scheduling computations on resources distributed across the world with varying quality of service (QoS). Our service-oriented grid computing system called Nimrod-G manages all operations associated with remote execution including resource discovery, trading, scheduling based on economic principles and a user-defined QoS requirement. The Nimrod-G resource broker is implemented by leveraging existing technologies such as Globus, and provides new services that are essential for constructing industrial-strength grids. We present the results of experiments using the Nimrod-G resource broker for scheduling parametric computations on the World Wide Grid (WWG) resources that span five continents.

Introduction

The accelerated development of grid computing systems has positioned them as promising next generation computing platforms. They enable the sharing, selection, and aggregation of geographically distributed resources for solving large-scale problems in science, engineering, and commerce [1], [7], [30]. However, application composition, resource management and scheduling in these environments is a complex undertaking. This is due to the geographic distribution of resources that are often owned by different organizations having different usage policies and cost models, and varying loads and availability patterns. To address these resource management challenges, we have proposed and developed a computational economy framework for resource allocation and regulation of supply and demand for resources. The new framework offers an incentive to resource owners for being part of the grid and motivates resource users to strike a balance between time for results delivery and heir economic cost, i.e., deadline and budget [5].

We are exploring the use of an economic paradigm for grid computing. We have developed an economy-driven grid resource broker within the Nimrod-G system that supports soft-deadline and budget-based scheduling of applications on the computational grid [7]. Depending on users’ quality of service (QoS) requirements, our resource broker dynamically leases grid services at runtime depending on their cost, quality, and availability. The scheduler allows minimization of time or cost within specific deadline and budget constraints.

Resource management systems need to provide mechanisms and tools that realize the goals of both service providers and consumers. The resource consumers need a utility model, representing their resource demand and preferences, and brokers that automatically generate strategies for choosing providers based on this model. Further, the brokers need to manage all issues associated with the execution of the underlying application. The service providers need price generation schemes that increase system utilization, as well as economic protocols that help them to offer competitive services. For the market to be competitive and efficient, coordination mechanisms are required that help the market reach an equilibrium price, that is, the market price at which the supply of a service equals the quantity demanded [13]. Numerous economic theories have been proposed in the literature and many commonly used economic models for selling goods and services can be employed as negotiation protocols in grid computing. Some of these market- or social-driven economic models are shown in Table 1 along with the identity of the distributed system that adopted the approach [8].

These economic models regulate the supply and demand for resources in grid-based virtual enterprises. We demonstrate the power of these models in scheduling computations using the Nimrod-G resource broker on a grid testbed, called the World Wide Grid (WWG) spanning across five continents. Whilst it is not the goal of the system to earn revenue for the resource providers, this approach does provide an economic incentive for resource owners to share their facilities on the grid. Further, it encourages the emergence of a new service-oriented computing industry. Importantly, it provides mechanisms to trade-off QoS parameters such as deadline and computational cost, and offers incentive for users to relax their requirements. For example, a user may be prepared to accept a later deadline if the computation can be achieved more cheaply. Current grid computing toolkits and applications do not provide this functionality.

Section snippets

Objectives and goals

Nimrod-G is a tool for automated modeling and execution of parameter sweep applications (parameter studies) over global computational grids [1], [2], [3]. It provides a simple declarative parametric modeling language for expressing parametric experiments. A domain expert can easily create a plan for a parametric experiment and use the Nimrod system to submit jobs for execution. It uses novel resource management and scheduling algorithms based on economic principles. Specifically, it supports

Scheduling experiments on the WWG testbed

We have performed deadline and budget constrained scheduling experiments at two different times (Australian peak and off-peak hours) on resources distributed in two major time zones [7] using a “cost-optimization scheduling algorithm” [6] on the WWG [14] testbed shown in Fig. 5. Currently, the testbed has heterogeneous computational resources owned by different organizations distributed across five continents: Asia, Australia, Europe, North America, and South America. This testbed contains

Conclusions and future work

The emerging grid computing technologies are enabling the creation of virtual organizations and enterprises for sharing distributed resources for solving large-scale problems in science, engineering, and commerce. The resource management and scheduling systems in grid environments need to be adaptive to handle dynamic changes in availability of resources and user requirements. At the same time, they need to provide scalable, controllable, measurable, and easily enforceable policies for

David Abramson is a Professor and the Head of the School of Computer Science and Software Engineering (CSSE) at Monash University, Melbourne, Australia. He has been involved in computer architecture and high performance computing research since 1979. Previous to joining Monash University in 1997, he has held appointments at Griffith University, CSIRO, and RMIT. He is currently project leader in the Co-operative Research Centre for Distributed Systems Nimrod Project. He is also Chief

References (30)

  • D. Abramson, J. Giddy, L. Kotler, High performance parametric modeling with Nimrod-G: killer application for the Global...
  • D. Abramson, R. Sosic, J. Giddy, B. Hall, Nimrod: a tool for performing parameterised simulations using distributed...
  • D. Abramson, I. Foster, J. Giddy, A. Lewis, R. Sosic, R. Sutherst, N. White, The Nimrod Computational Workbench: a case...
  • R. Buyya, D. Abramson, J. Giddy, Nimrod-G: an architecture for a resource management and scheduling system in a global...
  • R. Buyya, D. Abramson, J. Giddy, An economy driven resource management architecture for global computational power...
  • R. Buyya, J. Giddy, D. Abramson, An evaluation of economy-based resource trading and scheduling on computational power...
  • R. Buyya, D. Abramson, J. Giddy, A case for economy grid architecture for service-oriented grid computing, in:...
  • R. Buyya, H. Stockinger, J. Giddy, D. Abramson, Economic models for management of resources in peer-to-peer and grid...
  • K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W. Smith, S. Tuecke, A resource management architecture...
  • S. Chapin, J. Karpovich, A. Grimshaw, The Legion resource management system, in: Proceedings of the Fifth Workshop on...
  • J. Postel, C. Sunshine, D. Cohen, The ARPA Internet Protocol, Computer Networks 5 (4)...
  • M. Stonebraker, R. Devine, M. Kornacker, W. Litwin, A. Pfeffer, A. Sah, C. Staelin, An economic paradigm for query...
  • D. Ferguson, C. Nikolaou, J. Sairamesh, Y. Yemini, Economic models for allocating resources in computer systems, in:...
  • R. Buyya, World Wide Grid testbed, June 2001,...
  • G. Heiser, F. Lam, S. Russell, Resource management in the Mungi single-address-space operating system, in: Proceedings...
  • Cited by (319)

    View all citing articles on Scopus

    David Abramson is a Professor and the Head of the School of Computer Science and Software Engineering (CSSE) at Monash University, Melbourne, Australia. He has been involved in computer architecture and high performance computing research since 1979. Previous to joining Monash University in 1997, he has held appointments at Griffith University, CSIRO, and RMIT. He is currently project leader in the Co-operative Research Centre for Distributed Systems Nimrod Project. He is also Chief Investigator in the following ARC funded research project: Guard—a relative debugger and a software environment for building high performance optimizing decision support systems from computational models.

    Rajkumar Buyya is a lecturer in the School of Computer Science and Software Engineering at the University of Melbourne, Australia. He directs research activities of the Grid Computing and Distributed Systems (GRIDS) Laboratory. He was awarded Dharma Ratnakara Memorial Trust Gold Medal for his academic excellence during 1992 by Mysore and Kuvempu Universities. He has authored three books Microprocessor x86 Programming, Mastering C++, and Design of PARAS Microkernel. He has edited a popular two volumes book on High Performance Cluster Computing published by Prentice-Hall, USA. He also edited proceedings of six international conferences and served as guest editor for major research journals. He has contributed to the development of system software for PARAM supercomputers produced by the Centre for Development of Advanced Computing (C-DAC), India. At Monash University, he is conducting R&D on next generation Internet/grid computing technologies and its applications.

    Jonathan Giddy recently joined as a coordinator for grid technologies in the Welsh e-Science Centre at Cardiff University, UK. Prior to that, he worked as Research Scientist at the Distributed Systems Technology Centre (DSTC), Monash University, Melbourne, Australia. He holds BSc honours in Computer Science from the University of Wollongong. He worked on a number of DSTC funded projects—Security Unit at the Queensland University of Technology and Tools Unit at the Griffith University. In the context of Nimrod project, he is involved in design and development of number of software tools for high performance distributed computing. He enjoys programming in Python!

    View full text