Abstract
It is often difficult to perform efficiently a collection of jobs with complex job dependencies due to temporal unpredictability of the Grid. One way to mitigate the unpredictability is to schedule job execution in a manner that constantly maximizes the number of jobs that can be sent to workers. A recently developed scheduling theory provides a basis to meet that optimization goal. Intuitively, when the number of such jobs is always large, high parallelism can be maintained, even if the number of workers changes over time in an unpredictable manner. In this paper we present the design, implementation, and evaluation of a practical scheduling tool inspired by the theory. Given a DAGMan input file with interdependent jobs, the tool prioritizes the jobs. The resulting schedule significantly outperforms currently used schedules under a wide range of system parameters, as shown by simulation studies. For example, a scientific data analysis application, AIRSN, was executed at least 13% faster with 95% confidence. An implementation of the tool was integrated with the Condor high-throughput computing system.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aho, A.V., Garey, M.R., Ullman, J.D.: The transitive reduction of a directed graph. SIAM J. Comput. 1, 131–137 (1972)
Annis, J., Zhao, Y., Voeckler, J., Wilde, M., Kent, S., Foster, I.: Applying chimera virtual data concepts to cluster finding in the sloan sky survey. ACM/IEEE Conference on Supercomputing, pp. 1–14 (2002)
Berriman, B., Bergou, A., Deelman, E., Good, J., Jacob, J., Katz, D., Kesselman, C., Laity, A., Singh, G., Su, M.-H., Williams, R.: Montage: a Grid-enabled image mosaic service for the NVO. Astron. Data Anal. Softw. Syst. (2003)
Buyya, R., Abramson, A., Giddy, J.: A case for economy Grid architecture for service oriented Grid computing. In: 10th Heterogeneous Computing Wkshp. (2001)
Cohen, P.R.: Empirical Methods for Artificial Intelligence. MIT, Cambridge, MA (1995)
Cordasco, G., Malewicz, G., Rosenberg, A.L.: Applying IC-scheduling theory to familiar classes of computations. In: Workshop on Large-scale and Volatile Desktop Grids (2007)
Cordasco, G., Malewicz, G., Rosenberg, A.L.: Extending IC-scheduling theory via the sweep algorithm. Typescript, Univ. Massachusetts (2006)
Cordasco, G., Malewicz, G., Rosenberg, A.L.: Advances in IC-scheduling theory: scheduling expansive and reductive dags and scheduling dags via duality. IEEE Trans. Parallel Distrib. Syst. (2007) (to appear)
Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms (2nd edn.). MIT, Cambridge, MA (2001)
Deelman, E., Kesselman, C., Mehta, G., Meshkat, L., Pearlman, L., Blackburn, K., Ehrens, P., Lazzarini, A., Williams, R., Koranda, S.: GriPhyN and LIGO, building a virtual data Grid for gravitational wave scientists. In: Proc. IEEE Int. Symp. High Perform. Distrib. Comput. (HPDC’02), pp. 225–234 (2002)
Foster, I., Kesselman, C. (eds.): The Grid: Blueprint for a New Computing Infrastructure (2nd edn.). Morgan-Kaufmann, San Francisco (2004)
Foster, I. et al.: The Grid2003 production Grid: principles and practice. In: 13th IEEE International Symposium on High Performance Distributed Computing (HPDC’04), pp. 236–245 (2004)
Hall, R., Rosenberg, A.L., Venkataramani, A.: A comparison of dag-scheduling strategies for Internet-based computing. In: 21st Intl. Parallel and Distributed Processing Symp. (2007)
Horn, J.V., Dobson, J., Woodward, J., Wilde, M., Zhao, Y., Voeckler, J., Foster, I.: Grid-based computing and the future of Neuroscience computation. Methods in Mind, MIT, Cambridge, MA (2005)
Hsu, H.T.: An algorithm for finding a minimal equivalent graph of a digraph. J. ACM 22, 11–16 (1975)
Kondo, D., Casanova, H., Wing, E., Berman, F.: Models and scheduling mechanisms for global computing applications. In: 16th Intl. Parallel and Distributed Processing Symp. (2002)
Malewicz, G., Rosenberg, A.L.: On batch-scheduling dags for Internet-based computing. In: 11th European Conf. on Parallel Processing (Euro-Par’05), pp. 262–271 (2005)
Malewicz, G., Rosenberg, A.L., Yurkewych, M.: Toward a theory for scheduling dags in internet-based computing. IEEE Trans. Comput. 55(6), 757–768 (2006)
Rosenberg, A.L.: On scheduling mesh-structured computations for Internet-based computing. IEEE Trans. Comput. 53, 1176–1186 (2004)
Rosenberg, A.L., Yurkewych, M.: Guidelines for scheduling some common computation-dags for Internet-based computing. IEEE Trans. Comput. 54, 428–438 (2005)
Szalay, A.S., Kunszt, P.Z., Thakar, A., Gray, J., Slutz, D., Brunner, R.J.: Designing and mining multi-terabyte astronomy archives: the sloan digital sky survey. In: ACM SIGMOD International Conference on Management of Data, pp. 451–462 (2000)
Sun, X.-H., Wu, M.: Grid harvest service: a system for long-term, application-level task scheduling. In: IEEE Intl. Parallel and Distributed Processing Symp. (2003)
Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurrency and Computation: Practice and Experience 17, 323–356 (2005)
Author information
Authors and Affiliations
Corresponding author
Additional information
The research of G. Malewicz was supported in part by NSF Grant ITR-800864. The research of I. Foster and M. Wilde was supported in part by the NSF GriPhyN project and by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, U.S. Department of Energy, under Contract W-31-109-Eng-38. The research of A.L. Rosenberg was supported in part by NSF Grant CCF-0342417.
Rights and permissions
About this article
Cite this article
Malewicz, G., Foster, I., Rosenberg, A.L. et al. A Tool for Prioritizing DAGMan Jobs and its Evaluation. J Grid Computing 5, 197–212 (2007). https://doi.org/10.1007/s10723-007-9065-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10723-007-9065-9