Abstract
Adaptive clustering aims at improving cluster utilization for varying load and traffic patterns. Locality-based least-connection with replication (LBLCR) scheduling that comes with Linux is designed to help improve cluster utilization through adaptive clustering. A key issue with LBLCR, however, is that cluster performance depends much on a single threshold value that is used to determine adaptation. Once set, the threshold remains fixed regardless of the load and traffic patterns. If a cluster of PCs is to adapt to different traffic patterns for high utilization, a good threshold has to be selected and used dynamically. We present in this report an adaptive clustering framework that autonomously learns and adapts to different load and traffic patterns at runtime with no administrator intervention. Cluster is configured once and for all. As the patterns change the cluster automatically expands/contracts to meet the changing demands. At the same time, the patterns are proactively learned that when similar patterns emerge in the future, the cluster knows what to do to improve utilization. We have implemented this autonomous learning method and compared with LBLCR using published Web traces. Experimental results indicate that our autonomous learning method shows high performance scalability and adaptability for different patterns. On the other hand LBLCR-based clustering suffers from performance scalability and adaptability for different traffic patterns since it is not designed to obtain good threshold values and use them at runtime.
Hukeun Kwak and Kyusik Chung are supported in part by the Basic Research Program of the Korea Science & Engineering Foundation grant No.R01-2006-000-11167-0.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Amdahl, G.M., Blaauw, G.A., Brooks, F.P.: Architecture of the IBM System 360. IBM Journal of Research and Development 8(2), 87–101 (1964)
Arpaci-Dusseau, A.C., Arpaci-Dusseau, R.H.: Information and Control in Gray-Box Systems. In: 8th ACM Symposium on Operating Systems Principles, pp. 43–56. ACM, New York (2001)
Barford, P., Crovella, M.: Generating Representative Web Workloads for Network and Server Performance Evaluation. In: Proc. ACM SIGMETRICS Conf., Madison, WI, July. ACM Press, New York (1998)
Bozyigit, M., Wasiq, M.: User-Level Process Checkpoint and Restore for Migration, Vol. 35, No. 2, 86 - 96 (2001)
Chang, Y., Chen, J.: Designing an enhanced PC cluster system for scalable network services. In: 19th International Conference on Advanced Information Networking and Applications, pp. 163–166 (2005)
Christiansen, M., Jeffay, K., Otta, D., Smith, F.: Tuning RED for Web traffic. IEEE/ACM Transactions on Networking 9(3), 249–264 (2001)
Ernemann, C., Hamscher, V., Vahyapour, R.: Benefits of global grid computing for job scheduling. In: 5th IEEE/ACM International Workshop on Grid Computing, pp. 374–379. ACM Press, New York (2004)
Ganek, A.G., Corbi, T.A.: The dawning of the autonomic computing era. IBM System Journal 42(1), 5–18 (2003)
Hong, Y., No, J., Han, I.: Evaluation of fault-tolerant distributed Web systems. In: 10th IEEE International Conference on Advanced Information Networking and Applications, pp. 163–166. IEEE Computer Society Press, Los Alamitos (2005)
HP Grid and Utility Computing. http://devresource.hp.com/drc/topics/utility_comp.jsp
Job Scheduling Algorithms in Linux Virtual Server. http://www.linuxvirtual-server.org/docs/scheduling.html
Karlsson, M., Covell, M.: Dynamic Black-Box Performance Model Estimation for Self-Tuning Regulators. In: 2nd International Conference on Autonomic Computing (ICAC’05), pp. 172–182 (2005)
Liao, W., Shih, P.: Architecture of proxy partial caching using HTTP for supporting interactive video and cache consistency. In: 11th International Conference Computer Communications and Networks, pp. 216–221 (2002)
Linux Virtual Server Project. http://www.linuxvirtualserver.org/
Litzkow, M., Tannenbaum, T., Basney, J., Livny, M.: Checkpoint and Migration of UNIX Processes in the Condor Distributed Processing System, Technical Report #1346, University of Wisconsin-Madison Computer Science Department (1997)
Lu, D., Qiao, Y., Dinda, P., Bustamante, F.: Modeling and Taming Parallel TCP on the Wide Area Network. In: Proceedings of 19th IEEE International Parallel and Distributed Processing Symposium, IEEE Computer Society Press, Los Alamitos (2005)
McClure, S., Wheeler, R.: MOSIX: How Linux Clusters Solve Real World Problems. In: Proc. 2000 USENIX Annual Tech. Conf., San Diego, CA., pp. 49–56 (2000)
N1 Grid: Managing n computers as 1, Sun Microsystems, http://wwws.sun.com/software/solutions/n1/
Olaru, V., Tichy, W.: Request distribution-aware caching in cluster-based Web servers. In: 3rd IEEE International Symposium on Network Computing and Applications, pp. 311–316. IEEE Computer Society Press, Los Alamitos (2004)
Yang, C., Luo, M.: Building an adaptable, fault tolerant, and highly manageable web server on clusters of non-dedicated workstations. In: IEEE International Conference on Parallel Processing, pp. 413–420. IEEE Computer Society Press, Los Alamitos (2000)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Sohn, A., Kwak, H., Chung, K. (2007). Autonomous Learning of Load and Traffic Patterns to Improve Cluster Utilization. In: Lukowicz, P., Thiele, L., Tröster, G. (eds) Architecture of Computing Systems - ARCS 2007. ARCS 2007. Lecture Notes in Computer Science, vol 4415. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-71270-1_17
Download citation
DOI: https://doi.org/10.1007/978-3-540-71270-1_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-71267-1
Online ISBN: 978-3-540-71270-1
eBook Packages: Computer ScienceComputer Science (R0)