Abstract
The development of big data challenges the computing power and communication capability of cloud architecture, but traditional resource-allocation algorithms perform poorly due to the large-scale communication among cloud nodes. In this paper, a dynamically hierarchical, resource-allocation algorithm is proposed for multiple cloud nodes collaborating in big data environment. Using fuzzy pattern recognition, the algorithm dynamically divides tasks and nodes into different levels based on computing power and storage factors. Thus a dynamically adjusted mapping is generated between tasks and nodes. When a new task arrives, only the nodes corresponding to the task level join in the bid. The algorithm takes advantages of dynamical hierarchy to reduce the communication traffic during resource allocation. Both theoretical and experimental results illustrate that the proposed algorithm outperforms the MinMin algorithm in terms of communication traffic and makespan.
Similar content being viewed by others
References
Michael A et al (2010) A view of cloud computing. Commun ACM 53(4):50–58
Seref S, Duygu S (2013) Big data: a review. In: 2013 international conference on collaboration technologies and systems, San Diego, CA, USA, pp 42–47
McAffe A, Brynolfsson E (2012) Strategy & competition big data: the management revolution. Harv Bus Rev 90(10):60–66, 68, 128
Buyya R, Yeo CS, Venugopal S, Broberg J, Brandic I (2009) Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility. Future Gener Comput Syst 25(6):599–616
Amazon EC2 (2015). http://aws.amazon.com/cn/ec2/. Accessed 7 Feb 2015
IBM SmartCloud (2015). http://www.ibm.com/cloud-computing/cn/zh/index.html. Accessed 7 Feb 2015
Seokho S, Gihun J, Sung CJ (2013) An SLA-based cloud computing that facilitates resource allocation in the distributed data centers of a cloud provider. J Supercomput 64:606–637
Nathuji R, Kansal A, Ghaffarkhah A (2010) Q-clouds: managing performance interference effects for QoS-aware clouds. In: 5th ACM European conference on computer systems (EuroSys 2010), Paris, April 13–16, 2010
Marx V (2013) The big challenges of big data. Nature 498(7453):255–260
Sam M (2012) From databases to big data. IEEE Intenet Comput 12:089–7801
JiSu P, Hyongsoon K, Young-Sik J, Eunyoung L (2014) Two-phase grouping-based resource management for big data processing in mobile cloud computing. Int J Commun Syst 27:839–851
Hassan MM, Song B, Hossain MS, Alamri A (2014) QoS-aware resource provisioning for big data processing in cloud computing environment. In: 2014 international conference on computational science and computational intelligence, Las Vegas, NV, USA, March 10–13, 2014
Simon SW, Jelena M (2014) Optimal application allocation on multiple public clouds. Comput Netw 68:138–148
Liang Q, Zhang J, Zhang YH, Liang JM (2014) The placement method of resources and applications based on request prediction in cloud data center. Inf Sci 279:735–745
Yin C, Huang BQ, Liu F et al (2011) Common key technology system of cloud manufacturing service platform for small and medium enterprises. Comput Integr Manuf Syst 17:495–503
Amit N, Sanjay C, Gaurav S (2012) Policy based resource allocation in IaaS cloud. Future Gener Comput Syst 28:94–103
Christian V, Rodrigo NC, Dileban K, Rajkumar B (2012) Deadline-driven provisioning of resources for scientific applications in hybrid clouds with Aneka. Future Gener Comput Syst 28:58–65
Guiyi W, Athanasios V, Vasilakos YZ, Naixue X (2010) A game-theoretic method of fair resource allocation for cloud computing services. J Supercomput 54:252–269
Anton B, Jemal A, Rajkumar B (2012) Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing. Future Gener Comput Syst 28:755–768
Young CL, Albert YZ (2012) Energy efficient utilization of resources in cloud computing systems. J Supercomput 60:268–280
Wu CM, Chang RS, Chan HY (2014) A green energy-efficient scheduling algorithm using the DVFS technique for cloud datacenters. Future Gener Comput Syst 31:141–147
Xiao Z, Song WJ, Chen Q (2013) Dynamic resource allocation using virtual machines for cloud computing environment. IEEE Trans Parallel Distrib Syst 24(6):1107–1117
Daniel W, Odej K (2011) Exploiting dynamic resource allocation for efficient parallel data processing in the cloud. IEEE Trans Parallel Distrib Syst 22(6):985–997
Ahmed S, Kenli L, Aijia O, Zhiyong L (2014) Proactive workload management in dynamic virtualized environments. J Comput Syst Sci 80:1504–1517
Haluk T, Salim H, Wu MY (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260–274
Wang ZJ, Liu JJ (2011) Research on multi-agent-based distributed task scheduling mechanism with multi-objective. J Dalian Univ Technol 51(5):755–760
Wang ZJ, Fang T (2014) Task scheduling model based on multi-agent and multi-objective dynamical scheduling algorithm. J Netw 9(6):1588–1595
Grekovs R (2002) Methods of fuzzy pattern recognition. In: Scientific proceedings of RIGA Technical University
Pedrycz W (1990) Fuzzy sets in pattern recognition: methodology and methods. Pattern Recognit 23(1–2):121–146
Bezdek JC (1981) Pattern recognition with fuzzy objective function algorithms. Plenum Press, New York
Zhao RH, Lu XL, Wang M (2011) Safety evaluation of urban flood control system based on variable fuzzy pattern recognition. Adv Mater Res 159:264–269
Cui XJ, Cao BY (2010) Using two-level fuzzy pattern recognition in the classification of convex quadrilateral. In: 2nd international conference on quantitative logic and soft computing, vol 82, pp 527–534
Fatma AO, Rasha MZ (2010) Dynamic task scheduling using fuzzy logic in distributed memory systems. In: 2010 7th international conference on informatics and systems, Cairo, Egypt, vol 5
Sepideh A, Ali M, Amir MR, Hamid B, Hengameh DT (2013) A new fuzzy negotiation protocol for grid resource allocation. J Newt Comput Appl. doi:10.1016/j.jnca.2012.12.030
Guo FY, Yu L, Tian S, Yu J (2014) A workflow task scheduling algorithm based on the resources’ fuzzy clustering in cloud computing environment. Int J Commun Syst. doi:10.1002/dac.2743
Akihiro K, Nathan S (2008) Optimized algorithms for multi-agent routing. In: AAMAS ’08 proceedings of the 7th international joint conference on autonomous agents and multiagent systems, New York, pp 1585–1588
Rodrigo NC, Rajiv R, Anton B, César AFDR, Rajkumar B (2011) CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithm. Softw Pract Exp 41:23–50
Acknowledgments
This research was supported by the National Natural Science Foundation of China (Grant No. 61173160), Intel Industrial Liaison and Comprehensive Reform Program of Intel Semiconductor (US) LLC in 2014 of Ministry of Education of the People’s Republic of China. We would like to thank the anonymous reviewers for their attentive reading and for their constructive comments that have helped to further strengthen this paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, Z., Su, X. Dynamically hierarchical resource-allocation algorithm in cloud computing environment. J Supercomput 71, 2748–2766 (2015). https://doi.org/10.1007/s11227-015-1416-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-015-1416-x