A taxonomy of software-based and hardware-based approaches for energy efficiency management in the Hadoop

Abstract

Apache Hadoop framework supports the storing and processing of big data datasets using simple programming models. Energy management has been recognized as one of the major issues in Hadoop, and many types of research have been conducted in this scope. However, despite the importance of this issue, there is no inclusive study about energy efficiency in Hadoop. In this paper, the techniques of energy efficiency in Hadoop are classified into two main categories. Moreover, the benefits and drawbacks of these methods and a systematic study of the conducted research are provided and examined in this paper. Another aim is to provide the visions for the descriptions of open issues and recommendations for future research.

Introduction

Nowadays, the ability to analyze the big data repositories remains a problem in many modern enterprises and research societies (Gonçalves et al., 2017; Khezr and Navimipour, 2017). Every day, large amounts of data are produced from numerous sources e.g. sensors, digital pictures, videos, purchase transaction records, and cell phone, but mining suitable information for making an appropriate decision from these massive data repositories is almost impractical for the traditional database management system (DBMS) technologies (Cuzzocrea et al., 2011).

The Hadoop as an applicable solution to big data (Rashmi and Basu, 2017) provides reliable, fault-tolerant, scalable, and efficient services for large amounts of data processing using MapReduce (Uzunkaya et al., 2015; Zhao, 2017). The simple programming interface, high scalability and the capability of processing a high amount of data in the distributed processing environments are considered as its main features (Khezr and Navimipour, 2017). The MapReduce has an important role in performing a very large number of data-intensive applications (Cassales et al., 2015; Chelliah, 2017).

Recently, the significant issue in the data centers is the efficiency of energy (Khan et al., 2016; Kurpicz et al., 2018). According to U.S. department of energy report in 2014, the U.S data centers spent about 70 billion kWh (1.8% of entire U.S. electricity consumption). It is estimated that about 73 billion kWh will be consumed by U.S data centers in 2020 (Shehabi et al., 2016). Given the environmental challenges and the limited resources of energy and high energy costs (Akhter and Othman, 2016; Babar et al., 2017), hardware and software techniques should be used to reduce the energy consumption. As a result, the energy reduction is a big challenge for Hadoop which consists of the large cluster (Usama et al., 2017).

This paper is a key systematic one about the energy utilization techniques for Hadoop. It discusses different software methods, such as scheduling, and hardware methods such as Dynamic Voltage and Frequency Scaling (DVFS) which are employed to reduce the energy consumption. Providing the conceptual aspects of energy efficiency in Hadoop is the main goal of this paper. The contributions of this study are listed below:

• Providing a review on the current energy-aware methods for Hadoop.

• Dividing energy efficiency techniques into two main classes, including software-based and hardware-based techniques.

• Providing the benefits and drawbacks of the existing energy efficiency techniques for Hadoop.

• Discussing and comparing the main challenges for the energy efficiency in the Hadoop.

• Highlighting the guidelines for future research and open issues about the energy efficiency in the Hadoop.

Furthermore, the techniques are compared in this paper using some performance measures and the Quality of Service (QoS) parameters (Conejero et al., 2016) such as data locality, fault tolerance, heterogeneity, scalability, makespan, performance, cost, and load balancing. Therefore, we provide a brief discussion of them.

• Data Locality: It means moving computation close to data rather than moving data towards computation (George et al., 2016).

• Heterogeneity: In a heterogeneous data center, there are some nodes with dissimilar abilities such as computing power (Rasooli and Down, 2014b).

• Fault tolerance: It provides continuous and correct operation of a system in the presence of the failure of its component(s) (Sampaio and Barbosa, 2018).

• Scalability: It is the capacity to be changed and reformed in numerous situations in a Hadoop cluster (Zhang et al., 2018).

• Makespan: It is the time variance between the beginning and the end of the job or task sequence (Kalra and Singh, 2015).

• Load balancing: It enhances the distribution of loads across multiple computing resources (Gao and Yu, 2017; Ghomi et al., 2017).

• Cost: Two types of costs can be considered, one is in term of manpower and the other in term of money (Majeed and Shah, 2015).

• Performance: The amount of useful estimated fulfilled work in terms of time needed, used resources, etc. (Cheng et al., 2017a).

The arrangement of various sections of the articles is as follows: Hadoop and its components are presented in Section 2. Section 3 reviews the related work. Section 4 provides the research selection process and a Systematic Literature Review (SLR). Section 5 systematically overviews the energy efficiency approaches in the Hadoop and classifies them. Furthermore, this section provides a comparison of the methods of the selected articles. Section 6 discusses the obtained findings. Some open issues are elaborated in Section 7. Finally, Section 8 presents the conclusion in addition to the paper limitations.