Elsevier

Internet of Things

Volume 12, December 2020, 100273
Internet of Things

Review article
Performance evaluation metrics for cloud, fog and edge computing: A review, taxonomy, benchmarks and standards for future research

https://doi.org/10.1016/j.iot.2020.100273Get rights and content

Highlights

  • We identify the performance metrics to evaluate cloud, fog and edge optimizations.

  • A taxonomy of the whole identified performance metrics are presented.

  • Performance metrics are defined, formulated and their potential use cases are identified.

  • A review of the literature on each performance metrics is conducted.

  • Future directions for evaluating the promising fog and edge computing optimization are highlighted.

Abstract

Optimization is an inseparable part of Cloud computing, particularly with the emergence of Fog and Edge paradigms. Not only these emerging paradigms demand reevaluating cloud-native optimizations and exploring Fog and Edge-based solutions, but also the objectives require significant shift from considering only latency to energy, security, reliability and cost. Hence, it is apparent that optimization objectives have become diverse and lately Internet of Things (IoT)-specific born objectives must come into play. This is critical as incorrect selection of metrics can mislead the developer about the real performance. For instance, a latency-aware auto-scaler must be evaluated through latency-related metrics as response time or tail latency; otherwise the resource manager is not carefully evaluated even if it can reduce the cost. Given such challenges, researchers and developers are struggling to explore and utilize the right metrics to evaluate the performance of optimization techniques such as task scheduling, resource provisioning, resource allocation, resource scheduling and resource execution. This is challenging due to (1) novel and multi-layered computing paradigm, e.g., Cloud, Fog and Edge, (2) IoT applications with different requirements, e.g., latency or privacy, and (3) not having a benchmark and standard for the evaluation metrics. In this paper, by exploring the literature, (1) we present a taxonomy of the various real-world metrics to evaluate the performance of cloud, fog, and edge computing; (2) we survey the literature to recognize common metrics and their applications; and (3) outline open issues for future research. This comprehensive benchmark study can significantly assist developers and researchers to evaluate performance under realistic metrics and standards to ensure their objectives will be achieved in the production environments.

Introduction

Cloud computing has emerged as a utility in which a wide range of services from software to platform and infrastructure are offered. By the widespread usage of cloud services, diverse orchestration concerns are rising. For instance, it is outlined in [1] that the typical web applications are facing a growing demand during the working hours, with a severe variability at around noon, and a decreasing demand during the night to the morning. If such variability is not well controlled by a dynamic resource provisioning, the users may suffer from long latencies to receive the responses. Orchestration techniques include, but not limited to, resource allocation, resource provisioning, task scheduling, load balancing, Virtual Machine (VM) migration, task placement, server consolidation, service composition, data caching, service discovery, etc.

The presence of Internet of Things (IoT) demands even more careful consideration of orchestrations due to latency-sensitivity of many IoT applications (or services) and limitations specific to IoT. To reduce overall delay for IoT application and address their latency requirement, Edge and fog computing (hereafter, fog and edge) bring computing resources close to users and IoT devices and offer computation at the edge of network. In consequence, fog and edge not only minimize latency, but also address privacy, data locality, bandwidth consumption of IoT applications. While fog and edge are used interchangeably, we differentiate them in this paper. The former is composed of physical servers closer than the cloud and inherits cloud characteristics. The latter is closer to IoT devices and user, mostly inheriting IoT devices’ characteristics. With fog and edge, orchestration is facing more challenges and diversities such as limited energy power and ephemeral resources. Challenges are increasing due to the added layers to the computation and the diversities are furthered as edge-specific techniques such as task offloading and resource sharing are appeared in such distributed environments.

Orchestration development efforts are required to fulfill certain objective(s). The literature reported that several objectives such as latency minimization, cost reduction, energy management are targeted by the existing techniques [2], [3], [4], [5]. For instance, the authors in [6] attempt to minimize the latency while reducing the cost for gaming applications through a dynamic resource provisioning technique. Several of such a technique have been developed for latency-sensitive applications, but targeting energy as an objective [7,8]. Not only the objectives, but also the stakeholders of the service are diverse. One technique may target cloud, fog or edge providers to fulfill their requirements such as energy consumption while another may target users’ requirements such as lower response time. Objectives fulfilling the users’ requirements are also known as Service Level Objectives (SLO) which come under the umbrella of Quality of Service (QoS). Service Level Agreement (SLA) which acts as an official contract between users and providers is based on the same objectives. Hence, objectives and considered stakeholders for orchestration techniques must be considered before moving to development of the solution and then production environments.

Given such complexity, this question is raised that how the performance of orchestration techniques can be evaluated to ensure the accomplishment of their mission? To answer, it is necessary to evaluate the performance of the proposed technique using certain metrics which are called performance evaluation metrics (or shortly metrics). Measuring the capability and accuracy of the techniques becomes even more challenging when the adopted technique is multi-objective by which not only more measurements are required, but also the objectives may conflict with each other. Imagine that a resource allocation technique is claimed to be cost-efficient and keeps the number of resources at minimum, thus maximizing the latency and violating the SLA. This is further important when violating the SLA (e.g., users’ requirements) can impose penalties for the provider. Resources which host a service can be a physical host/server, VM, container, etc. (hereafter, host). Motivated by this, it is apparent that, regardless of which orchestration technique is employed, identifying the right metrics for evaluating the performance and also selecting the most suitable metrics are essential.

Performance evaluation (or measurement) is an inseparable process for optimization and cloud to things (i.e., cloud, fog, edge and IoT) orchestration techniques are not an exception. Performance evaluation highlights the weaknesses and benefits of different approaches by which the developer can ensure if the technique is fulfilling the objectives. Here, certain Research Questions (RQ) are raised which we aim to answer in this paper:

RQ1: What are the potential performance metrics applicable to the orchestration techniques in cloud, fog and edge computing?

RQ2: How are the potential performance metrics implemented and measured in cloud to things?

RQ3: What are the challenges and open issues for future performance evaluation metrics?

We have conducted a review to identify and categorized the potential metrics, which can stand as a standard benchmark for the future research. Identified metrics are defined, elaborated and the measurement thereof is outlined, as well as touching their considerations in the literature. To make the review article further understandable, the identified performance metrics are categorized according to the well-known concept as MAPE-K (monitoring, analyzing, planning, and execution) introduced by IBM [6,9,10]. This effort helps cloud, fog and edge computing research community to select the right performance metrics to evaluate orchestration techniques. Further, they will be able to re-use and/or reevaluate the existing cloud-native metrics in the fog and edge as promising computing landscape for the future of IoT [11].

The rest of the paper is organized as follows: In Section 2, a taxonomy of performance metrics is discussed to answer the RQ1 and the identified metrics are individually outlined in details to answer to the RQ2 in Sections 3 to 6. The discussion on the performance evaluations metrics, exploring challenges and open issues are highlighted in Section 7 to answer the RQ3. Finally, Section 8 concludes the paper.

Section snippets

Performance evaluation metrics for cloud, fog and edge computing: a taxonomy

In this section, a generic model for cloud to things landscape is illustrated. The positioning of the optimization techniques in the model is depicted. Then, a taxonomy of measurable performance metrics for each layer of computing is presented. The cloud layer is divided to clarify different cloud models’ requirements. Lastly, a brief definition thereof is provided.

Resource utilization

Resource utilization represents the intensity of load on a VM, for instance, and can be indicated by percentage, for instance 78%. VM utilization is the proportion of resource capacity, e.g. CPU; that is being or has been used to execute tasks. It also describes the demand on the resource and shows whether the utilization level is low or high [13,14]. Under another definition, resource utilization is a ratio of actual time spent by resource to execute workload [9]. In cloud scale, utilization

Statistical analysis methods

Analyzer duty is to analyze monitored parameters and is in charge of producing more accurate values from monitored parameters to be used by the planner. At this phase, predictions methods and machine learning techniques are mostly used. Therefore, the most relevant metrics that can be identified here are statistical methods. Precisely, to evaluate the accuracy of predictions, metrics like MAPE, MAE, MSE, R2, Root Mean Square Error (RMSE), Average, Median, Tail, PRED [11] are widely used [5].

Orchestration decision

This is an effective metric to understand how many decisions have been made by orchestrator. Certainly, if the number for decisions is low, it has been more confident in decision makings. The higher the number of decisions is, the more overhead on the system will be. On the other hand, a higher number of decisions may imply that the decision maker is conscious of the changes in the services and hosts situations.

Contradictory decisions

Another metric in planner phase is measuring contradictory decisions, showing the

Orchestration actions

This metric is measured to evaluate the overhead of orchestrator's actions, meaning that if the orchestrator is capable of handling the fluctuations in load while mitigating unnecessary actions. It can be also a measure of accuracy in mitigating fault decisions made by planner. For instance, a faulty task offloading decision of planner can be rejected by executor if it perceives that destination host is not able to handle additional services.

Provisioned and de-provisioned resources

Provisioned resources metric shows the number of

Discussions

Table 1 depicts how the aforementioned metrics are utilized in the literature, which could help the researchers to relate each research work to a particular metric. While there exist hundreds of published works on optimizations for cloud to things computing, the paper selection policy for this table considers highly cited papers mostly to explore a wider range of research efforts. The spread of evaluated metrics across the table does not give an overall view of the popularity trend for

Summary and conclusions

Orchestration techniques such as resource allocation, resource provisioning, task scheduling, load balancing, Virtual Machine (VM) migration, task placement, server consolidation, service composition, data caching, service discovery, etc. play an important role for cloud, fog and edge computing optimizations. Developing the technique is half the story while evaluating its performance under specific conditions is critical to realize if the technique can comply with the objectives and is

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

We would like to thank the editors, area editor and anonymous reviewers for their valuable comments and suggestions to help and improve our research paper.

Mohammad Sadegh Aslanpour is a PhD student in Monash University and CSIRO’s DATA61, Australia. He obtained his MSc degree in Computer Engineering from Azad University, Tehran Science and Research (Sirjan) Branch, Iran in 2016. He also obtained his Bachelor and Associate degrees in Computer-Software in 2012 and 2010, respectively. From 2011 to 2019, he worked in the industry, IT Department of Jahrom Municipality, Iran as a Software Engineer. He is also serving as Editorial Board Member and

References (50)

  • S. Islam et al.

    Empirical prediction models for adaptive resource provisioning in the cloud

    Futur. Gener. Comput. Syst.

    (2012)
  • Y. Al-Dhuraibi et al.

    Elasticity in cloud computing: state of the art and research challenges

    IEEE Trans. Serv. Comput.

    (2018)
  • M.S. Aslanpour et al.

    Auto-scaling web applications in clouds: a cost-aware approach

    J. Netw. Comput. Appl.

    (2017)
  • S. Singh et al.

    A survey on resource scheduling in cloud computing: issues and challenges

    J. grid Comput.

    (2016)
  • T. Lorido-Botran et al.

    A review of auto-scaling techniques for elastic applications in cloud environments

    J. Grid Comput.

    (2014)
  • S.S. Gill et al.

    A taxonomy and future directions for sustainable cloud computing: 360 degree view

    ACM Comput. Surv.

    (2018)
  • M.S. Aslanpour et al.

    LARPA: a learning automata‐based resource provisioning approach for massively multiplayer online games in cloud environments

    Int. J. Commun. Syst.

    (2019)
  • L. Zhou et al.

    Joint Server and Network Energy Saving in Data Centers for Latency-Sensitive Applications

  • C.-.H. Chou et al.

    μDPM: dynamic Power Management for the Microsecond Era

  • S.S. Gill et al.

    CHOPPER: an intelligent QoS-aware autonomic resource management approach for cloud computing

    Cluster Comput

    (2017)
  • S. Singh et al.

    SOCCER: self-optimization of energy-efficient cloud resources

    Cluster Comput

    (2016)
  • S.S. Gill et al.

    RADAR: self‐configuring and self‐healing in resource management for enhancing quality of cloud services

    Concurr. Comput. Pract. Exp.

    (2018)
  • S.H.H. Madni et al.

    Recent advancements in resource allocation techniques for cloud computing environment: a systematic review

    Cluster Comput

    (2017)
  • E.F. Coutinho et al.

    Elasticity in cloud computing: a survey

    Ann. Telecommun. des télécommunications

    (2015)
  • Q. Zhang et al.

    Cloud computing: state-of-the-art and research challenges

    J. internet Serv. Appl.

    (2010)
  • Cited by (141)

    • Modern computing: Vision and challenges

      2024, Telematics and Informatics Reports
    View all citing articles on Scopus

    Mohammad Sadegh Aslanpour is a PhD student in Monash University and CSIRO’s DATA61, Australia. He obtained his MSc degree in Computer Engineering from Azad University, Tehran Science and Research (Sirjan) Branch, Iran in 2016. He also obtained his Bachelor and Associate degrees in Computer-Software in 2012 and 2010, respectively. From 2011 to 2019, he worked in the industry, IT Department of Jahrom Municipality, Iran as a Software Engineer. He is also serving as Editorial Board Member and Reviewer for high-ranked international journals: JNCA, SUPE, IJHPCN, IJISTA, IJSAMI and IJAC. His research interests include Cloud, Fog and Edge Computing (particularly, orchestration and automation thereof), Serverless Computing, and Autonomous Systems. For more details, please visit his homepage: Aslanpour.github.io.

    Sukhpal Singh Gill is a Lecturer (Assistant Professor) in Cloud Computing at School of Electronic Engineering and Computer Science, Queen Mary University of London, UK. Prior to this, Dr. Gill has held positions as a Research Associate at the School of Computing and Communications, Lancaster University, UK and also as a Postdoctoral Research Fellow at CLOUDS Laboratory, The University of Melbourne, Australia. His-research interests include Cloud Computing, Fog Computing, Software Engineering, Internet of Things and Healthcare. For further information, please visit www.ssgill.me.

    Adel N. Toosi is a lecturer (a.k.a. Assistant Professor) at Department of Software Systems and Cybersecurity, Faculty of Information Technology, Monash University, Australia. Before joining Monash, Dr Toosi was a Postdoctoral Research Fellow at the University of Melbourne from 2015 to 2018. He received his Ph.D. degree in 2015 from the School of Computing and Information Systems at the University of Melbourne. His Ph.D. thesis was nominated for CORE John Makepeace Bennett Award for the Australasian Distinguished Doctoral Dissertation and John Melvin Memorial Scholarship for the Best Ph.D. thesis in Engineering. Dr Toosi made significant contributions to the areas of resource management and software systems for cloud computing. His research interests include Cloud/Fog/Edge Computing, Software Defined Networking, Green Computing and Energy Efficiency. Currently, he is working on green energy harvesting for Edge/Fog computing environments. For further information, please visit his homepage: http://adelnadjarantoosi.info.

    View full text