Abstract

The cloud-fog-edge hybrid system is the evolution of the traditional centralized cloud computing model. Through the combination of different levels of resources, it is able to handle service requests from terminal users with a lower latency. However, it is accompanied by greater uncertainty, unreliability, and instability due to the decentralization and regionalization of service processing, as well as the unreasonable and unfairness in resource allocation, task scheduling, and coordination, caused by the autonomy of node distribution. Therefore, this paper introduces blockchain technology to construct a trust-enabled interaction framework in a cloud-fog-edge environment, and through a double-chain structure, it improves the reliability and verifiability of task processing without a big management overhead. Furthermore, in order to fully consider the reasonability and load balance in service coordination and task scheduling, Berger’s model and the conception of service justice are introduced to perform reasonable matching of tasks and resources. We have developed a trust-based cloud-fog-edge service simulation system based on iFogsim, and through a large number of experiments, the performance of the proposed model is verified in terms of makespan, scheduling success rate, latency, and user satisfaction with some classical scheduling models.

1. Introduction

The traditional centralized cloud service mode, with all tasks being handled in the center, faces big challenges in practical applications, including high latency, network dependency, single point of failure, and failure scale effect and cannot adapt to the instant transaction scenarios. Therefore, researchers put forward the idea of distributing some time-sensitive and low-resource demand services to be processed at the edge of the network, which is the prototype of fog or edge computing.

Based on the idea, this paper designs a cloud-fog-edge hybrid computing architecture, in which the fog layer is actually a management middleware between edge and cloud, helping the scheduler decide where to deploy and implement a service, thus to better achieve the resource balancing and low service latency [1].

The cloud-fog-edge computing model provides a superior framework for the distributed coordination of resources and the timely process of tasks. However, due to the different level of heterogeneity and resource asymmetry, the possible cross-layer task offloading, and the dynamics and mobility of edge nodes, the three-tier hybrid architecture has been facing with a more severe crisis in terms of service credibility and rational resource allocation and scheduling [25]. On the other hand, the hybrid service provision environment, which contains the edge or IoT layer, has more abundant, random, and diversified application scenarios than traditional cloud computing and requires the task scheduling strategy to be more adaptive and robust.

At present, researchers have proposed many valuable solutions for the efficient task scheduling in the distributed systems (see [611]). However, the existing strategies cannot achieve full functions in a cloud-fog-edge environment due to the following reasons: (1) the centralized trust management model cannot handle the identity authentication and behavior management under the heterogeneous and decentralized architecture, (2) the existing distributed or decentralized trust models cannot provide enough trust evidence with sufficient credibility to convince entities in the same domain or across domains, and (3) it is difficult to guarantee the relative fairness and global rationality of resource distribution in scheduling.

In response to these problems, this paper proposes a trust-enhanced fair scheduling model for the cloud-fog-edge environments. It firstly deploys the blockchain technology to build a decentralized trust framework to solve the problem of service credibility. And then, it introduces Berger’s theory of wealth distribution to propose the concept and evaluation method of service fairness. Considering the mobile application scenarios, we divide scheduling into two levels. The first level is user scheduling, which deals with the matching of mobile terminals and access points, and the second one is task scheduling, which realizes the matching of tasks and resources in the local pool. Task scheduling in a cloud-fog-edge environment is treated as a multiobjective optimization problem, which comprehensively considers terminal mobility, QoS service requirements, and workload.

The main contributions of this paper are (1) it proposes a novel distributed decentralized trust management model based on blockchain technology; (2) it introduces Berger’s fairness theory to design a preference-based fair task scheduling model for the cloud-fog-edge environments; and (3) it designs a new task scheduling algorithm that comprehensively considers user mobility, load balancing, and trust.

The rest of the paper is organized as follows. Section 2 briefly introduces the related work. Section 3 introduces the system architecture along with the design details of the double-blockchain structure-based service transaction framework. The trust-enhanced location-aware fair scheduling algorithm is proposed in Section 4. And performance evaluation is presented in Section 5. The last section concludes the paper.

2.1. Task Scheduling Model

Task scheduling refers to the process of unified resource allocation among all resources and users based on a certain resource usage rules and user requirements in a specific service provision environment. Task scheduling is the core of cloud computing. The existing task scheduling models can generally be divided into three categories: performance-centric, user-centric, and hybrid.

The goal of the performance-centric models is to improve the performance of the systems, such as system throughput, makespan, and energy consumption. The classical task algorithms like min-min, max-min, greedy algorithm, swarm intelligence, and genetic algorithm all belong to this group. Aiming at solving the energy efficiency (EE) problem in fog computing, Y. Yang et al. proposed a collaborative task offloading algorithm named MEETS [12, 13]. In [14], they developed a delay and energy consumption balanced scheduling algorithm called DEBTS. H. Sun et al. designed a contract and cluster-based resource allocation model for fog-cloud hybrid platform [15]. N. Auluck et al. proposed a security aware task scheduling model for fog computing, through the classification of users and providers, achieving a better system performance [16].

Improving service QoS and enhancing user service experience are the primary tasks of the user-centric models [1724]. In order to minimize the service delay, Z. Liu et al. proposed a dispersive stable task scheduling model named DATS [25]. M. Mukherjee et al. introduced a deadline-aware task scheduling algorithm for fog computing to complete as many tasks as possible before their deadlines [26]. G. Zhang et al. proposed a task offloading algorithm for fog computing systems to reduce the service delay [27]. G. Zhang et al. put forward a fog task offloading algorithm named DOTS to decrease latency with the help of the voluntary nodes (VNs) [28]. H. Apat et al. proposed a three-layered priority-based fog task scheduling model to meet the different deadline requirement of tasks [29].

In recent years, researchers have proposed many hybrid scheduling models considering multifactors, of which energy consumption and delay were their focus. For example, C. Tang et al. used genetic algorithms to design a hybrid task scheduling algorithm for mobile cloud computing taking into consideration multifactors such as energy consumption task requirement like deadline and cost [30]. J. Xu et al. proposed a task scheduling model combining laxity and ant colony algorithm to satisfy both energy consumption and latency [31]. In order to improve the makespan and execution cost, M. Yang et al. put forward an evolutionary heuristic-based multiobjective task scheduling model for fog environment [32]. Caching is an effective way to reduce the execution delay of tasks in edge computing, and thus some novel caching strategies have been proposed by researchers [3335].

However, indicators considered by most current scheduling models are simply makespan, service QoS, load balancing, or economic principles, and the fairness of scheduling is usually ignored.

2.2. Trust Management in Distributed Systems

Trust is an efficient mechanism in dealing with the reputation and reliability issues in the distributed open network environment. A large number of highlight resulting in trust and trust-enabled interactions are achieved [36, 37].

W. Tian et al. made a survey on trust evaluation methods in sensor cloud systems [38]. J. Wang et al. proposed a recommendation trust evaluation method based on the cloud model and attribute weighted clustering [39]. P. Zhang et al. introduced a domain-based trust management model for the public cloud [40]. For the trust-base service management, Y. Li put forward an online learning aided service offloading model for mobile edge computing [41]. H. Yang et al. designed a context-aware trust prediction model for vehicle edge computing [42]. S. Jian et al. proposed a trust-based multiobjective task allocation model for cloud service transactions [43]. C. Hu et al. designed an e-commerce recommendation algorithm combining trust and distrust factors [44]. X. Meng et al. proposed a two-tier service selection method that matched credibility and behavior patterns [45]. Z. Ma et al. introduced a trust-enabled edge data management model based on blockchain technology [46]. L. Cui et al. put forward an edge service configuration method combined with the decentralized trust [47]. We have also done some interesting work in this area, such as the service trust architecture for open cloud environment [48], trust decision strategy based on fuzzy clustering [49], and trust and preference learning-based service matching and combination models [50, 51].

However, the existing solutions cannot achieve full functions in mobile fog computing systems due to the following limitations: (1) the centralized trust framework cannot be accurately integrated with the cloud-fog-edge hybrid systems characterized by node dynamic distributed autonomy and topology loosely coupled, (2) trust crisis of the center node, which easily leads to single point of failure, (3) huge trust management overhead prevents it from being used in the instant trading scenario, and (4) trust evidence lacks transparency. Therefore, the decentralized trust management model and trust-enabled transaction mechanism for the cloud-fog-edge hybrid environments require further exploration.

3. System Overview

3.1. Trust-Enhanced Cloud-Fog-Edge Hybrid Framework

In a cloud-fog-edge hybrid framework, fog computing architects a management middleware between cloud servers and edge devices to coordinate resource allocation, thus improving system performance and fulfilling the customized demand of different devices and users. Figure 1 shows the hybrid scheduling framework of the proposed model. Blockchain technology is introduced to construct a decentralized trust model in the IoT device layer, thereby enhancing the interactive credibility of the entire cloud-fog-edge hybrid architecture.

The framework consists of three layers: (1) trust-enhanced edge/IoT layer, (2) fog layer, and (3) cloud layer. The trust-enhanced edge/IoT layer implements the peer-to-peer interconnections through the ubiquitous sensors and communication protocols over the traditional IoT infrastructure layer and constructs the distributed and decentralized trust management with a blockchain architecture. The fog layer is introduced to decide the allocation and management of resources and handle complex transactions such as cross-group/cross-cloud tasks. The fog layer consists of a large number of fog servers deployed at the network edge. Fog servers are much closer to users or terminals than the centralized cloud server and are more capable than the terminals. Therefore, they can ensure lower latency and meet the needs of cross-group or cross-domain interactions. Trust management becomes one part of the responsibility of edge/fog servers. The cloud layer is located at the top of the framework. It is mainly used to deal with some high-level and highly complex tasks, such as data mining and behavior or preference analysis, which impose high requirements on computing and storage capacity and relatively loose requirement on the response time.

3.2. Service Transaction Model Based on a Double-Blockchain Structure

Generally speaking, a complete trust authentication contains two parts: identity authentication and behavior evaluation. And the first part is easy to obtain or evaluate because the identity information of a node is relatively statically stable even in a P2P network topology. In contrast, trading behavior is dynamic, requiring a lot of computing power to record and assess. Therefore, to improve the integrity and efficiency of the trust management in a real-time trading environment, a cloud service transaction model based on a double-blockchain structure is proposed, as shown in Figure 2.

The double-blockchain structure contains two blockchains, including a trust authentication blockchain and a trading behavior blockchain.

Definition 1. Trust Authentication Blockchain (TAB). TAB is a blockchain structure that stores the trust data of the cloud-fog-edge hybrid transaction system and assists trust transaction decisions. TAB adopts an alliance chain structure, in which blocks store the trust data of nodes in the transaction system.
TAB is responsible for managing trust data in the cloud service markets and provides trust evaluation results to other nodes. Each block in TAB contains two parts: identity trust data and behavior trust data. When a node initially joins, only the identity part is written; however, as time goes by with the transactions progressing, the behavior part is continuously written. Authentication is completed by a small number of supervisors, who can be the normal miners or some special nodes elected by the market authority. Miners are responsible for storing and authenticating trust data and ensuring the consistency of the data through some specifically designed consensus mechanisms. When nodes apply to enter the trading network, they must pay a fee to run a smart contract for the initial identity authentication. In addition, when they want to obtain the trust data of the other nodes, they also have to pay a fee. The funding provides the incentive fee for the miners. Figure 3 shows the basic content of a trust block of TAB.

Definition 2. Trading Behavior Blockchain (TBB). TBB is a blockchain structure that stores the transaction data in the cloud-fog-edge transaction system. A block in TBB contains both the transaction data and the evaluation data, which will assist in generating the behavioral trust data.
TBB is responsible for generating and storing the trading data. In TBB, the miners have two tasks, one is to generate the new transaction blocks based on the latest transaction results and the other is to evaluate the behavior trust, generate a trust block, and then forward it to TAB. The corresponding trust block will be confirmed and stored by the miners in TAB. Figure 4 shows the basic content of a trading block in TBB. The block structure of TBB is very similar to that of TAB, and the only difference is the storage content.
TAB and TBB jointly ensure the credibility of cloud-fog-edge interaction. When an entity registers for the first time, its identity trust will be written into TAB. As the transaction progresses, the trading record between entities will be written into TBB one by one and then the behavior trust will be evaluated in TBB and be forwarded into TAB. When TAB gradually grasps enough trust data of the entities in the transaction chain, it can help entities make trust decisions more accurately, which improves the reliability and success rate of interactions. Furthermore, with the benefits of the double-blockchain structure, an efficient parallel computing is realized. Because trust value is provided by TAB, while the large-scale calculation or evaluation is done on the TBB side, this effectively reduces the latency caused by trust management and makes it possible for the application of blockchain in a real-time and high-reliability scenarios.

4. Trust-Enhanced Location-Aware Fair Scheduling Model in a Cloud-Fog-Edge Computing System

4.1. Service Justice

The theory of distribution justice (Berger’s theory) proposed by Berger et al., is a theory of the distribution of social wealth. Berger et al. believed that justice is a subjective behavior, of which the conclusion is drawn from the comparison with the comparable objects. In the real world, people usually refer to the surrounding social information, such as the status and compensation of others, to generate their own expectations, which are used to judge whether they are fairly treated or not. Furthermore, the satisfaction of these expectations affects their future behaviors. Figure 5 is the diagram of Berger’s model.

Here, c or C is the attribute set and go or GO is the expectation set; go represents the expectation value, while GO means the actually obtained value. In Berger's theory, the distribution justice holds only when the local self-structure and the reference structure are significantly related.

In a cloud-fog-edge hybrid scheduling framework, fog provides services to users by coordinating resources from edge or cloud and users pay a service fee according to the usage of resources or time. Therefore, service fairness of users is reflected in the following: the scheduling system is able to provide services on user demand and preferences, ensuring a high user satisfaction and experience.

Definition 3. Service Justice (SJ). It represents the degree of agreement between the QoS of user actually obtained and the expected. How to evaluate SJi is given in the following formula:where represents the equilibrium coefficient, means the actual obtained service quality vector of useri, and is the demand vector of useri.
Obviously, the overall service justice of the scheduling system is the average of all system users’ SJ, which is illustrated in the following formula:The overall system balance is achieved by constraining the overall system service justice, namely, .
Suppose the capability vector of a resource can be described by , where represent the CPU, memory, and bandwidth capability of the resource/vm. The resource expectation of a user to the resource is described by , where represent the expected CPU, memory, and bandwidth capability of the resource. In order to eliminate the influence of dimensions of different capability, it uses the following formula for normalization:After normalization, performance or expectation parameters of the virtual machine are mapped to the [0, 1]. However, different users or tasks have different service preferences, for example, some are CPU or computing power enthusiasts and some are bandwidth pursuers. Let describe the service preference or weight of the attribute. It uses modified cosine function to compare the similarity and distance between expectation and obtained. Let denote the service justice when taski is scheduled to resourcej, and can be calculated by the following formula:Accordingly, the SJ of the task set can be obtained by the following formula:Here, denotes the number of tasks in the scheduling system.

4.2. Trust-Based Location-Aware Fair Scheduling Model
4.2.1. User/Smart Thing Model

A user or a smart thing is an intelligent entity who requests for mobile services. Let ST = {st0, st1, …, stk−1} represents a user or a smart thing set. The ith user sti (i ∈ [0, k − 1]) can be further described as sti= {stID, stName, stLocation, stTaskSet, stTrust}. The meaning of each attribute is as follows:(i)stID represents the unique system identity of a user/smart thing(ii)stName represents the name of the user/smart thing(iii)stLocation represents the location of the user/smart thing(iv)stTaskSet represents the task set submitted by the user/smart thing(v)stTrust represents the trust requirement of the user/smart thing

4.2.2. Task Model

T={t0, t1, …, tn−1} represents a task set, where the ith task ti (i∈ [0, n − 1]) can be further described as ti = {tID, tState, tRRes, tORSet, tDeadLine}. The meaning of each attribute is as follows:(i)tID represents the identity of a task(ii)tState represents the state of the task(iii)tRRes represents the resource requirements of the task, tRRes can be further described as tRRes = {tComp, tBW, tStor}, where tComp, tBW, and tStor represent the computation, network, and storage requirement of the task(iv)tORSet represents the obtained resource set of the task(v)tDeadLine represents the latest completion time of the task

4.2.3. Provider Model

AP= {ap0,ap1,…,apk−1} represents the provider set. In this paper, service provider refers to the access point or gateway server. In a cloud-fog-edge hybrid platform, a service provider can be a cloud server, a fog server, or even an edge server.

The ith provider api (i ∈ [0, k − 1]) can be further described as api= {apID, apName,apResouceSet, apLoad, apLocation}. The meaning of each attribute is as follows:(i)apID represents the unique system identity of a provider(ii)apName represents the name of the provider(iii)apResourceSet represents the resource set managed by the provider(iv)apLoad represents the load of the provider(v)apLocation represents the location of the provider

4.2.4. Resource Model

R = {r0, r1, …, rm−1} represents the resource set. The jth resource rj (j ∈ [0, m − 1]) can be further described as rj = {rID, rName, rProvider, rCap}. The meaning of each attribute is as follows:(i)rID represents the unique system identity of a resource(ii)rName represents the name of the resource(iii)rProvider represents the manage provider of the resource(iv)rCap represents the capability of the resource, rCap = {rComp, rBW, rStor}, where rComp, rBW, and rStor represent the computation, network, and storage capability of the resource.

4.2.5. Two-Level Service Scheduling Model

This paper divides scheduling in a cloud-fog-edge hybrid environment into two levels, user scheduling and task scheduling. User scheduling realizes the binding of the smart thing (mobile user or terminal) to the credible nearest access point, while task scheduling realizes the matching of tasks issued by the user with resources/vms managed by the access point.

The user scheduling model is defined as USM = (ST, AP, ACS), which selects the most suitable gateway or access point for the user from the list of providers:(i)ST means the smart thing set, as defined in 4.2.1.(ii)AP represents the gateway or access point set, as defined in 4.2.3.(iii)ACS: AP->APi represents the selection of the access point ap for smart thing st using a certain strategy. In a cloud-fog-edge environment, since the location of a user or terminal is constantly changing and a fog or edge server also has a limited service capability, distance between st and ap must be considered. Besides, for a trustworthy trading, credibility is another important factor. Therefore, the primary principles of ACS include distance and trust.

The task scheduling model is defined as TSM = (T, R, RCS), which matches the most suitable resources for the task set of the user.(i)T means the task set, as defined in 4.2.2.(ii)R means the resource set managed by the service provider, as defined in 4.2.4.(iii)RCS: R->Ri represents the selection of the resource r for task t using a certain strategy. The main selection principle of RCS in this paper is service justice defined in 4.1 and work load.

Figure 6 shows the general steps of the proposed model in this paper.(1)The user scheduler searches appropriate access points for a specific user/smart thing according to the current location and makes recommendation according to distance and trust;(2)When the recommendation arrives, the user/smart thing requests a trust service from the TAB and makes decision after obtaining the trust data. If it agrees to trade, then proceed to step (3); otherwise, it asks the scheduler to resume recommendation and returns to step (1);(3)The access point (gateway) starts the task scheduler to select specific resources for the user;(4)The task scheduler selects the appropriate virtual machines to perform tasks according to the principle of service justice and load balance;(5)The user obtains the service and makes service evaluation. The transaction data enter the TBB;(6)TBB regularly performs trust evaluation and forwards trust blocks into TAB.

4.3. User Scheduling Algorithm

The user scheduler chooses the most suitable access point for user or smart thing as shown in Algorithm 1. The basic principle is distance and trust.

Input: AP, sti, coverage, trust_threshold
Output: the matching of sti to an appropriate ap
(1)for each apj in AP do
(2)   Calculate the distance from sti to apj;
(3)   Calculate trust from sti to apj;
(4)   if (distancesti_apj ≤ coverage) && (truststi_apj ≥ trust_threshold) then
(5)    apj − > APcandidate
(6)   end if
(7)done
(8)for each apk in APcandidatedo
(9)   if ( distancesti_apk < min_distance) then
(10)    min_distance = distancesti_apk;
(11)    apchosen=apk;
(12)   end if
(13)done
(14)if apchosen! = NULL then
(15)  connect sti to apchosen;
(16)end if

In Algorithm 1, coverage means the service coverage of an ap, and trust_threshold is the minimum required trust value of the smart thing.

The user scheduler chooses the nearest and reliable access point for a user. The general steps of user scheduling are as follows: (1) the user scheduler selects the candidate subset APcandidate from AP that can cover sti and meet its minimum trust requirement, (2) it calculates the distance between ap and sti and chooses the nearest ap, namely, apchosen from APcandidate, and (3) it connects sti with apchosen.

4.4. Task Scheduling Algorithm

The task scheduling algorithm references Berger’s model to ensure the overall system justice.

Algorithm 2 shows how task scheduler (running on an access point) chooses the suitable resources/servers for mobile tasks triggered by a user or a smart thing.

(i)Input: R, ti, load_safe_range
(ii)Output: the matching of ti to an appropriate r
(1)for each rj in R do
(2)   observe the workload of rj;
(3)   if (workloadrj ≤ load_safe_range) then
(4)    rj - > Rcandidate
(5)   end if
(6)done
(7)for each rk in Rcandidatedo
(8)   Calculate service justice SJti_rk of ti in rj according to formula (4);
(9)   If ( SJti_rk < min_sj) then
(10)    min_sk = SJti_rk;
(11)    rchosen = rk;
(12)   end if
(13)done
(14)if rchosen! = NULL then
(15)    schedule ti to rchosen;
(16)    update the workload of rchosen;
(17)end if

In Algorithm 2, load_safe_range represents the safe load limit of a server or a resource. The general steps of task scheduling are as follows: (1) it puts the load-safe virtual resources into the candidate resource set Rcandidate, (2) it calculates the service fairness of each resource rk in Rcandidate according to the demand of ti and the capability of rk, and selects the most suitable resource rchosen, which achieves the maximum SJ, and (3) it schedules ti to rchosen and updates the workload of rchosen.

4.5. Blockchain-Architected Lightweight Trust Algorithm

Trust runs through the entire process of scheduling. The node initiates the transaction requests trust service from the trust blockchain TAB. After paying a certain fee, the trustworthiness of the candidate provider (the access server or the application server) is obtained, thereby helping it to make trustworthy decisions. After each transaction, transaction-related data (transaction type, content, and evaluation data) will be used to generate transaction blocks and be added to the transaction blockchain TBB after consensus. Thereafter, TBB initiates a trust evaluation transaction at regular intervals and pushes the new trust block to TAB.

This paper uses a lightweight method to compute the trust of a specific trading node as shown in the following formula:where is the trust value of node i, is the evaluation weight of trustor node k, and represents the trust evaluation value from k to i.

In the proposed mode, in order to deploy blockchain-based framework, we defined some transaction-related classes and trust-related classes. The transaction-related classes are used to maintain the transaction data and keep it traceable and tamper-proof. The trust-related classes are used to maintain trust data. They are mainly generated offline and can support trust decisions in the real-time transactions. Following is the definition of the critical data structures. The class “transaction” defined in Algorithm 3 stores transaction data of the trading system, through which trust value from trustor to trustee can be obtained. And the class “TrustBlock” defined in Algorithm 4 is used to store the trust value of a certain node. Each block in TAB or TBB has the normal content of a blockchain block (timestamp, hash value, and the pointer to the previous block) and some special functions (mineBlock, calculateHash, and addTransaction). (Algorithms 3 and 4)

class Transaction {
(1)String transactionId;
(2)PublicKey trustor;//the public key of trustor
(3)PublicKey trustee;//the public key of trustee
(4)double trust;//trust value from trustor to trustee
(5)byte[] signature;
(6)ArrayList < Transaction > transactions;
(7)boolean processTransaction() {
(8)    verifySignature();
(9)    transactionId = calulateHash();
(10)    generateTrust();//generate trust according to transaction details
(11)    TransactionOutput(this.recipient, trust, transactionId));
(12)    return true;
(13)} //add transactions into transactionChain TBB
(14)boolean addTransaction(Transaction transaction) {
(15)    verifyTransaction();
(16)    processTransaction();
(17)    transactions.add(transaction);
(18)    return true;
(19)}
}
TrustBlock {
(1)String hash;
(2)String previousHash;
(3)String merkleRoot;
(4)long timeStamp;
(5)int nonce;//the proof of miners
(6)double trust;//the integrated trust of a certain node
(7)TrustBlock(String previousHash) {//the basic structure of a trust block
(8)   this.previousHash = previousHash;
(9)   this.timeStamp = new Date().getTime();
(10)   this.hash = calculateHash(); }
(11)void mineBlock() {
(12)   merkleRoot = StringUtil.getMerkleRoot(transactions);
(13)   while(!hash.substring(0, difficulty).equals(target)) {
(14)    nonce ++;
(15)    hash = calculateHash();
(16)}
(17)}
}

5. Performance Evaluation

5.1. Experimental Design

This paper designed a task scheduling prototype for performance test in a cloud-fog-edge environment based on iFogsim [52]. In iFogsim, FogDevice is used to represent a common device from terminal to fog server. In order to reflect the mobility and distinguish the functions of different devices, new entities are added: MobileDevice, AccessPointDevice, MobileSensor, and MobileActuator. Inheriting from parent class FogDevice, MobileDevice handles mobile tasks, making trust decision and service selection. A mobile device randomly appears in a specific location and has its own specific resource preferences. MobileSensor inherits from sensor, acting as a mobile sensor, and is able to generate and transmit tasks. While MobileActuator inherits from Actuator to deal with the execution and output of tasks. AccessPoint also inherits from FogDevice, which is an intermediary for resource allocation and service scheduling according to the location of the mobile devices. It helps a mobile device to choose the most cost-effective service resources and handles real-time task migration. The resource pool that actually executes tasks is a collection of traditional fog devices, and each fog device is configured with certain capabilities. Figure 7 shows the key classes and their inherent relationships in the scheduling model.

This paper draws on the idea of blockchain to construct a trust-enhanced task scheduling model. The proposed model contains a dual-blockchain structure, including a transaction blockchain and a trust blockchain. Thus, the new classes including the class of TransactionBlock and TrustBlock class are derived. The transaction block records the actual transactions of the system in a traceable and nontamperable manner. And the trust block records the trust value of the nodes, where the identity trust depends on the authenticity of the node, while the behavior trust is continuously updated by the transaction evaluation results. The generation of the trust block mainly adopts the offline mode, and a trust block is generated according to the feedback data obtained from the transaction blockchain. Trust management contains three parts: trust initialization, trust decision, and trust maintenance. Trust decision helps entities to choose the credible trading partners. Figure 8 is the UML diagram of the trust-related classes and their relationships.

In order to serve mobile devices, we designed the mapping and migration rules for tasks in virtual machines, which were implemented in the class MigrationPolicy. The main strategies include load balance, shortest distance between smartThing and AccessPoint or between smartThing and foglets (vms), lowest latency, and trust-based. Figure 9 shows the migration-related classes and their relationships.

5.2. Simulation Indicator and Benchmarks

In the experiments, we use various evaluation indicators for performance test of the different scheduling models, such as makespan, tuple loss rate, distribution justice, latency, and transaction success rate. Here, makespan represents the overall execution time of the task set. Tuple loss rate refers to the percentage of tasks dropped due to the limited processing capacity of the edge server or task migration. And the system justice is defined in formula (5).

The mentioned strategies were implemented and tested in the EEG Tractor Beam Game [53]. It is a latency-critical game requiring each player to wear an EEG headset to process the EEG signals and obtain his brain state. In EEG Tractor Beam Game, there are seven types of tuples carried between the different modules of the application, as shown in Table 1. The capabilities of cloud, fog, and edge devices are shown in Table 2, and the task sending interval of two different EEG headsets is 10 ms and 5 ms, respectively.

The speed of MobileDevice is set to be 20 m/s, including nine different moving directions: NONE, EAST, NORTH, NORTH EAST, NORTHWEST, WEST, SOUTHWEST, SOUTH, and SOUTHEAST. The maximum switch radius of vm to a certain access point is set to be 40 meters. Various fog devices (resources and access points) are randomly and evenly distributed on the map.

This paper aims at improving the performance of resource scheduling in a cloud-fog-edge hybrid computing architecture, including the total execution time, service delay, the fairness, and reliability. Therefore, we chose the min-min and Berger’s model as the benchmark comparison models. At the same time, in order to measure the influence of the different factors in the proposed model, we also compared it with the subtraction models, namely, locate-ware and load-balance. The locate-ware is the subtraction model which only remains the location switching strategy of the proposed model in scheduling, and the load-balance is the subtraction model that only maintains the load-sensitive strategy.

The task scheduling algorithms including min-min, Berger’s Model, proposed model, location-aware, and load-balance model are compared from various perspectives.

5.3. Simulation Result

The experimental results are shown in Figures 1017. From the results, we can see that the proposed model has achieved better effect than the other models on almost all the experimental metrics. And two subtraction models also gained better results than the other benchmark models.

5.3.1. Comparison on System Performance

For the test of system performance, we compared the mentioned models in terms of makespan, service latency, and tuple loss rate.

The proposed model and the location-aware model work well in the total completion time and service delay. Because the built-in location-sensitive scheduling strategy takes into account the mobility of nodes to match it with the closer access point, which is able to effectively reduce the transmission overhead, the service delay, and the total execution time. The load-balance model gains a good performance in the index of the tuple loss rate because it considers the load status of resources in the task scheduling stage, which can effectively reduce the waiting time in queue and avoid tuple loss caused by overload.

The overall system performance of the min-min model is not bad. However, since it does not consider the resource preferences, mobility, and load, it may cause tuple loss and redistribution, which ultimately affects the overall execution time. Berger’s model does not work well because it focuses only on the fairness of scheduling, which tends to satisfy the needs of users rather than the overall performance of the system. In a cloud-fog-edge hybrid environment with nodes moving frequently, although the global fairness algorithm can find the most suitable resource for nodes, the scheduling result may not be implemented due to the distance, which increases the possibility of tuple loss and redistribution, thus eventually increasing in the total execution time. In particular, the overhead of Berger’s model is relatively large. When the total number of tasks increases, its execution time is very long, which may affect its deployment in the actual applications.

5.3.2. Comparison on User Satisfaction

User satisfaction, measured by service justice (SJ) in this paper, is an important index to evaluate the quality of scheduling algorithms. Berger’s model performed the best in this respect because scheduling fairness was its core concern. However, using the traditional Berger’s model to perform global matching and fairness calculations during task scheduling often leads to high computational overhead, which may cause the paralysis of the resource allocation system when the task scale is large. The experimental results indicate that when the number of tuples is greater than 125, Berger’s model cannot give a resource allocation decision within the tolerable time.

The proposed model still performs well in terms of user satisfaction because it adopts a two-level scheduling mode, and through the fairness factors, it takes into consideration the specific requirements of tasks in the process of task scheduling.

5.3.3. Influence of the Decentralized Trust Mechanism

This section tested the performance of the double-blockchain-based trust mechanism proposed in this paper.

Figure 17 shows the result of the transaction success rate. It can be seen that the decentralized trust can maintain a high degree of the transaction success rate despite the increase in the malicious nodes, indicating that it can effectively assist a reliable transaction decisions. In sharp contrast, in the random transaction scenario, the transaction success rate drops sharply as the number of malicious nodes increases.

This paper also used NetLogo [54] to test the efficiency of the proposed trust mechanism. Table 3 shows the parameters of the simulations.

Several experiments were done to observe the impact of trust on the success rate of transactions when the proportion of malicious providers was fixed to 30%. Figure 15 is the experimental result when trust mechanism was loaded, and Figure 16 is the experimental data without any trust mechanisms (in the case of random transactions).

In the simulation chart, users are represented by the blue color person-shaped turtles, and the fog providers (access point or service providers) are the red color star-shaped turtles. The links in the charts represent the relationship between the different entities, including the trading relationship (blue edges), the recommendation relationship (yellow edges), and the cooperation between service providers (red edges).

Figures 15(a) and 16(a) are the initial state of the simulation, Figures 15(b) and 16(b) are the final state of the trading cycle, and Figures 15(c) and 16(c) are the records of transaction success ratio output from NetLogo reporter. It can be seen from the figures that, when the trust mechanism is loaded, the transaction success rate gradually increases with the increasing of the number of transactions, which remains almost the same when no trust is loaded.

In order to measure the overhead of trust, we also evaluated the influence of trust on the total execution time. Figure 17 shows the additional time overhead brought by the loading of trust, from which we can see that trust does not have much influence on the total execution time, with even a few nodes showing the shorter execution time. The reason is that after trust has been running for a period of time, when entities in the system are able to correctly select the credible trading partner, the reselection or transaction failure in the random transactions can be effectively avoided, thus, to a certain extent, reducing the total execution time.

In summary, in the cloud-fog-edge hybrid environment, in order to improve the effective matching of terminals, access points, and their expected resources, it is necessary to fully consider the location, the workload, and also the credibility of nodes. The proposed decentralized trust strategy is able to guarantee the safety and the reliability of interaction.

6. Conclusion and Future Work

This paper proposes a trust-enhanced location-aware fair task scheduling model for the cloud-fog-edge hybrid environment. The new model contains a three-layer architecture of IoT, fog, and cloud. The fog layer is utilized to achieve the cloud-fog or fog-edge resource coordination and unified scheduling. The proposed task scheduling algorithm comprehensively considers user mobility, system justice, load balance, and trust requirement. Berger’s theory is introduced to solve the fairness problem in task scheduling. In resource allocation, it comprehensively considers the location, task’s QoS requirements, the capability, and the load of the resources. In addition, to improve the credibility of service interaction, it draws on the idea of blockchain to build a decentralized trust framework. The performance of the new model and the related strategies was evaluated by a series of related experiments.

However, the proposed model still has some imperfections. For example, it adopts a two-level scheduling mode to solve the problem of mobile access. However, the location of the mobile terminals is constantly changing, requiring the timely replacement of the access point. Therefore, it is necessary to determine where and when to switch and also the resource reservation algorithms. Things like the moving direction, speed, and preferences can be used to predict the next access point and make resource reservations for reducing the handover delay. In addition, the dual-blockchain-based trust management, with all the transactions recorded in the trading behavior blockchain, needs to find the most appropriate time to generate the trust blocks, whether in a fix time period or a certain number of transactions. All these problems are our future work.

Data Availability

Data is available upon request to the corresponding author.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under grant 61702151, 61702320, 61772334, the National Key Research and Development Plan under grant 2018YFB1003800, and the Joint Funds of the Zhejiang Provincial Natural Science Foundation of China under grant LHY21E090004.