Dynamic Self-Learning Artificial Bee Colony Optimization Algorithm for Flexible Job-Shop Scheduling Problem with Job Insertion

Abstract

To solve the problem of inserting new job into flexible job-shops, this paper proposes a dynamic self-learning artificial bee colony (DSLABC) optimization algorithm to solve dynamic flexible job-shop scheduling problem (DFJSP). Through the reasonable arrangement of the processing sequence of the jobs and the corresponding relationship between the operations and the machines, the makespan can be shortened, the economic benefit of the job-shop and the utilization rate of the processing machine can be improved. Firstly, the Q-learning algorithm and the traditional artificial bee colony (ABC) algorithm are combined to form the self-learning artificial bee colony (SLABC) algorithm. Using the learning characteristics of the Q-learning algorithm, the update dimension of each iteration of the ABC algorithm can be dynamically adjusted, which improves the convergence accuracy of the ABC algorithm. Secondly, the specific method of dynamic scheduling is determined, and the DSLABC algorithm is proposed. When a new job is inserted, the new job and the operations that have not started processing will be rescheduled. Finally, through solving the Brandimarte instances, it is proved that the convergence accuracy of the SLABC algorithm is higher than that of other optimization algorithms, and the effectiveness of the DSLABC algorithm is demonstrated by solving a specific example with a new job inserted.

1. Introduction

The Flexible Job Shop Scheduling Problem (FJSP) is a non-deterministic polynomial (NP)-hard problem, which is an extension of the classic JSP [1]. In FJSP, the operations of the job can be processed on multiple machines, that is, the corresponding relationship between the operation and the machine is uncertain [2]. In today’s world, with the diversification of user needs, flexible job-shops are faced with frequent insertion of new jobs. When a new job is inserted at a certain point, to reduce the makespan, it is necessary to dynamically form a new scheduling scheme by reasonably arranging the processing sequence of the new job and the old jobs, as well as the corresponding relationship between the operations and the machines, so as to improving economic efficiency and machine utilization. This is the DFJSP studied in this paper, which is a further study of FJSP. For combinatorial optimization problems, an efficient meta-heuristics algorithm can solve DFJSP. In addition, through reinforcement learning (RL) and other learning-based algorithms, the meta-heuristic can be optimized to improve the accuracy of flexible job-shop scheduling and obtain the optimal scheduling scheme.

With the continuous development of computer technology, many intelligent algorithms have been applied to combinatorial optimization problems, such as the particle swarm optimization (PSO) algorithm [3,4], genetic algorithm (GA) [5,6], artificial fish swarm algorithm (AFSA) [7], Bayesian algorithm [8], ant colony optimization (ACO) algorithm [9], gray wolf optimization (GWO) algorithm [10], lion swarm algorithm [11], ABC algorithm [12,13], etc. Sabharwal et al. [14] proposed an improved GA algorithm with excellent performance, which generated the initial population through probability, avoided the error caused by the interaction between the input parameters, which solved the combinatorial optimization problem well. Wang et al. [15] optimized the ACO algorithm, changed the pheromone update mechanism, and optimized the makespan of the FJSP, overcame the shortcoming of the ACO algorithm falling into local optimum, and improved its computational efficiency. Yao et al. [16] improved the population state of GWO algorithm, and proposed the IGWO algorithm through the position-based learning strategy, which enhanced the global search ability of GWO algorithm. Wang et al. [17] proposed an adaptive multi-objective PSO algorithm with cost and tardiness as objective functions. The algorithm adopts an elite strategy and small probability mutation mechanism to avoid premature convergence of PSO and improve the convergence accuracy. Ge et al. [7] improved the AFSA algorithm with the goal of minimizing the makespan, and improved the diversity of the population by adjusting the arrangement mechanism of machines and operations. The algorithm improves its local and global search ability through attracting behavior and path search strategy, and improves the performance of the algorithm to solve FJSP. Park et al. [18] applied the same crossover strategy to the categorical part and sequential part, and proposed a unified GA algorithm, which simplified the structure of GA and effectively explored the search space of FJSP. It can be seen from the above literature that the key parameters of most algorithms are set in advance, and their adjustment methods are difficult to reasonably determine [19]. In addition, some improved algorithms have too many input parameters, and their stability needs further research and verification [20].

The artificial bee colony algorithm was proposed by Karaboga et al. [21] in 2005. Due to its advantages of strong stability and few parameters, the ABC algorithm has received extensive attention and research by scholars [20]. For more than ten years, many researchers have applied the ABC algorithm and its improved algorithm to FJSP, and have achieved practical achievements. Li et al. [22] proposed an improved ABC algorithm using a two-dimensional vector coding method, aiming at the minimum energy consumption and the shortest makespan, taking into account important factors such as job preparation time, which improved the local and global search capabilities of the ABC algorithm. Pan et al. [23] developed an adaptive strategy aiming at the shortest total time of earliness and tardiness, which improved the diversity of the population and enhanced the local reinforcement ability, and solved the job-shop scheduling problem using the discrete artificial bee colony (DABC) algorithm. Meng et al. [24] proposed a hybrid ABC algorithm with the goal of minimizing the total flowtime, which balances the local and global search by dynamically adjusting the search range and increases the diversification of the population. Zhang et al. [25] improved the ABC algorithm to solve stochastic FJSP, and finally obtained the solution with the shortest lateness by screening the solutions and reducing the computational load with the K-armed bandit model. Zheng et al. [26] optimized the population of ABC algorithm by using chaos theory, left-shift strategy and crossover operation with the goal of minimum makespan. This method accelerates the convergence speed of the algorithm, improves the ability of global search, and solves the fuzzy FJSP problem. Gu [27] used adaptive neighborhood search strategy and greedy method to optimize the population of ABC and retain the optimal solution, which prevented the loss of the optimal solution and solved the multi-objective low-carbon FJSP. However, it can be seen from the above researches that most of the parameters of the improved ABC algorithm are set in advance or unchanged, and are not adjusted with the change of the population state, which limits the performance improvement of the ABC algorithm. The Q-learning algorithm, as a reinforcement learning algorithm, focuses on online learning and can maintain a balance between exploration and exploitation. Learning information and updating parameters can be obtained by receiving rewards from the environment for actions [28]. Therefore, the Q-learning algorithm provides the possibility to dynamically adjust the update dimension of the ABC algorithm by virtue of its learning ability. In addition, in view of the frequent insertion of new jobs in the actual flexible job-shop, it is necessary to study an algorithm to solve the dynamic scheduling problem of the job-shop.

To solve the problem that the update dimension of the ABC algorithm cannot be dynamically adjusted and a new job is frequently inserted into the flexible job-shop, this paper proposes the DSLABC algorithm with the goal of the shortest makespan. Firstly, to dynamically adjust the update dimension m, this paper combines the ABC algorithm with the Q-learning algorithm, and proposes the SLABC algorithm. In each iteration of the ABC algorithm, the Q-learning algorithm can use the characteristics of exploration and exploitation to select the appropriate update dimension according to the population state, realize the dynamic adjustment of the update dimension, and improve the convergence accuracy of the ABC algorithm. Secondly, this paper determines the specific method of dynamic scheduling, and proposes the DSLABC algorithm. When a new job is inserted at a certain time, the DSLABC algorithm reschedules the new job and the operations that has not started processing, and dynamically generates a new scheduling scheme, which solves the rescheduling problem in the flexible job-shop and reduces the makespan. Finally, this paper compares the DSLABC algorithm with other meta-heuristics algorithms, and proves the convergence performance and advantages of the DSLABC algorithm by solving 10 Brandimarte instances. And the feasibility of DSLABC algorithm to solve DFJSP is proved through a specific instance. In conclusion, the DSLABC algorithm proposed in this paper successfully realizes the dynamic adjustment of the update dimension, improves the convergence accuracy of the ABC algorithm, and successfully solves the DFJSP.

In this paper, Section 2 introduces the flexible job shop scheduling problem with new job insertion, including the mathematical representation, constraint conditions, and coding of the problem. Section 3 introduces the two basic algorithms needed to propose the DSLABC algorithm: ABC and Q-learning algorithms. Section 4 introduces the flow and specific settings of the DSLABC algorithm. Section 5 verifies the effectiveness and practicability of the DSLABC algorithm. Section 6 is the conclusion of this paper.

2. FJSP with New Job Insertion

2.1. FJSP

The operations of each job in the flexible job-shop can be processed on multiple machines. Therefore, the purpose of this study is to minimize the makespan through the reasonable arrangement of operations and machines [29,30,31]. The various parameters and their meanings of FJSP are shown in

Table 1. The symbols of FJSP.

Symbols	Meaning
n	The total number of jobs.
$J_i$ (i = 1, 2, …, n)	The i-th job.
$O_{ij}$ (j = 1, 2, …, l)	The j-th operation of the i-th job.
m	The total number of machines.
$M_k$ (k = 1, 2, …, m)	The k-th machine.
$C_{max}$	Makespan
$b_{ijk}$	The beginning time of the $O_{ij}$ processing on the $M_k$.
$t_{ijk}$	The duration time of the $O_{ij}$ processing on the $M_k$.
$W_{ijk}$	$\left\{\begin{array}{cc}1,\hfill & if{O}_{ij}isassignedto{M}_k\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right.$
$Y_{ijaqk}$	$\left\{\begin{array}{cc}1,\hfill & O_{ij}isprocessedonequipment{M}_{k}before{O}_{aq}\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right.$

. In the flexible job-shop, each job consists of multiple operations, and each operation can be processed on more than one machine. In this paper, through adjusting the corresponding relationship between $O_{ij}$ and $M_k$, an optimal scheduling scheme is formed and the makespan is reduced. Equation (1) is the objective function of solving FJSP in this paper.

This paper summarizes four constraints for solving FJSP. Condition I: The processing time of any operation $O_{ij}$ is greater than 0. Condition II: The operations of the same job must be processed in strict accordance with the sequence. Condition III: Any operation $O_{ij}$ can be processed on at least one machine. Condition IV: Any machine can only process at most one operation at a certain time. In addition, there is no order requirement between different jobs. The formulas in

Table 2. The mathematical expression of constraints.

$t_{ijk}$>0	I
$b_{ijk}$+$t_{ijk}\le b_{i(j+1)k}$	II
${\sum }_{k=1}^m{W}_{ijk}\ge 1$	III
$b_{ijk}+t_{ijk}\le b_{aqk}+Z(1-Y_{ijaqk})$	IV

are the mathematical representations of the above four constraints.

$$Min{C}_{max}=Min(Max\sum _{i=1}^n\sum _{j=1}^l(b_{ijk}+t_{ijk}))$$

(1)

A suitable coding method can effectively solve the combinatorial optimization problem. This paper illustrates the coding method with a specific instance.

Table 3. An instance of FJSP with a scale of 3 × 3.

Jobs	Operation	${\mathit{M}}_1$	${\mathit{M}}_2$	${\mathit{M}}_3$
Job 1	$O_{11}$	2	3	-
	$O_{12}$	3	-	4
	$O_{13}$	4	3	5
Job 2	$O_{21}$	2	-	-
	$O_{22}$	5	3	-
	$O_{23}$	-	4	3
Job 3	$O_{31}$	3	2	-
	$O_{32}$	-	4	2
	$O_{33}$	2	-	3

is an instance of FJSP with a scale of 3 × 3 (job × machine), and Figure 1 is the optimal scheduling scheme of Table 3. Figure 2 is the coding method of the scheduling scheme of Figure 1, which is divided into the corresponding operation sequence and machine sequence. The second “2” of the operation sequence means the second operation of job 2, and the corresponding “2” of the machine sequence means that $O_{22}$ is processed on machine $M_2$. The processing order on machine $M_1$ is $O_{21}\to O_{11}\to O_{12}\to O_{33}$, the processing order on machine $M_2$ is $O_{31}\to O_{22}\to O_{13}$, and the processing order on machine $M_3$ is $O_{32}\to O_{23}$.

2.2. DFJSP

At present, there will be unexpected situations of new job insertion in real job-shops [32,33]. When a new job is inserted at a certain moment, through reasonably adjusting the processing sequence of the new and old jobs, and the corresponding relationship between the operations and the machines, the makespan can be effectively reduced and the efficiency in the flexible job-shop can be improved. Through the description of three methods for dynamic scheduling of job-shop tasks [20,34,35], the dynamic scheduling method adopted in this paper is determined. Table 3 is an instance of FJSP, and

Table 4. The inserted job 4.

New Job	Operation	${\mathit{M}}_1$	${\mathit{M}}_2$	${\mathit{M}}_3$
Job 4	$O_{41}$	4	5	3
	$O_{42}$	2	-	3
	$O_{43}$	3	4	-

shows the newly inserted job.

Method I: After the original scheduling scheme is completed, and the machines are in an idle state, that is, when the machines have no processing operations, the operations of the new job are processed after the original scheduling scheme, as shown in Figure 3. Method I simply schedules twice in a row, and the makespan is long.

Method II: The operations of the new job are inserted into the original scheduling scheme, and the operations that cannot be inserted are scheduled according to method I. Method II relies too much on the original scheduling scheme, with very large randomness and low stability.

Method III: Reschedule the newly inserted job and the operations that have not started processing. According to Equation (2), determine the operations that have not started processing in the original scheduling plan, and $t_{insert}$ represents the insertion time of the new job. Figure 4 shows the new scheme obtained by rescheduling when a new job is inserted. Figure 5 is the coding method corresponding to Figure 4, and the red dashed box is the operations after rescheduling. Compared with methods I and II, method III can complete the processing of all jobs in the shortest time and has higher stability. Therefore, this paper uses method III to study dynamic scheduling.

$$b_{ijk}<t_{insert}$$

(2)

3. Basic Algorithm

This paper combines the ABC algorithm with the Q-learning algorithm and proposes the SLABC algorithm. Then, the rescheduling problem with new job insertion is studied, and the DSLABC algorithm is proposed. The following is an introduction to the two basic algorithms.

3.1. ABC Algorithm

As an optimization algorithm, the ABC algorithm effectively solves constraint problems [36,37,38]. The ABC algorithm consists of nectar sources, scouts, employed bees, and onlooker bees.

Table 5. Parameters of ABC algorithm.

Parameters	Meaning
SN	The total number of nectar sources.
$i,k\in \{1,2,\dots ,SN\}$	The i-th nectar source.
$j,d\in \{1,2,\dots ,D\}$	The number of updated dimensions.
$X_{ij}$	Original nectar source.
$V_{ij}$	The new nectar source obtained near the original nectar source $X_{ij}$.
$X_{kj}$	Randomly selected nectar source different from the original nectar source $X_{ij}$.
${\varphi }_1,{\varphi }_2$	Random number generated in the range [0,1].
$F_i$	The fitness of $X_{ij}$.
$P_i$	Probability of $X_{ij}$ being selected.
$X_{id}$	The new nectar source was obtained by updating the d dimension of the i-th nectar source.
$X_d^{min}$	The lower bound of the d-th dimension.
$X_d^{max}$	The upper bound of the d-th dimension.

shows the meaning of each parameter in the ABC algorithm. The higher the fitness of the nectar source is, the higher the quality of the solution of the algorithm. The principle of the ABC algorithm mainly includes four stages [39,40]:

(1) The initial stage of population: The initial population is generated through Equation (3), and new nectar sources are generated.

$$X_{id}=X_d^{max}+{\varphi }_1(X_d^{max}-X_d^{min})$$

(3)

(2) The stage of employed bees: The employed bees generate a new nectar source $V_{ij}$ in the neighborhood of $X_{ij}$ according to Equation (4), and retain the nectar sources with greater fitness through comparison $X_{ij}$ with $V_{ij}$ to optimize the population.

$$V_{ij}=X_{ij}+{\varphi }_2(X_{ij}-X_{kj})$$

(4)

(3) The stage of onlooker bees: The nectar sources generated in the stage of employed bees were selected through Equation (5) to exclude nectar sources with low fitness.

$$P_i=\frac{F_i}{{\sum }_{i=1}^{SN}{F}_i}$$

(5)

(4) The stage of scout: If the nectar source cannot be further promoted, then new nectar sources are searched randomly by Equation (3) to avoid the algorithm falling into a local optimum.

3.2. Q-Learning Algorithm

The Q-learning algorithm has a strong real-time learning ability as an RL algorithm. Its goal is to learn a strategy in interaction with the environment to obtain the maximum long-term reward [41,42]. Figure 6 shows the principle of the Q-learning algorithm, in which the agent and environment are the main components of Q-learning. First, the environment passes its state $s_t$ to the agent. Then, the agent selects the corresponding $a_t$ according to $s_t$, and the environment generates the corresponding reward $r_{t+1}$ according to $a_t$. Then, the agent selects $a_{t+1}$ according to reward $r_{t+1}$ and state $s_{t+1}$ of the environment at the next moment. In this way, Q-learning will eventually choose the most appropriate action according to the state at a certain moment.

Table 6. Initial Q value table.

is the initial Q value table, that is, the corresponding table of states and actions. The initial Q value table is a zero matrix. With further learning, the Q value table is constantly enriched and updated. The update function of Q value is shown in Equation (6), where $\alpha$ represents the learning rate and $\gamma$ represents the discount rate [28].

$$Q(s_t,a_t)\leftarrow (1-\alpha )Q(s_t,a_t)+\alpha (r_{t+1}+\gamma \underset{a}{max}Q(s_{t+1},a_{t+1}))$$

(6)

In the above formula, $Q(s_t,a_t)$ represents the Q value for performing action $a_t$ in state $s_t$, and ${max}_{a}Q(s_{t+1},a_{t+1})$ represents the maximum expected Q value at $s_{t+1}$ when $a_{t+1}$ is performed. In the early stage, the Q-learning algorithm learns and accumulates to enrich the Q value table. In the later stage, the Q-learning algorithm will select the action corresponding to the maximum Q value of the current state in the Q value table to execute, so as to achieve the effect of optimization.

4. Dslabc Algorithm

First, this section introduces the principle of combining ABC and Q-learning algorithms. The combination of the two algorithms can form the SLABC algorithm. Second, it introduces the specific design of the state, action, reward, and action selection strategy in the SLABC algorithm. Finally, the dynamic scheduling problem is studied when a new job is inserted, and the DSLABC algorithm is proposed.

4.1. The Combination of ABC and Q-Learning

The ABC algorithm is a widely used intelligent optimization algorithm. In each iteration, the population’s overall fitness is optimized by updating the m dimension. The m of the traditional ABC algorithm is unchanged or set in advance, which makes it have low convergence accuracy. As a reinforcement learning algorithm, the Q-learning algorithm can select the most suitable update dimension from the Q value table and give it to the ABC algorithm after a certain amount of learning and accumulation, which can solve the shortcomings of the ABC algorithm. Therefore, this paper applies the Q-learning algorithm to run the ABC algorithm. In the early stage, through continuous learning and recording of the dimension m in the iterative process, the optimal value of m suitable for each iteration is found to speed up the convergence and improve the accuracy of the ABC algorithm. Figure 7 is the combination model of the ABC and Q-learning algorithms. The Q-learning algorithm selects an action randomly or from the Q value table according to the state of the ABC algorithm, where the action is m during the iteration of the ABC algorithm. After the ABC algorithm executes the action, a reward will be generated. The positive or negative reward depends on whether the population’s fitness improved. Q-learning updates the Q value table according to the reward and finally forms an optimal correspondence between state and action. The action selection strategy determines whether Q-learning randomly selects an action or uses the Q-value table to select an action. The following paper will make a specific design of the state, action, reward, and action selection strategy.

Figure 8 is a schematic diagram of the SLABC algorithm, a combined algorithm of ABC and Q-learning. The agent of the Q-learning algorithm selects an action $a_t$ (i.e., m) according to the state $s_t$ of the ABC algorithm. Then, the ABC algorithm changes its state by updating the number of m dimensions in the t-th iteration, resulting in a state $s_{t+1}$ at a new moment. Compare state $s_t$ with $s_{t+1}$, generate a positive or negative reward $r_t$, update the Q value table by the reward, and then the agent will choose action $a_{t+1}$ according to $s_{t+1}$. Follow the above steps to finally complete all the iterations of the ABC algorithm.

4.2. The Design of State, Action, Reward, and Action Selection Strategy

The population state S of the ABC algorithm is determined by the fitness and diversity of the population, and the average nectar fitness can better reflect the overall quality of the population compared to the optimal nectar fitness. Therefore, in this paper, the average nectar fitness ($f_{avg}$), optimal nectar fitness ($f_{max}$), and population diversity (d) is used as the factors to calculate the population state S. Then, the three factors are weighted, and the corresponding weighting coefficients are 0.35, 0.3, and 0.35 [42]. Equations (7)–(9) are the calculation methods for the three factors after the first-generation population standardization. Equation (10) is the calculation formula for state S. The Q-learning algorithm needs to select the appropriate action a according to the value of the state S. The dimension of the nectar source is D. If the number m of each iteration is too large, a new nectar source that is very different from the original nectar source will be generated. Therefore, in this paper, m∈ [1, D/10], when the action is $a_1$, m = 1.

The positive or negative reward is determined by the optimal nectar fitness and the average nectar fitness, and the average nectar fitness represents the quality of the entire population. Equations (11) and (12) represent whether the optimal nectar fitness and average nectar fitness of the t-th generation relative to the $t-1$ generation have been improved, represented by $r_{max}$ and $r_{avg}$, respectively. In this paper, the corresponding weighting coefficients are set to 0.45 and 0.55 [42], and the calculation formula of the reward value is shown in Equation (13). A positive or negative indicates whether the current action a (i.e., m) is beneficial to improving population fitness. Equation (14) is the action selection strategy of the SLABC algorithm, introducing the $\epsilon$-greedy strategy, where $x_{0-1}$ is a random number in the range [0,1], and $\epsilon$ is the greedy rate [43]. In the early stage of the ABC algorithm, that is, $\epsilon$<$x_{0-1}$, Q-learning selects actions randomly. When the Q value table has a certain amount of learning and accumulation, that is, $\epsilon$≥$x_{0-1}$, Q-learning selects a maximum action corresponding to the current state. When running the algorithm, the Q value table becomes increasingly abundant, and the corresponding relationship between the state and the action becomes increasingly abundant. In each iteration, the probability that the ABC algorithm obtains a suitable action becomes increasingly larger, which is beneficial to improving the algorithm’s convergence accuracy.

$$f_{avg}=\frac{{\sum }_{i=1}^{SN}f\left(x_i^t\right)}{{\sum }_{i=1}^{SN}f\left(x_i^1\right)}$$

(7)

$$f_{max}=\frac{maxf\left(x_i^t\right)}{maxf\left(x_i^1\right)}$$

(8)

$$d=\frac{{\sum }_{i=1}^{SN}|f\left(x_i^t\right)-\frac{{\sum }_{i=1}^{SN}f\left(x_i^t\right)}{SN}|}{{\sum }_{j=1}^{SN}|f\left(x_j^1\right)-\frac{{\sum }_{j=1}^{SN}f\left(x_j^1\right)}{SN}|}$$

(9)

$$S=0.35f_{avg}+0.3f_{max}+0.35d$$

(10)

$$r_{max}=\frac{maxf\left(x_i^t\right)-maxf\left(x_i^{t-1}\right)}{maxf\left(x_i^{t-1}\right)}$$

(11)

$$r_{avg}=\frac{{\sum }_{i=1}^{SN}f\left(x_i^t\right)-{\sum }_{i=1}^{SN}f\left(x_i^{t-1}\right)}{{\sum }_{i=1}^{SN}f\left(x_i^{t-1}\right)}$$

(12)

$$r=0.45r_{max}+0.55r_{avg}$$

(13)

$$\pi (s,a)=\left\{\begin{array}{cc}\underset{a}{max}Q(s,a)\hfill & \epsilon \ge x_{0-1}\hfill \\ a\left(Randomly\right)\hfill & \epsilon <x_{0-1}\hfill \end{array}\right.$$

(14)

4.3. The Design and Process of the DSLABC Algorithm

If a new job is inserted when the original scheduling scheme is running, it needs to be rescheduled according to method III above to reduce the makespan.

This paper proposes a DSLABC algorithm for dynamic scheduling. Its operating principle is as follows: According to the beginning time of each operation in the original scheduling scheme, the operation whose begin time is less than the insertion time is judged as a finished operation, and the rest are unfinished operations and need to be rescheduled with the new job. First, delete the operation-machine information in the original scheduling scheme, calculate the end time of the finished operation for each machine after the insertion time, reinitialize the operation-machine information according to the unfinished operation, the new job and the end time of each machine, and finally rescheduling according to the SLABC algorithm to obtain a new scheduling scheme. Among them, the principle of rescheduling is also shown in Section 4.1. The DSLABC program is shown in the Algorithm 1, and Figure 9 shows the flow of the DSLABC algorithm.

Algorithm 1. DSLABC.

5. The Experimental Verification of Algorithm Performance

This section presents the experimental verification of the dynamic scheduling problem and DSLABC algorithm. First, the experiment of selecting parameters is designed, and various parameters needed to run the algorithm are determined. Second, the performance and advantages of the SLABC (i.e., DSLABC) algorithm are verified through experimental comparison. Finally, a specific example is used to verify the effectiveness of the DSLABC algorithm in solving the dynamic scheduling problem.

5.1. The Design of the Experiment

All experiments in this paper are carried out under the same experimental conditions and platform. The computer model is an Intel(R) Core(TM) i5-6500 CPU @ 3.20 GHz, Win10, and the software platform is MATLAB 2015b.

This paper determines the parameters required to run the DSLABC algorithm, including the discount rate, iteration times, learning rate, and greed rate. Experiments are carried out to determine the specific values of the parameters, as shown in Figure 10, Figure 11, Figure 12 and Figure 13. In Figure 10, Figure 11 and Figure 12, the variables are the learning rate, discount rate, and greed rate, respectively, while keeping the other variables unchanged. To maintain accurate convergence accuracy and faster convergence speed, the specific values of the parameters are shown in

Table 7. The parameter values of the SLABC algorithm.

Parameter	Specific Value
discount rate ($\gamma$)	0.2
iteration times	1000
learning rate ($\alpha$)	0.75
greed rate ($\epsilon$)	0.85
initial reward	1
number of runs	20
population size	400

. Figure 13 is the convergence diagram of the algorithm for 2000 iterations. The algorithm reaches convergence after approximately 500 iterations. To ensure that the algorithm can reach the final convergence and shorten the algorithm’s running time as much as possible, the number of iterations is selected as 1000 in this paper. In addition, the population size is set to 400, the initial reward is set to 1, and the number of runs is set to 20.

5.2. The Performance Comparison of the SLABC Algorithm

The SLABC algorithm is the core of the DSLABC algorithm, and its performance determines the performance of the DSLABC algorithm. In this paper, the performance of the SLABC algorithm is first verified by experiments. By solving 10 Brandimarte instances [44] with the shortest makespan as the goal, the comparison results of the SLABC algorithm and other algorithms are obtained.

Table 8. The best value (BV) obtained by 11 algorithms to solve 10 Brandimarte instances.

Instance	Scale	HOV	Algorithms
			ABC	ABC-S	ABC-S*Q	MATS-PSO	MA-CROG	HA	TS	GWO	GA	SLGA	SLABC
MK1	10 × 6	36	51	43	43	(40)	42	43	44	(40)	44	42	42
MK2	10 × 6	24	38	31	31	(29)	35	32	34	31	38	31	(29)
MK3	15 × 8	204	269	242	242	210	206	207	213	(204)	225	207	(204)
MK4	15 × 8	48	96	82	82	74	69	76	82	72	88	73	(67)
MK5	15 × 4	168	218	198	201	(175)	183	180	189	177	195	188	(175)
MK6	10 × 15	33	109	98	98	82	87	85	87	84	83	84	(80)
MK7	20 × 5	133	189	156	158	161	173	172	157	157	178	156	(155)
MK8	20 × 10	523	590	548	542	(523)	552	560	530	(523)	542	(523)	(523)
MK9	20 × 10	299	465	418	418	435	441	421	393	(347)	380	371	364
MK10	20 × 15	165	345	323	320	328	377	(283)	296	316	311	295	(283)

shows the optimal solutions (second) obtained by 11 algorithms to solve 10 instances. Except for the SLABC algorithm, this paper verifies the performance of ten algorithms of TS [44], HA [45], GWO [10], GA [46], SLGA [42], MA-CROG [47], MATS-PSO [48], ABC [49], ABC-S and ABC-S*Q [41] under the same experimental conditions The optimal results are marked in brackets, and HOV represents the optimal historical value of 10 calculation instances [10]. The SLABC algorithm obtains 8 optimal solutions in 10 calculation instances, indicating that the SLABC algorithm has excellent convergence performance. By counting the results of running the MK8 instance 20 times with 11 algorithms, the boxplot shown in Figure 14 is obtained. It can be seen intuitively that the optimal value and average value obtained by the SLABC algorithm have great advantages, which further verify the convergence of the algorithm.

Figure 15, Figure 16 and Figure 17 show the scheduling schemes obtained by the SLABC algorithm to solve MK2, MK4 and MK9. The scales of the job-machine are 10 × 6, 15 × 8, and 20 × 10. The different colors in the figures represent different jobs, the vertical axis represents the machine, the horizontal axis represents the processing time, and $Cmax$ represents the shortest makespan achievable through the SLABC algorithm. It can be seen from the figures that the shortest makespan of instances MK2, MK4, and MK9 reach 29, 67, and 364, respectively.

Table 9. The RPD (%) obtained by 11 algorithms to solve 10 instances.

Instance	Algorithms
	ABC	ABC-S	ABC-S*Q	MATS-PSO	MA-CROG	HA	TS	GWO	GA	SLGA	SLABC
MK1	41.67	19.44	19.44	(11.11)	16.67	19.44	22.22	(11.11)	22.22	16.67	16.67
MK2	58.33	29.16	19.44	(20.83)	45.83	33.33	41.67	29.17	58.33	29.17	(20.83)
MK3	31.86	18.63	18.63	2.94	0.98	1.47	4.41	(0)	10.29	1.47	(0)
MK4	100	70.83	70.83	54.17	43.75	58.33	70.83	50	83.33	52.08	(39.58)
MK5	29.76	17.86	19.64	(4.17)	8.93	7.14	12.5	5.36	16.07	11.9	(4.17)
MK6	230.3	197.0	197.0	148.5	163.6	157.6	163.6	154.4	151.5	154.4	(142.4)
MK7	42.11	17.29	18.80	21.05	30.08	29.32	18.05	18.05	33.83	17.29	(16.54)
MK8	12.81	4.78	3.63	(0)	5.54	7.07	1.34	(0)	3.63	(0)	(0)
MK9	55.52	39.80	39.80	45.48	47.49	40.80	31.44	(16.05)	27.09	24.08	21.74
MK10	109.1	95.76	93.94	98.79	128.5	(71.52)	79.39	91.52	88.48	78.79	(71.52)
Average	71.15	51.06	50.12	40.70	49.14	42.60	44.55	37.57	49.48	38.59	(33.35)

shows the relative percentage deviation (RPD) calculated by Equation (15). The value of BV is shown in Table 8, and the optimal RPD is marked in brackets. In the last row of Table 9, the average RPD of 11 algorithms to solve 10 instances are calculated, and the gap between the average RPD of SLABC and the other 10 algorithms is compared, as shown in Figure 18. This shows that SLABC will obtain a better solution than other algorithms when solving the same scheduling problem.

$$RPD=\frac{BV-HOV}{HOV}\times 100\%$$

(15)

5.3. Experiment on Dynamic Scheduling

This paper provides information on instance MK2 according to the coding method.

Table 10. Machines corresponding to each operation of MK2.

		Operation
		${\mathit{O}}_{\mathbf{1}}$	${\mathit{O}}_{\mathbf{2}}$	${\mathit{O}}_{\mathbf{3}}$	${\mathit{O}}_{\mathbf{4}}$	${\mathit{O}}_{\mathbf{5}}$	${\mathit{O}}_{\mathbf{6}}$
Job	$J_1$	1,2,3,4,5,6	3,6	1,2,3,4,5,6	2	5,6	2,3,5
	$J_2$	1,2,4,5,6	5,6	5	2,3	2,3,5	1,2,3,4,5,6
	$J_3$	1,2,3,4,5,6	1,2,3,4,5,6	2,3,5	1,2,3,5,6	1,2,3,4,5,6	1,3,4,5,6
	$J_4$	1,2,3,5,6	1,2,4,5,6	2	1,2,3,4,5,6	1,2,4,5,6	1,2,3,4,5,6
	$J_5$	1,2,3,4,5,6	1,2,4,5,6	4	1,3,4,5,6	1,2,4,5,6	2,3
	$J_6$	1,3,4,5,6	3,6	1,2,3,5,6	1,2,3,4,5,6	2	1,2,4,5,6
	$J_7$	1,2,3,4,5,6	5	1,2,3,4,5,6	1,2,3,4,5,6	1,3,4,5,6	-
	$J_8$	2,3	1,2,3,5,6	1,2,3,4,5,6	1,3,4,5,6	1,2,4,5,6	1,2,4,5,6
	$J_9$	2	3,6	1,2,4,5,6	1,2,4,5,6	2,3	-
	$J_{10}$	4	1,2,3,4,5,6	1,3,4,5,6	1,2,3,4,5,6	5,6	1,2,4,5,6

shows the machines corresponding to each operation in MK2,

Table 11. The corresponding time of the machine in Table 10.

		Operation
		${\mathit{O}}_{\mathbf{1}}$	${\mathit{O}}_{\mathbf{2}}$	${\mathit{O}}_{\mathbf{3}}$	${\mathit{O}}_{\mathbf{4}}$	${\mathit{O}}_{\mathbf{5}}$	${\mathit{O}}_{\mathbf{6}}$
Job	$J_1$	3,2,3,5,3,6	4,5	1,6,3,3,6,5	6	6,3	1,2,4
	$J_2$	3,4,2,6,1	3,6	2	4,3	1,2,4	3,2,3,5,3,6
	$J_3$	1,6,3,3,6,5	2,4,6,6,3,6	1,2,4	4,3,5,2,3	5,4,3,1,5,3	4,6,6,3,3
	$J_4$	4,3,5,2,3	3,4,2,6,1	6	1,6,3,3,6,5	4,3,5,4,3	5,4,3,1,5,3
	$J_5$	2,4,6,6,3,6	4,3,5,4,3	3	4,6,6,3,3	3,4,2,6,1	4,3
	$J_6$	4,6,6,3,3	4,5	4,3,5,2,3	2,4,6,6,3,6	6	3,4,2,6,1
	$J_7$	5,4,3,1,5,3	2	2,4,6,6,3,6	3,2,3,5,3,6	4,6,6,3,3	-
	$J_8$	4,3	4,3,5,2,3	2,4,6,6,3,6	4,6,6,3,3	4,3,5,4,3	3,4,2,6,1
	$J_9$	6	4,5	3,4,2,6,1	4,3,5,4,3	4,3	-
	$J_{10}$	3	2,4,6,6,3,6	4,6,6,3,3	5,4,3,1,5,3	6,3	3,4,2,6,1

shows the corresponding time of the machines in Table 10, and there is a one-to-one correspondence between Table 10 and Table 11. For example, operation $O_1$ of job $J_1$ can be processed on all six machines. The processing time on machine $M_1$ is “3”.

Table 12. Information about the newly inserted job.

New Job	Operation	${\mathit{M}}_1$	${\mathit{M}}_2$	${\mathit{M}}_3$	${\mathit{M}}_4$	${\mathit{M}}_5$	${\mathit{M}}_6$
Job 11	$O_{11-1}$	2	-	5	3	3	-
	$O_{11-2}$	-	3	-	3	5	4
	$O_{11-3}$	-	6	3	5	-	-
	$O_{11-4}$	4	1	-	2	5	3
	$O_{11-5}$	-	5	2	4	-	3
	$O_{11-6}$	5	6	3	2	-	4

is the newly inserted job designed in this paper, and Figure 19 is the scheduling scheme in the case of no emergent job. Figure 20 represents the new scheduling scheme obtained by adding the emergent Job 11 when the time is “12” and rescheduling the new job with the unfinished operation in the original scheduling scheme. White represents the newly inserted job. Figure 19 and Figure 20 show that the DSLABC algorithm can deal with the job-shop scheduling problem in emergencies, proving the DSLABC algorithm’s practicability.

6. Conclusions

Given the frequent insertion of new jobs in flexible job-shops, this paper determines the specific method of job-shop rescheduling and verifies it by experiments. First, a dynamic scheduling model is proposed. By comparing the insertion time and the beginning time of each operation in the original scheduling scheme, the operations that need to be rescheduled are screened out, and the coding method between the operations and the machines is designed. Second, the DSLABC algorithm for dynamic scheduling is proposed, which realizes the combination of the ABC and the Q-learning algorithms. The specific equations for the state, reward, and action selection strategy are designed. Through the Q value table, the algorithm selects appropriate action according to the specific state of the population during each iteration, which improves the convergence accuracy and stability of the algorithm. Finally, by comparing 11 algorithms to solve the Brandimarte instances, it is verified that the SLABC algorithm (i.e., DSLABC) has good convergence accuracy and stability. Through the dynamic scheduling of specific examples, the effectiveness of the DSLABC algorithm in solving dynamic problems is verified, and a new method with high precision is provided for the current dynamic scheduling of flexible job-shops.

The novelties of this paper are: 1. Since the update dimension of the ABC algorithm cannot be adjusted dynamically, this paper realizes that the ABC algorithm can obtain a suitable update dimension in each iteration through the Q-learning algorithm, which improves the convergence accuracy of the ABC algorithm. 2. In view of the situation that the real flexible job-shop faces the insertion of new jobs, this paper reschedules the new job and the operations that have not started processing to generate a new scheduling scheme. The proposed DSLABC algorithm realizes the dynamic scheduling of flexible job-shop, and the effectiveness and advantages of the DSLABC algorithm are proved by experiments.

The next research plan is as follows: 1. Research on dynamic scheduling with multi jobs and multiperiod insertion to further improve the application scope and field of the DSLABC algorithm. 2. Research on scheduling problems with multiple objective, such as priority and energy consumption. 3. Research on algorithm verification on platforms such as Python. 4. Apply the algorithm to the actual flexible job-shop to verify its actual application performance.