A novel stacked generalization ensemble-based hybrid LGBM-XGB-MLP model for Short-Term Load Forecasting

Highlights

• A novel Stacked XGB-LGBM-MLP model is proposed to improve the overall regression performance.

• A novel STLF technique is explored and developed in this study.

• A comparative analysis of five hyperparameter optimization algorithms was comprehensively presented for STLF.

• An assessment of the proposed technique is conducted using two real datasets.

• A comparative study with the recent benchmark techniques is performed.

Abstract

This paper proposes an effective computing framework for Short-Term Load Forecasting (STLF). The proposed technique copes with the stochastic variations of the load demand using a stacked generalization approach. This approach combines three models, namely, Light Gradient Boosting Machine (LGBM), eXtreme Gradient Boosting machine (XGB), and Multi-Layer Perceptron (MLP). The inner mechanism of Stacked XGB-LGBM-MLP model consists of generating a meta-data from XGB and LGBM models to compute the final predictions using MLP network. The performance of the proposed Stacked XGB-LGBM-MLP model is validated using two datasets from different locations: Malaysia and New England. The main contributions of this paper are: 1) A novel stacking ensemble-based algorithm is proposed; 2) An effective STLF technique is introduced; 3) A critical multi-study analysis for hyperparameter optimization with five techniques is comprehensively performed; 4) A performance comparative study using two datasets and reference models is conducted. Several case studies have been carried out to prove the performance superiority of the proposed model compared to both existing benchmark techniques and hybrid models.

Introduction

With the recent waves of digitization and the fast pervasiveness of information and communication technologies, many initiatives have placed an added emphasis on the development and modernization of the traditional centric power grids [1]. The balance between electricity generation and load demand must be optimally maintained to avoid fatal disturbances on the grid due to overloads [2]. To achieve that aim, electric Load Forecasting (LF) offers the necessary tools for stakeholders and energy suppliers to increase their profitability from Renewable Energy Sources (RES) and meet the ever-growing electricity demand. The high complexity of the utility grid operations paved the way for LF to monitor, control, and manage the electric system operations with high efficiency [3].

Recently, LF reached an overall state of maturity that guarantees its safe applicability and profitability in smart grids and traditional utility grids with satisfactory results. LF reveals a cost-effective, efficient, and reliable technique within the energy management framework [4]. Electrical LF assists the scheduling of load response and maintains the fast and economic dispatch to its optimal. Furthermore, it provides a reliable indicator for managing the complex pricing strategies in liberalized and deregulated energy markets with higher financial benefits [5]. Efficient energy management systems strongly require intelligent algorithms to effectively support the electric operations strategy, decrease the electricity bills, and enhance the energy trading and planning [5]. LF is conducted using a variety of features’ inputs including social, economic, and weather conditions [3]. Based on the application type, the LF is divided into two categories: 1) the time horizon 2) scope of the variables employed [3].

For the time horizon, most of the LF employed techniques can be divided into four major classes; long-term LF valid for years, medium-term LF valid from months to years, Short-Term Load Forecasting (STLF) used from minutes up to one week, and very short-term LF valid from seconds to minutes [6]. The STLF ensures high asset commitment and flexibility of the grid to enhance the serviceability of electricity operations over several time scales for day-to-day operations [7]. In Ref. [3], it was concluded that the most useful LF horizon is STLF.

STLF models are classified into three categories: soft computing techniques, conventional forecasting techniques, and modified traditional techniques [8,9]. Traditional techniques involve regression methods, Iterative reweighted Least Squares, and Exponential Smoothing [10]. However, there are three major drawbacks associated with these methods such as overfitting problems with a massive amount of data, difficulties in feature engineering processing, and relatively less accuracy compared to advanced techniques. Modified techniques include adaptative demand forecasting, stochastic times series, auto-regressive, and moving average-based models, support vector machine-based techniques [3]. Soft computing techniques essentially consist of genetic algorithms, fuzzy logic, neural networks, and knowledge-based expert systems [11]. However, there are some major drawbacks associated with these methods including loss of model interpretability, higher execution time and computational burden, and limited generalization capabilities. Moreover, Hyperparameter Optimization (HO) for soft techniques is a very computationally expensive and time-consuming task [12].

Various algorithms were proposed to solve the optimization problems and enhance the ML model performance. In Refs. [13], the authors proposed an Exchange Market-Genetic Algorithm (EMGA) technique to solve optimization problems with less iterations and better-quality results. The proposed technique combines the merits of the genetic algorithm, and exchange market algorithm [14]. The execution time of the EMGA algorithm took 2.82 min for 641 iterations in solving twelve benchmark functions. Despite the fast execution time and low error rate, the simulation results show that EMGA exhibits a high time iteration ratio. Authors in Ref. [15] used a Simulated Annealing (SA) algorithm for HO of Deep Neural Network (DNN). The proposed SA-DNN achieves accurate results in terms of RMSE. However, the search space of the SA-DNN HO is very limited to avoid computational burden (it only includes the neuron numbers). This leads to low accuracy improvement compared to DNN. In Ref. [16], the authors used the spearmint Bayesian Optimization (BO) method for the HO of recurrent neural networks. The proposed technique is hyper-effective for both short/long-term forecasting. The authors in Ref. [17] used a Derivative-Free Optimization (DFO) technique with deep learning models. The proposed technique uses an efficient feature selection via ensemble structures to predict a variety of RES. However, the comparative analysis of DFO with other HO benchmark techniques is missing.

Ensemble methods have been widely deployed for forecasting applications due to their ease of implementation. In Refs. [18], the authors employed the extreme Gradient Boosting technique (XGB) to predict the load based on similar days using cluster analysis. The presented results confirmed the superiority of the ensemble methods in terms of high accuracy and generalization capabilities compared to the deep learning techniques such as Long Short-Term Memory (LSTM). Nevertheless, the results are unsatisfactory with a relatively poor Mean Square Error (MSE). In Ref. [19], a combination between Convolutional Neural Network (CNN) and Light Gradient Boosting Machine (LGBM) was proposed. The feature extraction process was carried out using CNN model from five wind turbines. The reliability of the proposed technique was verified according to the low registered error metrics values. However, this technique shows sensitivity to time-order character size and requires high computing resources.

In this paper, a Stacked Generalization approach between XGB, LGBM, and Multi-Layer Perceptron (MLP) models named Stacked XGB-LGBM-MLP, is firstly explored for STLF. To the best of the authors’ knowledge, no prior work has addressed this architecture for STLF. The Stacked XGB-LGBM-MLP model is characterized by high accuracy, excellent performance, and ease of implementation. Moreover, the simulation results with an open data portal demonstrated that the Stacked XGB-LGBM-MLP model manages to outperform 11 benchmarks for STLF application. The main contributions of this paper are given as follows:

• A novel Stacked XGB-LGBM-MLP model is proposed to improve the overall regression performance. Despite the potential learning ability and rigorous mathematical theory of XGB and LGBM models, they can only use tree models with the same category, and it is difficult to fundamentally overcome the inherent defects of the tree models. Using MLP model, the meta-data enhances its training performance to generate a better-quality result.

• A novel STLF technique is explored and developed in this study. Most of the existing techniques use ensemble models or neural networks for STLF. However, mixing both ensembles and neural networks in one single framework has not received enough attention in the previous studies.

• A comparative analysis of five HO algorithms was comprehensively presented for STLF. Previous studies focused mainly on using various HO techniques to enhance the performance of the ML models. However, selecting the most appropriate HO technique received little attention in the field of ML.

• An assessment of the proposed technique is conducted using two real datasets. The sensibility of the proposed technique to the size and nature of the data has been given significant importance in this research study.

• A comparative study with the recent benchmark techniques is performed. A large comparative study with 11 benchmarks has been conducted to demonstrate the high performance of the proposed stacked XGB-LGBM-MLP model.

Therefore, the paper is organized as follows: Section 2 presents the preliminaries of the proposed technique. In Section 3, two case studies have been conducted. Several existing Machine Learning (ML) models are compared to the proposed technique. Furthermore, a comparative study with the recent benchmark technique regarding the same dataset has been discussed. Finally, section 3 draws conclusions to end this paper.