Elsevier

Neurocomputing

Volume 70, Issues 13–15, August 2007, Pages 2371-2378
Neurocomputing

Time series prediction with ensemble models applied to the CATS benchmark

https://doi.org/10.1016/j.neucom.2005.12.136Get rights and content

Abstract

We describe the use of ensemble methods to build models for time series prediction. Our approach extends the classical ensemble methods for neural networks by using several different model architectures. We further suggest an iterated prediction procedure to select the final ensemble members.

Introduction

Ensemble building is a common way to improve the performance of the resulting model for classification and regression tasks, since it was noticed that an ensemble of individual predictor performs better than a single predictor in the average [6], [10]. Usually an ensemble consists of models taken from one single class, e.g. neural networks [7], [17], [11], [16], support vector machines [24] or regression trees [2]. We suggest a different strategy. We train several models from different model classes and combine them to build the final ensemble. This is done in order to introduce model diversity which is the central feature of the ensemble approach [11]. The novelty of our approach consists of building heterogeneous ensembles with several model classes combined with an iterated prediction scheme for final model selection. For the CATS Benchmark [13] we propose a combined model strategy in order to cope with the different timescales of the data set. In Section 3 we present our investigation that leads to the assumption, that the time series has two different timescales. We noticed that other research groups had the same idea while dealing with the CATS data set [19], [3].

Section snippets

Method of prediction

We build a simple polynomial model to cover the long term oscillations and combine that with an ensemble model for the small scale dynamics. In the following section we like to introduce the ensemble approach that we used to build the small scale model.

Results obtained on the initial values of the benchmark

The data set of the CATS Benchmark was provided by Lendasse et al. [13] and consists of an artificial time series with 5000 data points, wherein 100 values are missing. These missing values are divided in five blocks of 20 points that have to be predicted. Visual inspection of the time series shows an oscillatory regime on the large scale that seems to be corrupted with a kind of noisy random process on the small scale (see Fig. 1). A further investigation leads to the conclusion that this

Analysis of the results: advantages and disadvantages of the current method

If we take a look at the MSE in the five missing blocks, we realize that our method shows good results in the first four blocks (see Fig. 5) but it fails in predicting the last 20 points in the fifth block (see Table 2).This is an effect of the missing polynomial fit which we did not perform for the last block. In general polynomials have good interpolation properties but it can be difficult to use them for extrapolation.

In our approach we use them to support the iterated prediction scheme that

Improved method for prediction

The discussion in the previous section has shown, that the iterated prediction works well with a polynomial fit as backbone. We like to improve our method by adding a polynomial fit to the last of the five missing blocks. In order to find a decent extrapolation of the last 20 points, we run a modified version of the brute force search of the parameter space as described in Section 3.1 that can cope with the open end of the time series. We find a minimum of the MSE for a window length of 50

Conclusion

We have demonstrated that ensembles models together with an iterated prediction procedure for model selection provides a powerful tool for time series prediction. The performance on CATS Competition shows that this method can compete against other “state of the art” prediction schemes. For the CATS Benchmark we used an advanced model building procedure that combines the robust polynomial fit of the large scale dynamics with an iterated ensemble prediction of the small scale dynamics. The major

Acknowledgments

The authors like to thank T. Roska and the members of the Analogic and Neural Computing Laboratory in Budapest for their hospitality. The work was done as part of the Research Training Network COSYC of SENS No. HPRN-CT-2000-00158 within the 5th EU Framework Program of the European Community. This research has been also supported by AGH-University of Science and Technology Grant 11.11.120.182.

Jörg D. Wichard received the Diploma degree in 1997 and the Ph.D. degree in Physics in 2000 both from the University of Göttingen, Germany.

Currently he has a PostDoc position at the Research Institute of Molecular Pharmacology and at the Schering AG in Berlin, Germany.

His research interest include nonlinear dynamics, time series analysis, neural networks, machine learning, computational drug design and cheminformatics.

He was PostDoc at the AGH University of Science and Technology in Kraków,

References (25)

  • T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning, Springer Series in Statistics, Springer,...
  • C. Igel et al.

    Improving the Rprop learning algorithm

  • Cited by (23)

    • A self-organizing deep belief network based on information relevance strategy

      2020, Neurocomputing
      Citation Excerpt :

      As can be seen in Fig. 12, the predicting results of modules 1–4 are better than those of module 5. A comparison of the proposed method and the other methods - the Kalman Smoother [46], Ensemble Models [47], DBN, continuous DBN (CDBN), PSO-DBN [21], PSO-based MLP [21], the multilayer perceptron (MLP) [21], a hierarchical Bayesian learning scheme [21] and ARIMA [48], is shown in Table 5. Form the results in Table 5, E1 and E2 of S-DBN are less than other methods (except the Kalman Smoother [46] and Ensemble Models [47]).

    • An intelligent, uncertainty driven management scheme for software updates in pervasive IoT applications

      2018, Future Generation Computer Systems
      Citation Excerpt :

      Ensemble forecasting is a common way to improve the performance of an estimation compared to single models. An ensemble of individual estimators performs better, in average, than a single estimator [58]. An ensemble scheme tries to deal with errors arising from the uncertainty associated with the dynamics of the environment (where observations are originated in) as sources of estimation uncertainty.

    • Forecasting the NN5 time series with hybrid models

      2011, International Journal of Forecasting
    View all citing articles on Scopus

    Jörg D. Wichard received the Diploma degree in 1997 and the Ph.D. degree in Physics in 2000 both from the University of Göttingen, Germany.

    Currently he has a PostDoc position at the Research Institute of Molecular Pharmacology and at the Schering AG in Berlin, Germany.

    His research interest include nonlinear dynamics, time series analysis, neural networks, machine learning, computational drug design and cheminformatics.

    He was PostDoc at the AGH University of Science and Technology in Kraków, Poland and he was a guest scientist at the MTA Sztaki (Computer and Automation Research Institute of the Hungarian Academy of Sciences) in Budapest, Hungary.

    Maciej Ogorzałek received the M.Sc. degree in 1979, Ph.D. degree in 1987 and Habilitation degree in 1992, all from the AGH University of Science and Technology, Kraków, Poland.

    Currently he is employed as full Professor of Electrical Engineering at the Department of Electrical Engineering, AGH University of Science and Technology, Kraków, Poland and holds a joint appointment as Head of the Department of Information Technologies at the Jagiellonian University in Kraków.

    His research and teaching interests include circuit theory with an emphasis on Nonlinear and dynamic circuits, complex phenomena and chaos, neural networks, nonlinear signal analysis and processing, nonlinear methods for mixed signal circuit design, biomedical signal analysis and modeling.

    He held several visiting positions: Swiss Federal Institute of Technology Lausanne, The Technical University of Denmark, Artificial Brain Systems Laboratory, Institute of Physical and Chemical Research (RIKEN), Wako, Japan; Electronics Research Laboratory, University of California, Berkeley. Centro Nacional de Microelectronica, Sevilla, Spain, Kyoto University, Japan (Senior JSPS Award) and Goethe University Frankfurt-am-Main, Germany.

    He is a Member of The Association of Polish Electrical Engineers, Polish Society of Theoretical and Applied Electrical Sciences, Member of the Committee on Electrical Engineering, Computer Science and Automatic Control of the Polish Academy of Sciences, Krakow Section, Member of the Section of Electronic Signal and Systems, Committee on Electronics and Telecommunication of the Polish Academy of Sciences. He is currently the Vice-President of the Executive Board of Sniadecki Science Foundation.

    Awards: Recipient of the Guillemin-Cauer (Best Paper) Award 2002. Distinguished Lecturer of the CAS Society 2001–2003. Premio Ano Sabatico from the Spanish Ministry of Education and Sport; Senior Award—Japan Society for Promotion of Science; Award of the Minister of National Education of Poland; National Educations Medal—Poland; Silver Cross of Merit—Poland.

    Editorial activities: Associate Editor Int. J. Circuit Theory and Applications (1999- ) Associate Editor—Journal of The Franklin Institute (1997- ), Secretary of the Editorial Board for the Quarterly of Electrical Engineering and Electronics (Poland) (1993- ), Member of the Editorial Board of Automatics (in Polish Automatyka). Member of the Editorial board of the International Journal of Bifurcation and Chaos.

    He was the Vice-chairman of the IEEE CAS Chapter Poland, recipient of the Chapter of the Year Award 1995, Chairman of the Technical Committee of Nonlinear Circuits and Systems of CAS Society 1997/1998, Chairman of the Organizing Committee 1994 Workshop on Nonlinear Dynamics of Electronic Systems, member of technical committees of several IEEE sponsored conferences, Special Sessions chairman for ISCAS’2000. General Chairman European Conference on Circuit Theory and Design 2003. Recipient of the IEEE-CAS Golden Jubilee Award. He has been elected CAS Society Vice-President for Region 8 for 2002–2004 and CAS Society Administrative Vice-president since 2004. Region 8 Chapter Coordinator for CAS, CS and IM Societies. Associate Editor for IEEE Transactions on Circuits and Systems Part I 1993–1995 (EIC Martin Hasler) and 1999–2001 (EIC—M.N.S. Swamy), Since 2004 member of the Editorial Board of the Proceedings of the IEEE. Since 2004 he is the Editor-in-Chief of the Circuits and Systems Magazine.

    View full text