A comparison of methods to avoid overfitting in neural networks training in the case of catchment runoff modelling

Summary

Artificial neural networks (ANNs) becomes very popular tool in hydrology, especially in rainfall–runoff modelling. However, a number of issues should be addressed to apply this technique to a particular problem in an efficient way, including selection of network type, its architecture, proper optimization algorithm and a method to deal with overfitting of the data. The present paper addresses the last, rarely considered issue, namely comparison of methods to prevent multi-layer perceptron neural networks from overfitting of the training data in the case of daily catchment runoff modelling. Among a number of methods to avoid overfitting the early stopping, the noise injection and the weight decay have been known for about two decades, however only the first one is frequently applied in practice. Recently a new methodology called optimized approximation algorithm has been proposed in the literature.

Overfitting of the training data leads to deterioration of generalization properties of the model and results in its untrustworthy performance when applied to novel measurements. Hence the purpose of the methods to avoid overfitting is somehow contradictory to the goal of optimization algorithms, which aims at finding the best possible solution in parameter space according to pre-defined objective function and available data. Moreover, different optimization algorithms may perform better for simpler or larger ANN architectures. This suggest the importance of proper coupling of different optimization algorithms, ANN architectures and methods to avoid overfitting of real-world data – an issue that is also studied in details in the present paper.

The study is performed for Annapolis River catchment, characterized by significant seasonal changes in runoff, rapid floods during winter and spring, moderately dry summers, severe winters with snowfall, snow melting, frequent freeze and thaw, and presence of river ice. The present paper shows that the elaborated noise injection method may prevent overfitting slightly better than the most popular early stopping approach. However, the implementation of noise injection to real-world problems is difficult and the final model performance depends significantly on a number of very technical details, what somehow limits its practical applicability. It is shown that optimized approximation algorithm does not improve the results obtained by older methods, possibly due to over-simplified criterion of stopping the algorithm. Extensive calculations reveal that Evolutionary Computation-based algorithm performs better for simpler ANN architectures, whereas classical gradient-based Levenberg–Marquardt algorithm is able to benefit from additional input variables, representing precipitation and snow cover from one more previous day, and from more complicated ANN architectures. This confirms that the curse of dimensionality has severe impact on the performance of Evolutionary Computing methods.

Introduction

During the last 20 years artificial neural networks (ANNs) (Haykin, 1999) have become very popular in various scientific disciplines (Paliwar and Kumar, 2009, Wen et al., 2009, Al-Garni, 2010). Within the field of hydrology different Artificial Intelligence methods, including ANNs, also gained much popularity (Maier and Dandy, 2000, Cheng et al., 2002, Cheng et al., 2008, Muttil and Chau, 2006, Lin et al., 2006, Piotrowski et al., 2007, Maier et al., 2010, Acharya et al., 2012, Huo et al., 2012, Nourani et al., 2012). ANN applications to rainfall–runoff modelling are plentiful and include: ASCE Task Committee, 2000, Solomatine, 2003, Cherkassy et al. (2006), Dawson et al., 2006, Piotrowski et al., 2006, Solomatine and Ostfeld, 2008, Wu et al., 2009, Siou et al., 2011 and Wu and Chau (2011). Among different ANN types, multi-layer perceptron neural networks (MLPs) are especially popular due to their simplicity, relatively low number of parameters, clear biological inspirations and a debate whether they may be considered as universal approximators or not (Hecht-Nielsen, 1987, Girosi and Poggio, 1989, Nakamura et al., 1993, Braun and Griebel, 2009). MLPs are of special interest also in the present paper.

Recently Wang et al. (2009) and Elshorbagy et al. (2010) showed a comparison of various Artificial Intelligence techniques for rainfall–runoff forecasting, encouraging the search for novel methods to improve ANN training and selecting their different features. Apart from choosing the neural network type, the successful application of a neural networks to a particular problem requires the determination of a model architecture (which defines number of parameters), an optimization algorithm and a method to avoid overfitting. However, in practice ANNs are frequently used at hand without discussing such details, what may have significant impact on model performance.

The present paper is a continuation of Piotrowski and Napiorkowski (2011) study, which aimed at choosing the best optimization method for MLPs training applied for daily catchment runoff forecasting in colder climate zones. The main objective of the current paper is the comparison of different methods to avoid overfitting when MLPs are applied to similar task. We put special attention to noise injection approach, which is rarely considered in hydrological applications. Also the performance of a new method called optimized approximation algorithm and of early stopping, the most popular approach to deal with overfitting, are studied in details. However, the methods to avoid overfitting cannot be compared or discussed apart of ANN architecture and training algorithms. For instance some optimization algorithms perform poorly in uncertain environments (Jin and Branke, 2005) of which neural networks with noise injection method to avoid overfitting may be an example. This emphasizes the importance of proper coupling methods to avoid overfitting with training algorithms. On the other hand some optimization methods may quickly converge to good solutions for simple ANN architecture with small number of parameters, but perform poorly for more complicated ones with more parameters. One may note that different ANN architectures means different number of parameters and different fitness landscapes, hence formally different problems. It is well known that the performance of optimization algorithms depends on the problem. It was verified empirically (e.g. Epitropakis et al., 2011); it was also proved (Wolpert and Macready, 1997, Wolpert and Macready, 2005) that under certain assumptions the performance of any two algorithms averaged over all possible problems (fitness landscapes) is equal. Of course, in practice few people may be interested in all problems, but the proof presented in Wolpert and Macready (1997) results in important warning: an optimization algorithm applied to a novel task can fail even if it was successful in solving some other problems. In the present paper at first based on the previous findings in the literature two training methods are chosen, one gradient-based and one based on Evolutionary Computation (EC). Then the optimal set of input variables and MLP architecture for each training algorithm is found experimentally. Finally three different methods to avoid ANN overfitting of data are compared for chosen ANN architectures and training algorithms. Below we very briefly introduce the main features connected with MLP training algorithms, architecture and methods to avoid overfitting.

A number of studies addressed the problem of application of different optimization algorithms to the ANN training for various regression problems. The most popular optimization approaches are gradient-based methods and among them the Levenberg–Marquardt (LM) algorithm (Press et al., 2006) is considered as one of the most efficient (see e.g. Adamowski and Karapataki, 2010). In a few studies EC methods were applied to the same problems – with various opinions on their performance. In some papers the advantage of the EC algorithms over the gradient-based methods applied to different ANN training was suggested (Sexton and Gupta, 2000, Jain and Srinivasulu, 2004, Martinez-Estudillo et al., 2006, Chau, 2006, Zhang et al., 2007, Huang et al., 2009), but a number of other studies showed that EC approaches are at least not better than the gradient-based algorithms in terms of ANN model performance and are of course much slower (Mandischer, 2002; Ilonen et al., 2003; Socha and Blum, 2007). Motivated by this diversity of opinions, authors of the present study conducted a detailed survey of recently developed EC methods from two “families” – Differential Evolution (DE) (Storn and Price, 1995) and Particle Swarm Optimization (Kennedy and Eberhart, 1995). Eight EC algorithms were compared with LM method for MLP training with early stopping approach for rainfall–runoff modelling at Annapolis River, Nova Scotia, Canada (Piotrowski and Napiorkowski, 2011). The results are generally in agreement with Mandischer (2002), Ilonen et al. (2003) and Socha and Blum (2007) opinions. Only one of the EC algorithms, namely the Differential Evolution with Global and Local neighbors (DEGL, Das et al., 2009) showed similar performance to LM method, with much slower convergence. It is worth noting that DEGL was also found to be among the best EC-based algorithms, along with Grouped Differential Evolution (GDE) (Piotrowski and Napiorkowski, 2010) and Self-Adaptive Differential Evolution (SADE) (Qin et al., 2009) in training MLP applied to estimation of longitudinal dispersion coefficient in rivers (Piotrowski et al., 2012b). In that study number of data was very small (under 100 observations) imposing the use of very simple MLP architecture, objective function was non-differentiable and a kind of noise injection method to avoid overfitting was used. This suggests that DEGL may be well suited for MLP training in general. Based on above findings, in the present paper LM and DEGL methods are used as training algorithms.

The architecture of MLP defines the number of parameters to be optimized. This architecture should always be adopted to the problem (Zhang et al., 1998, Mahmoud and Ben-Nahki, 2003, Siou et al., 2011, De et al., 2011), as it depends on the number of input and output variables, the number and the quality of available data, the presence of noise in the data, etc. Over-parameterization may have significant negative impact on the performance of neural networks, also in the case of rainfall–runoff modelling (Gaume and Gosset, 2003). The smaller architecture usually allows to obtain better generalization properties (Haykin, 1999) and is easier to train, especially by means of the non-gradient-based methods. Some Evolutionary Computation algorithms were proposed to determine an optimal architecture of ANN (Castillo et al., 2000, Huang et al., 2009) and were applied to hydrological problems (Chen and Chang, 2009), however they are applicable rather to problems when neither expert-knowledge is available nor physically-based choice of input variables and model complexity is possible. Although a number of other methods how to develop ANN architecture exists (Sietsma and Dow, 1991; Wang et al., 1994, Islam et al., 2009, Ssegane et al., 2012, Nourani and Sayyah Frad, 2012), they usually rely on heuristic or subjective decisions and none is widely applied (Zhang et al., 1998).

Impact of different methods used to avoid overfitting on ANN performance, which is the main focus of the present paper, was rarely studied in the literature and such papers usually dealt with classification problems (Holmstrom and Koistinen, 1992, Hua et al., 2006, Zur et al., 2009) or used artificial functions for comparison (Holmstrom and Koistinen, 1992, Reed et al., 1995). Only Giustolisi and Laucelli (2005) studied the impact of a number of methods to avoid ANN overfitting for hydrological data, namely the case of rainfall–runoff modelling for two very small catchments (up to 5 km²) in Italy. However, EC-based optimization methods were not used and the popular noise injection method based on maximization of the cross-validated likelihood (Holmstrom and Koistinen, 1992) was not compared. The early stopping technique led to poor results in Giustolisi and Laucelli (2005) what may be surprising as this is very popular and usually successful approach to avoid overfitting. Moreover, recently a novel methodology called optimized approximation algorithm (Liu et al., 2008) was proposed and gained much interest. The present paper tries to fill the gaps left by Giustolisi and Laucelli, 2005, Hua et al., 2006 and Zur et al. (2009) and presents the comparison of the catchment runoff modelling results obtained when the mentioned three techniques designed to avoid overfitting are coupled with MLPs of different architectures and gradient-based or EC-based optimization algorithms. Neural networks are applied to Annapolis River runoff forecasting, which is located in moderate climate zone.