Abstract
Bagging, boosting, rotation forest and random subspace methods are well known re-sampling ensemble methods that generate and combine a diversity of learners using the same learning algorithm for the base-classifiers. Boosting and rotation forest algorithms are considered stronger than bagging and random subspace methods on noise-free data. However, there are strong empirical indications that bagging and random subspace methods are much more robust than boosting and rotation forest in noisy settings. For this reason, in this work we built an ensemble of bagging, boosting, rotation forest and random subspace methods ensembles with 6 sub-classifiers in each one and then a voting methodology is used for the final prediction. We performed a comparison with simple bagging, boosting, rotation forest and random subspace methods ensembles with 25 sub-classifiers, as well as other well known combining methods, on standard benchmark datasets and the proposed technique had better accuracy in most cases.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bauer E, Kohavi R (1999) An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach Learn 36: 105–139
Bosch A, Daelemans W (1999) Memory-based morphological analysis. In: Proceedings of 37th annual meeting of the ACL. University of Maryland, pp 285–292 (http://ilk.kub.nl/~antalb/ltuia/week10.html)
Breiman L (1996) Bagging predictors. Mach Learn 24(3): 123–140
Breiman L (2001) Random forests. Mach Learn 45(1): 5–32
Bryll R, Gutierrez-Osuna R, Quek F (2003) Attribute bagging: improving accuracy of classifier ensemble by using random feature subsets. Pattern Recognit 36: 1291–1302
Bühlman P, Yu B (2002) Analyzing bagging. Ann Stat 30: 927–961
Buja WS (2006) Observations on bagging. Statistica Sinica 16: 323–351
Cai Q-T, Peng C-Y, Zhang C-S (2008) A weighted subspace approach for improving bagging performance. In: IEEE ICASSP, pp 3341–3344
Dietterich T (2000) An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Mach Learn 40: 139–157
Dietterich TG (2001) Ensemble methods in machine learning. In: Kittler J, Roli F (eds) Multiple classifier systems. LNCS, vol 1857, pp 1–15
Domingos P, Pazzani M (1997) On the optimality of the simple Bayesian classifier under zero-one loss. Mach Learn 29: 103–130
Frank E, Witten IH (1998) Generating Accurate Rule Sets Without Global Optimization. In: Fifteenth international conference on machine learning, pp 144–151
Frank A, Asuncion A (2010) UCI machine learning repository [http://www.archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science
Freidman JH, Hall P (2007) On bagging and nonlinear estimation. J Stat Plan Inference 137(3): 669–683
Freidman J, Hastie T, Tibshirani R (2000) Additive logistic regression: a statistical view of boosting. Ann Stat 28: 307–337
Freund Y, Robert E Schapire (1996) Experiments with a new boosting algorithm. In: Proceedings of ICML’96, pp 148–156
Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1): 119–139
Fumera G, Roli F, Serrau A (2005) Dynamics of variance reduction in bagging and other techniques based on randomisation. MCS 2005, LNCS vol 3541, pp 316–325
Fumera G, Roli F, Serrau A (2008) A theoretical analysis of bagging as a linear combination of classifiers. IEEE Trans Pattern Anal Mach Intell 30(7):1293–1299
Garcıa-Pedrajas N, Ortiz-Boyer D (2008) Boosting random subspace method. Neural Netw 21: 1344–1362
Grandvalet Y (2004) Bagging equalizes influence. Mach Learn 55: 251–270
Ho TK (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8): 832–844
Kotsiantis S, Pierrakeas C, Pintelas P (2003) Preventing student dropout in distance learning systems using machine learning techniques. In: Proceedings of 7th international conference on knowledge-based intelligent information and engineering systems (KES), Oxford, Sept. 3–5, Lecture notes series, vol 2774. Springer, pp 267–274
Kuncheva LI, Skurichina M, Duin RPW (2002) An experimental study on diversity for bagging and boosting with linear classifiers. Inf Fusion 3: 245–258
Latinne P, Debeir O, Decaestecker Ch (2000) Mixing bagging and multiple feature subsets to improve classification accuracy of decision tree combination. In: Proceedings of the 10th Belgian-Dutch conference on machine learning, Tilburg University
Melville P, Mooney R (2003) Constructing diverse classifier ensembles using artificial training examples. In: Proceedings of IJCAI-2003, pp 505–510, Acapulco, Mexico, August 2003
Murthy RPW (1998) Automatic construction of decision trees from data: a multi-disciplinary survey. Data Min Knowl Discov 2: 345–389
Opitz D, Maclin R (1999) Popular ensemble methods: an empirical study. Artif Intell Res 11: 169–198
Panov P, Dzeroski S (2007) Combining bagging and random subspaces to create better ensembles, IDA 2007, LNCS 4723, pp 118–129
Redpath DB, Lebart K (2005) Boosting feature selection. In: The third international conference on advances in pattern recognition, Bath, UK, Springer, Berlin, pp 305–314
Rodríguez JJ, Kuncheva LI, Alonso CJ (2006) Rotation forest: a new classifier ensemble method. IEEE Trans Pattern Anal Mach Intell 28(10): 1619–1630
Salzberg S (1997) On comparing classifiers: pitfalls to avoid and a recommended approach. Data Min Knowl Discov 1: 317–328
Schapire RE, Freund Y, Bartlett P, Lee WS (1998) Boosting the margin: a new explanation for the effectiveness of voting methods. Ann Stat 26: 1651–1686
Schapire RE, Singer Y (1999) Improved boosting algorithms using confidence-rated predictions. Mach Learn 37: 297–336
Shirai S, Kudo M, Nakamura A (2008) Bagging, random subspace method and biding. SSPR&SPR 2008, LNCS vol 5342, pp 801–810
Stefanowski J (2007) Combining answers of sub-classifiers in the bagging-feature ensembles, RSEISP 2007, LNAI vol 4585, pp 574–583
Ting KM, Witten IH (1997) Stacking bagged and dagged models. In: Fourteenth international conference on machine learning, San Francisco, CA, pp 367–375
Webb GI (2000) MultiBoosting: a technique for combining boosting and wagging. Mach Learn 40: 159–196
Yin X-C, Liu C-P, Zhi H (2005) Feature combination using boosting. Pattern Recognit Lett 26: 2195–2205
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kotsiantis, S. Combining bagging, boosting, rotation forest and random subspace methods. Artif Intell Rev 35, 223–240 (2011). https://doi.org/10.1007/s10462-010-9192-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-010-9192-8