Abstract
Along with the increase of data and information, incremental learning ability turns out to be more and more important for machine learning approaches. The online algorithms try not to remember irrelevant information instead of synthesizing all available information (as opposed to classic batch learning algorithms). Today, combining classifiers is proposed as a new road for the improvement of the classification accuracy. However, most ensemble algorithms operate in batch mode. For this reason, we propose an incremental ensemble that combines five classifiers that can operate incrementally: the Naive Bayes, the Averaged One-Dependence Estimators (AODE), the 3-Nearest Neighbors, the Non-Nested Generalised Exemplars (NNGE) and the Kstar algorithms using the voting methodology. We performed a large-scale comparison of the proposed ensemble with other state-of-the-art algorithms on several datasets and the proposed method produce better accuracy in most cases.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Auer P, Warmuth M (1998) Tracking the best disjunction. Mach Learn 32: 127–150, Kluwer Academic Publishers
Chu F, Zaniolo C (2004) Fast and light boosting for adaptive mining of data streams. In: Advances in knowledge discovery and data mining, Lecture notes in computer science 3056. pp 282–292
Cleary JG, Trigg LE (1995) K*: an instance-based learner using an entropic distance measure. In: 12th international conference on machine learning. pp 108–114
Cohen W (1995) Fast effective rule induction. In: Proceedings of International Conference of ML-95. pp 115–123
Dietterich T. (2000) Ensemble Methods in Machine Learning, Lecture Notes in Computer Science, vol 1857, pp 1–15
Domingos P, Pazzani M (1997) On the optimality of the simple Bayesian classifier under zero-one loss. Mach Learn 29: 103–130
Fan W, Stolfo S, Zhang J (1999) The application of AdaBoost for distributed, scalable and on-line learning. In: Proceedings of the fifth ACM SIGKDD international conference on knowledge discovery and data mining. ACM Press, New York, pp 362–366
Fern A, Givan R (2000) Online ensemble learning: an empirical study. In: Proceedings of the seventeenth international conference on ML. Morgan Kaufmann, pp 279–286
Frank A, Asuncion A (2010) UCI Machine learning repository. [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science
Freund Y, Schapire R (1999) Large margin classification using the perceptron algorithm. Mach Learn 37: 277–296. Kluwer Academic Publishers
Gangardiwala, A.; Polikar, R.; Dynamically weighted majority voting for incremental learning and comparison of three boosting based approaches, 2005 IEEE international joint conference on neural networks, IJCNN ’05, vol 2, pp. 1131–1136, 31 July–4 Aug 2005
Janssens D, Brijs T, Vanhoof K, Wets G (2006) Evaluating the performance of cost-based discretization versus entropy- and error-based discretization. Comput Oper Res 33(11): 3107–3123
Katagiri S, Abe S (2006) Incremental training of support vector machines using hyperspheres. Pattern Recognit Lett 27(13): 1495–1507
Kotsiantis S, Zaharakis I, intelas P (2006) Machine learning: a review of classification and combining techniques. Artificial Intell Rev 26(3): 159–190 (Springer)
Kuncheva LI (2004) Classifier ensembles for changing environments. In: Multiple classifier systems (MCS 2004), Lecture notes in computer science 3077. pp 1–15
Littlestone N, Warmuth M (1994) The weighted majority algorithm. Inf Comput 108: 212–261
Menahem E, Rokach L, Elovici Y (2009) Troika—an improved stacking schema for classification tasks. Inf Sci. doi:10.1016/j.ins.2009.08.025
Oza NC, Russell S (2001) Online bagging and boosting. In: Richardson T, Jaakkola T (eds) Artificial intelligence and statistics. pp 105–112
Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann, San Francisco
Rokach L (2009) Taxonomy for characterizing ensemble methods in classification tasks: a review and annotated bibliography. Comput Statist Data Anal 53(12): 4046–4072
Rokach L (2010) Ensemble-based classifiers. Artif Intell Rev 33(1–2): 1–39
Roy S (2002) Nearest neighbor with generalization. Christchurch, New Zealand
Saad D (1998) Online learning in neural networks. Cambridge University Press, London
Sahami M (1996) Learning limited dependence Bayesian classifiers. In: Proceedings of the 2nd international conference on knowledge discovery in databases. pp 335–338
Salzberg S (1997) On comparing classifiers: pitfalls to avoid and a recommended approach. Data Min Knowl Discov 1: 317–328
Ulaş A, Semerci M, Yıldız O T, Alpaydın E (2009) Incremental construction of classifier and discriminant ensembles. Inf Sci 179(9): 1298–1318
Utgoff P, Berkman N, Clouse J (1997) Decision tree induction based on efficient tree restructuring. Mach Learn 29: 5–44
Webb GI, Boughton JR, Wang Z (2005) Not so naive Bayes: aggregating one-dependence estimators. Mach Learn 58: 5–24
Widmer G, Kubat M (1996) Learning in the presence of concept drift and hidden contexts. Mach Learn 23: 69–101
Witten I, Frank E (2005) Data mining: practical machine learning tools and techniques. 2. Morgan Kaufmann, San Francisco
Wu Y, Ianakiev K, Govindaraju V (2002) Improved k-nearest neighbor classification. Pattern Recognit 35(10): 2311–2318
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kotsiantis, S.B. An incremental ensemble of classifiers. Artif Intell Rev 36, 249–266 (2011). https://doi.org/10.1007/s10462-011-9211-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-011-9211-4