Skip to main content
SpringerLink
Log in

An improved multiple fuzzy NNC system based on mutual information and fuzzy integral

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Multiple nearest neighbor classifier system (MNNCS) is a popular method to relax the curse of dimensionality. In previous work, most of the MNNCSs are designed by random methods. Random methods may generate unstable component classifiers. In order to relax the randomness, large amount of component classifiers are needed. This paper first extends nearest neighbor classifier into fuzzy nearest neighbor classifier, and proposes a new multiple fuzzy nearest neighbor classifier system based on mutual information and fuzzy integral, called MIFI-MFNNCS. MIFI-MFNNCS adopts target perturbation. Target perturbation decomposes the original classification problem into several sub-problems, where one sub-problem represents one class data. Each sub-problem is described by the relevant data and features. Then it is classified by one component classifier. Therefore, the number of component classifiers can be fixed and reduced. For one component classifier, data may be selected according to its class. And feature is needed to be selected by mutual information. Mutual information can reduce the uncertainty of each component classifier. Feature selection by mutual information in MIFI-MFNNCS may be less affected by the interaction among different classes. The diversity decisions from sub-problem classifiers are combined by fuzzy integral to get the final decision. Here we propose a new method to compute density value according to mutual information, which is a simple method. To demonstrate the performance of the proposed MIFI-MFNNCS, we perform experimental comparisons using five UCI datasets. The results of component classifiers in MIFI-MFNNCS for Ionosphere are shown and analyzed. MIFI-MFNNCS is compared with (1) NNC (2) NNC after feature selection by mutual information (MI-FS-NNC). In multiple fuzzy nearest neighbor classifier system (MFNNCS), mutual information is compared with attribute bagging. And three combination methods are compared, including fuzzy integral, majority voting rule and average. The experimental results show that the accuracy of MIFI-MFNNCS is better than other methods. And mutual information is superior to attribute bagging. Fuzzy integral shows a better performance than majority voting rule and average.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Ludmila I (2004) Kuncheva, combining pattern classifiers: methods and algorithms, wiles

  2. Saranlm A, Demirekler M (2001) On output independence and complementariness in rank-based multiple classifer decision systems. Pattern Recogn 34:2319–2330

    Article  Google Scholar 

  3. Zhou ZH (2005) Ensembling local learners through multimodal perturbation. IEEE Trans SMC Part B 35(4):725–735

    Google Scholar 

  4. Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140

    MathSciNet  MATH  Google Scholar 

  5. Freund Y, Schapire RE (1995) A decision-theoretic generalization of online learning and an application to boosting. Proceedings of the 2nd European conference on computational learning theory, Barcelona, Spain, pp 23–37

  6. Ho TK (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8):832–844

    Article  Google Scholar 

  7. Brylla R, Gutierrez-Osunab R, Queka F (2003) Attribute bagging: improving accuracy of classifer ensembles by using random feature subsets. Pattern Recogn 36:1291–1302

    Article  Google Scholar 

  8. Langley P, Iba W. Average-case analysis of a nearest neighbor algorithm. Proceedings of the thirteenth international joint

  9. Bay SD (1998) Combining nearest neighbor classifiers through multiple feature subsets. In: Proceedings of the 15th international conference on machine learning, Madison, WI, pp 37–45

  10. García-Pedrajas N, Ortiz-Boyer D (2009) Boosting k-nearest neighbor classifier by means of input space projection. Expert Syst Appl 36(7):10570–10582

    Google Scholar 

  11. Oza NC, Tumer K (2001) Input decimation ensembles: decorrelation through dimensio-nality reduction, 2nd international workshop on multiple classifier systems. In: Kittler J, Roli F (eds) Lecture notes in computer science. 2096:238–247

  12. Chow TWS, Huang D (2005) Estimating optimal feature subsets using efficient estimation of high-dimensional mutual information. IEEE Trans Neural Netw 16(1):213–224

    Google Scholar 

  13. Batttiti R (1994) Using mutual information for selecting features in supervised neural net learning. IEEE Trans Neural Netw 5(4):537–550

    Google Scholar 

  14. Kwak N, Choi CH (2002) Input feature selection for classification problems. IEEE Trans Neural Netw 13(1):143–159

    Google Scholar 

  15. Kwak N, Choi CH (2002) Input feature selection by mutual information based on parzen window. IEEE Trans PAMI 24(12):1667–1671

    Google Scholar 

  16. Huang J, Cai Y, Xu X (2007) A hybrid genetic algorithm for feature selection wrapper based on mutual information. Pattern Recogn Lett 28(13):1825–1844

    Google Scholar 

  17. François D, Rossib F, Wertza V, Verleysen M (2007) Resampling methods for parameter-free and robust feature selection with mutual information. Neurocomputing 70(7–9):1276–1288

    Google Scholar 

  18. Liu HW, Sun JG, Liu L, Zhang HJ (2009) Feature selection with dynamic mutual information. Pattern Recogn 42(7):1330–1339

    Google Scholar 

  19. Wang ZY, Klir GJ (1992) Fuzzy measure theory. Plenum Press, New York

    MATH  Google Scholar 

  20. Keller JM, sborn OJ (1996) Training the fuzzy integral. J Approx Reason 15:1–24

    Article  MATH  Google Scholar 

  21. Wang ZY, Leung KS, Wang J (1999) A genetic algorithm for determining nonadditive set functions in information fusion. Fuzzy Sets Syst 102(3):463–469

    Article  MathSciNet  MATH  Google Scholar 

  22. Wang J, Wang ZY (1997) Using neural networks to determine Sugeno measures by statics. Neural Netw 10(1):183–195

    Article  Google Scholar 

  23. Hu BG, Wang Y (2008) Evaluation criteria based on mutual information for classifications including rejected class. Acta Automatica Sin 34(11):1396–1403

    Article  Google Scholar 

  24. Wang LJ (2006) Combination of multiple K-NNCs by Fuzzy integral. Proceedings of the fifth international conference on machine learning and cybernetics, pp 1774–1778

  25. Sungeno M (1997) Fuzzy measures and fuzzy integrals—a survey. In: Gupta MM, Saridis GN, Gaines BR (eds) Fuzzy automata and decision processes. North-Holland, Amsterdam, pp 89–102

Download references

Acknowledgments

This paper is supported by Doctor initial fund of Guangdong university of technology (093031), Key Basic Research Project of Hebei Province Applied basic research plan (08963522D), and Hebei Province Department of Education Planning Project (Z2008302). Some parts of this paper has been appeared in Multiple Nearest neighbor classifiers system based on feature perturbation by mutual information, which has published in Proc of ICMLC 10.

Author information

Authors and Affiliations

  1. Faculty of Computer, Guangdong University of Technology, Guangzhou, 510006, China

    Li Juan Wang

Authors
  1. Li Juan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Juan Wang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wang, L.J. An improved multiple fuzzy NNC system based on mutual information and fuzzy integral. Int. J. Mach. Learn. & Cyber. 2, 25–36 (2011). https://doi.org/10.1007/s13042-010-0006-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-010-0006-8

Keywords

Search

Navigation