Boosted K-nearest neighbor classifiers based on fuzzy granules

Abstract

K-nearest neighbor (KNN) is a classic classifier, which is simple and effective. Adaboost is a combination of several weak classifiers as a strong classifier to improve the classification effect. These two classifiers have been widely used in the field of machine learning. In this paper, based on information fuzzy granulation, KNN and Adaboost, we propose two algorithms, a fuzzy granule K-nearest neighbor (FGKNN) and a boosted fuzzy granule K-nearest neighbor (BFGKNN), for classification. By introducing granular computing, we normalize the process of solving problem as a structured and hierarchical process. Structured information processing is focused, so the performance including accuracy and robust can be enhanced to data classification. First, a fuzzy set is introduced, and an atom attribute fuzzy granulation is performed on samples in the classified system to form fuzzy granules. Then, a fuzzy granule vector is created by multiple attribute fuzzy granules. We design the operators and define the measure of fuzzy granule vectors in the fuzzy granule space. And we also prove the monotonic principle of the distance of fuzzy granule vectors. Furthermore, we also give the definition of the concept of K-nearest neighbor fuzzy granule vector and present FGKNN algorithm and BFGKNN algorithm. Finally, we compare the performance among KNN, Back Propagation Neural Network (BPNN), Support Vector Machine (SVM), Logistic Regression (LR), FGKNN and BFGKNN on UCI data sets. Theoretical analysis and experimental results show that FGKNN and BFGKNN have better performance than that of the methods mentioned above if the appropriate parameters are given.

Introduction

As early as the 1960s, Zadeh, a famous American cybernetic expert, proposed fuzzy set theory and in 1979 first presented the problem of fuzzy information granulation [1]. He believed that human cognition can be summarized as three main characteristics of granulation, organization and causality [2], [3], [4], [5]. In 1985, Hobbs [6] presented the concept of granularity. Later, Yager and Filev [7] further pointed out that “people have formed a granular view of the world”. From this point of view, human observation, measurement, conceptualization and reasoning are carried out in a granular sense. The granular computing was first proposed by T.Y. Lin [8], [9]. Information granules are not only reflective of the nature of the data but can efficiently capture some auxiliary domain knowledge conveyed by the user and in this way reflect the human-centricity aspects of the investigations and enhances the actionability aspects of the results [10].

The concept of information granularity is ubiquitous, and granular computing has also promoted the development of many concepts, which are as the follows: graphs [11], [12], [13], information tables [14], knowledge representation [15], association discovery and data mining [16], clustering [17] and rule clustering [18], classification [19] and so on. Granule computing is also widely applied in forecasting time series [20], prediction tasks [21], concept learning [22], perception [23], optimization [24], credit scoring [25] etc. Many scholars have conducted extensive and in-depth research from various angles. Miao discussed the structure of granular computing from the perspective of set theory [26]. Wang analyzed the uncertainty measure in granular computing and its application in big data [27], [28]. Yao proposed neighborhood system and neighborhood granular computing [29], [30]. Hu analyzed neighborhood reduction and classification [31], [32], [33]. Chen studied feature dimension reduction and optimization from the perspective of group intelligence [34], [35] and so on. These views suggest that granulation, as one of the important features of human cognition, plays an important role in modeling complex data.

KNN algorithm was first proposed by Hart in 1968 [36]. KNN is a non-parametric statistical method for classification and regression in the field of pattern recognition [37]. KNN uses a vector space model to classify cases with the same category, with high similarity to each other, and can calculate the similarity with known category cases to evaluate the possible classification of unknown category cases. It is a simple and effective non-parametric classification method. Its advantages include that it is very suitable for incremental learning without knowing the sample distribution in advance. Explicit rules are not required, and classification accuracy is high, so it is widely used in many fields such as clustering, big data, and multi-label learning [38], [39], [40], [41], [42], [43]. The classical KNN algorithm has high time and space complexity. The same weight of K neighbor samples affects classification accuracy. It is noise sensitivity and has low classification accuracy to unbalanced samples, and is also difficult in determining K value. Many scholars have proposed improvements from the above aspects and improved the its performance [44], [45], [46], [47], [48], [49], [50], [51], [52].

It is difficult to construct a single classifier with high accuracy. However, it is possible to construct a strong classifier with high accuracy via integrating some weak classifiers. The weak learning theorem [53] theoretically supports this possibility. It is the main content of integrated learning research how to construct weak classifiers and how to integrate them. At present, the more successful integrated learning algorithm is AdaBoost [37], first proposed by Freund and Schapire in 1995. In 1999, Schapire et al. extended AdaBoost, which deals with binary judgments, to continuous AdaBoost with continuous confidence output. Thus, it can more accurately describe classification boundaries and have better classification effects [54]. AdaBoost algorithm is simple and widely applied in many fields, such as face recognition, water quality detection, protein prediction, pedestrian detection, EEG signal analysis, urban rail transit, etc. [55], [56], [57], [58], [59], [60], [61], [62], [63], [64]. At the same time, it also attracted a large number of scholars to research and improve its generalization ability [65], [66], [67], [68].

In this paper, we define fuzzy granule vector based on fuzzy information granulation from a new perspective, and design two new classification models: FGKNN and BFGKNN. On the basis of fuzzy granulation of various attributes of a classification system, we define fuzzy granule vector distance, and propose the K-nearest fuzzy granule vector concept, and transform the classification problem into the K-nearest fuzzy granule vector search problem. Moreover, we present FGKNN classification model. Based on FGKNN, we furthermore design BFGKNN model. We employ 10-fold cross-validation to test the performance of the two algorithms on UCI data sets. Theoretical analysis and experimental results show that FGKNN and BFGKNN can achieve better performance under appropriate parameters.