CO²RBFN: an evolutionary cooperative–competitive RBFN design algorithm for classification problems

Abstract

This paper presents a new evolutionary cooperative–competitive algorithm for the design of Radial Basis Function Networks (RBFNs) for classification problems. The algorithm, CO²RBFN, promotes a cooperative–competitive environment where each individual represents a radial basis function (RBF) and the entire population is responsible for the final solution. The proposal considers, in order to measure the credit assignment of an individual, three factors: contribution to the output of the complete RBFN, local error and overlapping. In addition, to decide the operators’ application probability over an RBF, the algorithm uses a Fuzzy Rule Based System. It must be highlighted that the evolutionary algorithm considers a distance measure which deals, without loss of information, with differences between nominal features which are very usual in classification problems. The precision and complexity of the network obtained by the algorithm are compared with those obtained by different soft computing methods through statistical tests. This study shows that CO²RBFN obtains RBFNs with an appropriate balance between accuracy and simplicity, outperforming the other methods considered.

Appendices

Appendix 1: Detailed results obtained for CO²RBFN

Car dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
4	0.129	0.007	0.207	0.061	87.104	79.350
5	0.122	0.007	0.190	0.048	87.823	81.007
6	0.116	0.008	0.204	0.045	88.354	79.572
7	0.112	0.007	0.198	0.046	88.753	80.197
8	0.108	0.009	0.198	0.051	89.186	80.241
9	0.102	0.009	0.200	0.056	89.784	79.998
10	0.098	0.009	0.207	0.053	90.203	79.270
11	0.095	0.009	0.191	0.052	90.543	80.925
12	0.093	0.011	0.200	0.040	90.678	80.011
13	0.089	0.008	0.202	0.047	91.075	79.804
14	0.087	0.010	0.207	0.049	91.279	79.261
15	0.079	0.008	0.220	0.061	92.085	77.974
16	0.079	0.010	0.199	0.045	92.103	80.127

Credit dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
2	0.123	0.006	0.158	0.065	87.655	84.232
3	0.120	0.004	0.158	0.081	87.974	84.203
4	0.118	0.004	0.177	0.100	88.235	82.261
5	0.116	0.005	0.175	0.096	88.432	82.522
6	0.115	0.003	0.172	0.088	88.470	82.812
7	0.114	0.004	0.167	0.084	88.577	83.275
8	0.113	0.004	0.179	0.094	88.686	82.116

Glass dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
7	0.328	0.020	0.358	0.113	67.223	64.216
8	0.319	0.016	0.373	0.103	68.145	62.699
9	0.310	0.017	0.354	0.104	68.987	64.575
10	0.296	0.014	0.360	0.120	70.410	63.990
11	0.290	0.015	0.354	0.111	71.034	64.635
12	0.282	0.017	0.333	0.105	71.812	66.694
13	0.277	0.014	0.332	0.111	72.299	66.778
14	0.275	0.016	0.356	0.116	72.547	64.389
15	0.266	0.015	0.330	0.109	73.399	66.976
16	0.262	0.015	0.343	0.107	73.826	65.654
17	0.258	0.015	0.346	0.104	74.230	65.425
18	0.255	0.016	0.335	0.103	74.490	66.487
19	0.251	0.014	0.340	0.117	74.925	65.980
20	0.250	0.017	0.349	0.109	74.977	65.086
21	0.248	0.014	0.354	0.118	75.164	64.602
22	0.248	0.014	0.323	0.111	75.175	67.710
23	0.242	0.016	0.328	0.114	75.798	67.223
24	0.240	0.015	0.332	0.118	75.997	66.763
25	0.236	0.014	0.342	0.116	76.389	65.820
26	0.233	0.015	0.326	0.102	76.700	67.391
27	0.234	0.019	0.329	0.115	76.587	67.065
28	0.235	0.015	0.337	0.107	76.536	66.332

Hepatitis dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
2	0.087	0.016	0.168	0.145	91.270	83.228
3	0.074	0.009	0.168	0.138	92.632	83.157
4	0.067	0.011	0.151	0.094	93.261	84.905
5	0.065	0.011	0.151	0.071	93.548	84.914
6	0.064	0.008	0.128	0.107	93.563	87.187
7	0.063	0.010	0.139	0.075	93.749	86.137
8	0.057	0.007	0.126	0.074	94.280	87.399

Ionosphere dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
2	0.149	0.017	0.171	0.053	85.078	82.926
3	0.134	0.020	0.160	0.049	86.648	84.003
4	0.105	0.015	0.134	0.050	89.485	86.579
5	0.097	0.017	0.119	0.049	90.294	88.069
6	0.087	0.014	0.111	0.043	91.270	88.907
7	0.074	0.013	0.099	0.048	92.643	90.111
8	0.065	0.011	0.086	0.039	93.485	91.411

Iris dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
3	0.017	0.008	0.045	0.047	98.252	95.467
4	0.012	0.004	0.048	0.055	98.770	95.200
5	0.011	0.004	0.045	0.052	98.919	95.467
6	0.010	0.004	0.037	0.042	99.007	96.267
7	0.010	0.005	0.037	0.050	99.007	96.267
8	0.009	0.004	0.044	0.054	99.067	95.600
9	0.009	0.004	0.052	0.050	99.081	94.800
10	0.009	0.004	0.044	0.049	99.141	95.600
11	0.009	0.004	0.048	0.050	99.111	95.200
12	0.008	0.004	0.040	0.046	99.230	96.000

Pima dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
2	0.239	0.024	0.260	0.051	76.068	74.001
3	0.227	0.010	0.248	0.043	77.341	75.218
4	0.221	0.007	0.240	0.049	77.873	75.950
5	0.218	0.006	0.247	0.047	78.224	75.252
6	0.215	0.006	0.243	0.047	78.516	75.716
7	0.213	0.006	0.244	0.045	78.675	75.638
8	0.212	0.006	0.242	0.044	78.814	75.796

Sonar dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
2	0.247	0.020	0.282	0.111	75.299	71.762
3	0.235	0.019	0.261	0.114	76.517	73.910
4	0.222	0.015	0.283	0.097	77.756	71.705
5	0.219	0.020	0.285	0.092	78.120	71.514
6	0.210	0.013	0.279	0.099	78.952	72.114
7	0.206	0.016	0.271	0.090	79.412	72.948
8	0.201	0.012	0.249	0.098	79.882	75.086

Vehicle dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
4	0.432	0.021	0.446	0.048	56.782	55.391
5	0.400	0.019	0.415	0.054	60.032	58.489
6	0.389	0.021	0.405	0.054	61.135	59.520
7	0.369	0.026	0.381	0.045	63.092	61.912
8	0.357	0.019	0.386	0.043	64.303	61.390
9	0.343	0.016	0.370	0.050	65.721	62.954
10	0.332	0.016	0.350	0.047	66.840	64.979
11	0.318	0.015	0.353	0.046	68.201	64.703
12	0.311	0.013	0.334	0.045	68.873	66.645
13	0.307	0.010	0.326	0.041	69.320	67.414
14	0.296	0.014	0.318	0.038	70.433	68.201
15	0.292	0.014	0.316	0.044	70.846	68.433
16	0.287	0.012	0.308	0.047	71.253	69.192

Wbcd dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
2	0.024	0.004	0.032	0.023	97.581	96.769
3	0.023	0.002	0.033	0.021	97.743	96.741
4	0.023	0.002	0.033	0.021	97.749	96.739
5	0.022	0.002	0.029	0.018	97.797	97.083
6	0.022	0.002	0.032	0.019	97.794	96.792
7	0.022	0.002	0.033	0.020	97.803	96.740
8	0.022	0.002	0.029	0.020	97.813	97.054

Wine dataset

# Nodes	Error training	Standard dev	Error test	Stand dev	% Training	% Test
3	0.008	0.006	0.051	0.060	99.189	94.915
4	0.004	0.004	0.057	0.066	99.575	94.261
5	0.003	0.003	0.043	0.061	99.738	95.732
6	0.002	0.003	0.038	0.045	99.813	96.157
7	0.001	0.002	0.033	0.046	99.913	96.739
8	0.001	0.002	0.036	0.040	99.950	96.412
9	0.000	0.001	0.037	0.046	99.975	96.281
10	0.000	0.000	0.045	0.054	100.000	95.484
11	0.000	0.001	0.038	0.047	99.975	96.196
12	0.000	0.000	0.038	0.050	100.000	96.190

Appendix 2: GeneticRBFN description

To test our cooperative–competitive method against a Pittsburgh based proposal, a typical genetic algorithm for the RBFN design, GeneticRBFN, has been developed. The design lines of GeneticRBFN are the classical ones for these kinds of algorithms (Harpham et al. 2004). In order to establish similar operating conditions certain characteristics of CO²RBFN have been introduced in GeneticRBFN, like analogies in the operators and the HVDM dissimilarity measure.

This method follows the traditional Pittsburgh evolutionary approach for the design of RBFNs: each individual is a whole network. The objective of the evolutionary process is to minimise the classification error. The main steps of this algorithm are shown in Fig. 5.

Fig. 5

Main steps of GeneticRBFN algorithm

The main components of GeneticRBFN algorithm are described below.

2.1 Initialization

The initialization stage is the same as that in CO²RBFN. So the RBFs will be centred, in an equidistributed way, for each RBFN/individual.

2.2 Genetic operators: selection, recombination and mutation

With the crossover operator, two individuals/RBFNs parents are randomly chosen to obtain an RBFN offspring. The number of RBFs of the new individual will be delimited between a minimum and a maximum value. The minimum value is set to the number of RBFs of the parent with fewer RBFs. In the same way, the maximum value is set to the number of RBFs of the parent with more RBFs. In order to generate the offspring RBFs will be chosen in a random way from the parents.

Six mutation operators, usually considered in the specialised bibliography (Harpham et al. 2004) have been implemented. They can be classified as random operators or biased operators. The random operators are:

• DelRandRBFs: randomly eliminates k RBFs, where k is a pm percent of the total number of RBFs in the RBFN.

• InsRandRBFs: randomly aggregates k RBFs, where k is a pm percent of the total number of RBFs in the RBFN.

• ModCentRBFs: randomly modifies the centre of k RBFs, where k is a pm percent of the total number of RBFs in the RBFN. The centre of the basis function will be modified in a pr percent of its width.

• ModWidtRBFs: randomly modifies the centre of k RBFs, where k is a pm percent of the total number of RBFs in the RBFN. The width of the basis function will be modified in a pr percent of its width.

Biased operators, which exploit local information are:

• DelInfRBFs: deletes the k RBFs of the RBFN with a lower weight. k is a pm percent of the total number of RBFs in the RBFN.

• InsInfRBFs: inserts the k RBFs in the RBFN outside the width of any RBF present in the RBFN. k is a pm percent of the total number of RBFs in the RBFN.

An intermediate population with the parents and the offspring is considered and a tournament selection mechanism is used to determine the new population. The diversity of the population is promoted by using a low value for the tournament size (k = 3).

2.3 Training weights

In order to train the weights, the LMS algorithm is used. Its parameters are set to their standard values.

2.4 Individual evaluation

The fitness defined for each individual/RBFN is its classification error for the given problem.

In order to increase the efficiency of the GA, the search space of this method has been drastically reduced. As is well known, in Pittsburgh GAs, where the only objective to optimise is the classification error, the complexity of the individuals (i.e. number of RBFs) grows in an uncontrolled way (because normally an RBFN with more RBFs gives a lower error percentage than an RBFN with few RBFs). In this experimentation, the search space has been reduced by fixing the maximum complexity (and so chromosome size) between a minimum and a maximum number of RBFs. The minimum number of RBFs has been set to the number of classes for the problem and the maximum to four times this number.

Appendix 3: Statistical methods

Non-parametric methods are often referred to as distribution free methods, as they do not rely on assumptions that the data are drawn from a given probability distribution.

Non-parametric methods are widely used for studying populations which take a ranked order (such as movie reviews receiving one to four stars). The use of non-parametric methods may be necessary when data has a ranking but no clear numerical interpretation, such as when assessing preferences.

As non-parametric methods make fewer assumptions, their applicability is much wider than the corresponding parametric methods. In particular, they may be used in situations where less is known about the application in question. In addition, due to the reliance on fewer assumptions, non-parametric methods are more robust.

Another justification for the use of non-parametric methods is simplicity. In certain cases, even when the use of parametric methods is justified, non-parametric methods may be easier to use. Due both to this simplicity and to their greater robustness, non-parametric methods are seen by some statisticians as leaving less room for improper use and misunderstanding.

Now we will explain the non-parametric methods used:

3.1 Friedman’s test

This is a non-parametric equivalent of the test of repeated-measures ANOVA. It computes the ranking of the observed results for algorithm (r _j for the algorithm j with k algorithms) for each data-set, assigning to the best the ranking 1, and to the worst the ranking k. Under the null hypothesis, formed from supposing that the results of the algorithms are equivalents and, therefore, their rankings are also similar, Friedman’s statistic

$$ \chi_{F}^{2} = {\frac{{12N_{ds} }}{k(k + 1)}}\left[ {\sum\limits_{j} {R_{j}^{2} - {\frac{{k(k + 1)^{2} }}{4}}} } \right], $$

is distributed according to $\chi_{F}^{2}$ with k − 1 degrees or freedom, being $R_{j} = {\frac{1}{{N_{ds} }}}\sum\nolimits_{i} {r_{i}^{2} } ,$ and N _ds the number of data-set. The critical values for Friedman’s statistic coincide with those established in the $\chi^{2}$ distribution when N _ds > 10 and k > 5.

3.2 Iman and Davenport’s test

This is a metric derived from Friedman’s statistic, given that this last metric produces a conservative undesirable effect. The statistic is:

$$ F_{F} = {\frac{{\left( {N_{\text{ds}} - 1} \right)\chi_{F}^{2} }}{{N_{\text{ds}} (k - 1) - \chi_{F}^{2} }}}, $$

and it is distributed according to a F-distribution with k − 1 and (k − 1)(N _ds − 1) degrees of freedom.

3.3 Wilcoxon’s signed-rank test

This is analogous to the paired t test in non-parametrical statistical procedures; therefore, it is a pair test that aims to detect significant differences between the behaviour of two algorithms.

Let d _i be the difference between the performance score of the two classifiers on ith out of N _ds data-sets. The differences are ranked according to their absolute values; average ranks are assigned in case of ties. Let R ⁺ be the sum of ranks for the data-sets is which the first algorithm outperformed the second, and R ⁻ the sum of ranks for the opposite. For ranks of d _i  = 0 are split evenly the sums; if there is an odd number of them, one is ignored:

$$ \begin{gathered} R^{ + } = \sum\limits_{{d_{i} > 0}} {{\text{rank}}(d_{i} ) + {\frac{1}{2}}\sum\limits_{{d_{i} = 0}} {{\text{rank}}(d_{i} )} } , \hfill \\ R^{ - } = \sum\limits_{{d_{i} < 0}} {{\text{rank}}(d_{i} ) + {\frac{1}{2}}\sum\limits_{{d_{i} = 0}} {{\text{rank}}(d_{i} )} } . \hfill \\ \end{gathered} $$

Let T be the smallest of the sums, T = min(R ⁺ , R ⁻). If T is less than or equal to the value of the distribution of Wilcoxon for N _ds degrees of freedom, the null hypothesis of equality of means is rejected.