A novel approach for classifying imbalance welding data: Mahalanobis genetic algorithm (MGA)

Abstract

Feature selection from imbalance data plays an important role in building efficient support decision systems, improving the machine learning process performance and enhancing the classification accuracy. The problem of feature selection becomes even more difficult with imbalance data, which occurs in real-world domains when the classes representing the data set are not equally distributed. Using the traditional classifiers to seek an accurate performance over a full range of instances is not suitable to deal with imbalanced learning tasks, since they tend to classify all the data into one class. In this paper, the Mahalanobis genetic algorithm (MGA) classifier is proposed to address the problem of feature selection for imbalance welding data. The MGA classifier was benchmarked with the Mahalanobis-Taguchi system (MTS) classifier, in terms of the following metrics: the total misclassification errors, the area under the curve (AUC) for receiver operating characteristic (ROC) curves, and the signal-to-noise (S/N) ratio. A real-life data set from the spot welding process was used as a pilot study. The results in terms of the total misclassification error and the AUC metrics showed that the MGA had better classification performance than MTS. Very close results were obtained when the training data set was balanced by using the Synthetic Minority Oversampling Technique (SMOTE) which indicates the suitability of the MGA and MTS classifiers to be used for the imbalance data set without using any preprocessor approach. Regarding the S/N ratio, the results were inconsistent with the other classification metrics, which raises the question about its credibility.

Appendix

1.1 Mahalanobis distance

The MD is a distance measurement introduced by P.C. Mahalanobis in 1936. It is a useful way of finding the similarity of an unknown sample set to a known one. It differs from the Euclidian distance in that it takes into consideration the correlation between variables. MD has been used in multivariate outlier detection [86, 87], process control [88, 89], and pattern recognition [90, 91].

To explain the difference between the Euclidian distance and the MD, consider the data shown in Fig. 10. Suppose a new alias data point “a” is obtained, and a decision should be made whether this point belongs to the group or not. If the Euclidean distance is used (left figure) as the decision criterion, point “a” will be classified to the group because it has the same distance from the center of the group as point “b.” On the other hand, if the MD is used as a decision criterion (right figure), then point “a” will not be classified to the group because it has a larger MD due to the weak correlation to the group data.

Mathematically, the MD is the squared distance (denoted as D²) calculated for the jth observation in a sample of size n with p features (or variables) as follows:

$$ {\mathrm{MD}}_j={D}_1^2={Z}_{ij}^T{R}^{-1}{Z}_{ij} $$

where j = 1…n, i = 1…p, Z _ij is the standardized vector obtained by standardizing the values of X _ij (i.e., ${Z}_{ij}=\left({X}_{ij}-{\overline{X}}_i\right)/{S}_i$, ${\overline{X}}_i$ is the mean of feature i, S _i is the sample standard deviation of feature i, Z ^T _ij is the transpose of Z _ij , and R ^− 1 is the inverse of the correlation matrix of the features.

1.2 Genetic algorithms

The GA, which was inspired by evolution theory, is a global meta-heuristic technique first introduced by Holland in 1975 [92]. According to this theory, living beings compete with each other to survive. Weak beings are faced with extinction within their environment by the natural selection process. The strong ones have better opportunity to pass their genes to the future generations via reproduction. In the long run, beings carrying the correct combination in their genes become dominant in their population. Sometimes, random changes may occur in genes during the slow process of evolution. If these changes provide additional advantages in the challenge for survival, new beings evolve from the old ones. Unsuccessful changes are eliminated by the natural selection.

In GA terminology, a candidate solution is called a chromosome or an individual. These chromosomes consist of discrete units called genes. Each gene controls one or more features of the chromosome. Genes are assumed to be binary digits in the original implementation of GA by Holland. In later implementations, more varied gene types have been introduced. Normally, a chromosome corresponds to a unique solution in the solution space. This requires a mapping mechanism between the solution space and the chromosomes. This mapping is called an encoding. In fact, GA works on the encoding of a problem, not on the problem itself.

GA works on a group of chromosomes, called a population. The population is normally randomly initialized. As the search evolves, the population contains fitter and fitter solutions, and eventually it converges, meaning that it is dominated by a single solution.

There are two operators used to generate new solutions from existing ones: crossover and mutation. The crossover operator is the most important operator of GA. In crossover (Fig. 11), generally two chromosomes, called parents, are combined together to form new chromosomes, called offspring. The parents are selected among existing chromosomes in the population with preference toward fitness so that offspring is expected to inherit good genes which make the parents fitter. By iteratively applying the crossover operator, genes of good chromosomes are expected to appear more frequently in the population, eventually leading to convergence to an overall good solution.

Fig. 10

Euclidian and Mahalanobis distance concepts

On the other hand, the mutation operator introduces random changes into the characteristics of chromosomes. Mutation is generally applied at the gene level (Fig. 12). In typical GA implementations, the mutation rate (probability of changing the properties of a gene) is very small and depends on the length of the chromosome. Therefore, the new chromosome produced by mutation will not be very different from the original one. Mutation plays a critical role in GA.

Fig. 11

Demonstration of crossover operation in genetic algorithm

In summary, crossover, which is by making the chromosomes in the population alike, leads the population to converge while mutation returns genetic diversity back into the population and assists the search to escape from local optima.

Reproduction involves selection of chromosomes for the next generation. In the most general case, the fitness of an individual determines the probability of its survival for the next generation. There are different selection procedures in GA depending on how the fitness values are used. Proportional selection, ranking, and tournament selection are the most popular selection procedures. The general procedure of GA [93] is given as follows (Fig. 13):

Fig. 12

Demonstration of mutation operation in genetic algorithm

• Step 1 Set t = 1. Randomly generate N solutions to form the first population, P1. Evaluate the fitness of solutions in P1.

• Step 2 Crossover: Generate an offspring population Qt as follows:

• 2.1 Choose two solutions x and y from Pt based on the fitness values.

• 2.2 Using a crossover operator, generate offspring and add them to Qt.

• Step 3 Mutation: Mutate each solution x ∈ Qt with a predefined mutation rate.

• Step 4 Fitness assignment: Evaluate and assign a fitness value to each solution x ∈ Qt based on its objective function value and infeasibility.

• Step 5 Selection: Select N solutions from Qt based on their fitness and copy them to Pt + 1.

• Step 6 If the stopping criterion is satisfied, terminate the search and return to the current population. Otherwise, set t = t + 1 and go to Step 2.

More details about the history, theory and mathematical background, applications, and the current direction of genetic algorithms can be found in [94]

Fig. 13

Genetic algorithm procedures