Feature Selection for Facial Emotion Recognition Using Cosine Similarity-Based Harmony Search Algorithm

Abstract

Nowadays, researchers aim to enhance man-to-machine interactions by making advancements in several domains. Facial emotion recognition (FER) is one such domain in which researchers have made significant progresses. Features for FER can be extracted using several popular methods. However, there may be some redundant/irrelevant features in feature sets. In order to remove those redundant/irrelevant features that do not have any significant impact on classification process, we propose a feature selection (FS) technique called the supervised filter harmony search algorithm (SFHSA) based on cosine similarity and minimal-redundancy maximal-relevance (mRMR). Cosine similarity aims to remove similar features from feature vectors, whereas mRMR was used to determine the feasibility of the optimal feature subsets using Pearson’s correlation coefficient (PCC), which favors the features that have lower correlation values with other features—as well as higher correlation values with the facial expression classes. The algorithm was evaluated on two benchmark FER datasets, namely the Radboud faces database (RaFD) and the Japanese female facial expression (JAFFE). Five different state-of-the-art feature descriptors including uniform local binary pattern (uLBP), horizontal–vertical neighborhood local binary pattern (hvnLBP), Gabor filters, histogram of oriented gradients (HOG) and pyramidal HOG (PHOG) were considered for FS. Obtained results signify that our technique effectively optimized the feature vectors and made notable improvements in overall classification accuracy.

1. Introduction

In recent days, due to the rapid growth of technology, human–computer interaction has begun to gain courtesy in the research domain. Expression of emotions through facial expressions is an important aspect of human communication that serves social interaction. In the work described by Shan et al. in [1], researchers have validated seven universal feature expressions, namely disgust, anger, happiness, sadness, neutral, fear and surprise. According to facial action coding system [2], various positions of the muscles on the face are responsible for the facial expression as explained by Ekman and Rosenberg [3]. The total number of action units in facial action coding system is found to be 46. These actions are associated with defined set of muscle movement [4]. The main use of Facial Emotion Recognition (FER) [5] lies in determining the emotional state of a person which further helps in depicting the mental state identification and mental disorders. Besides, FER also contributes to video indexing and data driven animation [6]. FER system is capable of making human–robot communication much more significant and effective. There are various challenges involved in FER. Predominantly, the camera angle is one of the key issues in data collection part. Wrong camera angle can hamper the recognition accuracy as it can distress the small muscle movements. Besides, there are many types of noises that also affect the minute muscle movements in eyes, nose or other parts of the face.

For a robust FER system, the primary need is to design a competent-feature vector. As the features are extracted from facial images showing different human emotions, in many cases, irrelevant or redundant features are generated. This ultimately stretches the dimension of feature set and brings down the overall accuracy in making predictions. Feature selection (FS)—an initiative to optimize feature dimensions undertaken by various researchers—aims in extricating redundant/irrelevant features that do not make any significant contribution in the overall prediction process [7,8]. FS is a useful way for substantial reduction of the size of original-feature vectors used to predict target facial emotions expressed by humans. This not only reduces computation time required to make the prediction, but also improves final accuracy by removing redundant and irrelevant features. Redundant and noisy features lead to misclassification and higher memory requirements, which, in turn, significantly increase model building time [9]. FS requires choosing a best-feature subset from $2^N$ feature subset combinations, where $N$ is the size of the feature subset [10]. Some of the techniques of FS may prove to be very costly. As a result, in many cases, random search and heuristic search techniques are employed. In [11], an algorithm for unsupervised FS using feature similarity measure, called maximum information compression index was elucidated. Another FS technique was proposed in [12] which uses sequential search methods characterized by dynamically changing the number of features, i.e., “floating” methods. The works described in [13,14,15,16] are some other established applications of FS in solving problems like handwriting recognition, handwritten script classification [17], handwritten numeral classification, etc. Another instance of FS, called histogram-based fuzzy ensemble technique was applied to UCI datasets for evaluation in [18].

FS algorithms are segregated into three categories [19]. The filter method applies a statistical or probabilistic approach to assign a scoring to each feature and these features are selected to be either kept or removed from the original-feature set. The wrapper method is often used in conjunction with a machine learning algorithm (which works as a classifier), where the learning algorithm plays the part of the feature validation process and selects an optimal feature subset which enhances the classification accuracy. The hybrid method tends to perform computationally better than wrapper method. Both wrapper and hybrid methods make selections based on classifier that may or may not work well with any other classifier. That is because the optimal feature subset is built when the classifier is built, and the selection depends on the hypotheses which the classifier makes. In our work, we have focused mainly on the supervised filter-based FS method [20] approach, where the feasibility check for a feature subset is based on a popular statistical measure called Pearson’s correlation coefficient (PCC) [21] and the minimal redundancy maximal relevance (mRMR) [22], which takes into the account of relation of features with other features and also with the classes.

Our adopted technique applies a supervised filter harmony search algorithm (SFHSA) in order to perform FS. Keeping the fundamentals of the algorithm intact, we have made modifications to the pitch adjustment procedure of the algorithm where we have incorporated the concept of cosine similarity for adjusting values of the variables. The details of implementation were elucidated in Section 4. Some applications of HSA on FS was discussed in [23,24,25,26,27,28,29]. The proposed HSA-based FS was applied on a set of on five different state-of-the-art feature descriptors that include uniform local binary pattern (uLBP), horizontal–vertical neighborhood local binary pattern (hvnLBP), Gabor filters, histogram of oriented gradients (HOG) and pyramidal HOG (PHOG). The method is tested on two popular FER datasets, namely Japanese female facial expression (JAFFE) [30] and Radboud faces database (RaFD) [31].

The flow of the paper is arranged as follows: Section 2 describes motivation behind the present work along with some state-of-the-art methods related to FER. Section 3 presents the detailed description of the datasets used. It also explains the five feature descriptors briefly. The proposed optimal feature subset selection method, called SFHSA, was detailed in Section 4. The performance evaluation of the SFHSA is reported in Section 5. Section 6 finally concludes the paper. The supporting codes are uploaded in the Github link: https://github.com/Soumyajit-Saha/Cosine-Similarity-based-Harmony-Search-Algorithm.

2. Motivation and Related Work

High dimensional datasets often impose computational overheads especially in cases where achieving a near perfect classification accuracy score is the primary concern. Hence, researchers focus mainly on reducing the dimensions of feature sets by filtering out irrelevant features that do not have a significant impact on the classification process. The overheads imposed by high dimensional feature sets mainly include time and space overheads [9,10,23]. Nowadays—especially in fields of bioinformatics, medicine and genetics [24], which include features or genes denoted in the form of microarray data of humongous dimension—FS is a necessity in countering the underlying challenges of space and execution time. Many efficient algorithms have been established for this purpose. With an objective to serve the same purpose in FER domain, we have established a FS algorithm to increase the efficiency of the machine learning algorithm and reducing the dimension of the feature set simultaneously. We developed a novel HSA-based FS technique—SFHSA—and applied it on the FER datasets and demonstrated a clear comparison in juxtaposition with other established FS techniques.

In FER, the features are obtained from datasets having a large dimension. However, there are many redundant features present within the feature sets that bring down the classification accuracy. To eradicate those redundant and irrelevant features as well as make the predictions more accurate, we have taken the initiative to apply our proposed FS technique over FER datasets. The features, extracted from facial expression images include uLBP, hvnLBP, Gabor filters, HOG and PHOG. There are few FS works found in the literature on FER. In [25], a wrapper-based approach to FS using genetic algorithm (GA) was applied to features for FER, obtained using log-Gabor filters. A linear programming technique was used for FS in [26], where the features for FER was extracted using Gabor filters. A FS based on random forest classifiers was incorporated in the FER system where visual-appearance based features were extracted from Gabor filter bank in [27]. For the FS, the mRMR method based on mutual information (MI) quotient was used. Li et al. in [28] use the combination of fixed filters and trainable non-linear 2D filters based on the biologic mechanism of shunting inhibition. Finally, FS was performed using MI and class separation scores. The work described in [29] proposes automatic FER where features were generated using methods like Gabor filters, log Gabor filter, LBP operator, higher-order local autocorrelation and higher order local autocorrelation-like features. A self-learning attribute reduction algorithm was proposed for FS in [32] which is based on rough set and domain oriented data-driven data mining. The authors in [33] describe an efficient FS technique, late-hill-climbing-based memetic algorithm (LHCMA) on feature sets for FER, which indeed outperforms many previously established FS algorithms. A facial expression recognition system with hvnLBP-based feature extraction and micro GA—along with the particle swarm optimization (PSO)-based feature optimization technique—was proposed in [34] by Mistry et al. The FER method, based on a wrapper-based FS technique called multi-objective differential evolution algorithm, was proposed in [35]. The evaluation was done using support vector machine (SVM) classifiers. We have seen that HSA has provided satisfactory results in the case of FS for holistic Bangla word recognition [36], digit classification [37], email classification [38], epileptic seizure detection [39] and protein sequence classification [40]. However, to best of our knowledge, to date, the HSA-based FS technique has not been applied to FER systems. This has served as a motivation for us to perform the SFHSA for FER systems.

3. Dataset and Feature Description

This section is divided into two subsections: dataset description and feature extraction. The first subsection deals with the dataset and preprocessing information, whereas the next subsection puts forward a brief description of the feature descriptors used here.

3.1. Dataset Description

Two popular benchmark FER datasets were included in the present work, namely JAFFE and RaFD. Details of those databases and their preprocessing algorithms are demonstrated in the following sections.

3.1.1. JAFFE

The JAFFE dataset [30] includes facial expression of 10 different Japanese females. It consists of 7 basic facial expressions: surprise, angry, happy, disgusted, fearful, sad and neutral. In total, there were 213 images, which result in an unequal number of samples per class. Hence, data augmentation [41] was performed to handle this issue. The augmentation was achieved by introducing Gaussian white noise of constant variance and mean to 11 sample images. In the end, the dataset contained a total of $224 \left(=32\times 7\right)$ images, i.e., 32 images per class. Sample images from the JAFFE dataset are shown in Figure 1.

3.1.2. RaFD

The RaFD facial emotion dataset [31] was constructed from 67 models (consisting of Caucasian males and females, Moroccan Dutch males and Caucasian boys and girls). It consists of 8 various expression classes: disgust, fear, happy, neutral, contempt, surprise, sadness and anger. While capturing each facial expression, five different camera angles and three different gaze directions (frontal, left and right) were used. In this dataset, each class contains 201 images. Figure 2 shows sample images taken from RaFD dataset.

3.1.3. Preprocessing

Facial emotion images generally suffer from various extraneous noises, as they are captured in different environmental conditions. These affect the feature extraction process. A suitable preprocessing technique was required to overcome this problem. Hence, Viola Jones [42] was applied to focus attention on the important region within the whole image. The important region or point of interest was mainly concerned with the areas containing facial expressions such as the lips, eyebrows, nose, eyes, etc. The Viola Jones algorithm returns the coordinates of the bounding box that contains the regions of interest. To make the method more robust and comparable with real-world scenario, the technique was assessed with various image dimensions. In the present work, facial images of three different dimensions that include $32\times 32, 48\times 48 \mathrm{and} 64\times 64$ were considered for the evaluation purpose. After preprocessing, the facial images were resized to their corresponding resolutions. Then, the images were used for the feature extraction. Finally, the extracted features were fed to the proposed feature selection algorithm. The use of three image sizes took into account the variation in image quality in practical applications for emotion recognition.

3.2. Feature Description

In this section, all the feature extraction methods are explained briefly. In the present work, we considered HOG, PHOG, uLBP, hvnLBP and Gabor filter-based feature extraction methodologies.

3.2.1. Histogram of Oriented Gradients

HOG [43] is a texture-based feature descriptor that adopts the histogram of gradients as a statistical measure. The primary idea behind the concept is that any local shape and object can be demonstrated using the gradient intensity distribution or edge direction. As HOG is invariant to geometric transformation, it is widely used in pattern recognition domain. In the computation part, first of all the entire facial image was divided into cells. After that, the gradients were calculated according to Equation (1). In this work, one dimensional gradient was taken. A matrix $\mathrm{M}=\left[\begin{array}{ccc}1& 0& -1\end{array}\right]$ was used to calculate the gradient in the X direction denoted by ${\mathrm{Grad}}_{\mathrm{X}}$. Subsequently, ${\mathrm{M}}^{\mathrm{T}}$ was adopted for gradient in Y direction denoted by ${ \mathrm{Grad}}_{\mathrm{Y}}$. In this work, the matrix $M$ was used as a mask which was passed through the entire image and the variable here was the pixel intensity. The gradient was computed by taking the intensity difference of the neighboring pixels in particular direction. In case of ${\mathrm{Grad}}_{\mathrm{X}}$, the intensity difference along horizontal neighboring pixels was considered. Similarly, for ${\mathrm{Grad}}_{\mathrm{y}}$ the intensity difference along vertical neighboring pixels was considered. The final gradient direction ${\mathrm{Grad}}_{\mathrm{Dir}}$ was calculated as follows:

$${\mathrm{Grad}}_{\mathrm{Dir}}={ \tan }^{-1}\frac{{\mathrm{Grad}}_{\mathrm{Y}}}{{\mathrm{Grad}}_{\mathrm{X}}}$$

(1)

Next, the entire gradient direction domain (lying between $0^{\circ }$ to ${360}^{\circ }$) was divided into 8 histogram bins. Initially, the entire image was divided into cells. For every cell, the bin count for each histogram bins was obtained. We increment the bin count if the value of the gradient direction falls in the range of a particular bin. This way, we achieve the histogram of the gradients for each cell. Finally, those histograms obtained from each cell were concatenated to get the final feature vector. For the extraction of HOG feature, three image dimensions were considered. HOG feature descriptors were applied on each facial emotion image and the final feature vectors were obtained. The feature size of HOG depends on the image size and the cell dimension. The feature dimension for $32\times 32, 48\times 48 \mathrm{and} 64\times 64$ images are 324, 900 and 1764 respectively.

3.2.2. Pyramidal HOG

Researchers have mainly relied on PHOG feature descriptors [44,45] for object recognition. The spatial pyramid representation of HOG is computed here. It mainly sets up the local shape and retains the spatial information by segregating it into various levels. The spatial information was retained by using HOG descriptor in every level. In this work, Canny edge detector [46] was applied to obtain the contour of each region. The edge detector was mainly used to capture the local shapes. An arrangement of minute spatial grids was shaped by doubling the separations in each axis direction for every region. For each resolution level $L=l$, the grid consists of $2^l$ cells along each dimension. For example, at $L=2$, the number of grids along $X axis$ will be $4$. As a result, the total number of grids becomes $4\ast 4=16$. Further, a Sobel mask [47] of window size $3\times 3$ was applied for extracting the orientation of the gradients along the edge contours. At this stage, the procedure of gradient binning was performed similar to HOG descriptor. The gradients corresponding to same cell are quantized and combined into $N$ histogram bins. We increment the bin count if the gradient direction value lies in the range of that particular bin. Mainly, the orientation for binning was performed using either $\left[0^{\circ } \mathrm{to} {360}^{\circ }\right]$ range or $\left[0^{\circ } \mathrm{to} {180}^{\circ }\right]$ range where the contrast sign was neglected [36]. These bins were sorted and concatenated into a single sequence corresponding to the same level. For each level, a histogram will be obtained. Finally, all the histograms corresponding to each level were merged to get the final feature vector. In the present work, we have used the PHOG descriptor to capture facial features from the images (having three different dimensions of $32\times 32, 48\times 48 \mathrm{and} 64\times 64$) of the JAFFE and RaFD datasets. The number of pyramids was kept up to level 3 ($L=3$) and number of histogram bins was 8 ($N=8$). The orientation of the gradient ranges between $0^{\circ } \mathrm{to} {360}^{\circ }.$ The number of features obtained from PHOG can be formulated as: $N\ast \sum _{l\in L}{4}^l$. Putting $L=3 \mathrm{and} N=8$, we obtain final feature vector of size 680.

3.2.3. Gabor Filter

In this work, we have also used the Gabor filter, a well-known frequency-based feature descriptor [48,49]. It was a linear filter that was mainly applied for texture study. The Gabor filter mainly analyzes the presence of an explicit frequency content in a specific direction within a localized region around the point or region of interest in the image. It is also invariant to rotation, translation and scale. Besides, it is also robust towards any kind of photometric disorders, mainly occur as illumination changes and image noise [37]. In the spatial domain, two dimensional Gabor filter includes the sinusoidal plane modulated Gaussian kernel function [50]. Equations of the kernel function for calculating Gabor filter-based features in the spatial domain are given in Equations (2)–(4).

$$Gabor\left(x,y\right)=\frac{f^2}{\pi \gamma \eta }exp\left(-\frac{x^{\prime 2}+{\gamma }^2{y}^{\prime 2}}{2{\sigma }^2}\right)\times exp\left(i. \left(2\pi f{x}^{\prime }+\omega \right)\right)$$

(2)

$$x^{\prime }=xcos\theta +ysin\theta$$

(3)

$$y^{\prime }=-xsin\theta +ycos\theta$$

(4)

Here, the standard deviation of the Gaussian envelope is expressed as $\sigma$. $\gamma$ is the spatial aspect ratio and the ellipticity of the support of the Gabor function. $i$ denotes the imaginary number. Phase offset is specified as $\omega$. $f$ stands for sinusoid frequency and $\theta$ symbolizes the alignment of normal to parallel stripes of the Gabor function. Eight different orientations and five distinct scales were taken in the Gabor model that results in 40 diverse Gabor filters.

Multiple spatial resolutions and orientations were measured from the set of 2D Gabor filter bank. Then, these were used for convolution of each facial image sample. Let us consider a sample facial image, $img\left(x,y\right)$ and the corresponding Gabor filter kernel is ${\Delta }_{u,v}\left(x,y\right)$. The characterization of the output image,$Ou{t}_{u,v}\left(x,y\right)$ is given in Equation (5) [39] as follows:

$$Ou{t}_{u,v}\left(x,y\right)=img\left(x,y\right). {\Delta }_{u,v}\left(x,y\right)$$

(5)

Finally, a down sample by a factor of 8 was performed on the obtained Gabor features. The size of the feature vector varies with the image dimension. In the present work, image dimensions which were taken into account are $32\times 32, 48\times 48 \mathrm{and} 64\times 64$.

Table 1. Estimation of feature vector size of Gabor filter bank (applied in 8 different orientations and 5 distinct scales) for various image dimensions with a down sampling factor of 8.

Image Dimension	Feature Dimension	Final Feature Dimension after down Sampling
$32\times 32$	$\mathrm{40,960} \left(=40\times 32\times 32\right)$	$640\left(=\frac{\mathrm{40,960}}{8\times 8 }\right)$
$48\times 48$	$\mathrm{92,160} \left(=40\times 48\times 48\right)$	$1440\left(=\frac{\mathrm{92,160}}{8\times 8}\right)$
$64\times 64$	$\mathrm{163,840} \left(=40\times 64\times 64\right)$	$2560\left(=\frac{\mathrm{163,840}}{8\times 8} \right)$

describes the feature dimension corresponds to the previously mentioned image size.

3.2.4. Uniform Local Binary Pattern

LBP, which was first introduced by Ojala et al. [51], is a useful texture-based feature. It mainly captures the edge properties by taking the intensity differences of the center pixel with its surrounding pixels. In this work, we have considered a window size of $3\times 3$ around a center pixel. As a result, a total of eight neighboring pixels were considered. The difference between the center pixel and each of the surrounding pixels were calculated. If the difference was greater than zero, we assigned 1; else 0. In this way, each center pixel was represented in a two-digit LBP code. The calculation of the LBP code is shown in Figure 3. In this figure, the top left corner is taken as the $7\mathrm{th}$ bit and the bits were considered in a clockwise fashion until it reaches to the $0\mathrm{th}$ bit. The resultant 8-bit binary number is formed which is then converted to its equivalent decimal number. This process is formulated in Equation (6), where, $(x_{cen},y_{cen})$ is the center pixel and $i_k$ is one of the surrounding pixels.

$$LBP\left(x_{cen},y_{cen}\right)={\displaystyle \sum }_{k=0}^{7}new\left(i_{cen}-i_k\right)2^k$$

(6)

A transition in a binary string is defined as the change from 0 to 1 or change from 1 to 0. The strings that comprise less than or equal to two transitions are known as uniform strings and others are called as non-uniform strings. The main purpose of using uniform pattern was to eliminate the redundant features and capture the information properly. In this work, uniform property was incorporated with the binary strings obtained by LBP. Therefore, most of the redundant binary strings will be eliminated as they were non-uniform. The obtained uniform strings were converted to their respective decimal values. The histogram of those values were taken as the feature vector as formulated in Equation (7), where, $L$ denotes the number of bins.

$$His{t}_n={\displaystyle \sum }_{x,y}I\left(Imag{e}_{label}\left(x,y\right)=i\right), i=0,\dots , L-1$$

(7)

In this work, initially all the images were divided into $16 \left(=4\times 4\right)$ blocks irrespective of the image dimension. The main purpose of the blocking was to preserve the local information of the image. Divisions were made uniformly in the entire image. Then uLBP was applied in each sub blocks. The obtained feature vectors from each sub block were concatenated to get the final feature vector. Here, the feature dimension of uLBP was $59 \left(=8\ast \left(8-1\right)+3\right)$. As the image was divided in 16 sub blocks, the final feature dimension becomes $944$.

3.2.5. Horizontal vertical Neighborhood Local Binary Pattern

hvnLBP is a very useful texture feature which was first proposed by Mistry et al. [34]. It acquires better contrast information among the neighborhood pixels such as edges and corners. Similarly, as uLBP, a $3\times 3$ window was considered for hvnLBP that contains eight neighboring pixels. The surrounding pixels are denoted as $L=\left\{l_0,l_1,l_2,l_3,l_4,l_5,l_6,l_7\right\}$. In case of hvnLBP, the comparison was done among the neighboring pixels. By comparing those surrounding values, we get a binary string of 8 bits which was further converted to its equivalent decimal value. The process is formulated in Equations (8) and (9). An example for calculating hvnLBP is also shown in Figure 4.

$$\begin{gathered} hvnLBP\left(x,y\right)=\{ fun\left(\max \left(l_0,l_1,l_2\right)\right),\hspace{1em}fun\left(\max \left(l_7,l_3\right)\right), \\ fun\left(\max \left(l_6,l_5,l_4\right)\right),\hspace{1em}fun\left(\max \left(l_0,l_7,l_6\right)\right), \\ fun\left(\max \left(l_1,l_5\right)\right), fun\left(\max \left(l_2,l_3,l_4\right)\right) \} \end{gathered}$$

(8)

$$C\left(\max \left(l_a,l_b,l_c\right)\right)=\{\begin{array}{cc}1& if maximum\\ 0& otherwise\end{array}$$

(9)

In Equation (9), the $l_b$ can also be absent. In that case, only two-pixel intensities will participate in the comparison.

As the binary string consists of 8 bits, the total number of possible values are 256. After obtaining the equivalent decimal values, the histogram was considered as described in Equation (7).

For generating discriminative facial representation, hvnLBP was combined with the 2-D Gabor filter. In this work, a total of 16 magnitude images of various wavelengths and orientations from Gabor filter were taken at the beginning. Finally, hvnLBP was applied on those 16 magnitude images. The feature vector obtained from each magnitude images were concatenated to get hold of the final feature vector. In a typical hvnLBP, there were total 256 features. As there were 16 magnitude images, the final feature dimension becomes $4096$.

4. Proposed Work

In this paper, we have proposed a FS technique based on HSA, which we have named as supervised filter harmony search algorithm (SFHSA). SFHSA uses cosine similarity and mRMR combined with PCC. HSA has proved to be an efficient technique [52,53,54,55] for providing an optimal solution to real life problems, in terms of feasible computation time and usage of memory, as proposed by Lee and Geem in [53]. It simulates the procedure opted by the musicians to find out the finest tune by selecting a particular combination of frequencies produced by sundry musical instruments. It optimizes an objective function by selecting an appropriate combination of solutions from existing set of solutions by employing random search. In [56], an improved version of HSA has been formulated which uses fine-tuning parameters of mathematical techniques to enhance the performance of HSA. Due to the efficient nature of HSA in finding the global optimum solution, it has been exploited in various works related to FS. The traditional approach of HSA involves adjustment of the pitch considering a parameter named Bandwidth (BW). In this phase of the algorithm, we have made a modification in adjustment of the pitch. In our case it was the adjustment of the features by replacing their values with the values of their cosine similar features subject to satisfaction of certain conditions rather than selecting adjacent features as in conventional way. The goodness of a subset was determined using mRMR.

The proposed algorithm is a supervised filter method. As mentioned before, the proposed algorithm is used to optimize the features extracted for FER; the objective of SFHSA is to reduce the feature dimensions while maintaining or increasing the accuracy score. Prior to execution of the algorithm, each feature vector was divided into training and testing sets in the ratio of 2:1. The training set was then used to find the optimal feature subset. The algorithm used is provided in Algorithm 1.

Algorithm 1 Selection of Optimal Feature Subset using SFHSA

1: Input: Original-feature set 2: Output: Reduced-feature subset 3: User defined parameters: HMS = 15, HMCR = 0.8 and PAR = 0.5 4: Determine the worst feature subset from HM 5: while($t<max\_iterations$) { 6:   while($i<=HMS$) { 7:  while($j<=total\_no\_of\_features\_in\_subset$) { 8:   Generate random value for P1 in $\left[0,1\right]$ 9:   if($P1<HMCR$) { 10:    Choose a feature $f_j$ from the subset 11:    Generate a random value for P2 in $\left[0,1\right]$ 12:    if($P2<PAR$) { 13:     Generate a random value for $\mathsf{\epsilon }$ in $\left[-1,1\right]$ 14:     Randomly choose a feature $f_k$ from subset such that cosine similarity $\left(f_j,f_k\right)$ is in $\left(-\mathsf{\epsilon },\mathsf{\epsilon }\right)$ 15:    } 16:   } 17:   else { 18:    Select any feature $f_r$ randomly from the original-feature set 19:   } 20:   $j=j+1$ 21:  } 22:  if( mRMR value of new subset >mRMR value of worst subset) { 23:   Replace worst subset with new 24:   Find the worst feature subset in the updated HM 25:  } 26:  $i=i+1$ 27: } 28: $t=t+1$ 29: }

The parameters that we have used include:

• Harmony Memory Size(HMS)
• Harmony Memory Consideration Rate (HMCR)
• Pitch Adjustment Rate (PAR)
• Number of Iterations

The HMS determines the size of the harmony memory (HM), i.e., the number of feature subsets present in the HM. Throughout the process, HMCR was applied in order to decide whether the feature to be selected from a feature subset in the HM or not. Its value is in the range of $\left[0,1\right]$ and based on experiments we have initialized it to 0.8. The parameter PAR was used (upon satisfaction of condition) to select a feature randomly whose cosine similarity measure with the pre-selected feature is within the range (−ɛ, ɛ), where ɛ is a randomly generated value of range $\left[-1, 1\right]$. Its value lies in the range $\left[0, 1\right]$ and we have fixed it to 0.5 on an experimental basis, in order to set the probability of PAR being satisfied equal to the probability of PAR not being satisfied. At the commencement of SFHSA, all the parameter values were initialized. The value of HMS was set to 15 in order to keep 15 feature subsets in HM and to have more diverse feature subsets for obtaining better results and the maximum number of iterations was set to 20. The initialization phase was followed by the random selection of feature subsets from the training set. In this phase, we have randomly created $m$ feature subsets, where $m=HMS$. The feature subsets were considered to have a dimension ranging from 80% to 90% of the dimension $n$ of the actual feature set, which is to be reduced in the subsequent phases. These $m$ feature subsets were used to populate the HM initially.

To find the cosine similarity of the features, the each of the feature sets was normalized. The normalization was performed column wise for each feature present in the feature set. The normalized value, $N\left(x\right)$ was calculated using Equation (10), where $x$ is an attribute, $curr\left(x\right)$ denotes the value of an attribute corresponding to the current instance; and $min\left(x\right)$ and $max\left(x\right)$ denote the minimum and maximum values of an attribute, respectively, corresponding to all the instances.

$$N\left(x\right)=\frac{curr\left(x\right)-min\left(x\right)}{max\left(x\right)-min\left(x\right)}$$

(10)

Thereafter, the features that had maximum and minimum values equal for all instances were filtered out as they were not considered to have significant contribution in the classification stage. The normalized values of the features were utilized to find out the cosine similarity [57] between each of two features from the feature set. Equation (11) measures the similarity between features $p$ and $q$, where they represent any two features from a feature set, $p_i$ and $q_i$ denote the values for features $p$ and $q$ respectively, corresponding to $i\mathrm{th}$ instance and $n$ denotes the total number of instances. The values of the cosine similarities were stored in the form of a matrix for faster computation.

$$similarity\left(p,q\right)=cos\theta =\frac{p.q}{\left|\left|p\right|\left|q\right|\right|}=\frac{{\sum }_{i=1}^n{p}_i{q}_i}{\sqrt{{\sum }_{i=1}^n{p}_i^2{\sum }_{i=1}^n{q}_i^2}}$$

(11)

For instance, let there be a feature vector $F=\{f_1, f_2, f_3,\dots f_n$}, where $f_1, f_2, f_3, \dots f_n$ are the features in a feature set with dimension $n$, then the cosine similarity matrix is represented as given in Equation (12), where $cos{\theta }_{a,b}$ is the cosine similarity between $a\mathrm{th}$ and $b\mathrm{th}$ feature and $a,b\le n$.

$$S_{n\times n}=\left(\begin{array}{ccc}cos{\theta }_{1, 1}& \cdots & cos{\theta }_{1, n}\\ ⋮& \ddots & ⋮\\ cos{\theta }_{n, 1}& \cdots & cos{\theta }_{n, n}\end{array}\right)$$

(12)

For each feature subset, a new feature subset was created using HMCR and PAR. Either a feature is selected from the feature subset in HM (based on cosine similarity value) or a random feature is selected from the existing feature set. For example, let there be a feature subset $S=\left\{f_2,f_8,f_6,f_5,f_3\right\}$ in HM, having a combination of features from the existing feature set say $F=\left\{f_1,f_2,f_3,f_4,f_5,f_{6, }{f}_7,f_8,f_{9,}{f}_{10}\right\}$. If the HMCR is satisfied in an iteration, we select a feature, say $f_5$. Again, if PAR is satisfied then the feature $f_5$ is replaced by selecting a cosine similar feature, say $f_2$ and removing $f_5$. Another option is to select a complete random feature, say $f_7$ from the global feature set, if neither of the condition is satisfied. Suppose $f_2$ has been used to replace $f_5$ and for another feature say $f_3$, again $f_2$ is found to be its cosine similar feature. Therefore, in this case both $f_5$ and $f_3$ are replaced by a single feature $f_2$; thus, reducing the feature dimension. This selection of features for replacement has been done n times to improvise the new feature subset from the existing feature subset in HM, where $n$ is the dimension of $S$ and the improvisation of the new feature subset carries on maximum number of iteration times. Thus, in each iteration a new improvised feature subset (with reduced or same dimension) was generated. The selection method of an optimal feature subset using SFHSA is explained in Algorithm 1 whereas the flowchart of the same is provided in Figure 5.

A decision is then made, whether the improvised subset is better in quality than the previous feature subset. For determining the quality of the feature subsets, mRMR [58] was applied. In evaluation of the feature subsets, mRMR has proved to be an efficient technique [33,59,60,61]. The concept of mRMR involves maximizing the Relevancy-R (PCC [62] between the feature and the class) and minimizing the Redundancy-D (PCC between the features in the subset). The features which were highly correlated with the class were considered to be relevant and the features those were highly correlated with each other were considered to be redundant. The calculation of PCC is performed using Equation (13), where $x$ and $y$ were two sets of values and $\overline{x}$ and $\overline{y}$ are their expectation values respectively. The values of both $R$ and $D$ were calculated using Equations (14) and (15) respectively. $x_i$ and $x_j$ represent features from the feature subset $S$ and $c$ represents the facial class label.

$$PCC\left(x,y\right)=\frac{\sum \left(x-\overline{x}\right)\left(y-\overline{y}\right)}{\sqrt{\sum {\left(x-\overline{x}\right)}^2{\left(y-\overline{y}\right)}^2}}$$

(13)

$$R=\frac{1}{\left|S\right|}{\displaystyle \sum }_{x_i\in S}PCC\left(x_{i,}c\right)$$

(14)

$$D=\frac{1}{\left|S^2\right|}{\displaystyle \sum }_{x_i, x_j\in S}PCC\left(x_{i,}{x}_j\right)$$

(15)

The value of mRMR is calculated based on finding out the quality score of the feature subset defined by $V\left(S\right)$, defined in Equation (16) as follows:

$$V\left(S\right)=R-D$$

(16)

During the decision-making process, if the quality score of the newly generated feature subset is found to be better than the worst subset in HM, it replaces the worst subset and again the worst subset is found out from the updated HM. The entire process is iterated until the stopping criterion is met. The finishing stage of the process includes calculating the accuracy score using a classifier and generating the results which is discussed in the next section. Therefore, the proposed HSA is a filter-based FS method based on mRMR. The use of PCC in mRMR makes the algorithm quite effective in selecting the best subsets. The extracted feature descriptors were refined using SFHSA, and the final selected features were chosen from the testing feature sets and passed through a classifier to find the recognition accuracy of the classification problem under consideration (here, FER). A schematic diagram of the proposed model is shown in Figure 6.

5. Results and Discussion

The SFHSA-based FS technique was applied to the features obtained from the images of two standard FER datasets, namely JAFFE and RaFD. The five feature sets obtained include uLBP, hvnLBP, Gabor-filter-based, HOG, PHOG features. We considered the facial images of three dimensions: $32\times 32, 48\times 48 \mathrm{and} 64\times 64$. As a result, for the present FS problem, the total number of feature sets taken under consideration becomes 30 ($=2\times 5\times 3$), 2 FER datasets, 5 feature sets and facial images of 3 dimensions.

Table 2. Tabular representation of the details of original-feature size, reduced-feature size, accuracy score and their corresponding percentage of reduced-feature size with respect to original-feature size and change in accuracy score of 5 different feature vectors extracted from facial images of 3 different dimensions for two popular FER datasets.

Feature Descriptor	Dataset	Dimension of Facial Images	Size of the Original-Feature Vector	Accuracy Score (%)	Size of the Optimal Feature Vector Obtained by SFHSA [Reduction in %]	Accuracy Score (%)[Change in %]
uLBP	JAFFE	32 × 32	944	62.34	570[60.38%]	78.95[+16.61%]
		48 × 48		62.34	339[35.91%]	75.32[+12.98%]
		64 × 64		59.74	541[57.31%]	74.13[+14.39%]
	RaFD	32 × 32		83.58	608[64.41%]	87.75[+4.17%]
		48 × 48		88.62	445[47.14%]	85.48[−3.14%]
		64 × 64		86.38	573[60.70%]	92.16[+5.78%]
hvnLBP	JAFFE	32 × 32	4096	57.14	1380[33.69%]	67.41[+10.27%]
		48 × 48		46.75	1416[34.57%]	55.84[+12.09%]
		64 × 64		44.16	1433[34.99%]	67.42[+23.26%]
	RaFD	32 × 32		66.42	1513[36.94%]	72.39[+5.97%]
		48 × 48		74.07	1493[36.45%]	75.74[+1.37%]
		64 × 64		69.4	1494[36.47%]	75.81[+6.41%]
Gabor	JAFFE	32 × 32	640	67.53	197[30.78%]	81.82[+14.29%]
		48 × 48	1440	72.73	560[38.89%]	92.21[+19.48%]
		64 × 64	2560	71.43	818[31.95%]	90.91[+19.48%]
	RaFD	32 × 32	640	90.49	241[37.66%]	91.91[+1.42%]
		48 × 48	1440	95.71	341[23.68%]	96.51[+0.8%]
		64 × 64	2560	98.51	462[18.04%]	97.79[−0.72%]
HOG	JAFFE	32 × 32	324	71.43	189[58.33%]	87.94[+16.51%]
		48 × 48	900	74.03	403[44.78%]	92.21[+18.18%]
		64 × 64	1764	71.43	1411[79.99%]	85.71[+14.28%]
	RaFD	32 × 32	324	88.43	205[63.27%]	89.15[+0.72%]
		48 × 48	900	94.22	385[42.78%]	95.40[+1.18%]
		64 × 64	1764	93.66	544[30.83%]	96.32[+2.66%]
PHOG	JAFFE	32 × 32	680	53.25	321[47.21%]	76.32[+20.07%]
		48 × 48		66.23	408[60.00%]	85.27[+19.04%]
		64 × 64		59.74	409[60.15%]	84.39[+24.65%]
	RaFD	32 × 32		78.54	429[63.09%]	85.01[+6.17%]
		48 × 48		85.45	271[39.85%]	89.03[+3.58%]
		64 × 64		88.81	300[44.12%]	88.30[−0.51%]

represents the measurement of sizes of the 5 feature vectors produced using 3 different dimensions of the facial images along with its recognition accuracy. This table also highlights the size of reduced-feature vector with recognition accuracy from two standard FER datasets.

As mentioned previously, each feature set was segregated into training and testing sets with ratio of 2:1 and SFHSA was applied on the training set only. The testing set was used to get the accuracy score by only selecting the attributes (feature indices) that were present in the reduced version of training feature set, defined as follows:

$$Accuracy score(\%)=\frac{\#facial images successfully recognized}{\#total faical images in the test set}\times 100$$

(17)

The initial value of HM consists of HMS (=15), which is randomly generated feature subsets having dimensions ranging from 80% to 90% of the original-feature vector. The detail evaluation on the reduced-feature subsets obtained as a result of using SFHSA on uLBP, hvnLBP, Gabor-filter-based, HOG, PHOG feature descriptors and their corresponding accuracy score were presented in

Table 3. Performance of SFHSA with respect to No FS, SA, GA, MA, ME-BPSO, WAO-CM and LHCMA for uLBP features.

Dataset	Image Size	No FS		SA		GA		MA
		Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)
JAFFE	32 × 32	944	62.34	481	37.66	661	71.43	542	76.62
	48 × 48		62.34	487	45.45	678	79.22	672	76.62
	64 × 64		59.74	493	45.45	611	70.13	703	71.43
RAFD	32 × 32		83.58	485	69.40	791	86.38	716	86.75
	48 × 48		88.62	441	78.73	696	92.16	696	91.47
	64 × 64		86.38	487	76.49	677	90.11	554	91.42
Dataset	ME-BPSO			WAO-CM		LHCMA		SFHSA
	Feature Dimension		Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)
JAFFE	576		64.93	911	61.04	594	76.62	570	78.95
	472		62.34	670	74.03	574	76.62	339	75.32
	603		72.34	869	63.64	570	74.03	541	74.13
RAFD	620		78.17	915	82.28	600	87.13	608	87.75
	533		84.89	883	92.72	552	91.61	445	85.48
	638		83.40	821	86.01	555	90.30	573	92.16

,

Table 4. Performance of SFHSA with respect to No FS, SA, GA, MA, ME-BPSO, WAO-CM and LHCMA for hvnLBP features.

Dataset	Image Size	No FS		SA		GA		MA
		Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)
JAFFE	32 × 32	4096	57.14	2059	51.95	2613	70.13	2232	70.13
	48 × 48		46.75	2011	42.86	2284	61.04	2451	58.44
	64 × 64		44.16	2132	38.96	2254	57.14	2208	58.44
RAFD	32 × 32		66.42	2024	61.19	2721	70.15	2081	72.01
	48 × 48		74.07	2049	68.10	2580	75.00	2584	76.49
	64 × 64		69.4	2026	64.18	2457	72.76	2090	74.44
Dataset	ME-BPSO			WAO-CM		LHCMA		SFHSA
	Feature Dimension		Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)
JAFFE	2118		61.04	3894	61.04	2235	72.73	1380	67.41
	1975		50.65	2921	51.95	2158	63.64	1416	55.84
	2595		50.65	1469	54.55	2060	58.44	1433	67.42
RAFD	2758		66.98	3772	70.71	2211	70.34	1513	72.39
	2615		72.57	3489	77.61	2279	75.19	1493	75.74
	2584		69.22	3914	75.56	2383	73.69	1494	75.81

,

Table 5. Performance of SFHSA with respect to No FS, SA, GA, MA, ME-BPSO, WAO-CM and LHCMA for Gabor filter-based features.

Dataset	Image Size	No FS		SA		GA		MA
		Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)
JAFFE	32 × 32	640	67.53	323	66.23	375	79.22	377	84.42
	48 × 48	1440	72.73	704	68.83	910	80.52	836	84.42
	64 × 64	2560	71.43	1293	71.43	1541	81.82	1408	83.12
RAFD	32 × 32	640	90.49	320	83.21	400	93.28	429	94.03
	48 × 48	1440	95.71	683	91.60	851	98.32	894	98.88
	64 × 64	2560	98.51	1333	95.90	1613	98.32	1414	98.75
Dataset	ME-BPSO			WAO-CM		LHCMA		SFHSA
	Feature Dimension		Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension		Accuracy Score (%)
JAFFE	301		84.03	217	79.22	319	84.42	197		81.82
	701		84.42	630	83.12	767	83.12	560		92.21
	1557		83.12	419	89.61	1428	83.12	818		90.91
RAFD	344		92.36	557	93.74	337	94.59	241		91.91
	770		95.52	1074	96.46	758	98.88	341		96.51
	1300		98.13	1186	97.01	1271	99.25	462		97.79

,

Table 6. Performance of SFHSA with respect to No FS, SA, GA, MA, ME-BPSO, WAO-CM and LHCMA for HOG features.

Dataset	Image Size	No FS		SA		GA		MA
		Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)
JAFFE	32 × 32	324	71.43	178	70.13	195	85.71	206	83.12
	48 × 48	900	74.03	444	67.53	530	89.61	507	87.01
	64 × 64	1764	71.43	887	58.44	1097	80.52	1105	83.12
RAFD	32 × 32	324	88.43	143	85.74	186	92.16	167	92.16
	48 × 48	900	94.22	538	91.54	480	97.01	390	97.01
	64 × 64	1764	93.66	816	92.66	867	96.27	675	96.27
Dataset	ME-BPSO			WAO-CM		LHCMA		SFHSA
	Feature Dimension		Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)
JAFFE	195		82.32	281	81.03	182	87.01	189	87.94
	440		85.32	371	88.22	482	89.61	403	92.21
	923		82.12	1446	81.62	1008	83.12	1411	85.71
RAFD	211		85.07	295	86.94	160	92.35	205	89.15
	530		93.91	605	93.47	455	97.20	385	95.40
	1039		95.15	1041	94.96	800	97.57	544	96.32

and

Table 7. Performance of SFHSA with respect to No FS, SA, GA, MA, ME-BPSO, WAO-CM and LHCMA for PHOG features.

Dataset	Image Size	No FS		SA		GA		MA
		Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)
JAFFE	32 × 32	680	53.25	357	46.75	419	64.94	374	68.83
	48 × 48		66.23	344	58.44	396	81.82	405	80.52
	64 × 64		59.74	342	62.34	423	79.22	412	79.22
RAFD	32 × 32		78.54	351	75.19	489	82.46	416	84.14
	48 × 48		85.45	354	84.89	398	90.49	344	91.98
	64 × 64		88.81	366	87.87	411	91.23	364	93.84
Dataset	ME-BPSO			WAO-CM		LHCMA		SFHSA
	Feature Dimension		Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)	Feature Dimension	Accuracy Score (%)
JAFFE	361		63.64	633	59.74	359	72.73	321	76.32
	440		71.43	441	67.53	373	80.52	408	85.27
	334		74.03	178	71.43	374	81.82	409	84.39
RAFD	420		78.92	491	80.41	396	83.21	429	85.01
	391		87.87	541	90.11	331	91.04	271	89.03
	352		90.67	503	92.35	395	93.10	300	88.30

respectively. We have used the sequential minimal optimization (SMO) classifier with linear kernel [33] to evaluate the recognition performance on the reduced-feature sets. This was done with an aim to achieve higher accuracy scores and also to make convenient to compare with the past experimental results. In Table 3, detailed outcomes for different FS techniques on feature sets extracted using uLBP method were provided in terms of both reduced-feature dimensions and recognition accuracy. Table 4, Table 5, Table 6 and Table 7 show the similar comparison of the different FS techniques on feature sets extracted using hvnLBP, Gabor-filter-based, HOG and PHOG feature vectors respectively.

In case of uLBP, hvnLBP, Gabor-filter-based, HOG and PHOG features, our proposed SFHSA algorithm produces reduced-feature sets which were 65%, 67%, 82%, 69% and 60% less than the original-feature vectors, respectively, and also increased the recognition accuracies up to 17%, 24%, 20%, 18% and 25%, respectively, than the original ones. Thus, it can be concluded that SFHSA demonstrates the best performance in case of PHOG features in terms of accuracy score and dimension reduction for both JAFFE and RAFD datasets. Table 6 reflects the detailed outcomes after applying different optimization techniques on PHOG features. The content of the HM was presented to provide a better understanding of reduced-feature subsets obtained at the end of execution of algorithm. This shows how the HM appears after the SFHSA was executed on feature vectors. In this regard, it is worth mentioning that for all other feature vectors, we have obtained similar content in the HM.

Table 3, Table 4, Table 5, Table 6 and Table 7 also highlight the detailed comparative results observed in the present experiment with some other standard optimization algorithms such as simulated annealing (SA), GA, memetic algorithm (MA), mutation enhanced binary particle swarm optimization (ME-BPSO) [63], whale optimization algorithm–crossover mutation (WOA-CM) [64] and LHCMA [33]. The achieved reduced-feature sets along with highest accuracy were marked in bold in the Table Analyzing the observed outcomes, it can be said that our SFHSA-based FS technique has surpassed the above mentioned techniques. It can be observed that there was a significant increment in accuracy score as compared to previous techniques. Out of 30 cases, our proposed technique achieves finer results in 16 cases as compared to all other techniques (2 cases in Gabor-filter-based feature vectors (up to 8% increment), 3 cases of HOG feature vectors (up to 2.60% increment), 4 cases of PHOG feature vectors (up to 3.60% increment), 3 cases of hvnLBP feature vectors (up to 9% increment) and in 4 cases of uLBP feature vectors (up to 2.40% increment). Thus, it is evident that our technique of feature optimization reduced feature dimensionality and improved recognition accuracy. Proposed SFHSA performs quite well against wrapper-based algorithms. This supports the effectiveness of the proposed method. It was observed from the preceding experiment that the second and third best performing algorithms (in terms of accuracy scores) were LHCMA, WAO-CM. Therefore, we have also presented the performance comparison of these techniques with our proposed SFHSA (which achieves the highest accuracy score) in terms of three well-known statistical parameters, namely precision, recall and F-measure. This is done in order to enhance the clarity of the comparison.

Table 8. Performance of SFHSA with respect to WAO-CM and LHCMA for uLBP features in terms of Precision, Recall and F-measure.

		WAO-CM			LHCMA			SFHSA
Dataset	Image Size	Precision	Recall	F-Measure	PRECISION	Recall	F-Measure	Precision	Recall	F-Measure
JAFFE	32 × 32	0.613	0.610	0.611	0.767	0.766	0.766	0.797	0.789	0.791
	48 × 48	0.738	0.740	0.739	0.765	0.766	0.765	0.757	0.753	0.753
	64 × 64	0.634	0.636	0.635	0.743	0.740	0.741	0.750	0.741	0.742
RaFD	32 × 32	0.826	0.823	0.825	0.870	0.871	0.871	0.875	0.878	0.877
	48 × 48	0.930	0.927	0.928	0.918	0.916	0.917	0.853	0.855	0.853
	64 × 64	0.857	0.860	0.859	0.906	0.903	0.905	0.928	0.921	0.923

,

Table 9. Performance of SFHSA with respect to WAO-CM and LHCMA for hvnLBP features in terms of Precision, Recall and F-measure.

		WAO-CM			LHCMA			SFHSA
Dataset	Image Size	Precision	Recall	F-Measure	Precision	Recall	F-Measure	Precision	Recall	F-Measure
JAFFE	32 × 32	0.609	0.610	0.609	0.725	0.727	0.727	0.680	0.674	0.675
	48 × 48	0.523	0.520	0.521	0.635	0.636	0.635	0.561	0.558	0.559
	64 × 64	0.548	0.546	0.546	0.587	0.584	0.585	0.667	0.674	0.672
RaFD	32 × 32	0.710	0.707	0.709	0.708	0.703	0.706	0.721	0.724	0.723
	48 × 48	0.773	0.776	0.775	0.748	0.752	0.749	0.766	0.757	0.760
	64 × 64	0.757	0.756	0.756	0.740	0.737	0.737	0.757	0.758	0.757

,

Table 10. Performance of SFHSA with respect to WAO-CM and LHCMA for Gabor-based features in terms of Precision, Recall and F-measure.

		WAO-CM			LHCMA			SFHSA
Dataset	Image Size	Precision	Recall	F-Measure	Precision	Recall	F-Measure	Precision	Recall	F-Measure
JAFFE	32 × 32	0.797	0.792	0.794	0.847	0.844	0.845	0.821	0.818	0.819
	48 × 48	0.827	0.831	0.831	0.836	0.831	0.833	0.927	0.922	0.924
	64 × 64	0.898	0.896	0.897	0.827	0.831	0.830	0.913	0.909	0.910
RaFD	32 × 32	0.939	0.937	0.938	0.949	0.946	0.946	0.923	0.919	0.920
	48 × 48	0.971	0.965	0.968	0.991	0.989	0.989	0.973	0.965	0.966
	64 × 64	0.968	0.970	0.969	0.989	0.992	0.990	0.986	0.978	0.982

,

Table 11. Performance of SFHSA with respect to WAO-CM and LHCMA for HOG features in terms of Precision, Recall and F-measure.

		WAO-CM			LHCMA			SFHSA
Dataset	Image Size	Precision	Recall	F-Measure	Precision	Recall	F-Measure	Precision	Recall	F-Measure
JAFFE	32 × 32	0.808	0.810	0.809	0.864	0.870	0.868	0.875	0.879	0.876
	48 × 48	0.886	0.882	0.883	0.898	0.896	0.897	0.929	0.922	0.924
	64 × 64	0.817	0.816	0.816	0.835	0.831	0.832	0.863	0.857	0.859
RaFD	32 × 32	0.865	0.869	0.868	0.926	0.924	0.924	0.894	0.891	0.892
	48 × 48	0.938	0.935	0.937	0.970	0.972	0.971	0.960	0.954	0.956
	64 × 64	0.953	0.950	0.951	0.973	0.976	0.974	0.961	0.963	0.962

and

Table 12. Performance of SFHSA with respect to WAO-CM and LHCMA for PHOG features in terms of Precision, Recall and F-measure.

		WAO-CM			LHCMA			SFHSA
Dataset	Image Size	Precision	Recall	F-Measure	Precision	Recall	F-Measure	Precision	Recall	F-Measure
JAFFE	32 × 32	0.601	0.597	0.599	0.725	0.727	0.726	0.762	0.763	0.762
	48 × 48	0.672	0.675	0.674	0.802	0.805	0.803	0.856	0.853	0.853
	64 × 64	0.718	0.714	0.716	0.820	0.818	0.818	0.845	0.844	0.844
RaFD	32 × 32	0.806	0.804	0.804	0.835	0.832	0.834	0.861	0.850	0.851
	48 × 48	0.900	0.901	0.900	0.912	0.910	0.911	0.894	0.890	0.892
	64 × 64	0.927	0.924	0.925	0.937	0.931	0.933	0.885	0.883	0.883

present the performance comparison of SFHSA with respect to LHCMA and WAO-CM for uLBP, hvnLBP, Gabor filter, HOG and PHOG features respectively. The comparison demonstrates that SFHSA has outperformed the rest two techniques in 16 out of 30 cases, from which a very high capability of FS of our technique can be inferred.

For each feature vector, we have presented the best reduced-feature set in term of both achieved recognition accuracy and reduced dimensionality out of all the HMS (=15) feature subsets in HM. Since, we have obtained the best results for PHOG features, considering facial images of dimension 32 × 32, so through the visual presentations of Figure 7 and Figure 8, we have shown the different reduced-feature sets with recognition accuracies for both JAFFE and RAFD databases respectively. In Figure 7, different colors were used to denote the dimension (number of features) of the reduced-feature sets obtained as a result of applying SFHSA on PHOG features extracted from images of dimension 32 × 32 in JAFFE database and the corresponding recognition accuracies were presented on the top of each bar, which represents a specific reduced-feature set in HM. Similarly in Figure 8, distinguishable colors were used to specify the dimension (number of features) of the reduced-feature sets in HM obtained as a result of applying SFHSA on PHOG features extracted from images of dimension 32 × 32 in RaFD database along with corresponding recognition accuracies, represented in similar fashion as in Figure 7.

6. Conclusions

In this paper, we have focused our attention in reducing the dimension of the feature sets obtained from facial expression images extracted using five feature descriptors—uLBP, hvnLBP, Gabor filters, HOG and PHOG. The evaluation of the proposed methodology, called SFHSA, is done on two benchmark FER datasets, namely RaFD and JAFFE. The proposed algorithm was applied on the extracted feature sets and reduced-feature subsets were obtained with higher accuracy score. It is evident from the results that our FS technique has effectively filtered out redundant/irrelevant features and also outperformed many existing FS techniques like SA, GA and MA. Following the primary backbone of traditional HSA, we have proposed filter variant of the HSA. Cosine similarity was used for the purpose of adjustment of features. On the other hand, both the mRMR and PCC values were used as a way for determining the feasibility of the optimal feature subsets. The performance of the SFHSA was evaluated using SMO classifier. The comparison presented has enabled us to conclude that our algorithm can be applied for FS in domains where the curse of dimensionality puts forth a concerning challenge to the researchers.