Hybrid Malware Classification Method Using Segmentation-Based Fractal Texture Analysis and Deep Convolution Neural Network Features

Abstract

As the number of internet users increases so does the number of malicious attacks using malware. The detection of malicious code is becoming critical, and the existing approaches need to be improved. Here, we propose a feature fusion method to combine the features extracted from pre-trained AlexNet and Inception-v3 deep neural networks with features attained using segmentation-based fractal texture analysis (SFTA) of images representing the malware code. In this work, we use distinctive pre-trained models (AlexNet and Inception-V3) for feature extraction. The purpose of deep convolutional neural network (CNN) feature extraction from two models is to improve the malware classifier accuracy, because both models have characteristics and qualities to extract different features. This technique produces a fusion of features to build a multimodal representation of malicious code that can be used to classify the grayscale images, separating the malware into 25 malware classes. The features that are extracted from malware images are then classified using different variants of support vector machine (SVM), k-nearest neighbor (KNN), decision tree (DT), and other classifiers. To improve the classification results, we also adopted data augmentation based on affine image transforms. The presented method is evaluated on a Malimg malware image dataset, achieving an accuracy of 99.3%, which makes it the best among the competing approaches.

1. Introduction

Malware programs are undesirable harmful threats that are intended to damage the security of a computer system. Malware detection has become an essential concern in the cybersecurity community since malware is capable of causing excessive loss and harm to computer security. Every day, a huge amount of malware is generated intentionally. A recent Symantec report in 2019 [1] demonstrated that malware is growing by 36% annually and the total samples of malware are estimated to be beyond 430 million. The rapid growth of malware causes an extensive threat in our daily life. Ransomware locks or encodes the files on a user’s device and asks for a payment to restore them, which may lead to large productivity loses [2]. Data breaches due to malware activity often incur huge financial losses for major corporations [3]. Trojans and spyware are used in cyber espionage resulting in damage of geopolitical and international relations [4]. Malware has become a primary concern for computer network [5] and smartphone users [6].

To counter these threats, deep learning, a technique based on artificial neural networks (ANN), can be employed successfully [7,8]. Deep learning with its multilayer architecture has a great ability to learn the features of the labeled and unlabeled data. However, when using a deep learning model, we need to train the model on a large data set every time, which takes a lot of time and computational resources. To overcome this learning difficulty, we can use pre-trained deep learning network architectures and extract features to build the classification model [9]. This motivates us to develop pre-trained network-based deep learning architecture for malware class identification and classification.

In this paper, we utilize binary files of malware as black-and-white (grayscale) images, which are generated using the byte code information and later used for malware classification. Visualizations of malware [10,11] in such form prove to be efficient in getting a complete view structure of images.

The conversion of textual data into image form is one of the major problems of malware detection [12]. In parallel, most of the malware detection methods adopt machine learning techniques by analyzing the different features within the malicious codes. These techniques generally experience two major problems: (1) extraction of robust and efficient features effectively for malware detection [13] and (2) data imbalance [14] preventing accurate malware detection and classification. Hence, keeping these problems in the mind, the ultimate challenge is to develop a malware identification model that can address the enormous variations in malware families.

In this paper, we propose a feature fusion-based solution to classify malware into different classes using features extracted from images as well as from pre-trained deep convolutional networks. The contributions of our proposed method are as follows:

• The conversion of malware binaries into the grayscale image.
• Data augmentation performed on malware images to overcome the data imbalance within the malware datasets for robust feature extraction to enhance the classifier performance.
• An optimized multimodal feature representation to combine the segmentation-based fractal texture analysis (SFTA) features [15] and deep convolutional neural network (DCNN) features into a single feature vector to obtain a robust malware classification model.

The other parts of the paper are prepared as follows: Section 2 discusses the related work, whereas Section 3 defines the detail of the proposed methodology. The results and analysis of experimental results and their significance are given in Section 4. Section 5 concludes the paper.

2. Related Work

In recent times, deep learning has confirmed its supremacy for many computer vision and machine learning applications like action recognition [16], gait recognition [17,18], object detection [19,20], and many more [21,22,23]. For malware detection and classification, different researchers have applied deep learning and image processing techniques to accomplish high accuracy because of their ground-breaking capacity to learn the best features.

For example, Cui et al. [12] presented a novel deep learning-based method for malware variant detection. They generated grayscale images from malicious code and effectively addressed the data imbalance problem by using bat algorithm. Then, a convolutional neural network (CNN) was used for identification and classification of malware images and the results show that the model achieved a good accuracy rate. Vinayakumar et al. [7] presented a more scalable and hybrid deep learning framework method for effective visual detection of malware with the same setup. Namavar Jahromi et al. [24] used an extreme learning machine with two hidden layers to detect malware threats in safety-critical systems. Zhu et al. [25] propose deep flow, a novel deep learning-based methodology for detecting malware straightforwardly from the information streams in the Android application. The results show that deep flow can accomplish a high discovery F1 score of 95.05%, outflanking customary deep learning based approaches, which uncovers the upside of deep learning procedure in malware detection. Jeon and Moon [26] extracted the opcode sequences from a malware binary file and adopted a deep recurrent neural network (DRNN) as a classifier to detect malware. Sung et al. [27] used word embedding techniques to embed the malware codes into lower dimensionality feature space. Then, they applied a bi-directional long short-term memory (Bi-LSTM) network model to categorize the malware files by attack family. Gibert et al. [28] combined hand-engineered features and neural networks for malware classification. Each of the subnetworks can be trained separately, while the learned features are fused into a common representation.

To increase the classifier efficiency, researchers adopted deep learning as a tool for feature extraction to detect the malware code. Venkatraman et al. [29] presented a hybrid deep learning visualization approach and the result showed that the deep learning-based model effectively differentiates the behavioral pattern of different malware families. Zhong et al. [30] presented a technique which robustly handles the malware dataset complexities. This technique used a multi-level deep model, which organizes the tree structure of multiple deep models for enhancing the scalability of anonymous malware detection. Ye et al. [31] presented a heterogeneous deep model, which is composed of different layers of associative memory and a weighted auto encoder with multilayer restricted Boltzmann machines to identify malware. For unsupervised feature extraction, the deep learning model is forced to perform a training operation as a greedy layer-wise methodology followed by fine-tuning of a supervised parameter to efficiently detect the malware. Yuxin et al. [32] used a deep belief network (DBN) to train a multilayer generative model using the unlabeled data, which represents the better characteristics of employed data samples. DBNs are used as an auto encoder for feature extraction to detect malware. Vasan et al. [33] used handcrafted features as well as those of VGG16 and ResNet-50 CNNs to perform image-based malware classification. Čeponis and Goranin [34] analyzed the use of dual-flow deep learning methods, such as gated recurrent unit fully convolutional network (GRU-FCN) vs single-flow convolutional neural network (CNN) models for detection of malware signatures.

Over the last few years, the Android operating system (OS) has experienced tremendous popularity. However, this popularity comes at the expense of security, because it is an alluring target for malicious apps. Billah et al. [35] used deep learning as sequences for classification and proposed MalDozer, which is a family attribution and automatic Android malware detection model. MalDozer detects and learns the patterns of malicious and benign data from the actual dataset to expose Android malware. Pektaş and Acarman [36] used deep learning to recognize malware based on the application programming interface (API) call graphs transformed into a numeric feature set as a representation of the malware’s execution paths. D’Angelo et al. [37] encoded the sequences of API calls invoked by apps as sparse image-like matrices (API images). Then, they used autoencoders to get the most informative features from these images, which were provided to an ANN-based classifier for malware detection. Naeem et al. [38] converted a raw Android file into a color image and then submitted it to a DCNN model, achieving 97.81% accuracy on a Leopard Mobile malware dataset and 98.47% accuracy on a Windows dataset. Vidal et al. [39] used genetic sequence alignment methods and statistical hypothesis testing for Android malware recognition, achieving an average hit rate of 98.61% on several public datasets.

The related works are summarized in

Table 1. Summary of some recent malware detection approaches.

Case Study	Year	Data Analysis	Dataset	Classification Approach	Accuracy
Malicious Code Localization [41]	2018	Static	Malware Dataset Benign Dataset Wild Dataset	Graph Kernels Support Vector Machine (SVM)	94%
Malware Variants Classified Behaviors [42]	2018	Dynamic	1220 malware samples	SVM, Decision Tree, Naïve Bayes	88.3%
Detection of Android Native Code Malware Variants [43]	2018	Static	Benign Dataset Malware Dataset	SVM	93.22%
Detecting and Classifying Android Malware [44]	2017	Static	Benign Dataset Malware Dataset	SVM	90%
Android Malware Detection System Based on Machine Learning [45]	2018	Hybrid	Benign Dataset Malware Dataset	SVM	95.2%
Malicious Code Variants Based on Deep Learning [12]	2018	Hybrid	Malware Dataset	SVM	94.5%
Image-Based malware classification [33]	2020	Static	Malware Dataset	Ensemble of convolutional neural networks	99.5%

. Here, the static analysis uses only static resources, which are available before the installation and execution of malicious applications [40].

Different from the earlier proposed methods, the proposed hybrid method based on feature fusion from pre-trained DCNN and SFTA multimodal feature representation is used for malware detection. The foremost objective of the proposed methodology is to extract the robust feature using malicious images to improve the classification accuracy after malware detection.

3. Methodology

3.1. Outline of Methodology

The proposed methodology for malware classification has four major steps: (a) visualization of binary to grayscale image, (b) image augmentation, (c) feature extraction and fusion, and (d) feature selection and classification. The flow diagram of the proposed methodology is illustrated in Figure 1.

The proposed method uses DCNN architecture with a combination of SFTA (scale feature texture analyzer) features [15] for malware classification. Firstly, we convert the binary dataset files into grayscale images for accurate malware classification. To overcome the challenge of the data imbalance problem between different malware classes, we use a data augmentation method [46]. In the feature extraction step, the SFTA features are extracted from grayscale images and the pre-trained DCNN models of AlexNet [47] and Inception-v3 [48] are used for deep feature extraction. Both SFTA features and DCNN features are combined into a single feature vector using a serial-based feature fusion method, and the most informative and robust features are selected using a principal component analysis (PCA)-based feature selection method for accurate malware classification. The complete details of each step are listed in further subsections below.

3.2. Visualization of Binary to a Grayscale Image

While dealing with malware, it is difficult to decide the type of attack by viewing the binary malware data. We convert the parallel malware executable file into 8-bit decimal value with a scope of 0–255, which is saved in a decimal number vector, representing a malware sample. We select the dimensions of the 2D matrix depending on the size of the malware binary file. At that point, we convert the grayscale image into three channel (red (R), green (G), blue (B)) images by duplicating the grayscale channels for three slices. A well-known change technique to change RGB (red, green, blue) to grayscale images is shown in Equation (1).

$$Gra{y}_{image}=Red\ast 0.299+Green\ast 0.587+Blue\ast 0.114$$

(1)

The process of visualization of binaries into a grayscale image is shown in Figure 2.

The width of each gray scale malware image was 256 pixels, while the height relied upon the file size.

3.3. Image Augmentation

The imbalanced datasets are a communal problem in computer vision and harm classification problems. A deficiency of images in each class may cause under-fitting and over-fitting [10], which has a large influence on the performance of the DCNNs. Hence, to overcome this problem, a data augmentation method is proposed within the malware datasets to enhance the classifier performance. Image augmentation is a method that is used to increase the dataset that is commonly used to train the neural network [49]. In the augmentation phase, we generate new data from classes that have less population in the dataset [50]. This process overcomes the restrictive impact on data to avoid uneven representation. To escape over-fitting complications successfully, proper data augmentation schemes can be helpful and improve the strength of the model.

Let $D_1$ be a given malware dataset, $D_1=\left\{a_i\dots \dots \dots a_n\right\}$, where $a_n$ represents the $n\mathrm{th}$ image sample in the dataset. Assume that $a_n$ has N pixels. The pixel coordinate matrix $A_k$ for $a_j$ is given below in Equation (2).

$$A_k=\left[\begin{array}{ccc}{x}_1& y_1& 1\\ x_2& y_2& 1\\ \cdot & \cdot & \cdot \\ x_N& y_N& 1\end{array}\right]$$

(2)

here each row shows similar coordinates of one pixel. For data augmentation on image $a_n$, we apply an affine transform matrix, $p$, on the coordinate matrix, $M_k$, and obtain a transformed coordinate matrix, $M_t^k$, for the image. The operation is given in Equation (3) as follows:

$$M_t^k=p\ast M_k,$$

(3)

where each row of $M_t^k$ is the transformed coordinate for one pixel. The transform matrix operations are also shown in

Table 2. Affine transformation matrices for image augmentation.

Operations	Flipping	Scaling	Rotation
Matrix transform	$\left[\begin{array}{ccc}-1& 0& 0\\ 0& 1& 0\\ 0& 0& 0\end{array}\right]$	$\left[\begin{array}{ccc}\cos \beta & -\sin \beta & 0\\ \sin \beta & \cos \beta & 0\\ 0& 0& 1\end{array}\right]$	$\left[\begin{array}{ccc}-1& 0& 0\\ 0& 1& 0\\ T_x& T_y& 1\end{array}\right]$

.

There are different approaches to decide the affine transformation matrix M. We utilize three sorts of arbitrary irritations for creating new augmentation data as follows. Flipping operation flips the image along the horizontal dimension. Rotation operation pivots the image with a point inspected from 0° to 180°. Scaling operation moves the image in both the x and y dimension of the image. The respective affine transform matrices are shown in Table 2, where β shows the angle of rotation $T_x$, and $T_y$ shows the balances on the arrange hub.

Let $a_k$ be a given image the augmented details denote as $A_k$, and the dataset is denoted as $D_1$. The augmentation process is formulated in the following equations:

$$A_k=\left\{T_1\left(a_k\right),T_2\left(a_k\right),T_3\left(a_k\right)\right\},$$

(4)

$$D_1=\left\{A_1\dots \dots \dots A_k\right\}={{\displaystyle \cup }}_{k=1}^k{{\displaystyle \cup }}_{j=1}^3 T_i\left(a_k\right),$$

(5)

where $T_1$ is flipping, $T_2$ is rotation, and $T_3$ is scaling operations.

The expanded dataset, $D_1$, alongside the compared class names, is then employed to train a deep CNN with the end goal of malware recognition and classification. It ought to be noticed that we misuse flips, rotation, and scaling as the data augmentation tasks since they do not modify the texture topologies in malware classes, as shown in Figure 3.

The augmentation process for example flips, rotates, and scales up a decent variety of training data and improves the classification accuracy of deep CNNs. The data augmentation method in Algorithm 1 describes how the malware images are rotated and flipped to deal with the problem of class imbalance, as given in Algorithm 1.

Algorithm 1. Malware Image Augmentation

Input:    Malware images $Y=\left\{\left(y_i\right),i=1,2,3\dots \dots \dots ,X_y\right\}$, image flipping (Vertical and Horizontal) and rotation $MR=m{r}_k,k=1,2,\dots \dots \dots ,m{i}_{xi}$ where $m{r}_k$ is the rotation angle. Output:  Rotation of malware image $RI$ and flipped malware image $FI$. 1: for each Image $y_i\in Y$ do 2:    for $k=1$ to 3:      Angle of rotation $m{r}_k$ to $X_y$ 4:   end for 5:    for $MI=\left\{Horizontal,Vertical\right\}$ 6:       Flipping 7:   end for 8:   end for 9:  return $RI$ and $FI$

3.4. Feature Extraction

Feature extraction is an important step in Malware classification. When the size of the data is large and difficult to manage, then the data is transformed into a reduced set of feature representations. A feature holds data about the size, texture, color, and shape of an image. Here, we used local features, global features, and texture features. The proposed framework adopts two stages for feature extraction, which are performed using the equivalent stream. In the first stage, from grayscale images, SFTA is used for texture feature extraction because of its robustness and low computation cost. In the second stage, the pre-trained CNNs, like Alexnet and Inception-v3, are used to extract deep features in depth to obtain a robust classification model. Each feature extraction method is discussed in the following sections.

3.4.1. Texture Feature (SFTA)

Segmentation-based fractal texture analysis (SFTA) is a texture-based feature acquisition method [15]. The texture element is the most noteworthy trait of an image, which is utilized to identify and characterize the malware images and discover the similitudes between images of various malware families. SFTA is utilized for texture feature extraction because of its robustness and low computation cost. The SFTA extraction method can be partitioned into two parts, which are discussed below.

First, the input grayscale image $i\left(v,u\right)$ is broken down into a binary image. We utilize two-threshold binary deterioration (TTBD) for binary decomposition. Binary images are formulated in Equation (6).

$$i_b\left(v,u\right)=\left\{\begin{array}{c}{\&}_0^1 if t_{low}\le 1\left(v,u\right)\le t_{up}\\ \mathrm{otherwise}\end{array}\right\},$$

(6)

where $t_{low}$ is the lower threshold and $t_{up}$ represents the upper threshold value.

After applying TTBD to the gray level input image, the SFTA feature vector is developed as the fractal dimension size, mean gray level, and, furthermore, the SFTA feature vector. The fractal estimations are utilized to portray the complexity of malware image structures fragmented in the input image. The fractal dimension $D_i$ is determined by using the boundary around an image. The boundary image, $\Delta \left(v,u\right)$, is computed as follows:

$$\left\{\begin{array}{l}1 \mathrm{if} \exists \left(v^{\prime },u^{\prime }\right)\in N_{I=8}\left[\left(v,u\right)\right]:\\ {\hspace{1em}I}_b\left(v^{\prime },u^{\prime }\right)=0\wedge \\ {\hspace{1em}I}_b\left(v^{\prime },u^{\prime }\right)=1\\ 0 \mathrm{otherwise}\end{array}\right\},$$

(7)

where $N_{I=8}\left[\left(v,u\right)\right]$ is the connected 8 pixels to $\left(v,u\right)$, $\Delta \left(v,u\right)$ in the event that the pixel at $\left(v,u\right)$ in the binary image $I_b\left(v^{\prime },u^{\prime }\right)$ has the value of 1 and having in any event one neighboring pixel with esteem 0. Otherwise, the value is 0. Δ(x, y) takes the value of 0.

Consequently, one can understand that the comings about outskirts are one pixel wide. The gray level mean size (pixel tally) supplements the data extricated from every binary image without essentially expanding the calculation time. Algorithm 2 and Figure 4 show the SFTA feature extraction process.

Algorithm 2. SFTA Feature Extraction Algorithm

Require:  Grayscale image$i\left(v,u\right)$ and two thresholds $t_{low}$ and $t_{up}$. Ensure:  SFTA feature vector $s_{1\times m}$. 1: $T_a\leftarrow$ MultiLevelOtsus $i\left(t_{low},t_{up}\right)$ 2:  $T_b$ $\leftarrow$ $\left\{\left\{t_i,t_{i+1}\right\}:t_i,t_{i+1}\in T_a,i\in |1\dots ]\left|T\right|\left|-1\right|\right\}$ 3:  $T_c$ $\leftarrow$ $\left\{\left\{t_c,t_l\right\}:t_i\in T_a,i\in |1\dots ]\left|T\right|\right\}$ 4:  $i\leftarrow 0$ 5:   for  $\left\{\left\{t_l,t_u\right\}:\left\{t_l,t_u\right\}\in T_b{\displaystyle \cup }{T}_c\right\}$ do 6:    $I_b\leftarrow$ Two Thresholds $\left(i,t_{low},t_{up}\right)$ 7:      $\Delta \left(v,u\right)\leftarrow$ Find Borders $I_b$ 8:     $s_{1\times m}\left[i\right]\leftarrow$ Box Counting $(\Delta )$ 9:      $s_{1\times m}\left[i+1\right]\leftarrow$ MeanGrayLevel $\left(i,i_b\right)$ 10:    $s_{1\times m}\left[i+2\right]\leftarrow$ Pixel Count $I_b$ 11:      $i\leftarrow i+3$ 12:   end for 13:  return $s_{1\times m}$

3.4.2. Deep Convolution Neural Network (DCNN)

Here we use a deep convolutional neural network (DCNN) model for malware classification and recognition. The network architecture has a multilayer structure comprising of convolutional layers, pooling layers, and fully connected layers. Every neuron is processed by a spot item between their little areas and weight, which are associated with the information volume. From that point, activation is performed utilizing the ReLu layer. The ReLu layer does not change the size of an image. Thereafter, the pooling layer is performed to lessen the noise impacts in the separated features. Lastly, significant level features are determined by an associated fully convolutional (FC) layer. The architecture of the neural network is shown in Figure 5.

In this paper we utilize two pre-trained CNN models (AlexNet [47] and Inception-V3 [48]) for feature extraction. These models link a convolution layer, pooling layer, normalization layer, ReLu layer, and FC layer. Deep convolution layer local feature extraction from an image is expressed in the following equations.

Let us denote the $M_i^N$ output layer $N$, and $b_i^N$ the base value, then, the filter for the $j\mathrm{th}$ feature map is ${\psi }_{i,j}^N$, and $h_j$ denotes the $N-1 \mathrm{th}$ output layer. Then, the deep convolutional layers have the formulation as stated in Equation (8).

$$M_i^N=b_i^N+{\displaystyle \sum }_{j=1}^{m_{1 }\left(N-1\right)}{\psi }_{i,j}^N\times h_j^{N-1}$$

(8)

The pooling layer extricates the most extreme returns from the lower convolutional layer with the goal of removing unnecessary features. The maximum pooling settles the issue of over-fitting and a 2 × 2 matrix is performed on the extricated framework. Max pooling is depicted in Equations (9)–(11).

$$f_1^K=f_1^{k-1},$$

(9)

$$f_2^k=\frac{f_2^{k-1}-F\left(k\right)}{S^k}+1,$$

(10)

$$f_3^k=\frac{f_3^{k-1}-F\left(k\right)}{S^k}+1,$$

(11)

Here, $S^k$ denotes the stride and $f_1^k$, $f_2^k$, and $f_3^k$ are $2\times 2$ and $3\times 3$ filters for feature maps. The ReLu and FC (fully connected) layers are defined as:

$$R_i^l=\max \left(h,h_i^{l-1}\right),$$

(12)

$$F{C}_i^l=f\left(z_i^l\right)with{ z}_i^l={\displaystyle \sum _{q=1}^{m_{1 }(l-1)}{\displaystyle \sum _{r=1}^{m_{2 }(l-1)}{\displaystyle \sum _{s=1}^{m_{3 }(l-1)}{x}_{i,j,r,s}^l}}}{\left(F{C}_i^{l-1}\right)}_{r,s}$$

(13)

Here, $R_i^l$ represents the ReLu layer and $F{C}_i^l$ represents the FC layer. The FC layers follow the convolution and pooling layers. The FC layer is like a convolution layer, and the greater part of the analysts perform activation on the FC layer for deep features extraction.

3.4.3. DCNN Feature Extraction and Fusion

Here we use two distinctive pre-trained models (AlexNet, Inception-V3) for feature extraction. These models have different structures, but they have been trained with the same dataset of malware.

AlexNet [47] contains five progressive convolutional layers, Conv1 · Conv5, succeeded by three max-pooling layers, and three FC layers, FC7 with a last softmax classifier. The AlexNet CNN has been trained on 1.2 million images of 1000 classes, with around 60 million network parameters. The network only accepts RGB images with a dimension size of 224 × 224 × 3.

The Inception-V3 [48] model is composed of well-structured convolution modules that can both produce preferential features and decrease the quantity of the parameters. Every Inception module is made of a few convolutional layers and pooling layers in equal numbers. Convolutional layers, for example, 3 × 3, 1 × 3, 3 × 1, and 1 × 1 layers, are utilized in the Inception modules to diminish the quantity of parameters.

The purpose of DCNN feature extraction from two network models is to improve the classifier accuracy. Since both models have characteristics and qualities of different extract features. Therefore, three kinds of features are extracted and used: AlexNet features, Inception-V3 features and texture (SFTA) features. We extract $M\times 87$ features from SFTA and deep features by performing an activation function on the FC7 layer and applying max pooling to expel the noise. The FC7 layer extracted $N\times 4096$ features, and $O\times 2048$ features are extracted through the average pooling layer, where $M$, $N$, and $O$ represent the number of images passed to the model.

To get efficient results, we perform serial based fusion to fuse the deep features and texture feature SFTA. Three feature vectors are defined as follows. Let $1\times M$ represent the SFTA feature vector, $1\times N$ represent the AlexNet feature vector, and $1\times O$ represent the Inception-V3 feature vector, then, Equations (14)–(16) are formulated for the features vector:

$$f_{s\times m}=\left\{s_{1\times 1},s_{1\times 2},s_{1\times 3},s_{1\times 4},\dots \dots ,s_{1\times m}\right\},$$

(14)

$$f_{a\times n}=\left\{a_{1\times 1},a_{1\times 2},a_{1\times 3},a_{1\times 4},\dots \dots ,a_{1\times n}\right\},$$

(15)

$$f_{i\times o}=\left\{i_{1\times 1},i_{1\times 2},i_{1\times 3},i_{1\times 4},\dots \dots \dots ,i_{1\times o}\right\},$$

(16)

Then, the resulting feature vectors from SFTA, AlexNet and Inception-v3 are concatenated. These feature vectors are shown below:

$$Fuse{d}_{fv}={{\displaystyle \sum }}_{i=1}^3\left\{f_{s\times m},f_{a\times n},f_{i\times o}\right\},$$

(17)

$${\displaystyle \sum }\left(f_s,f_a,f_i\right)=\left(\begin{array}{c}{s}_{1\times 1},s_{1\times 2},s_{1\times 3},s_{1\times 4},\dots \dots .,s_{1\times m},\\ a_{1\times 1},a_{1\times 2},a_{1\times 3},a_{1\times 4},\dots \dots ,a_{1\times n},\\ i_{1\times 1},i_{1\times 2},i_{1\times 3},i_{1\times 4},\dots \dots ,i_{1\times n}\end{array}\right),$$

(18)

$$f_{Res{u}_{1\times q}}=\left\{p_{1\times 1},p_{1\times 2},p_{1\times 3},p_{1\times 4},\dots \dots ,p_{1\times q}\right\},$$

(19)

where $f_{Res{u}_{1\times q}}$ is the resultant feature vector, and $1\times q$ represents the dimension $q=\left(m,n,o\right)$.

3.5. Feature Selection

The inspiration behind the employment of the feature selection is to select the most prominent features to refine the accuracy and eliminate the unnecessary features to make the system fast in terms of execution time and increase the accuracy rate. In this paper, from the fused feature vector (FV), principal component analysis (PCA) is used for optimal feature selection. PCA has the following steps: (a) calculating the mean of a fused feature vector, (b) subtracting the mean from each feature, (c) calculating the covariance matrix, and (d) calculating the eigenvalues and eigenvector from covariance matrix. Finally, PCA returns a score value and a principal components score.

The first process before applying PCA is normalizing the FV so that it works adequately. It can be done by subtracting the relevant means from the corresponding column in the FV. Let us have FV values denoted by $x$, and $N$ is the total number of extracted feature values in FV:

$$x_0=\frac{1}{N}{{\displaystyle \sum }}_{i=0}^N{x}_i,$$

(20)

$$K_0\left(x_0\right)={{\displaystyle \sum }}_{i=0}^n|\left(x_0-x_1\right)|2,$$

(21)

where $K_0$ will be the normalized FV that will be used further for finding the covariance matrix.

$$conv\left(x,y\right)=\frac{1}{n-1}{\displaystyle \sum }_{i=1}^n\left(X_i-x^{\prime }\right)\left(Y_i-y^{\prime }\right)$$

(22)

The covariance of $conv\left(x,y\right)$ matrix calculates the eigenvectors with their corresponding eigenvalues. The selected optimal features from both models after applying PCA are fused using a parallel-based method and a final fused vector is obtained.

4. Results and Analysis

4.1. Dataset

The Malimg [51] dataset (available for download from https://www.dropbox.com/s/ep8qjakfwh1rzk4/malimg_dataset.zip) consists of malware images that were processed from malware binaries, and we trained the deep learning models to classify each malware family. The dataset comprised of 25 malware families and has 9348 grayscale images. Each family of the Malimg dataset contains a different number of samples, so the dataset is imbalanced (see Figure 6). The dataset has an unfair distribution of classes, for example, 2949 images represent the Allaple malware family, while only 80 images are present in the Skintrim family. The image augmentation techniques are used to stabilize the number of training samples to recover the superiority of data trials. By applying the data augmentation technique, the data imbalance problem is resolved, and each family of malware contains 1000 malware samples. Hence, the number of total images becomes 25,000.

4.2. Settings

For classification, we used a support vector machine (SVM) with linear, quadratic, polynomial (cubic), and Gaussian kernels, k-nearest neighbor (KNN) with cosine similarity as distance metrics (cosine KNN), KNN with the count of neighbors set to 1 (fine KNN), KNN with the count of neighbors set to 10 (medium KNN), KNN with the count of neighbors set to 100 (coarse KNN), KNN with distance weighting (weighted KNN), decision tree (DT), DT with the maximum count of splits set to four (fine DT), 20 (medium DT), and 100 (coarse DT), boosted DT, cosine DT, Adaboost classifier, and linear discriminate (LD) classifiers.

Each classifier performance is evaluated using recall, error rate (ER), and area under the curve (AUC) and accuracy metrics (Equations (23)–(25)). All experiments were done with MATLAB 2018b (Math Works, Inc., Natick, MA, USA) having machine specifications of 1.00 GHz CPU with 8 GB RAM.

$$\mathrm{Recall}=\frac{TP}{TP+FN}$$

(23)

$$\mathrm{ER}=\frac{FP+FN}{TP+TN+FP+FN}$$

(24)

$$\mathrm{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$

(25)

4.3. Results

We use the multi-representation of various features and selection methods incorporated to obtain diverse classifier results to verify the efficiency of the proposed method. The results are generated into three groups: (a) selection of features using traditional method (SFTA), (b) selection of features using traditional approach (SFTA) and pre-trained network (AlexNet, Inception-v3), and (c) deep CNN and SFTA feature fusion lengthwise with PCA selection method. All these experiments are employed on the malware dataset by applying PCA and the planned features collection method.

We claim that we can build a high level of accuracy in recognizing malware from trusted samples deep learning with features derived from the pre-trained network and handcrafted method. These features are continuous and allow us to be more flexible with the classification. We use these continuous data and develop a novel classification method using a pre-trained network and handcrafted method to reduce over-fitting during training. We compare our method to a set of machine classification methods that have been applied in previous research and demonstrate an increase of classification accuracy using our method and an unseen dataset over the range of other machine classification methods that have been applied in previous research.

Four experiments are performed using the Malimg balanced and imbalanced dataset to check the effectiveness of results, and the results are compared with the existing methods.

4.3.1. Experiment 1: With Imbalanced Data and SFTA Features without Feature Fusion and Feature Optimization

Experiment 1 is performed using the features obtained by applying SFTA on the Malimg dataset [51] and using the state-of-the-art classifiers. The imbalanced dataset has large different sized images in 25 classes. This shows the problem of over-fitting and has a negative influence on the results. The results based on imbalanced data and without feature fusion and feature optimization using SFTA are depicted in

Table 3. Classification results using imbalanced dataset with SFTA features. Best values are shown in bold.

Classifier	Recall (%)	Accuracy (%)	AUC (%)	ER (%)
Cubic Support Vector Machine (SVM)	94.0	92.1	90.0	7.9
Decision Tree	36.0	72.4	81.0	27.6
Weighted k-Nearest Neighbor (KNN)	86.0	87.2	75.0	12.8
Fine Gaussian	49.0	81.4	93.1	18.6
Fine KNN	95.0	93.7	92.0	6.3
Boosted Decision Tree	85.9	85.2	92.0	14.5
Adaboost	91.0	90.2	89.0	9.8
Linear Discriminate	76.0	74.4	78.0	25.6
Quadratic SVM	96.0	95.2	90.0	4.8

. The best accuracy is achieved using the quadratic SVM.

The quadratic SVM on an imbalanced dataset obtained a maximum classification accuracy of 95.2%, recall of 69.0%, AUC of 90.0%, and ER of 4.8% on 10-fold cross-validation. Quadratic SVM obtained minimum ER and maximum AUC as compared to other classifiers such as weighted KNN, linear discriminate and some more in Table 3. The recall, ER, accuracy and AUC values of certain classifiers are also plotted in the graph shown in Figure 7.

4.3.2. Experiment 2: With Balanced (Augmented) Data and SFTA Features

Experiment 2 was performed on balanced (augmented) data, using the SFTA features but without any feature fusion and optimization. The classification results are depicted in

Table 4. Classification results using balanced data and SFTA features. Best values are shown in bold.

Classifier	Recall (%)	Accuracy (%)	AUC (%)	ER (%)
Gaussian SVM	98.0	98.5	100.0	1.5
Decision Tree	72.0	76.8	99.0	23.2
Weighted KNN	95.0	96.3	99.0	3.7
Fine Gaussian	94.0	95.7	100.0	4.3
Boosted Decision Tree	98.0	98.4	100.0	1.6
Adaboost	96.0	95.2	99.0	4.8
Linear Discriminate	97.6	99.3	60.0	0.7
Cubic SVM	98.0	98.2	100.0	1.8

.

The linear discriminant (LD) classifier used on a balanced dataset obtained a maximum classification accuracy of 99.3%, recall of 97.60%, AUC of 60.0%, and ER of 0.7% on 10-fold cross-validation. In the balanced data, accuracy is improved, and ER is smaller as compared to the imbalanced dataset. The image augmentation to obtain the balanced dataset has a positive influence on the results and solves the problem of overfitting, so accuracy is improved. The recall, ER, accuracy and AUC values of classifiers are also plotted in the graph shown in Figure 8.

4.3.3. Experiment 3: With Imbalanced Data and Using Featured Obtained from a Pre-Trained Network Model

Two pre-trained neural networks (AlexNet and Inception-v3) are used as deep feature extractors, and the classification results are presented in

Table 5. Imbalanced data without feature fusion and optimization results on pre-trained network. Best values are shown in bold.

Classifier	Recall (%)	Accuracy (%)	AUC (%)	ER (%)
Linear SVM	96.0	97.5	85.0	2.5
Fine Decision Tree	96.0	97.4	85.0	2.6
Weighted KNN	66.0	44.6	60.0	55.4
Fine KNN	67.0	54.8	63.0	45.2
Cosine KNN	97.0	98.7	88.0	1.3
Boosted Decision Tree	78.0	68.9	72.0	31.1
Quadratic SVM	96.0	97.5	85.0	2.5
Linear Discriminate	96.0	98.1	88.0	1.9
Cosine Decision Tree	72.0	58.6	63.0	41.8
Medium Decision Tree	69.0	51.4	60.0	48.6
Cubic SVM	97.0	98.3	88.0	1.7

.

The quadratic SVM used an on imbalanced dataset performed well and obtained a maximum classification accuracy of 98.3%, recall of 97.0%, AUC of 88.0%, and ER of 1.7% on 10-fold cross-validation. The quadratic SVM obtained minimum ER and maximum AUC as compared to other classifiers such as weighted KNN, linear discriminate, and some more in Table 5. The recall, ER, accuracy and AUC of certain classifiers are also plotted in the graph shown in Figure 9.

4.3.4. Experiment 4: With Balanced Data and Fusion of SFTA and Pre-Trained Network Features

Experiment 4 aims to confirm the usefulness of a balanced dataset on a DCNN. For a DCNN, all malware balanced dataset images are resized to 227 × 227 × 3. This experiment takes less training time as compared to the above experiments. Two pre-trained networks, AlexNet and Inception-V3, are used as deep feature extractors. The feature fusion of the pre-trained network and handcrafted method (SFTA) results are revealed in

Table 6. Balanced (augmented) data with feature fusion of SFTA and a deep convolutional neural network (DCNN). Best values are shown in bold.

Classifier	Recall (%)	Accuracy (%)	AUC (%)	ER (%)
Linear SVM	98.0	98.5	100.0	1.5
Fine Decision Tree	77.0	82.8	97.0	17.2
Weighted KNN	95.0	96.3	99.0	4.3
Medium KNN	93.0	94.7	99.0	6.7
Fine KNN	97.0	97.7	98.0	3.7
Cosine KNN	96.0	96.9	99	4.9
Boosted Decision Tree	97.0	72.8	99.0	27.2
Quadratic SVM	95.0	99.1	100.0	1.0
Linear Discriminate	97.0	97.6	99.0	3.6
Cosine Decision Tree	21.0	33.5	37.0	66.5
Medium Decision Tree	48.0	66.8	84.0	33.2
Cubic SVM	99.0	99.3	100.0	1.3

, and their visualization is shown in Figure 10.

Table 6 shows a maximum accuracy level of 99.3%, recall of 99.0%, AUC of 100.0%, and ER of 1.3% on cubic SVM, as compared to some more classifiers depicted in Table 6, which shows the effectiveness of balanced data and the robustness of the proposed technique. The pre-trained network works well with balanced datasets and obtained maximum accuracy on certain classifiers. The recall, accuracy, and AUC values of the balanced dataset are higher than for the imbalanced dataset.

4.4. Statistical Analysis

To assess the statistical significance of the results, we adopted the non-parametric Friedman test for the compared methods on all cross-validation folds. As the classification results are not Gaussian distributed, the Friedman test was applied to detect differences in the classification results of various methods across multiple runs. The results of the Friedman test regarding accuracy (see Figure 11, Figure 12, Figure 13 and Figure 14) show that the differences between the methods are statistically significant (p < 0.01). Critical difference (CD) shows the smallest difference in mean ranks, where the difference is not statistically significant.

The confusion matrix for the best classifier (cubic SVM with a fusion of SFTA and pre-trained neural network features applied on the balanced dataset) is presented in Figure 15. We can see only a few misclassifications mostly between the two most numerous classes.

The best classification results (based on accuracy) for all experiments are summarized in Figure 16. Note that image augmentation allowed a statistically significant improvement of results to be achieved for the SFTA features (by 4.1%, p < 0.05, using non-parametric Wilcoxon rank-sum test). However, for the fused feature set, the improvement was not statistically significant. On the other hand, using feature fusion on the original (imbalanced) dataset also allowed a statistically significant improvement to be achieved (by 3.5%, p < 0.05).

4.5. Comparison of the Results of the Proposed Technique with Other Existing Methods

The results on the Malimg dataset are compared with the results of other authors using the same dataset in

Table 7. Comparison with existing state-of-the-art algorithms.

Author	Year	Features	Techniques	Accuracy (%)
Nataraj et al. [51]	2011	GIST feature	Nearest Neighbor	97.18
Anderson et al. [52]	2012	Gaussian kernel features	SVM	98.0
Dahl et al. [53]	2013	Sparse binary features	Neural networks and logistic regression	86.0
Zhang et al. [54]	2014	Graph-based feature	Semantics-based	93.0
Pascanu et al. [55]	2015	Echo state networks (ESNs) and recurrent neural networks (RNNs) for feature extraction	Logistic regression and multilayer perceptron classifier	98.3
Garcia [56]	2016	Texture features	Random Forest	95.0
Moshiri et al. [57]	2017	Filter-based feature	Machine learning techniques	99.0
Liu et al. [58]	2017	N-gram-based texture feature	Shared nearest neighbor (SNN) clustering algorithm	98.9
Cakir et al. [59]	2018	Deep learning-based feature	Gradient boosting	96.0
Kalash et al. [60]	2018	GIST features	CNN-based architecture	98.5
Naeem et al. [61]	2019	Local and global malware pattern (LGMP) features	SVM, KNN	98.0
Cui et al. [12]	2019	CNN-based features	CNN	97.6
Naeem et al. [62]	2019	Combined local and global malware (CLGM) features	DCNN	98.18
This paper	2020	Fused SFTA and deep network features	DCNN (AlexNet, Inception v3)	99.3

. Note that our method allowed the best results to be achieved on this dataset.

Complexity: As shown in Figure 1, our proposed method consists of five key steps such as data augmentation, features extraction, fusion, selection, and classification. From these, the fusion and selection steps are much important, and the efficiency of our system depends on these. We compute the complexity in the form of Big-O (O) notation for each step. For data augmentation, features extraction, and classification, the complexity is $O\left(1\right)$, while for features fusion and selection, the complexity is $O\left(n\right)+3$ and $O\left(n\right)+k$, where $k$ denotes the number of selected parameters. Hence, our method is working under one major loop, so the overall complexity is $O\left(n\right)=C$.

5. Conclusions

In this paper, we have demonstrated a hybrid method which uses image augmentation and a fusion of segmentation-based fractal texture analysis (SFTA) and pre-trained deep neural network (AlexNet and Inception-v3) features to classify malware images into various classes using different state-of-the-art classifiers. In the first step, the grayscale images are generated using the bytecodes from the malicious programs. Then, in the second step, image augmentation is performed to balance the malware classes and resize the images to 227 × 227 × 3. In the last phase, the pre-trained network features and SFTA best features are selected and fused. Then, the selected features are forwarded to several versions of KNN, SVM, DT, and LD classifiers for malware image identification. The key advantage of the proposed method is the ability to achieve a high level of classification accuracy when compared to the existing methods. When faced with imbalanced data sets there is no one stopping solution to improve the accuracy of the prediction model. In order to get rid of the imbalanced data sets problem, we perform augmentation and balance the dataset and improve the performance of the proposed model. Malware classification based on deep learning introduced an extraordinary improvement when compared with the existing strategies and the experiment result proves the efficiency of the proposed technique as the accuracy was improved and model loss was decreased. The proposed method achieved an accuracy of 99.3% on the cubic SVM classifier, which displays exceptional performance when compared to other current malware classification approaches.