Sub-curve HMM: A malware detection approach based on partial analysis of API call sequences

Abstract

Malicious software (Malware) plays an important role in penetrating and extracting sensitive information. Based on dynamic program’s behavior monitoring, existing solutions have shown that the Hidden Markov Model (HMM) is efficient in detecting malware using sequences of API calls. However, an obfuscation technique could insert minimal data stealing code into a large set of legitimate instructions, which makes the detector ineffective. Additionally, existing solutions require a whole picture of a program’s actions, and hence a small chunk of activities is much harder to detect. Substantial performance degradation can occur during the detection when a long sequence of APIs is used. This paper proposes the Sub-Curve HMM feature extraction approach that focuses on matching subsets of activities from the running processes that potentially lead to data exfiltration incidents. Sequences of API calls are used to train HMMs and test the likelihood of matching to the model. Malicious and benign activities gain different matching scores over an adjoining set of API calls. By projecting a sequence of matching score into a curve, our approach discriminates malicious actions using discontinuities in the slope of the curve. The experimental results show that the proposed approach outperforms existing solutions in detecting six (6) families of malware: the detection accuracy of Sub-Curve HMM is over 94% compared to 83% for the baseline HMM approach and 73% for Information Gain.

Introduction

The high penetration of data-centric services, as mobile computing or online transactions, has markedly increased the risk of exposing the sensitive data of legitimate users to sophisticated malware and tools (Ullah et al., 2018). The main purpose of this malware and tools is to access and steal sensitive data from the users. Consequently, detecting malicious software is a crucial topic to prevent user’s privacy. This is even more critical for specific organizational targets, such as government bodies or military authorities (e.g., leakage of sub-contractor database (Conifer, 2017)). Recent outbreaks of ransomware malware are good examples of devastating consequences.

Some important limitations of existing malware detection approaches are: (1) the signature of the malware is new to Antivirus software, and (2) the anomaly-based systems cannot detect variations in the malware behavior; and they therefore cannot differentiate between legitimate and malignant activities. Intuitively, attackers develop malicious programs using various common techniques, which are performed using similar series of application programming interface (API) calls. Hence, observing the behavior of all running processes (to identify potential malware) is the key to detecting data breaches.

Existing work (Annachhatre, Austin, Stamp, 2015, Canfora, Mercaldo, Visaggio, 2016, Damodaran, Di Troia, Visaggio, Austin, Stamp, 2017) have shown that Hidden Markov Model (HMM) can accurately discriminate between the behavior of a malware and a benign software. HMM models a program’s activities as probabilities of sequences of API calls. Although previous studies (e.g., Annachhatre, Austin, Stamp, 2015, Ki, Kim, Kim, 2015) show promising accuracy in detecting malware, experiments however were carried on short API sequences (i.e., around 280 instructions on average). Dynamic behavior analysis of a malware can generate though very long sequences of API calls. Indeed, our experiments on Keylogger malware (MD5 hash value d4259130a53dae8ddce89ebc8e80b560) generated more than 300,000 series of API calls in less than 2 min. This study found that HMM performs poorly with long API sequences. This is because the HMM only gives a matching probability P(O|λ) between the model λ and all data in the testing sequence O for a whole API sequence. Since the matching score is computed using joint probabilities of each API instruction (INS) in a given sequence, malware could evade the detection by posing as a benign software for the majority of the time and only performs malicious activities for a brief period of time to make the final matching score smaller and indistinguishable from benign software. Nevertheless, to the best of our knowledge, existing work has not properly addressed the impact of the length of API sequences on the detection accuracy. Hence, this work focuses on improving the detection accuracy of long API sequences, where API calls datasets varies from around 5,000 to nearly 80,000 instructions per API sequence on average.

This paper describes the proposed Sub-Curve HMM approach that addresses the HMM limitations by focusing on subsets of API calls sequences. It focuses on finding subsets of matching patterns rather than the average probabilities from the whole sequence. Specifically, the proposed approach observes subsets of a sequence from API INS, which looks for high matching probabilities with the training sequences. These probabilities (i.e., Likelihood) scores are converted to a curve. The proposed approach also uses changing trends (or slope) of the curve to detect discontinuities in the series of probability scores, called Sub-Curve. These curves are selected as features, and they are used by classifiers to detect fragments of malware’s suspicious behaviors. The experimental results show that the proposed approach outperforms existing ones in malware detection accuracy, especially for the Keylogger family that had the longest average API sequence length in our study (76,101 INS). The detection accuracy is over 94% compared to the baseline HMM accuracy of 83%, which includes the whole API sequence.

This paper is organized as follows. Section 2 describes the proposed approach in detail. Section 3 shows the experimental setup, and the results are discussed in Section 4. Related work and conclusion are presented in Sections 6 and 7 respectively.