An Effective Method for Protecting Native API Hook Attacks in User-mode

K Muthumanickam,E Ilavarasan

doi:10.19026/rjaset.9.1373

Abstract

Today, many modern malware developers is taking the advantage of Application Programming Interface (API) hook technique to take the control of the victim computer which making it difficult to detect their presence. Because of the sophistication of rootkit tools, a remote attacker can use native API to compromise any computer which can later be used for many illegal activities such as sniffing network lines, capturing passwords, sending spam and DDoS attack, etc. Thus to protect end-system by identifying and preventing native API malicious code hooking is a challenging problem to the defenders. Today, many different malware-analysis tools incur specific features against malwares but manual and error-prone. In this study, we proposed a behavior-based monitoring detection system to effectively deal native API hooks in user-mode. Unlike other malware identification techniques, our approach involved dynamically analyzing the behavior of native API call hooking malwares. Comparing our experimental evaluation results with existing tools show better performance with no false positive.

Highlights

Today malicious software code which integrates stealthy rootkit technique has posed a serious challenge to computer security defenders
According to the history of information produced by Microsoft, 20% of malicious malware were removed from Windows XP operating system are stealthy rootkits
We propose a user-mode native Application Programming Interface (API) hook detection system to protect system resources from injecting malicious code that uses Import Address Table (IAT) hook and inline hook

Summary

Introduction

Today malicious software code which integrates stealthy rootkit technique has posed a serious challenge to computer security defenders. We use terms malicious code and rootkit, interchangeably. According to the history of information produced by Microsoft, 20% of malicious malware were removed from Windows XP operating system are stealthy rootkits (wikipedia). In order to achieve their programmed tasks, rootkits try to alter the customary execution flow of the Operating System (OS) that can hide system resources such as processes, threads, files, kernel data structures and other key information from the enduser. After the malicious instructions are deposited in a victim computer, code-injection attacks must use native API calls to do further damage. Hooking is a set of code which alters the normal behavior of the operating system by intercepting the system API functions or information exchange passed between different system resources. Rootkits use different types of hooking techniques in order to remain hidden. We focus two user-mode API hooking techniques: IAT hooking and Inline function hooking

Methods

Results

Conclusion