A Novel Disassemble Algorithm Designed for Malicious File

Jianmin Pang,Qiang Wang,Yichi Zhang,Di Sun,Chenyun Dai

doi:10.19026/rjaset.5.4749

Abstract

In order to avoid being static analyzed, hacker rely on various obfuscation techniques to hide its malicious characters. These techniques are very effective against common disassembles, preventing binary file from being disassembled correctly. The study presents novel disassemble algorithm which based on analyzed Control Flow Graph (CFG) and Data Flow Graph (DFG) information improve the ability of the disassembly. The proposed algorithm was verified on varied binary files. The experiment shows that the method not only improves the accuracy of disassemble but also greatly deal with malicious files.

Highlights

Modern reverse engineering techniques automatically recognize library functions, local variables, stack arguments, data types, branches, loops, etc
We present a novel disassemble algorithm which based on analyzed Control Flow Graph (CFG) and Data Flow Graph (DFG) information improve the ability of the disassembly
The experiment shows that the method improves the accuracy of disassemble and greatly deal with malicious files

Summary

Introduction

Modern reverse engineering techniques automatically recognize library functions, local variables, stack arguments, data types, branches, loops, etc. The linear sweep algorithm (Cullen and Saumya, 2003) disassembles from the start of a program’s entry point and sweeps on the whole code section and disassemble each instruction encountered in the process. The obfuscation with indirect jumps just makes use of the disposal of control transfer instruction to achieve the goal of confusing the disassemble. Any address in the code section is a potential target of an indirect branch.

Results

Conclusion