An Observational Investigation of Reverse Engineers' Process and Mental Models

Abstract

Reverse engineering is a complex task essential to several software security jobs like vulnerability discovery and malware analysis. While traditional program comprehension tasks (e.g., program maintenance or debugging) have been thoroughly studied, reverse engineering diverges from these tasks as reverse engineers do not have access to developers, source code, comments, or internal documentation. Further, reverse engineers often have to overcome countermeasures employed by the developer to make the task harder (e.g., symbol stripping, packing, obfuscation). Significant research effort has gone into providing program analysis tools to support reverse engineers. However, little work has been done to understand the way they think and analyze programs, potentially leading to the lack of adoption of these tools among practitioners. This paper reports on a first step toward better understanding the reverse engineer's process and mental models and provides directions for improving program analysis tools to better fit their users. We present the initial results of a semi-structured, observational interview study of reverse engineers (n=15). Each interview investigated the questions they asked while probing the program, how they answered these questions, and the decisions made throughout. Our initial observations suggest that reverse engineers rely on a variety of reference points in both the program text and structure as well as its dynamic behavior to build hypotheses about the program's function and identify points of interest for future exploration. In most cases, our reverse engineers used a mix of static and dynamic analysis---mostly manually---to verify these hypotheses. Throughout, they rely on intuition built up over past experience. From these observations, we provide recommendations for user interface and program analysis improvements to support the reverse engineer.