Modeling information flow for an autonomous agent to support reverse engineering work

Abstract

Reverse engineering is a cyber defense task used to investigate malware, reconstruct functionality of compiled software, and identify vulnerabilities from closed-source software code already being used in operational contexts. While research in this area has mainly focused on techniques to extract information from binary code, it is also important to understand the capabilities and limitations of the human involved in the reverse engineering process (both defensively and offensively), so we can design better information representations and effectively allocate appropriate tasks to autonomous agents. In this paper, we describe our introductory work in developing agent models of reverse engineering. We review what is known about reverse engineers’ mental models, then describe and characterize four human–computer interaction patterns involved in reverse engineering from a cognitive task analysis. Finally, we present a category theoretic model to describe how reverse engineers trace information flow when performing static analysis. Our approach is a first step in modeling, simulating, and optimizing the human interaction components of these tasks to increase the speed, scale, and accuracy of cyber defense efforts.