A Survey on Function and System Call Hooking Approaches

Abstract

Functions and system calls are effective indicators of the behavior of a process. These subroutines are useful for identifying unauthorized behavior caused by malware or for developing a better understanding of the lower-level operations of an application. Code obfuscation, however, often prevents user monitoring and modification of subroutine calls. Subroutine hooking offers a solution to this limitation. Function and system call hooking approaches allow for subroutine instrumentation, making hooking a valuable and versatile skill across industry and academia. In this survey, we present several criteria for the classification and selection of hooking tools and techniques as well as an examination of the major hooking approaches used on Windows, Linux, macOS, iOS, and Android operating systems. We also evaluate and compare the performance of different subroutine hooking tools and techniques based on computing resource utilization such as CPU time, memory, and wall-clock time. To the best of our knowledge, this is the first paper that encompasses both system call and function hooking techniques and tools across the major desktop and mobile operating systems.