Verification of dynamic memory allocators based on symbolic program execution

Abstract

Subject of Research. The paper presents the study of vulnerability exploitation techniques in the implementation of dynamic memory allocation algorithms (glibc library allocator): Poisoned Null-byte, Overlapped Chunks, Fastbin Attack,Unsafe Unlink, House of Einherjar, House of Force, House of Spirit, House of Lore, Unsorted Bin Attack. Examples of vulnerability exploitation code and classification of the presented techniques are given in accordance with the Common Weakness Enumeration list. The modern methods and means of vulnerabilities detection are studied; their advantages and disadvantages are shown using the Heap Hopper framework as an example. Modern methods of appropriate software verification are considered. Method. The proposed software verification method combines the approaches of static analysis and symbolic execution using an accurate model of algorithms for dynamic memory allocation. In the compilation process of program being tested, the Kripke structure is created. Dynamic memory vulnerabilities are described by temporal logic formulas. The resulting structure and formulas are passed at the input of the model checking algorithm. Concrete-symbolic execution of the assembled binary file is performed. Vulnerability conditions expressed in the form of propositional logic formulas are checked for symbolic execution paths. Main Results. The practical use of the proposed approach to detection of dynamic memory vulnerabilities in software applications is shown. Symbolic execution is implemented in the form of a low-level debugger, which reduces the operating time of algorithms due to the execution of the application being tested on a real processor. Practical Relevance. The paper presents an integrated approach for solving the problem of automatic vulnerabilities detecting at different stages of the software development life cycle. This approach is applicable for verification of the similar implementations of dynamic memory allocators, such as ptmalloc, dlmalloc, tcmalloc, jemalloc and musl.