SHFuzz: Selective Hybrid Fuzzing with Branch Scheduling Based on Binary Instrumentation

Baosheng Wang,Xianya Mi,Pengfei Wang,Yong Tang,Bo Yu

doi:10.3390/app10165449

Abstract

Hybrid fuzzing is a popular software testing technique that combines random fuzzing with concolic execution. It is widely used in the security domain known for its ability to find deeply hidden vulnerabilities and reach high code coverage. Hybrid fuzzing is based on negating branches in the execution path of a specific input to generate new test cases. However, due to numerous inputs and related branches, it does not show the best of its effectiveness without input and branch selection methods. In this paper, we systematically analyze the branch scheduling problem in the internal attributes of hybrid fuzzing, focusing on the synchronization mechanism. To solve the problems, we propose the Selective Hybrid Fuzzing (SHF) approach with branch scheduling based on binary instrumentation. There are two major parts to the SHF approach: (1) we propose a critical branch selection algorithm to select critical branches by three metrics: hit accuracy, solvability, and complexity; (2) we propose a priority score calculation algorithm to select inputs by the number of critical branches. With the SHF approach, we choose only the branches that can be negated to generate new coverage, instead of repeatedly executing the same branches and generating duplicates of inputs. We implement a hybrid fuzzer called SHFuzz with our SHF approach and compare it with the state-of-the-art hybrid fuzzer QSYM. In the evaluation, SHFuzz outperforms QSYM in 20 real-world applications from the Google Fuzzer Test Suite and other program suites in a 12 h test. On average, SHFuzz achieves 8.40% more code coverage and 100 more unique crashes in each application. Our work also finds existing vulnerabilities 7.85× faster than QSYM. We also find new bugs by SHFuzz, which QSYM fails to find. Our evaluation shows that the selective hybrid fuzzing approach can reduce the number of branches executed in concolic execution, enhancing hybrid fuzzing on code coverage and bug finding capabilities.

Highlights

Fuzzing [1] is a software testing method that randomly mutates the inputs, trying to reach the target execution paths
We propose the Selective Hybrid Fuzzing (SHF) approach with two algorithms: (1) the critical branch selection algorithm to calculate the critical score of each branch by three metrics: hit accuracy, solvability, and complexity; (2) the priority score calculation algorithm to select the input with the most critical branches
The results show that after applying our SHF approach, hybrid fuzzing can generate more coverage and unique crashes to find more bugs in less time

Summary

Introduction

Fuzzing [1] is a software testing method that randomly mutates the inputs, trying to reach the target execution paths. It is widely used in the software security domain, especially in finding vulnerabilities and increasing program coverage. If we consider each basic block as a different coverage, for the simple program shown, the full coverage of the program can be represented as set {A, B, C, D, E, F}. This metric is the least accurate one compared with the other two methods. If we discover a new execution path (A, B, D, E, F), it will not be considered as a new path because all basic blocks in this path have already been recorded as explored coverage, and as a result, we will lose context

Results

Discussion

Conclusion