Fuzzing Tool Research Articles

TR-Fuzz: A syntax valid tool for fuzzing C compilers

Compilers play a critical role in current software construction. However, the vulnerabilities or bugs within the compiler can pose significant challenges to ensuring the security of the resultant software. In recent years, many compilers have made use of testing techniques to address and mitigate such concerns. Fuzzing is widely used among these techniques to detect software bugs. However, when fuzzing compilers, there are still shortcomings in terms of the diversity and validity of test cases. This paper introduces TR-Fuzz, a fuzzing tool specifically designed for C compilers based on Transformer. Leveraging position embedding and multi-head attention mechanisms, TR-Fuzz establishes relationships among data, facilitating the generation of well-formed C programs for compiler testing. In addition, we use different generation strategies in the process of program generation to improve the performance of TR-Fuzz. We validate the effectiveness of TR-Fuzz through the comparison with existing fuzzing tools for C compilers. The experimental results show that TR-Fuzz increases the pass rate of the generated C programs by an average of about 12% and improves the coverage of programs under test compared with the existing tools. Benefiting from the improved pass rate and coverage, we found five bugs in GCC-9.

Towards Unveiling Exploitation Potential With Multiple Error Behaviors for Kernel Bugs

Nowadays, fuzz testing has significantly expedited the vulnerability discovery of Linux kernel. Security analysts use the manifested error behaviors to infer the exploitability of one bug and thus prioritize the patch development. However, only using an error behavior in the report, security analysts might underestimate the exploitability of the kernel bug because it could manifest various error behaviors indicating different exploitation potentials. In this work, we conduct an empirical study on multiple error behaviors of kernel bugs to understand 1) the prevalence of multiple error behaviors and the possible impact of multiple error behaviors towards the exploitation potential; 2) the factors that manifest multiple error behaviors with different exploitation potential. We collected <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">all the fixed kernel bugs</i> reported on Syzbot from September 2017 to January 2022, including 3,352 bug reports. We observed that multiple error behaviors manifested by kernel bugs are prevalent in the real world, and more error behaviors help unveil the exploitability of kernel bugs. Then we organized Linux kernel experts to analyze a sample of kernel bug dataset (484 bug reports, unique 162 bugs) and identified 6 key contributing factors to the mutiple error behaviors. Finally, based on the empirical findings, we propose an object-driven fuzzing technique to explore all possible error behaviors that a kernel bug might bring about. To evaluate the utility of our proposed technique, we implement our fuzzing tool <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GREBE</monospace> and apply it to 60 real-world Linux kernel bugs. On average, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GREBE</monospace> could manifest 2+ additional error behaviors for each of the kernel bugs. For 26 kernel bugs, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GREBE</monospace> discovers higher exploitation potential. We report to kernel vendors some of the bugs – the exploitability of which was wrongly assessed and the corresponding patch has not yet been carefully applied – resulting in their rapid patch adoption.

Bootstrapping Automated Testing for RESTful Web Services

Modern RESTful services expose RESTful APIs to integrate with diversified applications. Most RESTful API parameters are weakly typed, which greatly increases the possible input value space. Weakly-typed parameters pose difficulties for automated testing tools to generate effective test cases to reveal web service defects related to parameter validation. We call this phenomenon the type collapse problem. To remedy this problem, we introduce FET (Format-encoded Type) techniques, including the FET, the FET lattice, and the FET inference to model fine-grained information for API parameters. Inferred FET can enhance parameter validation, such as generating a parameter validator for a certain RESTful server. Enhanced by FET techniques, automated testing tools can generate targeted test cases. We demonstrate Leif, a trace-driven fuzzing tool, as a proof-of-concept implementation of FET techniques. Experiment results on 27 commercial services show that FET inference precisely captures documented parameter definitions, which helps Leif discover 11 new bugs and reduce <inline-formula><tex-math notation="LaTeX">$72\% - 86\%$</tex-math></inline-formula> fuzzing time compared to state-of-the-art fuzzers. Leveraged by the inter-parameter dependency inference, Leif saves 15% fuzzing time.

Optimizing WinAFL for Image Parsing Engine Vulnerability Discovery in PDF Readers

Fuzzing is a kind of automated vulnerability discovering technique using black-box testing ideas. The PDF file format is very complex and can be embedded in many other formats, providing opportunities for malicious code to hide. In this paper, to solve the problem of high blindness in fuzzing for PDF files by the fuzzing tool WinAFL, we propose a targeted fuzzing scheme for the image parsing engine in PDF readers, optimize WinAFL purposefully, and conduct comparison experiments with the original WinAFL. The experiments show that the optimized fuzzing tool can find an average of 69.43% more unique crashes and 43.28% more path discoveries per unit of time for commonly used PDF readers. So, the method can improve the number of path discoveries and unique crash discoveries, proving the effectiveness and practicality of the method and using this method as an inspiration to propose an improved method for other embedded formats in PDF as the next research direction.

Security Analysis of Zigbee Protocol Implementation via Device-agnostic Fuzzing

Zigbee is widely adopted as a resource-efficient wireless protocol in the IoT network. IoT devices from manufacturers have recently been affected due to major vulnerabilities in Zigbee protocol implementations. Security testing of Zigbee protocol implementations is becoming increasingly important. However, applying existing vulnerability detection techniques such as fuzzing to the Zigbee protocol is not a simple task. Dealing with low-level hardware events still remains a big challenge. For the Zigbee protocol, which communicates over a radio channel, many existing protocol fuzzing tools lack a sufficient execution environment. To narrow the gap, we designed Z-Fuzzer , a device-agnostic fuzzing tool for detecting security flaws in Zigbee protocol implementations. To simulate Zigbee protocol execution, Z-Fuzzer leverages a commercial embedded device simulator with pre-defined peripherals and hardware interrupt setups to interact with the fuzzing engine. Z-Fuzzer generates more high-quality test cases with code-coverage heuristics. We compare Z-Fuzzer with advanced protocol fuzzing tools, BooFuzz and Peach fuzzer, on top of Z-Fuzzer’s simulation platform. Our findings suggest that Z-Fuzzer can achieve greater code coverage in Z-Stack, a widely used Zigbee protocol implementation. Compared to BooFuzz and Peach, Z-Fuzzer found more vulnerabilities with fewer test cases. Three of them have been assigned CVE IDs with high CVSS scores (7.5~8.2).

Semantic-Fuzzing-Based Empirical Analysis of Voice Assistant Systems of Asian Symbol Languages

Recently, smart voice assistants (VAs) are widely deployed to provide control services via voice commands in IoT systems, e.g., smart home, industrial IoT systems, etc. However, due to the complexity of the application environment and the diversity of voice commands, more and more attacks against VAs cause severe security problems. As voice development platforms allow third-party voice skills to be accessed, adversaries are able to obtain users’ private information by squatting attacks using confusing names. The existing work studied the exploitability of semantic misinterpretation in VA systems on phonetic languages such as English. However, due to the semantic structural difference between phonetic English and symbol-based Asian languages, such as Chinese, the linguistic-model-guided fuzzing tool proposed by the previous work is insufficient to conduct semantic analysis on the VAs of Asian Languages. In this article, we conduct a systematic analysis to evaluate the feasibility of voice misinterpretation attacks to typical Asian language VAs through semantic fuzzing. We develop Harmony-Fuzzer, the semantic fuzzing tool that the fuzzing process is under the guidance of fuzzing rules abstracted from phenomena of speech errors, disfluency, or semantically similar expressions in Chinese corpus. We use Bayesian networks to formulate fuzzing models statistically so that the fuzzing space can be controlled by the probability of fuzzing processing. We use our results to test VAs and design malicious skills to empirically verify the feasibility of squatting attacks. We found that squatting attacks on Chinese VAs are feasible when attackers leverage some linguistic phenomena delicately.

Fully automated functional fuzzing of Android apps for detecting non-crashing logic bugs

Android apps are GUI-based event-driven software and have become ubiquitous in recent years. Obviously, functional correctness is critical for an app’s success. However, in addition to crash bugs, non-crashing functional bugs (in short as “non-crashing bugs” in this work) like inadvertent function failures, silent user data lost and incorrect display information are prevalent, even in popular, well-tested apps. These non-crashing functional bugs are usually caused by program logic errors and manifest themselves on the graphic user interfaces (GUIs). In practice, such bugs pose significant challenges in effectively detecting them because (1) current practices heavily rely on expensive, small-scale manual validation ( the lack of automation ); and (2) modern fully automated testing has been limited to crash bugs ( the lack of test oracles ). This paper fills this gap by introducing independent view fuzzing , a novel, fully automated approach for detecting non-crashing functional bugs in Android apps. Inspired by metamorphic testing, our key insight is to leverage the commonly-held independent view property of Android apps to manufacture property-preserving mutant tests from a set of seed tests that validate certain app properties. The mutated tests help exercise the tested apps under additional, adverse conditions. Any property violations indicate likely functional bugs for further manual confirmation. We have realized our approach as an automated, end-to-end functional fuzzing tool, Genie. Given an app, (1) Genie automatically detects non-crashing bugs without requiring human-provided tests and oracles (thus fully automated ); and (2) the detected non-crashing bugs are diverse (thus general and not limited to specific functional properties ), which set Genie apart from prior work. We have evaluated Genie on 12 real-world Android apps and successfully uncovered 34 previously unknown non-crashing bugs in their latest releases — all have been confirmed, and 22 have already been fixed. Most of the detected bugs are nontrivial and have escaped developer (and user) testing for at least one year and affected many app releases, thus clearly demonstrating Genie’s effectiveness. According to our analysis, Genie achieves a reasonable true positive rate of 40.9%, while these 34 non-crashing bugs could not be detected by prior fully automated GUI testing tools (as our evaluation confirms). Thus, our work complements and enhances existing manual testing and fully automated testing for crash bugs.

Exploratory Review of Hybrid Fuzzing for Automated Vulnerability Detection

Recently, software testing has become a significant component of information security. The most reliable technique for automated software testing is a fuzzing tool that feeds programs with random test-input and detects software vulnerabilities that are critical to security. Similarly, symbolic execution has gained the most attention as an efficient testing tool for producing smart test-inputs and discovering hard-to-reach bugs using search-based heuristics and compositional approaches. The combination of fuzzing and symbolic execution makes software testing more efficient by mitigating the limitations in each other. Although several studies have been conducted on hybrid fuzzing in recent years, a comprehensive and consistent review of hybrid fuzzing techniques has not been explored. To add coherence to the extensive literature on hybrid fuzzing and to make it reach a large audience, this study provides an overview of key concepts along with the taxonomy of existing hybrid fuzzing tools, problems, and solutions that have been developed in this sphere. It also includes evaluations of the proposed approaches and a number of suggestions for the development of hybrid fuzzing in the future.

Fuzzing With Optimized Grammar-Aware Mutation Strategies

Fuzzing is a widely used technique to discover vulnerabilities in software. However, for programs requiring highly structured inputs, the byte-based mutation strategies in existing fuzzers have difficulties in generating valid inputs. To resolve this challenge, Grammar-Based Fuzzing (GBF) utilizes existing grammar specifications to generate new inputs. Some GBFs perform mutation based on Abstract Syntax Trees (ASTs), which can generate inputs conforming to grammars. However, the existing GBFs neglect using feedback to optimize mutation strategies, and blindly generate inputs without considering the effectiveness of those inputs. In this paper, we use the power schedule and the subtree pool to optimize mutation strategies. Specifically, we first translate input files into ASTs, and extract subtrees from ASTs into a subtree pool. Then, we optimize the power schedule on AST nodes based on a probabilistic model. That is, we adaptively determine the time budget for mutating an AST node. Finally, we replace AST nodes along with their subtrees using the ones we select from the subtree pool. We implement a fuzzing tool to demonstrate our strategies. The experiment results show that our method outperforms the state-of-the-art methods in fuzzing efficiency.

BugMiner: Mining the Hard-to-Reach Software Vulnerabilities through the Target-Oriented Hybrid Fuzzer

Greybox Fuzzing is the most reliable and essentially powerful technique for automated software testing. Notwithstanding, a majority of greybox fuzzers are not effective in directed fuzzing, for example, towards complicated patches, as well as towards suspicious and critical sites. To overcome these limitations of greybox fuzzers, Directed Greybox Fuzzing (DGF) approaches were recently proposed. Current DGFs are powerful and efficient approaches that can compete with Coverage-Based Fuzzers. Nevertheless, DGFs neglect to accomplish stability between usefulness and proficiency, and random mutations make it hard to handle complex paths. To alleviate this problem, we propose an innovative methodology, a target-oriented hybrid fuzzing tool that utilizes a fuzzer and dynamic symbolic execution (also referred to as a concolic execution) engine. Our proposed method aims to generate inputs that can quickly reach the target sites in each sequence and trigger potential hard-to-reach vulnerabilities in the program binary. Specifically, to dive deep into the target binary, we designed a proposed technique named BugMiner, and to demonstrate the capability of our implementation, we evaluated it comprehensively on bug hunting and crash reproduction. Evaluation results showed that our proposed implementation could not only trigger hard-to-reach bugs 3.1, 4.3, 2.9, 2.0, 1.8, and 1.9 times faster than Hawkeye, AFLGo, AFL, AFLFast, QSYM, and ParmeSan respectively but also scale to several real-world programs.

FairFuzz-TC: a fuzzer targeting rare branches

FairFuzz is a coverage-guided mutational fuzzing tool based on AFL, which targets its mutation strategy towards rare branches in the program. FairFuzz was built to run on command-line C $${\backslash }$$ C++ programs which accept a single file as input. We introduce the modifications to FairFuzz which enable it to run on Test-Comp benchmarks; we refer to this altered version as FairFuzz-TC. FairFuzz-TC placed in the middle of the testing competition. FairFuzz-TC had better performance on the error-finding benchmarks than on the branch coverage benchmarks. We analyze the results and find that the benchmarks on which FairFuzz-TC has the most difficulties are those where (a) most functionality is under hard comparisons (requiring precise input values), (b) getting a seed input on which the program does not crash or time out is difficult, or (c) the program takes too much time to execute.

DeepDiver: Diving into Abysmal Depth of the Binary for Hunting Deeply Hidden Software Vulnerabilities

Fuzz testing is a simple automated software testing approach that discovers software vulnerabilities at a high level of performance by using randomly generated seeds. However, it is restrained by coverage and thus, there are chances of finding bugs entrenched in the deep execution paths of the program. To eliminate these limitations in mutational fuzzers, patching-based fuzzers and hybrid fuzzers have been proposed as groundbreaking advancements which combine two software testing approaches. Despite those methods having demonstrated high performance across different benchmarks such as DARPA CGC programs, they still present deficiencies in their ability to analyze deeper code branches and in bypassing the roadblocks checks (magic bytes, checksums) in real-world programs. In this research, we design DeepDiver, a novel transformational hybrid fuzzing tool that explores deeply hidden software vulnerabilities. Our approach tackles limitations exhibited by existing hybrid fuzzing frameworks, by negating roadblock checks (RC) in the program. By negating the RCs, the hybrid fuzzer can explore new execution paths to trigger bugs that are hidden in the abysmal depths of the binary. We combine AFL++ and concolic execution engine and leveraged the trace analyzer approach to construct the tree for each input to detect RCs. To demonstrate the efficiency of DeepDiver, we tested it with the LAVA-M dataset and eight large real-world programs. Overall, DeepDiver outperformed existing software testing tools, including the patching-based fuzzer and state-of-the-art hybrid fuzzing techniques. On average, DeepDiver discovered vulnerabilities 32.2% and 41.6% faster than QSYM and AFLFast respectively, and it accomplished in-depth code coverage.

FuzzE: Fuzzy Fairness Evaluation of Offensive Language Classifiers on African-American English

Hate speech and offensive language are rampant on social media. Machine learning has provided a way to moderate foul language at scale. However, much of the current research focuses on overall performance. Models may perform poorly on text written in a minority dialectal language. For instance, a hate speech classifier may produce more false positives on tweets written in African-American Vernacular English (AAVE). To measure these problems, we need text written in both AAVE and Standard American English (SAE). Unfortunately, it is challenging to curate data for all linguistic styles in a timely manner—especially when we are constrained to specific problems, social media platforms, or by limited resources. In this paper, we answer the question, “How can we evaluate the performance of classifiers across minority dialectal languages when they are not present within a particular dataset?” Specifically, we propose an automated fairness fuzzing tool called FuzzE to quantify the fairness of text classifiers applied to AAVE text using a dataset that only contains text written in SAE. Overall, we find that the fairness estimates returned by our technique moderately correlates with the use of real ground-truth AAVE text. Warning: Offensive language is displayed in this manuscript.

FuzzFactory: domain-specific fuzzing with waypoints

Coverage-guided fuzz testing has gained prominence as a highly effective method of finding security vulnerabilities such as buffer overflows in programs that parse binary data. Recently, researchers have introduced various specializations to the coverage-guided fuzzing algorithm for different domain-specific testing goals, such as finding performance bottlenecks, generating valid inputs, handling magic-byte comparisons, etc. Each such solution can require non-trivial implementation effort and produces a distinct variant of a fuzzing tool. We observe that many of these domain-specific solutions follow a common solution pattern. In this paper, we present FuzzFactory, a framework for developing domain-specific fuzzing applications without requiring changes to mutation and search heuristics. FuzzFactory allows users to specify the collection of dynamic domain-specific feedback during test execution, as well as how such feedback should be aggregated. FuzzFactory uses this information to selectively save intermediate inputs, called waypoints, to augment coverage-guided fuzzing. Such waypoints always make progress towards domain-specific multi-dimensional objectives. We instantiate six domain-specific fuzzing applications using FuzzFactory: three re-implementations of prior work and three novel solutions, and evaluate their effectiveness on benchmarks from Google's fuzzer test suite. We also show how multiple domains can be composed to perform better than the sum of their parts. For example, we combine domain-specific feedback about strict equality comparisons and dynamic memory allocations, to enable the automatic generation of LZ4 bombs and PNG bombs.

DeepFuzz: Automatic Generation of Syntax Valid C Programs for Fuzz Testing

Compilers are among the most fundamental programming tools for building software. However, production compilers remain buggy. Fuzz testing is often leveraged with newlygenerated, or mutated inputs in order to find new bugs or security vulnerabilities. In this paper, we propose a grammarbased fuzzing tool called DEEPFUZZ. Based on a generative Sequence-to-Sequence model, DEEPFUZZ automatically and continuously generates well-formed C programs. We use this set of new C programs to fuzz off-the-shelf C compilers, e.g., GCC and Clang/LLVM. We present a detailed case study to analyze the success rate and coverage improvement of the generated C programs for fuzz testing. We analyze the performance of DEEPFUZZ with three types of sampling methods as well as three types of generation strategies. Consequently, DEEPFUZZ improved the testing efficacy in regards to the line, function, and branch coverage. In our preliminary study, we found and reported 8 bugs of GCC, all of which are actively being addressed by developers.

JDriver: Automatic Driver Class Generation for AFL-Based Java Fuzzing Tools

AFL (American Fuzzy Lop) is a powerful fuzzing tool that has discovered hundreds of real-world vulnerabilities. Recent efforts are seen to port AFL to a fuzzing Java program and have shown to be effective in Java testing. However, these tools require humans to write driver classes, which is not plausible for testing large-scale software. In addition, AFL generates files as input, making it limited for testing methods that process files. In this paper, we present JDriver, an automatic driver class generation framework for AFL-based fuzzing tools, which can build driver code for methods’ processing files as well as ordinary methods not processing files. Our approach consists of three parts: a dependency-analysis based method to generate method sequences that are able to change the instance’s status so as to exercise more paths, a knowledge assisted method to make instance for the method sequences, and an input-file oriented driver class assembling method to handle the method parameters for ordinary methods. We evaluate JDriver on commons-imaging, a widely used image library provided by the Apache organization. JDriver has successfully generated 149 helper methods which can be used to make instances for 110 classes. Moreover, 99 driver classes are built to cover 422 methods.

A Practical Intent Fuzzing Tool for Robustness of Inter-Component Communication in Android Apps

This research aims at a new practical Intent fuzzing tool for detecting Intent vulnerabilities of Android apps causing the robustness problem. We proposed two new ideas. First, we designed an Intent specification language to describe the structure of Intent, which makes our Intent fuzz testing tool flexible. Second, we proposed an automatic tally method classifying unique failures. With the two ideas, we implemented an Intent fuzz testing tool called Hwacha, and evaluated it with 50 commercial Android apps. Our tool offers an arbitrary combination of automatic and manual Intent generators with executors such as ADB and JUnit due to the use of the Intent specification language. The automatic tally method excluded almost 80% of duplicate failures in our experiment, reducing efforts of testers very much in review of failures. The tool uncovered more than 400 unique failures including what is unknown so far. We also measured execution time for Intent fuzz testing, which has been rarely reported before. Our tool is practical because the whole procedure of fuzz testing is fully automatic and the tool is applicable to the large number of Android apps with no human intervention.

Комбинирование динамического символьного исполнения, статического анализа кода и фаззинга

This paper describes a new approach for dynamic code analysis. It combines dynamic symbolic execution and static code analysis with fuzzing to increase efficiency of each component. During fuzzing we recover indirect function calls and pass that information to the static analysis engine. This improves static path detection in the control flow graph of a program. Detected paths are used in dynamic symbolic execution to construct inputs which will cover new paths during execution. These inputs are used by the fuzzing tool to improve test-case generation and increase code coverage. The proposed approach can be used for classic fuzzing when the main goal is achieving high code coverage. As well it can be used for targeted analysis of paths and code fragments in the program. In this case the fuzzing tool accepts a set of programs addresses with potential defects and passes them to the static analysis engine. The engine constructs all paths connecting program entry point to the given addresses. Finally, dynamic symbolic execution is used to construct the set of inputs, which will cover these paths. Experimental results have shown that the proposed method can effectively detect different program defects.

A Novel Fuzzing Method for Zigbee Based on Finite State Machine

With the extensive application of Zigbee, some bodies of literature were devoted into finding the vulnerabilities of Zigbee by fuzzing. According to earlier test records, the majority of defects were exposed due to a series of testing cases. However, the context of malformed inputs is not taken account into the previous algorithms. In this paper, we propose a refined structure-based fuzzing algorithm for Zigbee based on FSM, FSM-fuzzing. Any malformed input in FSM-Fuzzing is injected to the tested sensor against a specific initial state. If the sensor transferred to the next state of FMS or crashed, there would be a defect of Zigbee in dealing with the input under the state. The final state of the sensor is verified by an UIO sequence. After a round of tests, the sensor is regressed to the specific state to prepars for receiving the next mutation. All of the states would be traversed in FSM-fuzzing. A fuzzing tool, ZFSM-fuzzer, is designed for evaluating the performance of FSM-fuzzing. Experiment results show that there is a vulnerability of Zigbee in dealing with the frames without destination addresses. Further, the quality of cases of FSM-fuzzing is higher than the previous algorithms. Therefore, FSM-fuzzing is powerful in finding the vulnerabilities of Zigbee.

Analysis of HTTP Protocol Implementation in Smart Card Embedded Web Server

The latest generation of smart card embeds an HTTP web server which facilitates the integration of smart card into the existing networks and provides more services and custom interfaces. It also helps the developers to simplify the use of new programming model (servlets). However, due to the sensitive information stored and the resource constraints with which the technology is running, it is necessary to test it deeply. Our aim is to detect bugs and vulnerabilities and non-compliance of the HTTP embedded web server. For that purpose, we used the fuzzing technique which consists of injecting invalid or random data on various inputs of the software to be tested. Our fuzzing tool, Smart-Fuzz is based on the Peach framework customised to our needs. Moreover, working in black box, we created the PyHAT application to collect maximum information of the target features. Thus, we can reduce the amount of protocol functionalities to be analysed. The results generated in the log files are finally analyzed to understand the behaviour of the application and to detect if some fuzzed data has succeeded to take up the vulnerabilities.