Symbolic Execution Tool Research Articles

Symbolic execution of floating-point programs: How far are we?

Floating-point programs are challenging for symbolic execution due to the constraint solving problem. This paper empirically studies five existing symbolic execution methods for floating-point programs to evaluate their effectiveness and limitations. We have implemented the existing methods based on the state-of-the-art symbolic execution tool KLEE and constructed a real-world floating-point program benchmark for evaluation. We evaluate the existing methods with respect to statement coverage and the ability to detect floating-point exceptions. The results demonstrate that the existing methods complement each other. Based on the evaluation results, we propose a synergistic approach to improving the efficiency of the symbolic execution for floating-point programs. The experimental results indicate our synergistic method’s effectiveness in finding floating-point exceptions.

SmartExecutor: Coverage-Driven Symbolic Execution Guided via State Prioritization and Function Selection

Symbolic execution of smart contracts suffers from sequence explosion. Some existing tools limit the sequence length, thus being unable to adequately evaluate some functions. In this paper, we propose a symbolic execution approach without limiting the sequence length. In our approach, the symbolic execution process is a two-phase model that maximizes code coverage while reducing the number of sequences to be executed. The first phase executes all sequences up to a length limit to identify the not-fully covered functions while the second attempts to cover these functions according to state evaluation and a function graph structure. We have developed a tool called SmartExecutor and conducted an experimental evaluation on the SGUARD dataset. The experimental results indicate that compared with state-of-the-art tools, SmartExecutor achieves higher code coverage with less time. It also detects more vulnerabilities than Mythril, a state-of-the-art symbolic execution tool.

Enhancing Dynamic Symbolic Execution by Automatically Learning Search Heuristics

We present a technique to automatically generate search heuristics for dynamic symbolic execution. A key challenge in dynamic symbolic execution is how to effectively explore the program's execution paths to achieve high code coverage in a limited time budget. Dynamic symbolic execution employs a search heuristic to address this challenge, which favors exploring particular types of paths that are most likely to maximize the final coverage. However, manually designing a good search heuristic is nontrivial and typically ends up with suboptimal and unstable outcomes. The goal of this paper is to overcome this shortcoming of dynamic symbolic execution by automatically learning search heuristics. We define a class of search heuristics, namely a parametric search heuristic, and present an algorithm that efficiently finds an optimal heuristic for each subject program. Experimental results with industrial-strength symbolic execution tools (e.g., KLEE) show that our technique can successfully generate search heuristics that significantly outperform existing manually-crafted heuristics in terms of branch coverage and bug-finding.

Compositional noninterference on hardware weak memory models

Weak memory models are employed by all modern multicore processors to improve their performance. For most code, the effects of such memory models can be largely ignored by the programmer. However, for low-level operating system or library code which can include data races for efficiency, these effects may lead to information leaks which cannot be detected without taking the specific memory model into account. While there have been some efforts to develop information flow logics which can detect such leaks, the existing approaches are either not compositional, hindering scalability, or support only a limited form of compositionality, reducing applicability to programs with only simple interactions between threads. This paper is the first to provide a compositional logic for enforcing noninterference properties on more complex concurrent algorithms, while taking into account the underlying hardware memory model. It uses rely/guarantee reasoning to establish how security classifications may be modified by concurrent threads, and considers effects of out-of-order execution allowed on modern multicore processors. The results have been formalised and proved sound in the Isabelle/HOL theorem prover, and automated in a prototype symbolic execution tool. • Weak memory models can invalidate rely/guarantee reasoning. • Structured rely/guarantees with additional checks can support general reasoning. • Rely/guarantee reasoning supports compositional information flow analysis.

Benchmarking the Capability of Symbolic Execution Tools with Logic Bombs

Symbolic execution has become an indispensable technique for software testing and program analysis. However, since several symbolic execution tools are presently available off-the-shelf, there is a need for a practical benchmarking approach. This paper introduces a fresh approach that can help benchmark symbolic execution tools in a fine-grained and efficient manner. The approach evaluates the performance of such tools against known challenges faced by general symbolic execution techniques, e.g., floating-point numbers and symbolic memories. We first survey related papers and systematize the challenges of symbolic execution. We extract 12 distinct challenges from the literature and categorize them into two categories: symbolic-reasoning challenges and path-explosion challenges . Next, we develop a dataset of logic bombs and a framework for benchmarking symbolic execution tools automatically. For each challenge, our dataset contains several logic bombs, each addressing a specific challenging problem. Triggering one or more logic bombs confirms that the symbolic execution tool in question is able to handle the corresponding problem. Real-world experiments with three popular symbolic execution tools, namely, KLEE, angr, and Triton have shown that our approach can reveal the capabilities and limitations of the tools in handling specific issues accurately and efficiently. The benchmarking process generally takes only a few dozens of minutes to evaluate a tool. We have released our dataset on GitHub as open source, with an aim to better facilitate the community to conduct future work on benchmarking symbolic execution tools.

Modified condition/decision coverage (MC/DC) oriented compiler optimization for symbolic execution

Symbolic execution is an effective way of systematically exploring the search space of a program, and is often used for automatic software testing and bug finding. The program to be analyzed is usually compiled into a binary or an intermediate representation, on which symbolic execution is carried out. During this process, compiler optimizations influence the effectiveness and efficiency of symbolic execution. However, to the best of our knowledge, there exists no work on compiler optimization recommendation for symbolic execution with respect to (w.r.t.) modified condition/decision coverage (MC/DC), which is an important testing coverage criterion widely used for mission-critical software. This study describes our use of a state-of-the-art symbolic execution tool to carry out extensive experiments to study the impact of compiler optimizations on symbolic execution w.r.t. MC/DC. The results indicate that instruction combining (IC) optimization is the important and dominant optimization for symbolic execution w.r.t. MC/DC. We designed and implemented a support vector machine based optimization recommendation method w.r.t. IC (denoted as auto). The experiments on two standard benchmarks (Coreutils and NECLA) showed that auto achieves the best MC/DC on 67.47% of Coreutils programs and 78.26% of NECLA programs.

A Survey of Symbolic Execution and Its Tool KLEE

Abstract Symbolic execution is a highly practical program analysis technology. With the gradual deepening of its research and the continuous maturity of technology itself, it has been widely used in software testing and other fields. KLEE is a symbolic execution tool built on the LLVM compilation framework that automatically generates test cases for high coverage of complex and environmentally intensive programs. This article comprehensively describes the basic principle, development situation, current issues of symbolic execution, and KLEE’s infrastructure as well as the generation of test cases. To some extent, it highlights the unique advantages of symbolic execution in software testing and analysis.

Automatic Loop Summarization via Path Dependency Analysis

Analyzing loops is very important for various software engineering tasks such as bug detection, test case generation and program optimization. However, loops are very challenging structures for program analysis, especially when (nested) loops contain multiple paths that have complex interleaving relationships. In this paper, we propose the path dependency automaton (PDA) to capture the dependencies among the multiple paths in a loop. Based on the PDA, we first propose a loop classification to understand the complexity of loop summarization. Then, we propose a loop analysis framework, named Proteus, which takes a loop program and a set of variables of interest as inputs and summarizes path-sensitive loop effects (i.e., disjunctive loop summary) on the variables of interest. An algorithm is proposed to traverse the PDA to summarize the effect for all possible executions in the loop. We have evaluated Proteus using loops from five open-source projects and two well-known benchmarks and applying the disjunctive loop summary to three applications: loop bound analysis, program verification and test case generation. The evaluation results have demonstrated that Proteus can compute a more precise bound than the existing loop bound analysis techniques; Proteus can significantly outperform the state-of-the-art tools for loop program verification; and Proteus can help generate test cases for deep loops within one second, while symbolic execution tools KLEE and Pex either need much more time or fail.

Analysis to Heap Overflow Exploit in Linux with Symbolic Execution

Heap overflow is a common error of buffer overflow in Linux. The control flow of a program may be hijacked when the program satisfies several specific conditions. The existing automatic exploit generation technologies for buffer overflow find vulnerability trigger point and generate exploit by checking the control flow state. However, the heap overflow data rarely lead to a control flow hijacking as well as protection mechanisms limit the trigger condition. It is difficult to analyze the exploitability of heap overflow automatically through the existing analysis technology. For the heap overflow errors in Linux, we summarize the features of exploit on the basis of analyzing the instances, building the detection model of the exploitability of heap overflow, and proposing a method for analyzing the exploitability of heap overflow based on the model. The proposed method monitors the input data and insecurity functions of the program by using taint analysis; builds the path constraints and data constraints which satisfy the conditions of heap overflow exploit through selective symbolic execution; solves the abovementioned constraints and generates the test case automatically. All the steps of our method can be finished automatically by using the symbolic execution tool S2E. The experiments show that this method can automatically analyze and detect the exploitability of heap overflow errors.

JaVerT 2.0: compositional symbolic execution for JavaScript

We propose a novel, unified approach to the development of compositional symbolic execution tools, bridging the gap between classical symbolic execution and compositional program reasoning based on separation logic. Using this approach, we build JaVerT 2.0, a symbolic analysis tool for JavaScript that follows the language semantics without simplifications. JaVerT 2.0 supports whole-program symbolic testing, verification, and, for the first time, automatic compositional testing based on bi-abduction. The meta-theory underpinning JaVerT 2.0 is developed modularly, streamlining the proofs and informing the implementation. Our explicit treatment of symbolic execution errors allows us to give meaningful feedback to the developer during whole-program symbolic testing and guides the inference of resource of the bi-abductive execution. We evaluate the performance of JaVerT 2.0 on a number of JavaScript data-structure libraries, demonstrating: the scalability of our whole-program symbolic testing; an improvement over the state-of-the-art in JavaScript verification; and the feasibility of automatic compositional testing for JavaScript.

Side-Channel Analysis via Symbolic Execution and Model Counting

An important problem in computer security is the detection of side-channel vulnerabilities. Information gained by observing nonfunctional properties of program executions (i.e., sidechannels such as execution time or memory usage) can enable attackers to infer secrets that the program accesses (such as a password). In this talk, I will discuss how symbolic execution, combined with a model counting constraint solver, can be used for quantifying side- channel leakage in Java programs. I will also discuss automata-based model counting techniques. We have implemented these techniques by integrating our modelcounting constraint solver, called Automata-Based model Counter (ABC), with the symbolic execution tool Symbolic Path Finder (SPF).

Tuning parallel symbolic execution engine for better performance

Symbolic execution is widely used in many code analysis, testing, and verification tools. As symbolic execution exhaustively explores all feasible paths, it is quite time consuming. To handle the problem, researchers have paralleled existing symbolic execution tools (e.g., KLEE). In particular, Cloud9 is a widely used paralleled symbolic execution tool, and researchers have used the tool to analyze real code. However, researchers criticize that tools such as Cloud9 still cannot analyze large scale code. In this paper, we conduct a field study on Cloud9, in which we use KLEE and Cloud9 to analyze benchmarks in C. Our results confirm the criticism. Based on the results, we identify three bottlenecks that hinder the performance of Cloud9: the communication time gap, the job transfer policy, and the cache management of the solved constraints. To handle these problems, we tune the communication time gap with better parameters, modify the job transfer policy, and implement an approach for cache management of solved constraints. We conduct two evaluations on our benchmarks and a real application to understand our improvements. Our results show that our tuned Cloud9 reduces the execution time significantly, both on our benchmarks and the real application. Furthermore, our evaluation results show that our tuning techniques improve the effectiveness on all the devices, and the improvement can be achieved upto five times, depending upon a tuning value of our approach and the behaviour of program under test.

Guest Editorial

Guest Editorial

A Branch History Directed Heuristic Search for Effective Binary Level Dynamic Symbolic Execution

Heuristic search is an important part of modern dynamic symbolic execution (DSE) tools, as heuristic search can be used to effectively explore the large program input space. Searching task remains one of several research challenges due to the fact that the input space grows exponentially with the increase of program size, and different programs may have very different structures. The challenge is compounded in a cyber-physical system or cloud-based Internet of Things environment. In this paper, we propose a novel heuristic search algorithm, which analyzes the program execution history and uses the refined history information to inform the search. This paper is based on the observation that the branch and input history generated during dynamic symbolic execution can help memorize the explored input space, and infer the partial structure of the program. With a summarized branch history, the proposed heuristic search makes informed (and better) decisions about which input area to search next for better efficiency. To evaluate the search algorithm, we implement the core DSE engine, integrated with modules to perform execution history collection and analysis. To make our method practical, we incorporate taint analysis and constraint solving statistics to guide the search algorithm. Experimental results demonstrate that with the rich history information, the new search algorithm can explore the input space more effectively, thus resulting in detecting software defects faster.

Вычисление входных данных для достижения определенной функции в программе методом итеративного динамического анализа

Dynamic symbolic execution is a well-known technique used for different tasks of program analysis: input generation for increasing test coverage for program, inputs of death generation, exploit generation and etc. But huge time costs of program analysis during dynamic symbolic execution for any real-life program is a well-known problem caused by path explosion and necessity of path constraint solving for every path with different SAT/SMT techniques which is a NP-complete task in general case. Brute force analysis of every path in program has limited practical sense for time limited analysis; instead different techniques and heuristics are used to improve analysis performance and reduce space of analysis for specific needs of analyst or while solving specific problem under analysis. We present our approach which combines static analysis of program binary code based on binutils library with dynamic symbolic execution tool based on Avalanche - an iterative dynamic analysis tool to perform targeted input data generation for reaching specific function in the program. As the first step of our algorithm we extract reduced program call graph which contains only calls to functions which ends with the function of interest, then we amplify this call graph with control flow graph inside of functions included into reduced call graph. Using the reduced control-flow graph of program which contain only calls and conditional jumps directions which lead to the function of interest we built the metric of best next analysis direction. This approach allows us to significantly (up to twelve times for some real world programs) reduce the time of reaching function of interest comparatively to brute force program paths analysis with inversion of every conditional jump at the execution path dependent on tainted data.

LCTD: Test-guided proofs for C programs on LLVM

Recently there has been much interest in combining underapproximation and overapproximation based approaches to software verification. Such a technique is employed by the Dash algorithm originally developed at Microsoft, which generates tests to gradually improve the accuracy of an underapproximation of the program under test. Simultaneously, an overapproximating abstraction of the program is refined with information gathered from the test generation.We present LCTD, an open source tool that implements the Dash algorithm for the verification of C programs compiled on the LLVM compiler framework. Our implementation is an extension of the dynamic symbolic execution tool LCT. We also present a detailed description of our method for constructing the weakest precondition based refinement operator employed by Dash for instructions of the LLVM internal representation. Our construction handles pointers and array indexing.To maintain a mapping between concrete executions and the abstraction Dash needs to evaluate predicates on the concrete states visited during test executions. A straightforward implementation might store the complete concrete states of each executed test or might employ expensive re-executions to recover the concrete states. We present a technique which allows only the concrete values of pointer variables to be stored while still requiring no re-executions.Finally we present a case study to show the viability of our tool. We also document a more powerful abstraction refinement method for Dash that exploits unsatisfiable regions and evaluate its effect.

User-defined backtracking criteria for symbolic execution

Symbolic execution is a path-sensitive program analysis technique that aids users with program verification. To avoid exploring infeasible paths, symbolic execution checks the prefix of a current path for feasibility by adding a branch constraint to the path prefix and passing the formula to an off-the-shelf SMT solver for an evaluation. If the solver returns SAT/UNSAT, then the prefix is marked as feasible/infeasible. However, the solver can also return an UNKNOWN result, which means it cannot evaluate the formula. In addition, an operation occurring before a constraint can cause over-approximation that propagates to the solver's result. Moreover, symbolic execution might time out the solver if it takes too long to run. A symbolic execution tool might handle these uncertainties by backtracking or by continuing its exploration of the prefix. This paper examines the behavior of path constraints beyond uncertain backtracking points. String and integer constraints are collected from concrete program execution via dynamic symbolic execution. These constraints are used to analyze how over- approximation in a path prefix affects the completeness of its extensions. We also examine variations in time required to decide a path constraint. Our findings suggest that a custom backtracking criteria defined by the user does improve the completeness of symbolic execution.

Design and implementation of a dynamic symbolic execution tool for windows executables

SUMMARYDynamic symbolic execution, or DSE for short, has become a promising technique in software testing. However, the implementation details of DSE have not been described in depth in existing works. Although some open‐source DSE tools are available nowadays, to design and implement a specific DSE tool from scratch is necessary for some reasons. To this end, we implement a Smart Fuzzing Tool for Windows Native Executables, or SMAFE for short, which utilizes Pin and STP for instrumentation and constraint solving, respectively. Advantages of Pin and STP make SMAFE portable. The major contribution of this paper is our detailed description of the implementation of DSE, including symbolization of inputs, tracking of symbols, synchronization of overlapped symbols, environment modeling, and so on. A practical case study validates the effectiveness of SMAFE. Then, the experiments with two benchmark sets present that the code coverage is above 90% on average. Benefits from this paper are at least twofold: moderating learning curve for scholars and shortening the development circle for practitioners. Copyright © 2013 John Wiley & Sons, Ltd.

Leading-edge Ada verification technologies

This tutorial presents a new approach to Spark/Ada contract checking using Bakar Kiasan--a highly automated, evidence-based symbolic execution tool. Bakar Kiasan aims to lower the barrier of entry and reduce the burden of engineers as they specify and verify Ada contracts. Even in the absence of contracts, Bakar Kiasan can check code for possible runtime exceptions and provide visualizations of semantic constraints along paths through procedures. As engineers progressively add contracts, Bakar Kiasan can verify the consistency of code and contracts, thus providing increased confidence, often proportional to the efforts made to capture fuller behavioral specifications via contracts. Bakar Kiasan also provides compositional checking; that is, it can be used on incomplete systems, where contracts are only present for some program components (which may not even have been implemented). This allows contract checking to be used as the program is being developed starting early in the software development process. Bakar Kiasan provides helpful feedback and evidence of its verification results. For example, it automatically generates counter examples as program test cases for illustrating how contracts are violated (this is very helpful when debugging code/contracts), as well as providing various visualization cues, for example, highlighting problematic code or contract segments similar to how modern Integrated Development Environments (IDEs) illustrate compile (type) errors. Kiasan also generates test cases for illustrating how contracts are satisfied, which is helpful for understanding code/contracts or confirming how a program should behave. Bakar Kiasan is integrated in the Eclipse IDE as a plug-in, and an integration with the GNAT Programming Studio (GPS) is currently being developed in collaboration with AdaCore.

Testing android apps through symbolic execution

There is a growing need for automated testing techniques aimed at Android apps. A critical challenge is the systematic generation of test cases. One method of systematically generating test cases for Java programs is symbolic execution. But applying symbolic execution tools, such as Symbolic Pathfinder (SPF), to generate test cases for Android apps is challenged by the fact that Android apps run on the Dalvik Virtual Machine (DVM) instead of JVM. In addition, Android apps are event driven and susceptible to path-divergence due to their reliance on an application development framework. This paper provides an overview of a two-pronged approach to alleviate these issues. First, we have developed a model of Android libraries in Java Pathfinder (JPF) to enable execution of Android apps in a way that addresses the issues of incompatibility with JVM and path-divergence. Second, we have leveraged program analysis techniques to correlate events with their handlers for automatically generating Android-specific drivers that simulate all valid events.