Semantic Bugs Research Articles

BatFix: Repairing language model-based transpilation

To keep up with changes in requirements, frameworks, and coding practices, software organizations might need to migrate code from one language to another. Source-to-source migration, or transpilation, is often a complex, manual process. Transpilation requires expertise both in the source and target language, making it highly laborious and costly. Languages models for code generation and transpilation are becoming increasingly popular. However, despite capturing code-structure well, code generated by language models is often spurious and contains subtle problems. We propose BatFix , a novel approach that augments language models for transpilation by leveraging program repair and synthesis to fix the code generated by these models. BatFix takes as input both the original program, the target program generated by the machine translation model, and a set of test cases and outputs a repaired program that passes all test cases. Experimental results show that our approach is agnostic to language models and programming languages. BatFix can locate bugs spawning multiple lines and synthesize patches for syntax and semantic bugs for programs migrated from Java to C++ and Python to C++ from multiple language models, including, OpenAI’s Codex .

JMLKelinci+: Detecting Semantic Bugs and Covering Branches with Valid Inputs Using Coverage-guided Fuzzing and Runtime Assertion Checking

Testing to detect semantic bugs is essential, especially for critical systems. Coverage-guided fuzzing (CGF) and runtime assertion checking (RAC) are two well-known approaches for detecting semantic bugs. CGF aims to generate test inputs with high code coverage. However, while CGF tools can be equipped with sanitizers to detect a fixed set of semantic bugs, they can otherwise only detect bugs that lead to a crash. Thus, the first problem we address is how to help fuzzers detect previously unknown semantic bugs that do not lead to a crash. Moreover, a CGF tool may not necessarily cover all branches with valid inputs, although invalid inputs are useless for detecting semantic bugs. So, the second problem is how to guide a fuzzer to maximize coverage using only valid inputs. However, RAC monitors the expected behavior of a program dynamically and can only detect a semantic bug when a valid test input shows that the program does not satisfy its specification. Thus, the third problem is how to provide high-quality test inputs for a RAC that can trigger potential bugs. The combination of a CGF tool and RAC solves these problems and can cover branches with valid inputs and detect semantic bugs effectively. Our study uses RAC to guarantee that only valid inputs reach the program under test using the program’s specified preconditions, and it also uses RAC to detect semantic bugs using specified postconditions. A prototype tool was developed for this study, named JMLKelinci+. Our results show that combining a CGF tool with RAC will lead to executing the program under test only with valid inputs and that this technique can effectively detect semantic bugs. Also, this idea improves the feedback given to a CGF tool, enabling it to cover all branches faster in programs with non-trivial preconditions. 1

Toward Using Fuzzers and Lightweight Specifications to Reveal Semantic Bugs.

Toward Using Fuzzers and Lightweight Specifications to Reveal Semantic Bugs.

Reliable Fix Patterns Inferred from Static Checkers for Automated Program Repair

Fix pattern-based patch generation is a promising direction in automated program repair (APR). Notably, it has been demonstrated to produce more acceptable and correct patches than the patches obtained with mutation operators through genetic programming. The performance of pattern-based APR systems, however, depends on the fix ingredients mined from fix changes in development histories. Unfortunately, collecting a reliable set of bug fixes in repositories can be challenging. In this article, we propose investigating the possibility in an APR scenario of leveraging fix patterns inferred from code changes that address violations detected by static analysis tools. To that end, we build a fix pattern-based APR tool, Avatar , which exploits fix patterns of static analysis violations as ingredients for the patch generation of repairing semantic bugs. Evaluated on four benchmarks (i.e., Defects4J, Bugs.jar, BEARS, and QuixBugs), Avatar presents the potential feasibility of fixing semantic bugs with the fix patterns inferred from the patches for fixing static analysis violations and can correctly fix 26 semantic bugs when Avatar is implemented with the normal program repair pipeline. We also find that Avatar achieves performance metrics that are comparable to that of the closely related approaches in the literature. Compared with CoCoNut, Avatar can fix 18 new bugs in Defects4J and 3 new bugs in QuixBugs. When compared with HDRepair, JAID, and SketchFix, Avatar can newly fix 14 Defects4J bugs. In terms of the number of correctly fixed bugs, Avatar is also comparable to the program repair tools with the normal fault localization setting and presents better performance than most program repair tools. These results imply that Avatar is complementary to current program repair approaches. We further uncover that Avatar can present different bug-fixing performances when it is configured with different fault localization tools, and the stack trace information from the failed executions of test cases can be exploited to improve the bug-fixing performance of Avatar by fixing more bugs with fewer generated patch candidates. Overall, our study highlights the relevance of static bug-finding tools as indirect contributors of fix ingredients for addressing code defects identified with functional test cases (i.e., dynamic information).

An Empirical Study on Bugs in Python Interpreters

Python is an interpreted programming language that has been widely used in many fields. The successful execution of a Python program depends on both the correctness of Python program and the correctness of Python interpreter. As an infrastructure software, there are many bugs in the Python interpreter. Exploring the bugs in Python interpreters can help developers and maintainers of Python interpreters detect and fix bugs and help users of Python avoid risks. In this article, we conduct an empirical study on the bugs in two mainstream Python interpreters: CPython and PyPy. By analyzing 25 958 fixed bugs, 18 824 revisions, 2 116 test cases, and root causes of randomly sampled 510 bugs, we have summarized the following findings. <list list-type="ordered" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><list-item><label>1)</label> The distribution of bugs in the Python interpreter is so uneven that the vast majority of bugs are distributed in a few components and source files. </list-item> <list-item><label>2)</label> The scales of the testing programs that reveal bugs are small. </list-item> <list-item><label>3)</label> The fixing works seem to be not complicated since the number of modified source files and lines of code are limited; however, most bugs need a long time to be fixed; nearly 15% of the bugs need more than one year to fix. </list-item> <list-item><label>4)</label> The priorities of bugs are independent of their locations, but they significantly correlate with duration of bugs. </list-item> <list-item><label>5)</label> Semantic bugs are the most frequent root causes of bugs, and their proportion exceeds other types of root causes. </list-item> </list> These results could indicate some potential problems during the detecting and fixing of Python interpreter’s bugs, and provide some assistance to developers and maintainers of Python interpreters, users of Python, as well as researchers in related fields.

An Empirical Study on Bugs in PHP

PHP (Hypertext Preprocessor) is a scripting language that has been widely used in web development. This paper conducts an empirical study on bugs in PHP. By analyzing 35,921 bug reports, 6524 revisions, and root causes of randomly selected 500 bugs, we find that: (1) Among all the 385 versions involved in these bugs, there are the most bugs in PHP 4.0.4, PHP 4.0.6, and PHP 4.0.3; Documentation bugs are mainly distributed in PHP 4.y.z and PHP 5.y.z; Security bugs are distributed primarily in the relatively later normal versions of PHP 5.y.z. (2) Documentation, Compile, and Scripting Engine packages are greatly affected by bugs; 73.71% of documentation bugs affect documentation; PHAR, EXIF, and GD are more affected by security bugs. (3) It may be not difficult to repair most bugs since the number of modified lines of code and files are limited; However, nearly 11% of bugs need more than one year to repair; Compared with documentation bugs, security bugs are more difficult to be repaired; The duration of bugs in PHP 8.y.z is shorter than in other versions. (4) Semantic bugs and documentation bugs are the more common root causes of bugs than others. Besides, among semantic bugs, the “Missing Features” bugs and “Processing” bugs are more than others. These results could indicate some potential problems during the detecting and repairing of PHP’s bugs. These findings reveal some laws of bugs in PHP. It could assist developers of PHP in improving their development quality, assist maintainers of PHP in detecting and repairing bugs more effectively, and suggest users of PHP evade potential risks.

Finding Bugs in File Systems with an Extensible Fuzzing Framework

File systems are too large to be bug free. Although handwritten test suites have been widely used to stress file systems, they can hardly keep up with the rapid increase in file system size and complexity, leading to new bugs being introduced. These bugs come in various flavors: buffer overflows to complicated semantic bugs. Although bug-specific checkers exist, they generally lack a way to explore file system states thoroughly. More importantly, no turnkey solution exists that unifies the checking effort of various aspects of a file system under one umbrella. In this article, to highlight the potential of applying fuzzing to find any type of file system bugs in a generic way, we propose H ydra , an extensible fuzzing framework. H ydra provides building blocks for file system fuzzing, including input mutators, feedback engines, test executors, and bug post-processors. As a result, developers only need to focus on building the core logic for finding bugs of their interests. We showcase the effectiveness of H ydra with four checkers that hunt crash inconsistency, POSIX violations, logic assertion failures, and memory errors. So far, H ydra has discovered 157 new bugs in Linux file systems, including three in verified file systems (FSCQ and Yxv6).

Detecting Semantic Bugs in Autopilot Software by Classifying Anomalous Variables

Like any software, manned-aircraft flight management systems and unmanned aerial system autopilots contain bugs. A large portion of bugs in autopilots are semantic bugs, where the autopilot does not behave according to the expectations of the programmer. A bug detector is constructed to detect semantic bugs for autopilot software. It is hypothesized that semantic bugs can be detected by monitoring a set of relevant variables internal to the autopilot. This paper formulates the problem of identifying these variables as an optimization problem aimed at minimizing the overhead for online bug detection. However, because the optimization problem is computationally prohibitive to solve directly, graph-based software models are used to identify a suboptimal solution. In analyzing real and injected bugs within a particular block of code (a program slice), our proof-of-concept approach resulted in a model using only 20% of the variables in the slice to detect real and synthetic bugs with a specificity of 95% and a sensitivity of at least 60% for all bugs tested (and 90% or higher for many of them).

Pallas

Software optimization is constantly a serious concern for developing high-performance systems. To accelerate the workflow execution of a specific functionality, software developers usually define and implement a fast path to speed up the critical and commonly executed functions in the workflow. However, producing a bug-free fast path is nontrivial. Our study on the Linux kernel discloses that a committed fast path can have up to 19 follow-up patches for bug fixing, and most of them are deep semantic bugs, which are difficult to be pinpointed by existing bug-finding tools. In this paper, we present such a new category of software bugs based on our fast-path bug study across various system software including virtual memory manager, file systems, network, and device drivers. We investigate their root causes and identify five error-prone aspects in a fast path: path state, trigger condition, path output, fault handling, and assistant data structure. We find that many of the deep bugs can be prevented by applying static analysis incorporating simple semantic information. We extract a set of rules based on our findings and build a toolkit PALLAS to check fast-path bugs. The evaluation results show that PALLAS can effectively reveal fast-path bugs in a variety of systems including Linux kernel, mobile operating system, software-defined networking system, and web browser.

Pallas

Software optimization is constantly a serious concern for developing high-performance systems. To accelerate the workflow execution of a specific functionality, software developers usually define and implement a fast path to speed up the critical and commonly executed functions in the workflow. However, producing a bug-free fast path is nontrivial. Our study on the Linux kernel discloses that a committed fast path can have up to 19 follow-up patches for bug fixing, and most of them are deep semantic bugs, which are difficult to be pinpointed by existing bug-finding tools. In this paper, we present such a new category of software bugs based on our fast-path bug study across various system software including virtual memory manager, file systems, network, and device drivers. We investigate their root causes and identify five error-prone aspects in a fast path: path state, trigger condition, path output, fault handling, and assistant data structure. We find that many of the deep bugs can be prevented by applying static analysis incorporating simple semantic information. We extract a set of rules based on our findings and build a toolkit PALLAS to check fast-path bugs. The evaluation results show that PALLAS can effectively reveal fast-path bugs in a variety of systems including Linux kernel, mobile operating system, software-defined networking system, and web browser.

Do I use the wrong definition?

Software bugs, such as concurrency, memory and semantic bugs, can significantly affect system reliability. Although much effort has been made to address this problem, there are still many bugs that cannot be detected, especially concurrency bugs due to the complexity of concurrent programs. Effective approaches for detecting these common bugs are therefore highly desired. This paper presents an invariant-based bug detection tool, DefUse, which can detect not only concurrency bugs (including the previously under-studied order violation bugs), but also memory and semantic bugs. Based on the observation that many bugs appear as violations to programmers' data flow intentions, we introduce three different types of definition-use invariants that commonly exist in both sequential and concurrent programs. We also design an algorithm to automatically extract such invariants from programs, which are then used to detect bugs. Moreover, DefUse uses various techniques to prune false positives and rank error reports. We evaluated DefUse using sixteen real-world applications with twenty real-world concurrency and sequential bugs. Our results show that DefUse can effectively detect 19 of these bugs, including 2 new bugs that were never reported before, with only a few false positives. Our training sensitivity results show that, with the benefit of the pruning and ranking algorithms, DefUse is accurate even with insufficient training.

MUVI

Software defects significantly reduce system dependability. Among various types of software bugs, semantic and concurrency bugs are two of the most difficult to detect. This paper proposes a novel method, called MUVI, that detects an important class of semantic and concurrency bugs. MUVI automatically infers commonly existing multi-variable access correlations through code analysis and then detects two types of related bugs: (1) inconsistent updates--correlated variables are not updated in a consistent way, and (2) multi-variable concurrency bugs--correlated accesses are not protected in the same atomic sections in concurrent programs.We evaluate MUVI on four large applications: Linux, Mozilla,MySQL, and PostgreSQL. MUVI automatically infers more than 6000 variable access correlations with high accuracy (83%).Based on the inferred correlations, MUVI detects 39 new inconsistent update semantic bugs from the latest versions of these applications, with 17 of them recently confirmed by the developers based on our reports.We also implemented MUVI multi-variable extensions to tworepresentative data race bug detection methods (lock-set and happens-before). Our evaluation on five real-world multi-variable concurrency bugs from Mozilla and MySQL shows that the MUVI-extension correctly identifies the root causes of four out of the five multi-variable concurrency bugs with 14% additional overhead on average. Interestingly, MUVI also helps detect four new multi-variable concurrency bugs in Mozilla that have never been reported before. None of the nine bugs can be identified correctly by the original race detectors without our MUVI extensions.

A Static Bug Detector for Uninitialized Field References in Java Programs

Correctness of Java programs is important because they are executed in distributed computing environments. The object initialization scheme in the Java programming language is complicated, and this complexity may lead to undesirable semantic bugs. Various tools have been developed for detecting program patterns that might cause errors during program execution. However, current tools cannot identify code patterns in which an uninitialized field is accessed when an object is initialized. We refer to such erroneous patterns as uninitialized field references. In this paper, we propose a static pattern detection algorithm for identifying uninitialized field references. We design a sound analysis for this problem and implement an analyzer using the Soot framework. In addition, we apply our algorithm to some real Java applications. From the experiments, we identify 12 suspicious field references in the applications, and among those we find two suspected errors by manual inspection.