Bug Detectors Research Articles

On the Impact of Lower Recall and Precision in Defect Prediction for Guiding Search-based Software Testing

Defect predictors, static bug detectors, and humans inspecting the code can propose locations in the program that are more likely to be buggy before they are discovered through testing. Automated test generators such as search-based software testing (SBST) techniques can use this information to direct their search for test cases to likely buggy code, thus speeding up the process of detecting existing bugs in those locations. Often the predictions given by these tools or humans are imprecise, which can misguide the SBST technique and may deteriorate its performance. In this article, we study the impact of imprecision in defect prediction on the bug detection effectiveness of SBST. Our study finds that the recall of the defect predictor, i.e., the proportion of correctly identified buggy code, has a significant impact on bug detection effectiveness of SBST with a large effect size. More precisely, the SBST technique detects 7.5 fewer bugs on average (out of 420 bugs) for every 5% decrements of the recall. However, the effect of precision, a measure for false alarms, is not of meaningful practical significance, as indicated by a very small effect size. In the context of combining defect prediction and SBST, our recommendation is to increase the recall of defect predictors as a primary objective and precision as a secondary objective. In our experiments, we find that 75% precision is as good as 100% precision. To account for the imprecision of defect predictors, in particular low recall values, SBST techniques should be designed to search for test cases that also cover the predicted non-buggy parts of the program, while prioritising the parts that have been predicted as buggy.

WELL: Applying bug detectors to bug localization via weakly supervised learning

AbstractBug localization, which is used to help programmers identify the location of bugs in source code, is an essential task in software development. Researchers have already made efforts to harness the powerful deep learning (DL) techniques to automate it. However, training bug localization model is usually challenging because it requires a large quantity of data labeled with the bug's exact location, which is difficult and time‐consuming to collect. By contrast, obtaining bug detection data with binary labels of whether there is a bug in the source code is much simpler. This paper proposes a WEakly supervised bug LocaLization (WELL) method, which only uses the bug detection data with binary labels to train a bug localization model. With CodeBERT finetuned on the buggy‐or‐not binary labeled data, WELL can address bug localization in a weakly supervised manner. The evaluations on three method‐level synthetic datasets and one file‐level real‐world dataset show that WELL is significantly better than the existing state‐of‐the‐art model in typical bug localization tasks such as variable misuse and other bugs.

How About Bug-Triggering Paths? - Understanding and Characterizing Learning-Based Vulnerability Detectors

Machine learning and its promising branch deep learning have proven to be effective in a wide range of application domains. Recently, several efforts have shown success in applying deep learning techniques for automatic vulnerability discovery, as alternatives to traditional static bug detection. In principle, these learning-based approaches are built on top of classification models using supervised learning. Depending on the different granularities to detect vulnerabilities, these approaches rely on learning models which are typically trained with well-labeled source code to predict whether a program method, a program slice, or a particular code line contains a vulnerability or not. The effectiveness of these models is normally evaluated against conventional metrics including precision, recall and F1 score. In this paper, we show that despite yielding promising numbers, the above evaluation strategy can be insufficient and even misleading when evaluating the effectiveness of current learning-based approaches. This is because the underlying learning models only produce the classification results or report individual/isolated program statements, but are unable to pinpoint bug-triggering paths, which is an effective way for bug fixing and the main aim of static bug detection. Our key insight is that a program method or statement can only be stated as vulnerable in the context of a bug-triggering path. In this work, we systematically study the gap between recent learning-based approaches and conventional static bug detectors in terms of fine-grained metrics called BTP metrics using bug-triggering paths. We then characterize and compare the quality of the prediction results of existing learning-based detectors under different granularities. Finally, our comprehensive empirical study reveals several key issues and challenges in developing classification models to pinpoint bug-triggering paths and calls for more advanced learning-based bug detection techniques.

Debugging Debug Information With Neural Networks

The correctness of debug information included in optimized binaries has been the subject of recent attention by the research community. Indeed, it represents a practically important problem, as most of the software running in production is produced by an optimizing compiler. Current solutions rely on invariants, human-defined rules that embed the desired behavior, whose violation may indicate the presence of a bug. Although this approach proved to be effective in discovering several bugs, it is unable to identify bugs that do not trigger invariants. In this paper, we investigate the feasibility of using Deep Neural Networks (DNNs) to discover incorrect debug information. We trained a set of different models borrowed from the NLP community in an unsupervised way on a large dataset of debug traces and tested their performance on two novel datasets that we propose. Our results are positive and show that DNNs are capable of discovering bugs in both synthetic and real datasets. More interestingly, we performed a <i>live analysis</i> of our models by using them as bug detectors in a fuzzing system. We show that they were able to report 12 unknown bugs in the latest version of the widely used LLVM toolchain, 2 of which have been confirmed.

Understanding and Analyzing Java Reflection

Java reflection has been widely used in a variety of applications and frameworks. It allows a software system to inspect and change the behaviour of its classes, interfaces, methods, and fields at runtime, enabling the software to adapt to dynamically changing runtime environments. However, this dynamic language feature imposes significant challenges to static analysis, because the behaviour of reflection-rich software is logically complex and statically hard to predict. As a result, existing static analysis tools either ignore reflection or handle it partially, resulting in missed, important behaviours, i.e., unsound results. Therefore, improving or even achieving soundness in static reflection analysis—an analysis that infers statically the behaviour of reflective code—will provide significant benefits to many analysis clients, such as bug detectors, security analyzers, and program verifiers. In this article, we provide a comprehensive understanding of Java reflection through examining its underlying concept, API, and real-world usage, and, building on this, we introduce a new static approach to resolving Java reflection effectively in practice. We have implemented our reflection analysis in an open-source tool, called S OLAR , and evaluated its effectiveness extensively with large Java programs and libraries. Our experimental results demonstrate that S OLAR is able to (1) resolve reflection more soundly than the state-of-the-art reflection analyses; (2) automatically and accurately identify the parts of the program where reflection is resolved unsoundly or imprecisely; and (3) guide users to iteratively refine the analysis results by using lightweight annotations until their specific requirements are satisfied.

DeepBugs: a learning approach to name-based bug detection

Natural language elements in source code, e.g., the names of variables and functions, convey useful information. However, most existing bug detection tools ignore this information and therefore miss some classes of bugs. The few existing name-based bug detection approaches reason about names on a syntactic level and rely on manually designed and tuned algorithms to detect bugs. This paper presents DeepBugs, a learning approach to name-based bug detection, which reasons about names based on a semantic representation and which automatically learns bug detectors instead of manually writing them. We formulate bug detection as a binary classification problem and train a classifier that distinguishes correct from incorrect code. To address the challenge that effectively learning a bug detector requires examples of both correct and incorrect code, we create likely incorrect code examples from an existing corpus of code through simple code transformations. A novel insight learned from our work is that learning from artificially seeded bugs yields bug detectors that are effective at finding bugs in real-world code. We implement our idea into a framework for learning-based and name-based bug detection. Three bug detectors built on top of the framework detect accidentally swapped function arguments, incorrect binary operators, and incorrect operands in binary operations. Applying the approach to a corpus of 150,000 JavaScript files yields bug detectors that have a high accuracy (between 89% and 95%), are very efficient (less than 20 milliseconds per analyzed file), and reveal 102 programming mistakes (with 68% true positive rate) in real-world code.

Surveying concurrency bug detectors based on types of detected bugs

Concurrency bugs widely exist in concurrent programs and have caused severe failures in the real world. Researchers have made significant progress in detecting concurrency bugs, which improves software reliability. In this paper, we survey the most up-to-date and well-known concurrency bug detectors. We categorize the existing detectors based on the types of concurrency bugs. Consequently, we analyze data race detectors, atomicity violation detectors, order violation detectors, and deadlock detectors, respectively. We also discuss some other techniques which are mostly related to concurrency bug detection, including schedule bounding techniques, interleaving optimizing techniques, path expanding techniques, and deterministic replay techniques. Additionally, we statistically analyze the reviewed detectors and get some interesting findings, for instance, nearly 86% of previous detectors focus on data races and atomicity violations, and dynamic approaches are popular (74%). We also discuss the limitations of previous detectors, finding that 91% of previous detectors suffer from false negatives and 64% of previous detectors suffer from runtime overhead. Based on the reviewed detectors and statistical analysis, we conclude some future research directions, including accuracy, performance, applicability, and integrality.

효과적인 내장형 소프트웨어의 정수 확장 (Integer Promotion) 버그 검출 기법

세탁기, 냉장고 등의 가전제품에 탑재되는 8-bit MCU용 C 컴파일러는 소프트웨어 실행 속도를 높이기 위해 표준 C 언어 규칙을 따르지 않고 컴파일을 수행할 수 있다. 개발자가 일반 C 컴파일러와 8-bit MCU용 C 컴파일러의 차이를 정확하게 이해하지 못할 경우 표준 C 언어 환경에서는 발생하지 않으나 8-bit MCU를 사용하는 내장형 시스템에서는 발생하는 버그를 야기할 수 있으며 이런 버그는 표준 C언어 환경을 가정하는 버그 검출 도구로는 찾기 어렵다. 본 논문에서는 표준 C 정수 확장 규칙을 따르지 않는 8-bit MCU용 컴파일러를 사용할 때 발생하는 정수 확장 버그를 소개하고 정수 확장 버그를 탐지하기 위한 다섯 종류의 버그 패턴을 제안한다. 정수 확장 버그 패턴 검출 도구를 개발하여 LG전자 세탁기 소프트웨어를 분석한 결과 컴파일러 옵션을 잘못 선택한 경우 발생하는 27개의 정수 확장 버그를 발견하였다.

ASP: Abstraction Subspace Partitioning for Detection of Atomicity Violations with an Empirical Study

Dynamic concurrency bug detectors predict and then examine suspicious instances of atomicity violations from executions of multithreaded programs. Only few predicted instances are real bugs. Prioritizing such instances can make the examinations cost-effective, but is there any design factor exhibiting significant influence? This work presents the first controlled experiment that studies two design factors, abstraction level and subspace, in partitioning such instances through 35 resultant partition-based techniques on 10 benchmarks with known vulnerability-related bugs. The empirical analysis reveals significant findings. First, partition-based prioritization can significantly improve the fault detection rate. Second, coarse-grained techniques are more effective than fine-grained ones, and using some one-dimensional subspaces is more effective than using other dimensional subspaces. Third, eight previously unknown techniques can be more effective than the technique modeled after a state-of-the-art dynamic detector.

Efficient context sensitivity for dynamic analyses via calling context uptrees and customized memory management

State-of-the-art dynamic bug detectors such as data race and memory leak detectors report program locations that are likely causes of bugs. However, programmers need more than static program locations to understand the behavior of increasingly complex and concurrent software. Dynamic calling context provides additional information, but it is expensive to record calling context frequently, e.g., at every read and write. Context-sensitive dynamic analyses can build and maintain a calling context tree (CCT) to track calling context--but in order to reuse existing nodes, CCT-based approaches require an expensive lookup. This paper introduces a new approach for context sensitivity that avoids this expensive lookup. The approach uses a new data structure called the calling context uptree (CCU) that adds low overhead by avoiding the lookup and instead allocating a new node for each context. A key contribution is that the approach can mitigate the costs of allocating many nodes by extending tracing garbage collection (GC): GC collects unused CCU nodes naturally and efficiently, and we extend GC to merge duplicate nodes lazily. We implement our CCU-based approach in a high-performance Java virtual machine and integrate it with a staleness-based memory leak detector and happens-before data race detector, so they can report context-sensitive program locations that cause bugs. We show that the CCU-based approach, in concert with an extended GC, provides a compelling alternative to CCT-based approaches for adding context sensitivity to dynamic analyses.

Efficient concurrency-bug detection across inputs

In the multi-core era, it is critical to efficiently test multi-threaded software and expose concurrency bugs before software release. Previous work has made significant progress in detecting and validating concurrency bugs under a given input. Unfortunately, software testing always faces large sets of test inputs, and existing techniques are still too expensive to be applied to every test input in practice. In this paper, we use open-source software to study how existing concurrency-bug detection tools work for a set of inputs. The study shows that an interleaving pattern, such as a data race or an atomicity violation, can often be exposed by many inputs. Consequently, existing bug detectors would inevitably waste their bug detection effort to generate duplicate bug reports, when applied to a set of inputs. Guided by the above study, we propose a coverage metric, Concurrent Function Pairs (CFP), to efficiently approximate how interleavings overlap across inputs. Using CFP, we have designed a new approach to detecting data races and atomicity-violation bugs for a set of inputs. Our evaluation on open-source C/C++ applications shows that our CFP-guided approach can effectively accelerate concurrency-bug detection for a set of inputs by reducing redundant detection effort across inputs.

Detection and Titer Estimation of Escherichia coli Using Aptamer‐Functionalized Single‐Walled Carbon‐Nanotube Field‐Effect Transistors

Bug detectors: Multiple arrays of aptamer-functionalized single-walled carbon-nanotube field-effect transistors (SWNT-FETs) can be combined with the most probable number method as a simple, fast screening platform for the detection of microorganisms. The picture indicates the binding events of Escherichia coli on such SWNT-FET arrays, which showed a conductance decrease of more than 50% after binding.

Late Night Talk Radio, Albuquerque, and: Behind My Ear is a Little Palace in Broad Daylight

Late Night Talk Radio, Albuquerque Each night, a drunk or Jew, a poor schmuck is stuck in calamity and dramatized à la The Lone Ranger by Chicago's Pacific Garden Mission. Still, "Unshackled" is more soothing than the paranoid who peddles survival water pellets and bug detectors at a slightly higher frequency. Or the connect-the-dot melodics of alien sightseers, and worse, the liberal bashers' cacophony. It's all against me across the dial – radio has yet to progress from black & white. Maybe I would get some sleep if I accepted Jesus as my personal savior. In the dark I'm drawn to stories even though I know the end from the beginning, this cop (to the swell of organ music) saves the believer by changing his testimony the day after God wrestles him in sheets. Or Hyman Applebaum, the evangelical pastor who finds peace despite exile from my tribe. I'll tell you a secret. I sometimes try on Christianity at 1:58 am, still fish-hooked to consciousness, and squirming. So far Christ hasn't risen to the bait and, feeling vaguely guilty at casting my lot with the majority, I switch off [End Page 76] the radio and move on to remodeling my house in my head or replaying a movie or imagining leaves gently floating toward snow, and the next thing I know I'm waking up, having no memory of being abducted by sleep, emerging from a cleansing static, as if from a prayer. Behind My Ear is a Little Palace in Broad Daylight Naturally I think of Him when she taps the needle into the top of my foot. So this is the oft-spoken-of willingness to be pricked on a deity's meridian, blood beading up waxy like a bindi. Under my skin, I picture migraine demons growing frantic as steel pokes through a pore. Godzilla looms over Tokyo. Dr. Li posts a lightning rod on the top of my head as if she were a pilgrim to the North Pole. This is where pain pools after commuting from my neck in tiny axiomatic taxies. Listen. Behind my knee, the universe hums in its velvet bag. Through my wrists, a pulse shimmers with electric eels. I imagine leaking out through the needles, diffusing into the little room papered in Chinese music. Imagine sleep gently tacked to the table like a beetle specimen. How dream minions shriek and scatter when Dr. Li returns, bursting into the dark. I have not yet been resurrected I want to proclaim but she is already extracting that desire. Seven times she carries the needles, like offerings, to the red box. Traffic outside is relentless. She says go home little godling. Put on your socks.

Valgrind: A Program Supervision Framework

Valgrind is a programmable framework for creating program supervision tools such as bug detectors and profilers. It executes supervised programs using dynamic binary translation, giving it total control over their every part without requiring source code, and without the need for recompilation or relinking prior to execution.New supervision tools can be easily created by writing skins that plug into Valgrind's core. As an example, we describe one skin that performs Purify-style memory checks for C and C++ programs.

Sem1 Presents Pathogens to Toll

How cells can recognize foreign matter from self is a critical question in immune responses in organisms as diverse as flies and humans. Michel et al. describe a circulating peptidoglycan recognition protein-encoding gene (sem1) in Drosophila that is important for mediating Toll receptor-mediated responses to Gram-positive bacteria, but apparently is not involved in Toll-mediated responses to fungal infection. Genetic analysis suggested that sem1 is upstream of the Toll receptor, consistent with its putative location in Drosophila hemolymph (injection of wild-type hemolymph into sem1 mutant flies restored expression of the antimicrobial peptide drosomycin in response to Gram-positive bacteria). Kaisho and Akira discuss the similarities and differences in Toll and Toll-like receptor-mediated immune responses in mammals and flies. For a more comprehensive analysis of Toll-like receptors, see O'Neill.T. Michel, J.-M. Reichhart, J. A. Hoffmann, J. Royet, Drosophila Toll is activated by Gram-positive bacteria through a circulating peptidoglycan recognition protein. Nature 414, 756-759 (2001). [Online Journal]T. Kaisho, S. Akira, Immunology: Bug detectors. Nature 414, 701-703 (2001). [Online Journal]L. A. J. O'Neill, The interleukin-1 receptor/Toll-like receptor superfamily: Signal transduction during inflammation and host defense. Science's STKE (2000), http://www.stke.org/cgi/content/full/OC_sigtrans;2000/44/re1. [Abstract] [Full Text]

Sem1 Presents Pathogens to Toll

How cells can recognize foreign matter from self is a critical question in immune responses in organisms as diverse as flies and humans. Michel et al. describe a circulating peptidoglycan recognition protein-encoding gene ( sem1 ) in Drosophila that is important for mediating Toll receptor-mediated responses to Gram-positive bacteria, but apparently is not involved in Toll-mediated responses to fungal infection. Genetic analysis suggested that sem1 is upstream of the Toll receptor, consistent with its putative location in Drosophila hemolymph (injection of wild-type hemolymph into sem1 mutant flies restored expression of the antimicrobial peptide drosomycin in response to Gram-positive bacteria). Kaisho and Akira discuss the similarities and differences in Toll and Toll-like receptor-mediated immune responses in mammals and flies. For a more comprehensive analysis of Toll-like receptors, see O'Neill . T. Michel, J.-M. Reichhart, J. A. Hoffmann, J. Royet, Drosophila Toll is activated by Gram-positive bacteria through a circulating peptidoglycan recognition protein . Nature 414 , 756-759 (2001). [Online Journal] T. Kaisho, S. Akira, Immunology: Bug detectors . Nature 414, 701-703 (2001). [Online Journal] L. A. J. O'Neill, The interleukin-1 receptor/Toll-like receptor superfamily: Signal transduction during inflammation and host defense . Science's STKE (2000), http://www.stke.org/cgi/content/full/OC_sigtrans;2000/44/re1. [Abstract] [Full Text]