Bug Detection Research Articles

How About Bug-Triggering Paths? - Understanding and Characterizing Learning-Based Vulnerability Detectors

Machine learning and its promising branch deep learning have proven to be effective in a wide range of application domains. Recently, several efforts have shown success in applying deep learning techniques for automatic vulnerability discovery, as alternatives to traditional static bug detection. In principle, these learning-based approaches are built on top of classification models using supervised learning. Depending on the different granularities to detect vulnerabilities, these approaches rely on learning models which are typically trained with well-labeled source code to predict whether a program method, a program slice, or a particular code line contains a vulnerability or not. The effectiveness of these models is normally evaluated against conventional metrics including precision, recall and F1 score. In this paper, we show that despite yielding promising numbers, the above evaluation strategy can be insufficient and even misleading when evaluating the effectiveness of current learning-based approaches. This is because the underlying learning models only produce the classification results or report individual/isolated program statements, but are unable to pinpoint bug-triggering paths, which is an effective way for bug fixing and the main aim of static bug detection. Our key insight is that a program method or statement can only be stated as vulnerable in the context of a bug-triggering path. In this work, we systematically study the gap between recent learning-based approaches and conventional static bug detectors in terms of fine-grained metrics called BTP metrics using bug-triggering paths. We then characterize and compare the quality of the prediction results of existing learning-based detectors under different granularities. Finally, our comprehensive empirical study reveals several key issues and challenges in developing classification models to pinpoint bug-triggering paths and calls for more advanced learning-based bug detection techniques.

AI-Assisted Programming Tasks Using Code Embeddings and Transformers

This review article provides an in-depth analysis of the growing field of AI-assisted programming tasks, specifically focusing on the use of code embeddings and transformers. With the increasing complexity and scale of software development, traditional programming methods are becoming more time-consuming and error-prone. As a result, researchers have turned to the application of artificial intelligence to assist with various programming tasks, including code completion, bug detection, and code summarization. The utilization of artificial intelligence for programming tasks has garnered significant attention in recent times, with numerous approaches adopting code embeddings or transformer technologies as their foundation. While these technologies are popular in this field today, a rigorous discussion, analysis, and comparison of their abilities to cover AI-assisted programming tasks is still lacking. This article discusses the role of code embeddings and transformers in enhancing the performance of AI-assisted programming tasks, highlighting their capabilities, limitations, and future potential in an attempt to outline a future roadmap for these specific technologies.

Predictive Monitoring against Pattern Regular Languages

While current bug detection techniques for concurrent software focus on unearthing low-level issues such as data races or deadlocks, they often fall short of discovering more intricate temporal behaviours that can arise even in the absence of such low-level issues. In this paper, we focus on the problem of dynamically analysing concurrent software against high-level temporal specifications such as LTL. Existing techniques for runtime monitoring against such specifications are primarily designed for sequential software and remain inadequate in the presence of concurrency — violations may be observed only in intricate thread interleavings, requiring many re-runs of the underlying software in conjunction with the analysis. Towards this, we study the problem of predictive runtime monitoring , inspired by the analogous problem of predictive data race detection studied extensively recently. The predictive runtime monitoring question asks, given an execution σ, if it can be soundly reordered to expose violations of a specification. In general, this problem may become easily intractable when either the specifications or the notion of reorderings used is complex. In this paper, we focus on specifications that are given in regular languages. Our notion of reorderings is trace equivalence , where an execution is considered a reordering of another if it can be obtained from the latter by successively commuting adjacent independent actions. We first show that, even in this simplistic setting, the problem of predictive monitoring admits a super-linear lower bound of O ( n α ), where n is the number of events in the execution, and α is a parameter describing the degree of commutativity, and typically corresponds to the number of threads in the execution. As a result, predictive runtime monitoring even in this setting is unlikely to be efficiently solvable, unlike in the non-predictive setting where the problem can be checked using a deterministic finite automaton (and thus, a constant-space streaming linear-time algorithm). Towards this, we identify a sub-class of regular languages, called pattern languages (and their extension generalized pattern languages ). Pattern languages can naturally express specific ordering of some number of (labelled) events, and have been inspired by popular empirical hypotheses underlying many concurrency bug detection approaches such as the “small bug depth” hypothesis. More importantly, we show that for pattern (and generalized pattern) languages, the predictive monitoring problem can be solved using a constant-space streaming linear-time algorithm. We implement and evaluate our algorithm PatternTrack on benchmarks from the literature and show that it is effective in monitoring large-scale applications.

Logic Bug Detection and Localization Using Symbolic Quick Error Detection

We present Symbolic Quick Error Detection (Symbolic QED), a structured approach for logic bug detection and localization which can be used both during pre-silicon design verification as well as post-silicon validation and debug. This new methodology leverages prior work on Quick Error Detection (QED) which has been demonstrated to drastically reduce the latency, in terms of the number of clock cycles, of error detection following the activation of a logic (or electrical) bug. QED works through software transformations, including redundant execution and control flow checking, of the applied tests. Symbolic QED combines these error-detecting QED transformations with bounded model checking-based formal analysis to generate minimal-length bug activation traces that detect and localize any logic bugs in the design. We demonstrate the practicality and effectiveness of Symbolic QED using the OpenSPARC T2, a 500-million-transistor open-source multicore System-on-Chip (SoC) design, and using “difficult” logic bug scenarios observed in various state-of-the-art commercial multicore SoCs. Our results show that Symbolic QED: (i) is fully automatic, unlike manual techniques in use today that can be extremely time-consuming and expensive; (ii) requires only a few hours in contrast to manual approaches that might take days (or even months) or formal techniques that often take days or fail completely for large designs; and (iii) generates counter-examples (for activating and detecting logic bugs) that are up to 6 orders of magnitude shorter than those produced by traditional techniques. Significantly, this new approach does not require any additional hardware.

On the effectiveness of graph data augmentation for source code learning

The methodology that employs deep learning to handle software engineering tasks, such as bug detection, is commonly referred to as source code learning. Given the inherent graph nature of source code, graph learning, bolstered by graph neural networks (GNNs), has seen an increasing adoption in the domain of source code learning. Similar to other contexts within deep learning, source code learning also relies on extensive high-quality training data, and the scarcity of such data has become a primary impediment that leads to performance bottlenecks. In practice, data augmentation is often used as a countermeasure to mitigate this issue, by synthesizing additional training data based on existing ones. However, most existing practice of data augmentation in source code learning is limited to simple program transformation methods, such as code refactoring, thus not sufficiently effective. In this work, in light of the graph nature of source code, we propose to apply the data augmentation methods used for graph-structured data in graph learning to the tasks of source code learning, and we conduct a comprehensive empirical study to evaluate whether such new data augmentation approaches bring better effectiveness, in terms of producing more accurate and robust models. Specifically, we evaluate four critical software engineering tasks and seven neural network architectures to assess the effectiveness of five data augmentation methods. Experimental results identify that, compared to the data augmentation-free training approach, the Manifold-Mixup method can significantly improve both the accuracy and robustness of the trained models of source code learning, for up to 1.60% and 4.09%, respectively.

CLFuzz: Vulnerability Detection of Cryptographic Algorithm Implementation via Semantic-aware Fuzzing

Cryptography is a core component of many security applications, and flaws hidden in its implementation will affect the functional integrity or, more severely, pose threats to data security. Hence, guaranteeing the correctness of the implementation is important. However, the semantic characteristics (e.g., diverse input data and complex functional transformation) challenge those traditional program validation techniques (e.g., static analysis and dynamic fuzzing). In this article, we propose CLFuzz, a semantic-aware fuzzer for the vulnerability detection of cryptographic algorithm implementation. CLFuzz first extracts the semantic information of targeted algorithms including their cryptographic-specific constraints and function signatures. Based on them, CLFuzz generates high-quality input data adaptively to trigger error-prone situations efficiently. Furthermore, CLFuzz applies innovative logical cross-check that strengthens the logical bug detection ability. We evaluate CLFuzz on the widely used implementations of 54 cryptographic algorithms. It outperforms state-of-the-art cryptographic fuzzing tools. For example, compared with Cryptofuzz, it achieves a coverage speedup of 3.4× and increases the final coverage by 14.4%. Furthermore, CLFuzz has detected 12 previously unknown implementation bugs in 8 cryptographic algorithms (e.g., CMAC in OpenSSL and Message Digest in SymCrypt), most of which are security-critical and have been successfully collected in the national vulnerability database (7 in NVD/CNVD) and is awarded by the Microsoft bounty program (2 for $1,000).

Characterizing and Detecting WebAssembly Runtime Bugs

WebAssembly (abbreviated WASM) has emerged as a promising language of the Web and also been used for a wide spectrum of software applications such as mobile applications and desktop applications. These applications, named WASM applications, commonly run in WASM runtimes. Bugs in WASM runtimes are frequently reported by developers and cause the crash of WASM applications. However, these bugs have not been well studied. To fill in the knowledge gap, we present a systematic study to characterize and detect bugs in WASM runtimes. We first harvest a dataset of 311 real-world bugs from hundreds of related posts on GitHub. Based on the collected high-quality bug reports, we distill 31 bug categories of WASM runtimes and summarize their common fix strategies. Furthermore, we develop a pattern-based bug detection framework to automatically detect bugs in WASM runtimes. We apply the detection framework to seven popular WASM runtimes and successfully uncover 60 bugs that have never been reported previously, among which 13 have been confirmed and 9 have been fixed by runtime developers.

Extending the range of bugs that automated program repair can handle

Modern automated program repair (APR) is well-tuned to finding and repairing bugs that introduce observable erroneous behavior to a program. However, a significant class of bugs does not lead to observable behavior (e.g., termination bugs and non-functional bugs). Such bugs can generally not be handled with current APR approaches, so complementary techniques are needed. To stimulate the systematic study of alternative approaches and hybrid combinations, we devise a novel bug classification system that enables methodical analysis of their bug detection power and bug repair capabilities. To demonstrate the benefits, we study the repair of termination bugs in sequential and concurrent programs. Our analysis shows that integrating dynamic APR with formal analysis techniques, such as termination provers and software model checkers, reduces complexity and improves the overall reliability of these repairs. We empirically investigate how well the hybrid approach can repair termination and performance bugs by experimenting with hybrids that integrate different APR approaches with termination provers and execution time monitors. Our findings indicate that hybrid repair holds promise for handling termination and performance bugs. However, the capability of the chosen tools and the completeness of the available correctness specification affects the quality of the patches that can be produced.

The progress, challenges, and perspectives of directed greybox fuzzing

SummaryGreybox fuzzing is a scalable and practical approach for software testing. Most greybox fuzzing tools are coverage‐guided as reaching high code coverage is more likely to find bugs. However, since most covered codes may not contain bugs, blindly extending code coverage is less efficient, especially for corner cases. Unlike coverage‐guided greybox fuzzing which increases code coverage in an undirected manner, directed greybox fuzzing (DGF) spends most of its time allocation on reaching specific targets (e.g. the bug‐prone zone) without wasting resources stressing unrelated parts. Thus, DGF is particularly suitable for scenarios such as patch testing, bug reproduction, and special bug detection. For now, DGF has become an active research area. However, DGF has general limitations and challenges that are worth further studying. Based on the investigation of 42 state‐of‐the‐art fuzzers that are closely related to DGF, we conducted the first in‐depth study to summarize the empirical evidence on the research progress of DGF. This paper studies DGF from a broader view, which takes into account not only the location‐directed type that targets specific code parts but also the behavior‐directed type that aims to expose abnormal program behaviors. By analyzing the benefits and limitations of DGF research, we try to identify gaps in current research, meanwhile, reveal new research opportunities and suggest areas for further investigation.

Moving from Cloud to Fog/Edge: The Smart Agriculture Experience

Integrating cloud with fog/edge is a main trend in networking. Many cloud computing applications have been shifted to the edge/fog domain. Such paradigm shift offers new opportunities for pervasive computing. An example is AgriTalk, an Internet of Things (IoT) application development platform for smart agriculture. By integrating cloud with edge/fog, this article describes how AgriTalk addresses six issues for developing edge/fog agriculture applications. These issues include device domain development, application generation and bug detection, sensor failure detection and calibration, big data management, Artificial Intelligence (AI) provisioning, and data privacy. We show how AgriTalk integrates fog/edge applications and use rice blast detection and piglet crushing mitigation as two examples to demonstrate that fog/edge computing is a better solution than cloud computing. Compared with cloud computing, fog/edge computing reduces the delays by 50% in AgriTalk. Through the low-code no-code approach, AgriTalk allows the farmers to create and maintain fog/edge agriculture applications by themselves.

An open-source natural language processing toolkit to support software development: addressing automatic bug detection, code summarisation and code search.

This paper aims to introduce the innovative work carried out in the Horizon 2020 DECODER project - acronym for "DEveloper COmpanion for Documented and annotatEd code Reference" - (Grant Agreement no. 824231) by linking the fields of natural language processing (NLP) and software engineering. The project as a whole addresses the development of a framework, namely the Persistent Knowledge Monitor (PKM), that acts as a central infrastructure to store, access, and trace all the data, information and knowledge related to a given software or ecosystem. This meta-model defines the knowledge base that can be queried and analysed by all the tools integrated and developed in DECODER. Besides, the DECODER project offers a friendly user interface where each of the predefined three roles (i.e., developers, maintainers and reviewers) can access and query the PKM with their personal accounts. The paper focuses on the NLP tools developed and integrated in the PKM, namely the deep learning models developed to perform variable misuse, code summarisation and semantic parsing. These were developed under a common work package - "Activities for the developer" - intended to precisely target developers, who can perform tasks such as detection of bugs, automatic generation of documentation for source code and generation of code snippets from natural languages instructions, among the multiple functionalities that DECODER offers. These tools assist and help the developers in the daily work, by increasing their productivity and avoiding loss of time in tedious tasks such as manual bug detection. Training and validation were conducted for four use cases in Java, C and C++ programming languages in order to evaluate the performance, suitability, usability, etc. of the developed tools.

Toward Understanding Deep Learning Framework Bugs

DL frameworks are the basis of constructing all DL programs and models, and thus their bugs could lead to the unexpected behaviors of any DL program or model relying on them. Such a wide effect demonstrates the necessity and importance of guaranteeing DL frameworks’ quality. Understanding the characteristics of DL framework bugs is a fundamental step for this quality assurance task, facilitating designing effective bug detection and debugging approaches. Hence, in this work, we conduct the most large-scale study on 1,000 bugs from four popular and diverse DL frameworks (i.e., TensorFlow, PyTorch, MXNet, and DL4J). By analyzing the root causes and symptoms of DL framework bugs associated with five components decomposed from DL frameworks, as well as measuring test coverage achieved by three state-of-the-art testing techniques, we obtain 12 major findings for the comprehensive understanding of DL framework bugs and the current status of existing DL framework testing practice, and then provide a series of actionable guidelines for better DL framework bug detection and debugging. Finally, based on the guidelines, we design and implement a prototype DL-framework testing tool, called TenFuzz , which is evaluated to be effective and finds three unknown bugs on the latest TensorFlow framework in a preliminary study, indicating the significance of our guidelines.

RNNtcs: A test case selection method for Recurrent Neural Networks

Recurrent Neural Network (RNN) is a typical feedback neural network, which is particularly effective in processing time-series data tasks such as image description, text generation and classification, etc. However, its safety and quality issues have aroused strong concern in academia and industry. One of the most commonly used methods to ensure RNN-driven software quality is testing and optimizing RNN. Its core idea is to introduce human efforts to label test cases so that many labeled test cases can be used to trigger faults in RNNs. But this brings a huge overhead. Therefore, it is of great importance to study the test case selection method for RNN. Current research on test case selection for RNN has achieved some results from the stateful view of RNN, but there are still some problems, such as a low bug detection rate, poor inclusiveness, and lack of diversity.In this paper, according to the structural characteristics of RNN, we propose a method of RNN test case selection, RNNtcs, which combines clustering and uncertainty, in order to select test cases that can touch RNN bugs as much as possible, thus reducing the cost of labeling. In order to evaluate RNNtcs, we conduct experiments on popular image datasets, text datasets and RNN models. The experimental results show that RNNtcs is superior to the state-of-the-art approach DeepState in bug detection effectiveness, bug detection inclusiveness and bug detection diversity. The test cases selected by RNNtcs can effectively improve the robustness of the RNN model.

Automatic software bug prediction using adaptive golden eagle optimizer with deep learning

In the software maintenance and development process, the software bug detection is an essential problem because it related with the complete software successes. So, the earlier software bug detection is essential to enhance the software efficiency, reliability, software quality and software cost. Moreover, the efficient software bug prediction is a critical as well as challenging operation. Hence, the efficient software bug prediction model is developed in this article. To achieve this objective, optimized long short-term memory is developed. The important stages of the proposed model is preprocessing, feature selection and bug detection. At first the input bug dataset is preprocessed. In preprocessing, the duplicate data instances are removed from the dataset. After the preprocessing, the feature selection is done by Adaptive Golden Eagle Optimizer (AGEO). Here the traditional GEO algorithm is altered by means of opposition-based learning (OBL). Finally, the proposed approach utilizes a long short-term memory (LSTM) based recurrent neural network (RNN) for bug prediction. Long Short-Term Memory (LSTM) network is a type of recurrent neural network. The promise and NASA dataset are considered as the input for bug prediction. the performance of proposed approach is analysed based on various metrics namely, accuracy, F- measure, G-measure and Matthews Correlation Coefficient (MCC).

Automatic Software Bug Prediction Using Adaptive Artificial Jelly Optimization With Long Short-Term Memory

AbstractIn the software maintenance and development process, software bug detection is an essential problem because it is related to complete software success. It is recommended to begin anticipating defects at the early stages of creation rather than during the assessment process due to the high expense of fixing the found bugs. The early stage software bug detection is used to enhance software efficiency, reliability, and software quality. Nevertheless, creating a reliable bug-forecasting system is a difficult challenge. Therefore, in this paper, an efficient, software bug forecast is developed. The presented technique consists of three stages namely, pre-processing, feature selection, and bug prediction. At first, the input datasets are pre-processed to eliminate the identical data from the dataset. After the pre-processing, the important features are selected using an adaptive artificial jelly optimization algorithm (A2JO) to eliminate the possibility of overfitting and reduce the complexity. Finally, the selected features are given to the long short-term memory (LSTM) classifier to predict whether the given data is defective or non-defective. In this paper, investigations are shown on visibly obtainable bug prediction datasets namely, promise and NASA which is a repository for most open-source software. The efficiency of the presented approach is discussed based on various metrics namely, accuracy, F- measure, G-measure, and Matthews Correlation Coefficient (MCC). The experimental result shows our proposed method achieved the extreme accuracy of 93.41% for the Promise dataset and 92.8% for the NASA dataset.

Runtime Verification for Programmable Switches

We introduce a runtime verification framework for programmable switches that complements static analysis. To evaluate our approach, we design and develop, a runtime verification system that automatically detects, localizes, and patches software bugs in P4 programs. Bugs are reported via a violation of pre-specified expected behavior that is captured by . is based on machine learning-guided fuzzing that tests P4 switch non-intrusively, i.e., without modifying the P4 program for detecting runtime bugs. This enables an automated and real-time localization and patching of bugs. We used a prototype to detect and patch existing bugs in various publicly available P4 application programs deployed on two different switch platforms, namely, behavioral model (bmv2) and Tofino. Our evaluation shows that significantly outperforms bug detection baselines while generating fewer packets and patches bugs in large P4 programs, e.g., switch.p4 without triggering any regressions.

A Demonstration of DLBD: Database Logic Bug Detection System

Database management systems (DBMSs) are prone to logic bugs that can result in incorrect query results. Current debugging tools are limited to single table queries and struggle with issues like lack of ground-truth results and repetitive query space exploration. In this paper, we demonstrate DLBD, a system that automatically detects logic bugs in databases. DLBD offers holistic logic bug detection by providing automatic schema and query generation and ground-truth query result retrieval. Additionally, DLBD provides minimal test cases and root cause analysis for each bug to aid developers in reproducing and fixing detected bugs. DLBD incorporates heuristics and domain-specific knowledge to efficiently prune the search space and employs query space exploration mechanisms to avoid the repetitive search. Finally, DLBD utilizes a distributed processing framework to test database logic bugs in a scalable and efficient manner. Our system offers developers a reliable and effective way to detect and fix logic bugs in DBMSs.

SEAL: Integrating Program Analysis and Repository Mining

Software projects are complex technical and organizational systems involving large numbers of artifacts and developers. To understand and tame software complexity, a wide variety of program analysis techniques have been developed for bug detection, program comprehension, verification, and more. At the same time, repository mining techniques aim at obtaining insights into the inner socio-technical workings of software projects at a larger scale. While both program analysis and repository mining have been successful on their own, they are largely isolated, which leaves considerable potential for synergies untapped. We present SEAL, the first integrated approach that combines low-level program analysis with high-level repository information. SEAL maps repository information, mined from the development history of a project, onto a low-level intermediate program representation, making it available for state-of-the-art program analysis. SEAL’s integrated approach allows us to efficiently address software engineering problems that span multiple levels of abstraction, from low-level data flow to high-level organizational information. To demonstrate its merits and practicality, we use SEAL to determine which code changes modify central parts of a given software project, how authors interact (indirectly) with each other through code, and we demonstrate that putting static analysis’ results into a socio-technical context improves their expressiveness and interpretability.

Predicting Aging-Related Bugs Using Network Analysis on Aging-Related Dependency Networks

Software aging, a phenomenon that exhibits an increasing failure rate or progressive performance degradation in long-running software systems, has caused serious cost damage or even loss of human lives. To aid aging-related bug (ARB, whose activation can result in software aging) detection and removal before software release, ARB prediction was proposed. Based on the prediction results, software teams can allocate limited testing resources to ARB-prone modules. Previous research has proposed several methods for both within-project and cross-project ARB prediction. However, they are based on the same set of metrics focusing on the contents of a single module, and only six metrics are aging-related. In this paper, we develop aging-related network measures by constructing an aging-related dependency network to model the flow of aging-related information in the software. Our evaluation on three commonly used open-source projects reveals that aging-related network measures show an inconsistent association with ARB-proneness in three projects, and the performance of aging-related network measures varies under different ARB prediction settings.

Performance Bug Analysis and Detection for Distributed Storage and Computing Systems

This article systematically studies 99 distributed performance bugs from five widely deployed distributed storage and computing systems (Cassandra, HBase, HDFS, Hadoop MapReduce and ZooKeeper). We present the TaxPerf database, which collectively organizes the analysis results as over 400 classification labels and over 2,500 lines of bug re-description. TaxPerf is classified into six bug categories (and 18 bug subcategories) by their root causes; resource, blocking, synchronization, optimization, configuration, and logic. TaxPerf can be used as a benchmark for performance bug studies and debug tool designs. Although it is impractical to automatically detect all categories of performance bugs in TaxPerf, we find that an important category of blocking bugs can be effectively solved by analysis tools. We analyze the cascading nature of blocking bugs and design an automatic detection tool called PCatch , which (i) performs program analysis to identify code regions whose execution time can potentially increase dramatically with the workload size; (ii) adapts the traditional happens-before model to reason about software resource contention and performance dependency relationship; and (iii) uses dynamic tracking to identify whether the slowdown propagation is contained in one job. Evaluation shows that PCatch can accurately detect blocking bugs of representative distributed storage and computing systems by observing system executions under small-scale workloads.