Bug Detection Research Articles

Leveraging Modular Architecture for Bug Characterization and Analysis in Automated Driving Software

With the rapid advancement of automated driving technology, numerous manufacturers deploy vehicles with auto-driving features. This highlights the importance of ensuring the quality of automated driving software. To achieve this, characterizing bugs in automated driving software is important, as it can facilitate bug detection and bug fixes, thereby ensuring software quality. Automated driving software typically has a modular architecture, where software is divided into multiple modules, each designed for its own functionality for automated driving. This may lead to varying bug characteristics. Additionally, our recent study has shown a correlation between bugs caused by code clones and the functionalities of modules in automated driving software. Hence, we consider the modular structure when analyzing bug characteristics. In this paper, we analyze 3,078 bugs from two representative open-source Level-4 automated driving systems, Apollo and Autoware. By analyzing the bug report description, title, and developers’ discussions, we have identified 20 bug symptoms and 17 bug-fixing strategies, and analyzed their relationships with the respective modules. Our analysis achieves 12 main findings offering a comprehensive view of bug characteristics in automated driving software. We believe our findings can help developers better understand and manage bugs in automated driving software, thereby improving software quality and reliability.

Early Bug Detection through Shift Left Testing

Shift-Left Testing is a preventive approach in the SW development process of identifying and handling defects where testing is performed before the flow proceeds to the subsequent phases of SDLC. In most situations, testing is done after development, and this means that any defects get discovered later contributing to high costs and more time to complete the project. Essentially, Shift-Left Testing implies that testing should be conducted during the design or the coding stage and is beneficial due to the fact that in those stages of development, it is considerably less expensive to rectify problems that are detected. It uses integrated strategies including continuous integration, static code analysis and automated testing, in which the development and the test team work together from the start. Consequently, the approach results in enhanced quality of the software, their development time, and minimization of the post-release faults. Although Shift-Left Testing is changing many ways in software development for the better, it has some problems, for instance, changing organizational culture and has high demands to test automation frameworks.

SPATA: Effective OS Bug Detection with Summary-Based, Alias-Aware, and Path-Sensitive Typestate Analysis

The operating system (OS) is the cornerstone for computer systems. It manages hardware and provides fundamental service for user-level applications. Thus, detecting bugs in OSes is important to improve the reliability of computer systems. Static typestate analysis is a common technique for detecting various types of bugs, but it is often inaccurate or unscalable for large-size OS code, due to imprecision of identifying alias relationships as well as high costs of typestate tracking, path-feasibility validation, and inter-procedural analysis. In this article, 1 we present SPATA, a novel summary-based, alias-aware, and path-sensitive typestate analysis framework to detect OS bugs. To identify precise alias relationships in the OS code, SPATA performs a path-based alias analysis based on control-flow paths and access paths. With these alias relationships, SPATA reduces the costs of typestate tracking and path-feasibility validation, to accelerate path-sensitive typestate analysis for accurate bug detection. Moreover, SPATA uses an alias-summary-based analysis to accelerate inter-procedural bug detection, without time-consuming alias analysis across functions. We have evaluated SPATA on the Linux kernel and three popular IoT OSes, and it finds 651 real bugs with a false-positive rate of 18%. Besides, our alias-summary-based analysis achieves a 6.7x speedup in bug detection compared to non-summary-based analysis.

The Necessity of System Integration Testing (SIT) and User Acceptance Testing (UAT) in Project Lifecycle

SIT and UAT are the major milestones in the project lifecycle. This paper was able to show the importance of such testing phases in helping to identify bugs early through continuous data-driven reporting and monitoring toward the successful execution of a project. Additionally, this session will cover the importance of regression and end-to-end testing, setting target percentages for the completion of test cases, and putting time aside for retesting. Keywords SIT: System Integration Testing, UAT: User Acceptance Testing, Project Life Cycle, Data-Driven Reporting, Monitoring, Bug Detection, Regression Testing, End-to-End Testing, Target Percentages, Retesting

Bug Detection at Scale

Bug Detection at Scale

Enhancing chip design verification through AI-powered bug detection in RTL code

This paper presents a novel AI-driven approach for enhancing chip design verification through automated bug detection in Register Transfer Level (RTL) code. The proposed method integrates advanced machine learning techniques with domain-specific knowledge of chip design to address the challenges of increasing complexity and time-to-market pressures in modern integrated circuit development. Our system employs a comprehensive data preprocessing pipeline that effectively captures syntactic and semantic features of RTL code, feeding into an innovative attention-based neural network model. The model demonstrates superior bug detection accuracy across diverse design categories and bug types compared to traditional methods and existing AI-assisted approaches. Extensive experimental evaluation on a large-scale dataset of RTL designs, including both open-source projects and industry collaborations, validates the effectiveness of our method. The proposed approach achieves a 95% accuracy in bug detection, with a 28-35% reduction in verification time when applied to real-world chip design projects. The paper addresses the interpretability of AI decisions in the context of chip design, presenting novel visualization techniques that enhance trust and facilitate adoption among RTL designers. While acknowledging current limitations, we discuss future research directions, including integration with formal verification methods and extension to system-level verification scenarios. This work contributes significantly to AI-assisted chip design, paving the way for more efficient and reliable development of complex integrated circuits.

Bug detection in software engineering: which incentives work best?

Bug detection in software engineering: which incentives work best?

Transformer-based models application for bug detection in source code

This paper explores the use of transformer-based models for bug detection in source code, aiming to better understand the capacity of these models to learn complex patterns and relationships within the code. Traditional static analysis tools are highly limited in their ability to detect semantic errors, resulting in numerous defects passing through to the code execution stage. This research represents a step towards enhancing static code analysis using neural networks. The experiments were designed as binary classification tasks to detect buggy code snippets, each targeting a specific defect type such as NameError, TypeError, IndexError, AttributeError, ValueError, EOFError, SyntaxError, and ModuleNotFoundError. Utilizing the «RunBugRun» dataset, which relies on code execution results, the models – BERT, CodeBERT, GPT-2, and CodeT5 – were fine-tuned and compared under identical conditions and hyperparameters. Performance was evaluated using F1-Score, Precision, and Recall. The results indicated that transformer-based models, especially CodeT5 and CodeBERT, were effective in identifying various defects, demonstrating their ability to learn complex code patterns. However, performance varied by defect type, with some defects like IndexError and TypeError being more challenging to detect. The outcomes underscore the importance of high-quality, diverse training data and highlight the potential of transformer-based models to achieve more accurate early defect detection. Future research should further explore advanced transformer architectures for detecting complicated defects, and investigate the integration of additional contextual information to the detection process. This study highlights the potential of modern machine learning architectures to advance software engineering practices, leading to more efficient and reliable software development.

Risky Dynamic Typing-related Practices in Python: An Empirical Study

Python’s dynamic typing nature provides developers with powerful programming abstractions. However, many type-related bugs are accumulated in code bases of Python due to the misuse of dynamic typing. The goal of this article is to aid in the understanding of developers’ high-risk practices toward dynamic typing and the early detection of type-related bugs. We first formulate the rules of six types of risky dynamic typing-related practices (type smells for short) in Python. We then develop a rule-based tool named RUPOR, which builds an accurate type base to detect type smells. Our evaluation shows that RUPOR outperforms the existing type smell detection techniques (including the Large Language Models–based approaches, Mypy, and PYDYPE) on a benchmark of 900 Python methods. Based on RUPOR, we conduct an empirical study on 25 real-world projects. We find that type smells are significantly related to the occurrence of post-release faults. The fault-proneness prediction model built with type smell features slightly outperforms the model built without them. We also summarize the common patterns, including inserting type check to fix type smell bugs. These findings provide valuable insights for preventing and fixing type-related bugs in the programs written in dynamic-typed languages.

On the Impact of Lower Recall and Precision in Defect Prediction for Guiding Search-based Software Testing

Defect predictors, static bug detectors, and humans inspecting the code can propose locations in the program that are more likely to be buggy before they are discovered through testing. Automated test generators such as search-based software testing (SBST) techniques can use this information to direct their search for test cases to likely buggy code, thus speeding up the process of detecting existing bugs in those locations. Often the predictions given by these tools or humans are imprecise, which can misguide the SBST technique and may deteriorate its performance. In this article, we study the impact of imprecision in defect prediction on the bug detection effectiveness of SBST. Our study finds that the recall of the defect predictor, i.e., the proportion of correctly identified buggy code, has a significant impact on bug detection effectiveness of SBST with a large effect size. More precisely, the SBST technique detects 7.5 fewer bugs on average (out of 420 bugs) for every 5% decrements of the recall. However, the effect of precision, a measure for false alarms, is not of meaningful practical significance, as indicated by a very small effect size. In the context of combining defect prediction and SBST, our recommendation is to increase the recall of defect predictors as a primary objective and precision as a secondary objective. In our experiments, we find that 75% precision is as good as 100% precision. To account for the imprecision of defect predictors, in particular low recall values, SBST techniques should be designed to search for test cases that also cover the predicted non-buggy parts of the program, while prioritising the parts that have been predicted as buggy.

MTL-TRANSFER: Leveraging Multi-task Learning and Transferred Knowledge for Improving Fault Localization and Program Repair

Fault localization (FL) and automated program repair (APR) are two main tasks of automatic software debugging. Compared with traditional methods, deep learning-based approaches have been demonstrated to achieve better performance in FL and APR tasks. However, the existing deep learning-based FL methods ignore the deep semantic features or only consider simple code representations. And for APR tasks, existing template-based APR methods are weak in selecting the correct fix templates for more effective program repair, which are also not able to synthesize patches via the embedded end-to-end code modification knowledge obtained by training models on large-scale bug-fix code pairs. Moreover, in most of FL and APR methods, the model designs and training phases are performed separately, leading to ineffective sharing of updated parameters and extracted knowledge during the training process. This limitation hinders the further improvement in the performance of FL and APR tasks. To solve the above problems, we propose a novel approach called MTL-TRANSFER, which leverages a multi-task learning strategy to extract deep semantic features and transferred knowledge from different perspectives. First, we construct a large-scale open-source bug datasets and implement 11 multi-task learning models for bug detection and patch generation sub-tasks on 11 commonly used bug types, as well as one multi-classifier to learn the relevant semantics for the subsequent fix template selection task. Second, an MLP-based ranking model is leveraged to fuse spectrum-based, mutation-based and semantic-based features to generate a sorted list of suspicious statements. Third, we combine the patches generated by the neural patch generation sub-task from the multi-task learning strategy with the optimized fix template selecting order gained from the multi-classifier mentioned above. Finally, the more accurate FL results, the optimized fix template selecting order, and the expanded patch candidates are combined together to further enhance the overall performance of APR tasks. Our extensive experiments on widely-used benchmark Defects4J show that MTL-TRANSFER outperforms all baselines in FL and APR tasks, proving the effectiveness of our approach. Compared with our previously proposed FL method TRANSFER-FL (which is also the state-of-the-art statement-level FL method), MTL-TRANSFER increases the faults hit by 8/11/12 on Top-1/3/5 metrics (92/159/183 in total). And on APR tasks, the number of successfully repaired bugs of MTL-TRANSFER under the perfect localization setting reaches 75, which is 8 more than our previous APR method TRANSFER-PR. Furthermore, another experiment to simulate the actual repair scenarios shows that MTL-TRANSFER can successfully repair 15 and 9 more bugs (56 in total) compared with TBar and TRANSFER, which demonstrates the effectiveness of the combination of our optimized FL and APR components.

Adaptive scheduling-based fine-grained greybox fuzzing for cloud-native applications

Coverage-guided fuzzing is one of the most popular approaches to detect bugs in programs. Existing work has shown that coverage metrics are a crucial factor in guiding fuzzing exploration of targets. A fine-grained coverage metric can help fuzzing to detect more bugs and trigger more execution states. Cloud-native applications that written by Golang play an important role in the modern computing paradigm. However, existing fuzzers for Golang still employ coarse-grained block coverage metrics, and there is no fuzzer specifically for cloud-native applications, which hinders the bug detection in cloud-native applications. Using fine-grained coverage metrics introduces more seeds and even leads to seed explosion, especially in large targets such as cloud-native applications. Therefore, we employ an accurate edge coverage metric in fuzzer for Golang, which achieves finer test granularity and more accurate coverage information than block coverage metrics. To mitigate the seed explosion problem caused by fine-grained coverage metrics and large target sizes, we propose smart seed selection and adaptive task scheduling algorithms based on a variant of the classical adversarial multi-armed bandit (AMAB) algorithm. Extensive evaluation of our prototype on 16 targets in real-world cloud-native infrastructures shows that our approach detects 233% more bugs than go-fuzz, achieving an average coverage improvement of 100.7%. Our approach effectively mitigates seed explosion by reducing the number of seeds generated by 41% and introduces only 14% performance overhead.

DinoDroid: Testing Android Apps Using Deep Q-Networks

The large demand of mobile devices creates significant concerns about the quality of mobile applications (apps). Developers need to guarantee the quality of mobile apps before it is released to the market. There have been many approaches using different strategies to test the GUI of mobile apps. However, they still need improvement due to their limited effectiveness. In this article, we propose DinoDroid, an approach based on deep Q-networks to automate testing of Android apps. DinoDroid learns a behavior model from a set of existing apps and the learned model can be used to explore and generate tests for new apps. DinoDroid is able to capture the fine-grained details of GUI events (e.g., the content of GUI widgets) and use them as features that are fed into deep neural network, which acts as the agent to guide app exploration. DinoDroid automatically adapts the learned model during the exploration without the need of any modeling strategies or pre-defined rules. We conduct experiments on 64 open-source Android apps. The results showed that DinoDroid outperforms existing Android testing tools in terms of code coverage and bug detection.

Generating Python Type Annotations from Type Inference: How Far Are We?

In recent years, dynamic languages such as Python have become popular due to their flexibility and productivity. The lack of static typing makes programs face the challenges of fixing type errors, early bug detection, and code understanding. To alleviate these issues, PEP 484 introduced optional type annotations for Python in 2014, but unfortunately, a large number of programs are still not annotated by developers. Annotation generation tools can utilize type inference techniques. However, several important aspects of type annotation generation are overlooked by existing works, such as in-depth effectiveness analysis, potential improvement exploration, and practicality evaluation. And it is unclear how far we have been and how far we can go. In this paper, we set out to comprehensively investigate the effectiveness of type inference tools for generating type annotations, applying three categories of state-of-the-art tools on a carefully-cleaned dataset. First, we use a comprehensive set of metrics and categories, finding that existing tools have different effectiveness and cannot achieve both high accuracy and high coverage. Then, we summarize six patterns to present the limitations in type annotation generation. Next, we implement a simple but effective tool to demonstrate that existing tools can be improved in practice. Finally, we conduct a controlled experiment showing that existing tools can reduce the time spent annotating types and determine more precise types, but cannot reduce subjective difficulty. Our findings point out the limitations and improvement directions in type annotation generation, which can inspire future work.

Graphuzz: Data-driven Seed Scheduling for Coverage-guided Greybox Fuzzing

Seed scheduling is a critical step of greybox fuzzing, which assigns different weights to seed test cases during seed selection, and significantly impacts the efficiency of fuzzing. Existing seed scheduling strategies rely on manually designed models to estimate the potentials of seeds and determine their weights, which fails to capture the rich information of a seed and its execution and thus the estimation of seeds’ potentials is not optimal. In this paper, we introduce a new seed scheduling solution, Graphuzz, for coverage-guided greybox fuzzing, which utilizes deep learning models to estimate the potentials of seeds and works in a data-driven way. Specifically, we propose an extended control flow graph called e-CFG to represent the control-flow and data-flow features of a seed’s execution, which is suitable for graph neural networks (GNN) to process and estimate seeds’ potential. We evaluate each seed’s code coverage increment and use it as the label to train the GNN model. Further, we propose a self-attention mechanism to enhance the GNN model so that it can capture overlooked features. We have implemented a prototype of Graphuzz based on the baseline fuzzer AFLplusplus. The evaluation results show that our model can estimate the potential of seeds and has the robust capability to generalize to different targets. Further, the evaluation using 12 benchmarks from FuzzBench shows that Graphuzz outperforms AFLplusplus and the state-of-the-art seed scheduling solution K-Scheduler and other coverage-guided fuzzers in terms of code coverage, and the evaluation using 8 benchmarks from Magma shows that Graphuzz outperforms the baseline fuzzer AFLplusplus and SOTA solutions in terms of bug detection.

Mindfulness and Emotional Intelligence: A Study on Millennials on Hospitality

In software development, dependability and quality are critical. A Bug Tracking System (BTS) designed specifically for large-scale software projects is introduced in this significant project with the goal of streamlining bug tracking, reporting, detection, and resolution. The system provides secure authentication, a backend database that is reliable, and an easy-to-use web interface. Using cutting-edge web technologies, it offers critical features including automated notifications, progress tracking, severity assignment, and thorough bug reporting together with user-friendly interfaces for a range of user roles. Tools for data visualization help identify defect patterns and monitor projects. The project follows best practices recommended by the industry for requirements analysis, design, implementation, testing, and deployment, and it solicits ongoing input. The result is a thorough BTS that addresses defect management difficulties. Thorough testing and assessment in comparison to standards confirms its efficacy. This approach could have a big impact on software development, improving user satisfaction and quality. Furthermore, issues with bug tracking are covered, including distributed team communication, scalability, and automation balancing. Proposals for strategies to address these issues are grounded in scholarly research and professional opinions. This summary highlights the critical role that bug tracking systems play in contemporary software development, making it an invaluable tool for researchers, practitioners, and teams looking to improve bug tracking procedures and raise the caliber and effectiveness of the software development lifecycle. Keywords— Bug Tracking System, Software Development, Defect Management, Quality Assurance, Automation, Collaboration, Large-scale Projects.

Bug Tracking System

For many years, bug-tracking mechanisms have been employed only in some of the large software development houses. Most of the others never bothered with bug tracking at all, and instead simply relied on shared lists and email to monitor the status of defects. This procedure is error-prone and tends to cause those bugs judged least significant by developers to be dropped or ignored. Bug Tracking System is an ideal solution to track the bugs of a product, solution or an application. Bug Tracking System allows individuals or groups of developers to keep track of outstanding bugs in their product effectively. This can also be called the Defect Tracking System. The Bug Tracking System can dramatically increase the productivity and accountability of individual employees by providing a documented workflow and positive feedback for good performance. Moreover, the project explores the use of machine learning for automated root cause analysis, assisting developers in identifying underlying issues contributing to bug occurrences. Recommendation systems are also developed to suggest potential fixes or solutions for reported bugs based on past incidents and code repositories, aiding developers in efficient bug resolution. Through these ML-driven functionalities, the proposed bug tracking software aims to revolutionize the bug management process, enabling faster bug detection, prioritization, and resolution, thereby enhancing software quality, user satisfaction, and overall development productivity. Key Words: Bug Tacking system, apps, users.

A Learning-Based Approach to Static Program Slicing

Traditional program slicing techniques are crucial for early bug detection and manual/automated debugging of online code snippets. Nevertheless, their inability to handle incomplete code hinders their real-world applicability in such scenarios. To overcome these challenges, we present NS-Slicer, a novel learning-based approach that predicts static program slices for both complete and partial code Our tool leverages a pre-trained language model to exploit its understanding of fine-grained variable-statement dependencies within source code. With this knowledge, given a variable at a specific location and a statement in a code snippet, NS-Slicer determines whether the statement belongs to the backward slice or forward slice, respectively. We conducted a series of experiments to evaluate NS-Slicer's performance. On complete code, it predicts the backward and forward slices with an F1-score of 97.41% and 95.82%, respectively, while achieving an overall F1-score of 96.77%. Notably, in 85.20% of the cases, the static program slices predicted by NS-Slicer exactly match entire slices from the oracle. For partial programs, it achieved an F1-score of 96.77%–97.49% for backward slicing, 92.14%–95.40% for forward slicing, and an overall F1-score of 94.66%–96.62%. Furthermore, we demonstrate NS-Slicer's utility in vulnerability detection (VD), integrating its predicted slices into an automated VD tool. In this setup, the tool detected vulnerabilities in Java code with a high F1-score of 73.38%. We also include the analyses studying NS-Slicer’s promising performance and limitations, providing insights into its understanding of intrinsic code properties such as variable aliasing, leading to better slicing.

Python source code Analysis for Bug Detection using Transformers

Effective bug detection is pivotal in software development, with the identification and localization of defects being crucial for robust applications. In Python-based programs, the conventional bug detection process relies on a Python interpreter, causing workflow interruptions due to sequential error detection. As Python's adoption surges, the demand for efficient bug detection tools intensifies. This paper addresses the challenges associated with bug detection in Python, focusing on the prevalence of built-in type bugs that can lead to code crashes. Building on recent advancements, this survey explores bug detection methodologies across programming languages, emphasizing Python, JavaScript, and C. The diverse array of techniques covered includes static and dynamic analyses, machine learning-based bug detection, and predictive analysis engines(specifically deep-learning based). The survey provides insights into bug detection in Python programs, offering perspectives on addressing built-in type bugs and optimizing tools within the language's constraints.

Drone-Based Bug Detection in Orchards with Nets: A Novel Orienteering Approach

The use of drones for collecting information and detecting bugs in orchards covered by nets is a challenging problem. The nets help in reducing pest damage, but they also constrain the drone’s flight path, making it longer and more complex. To address this issue, we model the orchard as an aisle-graph, a regular data structure that represents consecutive aisles where trees are arranged in straight lines. The drone flies close to the trees and takes pictures at specific positions for monitoring the presence of bugs, but its energy is limited, so it can only visit a subset of positions. To tackle this challenge, we introduce the Single-drone Orienteering Aisle-graph Problem (SOAP), a variant of the orienteering problem, where likely infested locations are prioritized by assigning them a larger profit. Additionally, the drone’s movements have a cost in terms of energy, and the objective is to plan a drone’s route in the most profitable locations under a given drone’s battery. We show that SOAP can be optimally solved in polynomial time, but for larger orchards/instances, we propose faster approximation and heuristic algorithms. Finally, we evaluate the algorithms on synthetic and real datasets to demonstrate their effectiveness and efficiency.