Concurrency Bugs Research Articles

Cross-project concurrency bug prediction using domain-adversarial neural network

In recent years, software bug prediction has shown to be effective in narrowing down the potential bug modules and boosting the efficiency and precision of existing testing and analysis tools. However, due to its non-deterministic nature and low presence, concurrency bug labeling is a challenging task, which limits the implementation of within-project concurrency bug prediction. This paper proposes DACon, a Domain-Adversarial neural network-based cross-project Concurrency bug prediction approach to tackle this problem by leveraging information from another related project. By combining a set of designed concurrency code metrics with widely used sequential code metrics, DACon uses SMOTE (Synthetic Minority Over-sampling TEchnique) and domain-adversarial neural network to mitigate two challenges including the severe class imbalance between concurrency bug-prone samples and concurrency bug-free samples, and shift between source and target distribution during bug prediction implementation. Our evaluation on 20 pair-wise groups of experiments constructed from 5 real-world projects indicates that cross-project concurrency bug prediction is feasible, and DACon can effectively predict concurrency bugs across different projects.

Predictive Monitoring against Pattern Regular Languages

While current bug detection techniques for concurrent software focus on unearthing low-level issues such as data races or deadlocks, they often fall short of discovering more intricate temporal behaviours that can arise even in the absence of such low-level issues. In this paper, we focus on the problem of dynamically analysing concurrent software against high-level temporal specifications such as LTL. Existing techniques for runtime monitoring against such specifications are primarily designed for sequential software and remain inadequate in the presence of concurrency — violations may be observed only in intricate thread interleavings, requiring many re-runs of the underlying software in conjunction with the analysis. Towards this, we study the problem of predictive runtime monitoring , inspired by the analogous problem of predictive data race detection studied extensively recently. The predictive runtime monitoring question asks, given an execution σ, if it can be soundly reordered to expose violations of a specification. In general, this problem may become easily intractable when either the specifications or the notion of reorderings used is complex. In this paper, we focus on specifications that are given in regular languages. Our notion of reorderings is trace equivalence , where an execution is considered a reordering of another if it can be obtained from the latter by successively commuting adjacent independent actions. We first show that, even in this simplistic setting, the problem of predictive monitoring admits a super-linear lower bound of O ( n α ), where n is the number of events in the execution, and α is a parameter describing the degree of commutativity, and typically corresponds to the number of threads in the execution. As a result, predictive runtime monitoring even in this setting is unlikely to be efficiently solvable, unlike in the non-predictive setting where the problem can be checked using a deterministic finite automaton (and thus, a constant-space streaming linear-time algorithm). Towards this, we identify a sub-class of regular languages, called pattern languages (and their extension generalized pattern languages ). Pattern languages can naturally express specific ordering of some number of (labelled) events, and have been inspired by popular empirical hypotheses underlying many concurrency bug detection approaches such as the “small bug depth” hypothesis. More importantly, we show that for pattern (and generalized pattern) languages, the predictive monitoring problem can be solved using a constant-space streaming linear-time algorithm. We implement and evaluate our algorithm PatternTrack on benchmarks from the literature and show that it is effective in monitoring large-scale applications.

R 3 : Record-Replay-Retroaction for Database-Backed Applications

Developers would benefit greatly from time travel: being able to faithfully replay past executions and retroactively execute modified code on past events. Currently, replay and retroaction are impractical because they require expensively capturing fine-grained timing information to reproduce concurrent accesses to shared state. In this paper, we propose practical time travel for database-backed applications , an important class of distributed applications that access shared state through transactions. We present R 3 , a novel Record-Replay-Retroaction tool. R 3 implements a lightweight interceptor to record concurrency information for applications at transaction-level granularity, enabling replay and retroaction with minimal overhead. We address key challenges in both replay and retroaction. First, we design a novel algorithm for faithfully reproducing application requests running with snapshot isolation, allowing R 3 to support most production DBMSs. Second, we develop a retroactive execution mechanism that provides high fidelity with the original trace while supporting nearly arbitrary code modifications. We demonstrate how R 3 simplifies debugging for real, hard-to-reproduce concurrency bugs from popular open-source web applications. We evaluate R 3 using TPC-C and microservice workloads and show that R 3 always-on recording has a small performance overhead (<25% for point queries but <0.1% for complex transactions like in TPC-C) during normal application execution and that R 3 can retroactively execute bugfixed code over recorded traces within 0.11--0.78× of the original execution time.

Sound Dynamic Deadlock Prediction in Linear Time

Deadlocks are one of the most notorious concurrency bugs, and significant research has focused on detecting them efficiently. Dynamic predictive analyses work by observing concurrent executions, and reason about alternative interleavings that can witness concurrency bugs. Such techniques offer scalability and sound bug reports, and have emerged as an effective approach for concurrency bug detection, such as data races. Effective dynamic deadlock prediction, however, has proven a challenging task, as no deadlock predictor currently meets the requirements of soundness, high-precision, and efficiency. In this paper, we first formally establish that this tradeoff is unavoidable, by showing that (a) sound and complete deadlock prediction is intractable, in general, and (b) even the seemingly simpler task of determining the presence of potential deadlocks, which often serve as unsound witnesses for actual predictable deadlocks, is intractable. The main contribution of this work is a new class of predictable deadlocks, called sync(hronization)-preserving deadlocks. Informally, these are deadlocks that can be predicted by reordering the observed execution while preserving the relative order of conflicting critical sections. We present two algorithms for sound deadlock prediction based on this notion. Our first algorithm SPDOffline detects all sync-preserving deadlocks, with running time that is linear per abstract deadlock pattern, a novel notion also introduced in this work. Our second algorithm SPDOnline predicts all sync-preserving deadlocks that involve two threads in a strictly online fashion, runs in overall linear time, and is better suited for a runtime monitoring setting. We implemented both our algorithms and evaluated their ability to perform offline and online deadlock-prediction on a large dataset of standard benchmarks. Our results indicate that our new notion of sync-preserving deadlocks is highly effective, as (i) it can characterize the vast majority of deadlocks and (ii) it can be detected using an online, sound, complete and highly efficient algorithm.

Hippodrome: Data Race Repair Using Static Analysis Summaries

Implementing bug-free concurrent programs is a challenging task in modern software development. State-of-the-art static analyses find hundreds of concurrency bugs in production code, scaling to large codebases. Yet, fixing these bugs in constantly changing codebases represents a daunting effort for programmers, particularly because a fix in the concurrent code can introduce other bugs in a subtle way.In this work, we show how to harness compositional static analysis for concurrency bug detection, to enable a new Automated Program Repair (APR) technique for data races in large concurrent Java codebases. The key innovation of our work is an algorithm that translates procedure summaries inferred by the analysis tool for the purpose of bug reporting into small local patches that fix concurrency bugs (without introducing new ones). This synergy makes it possible to extend the virtues of compositional static concurrency analysis to APR, making our approacheffective(it can detect and fix many more bugs than existing tools for data race repair),scalable(it takes seconds to analyze and suggest fixes for sizeable codebases), andusable(generally, it does not require annotations from the users and can performcontinuousautomated repair). Our study, conducted on popular open-source projects, has confirmed that our tool automatically produces concurrency fixes similar to those proposed by the developers in the past.

GC-MCR: Directed Graph Constraint-guided Concurrent Bug Detection Method

GC-MCR: Directed Graph Constraint-guided Concurrent Bug Detection Method

Achieving High MAP-Coverage Through Pattern Constraint Reduction

Testing multi-threaded programs is challenging due to the enormous space of thread interleavings. Recently, a code coverage criterion for multi-threaded programs called MAP-coverage has been proposed and shown to be effective for testing concurrent programs. Existing approaches for achieving high MAP-coverage are based on random testing with simple heuristics, which is ineffective in systematically triggering rare thread interleavings. In this study, we propose a novel approach called pattern constraint reduction (PCR), which employs optimized constraint solving to generate thread interleavings for high MAP-coverage. The idea is to iteratively encode and solve path conditions to generate thread interleavings which are guaranteed to improve MAP-coverage. Furthermore, we effectively apply interpolation techniques to reduce the efforts of constraint solving by avoiding solving infeasible constraints. The experiment results on 20 benchmark programs show that our approach complements existing random testing based approaches when there are rare failure-inducing interleaving in the whole search space. Specifically, PCR finds concurrency bugs faster in 18 out of 20 programs, with an average speedup of 4.2x and a maximum speedup of 11.4x.

The Discourje project: run-time verification of communication protocols in Clojure

To simplify shared-memory concurrent programming, languages have started to offer core support for high-level communications primitives, in the form of message passing though channels, in addition to lower-level synchronisation primitives. Yet, a growing body of evidence suggests that channel-based programming abstractions also have their issues. The Discourje project aims to help programmers cope with channels and concurrency bugs in Clojure programs, based on dynamic analysis. The idea is that programmers write not only implementations of communication protocols in their Clojure programs, but also specifications. Discourje then offers a run-time verification library to ensure that channel actions in implementations are safe relative to specifications. The aim of this paper is to provide a comprehensive overview of the current state of Discourje, including case studies, theoretical foundations, and practical aspects.

BiRD: Race Detection in Software Binaries under Relaxed Memory Models

Instruction reordering and interleavings in program execution under relaxed memory semantics result in non-intuitive behaviors, making it difficult to provide assurances about program correctness. Studies have shown that up to 90% of the concurrency bugs reported by state-of-the-art static analyzers are false alarms. As a result, filtering false alarms and detecting real concurrency bugs is a challenging problem. Unsurprisingly, this problem has attracted the interest of the research community over the past few decades. Nonetheless, many of the existing techniques rely on analyzing source code, rarely consider the effects introduced by compilers, and assume a sequentially consistent memory model. In a practical setting, however, developers often do not have access to the source code, and even commodity architectures such as x86 and ARM are not sequentially consistent. In this work, we present B i rd , a prototype tool, to dynamically detect harmful data races in x86 binaries under relaxed memory models, TSO and PSO. B i rd employs source-DPOR to explore all distinct feasible interleavings for a multithreaded application. Our evaluation of B i rd on 42 publicly available benchmarks and its comparison with the state-of-the-art tools indicate B i rd ’s potential in effectively detecting data races in software binaries.

Fuzzing with automatically controlled interleavings to detect concurrency bugs

Concurrency vulnerabilities are an irresistible threat to security, and detecting them is challenging. Triggering the concurrency vulnerabilities requires a specific thread interleaving and a bug-inducing input. Existing methods have focused on one of these but not both or have experimented with small programs, which raises scalability issues.This paper introduces AutoInter-fuzzing, a fuzzer controlling thread interleavings elaborately and providing an interleaving-aware power schedule to detect vulnerabilities in a multi-threaded program. AutoInter-fuzzing consists of static analysis and dynamic fuzzing. At the static analysis, the fuzzer extracts and optimizes the interleaving space to be explored and adds instrumentation to control thread interleavings. We apply the power schedule in the dynamic fuzzing to focus on the seeds that reveal the new interleaving space. The fuzzer records the interleaving information in a log when a crash occurs and uses it to reproduce and validate the crash.Experiments with 13 real-world multi-threaded programs show that the interleaving-aware power schedule effectively enlarges the untested interleaving space, and AutoInter-fuzzing outperforms AFL and ConAFL in detecting interleaving-relevant vulnerabilities. AutoInter-fuzzing has detected six interleaving-relevant vulnerabilities, including two new vulnerabilities and four interleaving-irrelevant vulnerabilities.

Model checking, testing and debugging

This issue contains four papers. The first paper focuses on model checking, the second and third papers focus on testing and the last paper focuses on debugging. The first paper, ‘Model checking C++ programs’ by Felipe R. Monteiro, Mikhail R. Gadelha and Lucas C. Cordeiro, is motivated by memory safety issues and how these have proved to be a source of security vulnerabilities. The authors devised a novel bounded model checking approach. The first step was to encode a number of C++ features in a decidable fragment of first-order logic. SMT solvers were then used to carry out verification. In experiments, the proposed approach was found to outperform state-of-the-art verifiers. The prototype tool also found arithmetic-overflow errors in a commercial application. (Recommended by Professor Pretscher) The second paper, ‘GPU acceleration of finite state machine input execution: Improving scale and performance’, by Vanya Yaneva, Ajitha Rajan and Christophe Dubach looks at the problem of executing a large number of tests on a finite state machine (FSM). The motivation for this work is model validation. The approach devised uses GPUs to allow multiple tests to be run in parallel. The authors built on their previous work, which showed how FSM execution can be performed on a GPU, by addressing a number of limitations. In particular, the authors addressed the data transfer overhead and they also performed experiments with FSMs that were too large to fit into GPU memory. In the experiments, the novel optimisations led to further improvements, with the GPU being over four times faster, on average, than a 16-core CPU. (Recommended by Professor Pretscher) The third paper, ‘Survey on test case generation, selection and prioritization for cyber-physical systems’, by Zahra Sadri-Moshkenani, Justin Bradley and Gregg Rothermel, presents a survey of approaches that generate, select or prioritise test cases for cyber-physical systems. The authors identified 34 related papers (26 papers on test generation, 6 papers on test selection and 7 papers on test prioritisation) and classified them according to 8 properties distilled by the authors from past experience. From the survey results, the authors identified a number of open challenges. To address some of these challenges, existing approaches may be adapted or new approaches may be developed. (Recommended by Professor Phil McMinn) The fourth paper, ‘Effective fault localization and context-aware debugging for concurrent programs’, by Justin Chu, Tingting Yu, Jane Huffman Hayes, Xue Han and Yu Zhao, presents Coadec, an approach for automatically generating interthread control flow paths to diagnose concurrency bugs. Coadec consists of two phases: concurrency fault localization and context-aware debugging. The authors evaluated Coadec on 10 real-world multithreaded Java applications and showed that Coadec outperforms state-of-the-art approaches for localising concurrency faults and that Coadec's context debugging can help developers understand concurrency fault by inspecting a small percentage of code. (Recommended by Marc Roper)

Effective fault localization and context‐aware debugging for concurrent programs

SummaryConcurrent programs are difficult to debug because concurrency faults usually occur under specific inputs and thread interleavings. Fault localization techniques for sequential programs are often ineffective because the root causes of concurrency faults involve memory accesses across multiple threads rather than single statements. Previous research has proposed techniques to analyse passing and failing executions obtained from running a set of test cases for identifying faulty memory access patterns. However, stand‐alone access patterns do not provide enough contextual information, such as the path leading to the failure, for developers to understand the bug. We present an approach, Coadec, to automatically generate interthread control flow paths that can link memory access patterns that occurred most frequently in the failing executions to better diagnose concurrency bugs. Coadec consists of two phases. In the first phase, we use feature selection techniques from machine learning to localize suspicious memory access patterns based on failing and passing executions. The patterns with maximum feature diversity information can point to the most suspicious pattern. We then apply a data mining technique and identify the memory access patterns that occurred most frequently in the failing executions. Finally, Coadec identifies faulty program paths by connecting both the frequent patterns and the suspicious pattern. We also evaluate the effectiveness of fault localization using test suites generated from different test adequacy criteria. We introduce and have evaluated Coadec on 10 real‐world multithreaded Java applications. Results indicate that Coadec outperforms state‐of‐the‐art approaches for localizing concurrency faults and that Coadec's context debugging can help developers understand concurrency fault by inspecting a small percentage of code.

On interleaving space exploration of multi-threaded programs

Exploring the interleaving space of a multithreaded program to efficiently detect concurrency bugs is important but also difficult because of the astronomically many thread schedules. This paper presents a novel framework to decompose a thread schedule generator that explores the interleaving space into the composition of a basic generator and its extension under the “small interleaving hypothesis”. Under this framework, we in-depth analyzed research work on interleaving space exploration, illustrated how to design an effective schedule generator, and shed light on future research opportunities.

GoDetector: Detecting Concurrent Bug in Go

Go provides a new concurrency mechanism based on message-passing, which brings a series of new concurrency bugs. The existing tools are highly dependent on model-checking methods in which they take the source program as the input of model-checking tools to detect the potential concurrent bugs. However, these methods can only qualitatively obtain the concurrency of the program but not obtain the exact number of concurrency bugs in the program. Meanwhile, these approaches are not sensitive to dead codes, which are prone to false negatives. Furthermore, few works have been done on detecting WaitGroup-related bugs in Go concurrent programming, which may cause deadlock. To this end, this paper proposes a novel approach GoDetector to detect concurrent bugs. GoDetector uses a detecting algorithm and a pushdown automaton to verify the channel-related concurrent bugs and WaitGroup-related concurrent bugs separately. To support the evaluation of the tool, 30 benchmarks are collected from the existing tools. GoDetector reports 2 channel safety errors, 29 deadlocks, and 11 WaitGroup-related errors. Compared with the existing tool Godel, GoDetector has the capacity to report more concurrency bugs and fewer false positives. GoDetector also offers additional support for the detection of WaitGroup variables. The experimental results indicate that GoDetector is more efficient at detecting the concurrency in regards to accuracy and diversity.

Learning-based controlled concurrency testing

Concurrency bugs are notoriously hard to detect and reproduce. Controlled concurrency testing (CCT) techniques aim to offer a solution, where a scheduler explores the space of possible interleavings of a concurrent program looking for bugs. Since the set of possible interleavings is typically very large, these schedulers employ heuristics that prioritize the search to “interesting” subspaces. However, current heuristics are typically tuned to specific bug patterns, which limits their effectiveness in practice. In this paper, we present QL, a learning-based CCT framework where the likelihood of an action being selected by the scheduler is influenced by earlier explorations. We leverage the classical Q-learning algorithm to explore the space of possible interleavings, allowing the exploration to adapt to the program under test, unlike previous techniques. We have implemented and evaluated QL on a set of microbenchmarks, complex protocols, as well as production cloud services. In our experiments, we found QL to consistently outperform the state-of-the-art in CCT.

Actor concurrency bugs: a comprehensive study on symptoms, root causes, API usages, and differences

Actor concurrency is becoming increasingly important in the development of real-world software systems. Although actor concurrency may be less susceptible to some multithreaded concurrency bugs, such as low-level data races and deadlocks, it comes with its own bugs that may be different. However, the fundamental characteristics of actor concurrency bugs, including their symptoms, root causes, API usages, examples, and differences when they come from different sources are still largely unknown. Actor software development can significantly benefit from a comprehensive qualitative and quantitative understanding of these characteristics, which is the focus of this work, to foster better API documentation, development practices, testing, debugging, repairing, and verification frameworks. To conduct this study, we take the following major steps. First, we construct a set of 186 real-world Akka actor bugs from Stack Overflow and GitHub via manual analysis of 3,924 Stack Overflow questions, answers, and comments and 3,315 GitHub commits, messages, original and modified code snippets, issues, and pull requests. Second, we manually study these actor bugs and their fixes to understand and classify their symptoms, root causes, and API usages. Third, we study the differences between the commonalities and distributions of symptoms, root causes, and API usages of our Stack Overflow and GitHub actor bugs. Fourth, we discuss real-world examples of our actor bugs with these symptoms and root causes. Finally, we investigate the relation of our findings with those of previous work and discuss their implications. A few findings of our study are: (1) symptoms of our actor bugs can be classified into five categories, with Error as the most common symptom and Incorrect Exceptions as the least common, (2) root causes of our actor bugs can be classified into ten categories, with Logic as the most common root cause and Untyped Communication as the least common, (3) a small number of Akka API packages are responsible for most of API usages by our actor bugs, and (4) our Stack Overflow and GitHub actor bugs can differ significantly in commonalities and distributions of their symptoms, root causes, and API usages. While some of our findings agree with those of previous work, others sharply contrast.

VERIFICATION AND VALIDATION OF A SOFTWARE: A REVIEW OF THE LITERATURE

With the development of the Internet, making software is often essential, also it is complicated to succeed in the project’s development. There is a necessity in delivering software of top quality. It might be accomplished through using the procedures of Verification and Validation (V&V) via development processes. The main aim of the V&V has been checking if the created software is meeting the needs and specifications of clients. V&V has been considered as collections related to testing as well as analysis activities across the software’s full life cycle. Quick developments in software V&V were of high importance in developing approaches and tools for identifying possible concurrent bugs and therefore verifying the correctness of software. It has been reflecting the modern software V&V concerning efficiency. The main aim of this study has been retrospective review related to various researches in software V&V and conduct a comparison between them.
 In the modern competitive world related to the software, the developers of software must be delivering on-time quality products, also the developers should be verifying that the software has been properly functioning and validating the product for each one of the client’s requirements. The significance of V&V in the development of software has been maintaining the quality of software. The approaches of V&V have been utilized in all stages of the System Development Life Cycle. Furthermore, the presented study also provides objectives of V&V and describes V&V tools that can be used in the process of software development, the way of improving the software’s quality.

CloudRaid: Detecting Distributed Concurrency Bugs via Log Mining and Enhancement

Cloud systems suffer from distributed concurrency bugs, which often lead to data loss and service outage. This paper presents <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CloudRaid</small> , a new automatical tool for finding distributed concurrency bugs efficiently and effectively. Distributed concurrency bugs are notoriously difficult to find as they are triggered by untimely interaction among nodes, i.e., unexpected message orderings. To detect concurrency bugs in cloud systems efficiently and effectively, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CloudRaid</small> analyzes and tests automatically only the message orderings that are likely to expose errors. Specifically, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CloudRaid</small> mines the logs from previous executions to uncover the message orderings that are feasible but inadequately tested. In addition, we also propose a log enhancing technique to introduce new logs automatically in the system being tested. These extra logs added improve further the effectiveness of <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CloudRaid</small> without introducing any noticeable performance overhead. Our log-based approach makes it well-suited for live systems. We have applied <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CloudRaid</small> to analyze six representative distributed systems: Hadoop2/Yarn, HBase, HDFS, Cassandra, Zookeeper, and Flink. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CloudRaid</small> has succeeded in testing 60 different versions of these six systems (10 versions per system) in 35 hours, uncovering 31 concurrency bugs, including nine new bugs that have never been reported before. For these nine new bugs detected, which have all been confirmed by their original developers, three are critical and have already been fixed.

Dynamic Analysis Method for Concurrency Bugs in Multi-process/Multi-thread Environments

In this paper, we propose a method of analyzing the types and causes of concurrency bugs that can occur in multi-process/multi-thread environments by analyzing the information gathered in the execution environment. Our method reduces the false detection rate by employing a hooking technique that intercepts the software’s execution at runtime and reduces the overhead that can occur in the original software due to defect detection methods using optimized data collection. We implemented the proposed method as a tool and demonstrated its effectiveness by applying it to weapon system software that previously had concurrency bugs. In addition, the proposed method’s high performance was proven by applying it to software into which faults were injected and comparing the results with those obtained using other tools.

Concurrent Bug Finding Based on Bounded Model Checking

Automated and reliable software verification is of crucial importance for development of high-quality software. Formal methods can be used for finding different kinds of bugs without executing the software, for example, for finding possible run-time errors. The methods like model checking and symbolic execution offer very precise static analysis but on real world programs do not always scale well. One way to tackle the scalability problem is to apply new concurrent and sequential approaches to complex algorithms used in these kinds of software analysis. In this paper, we compare different variants of bounded model checking and propose two concurrent approaches: concurrency of intra-procedural analysis and concurrency of inter-procedural analysis. We implemented these approaches in a software verification tool LAV, a tool that is based on bounded model checking and symbolic execution. For assessing the improvements gained, we experimentally compared the concurrent approaches with the standard bounded model checking approach (where all correctness conditions are put into a single compound formula) and with a sequential approach (where correctness conditions are checked separately, one after the other). The results show that, in many cases, the proposed concurrent approaches give significant improvements.