Concurrency Bugs Research Articles

Disentanglement in nested-parallel programs

Nested parallelism has proved to be a popular approach for programming the rapidly expanding range of multicore computers. It allows programmers to express parallelism at a high level and relies on a run-time system and a scheduler to deliver efficiency and scalability. As a result, many programming languages and extensions that support nested parallelism have been developed, including in C/C++, Java, Haskell, and ML. Yet, writing efficient and scalable nested parallel programs remains challenging, primarily due to difficult concurrency bugs arising from destructive updates or effects. For decades, researchers have argued that functional programming can simplify writing parallel programs by allowing more control over effects but functional programs continue to underperform in comparison to parallel programs written in lower-level languages. The fundamental difficulty with functional languages is that they have high demand for memory, and this demand only grows with parallelism. In this paper, we identify a memory property, called disentanglement, of nested-parallel programs, and propose memory management techniques for improved efficiency and scalability. Disentanglement allows for (destructive) effects as long as concurrently executing threads do not gain knowledge of the memory objects allocated by each other. We formally define disentanglement by considering an ML-like higher-order language with mutable references and presenting a dynamic semantics for it that enables reasoning about computation graphs of nested parallel programs. Based on this graph semantics, we formalize a classic correctness property---determinacy race freedom---and prove that it implies disentanglement. This establishes that disentanglement applies to a relatively broad class of parallel programs. We then propose memory management techniques for nested-parallel programs that take advantage of disentanglement for improved efficiency and scalability. We show that these techniques are practical by extending the MLton compiler for Standard ML to support this form of nested parallelism. Our empirical evaluation shows that our techniques are efficient and scale well.

Efficient noise injection for exposing hidden data races

The happens-before (HB) relation is widely used to precisely detect data races, which are among the major causes of concurrency bugs. However, the HB relation is sensitive to thread interleaving; thus, hidden races that can be revealed in other thread interleavings may remain despite the repeated running of a program with an HB detector. In this paper, we propose a noise injection-based race detection technique (RaceInducer) that can expose and detect hidden races by using the runtime HB analysis result to determine the noise injection instants. RaceInducer delays a thread execution after the occurrence of the first no-lock non-racing accesses of three types (read-exclusive, write-exclusive, and read-shared) to a shared variable during a precise HB detection (FastTrack). Using this technique, RaceInducer can efficiently disturb the normal locking sequences between two threads that generate HB edges hiding data races. We implemented RaceInducer on the RoadRunner instrumentation framework for Java and evaluated its detection capability and runtime overheads on 12 real-world benchmark programs. The evaluation results showed that RaceInducer generated a lower runtime overhead compared to random noise injection that randomly injected noise for no-lock non-racing accesses and can detect more races with the same noise strength. RaceInducer generates particularly lower runtime overhead than the random noise injection for large programs that generates a high number of shared-memory accesses. In addition, we experimentally demonstrated that RaceInducer incurred lower runtime overheads than a current hidden-race detector (DrFinder) for programs with a relatively large number of locking operations.

Disclosing and Locating Concurrency Bugs of Interrupt-Driven IoT Programs

The Internet of Things (IoT) is envisioned as a distributed network formed by many end devices, e.g., the motes of wireless sensor network (WSN). These important IoT end devices enable ubiquitous sensing of environments and provide reliable services for mission-critical applications. However, programs running on WSN devices are typically interrupt-driven and prone to interrupt-induced concurrency bugs, which are primarily caused by erroneous interleavings among interrupt procedure instances (IPIs) (namely, executions of interrupt processing logic). In this paper, we use a set of dynamic bug patterns to characterize the concurrency bugs due to buggy access-interleavings among IPIs to shared resources, including shared memory locations and shared communication channels. By matching the above bug patterns, a dynamic analysis approach called disclosing and locating concurrency bugs of interrupt-driven IoT programs based on dynamic bug patterns (Daemon) is proposed to automatically detect and locate concurrency bugs in WSN programs. A GUI tool of Daemon is developed. As the empirical studies exhibit, the tool can discover concurrency bugs effectively and locate the buggy source lines visually.

Convoider: A Concurrency Bug Avoider Based on Transparent Software Transactional Memory

Software transactional memory is an effective mechanism to avoid concurrency bugs in multi-threaded programs. However, two problems hinder the adoption of traditional such systems in wild world: high human cost for equipping programs with transaction functionality and low compatibility with I/O calls and conditional variables. This paper presents Convoider to try to solve these problems. By intercepting inter-thread operations and designating code among them as transactions in each thread, Convoider automatically transactionalizes target programs without any source code modification and recompiling. By saving/restoring stack frames and CPU registers on beginning/aborting a transaction, Convoider makes execution flow revocable. By turning threads into processes, leveraging virtual memory protection and customizing memory allocation/deallocation, Convoider makes memory manipulations revocable. By maintaining virtual file systems and redirecting I/O operations onto them, Convoider makes I/O effects revocable. By converting lock/unlock operations to no-ops, customizing signal/wait operations on condition variables and committing memory changes transactionally, Convoider makes deadlocks, data races and atomicity violations impossible. Experimental results show that Convoider succeeds in transparently transactionalizing twelve real-world applications and perfectly avoid 94% of thirty-one concurrency bugs used in our experiments. This study can help efficiently transactionalize legacy multi-threaded applications and effectively improve the runtime reliability of them.

Concurrency Debugging with MaxSMT

Current Maximum Satisfiability (MaxSAT) algorithms based on successive calls to a powerful Satisfiability (SAT) solver are now able to solve real-world instances in many application domains. Moreover, replacing the SAT solver with a Satisfiability Modulo Theories (SMT) solver enables effective MaxSMT algorithms. However, MaxSMT has seldom been used in debugging multi-threaded software.Multi-threaded programs are usually non-deterministic due to the huge number of possible thread operation schedules, which makes them much harder to debug than sequential programs. A recent approach to isolate the root cause of concurrency bugs in multi-threaded software is to produce a report that shows the differences between a failing and a non-failing execution. However, since they rely solely on heuristics, these reports can be unnecessarily large. Hence, reports may contain operations that are not relevant to the bug’s occurrence.This paper proposes the use of MaxSMT for the generation of minimal reports for multi-threaded software with concurrency bugs. The proposed techniques report situations that the existing techniques are not able to identify. Experimental results show that using MaxSMT can significantly improve the accuracy of the generated reports and, consequently, their usefulness in debugging the root cause of concurrency bugs.

A bidirectional trace simplification approach based on a context switch linked list for concurrent programs

SummaryConcurrent programs are notoriously difficult to debug due to shared memory and the non‐determined nature of thread scheduling. Frequent context switches add a huge burden on developers in reasoning about concurrency bugs. To alleviate this problem, we present a bidirectional trace simplification approach based on a context switch linked list. First, we calculate the dependence relations, including local dependences, synchronization dependences, and remote read/write dependences. Second, we construct a context switch linked list according to the original buggy trace. Then, we backward refactor the context switch linked list in sequence to extend thread execution intervals. Finally, we forward check the context switch linked list in sequence to ensure that no nodes can be further merged. We have conducted experiments on eight Java multi‐threaded programs to evaluate our approach. The results show that our approach performs better than or is comparable to the compared static approaches in effectiveness and efficiency.

Predictive analysis for race detection in software-defined networks

Race condition remains one kind of the most common concurrency bugs in software-defined networks (SDNs). The race conditions can be exploited to lead tosecurity and reliability risks. However, the race conditions are notoriously difficult to detect. The existing race detectors for SDNs have limited detection capability. They can only detect the races in the original traces (observed traces) and cause false negatives. In this study, we present a predictive analysis framework called SDN-predict for race detection in SDNs.By encoding the order between the specified network events in SDNs as constraint, we formulate race detection as a constraint solving problem. In addition to detectingthe races in the original trace, our framework can also detect the races in the feasible traces got from reordering the events in the original trace while satisfying the consistency requirements of trace. Moreover, we formally prove that our predictive analysis framework is sound and can achieve the maximal possible detectioncapability for any sound dynamic race detector with respect to the same trace. We evaluate our framework on a set of traces collected from three SDN controllers (POX, Floodlight, ONOS), running 5 representative applicationsincluding reactive and proactive applications in large networks, on three different network topologies. These experiments show that our framework has higherrace detection capability than exisiting SDN race detector-SDNRacer, and detects more 1173 races. These 1173 races were previously undetected and confirmed by checking the race graphs.

Applying Transactional Memory for Concurrency-Bug Failure Recovery in Production Runs

Concurrency bugs widely exist and severely threaten system availability. Techniques that help recover from concurrency-bug failures during production runs are highly desired. This paper proposes BugTM, an approach that applies transactional memory techniques for concurrency-bug recovery in production runs. Requiring no knowledge about where are concurrency bugs, BugTM uses static analysis and code transformation to enable BugTM-transformed software to recover from a concurrency-bug failure by rolling back and re-executing the recent history of a failure thread. BugTM is instantiated as three schemes that have different trade-offs in performance and recovery capability: BugTM$_H$H uses existing hardware transactional memory (HTM) support, BugTM$_S$S leverages software transactional memory techniques, and BugTM$_{\mathrm{HS}}$ HS is a software-hardware hybrid design. BugTM greatly improves the recovery capability of state-of-the-art techniques with low run-time overhead and no changes to OS or hardware, while guarantees not to introduce new bugs.

DFTracker: detecting double-fetch bugs by multi-taint parallel tracking

A race condition is a common trigger for concurrency bugs. As a special case, a race condition can also occur across the kernel and user space causing a double-fetch bug, which is a field that has received little research attention. In our work, we first analyzed real-world double-fetch bug cases and extracted two specific patterns for double-fetch bugs. Based on these patterns, we proposed an approach of multi-taint parallel tracking to detect double-fetch bugs. We also implemented a prototype called DFTracker (double-fetch bug tracker), and we evaluated it with our test suite. Our experiments demonstrated that it could effectively find all the double-fetch bugs in the test suite including eight real-world cases with no false negatives and minor false positives. In addition, we tested it on Linux kernel and found a new double-fetch bug. The execution overhead is approximately 2x for single-file cases and approximately 9x for the whole kernel test, which is acceptable. To the best of the authors’ knowledge, this work is the first to introduce multi-taint parallel tracking to double-fetch bug detection—an innovative method that is specific to double-fetch bug features—and has better path coverage as well as lower runtime overhead than the widely used dynamic approaches.

Detecting semantic violations of lock-free data structures through C++ contracts

The use of synchronization mechanisms in multithreaded applications is essential on shared-memory multi-core architectures. However, debugging parallel applications to avoid potential failures, such as data races or deadlocks, can be challenging. Race detectors are key to spot such concurrency bugs; nevertheless, if lock-free data structures are used, these may emit a significant number of false positives. In this paper, we present a framework for semantic violation detection of lock-free data structures which makes use of contracts, a novel feature of the upcoming C++20, and a customized version of the ThreadSanitizer race detector. We evaluate the detection accuracy of the framework in terms of false positives and false negatives leveraging some synthetic benchmarks which make use of the SPSC and MPMC lock-free queue structures from the Boost C++ library. Thanks to this framework, we are able to check the correct use of lock-free data structures, thus reducing the number of false positives.

Concurrency Bug Avoiding Based on Optimized Software Transactional Memory

Software transactional memory is an effective mechanism to avoid concurrency bugs in multithreaded programs. However, two problems hinder the adoption of such traditional systems in the wild world: high human cost for equipping programs with transaction functionality and low compatibility with I/O calls and conditional variables. This paper presents Convoider to solve these problems. By intercepting interthread operations and designating code among them as transactions in each thread, Convoider automatically transactionalizes target programs without any source code modification and recompiling. By saving/restoring stack frames and CPU registers on beginning/aborting a transaction, Convoider makes execution flow revocable. By turning threads into processes, leveraging virtual memory protection and customizing memory allocation/deallocation, Convoider makes memory manipulations revocable. By maintaining virtual file systems and redirecting I/O operations onto them, Convoider makes I/O effects revocable. By converting lock/unlock operations to no-ops, customizing signal/wait operations on condition variables, and committing memory changes transactionally, Convoider makes deadlocks, data races, and atomicity violations impossible. Experimental results show that Convoider succeeds in transparently transactionalizing twelve real-world applications with averagely incurring only 28% runtime overhead and perfectly avoid 94% of thirty-one concurrency bugs used in our experiments. This study can help efficiently transactionalize legacy multithreaded applications and effectively improve the runtime reliability of them.

Efficient Test Case Generation for Thread-Safe Classes

Generating test cases automatically for thread-safe classes is an effective approach to validating their correctness. However, the existing concurrent test generation techniques usually consume a large amount of time and efforts before finding concurrency bugs. To alleviate this problem, we present an automatic and efficient approach which combines the advantages of both the bug-driven and coverage-guided techniques to generate test cases for thread-safe classes. First, method pairs that cannot be executed concurrently are removed by the static analysis. Then, a strategy of the bug-driven grouping of method pairs is designed to divide the remaining method pairs into two groups. One group contains the method pairs with a high priority, and another group contains the method pairs with a low priority. Finally, iterative generation of concurrent test cases, which consists of the coverage-guided generation of concurrent tests and concurrency bug detection, is conducted to find concurrency bugs. Our evaluation is on 20 thread-safe classes. Compared with four state-of-the-art approaches, the results show that our approach can obtain a significant improvement in efficiency without impairing bug finding capacities.

Effective Detection of Sleep-in-atomic-context Bugs in the Linux Kernel

Atomic context is an execution state of the Linux kernel in which kernel code monopolizes a CPU core. In this state, the Linux kernel may only perform operations that cannot sleep, as otherwise a system hang or crash may occur. We refer to this kind of concurrency bug as a sleep-in-atomic-context (SAC) bug. In practice, SAC bugs are hard to find, as they do not cause problems in all executions. In this article, we propose a practical static approach named DSAC to effectively detect SAC bugs in the Linux kernel. DSAC uses three key techniques: (1) a summary-based analysis to identify the code that may be executed in atomic context, (2) a connection-based alias analysis to identify the set of functions referenced by a function pointer, and (3) a path-check method to filter out repeated reports and false bugs. We evaluate DSAC on Linux 4.17 and find 1,159 SAC bugs. We manually check all the bugs and find that 1,068 bugs are real. We have randomly selected 300 of the real bugs and sent them to kernel developers. 220 of these bugs have been confirmed, and 51 of our patches fixing 115 bugs have been applied.

How to implement any concurrent data structure

We propose a method called Node Replication (NR) to implement any concurrent data structure. The method takes a single-threaded implementation of a data structure and automatically transforms it into a concurrent (thread-safe) implementation. The result is designed to work well with and harness the power of modern servers, which are complex Non-Uniform Memory Access (NUMA) machines with many processor sockets and subtle performance characteristics. Using NR requires no expertise in concurrent data structure design, and the result is free of concurrency bugs. NR represents a paradigm shift of how concurrent algorithms are developed: rather than designing for a data structure, we design for the architecture.

Using Local Clocks to Reproduce Concurrency Bugs

Multi-threaded programs play an increasingly important role in current multi-core environments. Exposing concurrency bugs and debugging such multi-threaded programs are quite challenging due to their inherent non-determinism. In order to mitigate such non-determinism, many approaches such as record-and-replay have been proposed. However, those approaches often suffer significant performance degradation because they require a large amount of recorded information and/or long analysis and replay time. In this paper, we propose an efficient and effective approach, ReCBuLC (reproducing concurrency bugs using local clocks), to take advantage of the hardware clocks available on modern processors. The key idea is to reduce the recording overhead and the time to analyze events’ global order by recording timestamps in each thread. These timestamps are used to determine the global order of shared accesses. To avoid the large overhead in accessing system-wide global clock, we opt to use local per-core clocks that incur much less access overhead. We then propose techniques to resolve skews among local clocks and obtain an accurate global event order. By using per-core clocks, state-of-the-art bug reproducing systems such as PRES and CLAP can reduce their recording overheads by up to 85 percent, and the analysis time up to 84.66% $\sim$ 99.99%, respectively.

A Constraint-Aware Optimization Method for Concurrency Bug Diagnosis Service in a Distributed Cloud Environment

The advent of cloud computation and big data applications has enabled data access concurrency to be prevalent in the distributed cloud environment. In the meantime, security issue becomes a critical problem for researchers to consider. Concurrency bug diagnosis service is to analyze concurrent software and then reason about concurrency bugs in them. However, frequent context switches in concurrent program execution traces will inevitably impact the service performance. To optimize the service performance, this paper presents a static constraint-aware method to simplify concurrent program buggy traces. First, taking the original buggy trace as the operation object, we calculate the maximal sound dependence relations based on the constraint models. Then, we iteratively check the dependent constraints and move forward current event to extend thread execution intervals. Finally, we obtain the simplified trace that is equivalent to the original buggy trace. To evaluate our approach, we conduct a set of experiments on 12 widely used Java projects. Experimental results show that our approach outperforms other state-of-the-art approaches in terms of execution time.

Concurrent Bug Detection Using Invariant Analysis

In concurrent programs, bug detection is a tedious job due to non-determinism and multiple thread control. The bug detection is done by checking the interleaving of threads which is not available in operational phases. So, static analysis is one of the preferred approaches for detection of concurrent bug. Invariant based testing technique is one approach of static analysis used for detecting the concurrent bugs. In this paper, we discuss an invariant based testing approach using three steps: (i) the invariants of a given concurrent program are generated using Daikon tool. (ii) The bad invariants are removed by using the static call graph of the source code, where the static call graph is generated by the javacg tool. (iii) The reduced invariant set is obtained by eliminating the bad and redundant invariants which is used for testcase generation. Using the reduced invariant set, we generate the testcases that are used to find the various concurrent bugs such as Deadlock, Atomicity violation and Bad composition. We conducted an experiment on a well-known concurrent program called the Dining Philosopher Problem. The experimental results show that, the testcases obtained from the reduced invariant set is able to detect more types and no. of concurrent bugs than the existing approach on invariant based testing.

D4: fast concurrency debugging with parallel differential analysis

We present D4, a fast concurrency analysis framework that detects concurrency bugs (e.g., data races and deadlocks) interactively in the programming phase. As developers add, modify, and remove statements, the code changes are sent to D4 to detect concurrency bugs in real time, which in turn provides immediate feedback to the developer of the new bugs. The cornerstone of D4 includes a novel system design and two novel parallel differential algorithms that embrace both change and parallelization for fundamental static analyses of concurrent programs. Both algorithms react to program changes by memoizing the analysis results and only recomputing the impact of a change in parallel. Our evaluation on an extensive collection of large real-world applications shows that D4 efficiently pinpoints concurrency bugs within 100ms on average after a code change, several orders of magnitude faster than both the exhaustive analysis and the state-of-the-art incremental techniques.

CURD: a dynamic CUDA race detector

As GPUs have become an integral part of nearly every pro- cessor, GPU programming has become increasingly popular. GPU programming requires a combination of extreme levels of parallelism and low-level programming, making it easy for concurrency bugs such as data races to arise. These con- currency bugs can be extremely subtle and di cult to debug due to the massive numbers of threads running concurrently on a modern GPU. While some tools exist to detect data races in GPU pro- grams, they are often prohibitively slow or focused only on a small class of data races in shared memory. Compared to prior work, our race detector, CURD, can detect data races precisely on both shared and global memory, selects an appropriate race detection algorithm based on the synchronization used in a program, and utilizes efficient compiler instrumentation to reduce performance overheads. Across 53 benchmarks, we find that using CURD incurs an aver- age slowdown of just 2.88x over native execution. CURD is 2.1x faster than Nvidia’s CUDA-Racecheck race detector, de- spite detecting a much broader class of races. CURD finds 35 races across our benchmarks, including bugs in established benchmark suites and in sample programs from Nvidia.

AdaptiveLock: Efficient Hybrid Data Race Detection Based on Real-World Locking Patterns

Among the various types of concurrency bugs, the data race is one of the primary causes of other concurrency bugs. Thus, it is important to detect as many data races as possible during the development step of multithreaded programs. A hybrid data race detection technique that uses the Lockset algorithm and happens-before relation, can detect actually occurred and hidden data races in one execution trace. However, high runtime slowdown obstructs the frequent use of hybrid detectors. In this paper, we empirically demonstrate that most data race bugs are caused by the absence of a lock, and that multiple locks are rarely involved in a data race bug in the real world. Thus, we propose a fast hybrid detection algorithm that does not introduce additional false positives and false negatives to the current hybrid detectors. The suggested algorithm replaces the lock-set intersection by a simple comparison operation that focuses on exploring data-race-prone locking patterns. The experimental results indicate that the proposed algorithm detects the same data races as Multilock-HB, which is the most accurate hybrid detector, with a 1.18\(\times \) slowdown of FastTrack for eight large-scale benchmark programs.