Demonic Testing of Concurrent Programs

We present a lightweight library for testing concurrent Scala programs by systematically exploring multiple interleavings between user-specified operations on shared objects. Our library is targeted at beginners of concurrent programming in Scala, runs on a standard JVM, and supports conventional synchronization primitives such as wait, notify, and synchronized. The key component of the library is the trait SchedulableMonitor that accepts a thread schedule, and interleaves as per the schedule all user-specified operations invoked through multiple threads on objects implementing the trait. Using our library, we developed a unit test engine that tests concurrent operations on shared objects on thousands of schedules obtained by bounding the number of context-switches. If a unit test fails on a schedule, the test engine offers as feedback the interleaved traces of execution that resulted in the failure. We used our test engine to automatically test and evaluate two assignments: (a) lock-based producer/consumer problem, and (b) lock-free sorted list implementation, offered to a class of 150 under-graduate students of EPFL. Our evaluations show that the system is effective in detecting bugs in students' solutions.

Unit testing is a methodology for testing small parts of an application independently of whatever application uses them. It is time consuming and tedious to write unit tests, and it is especially difficult to write unit tests that model the pattern of usage of the application. Aspect-oriented programming (AOP) addresses the problem of separation of concerns in programs which is well suited to unit test problems. What's more, unit tests should be made from different concerns in the application instead of just from functional assertions of correctness or error. In this paper, we firstly present a new concept, application-specific Aspects, which mean top-level aspects picked up from generic low-level aspects in AOP for specific use. It can be viewed as the separation of concerns on applications of generic low-level aspects. Second, this paper describes an aspect-oriented test description language (AOTDL) and techniques to build top-level aspects for testing on generic aspects. Third, we generate JUnit unit testing framework and test oracles from AspectJ programs by integrating our tool with AspectJ and JUnit. We use runtime exceptions thrown by testing aspects to decide whether methods work well. Finally, we present a double-phase testing way to filter out meaningless test cases in our framework.

Context-bounded analysis is an attractive approach to verification of concurrent programs. Bounding the number of contexts executed per thread not only reduces the asymptotic complexity, but also the complexity increases gradually from checking a purely sequential program. Lal and Reps [14] provided a method for reducing the context-bounded verification of a concurrent boolean program to the verification of a sequential boolean program, thereby allowing sequential reasoning to be employed for verifying concurrent programs. In this work, we adapt the encoding to work for systems programs written in C with the heap and accompanying low-level operations such as pointer arithmetic and casts. Our approach is completely automatic: we use a verification condition generator and SMT solvers, instead of a boolean model checker, in order to avoid manual extraction of boolean programs and false alarms introduced by the abstraction. We demonstrate the use of field slicing for improving the scalability and (in some cases) coverage of our checking. We evaluate our tool Storm on a set of real-world Windows device drivers, and has discovered a bug that could not be detected by extensive application of previous tools.

The OpenWorm Project is an international open-source collaboration to create a multi-scale model of the organism Caenorhabditis elegans At each scale, including subcellular, cellular, network and behaviour, this project employs one or more computational models that aim to recapitulate the corresponding biological system at that scale. This requires that the simulated behaviour of each model be compared with experimental data both as the model is continuously refined and as new experimental data become available. Here we report the use of SciUnit, a software framework for model validation, to attempt to achieve these goals. During project development, each model is continuously subjected to data-driven 'unit tests' that quantitatively summarize model-data agreement, identifying modelling progress and highlighting particular aspects of each model that fail to adequately reproduce known features of the biological organism and its components. This workflow is publicly visible via both GitHub and a web application and accepts community contributions to ensure that modelling goals are transparent and well-informed.This article is part of a discussion meeting issue 'Connectome to behaviour: modelling C. elegans at cellular resolution'.

Performing unit testing on concurrent programs is a known challenge. Several solutions have been proposed for the challenges, that provide only a partial answer. We argue that there are two kinds of concurrent code writing, namely concurrency protocols and concurrent and that they are different not only in the coding itself, but also in the way unit testing should be done for them. We show that the current solutions to concurrent unit testing only give a satisfactory answer to concurrency protocols, but not to concurrent business code. We have designed and implemented a unit testing framework suitable for concurrent business code, as part of the IBM Multicore Software Development Kit. This paper presents the distinction between the types of concurrent programming, and describes the new framework.

The Java language supports the use of monitors, sockets, and remote method invocation for concurrent programming. Also, Java classes can be defined to simulate other types of concurrent constructs. However, concurrent Java programs, like other concurrent programs, are difficult to specify, design, code, test and debug. In this paper, we describe the design of a toolset, called JaDA (Java Dynamic Analyzer), that provides testing and debugging tools for concurrent Java programs. To collect run-time information or control program execution, JaDA requires transformation of a concurrent Java program into a slightly different Java program. We show that by modifying Java classes that support concurrent programming, Java application programs only need minor modifications. We also present a novel approach to managing threads that are needed for testing and debugging of concurrent Java programs.

An execution of a concurrent program P exercises a sequence of synchronization events, called a synchronization sequence or a SYN-sequence. This non-deterministic execution behavior creates a problem during the testing phase of P: when testing P with input X, a single execution is insufficient to determine the correctness of P with input X.In this paper, we show how to accomplish deterministic execution testing of a concurrent Ada program according to a given SYN-sequence. We first define the format of a SYN-sequence which provides sufficient information for deterministic execution. Then we show how to transform a concurrent Ada program P into a slightly different program P* (also written in Ada) so that an execution of P* with (X,S) as input, where X and S are an input and SYN-sequence of P respectively, determines whether or not S is feasible for P with input X and produces the same result as P with input X and SYN-sequence S would, provided that S is feasible. Tools for transforming concurrent Ada programs for deterministic execution testing are described.

A language-based approach to deterministic execution debugging of concurrent Ada programs is presented. The approach is to define synchronization (SYN)-sequences of a concurrent Ada program in terms of Ada language constructs and to replay such SYN-sequences without the need for system-dependent debugging tools. It is shown how to define a SYN-sequence of a concurrent Ada program in order to provide sufficient information for deterministic execution. It is also shown how to transform a concurrent Ada program P so that the SYN-sequences of previous executions of P can be replayed. This transformation adds an Ada task to P that controls program execution by synchronizing with the original tasks in P. A brief description is given of the implementation of tools supporting deterministic execution debugging of concurrent Ada programs. >

This lecture captures industrial and academic experience in teaching concurrent programming for several years. While the statement of a concurrent protocol is typically small taking only a few pages to complete, the statement implicitly introduces an exponential or possibly infinite space of possible interleavings. The novice is not aware of this, and as a result sometimes does not understand the lecture. To overcome this pitfall, exploration of the protocol interleaving is practiced from day one. A classical example being how many results does 100 threads executing i++ have?Separation of concerns is used to minimize the interleaving space at design time. Specifically, abstract primitives, such as atomic block and a conditional atomic block are introduced as a design tool. They are typically not to be found in practical programming languages. This reduces the space of interleavings to be considered at design time and facilitate verification and review of the protocol. On the other hand, the approach introduces pitfalls encountered in the translation/implementation stage.Once the abstract design is verified and/or tested, it needs to be translated to a commercial language where the synchronization primitives, for performance reasons, are weaker. For example, an atomic block is implemented using a lock unlock pair. But now atomicity needs to be ensured by the protocol implementer---a typical pitfall being the confusion of a lock/unlock code segment with an atomic block. Locations in the code that access the protected shared resources should be identified and access needs to follow the lock/unlock protocol. Exceptions and signals needs to be handled appropriately and adhere to the lock/unlock protocol. To avoid translation pitfalls, bug patterns are taught and precise understanding of the synchronization primitives is emphasized.Testing of the implemented protocol is taught. Testing should be aggressive, at unit test, and attempt to cover the interleaving space instead of common practice of testing concurrency at system test in a round about way through stress.Tools like ConTest or CHESS can be applied to this end.

Mutation-based software testing is a powerful technique for testing software systems. It requires executing many slightly different versions of the same program to evaluate the quality of the test cases used to test the program. Mutation-based testing has been applied to sequential software; however, problems are encountered when it is applied to concurrent programs. These problems are a product of the nondeterminism inherent in the executions of concurrent programs. In this paper, we describe a general approach to testing and debugging concurrent programs, called deterministic execution testing and debugging. We then describe a combination of deterministic execution testing and mutation-based testing, called deterministic execution mutation testing (DEMT), and illustrate the DEMT approach with an example. >

Testing is a validation process that determines the conformance of the software’s implementation to its specification. It is an important phase in unit test case design and is even more important in object-oriented systems. We want to develop test case designing criteria that give confidence in unit testing of object-oriented system. The main aim testing can be viewed as a means of assessing the existing quality of the software to probe the software for defect and fix them. We also want to develop and executing our test case automatically since this decreases the effort (and cost) of software development cycle (maintenance), and provide re-usability in Test Driven Development framework. We believe such approach is necessary for reaching the levels of confidence required in unit testing. The main goal of this paper is to assist the developers/testers to improve the quality of the ATCUT by accurately design the test case for unit testing of object-oriented software based on the test results. A blend of unit testing assisted by the domain knowledge of the test case designer is used in this paper to improve the design of test case. This paper outlines a solution strategy for deriving Automated Test Case for Unit Testing (ATCUT) metrics from object-oriented metrics via TDD concept.

A policy of rigorous unit testing as a foundation for the integration of software systems has been found to be effective, resulting in rapid integration and good overall quality of the final software and systems. However, there is little detailed information on how effective individual components of the code and unit test phase of the software lifecycle are. IPL have conducted an exercise to gather and analyse detailed metrics for the module design and unit code and test phases. This will assist the improvement of estimation and planning, identify areas where quality can be improved and costs reduced, enhance project management with improvements in control over the development, and develop indicators that can be used to anticipate problem areas and facilitate risk management.

<p>This thesis explores two kinds of program logics that have become important for modern program verification - separation logic, for reasoning about programs that use pointers to build mutable data structures, and rely guarantee reasoning, for reasoning about shared variable concurrent programs. We look more closely into the motivations for merging these two kinds of logics into a single formalism that exploits the benefits of both approaches - local, modular, and explicit reasoning about interference between threads in a shared memory concurrent program. We discuss in detail two such formalisms - RGSep and Local Rely Guarantee (LRG), in particular we analyse how each formalism models program state and treats the distinction between global state (shared by all threads) and local state (private to a given thread) and how each logic models actions performed by threads on shared state, and look into the proof rules specifically for reasoning about atomic blocks of code. We present full examples of proofs in each logic and discuss their differences. This thesis also illustrates how a weakest precondition semantics for separation logic can be used to carry out calculational proofs. We also note how in essence these proofs are data abstraction proofs showing that a data structure implements some abstract data type, and relate this idea to a classic data abstraction technique by Hoare. Finally, as part of the thesis we also present a survey of tools that are currently available for doing manual or semi-automated proofs as well as program analyses with separation logic and rely guarantee.</p>

<p>This thesis explores two kinds of program logics that have become important for modern program verification - separation logic, for reasoning about programs that use pointers to build mutable data structures, and rely guarantee reasoning, for reasoning about shared variable concurrent programs. We look more closely into the motivations for merging these two kinds of logics into a single formalism that exploits the benefits of both approaches - local, modular, and explicit reasoning about interference between threads in a shared memory concurrent program. We discuss in detail two such formalisms - RGSep and Local Rely Guarantee (LRG), in particular we analyse how each formalism models program state and treats the distinction between global state (shared by all threads) and local state (private to a given thread) and how each logic models actions performed by threads on shared state, and look into the proof rules specifically for reasoning about atomic blocks of code. We present full examples of proofs in each logic and discuss their differences. This thesis also illustrates how a weakest precondition semantics for separation logic can be used to carry out calculational proofs. We also note how in essence these proofs are data abstraction proofs showing that a data structure implements some abstract data type, and relate this idea to a classic data abstraction technique by Hoare. Finally, as part of the thesis we also present a survey of tools that are currently available for doing manual or semi-automated proofs as well as program analyses with separation logic and rely guarantee.</p>

SMART NoCs achieve ultra-low latency by enabling single-cycle multiple-hop transmission via bypass channels. However, contention along bypass channels can seriously degrade the performance of SMART NoCs by breaking the bypass paths. Therefore, contention-free task mapping and scheduling are essential for optimal system performance. In this article, we propose an SMT (Satisfiability Modulo Theories)-based framework to find optimal contention-free task mappings with minimum application schedule lengths on 2D/3D SMART NoCs with mixed dimension-order routing. On top of SMT’s fast reasoning capability for conditional constraints, we develop efficient search-space reduction techniques to achieve practical scalability. Experiments demonstrate that our SMT framework achieves 10× higher scalability than ILP (Integer Linear Programming) with 931.1× (ranges from 2.2× to 1532.1×) and 1237.1× (ranges from 4× to 4373.8×) faster average runtimes for finding optimum solutions on 2D and 3D SMART NoCs and our 2D and 3D extensions of the SMT framework with mixed dimension-order routing also maintain the improved scalability with the extended and diversified routing paths, resulting in reduced application schedule lengths throughout various application benchmarks.