Replay Debugging Research Articles

Replay debugging

Hardware-assisted Record and Deterministic Replay (RnR) of programs has been proposed as a primitive for debugging hard-to-repeat software bugs. However, simply providing support for repeatedly stumbling on the same bug does not help diagnose it. For bug diagnosis, developers typically want to modify the code, e.g., by creating and operating on new variables, or printing state. Unfortunately, this renders the RnR log inconsistent and makes Replay Debugging (i.e., debugging while using an RnR log for replay) dicey at best This paper presents rdb, the first scheme for replay debugging that guarantees exact replay. rdb relies on two mechanisms. The first one is compiler support to split the instrumented application into two executables: one that is identical to the original program binary, and another that encapsulates all the added debug code. The second mechanism is a runtime infrastructure that replays the application and, without affecting it in any way, invokes the appropriate debug code at the appropriate locations. We describe an implementation of rdb based on LLVM and Pin, and show an example of how rdb's replay debugging helps diagnose a real bug

Multithread Deterministic Replay Debugging: The State of The Art

Multithread Deterministic Replay Debugging: The State of The Art

A Comprehensive Solution for Deterministic Replay Debugging of SoftPLC Applications

Deterministic replay debugging is an approach to finding bugs in deployed software. It records an application run in the field so that it can deterministically be replayed offline in a development system for debugging purposes. To enable deterministic replay debugging, it is necessary to record all external influences and sources of nondeterminism in the original program run. From that trace log and from a known initial state, the program can be replayed deterministically without requiring any connection to the original environment. In this paper, we present a solution for deterministic replay debugging of hard real-time multitasking SoftPLC applications written in the IEC 61131-3 languages. By taking advantage of the special properties of these programs and by careful engineering, our technique allows recording a SoftPLC application run in the field with minimal overhead and obeying real-time constraints. In later phases, which are offline, the original program run is reconstructed from the minimal information recorded so it can be replayed for debugging. In comparison to previous work, our solution has several advantages: Instead of recording task scheduling information, it reconstructs the task interleaving based on data dependencies, thereby significantly simplifying the recording phase. Additionally, it incorporates a technique for periodically capturing the complete internal state of the system, which can later be used as a starting point for replay. We present the conceptual basis of our approach, a tool chain which provides deterministic replay debugging to the user as a set of fully automated tools, and an evaluation as well as an industrial case study for validating the approach.

Replay debugging of non‐deterministic executions in the Kernel‐based Virtual Machine

SUMMARYLogging and replay provide numerous benefits for debugging today's increasingly distributed and parallel applications, as well as for debugging within cloud computing environments. Tracking down faults, particularly in applications that involve non‐deterministic executions, becomes much easier if such a technique is used. This article presents a system that leverages the functionality of modern x86 hardware extensions present in Intel and AMD processors to implement logging and replay of the Kernel‐based Virtual Machine (KVM). With our KVM‐based logging and replay system, virtual machines can be logged at will. The system can record the behavior of the operating system, the applications, the network traffic, and the user input. At a later time, and even on another machine, the problematic execution can be replayed and analyzed. The implementation incurs a minimal amount of processing and memory overhead, and thus, its effects are largely imperceptible to users of the system. Copyright © 2011 John Wiley & Sons, Ltd.

Multi-stage replay with crosscut

Deterministic record-replay has many useful applications, ranging from fault tolerance and forensics to reproducing and diagnosing bugs. When choosing a record-replay solution, the system administrator must choose a priori how comprehensively to record the execution and at what abstraction level to record it. Unfortunately, these choices may not match well with how the recording is eventually used. A recording may contain too little information to support the end use of replay, or it may contain more sensitive information than is allowed to be shown to the end user of replay. Similarly, fixing the abstraction level at the time of recording often leads to a semantic mismatch with the end use of replay. This paper describes how to remedy these problems by adding customizable replay stages to create special-purpose logs for the end users of replay. Our system, called Crosscut, allows replay logs to be "sliced" along time and abstraction boundaries. Using this approach, users can create slices that include only the processes, applications, or components of interest, excluding parts that handle sensitive data. Users can also retarget the abstraction level of the replay log to higher-level platforms, such as Perl or Valgrind. Execution can then be augmented with additional analysis code at replay time, without disturbing the replayed components in the slice. Crosscut thus uses replay itself to transform logs into a more efficient, secure, and usable form for replay-based applications. Our current Crosscut prototype builds on VMware Workstation's record-replay capabilities, and supports a variety of different replay environments. We show how Crosscut can create slices of only the parts of the computation of interest and thereby avoid leaking sensitive information, and we show how to retarget the abstraction level of the log to enable more convenient use during replay debugging.

Recording shared memory dependencies using strata

Significant time is spent by companies trying to reproduce and fix bugs. BugNet and FDR are recent architecture proposals that provide architecture support for deterministic replay debugging. They focus on continuously recording information about the program's execution, which can be communicated back to the developer. Using that information, the developer can deterministically replay the program's execution to reproduce and fix the bugs.In this paper, we propose using Strata to efficiently capture the shared memory dependencies. A stratum creates a time layer across all the logs for the running threads, which separates all the memory operations executed before and after the stratum. A strata log allows us to determine all the shared memory dependencies during replay and thereby supports deterministic replay debugging for multi-threaded programs.

Recording shared memory dependencies using strata

Significant time is spent by companies trying to reproduce and fix bugs. BugNet and FDR are recent architecture proposals that provide architecture support for deterministic replay debugging. They focus on continuously recording information about the program's execution, which can be communicated back to the developer. Using that information, the developer can deterministically replay the program's execution to reproduce and fix the bugs.In this paper, we propose using Strata to efficiently capture the shared memory dependencies. A stratum creates a time layer across all the logs for the running threads, which separates all the memory operations executed before and after the stratum. A strata log allows us to determine all the shared memory dependencies during replay and thereby supports deterministic replay debugging for multi-threaded programs.

Recording shared memory dependencies using strata

Significant time is spent by companies trying to reproduce and fix bugs. BugNet and FDR are recent architecture proposals that provide architecture support for deterministic replay debugging. They focus on continuously recording information about the program's execution, which can be communicated back to the developer. Using that information, the developer can deterministically replay the program's execution to reproduce and fix the bugs.In this paper, we propose using Strata to efficiently capture the shared memory dependencies. A stratum creates a time layer across all the logs for the running threads, which separates all the memory operations executed before and after the stratum. A strata log allows us to determine all the shared memory dependencies during replay and thereby supports deterministic replay debugging for multi-threaded programs.

Event-based debugging system for a parallel object-oriented language A-NETL

Debugging of parallel programs is generally very difficult to undertake since the execution of programs is nondeterministic and nonrepeatable. Considering a parallel object-oriented language, this paper proposes a method of preserving the repeatability of the order of execution in a replay by defining a logic time in terms of events in the message communication system and state transitions, and by representing the execution history of events. A debugger that mainly uses a two-dimensional (2-D) representation and replay debugging has been designed based on the proposed method. A high-performance system has been constructed based on the design. The debugger functions are distributed between a host workstation and multicomputer nodes, and between local OS on nodes and in firmware so that a high-performance system has been realized. The experiments show that the method is successful for practical applications; the time and space overhead of event recording is 0.04 ms (mean value) and 35 B per event, respectively; the display of 16 objects for one logical time is 0.94 s; and the overhead for a breakpoint is 3.1 ms per point. © 1998 Scripta Technica, Syst Comp Jpn, 28(11): 53–63, 1997