Debugging Techniques Research Articles

Toward More Efficient Statistical Debugging with Abstraction Refinement

Debugging is known to be a notoriously painstaking and time-consuming task. As one major family of automated debugging, statistical debugging approaches have been well investigated over the past decade, which collect failing and passing executions and apply statistical techniques to identify discriminative elements as potential bug causes. Most of the existing approaches instrument the entire program to produce execution profiles for debugging, thus incurring hefty instrumentation and analysis cost. However, as in fact a major part of the program code is error-free, full-scale program instrumentation is wasteful and unnecessary. This article presents a systematic abstraction refinement-based pruning technique for statistical debugging. Our technique only needs to instrument and analyze the code partially. While guided by a mathematically rigorous analysis, our technique is guaranteed to produce the same debugging results as an exhaustive analysis in deterministic settings. With the help of the effective and safe pruning, our technique greatly saves the cost of failure diagnosis without sacrificing any debugging capability. We apply this technique to two different statistical debugging scenarios: in-house and production-run statistical debugging. The comprehensive evaluations validate that our technique can significantly improve the efficiency of statistical debugging in both scenarios, while without jeopardizing the debugging capability.

A Debugging Game for Probabilistic Models

One of the major advantages of model checking over other formal methods is its ability to generate a counterexample when a model does not satisfy is its specification. A counterexample is an error trace that helps to locate the source of the error. Therefore, the counterexample represents a valuable tool for debugging. In Probabilistic Model Checking (PMC), the task of counterexample generation has a quantitative aspect. Unlike the previous methods proposed for conventional model checking that generate the counterexample as a single path ending with a bad state representing the failure, the task in PMC is completely different. A counterexample in PMC is a set of evidences or diagnostic paths that satisfy a path formula, whose probability mass violates the probability threshold. Counterexample generation is not sufficient for finding the exact source of the error. Therefore, in conventional model checking, many debugging techniques have been proposed to act on the counterexamples generated to locate the source of the error. In PMC, debugging counterexamples is more challenging, since the probabilistic counterexample consists of multiple paths and it is probabilistic. In this article, we propose a debugging technique based on stochastic games to analyze probabilistic counterexamples generated for probabilistic models described as Markov chains in PRISM language. The technique is based mainly on the idea of considering the modules composing the system as players of a reachability game, whose actions contribute to the evolution of the game. Through many case studies, we will show that our technique is very effective for systems employing multiple components. The results are also validated by introducing a debugging tool called GEPCX (Game Explainer of Probabilistic Counterexamples).

Fundamentals of a Novel Debugging Mechanism for Orchestrated Cloud Infrastructures with Macrosteps and Active Control

In Infrastructure-as-a-Service (IaaS) clouds, the development process of a ready-to-use and reliable infrastructure might be a complex task due to the interconnected and dependent services that are deployed (and operated later on) in a concurrent way on virtual machines. Different timing conditions may change the overall initialisation method, which can lead to abnormal behaviour or failure in the non-deterministic environment. The overall motivation of our research is to improve the reliability of cloud-based infrastructures with minimal user interactions and significantly accelerate the time-consuming debugging process. This paper focuses on the behaviour of cloud-based infrastructures during their deployment phase and introduces the adaption of a replay, and active control enriched debugging technique, called macrostep, in the field of cloud orchestration in order to provide support for developers troubleshooting deployment-related errors. The fundamental macrostep mechanisms, including the generation of collective breakpoint sets as well as the traversal method for such consistent global states, have been combined with the Occopus cloud orchestrator and the Neo4J graph database. The paper describes the novel approach, the design choices as well as the implementation of the experimental debugger tool with a use case for validation purposes by providing some preliminary numerical results.

Spreadsheet debugging: The perils of tool over-reliance

Spreadsheets are widely used in organizations for various purposes such as data aggregation, reporting and decision-making. Since spreadsheets, like other types of software, can contain faulty formulas, it is important to provide developers with appropriate methods to find and fix such faults. Recently, various heuristic and statistics-based fault identification methods were proposed, which point developers to potentially faulty parts of the spreadsheets. Due to their heuristic nature, these methods might, however, miss some faults. As a result, if spreadsheet developers rely too strongly on these methods, they might not pay sufficient attention to problems that are not pinpointed by the methods. In this research, we are the first to study this potential problem of over-reliance in spreadsheet debugging, which may lead to limited debugging effectiveness. We report the outcome of a controlled experiment where 59 participants were tasked to find faulty formulas in a given spreadsheet with and without support of a novel spreadsheet debugging tool. Our results indicate that tool over-reliance can indeed result as a phenomenon of using heuristic debugging techniques. However, the study also provides evidence that making users aware of potential tool limitations within the debugging environment may help to address this problem.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

SAT-Based Integrated Hardware Trojan Detection and Localization Approach Through Path-Delay Analysis

With the rapid growth in IC outsourcing in the semiconductors industry, concerns have increased about the weakening ICs security against hardware Trojan attacks. In this brief, an integrated hardware Trojan detection and localization methodology is presented by employing the proposed SAT-based test pattern generation scheme and the MUX-based debugging technique. The experimental results show that our methodology can effectively detect timing anomalies in the path-delays caused by hardware Trojans with node coverage around 97% as well as localizing all Trojan's gates with a localization resolution around 99.6%. Moreover, all timing error sites are successfully identified with zero False Negative and 0.56% False Positive rates.

An empirical study of optimization bugs in GCC and LLVM

Optimizations are the fundamental component of compilers. Bugs in optimizations have significant impacts, and can cause unintended application behavior and disasters, especially for safety-critical domains. Thus, an in-depth analysis of optimization bugs should be conducted to help developers understand and test the optimizations in compilers. To this end, we conduct an empirical study to investigate the characteristics of optimization bugs in two mainstream compilers, GCC and LLVM. We collect about 57K and 22K bugs of GCC and LLVM, and then exhaustively examine 8,771 and 1,564 optimization bugs of the two compilers, respectively. The results reveal the following five characteristics of optimization bugs: (1) Optimizations are the buggiest component in both compilers except for the C++ component; (2) the value range propagation optimization and the instruction combine optimization are the buggiest optimizations in GCC and LLVM, respectively; the loop optimizations in both GCC and LLVM are more bug-prone than other optimizations; (3) most of the optimization bugs in both GCC and LLVM are misoptimization bugs, accounting for 57.21% and 61.38% respectively; (4) on average, the optimization bugs live over five months, and developers take 11.16 months for GCC and 13.55 months for LLVM to fix an optimization bug; in both GCC and LLVM, many confirmed optimization bugs have lived for a long time; (5) the bug fixes of optimization bugs involve no more than two files and three functions on average in both compilers, and around 99% of them modify no more than 100 lines of code, while 90% less than 50 lines of code.Our study provides a deep understanding of optimization bugs for developers and researchers. This could provide useful guidance for the developers and researchers to better design the optimizations in compilers. In addition, the analysis results suggest that we need more effective techniques and tools to test compiler optimizations. Moreover, our findings are also useful to the research of automatic debugging techniques for compilers, such as automatic compiler bug isolation techniques.

Vibroacoustic debugging technique for vehicle exhaust systems

Influence of the exhaust system on the vibroacoustic comfort of a passenger car with a transverse power unit is considered. The methodology is presented which, when using road tests and calculating the exhaust system on the finite element model, allows to reduce the levels of vibration and noise from the exhaust system when changing to more powerful engines in serial passenger cars.

IDD – A Platform Enabling Differential Debugging

Abstract Debugging is a very time consuming task which is not well supported by existing tools. The existing methods do not provide tools enabling optimal developers’ productivity when debugging regressions in complex systems. In this paper we describe a possible solution aiding differential debugging. The differential debugging technique performs analysis of the regressed system and identifying the cause of the unexpected behavior by comparing to a previous version of the same system. The prototype, idd, inspects two versions of the executable – a baseline and a regressed version. The interactive debugging session runs side by side both executables and allows to examine and to compare various internal states. The architecture can work with multiple information sources comparing data from different tools. We also show how idd can detect performance regressions using information from third-party performance facilities. We illustrate how in practice we can quickly discover regressions in large systems such as the clang compiler.

Dynamic Slicing for Concurrent Constraint Languages

Concurrent Constraint Programming (CCP) is a declarative model for concurrency where agents interact by telling and asking constraints (pieces of information) in a shared store. Some previous works have developed (approximated) declarative debuggers for CCP languages. However, the task of debugging concurrent programs remains difficult. In this paper we define a dynamic slicer for CCP (and other language variants) and we show it to be a useful companion tool for the existing debugging techniques. We start with a partial computation (a trace) that shows the presence of bugs. Often, the quantity of information in such a trace is overwhelming, and the user gets easily lost, since she cannot focus on the sources of the bugs. Our slicer allows for marking part of the state of the computation and assists the user to eliminate most of the redundant information in order to highlight the errors. We show that this technique can be tailored to several variants of CCP, such as the timed language ntcc, linear CCP (an extension of CCPbased on linear logic where constraints can be consumed) and some extensions of CCP dealing with epistemic and spatial information. We also develop a prototypical implementation freely available for making experiments.

Effective spectrum-based technique for software fault finding

The process of software testing can be divided into two phases, first is the testing of software where we search for bugs in the software and second is the software debugging which helps to find the root cause of the bug. Finding and correcting the bug takes much more time and effort than coding. In this paper, we present a study of different existing debugging techniques and their classifications. We found that most of the debugging techniques are manual, but the spectrum-based debugging technique is automatic. The spectrum-based software debugging is based on statistical formulas. Hence, we have presented the implementation and comparison of the six widely used statistical formulas and developed a debugging tool for JAVA programs.

An evaluation of pure spectrum‐based fault localization techniques for large‐scale software systems

SummaryPure spectrum‐based fault localization (SBFL) is a well‐studied statistical debugging technique that only takes a set of test cases (some failing and some passing) and their code coverage as input and produces a ranked list of suspicious program elements to help the developer identify the location of a bug that causes a failed test case. Studies show that pure SBFL techniques produce good ranked lists for small programs. However, our previous study based on the iBugs benchmark that uses the AspectJ repository shows that, for realistic programs, the accuracy of the ranked list is not suitable for human developers. In this paper, we confirm this based on a combined empirical evaluation with the iBugs and the Defects4J benchmark. Our experiments show that, on average, at most ∼40%, ∼80%, and ∼90% of the bugs can be localized reliably within the first 10, 100, and 1000 ranked lines, respectively, in the Defects4J benchmark. To reliably localize 90% of the bugs with the best performing SBFL metric D∗, ∼450 lines have to be inspected by the developer. For human developers, this remains unsuitable, although the results improve compared with the results for the AspectJ benchmark. Based on this study, we can clearly see the need to go beyond pure SBFL and take other information, such as information from the bug report or from version history of the code lines, into consideration.

An Analysis on the Negative Effect of Multiple-Faults for Spectrum-Based Fault Localization

This paper aims to investigate the negative effects of multiple-faults on spectrum-based fault localization (SBFL). Previously, researchers validated the fact that the occurrence of multiple-faults could have a significant negative impact on fault localization. However, a very little current research addresses the degree of these impacts through a systemic analysis. Furthermore, the fundamental causes underlying that negative impact have not been investigated and are not fully understood. We conducted experiments on fourteen real-life open source programs to explore and possibly solve these problems. Our results indicate that: 1) although multiple-faults generally do have a negative impact on fault localization, different fault localizations displayed various levels of robustness against that negative impact; 2) restoring pass/fail fault interactions only has a modest effect on that negative impact; 3) our investigation of twelve Fault Localization Interactions (FLI) shows that there is a dominant FLI-1 interaction in multiple-fault programs which should be responsible for that negative impact; 4) restoring FLI-1 can significantly improve the performance of both SBFL and parallel debugging techniques; and 5) furthermore, this paper practically validated the revised Kendall Tau distance as an efficient measure to help locate test cases, which have triggered FLI-1. Based on the revised Kendall Tau distance, a fast search algorithm has been suggested to locate FLI-1 test cases. It is expected that this paper can provide some insight into the fundamental causes of multiple-faults’ negative impact on fault localization and drive the development of more efficient fault localization techniques to improve the identification and handling of multiple-faults.

Fragment-based spreadsheet debugging

Faults in spreadsheets can represent a major risk for businesses. To minimize such risks, various automated testing and debugging approaches for spreadsheets were proposed. In such approaches, often one main assumption is that the spreadsheet developer is able to indicate if the outcomes of certain calculations correspond to the intended values. This, however, might require that the user performs calculations manually, a process which can easily become tedious and error-prone for more complex spreadsheets. In this work, we propose an interactive spreadsheet algorithmic debugging method, which is based on partitioning the spreadsheet into fragments. Test cases can then be automatically or manually created for each of these smaller fragments, whose correctness or faultiness can be easier assessed by users than test cases that cover the entire spreadsheet. The annotated test cases are then fed into an algorithmic debugging technique, which returns a set of formulas that could have caused any observed failures, i.e., discrepancies between the expected and computed calculation outcomes. Simulation experiments demonstrate that the suggested decomposition approach can speed up the algorithmic debugging process and significantly reduce the number of fault candidates returned by the algorithm. An additional laboratory study shows that fragmenting a spreadsheet with our method furthermore reduces the time needed by users for creating test cases for a spreadsheet.

VFL: Variable-based fault localization

ABSTRACT Context Fault localization is one of the most important debugging tasks. Hence, many automatic fault localization techniques have been proposed to reduce the burden on developers for such tasks. Among them, Spectrum-based Fault Localization (SFL) techniques leverage coverage information and localize faults based on the coverage difference between the failed and passed test cases. Objective However, such SFL techniques cannot localize faults effectively when coverage differences are not clear. To address this issue and improve the fault localization performance of the SFL techniques, we propose a Variable-based Fault Localization (VFL) technique. Method The VFL technique identifies suspicious variables and uses them to generate a ranked list of suspicious source code lines. Since it only requires additional information about variables that are also available in the SFL techniques, the proposed technique is lightweight and can be used to improve the performance of existing the SFL techniques. Results In an evaluation with 224 real Java faults and 120 C faults, the VFL technique outperforms the SFL techniques using the same similarity coefficient. The average Exam scores of the VFL techniques are reduced by more than 55% compared to the SFL techniques, and the VFL techniques localize faults at a lower rank than the SFL techniques for about 73% of the 344 faults. Conclusion We proposed a novel variable-based fault localization technique for more effective debugging. The VFL technique has better performance than the existing techniques and the results were more useful for actual fault localization tasks. In addition, this technique is very lightweight and scalable, so it is very easy to collaborate with other fault localization techniques.

Out-Of-Place debugging: a debugging architecture to reduce debugging interference

Context. Recent studies show that developers spend most of their programming time testing, verifying and debugging software. As applications become more and more complex, developers demand more advanced debugging support to ease the software development process. Inquiry. Since the 70's many debugging solutions were introduced. Amongst them, online debuggers provide a good insight on the conditions that led to a bug, allowing inspection and interaction with the variables of the program. However, most of the online debugging solutions introduce \textit{debugging interference} to the execution of the program, i.e. pauses, latency, and evaluation of code containing side-effects. Approach. This paper investigates a novel debugging technique called \outofplace debugging. The goal is to minimize the debugging interference characteristic of online debugging while allowing online remote capabilities. An \outofplace debugger transfers the program execution and application state from the debugged application to the debugger application, both running in different processes. Knowledge. On the one hand, \outofplace debugging allows developers to debug applications remotely, overcoming the need of physical access to the machine where the debugged application is running. On the other hand, debugging happens locally on the remote machine avoiding latency. That makes it suitable to be deployed on a distributed system and handle the debugging of several processes running in parallel. Grounding. We implemented a concrete out-of-place debugger for the Pharo Smalltalk programming language. We show that our approach is practical by performing several benchmarks, comparing our approach with a classic remote online debugger. We show that our prototype debugger outperforms by a 1000 times a traditional remote debugger in several scenarios. Moreover, we show that the presence of our debugger does not impact the overall performance of an application. Importance. This work combines remote debugging with the debugging experience of a local online debugger. Out-of-place debugging is the first online debugging technique that can minimize debugging interference while debugging a remote application. Yet, it still keeps the benefits of online debugging ( e.g. step-by-step execution). This makes the technique suitable for modern applications which are increasingly parallel, distributed and reactive to streams of data from various sources like sensors, UI, network, etc.

Exoneration-based fault localization for SQL predicates

Spectrum-based fault localization (SFL) techniques automatically localize faults in program entities (statements, predicates, SQL clauses, etc.) by analyzing information collected from test executions. One application of SFL techniques is to find faulty SQL statements in database applications. However, prior techniques treated each SQL statement as one program entity, thus they could not find faulty elements inside SQL statements. Since SQL statements can be complex, identifying the faulty element within a faulty SQL statement is still time-consuming.In our previous paper, we developed a novel fault localization method based on row-based dynamic slicing and delta debugging techniques that can localize faults in individual clauses within SQL predicates. We call this technique exoneration-based fault localization because it can exonerate “innocent” elements and precisely identify the faulty element, whereas previous SFL techniques simply ranked all the elements in an SQL statement based on suspiciousness.This paper improves the exoneration-based fault localization technique with a new algorithm that considerably reduces the execution time. We also conducted an empirical study that compared nine existing SFL techniques with the exoneration-based technique in localizing faulty clauses in SQL predicates. Results indicate that the new exoneration-based technique surpasses the other techniques both in terms of effectiveness and efficiency.

Algorithmic debugging generalized

Algorithmic debugging is a semi-automatic debugging technique that abstracts the operational details of computations, allowing the programmers to debug their code from an abstract point of view. However, its use in practice is still marginal, and one of the reasons is the lack of precision of this technique when reporting errors (current algorithmic debuggers do not point an expression or line as buggy, but they point a whole procedure/function/method as containing the bug). In this paper, we make a step forward to overcome this problem. We identify two specific causes of that problem in the standard formulation and implementations of algorithmic debugging, and we present a reformulation to solve both problems. We show that the novel ideas included in the reformulation proposed cannot be supported by the standard internal data structures (such as the Execution Tree) used in this technique and, hence, a generalisation of the standard definitions and algorithms is needed. The reformulation has been done in a language-independent manner to make it useful and reusable in different programming languages.

Contextualizing spectrum-based fault localization

Context: Fault localization is among the most expensive tasks in software development. Spectrum-based fault localization (SFL) techniques seek to pinpoint faulty program elements (e.g., statements), by sorting them only by their suspiciousness scores. Developers tend to fall back on another debugging strategy if they do not find the bug in the first positions of a suspiciousness list.Objective: In this study, we assess techniques to contextualize code inspection whose goal is two-fold: to provide guidance during fault localization, and to improve the effectiveness of SFL techniques in classifying bugs within the first picks. Code Hierarchy (CH) and Integration Coverage-based Debugging (ICD) techniques provide a search roadmap—a list of methods—that guide the developer toward faults. CH assigns a method with the highest suspiciousness score of its blocks, and ICD captures method call relationships from testing to establish the roadmap. Two new filtering strategies—Fixed Budget (FB) and Level Score (LS)—are combined with ICD and CH for reducing the amount of blocks to inspect in each method.Method: We evaluated the effectiveness of ICD, CH, FB, LS, and a suspiciousness block list (BL) on 62 bugs from 7 real programs.Results: ICD and CH using FB found more faults inspecting less blocks than BL with statistical significance. More than 50% of the faults were found inspecting at most 10 blocks using ICD-FB and CH-FB. Moreover, ICD and CH located 70% of the faults by inspecting, at most, 4 methods.Conclusions: These results suggest that the contextualization provided by roadmaps and filtering strategies is useful for guiding developers toward faults and improves the performance of SFL techniques.

Performance Prediction for Large-Scale Parallel Applications Using Representative Replay

Automatically predicting performance of parallel applications has been a long-standing goal in the area of high performance computing. However, accurate performance prediction is challenging, since the execution time of parallel applications is determined by several factors, such as sequential computation time, communication time and their complex interactions. Despite previous efforts, accurately estimating the sequential computation time in each process for large-scale parallel applications remains an open problem. In this paper, we propose a novel approach to acquiring accurate sequential computation time using a parallel debugging technique called deterministic replay. The main advantage of our approach is that we only need a single node of a target platform but the whole target platform does not need to be available. Therefore, with this approach we can simply measure the real sequential computation time on a target node for each process on by one. Moreover, we observe that there is great computation similarity in parallel applications, not only within each process but also among different processes. Based on this observation, we further propose representative replay that can significantly reduce replay overhead, because we only need to replay partial iterations for representative processes instead of all of them. Finally, we implement a complete performance prediction system, called Phantom , which combines the above computation-time acquisition approach and a trace-driven simulator. We validate our approach on both traditional HPC platforms and the latest Amazon EC2 cloud platform. On both types of platforms, prediction error of our approach is less than 7 percent on average up to 2,500 processes.

Visual Debugging Techniques for Reactive Data Visualization

AbstractInteraction is critical to effective visualization, but can be difficult to author and debug due to dependencies among input events, program state, and visual output. Recent advances leverage reactive semantics to support declarative design and avoid the “spaghetti code” of imperative event handlers. While reactive programming improves many aspects of development, textual specifications still fail to convey the complex runtime dynamics. In response, we contribute a set of visual debugging techniques to reveal the runtime behavior of reactive visualizations. A timeline view records input events and dynamic variable updates, allowing designers to replay and inspect the propagation of values step‐by‐step. On‐demand annotations overlay the output visualization to expose relevant state and scale mappings in‐situ. Dynamic tables visualize how backing datasets change over time. To evaluate the effectiveness of these techniques, we study how first‐time Vega users debug interactions in faulty, unfamiliar specifications; with no prior knowledge, participants were able to accurately trace errors through the specification.