Automated Program Repair Research Articles

Automated Program Repair: Comparative Analysis of Approaches

Automated Program Repair: Comparative Analysis of Approaches

Automated program repair

Automated program repair can relieve programmers from the burden of manually fixing the ever-increasing number of programming mistakes.

APRSuite: A suite of components and use cases based on categorical decomposition of automatic program repair techniques and tools

During the last decade, we are witnessing the advent of a proliferation of techniques and associated tools for automatic program repair (APR). The current techniques and tools provide rich sources of knowledge that should be taken into consideration for future research. An overview of the current APR techniques and tools can serve the research community as a knowledge accumulator. However, APR techniques and tools differ in many aspects making knowledge accumulation challenging. To overcome this challenge, in this paper, we propose to leverage common components that constitute the APR techniques and tools. To achieve this objective, we surveyed current APR techniques and tools to identify the APR Suite of common constituent components, namely as APRSuite. Repair source and defect class are examples of identified components. We grouped these components into several categories such as patch evaluation and target defects. We have also identified some of the possible use cases per component as well as different lessons learned in studies for each component and for each use case. In addition, we developed a principled way for application of the components. The APRSuite and the principled way to apply it comprise a framework for knowledge accumulation, evaluation, and comparison of APR techniques and tools. The novelty of our work lies in its original viewpoint to the process of literature review in the APR research field. To demonstrate the applicability of the framework, we mapped out several concrete APR techniques, as a first instantiation of the framework. We observed that the framework brings discipline into the evaluation and/or comparison of APR techniques and tools. The framework offers these benefits objectively and systematically. We concluded that knowledge accumulation and characterization through literature reviews can be therefore facilitated through the identified suite of components while at the same time the existing component suite can be modified, augmented, or improved.

Automatic Repair of Timestamp Comparisons

Automated program repair has the potential to reduce the developers’ effort to fix errors in their code. In particular, modern programming languages, such as Java, C, and C#, represent time as integer variables that suffer from integer overflow, introducing subtle errors that are hard to discover and repair. Recent researches on automated program repair rely on test cases to discover failures to correct, making them suitable only for regression errors. We propose a new strategy to automatically repair programs that suffer from timestamp overflows that are manifested in comparison expressions. It unifies the benefits of static analysis and automatic program repair avoiding dependency on testing to identify and correct defected code. Our approach performs an abstract analysis over the time domain of a program using a Time Type System to identify the problematic comparison expressions. The repairing strategy rewrites the timestamp comparisons exploiting the binary representation of machine numbers to correct the code. We have validated the applicability of our approach with 20 open source Java projects. The results show that it is able to correctly repair all 246 identified errors. To further validate the reliability of our approach, we have proved the soundness of both, type system and repairing strategy. Furthermore, several patches for three open source projects have been acknowledged and accepted by their developers.

Automated Program Repair: Basic Concepts and Approaches

Automated Program Repair: Basic Concepts and Approaches

Automated program repair: a step towards software automation

Automated program repair: a step towards software automation

A manual inspection of Defects4J bugs and its implications for automatic program repair

Automatic program repair techniques, which target to generate correct patches for real-world defects automatically, have gained a lot of attention in the last decade. Many different techniques and tools have been proposed and developed. However, even the most sophisticated automatic program repair techniques can only repair a small portion of defects while producing a large number of incorrect patches. A possible reason for the low performance is the test suites of real-world programs are usually too weak to guarantee the behavior of a program. To understand to what extent defects can be fixed with exiting test suites, we manually analyzed 50 real-world defectsfrom Defects4J, where a large portion (i.e., 82%) of them were correctly fixed This result suggests that there is much roomfor the current automatic program repair techniques to improve. Furthermore, we summarized seven fault localization and seven patch generation strategies that are useful in localizing and fixing these defects, and compared those strategies with current techniques. The results indicate potential directions to improve automatic program repair in the future.

The effectiveness of context-based change application on automatic program repair

An Automatic Program Repair (APR) technique is an implementation of a repair model to fix a given bug by modifying program behavior. Recently, repair models which collect source code and code changes from software history and use such collected resources for patch generation became more popular. Collected resources are used to expand the patch search space and to increase the probability that correct patches for bugs are included in the space. However, it is also revealed that navigation on such expanded patch search space is difficult due to the sparseness of correct patches in the space. In this study, we evaluate the effectiveness of Context-based Change Application (CCA) technique on change selection, fix location selection and change concretization, which are the key aspects of navigating patch search space. CCA collects abstract subtree changes and their AST contexts, and applies them to fix locations only if their contexts are matched. CCA repair model can address both search space expansion and navigation issues, by expanding search space with collected changes while narrowing down search areas in the search space based on contexts. Since CCA applies changes to a fix location only if their contexts are matched, it only needs to consider the same context changes for each fix location. Also, if there is no change with the same context as a fix location, this fix location can be ignored since it means that past patches did not modify such locations. In addition, CCA uses fine-grained changes preserving changed code structures, but normalizing user-defined names. Hence change concretization can be simply done by replacing normalized names with concrete names available in buggy code. We evaluated CCA’s effectiveness with over 54K unique collected changes (221K in total) from about 5K human-written patches. Results show that using contexts, CCA correctly found 90.1% of the changes required for test set patches, while fewer than 5% of the changes were found without contexts. We discovered that collecting more changes is only helpful if it is supported by contexts for effective search space navigation. In addition, CCA repair model found 44-70% of the actual fix locations of Defects4j patches more quickly compared to using SBFL techniques only. We also found that about 48% of the patches can be fully concretized using concrete names from buggy code.

Repairnator patches programs automatically

Repairnator is a bot. It constantly monitors software bugs discovered during continuous integration of open-source software and tries to fix them automatically. If it succeeds in synthesizing a valid patch, Repairnator proposes the patch to the human developers, disguised under a fake human identity. To date, Repairnator has been able to produce patches that were accepted by the human developers and permanently merged into the code base. This is a milestone for human-competitiveness in software engineering research on automatic program repair.

Discovering common bug‐fix patterns: A large‐scale observational study

AbstractBackground: Automatic program repair aims to reduce costs associated with defect repair. The detection and characterization of common bug‐fix patterns in software repositories play an important role in advancing this field. Aim: In this paper, we characterize the occurrence of known bug‐fix patterns in Java repositories at an unprecedented large scale. Furthermore, we propose a novel automatic technique for unveiling frequent and isolated repair actions corresponding to realistic bug fixes in Java. Method: The study was conducted for Java GitHub projects organized in two distinct data sets. The first data set (Boa) contains more than 4 million bug‐fix commits from 101 471 projects. The second data set (Defects4J) contains 369 real bug fixes from five open‐source projects. Results: We characterized the prevalence of the five most common bug‐fix patterns (identified in the work of Pan et al) in those bug fixes. The combined results showed direct evidence that developers often forget to add IF preconditions in the code.Conclusion: We discover a total of 155 repair actions from Defects4J patches and discuss 10 pervasive repair actions that occur across all analyzed Java projects. Moreover, the overall Precision and Recall values for the clustering approach were 0.62 and 0.64, respectively.

Automated Test Generation for Debugging Multiple Bugs in Arithmetic Circuits

Optimized and custom arithmetic circuits are widely used in embedded systems such as multimedia applications, cryptography systems, signal processing and console games. Debugging of arithmetic circuits is a challenge due to increasing complexity coupled with non-standard implementations. Existing algebraic rewriting techniques produce a remainder to indicate the presence of a potential bug. However, bug localization remains a major bottleneck. Simulation-based validation using random or constrained-random tests are not effective for complex arithmetic circuits due to bit-blasting. In this paper, we present an automated test generation and bug localization technique for debugging arithmetic circuits. This paper makes four important contributions. We propose an automated approach for generating directed tests by suitable assignments of input variables to make the remainder non-zero. The generated tests are guaranteed to activate bugs. We also propose an automatic bug fixing technique by utilizing the patterns of the remainder terms as well as by analyzing the regions activated by the generated tests to detect and correct the error(s). We also propose an efficient debugging algorithm that can handle multiple dependent as well as independent bugs. Finally, our proposed framework, consisting of directed test generation, bug localization and bug correction, is fully automated. In other words, our framework is capable of producing a corrected implementation of arithmetic circuits without any manual intervention. Our experimental results demonstrate that the proposed approach can be used for automated debugging of large and complex arithmetic circuits.

Genetic improvement events in 2018

Genetic improvement uses optimisation and machine learning techniques, particularly heuristic search and evolutionary algorithms, to improve existing software. The main application is automatic bug fixing, where reducing or eliminating buggy behaviour improves a program. Other applications involve automatically producing a better program that runs faster, uses less memory, uses less energy, or runs on a different type of computer. This article summarise two international events exclusively devoted to this topic in 2018.

Automatic Software Repair: A Survey

Despite their growing complexity and increasing size, modern software applications must satisfy strict release requirements that impose short bug fixing and maintenance cycles, putting significant pressure on developers who are responsible for timely producing high-quality software. To reduce developers workload, repairing and healing techniques have been extensively investigated as solutions for efficiently repairing and maintaining software in the last few years. In particular, repairing solutions have been able to automatically produce useful fixes for several classes of bugs that might be present in software programs. A range of algorithms, techniques, and heuristics have been integrated, experimented, and studied, producing a heterogeneous and articulated research framework where automatic repair techniques are proliferating. This paper organizes the knowledge in the area by surveying a body of 108 papers about automatic software repair techniques, illustrating the algorithms and the approaches, comparing them on representative examples, and discussing the open challenges and the empirical evidence reported so far.

Predicting Effectiveness of Generate-and-Validate Patch Generation Systems Using Random Forest

One way to improve practicability of automatic program repair (APR) techniques is to build prediction models which can predict whether an application of a APR technique on a bug is effective or not. Existing prediction models have some limitations. First, the prediction models are built with hand crafted features which usually fail to capture the semantic characteristics of program repair task. Second, the performance of the prediction models is only evaluated on Genprog, a genetic-programming based APR technique. This paper develops prediction models, i.e., random forest prediction models for SPR, another kind of generate- and-validate APR technique, which can distinguish ineffective repair instances from effective repair instances. Rather than handcrafted features, we use features automatically learned by deep belief network (DBN) to train the prediction models. The empirical results show that compared to the baseline models, that is, all effective models, our proposed models can at least improve the F1 by 9% and AUC(area under the receiver operating characteristics curve) by 19%. At the same time, the prediction model using learned features at least outperforms the one using hand-crafted features in terms of F1 by 11%

Test-Equivalence Analysis for Automatic Patch Generation

Automated program repair is a problem of finding a transformation (called a patch) of a given incorrect program that eliminates the observable failures. It has important applications such as providing debugging aids, automatically grading student assignments, and patching security vulnerabilities. A common challenge faced by existing repair techniques is scalability to large patch spaces, since there are many candidate patches that these techniques explicitly or implicitly consider. The correctness criteria for program repair is often given as a suite of tests. Current repair techniques do not scale due to the large number of test executions performed by the underlying search algorithms. In this work, we address this problem by introducing a methodology of patch generation based on a test-equivalence relation (if two programs are “test-equivalent” for a given test, they produce indistinguishable results on this test). We propose two test-equivalence relations based on runtime values and dependencies, respectively, and present an algorithm that performs on-the-fly partitioning of patches into test-equivalence classes. Our experiments on real-world programs reveal that the proposed methodology drastically reduces the number of test executions and therefore provides an order of magnitude efficiency improvement over existing repair techniques, without sacrificing patch quality.

A Survey of Test Based Automatic Program Repair

A Survey of Test Based Automatic Program Repair

Editor’s Note: Special Issue on Automatic Software Repair

Editor’s Note: Special Issue on Automatic Software Repair

Automated clustering and program repair for introductory programming assignments

Providing feedback on programming assignments is a tedious task for the instructor, and even impossible in large Massive Open Online Courses with thousands of students. Previous research has suggested that program repair techniques can be used to generate feedback in programming education. In this paper, we present a novel fully automated program repair algorithm for introductory programming assignments. The key idea of the technique, which enables automation and scalability, is to use the existing correct student solutions to repair the incorrect attempts. We evaluate the approach in two experiments: (I) We evaluate the number, size and quality of the generated repairs on 4,293 incorrect student attempts from an existing MOOC. We find that our approach can repair 97% of student attempts, while 81% of those are small repairs of good quality. (II) We conduct a preliminary user study on performance and repair usefulness in an interactive teaching setting. We obtain promising initial results (the average usefulness grade 3.4 on a scale from 1 to 5), and conclude that our approach can be used in an interactive setting.

Mining repair model for exception-related bug

It has long been a hot research topic to detect and to repair bugs automatically. As a common practice, researchers propose approaches for specific bugs, and their approaches typically are limited in handling the variety among bugs. Recently, researchers start to explore automatic program repair. With predefined repair operators and test cases, test-based repair approaches use search algorithms to generate patches for a bug, until a patch passes all the test cases. To improve the effectiveness to generate patches, Martinez and Monperrus (2013b) proposed an approach that mines repair models from past fixes. Although their approach produces positive results, we argue that it can be feasible to further improve their approach, if we mine repair models for bug categories, instead of all bugs. However, the benefits are still unclear, since existing benchmarks do not classify bugs into categories and existing approaches cannot mine repair models for bug categories. In this paper, we implement a tool, called ExFi, that classifies bugs into categories based on their related exceptions. With its support, we construct a benchmark, in which bug categories are marked. Furthermore, we propose an approach, called MiMo, that mines a repair model for each exception. We compared the general repair model with our mined repair models. Our results show that our mined models are all significantly different from the general model. Outside of the projects where our models are mined, we selected 59 real bugs. For each bug, we used our models and the general model to generate correct repair shapes for these bugs. The results show that for 43 out of 59 bugs, our models found faster a correct shape than the general repair model (Martinez and Monperrus, 2013b), and for 5 bugs, our models were able to find correct shapes that were not found by the compared model.

Overfitting in semantics-based automated program repair

The primary goal of Automated Program Repair (APR) is to automatically fix buggy software, to reduce the manual bug-fix burden that presently rests on human developers. Existing APR techniques can be generally divided into two families: semantics- vs. heuristics-based. Semantics-based APR uses symbolic execution and test suites to extract semantic constraints, and uses program synthesis to synthesize repairs that satisfy the extracted constraints. Heuristic-based APR generates large populations of repair candidates via source manipulation, and searches for the best among them. Both families largely rely on a primary assumption that a program is correctly patched if the generated patch leads the program to pass all provided test cases. Patch correctness is thus an especially pressing concern. A repair technique may generate overfitting patches, which lead a program to pass all existing test cases, but fails to generalize beyond them. In this work, we revisit the overfitting problem with a focus on semantics-based APR techniques, complementing previous studies of the overfitting problem in heuristics-based APR. We perform our study using IntroClass and Codeflaws benchmarks, two datasets well-suited for assessing repair quality, to systematically characterize and understand the nature of overfitting in semantics-based APR. We find that similar to heuristics-based APR, overfitting also occurs in semantics-based APR in various different ways.