Automated Program Repair Techniques Research Articles

E-APR: Mapping the effectiveness of automated program repair techniques

E-APR: Mapping the effectiveness of automated program repair techniques

CGenProg: Adaptation of cartesian genetic programming with migration and opposite guesses for automatic repair of software regression faults

In the last decade, the research community has been actively working to develop the techniques that can automatically find a solution to a software fault, namely, automatic program repair (APR). As of today, a multitude of APR techniques has been proposed. The techniques could have effectively repaired a wide variety of fault classes. However, the development of effective APR techniques for software regression faults, which are prevalently occurred in the maintenance stage of the software lifecycle, have received little attention. By incorporating specific knowledge in the domain of software regression faults, we have developed a novel technique for automatic repair of software regression faults in Java programs, which we call CGenProg. To achieve this, we have extensively adapted and modified the original cartesian genetic programming (CGP), biogeography-based optimization (BBO), and opposition-based learning (OBL). The modified CGP serves us as the core evolutionary process while the modified BBO and OBL act as crossover and mutation, respectively. The significance of CGenProg is that it contributes to the solution of a practical problem faced by developers in the maintenance stage of the software lifecycle. For expert and intelligent systems, it extends what is known about the application of optimization algorithms in the context of APR. Further, it demonstrates a novel use of CGP, BBO, and OBL for automatic repair of software regression faults. To evaluate CGenProg, we have developed a prototype tool using the Java language. Then, we conducted experiments on several programs in Code4Bench where each program is released with multiple consecutive versions comprising software regression faults. In the experiments, CGenProg could repair 17 out of 30 faulty programs. We conclude that CGenProg proves relevant and effective for repairing software regression faults. The impact of this study is to incentivize researchers for further exploitation and adaptation of the wealth of existing metaheuristics to develop effective APR techniques for different fault classes.

Contract-Based Program Repair Without The Contracts: An Extended Study

Most techniques for automated program repair (APR) use tests to drive the repair process; this makes them prone to generating spurious repairs that overfit the available tests unless additional information about expected program behavior is available. Our previous work on <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> , an APR technique for Java programs, showed that constructing detailed state abstractions—similar to those employed by techniques for programs with contracts—from plain Java code without any special annotations provides valuable additional information, and hence helps mitigate the overfitting problem. This paper extends the work on <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> with a comprehensive experimental evaluation involving 693 bugs in three different benchmark suites. The evaluation shows, among other things, that: 1) <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> is effective: it produced correct fixes for over 15 percent of all bugs, with a precision of nearly 60 percent; 2) <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> is reasonably efficient: on average, it took less than 30 minutes to output a correct fix; 3) <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> is competitive with the state of the art, as it fixed more bugs than any other technique, and 11 bugs that no other tool can fix; 4) <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> is robust: its heuristics are complementary and their effectiveness does not depend on the fine-tuning of parameters. The experimental results also indicate the main trade-offs involved in designing an APR technique based on tests, as well as possible directions for further progress in this line of work.

Input Test Suites for Program Repair: A Novel Construction Method Based on Metamorphic Relations

Test-suite-based automated program repair (APR) techniques acquire information from an input test suite to guide the repair process, aiming to produce a repair that can pass all test cases of the input test suite. Obviously, the input test suite has a critical impact on the repair effectiveness of APR techniques. This article reports on a study of the APR input test suites from a new perspective. We first propose a novel method of constructing the APR input test suites, using information derived from violated metamorphic relations. We then empirically evaluate our construction method using three APR techniques (Angelix, CETI, and GenProg), comparing it with random and code-coverage-based construction methods that are used as the experimental control. The results show that our approach is complementary to these two input test suite construction methods. This article illustrates a new use of metamorphic relations for program repair.

Quality of Automated Program Repair on Real-World Defects

Automated program repair is a promising approach to reducing the costs of manual debugging and increasing software quality. However, recent studies have shown that automated program repair techniques can be prone to producing patches of low quality, overfitting to the set of tests provided to the repair technique, and failing to generalize to the intended specification. This paper rigorously explores this phenomenon on real-world Java programs, analyzing the effectiveness of four well-known repair techniques, GenProg, Par, SimFix, and TrpAutoRepair, on defects made by the projects’ developers during their regular development process. We find that: (1) When applied to real-world Java code, automated program repair techniques produce patches for between 10.6 and 19.0 percent of the defects, which is less frequent than when applied to C code. (2) The produced patches often overfit to the provided test suite, with only between 13.8 and 46.1 percent of the patches passing an independent set of tests. (3) Test suite size has an extremely small but significant effect on the quality of the patches, with larger test suites producing higher-quality patches, though, surprisingly, higher-coverage test suites correlate with lower-quality patches. (4) The number of tests that a buggy program fails has a small but statistically significant positive effect on the quality of the produced patches. (5) Test suite provenance, whether the test suite is written by a human or automatically generated, has a significant effect on the quality of the patches, with developer-written tests typically producing higher-quality patches. And (6) the patches exhibit insufficient diversity to improve quality through some method of combining multiple patches. We develop JaRFly, an open-source framework for implementing techniques for automatic search-based improvement of Java programs. Our study uses JaRFly to faithfully reimplement GenProg and TrpAutoRepair to work on Java code, and makes the first public release of an implementation of Par. Unlike prior work, our study carefully controls for confounding factors and produces a methodology, as well as a dataset of automatically-generated test suites, for objectively evaluating the quality of Java repair techniques on real-world defects.

Restore: Retrospective Fault Localization Enhancing Automated Program Repair

Fault localization is a crucial step of automated program repair, because accurately identifying program locations that are most closely implicated with a fault greatly affects the effectiveness of the patching process. An ideal fault localization technique would provide precise information while requiring moderate computational resources—to best support an efficient search for correct fixes. In contrast, most automated program repair tools use standard fault localization techniques—which are not tightly integrated with the overall program repair process, and hence deliver only subpar efficiency. In this paper, we present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">retrospective fault localization</i> : a novel fault localization technique geared to the requirements of automated program repair. A key idea of retrospective fault localization is to reuse the outcome of failed patch validation to support mutation-based dynamic analysis—providing accurate fault localization information without incurring onerous computational costs. We implemented retrospective fault localization in a tool called <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Restore</small> —based on the <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> Java program repair system. Experiments involving faults from the <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Defects4J</small> standard benchmark indicate that retrospective fault localization can boost automated program repair: <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Restore</small> efficiently explores a large fix space, delivering state-of-the-art effectiveness (41 <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Defects4J</small> bugs correctly fixed, 8 of which no other automated repair tool for Java can fix) while simultaneously boosting performance (speedup over 3 compared to <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> ). Retrospective fault localization is applicable to any automated program repair techniques that rely on fault localization and dynamic validation of patches.

Modification Point Aware Test Prioritization and Sampling to Improve Patch Validation in Automatic Program Repair

Recently, Automatic Program Repair (APR) has shown a high capability of repairing software bugs automatically. In general, most of the APR techniques require test suites to validate automatically generated patches. However, the test suites used for patch validation might contain thousands of test cases. Running these whole test suites to validate every program variant makes the validation process not only time-consuming but also expensive. To mitigate this issue and to enhance the patch validation in APR, we introduce (1) MPTPS (Modification Point-aware Test Prioritization and Sampling), which iteratively records test execution. Based on the failed test information, it performs test prioritization, then sampling to reduce the test execution time by moving forward the test cases that are most likely to fail in the test suite; and (2) a new fitness function that refines the existing one to improve repair efficiency. We implemented our MPPEngine approach in the Astor workspace by extending jGenProg. And the experiments on the Defects4j benchmark against jGenProg show that, on average, jGenProg consumes 79.27 s to validate one program variant, where MPPEngine takes only 33.70 s for results in 57.50% of validation time reduction. Also, MPPEngine outperforms jGenProg by finding patches for six more bugs than jGenProg.

APRSuite: A suite of components and use cases based on categorical decomposition of automatic program repair techniques and tools

During the last decade, we are witnessing the advent of a proliferation of techniques and associated tools for automatic program repair (APR). The current techniques and tools provide rich sources of knowledge that should be taken into consideration for future research. An overview of the current APR techniques and tools can serve the research community as a knowledge accumulator. However, APR techniques and tools differ in many aspects making knowledge accumulation challenging. To overcome this challenge, in this paper, we propose to leverage common components that constitute the APR techniques and tools. To achieve this objective, we surveyed current APR techniques and tools to identify the APR Suite of common constituent components, namely as APRSuite. Repair source and defect class are examples of identified components. We grouped these components into several categories such as patch evaluation and target defects. We have also identified some of the possible use cases per component as well as different lessons learned in studies for each component and for each use case. In addition, we developed a principled way for application of the components. The APRSuite and the principled way to apply it comprise a framework for knowledge accumulation, evaluation, and comparison of APR techniques and tools. The novelty of our work lies in its original viewpoint to the process of literature review in the APR research field. To demonstrate the applicability of the framework, we mapped out several concrete APR techniques, as a first instantiation of the framework. We observed that the framework brings discipline into the evaluation and/or comparison of APR techniques and tools. The framework offers these benefits objectively and systematically. We concluded that knowledge accumulation and characterization through literature reviews can be therefore facilitated through the identified suite of components while at the same time the existing component suite can be modified, augmented, or improved.

A manual inspection of Defects4J bugs and its implications for automatic program repair

Automatic program repair techniques, which target to generate correct patches for real-world defects automatically, have gained a lot of attention in the last decade. Many different techniques and tools have been proposed and developed. However, even the most sophisticated automatic program repair techniques can only repair a small portion of defects while producing a large number of incorrect patches. A possible reason for the low performance is the test suites of real-world programs are usually too weak to guarantee the behavior of a program. To understand to what extent defects can be fixed with exiting test suites, we manually analyzed 50 real-world defectsfrom Defects4J, where a large portion (i.e., 82%) of them were correctly fixed This result suggests that there is much roomfor the current automatic program repair techniques to improve. Furthermore, we summarized seven fault localization and seven patch generation strategies that are useful in localizing and fixing these defects, and compared those strategies with current techniques. The results indicate potential directions to improve automatic program repair in the future.

Predicting Effectiveness of Generate-and-Validate Patch Generation Systems Using Random Forest

One way to improve practicability of automatic program repair (APR) techniques is to build prediction models which can predict whether an application of a APR technique on a bug is effective or not. Existing prediction models have some limitations. First, the prediction models are built with hand crafted features which usually fail to capture the semantic characteristics of program repair task. Second, the performance of the prediction models is only evaluated on Genprog, a genetic-programming based APR technique. This paper develops prediction models, i.e., random forest prediction models for SPR, another kind of generate- and-validate APR technique, which can distinguish ineffective repair instances from effective repair instances. Rather than handcrafted features, we use features automatically learned by deep belief network (DBN) to train the prediction models. The empirical results show that compared to the baseline models, that is, all effective models, our proposed models can at least improve the F1 by 9% and AUC(area under the receiver operating characteristics curve) by 19%. At the same time, the prediction model using learned features at least outperforms the one using hand-crafted features in terms of F1 by 11%

Overfitting in semantics-based automated program repair

The primary goal of Automated Program Repair (APR) is to automatically fix buggy software, to reduce the manual bug-fix burden that presently rests on human developers. Existing APR techniques can be generally divided into two families: semantics- vs. heuristics-based. Semantics-based APR uses symbolic execution and test suites to extract semantic constraints, and uses program synthesis to synthesize repairs that satisfy the extracted constraints. Heuristic-based APR generates large populations of repair candidates via source manipulation, and searches for the best among them. Both families largely rely on a primary assumption that a program is correctly patched if the generated patch leads the program to pass all provided test cases. Patch correctness is thus an especially pressing concern. A repair technique may generate overfitting patches, which lead a program to pass all existing test cases, but fails to generalize beyond them. In this work, we revisit the overfitting problem with a focus on semantics-based APR techniques, complementing previous studies of the overfitting problem in heuristics-based APR. We perform our study using IntroClass and Codeflaws benchmarks, two datasets well-suited for assessing repair quality, to systematically characterize and understand the nature of overfitting in semantics-based APR. We find that similar to heuristics-based APR, overfitting also occurs in semantics-based APR in various different ways.

Do automated program repair techniques repair hard and important bugs?

Do automated program repair techniques repair hard and important bugs?

The impacts of techniques, programs and tests on automated program repair: An empirical study

Manual program repair is notoriously tedious, error-prone, and costly, especially for the modern large-scale projects. Automated program repair can automatically find program patches without much human intervention, greatly reducing the burden of developers as well as accelerating software delivery. Therefore, much research effort has been dedicated to design powerful program repair techniques. To date, although various program repair techniques have been proposed, to our knowledge, there lacks extensive study on the impacts of repair techniques, subject programs, and test suites on the repair effectiveness and efficiency. In this paper, we perform such an extensive study on repairing 180 seeded and real faults from 17 small to large sized programs. We study the impacts of five representative automated program repair techniques, including GenProg, RSRepair, Brute-force-based technique, AE and Kali, on the repair results. We further investigate the impacts of different subject programs and test suites on effectiveness and efficiency of program repair techniques. Our study demonstrates a number of interesting findings: Brute-force-based technique generates the maximum number of patches but is also the most costly technique, while Kali is the most efficient and has medium effectiveness among the studied techniques; techniques that work well with small programs become too costly or ineffective when applied to large sized programs; since tool-reported patches may overfit the selected test cases, we calculate the false positive rates and find that the influence of failed test cases is much larger than that of passed test cases; finally, surprisingly, all the studied techniques except RSRepair can find more than 80% of successful patches within the first 50% of search space.

A metamorphic testing approach for supporting program repair without the need for a test oracle

Test suite based automated program repair (APR) relies on a test oracle to determine the execution result of individual test cases. The applicability of APR techniques, therefore, is limited by the fact that test oracles may not exist. Metamorphic Testing (MT) is a testing approach that, rather than checking the correctness of individual test outputs, checks testing results through verification of relations among multiple test cases and their outputs: MT can therefore be applied without test oracles. This paper presents an integration of MT with APR that enables application of APR without the need for a test oracle. Two important issues for this integration which have been thoroughly investigated and addressed are: (1) feasibility — which is addressed by proposing a framework to support the integration, and then presenting MT-GenProg, a tool incorporating MT with the popular APR technique GenProg; and (2) effectiveness — which is confirmed through an empirical study of GenProg and MT-GenProg on 1,143 program versions from the IntroClass benchmark suite, demonstrating MT-GenProg’s comparable performance to GenProg, in terms of repair effectiveness. We conclude that the proposed integration is both practically feasible and effective, and thus successfully extends APR techniques to a broader application domain.