Correct Patches Research Articles

An Automatic Software Defect Repair Method Based on Multi-Objective Genetic Programming

Due to the explosive growth of software quantity and the mixed ability of software developers, a large number of software defects emerge during the later stages of software maintenance. The search method based on genetic programming is one of the most popular in search algorithms, but it also has some issues. The single-objective approach to validate and select offspring patches without considering other constraints can affect the efficiency of patch generation. To address this issue, this paper proposes an automatic software repair method based on Multi-objective Genetic Programming (MGPRepair). Firstly, the method adopts a lightweight context analysis strategy to find suitable repair materials. Secondly, it decouples the replacement statements and insertion statements in the repair materials, using a lower-granularity patch representation method to encode the patches in the search space. Then, the automatic software defect repair is treated as a multi-objective search problem, and the NSGA-II multi-objective optimization algorithm is used to find simpler repair patches. Finally, the test case filtering technique is used to accelerate the patch validation process and generate correct patches. MGPRepair was experimentally evaluated on 395 real Java software defects from the Defects4J dataset. The experimental results show that MGPRepair can generate test case-passing patches for 51 defects, of which 35 defect patches are equivalent to manually generated patches. Its repair the efficiency and success rate are higher to other excellent automatic software defect repair methods such as jGenProg, RSRepair, ARJA, Nopol, Capgen, and SequenceR.

On the acceptance by code reviewers of candidate security patches suggested by Automated Program Repair tools

ObjectiveWe investigated whether (possibly wrong) security patches suggested by Automated Program Repairs (APR) for real world projects are recognized by human reviewers. We also investigated whether knowing that a patch was produced by an allegedly specialized tool does change the decision of human reviewers.MethodWe perform an experiment with n=72\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$n= 72$$\\end{document} Master students in Computer Science. In the first phase, using a balanced design, we propose to human reviewers a combination of patches proposed by APR tools for different vulnerabilities and ask reviewers to adopt or reject the proposed patches. In the second phase, we tell participants that some of the proposed patches were generated by security-specialized tools (even if the tool was actually a ‘normal’ APR tool) and measure whether the human reviewers would change their decision to adopt or reject a patch.ResultsIt is easier to identify wrong patches than correct patches, and correct patches are not confused with partially correct patches. Also patches from APR Security tools are adopted more often than patches suggested by generic APR tools but there is not enough evidence to verify if ‘bogus’ security claims are distinguishable from ‘true security’ claims. Finally, the number of switches to the patches suggested by security tool is significantly higher after the security information is revealed irrespective of correctness.LimitationsThe experiment was conducted in an academic setting, and focused on a limited sample of popular APR tools and popular vulnerability types.

Confix: Combining node-level fix templates and masked language model for automatic program repair

Automatic program repair (APR) is a promising technique to fix program defects by generating patches. In the current APR techniques, template-based and learning-based techniques have demonstrated different advantages. Template-based APR techniques rely on pre-defined fix templates, providing higher controllability but limited by the variety of templates and edit expressiveness. In contrast, learning-based APR techniques treat repair as a neural machine translation task, improving the edit expressiveness through training neural networks. However, this technique also faces the influence of quality and variety of training data, leading to numerous errors and redundant code generation. To overcome their limitations, this paper proposes an innovative APR technique called Confix. Confix first constructs a code information tree to assist in mining edit changes during historical repair. It then further enriches the types of fix templates using node information in the tree. Afterward, Confix defines masked lines based on node-level fix templates to control the scope of patch generation, avoiding redundant semantic code generation. Finally, Confix leverages the powerful edit expressiveness of the masked language model and combines it with fix strategies to generate correct patches more efficiently and accurately. Experimental results show that Confix exhibits state-of-the-art performance on the Defects4J 1.2 and QuixBugs benchmarks.

Enhanced evolutionary automated program repair by finer‐granularity ingredients and better search algorithms

SummaryBug repair is time consuming and tedious, which hampers software maintenance. To alleviate the burden, automated program repair (APR) is proposed and has been fruitful in the last decade. Evolutionary repair is the seminal work of this field and proliferated a family of approaches. The performance of evolutionary repair approaches is affected by two main factors: (1) search space, which defines all possible patches, and (2) search algorithms, which navigate the space. Although recent approaches have achieved remarkable progress, the main challenges of the two factors still remain. On one hand, the different kinds of search space are very coarse for containing correct patches. On the other hand, the search process guided by genetic algorithms is inefficient in finding the correct patches in an appropriate time budget. In this paper, we propose MicroRepair, a new evolutionary repair approach to address the two challenges. Rather than finding statement‐level patches like existing genetic repair approaches, MicroRepair enlarges the search space by breaking the statements into finer‐granularity ingredients that consist of AST leaves. As the search space grows exponentially, the former search algorithms may become inefficient in navigating the larger space. We utilize the best multiobjective search algorithm selected from our empirical comparison of a set of search algorithms. At last, we find redundancies search in the existing genetic process, and we further design a history‐aware search strategy to accelerate the process.We evaluated MicroRepair on 224 bugs of real‐world from the benchmark Defects4J and compared it with several state‐of‐the‐art repair approaches. The evaluation results show that MicroRepair correctly repaired 26 bugs with a precision of 62%, which significantly outperforms the state‐of‐the‐art evolutionary APR approaches in terms of precision. Moreover, the history‐aware search boosts the repair execution speed by 4% on average.

The Best of Both Worlds: Combining Learned Embeddings with Engineered Features for Accurate Prediction of Correct Patches

A large body of the literature on automated program repair develops approaches where patches are automatically generated to be validated against an oracle (e.g., a test suite). Because such an oracle can be imperfect, the generated patches, although validated by the oracle, may actually be incorrect. While the state-of-the-art explores research directions that require dynamic information or rely on manually-crafted heuristics, we study the benefit of learning code representations in order to learn deep features that may encode the properties of patch correctness. Our empirical work investigates different representation learning approaches for code changes to derive embeddings that are amenable to similarity computations of patch correctness identification, and assess the possibility of accurate classification of correct patch by combining learned embeddings with engineered features. Experimental results demonstrate the potential of learned embeddings to empower Leopard (a patch correctness predicting framework implemented in this work) with learning algorithms in reasoning about patch correctness: a machine learning predictor with BERT transformer-based learned embeddings associated with XGBoost achieves an AUC value of about 0.803 in the prediction of patch correctness on a new dataset of 2,147 labeled patches that we collected for the experiments. Our investigations show that deep learned embeddings can lead to complementary/better performance when comparing against the state-of-the-art, PATCH-SIM, which relies on dynamic information. By combining deep learned embeddings and engineered features, Panther (the upgraded version of Leopard implemented in this work) outperforms Leopard with higher scores in terms of AUC, +Recall and -Recall, and can accurately identify more (in)correct patches that cannot be predicted by the classifiers only with learned embeddings or engineered features. Finally, we use an explainable ML technique, SHAP, to empirically interpret how the learned embeddings and engineered features are contributed to the patch correctness prediction.

Reliable Fix Patterns Inferred from Static Checkers for Automated Program Repair

Fix pattern-based patch generation is a promising direction in automated program repair (APR). Notably, it has been demonstrated to produce more acceptable and correct patches than the patches obtained with mutation operators through genetic programming. The performance of pattern-based APR systems, however, depends on the fix ingredients mined from fix changes in development histories. Unfortunately, collecting a reliable set of bug fixes in repositories can be challenging. In this article, we propose investigating the possibility in an APR scenario of leveraging fix patterns inferred from code changes that address violations detected by static analysis tools. To that end, we build a fix pattern-based APR tool, Avatar , which exploits fix patterns of static analysis violations as ingredients for the patch generation of repairing semantic bugs. Evaluated on four benchmarks (i.e., Defects4J, Bugs.jar, BEARS, and QuixBugs), Avatar presents the potential feasibility of fixing semantic bugs with the fix patterns inferred from the patches for fixing static analysis violations and can correctly fix 26 semantic bugs when Avatar is implemented with the normal program repair pipeline. We also find that Avatar achieves performance metrics that are comparable to that of the closely related approaches in the literature. Compared with CoCoNut, Avatar can fix 18 new bugs in Defects4J and 3 new bugs in QuixBugs. When compared with HDRepair, JAID, and SketchFix, Avatar can newly fix 14 Defects4J bugs. In terms of the number of correctly fixed bugs, Avatar is also comparable to the program repair tools with the normal fault localization setting and presents better performance than most program repair tools. These results imply that Avatar is complementary to current program repair approaches. We further uncover that Avatar can present different bug-fixing performances when it is configured with different fault localization tools, and the stack trace information from the failed executions of test cases can be exploited to improve the bug-fixing performance of Avatar by fixing more bugs with fewer generated patch candidates. Overall, our study highlights the relevance of static bug-finding tools as indirect contributors of fix ingredients for addressing code defects identified with functional test cases (i.e., dynamic information).

Predicting Patch Correctness Based on the Similarity of Failing Test Cases

How do we know a generated patch is correct? This is a key challenging question that automated program repair (APR) systems struggle to address given the incompleteness of available test suites. Our intuition is that we can triage correct patches by checking whether each generated patch implements code changes (i.e., behavior) that are relevant to the bug it addresses. Such a bug is commonly specified by a failing test case. Towards predicting patch correctness in APR, we propose a novel yet simple hypothesis on how the link between the patch behavior and failing test specifications can be drawn: similar failing test cases should require similar patches . We then propose BATS , an unsupervised learning-based approach to predict patch correctness by checking patch B ehavior A gainst failing T est S pecification. BATS exploits deep representation learning models for code and patches: For a given failing test case, the yielded embedding is used to compute similarity metrics in the search for historical similar test cases to identify the associated applied patches, which are then used as a proxy for assessing the correctness of the APR-generated patches. Experimentally, we first validate our hypothesis by assessing whether ground-truth developer patches cluster together in the same way that their associated failing test cases are clustered. Then, after collecting a large dataset of 1,278 plausible patches (written by developers or generated by 32 APR tools), we use BATS to predict correct patches: BATS achieves AUC between 0.557 to 0.718 and recall between 0.562 and 0.854 in identifying correct patches. Our approach outperforms state-of-the-art techniques for identifying correct patches without the need for large labeled patch datasets—as is the case with machine learning-based approaches. While BATS is constrained by the availability of similar test cases, we show that it can still be complementary to existing approaches: When combined with a recent approach that relies on supervised learning, BATS improves the overall recall in detecting correct patches. We finally show that BATS is complementary to the state-of-the-art PATCH-SIM dynamic approach for identifying correct patches generated by APR tools.

Quality Evaluation Method of Automatic Software Repair Using Syntax Distance Metrics

In recent years, test-based automatic program repair has attracted widespread attention. However, the test suites in practice are not perfect ways to guarantee the correctness of patches generated by repair tools, and weak test suites lead to a large number of incorrect patches produced by the existing repair tool. To reduce the number of incorrect patches generated by repair tools, we propose a patch quality evaluation method based on syntax distance metrics, which measures the syntax distance of patches through four evaluation features—variable, expression, structure, and repair location. By fusing the distance values of the four features, the quality of a patch can be evaluated. Our method evaluates 368 patches from multiple famous repair tools, such as jKali, Nopol, SimFix, DynaMoth, and CapGen; 95% of the correct patches were ranked in the top one of the plausible patches for each defect, which indicates our method can find high-quality patches.

Improving Search-Based Automatic Program Repair With Neural Machine Translation

The challenge of automatically repairing bugs in programs to reduce debugging expenses and increase program quality is known as automated program repair. To overcome this issue, test-suite-based repair techniques use a specified test suite as an oracle and alter the input faulty program to pass the full test suite. GenProg is a well-known example of this kind of repair, in which genetic programming is used to reorder the statements already present in the faulty program. However, recent practical experiments suggest that GenProg’s performance, notably for Java, is not sufficient. Improved program dependability necessitates the use of automatic program repair techniques. Template-based program repair techniques have recently been combined with search-based techniques to solve program issues automatically. Although intriguing, it has two fundamental drawbacks: Its search space often lacks the correct solution, and the technique disregards program expertise, such as precise code language. Compared with the template-based program repair approach, existing neural-machine-translation-based approaches are not limited by these constraints due to their ability to learn and generate new solutions. We propose an approach that combines a search-based automatic program repair technique with a neural-machine-translation-based approach. More specifically, we use both redundancy assumption and sequence-to-sequence learning of correct patches as the source for potential fix statements that feed into a multiobjective evolutionary search algorithm to find test-suite-adequate patches. In this work, a novel framework called ARJANMT is introduced for automatically repairing Java programs. Two sets of controlled experiments are conducted on 410 bugs from two benchmarks to investigate the repairability and correctness of our proposed framework. A comparison between state-of-the-art automatic program repair frameworks is made. The experimental results indicate that combining those two types of repair techniques (search-based and neural-machine-translation-based) produces better results or fixes bugs that they previously were unable to fix individually.

Identifying Incorrect Patches in Program Repair Based on Meaning of Source Code

Automatic Program Repair (APR) techniques have shown the potential of reducing debugging costs while improving software quality by generating patches for fixing bugs automatically. However, they often generate many overfitting patches which pass only a specific test-suite but do not fix the bugs correctly. This paper proposes MIPI, a novel approach to reducing the number of overfitting patches generated in the APR. We leverage recent advances in deep learning to exploit the similarity between the patched method’s name (which often encloses the developer’s intention about the code) and the semantic meaning of the method’s body (which represents the actual implemented behavior) for identifying and removing overfitting patches generated by APR tools. Experiments with a large dataset of patches for QuixBugs and Defects4J programs show the promise of our approach. Specifically, in a total of 1,191 patches generated by 23 existing APR tools, MIPI successfully filters out 254 (32%) of the total 797 overfitting patches with a precision of 90% while preserving 93% of the correct patches. MIPI is more precise and less damaging to the APR than existing heuristic patch assessment techniques, achieving a higher recall than automated testing-based techniques that do not have access to the test oracle. In addition, MIPI is highly complementary to existing automated patch assessment techniques.

Trident: Controlling Side Effects in Automated Program Repair

The goal of program repair is to eliminate a bug in a given program by automatically modifying its source code. The majority of real-world software is written in imperative programming languages. Each function or expression in imperative code may have side effects, observable effects beyond returning a value. Existing program repair approaches have a limited ability to handle side effects. Previous test-driven semantic repair approaches only synthesise patches without side effects. Heuristic repair approaches generate patches with side effects only if suitable code fragments exist in the program or a database of repair patterns, or can be derived from training data. This work introduces Trident, the first test-driven program repair approach that synthesizes patches with side effects without relying on the plastic surgery hypothesis, a database of patterns, or training data. Trident relies on an interplay of several parts. First, it infers a specification for synthesising side-effected patches using symbolic execution with a custom state merging strategy that alleviates path explosion due to side effects. Second, it uses a novel component-based patch synthesis approach that supports lvalues, values that appear on the left-hand sides of assignments. In an evaluation on open-source projects, Trident successfully repaired 6 out of 10 real bugs that require insertion of new code with side effects, which previous techniques do not therefore repair. Evaluated on the ManyBugs benchmark, Trident successfully repaired two new bugs that previous approaches could not. Adding patches with side effects to the search space can exacerbate test-overfitting. We experimentally demonstrate that the simple heuristic of preferring patches with the fewest side effects alleviates the problem. An evaluation on a large number of smaller programs shows that this strategy reduces test-overfitting caused by side-effects, increasing the rate of correct patches from 33.3% to 58.3%.

Priority Measurement of Patches for Program Repair Based on Semantic Distance

Automated program repair is an effective way to ensure software quality and improve software development efficiency. At present, there are many methods and tools of automated program reapir in real world, but most of them have low repair accuracy, resulting in a large number of incorrect patches in the generated patches. To solve this problem, we propose a patch quality evaluation method based on semantic distance, which measures the semantic distance of patches by using features of interval distance, output coverage, and path matching. For each evaluation feature, we give a quantitative formula to obtain a specific distance value and use the distance to calculate the recommended patch value to measure the quality of the patch. Our quality evaluation method evaluated 279 patches from previous program repair tools, including Nopol, DynaMoth, ACS, jGenProg, and CapGen. This quality evaluation method successfully arranged the correct patches before the plausible but incorrect patches, and it recommended the higher-ranked patches to users first. On this basis, we compared our evaluation method with the existing evaluation methods and judged the evaluation ability of each feature. We showed that our proposed patch quality evaluation method can improve the repair accuracy of repair tools.

Automatic Repair of Semantic Defects Using Restraint Mechanisms

Recently, software, especially CPS and Internet of Things (IoT), increasingly have high requirements for quality, while program defects exist inevitably duo to the high complexity. Program defect repair faces serious challenges in that such repairs require considerable manpower, and the existing automatic repair approaches have difficulty generating correct patches efficiently. This paper proposes an automatic method for repairing semantic defects in Java programs based on restricted sets which refer to the interval domains of related variables that can trigger program semantic defects. Our work introduces a repair mechanism symmetrically combining defect patterns and repair templates. First, the program semantic defects are summarized into defect patterns according to their grammar and semantic features. A repair template for each type of defect pattern is predefined based on a restricted-set. Then, for each specific defect, a patch statement is automatically synthesized according to the repair template, and the detected defect information is reported by the static detection tool (DTSJava). Next, the patch location is determined by the def-use chain of defect-related variables. Finally, we evaluate the patches generated by our method using DTSJava. We implemented the method in the defect automatic repair prototype tool DTSFix to verify the effect of repairing the semantic defects detected by DTSJava in 6 Java open-source projects. The experimental results showed that 109 of 129 program semantic defects were repaired.

A critical review on the evaluation of automated program repair systems

Automated Program Repair (APR) has attracted significant attention from software engineering research and practice communities in the last decade. Several teams have recorded promising performance in fixing real bugs and there is a race in the literature to fix as many bugs as possible from established benchmarks. Gradually, repair performance of APR tools in the literature has gone from being evaluated with a metric on the number of generated plausible patches to the number of correct patches. This evolution is necessary after a study highlighting the overfitting issue in test suite-based automatic patch generation. Simultaneously, some researchers are also insisting on providing time cost in the repair scenario as a metric for comparing state-of-the-art systems.In this paper, we discuss how the latest evaluation metrics of APR systems could be biased. Since design decisions (both in approach and evaluation setup) are not always fully disclosed, the impact on repair performance is unknown and computed metrics are often misleading. To reduce notable biases of design decisions in program repair approaches, we conduct a critical review on the evaluation of patch generation systems and propose eight evaluation metrics for fairly assessing the performance of APR tools. Eventually, we show with experimental data on 11 baseline program repair systems that the proposed metrics allow to highlight some caveats in the literature. We expect wide adoption of these metrics in the community to contribute to boosting the development of practical, and reliably performable program repair tools.

Toward Better Evolutionary Program Repair

Bug repair is a major component of software maintenance, which requires a huge amount of manpower. Evolutionary computation, particularly genetic programming (GP), is a class of promising techniques for automating this time-consuming and expensive process. Although recent research in evolutionary program repair has made significant progress, major challenges still remain. In this article, we propose ARJA-e, a new evolutionary repair system for Java code that aims to address challenges for the search space, search algorithm, and patch overfitting. To determine a search space that is more likely to contain correct patches, ARJA-e combines two sources of fix ingredients (i.e., the statement-level redundancy assumption and repair templates) with contextual analysis-based search space reduction, thereby leveraging their complementary strengths. To encode patches in GP more properly, ARJA-e unifies the edits at different granularities into statement-level edits and then uses a lower-granularity patch representation that is characterized by the decoupling of statements for replacement and statements for insertion. ARJA-e also uses a finer-grained fitness function that can make full use of semantic information contained in the test suite, which is expected to better guide the search of GP. To alleviate patch overfitting, ARJA-e further includes a postprocessing tool that can serve the purposes of overfit detection and patch ranking. We evaluate ARJA-e on 224 real Java bugs from Defects4J and compare it with the state-of-the-art repair techniques. The evaluation results show that ARJA-e can correctly fix 39 bugs in terms of the patches ranked first, achieving substantial performance improvements over the state of the art. In addition, we analyze the effect of the components of ARJA-e qualitatively and quantitatively to demonstrate their effectiveness and advantages.

SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair

This paper presents a novel end-to-end approach to program repair based on sequence-to-sequence learning. We devise, implement, and evaluate a system, called SequenceR, for fixing bugs based on sequence-to-sequence learning on source code. This approach uses the copy mechanism to overcome the unlimited vocabulary problem that occurs with big code. Our system is data-driven; we train it on 35,578 samples, carefully curated from commits to open-source repositories. We evaluate it on 4,711 independent real bug fixes, as well on the Defects4J benchmark used in program repair research. SequenceR is able to perfectly predict the fixed line for 950/4711 testing samples, and find correct patches for 14 bugs in Defects4J. It captures a wide range of repair operators without any domain-specific top-down design.

A manual inspection of Defects4J bugs and its implications for automatic program repair

Automatic program repair techniques, which target to generate correct patches for real-world defects automatically, have gained a lot of attention in the last decade. Many different techniques and tools have been proposed and developed. However, even the most sophisticated automatic program repair techniques can only repair a small portion of defects while producing a large number of incorrect patches. A possible reason for the low performance is the test suites of real-world programs are usually too weak to guarantee the behavior of a program. To understand to what extent defects can be fixed with exiting test suites, we manually analyzed 50 real-world defectsfrom Defects4J, where a large portion (i.e., 82%) of them were correctly fixed This result suggests that there is much roomfor the current automatic program repair techniques to improve. Furthermore, we summarized seven fault localization and seven patch generation strategies that are useful in localizing and fixing these defects, and compared those strategies with current techniques. The results indicate potential directions to improve automatic program repair in the future.

The effectiveness of context-based change application on automatic program repair

An Automatic Program Repair (APR) technique is an implementation of a repair model to fix a given bug by modifying program behavior. Recently, repair models which collect source code and code changes from software history and use such collected resources for patch generation became more popular. Collected resources are used to expand the patch search space and to increase the probability that correct patches for bugs are included in the space. However, it is also revealed that navigation on such expanded patch search space is difficult due to the sparseness of correct patches in the space. In this study, we evaluate the effectiveness of Context-based Change Application (CCA) technique on change selection, fix location selection and change concretization, which are the key aspects of navigating patch search space. CCA collects abstract subtree changes and their AST contexts, and applies them to fix locations only if their contexts are matched. CCA repair model can address both search space expansion and navigation issues, by expanding search space with collected changes while narrowing down search areas in the search space based on contexts. Since CCA applies changes to a fix location only if their contexts are matched, it only needs to consider the same context changes for each fix location. Also, if there is no change with the same context as a fix location, this fix location can be ignored since it means that past patches did not modify such locations. In addition, CCA uses fine-grained changes preserving changed code structures, but normalizing user-defined names. Hence change concretization can be simply done by replacing normalized names with concrete names available in buggy code. We evaluated CCA’s effectiveness with over 54K unique collected changes (221K in total) from about 5K human-written patches. Results show that using contexts, CCA correctly found 90.1% of the changes required for test set patches, while fewer than 5% of the changes were found without contexts. We discovered that collecting more changes is only helpful if it is supported by contexts for effective search space navigation. In addition, CCA repair model found 44-70% of the actual fix locations of Defects4j patches more quickly compared to using SBFL techniques only. We also found that about 48% of the patches can be fully concretized using concrete names from buggy code.

Cascaded Static and Dynamic Local Feature Extractions for Face Sketch to Photo Matching

The automatic identification of a corresponding photo from a face sketch can assist in criminal investigations. The face sketch is rendered based on the descriptions elicited by the eyewitness. This may cause the face sketch to have some degrees of shape exaggeration that make some parts of the face geometrically misaligned. In this paper, we attempt to address the effect of these influences by a cascaded static and dynamic local feature extraction method so that the constructed feature vectors are built based on the correct patches. In the proposed method, the feature vectors from the local static extraction on a sketch and photo are matched using the nearest neighbors. Then, some n most similar photos are shortlisted based on the nearest neighbors. These photos are eventually re-matched using feature vectors from the local dynamic extraction method. The feature vectors are matched using the L 1 -distance measure. The experimental results for The Chinese University of Hong Kong (CUHK) Face Sketch Database (CUFS) and CUHK Face Sketch FERET Database (CUFSF) datasets indicate that the proposed method outperforms the state-of-the-art methods.

Automatic patch generation by learning correct code

We present Prophet, a novel patch generation system that works with a set of successful human patches obtained from open- source software repositories to learn a probabilistic, application-independent model of correct code. It generates a space of candidate patches, uses the model to rank the candidate patches in order of likely correctness, and validates the ranked patches against a suite of test cases to find correct patches. Experimental results show that, on a benchmark set of 69 real-world defects drawn from eight open-source projects, Prophet significantly outperforms the previous state-of-the-art patch generation system.