Automated Program Repair Research Articles

Patching Locking Bugs Statically with Crayons

The Linux Kernel is a world-class operating system controlling most of our computing infrastructure: mobile devices, Internet routers and services, and most of the supercomputers. Linux is also an example of low-level software with no comprehensive regression test suite (for good reasons). The kernel’s tremendous societal importance imposes strict stability and correctness requirements. These properties make Linux a challenging and relevant target for static automated program repair (APR). Over the past decade, a significant progress has been made in dynamic APR. However, dynamic APR techniques do not translate naturally to systems without tests. We present a static APR technique addressing sequential locking API misuse bugs in the Linux Kernel. We attack the key challenge of static APR, namely, the lack of detailed program specification, by combining static analysis with machine learning to complement the information presented by the static analyzer. In experiments on historical real-world bugs in the kernel, we were able to automatically re-produce or propose equivalent patches in 85% of the human-made patches, and automatically rank them among the top three candidates for 64% of the cases and among the top five for 74%.

How do Developers Really Feel About Bug Fixing? Directions for Automatic Program Repair

Automatic program repair (APR) is a rapidly advancing field of software engineering that aims to supplement or replace manual bug fixing with an automated tool. For APR to be successfully adopted in industry, it is vital that APR tools respond to developer needs and preferences. However, very little research has considered developers' general attitudes to APR or developers' current bug fixing practices (the activity APR aims to replace). This paper responds to this gap by reporting on a survey of 386 software developers about their bug finding and fixing practices and experiences, and their instinctive attitudes towards APR. We find that bug finding and fixing is not necessarily as onerous for developers as has often been suggested, being rated as more satisfying than developers' general work. The fact that developers derive satisfaction and benefit from bug fixing indicates that APR adoption is not as simple as APR replacing an unwanted activity. When it comes to potential APR approaches, we find a strong preference for developers being kept in the loop (for example, choosing between different fixes or validating fixes) as opposed to a fully automated process. This suggests that advances in APR should be careful to consider the agency of the developer, as well as what information is presented to developers alongside fixes. It also indicates that there are key barriers related to trust that would need to be overcome for full scale APR adoption, supported by the fact that even those developers who stated that they were positive about APR listed several caveats and concerns. We find very few statistically significant relationships between particular demographic variables (for example, developer experience, age, education) and key attitudinal variables, suggesting that developers' instinctive attitudes towards APR are little influenced by experience level but are held widely across the developer community.

Effective Isolation of Fault-Correlated Variables via Statistical and Mutation Analysis

It is a widely-adopted strategy for developers to monitor the values of program variables when debugging in practice. In particular, developers often set breakpoints at specific locations or execute the program step by step in the debugging mode to inspect if abnormal values or status will be observed for concerned variables. Such a practical debugging strategy can facilitate developers in understanding and localizing the target fault. This study aims to identify suspicious program variables of a given fault (i.e., denoted as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">fault-correlated variables</i> ) automatically, thus facilitating the debugging activities for developers. To the best of our knowledge, this is the finest granularity in fault localization (FL) so far, which can address the limitations of being coarse-grained as faced by existing FL techniques. However, isolating fault-correlated variables precisely is challenging since there are usually substantially different variables used or defined in a program, and plenty of them are in the same basic block which cannot be well discriminated from each other since they will be either executed or not against the given test suite. To address such challenges, this study presents IsoVar, a two-phase model to isolate fault-correlated variables. Specifically, IsoVar first performs statistical analysis based on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">variable execution matrices</i> , which is a novel concept proposed in this study, to identify a set of suspicious variables. It then observes the impacts of those variables on the program dynamically after applying subtle mutations at the bytecode level, to further isolate fault-correlated variables. Extensive experiments on Defects4J and Bears demonstrate that IsoVar can outperform state-of-the-art techniques significantly ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$13.0\%$</tex-math></inline-formula> for MAP and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$19.3\%$</tex-math></inline-formula> for MRR). More importantly, we incorporated IsoVar into 11 existing FL techniques as well as 14 automated program repair techniques, and found that IsoVar can significantly boost their performance.

SSDTutor: A feedback-driven intelligent tutoring system for secure software development

Application Programming Interfaces (APIs) in cryptography typically impose concealed usage constraints. The violations of these usage constraints can lead to software crashes or security vulnerabilities. Several professional tools can detect these constraints (API misuses) in cryptography; however, in the educational programs, the focus has been less on helping students implement an application without cryptographic API misuses that are caused by either a lack of cryptographic knowledge or programming mistakes. To address the problem, we present an intelligent tutoring approach SSDTutor for educating Secure Software Development. Our tutoring approach helps students or developers repair cryptographic API misuse defects by leveraging an automated program repair technique based on the usage patterns of cryptographic APIs. We studied the best practices of cryptographic implementations and encoded eight cryptographic API usage patterns. For quality feedback, we leverage a clone detection technique to recommend related feedback for helping students understand why their programs are incorrect, rather than blindly accepting repairs. We evaluated SSDTutor on 456 open source subject projects implemented with cryptographic APIs. SSDTutor successfully detected 1,553 out of 1,573 misuse defects with 98.9% accuracy and repaired 1,551 out of 1,573 misuse defects with 99.3% accuracy. In a user study involving 22 students, the participants reported that interactive SSDTutor's feedback recommendation could be valuable for novice students to learn about the correct usages of cryptography APIs.

Hippodrome: Data Race Repair Using Static Analysis Summaries

Implementing bug-free concurrent programs is a challenging task in modern software development. State-of-the-art static analyses find hundreds of concurrency bugs in production code, scaling to large codebases. Yet, fixing these bugs in constantly changing codebases represents a daunting effort for programmers, particularly because a fix in the concurrent code can introduce other bugs in a subtle way.In this work, we show how to harness compositional static analysis for concurrency bug detection, to enable a new Automated Program Repair (APR) technique for data races in large concurrent Java codebases. The key innovation of our work is an algorithm that translates procedure summaries inferred by the analysis tool for the purpose of bug reporting into small local patches that fix concurrency bugs (without introducing new ones). This synergy makes it possible to extend the virtues of compositional static concurrency analysis to APR, making our approacheffective(it can detect and fix many more bugs than existing tools for data race repair),scalable(it takes seconds to analyze and suggest fixes for sizeable codebases), andusable(generally, it does not require annotations from the users and can performcontinuousautomated repair). Our study, conducted on popular open-source projects, has confirmed that our tool automatically produces concurrency fixes similar to those proposed by the developers in the past.

IBiR : Bug-report-driven Fault Injection

Much research on software engineering relies on experimental studies based on fault injection. Fault injection, however, is not often relevant to emulate real-world software faults since it “blindly” injects large numbers of faults. It remains indeed challenging to inject few but realistic faults that target a particular functionality in a program. In this work, we introduce iBiR , a fault injection tool that addresses this challenge by exploring change patterns associated to user-reported faults. To inject realistic faults, we create mutants by re-targeting a bug-report-driven automated program repair system, i.e., reversing its code transformation templates. iBiR is further appealing in practice since it requires deep knowledge of neither code nor tests, just of the program’s relevant bug reports. Thus, our approach focuses the fault injection on the feature targeted by the bug report. We assess iBiR by considering the Defects4J dataset. Experimental results show that our approach outperforms the fault injection performed by traditional mutation testing in terms of semantic similarity with the original bug, when applied at either system or class levels of granularity, and provides better, statistically significant estimations of test effectiveness (fault detection). Additionally, when injecting 100 faults, iBiR injects faults that couple with the real ones in around 36% of the cases, while mutation testing achieves less than 4%.

Automatic Repair of Java Programs with Mixed Granularity and Variable Mapping

During the process of software repair, since the granularity of repair is too coarse and the way of fixing ingredient is too simple, the repair efficiency needs to be further improved. To resolve the problems, we propose a Mixed Granularity and Variable Mapping based automatic software Repair (MGVMRepair). We adopt random search algorithm as the framework of program evolution, and utilize the mapping relationship between variables as an auxiliary specification. Firstly, fault localization is used to locate the suspicious statements and to form a list of modification points. Secondly, the ingredient of program repair at statement level is obtained, and the mapping relationship of variables is established. Then, the test case prioritization is improved from the perspective of the modification point. Finally, a program passes all test cases or the program iteration terminates. The experimental results show that MGVMRepair has a higher repair success rate than GenProg, CapGen, SimFix, jKali, jMutRepair and SketchFix on Defects4J.

Neural Machine Translation in Translation and Program Repair

Translation is a challenge for humans since it needs a good command of two or more languages. When it comes to computer programs, it is even more complex as it is difficult for computers to imitate human translators. With the emergence of deep learning algorithms, especially neural network architectures, neural machine translation (NMT) models gradually outperformed previous machine translation models and became the new mainstream in practical machine translation (MT) systems. Nowadays, NMT has been developing for several years and has been applied in many fields. This paper is focused on studies on four different application categories of NMT models: 1) Text NMT; 2) Automatic program repair (based on NMT); 3) Simultaneous translation. Our work provides a summary of the latest research on different applications of NMT and makes comments on their development in the future. This paper also mentioned the shortcomings of existing studies in this essay and pointed out some possible research directions.

Program Repair With Repeated Learning

A key challenge in generate-and-validate automated program repair is directing the search for fixes so that it can efficiently find those that are more likely to be correct. To this end, several techniques use machine learning to capture the features of programmer-written fixes. In existing approaches, fitting the model typically takes place <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">before</i> fix generation and is independent of it: the fix generation process uses the learned model as one of its inputs. However, the intermediate outcomes of an ongoing fix generation process often provide valuable information about which candidate fixes were “better”; this information could profitably be used to retrain the model, so that each new iteration of the fixing process would also learn from the outcome of previous ones. In this paper, we propose the <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Liana</small> technique for automated program repair, which is based on this idea of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">repeatedly</i> learning the features of generated fixes. To this end, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Liana</small> uses a fine-grained model that combines information about fix characteristics, their relations to the fixing context, and the results of test execution. The model is initially trained offline, and then repeatedly updated online as the fix generation process unravels; at any step, the most up-to-date model is used to guide the search for fixes—prioritizing those that are more likely to include the right ingredients. In an experimental evaluation on 732 real-world Java bugs from 3 popular benchmarks, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Liana</small> built correct fixes for 134 faults (83 ranked as first in its output)— improving over several other generate-and-validate program repair tools according to various measures.

Let’s Talk With Developers, Not About Developers: A Review of Automatic Program Repair Research

Automatic program repair (APR) offers significant potential for automating some coding tasks. Using APR could reduce the high costs historically associated with fixing code faults and deliver significant benefits to software engineering. Adopting APR could also have profound implications for software developers’ daily activities, transforming their work practices. To realise the benefits of APR it is vital that we consider how developers feel about APR and the impact APR may have on developers’ work. Developing APR tools without consideration of the developer is likely to undermine the success of APR deployment. In this paper, we critically review how developers are considered in APR research by analysing how human factors are treated in 260 studies from Monperrus’s Living Review of APR. Over half of the 260 studies in our review were motivated by a problem faced by developers (e.g., the difficulty associated with fixing faults). Despite these human-oriented motivations, fewer than 7% of the 260 studies included a human study. We looked in detail at these human studies and found their quality mixed (for example, one human study was based on input from only one developer). Our results suggest that software developers are often talked <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">about</i> in APR studies, but are rarely talked <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">with</i> . A more comprehensive and reliable understanding of developer human factors in relation to APR is needed. Without this understanding, it will be difficult to develop APR tools and techniques which integrate effectively into developers’ workflows. We recommend a future research agenda to advance the study of human factors in APR.

Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning

Automated program repair (APR) faces the challenge of test overfitting, where generated patches pass validation tests but fail to generalize. Existing methods for patch assessment involve generating new tests or manual inspection, which can be time-consuming or biased. In this paper, we propose a novel technique, INVALIDATOR, to automatically assess the correctness of APR-generated patches via semantic and syntactic reasoning. INVALIDATOR leverages program invariants to reason about program semantics while also capturing program syntax through language semantics learned from a large code corpus using a pre-trained language model. Given a buggy program and the developer-patched program, INVALIDATOR infers likely invariants on both programs. Then, INVALIDATOR determines that an APR-generated patch overfits if: (1) it violates correct specifications or (2) maintains erroneous behaviors from the original buggy program. In case our approach fails to determine an overfitting patch based on invariants, INVALIDATOR utilizes a trained model from labeled patches to assess patch correctness based on program syntax. The benefit of INVALIDATOR is threefold. First, INVALIDATOR leverages both semantic and syntactic reasoning to enhance its discriminative capability. Second, INVALIDATOR does not require new test cases to be generated, but instead only relies on the current test suite and uses invariant inference to generalize program behaviors. Third, INVALIDATOR is fully automated. Experimental results demonstrate that INVALIDATOR outperforms existing methods in terms of Accuracy and F-measure, correctly identifying 79% of overfitting patches and detecting 23% more overfitting patches than the best baseline.

Patch It If You Can: Increasing the Efficiency of Patch Generation Using Context

Although program repair is a tremendous aspect of a software system, it can be extremely challenging. An Automated Program Repair (APR) technique has been proposed to solve this problem. Among them, template-based APR shows good performance. One of the key properties of the template-based APR technique for practical use is its efficiency. However, because the existing techniques mainly focus on performance improvement, they do not sufficiently consider the efficiency. In this study, we propose EffiGenC, which efficiently explores the patch ingredient search space to improve the overall efficiency of the template-based APR. EffiGenC defines the context using the concept of extended reaching definition from compiler theory. EffiGenC constructs the search space by collecting the ingredient required for patching in the context. We evaluated EffiGenC on the Defects4j benchmark. EffiGenC decreases the number of candidate patches from 27% to 86% compared to existing techniques. EffiGenC also correctly/plausibly fixes 47/72 bugs. For Future work, we will solve the search space problem that exists in multiline bugs using context.

Automatic Repair of Java Programs Weighted Fusion Similarity via Genetic Programming

Recently, automated program repair techniques have been proven to be useful in the process of software development. However, how to reduce the large search space and the random of ingredient selection is still a challenging problem. In this paper, we propose a repair approach for buggy program based on weighted fusion similarity and genetic programming. Firstly, the list of modification points is generated by selecting modification points from the suspicious statements. Secondly, the buggy repair ingredient is selected according to the value of the weighted fusion similarity, and the repair ingredient is applied to the corresponding modification points according to the selected operator. Finally, we use the test case execution information to prioritize the test cases to improve individual verification efficiency. We have implemented our approach as a tool called WSGRepair. We evaluate WSGRepair in Defects4J and compare with other program repair techniques. Experimental results show that our approach improve the success rate of buggy program repair by 28.6%, 64%, 29%, 64% and 112% compared with the GenProg, CapGen, SimFix, jKali and jMutRepair.

Crex: Predicting patch correctness in automated repair of C programs through transfer learning of execution semantics

A significant body of automated program repair literature relies on test suites to assess the validity of generated patches. Because such oracles are weak, state-of-the-art repair tools can validate some patches that overfit the test cases but are actually incorrect. This situation has become a prime concern in APR, hindering its adoption by the industry. This work investigates execution semantic features based on micro-traces, a form of under-constrained dynamic traces. We build on transfer learning to explore function code representations that are amenable to semantic similarity computation and can therefore be leveraged for classifying patch correctness. Our Crex prototype implementation is based on the Trex framework. Experimental results on patches generated by the CoCoNut APR tool on CodeFlaws programs indicate that our approach can yield high accuracy in predicting patch correctness. The learned embeddings were proven to capture semantic similarities between functions, which was instrumental in training a classifier that identifies patch correctness by learning to discriminate between correctly patched code and incorrectly patched code based on their semantic similarity with the buggy function.

Mapping the structure and evolution of software testing research over the past three decades

Background:The field of software testing is growing and rapidly-evolving. Aims:Based on keywords assigned to publications, we seek to identify predominant research topics and understand how they are connected and have evolved. Methods:We apply co-word analysis to map the topology of testing research as a network where author-assigned keywords are connected by edges indicating co-occurrence in publications. Keywords are clustered based on edge density and frequency of connection. We examine the most popular keywords, summarize clusters into high-level research topics examine how topics connect, and examine how the field is changing. Results:Testing research can be divided into 16 high-level topics and 18 subtopics. Creation guidance, automated test generation, evolution and maintenance, and test oracles have particularly strong connections to other topics, highlighting their multidisciplinary nature. Emerging keywords relate to web and mobile apps, machine learning, energy consumption, automated program repair and test generation, while emerging connections have formed between web apps, test oracles, and machine learning with many topics. Random and requirements-based testing show potential decline. Conclusions:Our observations, advice, and map data offer a deeper understanding of the field and inspiration regarding challenges and connections to explore.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Predicting Patch Correctness Based on the Similarity of Failing Test Cases

How do we know a generated patch is correct? This is a key challenging question that automated program repair (APR) systems struggle to address given the incompleteness of available test suites. Our intuition is that we can triage correct patches by checking whether each generated patch implements code changes (i.e., behavior) that are relevant to the bug it addresses. Such a bug is commonly specified by a failing test case. Towards predicting patch correctness in APR, we propose a novel yet simple hypothesis on how the link between the patch behavior and failing test specifications can be drawn: similar failing test cases should require similar patches . We then propose BATS , an unsupervised learning-based approach to predict patch correctness by checking patch B ehavior A gainst failing T est S pecification. BATS exploits deep representation learning models for code and patches: For a given failing test case, the yielded embedding is used to compute similarity metrics in the search for historical similar test cases to identify the associated applied patches, which are then used as a proxy for assessing the correctness of the APR-generated patches. Experimentally, we first validate our hypothesis by assessing whether ground-truth developer patches cluster together in the same way that their associated failing test cases are clustered. Then, after collecting a large dataset of 1,278 plausible patches (written by developers or generated by 32 APR tools), we use BATS to predict correct patches: BATS achieves AUC between 0.557 to 0.718 and recall between 0.562 and 0.854 in identifying correct patches. Our approach outperforms state-of-the-art techniques for identifying correct patches without the need for large labeled patch datasets—as is the case with machine learning-based approaches. While BATS is constrained by the availability of similar test cases, we show that it can still be complementary to existing approaches: When combined with a recent approach that relies on supervised learning, BATS improves the overall recall in detecting correct patches. We finally show that BATS is complementary to the state-of-the-art PATCH-SIM dynamic approach for identifying correct patches generated by APR tools.

Quality Evaluation Method of Automatic Software Repair Using Syntax Distance Metrics

In recent years, test-based automatic program repair has attracted widespread attention. However, the test suites in practice are not perfect ways to guarantee the correctness of patches generated by repair tools, and weak test suites lead to a large number of incorrect patches produced by the existing repair tool. To reduce the number of incorrect patches generated by repair tools, we propose a patch quality evaluation method based on syntax distance metrics, which measures the syntax distance of patches through four evaluation features—variable, expression, structure, and repair location. By fusing the distance values of the four features, the quality of a patch can be evaluated. Our method evaluates 368 patches from multiple famous repair tools, such as jKali, Nopol, SimFix, DynaMoth, and CapGen; 95% of the correct patches were ranked in the top one of the plausible patches for each defect, which indicates our method can find high-quality patches.

Automatic Repair Method for Null Pointer Dereferences Guided by Program Dependency Graph

Automatic program repair (APR) is an effective technique for eliminating defects. The repair of null pointer dereferences, as the most common defects, requires accurate dependencies among statements to determine where to repair and how to repair. In order to precisely identify the data and control dependencies, the program dependency graph is adopted. Based on the symmetry among a large number of patches, we propose four repair mechanisms in this passage, namely the assignment mechanism, restraint mechanism, evading mechanism, and transfer mechanism, and employ the decision tree algorithm to match the best repair mechanism for defects. The four repair mechanisms locate the accurate repair position using the program dependency graph, and generate candidate patches by reassigning the null pointer with an appropriate value, making a judgment for the null value in advance, or throwing an exception. Our method was implemented in the repair tool DTSFix, which supports the automatic repair of null pointer dereference in Java programs. The experimental result on Defects4J shows that 73% of null pointer dereferences are successfully repaired by DTSFix, and that the generated candidate patches do not contain over-fitting patches.

Speeding up constraint-based program repair using a search-based technique

Constraint-based program repair has been developed as one of the main techniques for automated program repair. Given a buggy program and a test suite, constraint-based program repair first extracts a repair constraint φ, and then synthesizes a patch satisfying φ. Since a patch is synthesized in a correct-by-construction manner (rather than compiling and testing each repair candidate source code), the constraint-based approach, in theory, requires less runtime overhead than the G&V approach. Nevertheless, the performance of existing constraint-based approaches is still suboptimal.

Bug-Transformer: Automated Program Repair Using Attention-Based Deep Neural Network

In this paper, we propose a novel transformer-based deep neural network model to learn semantic bug patterns from a corpus of buggy/fixed codes, then generate correct ones automatically. Transformer is a deep learning model relying entirely on attention mechanism to model global dependencies between input and output. Although there are a few endeavors to repair programs by learning neural language models (NLM), many special program properties, such as structure and semantics of an identifier, are not considered in embedding input sequence and designing model effectively, which results in undesired performance. In the proposed Bug-Transformer, we design a novel context abstraction mechanism to better support neural language models. Specifically, it is capable of 1) compressing code information but preserving the key structure and semantics, which provides more thorough information for NLM models, 2) renaming identifiers and literals based on their lexical scopes, structural and semantic information, to reduce code vocabulary size and 3) reserving keywords and selected idioms (domain- or developer-specific vocabularies) for better understanding code structure and semantics. Hence, Bug-Transformer adequately embeds code structural and semantic information into input data and optimize attention-based transformer neural network to well handle code features in order to improve learning tasks for bug repair. We evaluate the performance of the proposed work comprehensively on three datasets (Java code corpora) and generate patches to buggy code using a beam search decoder. The experimental results show that our proposed work outperforms the-state-of-art techniques: Bug-Transformer can successfully predict 54.81%, 34.45%, and 42.40% of the fixed code in these three datasets, respectively, which outperform the baseline models. These success rates steadily increase along with the increase of beam size. Besides, the overall syntactic correctness of all patches remains above 97%, 96%, and 50% on the three benchmarks, respectively, regardless of the beam size.