Modern Code Review Research Articles

Deciphering refactoring branch dynamics in modern code review: An empirical study on Qt

Context:Modern code review is a widely employed technique in both industrial and open-source projects, serving to enhance software quality, share knowledge, and ensure compliance with coding standards and guidelines. While code review is extensively studied for its general challenges, best practices, outcomes, and socio-technical aspects, little attention has been paid to how refactoring is reviewed and what developers prioritize when reviewing refactored code in the ‘Refactor’ branch. Objective:The goal is to understand the review process for refactoring changes in the ‘Refactor’ branch and to identify what developers care about when reviewing code in this branch. Method:In this study, we present a quantitative and qualitative examination to understand the main criteria developers use to decide whether to accept or reject refactored code submissions and identify the challenges inherent in this process. Results:Analyzing 2154 refactoring and non-refactoring reviews across Qt open-source projects, we find that reviews involving refactoring from the ‘Refactor’ branch take significantly less time to resolve in terms of code review efforts. Additionally, documentation of developer intent is notably sparse within the ‘Refactor’ branch compared to other branches. Furthermore, through thematic analysis of a substantial sample of refactoring code review discussions, we construct a comprehensive taxonomy consisting of 12 refactoring review criteria. Conclusion:Our findings underscore the importance of developing precise and efficient tools and techniques to aid developers in the review process amidst refactorings.

Don’t forget to change these functions! recommending co-changed functions in modern code review

Context:Code review is effective and widely used, yet still time-consuming. Especially, in large-scale software systems, developers may forget to change other related functions that must be changed together (aka. co-changes). This may increase the number of review iterations and reviewing time, thus delaying the code review process. Based on our analysis of 66 projects from five open-source systems, we find that there are 16%–33% of code reviews where at least one function must be co-changed, but was not initially changed. Objectives:This study aims to propose an approach to recommend co-changed functions in the context of modern code review, which could reduce reviewing time and iterations and help developers identify functions that need to be changed together. Methods:We propose CoChangeFinder, a novel method that employs a Graph Neural Network (GNN) to recommend co-changed functions for newly submitted code changes. Then, we conduct a quantitative and qualitative evaluation of CoChangeFinder with 66 studied large-scale open-source software projects. Results:Our evaluation results show that our CoChangeFinder outperforms the state-of-the-art approach, achieving 3.44% to 40.45% for top-k accuracy, 2.00% to 26.07% for Recall@k, and 0.04 to 0.21 for mean average precision better than the baseline approach. In addition, our CoChangeFinder demonstrates the capacity to pinpoint the functions related to logic changes. Conclusion:Our CoChangeFinder outperforms the baseline approach (i.e., TARMAQ) in recommending co-changed functions during the code review process. Based on our findings, CoChangeFinder could help developers save their time and effort, reduce review iterations, and enhance the efficiency of the code review process.

MULTICR: Predicting Merged and Abandoned Code Changes in Modern Code Review Using Multi-Objective Search

Modern Code Review (MCR) is an essential process in software development to ensure high-quality code. However, developers often spend considerable time reviewing code changes before being merged into the main code base. Previous studies attempted to predict whether a code change was going to be merged or abandoned soon after it was submitted to improve the code review process. However, these approaches require complex cost-sensitive learning, which makes their adoption challenging since it is difficult for developers to understand the main factors behind the models’ predictions. To address this issue, we introduce in this paper, MULTICR, a multi-objective search-based approach that uses Multi-Objective Genetic Programming (MOGP) to learn early code review prediction models as IF-THEN rules. MULTICR evolves predictive models while maximizing the accuracy of both merged and abandoned classes, eliminating the need for misclassification cost estimation. To evaluate MULTICR, we conducted an empirical study on 146,612 code reviews from Eclipse, LibreOffice, and Gerrithub. The obtained results show that MULTICR outperforms existing baselines in terms of Mathew correlation coefficient (MCC) and F1 scores while learning less complex models compared to decision trees. Our experiments showed also how MULTICR allows identifying the main factors related to abandoned code reviews as well as their associated thresholds, making it a promising approach for early code review prediction with notable performance and interoperability. Additionally, we qualitatively evaluate MULTICR by conducting a user study through semi-structured interviews involving 10 practitioners from different organizations. The obtained results indicate that 90% of the participants find that MULTICR is useful and can help them to improve the code review process. Additionally, the learned IF-THEN rules of MULTICR are transparent.

Debunking the Myths of Code Velocity: The Elusive Search for Speed

We demonstrate that no empirical evidence supports long-held beliefs related to code velocity. The best-known pragmatic way to increase code velocity is to periodically nudge engineers to unblock the code review process. The industry needs to revisit the Modern Code Review process in light of the availability of various tools and techniques that analyze and prevalidate the proposed code changes. The code review bots are coming.

Automating modern code review processes with code similarity measurement

Context:Modern code review is a critical component in software development processes, as it ensures security, detects errors early and improves code quality. However, manual reviews can be time-consuming and unreliable. Automated code review can address these issues. Although deep-learning methods have been used to recommend code review comments, they are expensive to train and employ. Instead, information retrieval (IR)-based methods for automatic code review are showing promising results in efficiency, effectiveness, and flexibility. Objective:Our main objective is to determine the optimal combination of the vectorization method and similarity to measure what gives the best results in an automatic code review, thereby improving the performance of IR-based methods. Method:Specifically, we investigate different vectorization methods (Word2Vec, Doc2Vec, Code2Vec, and Transformer) that differ from previous research (TF-IDF and Bag-of-Words), and similarity measures (Cosine, Euclidean, and Manhattan) to capture the semantic similarities between code texts. We evaluate the performance of these methods using standard metrics, such as Blue, Meteor, and Rouge-L, and include the run-time of the models in our results. Results:Our results demonstrate that the Transformer model outperforms the state-of-the-art method in all standard metrics and similarity measurements, achieving a 19.1% improvement in providing exact matches and a 6.2% improvement in recommending reviews closer to human reviews. Conclusion:Our findings suggest that the Transformer model is a highly effective and efficient approach for recommending code review comments that closely resemble those written by humans, providing valuable insight for developing more efficient and effective automated code review systems.

Advancing modern code review effectiveness through human error mechanisms

Modern code reviews tend to take a lightweight process, in which the accuracy and efficiency of identifying defects rely heavily on code reviewers’ experience. The human errors of developers, as a significant cause of software defects, is a key to identifying defects. However, there is a lack of understanding of the human error mechanisms underlying defects in code. This paper proposes an innovative code review method for identifying defects by pinpointing the scenarios that developers tend to commit errors. The method was validated by two experimental studies that involved 40 participants of about 5 years’ programming experience and modest code review experience. The experiment shows that the proposed method has significantly improved True Positives and Sensitivity by about 400 %, improved Precision by approximately 200 %, and reduced around one-third of False Positives. The effects were consistent across different tasks and different code reviewers.

State‐of‐the‐practice in quality assurance in Java‐based open source software development

SummaryTo ensure the quality of software systems, software engineers can make use of a variety of quality assurance approaches, for example, software testing, modern code review, automated static analysis, and build automation. Each of these quality assurance practices have been studied in depth in isolation, but there is a clear knowledge gap when it comes to our understanding of how these approaches are being used in conjunction, or not. In our study, we broadly investigate whether and how these quality assurance approaches are being used in conjunction in the development of 1454 popular open source software projects on GitHub. Our study indicates that typically projects do not follow all quality assurance practices together with high intensity. In fact, we only observe weak correlation among some quality assurance practices. In general, our study provides a deeper understanding of how existing quality assurance approaches are currently being used in Java‐based open source software development. Besides, we specifically zoom in on the more mature projects in our dataset, and generally we observe that more mature projects are more intense in their application of the quality assurance practices, with more focus on their ASAT usage, and code reviewing, but no strong change in their CI usage.

Modern Code Reviews—Survey of Literature and Practice

Background: Modern Code Review (MCR) is a lightweight alternative to traditional code inspections. While secondary studies on MCR exist, it is u a nknown whether the research community has targeted themes that practitioners consider important. Objectives: The objectives are to provide an overview of MCR research, analyze the practitioners’ opinions on the importance of MCR research, investigate the alignment between research and practice, and propose future MCR research avenues. Method: We conducted a systematic mapping study to survey state of the art until and including 2021, employed the Q-Methodology to analyze the practitioners’ perception of the relevance of MCR research, and analyzed the primary studies’ research impact. Results: We analyzed 244 primary studies, resulting in five themes. As a result of the 1,300 survey data points, we found that the respondents are positive about research investigating the impact of MCR on product quality and MCR process properties. In contrast, they are negative about human factor– and support systems–related research. Conclusion: These results indicate a misalignment between the state of the art and the themes deemed important by most survey respondents. Researchers should focus on solutions that can improve the state of MCR practice. We provide an MCR research agenda that can potentially increase the impact of MCR research.

Learning to Predict Code Review Completion Time In Modern Code Review

Learning to Predict Code Review Completion Time In Modern Code Review

Empirical analysis of security-related code reviews in npm packages

Security issues are a major concern in software packages and their impact can be detrimental if exploited. Modern code review is a widely-used practice that project maintainers adopt to improve the quality of contributed code. Prior work has shown that code review has an important role in improving software quality, however, in-depth analyses on code review in relation to security issues are limited.Therefore, in this paper, we aim to explore the role of code review in finding and mitigating security issues. In particular, we investigate active and popular npm packages to understand what types of security issues are raised during code review, and what kind of mitigation strategies are employed by package maintainers to address them. With pull requests (PRs) being the medium of code review under study, we analyze 171 PRs with raised security issues. We find that such issues are discussed at length by package maintainers. Moreover, we find that code review is effective at identifying certain types of security concerns, e.g., Race Condition, Access Control, and ReDOS, as dealing with such issues requires in-depth knowledge of the project domain and implementation specifics. Interestingly, we also observe that some projects have automated tools integrated in the project development cycle, which enhances the identification of frequent cases of certain security issues. When analyzing how maintainers respond to the raised security issues, we find that most of the issues (55%) are frequently addressed and mitigated. In other cases, security concerns ended up not being fixed or are ignored by project maintainers. Leveraging our findings, we offer several implications for project maintainers to support the role of reviewing code in finding and fixing security concerns.

How social interactions can affect Modern Code Review

IntroductionModern Code Review (MCR) is a multistage process where developers evaluate source code written by others to enhance the software quality. Despite the numerous studies conducted on the effects of MCR on software quality, the non-technical issues in the MCR process have not been extensively studied. This study aims to investigate the social problems in the MCR process and to find possible ways to prevent them and improve the overall quality of the MCR process.MethodologyTo achieve the research objectives, we applied the grounded theory research shaped by GQM approach to collect data on the attitudes of developers from different teams toward MCR. We conducted interviews with 25 software developers from 13 companies to obtain the information necessary to investigate how social interactions affect the code reviewing process.ResultsOur findings show that interpersonal relationships within the team can have significant consequences on the MCR process. We also received a list of possible strategies to overcome these problems.DiscussionOur study provides a new perspective on the non-technical issues in the MCR process, which has not been extensively studied before. The findings of this study can help software development teams to address the social problems in the MCR process and improve the overall quality of their software products.ConclusionThis study provides valuable insights into the non-technical issues in the MCR process and the possible ways to prevent them. The findings of this study can help software development teams to improve the MCR process and the quality of their software products. Future research could explore the effectiveness of the identified strategies in addressing the social problems in the MCR process.

Quality Evaluation of Modern Code Reviews Through Intelligent Biometric Program Comprehension

Code review is an essential practice in software engineering to spot code defects in the early stages of software development. Modern code reviews (e.g., acceptance or rejection of pull requests with Git) have become less formal than classic Fagan's inspections, lightweight, and more reliant on individuals (i.e., reviewers). However, reviewers may encounter mentally demanding challenges during the code review, such as code comprehension difficulties or distractions that might affect the code review quality. This work proposes a novel approach that evaluates the quality of code reviews in terms of bug-finding effectiveness and provides the reviewers with a clear message of whether the review should be repeated, indicating the code regions that may not have been well-reviewed. The proposed approach utilizes biometric information collected from the reviewer during the review process using non-intrusive biofeedback devices (e.g., smartwatches). Biometric measures such as Heart Rate Variability (HRV) and task-evoked pupillary response are captured as a surrogate of the cognitive state of the reviewer (e.g., mental workload) and inexpensive desktop eye-trackers compatible with the software development settings. This work uses Artificial Intelligence techniques to predict the cognitive load from the extracted biomarkers and classify each code region according to a set of features. The final evaluation considers various factors such as code complexity, time of the code review, the experience level of the reviewer, and other factors. Our experimental results show the approach could predict the review quality with 87.77%±4.65 accuracy and a Spearman correlation coefficient of 0.85 (p-value < 0.001) between the predicted and the actual review performance. This evaluation validates the cognitive load measurement using electroencephalography (EEG) signals as ground truth for the HRV and pupil signals.

DeepLineDP: Towards a Deep Learning Approach for Line-Level Defect Prediction

Defect prediction is proposed to assist practitioners effectively prioritize limited Software Quality Assurance (SQA) resources on the most risky files that are likely to have post-release software defects. However, there exist two main limitations in prior studies: (1) the granularity levels of defect predictions are still coarse-grained and (2) the surrounding tokens and surrounding lines have not yet been fully utilized. In this paper, we perform a survey study to better understand how practitioners perform code inspection in modern code review process, and their perception on a line-level defect prediction. According to the responses from 36 practitioners, we found that 50% of them spent at least 10 minutes to more than one hour to review a single file, while 64% of them still perceived that code inspection activity is challenging to extremely challenging. In addition, 64% of the respondents perceived that a line-level defect prediction tool would potentially be helpful in identifying defective lines. Motivated by the practitioners’ perspective, we present DeepLineDP, a deep learning approach to automatically learn the semantic properties of the surrounding tokens and lines in order to identify defective files and defective lines. Through a case study of 32 releases of 9 software projects, we find that the risk score of code tokens varies greatly depending on their location. Our DeepLineDP is 17%-37% more accurate than other file-level defect prediction approaches; is 47%-250% more cost-effective than other line-level defect prediction approaches; and achieves a reasonable performance when transferred to other software projects. These findings confirm that the surrounding tokens and surrounding lines should be considered to identify the fine-grained locations of defective files (i.e., defective lines).

Data used in modern code review mapping study and survey

Data used in modern code review mapping study and survey

Data used in modern code review mapping study and survey

Data used in modern code review mapping study and survey

Data used in modern code review mapping study and survey

Data used in modern code review mapping study and survey

Code smells detection via modern code review: a study of the OpenStack and Qt communities

Code review plays an important role in software quality control. A typical review process involves a careful check of a piece of code in an attempt to detect and locate defects and other quality issues/violations. One type of issue that may impact the quality of software is code smells - i.e., bad coding practices that may lead to defects or maintenance issues. Yet, little is known about the extent to which code smells are identified during modern code review. To investigate the concept behind code smells identified in modern code review and what actions reviewers suggest and developers take in response to the identified smells, we conducted an empirical study of code smells in code reviews by analysing reviews from four, large open source projects from the OpenStack (Nova and Neutron) and Qt (Qt Base and Qt Creator) communities. We manually checked a total of 25,415 code review comments obtained by keywords search and random selection; this resulted in the identification of 1,539 smell-related reviews which then allowed the study of the causes of code smells, actions taken against identified smells, time taken to fix identified smells and reasons why developers ignored fixing identified smells. Our analysis found that 1) code smells were not commonly identified in code reviews, 2) smells were usually caused by violation of coding conventions, 3) reviewers usually provided constructive feedback, including fixing (refactoring) recommendations to help developers remove smells, 4) developers generally followed those recommendations and actioned the changes, 5) once identified by reviewers, it usually takes developers less than one week to fix the smells and 6) the main reason why developers chose to ignore the identified smells is that it is not worth fixing the smell. Our results suggest the following: 1) developers should closely follow coding conventions in their projects to avoid introducing code smells, 2) review-based detection of code smells is perceived to be a trustworthy approach by developers, mainly because reviews are context-sensitive (as reviewers are more aware of the context of the code given that they are part of the project’s development team) and 3) program context needs to be fully considered in order to make a decision of whether to fix the identified code smell immediately.

An empirical study on self-admitted technical debt in modern code review

Technical debt is a sub-optimal state of development in projects. In particular, the type of technical debt incurred by developers themselves (e.g., comments that mean the implementation is imperfect and should be replaced with another implementation) is called self-admitted technical debt (SATD). In theory, technical debt should not be left for a long period because it accumulates more cost over time, making it more difficult to process. Accordingly, developers have traditionally conducted code reviews to find technical debt. In fact, we observe that many SATD comments are often introduced during modern code reviews (MCR) that are light-weight reviews with web applications. However, it is uncertain about the nature of SATD comments that are introduced in the review process: impact, frequency, characteristics, and triggers. Herein, this study empirically examines the relationship between SATD and MCR.

Early prediction for merged vs abandoned code changes in modern code reviews

The modern code review process is an integral part of the current software development practice. Considerable effort is given here to inspect code changes, find defects, suggest an improvement, and address the suggestions of the reviewers. In a code review process, usually, several iterations take place where an author submits code changes and a reviewer gives feedback until is happy to accept the change. In around 12% cases, the changes are abandoned, eventually wasting all the efforts. In this research, our objective is to design a tool that can predict whether a code change would be merged or abandoned at an early stage to reduce the waste of efforts of all stakeholders (e.g., program author, reviewer, project management, etc.) involved. The real-world demand for such a tool was formally identified by a study by Fan et al. [1]. We have mined 146,612 code changes from the code reviews of three large and popular open-source software and trained and tested a suite of supervised machine learning classifiers, both shallow and deep learning based. We consider a total of 25 features in each code change during the training and testing of the models. The best performing model named PredCR (Predicting Code Review), a LightGBM-based classifier achieves around 85% AUC score on average and relatively improves the state-of-the-art [1] by 14-23%. In our empirical study on the 146,612 code changes from the three software projects, we find that (1) The new features like reviewer dimensions that are introduced in PredCR are the most informative. (2) Compared to the baseline, PredCR is more effective towards reducing bias against new developers. (3) PredCR uses historical data in the code review repository and as such the performance of PredCR improves as a software system evolves with new and more data.

The Impact of Peer Code Review on Software Maintainability in Open-Source Software: A Case Study

Recently, open-source software (OSS) has become a considerably popular and reliable source of functionality corrections. OSS also allows software developers to reduce technical debt in software development. However, previous studies have shown that the main problem within OSS development is the lack of systematic processes and formal documents related to system development, such as requirements, designs, and testing. This feature of OSS development causes problems in the software quality, such as those related to security and maintainability. In this research, the authors focused on the software’s maintainability because this attribute has to greatest potential to reduce the cost and increase the productivity of the software development process. There is currently no existing research that examines whether OSS developers pay attention to software maintainability. To better understand how OSS developers improve software maintainability, this research aims to answer the question: “Are developers interested in software maintainability under the modern code review of open-source software projects?” To answer the research question, the authors investigated the code review process in which the OSS developers changed the code based on a review of code comments related to maintenance and collected the sub-characteristics associated with software maintainability from the existing literature. The authors examined the review comments from two OSS projects: Eclipse and Qt. The results suggest that the number of code revisions due to maintenance issues was moderate and that the OSS developers tend to improve source code quality. This direction could be observed from the increasing number of modifications on given maintenance-based comments over the years. Therefore, an implication of this is the possibility that OSS project developers are interested in software maintainability.