Does Reviewer Recommendation Help Developers?

Vladimir Kovalenko,Christian Bird,Evgeny Pasynkov,Alberto Bacchelli,Nava Tintarev

doi:10.1109/tse.2018.2868367

Abstract

Selecting reviewers for code changes is a critical step for an efficient code review process. Recent studies propose automated reviewer recommendation algorithms to support developers in this task. However, the evaluation of recommendation algorithms, when done apart from their target systems and users (i.e., code review tools and change authors), leaves out important aspects: perception of recommendations, influence of recommendations on human choices, and their effect on user experience. This study is the first to evaluate a reviewer recommender in vivo . We compare historical reviewers and recommendations for over 21,000 code reviews performed with a deployed recommender in a company environment and set out to measure the influence of recommendations on users’ choices, along with other performance metrics. Having found no evidence of influence, we turn to the users of the recommender. Through interviews and a survey we find that, though perceived as relevant, reviewer recommendations rarely provide additional value for the respondents. We confirm this finding with a larger study at another company. The confirmation of this finding brings up a case for more user-centric approaches to designing and evaluating the recommenders. Finally, we investigate information needs of developers during reviewer selection and discuss promising directions for the next generation of reviewer recommendation tools. Preprint: https://doi.org/10.5281/zenodo.1404814 .

Highlights

C ODE Review, i.e., manual inspection of source code changes, has become a standard step in software engineering [1], [2]
Code review is typically performed on code changes, before these changes are put into production, and is done with dedicated tools [23]
Over the past eight years, CodeFlow — a homegrown code review tool at Microsoft — has achieved companywide adoption. As it represents a standard solution for code review at the company and offers an integrated reviewer recommendation engine, we focused on developers who use this tool for code review

Summary

Introduction

C ODE Review, i.e., manual inspection of source code changes, has become a standard step in software engineering [1], [2]. The inspection approaches have evolved over the last decades and these days developers commonly conduct change-based code reviews using dedicated tools [2], [3]. This lightweight, change- and tool-based approach to code review, commonly used in the software industry, is referred to as Modern Code Review (MCR) in literature [2], [4], [5], [6]. Code review tools provide developers with a convenient environment to read and discuss code changes. Examples of such features include code navigation (e.g., [8]), integration of static analysis results (e.g., [8], [24]), and code repository analytics (e.g., [8], [25])

Objectives

Results

Discussion

Conclusion