A Large-Scale Study of Modern Code Review and Security in Open Source Projects

Abstract

Background: Evidence for the relationship between code review process and software security (and software quality) has the potential to help improve code review automation and tools, as well as provide a better understanding of the economics for improving software security and quality. Prior work in this area has primarily been limited to case studies of a small handful of software projects. Aims: We investigate the effect of modern code review on software security. We extend and generalize prior work that has looked at code review and software quality. Method: We gather a very large dataset from GitHub (3,126 projects in 143 languages, with 489,038 issues and 382,771 pull requests), and use a combination of quantification techniques and multiple regression modeling to study the relationship between code review coverage and participation and software quality and security. Results: We find that code review coverage has a significant effect on software security. We confirm prior results that found a relationship between code review coverage and software defects. Most notably, we find evidence of a negative relationship between code review of pull requests and the number of security bugs reported in a project. Conclusions: Our results suggest that implementing code review policies within the pull request model of development may have a positive effect on the quality and security of software.