On the Differences between Unit and Integration Testing in the TravisTorrent Dataset

Abstract

Already from the early days of testing, practitioners distinguish between unit tests and integration tests as a strategy to locate defects. Unfortunately, the mining software engineering community rarely distinguishes between these two strategies, mainly because it is not straightforward to separate them in the code repositories under study. In this paper we exploited the TravisTorrent dataset provided for the MSR 2017 mining challenge, separated unit tests from integration tests, and correlated these against the workflow as recorded in the corresponding issue reports. Further analysis confirmed that it is worthwhile to treat unit tests and integration tests differently: we discovered that unit tests cause more breaking builds, that fixing the defects exposed by unit tests takes longer and implies more coordination between team members.