Are unit and integration test definitions still valid for modern Java projects? An empirical study on open-source projects

Abstract

ContextUnit and integration testing are popular testing techniques. However, while the software development context evolved over time, the definitions remained unchanged. There is no empirical evidence, if these commonly used definitions still fit to modern software development. ObjectiveWe analyze, if the existing standard definitions of unit and integration tests are still valid in modern software development contexts. Hence, we analyze if unit and integration tests detect different types of defects, as expected from the standard literature. MethodWe classify 38,782 test cases into unit and integration tests according to the definition of the IEEE and use mutation testing to assess their defect detection capabilities. All integrated mutations are classified into five different defect types. Afterwards, we evaluate if there are any statistically significant differences in the results between unit and integration tests. ResultsWe could not find any evidence that one test type is more capable of detecting certain defect types than the other one. Our results suggest that the currently used definitions do not fit modern software development contexts. ConclusionsThis finding implies that we need to reconsider the definitions of unit and integration tests and suggest that the current property-based definitions may be exchanged with usage-based definitions.