Generated Tests in the Context of Maintenance Tasks: A Series of Empirical Studies

Abstract

Maintenance tasks are often time-consuming and hard to manage. To cope with that, developers often use failing test cases to guide their maintenance efforts. Therefore, working with good test cases is essential to the success of maintenance. Automatically generated tests can save time and lead to higher code coverage. However, they often include test smells and are hard to read. Moreover, it is not clear whether generated tests can be effective when guiding maintenance tasks, nor if developers fully accept them. This paper presents a series of empirical studies that evaluates whether automatically generated tests can support developers when maintaining code. First, we ran an empirical study with 20 real developers to compare how they perform maintenance tasks with automatically generated (Evosuite or Randoop) and manually-written tests. Then, we applied a survey with 82 developers that assessed developers’ perceptions of using refactored Randoop tests. Finally, a third empirical study with 24 developers focused on evaluating the impact of refactored Randoop tests on the performance of maintenance. Our results and statistical analyses show that automatically generated tests can be a great help for identifying faults during maintenance. Developers were more accurate at identifying maintenance tasks when using Evosuite tests and equally effective to create bug fixes (manually-written, Evosuite, and Randoop). Developers preferred refactored Randoop tests to the original ones. However, the refactorings applied did not improve their performance on fault detection. On the other hand, developers were more effective to fix the faults using refactored Randoop tests.