Maintenance of Android Widget-Based GUI Testing: A Taxonomy of Test Case Modification Causes

Abstract

Automated GUI testing is a fundamental part of the V&V process of every software, but - especially for fast evolving applications - is linked to relevant maintenance costs. Android (and mobile, in general) apps are also particularly prone to fragility, i.e. even minor modifications in the graphical user interface may induce relevant adaptation effort on test code. Based on a preliminary exploration about the evolution and fragility of test suites that we made on a set of nearly 20 thousand Android open-source projects hosted on GitHub, we performed a deeper inspection about the causes underlying the modifications in test classes and methods. We manually examined a set of 423 projects featuring the Espresso automated GUI testing tool, and 945 diff files containing modifications in test methods. We derived a set of 27 different causes of modifications, that we grouped into nine different macro-categories, two related to modifications performed just in code and application logic, and seven pertaining modifications in the GUI functioning and appearance, and hence related to our definition of GUI testing fragility. Based on the set of projects we inspected, we conclude that modifications related to changes in the GUI occured in 55% of modified test methods, a higher frequency than the ones related to changes in test logic only (35%) and to adaptations to modified application logic (27%). The described taxonomy can serve as a basis for further research, aimed at giving guidelines to developers to avoid fragility, and reduce the maintenance costs for automated test suites for Android applications.