On the Accuracy, Efficiency, and Reusability of Automated Test Oracles for Android Devices

Abstract

Automated GUI testing consists of simulating user events and validating the changes in the GUI in order to determine if an Android application meets specifications. Traditional record-replay testing tools mainly focus on facilitating the test case writing process but not the replay and verification process. The accuracy of testing tools degrades significantly when the device under test (DUT) is under heavy load. In order to improve the accuracy, our previous work, SPAG, uses event batching and smart wait function to eliminate the uncertainty of the replay process and adopts GUI layout information to verify the testing results. SPAG maintains an accuracy of up to 99.5 percent and outperforms existing methods. In this work, we propose smart phone automated GUI testing tool with camera (SPAG-C), an extension of SPAG, to test an Android hardware device. Our goal is to further reduce the time required to record test cases and increase reusability of the test oracle without compromising test accuracy. In the record stage, SPAG captures screenshots from device's frame buffer and writes verification commands into the test case. Unlike SPAG, SPAG-C captures the screenshots from an external camera instead of frame buffer. In the replay stage, SPAG-C automatically performs image comparison while SPAG simply performs a string comparison to verify the test results. In order to make SPAG-C reusable for different devices and to allow bettersynchronization at the time of capturing images, we develop a new architecture that uses an external camera and Web services to decouple the test oracle. Our experiments show that recording a test case using SPAG-C's automatic verification is as fast as SPAG's but more accurate. Moreover, SPAG-C is 50 to 75 percent faster than SPAG in achieving the same test accuracy. With reusability, SPAG-C reduces the testing time from days to hours for heterogeneous devices.