Detecting anomalous device loads during exploratory testing of mobile applications

Abstract

Reputation and competitiveness of both mobile applications and mobile operating systems depend on their quality. Developers are using various techniques to ensure high quality. Recently, exploratory testing approaches have been gaining significant attention in this context. However, these approaches mostly do not consider one major non-functional requirement that affects quality – performance issues. Performance issues, such as sluggish UI or extensive battery consumption, are tightly connected to inefficient use of device resources. Detecting these issues is a non-trivial task. In this paper we present a novel approach to detect anomalous device resource usage and hence potential performance issues. Our approach is integrated with exploratory testing framework and uses information about previously executed test runs to build the expected resource usage model. The underlying model represents resource usage data as a multidimensional time series and is able to detect anomalous time intervals. We integrate our approach with exploratory testing tool for Android and empirically evaluate it on a set of real-world applications with injected performance issues. Our results show that suggested approach can be successfully applied to detect anomalous device resource usage and potential performance regressions.