Test Report Generation for Android App Testing Via Heterogeneous Data Analysis

Abstract

The rising of the Android market demands higher quality assurance of Android applications (apps) to sharpen the competitive edge, and techniques for traditional software have problems adapting for mobile apps. Android apps often require testing on a large-scale device cluster, which produces a large amount of test reports consisting of heterogeneous data, e.g., hardware information, GUI screenshots, runtime logs. Such data are hard to merge to be unified analyzed, while they serve as an essential basis for bug inspection and fixing. Existing test report generation or analysis techniques can only handle testing data from different devices separately. They simply list all the information to app developers and have no further processing to summarize test reports. Besides, they neglect the inner connection of the heterogeneous data. Such techniques cannot improve the report reviewing effectiveness and efficiency, and they can hardly find the inner links and rules of the bug occurrence on different devices. As a result, developers still need to devote many efforts to inspect and fix bugs. In this paper, a large amount of test reports are investigated by the authors, as to construct a structured bug model to analyze heterogeneous data of the testing results. According to the investigation, we also define the <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Bug Inconsistency</small> of testing results from multiple devices and build a novel bug taxonomy. In general, an automated approach is proposed to generate structured and comprehensible test reports from raw testing results from multiple devices. Based on the approach, a tool, namely <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BreGat</small> , is implemented to evaluate the classification and deduplication capability of our approach. The experimental results of 30 Android apps on 20 devices show that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BreGat</small> can successfully cover 83% bug categories and exclude 76% duplicate bugs. Furthermore, a user study involving 16 developers shows that our test reports are more comprehensible and <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BreGat</small> greatly improves the bug inspection efficiency compared to the state-of-the-art tool.