Model-based GUI Testing For HarmonyOS Apps

Smartphone users have come to expect high-quality apps. This has increased the importance of software testing in mobile software development. Unfortunately, testing apps—particularly the GUI—can be very time-consuming. Exercising every user interface element and verifying transitions between different views of the app under test quickly becomes problematic. For example, execution of iOS GUI test suites using Apple’s UI Automation framework can take an hour or more if the app’s interface is complicated. The longer it takes to run a test, the less frequently the test can be run, which in turn reduces software quality. This book describes how to accelerate the testing process for iOS apps using HadoopUnit, a distributed test execution environment that leverages the parallelism inherent in the Hadoop platform. HadoopUnit was previously used to run unit and system tests in the cloud. It has been modified to perform GUI testing of iOS apps on a small-scale cluste —a modest computing infrastructure available to almost every developer. Experimental results have shown that distributed test execution with HadoopUnit can significantly outperform the test execution on a single machine, even if the size of the cluster used for the execution is as small as two nodes. This means that the approach described in this book could be adopted without a huge investment in IT resources. HadoopUnit is a cost-effective solution for reducing lengthy test execution times of system-level GUI testing of iOS apps.

Background. Evidence suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI test fragility, i.e. GUI test classes failing due to minor modifications in the GUI, without the application functionalities being altered.Aims. The objective of our study is to examine the diffusion of GUI testing on Android, and the amount of changes required to keep test classes up to date, and in particular the changes due to GUI test fragility. We define metrics to characterize the modifications and evolution of test classes and test methods, and proxies to estimate fragility-induced changes.Method. To perform our experiments, we selected six widely used open-source tools for scripted GUI testing of mobile applications previously described in the literature. We have mined the repositories on GitHub that used those tools, and computed our set of metrics.Results. We found that none of the considered GUI testing frameworks achieved a major diffusion among the open-source Android projects available on GitHub. For projects with GUI tests, we found that test suites have to be modified often, specifically 5%--10% of developers' modified LOCs belong to tests, and that a relevant portion (60% on average) of such modifications are induced by fragility.Conclusions. Fragility of GUI test classes constitute a relevant concern, possibly being an obstacle for developers to adopt automated scripted GUI tests. This first evaluation and measure of fragility of Android scripted GUI testing can constitute a benchmark for developers, and the basis for the definition of a taxonomy of fragility causes, and actionable guidelines to mitigate the issue.

Pervasiveness of smartphones and the vast number of corresponding apps have underlined the need for applicable automated software testing techniques. A wealth of research has been focused on either unit or GUI testing of smartphone apps, but little on automated support for end-to-end system testing. This paper presents SIG-Droid, a framework for system testing of Android apps, backed with automated program analysis to extract app models and symbolic execution of source code guided by such models for obtaining test inputs that ensure covering each reachable branch in the program. SIG-Droid leverages two automatically extracted models: Interface Model and Behavior Model. The Interface Model is used to find values that an app can receive through its interfaces. Those values are then exchanged with symbolic values to deal with constraints with the help of a symbolic execution engine. The Behavior Model is used to drive the apps for symbolic execution and generate sequences of events. We provide an efficient implementation of SIG-Droid based in part on Symbolic PathFinder, extended in this work to support automatic testing of Android apps. Our experiments show SIG-Droid is able to achieve significantly higher code coverage than existing automated testing tools targeted for Android.

Proliferation of Android devices and apps has created a demand for applicable automated software testing techniques. Prior research has primarily focused on either unit or GUI testing of Android apps, but not their end-to-end system testing in a systematic manner. We present EvoDroid, an evolutionary approach for system testing of Android apps. EvoDroid overcomes a key shortcoming of using evolutionary techniques for system testing, i.e., the inability to pass on genetic makeup of good individuals in the search. To that end, EvoDroid combines two novel techniques: (1) an Android-specific program analysis technique that identifies the segments of the code amenable to be searched independently, and (2) an evolutionary algorithm that given information of such segments performs a step-wise search for test cases reaching deep into the code. Our experiments have corroborated EvoDroid’s ability to achieve significantly higher code coverage than existing Android testing tools.

GUI testing is an essential part of regression testing for Android apps. For regression GUI testing to remain effective, it is important that obsolete GUI test scripts get repaired after the app has evolved. In this paper, we propose a novel approach named GUIDER to automated repair of GUI test scripts for Android apps. The key novelty of the approach lies in the utilization of both structural and visual information of widgets on app GUIs to better understand what widgets of the base version app become in the updated version. A supporting tool has been implemented for the approach. Experiments conducted on the popular messaging and social media app WeChat show that GUIDER is both effective and efficient. Repairs produced by GUIDER enabled 88.8% and 54.9% more test actions to run correctly than those produced by existing approaches to GUI test repair that rely solely on visual or structural information of app GUIs.

Evidence from empirical studies suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular, GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI testing fragility, i.e., GUI test classes failing or needing interventions because of modifications in the AUT or in its GUI arrangement and definition. The objective of our study is to examine the diffusion of test classes created with a set of popular GUI Automation Frameworks for Android apps, the amount of changes required to keep test classes up to date, and the amount of code churn in existing test suites, along with the underlying modifications in the AUT that caused such modifications. We defined 12 metrics to characterize the evolution of test classes and test methods, and a taxonomy of 28 possible causes for changes to test code. To perform our experiments, we selected six widely used open-source GUI Automation Frameworks for Android apps. We evaluated the diffusion of the tools by mining the GitHub repositories featuring them, and computed our set of metrics on the projects. Applying the Grounded Theory technique, we then manually analyzed diff files of test classes written with the selected tools, to build from the ground up a taxonomy of causes for modifications of test code. We found that none of the considered GUI automation frameworks achieved a major diffusion among open-source Android projects available on GitHub. For projects featuring tests created with the selected frameworks, we found that test suites had to be modified often – specifically, about 8% of developers’ modified LOCs belonged to test code and that a relevant portion (around 50% on average) of those modifications were induced by modifications in GUI definition and arrangement. Test code written with GUI automation fromeworks proved to need significant interventions during the lifespan of a typical Android open-source project. This can be seen as an obstacle for developers to adopt this kind of test automation. The evaluations and measurements of the maintainance needed by test code wrtitten with GUI automation frameworks, and the taxonomy of modification causes, can serve as a benchmark for developers, and the basis for the formulation of actionable guidelines and the development of automated tools to help mitigating the issue.

The rising popularity of Android and the GUI-driven nature of its apps have motivated the need for applicable automated GUI testing techniques. Although exhaustive testing of all possible combinations is the ideal upper bound in combinatorial testing, it is often infeasible, due to the combinatorial explosion of test cases. This paper presents TrimDroid, a framework for GUI testing of Android apps that uses a novel strategy to generate tests in a combinatorial, yet scalable, fashion. It is backed with automated program analysis and formally rigorous test generation engines. TrimDroid relies on program analysis to extract formal specifications. These specifications express the app’s behavior (i.e., control flow between the various app screens) as well as the GUI elements and their dependencies. The dependencies among the GUI elements comprising the app are used to reduce the number of combinations with the help of a solver. Our experiments have corroborated TrimDroid’s ability to achieve a comparable coverage as that possible under exhaustive GUI testing using significantly fewer test cases.

Traditionally, end-to-end testing of mobile apps is either performed manually or automated with test scripts. However, manual GUI testing is expensive and slow, and test scripts are fragile for GUI changes, resulting in high maintenance costs. Scriptless testing attempts to address the costs associated with GUI testing. Existing scriptless approaches for mobile testing do not seem to fit the requirements of the industry, specifically those of the ING. This study presents an extension to open source TESTAR tool to support scriptless GUI testing of Android and iOS applications. We present an initial validation of the tool on an industrial setting at the ING. From the validation, we determine that the extended TESTAR outperforms two other state-of-the-art scriptless testing tools for Android in terms of code coverage, and achieves similar performance as the scripted test automation already in use at the ING. Moreover, we see that the scriptless approach covers parts of the application under test that the existing test scripts did not cover, showing the complementarity of the approaches, providing more value for the testers.

Chapter 6 - Trends in Model-based GUI Testing

Mobile apps are indispensable for people's daily life. Complementing with\nautomated GUI testing, manual testing is the last line of defence for app\nquality. However, the repeated actions and easily missing of functionalities\nmake manual testing time-consuming and inefficient. Inspired by the game candy\ncrush with flashy candies as hint moves for players, we propose an approach\nnamed NaviDroid for navigating testers via highlighted next operations for more\neffective and efficient testing. Within NaviDroid, we construct an enriched\nstate transition graph with the triggering actions as the edges for two\ninvolved states. Based on it, we utilize the dynamic programming algorithm to\nplan the exploration path, and augment the GUI with visualized hints for\ntesters to quickly explore untested activities and avoid duplicate\nexplorations. The automated experiments demonstrate the high coverage and\nefficient path planning of NaviDroid and a user study further confirms its\nusefulness. The NaviDroid can help us develop more robust software that works\nin more mission-critical settings, not only by performing more thorough testing\nwith the same effort that has been put in before, but also by integrating these\ntechniques into different parts of development pipeline.\n

GUI testing has been an effective means of validating Android apps. Meanwhile, it still faces a strong challenge about how to explore trails, i.e., unfrequented test sequences, as defects tend to reside on these unfrequented trails. This paper introduces FSMdroid, a novel, guided approach to GUI testing of Android apps. The essential idea of FSMdroid is to (1) construct an initial stochastic model for the app under test, (2) iteratively mutate the stochastic model and derive tests. The model mutations are guided by an MCMC sampling method such that the resulting test sequences can be diverse and also achieve high code coverage during testing. We have evaluated FSMdroid on 40 real-world Android apps. Compared with the traditional model- based testing approaches, FSMdroid enhances the diversity of test sequences by 85%, but reduces the number of them by 54%. Furthermore, we uncover 7 app bugs.

The increasing popularity of Android and the GUI-driven nature of its apps have motivated the need for applicable automated GUI testing techniques. This paper presents a proposed strategy and a supporting tool for GUI testing of Android apps. The strategy employs a model-based approach to capture the event-driven nature of Android apps. It includes two phases: Modeling Phase and Test Evaluation Phase. In the modeling phase, an event sequence diagram (ESD) is created for each activity in the app under test (AUT), which depicts its events and possible transitions between them, and used to generate event sequences (test cases). In the test evaluation phase, certain event-based coverage criteria are employed to measure the adequacy of the generated test cases. The proposed tool analyses the AUT, creates an ESD for each activity, and generates event sequences. It handles the event sequences explosion problem and ensures the event sequences feasibility. For each event sequence, the tool generates a test script and a corresponding Robotium test class, adds it to the AUT and executes it. The paper also presents a case study that illustrates the use of the proposed strategy and tool for testing a simple Android app.

Generating GUI tests for complex Web applications is hard. There is lots of functionality to explore: The eBay home page, for instance, sports more than 2,000 individual GUI elements that a crawler has to trigger in order to discover the core functionality. We show how to leverage tests of other applications to guide test generation for a new application: Given a test for payments on Amazon, for instance, we can guide test generation on eBay towards payment functionality, exploiting the semantic similarity between UI elements across both applications. Evaluated on three domains, our approach allows to discover "deep" functionality in a few steps, which otherwise would require thousands to millions of crawling interactions.

This paper presents a reinforcement learning approach to automated GUI testing of Android apps. We use a test generation algorithm based on Q-learning to systematically select events and explore the GUI of an application under test without requiring a preexisting abstract model. We empirically evaluate the algorithm on eight Android applications and find that the proposed approach generates test suites that achieve between 3.31% to 18.83% better block-level code coverage than random test generation.

In regression GUI testing for Android apps, test scripts often fail due to changes to, rather than faults in, those apps. To avoid such false positives while still retaining the value of the old test scripts as much as possible, programmers need an automatic way to maintain the tests after the corresponding GUI has evolved. In this paper, we propose the CHATEM approach to automate GUI test script maintenance for Android apps. Taking as input the models for the GUIs of the base and updated version app and the original test scripts, CHATEM automatically extracts the changes between the two GUIs and generates maintenance actions for each change, which are then combined to form the maintenance actions for affected test scripts. In an experimental evaluation on 16 Android apps, CHATEM was able to automatically maintain the test scripts so that overall more than 95% of the remaining behaviors tested before are still tested, and almost 80% of the reusable test actions are retained in the result tests.