Graphical User Interface Test Case Research Articles

Extraction and empirical evaluation of GUI-level invariants as GUI Oracles in mobile app testing

ContextMobile apps (software) are used in almost all aspects of daily life by billions of people. Given the widespread use of mobile apps in various domains, the demand for systematic testing of their Graphical User Interfaces (GUI) is crucial. Despite the significant advances in automated mobile app testing over the last decade, certain challenges remain, most notably the app-specific GUI test-oracle problem, which can significantly hinder the effective detection of defects in mobile apps. In this study, we introduce the use of GUI-level invariants, referred to as GUI invariants, as app-specific GUI oracles in GUI test cases to address this challenge. MethodsWe propose a semi-automatic solution to extract GUI invariants and use them as app-specific GUI oracles in test cases. We use the mutation testing technique to evaluate the (fault detection) effectiveness of the GUI oracles used. In addition, we evaluate their quality aspects, namely correctness, understandability, and compatibility, from the perspective of human experts using a questionnaire survey. ResultsThe empirical results show that the GUI oracles used are effective and helpful, as they improved the fault-detection effectiveness of the empirical test suites ranging from 18% to 32%. These results also highlight the efficacy of GUI oracles used in identifying various defects, including crashing and non-crashing functional issues, and surpassing the performance of existing tools in fault-detection rates. Additionally, the questionnaire survey outcomes indicate that the GUI oracles used are correct, understandable, and compatible. ConclusionsBased on the empirical results, we can conclude that using GUI invariants as GUI oracles can be useful and effective in mobile app testing.

Finite State GUI Testing with Test Case Prioritization Using Z-BES and GK-GRU

To deliver user-friendly experiences, modern software applications rely heavily on graphical user interfaces (GUIs). However, it is paramount to ensure the quality of these GUIs through effective testing. This paper proposes a novel “Finite state testing for GUI with test case prioritization using ZScore-Bald Eagle Search (Z-BES) and Gini Kernel-Gated recurrent unit (GK-GRU)” approach to enhance GUI testing accuracy and efficiency. First, historical project data is collected. Subsequently, by utilizing the Z-BES algorithm, test cases are prioritized, aiding in improving GUI testing. Attributes are then extracted from prioritized test cases, which contain crucial details. Additionally, a state transition diagram (STD) is generated to visualize system behavior. The state activity score (SAS) is then computed to quantify state importance using reinforcement learning (RL). Next, GUI components are identified, and their text values are extracted. Similarity scores between GUI text values and test case attributes are computed. Grounded on similarity scores and SAS, a fuzzy algorithm labels the test cases. Data representation is enhanced by word embedding using GS-BERT. Finally, the test case outcomes are predicted by the GK-GRU, validating the GUI performance. The proposed work attains 98% accuracy, precision, recall, f-measure, and sensitivity, and low FPR and FNR error rates of 14.2 and 7.5, demonstrating the reliability of the model. The proposed Z-BES requires only 5587 ms to prioritize the test cases, retaining less time complexity. Meanwhile, the GK-GRU technique requires 38945 ms to train the neurons, thus enhancing the computational efficiency of the system. In conclusion, experimental outcomes demonstrate that, compared with the prevailing approaches, the proposed technique attains superior performance.

DroidbotX: Test Case Generation Tool for Android Applications Using Q-Learning

Android applications provide benefits to mobile phone users by offering operative functionalities and interactive user interfaces. However, application crashes give users an unsatisfactory experience, and negatively impact the application’s overall rating. Android application crashes can be avoided through intensive and extensive testing. In the related literature, the graphical user interface (GUI) test generation tools focus on generating tests and exploring application functions using different approaches. Such tools must choose not only which user interface element to interact with, but also which type of action to be performed, in order to increase the percentage of code coverage and to detect faults with a limited time budget. However, a common limitation in the tools is the low code coverage because of their inability to find the right combination of actions that can drive the application into new and important states. A Q-Learning-based test coverage approach developed in DroidbotX was proposed to generate GUI test cases for Android applications to maximize instruction coverage, method coverage, and activity coverage. The overall performance of the proposed solution was compared to five state-of-the-art test generation tools on 30 Android applications. The DroidbotX test coverage approach achieved 51.5% accuracy for instruction coverage, 57% for method coverage, and 86.5% for activity coverage. It triggered 18 crashes within the time limit and shortest event sequence length compared to the other tools. The results demonstrated that the adaptation of Q-Learning with upper confidence bound (UCB) exploration outperforms other existing state-of-the-art solutions.

GUI Test Case Prioritization using Social Network Analysis

Graphical User Interfaces (GUIs) has become popular in today software products due to its user-friendly merit. However, GUI tests are costly and suffer explosive test cases. Moreover, software evolution causes the GUIs inevitably changed resulting in harder testing. Regression test has been widely used for GUIs software testing. It is considered to be the most expensive phase in the software testing process, though. Among several techniques, test case prioritization would help identify the reduced test cases in the regression testing suite, still maintain the same criteria as the original number of test cases in order to solve the limitation of time and cost in this phase. In this paper, we propose using network centrality from social network analysis to prioritize the test cases and compare each parameter to find a suitable parameter for GUI test case prioritization.

Towards behavior-driven graphical user interface testing

The majority of users interacts with an application through its graphical user interface (GUI). To ensure high quality and expected behavior, those graphical user interfaces have to be tested thoroughly. Yet, creating graphical user interface test cases is considered expensive in comparison to unit or integration tests. In addition, test cases are perceived to be expensive to run and brittle, therefore causing a lot of false negative test results. Behavior-driven test case design addresses this challenges by bringing requirement specifications and test cases closer together. Although industry-proven tools map test specifications automatically, test methods making test scripts executable need to be implemented manually. The specification language Slang introduced by this paper generates automatically executable test cases from BDD-like feature descriptions that integrate low-fidelity prototypes in form of wireframesketcher models. To quantify the economic advantage of our approach an AB/BA crossover designed experiment was conducted. The experiment showed that creating automatically executable test cases utilizing Slang takes 63% less time compared to the industry-proven tool JBehave. In addition to presenting the experiment's results, the paper elaborates on first experience from applying the approach in a large Swiss bank. The findings of our experiments are supported by results from applying our approach in real-world scenarios. In addition, experiment as well as case study participants appreciated the sophisticated editor support of Slang.

Event-Coverage and Weight based Method for Test Suite Prioritization

There are many challenges in testing of Graphical User Interface (GUI) applications due to its event driven nature and infinite input domain. Testing each and every possible combination of input require creating number of test cases to satisfy the adequacy criteria of GUI testing. It is not possible to test each and every test case within specified time frame. Therefore it is important to assign higher priority to test cases which have higher fault revealing capability than other test cases. Various methods are specified in literature for test suite prioritization of GUI based software and some of them are based on interaction coverage and weight of events. Weight based methods are defined namely fault prone weight based method, random weight based method and equal weight based method in which fault prone based method is most effective. In this paper we have proposed Event-Coverage and Weight based Method (EC-WBM) which prioritizes GUI test cases according to their event coverage and weight value. Weight value will be assigned based on unique event coverage and fault revealing capability of events. Event coverage based method is used to evaluate the adequacy of test cases. EC-WBM is evaluated for 2 applications one is Notepad and another is Calculator. Fault seeding method is used to create number of versions of application and these faults are evaluated using APFD (Average percentage of fault detection). APFD for prioritized test cases of Notepad is 98% and APFD for non-prioritized test cases is 62%.

Apply ant colony to event-flow model for graphical user interface test case generation

Aimed at the complex and inefficient testing issues of graphical user interface (GUI) software, this study describes an automatic framework for GUI test case generation. The framework includes a reverse engineering of executive GUI to create GUI information and event-flow model. According to these information and model, an ant colony algorithm is applied to generate test cases. The study shows that by identifying valid event interactions, the generated test cases are capable of finding deep faults without infeasible problems; the configuration of different parameters for ant colony and the good design of the proposed framework provide flexible interfaces to control test suites.

Test Case PrioritizationUsing Fuzzy Logic for GUI based Software

Testing of GUI (Graphical User Interface) applications has many challenges due to its event driven nature and infinite input domain. It is very difficult for any programmer to test for each and every possible input. When test cases are generated using automated testing tool it uses each and every possible combination to generate test cases hence generates numerous number of test case for any GUI based application. Within a defined time frame it is not possible to test every test case, that is why test cases prioritization is required. Test-case prioritization has been widely proposed and used in recent years as it can improve the rate of fault detection during the testing phase. Very few methods are defined for GUI Test case prioritization that usually consider single criteria for assigning priority for the test case which is not sufficient for the consideration of that test case as more fault revealing. In this paper we have proposed a method for assigning weight value on the basis of multiple factors as one of the criteria for test case prioritization for GUI based software. These factors are: The type of event, Event Interaction, and Parameter-value interaction coverage-based criteria. In the proposed approach priority is assigned based upon these factors using fuzzy logic model. Experimental results indicate that the proposed model is suitable for prioritizing the test cases of GUI based software.

Using ontology to generate test cases for GUI testing

Graphical User Interface (GUI) testing is a knowledge-intensive process. In this paper, ontology is introduced to generate user-centric GUI test cases. First, GUI and non-GUI components are captured by reverse engineering techniques. Next, relations among GUI components are analysed and a GUI ontology is constructed by representing all analysed results. Then, test case generation rules are defined and used to simplify test cases. After that, a case study is demonstrated on a general communication application, which shows that the proposed approach is technically feasible and ontology can facilitate GUI testing by utilising knowledge of GUI systems and experience of testers.

Automatically repairing event sequence-based GUI test suites for regression testing

Although graphical user interfaces (GUIs) constitute a large part of the software being developed today and are typically created using rapid prototyping, there are no effective regression testing techniques for GUIs. The needs of GUI regression testing differ from those of traditional software. When the structure of a GUI is modified, test cases from the original GUI's suite are either reusable or unusable on the modified GUI. Because GUI test case generation is expensive, our goal is to make the unusable test cases usable, thereby helping to retain the suite's event coverage. The idea of reusing these unusable ( obsolete ) test cases has not been explored before. This article shows that a large number of test cases become unusable for GUIs. It presents a new GUI regression testing technique that first automatically determines the usable and unusable test cases from a test suite after a GUI modification, then determines the unusable test cases that can be repaired so that they can execute on the modified GUI, and finally uses repairing transformations to repair the test cases. This regression testing technique along with four repairing transformations has been implemented. An empirical study for four open-source applications demonstrates that (1) this approach is effective in that many of the test cases can be repaired, and is practical in terms of its time performance, (2) certain types of test cases are more prone to becoming unusable, and (3) certain types of “dominator” events, when modified, make a large number of test cases unusable.

Designing and comparing automated test oracles for GUI-based software applications

Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. This article shows that the test oracle , a mechanism that determines whether a software is executed correctly for a test case, also significantly impacts the fault detection effectiveness and cost of a test case. Graphical user interfaces (GUIs), which have become ubiquitous for interacting with today's software, have created new challenges for test oracle development. Test designers manually “assert” the expected values of specific properties of certain GUI widgets in each test case; during test execution, these assertions are used as test oracles to determine whether the GUI executed correctly. Since a test case for a GUI is a sequence of events, a test designer must decide: (1) what to assert; and (2) how frequently to check an assertion, for example, after each event in the test case or after the entire test case has completed execution. Variations of these two factors significantly impact the fault-detection ability and cost of a GUI test case. A technique to declaratively specify different types of automated GUI test oracles is described. Six instances of test oracles are developed and compared in an experiment on four software systems. The results show that test oracles do affect the fault detection ability of test cases in different and interesting ways: (1) Test cases significantly lose their fault detection ability when using “weak” test oracles; (2) in many cases, invoking a “thorough” oracle at the end of test case execution yields the best cost-benefit ratio; (3) certain test cases detect faults only if the oracle is invoked during a small “window of opportunity” during test execution; and (4) using thorough and frequently-executing test oracles can compensate for not having long test cases.

Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software

Software is increasingly being developed/maintained by multiple, often geographically distributed developers working concurrently. Consequently, rapid-feedback-based quality assurance mechanisms such as daily builds and smoke regression tests, which help to detect and eliminate defects early during software development and maintenance, have become important. This paper addresses a major weakness of current smoke regression testing techniques, i.e., their inability to automatically (re)test graphical user interfaces (GUIs). Several contributions are made to the area of GUI smoke testing. First, the requirements for GUI smoke testing are identified and a GUI smoke test is formally defined as a specialized sequence of events. Second, a GUI smoke regression testing process called daily automated regression tester (DART) that automates GUI smoke testing is presented. Third, the interplay between several characteristics of GUI smoke test suites including their size, fault detection ability, and test oracles is empirically studied. The results show that: 1) the entire smoke testing process is feasible in terms of execution time, storage space, and manual effort, 2) smoke tests cannot cover certain parts of the application code, 3) having comprehensive test oracles may make up for not having long smoke test cases, and 4) using certain oracles can make up for not having large smoke test suites.

Automating regression testing for evolving GUI software

AbstractWith the widespread deployment of broadband connections worldwide, software development and maintenance are increasingly being performed by multiple engineers, often working around‐the‐clock to maximize code churn rates. To ensure rapid quality assurance of such software, techniques such as ‘nightly/daily building and smoke testing’ have become widespread since they often reveal bugs early in the software development process. During these builds, a development version of the software is checked out from the source code repository tree, compiled, linked, and (re)tested with the goal of (re)validating its basic functionality. Although successful for conventional software, smoke tests are difficult to develop and automatically re‐run for software that has a graphical user interface (GUI). In this paper, we describe a framework called DART (Daily Automated Regression Tester) that addresses the needs of frequent and automated re‐testing of GUI software. The key to our success is automation: DART automates everything from structural GUI analysis, smoke‐test‐case generation, test‐oracle creation, to code instrumentation, test execution, coverage evaluation, regeneration of test cases, and their re‐execution. Together with the operating system's task scheduler, DART can execute frequently with little input from the developer/tester to re‐test the GUI software. We provide results of experiments showing the time taken and memory required for GUI analysis, test case and test oracle generation, and test execution. We empirically compare the relative costs of employing different levels of detail in the GUI test oracle. We also show the events and statements covered by the smoke test cases. Copyright © 2005 John Wiley & Sons, Ltd.

Regression testing of GUIs

Although graphical user interfaces (GUIs) constitute a large part of the software being developed today and are typically created using rapid prototyping, there are no effective regression testing techniques for GUIs. The needs of GUI regression testing differ from those of traditional software. When the structure of a GUI is modified, test cases from the original GUI are either reusable or unusable on the modified GUI. Since GUI test case generation is expensive, our goal is to make the unusable test cases usable. The idea of reusing these unusable ( a.k.a. obsolete ) test cases has not been explored before. In this paper, we show that for GUIs, the unusability of a large number of test cases is a serious problem. We present a novel GUI regression testing technique that first automatically determines the usable and unusable test cases from a test suite after a GUI modification. It then determines which of the unusable test cases can be repaired so they can execute on the modified GUI. The last step is to repair the test cases. Our technique is integrated into a GUI testing framework that, given a test case, automatically executes it on the GUI. We implemented our regression testing technique and demonstrate for two case studies that our approach is effective in that many of the test cases can be repaired, and is practical in terms of its time performance.

Automated test oracles for GUIs

Graphical User Interfaces (GUIs) are critical components of today's software. Because GUIs have different characteristics than traditional software, conventional testing techniques do not apply to GUI software. In previous work, we presented an approach to generate GUI test cases, which take the form of sequences of actions. In this paper we develop a test oracle technique to determine if a GUI behaves as expected for a given test case. Our oracle uses a formal model of a GUI, expressed as sets of objects, object properties, and actions. Given the formal model and a test case, our oracle automatically derives the expected state for every action in the test case. We represent the actual state of an executing GUI in terms of objects and their properties derived from the GUI's execution. Using the actual state acquired from an execution monitor, our oracle automatically compares the expected and actual states after each action to verify the correctness of the GUI for the test case. We implemented the oracle as a component in our GUI testing system, called Planning Assisted Tester for grapHical user interface Systems (PATHS), which is based on AI planning. We experimentally evaluated the practicality and effectiveness of our oracle technique and report on the results of experiments to test and verify the behavior of our version of the Microsoft WordPad's GUI.