Graphical User Interface Testing Research Articles

Robotic Process Automation Efficiency for Mobile App Testing: An Empirical Investigation

In today’s rapidly evolving software environment, the graphical user interface (GUI) plays a crucial role in providing intuitive, user-friendly interaction. However, traditional intrusive GUI testing methods often face challenges such as interrupting user workflows, requiring significant manual effort and insufficient test scenario coverage. Non-intrusive testing methods, such as Robotic Process Automation (RPA), offer a solution to validate GUI functionality without modifying the application’s code or affecting the user experience. RPA systems automate repetitive tasks by simulating user interactions, becoming valuable tools in GUI testing. However, challenges like limited computational resources, time constraints, or restricted exploration capabilities may limit the efficiency of individual RPA agents, thus restricting coverage and effectiveness. To address this issue, this study explores the performance of a single RPA agent versus an RPA cluster under different testing conditions, using three popular testing methods: Monkey, Stoat and Q-testing. Experimental results indicate that an RPA cluster outperforms a single RPA in GUI coverage and error detection, making a significant contribution to the field of non-intrusive GUI exploration testing. The findings of this study provide directions for future research to ensure the delivery of high-quality mobile applications.

An Effective GDP-LSTM and SDQL-Based Finite State Testing of GUI

The Graphical User Interface (GUI) is the most promising factor in the Software Development Lifecycle (SDL), which allows the users to interact with the system. To ensure user-friendliness, GUI Testing (GT) is required. The traditional testing techniques attained flawed results due to having inappropriate functions. Hence, Global Decaying Probabilistic Long Short-Term Memory (GDP-LSTM) and Standard Deviation Q-Learning (SDQL)-based automatic testing for GUI are proposed as solutions. Initially, the Test Case (TC) and GUI are extracted from the historical data and are subjected to Region of Interest (ROI) analysis. Here, an appropriate ROI is analyzed by Module Coupling Slice (MCS), and it is fed into Hadoop Parallelization (HP). Now, Spectral Kernelized Gaussian Clustering (SKGC) and Non-Linear Elite Guided Optimized Ant Colony (NE-GO-AC) are used to perform mapping and reducing, respectively. Likewise, the parallelized output is utilized to construct the Document Object Model (DOM) tree. Then, the attributes are extracted and given to the GDP-LSTM classifier that effectively predicts whether GUIs are desirable or undesirable. Then, the undesirable results are inputted into a SDQL-based deviation analysis. If the deviation is low, it is assumed as an update; otherwise, it is considered as an error. The experimental analysis depicted that the proposed system attained high dominance with 98.89% accuracy in the prevailing models.

A novel code generator for graphical user interfaces

Graphical user interfaces (GUIs) are widely used in human–computer interaction, providing a convenient interface for operation. Automating the conversion of GUI design images into source code can significantly reduce the coding workload for front-end developers. Detecting elements in GUI images is a key challenge in achieving automatic GUI code generation and is crucial for tasks such as GUI automation and testing. However, current state-of-the-art methods do not fully consider the unique characteristics of GUI images and elements, and they lack the required high localization accuracy, resulting in low detection accuracy for GUI element boxes. In this paper, we propose GUICG, an automatic GUI code generator that combines deep neural networks with image processing techniques to efficiently detect GUI elements from GUI images and generate front-end code. We empirically investigate various deep learning approaches and image processing methods for GUI component detection. Based on a comprehensive understanding of their performance and characteristics, we design GUICG by fusing image processing with a deep learning-based target detection model, achieving state-of-the-art performance. GUICG outperforms existing methods in accuracy and F1 score for component detection tasks, while producing human-readable code with a logical structure. Furthermore, we conduct an ablation study to quantitatively assess the impact of each key element in GUICG.

Enhancing test reuse with GUI events deduplication and adaptive semantic matching

Developers typically employ Graphical User Interface (GUI) testing to ensure the expected behavior of applications, but they face the challenge of designing appropriate test cases with functional features. Recently, researchers have proposed several test reuse methods based on semantic matching to alleviate the burden. However, the limited text semantic information and semantic matching rules between events severely limit the existing test reuse methods. In this paper, we propose TREADROID (Test Reuse EnhAncer for anDROID applications), a framework that combines GUI events deduplication with the adaptive semantic matching strategy to enhance the usability of the reused tests. Considering the connection between widget attribute texts, we categorize attributes and measure widget similarity based on the same corresponding attributes as well as across attributes in the same group. In addition, we propose a deduplication strategy for GUI events to reduce the redundancy caused by reusing a test with unique functionality. To further bridge the semantic gap, we design a two-stage adaptive matching strategy to search for the target test with functionality closer to that of the source test. Experimental evaluation against the baseline methods on 25 applications demonstrates that: (i) the adaptive semantic matching strategy overall improves the performance of widget mapping; (ii) GUI events deduplication dramatically increases the precision of events on average, even reaching 100% for multiple tests; (iii) TREADROID can significantly reduce the manual effort of creating tests for similar applications.

Finite State GUI Testing with Test Case Prioritization Using Z-BES and GK-GRU

To deliver user-friendly experiences, modern software applications rely heavily on graphical user interfaces (GUIs). However, it is paramount to ensure the quality of these GUIs through effective testing. This paper proposes a novel “Finite state testing for GUI with test case prioritization using ZScore-Bald Eagle Search (Z-BES) and Gini Kernel-Gated recurrent unit (GK-GRU)” approach to enhance GUI testing accuracy and efficiency. First, historical project data is collected. Subsequently, by utilizing the Z-BES algorithm, test cases are prioritized, aiding in improving GUI testing. Attributes are then extracted from prioritized test cases, which contain crucial details. Additionally, a state transition diagram (STD) is generated to visualize system behavior. The state activity score (SAS) is then computed to quantify state importance using reinforcement learning (RL). Next, GUI components are identified, and their text values are extracted. Similarity scores between GUI text values and test case attributes are computed. Grounded on similarity scores and SAS, a fuzzy algorithm labels the test cases. Data representation is enhanced by word embedding using GS-BERT. Finally, the test case outcomes are predicted by the GK-GRU, validating the GUI performance. The proposed work attains 98% accuracy, precision, recall, f-measure, and sensitivity, and low FPR and FNR error rates of 14.2 and 7.5, demonstrating the reliability of the model. The proposed Z-BES requires only 5587 ms to prioritize the test cases, retaining less time complexity. Meanwhile, the GK-GRU technique requires 38945 ms to train the neurons, thus enhancing the computational efficiency of the system. In conclusion, experimental outcomes demonstrate that, compared with the prevailing approaches, the proposed technique attains superior performance.

Finite State Testing of Graphical User Interface using Genetic Algorithm

Graphical user interfaces are the key components of any software. Nowadays, the popularity of the software depends upon how easily the user can interact with the system. However, as the system becomes complex, this interaction is also complicated with many states. The testing of graphical user interfaces is an important phase of modern software. The testing of GUI is possible only by interacting with the system, which may be a time-consuming process and is generally automated based on the test suite. The test suite generation proposed in this paper is based on the genetic algorithm in which various test cases are generated heuristically. For performance validation of the proposed approach, the same has been compared with a variant of PSO, and it found that GA is slightly better in comparison to the PSO.

A systematic mapping study for graphical user interface testing on mobile apps

AbstractMobile apps with tested Graphical User Interface (GUI) tend to have higher downloads in the apps store. In recent years, few efforts were made to analyse the research community and research status of the literature for GUI testing on mobile apps, which brings an obstacle to characterise and understand this field. In this study, the authors propose a systematic mapping study to gain insights into the field. First, the authors conduct an extensive search of relevant literature over seven popular digital libraries. From 4427 candidate studies, 114 primary studies published between January 2011 and September 2022 were selected. Next, the authors analyse these primary studies from the perspectives of bibliometric and qualitative analysis. For the bibliometric analysis, first, the authors analyse the popular research topics and their relationships. Second, the authors study the authors' community. For the qualitative analysis, the authors analyse the objectives, approaches and evaluation metrics employed in these primary studies. Their investigation reports several major findings: (1) there are relatively more studies on two topics, that is, test case generation and the automated test; (2) the most productive authors tend to collaborate and often have relatively broad research interests; (3) the functionality is the main objective of GUI testing; the model‐based approach is the most widely used.

An Enhanced Testing Approach for Mobile Applications

Nowadays, there is an enormous number of mobile applications that are continuously being launched to the market. As a result of this rapid process, there is a need to increase the speed of testing process using enhanced approaches. This research aims to increase the effectiveness of the graphical user interface testing process of mobile applications. This is achieved by proposing an enhanced combinatorial-based metaheuristic approach. The proposed approach aims to maximize statement and branch coverage by applying Cuckoo search, for event selection. The approach was compared to monkey, frequency, random and greedy approaches. Experiments were conducted on different mobile applications. During the same testing time duration, the proposed approach achieved higher coverage than the other approaches. The proposed approach proved its effectiveness in mobile application testing compared to the other approaches. 

Editorial

Dear Readers,  It gives me great pleasure to announce the second regular issue of 2023. I would like to thank all the authors for their sound research and the editorial board for the extremely valuable reviews and suggestions for improvement. These contributions together with the generous support of the consortium members enable us to run our journal and maintain its quality.  Still, I would like to expand our editorial board: If you are a tenured associate professor or above with a good publication record, please apply to join our editorial board. We are also interested in high-quality proposals for special issues on new topics and emerging trends.  In this regular issue, I am very pleased to introduce 4 accepted papers involving 16 authors from 6 different countries.  In a collaboration between researchers from South Africa and the USA, Trienko Grobler, Manfred Habeck, Lynette van Zijl and Jaco Geldenhuys address improved algorithms for combinatorial generation of bordered box repetition-free words based on tree-based search space and graph-based search space. In another joint research between Algeria and France, Mehdi Rouissat, Mohammed Belkheir, Hicham Sid Ahmed Belkhira, Sofiane Boukli Hacen, Pascal Lorenz and Merahi Bouziani describe a new technique to mitigate the version number attack for IoT networks, reducing control overhead by 83% and energy consumption by 74%. Amira Samir, Huda Amin Maghawry and Nagwa Badr from Egypt aim in their research to increase the effectiveness of the graphical user interface testing process of mobile applications by proposing an enhanced combinatorial-based metaheuristic approach that has been compared with monkey, frequency, random and greedy approaches. Qusai Y. Shambour, Mosleh M. Abualhaj and Ahmad Adel Abu-Shareha from Jordan propose an effective multi-criteria recommender algorithm for personalized restaurant recommendations by exploiting users’ and items’ implicit similarities to eliminate the sparseness of rating information.   Enjoy Reading!  Cordially, 

Distributed state model inference for scriptless GUI testing

State model inference of software applications through the Graphical User Interface (GUI) is a technique that identifies GUI states and transitions, and maps them into a model. Scriptless GUI testing tools can benefit substantially from the availability of these state models, for example, to improve the exploration, or have sophisticated test oracles. However, inferring models for large systems requires a long execution time. Our goal is to improve the speed of the state model inference process. To achieve this goal, this paper presents a distributed state model inference approach with an open source scriptless GUI testing tool. Moreover, in order to be able to infer a suitable model, we design a set of strategies to deal with abstraction challenges and to distinguish GUI states and transitions in the model. To validate it, we conduct an experiment with two open source web applications that have been tested with the distributed architecture using one to six Docker containers sharing the same state model. With the obtained results, we can conclude that it is feasible to infer a model with a distributed approach and that using the distributed approach reduces the time required for inferring a state model.

Many-objective test case generation for graphical user interface applications via search-based and model-based testing

The majority of the studies that generate test cases for graphical user interface (GUI) applications are based on or address functional requirements only. In spite of the fact that interesting approaches have been proposed, they do not address functional and non-functional requirements of the GUI systems, and non-functional properties of the created test suites altogether to generate test cases. This is called a many-objective perspective where several desirable and different characteristics are considered together to generate the test cases. In this study, we show how to combine search-based (optimisation) with model-based testing to generate test cases for GUI applications taking into account the many-objective perspective. We rely on meta and hyper-heuristics and we address two particular issues (problems) considering code-driven and use case-driven GUI testing. As for the code-driven testing, we target desktop applications and automatically read the C++ source code of the system, translate it into an event flow graph (EFG), and use objective functions that are graph-based measures. As for the use case-driven testing, EFGs are created directly via use cases. A rigorous evaluation was performed using 32 problem instances where we considered three multi-objective evolutionary algorithms and six selection hyper-heuristics using those algorithms as low-level (meta)heuristics. The performance of the algorithms was compared based on five different indicators, and also a new Multi-Metric Indicator (MMI) utilising multiple indicators and providing a unique measure for all algorithms. Results show that the metaheuristics obtained better performances overall, particularly NSGA-II, while Choice Function was the most outstanding hyper-heuristic approach.

Fast Witness Generation for Readable GUI Test Scenarios via Generalized Experience Replay

Verifying the functional behavior of graphical user interface (GUI) applications is essential for reducing post-release issues. In practice, a developer/tester performs this verification by executing a sequence of GUI actions and then witnessing the expected behavior on the GUI screen. An automated witness generator facilitates the verification process. However, creating an unambiguous and monitorable specification for the correct behavior and then generating the correct GUI actions to trigger that behavior is challenging. In this study, we propose FARLEAD2, an automated witness generator that uses unambiguous, monitorable, and easy-to-read staged test scenarios (STSs) to specify expected behavior. FARLEAD2 maximizes its effectiveness and performance using generalized experienced replay (GER) to exploit the experience gathered from previously witnessed scenarios on new, unwitnessed test scenarios. To the best of our knowledge, STS and GER are novel improvements to GUI testing. Our evaluation of Android GUI applications shows that FARLEAD2 effectively generates a witness 95.7 times out of 100 and does it in 520 seconds, on average, indicating that FARLEAD2 is approximately 65 percent faster and 6.3 percent more effective than its best predecessor.

Comparison of Graphical User Interface Testing Tools

Graphical User Interface or better known as the user interface is the liaison of users with electronic devices such as computers. The Graphical User Interface uses icons, menus, and some other visual indicators to represent the information contained in the interface of the application being used. The Graphical User Interface I must pass the Graphical User Interface Testing stage to ensure that every element in the Graphical User Interface is not an error and by the specified one. Also, we know that Graphical User Interface Testing is a set of activities that aim to test the Graphical User Interface I of the test object to ensure that the Graphical User Interface complies with the specifications specified in the software design document. In this research, we try to compare four Graphical User Interface testing tools which ae: Robotium, Espresso, UI Automator, and Pix2Code. By exploring these 4 testing tools we find out that Pix2code can only identify objects, especially label objects. Pix2code can only meet 3 out of 7 predefined criteria. This indicates that there are still many objects of the android application that Pix2code has not been able to identify. In other words in the Graphical User Interface testing section, pix2code can play a role in identifying each object contained in the application and can be done at the design stage. The result that we get from this research is that the GUI testing tools could identify many parts and almost every object in the application except the Pix2code. For future development, Pix2code as a testing tool requires development in the form of a desktop display such as the UI Automatorviewer so that it can display every detail of the object including the attributes of the object.

Why many challenges with GUI test automation (will) remain

Automated testing is ubiquitous in modern software development and used to verify requirement conformance on all levels of system abstraction, including the system’s graphical user interface (GUI). GUI-based test automation, like other automation, aims to reduce the cost and time for testing compared to alternative, manual approaches. Automation has been successful in reducing costs for other forms of testing (like unit- or integration testing) in industrial practice. However, we have not yet seen the same convincing results for automated GUI-based testing, which has instead been associated with multiple technical challenges. Furthermore, the software industry has struggled with some of these challenges for more than a decade with what seems like only limited progress. This systematic literature review takes a longitudinal perspective on GUI test automation challenges by identifying them and then investigating why the field has been unable to mitigate them for so many years. The review is based on a final set of 49 publications, all reporting empirical evidence from practice or industrial studies. Statements from the publications are synthesized, based on a thematic coding, into 24 challenges related to GUI test automation. The most reported challenges were mapped chronologically and further analyzed to determine how they and their proposed solutions have evolved over time. This chronological mapping of reported challenges shows that four of them have existed for almost two decades. Based on the analysis, we discuss why the key challenges with GUI-based test automation are still present and why some will likely remain in the future. For others, we discuss possible ways of how the challenges can be addressed. Further research should focus on finding solutions to the identified technical challenges with GUI-based test automation that can be resolved or mitigated. However, in parallel, we also need to acknowledge and try to overcome non-technical challenges.

Testar – scriptless testing through graphical user interface

SummaryCovering all the possible paths of the graphical user interface (GUI) with test scripts would take too much effort and result in serious maintenance issues. We propose complementing scripted testing with scriptless test automation using the open‐source testar tool. This paper gives a comprehensive overview of testar and its latest extensions together with the ongoing and future research. With this paper, we hope we can help and encourage other researchers to use testar for their GUI testing‐related research and pave the way for an international research agenda in GUI testing built upon stable and open‐source infrastructure.

DroidbotX: Test Case Generation Tool for Android Applications Using Q-Learning

Android applications provide benefits to mobile phone users by offering operative functionalities and interactive user interfaces. However, application crashes give users an unsatisfactory experience, and negatively impact the application’s overall rating. Android application crashes can be avoided through intensive and extensive testing. In the related literature, the graphical user interface (GUI) test generation tools focus on generating tests and exploring application functions using different approaches. Such tools must choose not only which user interface element to interact with, but also which type of action to be performed, in order to increase the percentage of code coverage and to detect faults with a limited time budget. However, a common limitation in the tools is the low code coverage because of their inability to find the right combination of actions that can drive the application into new and important states. A Q-Learning-based test coverage approach developed in DroidbotX was proposed to generate GUI test cases for Android applications to maximize instruction coverage, method coverage, and activity coverage. The overall performance of the proposed solution was compared to five state-of-the-art test generation tools on 30 Android applications. The DroidbotX test coverage approach achieved 51.5% accuracy for instruction coverage, 57% for method coverage, and 86.5% for activity coverage. It triggered 18 crashes within the time limit and shortest event sequence length compared to the other tools. The results demonstrated that the adaptation of Q-Learning with upper confidence bound (UCB) exploration outperforms other existing state-of-the-art solutions.

A survey on software test automation return on investment, in organizations predominantly from Bengaluru, India

Software industry has adopted automated testing widely. The most common method adopted is graphical user interface test automation for the functional scenarios to reduce manual testing and increase the repeatability. Reducing execution time and redundancy to achieve quality “go/no go decisions” provides rational for the executive management to allocate funds to adopt automation and invest in the setup including people, process, and tools to achieve faster time to market. There are a variety of practices engaged by testers, like frameworks, tools, methods, procedures, models, and technologies to achieve automation. Nonetheless, the actual effectiveness in terms of return on investment (ROI) is not known, though there are various formulas to calculate ROI of test automation. The factors that determine the ROI are maturity of test automation, purpose, or intent, picking right tests for automation, knowledge of the tester to derive test coverage, domain expertise, defining right metrics like defects found by automation runs, cost versus reduction of time, labor versus quality of scripts, repeatability, and results. These factors form the base of the questions designed for the survey. The paper presents a survey and analysis to understand the ROI of test automation from industry test professionals from both product and services organizations.

An Empirical Analysis of Test Input Generation Tools for Android Apps through a Sequence of Events

Graphical User Interface (GUI) testing of Android apps has gained considerable interest from the industries and research community due to its excellent capability to verify the operational requirements of GUI components. To date, most of the existing GUI testing tools for Android apps are capable of generating test inputs by using different approaches and improve the Android apps’ code coverage and fault detection performance. Many previous studies have evaluated the code coverage and crash detection performances of GUI testing tools in the literature. However, very few studies have investigated the effectiveness of the test input generation tools, especially in the events sequence length of the overall test coverage and crash detection. The event sequence length generally shows the number of steps required by the test input generation tools to detect a crash. It is critical to highlight its effectiveness due to its significant effects on time, testing effort, and computational cost. Thus, this study evaluated the effectiveness of six test input generation tools for Android apps that support the system events generation on 50 Android apps. The generation tools were evaluated and compared based on the activity coverage, method coverage, and capability in detecting crashes. Through a critical analysis of the results, this study identifies the diversity and similarity of test input generation tools for Android apps to provide a clear picture of the current state of the art. The results revealed that a long events sequence performed better than a shorter events sequence. However, a long events sequence led to a minor positive effect on the coverage and crash detection. Moreover, the study showed that the tools achieved less than 40% of the method coverage and 67% of the activity coverage.

Comparing the effectiveness of capture and replay against automatic input generation for Android graphical user interface testing

SummaryExploratory testing and fully automated testing tools represent two viable and cheap alternatives to traditional test‐case‐based approaches for graphical user interface (GUI) testing of Android apps. The former can be executed by capture and replay tools that directly translate execution scenarios registered by testers in test cases, without requiring preliminary test‐case design and advanced programming/testing skills. The latter tools are able to test Android GUIs without tester intervention. Even if these two strategies are widely employed, to the best of our knowledge, no empirical investigation has been performed to compare their performance and obtain useful insights for a project manager to establish an effective testing strategy. In this paper, we present two experiments we carried out to compare the effectiveness of exploratory testing approaches using a capture and replay tool (Robotium Recorder) against three freely available automatic testing tools (AndroidRipper, Sapienz, and Google Robo). The first experiment involved 20 computer engineering students who were asked to record testing executions, under strict temporal limits and no access to the source code. Results were slightly better than those of fully automated tools, but not in a conclusive way. In the second experiment, the same students were asked to improve the achieved testing coverage by exploiting the source code and the coverage obtained in the previous tests, without strict temporal constraints. The results of this second experiment showed that students outperformed the automated tools especially for long/complex execution scenarios. The obtained findings provide useful indications for deciding testing strategies that combine manual exploratory testing and automated testing.

Functional test generation from UI test scenarios using reinforcement learning for android applications

SummaryWith the ever‐growing Android graphical user interface (GUI) application market, there have been many studies on automated test generation for Android GUI applications. These studies successfully demonstrate how to detect fatal exceptions and achieve high coverage with fully automated test generation engines. However, it is unclear how many GUI functions these engines manage to test. The current best practice for the functional testing of Android GUI applications is to design user interface (UI) test scenarios with a non‐technical and human‐readable language such as Gherkin and implement Java/Kotlin methods for every statement of all the UI test scenarios. Writing tests for UI test scenarios is hard, especially when some scenario statements are high‐level and declarative, so it is not clear what actions should the generated test perform. We propose the Fully Automated Reinforcement LEArning‐Driven specification‐based test generator for Android (FARLEAD‐Android). FARLEAD‐Android first translates the UI test scenario to a GUI‐level formal specification as a linear‐time temporal logic (LTL) formula. The LTL formula guides the test generation and acts as a specified test oracle. By dynamically executing the application under test (AUT), and monitoring the LTL formula, FARLEAD‐Android learns how to produce a witness for the UI test scenario, using reinforcement learning (RL). Our evaluation shows that FARLEAD‐Android is more effective and achieves higher performance in generating tests for UI test scenarios than three known engines: Random, Monkey and QBEa. To the best of our knowledge, FARLEAD‐Android is the first fully automated mobile GUI testing engine that uses formal specifications.