Graphical User Interface Testing Research Articles

Augmented testing to support manual GUI-based regression testing: An empirical study

ContextManual graphical user interface (GUI) software testing presents a substantial part of the overall practiced testing efforts, despite various research efforts to further increase test automation. Augmented Testing (AT), a novel approach for GUI testing, aims to aid manual GUI-based testing through a tool-supported approach where an intermediary visual layer is rendered between the system under test (SUT) and the tester, superimposing relevant test information.ObjectiveThe primary objective of this study is to gather empirical evidence regarding AT’s efficiency compared to manual GUI-based regression testing. Existing studies involving testing approaches under the AT definition primarily focus on exploratory GUI testing, leaving a gap in the context of regression testing. As a secondary objective, we investigate AT’s benefits, drawbacks, and usability issues when deployed with the demonstrator tool, Scout.MethodWe conducted an experiment involving 13 industry professionals, from six companies, comparing AT to manual GUI-based regression testing. These results were complemented by interviews and Bayesian data analysis (BDA) of the study’s quantitative results.ResultsThe results of the Bayesian data analysis revealed that the use of AT shortens test durations in 70% of the cases on average, concluding that AT is more efficient. When comparing the means of the total duration to perform all tests, AT reduced the test duration by 36% in total. Participant interviews highlighted nine benefits and eleven drawbacks of AT, while observations revealed four usability issues.ConclusionThis study presents empirical evidence of improved efficiency using AT in the context of manual GUI-based regression testing. We further report AT’s benefits, drawbacks, and usability issues. The majority of identified usability issues and drawbacks can be attributed to the tool implementation of AT and, thus, can serve as valuable input for future tool development.

Organizing Graphical User Interface tests from behavior‐driven development as videos to obtain stakeholders' feedback

AbstractDemonstrating software early and responding to feedback is crucial in agile development. However, it is difficult for stakeholders who are not on‐site customers but end users, marketing people, or designers, and so forth to give feedback in an agile development environment. Successful graphical user interface (GUI) test executions can be documented and then demonstrated for feedback. In our new concept, GUI tests from behavior‐driven development (BDD) are recorded, augmented, and demonstrated as videos. A GUI test is divided into several GUI unit tests, which are specified in Gherkin, a semi‐structured natural language. For each GUI unit test, a video is generated during test execution. Test steps specified in Gherkin are traced and highlighted in the video. Stakeholders review these generated videos and provide feedback, for example, on misunderstandings of requirements or on inconsistencies. To evaluate the impact of videos in identifying inconsistencies, we asked 22 participants to identify inconsistencies between (1) given requirements in regular sentences and (2) demonstrated behaviors from videos with Gherkin specifications or from Gherkin specifications alone. Our results show that participants tend to identify more inconsistencies from demonstrated behaviors, which are not in accordance with given requirements. They tend to recognize inconsistencies more easily through videos than through Gherkin specifications alone. The types of inconsistency are threefold: The mentioned feature can be incorrectly implemented, not implemented, or an unspecified new feature. We use a fictitious example showing how this feedback helps a product owner and her team manage requirements. We conclude that GUI test videos can help stakeholders give feedback more effectively. By obtaining early feedback, inconsistencies can be resolved, thus contributing to higher stakeholder satisfaction.

Extraction and empirical evaluation of GUI-level invariants as GUI Oracles in mobile app testing

ContextMobile apps (software) are used in almost all aspects of daily life by billions of people. Given the widespread use of mobile apps in various domains, the demand for systematic testing of their Graphical User Interfaces (GUI) is crucial. Despite the significant advances in automated mobile app testing over the last decade, certain challenges remain, most notably the app-specific GUI test-oracle problem, which can significantly hinder the effective detection of defects in mobile apps. In this study, we introduce the use of GUI-level invariants, referred to as GUI invariants, as app-specific GUI oracles in GUI test cases to address this challenge. MethodsWe propose a semi-automatic solution to extract GUI invariants and use them as app-specific GUI oracles in test cases. We use the mutation testing technique to evaluate the (fault detection) effectiveness of the GUI oracles used. In addition, we evaluate their quality aspects, namely correctness, understandability, and compatibility, from the perspective of human experts using a questionnaire survey. ResultsThe empirical results show that the GUI oracles used are effective and helpful, as they improved the fault-detection effectiveness of the empirical test suites ranging from 18% to 32%. These results also highlight the efficacy of GUI oracles used in identifying various defects, including crashing and non-crashing functional issues, and surpassing the performance of existing tools in fault-detection rates. Additionally, the questionnaire survey outcomes indicate that the GUI oracles used are correct, understandable, and compatible. ConclusionsBased on the empirical results, we can conclude that using GUI invariants as GUI oracles can be useful and effective in mobile app testing.

Probability maps for deep learning-based head and neck tumor segmentation: Graphical User Interface design and test

BackgroundThe different tumor appearance of head and neck cancer across imaging modalities, scanners, and acquisition parameters accounts for the highly subjective nature of the manual tumor segmentation task. The variability of the manual contours is one of the causes of the lack of generalizability and the suboptimal performance of deep learning (DL) based tumor auto-segmentation models. Therefore, a DL-based method was developed that outputs predicted tumor probabilities for each PET-CT voxel in the form of a probability map instead of one fixed contour. The aim of this study was to show that DL-generated probability maps for tumor segmentation are clinically relevant, intuitive, and a more suitable solution to assist radiation oncologists in gross tumor volume segmentation on PET-CT images of head and neck cancer patients. MethodA graphical user interface (GUI) was designed, and a prototype was developed to allow the user to interact with tumor probability maps. Furthermore, a user study was conducted where nine experts in tumor delineation interacted with the interface prototype and its functionality. The participants’ experience was assessed qualitatively and quantitatively. ResultsThe interviews with radiation oncologists revealed their preference for using a rainbow colormap to visualize tumor probability maps during contouring, which they found intuitive. They also appreciated the slider feature, which facilitated interaction by allowing the selection of threshold values to create single contours for editing and use as a starting point. Feedback on the prototype highlighted its excellent usability and positive integration into clinical workflows. ConclusionsThis study shows that DL-generated tumor probability maps are explainable, transparent, intuitive and a better alternative to the single output of tumor segmentation models.

Robotic Process Automation Efficiency for Mobile App Testing: An Empirical Investigation

In today’s rapidly evolving software environment, the graphical user interface (GUI) plays a crucial role in providing intuitive, user-friendly interaction. However, traditional intrusive GUI testing methods often face challenges such as interrupting user workflows, requiring significant manual effort and insufficient test scenario coverage. Non-intrusive testing methods, such as Robotic Process Automation (RPA), offer a solution to validate GUI functionality without modifying the application’s code or affecting the user experience. RPA systems automate repetitive tasks by simulating user interactions, becoming valuable tools in GUI testing. However, challenges like limited computational resources, time constraints, or restricted exploration capabilities may limit the efficiency of individual RPA agents, thus restricting coverage and effectiveness. To address this issue, this study explores the performance of a single RPA agent versus an RPA cluster under different testing conditions, using three popular testing methods: Monkey, Stoat and Q-testing. Experimental results indicate that an RPA cluster outperforms a single RPA in GUI coverage and error detection, making a significant contribution to the field of non-intrusive GUI exploration testing. The findings of this study provide directions for future research to ensure the delivery of high-quality mobile applications.

An Effective GDP-LSTM and SDQL-Based Finite State Testing of GUI

The Graphical User Interface (GUI) is the most promising factor in the Software Development Lifecycle (SDL), which allows the users to interact with the system. To ensure user-friendliness, GUI Testing (GT) is required. The traditional testing techniques attained flawed results due to having inappropriate functions. Hence, Global Decaying Probabilistic Long Short-Term Memory (GDP-LSTM) and Standard Deviation Q-Learning (SDQL)-based automatic testing for GUI are proposed as solutions. Initially, the Test Case (TC) and GUI are extracted from the historical data and are subjected to Region of Interest (ROI) analysis. Here, an appropriate ROI is analyzed by Module Coupling Slice (MCS), and it is fed into Hadoop Parallelization (HP). Now, Spectral Kernelized Gaussian Clustering (SKGC) and Non-Linear Elite Guided Optimized Ant Colony (NE-GO-AC) are used to perform mapping and reducing, respectively. Likewise, the parallelized output is utilized to construct the Document Object Model (DOM) tree. Then, the attributes are extracted and given to the GDP-LSTM classifier that effectively predicts whether GUIs are desirable or undesirable. Then, the undesirable results are inputted into a SDQL-based deviation analysis. If the deviation is low, it is assumed as an update; otherwise, it is considered as an error. The experimental analysis depicted that the proposed system attained high dominance with 98.89% accuracy in the prevailing models.

A novel code generator for graphical user interfaces

Graphical user interfaces (GUIs) are widely used in human–computer interaction, providing a convenient interface for operation. Automating the conversion of GUI design images into source code can significantly reduce the coding workload for front-end developers. Detecting elements in GUI images is a key challenge in achieving automatic GUI code generation and is crucial for tasks such as GUI automation and testing. However, current state-of-the-art methods do not fully consider the unique characteristics of GUI images and elements, and they lack the required high localization accuracy, resulting in low detection accuracy for GUI element boxes. In this paper, we propose GUICG, an automatic GUI code generator that combines deep neural networks with image processing techniques to efficiently detect GUI elements from GUI images and generate front-end code. We empirically investigate various deep learning approaches and image processing methods for GUI component detection. Based on a comprehensive understanding of their performance and characteristics, we design GUICG by fusing image processing with a deep learning-based target detection model, achieving state-of-the-art performance. GUICG outperforms existing methods in accuracy and F1 score for component detection tasks, while producing human-readable code with a logical structure. Furthermore, we conduct an ablation study to quantitatively assess the impact of each key element in GUICG.

Enhancing test reuse with GUI events deduplication and adaptive semantic matching

Developers typically employ Graphical User Interface (GUI) testing to ensure the expected behavior of applications, but they face the challenge of designing appropriate test cases with functional features. Recently, researchers have proposed several test reuse methods based on semantic matching to alleviate the burden. However, the limited text semantic information and semantic matching rules between events severely limit the existing test reuse methods. In this paper, we propose TREADROID (Test Reuse EnhAncer for anDROID applications), a framework that combines GUI events deduplication with the adaptive semantic matching strategy to enhance the usability of the reused tests. Considering the connection between widget attribute texts, we categorize attributes and measure widget similarity based on the same corresponding attributes as well as across attributes in the same group. In addition, we propose a deduplication strategy for GUI events to reduce the redundancy caused by reusing a test with unique functionality. To further bridge the semantic gap, we design a two-stage adaptive matching strategy to search for the target test with functionality closer to that of the source test. Experimental evaluation against the baseline methods on 25 applications demonstrates that: (i) the adaptive semantic matching strategy overall improves the performance of widget mapping; (ii) GUI events deduplication dramatically increases the precision of events on average, even reaching 100% for multiple tests; (iii) TREADROID can significantly reduce the manual effort of creating tests for similar applications.

Finite State GUI Testing with Test Case Prioritization Using Z-BES and GK-GRU

To deliver user-friendly experiences, modern software applications rely heavily on graphical user interfaces (GUIs). However, it is paramount to ensure the quality of these GUIs through effective testing. This paper proposes a novel “Finite state testing for GUI with test case prioritization using ZScore-Bald Eagle Search (Z-BES) and Gini Kernel-Gated recurrent unit (GK-GRU)” approach to enhance GUI testing accuracy and efficiency. First, historical project data is collected. Subsequently, by utilizing the Z-BES algorithm, test cases are prioritized, aiding in improving GUI testing. Attributes are then extracted from prioritized test cases, which contain crucial details. Additionally, a state transition diagram (STD) is generated to visualize system behavior. The state activity score (SAS) is then computed to quantify state importance using reinforcement learning (RL). Next, GUI components are identified, and their text values are extracted. Similarity scores between GUI text values and test case attributes are computed. Grounded on similarity scores and SAS, a fuzzy algorithm labels the test cases. Data representation is enhanced by word embedding using GS-BERT. Finally, the test case outcomes are predicted by the GK-GRU, validating the GUI performance. The proposed work attains 98% accuracy, precision, recall, f-measure, and sensitivity, and low FPR and FNR error rates of 14.2 and 7.5, demonstrating the reliability of the model. The proposed Z-BES requires only 5587 ms to prioritize the test cases, retaining less time complexity. Meanwhile, the GK-GRU technique requires 38945 ms to train the neurons, thus enhancing the computational efficiency of the system. In conclusion, experimental outcomes demonstrate that, compared with the prevailing approaches, the proposed technique attains superior performance.

Finite State Testing of Graphical User Interface using Genetic Algorithm

Graphical user interfaces are the key components of any software. Nowadays, the popularity of the software depends upon how easily the user can interact with the system. However, as the system becomes complex, this interaction is also complicated with many states. The testing of graphical user interfaces is an important phase of modern software. The testing of GUI is possible only by interacting with the system, which may be a time-consuming process and is generally automated based on the test suite. The test suite generation proposed in this paper is based on the genetic algorithm in which various test cases are generated heuristically. For performance validation of the proposed approach, the same has been compared with a variant of PSO, and it found that GA is slightly better in comparison to the PSO.

A systematic mapping study for graphical user interface testing on mobile apps

AbstractMobile apps with tested Graphical User Interface (GUI) tend to have higher downloads in the apps store. In recent years, few efforts were made to analyse the research community and research status of the literature for GUI testing on mobile apps, which brings an obstacle to characterise and understand this field. In this study, the authors propose a systematic mapping study to gain insights into the field. First, the authors conduct an extensive search of relevant literature over seven popular digital libraries. From 4427 candidate studies, 114 primary studies published between January 2011 and September 2022 were selected. Next, the authors analyse these primary studies from the perspectives of bibliometric and qualitative analysis. For the bibliometric analysis, first, the authors analyse the popular research topics and their relationships. Second, the authors study the authors' community. For the qualitative analysis, the authors analyse the objectives, approaches and evaluation metrics employed in these primary studies. Their investigation reports several major findings: (1) there are relatively more studies on two topics, that is, test case generation and the automated test; (2) the most productive authors tend to collaborate and often have relatively broad research interests; (3) the functionality is the main objective of GUI testing; the model‐based approach is the most widely used.

An Enhanced Testing Approach for Mobile Applications

Nowadays, there is an enormous number of mobile applications that are continuously being launched to the market. As a result of this rapid process, there is a need to increase the speed of testing process using enhanced approaches. This research aims to increase the effectiveness of the graphical user interface testing process of mobile applications. This is achieved by proposing an enhanced combinatorial-based metaheuristic approach. The proposed approach aims to maximize statement and branch coverage by applying Cuckoo search, for event selection. The approach was compared to monkey, frequency, random and greedy approaches. Experiments were conducted on different mobile applications. During the same testing time duration, the proposed approach achieved higher coverage than the other approaches. The proposed approach proved its effectiveness in mobile application testing compared to the other approaches. 

Editorial

Dear Readers,  It gives me great pleasure to announce the second regular issue of 2023. I would like to thank all the authors for their sound research and the editorial board for the extremely valuable reviews and suggestions for improvement. These contributions together with the generous support of the consortium members enable us to run our journal and maintain its quality.  Still, I would like to expand our editorial board: If you are a tenured associate professor or above with a good publication record, please apply to join our editorial board. We are also interested in high-quality proposals for special issues on new topics and emerging trends.  In this regular issue, I am very pleased to introduce 4 accepted papers involving 16 authors from 6 different countries.  In a collaboration between researchers from South Africa and the USA, Trienko Grobler, Manfred Habeck, Lynette van Zijl and Jaco Geldenhuys address improved algorithms for combinatorial generation of bordered box repetition-free words based on tree-based search space and graph-based search space. In another joint research between Algeria and France, Mehdi Rouissat, Mohammed Belkheir, Hicham Sid Ahmed Belkhira, Sofiane Boukli Hacen, Pascal Lorenz and Merahi Bouziani describe a new technique to mitigate the version number attack for IoT networks, reducing control overhead by 83% and energy consumption by 74%. Amira Samir, Huda Amin Maghawry and Nagwa Badr from Egypt aim in their research to increase the effectiveness of the graphical user interface testing process of mobile applications by proposing an enhanced combinatorial-based metaheuristic approach that has been compared with monkey, frequency, random and greedy approaches. Qusai Y. Shambour, Mosleh M. Abualhaj and Ahmad Adel Abu-Shareha from Jordan propose an effective multi-criteria recommender algorithm for personalized restaurant recommendations by exploiting users’ and items’ implicit similarities to eliminate the sparseness of rating information.   Enjoy Reading!  Cordially, 

Distributed state model inference for scriptless GUI testing

State model inference of software applications through the Graphical User Interface (GUI) is a technique that identifies GUI states and transitions, and maps them into a model. Scriptless GUI testing tools can benefit substantially from the availability of these state models, for example, to improve the exploration, or have sophisticated test oracles. However, inferring models for large systems requires a long execution time. Our goal is to improve the speed of the state model inference process. To achieve this goal, this paper presents a distributed state model inference approach with an open source scriptless GUI testing tool. Moreover, in order to be able to infer a suitable model, we design a set of strategies to deal with abstraction challenges and to distinguish GUI states and transitions in the model. To validate it, we conduct an experiment with two open source web applications that have been tested with the distributed architecture using one to six Docker containers sharing the same state model. With the obtained results, we can conclude that it is feasible to infer a model with a distributed approach and that using the distributed approach reduces the time required for inferring a state model.

Many-objective test case generation for graphical user interface applications via search-based and model-based testing

The majority of the studies that generate test cases for graphical user interface (GUI) applications are based on or address functional requirements only. In spite of the fact that interesting approaches have been proposed, they do not address functional and non-functional requirements of the GUI systems, and non-functional properties of the created test suites altogether to generate test cases. This is called a many-objective perspective where several desirable and different characteristics are considered together to generate the test cases. In this study, we show how to combine search-based (optimisation) with model-based testing to generate test cases for GUI applications taking into account the many-objective perspective. We rely on meta and hyper-heuristics and we address two particular issues (problems) considering code-driven and use case-driven GUI testing. As for the code-driven testing, we target desktop applications and automatically read the C++ source code of the system, translate it into an event flow graph (EFG), and use objective functions that are graph-based measures. As for the use case-driven testing, EFGs are created directly via use cases. A rigorous evaluation was performed using 32 problem instances where we considered three multi-objective evolutionary algorithms and six selection hyper-heuristics using those algorithms as low-level (meta)heuristics. The performance of the algorithms was compared based on five different indicators, and also a new Multi-Metric Indicator (MMI) utilising multiple indicators and providing a unique measure for all algorithms. Results show that the metaheuristics obtained better performances overall, particularly NSGA-II, while Choice Function was the most outstanding hyper-heuristic approach.

Fast Witness Generation for Readable GUI Test Scenarios via Generalized Experience Replay

Verifying the functional behavior of graphical user interface (GUI) applications is essential for reducing post-release issues. In practice, a developer/tester performs this verification by executing a sequence of GUI actions and then witnessing the expected behavior on the GUI screen. An automated witness generator facilitates the verification process. However, creating an unambiguous and monitorable specification for the correct behavior and then generating the correct GUI actions to trigger that behavior is challenging. In this study, we propose FARLEAD2, an automated witness generator that uses unambiguous, monitorable, and easy-to-read staged test scenarios (STSs) to specify expected behavior. FARLEAD2 maximizes its effectiveness and performance using generalized experienced replay (GER) to exploit the experience gathered from previously witnessed scenarios on new, unwitnessed test scenarios. To the best of our knowledge, STS and GER are novel improvements to GUI testing. Our evaluation of Android GUI applications shows that FARLEAD2 effectively generates a witness 95.7 times out of 100 and does it in 520 seconds, on average, indicating that FARLEAD2 is approximately 65 percent faster and 6.3 percent more effective than its best predecessor.

Comparison of Graphical User Interface Testing Tools

Graphical User Interface or better known as the user interface is the liaison of users with electronic devices such as computers. The Graphical User Interface uses icons, menus, and some other visual indicators to represent the information contained in the interface of the application being used. The Graphical User Interface I must pass the Graphical User Interface Testing stage to ensure that every element in the Graphical User Interface is not an error and by the specified one. Also, we know that Graphical User Interface Testing is a set of activities that aim to test the Graphical User Interface I of the test object to ensure that the Graphical User Interface complies with the specifications specified in the software design document. In this research, we try to compare four Graphical User Interface testing tools which ae: Robotium, Espresso, UI Automator, and Pix2Code. By exploring these 4 testing tools we find out that Pix2code can only identify objects, especially label objects. Pix2code can only meet 3 out of 7 predefined criteria. This indicates that there are still many objects of the android application that Pix2code has not been able to identify. In other words in the Graphical User Interface testing section, pix2code can play a role in identifying each object contained in the application and can be done at the design stage. The result that we get from this research is that the GUI testing tools could identify many parts and almost every object in the application except the Pix2code. For future development, Pix2code as a testing tool requires development in the form of a desktop display such as the UI Automatorviewer so that it can display every detail of the object including the attributes of the object.

Why many challenges with GUI test automation (will) remain

Automated testing is ubiquitous in modern software development and used to verify requirement conformance on all levels of system abstraction, including the system’s graphical user interface (GUI). GUI-based test automation, like other automation, aims to reduce the cost and time for testing compared to alternative, manual approaches. Automation has been successful in reducing costs for other forms of testing (like unit- or integration testing) in industrial practice. However, we have not yet seen the same convincing results for automated GUI-based testing, which has instead been associated with multiple technical challenges. Furthermore, the software industry has struggled with some of these challenges for more than a decade with what seems like only limited progress. This systematic literature review takes a longitudinal perspective on GUI test automation challenges by identifying them and then investigating why the field has been unable to mitigate them for so many years. The review is based on a final set of 49 publications, all reporting empirical evidence from practice or industrial studies. Statements from the publications are synthesized, based on a thematic coding, into 24 challenges related to GUI test automation. The most reported challenges were mapped chronologically and further analyzed to determine how they and their proposed solutions have evolved over time. This chronological mapping of reported challenges shows that four of them have existed for almost two decades. Based on the analysis, we discuss why the key challenges with GUI-based test automation are still present and why some will likely remain in the future. For others, we discuss possible ways of how the challenges can be addressed. Further research should focus on finding solutions to the identified technical challenges with GUI-based test automation that can be resolved or mitigated. However, in parallel, we also need to acknowledge and try to overcome non-technical challenges.

Testar – scriptless testing through graphical user interface

SummaryCovering all the possible paths of the graphical user interface (GUI) with test scripts would take too much effort and result in serious maintenance issues. We propose complementing scripted testing with scriptless test automation using the open‐source testar tool. This paper gives a comprehensive overview of testar and its latest extensions together with the ongoing and future research. With this paper, we hope we can help and encourage other researchers to use testar for their GUI testing‐related research and pave the way for an international research agenda in GUI testing built upon stable and open‐source infrastructure.

DroidbotX: Test Case Generation Tool for Android Applications Using Q-Learning

Android applications provide benefits to mobile phone users by offering operative functionalities and interactive user interfaces. However, application crashes give users an unsatisfactory experience, and negatively impact the application’s overall rating. Android application crashes can be avoided through intensive and extensive testing. In the related literature, the graphical user interface (GUI) test generation tools focus on generating tests and exploring application functions using different approaches. Such tools must choose not only which user interface element to interact with, but also which type of action to be performed, in order to increase the percentage of code coverage and to detect faults with a limited time budget. However, a common limitation in the tools is the low code coverage because of their inability to find the right combination of actions that can drive the application into new and important states. A Q-Learning-based test coverage approach developed in DroidbotX was proposed to generate GUI test cases for Android applications to maximize instruction coverage, method coverage, and activity coverage. The overall performance of the proposed solution was compared to five state-of-the-art test generation tools on 30 Android applications. The DroidbotX test coverage approach achieved 51.5% accuracy for instruction coverage, 57% for method coverage, and 86.5% for activity coverage. It triggered 18 crashes within the time limit and shortest event sequence length compared to the other tools. The results demonstrated that the adaptation of Q-Learning with upper confidence bound (UCB) exploration outperforms other existing state-of-the-art solutions.