Mobile App Testing Research Articles

Extraction and empirical evaluation of GUI-level invariants as GUI Oracles in mobile app testing

ContextMobile apps (software) are used in almost all aspects of daily life by billions of people. Given the widespread use of mobile apps in various domains, the demand for systematic testing of their Graphical User Interfaces (GUI) is crucial. Despite the significant advances in automated mobile app testing over the last decade, certain challenges remain, most notably the app-specific GUI test-oracle problem, which can significantly hinder the effective detection of defects in mobile apps. In this study, we introduce the use of GUI-level invariants, referred to as GUI invariants, as app-specific GUI oracles in GUI test cases to address this challenge. MethodsWe propose a semi-automatic solution to extract GUI invariants and use them as app-specific GUI oracles in test cases. We use the mutation testing technique to evaluate the (fault detection) effectiveness of the GUI oracles used. In addition, we evaluate their quality aspects, namely correctness, understandability, and compatibility, from the perspective of human experts using a questionnaire survey. ResultsThe empirical results show that the GUI oracles used are effective and helpful, as they improved the fault-detection effectiveness of the empirical test suites ranging from 18% to 32%. These results also highlight the efficacy of GUI oracles used in identifying various defects, including crashing and non-crashing functional issues, and surpassing the performance of existing tools in fault-detection rates. Additionally, the questionnaire survey outcomes indicate that the GUI oracles used are correct, understandable, and compatible. ConclusionsBased on the empirical results, we can conclude that using GUI invariants as GUI oracles can be useful and effective in mobile app testing.

Robotic Process Automation Efficiency for Mobile App Testing: An Empirical Investigation

In today’s rapidly evolving software environment, the graphical user interface (GUI) plays a crucial role in providing intuitive, user-friendly interaction. However, traditional intrusive GUI testing methods often face challenges such as interrupting user workflows, requiring significant manual effort and insufficient test scenario coverage. Non-intrusive testing methods, such as Robotic Process Automation (RPA), offer a solution to validate GUI functionality without modifying the application’s code or affecting the user experience. RPA systems automate repetitive tasks by simulating user interactions, becoming valuable tools in GUI testing. However, challenges like limited computational resources, time constraints, or restricted exploration capabilities may limit the efficiency of individual RPA agents, thus restricting coverage and effectiveness. To address this issue, this study explores the performance of a single RPA agent versus an RPA cluster under different testing conditions, using three popular testing methods: Monkey, Stoat and Q-testing. Experimental results indicate that an RPA cluster outperforms a single RPA in GUI coverage and error detection, making a significant contribution to the field of non-intrusive GUI exploration testing. The findings of this study provide directions for future research to ensure the delivery of high-quality mobile applications.

Take the Blue Pill: Pursuing Mobile App Testing Fidelity, Efficiency, and Accessibility with Virtual Device Farms

For more than a decade, researchers have been extensively exploring mobile app testing on virtual devices [1-9], which are software-emulated mobile devices running on commodity servers, in a similar vein as virtual machines (VM) in the cloud. Building on server virtualization, virtual devices naturally inherit the benefits of VM, such as scalability, elasticity, and cost efficiency. Moreover, virtualization enables useful features not offered by physical devices, such as service instrumentation [3], whole-system snapshot [9], and memory introspection [1,2], atop which a series of advanced testing and debugging techniques are developed.

Practical, Automated Scenario-based Mobile App Testing

Practical, Automated Scenario-based Mobile App Testing

Accessibility evaluation of major assistive mobile applications available for the visually impaired

People with visual impairments face numerous challenges in their daily lives, including mobility, access to information, independent living, and employment. Artificial Intelligence (AI) with Computer Vision (CV) has the potential to improve their daily lives, provide them with necessary independence, and it will also spawn new opportunities in education and employment. However, while many such AI/CV-based mobile applications are now available, these apps are still not the preferred choice amongst visually impaired persons and are generally limited to advanced users only, due to certain limitations. This study evaluates the challenges faced by visually impaired persons when using AI/CV-based mobile apps. Four popular AI/CV-based apps, namely Seeing AI, Supersense, Envision and Lookout, are assessed by blind and low-vision users. Hence these mobile applications are evaluated on a set of parameters, including generic parameters based on the Web Content Accessibility Guidelines (WCAG) and specific parameters related to mobile app testing. The evaluation not only focused on the guidelines but also on the feedback that was gathered from these users on parameters covering the apps' accuracy, response time, reliability, accessibility, privacy, energy efficiency and usability. The paper also identifies the areas of improvement in the development and innovation of these assistive apps. This work will help developers create better accessible AI-based apps for the visually impaired.

Association Between the Characteristics of mHealth Apps and User Input During Development and Testing: Secondary Analysis of App Assessment Data.

User involvement is increasingly acknowledged as a central part of health care innovation. However, meaningful user involvement during the development and testing of mobile health apps is often not fully realized. This study aims to examine in which areas user input is most prevalent and whether there is an association between user inclusion and compliance with best practices for mobile health apps. A secondary analysis was conducted on an assessment data set of 1595 health apps. The data set contained information on whether the apps had been developed or tested with user input and whether they followed best practices across several domains. Background information was also available regarding the apps' country of origin, targeted condition areas, subjective user ratings, download numbers, and risk (as per the National Institute for Health and Care Excellence Evidence Standards Framework [ESF]). Descriptive statistics, Mann-Whitney U tests, and Pearson chi-square analyses were applied to the data. User involvement was reported by 8.71% (139/1595) of apps for only the development phase, by 33.67% (537/1595) of apps for only the testing phase, by 21.88% (349/1595) of apps for both phases, and by 35.74% (570/1595) of apps for neither phase. The highest percentage of health apps with reported user input during development was observed in Denmark (19/24, 79%); in the condition areas of diabetes (38/79, 48%), cardiology (15/32, 47%), pain management (20/43, 47%), and oncology (25/54, 46%); and for high app risk (ESF tier 3a; 105/263, 39.9%). The highest percentage of health apps with reported user input during testing was observed in Belgium (10/11, 91%), Sweden (29/34, 85%), and France (13/16, 81%); in the condition areas of neurodiversity (42/52, 81%), respiratory health (58/76, 76%), cardiology (23/32, 72%), and diabetes (56/79, 71%); and for high app risk (ESF tier 3a; 176/263, 66.9%). Notably, apps that reported seeking user input during testing demonstrated significantly more downloads than those that did not (P=.008), and user inclusion was associated with better compliance with best practices in clinical assurance, data privacy, risk management, and user experience. The countries and condition areas in which the highest percentage of health apps with user involvement were observed tended to be those with higher digital maturity in health care and more funding availability, respectively. This suggests that there may be a trade-off between developers' willingness or ability to involve users and the need to meet challenges arising from infrastructure limitations and financial constraints. Moreover, the finding of a positive association between user inclusion and compliance with best practices indicates that, where no other guidance is available, users may benefit from prioritizing health apps developed with user input as the latter may be a proxy for broader app quality.

Mobile App Crowdsourced Test Report Consistency Detection via Deep Image-and-Text Fusion Understanding

Crowdsourced testing, as a distinct testing paradigm, has attracted much attention in software testing, especially in mobile application (app) testing field. Compared with in-house testing, crowdsourced testing shows superiority with the diverse testing environments when faced with the mobile testing fragmentation problem. However, crowdsourced testing also encounters the low-quality test report problem caused by unprofessional crowdworkers involved with different expertise. In order to handle the submitted reports of uneven quality, app developers have to distinguish high-quality reports from low-quality ones to help the bug inspection. One kind of typical low-quality test report is inconsistent test reports, which means the textual descriptions are not focusing on the attached bug-occurring screenshots. According to our empirical survey, only 18.07% crowdsourced test reports are consistent. Inconsistent reports cause waste on mobile app testing. To solve the inconsistency problem, we propose ReCoDe to detect the consistency of crowdsourced test reports via deep image-and-text fusion understanding. ReCoDe is a two-stage approach that first classifies the reports based on textual descriptions into different categories according to the bug feature. In the second stage, ReCoDe has a deep understanding of the GUI image features of the app screenshots and then applies different strategies to handle different types of bugs to detect the consistency of the crowdsourced test reports. We conduct an experiment on a dataset with over 22k test reports to evaluate ReCoDe, and the results show the effectiveness of ReCoDe in detecting the consistency of crowdsourced test reports. Besides, a user study is conducted to prove the practical value of ReCoDe in effectively helping app developers improve the efficiency of reviewing the crowdsourced test reports.

Wayfinding and path integration deficits detected using a virtual reality mobile app in patients with traumatic brain injury.

The ability to navigate is supported by a wide network of brain areas which are particularly vulnerable to disruption brain injury, including traumatic brain injury (TBI). Wayfinding and the ability to orient back to the direction you have recently come (path integration) may likely be impacted in daily life but have so far not been tested with patients with TBI. Here, we assessed spatial navigation in thirty-eight participants, fifteen of whom had a history of TBI, and twenty-three control participants. Self-estimated spatial navigation ability was assessed using the Santa Barbara Sense of Direction (SBSOD) scale. No significant difference between TBI patients and a control group was identified. Rather, results indicated that both participant groups demonstrated 'good' self-inferred spatial navigational ability on the SBSOD scale. Objective navigation ability was tested via the virtual mobile app test Sea Hero Quest (SHQ), which has been shown to predict real-world navigation difficulties and assesses (a) wayfinding across several environments and (b) path integration. Compared to a sub-sample of 13 control participants, a matched subsample of 10 TBI patients demonstrated generally poorer performance on all wayfinding environments tested. Further analysis revealed that TBI participants consistently spent a shorter duration viewing a map prior to navigating to goals. Patients showed mixed performance on the path integration task, with poor performance evident when proximal cues were absent. Our results provide preliminary evidence that TBI impacts both wayfinding and, to some extent, path integration. The findings suggest long-lasting clinical difficulties experienced in TBI patients affect both wayfinding and to some degree path integration ability.

Visual judgment approach of isomorphic GUI for automated mobile app testing

Currently, the rapid growth of mobile apps requires automated testing technology to ensure their quality. Automated testing of mobile apps is usually closely related to the recognition and judgment of their graphical user interface (GUI), but there usually are numerous isomorphic GUIs with different styles and contents, and similar structure and function in mobile apps. In automatic testing, isomorphic GUI is easy to cause the issue of state space explosion, which leads to low efficiency or failure of testing. In view of the limitations of traditional automatic recognition of isomorphic GUI, this paper presents a GUI similarity judgment approach based on visual feature information. Firstly, the GUI component elements are identified by object detection, and then the GUI skeleton is abstracted. Secondly, the visual features of the GUI skeleton are extracted by a convolutional autoencoder. Finally, the isomorphic GUI judgment is completed by comparing the similarity of GUI visual features. The experimental results show that the proposed approach can effectively shield the influence of GUI style and content, complete the isomorphic GUI recognition more accurately and optimize the efficiency of automated mobile app testing.

Use of Mobile Apps in Heart Failure Self-management: Qualitative Study Exploring the Patient and Primary Care Clinician Perspective.

BackgroundMobile apps have the potential to support patients with heart failure and facilitate disease self-management, but this area of research is recent and rapidly evolving, with inconsistent results for efficacy. So far, most of the published studies evaluated the feasibility of a specific app or assessed the quality of apps available in app stores. Research is needed to explore patients’ and clinicians’ perspectives to guide app development, evaluation, and implementation into models of care.ObjectiveThis study aims to explore the patient and primary care clinician perspective on the facilitators and barriers to using mobile apps, as well as desired features, to support heart failure self-management.MethodsThis is a qualitative phenomenological study involving face-to-face semistructured interviews. Interviews were conducted in a general practice clinic in Sydney, Australia. Eligible participants were adult patients with heart failure and health care professionals who provided care to these patients at the clinic. Patients did not need to have previous experience using heart failure mobile apps to be eligible for this study. The interviews were audio-recorded, transcribed, and analyzed using inductive thematic data analysis in NVivo 12.ResultsA total of 12 participants were interviewed: 6 patients (mean age 69 [SD 7.9] years) and 6 clinicians. The interviews lasted from 25 to 45 minutes. The main facilitators to the use of apps to support heart failure self-management were communication ability, personalized feedback and education, and automated self-monitoring. Patients mentioned that chat-like features and ability to share audio-visual information can be helpful for getting support outside of clinical appointments. Clinicians considered helpful to send motivational messages to patients and ask them about signs and symptoms of heart failure decompensation. Overall, participants highlighted the importance of personalization, particularly in terms of feedback and educational content. Automated self-monitoring with wireless devices was seen to alleviate the burden of tracking measures such as weight and blood pressure. Other desired features included tools to monitor patient-reported outcomes and support patients’ mental health and well-being. The main barriers identified were the patients’ unwillingness to engage in a new strategy to manage their condition using an app, particularly in the case of low digital literacy. However, clinicians mentioned this barrier could potentially be overcome by introducing the app soon after an exacerbation, when patients might be more willing to improve their self-management and avoid rehospitalization.ConclusionsThe use of mobile apps to support heart failure self-management may be facilitated by features that increase the usefulness and utility of the app, such as communication ability in-between consultations and personalized feedback. Also important is facilitating ease of use by supporting automated self-monitoring through integration with wireless devices. Future research should consider these features in the co-design and testing of heart failure mobile apps with patients and clinicians.

Learning-Replay Based Automated Robotic Testing for Mobile App

Record-replay testing is widely used in mobile app testing as an automated testing method. However, the current record-replay methods are closely dependent on the internal information of the device or app under test. Due to the diversity of mobile devices and system platforms, their practical use is limited. To break this limitation, this paper proposes an entirely black-box learning-replay testing approach by combining robotics and vision technology to achieve a record-replay testing that can support cross-device and cross-platform. Firstly, vision technology is used to extract the critical information of GUI and gesture actions during the tester’s testing process; secondly, the GUI composition and test actions are analyzed to form a test sequence; finally, the robotic arm is guided to complete the replay of the test sequence through visual judgment. On the one hand, the approach in this paper does not access the interior of the app, shielding the association between test actions and device; on the other hand, it captures more abstract test action information instead of simple operation location records and supports more flexible test action replay. We demonstrate the effectiveness of this approach by evaluating the learning-replay of 12 popular apps for 13 typical scenarios on the same device, across devices, and across platforms.

A Novel Framework to Monitor and Test the Performance of Mobile Applications

Mobile applications (apps) are being widely used around the world. The long-term success and diffusion of a mobile app depends on it being well-tested, with respect to its performance and its reliability. Mobile app testing is typically carried out in multiple stages. An app is first tested for performance and reliability in a controlled lab-like environment during its development. However, the heterogeneity of in-the-field environments (including many device profiles, network speeds, etc.) is far greater than in a lab setting, with a small group of users. For this reason, a so-called Beta version of the app is typically rolled-out to a limited number of users, to get deeper insights into its performance and reliability in the field. After the Beta test results are analyzed, and refinements are incorporated, the application is released to all of its intended users. However, this work is based on a belief that field testing should be an automatic and ongoing process. Field test data, representing the actual performance and reliability of the app in the field, should be continuously fed back to the developer. To accomplish this, this paper proposes a framework for ongoing monitoring and field testing of mobile apps. This framework includes a library that mobile app developers can use to collect performance and reliability data during ongoing extensive field testing. These data are collected and presented using a web form to inform developers about the performance and reliability of their apps as they are being used by end-users in the field.

Testing Challenges for Mobile Applications: An evaluation and comparative analysis of different testing approaches

A mobile application, sometimes known as an app, is a form of software designed to operate on a mobile device such as a mobile phone or tablet and is commonly used to deliver user services similar to those accessible on PCs. Mobile applications have become an important part of our lives, allowing us to communicate with people all over the world while sitting at home. However, these applications do more than just connect people; they also provide information, entertainment, and a way to learn new things at any time and from any location. Mobile application testing is very important before releasing any apps. However, since multimode mobile platforms such as handsets, smartphones, tablets, pads, and wearable electronics are now available, various device fragmentations, iOS platforms, and various customer specifications, mobile application testing has recently become incredibly challenging. This article provides an overview of several mobile testing approaches as well as the most recent mobile app testing challenges. In addition, we provide a testing cycle in which mobile apps are sent to clients for feedback. In this article, we also performed a survey of current mobile app users on various bugs in real-world mobile applications, which will be used as a factor in future development.

Mobile Technology for Improved Contraceptive Care in Morocco.

The fulfillment of unmet needs for contraception can help women reach their reproductive goals. Therefore, there is a growing concern worldwide about contraception and women's knowledge of making an advised choice about it. In this aspect, an outgrown number of apps are now available providing information concerning contraception whether it concerns natural contraception or modern contraception. However, vast amounts of these apps contain inaccurate sexual health facts and non-evidence-based information concerning contraception. On these bases, and in respect to: (1) the needs of women to effectively prevent unintended pregnancies while conducting a stress-free healthy lifestyle. (2) the World Health Organization (WHO) Medical Eligibility Criteria (MEC) for contraception's recommendations, and (3) the results/recommendations of a field study conducted in the reproductive health center 'Les Orangers' in Rabat to collect the app's requirements, we developed an evidence-based patient-centered contraceptive app referred to as 'MyContraception'. Thereafter, we conducted a set of functional tests to ensure that the MyContraception solution is performing as expected and is conform to the software functional requirements previously set before moving to non-functional requirements evaluation. Since customer's feedback is valuable to non-functional testing, we choose to evaluate potential users' feedback. Moreover, giving that mobile app testing is a complex process involving different skill sets, we elaborated a rigorous experimental design to conduct an empirical evaluation of the MyContraception solution, which will exhaustively assess the overall quality of this solution and examine its effects on improving the quality of patient-centered contraception care.

Unsupervised Detection of Changes in Usage-Phases of a Mobile App

Under the fierce competition and budget constraints, most mobile apps are launched without sufficient tests. Thus, there exists a great demand for automated app testing. Recent developments in various machine learning techniques have made automated app testing a promising alternative to manual testing. This work proposes novel approaches for one of the core functionalities of automated app testing: the detection of changes in usage-phases of a mobile app. Because of the flexibility of app development languages and the lack of standards, each mobile app is very different from other apps. Furthermore, the graphical user interfaces for similar functionalities are rarely consistent or similar. Thus, we propose methods detecting usage-phase changes through object recognition and metrics utilizing graphs and generative models. Contrary to the existing change detection methods requiring learning models, the proposed methods eliminate the burden of training models. This elimination of training is suitable for mobile app testing whose typical usage-phase is composed of less than 10 screenshots. Our experimental results on commercial mobile apps show promising improvement over the state-of-the-practice method based on SIFT (scale-invariant feature transform).

A COSMIC function points based test effort estimation model for mobile applications

With the substantial proliferation in demand for applications running on mobile devices, developers and testers are foreseen to release applications with high caliber, on time and inside budget. Mobile application testing is imperative and perilous activity in the application development lifecycle, which confirms quality and unswervingly impacts the development effort and affluence of the application. The estimation of effort for testing of apps is a key figure that helps test managers in making approximate accurate decisions for coordinating testing resources. In this study, a regression model is presented to predict test effort for mobile applications considering COSMIC Function Size Measurement (FSM), mobile app characteristics/factors and test factors. The major benefit of this model is that it tends to be utilized at the beginning of mobile app testing life cycle, and thus can assist testers to effectively lead early effort estimation. The model presented is further validated and evaluated for their effectiveness by using a k-fold cross-validation method. MRE, MMRE, MdMRE, PRED (0.25) and PRED (0.50) indices are used for measuring the accuracy of the model and findings suggest that the proposed model gives a good prediction and can be exercised in the mobile software industry for predicting test effort.

AMOGA: A Static-Dynamic Model Generation Strategy for Mobile Apps Testing

In the past few years, mobile devices have been increasingly replacing traditional computers as their capabilities such as CPU computation, memory, RAM size, and many more, are being enhanced almost to the level of conventional computers. These capabilities are being exploited by mobile apps developers to produce apps that offer more functionalities and optimized performance. To ensure acceptable quality and to meet their specifications (e.g., design), mobile apps need to be tested thoroughly. As the testing process is often tedious, test automation can be the key to alleviating such laborious activities. In the context of the Android-based mobile apps, researchers and practitioners have proposed many approaches to automate the testing process mainly on the creation of the test suite. Although useful, most existing approaches rely on reverse engineering a model of the application under test for test case creation. Often, such approaches exhibit a lack of comprehensiveness as the application model does not capture the dynamic behavior of the applications extensively due to the incompleteness of reverse engineering approaches. To address this issue, this paper proposes AMOGA, a strategy that uses a hybrid, static-dynamic approach for generating user interface model from mobile apps for model-based testing. AMOGA implements a novel crawling technique that uses the event list of UI element associated with each event to dynamically exercise the events ordering at the run-time to explore the applications' behavior. Results of the experimental assessment showed that AMOGA represents an alternative approach for model-based testing of mobile apps by generating comprehensive models to improve the coverage of the applications.

A SYSTEMATIC MAPPING STUDY ON CLOUD-BASED MOBILE APPLICATION TESTING

Mobile applications and devices have played a significant role in boosting global businesses that encompass various domains such as health, education, banking, and transportation. These tools have become indispensable for everyday activities, and its applications have been developing rapidly with diverse features and platforms. However, this has created new problems and security challenges. To ensure the quality and security of these applications, a rigorous and systematic testing using cloud-based environment is required. By employing systematic mapping study (SMS) method, this paper will examine the empirical studies that address the issues on cloud-based mobile application testing. This paper presents a total of 23 primary studies that investigate cloud based mobile application testing and the effect of Testing as a Service (TaaS). The majority of these studies (56.5%) contribute to literature with a number of framework proposals. A large proportion of the studies (60.9%) analyzed Android applications, and usually supported a single type of mobile app testing. Other than that, the majority of the studies (52.2%) have failed to investigate the outcomes of TaaS, despite a plethora of services that offers TaaS. The SMS method conducted in this paper has identified gaps in literature, which are: 1) there is a lack of general and scalable approaches to support the diverse types of mobile app testing for applications using various platforms, and 2) the lack of evaluation methods such as case study to validate the proposed approaches.

Characterizing mobile apps from a source and test code viewpoint

Context: while the mobile computing market has expanded and become critical, the amount and complexity of mobile apps have also increased. To assure reliability, these apps require software engineering methods, mainly verification, validation, and testing. However, mobile app testing is a challenging activity due to the diversity and limitations found in mobile devices. Thus, it would be interesting to characterize mobile apps in hopes of assisting in the definition of more efficient and effective testing approaches. Objective: this paper aims to identify and quantify the specific characteristics of mobile apps so that testers can draw from this knowledge and tailor software testing activities to mobile apps. We investigate the presence of automated tests, adopted frameworks, external connectivity, graphical user interface (GUI) elements, sensors, and different system configurations. Method: we developed a tool to support the automatic extraction of characteristics from Android apps. We conducted an empirical study with a sample of 663 open source mobile apps. Results: we found that one third of the projects perform automated testing. The frameworks used in these projects can be divided into three groups: unit testing, GUI testing, and mocking. There is a medium correlation between project size and test presence. Specific features of mobile apps (connectivity, GUI, sensors, and multiple configurations) are present in the projects, however, they are fully covered by tests. Conclusion: automated tests are still not developed in a systematic way. Interestingly, measures of app popularity (number of downloads and rating) do not seem to be correlated with the presence of tests. However, the results show a correlation of the project size and more critical domains with the existence of automated tests. Although challenges such as connectivity, sensors, and multiple configurations are present in the examined apps, only one tool has been identified to support the testing of these challenges.

Mobile App Testing: Challenges, Strategy and Approaches

Mobile App Testing: Challenges, Strategy and Approaches