Crowdsourced Testing Research Articles

A Test Report Optimization Method Fusing Reinforcement Learning and Genetic Algorithms

Filtering high-variability and high-severity defect reports from large test report databases is a challenging task in crowdtesting. Traditional optimization algorithms based on clustering and distance techniques have made progress but are limited by initial parameter settings and significantly decrease in efficiency with an increasing number of reports. To address this issue, this paper proposes a method that integrates reinforcement learning with genetic algorithms for crowdsourced testing report optimization, called Reinforcement Learning-based Genetic Algorithm for Crowdsourced Testing Report Optimization (RLGA). Its core goal is to identify distinct, high-severity defect reports from a large set. The method uses genetic algorithms to generate the optimal report selection sequence and adjusts the crossover probability (Pc) and mutation probability (Pm) dynamically with reinforcement learning based on the population’s average fitness, best fitness, and diversity. The reinforcement learning component uses a hybrid SARSA and Q-Learning strategy to update the Q-value table, allowing the algorithm to learn quickly in early iterations and expand the search space later to avoid local optima, thereby improving efficiency. To validate the RLGA method, this paper uses four public datasets and compares RLGA with six classic methods. The results indicate that RLGA outperforms BDDIV in terms of execution time and is less sensitive to the total number of test reports. In terms of optimization objectives, the test reports selected by RLGA have higher levels of defect severity and diversity than those selected by the random choice, BDDIV, and TSE methods. Regarding population diversity, RLGA effectively enhances the uniformity and diversity of individuals compared to random initialization. In terms of convergence speed, RLGA is superior to the GA, GA-SARSA, and GA-Q methods.

Clustering and Prioritization of Web Crowdsourced Test Reports Based on Text Classification

In crowdsourced testing, prioritizing numerous test reports is critical for improving developer review efficiency. Many researchers have proposed methods for prioritizing crowdsourced test reports for mobile applications, however, web crowdsourced test reports usually contain more detailed text and more complex screenshots, which makes prioritizing them especially necessary. This paper proposes a prioritization method named TCDiv, based on the data characteristics of web crowdsourcing test reports. First, using a pre-trained TextCNN model, the text of each report is segmented into two parts: reproduction steps and defect descriptions. Then, features are extracted from textual and image information respectively, and clustering technique is utilized to classify similar reports, and finally the clustering results are ranked and sampled. To validate the approach, experiments were conducted on 717 web crowdsourced test reports. The results show that TCDiv can detect more different defects in a limited time, thus improving the review efficiency of the development team.

Intellectual Property Management and Legal Protection Mechanism of Blockchain-based Crowdsourced Testing

While promoting the vigorous development of crowdsourced testing services, Internet technology has also generated problems such as non-standard intellectual property management, unclear ownership of rights, and lack of trustworthiness. Based on the characteristics of being decentralized, traceable and tamper-proof of blockchain, IPFS, SHA256 algorithm, and digital certificate technology, this paper puts forward a new intellectual property management framework for crowdsourced testing to meet the requirements of intellectual property traceability of crowdsourced testing, right confirmation and query, and proposes the idea of applying on the chain and off chain decomposition and integration technology to address the storage performance bottleneck problem caused by data expansion on the chain, thus realizing a trusted intellectual property management system of crowdsourced testing. Meanwhile, this paper puts that the intellectual property of crowdsourced testing cannot be fully maintained only by technical means, and corresponding legal support means should be supplemented, including improving the laws and regulations on judicial and administrative protection of intellectual property, and properly handling the legal risk control of intellectual property of crowdsourced testing.

Crowdsourced bug report severity prediction based on text and image understanding via heterogeneous graph convolutional networks

AbstractIn the process of crowdsourced testing, massive bug reports are submitted. Among them, the severity level of the bug report is an important indicator for traigers of crowdsourced platforms to arrange the order of reports efficiently so that developers can prioritize high‐severity defects. A lot of work has been devoted to the study of automatically assigning severity levels to a large number of bug reports in crowdsourcing test systems. The research objects of these works are standard bug reports, focusing on the text part of the report, using various feature engineering methods and classification techniques. However, while achieving good performance, these methods still need to overcome two challenges: no consideration of image information in mobile testing and discontinuous semantic information of words in bug reports. In this paper, we propose a new method of severity prediction by using heterogeneous graph convolutional networks with screenshots (SPHGCN‐S), which combines text features and screenshots information to understand the report more comprehensively. In addition, our approach applies the heterogeneous graph convolutional network (HGCN) architecture, which can capture the global word information to alleviate the semantic problem of word discontinuity and underlying relations between reports. We conduct a comprehensive study to compare seven commonly adopted bug report severity prediction methods with our approach. The experimental results show that our approach SPHGCN‐S can improve severity prediction performance and effectively predict reports with high severity.

Adaptive attention fusion network for cross-device GUI element re-identification in crowdsourced testing

The rapid growth of mobile devices has ushered in an era of different device platforms. Different devices require a consistent user experience, especially with similar graphical user interfaces (GUIs). However, the different code bases of the various operating systems as well as the different GUI layouts and resolutions of the various devices pose a challenge for automated software testing. Crowdsourced software testing (CST) has emerged as a viable solution where crowdsourced workers perform tests on their own devices and provide detailed bug reports. Although CST is cost-effective, it is not very efficient and requires a large number of workers for manual testing. The potential of optimizing CST reproduction testing through computer vision remains largely untapped, especially when considering the uniformity of GUI elements on different devices. In this study, we present a novel deep learning model specifically designed to re-identify GUI elements in CST reproduction test scenarios, regardless of the underlying code changes on different devices. The model features a robust backbone network for feature extraction, an innovative attention mechanism with learnable factors to enhance the features of GUI elements and minimize interference from their backgrounds, and a classifier to determine matching labels for these elements. Our approach was validated on a large GUI element dataset containing 31,098 element images for training, 115,704 element images from real apps for testing, and 67 different background images. The results of our experiments underline the excellent accuracy of the model and the importance of each component. This work is a major step forward in improving the efficiency of reproduction testing in CST. The innovative solutions we propose could further reduce labor costs for CST platforms.

Measurement and Calibration of EMF: A Study Using Phone and GBDT for Mobile Communication Signals

AbstractElectromagnetic exposure caused by mobile communication signals has always been a cause of concern. Due to the cost and inconvenience of professional measurement equipment, researchers have turned to smartphone APPs to study and assess the electric field strength caused by mobile communication signals. However, existing cell phone‐based measurements have two weaknesses. First, no system architecture suitable for large‐scale crowdsourced testing has been proposed. Second, since smartphone sensors cannot measure electric field strength directly, existing methods for converting the received signal power of the phone and electric field strength have errors of more than 5 dB. This paper proposes a measurement and calibration method for electric field strength of mobile communication signals based on a smartphone app and gradient boosting decision tree (GBDT). This method consists of a downlink signal acquisition system based on an APP and a calibration model based on GBDT to convert received signal power into electric field strength. The experimental results show that the proposed model achieves a R2 score of 0.93 and a MAE of 0.97 dB. Compared with the existing methods, our method improves the calibration accuracy by 4 dB, enabling large‐scale, low‐cost, and high‐precision direct measurement of the electric field strength of mobile communication signals.

Severity-Oriented Multi-Objective Crowdsourced Test Reports Prioritization

Crowdsourced testing for mobile applications is widespread. Efficiently and swiftly inspecting enormous test reports is a crucial challenge in crowdsourced testing. Most existing methods focus on finding different bugs earlier, i.e. the diversity of bugs, while ignoring bug severity. We present a multi-objective prioritization method for crowdsourced test reports that takes into account the diversity and severity of bugs. In the beginning, we ranked the test reports according to severity, then clustered the test reports by fusing textual and image features, and finally used the clustering results to adjust the initial ranking to ensure diversity. To validate our method, we conducted experiments using an industrial crowdsourced test report dataset comprised of six mobile application projects. The results demonstrate that our method can find more different high-severity bugs earlier than existing methods.

Optimizing Prioritization of Crowdsourced Test Reports of Web Applications through Image-to-Text Conversion

Collating vast test reports is a time-consuming and laborious task in crowdsourced testing. Crowdsourced test reports are usually presented in two ways, one as text and the other as images, which have symmetrical content. Researchers have proposed many text- and image-based methods for prioritizing crowdsourced test reports of mobile applications. However, crowdsourced test reports of web applications typically have clearer textual descriptions of errors but noisier screenshots of errors. This gap motivates us to propose a method for prioritizing crowdsourced test reports of web applications to detect all errors earlier. In this paper, we integrate text and image information from test reports to enhance the analysis process. First, we use the natural language processing (NLP) technique to extract textual features of error descriptions and then symmetrically extract image features of error screenshots, i.e., we use the optical character recognition (OCR) technique to obtain textual information in the screenshots and then also use the NLP technique to extract features. To validate our approach, we conduct experiments on 717 test reports. The experimental results show that our method has a higher APFD (average percentage fault detection) and shorter runtime than state-of-the-art prioritization methods.

众包测试研究综述

众包测试研究综述

Mobile App Crowdsourced Test Report Consistency Detection via Deep Image-and-Text Fusion Understanding

Crowdsourced testing, as a distinct testing paradigm, has attracted much attention in software testing, especially in mobile application (app) testing field. Compared with in-house testing, crowdsourced testing shows superiority with the diverse testing environments when faced with the mobile testing fragmentation problem. However, crowdsourced testing also encounters the low-quality test report problem caused by unprofessional crowdworkers involved with different expertise. In order to handle the submitted reports of uneven quality, app developers have to distinguish high-quality reports from low-quality ones to help the bug inspection. One kind of typical low-quality test report is inconsistent test reports, which means the textual descriptions are not focusing on the attached bug-occurring screenshots. According to our empirical survey, only 18.07% crowdsourced test reports are consistent. Inconsistent reports cause waste on mobile app testing. To solve the inconsistency problem, we propose ReCoDe to detect the consistency of crowdsourced test reports via deep image-and-text fusion understanding. ReCoDe is a two-stage approach that first classifies the reports based on textual descriptions into different categories according to the bug feature. In the second stage, ReCoDe has a deep understanding of the GUI image features of the app screenshots and then applies different strategies to handle different types of bugs to detect the consistency of the crowdsourced test reports. We conduct an experiment on a dataset with over 22k test reports to evaluate ReCoDe, and the results show the effectiveness of ReCoDe in detecting the consistency of crowdsourced test reports. Besides, a user study is conducted to prove the practical value of ReCoDe in effectively helping app developers improve the efficiency of reviewing the crowdsourced test reports.

A Crowdsourced Test Defect Number Prediction Model Based on Test Labor and Test Reports

A Crowdsourced Test Defect Number Prediction Model Based on Test Labor and Test Reports

Leveraging Android Automated Testing to Assist Crowdsourced Testing

Crowdsourced testing is an emerging trend in mobile application testing. The openness of crowdsourced testing provides a promising way to conduct large-scale and user-oriented testing scenarios on various mobile devices, while it also brings a problem, i.e., crowdworkers with different levels of testing experience severely threaten the quality of crowdsourced testing. Currently, many approaches have been proposed and studied to improve crowdsourced testing. However, these approaches do not fundamentally improve the ability of crowdworkers. In essence, the low-quality crowdsourced testing is caused by crowdworkers who are unfamiliar with the App Under Test (AUT) and do not know which part of the AUT should be tested. To address this problem, we propose a testing assistance approach, which leverages Android automated testing (i.e., dynamic and static analysis) to improve crowdsourced testing. Our approach constructs an Annotated Window Transition Graph (AWTG) model for the AUT by merging dynamic and static analysis results. Based on the AWTG model, our approach implements a testing assistance pipeline that provides the test task extraction, test task recommendation, and test task guidance to assist crowdworkers in testing the AUT. We experimentally evaluate our approach on real-world AUTs. The quantitative results demonstrate that our approach can effectively and efficiently assist crowdsourced testing. Besides, the qualitative results from a user study confirm the usefulness of our approach.

Mobile crowdsourced test report prioritization based on text and image understanding

AbstractIn the process of mobile crowdsourced testing, a large number of test reports are generated, which often consist of short text and rich image information. One of the critical issues is how to review these reports efficiently. Researchers have recently proposed clustering, classification, and prioritization techniques to solve this problem. However, existing studies directly use related technologies to text description and segment the text content without further understanding of the text content. By deeply digging into the text information and distinguishing it according to its actual meaning, the sentences described in the text can be categorized into two types: Describing abnormal system behavior and describing reproduction steps. This paper proposes a mobile crowdsourced test report prioritization technique to improve performance. First, we use a TextCNN trained on large‐scale projects to distinguish the text descriptions of reports, then extract features from the text and screenshot information, respectively. Then we apply a clustering technique to gather similar reports. Finally, the inspection order is sampled from the clustering results. To validate our approach, we conduct experiments on six industrial crowdsourced projects. The results show that our method can detect all bugs faster in a limited time than existing prioritization methods, which can improve the bug reports review efficiency.

Crowd Wisdom vs in-House Expertise: A Comprehensive Analysis of Quality Assurance Approaches

In the dynamic landscape of software quality assurance, this research paper conducts an insightful comparative analysis of the contrasting methodologies: crowd-sourced testing and in-house testing. By closely examining their individual merits, limitations, and adaptability across diverse project contexts, the study explores crucial metrics encompassing cost-effectiveness, comprehensive test coverage, and adept defect detection. Integrating quantitative data alongside real-world instances, the paper not only elucidates the intricate factors pivotal in making informed decisions between these methodologies but also provides practical, well-grounded recommendations for their efficacious implementation. The study serves as a compass, empowering software practitioners and decision-makers with a discerning outlook to deftly navigate and choose the testing strategy that impeccably aligns with their project requisites and overarching organizational goals.

Taming Android Fragmentation through Lightweight Crowdsourced Testing

Android fragmentation refers to the overwhelming diversity of Android devices and OS versions. These lead to the impossibility of testing an app on every supported device, leaving a number of compatibility bugs scattered in the community and thereby resulting in poor user experiences. To mitigate this, our fellow researchers have designed various works to automatically detect such compatibility issues. However, the current state-of-the-art tools can only be used to detect specific kinds of compatibility issues (i.e., compatibility issues caused by API signature evolution), i.e., many other essential types of compatibility issues are still unrevealed. For example, customized OS versions on real devices and semantic changes of OS could lead to serious compatibility issues, which are non-trivial to be detected statically. To this end, we propose a novel, lightweight, crowdsourced testing approach, to fill this research gap and enable the possibility of taming Android fragmentation through crowdsourced efforts. Specifically, crowdsourced testing is an emerging alternative to conventional mobile testing mechanisms that allow developers to test their products on real devices to pinpoint platform-specific issues. Experimental results on thousands of test cases on real-world Android devices show that is effective in automatically identifying and verifying API-induced compatibility issues. Also, after investigating the user experience through qualitative metrics, users' satisfaction provides strong evidence that is useful and welcome in practice.

False negative of defects estimation in crowdsourced testing

AbstractEstimating the population of defects is a key reference for completion evaluation in crowdsourced testing. Current studies commonly use the biological capture–recapture model (CRC model) to estimate the population of defects in crowdsourced tests with good results. However, the lack of consideration of false negatives for defects in existing studies leads to imprecise estimation of completion in crowdsourced testing, which in turn influences decision support during crowdsourced testing. This study analyzes the basic assumptions of the CRC model in biology and maps them to the field of crowdsourced testing, improves the Lincoln–Peterson estimator and sample coverage estimator utilizing false negative assumptions, and compares them with original estimators in 29 real crowdsourced testing projects. The experimental results show that our improved estimators can effectively reduce the estimation relative error of the population of defects caused by false negatives.

Estimate the Precision of Defects Based on Reports Duplication in Crowdsourced Testing

When analyzing the defects of crowdsourced testing, the testing reports need to be preprocessed, including removing duplicate and false positives. At present, most crowdsourced testing research focuses on duplication of the reports, which has achieved high accuracy. However, studies on reducing false positives of defects have rarely been conducted. Starting from the duplication of defects in the reports, this paper discusses the relationship between the duplication and the accuracy of defects and proposes an estimation approach based on the defect distribution in historical crowdsourced testing projects. Experiments have shown that our approach can provide a priori knowledge of defects and exhibits good stability. We applied this approach to the defect population estimation of crowdsourced testing. With better experimental results, our improved model is more accurate than the original model. We attempt to apply it to the estimation of the defect population of crowdsourced testing, which is more accurate than the original model.

CrowdIoT: The Crowd-Sourcing Test System for IoT Devices Based on Blockchain

In view of some problems existing in traditional software crowdsourcing tests and Internet of Things device tests, we developed a CrowdIoT crowdsourcing test system for the Internet of Things devices based on a block chain. CrowdIoT uses the core technology of blockchain to focus its testing on physical devices on the Internet of Things. CrowdIoT uses two different testing methods for different testing needs: online testing and offline testing. Online remote testing is our key function and research object. By opening the testing interface of Internet of Things devices to testers, testers can test the devices in the CrowdIoT system interface without having to get the hardware. At the same time, CrowdIoT introduced multi-threaded parallel testing technology to solve the conflict problem of multiple testers competing for hardware resources in online testing. Offline testing, as a supplement to online testing, is to send Internet of Things devices to testers with high credibility in deposit guarantee, so that testers can fully test the hardware devices and dig out as many vulnerabilities of the devices as possible. CrowdIoT has its own complete and scientific credibility system, combined with relevant incentive mechanism, consensus mechanism and connection mechanism, which not only effectively solves the centralization problem existing in traditional crowdsourcing testing platforms, but also solves the pain point of the lack of remote testing of Internet of Things devices in the market. Then it solves the problems of equipment limitation and testing cost limits in the field of Internet of Things equipment testing, and provides a platform for security testing and use for the Internet of Things participants.

Crowdsourced Test Report Prioritization Based on Text Classification

In crowdsourced testing, crowd workers from different places help developers conduct testing and submit test reports for the observed abnormal behaviors. Developers manually inspect each test report and make an initial decision for the potential bug. However, due to the poor quality, test reports are handled extremely slowly. Meanwhile, due to the limitation of resources, some test reports are not handled at all. Therefore, some researchers attempt to resolve the problem of test report prioritization and have proposed many methods. However, these methods do not consider the impact of duplicate test reports. In this paper, we focus on the problem of test report prioritization and present a new method named DivClass by combining a diversity strategy and a classification strategy. First, we leverage Natural Language Processing (NLP) techniques to preprocess crowdsourced test reports. Then, we build a similarity matrix by introducing an asymmetric similarity computation strategy. Finally, we combine the diversity strategy and the classification strategy to determine the inspection order of test reports. To validate the effectiveness of DivClass, experiments are conducted on five crowdsourced test report datasets. Experimental results show that DivClass achieves 0.8887 in terms of APFD (Average Percentage of Fault Detected) and improves the state-of-the-art technique DivRisk by 14.12% on average. The asymmetric similarity computation strategy can improve DivClass by 4.82% in terms of APFD on average. In addition, empirical results show that DivClass can greatly reduce the number of inspected test reports.

Advanced Crowdsourced Test Report Prioritization Based on Adaptive Strategy

Advanced Crowdsourced Test Report Prioritization Based on Adaptive Strategy