Crowdsourced Software Testing Research Articles

Adaptive attention fusion network for cross-device GUI element re-identification in crowdsourced testing

The rapid growth of mobile devices has ushered in an era of different device platforms. Different devices require a consistent user experience, especially with similar graphical user interfaces (GUIs). However, the different code bases of the various operating systems as well as the different GUI layouts and resolutions of the various devices pose a challenge for automated software testing. Crowdsourced software testing (CST) has emerged as a viable solution where crowdsourced workers perform tests on their own devices and provide detailed bug reports. Although CST is cost-effective, it is not very efficient and requires a large number of workers for manual testing. The potential of optimizing CST reproduction testing through computer vision remains largely untapped, especially when considering the uniformity of GUI elements on different devices. In this study, we present a novel deep learning model specifically designed to re-identify GUI elements in CST reproduction test scenarios, regardless of the underlying code changes on different devices. The model features a robust backbone network for feature extraction, an innovative attention mechanism with learnable factors to enhance the features of GUI elements and minimize interference from their backgrounds, and a classifier to determine matching labels for these elements. Our approach was validated on a large GUI element dataset containing 31,098 element images for training, 115,704 element images from real apps for testing, and 67 different background images. The results of our experiments underline the excellent accuracy of the model and the importance of each component. This work is a major step forward in improving the efficiency of reproduction testing in CST. The innovative solutions we propose could further reduce labor costs for CST platforms.

Crowdtesting Practices and Models: An Empirical Approach

Crowdsourced software testing (CST) has received significant attention. After these years, CST has made new progress and changes. While current literature lists many CST challenges, this paper analyzes industrial CST practices, finds that many challenges already have practical solutions, summarizes their commonalities, and comes up with new CST models and processes. We look for well-known CST websites to participate in and take a secret and unobtrusive approach where customers, platform managers, and fellow workers do not know that we are mainly interested in CST research. We then register at selected CST websites, collect any public documents such as whitepapers, open rules, and public training materials, and join as many test tasks as possible. We analyze the confrontation and collaboration among clients, platforms, and workers in the CST sessions. Clients want to get as much bug information as possible for a small amount of pay, but workers want to get paid as much as possible for a small amount of bug information. We also study the process and method of selecting suitable CST workers. Based on these, this paper proposes three future research directions. Data security and privacy at CST are paramount. If this problem can be overcome, CST will have wider applications. Additionally, the integration of workers, internal workers, software automation, and artificial intelligence will be major drivers for CST. It is also critical to develop a standardized CST structure and processes, and this will push the field to grow significantly.

Collaborative Crowdsourced Software Testing

Crowdsourced software testing (CST) uses a crowd of testers to conduct software testing. Currently, the microtasking model is used in CST; in it, a testing task is sent to individual testers who work separately from each other. Several studies mentioned that the quality of test reports produced by individuals was a drawback because a large number of invalid defects were submitted. Additionally, individual workers tended to catch the simple defects, not those with high complexity. This research explored the effect of having pairs of collaborating testers working together to produce one final test report. We conducted an experiment with 75 workers to measure the effect of this approach in terms of (1) the total number of unique valid defects detected, (2) the total number of invalid defects reported, and (3) the possibility of detecting more difficult defects. The findings show that testers who worked in collaborating pairs can be as effective in detecting defects as an individual worker; the differences between them are marginal. However, CST significantly affects the quality of test reports submitted in two dimensions: it helps reduce the number of invalid defects and also helps detect more difficult defects. The findings are promising and suggest that CST platforms can benefit from new mechanisms that allow for the formation of teams of two individuals who can participate in doing testing jobs.

Context-Aware Personalized Crowdtesting Task Recommendation

Crowdsourced software testing (short for crowdtesting) is a special type of crowdsourcing. It requires that crowdworkers master appropriate skill-sets and commit significant effort for completing a task. Abundant uncertainty may arise during a crowdtesting process due to imperfect information between the task requester and crowdworkers. For example, a worker frequently chooses tasks in an ad hoc manner in crowdtesting context, and an inappropriate task selection may lead to the worker's failing to detect any bugs, and significant testing effort unpaid and wasted. Recent studies have explored methods for supporting task requesters to make informed decisions on task pricing, worker recommendation, and so on. Unfortunately, very few study offers decision making support from the crowdworkers’ perspectives. We motivate this study through a pilot study, revealing the large portion (74 percent) of unpaid crowdworkers’ effort due to the inappropriate task choice. Drawn from our previous work on context-aware crowdworker recommendations, we advocate a more effective alternative to manual task selection would be to provide contextualized and personalized task recommendation considering the diverse distribution of worker preference and expertise, with objectives to increase their winning chances and to potentially reduce the frequency of unpaid crowd work. This paper proposes a context-aware personalized task recommendation approach <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PTRec</i> , consisting of a testing context model and a learning-based task recommendation model to aid dynamic worker decision in selecting crowdtesting tasks. The testing context model is constructed in two perspectives, i.e., process context and resource context, to capture the in-process progress-oriented information and crowdworkers’ characteristics respectively. Built on top of this context model, the learning-based task recommendation model extracts 60 features automatically, and employs random forest learner to generate dynamic and personalized task recommendation which matches workers’ expertise and interest. The evaluation is conducted on 636 crowdtesting tasks involving 2,404 crowdworkers from one of the largest crowdtesting platforms, and results show our approach can achieve an average precision of 82 percent, average recall of 84 percent, and save an estimated average of 81 percent effort originally spent on exploring, significantly outperforming four commonly-used and state-of-the-art baselines. This indicates its potential in recommending proper tasks to workers so as to improve bug detection efficiency and increase their monetary earnings.

A Study on Testers’ Learning Curve in Crowdsourced Software Testing

Recommending effective testers in crowdsourced software testing is a challenge. In this paper, we study the improvement of crowdsourced software testers’ skills over time. We propose the project difficulty coefficient to eliminate the influence of the item on the tester’s score. The hyperbolic learning curve model and exponential learning curve model are used to fit the learning ability of the testers. The experimental results show that when the test data is large, the exponential learning curve can better simulate the improvement of testers’ skills.

Crowdsourced software testing: A systematic literature review

Abstract Context Crowdsourced software testing (CST) refers to the use of crowdsourcing techniques in the domain of software testing. CST is an emerging area with its applications rapidly increasing in the last decade. Objective A comprehensive review on CST has been conducted to determine the current studies aiming to improve and assess the value of using CST as well as the challenges identified by these evaluation studies. Method We conducted a systematic literature review on CST by searching six popular databases. We identified 50 primary studies that passed our quality assessment criteria and defined two research questions covering the aim of the study. Results There are three main process activities that the current literature aims to improve, namely selection of suitable testers, reporting of defects, and validation of submitted defects. In addition, there are 23 CST evaluation studies and most of them involve a large group and real crowd. These studies have identified 27 different challenges encountered during the application of crowdsourcing in software testing. Conclusions The improvements achieved for the specific process activities in CST help explore other unexplored process activities. Similarly, knowing the characteristics of the evaluation studies can direct us on what other studies are worth investigating. Additionally, many of the challenges identified by the evaluation studies represent research problems that need better understanding and alternative solutions. This research also offers opportunities for practitioners to understand and apply new solutions proposed in the literature and the variations between them. Moreover, it provides awareness to the related parties regarding the challenges reported in the literature, which they may encounter during CST tasks.

Towards Better Crowdsourced Software Testing Process

Crowdsourced software testing (CST) is an emerging trend in software testing. Companies and developers assign testing tasks through CST platforms to thousands of online testers. Currently, CST platform managers are trying to find and resolve challenges in order to reach the best CST practice. Many features have been applied by CST platforms to improve CST activities, including notification emails, online chat rooms, forums and, most importantly, a CST platform dashboard to view all testing projects and tasks; these features have enabled CST to operate efficiently. Still, CST users find it difficult to stay abreast of test project updates, maintain their motivation, and avoid frustration. This aligns with the increasing studies in the literature that call for more efficient solutions to support CST process. This research aims to support CST by searching for potential process limitations and overcoming them. In order to do so, a five-stage approach is used: first, the current process of CST is investigated through reviewing 15 CST platforms. Second, the review has resulted in identifying eight possible activities to improve CST; among them six have been selected after a survey distributed to 30 domain experts in Stage 3. In Stage 4, we have designed and implemented five process models in a web-based system that can fulfill the requirements of the six activities identified earlier. In Stage 5, we have evaluated these process models through interviews with representatives of two CST platforms and an expert tester, and through making scenario-based evaluation with 20 domain experts who used the system and rated the value of the processes. The results show that the new improvements are sound and can strengthen the CST practice.

Crowdsourced Software Testing: A Timely Opportunity

The concept of crowdsourcing has gained a lot of attention lately. Many companies are making use of this concept for value creation, as well as the performance of varied tasks. Despite its wide application, little is known about crowdsourcing, especially when it comes to crowdsourced software testing. This paper explores the crowdsourced software testing concept from a wider perspective ranging from a cost-benefit analysis, crowdsourcing intermediaries, and the level of expertise in the crowd. Drawing from a varied range of sources, a systematic literature review is done, where the research narrows down to ten most relevant peer-reviewed sources of high impact rating. In a comparative analysis between crowdsourced software testing and in-house testing, it is found that crowd testing has numerous advantages when it comes to efficiency, user heterogeneity, and cost-effectiveness. The study indicates that intermediaries play a key role in managing the connection between the crowd and crowdsourcing companies despite various challenges. A comparison between novice testers and expert testers reveals that both the two have their unique capabilities in their respective domains.&#x0D;

Identify Severity Bug Report with Distribution Imbalance by CR-SMOTE and ELM

Manually inspecting bugs to determine their severity is often an enormous but essential software development task, especially when many participants generate a large number of bug reports in a crowdsourced software testing context. Therefore, boosting the capabilities of methods of predicting bug report severity is critically important for determining the priority of fixing bugs. However, typical classification techniques may be adversely affected when the severity distribution of the bug reports is imbalanced, leading to performance degradation in a crowdsourcing environment. In this study, we propose an enhanced oversampling approach called CR-SMOTE to enhance the classification of bug reports with a realistically imbalanced severity distribution. The main idea is to interpolate new instances into the minority category that are near the center of existing samples in that category. Then, we use an extreme learning machine (ELM) — a feedforward neural network with a single layer of hidden nodes — to predict the bug severity. Several experiments were conducted on three datasets from real bug repositories, and the results statistically indicate that the presented approach is robust against real data imbalance when predicting the severity of bug reports. The average accuracies achieved by the ELM in predicting the severity of Eclipse, Mozilla, and GNOME bug reports were 0.780, 0.871, and 0.861, which are higher than those of classifiers by 4.36%, 6.73%, and 2.71%, respectively.

When is Crowdsourcing Advantageous? The Case of Crowdsourced Software Testing

Crowdsourcing describes a novel mode of value creation in which organizations broadcast tasks that have been previously performed in-house to a large magnitude of Internet users that perform these tasks. Although the concept has gained maturity and has proven to be an alternative way of problem-solving, an organizational cost-benefit perspective has largely been neglected by existing research. More specifically, it remains unclear when crowdsourcing is advantageous in comparison to alternative governance structures such as in-house production. Drawing on crowdsourcing literature and transaction action cost theory, we present two case studies from the domain of crowdsourced software testing. We systematically analyze two organizations that applied crowdtesting to test a mobile application. As both organizations tested the application via crowdtesting and their traditional in-house testing, we are able to relate the effectiveness of crowdtesting and the associated costs to the effectiveness and costs of in-house testing. We find that crowdtesting is comparable in terms of testing quality and costs, but provides large advantages in terms of speed, heterogeneity of testers and user feedback as added value. We contribute to the crowdsourcing literature by providing first empirical evidence about the instances in which crowdsourcing is an advantageous way of problem solving.

Using Knowledge Transfer and Rough Set to Predict the Severity of Android Test Reports via Text Mining

Crowdsourcing is an appealing and economic solution to software application testing because of its ability to reach a large international audience. Meanwhile, crowdsourced testing could have brought a lot of bug reports. Thus, in crowdsourced software testing, the inspection of a large number of test reports is an enormous but essential software maintenance task. Therefore, automatic prediction of the severity of crowdsourced test reports is important because of their high numbers and large proportion of noise. Most existing approaches to this problem utilize supervised machine learning techniques, which often require users to manually label a large number of training data. However, Android test reports are not labeled with their severity level, and manual labeling is time-consuming and labor-intensive. To address the above problems, we propose a Knowledge Transfer Classification (KTC) approach based on text mining and machine learning methods to predict the severity of test reports. Our approach obtains training data from bug repositories and uses knowledge transfer to predict the severity of Android test reports. In addition, our approach uses an Importance Degree Reduction (IDR) strategy based on rough set to extract characteristic keywords to obtain more accurate reduction results. The results of several experiments indicate that our approach is beneficial for predicting the severity of android test reports.

Process Improvements for Crowdsourced Software Testing

Crowdsourced software testing has been a common practice lately. It refers to the use of crowdsourcing in software testing activities. Although the crowd testing is a collaborative process by nature, there is no available research that provides a critical assessment of the key collaboration activities offered by the current crowdsourced testing platforms. In this paper, we review the process used in the crowd testing platforms including identifying the workflow used in managing the crowd testing process starting from submitting testing requirements and ending with reviewing testing report. Understanding the current process is then utilized to identify a set of limitations in the current process and has led to propose three process improvements (improving assigning crowd manager, improving building test team, monitoring testing progress). We have designed and implemented these process improvements and then evaluated them using two techniques: 1) questionnaire and 2) workshop. The questionnaire shows that the process improvements are significantly sound and strong enough to be added to crowd testing platforms. In addition, the evaluation through conducting a workshop was useful to assess the design and implementation of the process improvements. The participants were satisfied with them but asked for further modifications. Nevertheless, because crowd testing requires participation from a large number of people, the automation suggested improving managing the current process which was highly appreciated.

How to Systematically Conduct Crowdsourced Software Testing? Insights from an Action Research Project

Nowadays, traditional testing approaches become less feasible – both economically and practicably - for several reasons, such as an increasingly dynamic environment, shorter product lifecycles, cost pressure, as well as a fast growing and increasingly segmented hardware market. With the surge towards new modes of value creation, crowdsourced software testing (CST) seems to be a promising solution to effectively solve these problems and was already applied in various software testing contexts. However, literature so far mostly neglected the perspective of an organization intending to crowdsource tasks. In this study, we present an ongoing action research project with a consortium of six companies and present a preliminary model for crowdsourced software testing in organizations. The model unfolds necessary activities, process changes, and the accompanied roles for crowdsourced software testing to enable organizations to systematically conduct such initiatives and illustrates how test departments can use crowdsourcing as a new tool in their department.

Managing crowdsourced software testing: a case study based insight on the challenges of a crowdsourcing intermediary

Crowdsourcing has gained much attention in practice over the last years. Numerous companies have drawn on this concept for performing different tasks and value creation activities. Nevertheless, despite its popularity, there is still comparatively little well-founded knowledge on crowdsourcing, particularly with regard to crowdsourcing intermediaries. Crowdsourcing intermediaries play a key role in crowdsourcing initiatives as they assure the connection between the crowdsourcing companies and the crowd. However, the issue of how crowdsourcing intermediaries manage crowdsourcing initiatives and the associated challenges has not been addresses by research yet. We address these issues by conducting a case study with a German start-up crowdsourcing intermediary called testCloud that offers software testing services for companies intending to partly or fully outsource their testing activities to a certain crowd. The case study shows that testCloud faces three main challenges, these are: managing the process, managing the crowd and managing the technology. For each dimension, we outline mechanisms that testCloud applies for facing the challenges associated with crowdsourcing projects.