Crowd Tasks Research Articles

Maximizing Revenue from Selfish Agents in Crowd Tasks: Indirect Incentive Strategies

Maximizing Revenue from Selfish Agents in Crowd Tasks: Indirect Incentive Strategies

On Dynamically Pricing Crowdsourcing Tasks

Crowdsourcing techniques have been extensively explored in the past decade, including task allocation, quality assessment, and so on. Most of professional crowdsourcing platforms adopt the fixed pricing scheme to offer a fixed price for crowd tasks. It is neither incentive for crowd workers to produce good performance, nor profitable for the requester to gain high utility with low budget. In this article, we study the problem of pricing crowdsourcing tasks with optional bonuses. We propose a dynamic pricing mechanism, named CrowdPricer for incentively delivering bonuses to the crowd workers of completing tasks, in addition to offering a base payment for completing a task. We leverage a deep time sequence model to learn the effect of bonuses on workers’ quality for crowd tasks. CrowdPricer makes decisions on whether to provide bonuses on workers, so as to maximize the requester’s utility in expectation. We present an efficient bonus delivery algorithm under the help of beam search technique, in order to efficiently solve the decision making problem. Extensive experiments using both a real crowdsourcing platform and simulations demonstrate that CrowdPricer yields the higher utility for the requester. It also obtains more correct crowd answers than the state-of-the-art pricing methods.

Crowdsourcing the identification of studies for COVID-19-related Cochrane Rapid Reviews.

BackgroundUtilisation of crowdsourcing within evidence synthesis has increased over the last decade. Crowdsourcing platform Cochrane Crowd has engaged a global community of 22,000 people from 170 countries. The COVID‐19 pandemic presented an opportunity to engage the community and keep up with the exponential output of COVID‐19 research.AimsTo test whether a crowd could accurately assess study eligibility for reviews under time constraints. Outcome measures: time taken to complete each task, time to produce required training modules, crowd sensitivity, specificity and crowd consensus.MethodsWe created four crowd tasks, corresponding to four Cochrane COVID‐19 Rapid Reviews. The search results of each were uploaded and an interactive training module was developed for each task. Contributors who had participated in another COVID‐19 task were invited to participate. Each task was live for 48‐h. The final inclusion and exclusion decisions made by the core author team were used as the reference standard.ResultsAcross all four reviews 14,299 records were screened by 101 crowd contributors. The crowd completed each screening task within 48‐h for three reviews and in 52 h for one. Sensitivity ranged from 94% to 100%. Four studies, out of a total of 109, were incorrectly rejected by the crowd. However, their absence ultimately would not have altered the conclusions of the reviews. Crowd consensus ranged from 71% to 92% across the four reviews.ConclusionCrowdsourcing can play a valuable role in study identification and offers willing contributors the opportunity to help identify COVID‐19 research for rapid evidence syntheses.

Multi-stage online task assignment driven by offline data under spatio-temporal crowdsourcing

In the era of the Internet of Things (IoT), the crowdsourcing process is driven by data collected by devices that interact with each other and with the physical world. As a part of the IoT ecosystem, task assignment has become an important goal of the research community. Existing task assignment algorithms can be categorized as offline (performs better with datasets but struggles to achieve good real-life results) or online (works well with real-life input but is difficult to optimize regarding in-depth assignments). This paper proposes a Cross-regional Online Task (CROT) assignment problem based on the online assignment model. Given the CROT problem, an Online Task Assignment across Regions based on Prediction (OTARP) algorithm is proposed. OTARP is a two-stage graphics-driven bilateral assignment strategy that uses edge cloud and graph embedding to complete task assignments. The first stage uses historical data to make offline predictions, with a graph-driven method for offline bipartite graph matching. The second stage uses a bipartite graph to complete the online task assignment process. This paper proposes accelerating the task assignment process through multiple assignment rounds and optimizing the process by combining offline guidance and online assignment strategies. To encourage crowd workers to complete crowd tasks across regions, an incentive strategy is designed to encourage crowd workers’ movement. To avoid the idle problem in the process of crowd worker movement, a drop-by-rider problem is used to help crowd workers accept more crowd tasks, optimize the number of assignments, and increase utility. Finally, through comparison experiments on real datasets, the performance of the proposed algorithm on crowd worker utility value and the matching number is evaluated.

Human+AI Crowd Task Assignment Considering Result Quality Requirements

This paper addresses the problem of dynamically assigning tasks to a crowd consisting of AI and human workers. Currently, crowdsourcing the creation of AI programs is a common practice. To apply such kinds of AI programs to the set of tasks, we often take the ``all-or-nothing'' approach that waits for the AI to be good enough. However, this approach may prevent us from exploiting the answers provided by the AI until the process is completed, and also prevents the exploration of different AI candidates. Therefore, integrating the created AI, both with other AIs and human computation, to obtain a more efficient human-AI team is not trivial. In this paper, we propose a method that addresses these issues by adopting a ``divide-and-conquer'' strategy for AI worker evaluation. Here, the assignment is optimal when the number of task assignments to humans is minimal, as long as the final results satisfy a given quality requirement. This paper presents some theoretical analyses of the proposed method and an extensive set of experiments conducted with open benchmarks and real-world datasets. The results show that the algorithm can assign many more tasks than the baselines to AI when it is difficult for AIs to satisfy the quality requirement for the whole set of tasks. They also show that it can flexibly change the number of tasks assigned to multiple AI workers in accordance with the performance of the available AI workers.

Intimate Partner Violence Exposure and Childhood Psychopathology: Associations with Discriminating Fearful and Angry Faces in Young Children.

Childhood exposure to traumatic violence may shape how children respond to threatening faces and increase risk for psychopathology. Maltreated children may exhibit altered processing of threatening faces; however, the effects of witnessing intimate partner violence (IPV) on children's discrimination of facial expressions is under-studied. Emotional face processing differentially relates to psychopathology, with some evidence suggesting improved detection of angry faces in children with fear-related anxiety symptoms, whereas externalizing symptoms are associated with poorer detection of fearful faces and perhaps emotional faces broadly. In this cross-sectional study, we examined discrimination of threatening emotional faces (angry, fearful) in relation to experiences of probable abuse and witnessing of physical IPV, as well as psychopathology. Children (N = 137, mean age = 5.01 years, SD = 0.81) completed a "face in the crowd task" designed to examine discrimination of angry and fearful faces. Children either searched for an angry face among fearful distractor faces or a fearful face among angry distractors. Probable child abuse, witnessed IPV, and symptoms were assessed in semi-structured maternal interviews. Children who witnessed violence showed poorer accuracy when fearful faces were the target; however, effects for probable abuse were non-significant. Greater fear-related anxiety symptoms were associated with poorer accuracy for fearful faces. Externalizing symptoms were associated with poorer overall accuracy. Findings suggest that IPV and fear-related anxiety symptoms were associated with difficulty detecting fearful faces when angry distractors were present, consistent with prior research. Implications of violence- and symptom-associated deficits in emotional face processing are discussed.

CrowdCog

While crowd workers typically complete a variety of tasks in crowdsourcing platforms, there is no widely accepted method to successfully match workers to different types of tasks. Researchers have considered using worker demographics, behavioural traces, and prior task completion records to optimise task assignment. However, optimum task assignment remains a challenging research problem due to limitations of proposed approaches, which in turn can have a significant impact on the future of crowdsourcing. We present 'CrowdCog', an online dynamic system that performs both task assignment and task recommendations, by relying on fast-paced online cognitive tests to estimate worker performance across a variety of tasks. Our work extends prior work that highlights the effect of workers' cognitive ability on crowdsourcing task performance. Our study, deployed on Amazon Mechanical Turk, involved 574 workers and 983 HITs that span across four typical crowd tasks (Classification, Counting, Transcription, and Sentiment Analysis). Our results show that both our assignment method and recommendation method result in a significant performance increase (5% to 20%) as compared to a generic or random task assignment. Our findings pave the way for the use of quick cognitive tests to provide robust recommendations and assignments to crowd workers.

Investigating Differences in Crowdsourced News Credibility Assessment

Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy to combat misinformation by attempting to replicate the assessments of experts at scale. In this work, we investigate news credibility assessments by crowds versus experts to understand when and how ratings between them differ. We gather a dataset of over 4,000 credibility assessments taken from 2 crowd groups---journalism students and Upwork workers---as well as 2 expert groups---journalists and scientists---on a varied set of 50 news articles related to climate science, a topic with widespread disconnect between public opinion and expert consensus. Examining the ratings, we find differences in performance due to the makeup of the crowd, such as rater demographics and political leaning, as well as the scope of the tasks that the crowd is assigned to rate, such as the genre of the article and partisanship of the publication. Finally, we find differences between expert assessments due to differing expert criteria that journalism versus science experts use---differences that may contribute to crowd discrepancies, but that also suggest a way to reduce the gap by designing crowd tasks tailored to specific expert criteria. From these findings, we outline future research directions to better design crowd processes that are tailored to specific crowds and types of content.

How Context Influences Cross-Device Task Acceptance in Crowd Work

Although crowd work is typically completed through desktop or laptop computers by workers at their home, literature has shown that crowdsourcing is feasible through a wide array of computing devices, including smartphones and digital voice assistants. An integrated crowdsourcing platform that operates across multiple devices could provide greater flexibility to workers, but there is little understanding of crowd workers’ perceptions on uptaking crowd tasks across multiple contexts through such devices. Using a crowdsourcing survey task, we investigate workers’ willingness to accept different types of crowd tasks presented on three device types in different scenarios of varying location, time and social context. Through analysis of over 25,000 responses received from 329 crowd workers on Amazon Mechanical Turk, we show that when tasks are presented on different devices, the task acceptance rate is 80.5% on personal computers, 77.3% on smartphones and 70.7% on digital voice assistants. Our results also show how different contextual factors such as location, social context and time influence workers decision to accept a task on a given device. Our findings provide important insights towards the development of effective task assignment mechanisms for cross-device crowd platforms.

A crowdsourcing method for online social networks security assessment based on human-centric computing

Crowdsourcing and crowd computing are a trend that is likely to be increasingly popular, and there remain a number of research and operational challenges that need to be addressed. The human-centric computational abstraction called situation may be used to cope with these difficulties. In this paper, we focus on one such challenge, which is how to assign crowd assessment tasks about security and privacy in online social networks to the most appropriate users efficiently, effectively and accurately. Specifically, here we propose a novel task assignment method to facilitate crowd assessment, which improves the security and trustworthiness of social networking platforms, as well as a task assignment algorithm based on SocialSitu, which is a social-domain-focused situational analytics. Findings from our crowd assessment experiments on a real world social network Shareteches show that the precision and recall of the proposed method and algorithm are 0.491 and 0.538 higher than those of a random algorithm’s, as well as 0.336 and 0.366 higher than users’ theme-aware algorithm’s, respectively. Moreover, these results further suggest that our experimental evaluation enhance the security and privacy of online social networks.

Walrasian Equilibrium-Based Multiobjective Optimization for Task Allocation in Mobile Crowdsourcing

With the rapid development of Industry 5.0 and mobile devices, the research of mobile crowdsensing networks has become an important research focus. Task allocation is an important research content that can inspire crowd workers to participate in crowd tasks and provide truthful sensed data in mobile crowdsourcing systems. However, how to inspire crowd workers to participate in crowd tasks and provide truthful sensed data still has many challenges. In this article, based on the Markov model and collaborative filtering model, the similarities, trajectory prediction, dwell time, and trust degree are considered to propose the Markov and Collaborative filtering-based Task Recommendation (MCTR) model. Then, based on the Walrasian equilibrium, the optimum solution is researched to maximize the social welfare of mobile crowdsourcing systems. Finally, the comparison experiments are carried out to evaluate the performance of the proposed multiobjective optimization and the Markov-based task allocation with other methods. Through comparison experiments, the efficiency and adaptation of mobile crowdsourcing systems could be improved by the proposed task allocation.

How Gamification Affects Crowdsourcing: The Case of Amazon Mechanical Turk

Since its very first appearance the concept of crowdsourcing has undergone major variations, coming to include highly heterogeneous phenomena such as Google’s data mining, exchanges on sharing economy platforms (e.g. Airbnb or eBay), contents production within creative communities online (e.g. Wikipedia) and much more. If one assumes a very broad perspective, it is eventually possible to extend the category of crowdsourcing to cover whatsoever phenomena involving the participation of the crowd online, as in fact has been done. On the contrary, I will argue that crowdsourcing – and in particular its microwork branch – represents the specific practice of extending outsourcing processes to a large, low-cost, scalable and flexible workforce, in order to generate greater added value for a supply chain. To develop this analysis, I will especially focus on the case of Amazon Mechanical Turk, and on how the operations carried out on this platform are primarily intended to manage the huge flow of information which spans across a supply chain. The practice of subcontracting to the crowd tasks previously carried out by employees or third-party suppliers highlights how crowdsourcing involves a reshaping of the supply chain, further extending it to a large network of individuals. Through crowdsourcing processes, companies are either able to replace or train AI, integrating human computation skills in algorithmic structures through simple, and oftentimes tedious, microtasks. In this context, processes of gamification are capable to put further downward pressure on already small piece-wages, as long as crowdworkers are rather willing to earn an even lower economic compensation, if it’s associated to challenging tasks; thus, to make a task more enjoyable through gamification could be an effective way to further reduce a supply chain’s expenditures in crowdsourcing, pushing forward labor exploitation practices structurally embedded in this phenomenon.

Gender differences in attention to pain body postures in a social context: a novel use of the bodies in the crowd task.

Pain signals the presence of potential harm, captures attention, and can inhibit performance on concurrent tasks. What is less well known, however, is whether such attentional capture also occurs in a wider social context, such as when observing people in pain. To explore this possibility, we adopted a novel social-cue detection methodology: the bodies-in-the-crowd task. Two experiments are reported that consider whether nonverbal cues of pain, happiness, and anger as expressed through body postures would capture and hold attention. Both experiments recruited 40 (20 male and 20 female) pain-free individuals. Overall, results show that pain postures do not capture attention any more than happiness or anger postures, but disengagement from pain postures was significantly slower across both studies. Gender differences were also found, and were more likely to be found when crowds comprised both men and women. Male pain postures were more likely to capture attention. However, female observers had faster target detection speed, and were quicker to disengage from distractors. They also showed slower disengagement from female expressions overall. Male observers showed no variation based on target or distractor gender. Implications and potential directions for future research are discussed.

A worker-selection incentive mechanism for optimizing platform-centric mobile crowdsourcing systems

With the development of Mobile Crowd Sensing Networks (MCSN), more and more Mobile Crowdsourcing applications emerge. The mobile sensing technologies and theories have become the research hotspot. How to design an effective incentive mechanism to maximize social welfare is the key research in mobile crowdsourcing. This paper proposes an online auction algorithm through combining multi-attribute auction and reverse auction to select the candidate set of crowd workers dynamically. In order to further optimize the utility of platform-centric mobile crowdsourcing, the candidate set of crowd workers is optimized through improving the Discrete Particle Swarm Optimization (DPSO), which can determine the winner set of crowd workers. Through comparison experiments on simulation dataset and real dataset, the effectiveness and performance of the proposed worker-selection incentive mechanism are verified. From experimental results, it can be inferred that the proposed worker-selection incentive mechanism can inspire users to participate in crowd tasks and maximize the utility of mobile crowdsourcing systems effectively.

Privacy-aware task data management using TPR*-Tree for trajectory-based crowdsourcing

Spatial crowdsourcing is a promising architecture for collecting various types of data online with the aid of participants’ powerful mobile devices. However, it is also associated with certain privacy and security issues, which can reduce the quality of the crowdsourcing service. Some crowd tasks require the collection of connected data points. When a location-anonymous method is employed to ensure the privacy of location data points, the location trajectory data may become meaningless. To solve the privacy problem for trajectory data in large-scale crowdsourcing systems, we proposed a spatial task management method for privacy-preserving trajectory-based crowdsourcing, using a 3DES encryption and compressive-sensing-based trajectory data decryption method which is called DES-TraVec (3DES-based trajectory vector) cryptography algorithm. To provide a real-time crowdsourcing service, we proposed the use of an extended TPR*-Tree to bulk load the crowdsourcing results and manage the benders service requests so that the proposed method could support participants’ privacy and ensure quick answers for crowdsourcing services. The experimental results demonstrated that the proposed method is efficient in preserving trajectory-based crowdsourcing data and is faster than the current method.

A differentially k-anonymity-based location privacy-preserving for mobile crowdsourcing systems

With the rapid development of mobile devices, the problem privacy leaking has become an important research focus in the field of mobile crowdsourcing. In order to guarantee the security and truthfulness of mobile crowdsourcing, this paper proposes a differentially k-anonymity location privacy-preserving for mobile crowdsourcing. Through combining k-anonymity and differential privacy-preserving, the differentially k-anonymity-based location privacy-preserving is proposed in order to prevent workers’ location information from being leaked. Through comparison experiments, the effectiveness, adaptation and flexibility of the proposed differentially k-anonymity-based location privacy-preserving is verified. The differentially k-anonymity-based location privacy-preserving can inspire workers to participate crowd tasks, and protect workers’ location privacy effectively.

Spatial task management method for location privacy aware crowdsourcing

Spatial crowdsourcing is a promising architecture that collects various types of data online with the help of participants powerful mobile devices. Humans are involved in the crowdsourcing process, thereby increasing its accuracy; however, it is also associated with some privacy and security problems. The crowd tasks are executed in participants mobile devices, and the results are send to the server through networks, so that attackers could eavesdrop participants location information. Thus, we studied and proposed a spatial task assignment method for privacy-aware spatial crowdsourcing using a secure grid-based index. The secure grid index used an encrypted grid number and grid cell-based local coordinate system to protect participants location privacy. By using the grid based index in spatial task management process, it also could increase the spatial task processing time. In the experimental test, we showed that the proposed method is faster than the current method and extremely efficient when the spatial crowdsourcing tasks are geometry based tasks.

Exploiting Massive D2D Collaboration for Energy-Efficient Mobile Edge Computing

In this article we propose a novel D2D Crowd framework for 5G mobile edge computing, where a massive crowd of devices at the network edge leverage network-assisted D2D collaboration for computation and communication resource sharing. A key objective of this framework is to achieve energy-efficient collaborative task executions at the network edge for mobile users. Specifically, we first introduce the D2D Crowd system model in detail, and then formulate the energy-efficient D2D Crowd task assignment problem by taking into account the necessary constraints. We next propose a graph-matching-based optimal task assignment policy, and further evaluate its performance through extensive numerical study, which shows superior performance of more than 50 percent energy consumption reduction over the case of local task executions. Finally, we also discuss the directions of extending the D2D Crowd framework by taking into account a variety of application factors.

Crowds of Crowds: Performance based Modeling and Optimization over Multiple Crowdsourcing Platforms

The dynamic nature of crowdsourcing platforms poses interesting problems for users who wish to schedule large batches of tasks on these platforms. Of particular interest is that of scheduling the right number of tasks with the right price at the right time, in order to achieve the best performance with respect to accuracy and completion time. Research results, however, have shown that the performance exhibited by online platforms are both dynamic and largely unpredictable. This is primarily attributed to the fact that, unlike a traditional organizational workforce, a crowd platform is inherently composed of a dynamic set of workers with varying performance characteristics. Thus, any effort to optimize the performance needs to be complemented by a deep understanding and robust techniques to model the behaviour of the underlying platform(s). To this end, the research in this paper studies the above interrelated facets of crowdsourcing in two parts. The first part comprises the aspects of manual and automated statistical modeling of the crowd-workers' performance; the second part deals with optimization via intelligent scheduling over multiple platforms. %based on simulation testbed generated by the statistical models.Detailed experimentation with competing techniques, under varying operating conditions, validate the efficacy of our proposed algorithms while posting tasks either on a single crowd platform or multiple platforms. Our research has led to the development of a platform recommendation tool that is now being used by a large enterprise for performance optimization of voluminous crowd tasks.

The trustworthiness of reference group and cooperative quality: implications for online crowdsourcing market

Crowdsourcing is one emerging business model featured by value cooperation of focal agent and crowds. The trustworthiness of first movers, they act as reference groups, would provide referent information for crowds' decision. It is necessary to develop an understanding of the relationship of reference groups and the cooperative quality of crowd task. The authors argue that reference groups can obtain trustworthiness in the eyes of the crowd, such as integrity trustworthiness and ability trustworthiness. We examine the influence mechanism of trustworthiness of reference groups, and testify our arguments with secondary data from a typical crowdsourcing website. We found that, when the perceived risk of task is high, the effect of reference groups' integrity trustworthiness on crowds' participation is greater than ability trustworthiness. While the perceived risk is low, the effect of reference groups' ability trustworthiness on crowds' participation is greater than integrity trustworthiness. And the degree increase of the participation of crowd will improve cooperative quality between focal agent and qualified crowd. The paper supposes management policies based on the empirical research.