Crowdsourcing Task Research Articles

Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platforms

Crowdsourcing tasks have been widely used to collect a large number of human labels at scale. While some of these tasks are deployed by requesters and performed only once by crowd workers, others require the same worker to perform the same task or a variant of it more than once, thus participating in a so-called longitudinal study . Despite the prevalence of longitudinal studies in crowdsourcing, there is a limited understanding of factors that influence worker participation in them across different crowdsourcing marketplaces. We present results from a large-scale survey of 300 workers on 3 different micro-task crowdsourcing platforms: Amazon Mechanical Turk, Prolific, and Toloka. The aim is to understand how longitudinal studies are performed using crowdsourcing. We collect answers about 547 experiences and we analyze them both quantitatively and qualitatively. We synthesize 17 take-home messages about longitudinal studies together with 8 recommendations for task requesters and 5 best practices for crowdsourcing platforms to adequately conduct and support such kinds of studies. We release the survey and the data at: https://osf.io/h4du9/.

Research on Express Crowdsourcing Task Allocation Considering Distribution Mode under Customer Classification

Research on Express Crowdsourcing Task Allocation Considering Distribution Mode under Customer Classification

Research on Multi-round First-price Sealed Bidding Strategy Based on Bidders' Incomplete Information Game

Bidding mechanisms are essential for the market and policies because bidding patterns have an impact on economic rationality and are crucial for the functioning of the construction industry. Thus, this paper consolidates the findings from the multiple rounds of combinatorial auctions, the winners curse in incomplete information auctions, and game theory. There is no doubt that bidding mechanisms play an important role in markets and policies in economic activities. This paper's related topics of interest would be multi-round combinatorial auctions, winners curses in project bidding, and game theory in incompletely informed auctions. This paper reviews the literature and employs the models based on the Nash equilibrium. There are many types of bidding strategies such as first, second, and successive bid bidding strategies, cooperative bidding, non-cooperative bidding, and how information asymmetry plays a critical role in these strategies. As mentioned before, criterion-based adaptive bidding strategies are not sensitive to market conditions and can provide near-optimal solutions. In the same way, iterative learning also prevents the occurrence of the winners curse in the multi-stage bidding context. UK auctions are seen as more efficient relative to other locations because information costs are lower, while game-theory-based strategies improve mechanisms for obtaining tasks and cost optimization in mobile crowdsourcing tasks. The use of electronic auctions can help a market become more fully rational and efficient. The methodology of game theory helps explain the outcomes of auctioning and emphasizes innovations in automation as well as policies. Therefore, this paper's implications are significant to the current and ongoing literature on efficiency, strategic behaviour, and policy in bidding mechanisms across different markets.

A clarity and fairness aware framework for selecting workers in competitive crowdsourcing tasks

A clarity and fairness aware framework for selecting workers in competitive crowdsourcing tasks

Framing is Mightier than the Sword: Detection of Episodic and Thematic Framing in News Media

Many people today watch news videos to get informed. However, news videos can frame information differently and be prone to bias that might lead to miscommunication. Bias is ubiquitous andinherent in interactions between news consumers’ groups, but framing can introduce additional bias in news communications. Citizens that interpret the news have different political orientations and, thus, understand them differently. Experts can be more capable of detecting biased or differently framed information. However, the ever-increasing amount of news videos also make it difficult for experts alone to analyze. While automated methods exist for identifying different types of bias, frame detection approaches, namely episodic and thematic framing, are scarce and focused on texts. In this work, we address the issue of scalable thematic and episodic frame detection in news videos through crowdsourcing and machine learning techniques. We design a crowdsourcing task for annotating thematic and episodic framing in videos with the help of domain experts in political and social sciences. We then use the annotations gathered from experts and crowds to investigate whether machine learning methods can scale up the annotation process by automatically labelling videos on episodic and thematic framing. Our results indicate that framing analysis is a challenging task, both for humans and machines, with high disagreement amongst experts and crowd annotators. Nevertheless, our results suggest that machine learning has potential by combining crowd and expert annotations and building upon them a classifier.

Revisiting Bundle Recommendation for Intent-aware Product Bundling

Product bundling represents a prevalent marketing strategy in both offline stores and e-commerce systems. Despite its widespread use, previous studies on bundle recommendation face two significant limitations. Firstly, they rely on noisy datasets, where bundles are defined by heuristics, e.g., products co-purchased in the same session. Secondly, they target specific tasks by holding unrealistic assumptions, e.g., the availability of bundles for recommendation directly. This paper proposes to take a step back and considers the process of bundle recommendation from a holistic user experience perspective. We first construct high-quality bundle datasets with rich metadata, particularly bundle intents, through a carefully designed crowd-sourcing task. We then define a series of tasks that together, support all key steps in a typical bundle recommendation process, from bundle detection, completion and ranking, to explanation and auto-naming, whereby 19 research questions are raised correspondingly to guide the analysis. Finally, we conduct extensive experiments and analyses with representative recommendation models and large language models (LLMs), demonstrating the challenges and opportunities, especially with the emergence of LLMs. To summarize, our study contributes by introducing novel data sources, paving the way for new research avenues, and offering insights to guide product bundling in real e-commerce platforms.

Leadership Through Citizen Science Games

The current paper examines how citizen science games, or games that build knowledge and enable the crowdsourcing of tasks, enable leadership. Specifically, the paper describes how both the creators and players of these knowledge‐building games are collectively engaged in the leadership process. Players are enacting leadership through collaboration, communication, and problem solving with other players, as well as alongside the creators of these games. The games' developers are also engaged in leadership in how they are framing and enabling problem solving and other collective activities.

What makes an awfully good oxymoron?

AbstractOxymorons combine two opposite terms in a paradoxical manner. They are closely intertwined with antonymy, since the union of antonymous items creates the paradoxical effect of the oxymoron and generates a new meaning. Compared to other forms of figurative language, oxymorons are largely underinvestigated. We explored what makes good oxymorons through a crowdsourcing task in which we asked participants to judge the acceptability, comprehensibility, effectiveness/aptness, commonness, pleasantness, and humoristic connotation of Italian adjective–noun oxymorons. We hypothesized that oxymorons featuring morphologically related antonyms (felice infelicità ‘happy unhappiness’) may be perceived to be better than oxymorons featuring morphologically unrelated antonyms (felice tristezza ‘happy sadness’) and that oxymorons constructed by complementaries (esatta inesattezza ‘exact inexactness’) may be perceived to be better than oxymorons constructed by contraries (bella bruttezza ‘beautiful ugliness’). The results confirmed only partially our hypotheses: oxymorons with complementaries were perceived as more acceptable, comprehensible, effective/apt, common, whereas no strong trend was found for the other two dimensions. Surprisingly, our analyses revealed that oxymoronic constructions containing morphologically unrelated words were perceived as more acceptable, comprehensible, effective/apt, common, pleasant, contradicting our initial expectations.

Your Cursor Reveals: On Analyzing Workers’ Browsing Behavior and Annotation Quality In Crowdsourcing Tasks

In this work, we investigate the connection between browsing behavior and task quality of crowdsourcing workers performing annotation tasks that require information judgements. Such information judgements are often required to derive ground truth answers to information retrieval queries. We explore the use of workers’ browsing behavior to directly determine their annotation result quality. We hypothesize <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">user attention</i> to be the main factor contributing to a worker’s annotation quality. To predict annotation quality at the task level, we model two aspects of task-specific user attention, also known as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">general</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">semantic user attentions</i> . Both aspects of user attention can be modeled using different types of browsing behavior features but most previous research mostly focuses on the former. This work therefore proposes to model semantic user attention by capturing the worker’s understanding of task content using task-semantics specific behavior features. We develop a web-based annotation interface for gathering user behavior data when workers perform a knowledge path retrieval task. With the collected data, we train several prediction models using behavior features corresponding to different aspects of user attention and conduct experiments on a set of annotation tasks performed by 51 Amazon Mechanical Turk workers. We show that the prediction model using both general and semantic user attention features can achieve the best performance of nearly 75% accuracy.

Mobile Crowdsourcing Task Allocation Based on Dynamic Self-Attention GANs

Mobile Crowdsourcing Task Allocation Based on Dynamic Self-Attention GANs

Multiplicative Consistent q-Rung Orthopair Fuzzy Preference Relations with Application to Critical Factor Analysis in Crowdsourcing Task Recommendation

This paper presents a group decision-making (GDM) method based on q-rung orthopair fuzzy preference relations (q-ROFPRs). Firstly, the multiplicative consistent q-ROFPRs (MCq-ROFPRs) and the normalized q-rung orthopair fuzzy priority weight vectors (q-ROFPWVs) are introduced. Then, to obtain q-ROFPWVs, a goal programming model under q-ROFPRs is established to minimize their deviation from the MCq-ROFPRs and minimize the weight uncertainty. Further, a group goal programming model of ideal MCq-ROFPRs is constructed to obtain the expert weights using the compatibility measure between the ideal MCq-ROFPRs and the individual q-ROFPRs. Finally, a GDM method with unknown expert weights is solved by combining the group goal programming model and the simple q-rung orthopair fuzzy weighted geometric (Sq-ROFWG) operator. The effectiveness and practicality of the proposed GDM method are verified by solving the crucial factors in crowdsourcing task recommendation. The results show that the developed GDM method effectively considers the important measures of experts and identifies the crucial factors that are more reliable than two other methods.

Pick-up point recommendation strategy based on user incentive mechanism

In recent years, with the development of spatial crowdsourcing technology, online car-hailing, as a typical spatiotemporal crowdsourcing task application scenario, has attracted widespread attention. Existing researches on spatial crowdsourcing are mainly based on the coordinate positions of user and worker roles to achieve task allocation with the goal of maximum matching number or lowest cost. However, they ignores the problem of the selection of the pick-up point which needs to be solved in the actual scene of online car booking. This problem needs to take into account the four-dimensional coordinate positions of users, workers, pick-up point and destination. Based on this, this study designs a pick-up point recommendation strategy based on user incentive mechanism. Firstly, a new four-dimensional crowdsourcing model is established, which is closer to the practical application of crowdsourcing problem. Secondly, taking cost optimization as the index, a user incentive mechanism is designed to encourage users to walk to the appropriate pick-up point within a certain distance. Thirdly, a concept of forward rate is proposed to reduce the computation time. Some key factors, such as the maximum walking distance limit of users and task cost, are considered as the recommendation index for measuring the pick-up point. Then, an effective pick-up point recommendation strategy is designed based on this index. Experiments show that the strategy proposed in this article can achieve reasonable recommendation for pick-up points and improve the efficiency of drivers and reduce the total trip cost of orders to the greatest extent.

CrowdDA: Difficulty-aware crowdsourcing task optimization for cleaning web tables

Web tables are rich sources of structured data for collecting and analyzing data, but there exist various data quality problems such as missing or inconsistent values. Crowdsourcing provides a new solution that leverages the human cognitive ability to clean tables, but existing crowdsourcing-based solutions suffer from cleaning quality. For most crowd are not experts, the difficulty degree of cleaning tasks will seriously affect the cleaning result. To help people clean web tables effectively and efficiently, it is important to reduce the overall difficulty of tasks. In this paper, we introduce a difficulty-aware crowdsourcing task optimization system CrowdDA, which aims to recommend the best task execution order from easy to difficult for crowd and support various kinds of cleaning tasks for web tables. CrowdDA takes both latency and space constraints into account for task optimization and generates the task execution order that minimizes the overall difficulty of tasks under two constraints. Furthermore, CrowdDA adopts partition strategies for large tables to improve system efficiency, and introduces independent task sequence to tolerate crowd’s inconsistent answers for system robustness. The experiments based on the real-world datasets demonstrate the performance superiority of CrowdDA for improving the cleaning quality of web tables.

Application of clustering cooperative differential privacy in spatial crowdsourcing task allocation

A framework for spatial crowdsourcing task allocation based on centralized differential privacy is proposed for addressing the problem of worker’s location privacy leakage. Firstly, by combining two stages of differential privacy noise addition and clustering matching, a spatial crowdsourcing worker dataset with high differential privacy protection can be obtained; Secondly, the dynamic problem of spatial crowdsourcing task allocation is transformed into a static combinatorial optimization problem by dividing the spatiotemporal units and the “delay matching” strategy; Finally, the improved discrete glowworm swarm optimization algorithm is used to calculate the results of spatial crowdsourcing task allocation. It has been demonstrated that, compared to the direct differential privacy noise-adding assignment method and the discrete glowworm swarm optimization assignment method, the proposed method achieves better task assignment results, with the total travel distance reduced by 12.42% and 3.56%, respectively, and the task assignment success rate increased by 11.75% and 3.34%, respectively.

Modeling and IAHA Solution for Task Scheduling Problem of Processing Crowdsourcing in the Context of Social Manufacturing

The paper addresses the discrete characteristics of the processing crowdsourcing task scheduling problem in the context of social manufacturing, divides it into two subproblems of social manufacturing unit selecting and subtask sorting, establishes its mixed-integer programming with the objective of minimizing the maximum completion time, and proposes an improved artificial hummingbird algorithm (IAHA) for solving it. The IAHA uses initialization rules of global selection, local selection, and random selection to improve the quality of the initial population, the Levy flight to improve guided foraging and territorial foraging, the simplex search strategy to improve migration foraging to enhance the merit-seeking ability, and the greedy decoding method to improve the quality of the solution and reduce solution time. For the IAHA, orthogonal tests are designed to obtain the optimal combination of parameters, and comparative tests are made with variants of the AHA and other algorithms on the benchmark case and a simulated crowdsourcing case. The experimental results show that the IAHA can obtain superior solutions in many cases with economy and effectiveness.

Are crowdsourcing announcements signals of customer orientation? A comparison of consumer responses to product- versus communication-related campaigns

PurposeThis study aims to examine consumers’ responses to crowdsourcing campaigns in the request initiation stage using the signaling theory from economics. The purpose of the research is threefold. First, it provides a comprehensive classification of various task types within crowdsourcing. Second, it conceptualizes crowdsourcing announcements as signals of customer orientation and empirically tests the differential effects of the two most common crowdsourcing task types (product- and communication-related) on customer orientation perceptions. Third, it illuminates the downstream behavioral consequences of crowdsourcing campaign announcements.Design/methodology/approachThe authors conducted secondary data analysis of 883 crowdsourcing campaigns (pilot study) to provide evidence on the differential effects of crowdsourcing task types. In addition, four laboratory experiments were conducted to test the theoretical arguments. To test the main effect of crowdsourcing task types, Study 1A (N = 252 MTurk workers) used a one-factor (product- vs communication-related crowdsourcing vs control) between-subject design, whereas Study 1B (N = 171 undergraduate students) used a 2 (task type: product- vs communication-related) by 2 (product category: restaurant vs fashion) between-subject design. Study 2 (N = 93 MTurk workers) explored the underlying mechanism using a one-factor (product- vs communication-related) between-subject design. Study 3 (N = 375 MTurk workers) investigated the boundary condition for the effect of task type with a 2 (task type: product- vs communication-related) by 3 (company credibility: low vs neutral vs high) between-subject design.FindingsThe pilot study provides evidence for the conceptualized typology and the differential effects of crowdsourcing task types. Study 1A reveals that product-related crowdsourcing tends to have a more substantial impact than communication-related crowdsourcing on how customer-oriented consumers perceive a company. Study 1B validates the results of Study 1A in a different product category and population sample. Study 2 shows that the differential customer-orientation effect is mediated by the perceived cost of implementing the crowdsourcing outcome and unravels the differences in consumers’ purchase and campaign participation intentions depending on task type. Study 3 highlights that the customer-orientation effect attenuates as company credibility increases.Research limitations/implicationsThis research contributes to the crowdsourcing literature by categorizing the various types of crowdsourcing campaigns companies undertake and revealing the differential impact of the different types of crowdsourcing campaigns on consumers’ perceptions and behavioral intentions. In doing so, this research converges two lines of consumer research on crowdsourcing, i.e. product- and communication-related crowdsourcing. The findings add to the debate over the returns from research and development (R&D) versus advertising and extend it from marketing strategy to crowdsourcing literature.Practical implicationsThe findings highlight the importance of choosing specific task types for crowdsourcing and lead to practical recommendations on designing crowdsourcing campaigns to maximize their benefits to crowdsourcing brands.Originality/valueTo the best of the authors’ knowledge, this is the first study that differentiates crowdsourcing task types and compares their effectiveness from a consumer perspective.

A Generative Answer Aggregation Model for Sentence-Level Crowdsourcing Tasks

Previous answer aggregation methods for sentence-level crowdsourcing tasks extract one sentence from a collection of redundant sentences. However, these extractive methods have a drawback in that they ignore the phenomenon of answer complementarity and the spammer dilemma in crowdsourcing. To alleviate this problem, in this paper, we generate new, comprehensive sentences that synthesize all redundant sentences. To achieve this goal, we design a sequence-to-sequence neural model composed of an encoder and decoder. Specifically, considering the complementarity phenomenon, the encoder synthesizes all the collected sentences into hidden states, which are then utilized by the decoder to generate the final sentence. Next, considering the spammer dilemma, we model the workers in the neural network to detect spammers. Furthermore, to train the neural model better, we construct pseudo-sentences to enrich the training data. The experimental results demonstrate that our method can efficiently aggregate redundant sentences and generate comprehensive sentences with increased quality.

BFCRI: A Blockchain-Based Framework for Crowdsourcing With Reputation and Incentive

With the rapid development of cloud computing and the sharing economy, crowdsourcing aroused widespread interest and adoption in providing intelligent and efficient services for humans. The majority of existing works focus on effective crowdsourcing task assignment and privacy protection, mostly relying on central servers and assuming that participants are <inline-formula><tex-math notation="LaTeX">$honest$</tex-math></inline-formula>-<inline-formula><tex-math notation="LaTeX">$and$</tex-math></inline-formula>-<inline-formula><tex-math notation="LaTeX">$curious$</tex-math></inline-formula> and proactive. However, in reality, workers may be unwilling to participate, and there may be malicious behavior among participants, thus harming the enthusiasm and interests of other participants. The central server has weaknesses such as single point of failure. To address above problems, we propose a blockchain-based framework for crowdsourcing with reputation and incentive. We first design a worker selection scheme to select credible and capable workers. We leverage reputation as a metric of workers&#x0027; credibility, which is calculated through the improved subjective logic model. Then we utilize contract theory to design incentive mechanisms to attract more workers, especially high-quality workers to participate. Experimental results show that our proposed method can detect and prevent malicious participants and resist malicious collusion when the proportion of malicious participants is no more than 1/3. And encourage more workers to actively, honestly and continuously participate in crowdsourcing.

Crowdsourcing Utilizing Subgroup Structure of Latent Factor Modeling

Crowdsourcing has emerged as an alternative solution for collecting large scale labels. However, the majority of recruited workers are not domain experts, so their contributed labels could be noisy. In this article, we propose a two-stage model to predict the true labels for multicategory classification tasks in crowdsourcing. In the first stage, we fit the observed labels with a latent factor model and incorporate subgroup structures for both tasks and workers through a multi-centroid grouping penalty. Group-specific rotations are introduced to align workers with different task categories to solve multicategory crowdsourcing tasks. In the second stage, we propose a concordance-based approach to identify high-quality worker subgroups who are relied upon to assign labels to tasks. In theory, we show the estimation consistency of the latent factors and the prediction consistency of the proposed method. The simulation studies show that the proposed method outperforms the existing competitive methods, assuming the subgroup structures within tasks and workers. We also demonstrate the application of the proposed method to real world problems and show its superiority. Supplementary materials for this article are available online.

On Dynamically Pricing Crowdsourcing Tasks

Crowdsourcing techniques have been extensively explored in the past decade, including task allocation, quality assessment, and so on. Most of professional crowdsourcing platforms adopt the fixed pricing scheme to offer a fixed price for crowd tasks. It is neither incentive for crowd workers to produce good performance, nor profitable for the requester to gain high utility with low budget. In this article, we study the problem of pricing crowdsourcing tasks with optional bonuses. We propose a dynamic pricing mechanism, named CrowdPricer for incentively delivering bonuses to the crowd workers of completing tasks, in addition to offering a base payment for completing a task. We leverage a deep time sequence model to learn the effect of bonuses on workers’ quality for crowd tasks. CrowdPricer makes decisions on whether to provide bonuses on workers, so as to maximize the requester’s utility in expectation. We present an efficient bonus delivery algorithm under the help of beam search technique, in order to efficiently solve the decision making problem. Extensive experiments using both a real crowdsourcing platform and simulations demonstrate that CrowdPricer yields the higher utility for the requester. It also obtains more correct crowd answers than the state-of-the-art pricing methods.