Crowdsourcing Task Research Articles

Bidirectional k-nearest neighbor spatial crowdsourcing allocation protocol based on edge computing.

Spatial crowdsourcing refers to the allocation of crowdsourcing workers to each task based on location information. K-nearest neighbor technology has been widely applied in crowdsourcing applications for crowdsourcing allocation. However, there are still several issues need to be stressed. Most of the existing spatial crowdsourcing allocation schemes operate on a centralized framework, resulting in low efficiency of crowdsourcing allocation. In addition, these spatial crowdsourcing allocation schemes are one-way allocation, that is, the suitable matching objects for each task can be queried from the set of crowdsourcing workers, but cannot query in reverse. In this article, a bidirectional k-nearest neighbor spatial crowdsourcing allocation protocol based on edge computing (BKNN-CAP) is proposed. Firstly, a spatial crowdsourcing task allocation framework based on edge computing (SCTAFEC) is established, which can offload all tasks to edge nodes in edge computing layer to realize parallel processing of spatio-temporal queries. Secondly, the positive k-nearest neighbor spatio-temporal query algorithm (PKNN) and reverse k-nearest neighbor spatio-temporal query algorithm (RKNN) are proposed to make the task publishers and crowdsourcing workers conduct two-way query. In addition, a road network distance calculation method is proposed to improve the accuracy of Euclidean distance in spatial query scenarios. Experimental results show that the proposed protocol has less time cost and higher matching success rate compared with other ones.

A Novel Crowdsourcing Task Recommendation Method Considering Workers’ Fuzzy Expectations: A Case of ZBJ.COM

Crowdsourcing recommendation is necessary to surmount information overload and assist workers to identify tasks of interest efficiently. However, existing studies mainly concentrate on investigating workers’ qualifications or requirements, neglecting their fuzzy expectations and psychological behavior. Besides, workers can only passively accept the results and do not have any right to change them even if they are not satisfactory. As workers will pay different attention to task aspects, we regard the crowdsourcing recommendation as a multi-criteria decision-making (MCDM) problem. In this paper, we propose an MCDM-based method for automated and interactive crowdsourcing task recommendations while comprehensively considering workers’ fuzzy expectations and psychological behavior. Given a large volume of available tasks on a crowdsourcing platform, the requirement of massive human efforts in traditional MCDM is not applicable in such situation. Therefore, we formulate a set of quantitative task evaluation criteria for automated task assessments based on the attributes that have the most impact on workers’ participation behavior. Further, to capture workers’ fuzziness in expectations, an intuitionistic fuzzy 2-tuple linguistic (IF2L) term is introduced, which are further leveraged as reference points to evaluate their “gains” and “losses” based on the prospect theory. Finally, the D-S evidence theory is adopted to gather workers’ uncertain prospect values. A case of ZBJ.COM, a popular crowdsourcing platform in China, shows that the proposed method is efficient and effective in recommending suitable and personalized tasks to workers.

How leaderboard positions shape our motivation: the impact of competence satisfaction and competence frustration on motivation in a gamified crowdsourcing task

PurposeGamification is a booming motivational approach in information systems. Leaderboards play a key role in gamification; however, there are mixed findings regarding the heterogeneous motivational impacts of leaderboard positions. This study aims to clarify the motivational effects of high and low leaderboard positions by assembling diverse behavioral measures and self-reports. The measures used in this study shed a light on the quantitative and qualitative dynamics of motivation facilitated by leaderboard positions. The authors inspect motivation in relation to satisfaction and frustration of competence need.Design/methodology/approachThe authors conducted an online experiment set in a crowdsourcing context, asking the participants to compete in an image tagging game. Participants' leaderboard positions were manipulated to be either high or low for five consecutive rounds. The number of clicks, tags, duration of tagging and persistence on the task were measured as indicators of motivation.FindingsHigh ranks on leaderboards induced complacent behaviors choosing easy ways to maintain their positions, while low ranks led the participants to stick to the right process of the task with intensified motivation round after round. However, neither of the motivations seemed to be of intrinsic nature.Originality/valueThe present study provides conclusive evidence on the varying motivational impact of leaderboard positions. The authors also demonstrate how the “needs-as-motive” model (Sheldon and Gunz, 2009) applies to gamification. Its implications in self-determination theory and gamification literature are discussed.

PMATE: A Privacy-Preserving Map Retrieval Task Assignment Scheme in Spatial Crowdsourcing

Spatial crowdsourcing (SC) task assignment is to find the optimal worker for the task from abundant alternative workers based on the information of the task and workers, such as location, time, and ability. This information will undoubtedly reveal the privacy of both the task and workers. The disclosure of private information is a crucial issue constraining the development of SC. To this end, various privacy-preserving task assignments have been proposed to protect privacy by obfuscating or encrypting information. Fuzzy processing will limit matching accuracy, while encrypted information will increase the time cost of data computation. Therefore, this paper proposes a privacy-preserving map retrieval task assignment scheme (pMATE), which can divide the map and accurately retrieve the optimal workers according to this division. In pMATE, relevant information about tasks and workers is encrypted, and neighboring workers are searched based on the task presence partition. The task location can also be hidden in that partition. Partitioned retrieval reduces the amount of encrypted data needed to be matched. Furthermore, to reduce the problem of multiple communications during encrypted data comparison, we propose the Find MinNumber (FMN) algorithm, which can determine the optimal worker or top-k optimal workers need only two communications. Experimental evaluations of real-world data show that pMATE is efficient and accurate.

Linguistic Profile of a Text and Human Ratings of Writing Quality: a Case Study on Italian L1 Learner Essays

This paper presents a study based on the linguistic profiling methodology to explore the relationship between the linguistic structure of a text and how it is perceived in terms of writing quality by humans. The approach is tested on a selection of Italian L1 learners essays, which were taken from a larger longitudinal corpus of essays written by Italian L1 students enrolled in the first and second year of lower secondary school. Human ratings of writing quality by Italian native speakers were collected through a crowdsourcing task, in which annotators were asked to read pairs of essays and rated which one they believed to be better written. By analyzing these ratings, the study identifies a variety of linguistic phenomena spanning across distinct levels of linguistic description that distinguish the essays considered as ‘winners’ and evaluates the impact of students’ errors on the human perception of writing quality.

Deep learning-based recommendation method for top-K tasks in software crowdsourcing systems

The task personalized recommendation problems in software crowdsourcing systems have unique characteristics, i.e., large task flow, high task complexity, long development cycle, winning task competitions, professional ability requirements, etc. However, existing software crowdsourcing recommendation mechanisms do not consider the contextual information of crowdsourcing tasks. In particular, the effects of crowdsourcing workers' interest changes and capability constraints on task selection are usually ignored. Therefore, this study proposes a new worker capability-correction long- and short-term attention network (CLSAN) recommendation framework. Firstly, explicit and implicit features are obtained from the feature data of crowdsourcing workers and historical crowdsourcing tasks. The long-term and short-term feature layers are extracted by adding LSTM to the historical tasks; the attention weight of user preference is calculated by integrating the attention mechanism to obtain each user's personalized preference. Secondly, this study considers that the capabilities of the worker need to be matched with the skills required for the software task. We design a worker capability correction model that uses Word2Vec software to obtain competency similarities between crowdsourcing workers and tasks, thereby determining the priority of tasks that satisfy the worker's capability. The experimental results show that CLSAN can accurately evaluate the changes of interest preferences of crowdsourcing workers, and effectively improve the quality and efficiency of crowdsourcing recommendations.

Collective Assignment of Complex Crowdsourcing Tasks Based on the KM Algorithm

Collective Assignment of Complex Crowdsourcing Tasks Based on the KM Algorithm

P&T-Inf: A Result Inference Method for Context-Sensitive Tasks in Crowdsourcing

Context-Sensitive Task (CST) is a complex task type in crowdsourcing, such as handwriting recognition, route plan, and audio transcription. The current result inference algorithms can perform well in simple crowdsourcing tasks, but cannot obtain high-quality inference results for CSTs. The conventional method to solve CSTs is to divide a CST into multiple independent simple subtasks for crowdsourcing, but this method ignores the context correlation among subtasks and reduces the quality of result inference. To solve this problem, we propose a result inference algorithm based on the Partially ordered set and Tree augmented naive Bayes Infer (P&T-Inf) for CSTs. Firstly, we screen the candidate results of context-sensitive tasks based on the partially ordered set. If there are parallel candidate sets, the conditional mutual information among subtasks containing context information in external knowledge (such as Google n-gram corpus, American Contemporary English corpus, etc.) will be calculated. Combined with the tree augmented naive (TAN) Bayes model, the maximum weighted spanning tree is used to model the dependencies among subtasks in each CST. We collect two crowdsourcing datasets of handwriting recognition tasks and audio transcription tasks from the real crowdsourcing platform. The experimental results show that our approach improves the quality of result inference in CSTs and reduces the time cost compared with the latest methods.

Batch Crowdsourcing for Complex Tasks Based on Distributed Team Formation in E-Markets

Team formation has been extensively studied for complex task crowdsourcing in E-markets, in which a set of workers are hired to form a team to complete a complex task collaboratively. However, existing studies have two typical drawbacks: 1) each team is created for only one task, which may be costly and cannot accommodate crowdsourcing markets with a large number of tasks; and 2) most existing studies form teams in a centralized manner by the requesters, which may place a heavy burden on requesters. In fact, we observe that many complex tasks at real-world crowdsourcing platforms have similar skill requirements and workers are often connected through social networks. Therefore, this paper explores distributed team formation-based batch crowdsourcing for complex tasks to address the drawbacks in existing studies, in which similar tasks can be addressed in a batch to reduce computational costs and workers can self-organize through their social networks to form teams. To solve such an NP-hard problem, this paper presents two approaches: one is to form a fixed team for all tasks in the batch; the other is to form a basic team that can be dynamically adjusted for each task in the batch. In comparison, the former approach has lower computational complexity but the latter approach performs better in reducing the total payments by requesters. With the experiments on a real-world dataset comparing with previous benchmark approaches, it is shown that the presented approaches have better performance in saving the costs of forming teams, payments by requesters, and communication among team members; moreover, the presented approaches have higher success rate of tasks and much better scalability.

Exploring Collection of Sign Language Videos through Crowdsourcing

Inadequate sign language data currently impedes advancement of sign language ML and AI. Training on existing datasets results in limited models due to small size, and lack of diverse signers in real-world settings. Complex labeling problems in particular often limit scale. In this work, we explore the potential for crowdsourcing to help overcome these barriers. To do this, we ran a user study with exploratory crowdsourcing tasks designed to support scalability: 1) to record videos of specific content -- thereby enabling automatic, scalable labeling -- and 2) to perform quality control checks for execution consistency -- further reducing post-processing requirements. We also provided workers with a searchable view of the crowdsourced dataset, to boost engagement and transparency and align with Deaf community values. Our user study included 29 participants using our exploratory tasks to record 1906 videos and perform 2331 quality control checks. Our results suggest that a crowd of signers may be able to generate high-quality recordings and perform reliable quality control, and that the signing community values visibility into the resulting dataset.

Reputation-Based Blockchain for Spatial Crowdsourcing in Vehicular Networks

The sharing of high-quality traffic information plays a crucial role in enhancing the driving experience and safety performance for vehicular networks, especially in the development of electric vehicles (EVs). The crowdsourcing-based real-time navigation of charging piles is characterized by low delay and high accuracy. However, due to the lack of an effective incentive mechanism and the resource-consuming bottleneck of sharing real-time road conditions, methods to recruit or motivate more EVs to provide high-quality information gathering has attracted considerable interest. In this paper, we first introduce a blockchain platform, where EVs act as the blockchain nodes, and a reputation-based incentive mechanism for vehicular networks. The reputations of blockchain nodes are calculated according to their historical behavior and interactions. Further, we design and implement algorithms for updating honest-behavior-based reputation as well as for screening low-reputation miners, to optimize the profits of miners and address spatial crowdsourcing tasks for sharing information on road conditions. The experimental results show that the proposed reputation-based incentive method can improve the reputation and profits of vehicle users and ensure data timeliness and reliability.

Research on incentive strategy based on service quality in spatial crowdsourcing task allocation

In order to further improve the enthusiasm of spatial crowdsourcing workers, considering the service quality of workers, different incentive strategies are proposed and tasks are assigned. Firstly, the incentive model is constructed from the unit time revenue of task and online idle time, and the evaluation function of the evaluation model is constructed; Secondly, the task allocation is transformed into a combinatorial optimization problem by delay matching, and an improved glowworm swarm algorithm is proposed to solve the problem by discrete coding, introducing six kinds of mobile modes, adaptive probability matching and infeasible solution processing; Finally, the algorithm is used to solve the task allocation. The experimental results show that compared with the travel cost minimization strategy and random allocation strategy, the positive incentive index of the proposed strategy is improved by 11.79% and 14.60% respectively, and the fair incentive index is improved by 0.83% and 0.22% respectively, which can effectively improve the positive incentive range and incentive fairness of workers.

Assuring quality and waiting time in real-time spatial crowdsourcing

With the rapid development of mobile devices, spatial crowdsourcing has become an important way to collect data. Task assignment is an important aspect of spatial crowdsourcing. How to improve the quality of the results and decrease the travel distance has been extensively studied in recent years. Existing studies often assume that moving speed is constant or real-time road network information is known. In this paper, the travel time is predicted based on historical data. A framework for time-prediction-based task assignment approach in spatial crowdsourcing (TP-TASC) is proposed. Firstly, a prediction model based on the light gradient boosting machine (LightGBM) is used to predict the travel time of workers with the consideration of the spatial features, the temporal features, and the climate features. Secondly, a heuristic algorithm is proposed to assign the spatial crowdsourcing tasks to appropriate workers. When a task is assigned to a worker, the payment of the worker is also determined automatically. Finally, the simulation experiments based on a real-world taxi-hailing dataset show that the proposed method can not only effectively minimize the task requesters’ waiting time, but also maximize the results’ quality.

A Graph Matching Model for Designer Team Selection for Collaborative Design Crowdsourcing Tasks in Social Manufacturing

In order to find a suitable designer team for the collaborative design crowdsourcing task of a product, we consider the matching problem between collaborative design crowdsourcing task network graph and the designer network graph. Due to the difference in the nodes and edges of the two types of graphs, we propose a graph matching model based on a similar structure. The model first uses the Graph Convolutional Network to extract features of the graph structure to obtain the node-level embeddings. Secondly, an attention mechanism considering the differences in the importance of different nodes in the graph assigns different weights to different nodes to aggregate node-level embeddings into graph-level embeddings. Finally, the graph-level embeddings of the two graphs to be matched are input into a multi-layer fully connected neural network to obtain the similarity score of the graph pair after they are obtained from the concat operation. We compare our model with the basic model based on four evaluation metrics in two datasets. The experimental results show that our model can more accurately find graph pairs based on a similar structure. The crankshaft linkage mechanism produced by the enterprise is taken as an example to verify the practicality and applicability of our model and method.

STUDY ON THE RELATIONSHIP BETWEEN CROWDSOURCING TASK DESCRIPTION AND WINNING THE BID

STUDY ON THE RELATIONSHIP BETWEEN CROWDSOURCING TASK DESCRIPTION AND WINNING THE BID

Crowdsourcing Map-Using and Map-Generating Tasks into OpenStreetMap

This article describes research to identify map-using and map-generating tasks in crowdsourcing. As a result, we identified a set of tasks from some users’ tests designed as two experiments. In Experiment 1, we interviewed eight individuals (in person) and seventy-eight individuals—by an online survey—within Experiment 2. They have different backgrounds and map-reading skills. We selected a geoinformation crowdsourcing platform, OpenStreetMap, for applying the tests. As exploratory research, the results revealed an initial set of eighty-two map-using and map-generating tasks that individuals perform while interacting with OpenStreetMap. In general, the tests’ results allowed us to assess the changes triggered by the technological developments in map-use research. Consequently, we conclude that we should complement the traditional list of map-reading tasks in this new context, where individuals can simultaneously use and produce maps. This new possibility of using and generating geoinformation could bring some progress for designing map use tests and understanding the technological advances in Cartography.

Blockchain-based multi-skill mobile crowdsourcing services

With the boom in 5G technology, mobile spatial crowdsourcing has shown great dynamism in industrial mobile communications and edge computing node management. But the traditional crowdsourcing system is not advanced enough to adapt to the new environment. Typically, traditional crowdsourcing workflow is hosted by a centralized crowdsourcing platform. However, the centralized crowdsourcing platform faces the following problems: (1) single point of failure, (2) user privacy leakage, (3) subjective arbitration, (4) additional service fee, and (5) non-transparent task assignment process. To improve those problems, we replaced the centralized crowdsourcing platform with a decentralized blockchain infrastructure. And we analyzed the challenge problems of multi-skilled spatial crowdsourcing tasks in the blockchain crowdsourcing system. In addition, a crowdsourcing task allocation algorithm has been proposed, which implements a transparent task distribution process and can adapt to the computing-constrained environment on the blockchain. Compared with the TSWCrowd blockchain-based crowdsourcing model, our system has a higher task allocation rate under the same conditions. And the experimental result shows our work has good economic feasibility, which decentralizes the crowdsourcing process and significantly reduces the additional consumption of the crowdsourcing process.

Dynamic incentive mechanism in mobile crowdsourcing networks by combining reputation and contract theory

By utilizing the mobile terminals’ sensing and computing capabilities, mobile crowdsourcing network is considered to be a promising technology to support the various large-scale sensing applications. However, considering the limited resources and security issue, mobile users may be unwilling to participate in crowdsourcing without any incentive. In this work, by combining reputation and contract theory, a dynamic long-term incentive mechanism is proposed to attract the mobile users to participate in mobile crowdsourcing networks. A two-period dynamic contract is first investigated to deal with the asymmetric information problem in the crowdsourcing tasks. Reputation strategy is then introduced to further attract the mobile users to complete the long-term crowdsourcing tasks. The optimal contracts are designed to obtain the maximum expected utility of service provider with reputation strategy and without reputation strategy, respectively. Simulation results demonstrate that the long-term crowdsourcing tasks can be guaranteed by combining the contract’s explicit incentive with the reputation’s implicit incentive. The incentive mechanism can gain a higher expected utility, the more implicit reputation effect factor.

Effect of crowdsourcing work characteristics on perceived work effort in competitive crowdsourcing markets

PurposeThis study explored whether crowdsourcing work characteristics are associated with perceived work effort in competitive crowdsourcing markets. The study also investigated the important contextual variables and internal mechanisms related to perceived work effort.Design/methodology/approachA questionnaire was posted as a crowdsourcing task on China's Time Fortune website. Data from 231 valid questionnaires were analyzed using SmartPLS 3.FindingsCrowdsourcing workers' intrinsic and extrinsic motivations were significantly and positively correlated with their perceived work effort. Task autonomy and feedback were significantly and positively correlated with intrinsic motivation. Skill variety, task significance, task identity, and task clarity had no significant correlations with intrinsic motivation. However, task clarity was significantly and positively correlated with perceived work effort. Moreover, the relationship between workers' trust in task requesters and perceived work effort was fully mediated by intrinsic motivation.Originality/valueThis study extended the job characteristic model into the virtual competitive crowdsourcing market. The authors verified the relationship between task clarity/trust in task requesters and workers' motivation and perceived work effort.

A Fine-Grain Batching-Based Task Allocation Algorithm for Spatial Crowdsourcing

Task allocation is a critical issue of spatial crowdsourcing. Although the batching strategy performs better than the real-time matching mode, it still has the following two drawbacks: (1) Because the granularity of the batch size set obtained by batching is too coarse, it will result in poor matching accuracy. However, roughly designing the batch size for all possible delays will result in a large computational overhead. (2) Ignoring non-stationary factors will lead to a change in optimal batch size that cannot be found as soon as possible. Therefore, this paper proposes a fine-grained, batching-based task allocation algorithm (FGBTA), considering non-stationary setting. In the batch method, the algorithm first uses variable step size to allow for fine-grained exploration within the predicted value given by the multi-armed bandit (MAB) algorithm and uses the results of pseudo-matching to calculate the batch utility. Then, the batch size with higher utility is selected, and the exact maximum weight matching algorithm is used to obtain the allocation result within the batch. In order to cope with the non-stationary changes, we use the sliding window (SW) method to retain the latest batch utility and discard the historical information that is too far away, so as to finally achieve refined batching and adapt to temporal changes. In addition, we also take into account the benefits of requesters, workers, and the platform. Experiments on real data and synthetic data show that this method can accomplish the task assignment of spatial crowdsourcing effectively and can adapt to the non-stationary setting as soon as possible. This paper mainly focuses on the spatial crowdsourcing task of ride-hailing.