Crowdsourcing System Research Articles

An Exploratory Study of the Impact of Task Selection Strategies on Worker Performance in Crowdsourcing Microtasks

In microtask crowdsourcing systems like Amazon Mechanical Turk (AMT) and Appen Figure-Eight, workers often employ task selection strategies, completing sequences of tasks to maximize earnings. While previous literature has explored the effects of sequential tasks with varying complexities of the same type, there is a lack of knowledge on the consequences when multiple types of tasks with similar levels of difficulty are performed. This study examines the impact of sequences of three frequently employed task types, namely image classification, text classification, and surveys, on workers' engagement, accuracy, and perceived workloads. In addition, we analyze the influence of workers' personality traits on their strategies for selecting tasks. Our study, which involved 558 participants using AMT, found that engaging in sequences of distinct task types had a detrimental effect on classification task engagement and accuracy. It also increases the perceived task load and the worker's frustration. Nevertheless, the precise order of tasks does not significantly impact these results. Moreover, we showed a slight association between personality traits and the workers' selection strategy for the tasks. The results offered valuable knowledge for designing an efficient and inclusive crowdsourcing platform.

Design and evaluation of crowdsourcing platforms based on users’ confidence judgments

Crowdsourcing deals with solving problems by assigning them to a large number of non-experts called crowd using their spare time. In these systems, the final answer to the question is determined by summing up the votes obtained from the community. The popularity of these systems has increased by facilitating access for community members through mobile phones and the Internet. One of the issues raised in crowdsourcing is how to choose people and how to collect answers. Usually, users are separated based on their performance in a pre-test. Designing the pre-test for performance calculation is challenging; The pre-test questions should be selected to assess characteristics in individuals that are relevant to the main questions. One of the ways to increase the accuracy of crowdsourcing systems is by considering individuals’ cognitive characteristics and decision-making models to form a crowd and improve the estimation of their answer accuracy to questions. People can estimate the correctness of their responses while making a decision. The accuracy of this estimate is determined by a quantity called metacognition ability. Metacoginition is referred to the case where the confidence level is considered along with the answer to increase the accuracy of the solution. In this paper, by both mathematical and experimental analysis, we would answer the following question: Is it possible to improve the performance of a crowdsourcing system by understanding individuals’ metacognition and recording and utilizing users’ confidence in their answers?

Dynamic Operations of a Mobile Charging Crowdsourcing Platform

This paper investigates the operation of a novel electric vehicles (EVs) charging service mode, that is, crowdsourced mobile charging service for EVs, whereby a crowdsourcing platform is established to arrange suppliers (crowdsourced chargers) to deliver charging service to customers’ electric vehicles (parked EVs) at low-battery levels. From the platform operator’s perspective, we aim to determine the optimal operation strategies for mobile charging crowdsourcing platforms to achieve specific objectives. A mathematical modeling framework is developed to capture the interactions among supply, demand, and service operations in the crowdsourced mobile charging market. To design an efficient solution method to solve the formulated model, we first analyze the model properties by rigorously proving that a crucial variable set for operating the mobile charging crowdsourcing system includes charging price, commission control, and period-specific aggregate demand control. Besides, we provide both an equivalent condition and a necessary condition for checking the feasibility of these crucial variables. On top of this, we construct a search tree according to the operation periods in a day to solve the optimal operation strategies, wherein a nondominated principle is adopted as an accelerating technique in the searching process. The solution obtained from the proposed solution algorithm is proved to be sufficiently close to the actual global optimal solutions of the formulated model up to the resolution of the discretization scheme adopted. Numerical examples provide evidence verifying the model’s validity and the solution method’s efficiency. Overall, the research outcome of this work can offer service operators structured and valuable guidelines for operating mobile charging crowdsourcing platforms. Funding: This work was supported by the Singapore Ministry of Education [Grant RG124/21]. Supplemental Material: The online appendices are available at https://doi.org/10.1287/trsc.2023.0126 .

Revolutionizing machine learning: Blockchain-based crowdsourcing for transparent and fair labeled datasets supply

With the development of 5G/6G communication networks, the industrial Internet of Things (IIoT) industry has generated a massive amount of data, presenting opportunities for advancements in the field of machine learning. The core of machine learning, labeled datasets, requires qualities such as diversity, quantity, and quality. However, the current collection of training datasets is mostly centralized, relying on crowdsourcing systems as platforms. Hence, there are four open challenges existing: (1) traditional crowdsourcing systems are mostly based on centralized platform which often suffers single point of failure, mischief attacks, DDoS attacks and are easy to be remotely hijacked; (2) the transparency of traditional crowdsourcing systems cannot be guaranteed, which results in the unfairness in assigning task or rewarding worker; (3) the quality of labeled training dataset cannot be guaranteed as the workers in traditional crowdsourcing systems are professional or non-professional; (4) the privacy of labels cannot be preserved as the platform can also learn the labeled result. In this paper, we utilize web 3.0 technology to propose a blockchain-based labeled training dataset supply system which can simultaneously supply annotation service and labeled training dataset with the challenges above overcome. Meanwhile we design a privacy-preserved truth discovery suitable for categorical data by combining it with Software Guard Extensions (SGX). Furthermore, we design a fair rewards distribution mechanism which is based on reputation system and Shapley value. The two mechanisms above can ensure the quality of labeled training dataset. Finally, to demonstrate the practicability of our design, we implement a prototype of the system deployed on Fabric test network and conduct extensive simulations. Compared to applications on public chains, the throughput of our blockchain application can reach 53 times higher.

Towards Efficient and Deposit-Free Blockchain-Based Spatial Crowdsourcing

Spatial crowdsourcing leverages the widespread use of mobile devices to outsource tasks to a crowd of users based on their geographical location. Despite its growing popularity, current crowdsourcing systems often suffer from a lack of transparency, centralization, and other security issues. Blockchain technology has revolutionized this sector with its potential for decentralization, security, and transparency. However, existing blockchain-based crowdsourcing systems often overlook efficient task assignment mechanisms and expose users to potential losses due to the obligatory deposit payments to smart contracts, which might be vulnerable or untrustworthy. This paper proposes EDF-Crowd, an E fficient and D eposit- F ree blockchain-based spatial crowdsoucing framework, to address these challenges. EDF-Crowd introduces an efficient, customizable task assignment mechanism based on smart contracts, operating periodically and batch-wise. We also design a fair compensation mechanism to compensate users for the extra overhead caused by invoking certain smart contracts. More importantly, we propose a series of linkage protocols. By linking users’ back-and-forth actions, EDF-Crowd can regulate user behavior without requiring users to deposit. The versatility of EDF-Crowd also allows its application to generic crowdsourcing systems with minimal modifications. We implement EDF-Crowd based on the EOS blockchain. Extensive evaluations show that EDF-Crowd achieves high task assignment efficiency and low cost.

CROWDMATCH: Optimizing Crowdsourcing Matching through the Integration of Matching Theory and Coalition Games

This paper tackles the challenges inherent in crowdsourcing dynamics by introducing the CROWDMATCH mechanism. Aimed at enabling crowdworkers to strategically select suitable crowdsourcers while contributing information to crowdsourcing tasks, CROWDMATCH considers incentives, information availability and cost, and the decisions of fellow crowdworkers to model the utility functions for both the crowdworkers and the crowdsourcers. Specifically, the paper presents an initial Approximate CROWDMATCH mechanism grounded in matching theory principles, eliminating externalities from crowdworkers’ decisions and enabling each entity to maximize its utility. Subsequently, the Accurate CROWDMATCH mechanism is introduced, which is initiated by the outcome of the Approximate CROWDMATCH mechanism, and coalition game-theoretic principles are employed to refine the matching process by accounting for externalities. The paper’s contributions include the introduction of the CROWDMATCH system model, the development of both Approximate and Accurate CROWDMATCH mechanisms, and a demonstration of their superior performance through comprehensive simulation results. The mechanisms’ scalability in large-scale crowdsourcing systems and operational advantages are highlighted, distinguishing them from existing methods and highlighting their efficacy in empowering crowdworkers in crowdsourcer selection.

Aligning indoor human route descriptions to facilitate the use of crowdsourced indoor navigation systems

ABSTRACT Crowdsourced indoor route descriptions are increasingly used to provide user-friendly navigation services at low cost. In this study, we propose an approach for aligning indoor route descriptions and use the triangular model to visualize the aligned route descriptions. Linear route segments are represented as two-dimensional triangles, allowing navigation knowledge to be derived. The results show that the proposed approach can help to represent the hierarchical structure of multiple indoor route descriptions, identify the critical decision points that users frequently describe, and aggregate indoor route descriptions from different people to generate new route descriptions.

Decreased Seasonal Influenza Rates Detected in a Crowdsourced Influenza-Like Illness Surveillance System During the COVID-19 Pandemic: Prospective Cohort Study.

Seasonal respiratory viruses had lower incidence during their 2019-2020 and 2020-2021 seasons, which overlapped with the COVID-19 pandemic. The widespread implementation of precautionary measures to prevent transmission of SARS-CoV-2 has been seen to also mitigate transmission of seasonal influenza. The COVID-19 pandemic also led to changes in care seeking and access. Participatory surveillance systems have historically captured mild illnesses that are often missed by surveillance systems that rely on encounters with a health care provider for detection. This study aimed to assess if a crowdsourced syndromic surveillance system capable of detecting mild influenza-like illness (ILI) also captured the globally observed decrease in ILI in the 2019-2020 and 2020-2021 influenza seasons, concurrent with the COVID-19 pandemic. Flu Near You (FNY) is a web-based participatory syndromic surveillance system that allows participants in the United States to report their health information using a brief weekly survey. Reminder emails are sent to registered FNY participants to report on their symptoms and the symptoms of household members. Guest participants may also report. ILI was defined as fever and sore throat or fever and cough. ILI rates were determined as the number of ILI reports over the total number of reports and assessed for the 2016-2017, 2017-2018, 2018-2019, 2019-2020, and 2020-2021 influenza seasons. Baseline season (2016-2017, 2017-2018, and 2018-2019) rates were compared to the 2019-2020 and 2020-2021 influenza seasons. Self-reported influenza diagnosis and vaccination status were captured and assessed as the total number of reported events over the total number of reports submitted. CIs for all proportions were calculated via a 1-sample test of proportions. ILI was detected in 3.8% (32,239/848,878) of participants in the baseline seasons (2016-2019), 2.58% (7418/287,909) in the 2019-2020 season, and 0.27% (546/201,079) in the 2020-2021 season. Both influenza seasons that overlapped with the COVID-19 pandemic had lower ILI rates than the baseline seasons. ILI decline was observed during the months with widespread implementation of COVID-19 precautions, starting in February 2020. Self-reported influenza diagnoses decreased from early 2020 through the influenza season. Self-reported influenza positivity among ILI cases varied over the observed time period. Self-reported influenza vaccination rates in FNY were high across all observed seasons. A decrease in ILI was detected in the crowdsourced FNY surveillance system during the 2019-2020 and 2020-2021 influenza seasons, mirroring trends observed in other influenza surveillance systems. Specifically, the months within seasons that overlapped with widespread pandemic precautions showed decreases in ILI and confirmed influenza. Concerns persist regarding respiratory pathogens re-emerging with changes to COVID-19 guidelines. Traditional surveillance is subject to changes in health care behaviors. Systems like FNY are uniquely situated to detect disease across disease severity and care seeking, providing key insights during public health emergencies.

Decision aggregation with reliability propagation

People often make decisions differently, even when faced with the same decision-making scenario and objectives, due to their varying abilities to access, process, and comprehend information relevant to the decisions at hand. To reconcile these differing perspectives and arrive at a unified decision, various approaches such as crowd-sourced systems have been developed to tap into the collective intelligence embodied in the opinions from a group of individuals. The diversity of opinions is both cure and curse for the effective use of crowd-sourced intelligence. To unify crowd-sourced intelligence for a well-informed decision, we propose an algorithmic approach for decision aggregation that accurately quantifies the reliability of information from multiple sources. The key idea behind this approach is to model the propagation of reliability in decisions based on an ensemble of relevance graphs, where the optimization of both the reliability propagation and the graph ensemble are mutually reinforced. The propagated reliability can be used to aggregate intelligence from multiple sources and facilitate decision-making by leveraging various types of inter-correlations of information sources and the subjects of the information. Meanwhile, the optimized graph ensemble can retain the relevance structures with respect to the crowd-sourced intelligence. We evaluate our approach with large-scale data sets, and the results show that, when aggregating analysts’ recommendations in stock markets, our approach not only outperforms alternative methods, but also provides interesting insights into the reliability of the information.

A dynamic state sharding blockchain architecture for scalable and secure crowdsourcing systems

Currently, the crowdsourcing system has serious problems such as single point of failure of the server, leakage of user privacy, unfair arbitration, etc. By storing the interactions between workers, requesters, and crowdsourcing platforms in the form of transactions on the blockchain, these problems can be effectively addressed. However, the improvement in total computing power on the blockchain is difficult to provide positive feedback to the efficiency of transaction confirmation, thereby limiting the performance of crowdsourcing systems. On the other hand, the increasing amount of data in blockchain further increases the difficulty of nodes participating in consensus, affecting the security of crowdsourcing systems. To address the above problems, in this paper we design a blockchain architecture based on dynamic state sharding, called DSSBD. Firstly, we solve the problems caused by cross sharding transactions and reconfiguration in blockchain state sharding through graph segmentation and relay transactions. Then, we model the optimal block generation problem as a Markov decision process. By utilizing deep reinforcement learning, we can dynamically adjust the number of shards, block spacing, and block size. This approach helps improve both the throughput of the blockchain and the proportion of non-malicious nodes. Security analysis has proven that the proposed DSSBD can effectively resist attacks such as transaction atomic attacks, double spending attacks, sybil attacks, replay attacks, etc. The experimental results show that the crowdsourcing system with the proposed DSSBD has better performance in throughput, latency, balancing, cross-shard transaction proportion, and node reconfiguration proportion, etc., while ensuring security.

BackTrace: A Human-AI Collaborative Approach to Discovering Studio Backdrops in Historical Photographs

In historical photo research, the presence of painted backdrops have the potential to help identify subjects, photographers, locations, and events surrounding certain photographs. However, there are few dedicated tools or resources available to aid researchers in this largely manual task. In this paper, we propose BackTrace, a human-AI collaboration system that employs a three-step workflow to retrieve and organize historical photos with similar backdrops. BackTrace is a content-based image retrieval (CBIR) system powered by deep learning that allows for the iterative refinement of search results via user feedback. We evaluated BackTrace with mixed-methods evaluation and found that it successfully aided users in finding photos with similar backdrops and grouping them into collections. Finally, we discuss how our findings can be applied to other domains, as well as implications of deploying BackTrace as a crowdsourcing system.

Crowdsourced Clustering via Active Querying: Practical Algorithm with Theoretical Guarantees

We consider the problem of clustering n items into K disjoint clusters using noisy answers from crowdsourced workers to pairwise queries of the type: “Are items i and j from the same cluster?” We propose a novel, practical, simple, and computationally efficient active querying algorithm for crowdsourced clustering. Furthermore, our algorithm does not require knowledge of unknown problem parameters. We show that our algorithm succeeds in recovering the clusters when the crowdworkers provide answers with an error probability less than 1/2 and provide sample complexity bounds on the number of queries made by our algorithm to guarantee successful clustering. While the bounds depend on the error probabilities, the algorithm itself does not require this knowledge. In addition to the theoretical guarantee, we implement and deploy the proposed algorithm on a real crowdsourcing platform to characterize its performance in real-world settings. Based on both the theoretical and the empirical results, we observe that while the total number of queries made by the active clustering algorithm is order-wise better than random querying, the advantage applies most conspicuously when the datasets have small clusters. For datasets with large enough clusters, passive querying can often be more efficient in practice. Our observations and practically implementable active clustering algorithm can inform and aid the design of real-world crowdsourced clustering systems. We make the dataset collected through this work publicly available (and the code to run such experiments).

Identifying active retrogressive thaw slumps from ArcticDEM

Permafrost degradation in the Arctic is accelerating and affects northern communities, ecosystems, and global soil carbon storage. However, the extent, distribution, and rates of permafrost degradation in the pan-Arctic remain unknown, contributing to the challenges of mapping and monitoring it in a harsh and remote environment. We applied feature extraction, deep learning, and crowdsourcing to an open-access, high-resolution (2 m), and multi-temporal digital elevation models (i.e. ArcticDEM), to identify retrogressive thaw slumps (RTSs), a dynamic form of permafrost degradation widespread across the Arctic. Specifically, we (1) developed an automated pipeline to process approximately 200 TB of ArcticDEM data; (2) designed feature extractors (pixel-wise elevation differences, polygons of elevation reductions, lines of narrow-steep slopes, and RTS headwall lines) to identify slumps; (3) trained a super-efficient object detection algorithm based on deep learning (YOLOv4) and used it to locate RTSs from composite imagery derived from the ArcticDEM; (4) combined the extracted features and the bounding boxes output by YOLOv4 to obtain mapping results at a manageable level; and (5) developed a crowdsourcing system and invited volunteers to validate and refine the results. The final map included 2494 RTSs (actively expanding during ArcticDEM observation) across the Arctic. The results also show that (1) it is necessary to combine the extracted features and deep learning to remove many false positives in the scenario with limited training data, but large regions to map; (2) some hotspots of RTSs need further and detailed investigation, including an RTS cluster in Greenland; (3) the crowdsourcing system is useful for the validation of a large dataset, but responses to this work were limited, possibly because RTSs are a quite specific topic that not many people are familiar with. The results likely miss many RTSs due to the limitation in the method for identifying RTSs and uncertainties as well as short observation periods of ArcticDEM at many locations. This study provides data and serves as a starting point to develop a global inventory and better understand permafrost thaw in the pan-Arctic using very high resolution remote sensing.

A Triple Real-Time Trajectory Privacy Protection Mechanism Based on Edge Computing and Blockchain in Mobile Crowdsourcing

With the rapid development of the Internet of Things (IoT) and the rapid popularization of 5G networks, the data that needs to be processed in Mobile Crowdsourcing (MCS) system is increasing every day. Traditional cloud computing can no longer meet the needs of crowdsourcing for real-time data and processing efficiency, thus, edge computing was born. Edge computing can be calculated at the edge of network so that greatly improve the efficiency and real-time performance of data processing. In addition, most of the existing privacy protection technologies are based on the trusted third parties. Therefore, in view of the semi-trustworthiness of edge servers and the transparency of blockchain, this paper proposes a triple real-time trajectory privacy protection mechanism (T-LGEB) based on edge computing and blockchain. Through combining the localized differential privacy and multiple probability extension mechanism, the T-LGEB mechanism is proposed to send the requests and data to the edge server in this paper. Then, through the spatio-temporal dynamic pseudonym mechanism proposed in the paper, the entire trajectory of task participants is divided into multiple unrelated trajectory segments with different pseudonymous identities in order to protect the trajectory privacy of task participants while ensuring high data availability and real-time data. Through a large number of experiments and comparative analysis on multiple real data sets, the proposed T-LGEB has extremely high privacy protection capabilities and data availability, and the resource consumption caused is relatively low.

A crowdsourced co-modality transportation system integrating passenger and freight

Co-modality transportation aims to integrate passenger and freight flows into a single system by sharing public transport between passengers and freight. This paper proposes a crowdsourced co-modality transportation system (CCTS) for enhancing the practical implementation of co-modality transportation in urban areas. Firstly, empirical investigations are conducted, including semi-structured interviews with industry practitioners and online surveys with passengers, to collect system design considerations. Secondly, an overview of the crowdsourced co-modality management platform (CCMP) is presented by highlighting three key components: co-modality services, information infrastructure, and decision support system. Thirdly, we formulate the parcel assignment problem in the CCTS and examine the factors that influence the performance of the CCTS. The empirical investigations suggest that crowdsourced passengers care about the potential risks and the degree of operation difficulty in crowdsourced activities but do not care about privacy issues. In addition, the experimental studies show that crowdsourced co-modality is more feasible and promising if more passengers are willing to provide crowdsourcing services.

Fair Allocation in Crowd-Sourced Systems

In this paper, we address the problem of fair sharing of the total value of a crowd-sourced network system between major participants (founders) and minor participants (crowd) using cooperative game theory. We use the framework of a Shapley allocation which is regarded as a fundamental method of computing the fair share of all participants in a cooperative game when the values of all possible coalitions could be quantified. To quantify the value of all coalitions, we define a class of value functions for crowd-sourced systems which capture the contributions of the founders and the crowd plausibly and derive closed-form expressions for Shapley allocations to both. These value functions are defined for different scenarios, such as the presence of oligopolies or geographic spread of the crowd, taking network effects, including Metcalfe’s law, into account. A key result we obtain is that under quite general conditions, the crowd participants are collectively owed a share between 12 and 23 of the total value of the crowd-sourced system. We close with an empirical analysis demonstrating the consistency of our results with the compensation offered to the crowd participants in some public internet content sharing companies.

A Spatio-Temporal Task Allocation Model in Mobile Crowdsensing Based on Knowledge Graph

With the increasing popularity of wireless networks and the development of smart cities, the Mobile Crowdsourcing System (MCS) has emerged as a framework for automatically assigning spatiotemporal tasks to workers. The study of mobile crowdsourcing makes a valuable research contribution to community service and urban route planning. However, previous algorithms have faced challenges in effectively addressing task allocation issues with massive spatial data. In this paper, we propose a novel solution to the spatiotemporal task allocation problem using a knowledge graph. Firstly, we construct a robust spatiotemporal knowledge graph (STKG) and employ a knowledge graph embedding algorithm to learn the representations of nodes and edges. Next, we utilize these representations to build a task transition graph, which is a weighted and learning-based graph that highlights important neighbors for each task. We then apply a simplified Graph Convolutional Network (GCN) and an RNN-based model to enhance task representations and capture sequential transition patterns on the task transition graph. Furthermore, we design a similarity function to facilitate personalized task allocation. Through experimental results, we demonstrate that our solution achieves higher accuracy compared to existing approaches when tested on three real datasets. These research findings are significant as they contribute to an 18.01% improvement in spatiotemporal task allocation accuracy.

TTAF: A two-tier task assignment framework for cooperative unit-based crowdsourcing systems

Traditional task assignment follows a direct recruitment model in which requesters recruit and select workers to complete tasks. Because of the unclear division of roles and the diversity of each role’s mission, this model is neither efficient nor scalable. This paper introduces the concept of cooperative unit (CU), in which workers are organized into cooperative units, whose proxies bid for tasks from requesters based on worker characteristics. However, because of the decentralization of task assignment, quality control is complicated, and the benefits of different roles must be balanced. As a result, we propose a novel two-tier task assignment framework (TTAF) that produces high-quality results while striking the appropriate balance between requesters, CUs, and workers. We first propose a vector-based expertise representation model that evaluates workers’ expertise based on previous answers. Then, we devise a higher-tier task assignment between tasks and CUs that maximizes answer quality while staying within budget. The quality of the answers is ensured by aspects such as keyword coverage, overall expertise, and the number of workers. We also devise lower-tier task assignment, which evenly distributes tasks among workers such that more workers have the opportunity to perform tasks. The extensive evaluation shows that our proposed approaches achieve promising results.

Hiring a Team From Social Network: Incentive Mechanism Design for Two-Tiered Social Mobile Crowdsourcing

Mobile crowdsourcing has become an efficient paradigm for performing large scale tasks. The incentive mechanism is important for the mobile crowdsourcing system to stimulate participants, and to achieve good service quality. In this paper, we focus on solving the insufficient participation problem for the budget constrained online crowdsourcing system. We present a two-tiered social crowdsourcing architecture, which can enable the selected registered users to recruit their social neighbors by diffusing the tasks to their social circles. We present three system models for two-tiered social crowdsourcing system based on the arrival modes of registered users and social neighbors: offline model, semi-online model, and full-online model. We consider the tasks are associated with different end times. We present an incentive mechanism for each of three system models. Through both rigorous theoretical analysis and extensive simulations, we demonstrate that the proposed incentive mechanisms achieve computational efficiency, individual rationality, budget feasibility, cost truthfulness, and time truthfulness. We further show that our incentive mechanisms for semi-online model and full-online model can obtain averagely 51.1% and 39.7% value of approximate optimal untruthful offline algorithm, respectively.

Eliciting Joint Truthful Answers and Profiles From Strategic Workers in Mobile Crowdsourcing Systems

Due to the openness of mobile crowdsourcing, workers may yield low-quality task answers. To alleviate this problem, substantial efforts have been devoted to eliciting truthful data from workers. On the other hand, to facilitate task assignment, workers are required to upload to the platform their profiles, such as locations and expertise. Therefore, task assignment outcomes and thus mobile crowdsourcing service accuracy are subject to the quality of worker's self-reported profiles. In this paper, we leverage incentive design to motivate workers to honestly reveal both task answers and their profiles. The challenge is to design one incentive payment for truth elicitation in two kinds of submissions. For this, we first derive the sufficient and necessary conditions for answer truthfulness and profile truthfulness separately. We then construct an incentive optimization problem that incorporates these conditions as constraints. Its optimal solution lists the payment to each worker that elicits answers and profiles jointly. Our proposed mechanism, with formally proved bounded approximation ratio, ensures that truth-telling is a Bayesian Nash equilibrium. We prototype the mechanism and conduct a series of experiments that involve 30 volunteers to validate the efficacy and efficiency of the proposed mechanism