Efficient Crowdsourcing Research Articles

Unveiling the Inter-Related Preferences of Crowdworkers: Implications for Personalized and Flexible Platform Design

Crowdsourcing platforms have traditionally been designed with a focus on workstation interfaces, restricting the flexibility that crowdworkers need. Recognizing this limitation and the need for more adaptable platforms, prior research has highlighted the diverse work processes of crowdworkers, influenced by factors such as device type and work stage. However, these variables have largely been studied in isolation. Our study is the first to explore the interconnected variabilities among these factors within the crowdwork community. Through a survey involving 150 Amazon Mechanical Turk crowdworkers, we uncovered three distinct groups characterized by their interrelated variabilities in key work aspects. The largest group exhibits a reliance on traditional devices, showing limited interest in integrating smartphones and tablets into their work routines. The second-largest group also primarily uses traditional devices but expresses a desire for supportive tools and scripts that enhance productivity across all devices, particularly smartphones and tablets. The smallest group actively uses and strongly prefers non-workstation devices, especially smartphones and tablets, for their crowdworking activities. We translate our findings into design insights for platform developers, discussing the implications for creating more personalized, flexible, and efficient crowdsourcing environments. Additionally, we highlight the unique work practices of these crowdworker clusters, offering a contrast to those of more traditional and established worker groups.

PanorAMS: Automatic Annotation for Detecting Objects in Urban Context

Large collections of geo-referenced panoramic images are freely available for cities across the globe, as well as detailed maps with location and meta-data on a great variety of urban objects. They provide a potentially rich source of information on urban objects, but manual annotation for object detection is expensive, laborious and challenging. Can we utilize such multimedia sources to automatically annotate street level images as an inexpensive alternative to manual labeling? With the PanorAMS framework we introduce a method to automatically generate bounding box annotations for panoramic images based on urban context information. Following this method, we acquire large-scale, albeit noisy, annotations for an urban dataset solely from open data sources in a fast and automatic manner. The dataset covers the City of Amsterdam and includes over 14 million noisy bounding box annotations of 22 object categories present in 771,299 panoramic images. For many objects further fine-grained information is available, obtained from geospatial meta-data, such as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">building value</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">function</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">average surface area</i> . Such information would have been difficult, if not impossible, to acquire via manual labeling based on the image alone. For detailed evaluation, we introduce an efficient crowdsourcing protocol for bounding box annotations in panoramic images, which we deploy to acquire 147,075 ground-truth object annotations for a subset of 7,348 images, the PanorAMS-clean dataset. For our PanorAMS-noisy dataset, we provide an extensive analysis of the noise and how different types of noise affect image classification and object detection performance. We make both datasets, PanorAMS-noisy and PanorAMS-clean, benchmarks and tools presented in this paper openly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://panorams.inskegroenen.nl/</uri> .

A Participatory Approach to Exploring Nexus Challenges: A Case Study on the Pinios River Basin, Greece

The conservation of natural resources is indispensable for supporting sustainable development. Water–Energy–Food–Ecosystem (WEFE) nexus management is widely acknowledged as the utmost priority of our time. Considering the specificities of the different nexus sectors, conflicting views of sustainability may arise from different stakeholders. This paper presents and analyses an effective stakeholders’ participation strategy aiming at identifying WEFE challenges in the Pinios River Basin (PRB), the most agriculturally productive and natural resource-stressed basin of Greece. The presented methodology was developed in the context of a Learning and Action Alliance (LAA) scheme adopted by the H2020 REXUS project, intended to support resilient nexus system management with the engagement of stakeholders. The proposed comprehensive methodology comprises multiple phases, including “framing”, “mapping”, “involvement”, “co-production”, and “sharing” phases. The involvement phase is the most interactive one, including the organisation and outputs of efficient crowdsourcing theme sessions organised in the framework of the first REXUS PRB stakeholders’ workshop. This paper illustrates and analyses stakeholders’ perceptions regarding nexus status in the PRB and reveals the most critical challenges in the pilot basin, along with their interdependencies and correlations. The determination of nexus challenges insights by stakeholders could significantly contribute to redefining policies so that they align with sustainable development aims. This methodology is proposed to form the baseline strategy in stakeholders’ engagement for future nexus management studies.

PrivBCS: a privacy-preserving and efficient crowdsourcing system with fine-grained worker selection based on blockchain

Crowdsourcing takes advantage of human intelligence to solve complex problems that computers cannot handle. People actively participate in computational tasks for rewards, especially those that are relatively simple for humans but challenging for computers. However, most traditional crowdsourcing systems must rely on a central organisation to handle the work associated with crowdsourcing. This may lead to privacy and security issues such as information leakage, single point of attack and unfair judgement. Meanwhile, the high service fees of existing central crowdsourcing platforms will reduce people's motivation to participate in crowdsourcing tasks. Consequently, PrivBCS is proposed in this paper, a blockchain-based decentralised crowdsourcing system that does not rely on third-party agencies and requires only low transaction fees. In this paper, PrivBCS protects the confidentiality of crowdsourcing data and the identity privacy of participants by cryptographic means, such as Paillier, El-Gamal and CPABE. It selects the most suitable worker to perform tasks by setting access control policies on task data and matching workers' attributes while ensuring the anonymity of workers' identity. Considering that the data volume of crowdsourcing tasks may be large and there is a certain upper limit to the block capacity of the blockchain, an on-chain and off-chain collaborative storage scheme is proposed, which can increase the throughput of the blockchain system and improve its scalability to a certain extent. In PrivBCS, we implement the transaction process of crowdsourcing tasks through smart contracts. The experimental results and security analysis in this paper demonstrate the effectiveness, efficiency, and feasibility of PrivBCS.

Efficient crowdsourcing of crowd-generated microtasks.

Allowing members of the crowd to propose novel microtasks for one another is an effective way to combine the efficiencies of traditional microtask work with the inventiveness and hypothesis generation potential of human workers. However, microtask proposal leads to a growing set of tasks that may overwhelm limited crowdsourcer resources. Crowdsourcers can employ methods to utilize their resources efficiently, but algorithmic approaches to efficient crowdsourcing generally require a fixed task set of known size. In this paper, we introduce cost forecasting as a means for a crowdsourcer to use efficient crowdsourcing algorithms with a growing set of microtasks. Cost forecasting allows the crowdsourcer to decide between eliciting new tasks from the crowd or receiving responses to existing tasks based on whether or not new tasks will cost less to complete than existing tasks, efficiently balancing resources as crowdsourcing occurs. Experiments with real and synthetic crowdsourcing data show that cost forecasting leads to improved accuracy. Accuracy and efficiency gains for crowd-generated microtasks hold the promise to further leverage the creativity and wisdom of the crowd, with applications such as generating more informative and diverse training data for machine learning applications and improving the performance of user-generated content and question-answering platforms.

Incentive Scheme for Cyber Physical Social Systems Based on User Behaviors

Cyber-physical social system (CPSS) has emerged as a new paradigm to help social users share and exchange data by the close association with the cyberspace and physical world. To further improve the performance of CPSS, the incentive computing scheme to provide efficient crowd sourcing in the CPPS becomes a challenge. Therefore, in this paper we propose a novel incentive scheme for CPSS based on the reputation of social users. First, we present a framework to provide crowd sourcing service in CPSS by dividing social users into three types, which are malicious users, speculative users and honest users, respectively. Second, based on the reputation of social users, an incentive scheme is proposed to encourage users to contribute sourcing data. Next, an auction game model is developed to help CPSS select the optimal social user to obtain the needed data. Finally, simulation results show that the proposal can obtain a lower cost and higher data accuracy than other conventional methods.

When Crowdsourcing Meets Social IoT: An Efficient Privacy-Preserving Incentive Mechanism

Crowdsourcing is an effective paradigm in human centric computing for addressing problems by utilizing human computation power, especially in booming social Internet of Things (IoT). By leveraging mutual friendship between computing entities (i.e., workers), collaborative tasks can thus be routed and finally fulfilled by multihop friends with high expertise. However, crowdsourcing in social IoT may reveal the privacy of task requesters which results in a large dilemma. In this paper, we focus on designing a multihop routing incentive mechanism which can also preserve task requester’s privacy. Specifically, a utility maximization problem under privacy and budget feasibility constraints is formulated. Defining the conditions for privacy insurance, we give guidelines on how many subtasks should an entire task be divided into, and analyze the tradeoff between privacy and task accuracy. To enable efficient crowdsourcing task routing in social IoT, we first consider 1-hop myopic routing case and propose a near-optimal task assignment algorithm with 1/2 approximation ratio for an arbitrary prior knowledge. We further design multihop payment policy to establish an equilibrium where workers are motivated to forward subtasks to their friends with the best expertise. The extensive simulations validate that our mechanism achieves a high level of average information gain with modest privacy guarantee.

Rapid, Efficient Crowdsourcing Using Social Media for the Surgical Management of Nephrolithiasis.

Introduction: The surgical management of nephrolithiasis can be challenging due to multiple appropriate and feasible approaches. One advantage of urologists' increasing social media use is the opportunity for crowdsourcing or aggregating feedback from remote participants. This study assesses the purposes and extent of online crowdsourcing by urologists for surgical stone disease. Methods: Twitter was queried with 22 stone-related keywords during a 60-day study period. All public posts were reviewed to identify crowdsourcing requests by urologists seeking clinical advice or feedback about surgical stone management. Content analysis of the crowdsourcing posts was performed by two reviewers to assess the purposes of and responses to the requests. Results: A total of 399 posts linked to 29 crowdsourcing requests were analyzed. Crowdsourcing requests were answered rapidly; the median time from initial post to final reply by participants was 2 days. In the majority of requests, the author of the initial request posted a follow-up question or clarification, underscoring a bidirectional exchange of ideas. Most crowdsourcing engaged an international audience; requests originated from 27 urologists in 12 countries, and all but one of the 29 requests (97%) received replies from users outside the origin country. Four purposes for crowdsourcing were observed: patient-specific surgical planning (206 posts, 52%); colleagues' experiences with a device or technique (86 posts, 22%); general advice about stone-related concerns (57 posts, 14%); and solutions to complications or quality improvement challenges (50 posts, 12%). Finally, protection of patient privacy appeared to be a concern; among the 17 (59%) crowdsourcing requests that were about individual patients or cases, all included patient-specific images or radiography, but none noted whether patient permission had been obtained. Conclusions: Social media crowdsourcing facilitates a range of general, patient-specific, and quality improvement challenges in surgical stone management. Participants should consider privacy concerns when publicly sharing imaging or patient data.

Applying efficient crowdsourcing techniques for increasing quality and transparency of election processes

Recently crowdsourcing is being established as the new platform for capturing ideas from multiple users, i.e., the crowd. Many companies have already shifted their approach towards utilising the power of the crowd. Transparency and quality of election process is the main factor for acknowledging the general election results. Voters, crowd feedback can be utilised to maintain a desired election process transparency and quality. This paper presents an efficient solution using crowdsourcing techniques for increasing transparency and the quality of election processes through a simple feedback web form in polling stations. These polling stations are securely connected to central election commission monitoring room, where the overall transparency and quality in national scale can be monitored. The survey conducted with more than 600 respondents shows that this approach will be acceptable from citizens and will increase the overall transparency, quality, and acceptance of election results.

Applying efficient crowdsourcing techniques for increasing quality and transparency of election processes

Applying efficient crowdsourcing techniques for increasing quality and transparency of election processes

Plexiglass: Multiplexing Passive and Active Tasks for More Efficient Crowdsourcing

Efficiently scaling continuous real-time crowdsourcing tasks — which engage crowd workers over long periods of time to complete tasks, such as monitoring video for critical events — is challenging largely because of the cost of keeping people consistently engaged. Worse, for many continuous tasks on which progress cannot be immediately made, which we term passive tasks, this engagement effort is wasted until something becomes true about the environment (e.g., the annotation of a critical event cannot happen until the event is seen). In this paper, we present the idea of passive-active task multiplexing, in which continuous tasks that involve waiting for a change in state are completed concurrently with traditional (offline/non-real time) tasks. We then implement this idea in Plexiglass, a system that answers visual queries made by blind and low-vision users more efficiently by concurrently presenting a (passive) real-time sensing task and an (active) offline visual question answering tasks to workers. We explore different approaches to accomplishing this, and validate that task multiplexing can lead to an improvement in efficiency in these settings of more than 40% in terms of overall worker time taken and 85% decrease in cost of running continuous real-time tasks. This work has implications on how real-time crowdsourcing tasks are presented to workers, and it increases the feasibility of deploying continuous real-time crowdsourcing systems in real-world settings.

Super-Fine Attributes with Crowd Prototyping

Recognising human attributes from surveillance footage is widely studied for attribute-based re-identification. However, most works assume coarse, expertly-defined categories, ineffective in describing challenging images. Such brittle representations are limited in descriminitive power and hamper the efficacy of learnt estimators. We aim to discover more relevant and precise subject descriptions, improving image retrieval and closing the semantic gap. Inspired by fine-grained and relative attributes, we introduce super-fine attributes, which now describe multiple, integral concepts of a single trait as multi-dimensional perceptual coordinates. Crowd prototyping facilitates efficient crowdsourcing of super-fine labels by pre-discovering salient perceptual concepts for prototype matching. We re-annotate gender, age and ethnicity traits from PETA, a highly diverse (19K instances, 8.7K identities) amalgamation of 10 re-id datasets including VIPER, CUHK and TownCentre. Employing joint attribute regression with the ResNet-152 CNN, we demonstrate substantially improved ranked retrieval performance with super-fine attributes in comparison to conventional binary labels, reporting up to a 11.2 and 14.8 percent mAP improvement for gender and age, further surpassed by ethnicity. We also find our 3 super-fine traits to outperform 35 binary attributes by 6.5 percent mAP for subject retrieval in a challenging zero-shot identification scenario.

Reply & Supply: Efficient crowdsourcing when workers do more than answer questions.

Crowdsourcing works by distributing many small tasks to large numbers of workers, yet the true potential of crowdsourcing lies in workers doing more than performing simple tasks—they can apply their experience and creativity to provide new and unexpected information to the crowdsourcer. One such case is when workers not only answer a crowdsourcer’s questions but also contribute new questions for subsequent crowd analysis, leading to a growing set of questions. This growth creates an inherent bias for early questions since a question introduced earlier by a worker can be answered by more subsequent workers than a question introduced later. Here we study how to perform efficient crowdsourcing with such growing question sets. By modeling question sets as networks of interrelated questions, we introduce algorithms to help curtail the growth bias by efficiently distributing workers between exploring new questions and addressing current questions. Experiments and simulations demonstrate that these algorithms can efficiently explore an unbounded set of questions without losing confidence in crowd answers.

An Efficient Crowdsourcing Search Scheme and Aggregation Techniques in Vehicular Ad hoc Networks

In recent years, Vehicular Ad-hoc Networks (VANets) have become a popular research field. In addition, conditions happened surrounding our life often have the demand to be supported by other people, and these conditions often occur on the road. This paper proposes an efficient crowdsourcing search scheme and aggregation techniques in the VANets to search objects and collect data. Our proposed mechanism is based on the virtual backbone construction. The regional data exchange can be limited, and the packet transmissions can be reduced. VANets nodes move with high speed and thus may cause dramatic topology changes, and it is not conducive to the frequent swap of information among nodes under limited network bandwidth. We have established coordinator and header mechanisms to manage the information of region nodes, reduce the amount of packet transmission and improve searching efficiency. On the other hand, aggregation and filtering mechanisms are designed and developed for the integrity of the information. Experimental results show that our schemes can effectively provide assistance in terms of search efficiency and satisfactory ratio.

Efficient crowdsourcing of unknown experts using bounded multi-armed bandits

Increasingly, organisations flexibly outsource work on a temporary basis to a global audience of workers. This so-called crowdsourcing has been applied successfully to a range of tasks, from translating text and annotating images, to collecting information during crisis situations and hiring skilled workers to build complex software. While traditionally these tasks have been small and could be completed by non-professionals, organisations are now starting to crowdsource larger, more complex tasks to experts in their respective fields. These tasks include, for example, software development and testing, web design and product marketing. While this emerging expert crowdsourcing offers flexibility and potentially lower costs, it also raises new challenges, as workers can be highly heterogeneous, both in their costs and in the quality of the work they produce. Specifically, the utility of each outsourced task is uncertain and can vary significantly between distinct workers and even between subsequent tasks assigned to the same worker. Furthermore, in realistic settings, workers have limits on the amount of work they can perform and the employer will have a fixed budget for paying workers. Given this uncertainty and the relevant constraints, the objective of the employer is to assign tasks to workers in order to maximise the overall utility achieved. To formalise this expert crowdsourcing problem, we introduce a novel multi-armed bandit (MAB) model, the bounded MAB. Furthermore, we develop an algorithm to solve it efficiently, called bounded ε-first, which proceeds in two stages: exploration and exploitation. During exploration, it first uses εB of its total budget B to learn estimates of the workers' quality characteristics. Then, during exploitation, it uses the remaining (1−ε)B to maximise the total utility based on those estimates. Using this technique allows us to derive an O(B23) upper bound on its performance regret (i.e., the expected difference in utility between our algorithm and the optimum), which means that as the budget B increases, the regret tends to 0. In addition to this theoretical advance, we apply our algorithm to real-world data from oDesk, a prominent expert crowdsourcing site. Using data from real projects, including historic project budgets, expert costs and quality ratings, we show that our algorithm outperforms existing crowdsourcing methods by up to 300%, while achieving up to 95% of a hypothetical optimum with full information.

A Division Strategy for Achieving Efficient Crowdsourcing Contest

A Division Strategy for Achieving Efficient Crowdsourcing Contest

Efficient crowdsourcing for multi-class labeling

Crowdsourcing systems like Amazon's Mechanical Turk have emerged as an effective large-scale human-powered platform for performing tasks in domains such as image classification, data entry, recommendation, and proofreading. Since workers are low-paid (a few cents per task) and tasks performed are monotonous, the answers obtained are noisy and hence unreliable. To obtain reliable estimates, it is essential to utilize appropriate inference algorithms (e.g. Majority voting) coupled with structured redundancy through task assignment. Our goal is to obtain the best possible trade-off between reliability and redundancy. In this paper, we consider a general probabilistic model for noisy observations for crowd-sourcing systems and pose the problem of minimizing the total price (i.e. redundancy) that must be paid to achieve a target overall reliability. Concretely, we show that it is possible to obtain an answer to each task correctly with probability 1-ε as long as the redundancy per task is O((K/q) log (K/ε)), where each task can have any of the $K$ distinct answers equally likely, q is the crowd-quality parameter that is defined through a probabilistic model. Further, effectively this is the best possible redundancy-accuracy trade-off any system design can achieve. Such a single-parameter crisp characterization of the (order-)optimal trade-off between redundancy and reliability has various useful operational consequences. Further, we analyze the robustness of our approach in the presence of adversarial workers and provide a bound on their influence on the redundancy-accuracy trade-off. Unlike recent prior work [GKM11, KOS11, KOS11], our result applies to non-binary (i.e. K&gt;2) tasks. In effect, we utilize algorithms for binary tasks (with inhomogeneous error model unlike that in [GKM11, KOS11, KOS11]) as key subroutine to obtain answers for K-ary tasks. Technically, the algorithm is based on low-rank approximation of weighted adjacency matrix for a random regular bipartite graph, weighted according to the answers provided by the workers.

WEB MAPPING SERVICES IN A CROWDSOURCE ENVIRONMENT FOR DISASTER MANAGEMENT: STATE-OF-THE-ART AND FURTHER DEVELOPMENT

Abstract. The Haiti earthquake in 2010 has been considered a turning point in disaster response because of the support provided by collaborative communities such as OpenStreetMap, Ushahidi, CrisisMappers, Google MapMaker and others. Taking advantage of post-disaster satellite imagery these virtual communities were able to give an impressive contribution in mapping affected areas and in damage assessment. Nowadays the crowd sourcing, whether made of experts or not, can play an important role in preparedness and disaster response. In recent years the increase of web applications’ capabilities have enabled more efficient crowd sourcing of environmental data. The most successful integration of tools and data between organizations, institutions or even individuals, happens when the most open standards for interoperability are adopted, such as Web Mapping Services and Web Feature Services, defined by the Open Geospatial Consortium. In order to be widely accepted, the applications should be likely open source. It is unlikely that a single platform can fit every need or terms of use can be broadly accepted (e.g. data ownership). Therefore some organizations could prefer having more control on the application by downloading and running it as their own shared data repository and possibly as a hub for external contributors. Nevertheless, other users could prefer to access to a hosted site where they can simply upload and share their contributions. This paper will illustrate the GeoNode open source application and the principles behind the developing of a community. GeoNode is an example of how an application can be use as a global platform or as a distributed data node to promote the collaborative use of spatial data.