Crowdsourcing System Research Articles

Trajectory recommendation for task accomplishment in crowdsourcing – a model to favour different actors

Crowdsourcing systems (CS) are platforms that enable users, called requesters, to publish tasks that others, called workers, are expected to accomplish. Usually, these are systems where workers perform tasks using desktop computers. Recently, some CS have appeared with location-based tasks that require a worker to be at a given location within a given time window to be accomplished. In this paper, we study the problem of matching these location-based tasks and mobile workers under a novel perspective where three actors (workers, requesters and the system itself) of the CS may benefit differently from a configuration of tasks and workers. We introduce the Trajectory Recommendation Problem (TRP) where a CS finds a trajectory (sequence of tasks) for a single mobile worker that maximises the satisfaction of one or more of the actors. We show that TRP is NP-complete and then propose an exact algorithm for solving it. Our experiments have proved that our algorithm is a feasible solution when up to a few hundred tasks must be analysed to find an optimal solution. Finally, we propose an architecture for the deployment of the recommendation in a CS.

An incentive mechanism with privacy protection in mobile crowdsourcing systems

In order to improve the efficiency and utility of mobile crowdsourcing systems, this paper proposes an incentive mechanism with privacy protection in mobile crowdsourcing systems. Combining the advantages of offline incentive mechanisms and online incentive mechanisms, this paper proposes an incentive mechanism that selects the worker candidates statically, and then dynamically selects winners after bidding. The proposed incentive mechanism includes two algorithms which are an improved two-stage auction algorithm (ITA) and a truthful online reputation updating algorithm (TORU). Through simulations, we verify the efficiency and effectiveness of the proposed incentive mechanism, which can solve the free-riding problem and improve the efficiency and utility of mobile crowdsourcing systems effectively.

Text mining stackoverflow

Purpose– The use of social media and in particular community Question Answering (Q & A) websites by learners has increased significantly in recent years. The vast amounts of data posted on these sites provide an opportunity to investigate the topics under discussion and those receiving most attention. The purpose of this paper is to automatically analyse the content of a popular computer programming Q & A website, StackOverflow (SO), determine the exact topics of posted Q & As, and narrow down their categories to help determine subject difficulties of learners. By doing so, the authors have been able to rank identified topics and categories according to their frequencies, and therefore, mark the most asked about subjects and, hence, identify the most difficult and challenging topics commonly faced by learners of computer programming and software development.Design/methodology/approach– In this work the authors have adopted a heuristic research approach combined with a text mining approach to investigate the topics and categories of Q & A posts on the SO website. Almost 186,000 Q & A posts were analysed and their categories refined using Wikipedia as a crowd-sourced classification system. After identifying and counting the occurrence frequency of all the topics and categories, their semantic relationships were established. This data were then presented as a rich graph which could be visualized using graph visualization software such as Gephi.Findings– Reported results and corresponding discussion has given an indication that the insight gained from the process can be further refined and potentially used by instructors, teachers, and educators to pay more attention to and focus on the commonly occurring topics/subjects when designing their course material, delivery, and teaching methods.Research limitations/implications– The proposed approach limits the scope of the analysis to a subset of Q & As which contain one or more links to Wikipedia. Therefore, developing more sophisticated text mining methods capable of analysing a larger portion of available data would improve the accuracy and generalizability of the results.Originality/value– The application of text mining and data analytics technologies in education has created a new interdisciplinary field of research between the education and information sciences, called Educational Data Mining (EDM). The work presented in this paper falls under this field of research; and it is an early attempt at investigating the practical applications of text mining technologies in the area of computer science (CS) education.

Efficient Collaborative Crowdsourcing

We consider the problem of making efficient quality-time-cost trade-offs in collaborative crowdsourcing systems in which different skills from multiple workers need to be combined to complete a task. We propose CrowdAsm - an approach which helps collaborative crowdsourcing systems determine how to combine the expertise of available workers to maximize the expected quality of results while minimizing the expected delays. Analysis proves that CrowdAsm can achieve close to optimal profit for workers in a given crowdsourcing system if they follow the recommendations.

Discovering Best Teams for Data Leak-Aware Crowdsourcing in Social Networks

Crowdsourcing is emerging as a powerful paradigm to help perform a wide range of tedious tasks in various enterprise applications. As such applications become more complex, crowdsourcing systems often require the collaboration of several experts connected through professional/social networks and organized in various teams. For instance, a well-known car manufacturer asked fans to contribute ideas for the kinds of technologies that should be incorporated into one of its cars. For that purpose, fans needed to collaborate and form teams competing with each others to come up with the best ideas. However, once teams are formed, each one would like to provide the best solution and treat that solution as a “trade secret,” hence preventing any data leak to its competitors (i.e., the other teams). In this article, we propose a data leak--aware crowdsourcing system called SocialCrowd . We introduce a clustering algorithm that uses social relationships between crowd workers to discover all possible teams while avoiding interteam data leakage. We also define a ranking mechanism to select the “best” team configurations. Our mechanism is based on the semiring approach defined in the area of soft constraints programming. Finally, we present experiments to assess the efficiency of the proposed approach.

Towards Successful Crowdsourcing Projects: Evaluating the Implementation of Governance Mechanisms

The last decade has witnessed the proliferation of crowdsourcing in various academic domains including strategic management, computer science, or IS research. Numerous companies have drawn on this concept and leveraged the wisdom of crowds for various purposes. However, not all crowdsourcing projects turn out to be a striking success. Hence, research and practice are on the lookout for the main factors influencing the success of crowdsourcing projects. In this context, proper governance is considered as the key to success by several researchers. However, little is known about governance mechanisms and their impact on project outcomes. We address this issue by means of a multiple case analysis in the scope of which we examine crowdsourcing projects on collaboration-based and/or competition-based crowdsourcing systems. Our initial study reveals that task definition mechanisms and quality assurance mechanisms have the highest impact on the success of crowdsourcing projects, whereas task allocation mechanisms are less decisive.

Massive Online Crowdsourced Study of Subjective and Objective Picture Quality.

Most publicly available image quality databases have been created under highly controlled conditions by introducing graded simulated distortions onto high-quality photographs. However, images captured using typical real-world mobile camera devices are usually afflicted by complex mixtures of multiple distortions, which are not necessarily well-modeled by the synthetic distortions found in existing databases. The originators of existing legacy databases usually conducted human psychometric studies to obtain statistically meaningful sets of human opinion scores on images in a stringently controlled visual environment, resulting in small data collections relative to other kinds of image analysis databases. Toward overcoming these limitations, we designed and created a new database that we call the LIVE In the Wild Image Quality Challenge Database, which contains widely diverse authentic image distortions on a large number of images captured using a representative variety of modern mobile devices. We also designed and implemented a new online crowdsourcing system, which we have used to conduct a very large-scale, multi-month image quality assessment (IQA) subjective study. Our database consists of over 350 000 opinion scores on 1162 images evaluated by over 8100 unique human observers. Despite the lack of control over the experimental environments of the numerous study participants, we demonstrate excellent internal consistency of the subjective data set. We also evaluate several top-performing blind IQA algorithms on it and present insights on how the mixtures of distortions challenge both end users as well as automatic perceptual quality prediction models. The new database is available for public use at http://live.ece.utexas.edu/research/ChallengeDB/index.html.

Constella: Crowdsourced system setting recommendations for mobile devices

The question “Where has my battery gone?” remains a common source of frustration for many smartphone users. With the increased complexity of smartphone applications, and the increasing number of system settings affecting them, understanding and optimizing battery use has become a difficult chore. The present paper develops a novel approach for constructing energy models from crowdsourced measurements. In contrast to previous approaches, which have focused on the effect of a specific sensor, system setting or application, our approach can simultaneously capture relationships between multiple factors, and provide a unified view of the energy state of the mobile device. We demonstrate the validity of using crowdsourced measurements for constructing battery models through a combination of large-scale analysis of a dataset containing battery discharge and system state measurements, and hardware power measurements. The results indicate that the models captured by our approach are both in line with previous studies on battery consumption and empirical measurements, providing a cost-effective way to construct energy models during normal operations of the device. The analysis also provides several new insights about battery consumption. For example, our analysis reveals the combined effect of high CPU activity and automatic screen brightness to be higher (resulting in 9 min shorter battery lifetime on average) than the effect of medium CPU load and manual screen brightness; a Wi-Fi signal strength drop of one bar can shorten battery life by over 13%; and a smartphone sitting in direct sunlight can witness over 50% shorter battery life than one indoors in cool conditions. Based on the crowdsourced energy models, we develop Constella, a novel recommender system for system settings. Constella provides actionable and human-readable recommendations on how to adjust system settings in order to reduce overall battery drain. We validate the effectiveness of Constella through a hardware power measurement experiment carried out using three application case studies. The results of the evaluation demonstrate that Constella is capable of generating recommendations that can provide up to 61% improvements in battery life.

Mapping an alternative community river: The case of the Ciliwung

Dense, self-built settlements along riverbanks within Asian cities are often excluded from the planning realm, which ensures governments lack knowledge of how particular communities function. The magnitude of land area and population, dynamic local economies, organic policy making processes, and scarcity and consistency of data challenge research on flood impacts and possible solutions in Asian cities. Resultantly, a deeper understanding of alternative and more dynamic forms of environmental management is necessary. The focus of this paper is to analyze the usefulness and challenges of participatory mapping in relation to urban floods, particularly community mapping and crowd-sourced mapping. This analysis is based on the assumption of participatory mapping discourse that participatory mapping increases communities’ negotiation power to improve their livelihood. This paper employs participant observation and ethnographic interviews within the Ciliwung River corridor in Jakarta. Specifically it focuses on activists and residents in river communities in relation to participatory community mapping exercises conducted since 2012 and a new crowd-sourced flood mapping system launched in December 2014. Participatory community mapping and crowd-sourced flood mapping, as two forms of community-based mapping approaches to floods, are viewed as potential tools to overcome urban flood hazards while raising disaster awareness among city residents. Community mapping is a method of visualizing a neighborhood's communal memories and embedded power relations, while a crowd-sourced flood map visualizes vulnerabilities and may become a tool for information sharing for the betterment of the spatially and socially fragmented city.

Book Review “Advances in Crowdsourcing”

With Web 2.0 providing a greater interaction among web users, crowdsourcing has received great attention from both practitioners and scholars over the years. Crowdsourcing is defined as the process of outsourcing tasks once done by employees to an undefined network of people in the form of an open call. While crowdsourcing is mainly used in a business context, it is applicable to many other areas, including research and education. This book provides a diversity of studies in crowdsourcing, covering its definition and how it is related to other concepts, such as citizen empowerment. It examines the components of a crowdsourcing system and their interaction. The application of crowdsourcing across difference areas, including tourism and education, will have great potential of development in the near future.

Using Expertise for Crowd-Sourcing

In this paper, we examine whether the use of expertise ratings can help crowd-sourcing systems. We show, using simulations, that a crowd-sourcing system based in social navigation works better when users’ expertise levels are taken into account.

Proposal of Grade Training Method in Private Crowdsourcing System

Current crowdsourcing platforms such as Amazon Mechanical Turk provide an attractive solution for processing of high-volume tasks at low cost. However, problems of quality control remain a major concern. We developed a private crowdsourcing system (PCSS) running in a intranetwork, that allow us to devise for quality control methods. In the present work, we designed a novel task allocation method to improve accuracy of task results in PCSS. PCSS analyzed relations between tasks from workers' behavior using Bayesian network, then created learning tasks according to analyzed relations. PCSS increased quality of task results by allocating learning tasks to workers before processing difficult tasks. PCSS created 8 learning tasks automatically for 2 target task categories and increased accuracy of task results by 10.77 point on average. We found that creating learning tasks according to analyzed relations is a practical method to improve the quality of workers.

LoRUS: A Mobile Crowdsourcing System for Efficiently Retrieving the Top-k Relevant Users in a Spatial Window

The prevalence of mobile devices and applications strongly motivate mobile crowdsourcing for facilitating location-dependent services. We propose LoRUS, a Location-based Relevant User determination System for efficiently retrieving the top-k relevant mobile users in a given spatial window.

Creative ideation in a crowdsourcing system: Effects of exposure to others’ criticism

Creative ideation in a crowdsourcing system: Effects of exposure to others’ criticism

CrowdOp: Query Optimization for Declarative Crowdsourcing Systems

We study the query optimization problem in declarative crowdsourcing systems. Declarative crowdsourcing is designed to hide the complexities and relieve the user of the burden of dealing with the crowd. The user is only required to submit an SQL-like query and the system takes the responsibility of compiling the query, generating the execution plan and evaluating in the crowdsourcing marketplace. A given query can have many alternative execution plans and the difference in crowdsourcing cost between the best and the worst plans may be several orders of magnitude. Therefore, as in relational database systems, query optimization is important to crowdsourcing systems that provide declarative query interfaces. In this paper, we propose CrowdOp , a cost-based query optimization approach for declarative crowdsourcing systems. CrowdOp considers both cost and latency in query optimization objectives and generates query plans that provide a good balance between the cost and latency. We develop efficient algorithms in the CrowdOp for optimizing three types of queries: selection queries, join queries, and complex selection-join queries. We validate our approach via extensive experiments by simulation as well as with the real crowd on Amazon Mechanical Turk.

Benefits of visualization in the Mammography Problem

Trying to make a decision between two outcomes, when there is some level of uncertainty, is inherently difficult because it involves probabilistic reasoning. Previous studies have shown that most people do not correctly apply Bayesian inference to solve probabilistic problems for decision making under uncertainty. In an effort to improve decision making with Bayesian problems, previous work has studied supplementing the textual description of problems with visualizations, such as graphs and charts. However, results have been varied and generally indicate that visualization is not an effective technique. As these studies were performed over many years with a variety of goals and experimental conditions, we sought to re-evaluate the use of visualization as an aid in solving Bayesian problems. Many of these studies used the classic Mammography Problem with visualizations portraying the problem structure, the quantities involved, or the nested-set relations of the populations involved. We selected three representative visualizations from this work and developed two hybrid visualizations, combining structure types and frequency with structure. We also included a text-only baseline condition and a text-legend condition where all nested-set problem values were given to eliminate the need for participants to estimate or calculate values. Seven hundred participants evaluated these seven conditions on the classic Mammography Problem in a crowdsourcing system, where micro-interaction data was collected from the participants. Our analysis of the user input and of the results indicates that participants made use of the visualizations but that the visualizations did not help participants to perform more accurately. Overall, static visualizations do not seem to aid a majority of people in solving the Mammography Problem. Graphical abstractDisplay Omitted HighlightsWe evaluate the use of visualization as an aid in solving Bayesian problems.We propose comparability criteria for consistency in visualization design.We created a design space including existing and novel visualizations.700 participants evaluated 7 different visualizations of the Mammography Problem.Participants made use of the visualizations but they did not improve performance.

Geocrowdsourcing and accessibility for dynamic environments

A consequence of modern society’s increasing reliance on digital communication is the concurrent multiplication and narrowing of information streams, with many channels of digital information, but channels which are difficult or impossible for some individuals to access. Blind and visually-impaired individuals are often left out of this communication, unless accommodations are carefully planned and made to present the information in a usable form. The context of this research is the realm of assistive geotechnology, where web mapping technology and geographic information systems are used to provide access to enabling information and services. This paper presents research on the development of tools to provide transient obstacle information to blind, visually-impaired, and mobility-impaired individuals through crowdsourcing, and research on the general accessibility of local pedestrian networks under constraints presented by transient and permanent navigation obstacles. We discuss the social and technological dynamics associated with the creation and use of our crowdsourcing system and present our effort for comprehensive quality assessment. We conclude that crowdsourcing is a crucial technique for successful deployment of assistive geotechnology, particularly those that involve navigation, wayfinding, and travel in public space. We find that many public health services and other important resources cannot be accessed dependably without the use of crowdsourcing and other techniques for capturing dynamic, changing environmental conditions. More broadly, this paper concludes by addressing the progression of user-generated and crowdsourced content from static data contributions to dynamic place-based services and the enabling role of assistive geotechnology in providing access and help to blind, visually-impaired, and mobility-impaired individuals.

Reliable Multiple-choice Iterative Algorithm for Crowdsourcing Systems

The appearance of web-based crowdsourcing systems gives a promising solution to exploiting the wisdom of crowds efficiently in a short time with a relatively low budget. Despite their efficiency, crowdsourcing systems have an inherent problem in that responses from workers can be unreliable since workers are low-paid and have low responsibility. Although simple majority voting can be a solution, various research studies have sought to aggregate noisy responses to obtain greater reliability in results through effective techniques such as Expectation-Maximization (EM) based algorithms. While EM-based algorithms get the limelight in crowdsourcing systems due to their useful inference techniques, Karger et al. made a significant breakthrough by proposing a novel iterative algorithm based on the idea of low-rank matrix approximations and the message passing technique. They showed that the performance of their iterative algorithm is order-optimal, which outperforms majority voting and EM-based algorithms. However, their algorithm is not always applicable in practice since it can only be applied to binary-choice questions. Recently, they devised an inference algorithm for multi-class labeling, which splits each task into a bunch of binary-choice questions and exploits their existing algorithm. However, it has difficulty in combining into real crowdsourcing systems since it overexploits redundancy in that each split question should be queried in multiple times to obtain reliable results. In this paper, we design an iterative algorithm to infer true answers for multiple-choice questions, which can be directly applied to real crowdsourcing systems. Our algorithm can also be applicable to short-answer questions as well. We analyze the performance of our algorithm, and prove that the error bound decays exponentially. Through extensive experiments, we verify that our algorithm outperforms majority voting and EM-based algorithm in accuracy.

Population as Auditor of an Election Process in Honduras: The Case of the VotoSocial Crowdsourcing Platform

For a country to become fully democratic, the majority of the population needs to be involved in the politics of that country. This requires people to be aware of what is happening, and to be able to participate in the country's political processes. VotoSocial is a crowdsourcing system launched during the 2013 Honduran government elections. It retrieved the official polling table records in digitized form from the Supreme Electoral Tribunal (the Government's elections authority) and provided them to the Internet community to be counted and the digitizations verified. VotoSocial was thereby able to verify the accuracy of the official polling results. We found no evidence of fraud in the digitization process, but statistical analysis revealed a data behavior usually associated with incremental fraud in an electoral process. VotoSocial shows that social media‐powered crowdsourcing systems can increase a population's political awareness during elections, thus providing a motive to build similar platforms in other countries with political systems marked by electoral fraud.

Efficient Task Sub-Delegation for Crowdsourcing

Reputation-based approaches allow a crowdsourcing system to identify reliable workers to whom tasks can be delegated. In crowdsourcing systems that can be modeled as multi-agent trust networks consist of resource constrained trustee agents (i.e., workers), workers may need to further sub-delegate tasks to others if they determine that they cannot complete all pending tasks before the stipulated deadlines. Existing reputation-based decision-making models cannot help workers decide when and to whom to sub-delegate tasks. In this paper, we proposed a reputation aware task sub-delegation (RTS) approach to bridge this gap. By jointly considering a worker's reputation, workload, the price of its effort and its trust relationships with others, RTS can be implemented as an intelligent agent to help workers make sub-delegation decisions in a distributed manner. The resulting task allocation maximizes social welfare through efficient utilization of the collective capacity of a crowd, and provides provable performance guarantees. Experimental comparisons with state-of-the-art approaches based on the Epinions trust network demonstrate significant advantages of RTS under high workload conditions.