Crowd-powered Systems Research Articles

Improving Search Quality in Crowdsourced Bib Number Tagging Systems Using Data Fusion

Today, crowd computing is successfully applied for many information processing problems in a variety of domains. One of the most acute issues with crowd-powered systems is the quality of results (as humans can make errors). Therefore, a number of methods have been proposed to process the results obtained from the crowd in order to compensate human errors. Most of the existing methods of processing contributions are constructed based on a (natural) assumption that the only information available is unreliable data obtained from the crowd. However, in some cases, additional information is available, and it can be utilized in order to improve the overall quality of the result. The paper describes a crowd computing application for community tagging of running race photos. It presents a utility analysis to identify situations in which community photo tagging is a reasonable choice. It also proposes a data fusion model making use of runners’ location information recorded in their Global Positioning System (GPS) tracks. Field experiments with the applications show that community-based tagging can collect enough contributors to process photosets from medium-sized running events. Simulation results confirm, that the use of data fusion in processing the results of crowd computing is a promising technique, and the use of probabilistic graphical models (e.g., Bayesian networks) for data fusion allows one to smoothly increase the quality of the results with an increase of the available information.

A Review of Judgment Analysis Algorithms for Crowdsourced Opinions

The crowd-powered systems have been shown to be highly successful in the current decade to manage collective contribution of online workers for solving different complex tasks. It can also be used for soliciting opinions from a large set of people working in a distributed manner. Unfortunately, the online community of crowd workers might involve non-experts as opinion providers. As a result, such approaches may give rise to noise making it hard to predict the appropriate (gold) judgment. Judgment analysis is in general a way of learning about human decision from multiple opinions. A spectrum of algorithms has been proposed in the last few decades to address this problem. They are broadly made up of supervised or unsupervised types. However, they have been readdressed in recent years having focus on different strategies for obtaining the gold judgment from crowdsourced opinions, viz., estimating the accuracy of opinions, difficulties of the problem, spammer identification, handling noise, etc. Besides this, investigation of various types of crowdsourced opinions to solve complex real-life problems provide new insights in this domain. In this survey, we provide a comprehensive overview of the judgment analysis problem and some of its novel variants, addressed with different approaches, where the opinions are crowdsourced.

Crowd of Oz: A Crowd-Powered Social Robotics System for Stress Management.

Coping with stress is crucial for a healthy lifestyle. In the past, a great deal of research has been conducted to use socially assistive robots as a therapy to alleviate stress and anxiety related problems. However, building a fully autonomous social robot which can deliver psycho-therapeutic solutions is a very challenging endeavor due to limitations in artificial intelligence (AI). To overcome AI’s limitations, researchers have previously introduced crowdsourcing-based teleoperation methods, which summon the crowd’s input to control a robot’s functions. However, in the context of robotics, such methods have only been used to support the object manipulation, navigational, and training tasks. It is not yet known how to leverage real-time crowdsourcing (RTC) to process complex therapeutic conversational tasks for social robotics. To fill this gap, we developed Crowd of Oz (CoZ), an open-source system that allows Softbank’s Pepper robot to support such conversational tasks. To demonstrate the potential implications of this crowd-powered approach, we investigated how effectively, crowd workers recruited in real-time can teleoperate the robot’s speech, in situations when the robot needs to act as a life coach. We systematically varied the number of workers who simultaneously handle the speech of the robot (N = 1, 2, 4, 8) and investigated the concomitant effects for enabling RTC for social robotics. Additionally, we present Pavilion, a novel and open-source algorithm for managing the workers’ queue so that a required number of workers are engaged or waiting. Based on our findings, we discuss salient parameters that such crowd-powered systems must adhere to, so as to enhance their performance in response latency and dialogue quality.

Crowdsourcing and Sensing for Indoor Localization in IoT: A Review

Among the emerging concepts in the context of IoT, crowdsourcing, and crowdsensing are known as two critical building blocks on the intersection point of things and human-based techniques. Although contribution and involvement of people is a key factor in both crowdsensing and crowdsourcing systems which provides reliability and data quality, level of human contribution is what makes crowd-powered systems different. To scrutinize this difference it is considered that user intervention in crowd-based schemes can be either implicit or explicit. Since one of the most important applications of crowdsourcing and crowdsensing systems is providing location services in indoor environments, in this paper, we study the level of user contribution in available crowd-powered techniques and propose a classification for crowd-powered indoor localization solutions to clarify which crowds-based approach is utilized in each indoor localization solution because in many cases the distinction is not clear and often leads to misconception. Dependency on site survey process is considered as another distinction point in our proposed classification. Hence, we consider indoor localization solutions with implicit user participation and latent mobile utilization as crowdsensing indoor localization systems. Respectively, indoor localization solutions with explicit user participation and manifest mobile utilization are classified as crowdsourcing indoor localization systems.

Deadline-Aware Fair Scheduling for Multi-Tenant Crowd-Powered Systems

Crowdsourcing has become an integral part of many systems and services that deliver high-quality results for complex tasks such as data linkage, schema matching, and content annotation. A standard function of such crowd-powered systems is to publish a batch of tasks on a crowdsourcing platform automatically and to collect the results once the workers complete them. Currently, these systems provide limited guarantees over the execution time, which is problematic for many applications. Timely completion may even be impossible to guarantee due to factors specific to the crowdsourcing platform, such as the availability of workers and concurrent tasks. In our previous work, we presented the architecture of a crowd-powered system that reshapes the interaction mechanism with the crowd. Specifically, we studied a push-crowdsourcing model whereby the workers receive tasks instead of selecting them from a portal. Based on this interaction model, we employed scheduling techniques similar to those found in distributed computing infrastructures to automate the task assignment process. In this work, we first devise a generic scheduling strategy that supports both fairness and deadline-awareness. Second, to complement the proof-of-concept experiments previously performed with the crowd, we present an extensive set of simulations meant to analyze the properties of the proposed scheduling algorithms in an environment with thousands of workers and tasks. Our experimental results show that, by accounting for human factors, micro-task scheduling can achieve fairness for best-effort batches and boosts production batches.

CrowdMask: Using Crowds to Preserve Privacy in Crowd-Powered Systems via Progressive Filtering

Crowd-powered systems leverage human intelligence to go beyond the capabilities of automated systems, but also introduce privacy and security concerns because unknown people must view the data that the system processes. While automated approaches cannot robustly filter private information from these datasets, people have the ability to do so if the risk from them viewing the data can be mitigated. We present a crowd-powered approach to masking private content in data by segmenting and distributing smaller segments to crowd workers so that individual workers can identify potentially private content without being able to fully view it themselves. We introduce a novel pyramid workflow for segmentation that uses segments at multiple levels of granularity to overcome problems with fixed-sized approaches. We implement our approach in CrowdMask, a system that allows images with potentially sensitive content to be masked by appearing in progressively larger, more identifiable segments, and masking portions of the image as soon as a risk is identified. Our experiments with 4134 Mechanical Turk workers show that CrowdMask can effectively mask private content from images without revealing sensitive content to constituent workers, while still enabling future systems to use the filtered result.

A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing

A number of interactive crowd-powered systems have been developed to solve difficult problems out of reach for automated solutions. To work interactively, such systems need access to on-demand labor. To meet this demand, workers can be (i) recruited when needed directly from the crowd marketplace, or (ii) recruited in advance and asked to wait in a retainer pool until they are needed. Most of the evaluations of these systems have been over a short time period, even though we know that marketplaces change and adapt over time. In this paper, we present the results of a 10-month deployment of a crowd-powered system that uses a hybrid approach to fast recruitment of workers that we call Ignition. We describe the Ignition approach and the observed times required to recruit workers from the marketplace and retainer over this long period of time. Our results demonstrate that it is possible to recruit workers with low latency even over long periods, and suggest a number of opportunities for future work for recruitment strategies and modeling that may further improve on-demand recruitment for deployed systems.

"Is There Anything Else I Can Help You With?" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent

Intelligent conversational assistants, such as Apple's Siri, Microsoft's Cortana, and Amazon's Echo, have quickly become a part of our digital life. However, these assistants have major limitations, which prevents users from conversing with them as they would with human dialog partners. This limits our ability to observe how users really want to interact with the underlying system. To address this problem, we developed a crowd-powered conversational assistant, Chorus, and deployed it to see how users and workers would interact together when mediated by the system. Chorus sophisticatedly converses with end users over time by recruiting workers on demand, which in turn decide what might be the best response for each user sentence. Up to the first month of our deployment, 59 users have held conversations with Chorus during 320 conversational sessions. In this paper, we present an account of Chorus' deployment, with a focus on four challenges: (i) identifying when conversations are over, (ii) malicious users and workers, (iii) on-demand recruiting, and (iv) settings in which consensus is not enough. Our observations could assist the deployment of crowd-powered conversation systems and crowd-powered systems in general.

A Crowd-Powered System for Fashion Similarity Search

Driven by the needs of customers and industry, online fashion search and analytics are recently gaining much attention. As fashion is mostly expressed by visual content, the analysis of fashion images in online social networks is a rich source of possible insights on evolving trends and customer preferences. Although a plethora of visual content is available, the modeling of clothes’ physics and movement, the implicit semantics in fashion designs, and the subjectivity of their interpretation pose difficulties to fully automated solutions for fashion search and analysis. In this article, we present the design and evaluation of a crowd-powered system for fashion similarity search from Twitter, supporting trend analysis for fashion professionals. The system enables fashion similarity search based on specific human-based similarity criteria. This is achieved by implementing a novel machine--crowd workflow that supports complex tasks requiring highly subjective judgments where multiple true solutions may coexist. We discuss how this leads to a novel class of crowd-powered systems for which the output of the crowd is not used to verify the automatic analysis but is the desired outcome. Finally, we show how this kind of crowd involvement enables a novel kind of similarity search and represents a crucial factor for the acceptance of system results by the end user.

Target Acquisition and the Crowd Actor

Work in human-computer interaction has generally assumed either a single user or a group of users working together in a shared virtual space. Recent crowd-powered systems use a different model in which a dynamic group of individuals (the crowd) collectively form a single actor that responds to real-time performance tasks, e.g., controlling an on-screen character, driving a robot, or operating an existing desktop interface. In this paper, we introduce the idea of the crowd actor as a way to model coordination strategies and resulting collective performance, and discuss how the crowd actor is influenced not only by the domain on which it is asked to operate but also by the personality endowed to it by algorithms used to combine the inputs of constituent participants. Nowhere is the focus on the individual performer more finely resolved than in the study of the human psychomotor system, a mainstay topic in psychology that, largely owing to Fitts' law, also has a legacy in HCI. Therefore, we explored our notion of a crowd actor by modeling the crowd as a individual motor system performing pointing tasks. We combined the input of 200 participants in a controlled offline experiment to demonstrate the inherent trade-offs between speed and errors based on personality, the number of constituent individuals, and the mechanism used to distribute work across the group. Finally, 10 workers participated in a synchronous experiment to explore how the crowd actor responds in a real online setting. This work contributes to the beginning of a predictive science for the general crowd actor model.

Architecting Real-Time Crowd-Powered Systems

Human computation allows computer systems to leverage human intelligence in computational processes. While it has primarily been used for tasks that are not time-sensitive, recent systems use crowdsourcing to get on-demand, real-time, and even interactive results. In this paper, we present techniques for building real-time crowdsourcing systems, and then discuss how and when to use them. Our goal is to provide system builders with the tools and insights they need to replicate the success of modern systems in order to further explore this new space.

Architecting Real-Time Crowd-Powered Systems

Human computation allows computer systems to leverage human intelligence in computational processes. While it has primarily been used for tasks that are not time-sensitive, recent systems use crowdsourcing to get on-demand, real-time, and even interactive results. In this paper, we present techniques for building real-time crowdsourcing systems, and then discuss how and when to use them. Our goal is to provide system builders with the tools and insights they need to replicate the success of modern systems in order to further explore this new space.

Automated Support for Collective Memory of Conversational Interactions

Maintaining consistency is a difficult challenge in crowd-powered systems in which constituent crowd workers may change over time. We discuss an initial outline for Chorus:Mnemonic, a system that augments the crowd's collective memory of a conversation by automatically recovering past knowledge based on topic, allowing the system to support consistent multi-session interactions. We present the design of the system itself, and discuss methods for testing its effectiveness. Our goal is to provide consistency between long interactions with crowd-powered conversational assistants by using AI to augment crowd workers.

Crowd-Powered Systems

Crowd-powered systems combine computation with human intelligence, drawn from large groups of people connecting and coordinating online. These hybrid systems enable applications and experiences that neither crowds nor computation could support alone. Unfortunately, crowd work is error-prone and slow, making it difficult to incorporate crowds as first-order building blocks in software systems. I introduce computational techniques that decompose complex tasks into simpler, verifiable steps to improve quality, and optimize work to return results in seconds. These techniques develop crowdsourcing as a platform so that it is reliable and responsive enough to be used in interactive systems. This thesis develops these ideas through a series of crowd-powered systems. The first, Soylent, is a word processor that uses paid micro-contributions to aid writing tasks such as text shortening and proofreading. Using Soylent is like having access to an entire editorial staff as you write. The second system, Adrenaline, is a camera that uses crowds to help amateur photographers capture the exact right moment for a photo. It finds the best smile and catches subjects in mid-air jumps, all in realtime. Moving beyond generic knowledge and paid crowds, I introduce techniques to motivate a social network that has specific expertise, and techniques to data mine crowd activity traces in support of a large number of uncommon user goals. These systems point to a future where social and crowd intelligence are central elements of interaction, software, and computation. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)