Human Intelligence Tasks Research Articles

Crowdsourcing: A Critical Reflection on This New Frontier of Participant Recruiting in Nutrition and Dietetics Research

Crowdsourcing: A Critical Reflection on This New Frontier of Participant Recruiting in Nutrition and Dietetics Research

Understanding the Impact of Text Highlighting in Crowdsourcing Tasks

Text classification is one of the most common goals of machine learning (ML) projects, and also one of the most frequent human intelligence tasks in crowdsourcing platforms. ML has mixed success in such tasks depending on the nature of the problem, while crowd-based classification has proven to be surprisingly effective, but can be expensive. Recently, hybrid text classification algorithms, combining human computation and machine learning, have been proposed to improve accuracy and reduce costs. One way to do so is to have ML highlight or emphasize portions of text that it believes to be more relevant to the decision. Humans can then rely only on this text or read the entire text if the highlighted information is insufficient. In this paper, we investigate if and under what conditions highlighting selected parts of the text can (or cannot) improve classification cost and/or accuracy, and in general how it affects the process and outcome of the human intelligence tasks. We study this through a series of crowdsourcing experiments running over different datasets and with task designs imposing different cognitive demands. Our findings suggest that highlighting is effective in reducing classification effort but does not improve accuracy - and in fact, low-quality highlighting can decrease it.

Deep Learning in Computer Vision: Methods, Interpretation, Causation and Fairness

Deep learning models have succeeded at a variety of human intelligence tasks and are already being used at commercial scale. These models largely rely on the standard gradient descent optimization of parameters W, which maps an input X to an output y =f(X;W). The optimization procedure minimizes the loss (difference) between the model output y and actual output y. As an example, in the cancer detection setting, X is an MRI image, while y is the presence or absence of cancer. Three key ingredients hint at the reason behind deep learning’s power. (1) Deep architectures better adapt to breaking down complex functions into a composition of simpler abstract parts. (2) Standard gradient descent methods that attain local minima on a nonconvex Loss(y,y ) function that are close enough to the global minima. (3) Architectures suited for execution on parallel computing hardware (e.g., GPUs), thus making the optimization viable over hundreds of millions of observations (X,y). Computer vision tasks, where input X is a high-dimensional image or video, are particularly suited to deep learning application. Recent advances in deep architectures, i.e., inception modules, attention networks, adversarial networks and DeepRL, have opened up completely new applications that were previously unexplored. However, the breakneck progress to replace human tasks with deep learning comes with caveats. These deep models tend to evade interpretation, lack causal relationships between input X and output y and may inadvertently mimic not just human actions but human biases and stereotypes. In this tutorial, we provide an intuitive explanation of deep learning methods in computer vision as well as limitations in practice.

Crowdsourcing for Hispanic Linguistics: Amazon’s Mechanical Turk as a source of Spanish data

Within the field of Linguistics, Amazon’s Mechanical Turk, a crowdsourcing marketplace specializes in computer-based Human Intelligence Tasks, has been praised as a cost efficient source of data for English and other major languages. Spanish is a good candidate due to its presence within the US and beyond. Still, detailed information concerning the linguistic and demographic profile of Spanish-speaking ‘Turkers’ is missing, thus making it difficult for researchers to evaluate whether the Mechanical Turk provides the right environment for their tasks. This paper addresses this gap in our knowledge by developing the first detailed study of the presence of Spanish-speaking workers, focusing on factors relevant for research planning, namely, (socio)linguistically relevant variables and information concerning work habits. The results show that this platform provides access to a fairly active participant pool of both L1 and L2Spanish speakers as well as bilinguals. A brief introduction to how Amazon’s Mechanical Turk works and overview of Hispanic Linguistics projects that have so far used the Mechanical Turk successfully is included.

Risk Tolerance, Legitimacy, and Perspective: Navigating Biases in Social Enterprise Evaluations

AbstractSocial enterprises viewed as viable from societal perspectives are often regarded differently from traditional business perspectives. To examine this difference, we undertook two empirical studies of risk tolerance and legitimacy perceptions among observers of social enterprise and for-profit ventures. In Study 1, participants (n= 115) drawn randomly from Amazon’s Mechanical Turk, an online crowdsourcing marketplace for human intelligence tasks, examined two hypothetical cases and completed the risk tolerance scale of theJackson Personality Inventory. Results show that social enterprises were seen as having lower industry legitimacy, especially by individuals with lower risk tolerance. Here, industry legitimacy mediated the effect of venture purpose on cognitive legitimacy. In Study 2, practicing entrepreneurs (n= 23) narratively interpreted Study 1 results from social enterprise and traditional business perspectives. Both studies demonstrate that social enterprise legitimacy evaluations vary based on risk tolerance and the type of legitimacy in question. Overall findings show that explicit observations of risk tolerance effects, and multidimensional conceptualizations of legitimacy, are important to accurate evaluations of social enterprises.

Analysing the characteristics of crowdsourcing platforms for improving throughput

Crowdsourcing leverages human intelligence to gather solutions on tasks that cannot be accomplished by automated tools. This system consists of components such as the requester, task, worker and the crowdsourcing platform. Studies do not explore the various features of these components and the dependencies among the same. Hence, we analyse the characteristics of the components of crowdsourcing systems using a trace-driven approach. Additionally, for reproducible research, we have introduced a workload generator for crowdsourcing platforms, which generates an unbiased workload similar to the empirical workload. Finally, the impact of various characteristics on the quality of answers has been analysed using both the empirical and synthetic workloads. The results demonstrate that success rate and activeness positively affect the productivity of workers, while the number of available human intelligence tasks (HITs) and the time duration of the same affect the productivity on each task.

The collective intelligence of random small crowds: A partial replication of Kosinski et al. (2012)

AbstractWe examined the trade-off between the cost of response redundancy and the gain in output quality on the popular crowdsourcing platform Mechanical Turk, as a partial replication of Kosinski et al. (2012) who demonstrated a significant improvement in performance by aggregating multiple responses through majority vote. We submitted single items from a validated intelligence test as Human Intelligence Tasks (HITs) and aggregated the responses from “virtual groups” consisting of 1 to 24 workers. While the original study relied on resampling from a relatively small number of responses across a range of experimental conditions, we randomly and independently sampled from a large number of HITs, focusing only on the main effect of group size. We found that – on average – a group of six MTurkers has a collective IQ one standard deviation above the mean for the general population, thus demonstrating a “wisdom of the crowd” effect. The relationship between group size and collective IQ was characterised by diminishing returns, suggesting moderately sized groups provide the best return on investment. We also analysed performance of a smaller subset of workers who had each completed all 60 test items, allowing for a direct comparison between a group’s collective IQ and the individual IQ of its members. This demonstrated that randomly selected groups collectively equalled the performance of the best-performing individual within the group. Our findings support the idea that substantial intellectual capacity can be gained through crowdsourcing, contingent on moderate redundancy built into the task request.

Automating Articulation: Applying Natural Language Processing to Post-Secondary Credit Transfer

Within the field of post-secondary student mobility, the assessment, and evaluation of transfer credit is a labor-intensive human intelligence task that is subject to time limits and human bias. This paper introduces a semi-automated approach to assessing transfer credit and generating articulation agreements between post-secondary institutions using natural language processing (NLP). The output from the NLP system is tested using a content expert generated an assessment of transfer credit between computer science programs at two separate post-secondary institutions. Initial testing with an unsupervised NLP algorithm, despite good results against standardized measures, assessed the percentage of course overlap as 71% similar to the percentages selected by human content experts. The application of an algorithm based on the Word2Vec model using domain-specific Wikipedia corpus and dependency parsing was applied to compensate for domain specific language and improved the relationship between content experts ratings and NLP output to 86% related overlap.

Context-Aware Hierarchical Online Learning for Performance Maximization in Mobile Crowdsourcing

In mobile crowdsourcing (MCS), mobile users accomplish outsourced human intelligence tasks. MCS requires an appropriate task assignment strategy, since different workers may have different performance in terms of acceptance rate and quality. Task assignment is challenging, since a worker's performance (i) may fluctuate, depending on both the worker's current personal context and the task context, (ii) is not known a priori, but has to be learned over time. Moreover, learning context-specific worker performance requires access to context information, which may not be available at a central entity due to communication overhead or privacy concerns. Additionally, evaluating worker performance might require costly quality assessments. In this paper, we propose a context-aware hierarchical online learning algorithm addressing the problem of performance maximization in MCS. In our algorithm, a local controller (LC) in the mobile device of a worker regularly observes the worker's context, her/his decisions to accept or decline tasks and the quality in completing tasks. Based on these observations, the LC regularly estimates the worker's context-specific performance. The mobile crowdsourcing platform (MCSP) then selects workers based on performance estimates received from the LCs. This hierarchical approach enables the LCs to learn context-specific worker performance and it enables the MCSP to select suitable workers. In addition, our algorithm preserves worker context locally, and it keeps the number of required quality assessments low. We prove that our algorithm converges to the optimal task assignment strategy. Moreover, the algorithm outperforms simpler task assignment strategies in experiments based on synthetic and real data.

It's "the Joneses": the influence of objective and subjective socioeconomic status on subjective perceptions of aging.

Positive subjective perceptions of aging predict various well-being, physical health, mental health, and longevity outcomes. Thus, it is important to consider what factors contribute to their formation. Socioeconomic status (SES) has been hypothesized to be one such factor, but past research has been mixed. We propose that subjective assessments of SES may better predict subjective perceptions of aging than traditional objective measures (e.g., income and education). The purpose of this study was to examine the effects of objective and subjective indicators of SES on subjective perceptions of aging. Participants (n = 296)from the Mindfulness and Anticipatory Coping Everyday study were recruited with a Human Intelligence Task on Amazon's Mechanical Turk. Participants reported on their SES (i.e., income, education, and subjective social status) and subjective perceptions of aging (i.e., attitudes toward own aging, subjective age, and awareness of age-related gains and losses). Data were analyzed via hierarchical multiple regression with demographic and health variables entered first, followed by income and education, then subjective social status. Results demonstrated that although objective measures of SES did not contribute to predicting subjective perceptions of aging, those who rated themselves subjectively higher in their community social standing were more likely to possess positive aging attitudes, younger subjective ages, more awareness of age-related gains, and fewer awareness of age-related losses. These findings suggest that perceptions of doing better than one's neighbors (i.e., "the Joneses") may matter more for understanding subjective perceptions of aging than objective indicators of social status.

Labour law for TOS and HITs? reflections on the potential for applying ‘labour law analogies’ to crowdworkers, focusing on employee representation

A growing number of people are performing work tasks via online platforms, referred to under various designations such as ‘Human Intelligence Tasks’ (HITs), under conditions set out not in an employment contract but through the standardised Terms of Service (TOS) of their contract partner. This article argues that, in spite of increasing evidence of precarious working conditions and circumvention of labour law and social protection standards in ‘turking’-style work environments, attempts to classify these forms of crowdwork as employment relationships are of limited practical use and benefit for those working in the industry. Instead, departing from much-debated concepts of a ‘purposive’ approach to labour law, it makes the case for a differential analysis of the aims of diverse elements of labour law and a consideration of whether, and to what degree, these can be instrumentalised for dealing with a contractual relationship that, notwithstanding socio-economic similarities, is fundamentally different from the employment relationship for which that law was developed, in several respects. It discusses the merits of rules on workplace employee representation and explores options for operationalising these for crowdworkers. For this purpose, it considers forms of collective organisation of crowdworkers via various networks as they already exist and are emerging in practice to question whether it is or should be reasonable for crowdworkers to be legally entitled to rights analogous to the workplace representation bodies such as works councils, which would entitle them to rights ranging from information and consultation to co-determination as well as veto rights on specific issues.

The Big, Gig Picture: We Can't Assume the Same Constructs Matter

I am concerned about industrial and organizational (I-O) psychology's relevance to the gig economy, defined here as the broad trends toward technology-based platform work. This sort of work happens on apps like Uber (where the app connects drivers and riders) and sites like MTurk (where human intelligence tasks, or HITs, are advertised to workers on behalf of requesters). We carry on with I-O research and practice as if technology comprises only things (e.g., phones, websites, platforms) that we use to assess applicants and complete work. However, technology has much more radically restructured work as we know it, to happen in a much more piecemeal, on-demand fashion, reviving debates about worker classification and changing the reality of work for many workers (Sundararajan, 2016). Instead of studying technology as a thing we use, it's critical that we “zoom out” to see and adapt our field to this bigger picture of trends towards a gig economy. Rather than a phone being used to check work email or complete pre-hire assessments, technology and work are inseparable. For example, working on MTurk requires constant Internet access (Brawley, Pury, Switzer, & Saylors, 2017; Ma, Khansa, & Hou, 2016). Alarmingly, some researchers describe these workers as precarious (Spretizer, Cameron, & Garrett, 2017), dependent on an extremely flexible (a label that is perhaps euphemistic for unreliable) source of work. Although it's unlikely that all workers consider their “gig” a full time job or otherwise necessary income, at least some workers do: An estimated 10–40% of MTurk workers consider themselves serious gig workers (Brawley & Pury, 2016). Total numbers for the broader gig economy are only growing, with recent tax-based estimates including 34% of the US workforce now and up to 43% within 3 years (Gillespie, 2017). It appears we're seeing some trends in work reverse and return to piece work (e.g., a ride on Uber, a HIT on MTurk) as if we've simply digitized the assembly line (Davis, 2016). Over time, these trends could accelerate, and we could potentially see total elimination of work (Morrison, 2017).

Artificial Intelligence-Based Triage for Patients with Acute Abdominal Pain in Emergency Department; a Diagnostic Accuracy Study.

Introduction:Artificial intelligence (AI) is the development of computer systems which are capable of doing human intelligence tasks such as decision making and problem solving. AI-based tools have been used for predicting various factors in medicine including risk stratification, diagnosis and choice of treatment. AI can also be of considerable help in emergency departments, especially patients’ triage. Objective:This study was undertaken to evaluate the application of AI in patients presenting with acute abdominal pain to estimate emergency severity index version 4 (ESI-4) score without the estimate of the required resources.Methods:A mixed-model approach was used for predicting the ESI-4 score. Seventy percent of the patient cases were used for training the models and the remaining 30% for testing the accuracy of the models. During the training phase, patients were randomly selected and were given to systems for analysis. The output, which was the level of triage, was compared with the gold standard (emergency medicine physician). During the test phase of the study, another group of randomly selected patients were evaluated by the systems and the results were then compared with the gold standard. Results:Totally, 215 patients who were triaged by the emergency medicine specialist were enrolled in the study. Triage Levels 1 and 5 were omitted due to low number of cases. In triage Level 2, all systems showed fair level of prediction with Neural Network being the highest. In Level 3, all systems again showed fair level of prediction. However, in triage Level 4, decision tree was the only system with fair prediction. Conclusion:The application of AI in triage of patients with acute abdominal pain resulted in a model with acceptable level of accuracy. The model works with optimized number of input variables for quick assessment.

Incremental Interactive Record Linkage using Human Intelligence Tasks (HITs)

ABSTRACTObjectiveWhen analyzing population data, there is a need to link data about organizations. One challenge in linking organization level data is that unlike a person, there can be many definitions for an entity. For example, for hospitals, depending on the dataset, an entity might represent any one of the following similar but different semantic types:(1) physical units, (2) billing units (3) legal units, (5) licensed units, or (5) reporting units. How these different entities relate to each other can be complex such as one billing unit can span across many physical units or multiple billing units can exist for one physical unit. Thus, linking organization level data requires human involvement to sort through these issues in heterogeneous data sources to make informed decisions on the messy data. We design and evaluate a general framework for a hybrid Human-Machine process for ongoing integration and cleaning of hospital level data when no common identifiers exist such that we highlight the decisions that need human judgement and document and track the full processes to ensure reproducibility. Such ongoing integration is often called incremental record linkage (RL).
 ApproachAccurate linkage in big data requires well-defined tasks that need automatic or human processing. In the human computer interaction (HCI) field, Human Intelligence Tasks (HITs) are defined as micro tasks requiring human judgment and are often used in designing crowdsourcing systems. We designed HITs for linking organization level data and embed them into automatic deterministic linkage algorithms that supports interactive stepwise RL. The hybrid system is a framework for reproducible incremental RL.
 ResultsWe illustrate this framework by integrating four databases of hospitals in Texas from 2008 to 2014(N=664). The IDs used in the databases are the Texas Provider ID, the National Provider ID, the Medicare ID, and the Facility ID. We link the databases using provider name, including dba (i.e., doing business as), addresses, and phone numbers. Similarities in hospital names and addresses and the dynamic nature of hospital attributes over time make it impossible to build a fully automated linkage system for hospitals. Using our system to iteratively standardize and clean the data, we linked the hospitals with 100% precision using HITs that required confirming 79 approximate linkages and manually linking 28 hospitals.
 ConclusionEffective software that can support the interactive and iterative process of RL with well-designed HITs can streamline the linkage processes to support high quality replicable research using big data.

Perceptions Toward the Federal Flight Deck Officer Program and Willingness to Fly

Abstract. In the aftermath of the September 11 terrorist attacks against the United States, numerous changes were implemented to aviation security. One of those programs was the arming of commercial airline pilots with handguns while they were on duty in a flight capacity. Since its inception, the Federal Flight Deck Officer (FFDO) program has been controversial in nature. The purpose of this study was to examine participant perceptions and their willingness to fly based on whether or not the pilot of their hypothetical flight was armed and participating in the FFDO program. A sample of 812 participants was selected from both India and the United States using Amazon’s Mechanical Turk® (MTurk). In general, the findings indicate that participants were more willing to fly when their pilot was armed; however, American females indicated no significant difference between the two conditions. Affect measures were also collected from participants and suggest that the relationship between the condition and the participants’ willingness to fly was mediated by emotions. The findings of the study provide information on participants’ view of the arming of pilots and suggest that, for the most part, participants in this study seem supportive of FFDO and the arming of pilots of commercial airliners. This finding is limited to participants who take part in online human intelligence tasks through services such as MTurk, who may or may not have been a commercial flight passenger.

Expediting Combinatorial Data Set Analysis by Combining Human and Algorithmic Analysis.

A challenge in combinatorial materials science remains the efficient analysis of X-ray diffraction (XRD) data and its correlation to functional properties. Rapid identification of phase-regions and proper assignment of corresponding crystal structures is necessary to keep pace with the improved methods for synthesizing and characterizing materials libraries. Therefore, a new modular software called htAx (high-throughput analysis of X-ray and functional properties data) is presented that couples human intelligence tasks used for "ground-truth" phase-region identification with subsequent unbiased verification by an algorithm to efficiently analyze which phases are present in a materials library. Identified phases and phase-regions may then be correlated to functional properties in an expedited manner. For the functionality of htAx to be proven, two previously published XRD benchmark data sets of the materials systems Al-Cr-Fe-O and Ni-Ti-Cu are analyzed by htAx. The analysis of ∼1000 XRD patterns takes less than 1 day with htAx. The proposed method reliably identifies phase-region boundaries and robustly identifies multiphase structures. The method also addresses the problem of identifying regions with previously unpublished crystal structures using a special daisy ternary plot.

Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation

Purpose. To evaluate the feasibility of using Mechanical Turk as a massively parallel platform to perform manual segmentations of macular spectral domain optical coherence tomography (SD-OCT) images using a MapReduce framework. Methods. A macular SD-OCT volume of 61 slice images was map-distributed to Amazon Mechanical Turk. Each Human Intelligence Task was set to $0.01 and required the user to draw five lines to outline the sublayers of the retinal OCT image after being shown example images. Each image was submitted twice for segmentation, and interrater reliability was calculated. The interface was created using custom HTML5 and JavaScript code, and data analysis was performed using R. An automated pipeline was developed to handle the map and reduce steps of the framework. Results. More than 93,500 data points were collected using this framework for the 61 images submitted. Pearson's correlation of interrater reliability was 0.995 (p < 0.0001) and coefficient of determination was 0.991. The cost of segmenting the macular volume was $1.21. A total of 22 individual Mechanical Turk users provided segmentations, each completing an average of 5.5 HITs. Each HIT was completed in an average of 4.43 minutes. Conclusions. Amazon Mechanical Turk provides a cost-effective, scalable, high-availability infrastructure for manual segmentation of OCT images.

Crowdsourced Software Development Process: Investigation and Modeling through Markov Decision Theory

Crowdsourcing is a name given to a revolution that marks the rise of online community made up of like minded individuals who work together or compete with each other to deliver solutions to problems. It has been used for accomplishing various Human Intelligence Tasks in the past and is an emerging form of outsourcing software development as it has the potential to significantly lower the production cost. This paper analyses the process of software development at a Crowdsourced platform. The work investigates and identifies the phase wise deliverables in a competitive software development process. We propose the use of Markov Decision Process to model the dynamics of the development process and present an illustration of the proposed model.

Acquiring Planning Knowledge via Crowdsourcing

Plan synthesis often requires complete domain models and initial states as input. In many real world applications, it is difficult to build domain models and provide complete initial state beforehand. In this paper we propose to turn to the crowd for help before planning. We assume there are annotators available to provide information needed for building domain models and initial states. However, there might be a substantial amount of discrepancy within the inputs from the crowd. It is thus challenging to address the planning problem with possibly noisy information provided by the crowd. We address the problem by two phases. We first build a set of Human Intelligence Tasks (HITs), and collect values from the crowd. We then estimate the actual values of variables and feed the values to a planner to solve the problem.

Work experiences on MTurk: Job satisfaction, turnover, and information sharing

Amazon's Mechanical Turk (MTurk) is an online marketplace for work, where Requesters post Human Intelligence Tasks (HITs) for Workers to complete for varying compensation. Past research has focused on the quality and generalizability of social and behavioral science research conducted using MTurk as a source of research participants. However, MTurk and other crowdsourcing platforms also exemplify trends toward extremely short-term contract work. We apply principles of industrial–organizational (I–O) psychology to investigate MTurk Worker job satisfaction, information sharing, and turnover. We also report the top best and worst Requester behaviors (e.g., building a relationship, unfair pay) that affect Worker satisfaction. Worker satisfaction was consistently negatively related to turnover as expected, indicating that this traditional variable operates similarly in the MTurk work context. However, few of the traditional predictors of job satisfaction were significant, signifying that new operational definitions or entirely new variables may be needed in order to adequately understand the experiences of crowdsourced workers. Coworker friendships consistently predicted information sharing among Workers. The findings of this study are useful for understanding the experiences of crowdsourced workers from the perspective of I–O psychology, as well as for researchers using MTurk as a recruitment tool.