Crowdsourced Experiments Research Articles

Comparison of Human-Drone Distancing Studies Across In-person and Online Modalities

Human-robot proxemics behaviors can vary based on personal, robot, and environmental factors which, along with their deployment in public-facing interactions, calls for an in-depth exploration. This paper explores the impact of altitude and safety modifications of small unmanned aerial vehicle (sUAV) on users’ comfortable interaction distance. By leveraging interaction techniques from literature like video, sound, and simulations, we explore personal space interactions in online studies (N=376) with the sUAV and the Double telepresence robot. We then compare the findings with our in-person interaction data (N=47). While in-person interactions are the ultimate goal, online methods can be used to reduce resources, allow larger sample sizes, and may lead to a more comprehensive sampling of population than would be expected from in-person studies. The lessons learned from this work are applicable broadly within the social robotics community, even outside those who are interested in proxemics interactions, to conduct future crowd-sourced experiments. The various modalities provided similar trends when compared with data from in-person studies. While the distances may not have been precise compared to those measured in the real world, these experiments are useful to detect patterns in human-robot interactions, and to conduct formative studies before committing resources to in-person testing.

From Delays to Densities: Exploring Data Uncertainty through Speech, Text, and Visualization

AbstractUnderstanding and communicating data uncertainty is crucial for making informed decisions in sectors like finance and healthcare. Previous work has explored how to express uncertainty in various modes. For example, uncertainty can be expressed visually with quantile dot plots or linguistically with hedge words and prosody. Our research aims to systematically explore how variations within each mode contribute to communicating uncertainty to the user; this allows us to better understand each mode's affordances and limitations. We completed an exploration of the uncertainty design space based on pilot studies and ran two crowdsourced experiments examining how speech, text, and visualization modes and variants within them impact decision‐making with uncertain data. Visualization and text were most effective for rational decision‐making, though text resulted in lower confidence. Speech garnered the highest trust despite sometimes leading to risky decisions. Results from these studies indicate meaningful trade‐offs among modes of information and encourage exploration of multimodal data representations.

AccuStripes: Visual exploration and comparison of univariate data distributions using color and binning

Understanding and analyzing univariate distributions of data in terms of their shapes as well as their specific characteristics, regarding gaps, spikes, or outliers, is crucial in many scientific disciplines. In this paper, we propose a design space composed of the visual channels position and color for representing accumulated distributions. The designs are a mixture of color-coded stripes with density lines. The width and coloring of the stripes is based on the applied binning technique. In a crowd-sourced experiment we explore a subspace, called the AccuStripes (i.e., “accumulated stripes”) design space, consisting of nine representations. These AccuStripes designs integrate three composition strategies (color only, overlay, filled curve) with three binning techniques, one uniform (UB) and two adaptive methods, namely Bayesian Blocks (BB) and Jenks’ Natural Breaks (NB). We evaluate the accuracy, efficiency, and confidence ratings of the nine AccuStripes designs for structural estimation and comparison tasks. Across all study tasks, the overlay composition was found to be most accurate and preferred by observers. Furthermore, the results demonstrate that while no binning method performed best in both identification and comparison, detection of structures using adaptive binning was the most accurate one. For validation we compared the best AccuStripes’ design, i.e., the overlay composition, to line charts. Our results show that the AccuStripes’ design outperformed the line charts in accuracy for all study tasks.

Interpretable Compositional Representations for Robust Few-Shot Generalization.

We propose Recognition as Part Composition (RPC), an image encoding approach inspired by human cognition. It is based on the cognitive theory that humans recognize complex objects by components, and that they build a small compact vocabulary of concepts to represent each instance with. RPC encodes images by first decomposing them into salient parts, and then encoding each part as a mixture of a small number of prototypes, each representing a certain concept. We find that this type of learning inspired by human cognition can overcome hurdles faced by deep convolutional networks in low-shot generalization tasks, like zero-shot learning, few-shot learning and unsupervised domain adaptation. Furthermore, we find a classifier using an RPC image encoder is fairly robust to adversarial attacks, that deep neural networks are known to be prone to. Given that our image encoding principle is based on human cognition, one would expect the encodings to be interpretable by humans, which we find to be the case via crowd-sourcing experiments. Finally, we propose an application of these interpretable encodings in the form of generating synthetic attribute annotations for evaluating zero-shot learning methods on new datasets.

My Model is Unfair, Do People Even Care? Visual Design Affects Trust and Perceived Bias in Machine Learning.

Machine learning technology has become ubiquitous, but, unfortunately, often exhibits bias. As a consequence, disparate stakeholders need to interact with and make informed decisions about using machine learning models in everyday systems. Visualization technology can support stakeholders in understanding and evaluating trade-offs between, for example, accuracy and fairness of models. This paper aims to empirically answer "Can visualization design choices affect a stakeholder's perception of model bias, trust in a model, and willingness to adopt a model?" Through a series of controlled, crowd-sourced experiments with more than 1,500 participants, we identify a set of strategies people follow in deciding which models to trust. Our results show that men and women prioritize fairness and performance differently and that visual design choices significantly affect that prioritization. For example, women trust fairer models more often than men do, participants value fairness more when it is explained using text than as a bar chart, and being explicitly told a model is biased has a bigger impact than showing past biased performance. We test the generalizability of our results by comparing the effect of multiple textual and visual design choices and offer potential explanations of the cognitive mechanisms behind the difference in fairness perception and trust. Our research guides design considerations to support future work developing visualization systems for machine learning.

Quantifying Emotional Responses to Immutable Data Characteristics and Designer Choices in Data Visualizations.

Emotion is an important factor to consider when designing visualizations as it can impact the amount of trust viewers place in a visualization, how well they can retrieve information and understand the underlying data, and how much they engage with or connect to a visualization. We conducted five crowdsourced experiments to quantify the effects of color, chart type, data trend, data variability and data density on emotion (measured through self-reported arousal and valence). Results from our experiments show that there are multiple design elements which influence the emotion induced by a visualization and, more surprisingly, that certain data characteristics influence the emotion of viewers even when the data has no meaning. In light of these findings, we offer guidelines on how to use color, scale, and chart type to counterbalance and emphasize the emotional impact of immutable data characteristics.

Rainbow Colormaps: What are they good and bad for?

Guidelines for color use in quantitative visualizations have strongly discouraged the use of rainbow colormaps, arguing instead for smooth designs that do not induce visual discontinuities or implicit color categories. However, the empirical evidence behind this argument has been mixed and, at times, even contradictory. In practice, rainbow colormaps are widely used, raising questions about the true utility or dangers of such designs. We study how color categorization impacts the interpretation of scalar fields. We first introduce an approach to detect latent categories in colormaps. We hypothesize that the appearance of color categories in scalar visualizations can be beneficial in that they enhance the perception of certain features, although at the cost of rendering other features less noticeable. In three crowdsourced experiments, we show that observers are more likely to discriminate global, distributional features when viewing colorful scales that induce categorization (e.g., rainbow or diverging schemes). Conversely, when seeing the same data through a less colorful representation, observers are more likely to report localized features defined by small variations in the data. Participants showed awareness of these different affordances, and exhibited bias for exploiting the more discriminating colormap, given a particular feature type. Our results demonstrate costs and benefits for rainbows (and similarly colorful schemes), suggesting that their complementary utility for analyzing scalar data should not be dismissed. In addition to explaining potentially valid uses of rainbow, our study provides actionable guidelines, including on when such designs can be more harmful than useful. Data and materials are available at https://osf.io/xjhtf.

Clarifying the Effect of Edge Targets in Touch Pointing through Crowdsourced Experiments

A prior work has recommended adding a 4-mm gap between a target and the edge of a screen, as tapping a target located at the screen edge takes longer than tapping non-edge targets. However, it is possible that this recommendation was created based on statistical errors, and unexplored situations existed in the prior work. In this study, we re-examine the recommendation by utilizing crowdsourced experiments to resolve the issues. If we observe the same results as the prior work through experiments including diversities, we can verify that the recommendation is suitable. We found that increasing the gap between the target and the screen edge decreased the movement time, which was consistent with the prior work. In addition, we newly found that increasing the gap decreased the error rate as well. On the basis of these results, we discuss how the gap and the target should be designed.

Modeling Bilingualism as a Dynamic Phenomenon in Healthy and Neurologically Affected Speakers Across the Lifespan: A Commentary on "Computational Modeling of Bilingual Language Learning: Current Models and Future Directions".

Modeling Bilingualism as a Dynamic Phenomenon in Healthy and Neurologically Affected Speakers Across the Lifespan: A Commentary on "Computational Modeling of Bilingual Language Learning: Current Models and Future Directions".

Designing With Pictographs: Envision Topics Without Sacrificing Understanding.

Past studies have shown that when a visualization uses pictographs to encode data, they have a positive effect on memory, engagement, and assessment of risk. However, little is known about how pictographs affect one's ability to understand a visualization, beyond memory for values and trends. We conducted two crowdsourced experiments to compare the effectiveness of using pictographs when showing part-to-whole relationships. In Experiment 1, we compared pictograph arrays to more traditional bar and pie charts. We tested participants' ability to generate high-level insights following Bloom's taxonomy of educational objectives via 6 free-response questions. We found that accuracy for extracting information and generating insights did not differ overall between the two versions. To explore the motivating differences between the designs, we conducted a second experiment where participants compared charts containing pictograph arrays to more traditional charts on 5 metrics and explained their reasoning. We found that some participants preferred the way that pictographs allowed them to envision the topic more easily, while others preferred traditional bar and pie charts because they seem less cluttered and faster to read. These results suggest that, at least in simple visualizations depicting part-to-whole relationships, the choice of using pictographs has little influence on sensemaking and insight extraction. When deciding whether to use pictograph arrays, designers should consider visual appeal, perceived comprehension time, ease of envisioning the topic, and clutteredness.

Relaxed Dot Plots: Faithful Visualization of Samples and Their Distribution.

We introduce relaxed dot plots as an improvement of nonlinear dot plots for unit visualization. Our plots produce more faithful data representations and reduce moiré effects. Their contour is based on a customized kernel frequency estimation to match the shape of the distribution of underlying data values. Previous nonlinear layouts introduce column-centric nonlinear scaling of dot diameters for visualization of high-dynamic-range data with high peaks. We provide a mathematical approach to convert that column-centric scaling to our smooth envelope shape. This formalism allows us to use linear, root, and logarithmic scaling to find ideal dot sizes. Our method iteratively relaxes the dot layout for more correct and aesthetically pleasing results. To achieve this, we modified Lloyd's algorithm with additional constraints and heuristics. We evaluate the layouts of relaxed dot plots against a previously existing nonlinear variant and show that our algorithm produces less error regarding the underlying data while establishing the blue noise property that works against moiré effects. Further, we analyze the readability of our relaxed plots in three crowd-sourced experiments. The results indicate that our proposed technique surpasses traditional dot plots.

Left, Right, and Gender: Exploring Interaction Traces to Mitigate Human Biases.

Human biases impact the way people analyze data and make decisions. Recent work has shown that some visualization designs can better support cognitive processes and mitigate cognitive biases (i.e., errors that occur due to the use of mental "shortcuts"). In this work, we explore how visualizing a user's interaction history (i.e., which data points and attributes a user has interacted with) can be used to mitigate potential biases that drive decision making by promoting conscious reflection of one's analysis process. Given an interactive scatterplot-based visualization tool, we showed interaction history in real-time while exploring data (by coloring points in the scatterplot that the user has interacted with), and in a summative format after a decision has been made (by comparing the distribution of user interactions to the underlying distribution of the data). We conducted a series of in-lab experiments and a crowd-sourced experiment to evaluate the effectiveness of interaction history interventions toward mitigating bias. We contextualized this work in a political scenario in which participants were instructed to choose a committee of 10 fictitious politicians to review a recent bill passed in the U.S. state of Georgia banning abortion after 6 weeks, where things like gender bias or political party bias may drive one's analysis process. We demonstrate the generalizability of this approach by evaluating a second decision making scenario related to movies. Our results are inconclusive for the effectiveness of interaction history (henceforth referred to as interaction traces) toward mitigating biased decision making. However, we find some mixed support that interaction traces, particularly in a summative format, can increase awareness of potential unconscious biases.

When Does Status Transfer between People? A Crowdsourced Experiment on the Scope of Status by Association

It is well known in social psychology that people are judged by the company they keep, but when and how does that company affect how individuals are evaluated? This article extends expectation states theory to explain associative status. The theory predicts that the status value of former coworkers will “spill over” to positively predict a person’s status position in a new task with new coworkers. A series of crowdsourced experiments finds that status spreads to a person from a former interaction partner. The status of one’s associates predicts deference behavior only when the previous and current task contexts rely on similar abilities. Meanwhile, explicitly evaluated status and performance expectations respond to the status of associates regardless of how interaction contexts are related. The present findings highlight the importance of role relationships and task contexts as moderators that regulate whether status transfers from one person to another.

Crowdsourcing and Evaluating Concept-driven Explanations of Machine Learning Models

An important challenge in building explainable artificially intelligent (AI) systems is designing interpretable explanations. AI models often use low-level data features which may be hard for humans to interpret. Recent research suggests that situating machine decisions in abstract, human understandable concepts can help. However, it is challenging to determine the right level of conceptual mapping. In this research, we explore granularity (of data features) and context (of data instances) as dimensions underpinning conceptual mappings. Based on these measures, we explore strategies for designing explanations in classification models. We introduce an end-to-end concept elicitation pipeline that supports gathering high-level concepts for a given data set. Through crowd-sourced experiments, we examine how providing conceptual information shapes the effectiveness of explanations, finding that a balance between coarse and fine-grained explanations help users better estimate model predictions. We organize our findings into systematic themes that can inform design considerations for future systems.

Longer Delays in Rehearsal-based Interfaces Increase Expert Use

Rehearsal-based interfaces are designed to encourage a transition from novice to expert, but many users fail to make this transition. Most of these interfaces activate novice mode after a short delay, between 150 and 500 ms. We investigate the impact of delay time on expert usage and learning in three crowdsourced experiments. The first experiment examines an 8-item marking menu with delay times from 200 ms to 2 s. Results show longer delays increase successful expert selections. The second and third experiments generalise this result to a different rehearsal-based menu, a desktop clone of FastTap with 8 items and 15 items. Together, our results show that expert use correlates positively with increased delay time, but can increase errors since users are less risk averse. We also find imperceptible delays of 200 ms can harm long-term retention of menu items. Designers should consider longer delays in rehearsal-based interfaces to encourage a transition to expert usage.

Revealing Perceptual Proxies with Adversarial Examples.

Data visualizations convert numbers into visual marks so that our visual system can extract data from an image instead of raw numbers. Clearly, the visual system does not compute these values as a computer would, as an arithmetic mean or a correlation. Instead, it extracts these patterns using perceptual proxies; heuristic shortcuts of the visual marks, such as a center of mass or a shape envelope. Understanding which proxies people use would lead to more effective visualizations. We present the results of a series of crowdsourced experiments that measure how powerfully a set of candidate proxies can explain human performance when comparing the mean and range of pairs of data series presented as bar charts. We generated datasets where the correct answer-the series with the larger arithmetic mean or range-was pitted against an "adversarial" series that should be seen as larger if the viewer uses a particular candidate proxy. We used both Bayesian logistic regression models and a robust Bayesian mixed-effects linear model to measure how strongly each adversarial proxy could drive viewers to answer incorrectly and whether different individuals may use different proxies. Finally, we attempt to construct adversarial datasets from scratch, using an iterative crowdsourcing procedure to perform black-box optimization.

Disentangling the Independent Contributions of Visual and Conceptual Features to the Spatiotemporal Dynamics of Scene Categorization.

Human scene categorization is characterized by its remarkable speed. While many visual and conceptual features have been linked to this ability, significant correlations exist between feature spaces, impeding our ability to determine their relative contributions to scene categorization. Here, we used a whitening transformation to decorrelate a variety of visual and conceptual features and assess the time course of their unique contributions to scene categorization. Participants (both sexes) viewed 2250 full-color scene images drawn from 30 different scene categories while having their brain activity measured through 256-channel EEG. We examined the variance explained at each electrode and time point of visual event-related potential (vERP) data from nine different whitened encoding models. These ranged from low-level features obtained from filter outputs to high-level conceptual features requiring human annotation. The amount of category information in the vERPs was assessed through multivariate decoding methods. Behavioral similarity measures were obtained in separate crowdsourced experiments. We found that all nine models together contributed 78% of the variance of human scene similarity assessments and were within the noise ceiling of the vERP data. Low-level models explained earlier vERP variability (88 ms after image onset), whereas high-level models explained later variance (169 ms). Critically, only high-level models shared vERP variability with behavior. Together, these results suggest that scene categorization is primarily a high-level process, but reliant on previously extracted low-level features.SIGNIFICANCE STATEMENT In a single fixation, we glean enough information to describe a general scene category. Many types of features are associated with scene categories, ranging from low-level properties, such as colors and contours, to high-level properties, such as objects and attributes. Because these properties are correlated, it is difficult to understand each property's unique contributions to scene categorization. This work uses a whitening transformation to remove the correlations between features and examines the extent to which each feature contributes to visual event-related potentials over time. We found that low-level visual features contributed first but were not correlated with categorization behavior. High-level features followed 80 ms later, providing key insights into how the brain makes sense of a complex visual world.

Information Theoretic Model to Simulate Agent-Signage Interaction for Wayfinding

Signage systems are critical for communicating spatial information during wayfinding among a plethora of noise in the environment. A proper signage system can improve wayfinding performance and user experience by reducing the perceived complexity of the environment. However, previous models of sign-based wayfinding do not incorporate realistic noise or quantify the reduction in perceived complexity from the use of signage. Drawing upon concepts from information theory, we propose and validate a new agent-signage interaction model that quantifies available wayfinding information from signs for wayfinding. We conducted two online crowd-sourcing experiments to compute the distribution of a sign’s visibility and an agent’s decision-making confidence as a function of observation angle and viewing distance. We then validated this model using a virtual reality (VR) experiment with trajectories from human participants. The crowd-sourcing experiments provided a distribution of decision-making entropy (conditioned on visibility) that can be applied to any sign/environment. From the VR experiment, a training dataset of 30 trajectories was used to refine our model, and the remaining test dataset of 10 trajectories was compared with agent behavior using dynamic time warping (DTW) distance. The results revealed a reduction of 38.76% in DTW distance between the average trajectories before and after refinement. Our refined agent-signage interaction model provides realistic predictions of human wayfinding behavior using signs. These findings represent a first step towards modeling human wayfinding behavior in complex real environments in a manner that can incorporate several additional random variables (e.g., environment layout).

Illusion of Causality in Visualized Data

Students who eat breakfast more frequently tend to have a higher grade point average. From this data, many people might confidently state that a before-school breakfast program would lead to higher grades. This is a reasoning error, because correlation does not necessarily indicate causation - X and Y can be correlated without one directly causing the other. While this error is pervasive, its prevalence might be amplified or mitigated by the way that the data is presented to a viewer. Across three crowdsourced experiments, we examined whether how simple data relations are presented would mitigate this reasoning error. The first experiment tested examples similar to the breakfast-GPA relation, varying in the plausibility of the causal link. We asked participants to rate their level of agreement that the relation was correlated, which they rated appropriately as high. However, participants also expressed high agreement with a causal interpretation of the data. Levels of support for the causal interpretation were not equally strong across visualization types: causality ratings were highest for text descriptions and bar graphs, but weaker for scatter plots. But is this effect driven by bar graphs aggregating data into two groups or by the visual encoding type? We isolated data aggregation versus visual encoding type and examined their individual effect on perceived causality. Overall, different visualization designs afford different cognitive reasoning affordances across the same data. High levels of data aggregation by graphs tend to be associated with higher perceived causality in data. Participants perceived line and dot visual encodings as more causal than bar encodings. Our results demonstrate how some visualization designs trigger stronger causal links while choosing others can help mitigate unwarranted perceptions of causality.

True contextuality in a psychophysical experiment

Recent crowdsourcing experiments have shown that true contextuality of the kind found in quantum mechanics can also be present in human behavior. In these experiments simple human choices were aggregated over large numbers of respondents, with each respondent dealing with a single context (set of questions asked). In this paper we present experimental evidence of contextuality in individual human behavior, in a psychophysical experiment with repeated presentations of visual stimuli in randomly varying conteXts (arrangements of stimuli). The analysis is based on the Contextuality-by-Default (CbD) theory whose relevant aspects are reviewed in the paper. CbD allows one to detect contextuality in the presence of direct influences, i.e., when responses to the same stimuli have different distributions in different contexts. The experiment presented is also the first one in which contextuality is demonstrated for responses that are not dichotomous, with five options to choose among. CbD requires that random variables representing such responses be dichotomized before they are subjected to contextuality analysis. A theorem says that a system consisting of all possible dichotomizations of responses has to be contextual if these responses violate a certain condition, called nominal dominance. In our experiment nominal dominance was violated in all data sets, with very high statistical reliability established by bootstrapping.