Mouse-tracking Data Research Articles

Clustering computer mouse tracking data with informed hierarchical shrinkage partition priors.

Mouse-tracking data, which record computer mouse trajectories while participants perform an experimental task, provide valuable insights into subjects' underlying cognitive processes. Neuroscientists are interested in clustering the subjects' responses during computer mouse-tracking tasks to reveal patterns of individual decision-making behaviors and identify population subgroups with similar neurobehavioral responses. These data can be combined with neuroimaging data to provide additional information for personalized interventions. In this article, we develop a novel hierarchical shrinkage partition (HSP) prior for clustering summary statistics derived from the trajectories of mouse-tracking data. The HSP model defines a subjects' cluster as a set of subjects that gives rise to more similar (rather than identical) nested partitions of the conditions. The proposed model can incorporate prior information about the partitioning of either subjects or conditions to facilitate clustering, and it allows for deviations of the nested partitions within each subject group. These features distinguish the HSP model from other bi-clustering methods that typically create identical nested partitions of conditions within a subject group. Furthermore, it differs from existing nested clustering methods, which define clusters based on common parameters in the sampling model and identify subject groups by different distributions. We illustrate the unique features of the HSP model on a mouse tracking dataset from a pilot study and in simulation studies. Our results show the ability and effectiveness of the proposed exploratory framework in clustering and revealing possible different behavioral patterns across subject groups.

Conflict in moral and nonmoral decision making: an empirical study coupled with a computational model.

While moral psychology research has extensively studied decision making using moral dilemmas, such high-conflict situations may not fully represent all moral decisions. Moreover, most studies on the effect of conflict have focused on nonmoral decisions, and it is unclear how it applies to the moral realm. The present mixed-method research investigates how conflict impacts moral compared to nonmoral decision making. In a preregistered empirical study ( ), participants made moral and nonmoral decisions with varying levels of conflict while their mouse trajectories were recorded. Results indicate that moral decisions were more stable in the presence of conflict, while still seeking compromise. In addition, decisions were more affected when conflict got higher. Mouse-tracking data further indicate that some factors are impacting the decision process earlier than others, supporting the relevance of tracing methods to dig into finer-grained decision dynamics. We also present a computational model that aims to capture decision mechanisms and how conflict and morality influence decision making. The model uses dynamic neural fields coupled with sensorimotor control to map a continuous decision space. Two model versions were compared: one with greater perceptual weight for moral information, and another with earlier processing of moral versus nonmoral information. The simulated data more successfully reproduced empirical patterns for the second version, thus providing insights into the underlying decision processes for both moral and nonmoral decisions, in the presence of conflict or not.

Investigating the role of bilingualism and gender on theory of mind using mouse-tracking and self-report

Aims and Objectives: Although there exists empirical support for the idea that bilingualism is associated with improved social cognition, the presence and the structure of this link remains inconclusive. There is currently a need to clarify the specific contexts in which these benefits may tend to emerge. Methodology: We sought to advance the field of bilingualism by examining the link between bilingualism and a specific form of social cognition, theory of mind. We measured theory of mind processing in a sample of adult bilinguals and monolinguals and measured patterns of decision-making using several different methods; task performance (accuracy and reaction time), implicit decision-making patterns (mouse-tracking), and self-report. We collected mouse-tracking data during two different tasks (false belief and cognitive–affective theory of mind) and self-report was measured using the perspective-taking (PT) subscale of the Interpersonal Reactivity Index (IRI). Data and Analysis: For each behavioral task, we compared reaction time, accuracy, and both mouse-tracking metrics between language groups. For self-report, we compared PT scores between language groups. We also tested for interactive effects of gender and language group on each dependent variable. Findings: Across all participants, bilingualism was not associated with any statistically significant improvements in theory of mind. However, we did observe that gender interacted with language group to affect maximal deviation values (mouse-tracking) during the false belief task and self-reported PT scores. Bilingualism tended to confer some improvements in self-reported theory of mind (PT subscale of IRI) in men, but not women. Originality: It is currently unknown if bilingualism is associated with improved theory of mind processing when measured using several different types of tasks and using mouse-tracking. Significance/Implications: These findings contribute to a growing body of evidence elucidating the complex relationship that exists between multiple language acquisition and social cognition.

Mouse tracking performance: A new approach to analyzing continuous mouse tracking data.

Mouse tracking is an important source of data in cognitive science. Most contemporary mouse tracking studies use binary-choice tasks and analyze the curvature or velocity of an individual mouse movement during an experimental trial as participants select from one of the two options. However, there are many types of mouse tracking data available beyond what is produced in a binary-choice task, including naturalistic data from web users. In order to utilize these data, cognitive scientists need tools that are robust to the lack of trial-by-trial structure in most normal computer tasks. We use singular value decomposition (SVD) and detrended fluctuation analysis (DFA) to analyze whole time series of unstructured mouse movement data. We also introduce a new technique for describing two-dimensional mouse traces as complex-valued time series, which allows SVD and DFA to be applied in a straightforward way without losing important spatial information. We find that there is useful information at the level of whole time series, and we use this information to predict performance in an online task. We also discuss how the implications of these results can advance the use of mouse tracking research in cognitive science.

When ethics also matter: Influence of taste, health, and ethical attributes on food decisions traced with a novel mouse-tracking paradigm

Understanding which food attributes influence food decisions is a matter of public health and a lever for interventions promoting healthy diets. Research shows that food decisions are strongly influenced by taste, with health having a weaker and later influence in the food decision process. Yet, the influence of other food attributes and specifically ethical attributes in food decision processes—as traceable in mouse-tracking data—has not been investigated. Furthermore, past research tracing food decision processes with classical mouse-tracking tools has artificially reduced the occurrence of neutral food items, particularly on the taste attribute. This represents an important limitation as neutral items on taste are particularly likely to be influenced by higher-order level attributes, such as health, but also ethics. Extending previous research, two preregistered studies (Study 1, N = 77; Study 2, N = 92) aimed at filling these gaps using a novel one-dimensional mouse-tracking paradigm. Results showed that taste, health, and ethics all influenced food decisions and interacted over time during decision processes. Taste still had the strongest influence, hence replicating previous findings with the present novel mouse-tracking paradigm. Of importance, ethics and health also influenced decisions—and sometimes had an early significant effect—especially for food items rated as neutral on taste. Beyond these effects and taking full advantage of the use of mixed effects models for all analyses, graphical representations of the influence of taste, health, and ethical attributes for all individual food items were provided. Results are discussed considering previous findings and suggested levers for interventions.

Privacy Attitudes toward Mouse-Tracking Paradata Collection.

Survey participants' mouse movements provide a rich, unobtrusive source of paradata, offering insight into the response process beyond the observed answers. However, the use of mouse tracking may require participants' explicit consent for their movements to be recorded and analyzed. Thus, the question arises of how its presence affects the willingness of participants to take part in a survey at all-if prospective respondents are reluctant to complete a survey if additional measures are recorded, collecting paradata may do more harm than good. Previous research has found that other paradata collection modes reduce the willingness to participate, and that this decrease may be influenced by the specific motivation provided to participants for collecting the data. However, the effects of mouse movement collection on survey consent and participation have not been addressed so far. In a vignette experiment, we show that reported willingness to participate in a survey decreased when mouse tracking was part of the overall consent. However, a larger proportion of the sample indicated willingness to both take part and provide mouse-tracking data when these decisions were combined, compared to an independent opt-in to paradata collection, separated from the decision to complete the study. This suggests that survey practitioners may face a trade-off between maximizing their overall participation rate and maximizing the number of participants who also provide mouse-tracking data. Explaining motivations for paradata collection did not have a positive effect and, in some cases, even reduced participants' reported willingness to take part in the survey.

Complex economic decisions from simple neurocognitive processes: the role of interactive attention.

Neurocognitive theories of value-based choice propose that people additively accumulate choice attributes when making decisions. These theories cannot explain the emergence of complex multiplicative preferences such as those assumed by prospect theory and other economic models. We investigate an interactive attention mechanism, according to which attention to attributes (like payoffs) depends on other attributes (like probabilities) attended to previously. We formalize this mechanism using a Markov attention model combined with an accumulator decision process, and test our model on eye-tracking and mouse-tracking data in risky choice. Our tests show that interactive attention is necessary to make good choices, that most participants display interactive attention and that allowing for interactive attention in accumulation-based decision models improves their predictions. By equipping established decision models with sophisticated attentional dynamics, we extend these models to describe complex economic choice, and in the process, we unify two prominent theoretical approaches to studying value-based decision making.

Temporal aspects of two types of backward crosstalk in dual-tasks: An analysis of continuous mouse-tracking data.

A common explanation for processing limitations in dual-tasking is the existence of a bottleneck during response selection, meaning that the selection of responses can only occur serially for different tasks. However, a large body of data shows that features of a (secondary) Task 2 can already influence the processing of a (primary) Task 1. Such effects are referred to as backward crosstalk effects (BCEs). In the present study, two types of such BCEs were investigated: the compatibility-based BCE, which depends on the dimensional (often spatial) overlap between task features, and a BCE based on a go/no-go task in Task 2 (no-go BCE). Joining a line of research that suggests different mechanisms for these two types of BCEs, we investigated them using a mouse-tracking setup. Time continuous analyses revealed that the compatibility-based BCE triggered a spatial activation of the Task 2 response early during Task 1 processing, whereas the no-go BCE triggered an inhibitory effect in the case of a no-go Task 2, which spills over to Task 1 execution. This occurred, however, earlier in the time course than expected. The results are discussed with regard to recent models of dual-task processing.

Taking the perspective of narrative characters

Abstract An ongoing debate in the interpretation of referring expressions concerns the degree to which listeners make use of perspective information during referential processing. We aim to contribute to this debate by considering perspective shifting in narrative discourse. In a web-based mouse-tracking experiment in Dutch, we investigated whether listeners automatically shift to a narrative character’s perspective when resolving ambiguous referring expressions, and whether different linguistic perspective-shifting devices affect how and when listeners switch to another perspective. We compared perspective-neutral, direct, and free indirect discourse, manipulating which objects are visible to the character. Our results do not show a clear effect of the perspective shifting devices on participants’ eventual choice of referent, but our online mouse-tracking data reveal processing differences that suggest that listeners are indeed sensitive to the conventional markers of perspective shift associated with direct and (to a lesser degree) free indirect discourse.

Integrating Eye- and Mouse-Tracking with Assistant Based Speech Recognition for Interaction at Controller Working Positions

Assistant based speech recognition (ABSR) prototypes for air traffic controllers have demonstrated to reduce controller workload and aircraft flight times as a result. However, two aspects of ABSR could enhance benefits, i.e., (1) the predicted controller commands that speech recognition engines use can be more accurate, and (2) the confirmation process of ABSR recognition output, such as callsigns, command types, and values by the controller, can be less intrusive. Both tasks can be supported by unobtrusive eye- and mouse-tracking when using operators’ gaze and interaction data. First, probabilities for predicted commands should consider controllers’ visual focus on the situation data display. Controllers will more likely give commands to aircraft that they focus on or where there was a mouse interaction on the display. Furthermore, they will more likely give certain command types depending on the characteristics of multiple aircraft being scanned. Second, it can be determined via eye-tracking instead of additional mouse clicks if the displayed ABSR output has been checked by the controller and remains uncorrected for a certain amount of time. Then, the output is assumed to be correct and is usable by other air traffic control systems, e.g., short-term conflict alert. If the ABSR output remains unchecked, an attention guidance functionality triggers different escalation levels to display visual cues. In a one-shot experimental case study with two controllers for the two implemented techniques, (1) command prediction probabilities improved by a factor of four, (2) prediction error rates based on an accuracy metric for three most-probable aircraft decreased by a factor of 25 when combining eye- and mouse-tracking data, and (3) visual confirmation of ABSR output promises to be an alternative for manual confirmation.

Predicting Question Difficulty in Web Surveys: A Machine Learning Approach Based on Mouse Movement Features

Survey research aims to collect robust and reliable data from respondents. However, despite researchers’ efforts in designing questionnaires, survey instruments may be imperfect, and question structure not as clear as could be, thus creating a burden for respondents. If it were possible to detect such problems, this knowledge could be used to predict problems in a questionnaire during pretesting, inform real-time interventions through responsive questionnaire design, or to indicate and correct measurement error after the fact. Previous research has used paradata, specifically response times, to detect difficulties and help improve user experience and data quality. Today, richer data sources are available, for example, movements respondents make with their mouse, as an additional detailed indicator for the respondent–survey interaction. This article uses machine learning techniques to explore the predictive value of mouse-tracking data regarding a question’s difficulty. We use data from a survey on respondents’ employment history and demographic information, in which we experimentally manipulate the difficulty of several questions. Using measures derived from mouse movements, we predict whether respondents have answered the easy or difficult version of a question, using and comparing several state-of-the-art supervised learning methods. We have also developed a personalization method that adjusts for respondents’ baseline mouse behavior and evaluate its performance. For all three manipulated survey questions, we find that including the full set of mouse movement measures and accounting for individual differences in these measures improve prediction performance over response-time-only models.

Usability Evaluation using Eye-Tracking on E-Commerce and Education Domains

The rapid consumption of content on the web and the creation of online businesses and educational platforms require improved usability for the user. It also calls for concerted effort to understand users' interactions with these digital products and services. This study was carried out to investigate the search and scan patterns employed by users interacting with the selected web interface from the e-commerce and education domains and also to establish the possibility of a localized interface design patterns in Nigeria. Usability evaluation was conducted using OGAMA software as an eye-tracking tool, to record as well as to analyze eye and mouse tracking data from slideshow eye-tracking experiments concurrently. During the evaluation period, fixation count, fixation length and saccade length of each website were taken into the consideration. Findings revealed that educational based site delivers users’ need than that of e-commerce due to less level of distraction as a result of fewer images and pictures. The study recommends that interface designs should be void of the display of excessive ad contents and include proper formatting of navigation items to contribute towards an enhanced user experience.

Implementation of real-time online mouse tracking on overseas quiz session

Mouse tracking serves as an alternative to eye tracking in measuring the learning process in education because of its affordability. Moreover, mouse tracking does not require extra hardware, as in the case of eye tracking, because it is a feature in personal computers by default. Therefore, it is possible to implement mouse tracking in a massive open scale. However, mouse tracking has only been implemented in a laboratory setting to date, ostensibly because of the associated extremely high running costs. Nonetheless, there is no available data to support the claim of high resource costs, which has resulted in much speculation among implementers. In general, the implementation of mouse tracking in a non-laboratory environment is still rare. Therefore, the authors developed an application to investigate real-time mouse tracking online. It was implemented on the Moodle learning management system and tested on an online quiz session accessed abroad. Additionally, the application can handle tracking on mobile devices. In this work, the main resources that include CPU, network, RAM, and storage costs were measured when mouse tracking was used. These results can serve as a reference for network and server administrators during future implementation of this technique. It was determined that the characteristics of mouse activities were dynamic in that occasional surges and lulls were observed. Additionally, this article also discussed the advantage of real-time and online implementation to regular online implementation and showed that there is a possibility of implementing mouse tracking on a large scale if mouse tracking data are not aggregated and transmitted as a single data package.

A State Space Approach to Dynamic Modeling of Mouse-Tracking Data.

Mouse-tracking recording techniques are becoming very attractive in experimental psychology. They provide an effective means of enhancing the measurement of some real-time cognitive processes involved in categorization, decision-making, and lexical decision tasks. Mouse-tracking data are commonly analyzed using a two-step procedure which first summarizes individuals' hand trajectories with independent measures, and then applies standard statistical models on them. However, this approach can be problematic in many cases. In particular, it does not provide a direct way to capitalize the richness of hand movement variability within a consistent and unified representation. In this article we present a novel, unified framework for mouse-tracking data. Unlike standard approaches to mouse-tracking, our proposal uses stochastic state-space modeling to represent the observed trajectories in terms of both individual movement dynamics and experimental variables. The model is estimated via a Metropolis-Hastings algorithm coupled with a non-linear recursive filter. The characteristics and potentials of the proposed approach are illustrated using a lexical decision case study. The results highlighted how dynamic modeling of mouse-tracking data can considerably improve the analysis of mouse-tracking tasks and the conclusions researchers can draw from them.

When the eyes say buy: visual fixations during hypothetical consumer choice improve prediction of actual purchases

Consumers typically overstate their intentions to purchase products, compared to actual rates of purchases, a pattern called “hypothetical bias”. In laboratory choice experiments, we measure participants’ visual attention using mousetracking or eye-tracking, while they make hypothetical as well as real purchase decisions. We find that participants spent more time looking both at price and product image prior to making a real “buy” decision than making a real “don’t buy” decision. We demonstrate that including such information about visual attention improves prediction of real buy decisions. This improvement is evident, although small in magnitude, using mousetracking data, but is not evident using eye-tracking data.

Design factors in mouse-tracking: What makes a difference?

Investigating cognitive processes by analyzing mouse movements has become a popular method in many psychological disciplines. When creating mouse-tracking experiments, researchers face many design choices-for example, whether participants indicate responses by clicking a button or just by entering the button area. Hitherto, numerous different settings have been employed, but little is known about how these methodological differences affect mouse-tracking data. We systematically investigated the influences of three central design factors, using a classic mouse-tracking paradigm in which participants classified typical and atypical exemplars. In separate experiments, we manipulated the response indication, mouse sensitivity, and starting procedure. The core finding that mouse movements deviate more toward the nonchosen option for atypical exemplars was replicated in all conditions. However, the size of this effect varied. Specifically, it was larger when participants indicated responses via click and when they were instructed to initialize the movement early. Trajectory shapes also differed between setups. For example, a dynamic start led to mostly curved trajectories, responses via click led to a mix of straight and "change-of-mind" trajectories, and responses via touch led to mostly straight trajectories. Moreover, the distribution of curvature indices was classified as bimodal in some setups and as unimodal in others. Because trajectory curvature and shape are frequently used to make inferences about psychological theories, such as differentiating between dynamic and dual-system models, this study shows that the specific design must be carefully considered when drawing theoretical inferences. All methodological designs and analyses were implemented using open-source software and are available from https://osf.io/xdp7a/.

User-interfaces layout optimization using eye-tracking, mouse movements and genetic algorithms

Establishing the best layout configuration for software-generated interfaces and control panels is a complex problem when they include many controls and indicators. Several methods have been developed for arranging the interface elements; however, the results are usually conceptual designs that must be manually adjusted to obtain layouts valid for real situations. Based on these considerations, in this work we propose a new automatized procedure to obtain optimal layouts for software-based interfaces. Eye-tracking and mouse-tracking data collected during the use of the interface is used to obtain the best configuration for its elements. The solutions are generated using a slicing-trees based genetic algorithm. This algorithm is able to obtain really applicable configurations that respect the geometrical restrictions of elements in the interface. Results show that this procedure increases effectiveness, efficiency and satisfaction of the users when they interact with the obtained interfaces.

Use of Mouse-tracking Method to Measure Cognitive Load

Integrated with computerized education platforms, the mouse-tracking technique could provide an inexpensive and less intrusive tool for assessing cognitive load. The present study examined whether mouse-tracking can quantify changes in cognitive load. Participants performed a dual-task, which required them to perform primary tasks of moving a computer mouse cursor along the vertical and the horizontal axes to a target, and secondary arithmetic tasks designed to impose different levels of cognitive load. Analyses of the mouse-tracking data indicated that slower mean response time and less trajectory deviation were observed when participants were given secondary tasks imposing a greater cognitive load, whereas slower mean response time and greater trajectory deviation were observed when participants moved a cursor toward a smaller-sized target. The cause behind the quantitative difference between the cognitive load effect, and the motor task difficulty (target size) is discussed, as are implications of these results for computerized education platforms.

Detecting Multiple Random Changepoints in Bayesian Piecewise Growth Mixture Models.

Piecewise growth mixture models are a flexible and useful class of methods for analyzing segmented trends in individual growth trajectory over time, where the individuals come from a mixture of two or more latent classes. These models allow each segment of the overall developmental process within each class to have a different functional form; examples include two linear phases of growth, or a quadratic phase followed by a linear phase. The changepoint (knot) is the time of transition from one developmental phase (segment) to another. Inferring the location of the changepoint(s) is often of practical interest, along with inference for other model parameters. A random changepoint allows for individual differences in the transition time within each class. The primary objectives of our study are as follows: (1) to develop a PGMM using a Bayesian inference approach that allows the estimation of multiple random changepoints within each class; (2) to develop a procedure to empirically detect the number of random changepoints within each class; and (3) to empirically investigate the bias and precision of the estimation of the model parameters, including the random changepoints, via a simulation study. We have developed the user-friendly package BayesianPGMM for R to facilitate the adoption of this methodology in practice, which is available at https://github.com/lockEF/BayesianPGMM . We describe an application to mouse-tracking data for a visual recognition task.

Stuck at the starting line: How the starting procedure influences mouse-tracking data.

Mouse-tracking is an increasingly popular method to trace cognitive processes. As is common for a novel method, the exact methodological procedures employed in an individual study are still relatively idiosyncratic and the effects of different methodological setups on mouse-tracking measures have not been explored so far. Here, we study the impact of one commonly occurring methodological variation, namely whether participants have to initiate their mouse movements to trigger stimulus presentation (dynamic starting condition) or whether the stimulus is presented automatically after a fixed delay and participants can freely decide when to initiate their movements (static starting condition). We compared data from a previous study in which participants performed a mouse-tracking version of a Simon task with a dynamic starting condition to data from a new study that employed a static starting condition in an otherwise identical setup. Results showed reliable Simon effects and Congruency Sequence effects on response time (RT) and discrete trial-level mouse-tracking measures (i.e., average deviation) in both starting conditions. In contrast, within-trial continuous measures (i.e., extracted temporal segments) were weaker and occurred in a more temporally compressed way in the static compared to the dynamic starting condition. This was in line with generally less consistent movements within and across participants in the static compared to the dynamic condition. Our results suggest that studies that use within-trial continuous measures to assess dynamic aspects of mouse movements should apply dynamic starting procedures to enhance the leakage of cognitive processing into the mouse movements.