Overt Visual Attention Research Articles

Crossmodal semantic congruence guides spontaneous orienting in real-life scenes.

In real-world scenes, the different objects and events are often interconnected within a rich web of semantic relationships. These semantic links help parse information efficiently and make sense of the sensory environment. It has been shown that, during goal-directed search, hearing the characteristic sound of an everyday life object helps finding the affiliate objects in artificial visual search arrays as well as in naturalistic, real-life videoclips. However, whether crossmodal semantic congruence also triggers orienting during spontaneous, not goal-directed observation is unknown. Here, we investigated this question addressing whether crossmodal semantic congruence can attract spontaneous, overt visual attention when viewing naturalistic, dynamic scenes. We used eye-tracking whilst participants (N = 45) watched video clips presented alongside sounds of varying semantic relatedness with objects present within the scene. We found that characteristic sounds increased the probability of looking at, the number of fixations to, and the total dwell time on semantically corresponding visual objects, in comparison to when the same scenes were presented with semantically neutral sounds or just with background noise only. Interestingly, hearing object sounds not met with an object in the scene led to increased visual exploration. These results suggest that crossmodal semantic information has an impact on spontaneous gaze on realistic scenes, and therefore on how information is sampled. Our findings extend beyond known effects of object-based crossmodal interactions with simple stimuli arrays and shed new light on the role that audio-visual semantic relationships out in the perception of everyday life scenarios.

Eye tracking insights into physician behaviour with safe and unsafe explainable AI recommendations

We studied clinical AI-supported decision-making as an example of a high-stakes setting in which explainable AI (XAI) has been proposed as useful (by theoretically providing physicians with context for the AI suggestion and thereby helping them to reject unsafe AI recommendations). Here, we used objective neurobehavioural measures (eye-tracking) to see how physicians respond to XAI with N = 19 ICU physicians in a hospital’s clinical simulation suite. Prescription decisions were made both pre- and post-reveal of either a safe or unsafe AI recommendation and four different types of simultaneously presented XAI. We used overt visual attention as a marker for where physician mental attention was directed during the simulations. Unsafe AI recommendations attracted significantly greater attention than safe AI recommendations. However, there was no appreciably higher level of attention placed onto any of the four types of explanation during unsafe AI scenarios (i.e. XAI did not appear to ‘rescue’ decision-makers). Furthermore, self-reported usefulness of explanations by physicians did not correlate with the level of attention they devoted to the explanations reinforcing the notion that using self-reports alone to evaluate XAI tools misses key aspects of the interaction behaviour between human and machine.

Less Is More: Higher-Skilled Sim Racers Allocate Significantly Less Attention to the Track Relative to the Display Features than Lower-Skilled Sim Racers.

Simulated (sim) racing is an emerging esport that has garnered much interest in recent years and has been a relatively under-researched field in terms of expertise and performance. When examining expertise, visual attention has been of particular interest to researchers, with eye tracking technology commonly used to assess visual attention. In this study, we examined the overt visual attention allocation of high- and low-skilled sim racers during a time trial task using Tobii 3 glasses. In the study, 104 participants were tested on one occasion, with 88 included in the analysis after exclusions. Participants were allocated to either group according to their fastest lap times. Independent t-tests were carried out with sidak corrections to test our hypotheses. Our results indicate that when eye tracking metrics were normalised to the lap time and corner sector time, there was a difference in the relative length of overt attention allocation (fixation behaviour) as lower-skilled racers had significantly greater total fixation durations in laps overall and across corner sectors when normalised (p = 0.013; p = 0.018). Interestingly, high- and low-skilled sim racers differed in where they allocated their attention during the task, with high-skilled sim racers allocating significantly less overt attention to the track relative to other areas of the display (p = 0.003). This would allow for higher-skilled racers to obtain relatively more information from heads-up display elements in-game, all whilst driving at faster speeds. This study provides evidence that high-skilled sim racers appear to need significantly less overt attention throughout a fast lap, and that high- and low-skilled sim racers differ in where they allocate their attention while racing.

Gestures cued by demonstratives in speech guide listeners' visual attention during spatial language comprehension.

Gestures help speakers and listeners during communication and thinking, particularly for visual-spatial information. Speakers tend to use gestures to complement the accompanying spoken deictic constructions, such as demonstratives, when communicating spatial information (e.g., saying "The candle is here" and gesturing to the right side to express that the candle is on the speaker's right). Visual information conveyed by gestures enhances listeners' comprehension. Whether and how listeners allocate overt visual attention to gestures in different speech contexts is mostly unknown. We asked if (a) listeners gazed at gestures more when they complement demonstratives in speech ("here") compared to when they express redundant information to speech (e.g., "right") and (b) gazing at gestures related to listeners' information uptake from those gestures. We demonstrated that listeners fixated gestures more when they expressed complementary than redundant information in the accompanying speech. Moreover, overt visual attention to gestures did not predict listeners' comprehension. These results suggest that the heightened communicative value of gestures as signaled by external cues, such as demonstratives, guides listeners' visual attention to gestures. However, overt visual attention does not seem to be necessary to extract the cued information from the multimodal message. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The process and product of translation revision: empirical data from student translators using eye tracking and screen recording

ABSTRACT As the final phase of the translation process, self-revision has traditionally been a relatively neglected component despite it being widely acknowledged as critical to achieving high-quality translation and in translator training. To identify effective educational interventions for self-revision, the current paper reports on a study that collected empirical data from student translators (n = 72) who were randomly assigned to different conditions (traditional, screen recording, and eye tracking) in highly ecologically valid bidirectional tasks. Six features of eye tracking and screen recording were identified as indicators of potential translation difficulties which could be used to direct student translator’s overt visual attention in the revision phase. We found that eye tracking was the most effective way of assisting student translators in self-revising their translations, regardless of the direction of translation. Further, average fixation duration and backward saccades in the eye tracking recording condition, and pauses and concurrent revision activity in the screen recording condition were the most useful features in helping the student translators to identify potential translation errors and thereby enhance the quality of their translation after revisions. Qualitative data showed that the student translators became more confident and had a clearer understanding of their own translation processes after watching the recordings.

Processing spatial configurations in visuospatial working memory is influenced by shifts of overt visual attention.

When memorizing multiple objects, humans process them in relation to each other, proposing a configuration benefit. Shifts in overt visual attention through eye movements might influence the processing of spatial configurations. Whereas some research suggests that overt visual attention aids the processing of spatial representations, other research suggests a snapshot-like processing of spatial configurations, thus likely not relying on eye movements. In the first experiment, we focused on the comparison between an enforced fixation and a free view condition regarding configurational effects. Participants encoded objects' locations and were asked for changes at retrieval. One object was displaced in half of the trials and was either accompanied by a configuration or was displayed alone. In the second experiment, we expanded this idea by enforcing fixation during different task phases, namely encoding, maintenance and retrieval. We investigated if a fixed gaze during one specific phase drives the influence of eye movements when processing spatial configurations. We observed reliable configuration benefits for the free view conditions. Whereas a fixed gaze throughout the whole trial reduced the effect, enforced fixations during the task phases did not break the configuration benefit. Our findings suggest that whereas the processing of spatial configurations in memory is supported by the ability of performing shifts of overt visual attention, configurational processing does not rely on these shifts occurring throughout the task. Our results indicate a reciprocal relationship of visuospatial working memory and eye movements.

Examining the validity of eye tracking during the computerized Wisconsin card sorting test in a sample of stroke patients and healthy controls

ABSTRACT Introduction Each year, approximately 50,000 Canadians, one million Americans, and millions of people worldwide are hospitalized for stroke. Cognitive impairment is common after experiencing a stroke and is known to affect functioning on daily tasks. While neuropsychological assessments are often employed to assess cognitive abilities and make inferences about functional capabilities, there is growing interest in integrating contemporary technologies to augment assessment. Eye tracking allows previously overlooked information, such as overt visual attention based on fixations and saccades, to be quantified to help elucidate how responses are made during testing. Method The current study investigated the validity of eye tracking during completion of the computerized Wisconsin Card Sorting Test (cWCST), a common test of higher level cognition, in a sample of inpatients recovering from stroke (n= 41) and a control group of healthy individuals (n = 46). Results Results provided supporting evidence for the construction, criterion, and ecological validity of eye tracking on the cWCST with inpatients recovering from a stroke. Specifically, eye tracking metrics differentiated between inpatients and controls; fixations on cWCST areas of interest differed between type of response (conceptual versus non-conceptual); and average time per fixation predicted functional status early after a stroke as well as recovery during inpatient rehabilitation, above-and-beyond cWCST scores. Time spent on testing negated the effects of fixation and saccade counts for predicting cWCST performance, due to the substantial overlap in variance. Conclusion Current findings of this preliminary study provided support for the validity of eye tracking, integrated with the cWCST, for inpatients recovering from a stroke. Implications and areas for future research are discussed.

Overt visual attention and between-limb asynchrony for bimanual reaching movements

Although synchrony between the limbs is an often-cited feature of bimanual coordination, recent studies have also highlighted the small asynchronies that can occur. The visuo-motor demands of any bimanual task are considered central to the emergence of asynchrony, but the relationship between the two remains largely unexplored. This study aimed to address this issue. Hand and eye movements were measured in 19 participants, while they made either unimanual or bimanual reach-to-point (aiming) movements to targets presented on a touchscreen. Bimanual movements were either congruent (same-sized targets) or incongruent (different-sized targets). Resulting hand data showed many of the typical patterns of movement previously reported. While temporal coupling between the limbs remained largely evident for bimanual movements, small between-limb asynchronies were apparent and demonstrated clear associations with the competing precision requirements of the targets and related visual attention. Participants mainly directed their gaze towards the more difficult target with corresponding reaching movements demonstrating greater precision than for the easier target. Additionally, there was a reliable tendency for the hand reaching towards the more difficult target to lead. Importantly, it was the competing visuo-motor demands of individual movements rather than overall difficulty that resulted in greater between-limb asynchrony; accordingly, where both targets were small (i.e., the most difficult condition), asynchrony was significantly less pronounced than for incongruent bimanual conditions. The results show how the visuo-motor system balances its apparent drive for synchrony in coordinating bimanual movements with the competing demands that characterise the constituent unimanual movements.

Guiding visual attention in deep convolutional neural networks based on human eye movements.

Deep Convolutional Neural Networks (DCNNs) were originally inspired by principles of biological vision, have evolved into best current computational models of object recognition, and consequently indicate strong architectural and functional parallelism with the ventral visual pathway throughout comparisons with neuroimaging and neural time series data. As recent advances in deep learning seem to decrease this similarity, computational neuroscience is challenged to reverse-engineer the biological plausibility to obtain useful models. While previous studies have shown that biologically inspired architectures are able to amplify the human-likeness of the models, in this study, we investigate a purely data-driven approach. We use human eye tracking data to directly modify training examples and thereby guide the models’ visual attention during object recognition in natural images either toward or away from the focus of human fixations. We compare and validate different manipulation types (i.e., standard, human-like, and non-human-like attention) through GradCAM saliency maps against human participant eye tracking data. Our results demonstrate that the proposed guided focus manipulation works as intended in the negative direction and non-human-like models focus on significantly dissimilar image parts compared to humans. The observed effects were highly category-specific, enhanced by animacy and face presence, developed only after feedforward processing was completed, and indicated a strong influence on face detection. With this approach, however, no significantly increased human-likeness was found. Possible applications of overt visual attention in DCNNs and further implications for theories of face detection are discussed.

Personality similarity predicts synchronous neural responses in fMRI and EEG data

Successful communication and cooperation among different members of society depends, in part, on a consistent understanding of the physical and social world. What drives this alignment in perspectives? We present evidence from two neuroimaging studies using functional magnetic resonance imaging (fMRI; N = 66 with 2145 dyadic comparisons) and electroencephalography (EEG; N = 225 with 25,200 dyadic comparisons) to show that: (1) the extent to which people’s neural responses are synchronized when viewing naturalistic stimuli is related to their personality profiles, and (2) that this effect is stronger than that of similarity in gender, ethnicity and political affiliation. The localization of the fMRI results in combination with the additional eye tracking analyses suggest that the relationship between personality similarity and neural synchrony likely reflects alignment in the interpretation of stimuli and not alignment in overt visual attention. Together, the findings suggest that similarity in psychological dispositions aligns people’s reality via shared interpretations of the external world.

The role of prediction in learned predictiveness.

Learning permits even relatively uninteresting stimuli to capture attention if they are established as predictors of important outcomes. Associative theories explain this “learned predictiveness” effect by positing that attention is a function of the relative strength of the association between stimuli and outcomes. In three experiments we show that this explanation is incomplete: learned overt visual-attention is not a function of the relative strength of the association between stimuli and an outcome. In three experiments, human participants were exposed to triplets of stimuli that comprised (a) a target (that defined correct responding), (b) a stimulus that was perfectly correlated with the presentation of the target, and (c) a stimulus that was uncorrelated with the presentation of the target. Participants’ knowledge of the associative relationship between the correlated or uncorrelated stimuli and the target was always good. However, eye-tracking revealed that an attentional bias toward the correlated stimulus only developed when it and target-relevant responding preceded the target stimulus. We propose a framework in which attentional changes are modulated during learning as a function the relative strength of the association between stimuli and the task-relevant response, rather than an association between stimuli and the task-relevant outcome.

Comparison of the importance of beef price labelling aspects: An eye-tracking approach

Self-reported approaches such as surveys are a common way in which consumers' preferences and needs can be assessed but this approach can provide incorrect results. Few studies combine data from cognitive psychology, cognitive neuroscience, and marketing. Among these techniques, eye-tracking can be used to elucidate how observers' overt visual attention is acted upon, and it can also be employed to evaluate and compare individuals' graphic search behaviour in a number of settings. Using eye-tracking technology, the purpose of this article is to identify how beef consumers pay attention to the information on price labels and to compare that to their self-reported attention ratings. The dataset used in this study consists of 307 participants. Kendall's W tests were used to identify and rank consumers' preferences from eye-tracking and post-test questionnaires. Pearson's correlations were used to correlate consumers' demographic information against their eye-tracking data. Results from the eye-tracking test showed that the aspects that received the most attention from the participants were the price of the pack, butchery name, and classification of the meat. The price of the pack was looked at more times by the largest share of consumers, and for the most extended period of all labelling, aspects presented. Results also showed that the butchery's name was the labelling aspect consumers mostly paid attention to first. Fewer consumers tended to look at the packaging date, sell-by date and cut name. A difference was found between the aspects that the participants indicated were the most important when buying red meat and what aspects they actually paid attention to when looking at packs of beef. Correlation results suggest that younger consumers are more likely to pay attention to the price labelling aspects and for a longer period. However, it will be at a later stage than older consumers. Also, price label information seems to be more important to higher educated beef consumers. The vital role that eye-tracking can play in improving the accuracy of research has been highlighted in the study. Marketing agents will also be able to use eye-tracking to ensure that products meet the demands of their customers. Future studies must be performed to confirm the results of this study before they can be generalised. It would be worthwhile to also test consumers' understanding of beef price label information.

Look at what I can do: Object affordances guide visual attention while speakers describe potential actions.

As we act on the world around us, our eyes seek out objects we plan to interact with. A growing body of evidence suggests that overt visual attention selects objects in the environment that could be interacted with, even when the task precludes physical interaction. In previous work, objects that afford grasping interactions influenced attention when static scenes depicted reachable spaces, and attention was otherwise better explained by general informativeness. Because grasping is but one of many object interactions, previous work may have downplayed the influence of object affordances on attention. The current study investigated the relationship between overt visual attention and object affordances versus broadly construed semantic information in scenes as speakers describe or memorize scenes. In addition to meaning and grasp maps-which capture informativeness and grasping object affordances in scenes, respectively-we introduce interact maps, which capture affordances more broadly. In a mixed-effects analysis of 5 eyetracking experiments, we found that meaning predicted fixated locations in a general description task and during scene memorization. Grasp maps marginally predicted fixated locations during action description for scenes that depicted reachable spaces only. Interact maps predicted fixated regions in description experiments alone. Our findings suggest observers allocate attention to scene regions that could be readily interacted with when talking about the scene, while general informativeness preferentially guides attention when the task does not encourage careful consideration of objects in the scene. The current study suggests that the influence of object affordances on visual attention in scenes is mediated by task demands.

An Eye-Movement Analysis of Overt Visual Attention During Consecutive and Simultaneous Interpreting Modes in a Remotely Interpreted Investigative Interview.

Remote interpreting via video-link is increasingly being employed in investigative interviews chiefly due to its apparent increased accessibility and efficiency. However, risks of miscommunication have been shown to be magnified in remote interpreting and empirical research specifically on video-link remote interpreting is in its infancy which greatly limits the evidence base available to inform and direct evidence-based policy and best practice, particularly in the identification of the optimal mode(s) of interpreting to be used, namely consecutive and simultaneous. Consecutive interpreting refers to a process in which the interpreter transfers short segments of speech from one language into the other as each person speaks in managed turn-taking, while simultaneous interpreting refers to the transfer of natural speech from one language into another in a concurrent manner without the need for speakers to segment their speech. This study provides novel empirical evidence by using eye tracking to compare the overt visual attention of interpreters working in a remote setting in which an English-speaking Interviewer interacts with a non-English-speaking Suspect in person, for whom interpretation is provided via video-link in real time. Using a within-subject design, we analyze eye-movement data from 28 professionally accredited interpreters who interpreted via video-link an investigative interview in which consecutive and simultaneous interpreting modes were counterbalanced. Taking interpreting performance into account, our results showed that, the consecutive mode yielded significantly less gaze time and therefore significantly less on-screen overt visual attention due to off-screen notetaking, an essential component of the consecutive interpreting mode. Relative to gaze time, the consecutive mode also resulted in significantly more and longer fixations and shifts of attention. Participants also allocated significantly more overt visual attention to the Interviewer than the Suspect, particularly in the consecutive mode. Furthermore, we found informative significant correlations between eye tracking measures and interpreting performance: accuracy, verbal rapport, and management. Finally, we found no significant differences between the three language pairs tested. We conclude with a discussion of limitations and the contributions of the study and an outline for future work on this topic of growing importance.

Note-taking effort in video remote interpreting: effects of source speech difficulty and interpreter work experience

ABSTRACT This paper reports on an investigation of the interactions between source speech difficulty, interpreters’ work experience and their note-taking behaviour in video remote interpreting. 20 professional interpreters and 29 student interpreters consecutively interpreted two easy segments and two difficult segments from English (L2) to Chinese (L1), with their eye and pen movements being registered by a Tobii eye-tracker and a Wacom digital pen. The results show that: (1) an increase of perceived interpreting difficulty led to a decrease in the overt visual attention and physical effort invested in note-taking, and a longer time to make note-taking decisions; (2) although the professionals and students reported a similar level of interpreting difficulty, the professionals devoted more overt visual attention and physical effort to completing the note-taking activity in interpreting than the students; and (3) the cognitive effort of note-taking was not affected by changes in source speech difficulty or interpreter work experience. These findings indicate that: (1) interpreting experience accumulation does not entail reductions in task-difficulty perception or the cognitive effort of note-taking but increases the amount of effort interpreters devote to note-taking; and (2) increased task difficulty affects time management during note-taking but not the cognitive effort of note-taking.

Supplemental Material for The Effects of Stimulus Pre-Exposure and Conditioning on Overt Visual Attention

Supplemental Material for The Effects of Stimulus Pre-Exposure and Conditioning on Overt Visual Attention

The effects of stimulus pre-exposure and conditioning on overt visual attention.

Three experiments (a, b, c) combined to provide a well-powered examination of the effects of stimulus pre-exposure and conditioning on visual attention using an eye tracker and a space-shooter video game where a colored flashing light predicted an attacking spaceship. In each, group "control" received no pre-exposure to the light, group "same" received pre-exposure in the same context as conditioning, and group "different" received pre-exposure in a different context. Experiments differed in visual details regarding the game (1a vs. 1b and 1c) or minor details in the setup of the eye tracker (1a and 1b vs. 1c). Overall, pre-exposure retarded acquisition of keyboard responding. That effect was enhanced, rather than attenuated, by a context change. Separating participants by sign and goal trackers showed the context change enhanced the pre-exposure effect in goal trackers and reduced it in sign trackers. Visual attention to the light declined during pre-exposure and did not recover with either conditioning or a context switch. Visual attention to the light decreased during conditioning. Visual goal tracking toward where the spaceship would appear was also retarded with pre-exposure. Unlike the keyboard responding, a context change led to more normal goal-tracking acquisition. Results are discussed in terms of theories of attention and the potential effects of demand characteristics on the task. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

The ‘hookability’ of multimodal impact captions

Abstract The use of captions has grown in recent years in both traditional and new media, particularly in terms of the diversity of style, content, and function. Impact captions have emerged as a popular form of captions for hearing viewers and contain rich multimodal information which is employed to capture viewer attention and enhance engagement, particularly in situations where there is competition for viewer attention. Drawing upon relevance theory, we argue how impact captions could effectively attract and hold visual attention owing to their balance between processing effort and contextual effects. This exploratory study employs a dual-task paradigm and uses authentic materials and viewing situations to further examine the ability of multimodal impact captions to attract and retain overt visual attention amongst a small sample of TV viewers. Our results provide novel insight into the apparent highly individualised efficacy of impact captions, where we identify several variables of interest in participants’ viewing behaviours. We conclude with a discussion of the study’s contributions, limitations, and an outline for future work.

A causal role for the right frontal eye fields in value comparison.

Recent studies have suggested close functional links between overt visual attention and decision making. This suggests that the corresponding mechanisms may interface in brain regions known to be crucial for guiding visual attention - such as the frontal eye field (FEF). Here, we combined brain stimulation, eye tracking, and computational approaches to explore this possibility. We show that inhibitory transcranial magnetic stimulation (TMS) over the right FEF has a causal impact on decision making, reducing the effect of gaze dwell time on choice while also increasing reaction times. We computationally characterize this putative mechanism by using the attentional drift diffusion model (aDDM), which reveals that FEF inhibition reduces the relative discounting of the non-fixated option in the comparison process. Our findings establish an important causal role of the right FEF in choice, elucidate the underlying mechanism, and provide support for one of the key causal hypotheses associated with the aDDM.

An Object-Based Bayesian Framework for Top-Down Visual Attention

We introduce a new task-independent framework to model top-down overt visual attention based on graph-ical models for probabilistic inference and reasoning. We describe a Dynamic Bayesian Network (DBN) that infers probability distributions over attended objects and spatial locations directly from observed data. Probabilistic inference in our model is performed over object-related functions which are fed from manual annotations of objects in video scenes or by state-of-the-art object detection models. Evaluating over ∼3 hours (appx. 315,000 eye fixations and 12,600 saccades) of observers playing 3 video games (time-scheduling, driving, and flight combat), we show that our approach is significantly more predictive of eye fixations compared to: 1) simpler classifier-based models also developed here that map a signature of a scene (multi-modal information from gist, bottom-up saliency, physical actions, and events) to eye positions, 2) 14 state-of-the-art bottom-up saliency models, and 3) brute-force algorithms such as mean eye position. Our results show that the proposed model is more effective in employing and reasoning over spatio-temporal visual data.