Visual Search Studies Research Articles

Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Using Fixation-Related Potentials and Eye Tracking Features

While Mixed Reality allows the seamless blending of digital content in users' surroundings, it is unclear if its fusion with physical information impacts users' perceptual and cognitive resources differently. While the fusion of digital and physical objects provides numerous opportunities to present additional information, it also introduces undesirable side effects, such as split attention and increased visual complexity. We conducted a visual search study in three manifestations of mixed reality (Augmented Reality, Augmented Virtuality, Virtual Reality) to understand the effects of the environment on visual search behavior. We conducted a multimodal evaluation measuring Fixation-Related Potentials (FRPs), alongside eye tracking to assess search efficiency, attention allocation, and behavioral measures. Our findings indicate distinct patterns in FRPs and eye-tracking data that reflect varying cognitive demands across environments. Specifically, AR environments were associated with increased workload, as indicated by decreased FRP - P3 amplitudes and more scattered eye movement patterns, impairing users' ability to identify target information efficiently. Participants reported AR as the most demanding and distracting environment. These insights inform design implications for MR adaptive systems, emphasizing the need for interfaces that dynamically respond to user cognitive load based on physiological inputs.

Reward-modulated attention deployment is driven by suppression, not attentional capture

One driving factor for attention deployment towards a stimulus is its associated value due to previous experience and learning history. Previous visual search studies found that when looking for a target, distractors associated with higher reward produce more interference (e.g., longer response times). The present study investigated the neural mechanism of such value-driven attention deployment. Specifically, we were interested in which of the three attention sub-processes are responsible for the interference that was repeatedly observed behaviorally: enhancement of relevant information, attentional capture by irrelevant information, or suppression of irrelevant information. We replicated earlier findings showing longer response times and lower accuracy when a target competed with a high-reward compared to a low-reward distractor. We also found a spatial gradient of interference: behavioral performance dropped with increasing proximity to the target. This gradient was steeper for high- than low-reward distractors. Event-related potentials of the EEG signal showed the reason for the reward-induced attentional bias: High-reward distractors required more suppression than low-reward distractors as evident in larger Pd components. This effect was only found for distractors near targets, showing the additional filtering needs required for competing stimuli in close proximity. As a result, fewer attentional resources can be distributed to the target when it competes with a high-reward distractor, as evident in a smaller target-N2pc amplitude. The distractor-N2pc, indicative of attentional capture, was neither affected by distance nor reward, showing that attentional capture alone cannot explain interference by stimuli of high value. In sum our results show that the higher need for suppression of high-value stimuli contributes to reward-modulated attention deployment and increased suppression can prevent attentional capture of high-value stimuli.

A limited visual search advantage for illusory faces.

The human visual system is very sensitive to the presence of faces in the environment, so much so that it can produce the perception of illusory faces in everyday objects. Growing research suggests that illusory faces and real faces are processed by similar perceptual and neural mechanisms, but whether this similarity extends to visual attention is less clear. A visual search study showed that illusory faces have a search advantage over objects when the types of objects vary to match the objects in the illusory faces (e.g., chair, pepper, clock) (Keys et al., 2021). Here, we examine whether the search advantage for illusory faces over objects remains when compared against objects that belong to a single category (flowers). In three experiments, we compared visual search of illusory faces, real faces, variable objects, and uniform objects (flowers). Search for real faces was best compared with all other types of targets. In contrast, search for illusory faces was only better than search for variable objects, not uniform objects. This result shows a limited visual search advantage for illusory faces and suggests that illusory faces may not be processed like real faces in visual attention.

Task-Dependent Visual Behavior in Immersive Environments: A Comparative Study of Free Exploration, Memory and Visual Search.

Visual behavior depends on both bottom-up mechanisms, where gaze is driven by the visual conspicuity of the stimuli, and top-down mechanisms, guiding attention towards relevant areas based on the task or goal of the viewer. While this is well-known, visual attention models often focus on bottom-up mechanisms. Existing works have analyzed the effect of high-level cognitive tasks like memory or visual search on visual behavior; however, they haveoften done so with different stimuli, methodology, metrics and participants, which makes drawing conclusions and comparisons between tasks particularly difficult. In this work we present a systematic study of how different cognitive tasks affect visual behavior in a novel within-subjects design scheme. Participants performed free exploration, memory and visual search tasks in three different scenes while their eye and head movements were being recorded. We found significant, consistent differences between tasks in the distributions of fixations, saccades and head movements. Our findings can provide insights for practitioners and content creators designing task-oriented immersive applications.

The Effect of Time Pressure on Visual Search Eye Tracking Metrics in Virtual Reality

The study of visual search has focused on various guiding factors, but less attention has been given to how environmental factors affect visual search in virtual reality (VR). The visual search literature has primarily been based on 2D laboratory tasks, which lack the complexity of real-life search tasks. Thus, this study studies the effect of time pressure on visual search in a naturalistic environment. To do that, participants were immersed in a virtual living room using VR and tasked with finding objects under a time constraint. Eye gaze data was collected, and convex hull volumes and scanning rates were calculated and analyzed. The results show that time pressure reduced convex hull volume and increased scanning rate, indicating faster search speed and a gaze tunneling effect. Understanding how time pressure affects visual search can help improve training strategies and design better user interfaces for visual search critical domains.

Suppressive Control of Incentive Salience in Real-World Human Vision.

Reward-related activity in the dopaminergic midbrain is thought to guide animal behaviour, in part by boosting the perceptual and attentional processing of reward-predictive environmental stimuli. In line with this incentive salience hypothesis, studies of human visual search have shown that simple synthetic stimuli - like lines, shapes, or Gabor patches - capture attention to their location when they are characterized by reward-associated visual features like colour. In the real world, however, we commonly search for members of a category of visually-heterogenous objects - like people, cars, or trees - where category examples do not share low-level features. Is attention captured to examples of a reward-associated real-world object category? Here, we have human participants search for targets in photographs of city- and landscapes that contain task-irrelevant examples of a reward-associated category. We use the temporal precision of EEG machine learning and ERPs to show that these distractors acquire incentive salience and draw attention, but do not capture it. Instead, we find evidence of rapid, stimulus-triggered attentional suppression, such that the neural encoding of these objects is degraded relative to neutral objects. Humans appear able to suppress the incentive salience of reward-associated objects when they know these objects will be irrelevant, supporting the rapid deployment of attention to other objects that might be more useful. Incentive salience is thought to underlie key behaviours in eating disorders and addiction, among other conditions, and the kind of suppression identified here likely plays a role in mediating the attentional biases that emerge in these circumstances.Significance StatementLike other animals, humans are prone to notice and interact with environmental objects that have proven rewarding in earlier experience. However, it is common that such objects have no immediate strategic use and are therefore distracting. Do these reward-associated real-world objects capture our attention, despite our strategic efforts otherwise? Or are we able to strategically control the impulse to notice them? Here we use machine learning classification of human electrical brain activity to show that we can establish strategic control over the salience of naturalistic reward-associated objects. These objects draw our attention, but do not necessarily capture it, and this kind of control may play an important role in mediating conditions like eating disorder and addiction.

Children With Amblyopia Make More Saccadic Fixations When Doing the Visual Search Task.

Individuals with amblyopia are known to have functional vision deficits (e.g., reduced reading speed) in spite of good visual acuity in the nonamblyopic eye. We studied and compared eye movements in children with and without amblyopia to examine how a visual scene is explored during visual search. Children (six to 16 years of age) in the control group (n = 14) and cases group with anisometropic amblyopia (n = 23) participated in a visual search study, in which they looked for targets in real-world images displayed on a computer monitor. Eyelink 1000 Plus was used to track the eye movements. Three viewing conditions were randomized: dominant/fellow eye, nondominant/amblyopic eye, and binocular viewing. Visual search performance was measured by combining search time and accuracy. As expected, poorer visual search performance was observed in the amblyopic eye when compared to the controls and fellow eye (P < 0.005). However, the reaction time was longer even in binocular and fellow eye viewing conditions than the controls (P < 0.028). Children with amblyopia made more saccades (17 vs. 12, P = 0.007), without the need to fixate longer (P = 0.312), but with more fixations in the target interest area (4.65 vs. 3.14, P = 0.002) when compared to controls. These eye movement patterns were observed in both the fellow eye and binocular viewing conditions. In spite of good visual acuity in the fellow eye, children with amblyopia needed to sample the scene with more fixations. Even upon gazing at the target location, they made more fixations before confirming a hit. These search patterns suggest a possible narrower spatial visual span to process the visual information in children with amblyopia.

Task-irrelevant filler items alter the dynamics of electrical brain activity during visual search.

In electroencephalography (EEG) studies of visual search, task-irrelevant fillers are included in displays to balance bottom-up stimulation across the visual field and generally considered as inconsequential for performance or EEG results. We examined the impact of fillers on target and distractor processing using lateralised event-related potentials (ERPs). Two task-relevant items (TRIs) were presented, with or without fillers. One TRI (target or target-colour distractor) was on the vertical midline and the other in a lateral position (left or right visual field) on an imaginary circle around fixation. An N2pc was elicited by lateral targets and task-relevant distractors, suggesting that attention was allocated to the lateral TRI because it possessed a target defining feature (colour). A Ptc was only elicited by lateral task-relevant distractors, in line with previous research suggesting that this component is associated with distractor processing. When fillers were also in the circular arrangement, alterations in performance and neural activity occurred. Fillers enhanced and delayed attentional deployment (N2pc) and delayed distractor processing (Ptc). Critically, we observed no difference in Ptc amplitude according to filler presence. Thus, if the Ptc reflects active suppression (or attentional disengagement), it was not required for fillers. ERPs were also modulated by the distance between TRIs (which could be separated by one or four filler positions), but differently according to the colour scheme (blue TRIs with grey fillers or vice versa). Our results suggest that fillers affect lateralised electrophysiological activity at multiple loci during visual search and should not be considered inconsequential.

How fixation durations are affected by search difficulty manipulations

ABSTRACT Many eye tracking studies of visual search have focused on the role of the number of fixations and the nature of scan paths. Less attention has been paid to fixation durations and to how those durations are affected by stimulus features. Previous studies have shown that fixation durations can be as important as the number of fixations in explaining search times with complex stimuli (e.g., in search for specific faces). In the present study, simple stimuli were used in a search experiment where participants searched for a closed ring among rings with a gap. We manipulated distractor heterogeneity (DH), target-distractor similarity (TDS), and stimulus density (SDY, set size within a constant search window), and estimated the contributions of these factors to gaze behaviour and trial search time. The results show that fixation durations contribute less to variation in overall search time with simple search stimuli as compared to previous studies with more complex stimuli. However, fixation durations still increased with DH, TDS, and SDY. These effects were mainly additive, and we did not find an interaction between DH and closer element spacing at high levels of SDY that might have been expected since both DH and SDY influence distractor grouping.

Where to display vital information? ERP evidence for background changes

Critical decision systems require expeditious and accurate responses to the displayed information. In addition to content, location and background are equally important. Several visual search studies have pointed out the differences and compared modes of top-down attention allocation: distractor suppression and attentional capture. Previous studies have used color (mainly) and shape as a feature but have overlooked luminance as a feature for studying underlying attention mechanisms. The present study attempts to bridge this gap. In this study, participants performed a target-distractor discrimination task by identifying a randomly appearing target from the pool of distractors based on defined luminance levels. Background change was noticed by manipulating the task such as making visible quadrant boundaries over the screen. The preliminary evidence suggested that displaying information at the top-left of the screen had higher percentage accuracy; whereas, response time (RT) remained unaffected. Improvement in RT and percentage accuracy was observed with task manipulation. Event-related potential (ERP) analysis revealed elicited Distractor Positivity (PD), providing evidence for the distractor suppression hypothesis. Further, differences emerged in the topographic plot of N2pc and PD. In sum, the result contributes to classic debate of capture vs. suppression and provides a crucial connection between display design and electrophysiological indices, emphasizing locations and background as equally important factors.

Contributed Session II: Visual Search in Virtual Reality (VSVR): A visual search toolbox for virtual reality

Our understanding of human visual attention has greatly benefited from a wealth of visual search studies conducted over the past few decades. Task observers with searching for a specific target embedded among a set of distractors, and the time taken for them to find the target can reveal much about the sensory and cognitive processes involved. These experiments have typically been conducted on 2D displays under tightly controlled viewing conditions. Recently however, there have been calls within the visual attention community to explore more ecologically valid means of data collection. Virtual reality (VR) is a promising methodological tool for such research as it offers improved visual realism and the possibility of participant interaction, while retaining a significant amount of the control afforded by a computerized and monitor presented experiment. Here we present the Visual Search in Virtual Reality (VSVR) ToolBox. VSVR is a set of functions, scripts and assets that can be combined within a visual scripting environment in the Unity game engine to design, replicate and extend visual search experiments in VR. We further demonstrate the utility of such a toolbox with three experiments: a replication of feature search behavior, a demonstration of wide field-of-view visual search and eccentricity effects, and replication of depth plane as a feature for search.

How feature context alters attentional template switching.

In real-world tasks visual attention is rarely aimed at a single object. Humans rather "forage" the visual scene for information, dynamically switching attentional templates. Several visual search studies have found that observers often use suboptimal attentional control strategies, possibly to avoid effort. Here, we investigated with a foraging paradigm if observers' reluctance to switch between attentional templates increases with template specificity. To that end, we manipulated the feature context of displays in which participants "foraged" moving stimuli on a tablet-PC. Experiment 1 (N = 35) revealed a decline in switching tendency and foraging efficiency with increasing feature-space distance between target alternatives. Experiment 2 (N = 36) found even lower flexibility with distractor color close to target colors and strongest impairments with distractor color in between target colors. Our results demonstrate that visual information sampling is most flexible when broad (instead of very specific) templates and relational search strategies are possible (e.g., attending to "redder" objects), with implications for both attention research and applications, especially in visual-foraging-like tasks, such as baggage screening or medical image assessment. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

A Study of Visual Search based Calibration Protocol for EEG Attention Detection.

Attention, a multi-faceted cognitive process, is essential in our daily lives. We can measure visual attention using an EEG Brain-Computer Interface for detecting different levels of attention in gaming, performance training, and clinical applications. In attention calibration, we use Flanker task to capture EEG data for attentive class. For EEG data belonging to inattentive class calibration, we instruct subject not focusing on a specific position on screen. We then classify attention levels using binary classifier trained with these surrogate ground-truth classes. However, subjects may not be in desirable attention conditions when performing repetitive boring activities over a long experiment duration. We propose attention calibration protocols in this paper that use simultaneous visual search with an audio directional change paradigm and static white noise as 'attentive' and 'inattentive' conditions, respectively. To compare the performance of proposed calibrations against baselines, we collected data from sixteen healthy subjects. For a fair comparison of classification performance; we used six basic EEG band-power features with a standard binary classifier. With the new calibration protocol, we achieved 74.37 ± 6.56% mean subject accuracy, which is about 3.73 ± 2.49% higher than the baseline, but there were no statistically significant differences. According to post-experiment survey results, new calibrations are more effective in inducing desired perceived attention levels. We will improve calibration protocols with reliable attention classifier modeling to enable better attention recognition based on these promising results.

Adults with higher social anxiety show avoidant gaze behaviour in a real-world social setting: A mobile eye tracking study.

Attentional biases are a core characteristic of social anxiety (SA). However, research has yielded conflicting findings and failed to investigate these biases in real, face-to-face social situations. Therefore, this study examined attentional biases in SA by measuring participants’ eye gaze within a novel eye-tracking paradigm during a real-life social situation. Student participants (N = 30) took part in what they thought was a visual search study, when a confederate posing as another participant entered the room. Whilst all participants avoided looking at the confederate, those with higher SA fixated for a shorter duration during their first fixation on him, and executed fewer fixations and saccades overall as well as exhibiting a shorter scanpath. These findings are indicative of additional avoidance in the higher SA participants. In contrast to previous experimental work, we found no evidence of social hypervigilance or hyperscanning in high SA individuals. The results indicate that in unstructured social settings, avoidance rather than vigilance predominates, especially in those with higher SA.

Preserved Extra-Foveal Processing of Object Semantics in Alzheimer's Disease.

Alzheimer's disease (AD) patients underperform on a range of tasks requiring semantic processing, but it is unclear whether this impairment is due to a generalised loss of semantic knowledge or to issues in accessing and selecting such information from memory. The objective of this eye-tracking visual search study was to determine whether semantic expectancy mechanisms known to support object recognition in healthy adults are preserved in AD patients. Furthermore, as AD patients are often reported to be impaired in accessing information in extra-foveal vision, we investigated whether that was also the case in our study. Twenty AD patients and 20 age-matched controls searched for a target object among an array of distractors presented extra-foveally. The distractors were either semantically related or unrelated to the target (e.g., a car in an array with other vehicles or kitchen items). Results showed that semantically related objects were detected with more difficulty than semantically unrelated objects by both groups, but more markedly by the AD group. Participants looked earlier and for longer at the critical objects when these were semantically unrelated to the distractors. Our findings show that AD patients can process the semantics of objects and access it in extra-foveal vision. This suggests that their impairments in semantic processing may reflect difficulties in accessing semantic information rather than a generalised loss of semantic memory.

Inhibitory control deficits in vascular cognitive impairment revealed using the MILO task

We used the MILO (Multi-Item Localization) task to characterise the performance of a group of older adults diagnosed with mild to moderate vascular cognitive impairment (VCI). The MILO task is designed to explore the temporal context of visual search and in addition to measuring overall completion time, provides a profile of serial reaction time (SRT) patterns across all items in a sequence. Of particular interest here is the Vanish/Remain MILO manipulation that can identify problems with inhibitory control during search. Typically, SRT functions closely overlap, regardless of whether items Vanish or Remain visible when selected, indicating an ability to ignore previously selected targets. Based on the distributed nature of VCI-related pathology and previous visual search studies from our group, we speculated that MILO performance would be compromised in this group of participants when items remained visible after being selected relative to when they vanished. Compared to cognitively healthy, age-matched control participants, the performance of VCI participants was characterised by overall slowing, increased error rates, and crucially, a compromised ability to ignore past locations. As predicted, the Vanish versus Remain SRT functions of VCI participants significantly diverged towards the end of the sequence, which was not the case for control groups. Overall, our findings suggest that the MILO task could be a useful tool for identifying non-age-related changes in behaviour with patient populations, and more generally hints at a possible inhibitory deficit in VCI.

Response of multiple demand network to visual search demands

Neuroimaging studies for human participants have shown that the activity in the multiple-demand (MD) network is associated with various kinds of cognitive demand. However, surprisingly, it remains unclear how this MD network is related to a core component of cognition, the process of searching for a target among distractors. This was because previous neuroimaging studies of visual search were confounded by task difficulty or time on task. To circumvent these limitations, we examined human brain activity while participants perform two different visual search tasks. The performance of a task was limited by increased attentional demand, while the other task was primarily limited by poor quality of input data or neural noise. Throughout the MD network, increased activity and strengthened functional connectivity among the MD regions were observed under the search task recruiting capacity-limited attentional resources. The present findings provide unequivocal evidence that the MD network mediates visual search, as well as other capacity-limited cognitive processes.

Towards Interactive Search: Investigating Visual Search in a Novel Real-World Paradigm.

An overwhelming majority of studies on visual search and selective attention were conducted using computer screens. There are arguably shortcomings in transferring knowledge from computer-based studies to real-world search behavior as findings are based on viewing static pictures on computer screens. This does not go well with the dynamic and interactive nature of vision in the real world. It is crucial to take visual search research to the real world in order to study everyday visual search processes. The aim of the present study was to develop an interactive search paradigm that can serve as a “bridge” between classical computerized search and everyday interactive search. We based our search paradigm on simple LEGO® bricks arranged on tabletop trays to ensure comparability with classical computerized visual search studies while providing room for easily increasing the complexity of the search environment. We found that targets were grasped slower when there were more distractors (Experiment 1) and there were sizable differences between various search conditions (Experiment 2), largely in line with classical visual search research and revealing similarities to research in natural scenes. Therefore, our paradigm can be seen as a valuable asset complementing visual search research in an environment between computerized search and everyday search.

Visual search for drowning swimmers: Investigating the impact of lifeguarding experience

SummaryHow does domain expertise influence dynamic visual search? Previous studies of visual search often use abstract search arrays that are devoid of applied context, with comparatively few studies exploring applied naturalistic and dynamic settings. The current research adds to this literature by examining lifeguard drowning‐detection across two studies using naturalistic, dynamic search tasks. Behavioural responses and eye‐movement data were recorded as participants watched staged video clips and attempted to identify if a swimmer was drowning. The results demonstrate lifeguard superiority in response times to drowning events, compared to non‐lifeguards. No differences between lifeguard and non‐lifeguard eye‐movements were noted however. This suggests that the experiential benefit in response times results from other underlying processes, rather than any scanning benefits. This research highlights the complex nature of naturalistic and dynamic searches, while demonstrating the robust nature of simulated videos in producing experience effects in visual search.

The anger superiority effect revisited: a visual crowding task

ABSTRACT Visual search studies have shown that threatening facial expressions are more efficiently detected among a crowd of distractor faces than nonthreatening expressions, known as the anger superiority effect (ASE). However, the opposite finding has also been documented. The present study investigated the ASE in the visual periphery with a visual crowding task. In the study, the target face either appeared alone (uncrowded condition) or was crowded by four neutral or emotional faces (crowded condition). Participants were instructed to determine whether the target face was happy or angry. Experiment 1 showed an ASE when crowded by neutral faces. Intriguingly, this superiority vanished when the target face was crowded by emotional faces that had a different expression from the target as well as when the target face was presented alone. Experiment 2 replicated this result in an independent sample of East Asians (vs. Caucasians in Experiment 1) and thus demonstrated the robustness and cross-cultural consistency of our findings. Together, these results suggest that the ASE in the visual periphery is contingent on task demands induced by visual crowding.