Visual Attention Model Research Articles

Supporting Driver Attention Toward Potential Hazards During Takeover: A Preliminary Result

This study investigates the impact of a supportive system on takeover transitions in conditionally automated driving (SAE Level 3). The supportive system is designed to direct drivers’ attention toward potential hazards in the environment when a takeover request occurs. The study comprises two components: (a) identifying various types of potential hazards using naturalistic driving data, and (b) conducting a driving simulator study to develop and assess a gaze guidance system based on the N-SEEV model of visual attention. Results indicate that drivers using a highly salient attention guidance system were less likely to collide with a secondary hazard during takeover transitions. This suggests that gaze guidance support is an effective approach for assisting drivers during takeover transitions.

Progressive Hierarchical Deep Reinforcement Learning for defect wafer test

Wafer testing plays a critical role in the wafer fabrication process. It involves using a probe to inspect all the wafers on a map, which presents a Coverage Path Planning (CPP) problem. Deep Reinforcement Learning (DRL) has emerged as a state-of-the-art (SOTA) solution for automating regular wafer testing. However, in the presence of production defects, the wafer maps become irregular, requiring more advanced probe strategies. Unfortunately, SOTA DRL methods cannot be directly applied to defect wafer testing due to high computational complexity, low sample efficiency, and limited transferability between tasks. To address these challenges, we propose a novel DRL solution called the Progressive Hierarchical DRL method for automated defect wafer testing. Specifically, our method utilizes a Visual Memory Attention Model (VMAM) to create a high-level strategy, referred to as a manager, which generates a goal specifying the most crucial image region for the task. Subsequently, a low-level strategy, referred to as a worker, generates precise cruising actions to navigate to the designated area. Additionally, we employ transfer learning to generalize the method to handle more complex defect wafer maps. Our approach outperforms seven SOTA DRL baselines in the domains of CPP and wafer testing. It achieves significant improvements in task duration and wafer visit overlap, ranging from 6.0% to 30.5%. Furthermore, we explore the application of our method to a dual-probe defect wafer testing system.

Distinct eye movement patterns to complex scenes in Alzheimer's disease and Lewy body disease.

Alzheimer's disease (AD) and Lewy body disease (LBD), the two most common causes of neurodegenerative dementia with similar clinical manifestations, both show impaired visual attention and altered eye movements. However, prior studies have used structured tasks or restricted stimuli, limiting the insights into how eye movements alter and differ between AD and LBD in daily life. We aimed to comprehensively characterize eye movements of AD and LBD patients on naturalistic complex scenes with broad categories of objects, which would provide a context closer to real-world free viewing, and to identify disease-specific patterns of altered eye movements. We collected spontaneous viewing behaviors to 200 naturalistic complex scenes from patients with AD or LBD at the prodromal or dementia stage, as well as matched control participants. We then investigated eye movement patterns using a computational visual attention model with high-level image features of object properties and semantic information. Compared with matched controls, we identified two disease-specific altered patterns of eye movements: diminished visual exploration, which differentially correlates with cognitive impairment in AD and with motor impairment in LBD; and reduced gaze allocation to objects, attributed to a weaker attention bias toward high-level image features in AD and attributed to a greater image-center bias in LBD. Our findings may help differentiate AD and LBD patients and comprehend their real-world visual behaviors to mitigate the widespread impact of impaired visual attention on daily activities.

Gaze-based attention network analysis in a virtual reality classroom

This article provides a step-by-step guideline for measuring and analyzing visual attention in 3D virtual reality (VR) environments based on eye-tracking data. We propose a solution to the challenges of obtaining relevant eye-tracking information in a dynamic 3D virtual environment and calculating interpretable indicators of learning and social behavior. With a method called "gaze-ray casting," we simulated 3D-gaze movements to obtain information about the gazed objects. This information was used to create graphical models of visual attention, establishing attention networks. These networks represented participants' gaze transitions between different entities in the VR environment over time. Measures of centrality, distribution, and interconnectedness of the networks were calculated to describe the network structure. The measures, derived from graph theory, allowed for statistical inference testing and the interpretation of participants' visual attention in 3D VR environments. Our method provides useful insights when analyzing students’ learning in a VR classroom, as reported in a corresponding evaluation article with N = 274 participants.•Guidelines on implementing gaze-ray casting in VR using the Unreal Engine and the HTC VIVE Pro Eye.•Creating gaze-based attention networks and analyzing their network structure.•Implementation tutorials and the Open Source software code are provided via OSF: https://osf.io/pxjrc/?view_only=1b6da45eb93e4f9eb7a138697b941198.

Systematic Review of Retinal Blood Vessels Segmentation Based on AI-driven Technique.

Image segmentation is a crucial task in computer vision and image processing, with numerous segmentation algorithms being found in the literature. It has important applications in scene understanding, medical image analysis, robotic perception, video surveillance, augmented reality, image compression, among others. In light of this, the widespread popularity of deep learning (DL) and machine learning has inspired the creation of fresh methods for segmenting images using DL and ML models respectively. We offer a thorough analysis of this recent literature, encompassing the range of ground-breaking initiatives in semantic and instance segmentation, including convolutional pixel-labeling networks, encoder-decoder architectures, multi-scale and pyramid-based methods, recurrent networks, visual attention models, and generative models in adversarial settings. We study the connections, benefits, and importance of various DL- and ML-based segmentation models; look at the most popular datasets; and evaluate results in this Literature.

A collective perception model for neighbor selection in groups based on visual attention mechanisms

The mechanisms by which individuals visually perceive and select others within a group to interact with are of fundamental importance to various collective motion behaviors. However, they remain poorly understood theoretically, partially due to the challenges of modeling neurological perception systems. Here, the classic zoom-lens visual attention model is introduced into collective perception, leading to an attention-based model with only a single parameter. Numerical experiments reveal that the proposed model offers a unified mechanism for collective perception, as reflected in three aspects. First, this model describes both conspecific (e.g. flocking) and heterospecific (e.g. predator-prey) collective behaviors. Second, this model unifies the well-known topological and visibility models in the context of neighbor selection in conspecific groups. Third, in the context of prey selection in heterospecific groups, the model can simultaneously replicate the well-known confusion and oddity effects. These findings demonstrate the fundamental role of visual attention underlying a diverse array of collective motion behaviors.

Physical adversarial attack in artificial intelligence of things

AbstractWith the continuous development of wireless communication and artificial intelligence technology, Internet of Things (IoT) technology has made great progress. Deep learning methods are currently used in IoT technology, but deep neural networks (DNNs) are notoriously susceptible to adversarial examples, and subtle pixel changes to images can result in incorrect recognition results from DNNs. In the real‐world application, the patches generated by the recent physical attack methods are larger or less realistic and easily detectable. To address this problem, a Generative Adversarial Network based on Visual attention model and Style transfer network (GAN‐VS) is proposed, which reduces the patch area and makes the patch more natural and less noticeable. A visual attention model combined with generative adversarial network is introduced to detect the critical regions of image recognition, and only generate patches within the critical regions to reduce patch area and improve attack efficiency. For any type of seed patch, an adversarial patch can be generated with a high degree of stylistic and content similarity to the attacked image by generative adversarial network and style transfer network. Experimental evaluation shows that the proposed GAN‐VS has good camouflage and outperforms state‐of‐the‐art adversarial patch attack methods.

Computer-aided autism diagnosis using visual attention models and eye-tracking: replication and improvement proposal

BackgroundAutism Spectrum Disorder (ASD) diagnosis can be aided by approaches based on eye-tracking signals. Recently, the feasibility of building Visual Attention Models (VAMs) from features extracted from visual stimuli and their use for classifying cases and controls has been demonstrated using Neural Networks and Support Vector Machines. The present work has three aims: 1) to evaluate whether the trained classifier from the previous study was generalist enough to classify new samples with a new stimulus; 2) to replicate the previously approach to train a new classifier with a new dataset; 3) to evaluate the performance of classifiers obtained by a new classification algorithm (Random Forest) using the previous and the current datasets.MethodsThe previously approach was replicated with a new stimulus and new sample, 44 from the Typical Development group and 33 from the ASD group. After the replication, Random Forest classifier was tested to substitute Neural Networks algorithm.ResultsThe test with the trained classifier reached an AUC of 0.56, suggesting that the trained classifier requires retraining of the VAMs when changing the stimulus. The replication results reached an AUC of 0.71, indicating the potential of generalization of the approach for aiding ASD diagnosis, as long as the stimulus is similar to the originally proposed. The results achieved with Random Forest were superior to those achieved with the original approach, with an average AUC of 0.95 for the previous dataset and 0.74 for the new dataset.ConclusionIn summary, the results of the replication experiment were satisfactory, which suggests the robustness of the approach and the VAM-based approaches feasibility to aid in ASD diagnosis. The proposed method change improved the classification performance. Some limitations are discussed and additional studies are encouraged to test other conditions and scenarios.

Visual attention predictive model of built colonial heritage based on visual behaviour and subjective evaluation

Although physiological measurements, subjective evaluation and other methods have been applied to visual attention research, architects still lack a systematic quantitative classification method when assessing the visual attention to built colonial heritage. This study aimed to investigate the relationship between people’s visual behaviour and subjective evaluation when observing built colonial heritage and to construct a prediction model based on eye-movement metrics and subjective evaluation indicators to distinguish the visual attention to built colonial heritage. This study recorded data from 54 participants while observing five scenes of built colonial heritage, and the results showed that participants had different visual behaviours and subjective evaluations when viewing built colonial heritage in different scenes. And visual attention to built colonial heritage was negatively correlated with the average saccades peak velocity and average saccades amplitude and positively correlated with the average pupil diameter; visual attention was correlated with 12 subjective evaluation indicators. The eye-movement metrics and subjective evaluation indicators with correlation to visual attention were used as input variables to construct a prediction model of visual attention to built colonial heritage based on the BP neural network. Different built colonial heritage’s low, middle and high visual attention were identified with high accuracy (74.46%). This quantitative method can help architects to measure the visual attention to built colonial heritage to develop conservation and renewal strategies.

Coreference resolution helps visual dialogs to focus

Visual Dialog is a multi-modal task involving both computer vision and dialog systems. The goal is to answer multiple questions in conversation style, given an image as the context. Neural networks with attention modules are widely used for this task, because of their effectiveness in reasoning the relevance between the texts and images. In this work, we study how to further improve the quality of such reasoning, which is an open challenge. Our baseline is the Recursive Visual Attention (RVA) model, which refines the vision-text attention by iteratively visiting the dialog history. Building on top of that, we propose to improve the attention mechanism with contrastive learning. We train a Matching-Aware Attention Kernel (MAAK) by aligning the deep feature embeddings of an image and its caption, to provide better attention scores. Experiments show consistent improvements from MAAK. In addition, we study the effect of using Multimodal Compact Bilinear (MCB) pooling as a three-way feature fusion for the visual, textual and dialog history embeddings. We analyze the performance of both methods in the discussion section, and propose further ideas to resolve current limitations.

Task-Dependent Visual Behavior in Immersive Environments: A Comparative Study of Free Exploration, Memory and Visual Search.

Visual behavior depends on both bottom-up mechanisms, where gaze is driven by the visual conspicuity of the stimuli, and top-down mechanisms, guiding attention towards relevant areas based on the task or goal of the viewer. While this is well-known, visual attention models often focus on bottom-up mechanisms. Existing works have analyzed the effect of high-level cognitive tasks like memory or visual search on visual behavior; however, they haveoften done so with different stimuli, methodology, metrics and participants, which makes drawing conclusions and comparisons between tasks particularly difficult. In this work we present a systematic study of how different cognitive tasks affect visual behavior in a novel within-subjects design scheme. Participants performed free exploration, memory and visual search tasks in three different scenes while their eye and head movements were being recorded. We found significant, consistent differences between tasks in the distributions of fixations, saccades and head movements. Our findings can provide insights for practitioners and content creators designing task-oriented immersive applications.

Driver's visual fixation attention prediction in dynamic scenes using hybrid neural networks

The traffic driving scenario is dynamically varying and experienced drivers can allocate visual attention effectively in brief time intervals, focusing on prominent targets and areas in advance, thus ensuring safe driving. Therefore, it is significant to research the behavior of driver's visual attention allocation in the development of driver assistance systems and autonomous vehicles. Recently, inspired by top-down and bottom-up attention mechanisms, most visual attention models based on neural networks have been proposed. However, these models tend to yield highly distributed predictions for the specific driving scenario, and have relatively low prediction accuracy, making them unfeasible for being used in practice. Hence, to overcome these challenges, we propose a driver's visual attention model (DVAM) built on the encoder-decoder architecture with the utilization of the convolutional neural network (CNN) and the recurrent neural networks (RNN). Specifically, the model leverages the temporal and spatial dimension information to better mimic realistic dynamic driving, and extract abundant feature information, thus ensuring that the final predictions are authentic and effective. Then, extensive experiments demonstrate that our suggested DVAM achieves better robustness and precision in predicting driver attention to areas or targets compared to state-of-the-art methods on the DR(eye)VE dataset. Finally, our model is deployed on the TDV dataset to validate its generalization capabilities, which prove the potential for field adaptation.

Multi-Difference Image Fusion Change Detection Using a Visual Attention Model on VHR Satellite Data

For very-high-resolution (VHR) remote sensing images with complex objects and rich textural information, multi-difference image fusion has been proven as an effective method to improve the performance of change detection. However, errors are superimposed during this process and a single spectral feature cannot fully utilize the correlation between pixels, resulting in low robustness. To overcome these problems and optimize the performance of multi-difference image fusion in change detection, we propose a novel multi-difference image fusion change detection method based on a visual attention model (VA-MDCD). First, we construct difference images using change vector analysis (CVA) and spectral gradient difference (SGD). Second, we use the visual attention model to calculate multiple color, intensity and orientation features of the difference images to obtain the difference saliency images. Third, we use the wavelet transform fusion algorithm to fuse two saliency images. Finally, we execute the OTSU threshold segmentation algorithm (OTSU) to obtain the final change detection map. To validate the effectiveness of VA-MDCD on VHR images, two datasets of Jilin 1 and Beijing 2 are selected for experiments. Compared with classical methods, the proposed method has a better performance with fewer missed alarms (MA) and false alarms (FA), which proves that the method has a strong robustness and generalization ability. The F-measure of the two datasets is 0.6671 and 0.7313, respectively. In addition, the results of ablation experiments confirm that the three feature extraction modules of the model all play a positive role.

An Effective Method of Infrared Maritime Target Enhancement and Detection with Multiple Maritime Scene

Aiming at maritime infrared target detection with low contrast influenced by maritime clutter and illumination, this paper proposes a Modified Histogram Equalization with Edge Fusion (MHEEF) pre-processing algorithm in backlight maritime scenes and establishes Local-Contrast Saliency Models with Double Scale and Modes (LCMDSM) for detecting a target with the properties of positive and negative contrast. We propose a local-contrast saliency mathematical model with double modes in the extension of only one mode. Then, the big scale and small scale are combined into one Target Detection Unit (TDU), which can approach the “from bottom to up” mechanism of the Visual Attention Model (VAM) better and identify the target with a suitable size, approaching the target’s actual shape. In the experimental results and analysis, clutter, foggy, backlight, and dim maritime scenes are chosen to verify the effectiveness of the target detection algorithm. From the enhancement result, the LCMDSM algorithm can achieve a Detection Rate (DR) with a value of 98.26% under each maritime scene on the average level and can be used in real-time detection with low computational cost.

Explaining the visual attention model in impulsive purchase behavior of tourism industry customers by theme analysis method

PurposeThis study aims to explain the model of impulsive purchase behavior in the Islamic Republic of Iran’s tourism industry.Design/methodology/approachThis study uses the inductive–deductive approach for data collection. Field (interviews) and library (articles, books and theses) methods have been used for data collection. After determining the research variables, content analysis units (theme, category and markers) are specified. Academic experts and professionals in the subject area helped during the model design stage.FindingsThe results reveal 14 visual attention variables which are classified into four categories: attractiveness in the market, emotional reactions, improving the shopping style of tourists and visual advertising.Research limitations/implicationsThis study focuses only on Iran. There is scope to expand the discussion with more interviews. The impact of the COVID-19 pandemic on tourism presented some unique challenges in participant recruitment. Future studies could focus on other Middle Eastern countries or other international areas. Moreover, future researchers could analyze other variables affecting impulsive purchase.Originality/valueNo other studies on tourism industry marketing have recognized the effective variables in impulsive purchase. To fill this gap, this paper explains the visual attention model in tourists’ impulsive purchase behavior.

Siamese Network with multi-scale fusion attention for Visual Tracking

The existing trackers based on the Siamese network have poor tracking robustness in the face of complex situations such as target occlusion, rapid target movement and scale changes. To address this problem, we propose a new tracking framework based on the Siamese network with multi-scale fusion attention. The algorithm proposed in this paper increases the receptive field of the last two layers of the backbone network, strengthens the ability to capture target information, and outputs the last three layers of the backbone for feature fusion. We add an improved visual attention model, and the multi-scale channel attention model is used for the first time in this work, so as to strengthen the long-range dependence of feature information and the learning of channel attention, so that the network can select better salient features of the target. In this paper, to avoid to generate complex hyper-parameters for the target candidate frame and speeds up the training speed of the network, we introduce an anchor-free classification and regression network model. The experimental evaluation conducted on the OTB100 and VOT2016 datasets shows that the algorithm in this paper has good robustness in the face of challenges such as target occlusion, rapid target movement and scale changes, and effectively improves the accuracy of our algorithm.

Retraction

Retraction: Tang T, Hu P, Wu G. Influence of promotion mode on purchase decision based on multilevel psychological distance dimension of visual attention model and data mining. J Concurrency Computat Pract Exper. 2021; 33(12).The above article, published online on 26 Nov 2019 Wiley Online Library (https://onlinelibrary.wiley.com/doi/full/10.1002/cpe.5587) has been retracted by agreement between the journal Editor, Professor David W. Walker, School of Computer Science and Informatics, Cardiff University, Cardiff CF24 3AA, UK and Wiley Periodicals, LLC. The retraction has been agreed following an investigation based on allegations raised by a third party. Several inconsistencies were found, including general logical flaws and irrelevant citations, making the motivation and conclusions of the article untrustworthy. The editorial office did not receive a response from the authors, and so requested underlying experimental data was not available for evaluation. Accordingly, the editors consider the conclusions of this manuscript invalid.

Modeling driver behavior in critical traffic scenarios for the safety assessment of automated driving

Objective Before market introduction, the safety of highly automated driving systems needs to be assessed prospectively. BMW has developed a holistic approach for the assessment of the traffic safety impact by these systems in which stochastic traffic simulations play a significant role. A driver behavior model which represents realistic driver behavior ranging from performance in non-critical everyday driving toward performance in critical situations is key for this approach. To ensure trustworthy results, validation of the driver model is needed. The paper aims at demonstrating that the presented driver model acts realistically in different critical real-world traffic scenarios. Methods BMW has been developing the Stochastic Cognitive Model (SCM) which models cognitive processes in traffic situations. These processes range from information acquisition by gaze behavior, mental representation of the environment, recognition of situations from the visual information and reaction to the situation. The driver model combines these cognitive processes with stochastic driver parameters to obtain a variation in driver behavior in simulations. Especially visual attention modeling is key to realistic traffic interactions in simulations as this is the input for the sequential cognitive processes, i.e., the recognition of situations and the reaction to the situation. Modeling of driver’s gaze behavior with SCM is thus shown in this paper. Results SCM is applied in three critical real-world traffic scenarios in which gaze behavior, brake reaction times and time-to-collisions are evaluated and compared to the real-world data. Due to the stochastic approach not only a single SCM agent but a collective of virtual SCM test drivers is assessed. Results show that SCM is capable to simulate the influence of visual inattention on collision risk. Conclusion Realistic driver behavior in simulations can be achieved by using SCM. Especially in the presented critical scenarios SCM is able to represent real-world driving behavior which is determined particularly by its gaze behavior and subsequent reaction. Driving performance varies over different SCM agents which mean that different driving behavior can be simulated with SCM as well. However, the investigation in this paper included only three real-world cases. Therefore, further critical, and additionally non-critical scenarios need to be investigated in the future.

A dynamical scan-path model for task-dependence during scene viewing.

In real-world scene perception, human observers generate sequences of fixations to move image patches into the high-acuity center of the visual field. Models of visual attention developed over the last 25 years aim to predict two-dimensional probabilities of gaze positions for a given image via saliency maps. Recently, progress has been made on models for the generation of scan paths under the constraints of saliency as well as attentional and oculomotor restrictions. Experimental research demonstrated that task constraints can have a strong impact on viewing behavior. Here, we propose a scan-path model for both fixation positions and fixation durations, which include influences of task instructions and interindividual differences. Based on an eye-movement experiment with four different task conditions, we estimated model parameters for each individual observer and task condition using a fully Bayesian dynamical modeling framework using a joint spatial-temporal likelihood approach with sequential estimation. Resulting parameter values demonstrate that model properties such as the attentional span are adjusted to task requirements. Posterior predictive checks indicate that our dynamical model can reproduce task differences in scan-path statistics across individual observers. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

FFR-SSD: feature fusion and reconstruction single shot detector for multi-scale object detection

Object detection is one of the most fundamental tasks toward image content understanding. Although numerous algorithms have been proposed, implementing effective and efficient object detection is still very challenging for now. To solve the challenges of small and multi-scale objects, we propose a hierarchical feature fusion and reconstruction method called feature fusion and reconstruction single shot detector (FFR-SSD). We first present a multi-scale visual attention model, which incorporates the channel-wise and space-wise information into the multi-branch feature enhancement module to improve the feature representation capacities. Second, a hierarchical feature map weighing mechanism is developed to fuse multi-layer feature maps, which contributes to describing the intact objects for the subsequent module. Third, we present an effective feature map reconstruction module to encourage the model to focus on pivotal information. Therefore, the overall contour information is preserved for the shallow enhanced response map and the semantic feature information is enriched for the deep enhanced feature map. Numerous experiments on two public benchmark datasets show that the proposed method achieves significant improvements over the state of the arts.