Human Visual Attention Research Articles

LeNo: Adversarial Robust Salient Object Detection Networks with Learnable Noise

Pixel-wise prediction with deep neural network has become an effective paradigm for salient object detection (SOD) and achieved remarkable performance. However, very few SOD models are robust against adversarial attacks which are visually imperceptible for human visual attention. The previous work robust saliency (ROSA) shuffles the pre-segmented superpixels and then refines the coarse saliency map by the densely connected conditional random field (CRF). Different from ROSA that rely on various pre- and post-processings, this paper proposes a light-weight Learnable Noise (LeNo) to defend adversarial attacks for SOD models. LeNo preserves accuracy of SOD models on both adversarial and clean images, as well as inference speed. In general, LeNo consists of a simple shallow noise and noise estimation that embedded in the encoder and decoder of arbitrary SOD networks respectively. Inspired by the center prior of human visual attention mechanism, we initialize the shallow noise with a cross-shaped gaussian distribution for better defense against adversarial attacks. Instead of adding additional network components for post-processing, the proposed noise estimation modifies only one channel of the decoder. With the deeply-supervised noise-decoupled training on state-of-the-art RGB and RGB-D SOD networks, LeNo outperforms previous works not only on adversarial images but also on clean images, which contributes stronger robustness for SOD. Our code is available at https://github.com/ssecv/LeNo.

Efficient Rate Control in Versatile Video Coding With Adaptive Spatial–Temporal Bit Allocation and Parameter Updating

Despite the fact that Versatile Video Coding (VVC) has achieved superior coding performance, two major problems remain for the rate control (RC) model in VVC. First, the regions concerned by human eyes are not clear enough in the coded video due to the deviation between the target bit allocation strategy of the coding tree unit (CTU) in RC and the human visual attention mechanism (HVAM). Second, there are significant quality fluctuations in the coded video frames due to the inappropriate updating speed. To address the above problems, we propose an efficient rate control (ERC) model. Specifically, in order to make the coded video more consistent with the attention of human eyes, we extract texture and motion-based spatial-temporal information to guide the bit allocation at the CTU level. Furthermore, based on the quasi-Newton algorithm and bit error, we propose an adaptive parameter updating (APU) method with the proper updating speed to precisely control the bits per frame. The proposed ERC outperforms the default RC model of VVC Test Model (VTM) 9.1 by saving the average Bjøntegaard Delta Rate (BD-Rate) on full-frame video sequences by 3.60% and 4.94% under low delay P (LDP) and random access (RA) configurations respectively, with higher bitrate accuracy. Moreover, the Peak Signal-to-Noise Ratio (PSNR) and actual coded bits per frame in the video coded by the proposed ERC are more stable.

A case study of left visual neglect after right pontine lesion: pathophysiological evidence for the infratentorial involvement in human visual attention.

Hemispatial neglect, the inability to attend to the contralesional side of space, is the most common disturbance of visuospatial attention. Both hemispatial neglect and visuospatial attention are typically associated with extended cortical networks. Nevertheless, recent accounts challenge this so-called corticocentric view and postulate the participation of structures well beyond the telencephalic cortex, in particular advocating the role of the brainstem. However, to the best of our knowledge, hemispatial neglect after a brainstem lesion has not yet been described. We describe, for the first time in a human, the occurrence and remission of contralesional visual hemispatial neglect after a focal lesion in the right pons. Hemispatial neglect was assessed by means of video-oculography during free visual exploration, a very sensitive and established method, and its remission was followed up until 3 wk after stroke. Moreover, by means of a lesion-deficit approach complemented by imaging, we identify a pathophysiological mechanism involving the disconnection of cortico-ponto-cerebellar and/or tecto-cerebellar-tectal pathways passing through the pons. Our findings offer, for the first time in a human, causal, lesion-based support for recent seminal accounts postulating the role of infratentorial structures participating in the activity of cerebral cortical attentional networks mediating attentional processes.NEW & NOTEWORTHY Visuospatial attention and its most common disturbance, hemispatial neglect, are typically associated with extended cortical networks. However, recent accounts challenge this corticocentric view and advocate the role of infratentorial structures. We describe, for the first time in a human, the occurrence of contralesional visual hemispatial neglect after a focal lesion in the right pons. We provide causal, lesion-based evidence for a pathophysiological mechanism involving the disconnection of cortico-ponto-cerebellar and/or tecto-cerebellar-tectal pathways passing through the pons.

Human Visual Attention Mechanism-Inspired Point-and-Line Stereo Visual Odometry for Environments with Uneven Distributed Features

Visual odometry is critical in visual simultaneous localization and mapping for robot navigation. However, the pose estimation performance of most current visual odometry algorithms degrades in scenes with unevenly distributed features because dense features occupy excessive weight. Herein, a new human visual attention mechanism for point-and-line stereo visual odometry, which is called point-line-weight-mechanism visual odometry (PLWM-VO), is proposed to describe scene features in a global and balanced manner. A weight-adaptive model based on region partition and region growth is generated for the human visual attention mechanism, where sufficient attention is assigned to position-distinctive objects (sparse features in the environment). Furthermore, the sum of absolute differences algorithm is used to improve the accuracy of initialization for line features. Compared with the state-of-the-art method (ORB-VO), PLWM-VO show a 36.79% reduction in the absolute trajectory error on the Kitti and Euroc datasets. Although the time consumption of PLWM-VO is higher than that of ORB-VO, online test results indicate that PLWM-VO satisfies the real-time demand. The proposed algorithm not only significantly promotes the environmental adaptability of visual odometry, but also quantitatively demonstrates the superiority of the human visual attention mechanism.

Emotion estimation model for cognitive state analysis of learners in online education using deep learning

AbstractFor Facial Expression Recognition, occlusion and position change that may drastically alter facial expressions are two important challenges (FER). Due to advances in automated FER over the last several decades, it has received less attention in the real world, where occlusion‐ and pose‐invariant aspects of FER are critical. Online education learners' cognitive states may be assessed using this paper's focus on real‐world stance and occlusion robust FER. The human visual system's attention mechanism inspired us to develop a new kind of spatial attention network (SAN‐CNN). Saliency characteristics and spatial importance between adjacent pixels are emphasized in the SAN‐CNN model. Preprocessing an input picture using a median contour filter is used here initially, followed by mask‐based ROI for segmentation. Using CNNs for facial recognition, landmark localization, and head position estimation based on spatial attention networks, it is possible to do emotional categorization. Using the Kaggle public video‐based facial expression datasets, we were able to demonstrate that our proposed approach is more accurate and faster than the usual techniques. In addition, we evaluated the suggested method's performance metrics with those of the already used approaches. FER with occlusion and variant posture performs better than standard approaches using our suggested method, as shown by the results of the experiments.

EHTask: Recognizing User Tasks From Eye and Head Movements in Immersive Virtual Reality.

Understanding human visual attention in immersive virtual reality (VR) is crucial for many important applications, including gaze prediction, gaze guidance, and gaze-contingent rendering. However, previous works on visual attention analysis typically only explored one specific VR task and paid less attention to the differences between different tasks. Moreover, existing task recognition methods typically focused on 2D viewing conditions and only explored the effectiveness of human eye movements. We first collect eye and head movements of 30 participants performing four tasks, i.e., Free viewing, Visual search, Saliency, and Track, in 15 360-degree VR videos. Using this dataset, we analyze the patterns of human eye and head movements and reveal significant differences across different tasks in terms of fixation duration, saccade amplitude, head rotation velocity, and eye-head coordination. We then propose EHTask - a novel learning-based method that employs eye and head movements to recognize user tasks in VR. We show that our method significantly outperforms the state-of-the-art methods derived from 2D viewing conditions both on our dataset (accuracy of 84.4% versus 62.8%) and on a real-world dataset ( 61.9% versus 44.1%). As such, our work provides meaningful insights into human visual attention under different VR tasks and guides future work on recognizing user tasks in VR.

Visual attention during neonatal pain assessment: A 2‐s exposure to a facial expression is sufficient

AbstractNewborn infants are exposed to painful experiences, routinely practised clinically, that might increase their short‐ and long‐term morbidity and mortality. Facial expression has been widely used in clinical practice to assess pain in newborns. However, the inherent human visual attention required to make such vital inference is poorly understood. It is also unknown whether this inference occurs differently when comparing health professionals with non‐health ones. To investigate these issues, the authors have recorded and monitored the pupil size signal of 102 subjects (44 experts, 29 parents, and 29 non‐experts) while visually analyzing 20 frontal face images of 10 distinct newborns after a painful procedure (positive stimulus) and at painless rest (negative stimulus). Our experimental results show that neonatal pain assessment is more cognitively demanding when analyzing the presence of pain rather than its absence. In addition, the pupil responses for both positive and negative stimuli and all sample groups of subjects present experimentally a common temporal pattern, disclosing that a 2‐s exposure to a facial expression is sufficient to make this assessment, regardless of whether performed by health professionals or non‐health ones.

An improved saliency model of visual attention dependent on image content.

Many visual attention models have been presented to obtain the saliency of a scene, i.e., the visually significant parts of a scene. However, some mechanisms are still not taken into account in these models, and the models do not fit the human data accurately. These mechanisms include which visual features are informative enough to be incorporated into the model, how the conspicuity of different features and scales of an image may integrate to obtain the saliency map of the image, and how the structure of an image affects the strategy of our attention system. We integrate such mechanisms in the presented model more efficiently compared to previous models. First, besides low-level features commonly employed in state-of-the-art models, we also apply medium-level features as the combination of orientations and colors based on the visual system behavior. Second, we use a variable number of center-surround difference maps instead of the fixed number used in the other models, suggesting that human visual attention operates differently for diverse images with different structures. Third, we integrate the information of different scales and different features based on their weighted sum, defining the weights according to each component's contribution, and presenting both the local and global saliency of the image. To test the model's performance in fitting human data, we compared it to other models using the CAT2000 dataset and the Area Under Curve (AUC) metric. Our results show that the model has high performance compared to the other models (AUC = 0.79 and sAUC = 0.58) and suggest that the proposed mechanisms can be applied to the existing models to improve them.

FESNet: Frequency-Enhanced Saliency Detection Network for Grain Pest Segmentation

As insect infestation is the leading factor accounting for nutritive and economic losses in stored grains, it is important to detect the presence and number of insects for the sake of taking proper control measures. Inspired by the human visual attention mechanism, we propose a U-net-like frequency-enhanced saliency (FESNet) detection model, resulting in the pixelwise segmentation of grain pests. The frequency clues, as well as the spatial information, are leveraged to enhance the detection performance of small insects from the cluttered grain background. Firstly, we collect a dedicated dataset, GrainPest, with pixel-level annotation after analyzing the image attributes of the existing salient object detection datasets. Secondly, we design a FESNet with the discrete wavelet transformation (DWT) and the discrete cosine transformation (DCT), both involved in the traditional convolution layers. As current salient object detection models will reduce the spatial information with pooling operations in the sequence of encoding stages, a special branch of the discrete wavelet transformation (DWT) is connected to the higher stages to capture accurate spatial information for saliency detection. Then, we introduce the discrete cosine transform (DCT) into the backbone bottlenecks to enhance the channel attention with low-frequency information. Moreover, we also propose a new receptive field block (NRFB) to enlarge the receptive fields by aggregating three atrous convolution features. Finally, in the phase of decoding, we use the high-frequency information and aggregated features together to restore the saliency map. Extensive experiments and ablation studies on our dataset, GrainPest, and open dataset, Salient Objects in Clutter (SOC), demonstrate that the proposed model performs favorably against the state-of-the-art model.

HVS-Based Perception-Driven No-Reference Omnidirectional Image Quality Assessment

Evaluating the quality of panoramic images has gradually become a hot research topic with the development of virtual reality (VR) technology. Therefore, a novel method is proposed to assess the quality of omnidirectional images without any reference information. Inspired by the characteristics of the human visual system (HVS) and visual attention mechanism, the proposed model is composed of the structure feature, statistical feature, and saliency feature to measure the panoramic image quality, in which structure information is expressed by combining the local Taylor series with the local binary pattern (LBP) operator, gradient-based statistical information of panoramic images are summarized comprehensively from three levels: the gradient measure, the relative gradient magnitude, and the relative gradient orientation, and the saliency detection by combining simple priors (SDSP)-based saliency information is extracted in this article to enrich perception feature of our model and improve the visibility of the saliency region in the omnidirectional image. Finally, according to the subjective scores provided and the above features, we use support vector regression (SVR) to predict the objective scores. The experiments indicate that our model has more substantial competitiveness and stability than other state-of-the-art methods on two reliable databases.

Detecting dynamic visual attention in augmented reality aided navigation environment based on a multi-feature integration fully convolutional network

ABSTRACT Visual attention detection, as an important concept for human visual behavior research, has been widely studied. However, previous studies seldom considered the feature integration mechanism to detect visual attention and rarely considered the differences due to different geographical scenes. In this paper, we use an augmented reality aided (AR-aided) navigation experimental dataset to study human visual behavior in a dynamic AR-aided environment. Then, we propose a multi-feature integration fully convolutional network (M-FCN) based on a self-adaptive environment weight (SEW) to integrate RGB-D, semantic, optical flow and spatial neighborhood features to detect human visual attention. The result shows that the M-FCN performs better than other state-of-the-art saliency models. In addition, the introduction of feature integration mechanism and the SEW can improve the accuracy and robustness of visual attention detection. Meanwhile, we find that RGB-D and semantic features perform best in different road routes and road types, but with the increase in road type complexity, the expressiveness of these two features weakens, and the expressiveness of optical flow and spatial neighborhood features increases. The research is helpful for AR-device navigation tool design and urban spatial planning.

Attentional bias towards social interactions during viewing of naturalistic scenes.

Human visual attention is readily captured by the social information in scenes. Multiple studies have shown that social areas of interest (AOIs) such as faces and bodies attract more attention than non-social AOIs (e.g., objects or background). However, whether this attentional bias is moderated by the presence (or absence) of a social interaction remains unclear. Here, the gaze of 70 young adults was tracked during the free viewing of 60 naturalistic scenes. All photographs depicted two people, who were either interacting or not. Analyses of dwell time revealed that more attention was spent on human than background AOIs in the interactive pictures. In non-interactive pictures, however, dwell time did not differ between AOI type. In the time-to-first-fixation analysis, humans always captured attention before other elements of the scene, although this difference was slightly larger in interactive than non-interactive scenes. These findings confirm the existence of a bias towards social information in attentional capture and suggest our attention values social interactions beyond the presence of two people.

A saliency-guided street view image inpainting framework for efficient last-meters wayfinding

Global Positioning Systems (GPS) have played a crucial role in various navigation applications. Nevertheless, localizing the perfect destination within the last few meters remains an important but unresolved problem. Limited by the GPS positioning accuracy, navigation systems always show users a vicinity of a destination, but not its exact location. Street view images (SVI) in maps as an immersive media technology have served as an aid to provide the physical environment for human last-meters wayfinding. However, due to the large diversity of geographic context and acquisition conditions, the captured SVI always contains various distracting objects (e.g., pedestrians and vehicles), which will distract human visual attention from efficiently finding the destination in the last few meters. To address this problem, we highlight the importance of reducing visual distraction in image-based wayfinding by proposing a saliency-guided image inpainting framework. It aims at redirecting human visual attention from distracting objects to destination-related objects for more efficient and accurate wayfinding in the last meters. Specifically, a context-aware distracting object detection method driven by deep salient object detection has been designed to extract distracting objects from three semantic levels in SVI. Then we employ a large-mask inpainting method with fast Fourier convolutions to remove the detected distracting objects. Experimental results with both qualitative and quantitative analysis show that our saliency-guided inpainting method11Codes are available at https://github.com/cbhu523/saliency_last_way_finding/. can not only achieve great perceptual quality in street view images but also redirect the human’s visual attention to focus more on static location-related objects than distracting ones. The human-based evaluation also justified the effectiveness of our method in improving the efficiency of locating the target destination.

Gaze Grammars - Is there an invariant hierarchical sequential structure of human visual attention in natural tasks?

Gaze Grammars - Is there an invariant hierarchical sequential structure of human visual attention in natural tasks?

All Two‐dimensional Integration‐Type Optoelectronic Synapse Mimicking Visual Attention Mechanism for Multi‐Target Recognition

AbstractThe human visual attention mechanism enables them to rapidly perceive important information and objects in a complex external scene; this effectively solves the problems of data redundancy, low‐resolution images, and substantial computing resources. The process by which the attention system reconstructs the visual information can be considered as integrating internal attention signals with external visual details in the postsynaptic neuron. However, electronic devices that simulate visual attention modulation by incorporating device characteristics into neuromorphic vision systems (NVSs) to achieve visual attention behavior are rarely reported. Herein, a synapse device that integrates optical and electrical stimulation is designed using ReS2/hBN/monolayer graphene heterojunction to mimic attention regulation and integrate multiple neuron signals successfully. The synapse array can imitate perceptual learning of the human visual system (HVS) to realize visual preprocessing, such as image contrast improvement and weak signal enhancement at the sensory terminal, and overcome data redundancy. Moreover, by applying gate voltage pulses, electric‐tunable synaptic plasticity is successfully observed, attributed to the carrier trapping and de‐trapping mechanism in the floating layer. Attention stabilization, fluctuation, distraction, and reinforcement are exhibited, simulating the attention behaviors of the HVS. Thus, an NVS with attention mechanism is established depending on the optoelectronic hybrid synaptic plasticity of the device, which successfully mimics the HVS to perform a multi‐target recognition task. Furthermore, the effect of device defects on the NVS is rarely evaluated, in which a method is provided to analyze the application results of the NVS when considering uniformity and fault rate. This study may provide new inspiration for developing neuromorphic vision systems for autonomous driving and brainwave control in the future.

Working memory biases early object discrimination and parietal activity during attentional selection.

The contents of working memory (WM) guide visual attention, but the neural mechanisms underlying WM biases remains unclear. Here, we used simultaneous electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) approaches to characterize the timing and location of the neural response underlying WM guidance during a visual search task. Behaviorally, we observed faster search performance when the WM contents matching targets (valid) compared to when WM contents did not reappear (neutral). The EEG data showed similar benefit effects of posterior N1 component, in which targets induced larger N1 amplitudes in the valid condition than in the neutral condition. Interestingly, the fMRI activation in left supramarginal gyrus (SMG)/inferior parietal lobule (IPL) and bilateral occipital cortex was lower in the valid compared to neutral conditions. Importantly, the magnitude of the increased N1 activity and the decreased fMRI activity in the left SMG/IPL predicted the extent of search improvement at an individual subject level. These results suggest that information held in WM enhances early object discrimination during attentional selection, and the left SMG/IPL may be a critical region in mediating goal-directed processing under WM biases in human visual attention.

Salient detection via the fusion of background-based and multiscale frequency-domain features

Salient object detection is a fundamental problem in image processing and computer vision. Many saliency detection algorithms based on the background and frequency-domain are used to extract salient object clues. However, the former causes the real object to be submerged in the detected object areas, especially in complex or small object scenes. While the latter will lead to the loss of some object information when detecting large objects. To solve these problems and achieve better object detection results, we propose a fusion framework for salient object detection by fusing background and frequency-domain features. The background features of the image are extracted by an improved background model. This model represents the spatial layout of the image area with respect to the image boundaries. Meanwhile, we present a new frequency-domain processing method to obtain multiscale frequency-domain features and mark the saliency of the object at different scales. Within our framework, inspired by human visual attention, we use the idea of a self-attention mechanism to capture the intrinsic relation between background and multiscale frequency-domain features. In addition, this fusion framework provides a three-dimensional Gaussian convolution kernel, which expands two-dimensional local information to three dimensions for feature fusion, thus producing more accurate salient objects. Experiment results demonstrate that the proposed method consistently outperforms eleven state-of-the-art methods on five challenging and complicated datasets in terms of four evaluation metrics.

Attention for Vision-Based Assistive and Automated Driving: A Review of Algorithms and Datasets

Driving safety has been a concern since the first cars appeared on the streets. Driver inattention has been singled out as a major cause of accidents early on. This is hardly surprising, as drivers routinely perform other tasks in addition to controlling the vehicle. Decades of research into what causes lapses or misdirection of drivers’ attention resulted in improvements in road safety through better design of infrastructure, driver training programs, in- vehicle interfaces, and, more recently, the development of driving assistance systems (ADAS) and driving automation. This review focuses on the methods for modeling and detecting spatio-temporal aspects of drivers’ attention, i. e. where and when they look, for the two latter categories of applications. We start with a brief theoretical background on human visual attention, methods for recording and measuring attention in the driving context, types of driver inattention, and factors causing it. We then discuss machine learning approaches for 1) modeling gaze for assistive and self-driving applications and 2) detecting gaze for driver monitoring. Following the overview of state-of-the-art models, we provide an extensive list of publicly available datasets that feature recordings of drivers’ gaze and other attention-related annotations. We conclude with a general overview of the remaining challenges, such as data availability and quality, evaluation methods, and the limited scope of attention modeling, and outline steps toward rectifying some of these issues. Categorized and annotated lists of the reviewed models and datasets are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ykotseruba/attention_and_driving</uri>

Multisource Adaption for Driver Attention Prediction in Arbitrary Driving Scenes

Driver attention cues contribute to the following intended maneuver prediction and provide a risk indicator for the advanced driver-assistance systems in complex driving scenarios. The diverse traffic scenes result in a challenging task to predict human visual attention with high generalization capability. The data heterogeneity caused by multiple sources with different data characteristics, such as video sources, was investigated in the current research and mitigated using the domain adaption modules (i.e., domain-specific batch normalization, Gaussian priors, and smoothing filter) and the domain-specific focal loss. Inspired by human attention mechanism, generic coders and task-driven attention modules were incorporated into a lightweight network to replicate human-like perceptual patterns, such as the perception of latent risk. Integrating these adaptive modules, we proposed the adaptive driver attention (ADA) model to predict salient regions in different traffic scenes. Consequently, the ADA model trained jointly on four driver attention datasets achieves the best performance against the state-of-the-art methods across seven metrics. Retrospective visualizations of the network and cross-validation results further explain the merit of the proposed method. Since these approaches are generic, adaptive modules are universally applicable for standard deep neural network architectures to alleviate the heterogeneity across datasets and easily extended for arbitrary traffic scenes in the real world.

What the Barn Owl Can Teach Us About Human Visual Attention

Visual attention is the brain’s process of extracting information we receive through the eyes. It guides our gaze to the most important things around us. How do scientists research such a complicated subject? And can birds, which are separated from humans by 300 million years of evolution, help us understand attention in the human brain? The barn owl is a member of the owl family, and its extremely sharp vision and hearing allow it to hunt in total darkness. Researchers can use the barn owl to learn about attention, both in animals and humans. This article will explain the visual attention experiments we performed using barn owls and what we learned from them. An understanding of visual attention may help doctors and scientists to treat important brain disorders and may also help with the development of technologies that need to “pay attention,” like self-driving cars.