Human Attention Mechanism Research Articles

Emotion classification with multi‐modal physiological signals using multi‐attention‐based neural network

AbstractThe ability to effectively classify human emotion states is critically important for human‐computer or human‐robot interactions. However, emotion classification with physiological signals is still a challenging problem due to the diversity of emotion expression and the characteristic differences in different modal signals. A novel learning‐based network architecture is presented that can exploit four‐modal physiological signals, electrocardiogram, electrodermal activity, electromyography, and blood volume pulse, and make a classification of emotion states. It features two kinds of attention modules, feature‐level, and semantic‐level, which drive the network to focus on the information‐rich features by mimicking the human attention mechanism. The feature‐level attention module encodes the rich information of each physiological signal. While the semantic‐level attention module captures the semantic dependencies among modals. The performance of the designed network is evaluated with the open‐source Wearable Stress and Affect Detection dataset. The developed emotion classification system achieves an accuracy of 83.88%. Results demonstrated that the proposed network could effectively process four‐modal physiological signals and achieve high accuracy of emotion classification.

Multisource Adaption for Driver Attention Prediction in Arbitrary Driving Scenes

Driver attention cues contribute to the following intended maneuver prediction and provide a risk indicator for the advanced driver-assistance systems in complex driving scenarios. The diverse traffic scenes result in a challenging task to predict human visual attention with high generalization capability. The data heterogeneity caused by multiple sources with different data characteristics, such as video sources, was investigated in the current research and mitigated using the domain adaption modules (i.e., domain-specific batch normalization, Gaussian priors, and smoothing filter) and the domain-specific focal loss. Inspired by human attention mechanism, generic coders and task-driven attention modules were incorporated into a lightweight network to replicate human-like perceptual patterns, such as the perception of latent risk. Integrating these adaptive modules, we proposed the adaptive driver attention (ADA) model to predict salient regions in different traffic scenes. Consequently, the ADA model trained jointly on four driver attention datasets achieves the best performance against the state-of-the-art methods across seven metrics. Retrospective visualizations of the network and cross-validation results further explain the merit of the proposed method. Since these approaches are generic, adaptive modules are universally applicable for standard deep neural network architectures to alleviate the heterogeneity across datasets and easily extended for arbitrary traffic scenes in the real world.

Attention in Reasoning: Dataset, Analysis, and Modeling.

While attention has been an increasingly popular component in deep neural networks to both interpret and boost the performance of models, little work has examined how attention progresses to accomplish a task and whether it is reasonable. In this work, we propose an Attention with Reasoning capability (AiR) framework that uses attention to understand and improve the process leading to task outcomes. We first define an evaluation metric based on a sequence of atomic reasoning operations, enabling a quantitative measurement of attention that considers the reasoning process. We then collect human eye-tracking and answer correctness data, and analyze various machine and human attention mechanisms on their reasoning capability and how they impact task performance. To improve the attention and reasoning ability of visual question answering models, we propose to supervise the learning of attention progressively along the reasoning process and to differentiate the correct and incorrect attention patterns. We demonstrate the effectiveness of the proposed framework in analyzing and modeling attention with better reasoning capability and task performance. The code and data are available at https://github.com/szzexpoi/AiR.

“Focusing on the right regions” — Guided saliency prediction for visual SLAM

Features play an important role in achieving robust visual simultaneous localization and mapping (SLAM) in complex environments. Although all scene features provide a certain amount of information, their importance to SLAM is different. Similar to the human attention mechanism, close attention should be paid to features in salient and important regions. Therefore, this paper proposes a saliency prediction-based SLAM (SP-SLAM), which represents a visual SLAM system that combines the ORB-SLAM3 with a saliency prediction model. The proposed combined saliency prediction model focuses on the right regions by considering geometric, semantic, and depth information, thus making visual SLAM more accurate. Moreover, a multi-level strategy is introduced to make the saliency prediction model continuously focus on the same regions, which can learn the temporally consistent information between adjacent images. Then, the predicted saliency map is used to provide salient weights for robust tracking and optimization to improve the accuracy of visual SLAM. Finally, comprehensive test results show that the proposed SP-SLAM has superior performance in terms of localization accuracy and saliency prediction performance.

Visually imperceptible adversarial patch attacks

The vulnerability of deep neural networks (DNNs) to adversarial examples has attracted more attention. Many algorithms have been proposed to craft powerful adversarial examples. However, most of these algorithms modified the global or local region of pixels without taking network explanations into account. Hence, the perturbations are redundant, which are easily detected by human eyes. In this paper, we propose a novel method to generate local region perturbations. The main idea is to find a contributing feature region (CFR) of an image by simulating the human attention mechanism and then add perturbations to CFR. Furthermore, a soft mask matrix is designed on the basis of an activation map to finely represent the contributions of each pixel in CFR. With this soft mask, we develop a new loss function with inverse temperature to search for optimal perturbations in CFR. Due to the network explanations, the perturbations added to CFR are more effective than those added to other regions. Extensive experiments conducted on CIFAR-10 and ILSVRC2012 demonstrate the effectiveness of the proposed method, including attack success rate, imperceptibility, and transferability.

Thermal Infrared Tracking Method Based on Efficient Global Information Perception.

To solve the insufficient ability of the current Thermal InfraRed (TIR) tracking methods to resist occlusion and interference from similar targets, we propose a TIR tracking method based on efficient global information perception. In order to efficiently obtain the global semantic information of images, we use the Transformer structure for feature extraction and fusion. In the feature extraction process, the Focal Transformer structure is used to improve the efficiency of remote information modeling, which is highly similar to the human attention mechanism. The feature fusion process supplements the relative position encoding to the standard Transformer structure, which allows the model to continuously consider the influence of positional relationships during the learning process. It can also generalize to capture the different positional information for different input sequences. Thus, it makes the Transformer structure model the semantic information contained in images more efficiently. To further improve the tracking accuracy and robustness, the heterogeneous bi-prediction head is utilized in the object prediction process. The fully connected sub-network is responsible for the classification prediction of the foreground or background. The convolutional sub-network is responsible for the regression prediction of the object bounding box. In order to alleviate the contradiction between the vast demand for training data of the Transformer model and the insufficient scale of the TIR tracking dataset, the LaSOT-TIR dataset is generated with the generative adversarial network for network training. Our method achieves the best performance compared with other state-of-the-art trackers on the VOT2015-TIR, VOT2017-TIR, PTB-TIR and LSOTB-TIR datasets, and performs outstandingly especially when dealing with severe occlusion or interference from similar objects.

Micro-expression spotting method based on human attention mechanism

Micro-expression spotting method based on human attention mechanism

Review of Visual Saliency Prediction: Development Process from Neurobiological Basis to Deep Models

The human attention mechanism can be understood and simulated by closely associating the saliency prediction task to neuroscience and psychology. Furthermore, saliency prediction is widely used in computer vision and interdisciplinary subjects. In recent years, with the rapid development of deep learning, deep models have made amazing achievements in saliency prediction. Deep learning models can automatically learn features, thus solving many drawbacks of the classic models, such as handcrafted features and task settings, among others. Nevertheless, the deep models still have some limitations, for example in tasks involving multi-modality and semantic understanding. This study focuses on summarizing the relevant achievements in the field of saliency prediction, including the early neurological and psychological mechanisms and the guiding role of classic models, followed by the development process and data comparison of classic and deep saliency prediction models. This study also discusses the relationship between the model and human vision, as well as the factors that cause the semantic gaps, the influences of attention in cognitive research, the limitations of the saliency model, and the emerging applications, to provide new saliency predictions for follow-up work and the necessary help and advice.

TDAF: Top-Down Attention Framework for Vision Tasks

Human attention mechanisms often work in a top-down manner, yet it is not well explored in vision research. Here, we propose the Top-Down Attention Framework (TDAF) to capture top-down attentions, which can be easily adopted in most existing models. The designed Recursive Dual-Directional Nested Structure in it forms two sets of orthogonal paths, recursive and structural ones, where bottom-up spatial features and top-down attention features are extracted respectively. Such spatial and attention features are nested deeply, therefore, the proposed framework works in a mixed top-down and bottom-up manner. Empirical evidence shows that our TDAF can capture effective stratified attention information and boost performance. ResNet with TDAF achieves 2.0% improvements on ImageNet. For object detection, the performance is improved by 2.7% AP over FCOS. For pose estimation, TDAF improves the baseline by 1.6%. And for action recognition, the 3D-ResNet adopting TDAF achieves improvements of 1.7% accuracy.

A robust RGBD saliency method with improved probabilistic contrast and the global reference surface

The human attention mechanism inspires salient object detection. Most of the saliency methods work on 2D perception mechanisms, while human attention systems work on 3D perception mechanisms. This proposed method makes use of depth information from RGBD to robustly and correctly detect the salient object in a complex and clutter background. The saliency of regions related to object border increases in Poisson probabilistic contrast space while distinguishing the conspicuous object in a complex and clutter background. This process produces a global concave reference surface. This global reference plane integrated with intra-regional spatial, structural, color, and depth information detects the salient object correctly. Background estimation and central saliency integration thoroughly remove the background. This algorithm generates a robust conspicuous object. The experimental result presented here shows that the proposed method performs better in comparison to the recent, highly referenced and closely related fourteen state-of-the-art methods, and the three publicly available complex RGBD datasets and six evaluation parameters.

Rethinking Image Salient Object Detection: Object-Level Semantic Saliency Reranking First, Pixelwise Saliency Refinement Later.

Human attention is an interactive activity between our visual system and our brain, using both low-level visual stimulus and high-level semantic information. Previous image salient object detection (SOD) studies conduct their saliency predictions via a multitask methodology in which pixelwise saliency regression and segmentation-like saliency refinement are conducted simultaneously. However, this multitask methodology has one critical limitation: the semantic information embedded in feature backbones might be degenerated during the training process. Our visual attention is determined mainly by semantic information, which is evidenced by our tendency to pay more attention to semantically salient regions even if these regions are not the most perceptually salient at first glance. This fact clearly contradicts the widely used multitask methodology mentioned above. To address this issue, this paper divides the SOD problem into two sequential steps. First, we devise a lightweight, weakly supervised deep network to coarsely locate the semantically salient regions. Next, as a postprocessing refinement, we selectively fuse multiple off-the-shelf deep models on the semantically salient regions identified by the previous step to formulate a pixelwise saliency map. Compared with the state-of-the-art (SOTA) models that focus on learning the pixelwise saliency in single images using only perceptual clues, our method aims at investigating the object-level semantic ranks between multiple images, of which the methodology is more consistent with the human attention mechanism. Our method is simple yet effective, and it is the first attempt to consider salient object detection as mainly an object-level semantic reranking problem.

Cross-Camera Person Re-Identification With Body-Guided Attention Network

Various challenges exist throughout person re-identification (ReID) process, including background clutters, illumination variation, pose variation, occlusion, etc. Addressing these problems, this paper explores the incorporation of human attention mechanism in person ReID and proposes an attention-aware model named Body-guided Attention Network (BANet). The proposed attention is based on the body masked images which are obtained by a reliable pixel-level segmentation strategy. To optimize the feature representation learning so as to pay more attention to the discriminative details of human body, BANet is built. It is composed of three attention branches. In order to guide attention learning layer by layer, these branches are applied to the convolution features of different levels. The proposed BANet aims to fully utilize fine-grained information of body region to guide the final process of feature extraction. Extensive experiments on benchmarks including CUHK03, Market1501 and DukeMTMC-reID show that BANet can achieve state-of-the-art performance, which validates the importance of attention mechanism in person ReID.

DSTP-RNN: A dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series prediction

Long-term prediction of multivariate time series is still an important but challenging problem. The key to solve this problem is capturing (1) the spatial correlations at the same time, (2) the spatio-temporal relationships at different times, and (3) long-term dependency of the temporal relationships between different series. Attention-based recurrent neural networks (RNN) can effectively represent and learn the dynamic spatio-temporal relationships between exogenous series and target series, but they only perform well in one-step time prediction and short-term time prediction. In this paper, inspired by human attention mechanism including the dual-stage two-phase (DSTP) model and the influence mechanism of target information and non-target information, we propose DSTP-based RNN (DSTP-RNN) and DSTP-RNN-Ⅱ respectively for long-term time series prediction. Specifically, we first propose the DSTP-based structure to enhance the spatial correlations between exogenous series. The first phase produces violent but decentralized response weight, while the second phase leads to stationary and concentrated response weight. Then, we employ multiple attentions on target series to boost the long-term dependency. Finally, we study the performance of deep spatial attention mechanism and provide interpretation. Experimental results demonstrate that the present work can be successfully used to develop expert or intelligent systems for a wide range of applications, with state-of-the-art performances superior to nine baseline methods on four datasets in the fields of energy, finance, environment and medicine, respectively. Overall, the present work carries a significant value not merely in the domain of machine intelligence and deep learning, but also in the fields of many applications.

Attention-mechanism-based tracking method for intelligent Internet of vehicles

Vehicle tracking task plays an important role on the Internet of vehicles and intelligent transportation system. Beyond the traditional Global Positioning System sensor, the image sensor can capture different kinds of vehicles, analyze their driving situation, and can interact with them. Aiming at the problem that the traditional convolutional neural network is vulnerable to background interference, this article proposes vehicle tracking method based on human attention mechanism for self-selection of deep features with an inter-channel fully connected layer. It mainly includes the following contents: (1) a fully convolutional neural network fused attention mechanism with the selection of the deep features for convolution; (2) a separation method for template and semantic background region to separate target vehicles from the background in the initial frame adaptively; (3) a two-stage method for model training using our traffic dataset. The experimental results show that the proposed method improves the tracking accuracy without an increase in tracking time. Meanwhile, it strengthens the robustness of algorithm under the condition of the complex background region. The success rate of the proposed method in overall traffic datasets is higher than Siamese network by about 10%, and the overall precision is higher than Siamese network by 8%.

Machine Listening for Park Soundscape Quality Assessment

The increasing importance attributed to soundscape quality in urban design generates a need for a system for automatic quality assessment that could be used for example in monitoring. In this work, the possibility for using machine listening techniques for this purpose is explored. The outlined approach detects the presence of particular sounds in a human-inspired way, and therefore allows to draw conclusions about how soundscapes are perceived. The system proposed in this paper consists of a partly recurrent artificial neural network modified to incorporate human attention mechanisms. The network is trained on sounds recorded in typical urban parks in the city of Antwerp, and thus becomes an auditory object creation and classification system particularly tuned to this context. The system is used to analyze a continuous sound level recording in different parks, resulting in a prediction of sounds that will most likely be noticed by a park visitor. Finally, it is shown that these indicators for noticed sounds allow to construct more powerful models for soundscape quality as reported in a survey with park visitors than indicators that are more regularly used in soundscape research.

Attentional biases and memory for emotional stimuli in men and male rhesus monkeys

We examined attentional biases for social and non-social emotional stimuli in young adult men and compared the results to those of male rhesus monkeys (Macaca mulatta) previously tested in a similar dot-probe task (King et al. in Psychoneuroendocrinology 37(3):396-409, 2012). Recognition memory for these stimuli was also analyzed in each species, using a recognition memory task in humans and a delayed non-matching-to-sample task in monkeys. We found that both humans and monkeys displayed a similar pattern of attentional biases toward threatening facial expressions of conspecifics. The bias was significant in monkeys and of marginal significance in humans. In addition, humans, but not monkeys, exhibited an attentional bias away from negative non-social images. Attentional biases for social and non-social threat differed significantly, with both species showing a pattern of vigilance toward negative social images and avoidance of negative non-social images. Positive stimuli did not elicit significant attentional biases for either species. In humans, emotional content facilitated the recognition of non-social images, but no effect of emotion was found for the recognition of social images. Recognition accuracy was not affected by emotion in monkeys, but response times were faster for negative relative to positive images. Altogether, these results suggest shared mechanisms of social attention in humans and monkeys, with both species showing a pattern of selective attention toward threatening faces of conspecifics. These data are consistent with the view that selective vigilance to social threat is the result of evolutionary constraints. Yet, selective attention to threat was weaker in humans than in monkeys, suggesting that regulatory mechanisms enable non-anxious humans to reduce sensitivity to social threat in this paradigm, likely through enhanced prefrontal control and reduced amygdala activation. In addition, the findings emphasize important differences in attentional biases to social versus non-social threat in both species. Differences in the impact of emotional stimuli on recognition memory between monkeys and humans will require further study, as methodological differences in the recognition tasks may have affected the results.

Neurophysiological mechanisms of auditory selective attention in humans.

This chapter reviews the main data on the physiological substrates of auditory selective attention and their contribution to theoretical models of cognitive psychology. While event-related potentials, magnetoencephalography, and more recently neuroimaging techniques have provided fundamental information on the neural correlates of attention in the central cortical system, measurements of the frequency-following responses in the brainstem and evoked otoacoustic emissions at the cochlea strongly suggest attentional phenomena at the auditory periphery. We propose an adaptive filtering mechanism for selective auditory attention that can be flexibly and dynamically tuned depending on the attentional demand.

Neurophysiological mechanisms of auditory selective attention in humans

Neurophysiological mechanisms of auditory selective attention in humans

PASAT performance and the pattern of uptake of 99mTc-exametazime in brain estimated with single photon emission tomography

The effect of the paced auditory serial addition test (PASAT) on the regional uptake of 99mTc-exametazime was determined by single photon emission computed tomography. Twenty insulin-treated diabetic outpatients were scanned at rest and during the performance of the PASAT task using split-dose injection of tracer. When resting and activation scans were compared there were significant decreases in tracer uptake in the right anterior cingulate and left posterior cingulate areas during PASAT activation. The findings are compared with previous studies which had implicated the anterior cingulate area in the mechanisms of attention in humans and other animals. The potentially confounding role of anxiety during attentional tasks is discussed.