Visual Attention Mechanism Research Articles

SVA-SSD: saliency visual attention single shot detector for building detection in low contrast high-resolution satellite images.

Building detection in high-resolution satellite images has received great attention, as it is important to increase the accuracy of urban planning. The building boundary detection in the desert environment is a real challenge due to the nature of low contrast images in the desert environment. The traditional computer vision algorithms for building boundary detection lack scalability, robustness, and accuracy. On the other hand, deep learning detection algorithms have not been applied to such low contrast satellite images. So, there is a real need to employ deep learning algorithms for building detection tasks in low contrast high-resolution images. In this paper, we propose a novel building detection method based on a single-shot multi-box (SSD) detector. We develop the state-of-the-art SSD detection algorithm based on three approaches. First, we propose data-augmentation techniques to overcome the low contrast images’ appearance. Second, we develop the SSD backbone using a novel saliency visual attention mechanism. Moreover, we investigate several pre-trained networks performance and several fusion functions to increase the performance of the SSD backbone. The third approach is based on optimizing the anchor-boxes sizes which are used in the detection stage to increase the performance of the SSD head. During our experiments, we have prepared a new dataset for buildings inside Riyadh City, Saudi Arabia that consists of 3878 buildings. We have compared our proposed approach vs other approaches in the literature. The proposed system has achieved the highest average precision, recall, F1-score, and IOU performance. Our proposed method has achieved a fast average prediction time with the lowest variance for our testing set. Our experimental results are very promising and can be generalized to other object detection tasks in low contrast images.

Visual vs internal attention mechanisms in deep neural networks for image classification and object detection

The so-called “attention mechanisms” in Deep Neural Networks (DNNs) denote an automatic adaptation of DNNs to capture representative features given a specific classification task and related data. Such attention mechanisms perform both globally by reinforcing feature channels and locally by stressing features in each feature map. Channel and feature importance are learnt in the global end-to-end DNNs training process. In this paper, we present a study and propose a method with a different approach, adding supplementary visual data next to training images. We use human visual attention maps obtained independently with psycho-visual experiments, both in task-driven or in free viewing conditions, or powerful models for prediction of visual attention maps. We add visual attention maps as new data alongside images, thus introducing human visual attention into the DNNs training and compare it with both global and local automatic attention mechanisms. Experimental results show that known attention mechanisms in DNNs work pretty much as human visual attention, but still the proposed approach allows a faster convergence and better performance in image classification tasks.

Multiscale face recognition in cluttered backgrounds based on visual attention

The human vision system can efficiently recognize multiscale objects in cluttered backgrounds. The scheme can be achieved with a visual attention mechanism by concentrating visual resources to the saliency area while ignoring other task-irrelevant areas. However, in the computer vision community, when recognizing multiscale faces in cluttered backgrounds, object detection modules are necessary to locate face regions and reduce the influence of complex backgrounds on the recognition model, which inevitably increases the computational complexity. Motivated by the human vision system, this study proposes the attention developmental network to recognize multiscale faces without using face detectors. A top-down attention mechanism is used to teach the network to focus on the face areas and ignore the backgrounds. An attention-based synapse maintenance mechanism is also introduced to further suppress the background pixels and improve the accuracy of face recognition. Comparative experiments show that our method can attain at least 13% of accuracy improvement over bionic neural networks and ResNet-based recognition networks on the same model scale with less training epochs.

Consumer Neuroscience: Attentional Preferences for Wine Labeling Reflected in the Posterior Contralateral Negativity.

During the decision-making process, consumers notice, inspect, and visually scan different products. External characteristics of a product, such as design, packaging, label, and logo, have been shown to strongly influence how customers perceive, assess, and select a product. Marketers have put a lot of effort into determining the factors that trigger consumers’ visual attention toward products, using traditional research methods, self-reports, or observations. The use of neuroscientific tools to study consumer behavior may improve our understanding of how external characteristics influence consumers’ visual attention. Consumer neuroscience research shows that preferences for a product may already be reflected in brain activity before customers make a final decision. Using electroencephalography (EEG), we investigated whether the design of different wine labeling influences individual preferences, reflected in the neural activity related to visual attention. More specifically, we examined whether the posterior contralateral negativity (PCN) can be used to assess and predict consumers’ preferences for a specific product based on its external characteristics. The PCN is commonly used to estimate attentional selection by focusing on stimulus-side dependent EEG lateralization above parieto-occipital areas. We computed the PCN to assess whether a certain wine label caught participants’ visual attention and additionally by comparing the PCN with behavioral data (wine preferences and reaction times) to determine whether early effects of visual attention could predict participants’ final preferences for a specific label. Our findings indicate that the PCN provides relevant information on visual attention mechanisms for external characteristics, as the view of the four labels modulated PCN amplitude. We hope this study can help researchers and practitioners in examining the effects of external product characteristics on consumer choice by estimating the changes in the EEG that are related to visual attention.

Temporal–spatial feature compensation combines with multi-feature discriminators for video super-resolution perceptual quality improvement

Generative adversarial network (GAN) for super-resolution (SR) has attracted enormous interest in recent years. It has been widely used to solve the single-image super-resolution (SISR) task and made superior performance. However, GAN is rarely used for video super-resolution (VSR). VSR aims to improve video resolution by exploiting the temporal continuity and spatial similarity of video sequence frames. We design a GAN with multi-feature discriminators and combine it with optical flow estimation compensation to construct an end-to-end VSR framework OFC-MFGAN. Optical flow estimation compensation makes use of temporal continuity and spatial similarity features of adjacent frames to provide rich detailed information for GAN. Multi-feature discriminators based on visual attention mechanism include the pixel discriminator, edge discriminator, gray discriminator, and color discriminator. GAN with multi-feature discriminators makes the data distribution and visually sensitive features (edge, texture, and color) of SR frames similar to high-resolution frames. OFC-MFGAN effectively integrates the time, space, and visually sensitive features of videos. Extensive experiments on public video datasets and surveillance videos show the effectiveness and robustness of the proposed method. Compared with several state-of-the-art VSR methods and SISR methods, the proposed method can not only recover prominent edges, clear textures, and realistic colors but also make a pleasant visual feeling and competitive perceptual index.

Fusion of the YOLOv4 network model and visual attention mechanism to detect low-quality young apples in a complex environment

Fusion of the YOLOv4 network model and visual attention mechanism to detect low-quality young apples in a complex environment

Client-Server Approach for Managing Visual Attention, Integrated in a Cognitive Architecture for a Social Robot

This paper proposes a novel system for managing visual attention in social robots. This system is based on a client/server approach that allows integration with a cognitive architecture controlling the robot. The core of this architecture is a distributed knowledge graph, in which the perceptual needs are expressed by the presence of arcs to stimuli that need to be perceived. The attention server sends motion commands to the actuators of the robot, while the attention clients send requests through the common knowledge representation. The common knowledge graph is shared by all levels of the architecture. This system has been implemented on ROS and tested on a social robot to verify the validity of the approach and was used to solve the tests proposed in RoboCup @ Home and SciROc robotic competitions. The tests have been used to quantitatively compare the proposal to traditional visual attention mechanisms.

Bioinspired Visual-Integrated Model for Multilabel Classification of Textile Defect Images

In modern textile industrial processes, textile defect recognition and classification are of vital importance for textile quality control. Recently, as the critical machine-learning method, the deep convolutional neural network (CNN) has shown superior performance in the single-label classification of textile defect images. However, accurately recognizing and classifying multilabel textile defect images are still an unsolved issue due to the complexity of intersected defects, as well as difficulty in distinguishing small-size defects and concerning the correlations amongst labels. To address these challenges, we propose a bioinspired visual-integrated model for multilabel classification of textile defects called BIVI-ML. Three bioinspired visual mechanisms (the visual gain mechanism, the visual attention mechanism, and the visual memory mechanism) are proposed and built within the BIVI-ML to: 1) enhance the resolution and feature discrimination; 2) attend to the textile defect; and 3) associate relevant labels. To evaluate the proposed method, a unique multilabel textile defect database is built as the benchmark for the multilabel classification of textile defect images. Compared with both the traditional and the state-of-the-art deep-learning classification methods, the proposed BIVI-ML achieves the best performance in both single-label and multilabel image classification.

Deep progressive attention for person re-identification

Person re-identification (Re-ID) aims to retrieve specific individuals across non-overlapping camera views. In recent years, attention-based models contribute to many computer vision tasks due to their great ability for learning discriminative features. We propose the deep progressive attention (DPA) in a more natural manner for person Re-ID. Similar to human visual attention mechanism, the proposed DPA progressively selects the most discriminative parts of a specific individual and formulates feature representation for comparison purpose. Concretely, on the one hand, the proposed DPA uses a long-term reward to optimize the discriminative feature selection. On the other hand, a deep convolutional architecture is integrated into a recurrent model for feature representation learning. Extensive experiments on three person Re-ID benchmarks Market-1501, DukeMTMC-reID, and CUHK03-NP demonstrate the proposed DPA is on par with the state-of-the-art. Moreover, the experiments on partial person Re-ID datasets indicate the proposed DPA is competitive with the specially designed partial person Re-ID methods.

Facial expression recognition with grid-wise attention and visual transformer

Facial Expression Recognition (FER) has achieved remarkable progress as a result of using Convolutional Neural Networks (CNN). Relying on the spatial locality, convolutional filters in CNN, however, fail to learn long-range inductive biases between different facial regions in most neural layers. As such, the performance of a CNN-based model for FER is still limited. To address this problem, this paper introduces a novel FER framework with two attention mechanisms for CNN-based models, and these two attention mechanisms are used for the low-level feature learning the high-level semantic representation, respectively. In particular, in the low-level feature learning, a grid-wise attention mechanism is proposed to capture the dependencies of different regions from a facial expression image such that the parameter update of convolutional filters in low-level feature learning is regularized. In the high-level semantic representation, a visual transformer attention mechanism uses a sequence of visual semantic tokens (generated from pyramid features of high convolutional layer blocks) to learn the global representation. Extensive experiments have been conducted on three public facial expression datasets, CK+, FER+, and RAF-DB. The results show that our FER-VT has achieved state-of-the-art performance on these datasets, especially with a 100% accuracy on CK + datasets without any extra training data.

Ship Object Detection of Remote Sensing Image Based on Visual Attention

At present, reliable and precise ship detection in high-resolution optical remote sensing images affected by wave clutter, thin clouds, and islands under complex sea conditions is still challenging. At the same time, object detection algorithms in satellite remote sensing images are challenged by color, aspect ratio, complex background, and angle variability. Even the results obtained based on the latest convolutional neural network (CNN) method are not satisfactory. In order to obtain more accurate ship detection results, this paper proposes a remote sensing image ship object detection method based on a brainlike visual attention mechanism. We refer to the robust expression mode of the human brain, design a vector field filter with active rotation capability, and explicitly encode the direction information of the remote sensing object in the neural network. The progressive enhancement learning model guided by the visual attention mechanism is used to dynamically solve the problem, and the object can be discovered and detected through time–space information. To verify the effectiveness of the proposed method, a remote sensing ship object detection data set is established, and the proposed method is compared with other state-of-the-art methods on the established data set. Experiments show that the object detection accuracy of this method and the ability to capture image details have been improved. Compared with other models, the average intersection rate of the joint is 80.12%, which shows a clear advantage. The proposed method is fast enough to meet the needs of ship detection in remote sensing images.

Leveraging colour-based pseudo-labels to supervise saliency detection in hyperspectral image datasets

Saliency detection mimics the natural visual attention mechanism that identifies an imagery region to be salient when it attracts visual attention more than the background. This image analysis task covers many important applications in several fields such as military science, ocean research, resources exploration, disaster and land-use monitoring tasks. Despite hundreds of models have been proposed for saliency detection in colour images, there is still a large room for improving saliency detection performances in hyperspectral imaging analysis. In the present study, an ensemble learning methodology for saliency detection in hyperspectral imagery datasets is presented. It enhances saliency assignments yielded through a robust colour-based technique with new saliency information extracted by taking advantage of the abundance of spectral information on multiple hyperspectral images. The experiments performed with the proposed methodology provide encouraging results, also compared to several competitors.

Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

Standard registration algorithms need to be independently applied to each surface to register, following careful pre-processing and hand-tuning. Recently, learning-based approaches have emerged that reduce the registration of new scans to running inference with a previously-trained model. The potential benefits are multifold: inference is typically orders of magnitude faster than solving a new instance of a difficult optimization problem, deep learning models can be made robust to noise and corruption, and the trained model may be re-used for other tasks, e.g. through transfer learning. In this paper, we cast the registration task as a surface-to-surface translation problem, and design a model to reliably capture the latent geometric information directly from raw 3D face scans. We introduce Shape-My-Face (SMF), a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model that we smoothly integrate with the mesh convolutions. Compared to the previous state-of-the-art learning algorithms for non-rigid registration of face scans, SMF only requires the raw data to be rigidly aligned (with scaling) with a pre-defined face template. Additionally, our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets. We extensively evaluate the quality of our registrations on diverse data. We demonstrate the robustness and generalizability of our model with in-the-wild face scans across different modalities, sensor types, and resolutions. Finally, we show that, by learning to register scans, SMF produces a hybrid linear and non-linear morphable model. Manipulation of the latent space of SMF allows for shape generation, and morphing applications such as expression transfer in-the-wild. We train SMF on a dataset of human faces comprising 9 large-scale databases on commodity hardware.

Accuracy vs. complexity: A trade-off in visual question answering models

Visual Question Answering (VQA) has emerged as a Visual Turing Test to validate the reasoning ability of AI agents. The pivot to existing VQA models is the joint embedding that is learned by combining the visual features from an image and the semantic features from a given question. Consequently, a large body of literature has focused on developing complex joint embedding strategies coupled with visual attention mechanisms to effectively capture the interplay between these two modalities. However, modelling the visual and semantic features in a high dimensional (joint embedding) space is computationally expensive, and more complex models often result in trivial improvements in the VQA accuracy. In this work, we systematically study the trade-off between the model complexity and the performance on the VQA task. VQA models have a diverse architecture comprising of pre-processing, feature extraction, multimodal fusion, attention and final classification stages. We specifically focus on the effect of “multi-modal fusion” in VQA models that is typically the most expensive step in a VQA pipeline. Our thorough experimental evaluation leads us to three proposals, one optimized for minimal complexity, one for balanced complexity-accuracy and the last one for state-of-the-art VQA performance.

Integrating Scene Semantic Knowledge into Image Captioning

Most existing image captioning methods use only the visual information of the image to guide the generation of captions, lack the guidance of effective scene semantic information, and the current visual attention mechanism cannot adjust the focus intensity on the image. In this article, we first propose an improved visual attention model. At each timestep, we calculated the focus intensity coefficient of the attention mechanism through the context information of the model, then automatically adjusted the focus intensity of the attention mechanism through the coefficient to extract more accurate visual information. In addition, we represented the scene semantic knowledge of the image through topic words related to the image scene, then added them to the language model. We used the attention mechanism to determine the visual information and scene semantic information that the model pays attention to at each timestep and combined them to enable the model to generate more accurate and scene-specific captions. Finally, we evaluated our model on Microsoft COCO (MSCOCO) and Flickr30k standard datasets. The experimental results show that our approach generates more accurate captions and outperforms many recent advanced models in various evaluation metrics.

Efficient residual attention network for single image super-resolution

The use of deep convolutional neural networks (CNNs) for image super-resolution (SR) from low-resolution (LR) input has achieved remarkable reconstruction performance with the utilization of residual structures and visual attention mechanisms. However, existing single image super-resolution (SISR) methods with deeper network architectures can encounter representational bottlenecks in CNN-based networks and neglect model efficiency in model statistical inference. To solve these issues, in this paper, we design a channel hourglass residual structure (CHRS) and explore an efficient channel attention (ECA) mechanism to extract more representative features and ease the computational burden. Specifically, our CHRS, consisting of several nested residual modules, is developed to learn more discriminative representations with fewer model parameters, and the ECA is presented to efficiently capture local cross-channel interaction by subtly applying 1D convolution. Finally, we propose an efficient residual attention network (ERAN), which not only fully learns more representative features but also pays special attention to network learning efficiency. Extensive experiments demonstrate that our ERAN achieves certain improvements in model performance and implementation efficiency compared to other previous state-of-the-art methods.

Application of printing defects detection based on visual saliency

Visual saliency means that when humans observe a certain area, humans automatically process the area of interest and selectively ignore the uninterested area. This local area is called the saliency area. In recent years, people have introduced image processing and pattern recognition detection methods based on visual saliency to simulate the biological visual attention mechanism, and achieved good results. This paper focuses on the process of printed product defect detection, analyzes the application of visual saliency in printed product detection in recent years, summarizes and compares the advantages and disadvantages of various printed product detection methods, and discusses future development trends.

Consensus emerges and biased competition wins: A commentary on Luck et al. (2021)

ABSTRACT As the debate about the automaticity of attentional capture has raged on over the years, knowledge of how selection functions has been accumulating in the literature. Luck et al. (2021) highlights emerging areas of consensus, including support for multiple mechanisms that influence attentional guidance. Here, I revisit the biased competition model of [Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18(1), 193–222. https://doi.org/10.1146/annurev.ne.18.030195.001205] to bring additional perspective. This aims to provide insight about open questions of “control” and to offer guidance in how to address other conceptual issues that hinder progress toward a better understanding of attentional functioning.

A Real-Time Apple Targets Detection Method for Picking Robot Based on Improved YOLOv5

The apple target recognition algorithm is one of the core technologies of the apple picking robot. However, most of the existing apple detection algorithms cannot distinguish between the apples that are occluded by tree branches and occluded by other apples. The apples, grasping end-effector and mechanical picking arm of the robot are very likely to be damaged if the algorithm is directly applied to the picking robot. Based on this practical problem, in order to automatically recognize the graspable and ungraspable apples in an apple tree image, a light-weight apple targets detection method was proposed for picking robot using improved YOLOv5s. Firstly, BottleneckCSP module was improved designed to BottleneckCSP-2 module which was used to replace the BottleneckCSP module in backbone architecture of original YOLOv5s network. Secondly, SE module, which belonged to the visual attention mechanism network, was inserted to the proposed improved backbone network. Thirdly, the bonding fusion mode of feature maps, which were inputs to the target detection layer of medium size in the original YOLOv5s network, were improved. Finally, the initial anchor box size of the original network was improved. The experimental results indicated that the graspable apples, which were unoccluded or only occluded by tree leaves, and the ungraspable apples, which were occluded by tree branches or occluded by other fruits, could be identified effectively using the proposed improved network model in this study. Specifically, the recognition recall, precision, mAP and F1 were 91.48%, 83.83%, 86.75% and 87.49%, respectively. The average recognition time was 0.015 s per image. Contrasted with original YOLOv5s, YOLOv3, YOLOv4 and EfficientDet-D0 model, the mAP of the proposed improved YOLOv5s model increased by 5.05%, 14.95%, 4.74% and 6.75% respectively, the size of the model compressed by 9.29%, 94.6%, 94.8% and 15.3% respectively. The average recognition speeds per image of the proposed improved YOLOv5s model were 2.53, 1.13 and 3.53 times of EfficientDet-D0, YOLOv4 and YOLOv3 and model, respectively. The proposed method can provide technical support for the real-time accurate detection of multiple fruit targets for the apple picking robot.

Real-Time Monocular Obstacle Detection Based on Horizon Line and Saliency Estimation for Unmanned Surface Vehicles

Recently, real-time obstacle detection by monocular vision exhibits a promising prospect in enhancing the safety of unmanned surface vehicles (USVs). Since the obstacles that may threaten USVs generally appear below the water edge, most existing methods first detect the horizon line and then search for obstacles below the estimated horizon line. However, these methods detect horizon line only using edge or line features, which are susceptible to interference edges from clouds, waves, and land, eventually resulting in poor obstacle detection. To avoid being affected by interference edges, in this paper, we propose a novel horizon line detection method based on semantic segmentation. The method assumes a Gaussian mixture model (GMM) with spatial smoothness constraints to fit the semantic structure of marine images and simultaneously generate a water segmentation mask. The horizon line is estimated from the water boundary points via straight line fitting. Further, inspired by human visual attention mechanisms, an efficient saliency detection method based on background prior and contrast prior is presented to detect obstacles below the estimated horizon line. To reduce false positives caused by sun glitter, waves and foam, the continuity of the adjacent frames is employed to filter the detected obstacles. An extensive evaluation was conducted on a large marine image dataset collected by our ‘Jinghai VIII’ USV. The experimental results show that the proposed method significantly outperformed the recent state-of-the-art marine obstacle method by 22.07% in terms of F-score while running over 24 fps on an NVIDIA GTX1080Ti GPU.