Visual Attention Mechanism Research Articles

Identifying Novel Coronavirus Pneumonia With CT Images: A Deep Learning Approach With Detail Upsampling and Attention Guidance

To construct a deep learning-based target detection method to help radiologists perform rapid diagnosis of lesions in the CT images of patients with novel coronavirus pneumonia (NCP) by restoring detailed information and mining local information. We present a deep learning approach that integrates detail upsampling and attention guidance. A linear upsampling algorithm based on bicubic interpolation algorithm was adopted to improve the restoration of detailed information within feature maps during the upsampling phase. Additionally, a visual attention mechanism based on vertical and horizontal spatial dimensions embedded in the feature extraction module to enhance the capability of the object detection algorithm to represent key information related to NCP lesions. Experimental results on the NCP dataset showed that the detection method based on the detail upsampling algorithm improved the recall rate by 1.07% compared with the baseline model, with the AP50 reaching 85.14%. After embedding the attention mechanism in the feature extraction module, 86.13% AP50, 73.92% recall, and 90.37% accuracy were achieved, which were better than those of the popular object detection models. The feature information mining of CT images based on deep learning can further improve the lesion detection ability. The proposed approach helps radiologists rapidly identify NCP lesions on CT images and provides an important clinical basis for early intervention and high-intensity monitoring of NCP patients.

Research on image classification leveraging deep convolutional neural networks and visual cognition

The field of image classification has experienced remarkable improvements with the advent of deep learning techniques, especially Deep Convolutional Neural Networks. The present study provides an extensive exploration of the junction where image classification based on Deep Convolutional Neural Networks meets human visual cognition. Utilizing the inherent ability of these networks to automatically learn hierarchical features from raw pixel data, this research examines their potential in classifying images from diverse complex datasets, emphasizing predominantly on the extensively utilized ImageNet dataset. The initial aspect of this study involves training and evaluating models based on Deep Convolutional Neural Networks on the ImageNet dataset, which comprises millions of labeled images spanning across thousands of categories. Well-established network architectures such as AlexNet, VGGNet, GoogLeNet, and ResNet are employed, and their performance in the challenging task of image classification is assessed. Rigorous experiments highlight the strengths and weaknesses of each model while emphasizing the prospects of transfer learning and fine-tuning. Following this, the interpretability of Deep Convolutional Neural Networks is explored by using visualization techniques to comprehend the learned feature representations. By visualizing activation maps and class-specific saliency maps, valuable insights are gained into the regions of interest that guide the decision-making of these models. Moreover, the correlation between the features extracted by these models and human visual attention mechanisms is examined to shed light on the focus of attention of the models. The study also addresses the difficulties that adversarial attacks, data bias, and generalization capabilities present to Deep Convolutional Neural Networks. Strategies to enhance the robustness and adaptability of the models across various domains are examined, linking these observations to human cognitive behavior.

The guiding significance of consumer visual perception on the packaging design of cultural and creative products

Abstract The packaging of cultural and creative products plays a crucial role in enhancing their competitiveness, with exquisite packaging significantly contributing to meeting the personalized needs of consumers and stimulating their desire to purchase. This paper examines the visual perception factors involved in the packaging design of cultural and creative products using the visual attention mechanism and categorizes the design factors according to demand using the Kano model. The hierarchical analysis method is used to determine the consumer demand weight of each design factor. The packaging solutions for cultural and creative products are finalized, and the TOPSIS method is employed to assess each solution. The hierarchical analysis model achieves consistency ratios of 0.076, 0.016, 0.003, and 0.004 for the target layer Z and the criterion layers M, O, and A, respectively, all of which are less than 0.1, thereby passing the consistency test. Among the three designed packaging schemes for cultural and creative products, scheme 1 has the largest S value, which is 0.899. Therefore, scheme 1 is the best design scheme. The existing sample scheme has a S value of 0.013, which is smaller than that of the other three design schemes. The integrated Kano-AHP-TOPSIS packaging design path’s design schemes for cultural and creative products are more likely to meet consumers’ needs, demonstrating the path’s practical feasibility and the importance of consumer visual perception in shaping packaging design for these products.

A collective perception model for neighbor selection in groups based on visual attention mechanisms

The mechanisms by which individuals visually perceive and select others within a group to interact with are of fundamental importance to various collective motion behaviors. However, they remain poorly understood theoretically, partially due to the challenges of modeling neurological perception systems. Here, the classic zoom-lens visual attention model is introduced into collective perception, leading to an attention-based model with only a single parameter. Numerical experiments reveal that the proposed model offers a unified mechanism for collective perception, as reflected in three aspects. First, this model describes both conspecific (e.g. flocking) and heterospecific (e.g. predator-prey) collective behaviors. Second, this model unifies the well-known topological and visibility models in the context of neighbor selection in conspecific groups. Third, in the context of prey selection in heterospecific groups, the model can simultaneously replicate the well-known confusion and oddity effects. These findings demonstrate the fundamental role of visual attention underlying a diverse array of collective motion behaviors.

Biological visual attention convolutional neural network for aircraft skin defect detection

Inspired by the biological visual attention mechanism, we propose a visual attention convolutional neural network to solve the problem of identifying aircraft skin, but showing that the aircraft s kin image collected by the camera is rarely illuminated, and the contrast between defects and background is low and difficult to identify. Firstly, the U-Net with the encoder-decoder structure is used to initially segment the skin image that are provided by Airlines. Then, the residual block is introduced into the U-Net to enhance the propagation ability of the features and extract more defect detail feature information. Finally, the visual attention mechanism is used to increase the weight of the defect area to reduce the influence of uneven illumination on the model. The experimental results show that the proposed model has better defect segmentation effects in both visual effects and objective evaluation indexes. Our algorithm not only enhances the accuracy and effectiveness of skin defect identification but also provides management personnel with a more relevant tool, promoting the intelligence and information of air maintenance. This contributes to excellent management support for aviation companies and related industries.

Spatial memory-augmented visual navigation based on hierarchical deep reinforcement learning in unknown environments

Visual navigation in unknown environments poses significant challenges due to the presence of many obstacles and low-texture scenes. These factors may cause frequent collisions and tracking failure of feature-based visual Simultaneous Localization and Mapping (vSLAM). To avoid these issues, this paper proposes a spatial memory-augmented visual navigation system that combines a vSLAM module, a conventional global planner module, and a Hierarchical Reinforcement Learning (HRL)-based local planner module. Firstly, a real-time vSLAM named Salient-SLAM is proposed to augment the performance of visual navigation. Salient-SLAM creates a navigation mapping thread by combining a saliency prediction model to build a navigation map that categorizes environmental regions as occupied, explored, or noticeable. Spatial memory that contains spatial abstraction and saliency information of the environment can be further formed by encoding navigation maps, which helps the agent determine an optimal path towards its destination. An open-sourced saliency dataset is proposed to train the saliency prediction model by mimicking the visual attention mechanism. Secondly, a HRL method is proposed to automatically decompose local planning into a high-level policy selector and several low-level policies, where the latter produces actions to interact with the environment. We maximize entropy and minimize option correlation in learning low-level policies, aiming at acquiring diverse and independent behaviors. The simulation results show that the proposed HRL method outperforms competitive baselines by 6.29–10.85 % on Success Rate (SR) and 3.87–11.1 % on Success weighted by Path Length (SPL) metrics. By incorporating the spatial memory, SR, and SPL metrics can be augmented by an average of 9.85 % and 10.89 %, respectively.

A deep learning-assisted visual attention mechanism for anomaly detection in videos

A deep learning-assisted visual attention mechanism for anomaly detection in videos

Multiobjective visual evolutionary neural network and related convolutional neural network optimization

Biological visual systems intrinsically include a variety of direction-selective neurons capable of detecting changes in visual motion. Some of the neurons have been used to construct computational models for problem-specific engineering applications. However, it remains unclear how these neurons' visual response mechanisms can serve an interdisciplinary topic - visual evolutionary neural networks. In this study, we draw inspiration from swarm evolution and the fly's visual response and attention mechanisms to develop a fast multiobjective visual evolutionary neural network for solving multiobjective optimization problems, specifically focusing on convolutional neural network optimization. Our approach involves designing a multi-channel input visual neural network that outputs global and local learning rates for state transition, proposing a new population-like state update strategy to move current states toward the Pareto front as quickly as possible, and exploiting a point-to-plane distance model to control the size of the external archive set. The computational complexity of the evolutionary neural network mainly depends on the size of the external archive set and the number of fitness evaluations. Comparative experiments demonstrate that our network is an extremely competitive optimizer, effectively solving multiple kinds of benchmark examples. Especially, when optimizing the 8,248 parameters of a conventional convolutional neural network, our approach quickly achieves large-scale optimization.

Fine-grained Species Recognition with Privileged Pooling: Better Sample Efficiency Through Supervised Attention.

We propose a scheme for supervised image classification that uses privileged information, in the form of keypoint annotations for the training data, to learn strong models from small and/or biased training sets. Our main motivation is the recognition of animal species for ecological applications such as biodiversity modelling, which is challenging because of long-tailed species distributions due to rare species, and strong dataset biases such as repetitive scene background in camera traps. To counteract these challenges, we propose a visual attention mechanism that is supervised via keypoint annotations that highlight important object parts. This privileged information, implemented as a novel privileged pooling operation, is only required during training and helps the model to focus on regions that are discriminative. In experiments with three different animal species datasets, we show that deep networks with privileged pooling can use small training sets more efficiently and generalize better.

CVANet: Cascaded visual attention network for single image super-resolution

Deep convolutional neural networks (DCNNs) have exhibited excellent feature extraction and detail reconstruction capabilities for single image super-resolution (SISR). Nevertheless, most previous DCNN-based methods do not fully utilize the complementary strengths between feature maps, channels, and pixels. Therefore, it hinders the ability of DCNNs to represent abundant features. To tackle the aforementioned issues, we present a Cascaded Visual Attention Network for SISR called CVANet, which simulates the visual attention mechanism of the human eyes to focus on the reconstruction process of details. Specifically, we first designed a trainable feature attention module (FAM) for feature-level attention learning. Afterward, we introduce a channel attention module (CAM) to reinforce feature maps under channel-level attention learning. Meanwhile, we propose a pixel attention module (PAM) that adaptively selects representative features from the previous layers, which are utilized to generate a high-resolution image. Satisfactory, our CVANet can effectively improve the resolution of images by exploring the feature representation capabilities of different modules and the visual perception properties of the human eyes. Extensive experiments with different methods on four benchmarks demonstrate that our CVANet outperforms the state-of-the-art (SOTA) methods in subjective visual perception, PSNR, and SSIM.The code will be made available https://github.com/WilyZhao8/CVANet.

Frescoes restoration via virtual-real fusion: Method and practice

In the era of artificial intelligence, image-based virtual restoration of cultural relics is one of the methods used in the restoration of cultural relics. As the most informative representative of cultural heritage in the study of historical materials, murals occupy a significant position in archaeology and ancient culture. Currently, most of the existing virtual restoration of murals is limited to the restoration of image information for local damage. The scenes of murals have large spatial scales and complex semantic contents. In order to enhance the semantic relevance of the virtual restoration of murals and the immersion of a visual perception, this paper proposes a kind of mural virtual-real fusion restoration display method based on visual attention mechanism. Based on the case study on the immersive fusion restoration of Dunhuang grotto murals, the feasibility of regional mural image restoration and real space scene fusion restoration is verified. A new paradigm of mural heritage protection in the context of virtual-real fusion is realized.

A Human-Like Trajectory Planning Method on a Curve Based on the Driver Preview Mechanism

With the development of intelligent vehicle technology, many studies have been focused on developing human-like trajectory planning methods for automated driving systems. Although data-driven methods are widely used for human driver behavior learning, there have been fewer studies on realizing human-like trajectory planning by using the generation mechanism of driving behavior, especially under curve conditions, where the lane centerline has been denoted as a reference trajectory. In this paper, thirty-two skilled drivers were recruited to collect data under different curve conditions on a self-designed driver-in-the-loop system. The collected data are processed by dynamic time warping, trajectories with different lengths are warped and the abnormal data are removed. Based on the warped data, common characteristics and differences between left and right turning trajectories are compared and explored from the perspectives of drivers’ demand for turning performance and their visual attention mechanism. Then, by introducing the driver preview mechanism, two features with a strong ability to represent the generation mechanism of the driver’s curve driving behavior are introduced. Finally, the preview-based human-like trajectory planning model (PHTPM) is proposed, and it is verified and analyzed by comparative tests and generalizability tests. The results show that the introduction of the driver preview mechanism enables PHTPM to match the characteristics of skilled drivers accurately on left turnings and outperform them on right turnings.

Objects guide human gaze behavior in dynamic real-world scenes.

The complexity of natural scenes makes it challenging to experimentally study the mechanisms behind human gaze behavior when viewing dynamic environments. Historically, eye movements were believed to be driven primarily by space-based attention towards locations with salient features. Increasing evidence suggests, however, that visual attention does not select locations with high saliency but operates on attentional units given by the objects in the scene. We present a new computational framework to investigate the importance of objects for attentional guidance. This framework is designed to simulate realistic scanpaths for dynamic real-world scenes, including saccade timing and smooth pursuit behavior. Individual model components are based on psychophysically uncovered mechanisms of visual attention and saccadic decision-making. All mechanisms are implemented in a modular fashion with a small number of well-interpretable parameters. To systematically analyze the importance of objects in guiding gaze behavior, we implemented five different models within this framework: two purely spatial models, where one is based on low-level saliency and one on high-level saliency, two object-based models, with one incorporating low-level saliency for each object and the other one not using any saliency information, and a mixed model with object-based attention and selection but space-based inhibition of return. We optimized each model's parameters to reproduce the saccade amplitude and fixation duration distributions of human scanpaths using evolutionary algorithms. We compared model performance with respect to spatial and temporal fixation behavior, including the proportion of fixations exploring the background, as well as detecting, inspecting, and returning to objects. A model with object-based attention and inhibition, which uses saliency information to prioritize between objects for saccadic selection, leads to scanpath statistics with the highest similarity to the human data. This demonstrates that scanpath models benefit from object-based attention and selection, suggesting that object-level attentional units play an important role in guiding attentional processing.

Crop disease recognition based on the YOLOv5 and transfer learning

Crop disease detection is an important factor in agricultural production. Traditional object detection methods can't effectively screen key features, resulting in weak crop disease control in many countries. In recent years, several convolutional neural networks for object detection have been proposed, which makes it possible to apply computer vision to crop disease identification through deep learning. YOLOv5 is an advanced object detection network, which can extract key features and use human visual attention mechanism as a reference. This paper would evaluate the performance of four pre-trained models of YOLOv5 in object detection of crop diseases. And transfer learning was used to train the corresponding dataset. The experiment results have showed that the F1 values of the four models all reached above 0.93, and the Yolov5x got the best result, which achieved 0.963. Furthermore, the detection accuracy of the four models has reached more than 98%. This shows that the YOLOv5 series network models have great application prospects in the identification of crop diseases. In the near future, the object detection model can be applied to various mobile devices, even unmanned aerial vehicles, which would play a significant role in crop disease prevention.

SAVAnet: Surgical Action-Driven Visual Attention Network for Autonomous Endoscope Control

An endoscope holder must understand the detailed surgical actions and the surgeons’ visual attention to keep important targets in the field of endoscopic view during operations. From an intensive analysis of the surgeons’ attention mechanism, we included that surgical actions, like cutting, suturing, etc., play an important role in determining the positions and weights of visual attention points during a dynamic surgical scene. To perform this process, this work proposes a Surgical Action-driven Visual Attention network (SAVAnet) and applies the network in autonomous endoscope control. Four scenarios are constructed in the da Vinci V-rep simulator: pick&place and needle exercise in a general laparoscopic training environment, needle driving with and without obstacle removal in an abdominal cavity, to create datasets for network training. The results show that the network has an outstanding performance in surgical action prediction with a high average accuracy of over 91%. Additionally, with surgical action guidance, the attention point prediction has higher accuracy and accords with surgeons’ visual attention. Finally, the acquired attention points are utilized to execute visual servoing in simulation. The results verify that the SAVAnet is feasible for autonomous endoscope control in real-time and lays a theoretical foundation for future sim-to-real execution. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper was motivated by the problem of endowing an endoscope with surgeons’ visual attention mechanism, which is affected by surgical actions, for autonomous endoscope control. An eye-tracking device has been utilized to detect surgeon’s visual attention in real-time and then control the endoscope to follow what the surgeon is looking at. However, this approach is susceptible to the surgical environment. Besides, many instrument detection and segmentation algorithms are developed for automatic surgical instrument tracking. However, surgeons’ visual attention does not always focus on the instruments during operations. In this work, we propose a novel SAVAnet to determine visual attention based on surgical actions. We prove from many qualitative and quantitative experiments that surgical actions play a significant role in determining visual attention. The designed SAVAnet can predict surgical actions correctly and then effectively guide the choice of visual attention. Finally, the simulation results show that the SAVAnet can endow endoscope with surgeons’ visual attention to perform self-control in real time. In future research, we will train the SAVAnet using real datasets and conduct more physical experiments on real surgical robots.

Spatio-Temporal Feature Pyramid Interactive Attention Network for Egocentric Gaze Prediction

Over the past few years, gaze prediction technology has rapidly developed in human visual attention mechanism research. However, there is still a large gap between the prediction models and the human visual system (HVS). This work makes four contributions to video egocentric gaze prediction research. First, we introduce a new benchmark named ECVG (egocentric video gaze), which consists of 3K high-quality, elaborately selected video sequences spanning a large range of scenes. Existing eye-tracking datasets limit human activities. In contrast, ECVG achieves the collection of eye movement data under real human activity. Second, we establish a visual attention-selection and gaze-tracking model, which summarizes the human visual attention process. Third, we propose a novel feature pyramid interactive attention 3D network (FPIANet) for egocentric gaze prediction, which directly captures the long-term relationship between spatiotemporal features at different time steps and achieves feature nonlocal interactions across temporal, spatial, and scales. In addition, we propose a multiscale interactive attention (MSIA) module, which first explicitly integrates the human top-down and bottom-up visual attention mechanisms to transform any spatiotemporal feature into another feature of the same size but with richer contexts. Furthermore, we propose a new gaze transfer path consistency evaluation and thoroughly examine the performance of our model on two large datasets (ECVG and LEDOV). The experimental results show that our model outperforms the existing state-of-the-art methods.

Object-oriented building extraction based on visual attention mechanism.

Buildings, which play an important role in the daily lives of humans, are a significant indicator of urban development. Currently, automatic building extraction from high-resolution remote sensing images (RSI) has become an important means in urban studies, such as urban sprawl, urban planning, urban heat island effect, population estimation and damage evaluation. In this article, we propose a building extraction method that combines bottom-up RSI low-level feature extraction with top-down guidance from prior knowledge. In high-resolution RSI, buildings usually have high intensity, strong edges and clear textures. To generate primary features, we propose a feature space transform method that consider building. We propose an object oriented method for high-resolution RSI shadow extraction. Our method achieves user accuracy and producer accuracy above 95% for the extraction results of the experimental images. The overall accuracy is above 97%, and the quantity error is below 1%. Compared with the traditional method, our method has better performance on all the indicators, and the experiments prove the effectiveness of the method.

Different Mechanisms for Visual Attention at the Hand-movement Goal and Endogenous Visual Attention.

Visual perception is closely related to body movements and action, and it is known that processing visual stimuli is facilitated at the hand or at the hand-movement goal. Such facilitation suggests that there may be an attentional process associated with the hands or hand movements. To investigate the underlying mechanisms of visual attention at a hand-movement goal, we conducted two experiments to examine whether attention at the hand-movement goal is a process independent from endogenous attention. Endogenous attention is attention that is intentionally focused on a location, feature, or object. We controlled the hand-movement goal and endogenous attention separately to investigate the spatial profiles of the two types of attention. A visual target was presented either at the goal of hand movement (same condition) or at its opposite side (opposite condition) while steady-state visual-evoked potential (SSVEP) was used to estimate the spatial distributions of the facilitation effect from the 2 types of attention around the hand-movement goal and around the visual target through EEG. We estimated the spatial profile of attentional modulation for the hand-movement goal by taking the difference in SSVEP amplitude between conditions with and without hand movement, thereby obtaining the effect of visual endogenous attention alone. The results showed a peak at the hand-movement goal, independent of the location of the visual target where participants intentionally focused their attention (endogenous attention). We also found differences in the spatial extent of attentional modulation. Spatial tuning was narrow around the hand-movement goal (i.e., attentional facilitation only at the goal location) but was broadly tuned around the focus of endogenous attention (i.e., attentional facilitation spreading over adjacent stimulus locations), which was obtained from the condition without hand movement. These results suggest the existence of two separate mechanisms, one underlying the attention at the hand-movement goal and another underlying endogenous attention.

Foreground Fusion-Based Liquefied Natural Gas Leak Detection Framework From Surveillance Thermal Imaging

A leak detection and repair survey (LDAR) is essential to ensure a reliable and safe liquefied natural gas (LNG) supply. Modern LDAR systems deploy numerous fixed thermal imaging devices to automatically monitor the risk of potential leaks empowered by computational intelligence frameworks. Existing frameworks employ either background subtraction-based (BGS-based) or deep neural network-based (DNN-based) frameworks for LNG leak detection from thermal images. However, the BGS-based frameworks feature high sensitivity to perceive LNG emissions with low precision. On the contrary, the DNN-based frameworks can precisely classify the LNG leak after training while the sensitivity is low. Additionally, conventional DNN-based frameworks are difficult in modeling non-rigid objects such as LNG gas due to limited perceptive fields. Therefore, this study proposes a hybrid framework, namely foreground fusion-based gas detection (FFBGD), combining the advantages of BGS-based and DNN-based detectors for improved detection robustness through newly introduced concept of information fusion to LNG industries. Specifically, a foreground fusion network (FFN) is designed to fuse information of original thermal and foreground images after BGS based on the visual attention mechanism. Meanwhile, several advanced modules, i.e. deformable convolution, feature pyramid network, and cascade region-of-interest (ROI) head are adopted to enhance leak detection by offering better perceptive fields. Extensive experiments are carried out in this study to demonstrate the significant improvement brought by the proposed FFBGD over leak detection accuracy and robustness. Hence, the proposed solution can be deployed in energy facilities and enable reliable visual surveillance of LNG leaks.

Multi-scale fusion visual attention network for facial micro-expression recognition.

Micro-expressions are facial muscle movements that hide genuine emotions. In response to the challenge of micro-expression low-intensity, recent studies have attempted to locate localized areas of facial muscle movement. However, this ignores the feature redundancy caused by the inaccurate locating of the regions of interest. This paper proposes a novel multi-scale fusion visual attention network (MFVAN), which learns multi-scale local attention weights to mask regions of redundancy features. Specifically, this model extracts the multi-scale features of the apex frame in the micro-expression video clips by convolutional neural networks. The attention mechanism focuses on the weights of local region features in the multi-scale feature maps. Then, we mask operate redundancy regions in multi-scale features and fuse local features with high attention weights for micro-expression recognition. The self-supervision and transfer learning reduce the influence of individual identity attributes and increase the robustness of multi-scale feature maps. Finally, the multi-scale classification loss, mask loss, and removing individual identity attributes loss joint to optimize the model. The proposed MFVAN method is evaluated on SMIC, CASME II, SAMM, and 3DB-Combined datasets that achieve state-of-the-art performance. The experimental results show that focusing on local at the multi-scale contributes to micro-expression recognition. This paper proposed MFVAN model is the first to combine image generation with visual attention mechanisms to solve the combination challenge problem of individual identity attribute interference and low-intensity facial muscle movements. Meanwhile, the MFVAN model reveal the impact of individual attributes on the localization of local ROIs. The experimental results show that a multi-scale fusion visual attention network contributes to micro-expression recognition.