Human Visual Attention Mechanism Research Articles

Research on image classification leveraging deep convolutional neural networks and visual cognition

The field of image classification has experienced remarkable improvements with the advent of deep learning techniques, especially Deep Convolutional Neural Networks. The present study provides an extensive exploration of the junction where image classification based on Deep Convolutional Neural Networks meets human visual cognition. Utilizing the inherent ability of these networks to automatically learn hierarchical features from raw pixel data, this research examines their potential in classifying images from diverse complex datasets, emphasizing predominantly on the extensively utilized ImageNet dataset. The initial aspect of this study involves training and evaluating models based on Deep Convolutional Neural Networks on the ImageNet dataset, which comprises millions of labeled images spanning across thousands of categories. Well-established network architectures such as AlexNet, VGGNet, GoogLeNet, and ResNet are employed, and their performance in the challenging task of image classification is assessed. Rigorous experiments highlight the strengths and weaknesses of each model while emphasizing the prospects of transfer learning and fine-tuning. Following this, the interpretability of Deep Convolutional Neural Networks is explored by using visualization techniques to comprehend the learned feature representations. By visualizing activation maps and class-specific saliency maps, valuable insights are gained into the regions of interest that guide the decision-making of these models. Moreover, the correlation between the features extracted by these models and human visual attention mechanisms is examined to shed light on the focus of attention of the models. The study also addresses the difficulties that adversarial attacks, data bias, and generalization capabilities present to Deep Convolutional Neural Networks. Strategies to enhance the robustness and adaptability of the models across various domains are examined, linking these observations to human cognitive behavior.

Crop disease recognition based on the YOLOv5 and transfer learning

Crop disease detection is an important factor in agricultural production. Traditional object detection methods can't effectively screen key features, resulting in weak crop disease control in many countries. In recent years, several convolutional neural networks for object detection have been proposed, which makes it possible to apply computer vision to crop disease identification through deep learning. YOLOv5 is an advanced object detection network, which can extract key features and use human visual attention mechanism as a reference. This paper would evaluate the performance of four pre-trained models of YOLOv5 in object detection of crop diseases. And transfer learning was used to train the corresponding dataset. The experiment results have showed that the F1 values of the four models all reached above 0.93, and the Yolov5x got the best result, which achieved 0.963. Furthermore, the detection accuracy of the four models has reached more than 98%. This shows that the YOLOv5 series network models have great application prospects in the identification of crop diseases. In the near future, the object detection model can be applied to various mobile devices, even unmanned aerial vehicles, which would play a significant role in crop disease prevention.

Spatio-Temporal Feature Pyramid Interactive Attention Network for Egocentric Gaze Prediction

Over the past few years, gaze prediction technology has rapidly developed in human visual attention mechanism research. However, there is still a large gap between the prediction models and the human visual system (HVS). This work makes four contributions to video egocentric gaze prediction research. First, we introduce a new benchmark named ECVG (egocentric video gaze), which consists of 3K high-quality, elaborately selected video sequences spanning a large range of scenes. Existing eye-tracking datasets limit human activities. In contrast, ECVG achieves the collection of eye movement data under real human activity. Second, we establish a visual attention-selection and gaze-tracking model, which summarizes the human visual attention process. Third, we propose a novel feature pyramid interactive attention 3D network (FPIANet) for egocentric gaze prediction, which directly captures the long-term relationship between spatiotemporal features at different time steps and achieves feature nonlocal interactions across temporal, spatial, and scales. In addition, we propose a multiscale interactive attention (MSIA) module, which first explicitly integrates the human top-down and bottom-up visual attention mechanisms to transform any spatiotemporal feature into another feature of the same size but with richer contexts. Furthermore, we propose a new gaze transfer path consistency evaluation and thoroughly examine the performance of our model on two large datasets (ECVG and LEDOV). The experimental results show that our model outperforms the existing state-of-the-art methods.

LeNo: Adversarial Robust Salient Object Detection Networks with Learnable Noise

Pixel-wise prediction with deep neural network has become an effective paradigm for salient object detection (SOD) and achieved remarkable performance. However, very few SOD models are robust against adversarial attacks which are visually imperceptible for human visual attention. The previous work robust saliency (ROSA) shuffles the pre-segmented superpixels and then refines the coarse saliency map by the densely connected conditional random field (CRF). Different from ROSA that rely on various pre- and post-processings, this paper proposes a light-weight Learnable Noise (LeNo) to defend adversarial attacks for SOD models. LeNo preserves accuracy of SOD models on both adversarial and clean images, as well as inference speed. In general, LeNo consists of a simple shallow noise and noise estimation that embedded in the encoder and decoder of arbitrary SOD networks respectively. Inspired by the center prior of human visual attention mechanism, we initialize the shallow noise with a cross-shaped gaussian distribution for better defense against adversarial attacks. Instead of adding additional network components for post-processing, the proposed noise estimation modifies only one channel of the decoder. With the deeply-supervised noise-decoupled training on state-of-the-art RGB and RGB-D SOD networks, LeNo outperforms previous works not only on adversarial images but also on clean images, which contributes stronger robustness for SOD. Our code is available at https://github.com/ssecv/LeNo.

Efficient Rate Control in Versatile Video Coding With Adaptive Spatial–Temporal Bit Allocation and Parameter Updating

Despite the fact that Versatile Video Coding (VVC) has achieved superior coding performance, two major problems remain for the rate control (RC) model in VVC. First, the regions concerned by human eyes are not clear enough in the coded video due to the deviation between the target bit allocation strategy of the coding tree unit (CTU) in RC and the human visual attention mechanism (HVAM). Second, there are significant quality fluctuations in the coded video frames due to the inappropriate updating speed. To address the above problems, we propose an efficient rate control (ERC) model. Specifically, in order to make the coded video more consistent with the attention of human eyes, we extract texture and motion-based spatial-temporal information to guide the bit allocation at the CTU level. Furthermore, based on the quasi-Newton algorithm and bit error, we propose an adaptive parameter updating (APU) method with the proper updating speed to precisely control the bits per frame. The proposed ERC outperforms the default RC model of VVC Test Model (VTM) 9.1 by saving the average Bjøntegaard Delta Rate (BD-Rate) on full-frame video sequences by 3.60% and 4.94% under low delay P (LDP) and random access (RA) configurations respectively, with higher bitrate accuracy. Moreover, the Peak Signal-to-Noise Ratio (PSNR) and actual coded bits per frame in the video coded by the proposed ERC are more stable.

Human Visual Attention Mechanism-Inspired Point-and-Line Stereo Visual Odometry for Environments with Uneven Distributed Features

Visual odometry is critical in visual simultaneous localization and mapping for robot navigation. However, the pose estimation performance of most current visual odometry algorithms degrades in scenes with unevenly distributed features because dense features occupy excessive weight. Herein, a new human visual attention mechanism for point-and-line stereo visual odometry, which is called point-line-weight-mechanism visual odometry (PLWM-VO), is proposed to describe scene features in a global and balanced manner. A weight-adaptive model based on region partition and region growth is generated for the human visual attention mechanism, where sufficient attention is assigned to position-distinctive objects (sparse features in the environment). Furthermore, the sum of absolute differences algorithm is used to improve the accuracy of initialization for line features. Compared with the state-of-the-art method (ORB-VO), PLWM-VO show a 36.79% reduction in the absolute trajectory error on the Kitti and Euroc datasets. Although the time consumption of PLWM-VO is higher than that of ORB-VO, online test results indicate that PLWM-VO satisfies the real-time demand. The proposed algorithm not only significantly promotes the environmental adaptability of visual odometry, but also quantitatively demonstrates the superiority of the human visual attention mechanism.

Emotion estimation model for cognitive state analysis of learners in online education using deep learning

AbstractFor Facial Expression Recognition, occlusion and position change that may drastically alter facial expressions are two important challenges (FER). Due to advances in automated FER over the last several decades, it has received less attention in the real world, where occlusion‐ and pose‐invariant aspects of FER are critical. Online education learners' cognitive states may be assessed using this paper's focus on real‐world stance and occlusion robust FER. The human visual system's attention mechanism inspired us to develop a new kind of spatial attention network (SAN‐CNN). Saliency characteristics and spatial importance between adjacent pixels are emphasized in the SAN‐CNN model. Preprocessing an input picture using a median contour filter is used here initially, followed by mask‐based ROI for segmentation. Using CNNs for facial recognition, landmark localization, and head position estimation based on spatial attention networks, it is possible to do emotional categorization. Using the Kaggle public video‐based facial expression datasets, we were able to demonstrate that our proposed approach is more accurate and faster than the usual techniques. In addition, we evaluated the suggested method's performance metrics with those of the already used approaches. FER with occlusion and variant posture performs better than standard approaches using our suggested method, as shown by the results of the experiments.

FESNet: Frequency-Enhanced Saliency Detection Network for Grain Pest Segmentation

As insect infestation is the leading factor accounting for nutritive and economic losses in stored grains, it is important to detect the presence and number of insects for the sake of taking proper control measures. Inspired by the human visual attention mechanism, we propose a U-net-like frequency-enhanced saliency (FESNet) detection model, resulting in the pixelwise segmentation of grain pests. The frequency clues, as well as the spatial information, are leveraged to enhance the detection performance of small insects from the cluttered grain background. Firstly, we collect a dedicated dataset, GrainPest, with pixel-level annotation after analyzing the image attributes of the existing salient object detection datasets. Secondly, we design a FESNet with the discrete wavelet transformation (DWT) and the discrete cosine transformation (DCT), both involved in the traditional convolution layers. As current salient object detection models will reduce the spatial information with pooling operations in the sequence of encoding stages, a special branch of the discrete wavelet transformation (DWT) is connected to the higher stages to capture accurate spatial information for saliency detection. Then, we introduce the discrete cosine transform (DCT) into the backbone bottlenecks to enhance the channel attention with low-frequency information. Moreover, we also propose a new receptive field block (NRFB) to enlarge the receptive fields by aggregating three atrous convolution features. Finally, in the phase of decoding, we use the high-frequency information and aggregated features together to restore the saliency map. Extensive experiments and ablation studies on our dataset, GrainPest, and open dataset, Salient Objects in Clutter (SOC), demonstrate that the proposed model performs favorably against the state-of-the-art model.

HVS-Based Perception-Driven No-Reference Omnidirectional Image Quality Assessment

Evaluating the quality of panoramic images has gradually become a hot research topic with the development of virtual reality (VR) technology. Therefore, a novel method is proposed to assess the quality of omnidirectional images without any reference information. Inspired by the characteristics of the human visual system (HVS) and visual attention mechanism, the proposed model is composed of the structure feature, statistical feature, and saliency feature to measure the panoramic image quality, in which structure information is expressed by combining the local Taylor series with the local binary pattern (LBP) operator, gradient-based statistical information of panoramic images are summarized comprehensively from three levels: the gradient measure, the relative gradient magnitude, and the relative gradient orientation, and the saliency detection by combining simple priors (SDSP)-based saliency information is extracted in this article to enrich perception feature of our model and improve the visibility of the saliency region in the omnidirectional image. Finally, according to the subjective scores provided and the above features, we use support vector regression (SVR) to predict the objective scores. The experiments indicate that our model has more substantial competitiveness and stability than other state-of-the-art methods on two reliable databases.

All Two‐dimensional Integration‐Type Optoelectronic Synapse Mimicking Visual Attention Mechanism for Multi‐Target Recognition

AbstractThe human visual attention mechanism enables them to rapidly perceive important information and objects in a complex external scene; this effectively solves the problems of data redundancy, low‐resolution images, and substantial computing resources. The process by which the attention system reconstructs the visual information can be considered as integrating internal attention signals with external visual details in the postsynaptic neuron. However, electronic devices that simulate visual attention modulation by incorporating device characteristics into neuromorphic vision systems (NVSs) to achieve visual attention behavior are rarely reported. Herein, a synapse device that integrates optical and electrical stimulation is designed using ReS2/hBN/monolayer graphene heterojunction to mimic attention regulation and integrate multiple neuron signals successfully. The synapse array can imitate perceptual learning of the human visual system (HVS) to realize visual preprocessing, such as image contrast improvement and weak signal enhancement at the sensory terminal, and overcome data redundancy. Moreover, by applying gate voltage pulses, electric‐tunable synaptic plasticity is successfully observed, attributed to the carrier trapping and de‐trapping mechanism in the floating layer. Attention stabilization, fluctuation, distraction, and reinforcement are exhibited, simulating the attention behaviors of the HVS. Thus, an NVS with attention mechanism is established depending on the optoelectronic hybrid synaptic plasticity of the device, which successfully mimics the HVS to perform a multi‐target recognition task. Furthermore, the effect of device defects on the NVS is rarely evaluated, in which a method is provided to analyze the application results of the NVS when considering uniformity and fault rate. This study may provide new inspiration for developing neuromorphic vision systems for autonomous driving and brainwave control in the future.

Research on defect detection algorithm of complex texture ceramic tiles based on visual attention mechanism

Aiming at the difficulty of detecting the surface defects of complex texture tiles, a salient target detection method based on the human visual attention mechanism is proposed and used for the detection of tile surface defects. Firstly, the image of ceramic tile surface is pretreated using the single-scale SSR light correction method and bilateral filtering method; Secondly, according to the principle of contrast and high-frequency suppression in the visual attention mechanism, aiming at the "imaging" and "aggregation" characteristics of complex background textures, a detection model based on the visual attention mechanism is established to determine and mark defects.According to the contrast principle and high-frequency suppression principle in visual attention mechanism, feature extraction of ceramic tile surface is carried out. Then, the image color patch weight salient map and image feature fused salient map are obtained, and the two maps are fused according to the image saliency criteria.Finally, the marked ceramic tile defects are determined and marked.Finally the marked ceramic tile defects are obtained. This defect detection algorithm and the other two algorithms are applied to three kinds of randomly selected complex texture ceramic tiles. The experimental results show that compared with other algorithms, our algorithm can achieve a comprehensive detection rate of more than 96% for complex texture ceramic tiles, and can obtain a good effect of ceramic tile defect detection as well.

Masking technique based attention mechanism for off-type identification in plants

Off-type plants, or other distinct varieties of the same plant, need to be detected and removed at early stage from the fields to preserve the genetic integrity and significant traits of a given plant variety. At present, these off-type plants are manually identified in the cultivation fields through morphological analysis of leaves. This method is time-consuming and involves skilled labor to define the off-type plant. Hence, masking technique based attention mechanism is proposed for identifying the off-type from the leaf images captured in the cultivation fields. The proposed method recognizes the plant variety by locating and visualizing the prominent parts of the leaf which is similar to the human visual attention mechanism. When the prominent parts of the leaf image are masked, there is a significant drop in the class probability of the convolutional network. This reflects the importance of such prominent parts which directly account for its variety. An attention module is designed using neural networks which focuses on such significant regions and improves the performance of the network. The proposed method is tested on a field image dataset consisting of 1235 images of Sunflower leaves and achieves the highest mean accuracy of 98.25% in identifying the plant variety.

An improved YOLOv5 model based on visual attention mechanism: Application to recognition of tomato virus disease

• A deep learning model based on attention mechanism is proposed for tomato virus disease recognition. • The recognition accuracy is improved while maintaining the same detection speed. • It provides technical support for other researches related to plant disease recognition. Traditional target detection methods cannot effectively screen key features, which leads to overfitting and produces a model with a weak generalization ability. In this paper, an improved SE-YOLOv5 network model is proposed for the recognition of tomato virus diseases. Images of tomato diseases in greenhouses were collected using a mobile phone, and the collected images were expanded. A squeeze-and-excitation (SE) module was added to a YOLOv5 model to realize the extraction of key features, using a human visual attention mechanism for reference. The trained network model was evaluated on the test set of tomato virus diseases. The accuracy was 91.07%, which was 7.12%, 17.85% and 8.91% higher than that of the Faster regions with convolutional neural network features (R-CNN) model, single-shot multiBox detector (SSD) model and YOLOv5 model, respectively. Meanwhile, the mean average precision (mAP @0.5 ) was 94.10%, which was 1.23%, 16.77% and 1.78% higher than that of the Faster R-CNN model, SSD model and YOLOv5 model. The proposed SE-YOLOv5 model can effectively detect regions of tomato virus disease, which provides disease identification and control theoretical research and technical support.

Deep progressive attention for person re-identification

Person re-identification (Re-ID) aims to retrieve specific individuals across non-overlapping camera views. In recent years, attention-based models contribute to many computer vision tasks due to their great ability for learning discriminative features. We propose the deep progressive attention (DPA) in a more natural manner for person Re-ID. Similar to human visual attention mechanism, the proposed DPA progressively selects the most discriminative parts of a specific individual and formulates feature representation for comparison purpose. Concretely, on the one hand, the proposed DPA uses a long-term reward to optimize the discriminative feature selection. On the other hand, a deep convolutional architecture is integrated into a recurrent model for feature representation learning. Extensive experiments on three person Re-ID benchmarks Market-1501, DukeMTMC-reID, and CUHK03-NP demonstrate the proposed DPA is on par with the state-of-the-art. Moreover, the experiments on partial person Re-ID datasets indicate the proposed DPA is competitive with the specially designed partial person Re-ID methods.

Real-Time Monocular Obstacle Detection Based on Horizon Line and Saliency Estimation for Unmanned Surface Vehicles

Recently, real-time obstacle detection by monocular vision exhibits a promising prospect in enhancing the safety of unmanned surface vehicles (USVs). Since the obstacles that may threaten USVs generally appear below the water edge, most existing methods first detect the horizon line and then search for obstacles below the estimated horizon line. However, these methods detect horizon line only using edge or line features, which are susceptible to interference edges from clouds, waves, and land, eventually resulting in poor obstacle detection. To avoid being affected by interference edges, in this paper, we propose a novel horizon line detection method based on semantic segmentation. The method assumes a Gaussian mixture model (GMM) with spatial smoothness constraints to fit the semantic structure of marine images and simultaneously generate a water segmentation mask. The horizon line is estimated from the water boundary points via straight line fitting. Further, inspired by human visual attention mechanisms, an efficient saliency detection method based on background prior and contrast prior is presented to detect obstacles below the estimated horizon line. To reduce false positives caused by sun glitter, waves and foam, the continuity of the adjacent frames is employed to filter the detected obstacles. An extensive evaluation was conducted on a large marine image dataset collected by our ‘Jinghai VIII’ USV. The experimental results show that the proposed method significantly outperformed the recent state-of-the-art marine obstacle method by 22.07% in terms of F-score while running over 24 fps on an NVIDIA GTX1080Ti GPU.

Automatic calibration of work coordinates for robotic wire and arc additive re-manufacturing with a single camera

Industrial robots are increasingly applied in the automatic die repair welding via the prevalent wire and arc additive manufacturing (WAAM) technology. However, the precise calibration of work coordinates is indispensable for the off-line programming of robotic welding paths, which often results in positioning error, path deviation, or even tool collisions. The die is pre-heated at about 500 °C before the robotic WAAM processes. Thus, it is challenging to calibrate work coordinates by touch sensing because those points on the X-axis and the Y-axis to determine the location of the part need to be caught by human eyes. In this paper, a camera vision calibration (CVC) method based on stereo vision is developed. Image feature points are extracted by a multi-saliency fusion algorithm based on the human visual attention mechanism. Through stereo vision, 3D information of the feature points is obtained, and the workpiece coordinate system (WCS) is finely calibrated. Compared with the random error of human vision calibration (HVC), the proposed method could improve the workpiece’s calibration accuracy, reduce the unexpected collisions in limited space, and improve the dimensional precision of the welding layer.

Human eye-fixation prediction based on Convolutional Neural Network in RGB images

Using neural networks to simulate and predict human visual attention mechanism is a hot topic in the field of computer vision. In this paper, we propose an end-to-end encoder-decoder network architecture to predict the fixation mechanism of human eyes, which consists of modules composed of multiple convolutional layers with different expansion rates to capture multi-scale features in parallel. In addition, the attention module is added on the basis of the encoder network structure, and a self-attention mechanism is introduced to capture the visual feature dependency in the channel size, and the semantic interdependence in the channel dimension is modeled to predict the visual salience more accurately. In this paper, five data sets and selected examples are used to demonstrate the effectiveness of the proposed method. Our method achieves competitive and consistent results on multiple evaluation indicators on MIT1003 and CAT2000 datasets.

CNN Based No‐Reference HDR Image Quality Assessment

Motivated by the problems of nonuniversality and over-reliance on the original reference image in High dynamic range (HDR) Image quality assessment (IQA), a convolutional neural network-based algorithm for no-reference HDR image quality assessment is proposed. The Salience detection by self-resemblance (SDSR) algorithm which extracts the salient regions of the HDR image, is used to simulate the human visual attention mechanism. Then a visual quality perception network for training quality prediction models is designed according to the visual characteristics of luminance and contrast sensitivity. And this network consists of an Error estimation network (Error-net), a Perceptual resistance network (PR-net) and a mixing function. The experimental results indicate that the method proposed has high consistency with subjective perception, and the value of assessment metrics Spearman rank-order correlation coefficient (SROCC), Pearson product-moment correlation coefficient (PLCC) and Root mean square error (RMSE) correspondingly reaches 0.941, 0.910 and 8.176 as well. It is comparable with classic full-reference HDR IQA methods.

A novel distant target region detection method using hybrid saliency-based attention model under complex textures

In this paper, a hybrid visual attention model to effectively detect a distant target is proposed. The model employs the human visual attention mechanism and consists of two models, the training model, and the detection model. In the training model, some of the features are selected to train in the process of extracting and combining the early visual features from the training image of the target by bottom-up manner, and these features are trained and accumulated as trained data. When the image containing the target is input into the detection model, a task of selectively promoting only features of the target using pre-trained data is performed. As a result, the desired target is detected through the saliency map created as a result of the feature combination. The model has been tested on various images, and the experimental results demonstrate that the proposed model detected the target more accurately and faster than other previous models.

Spatiotemporal Convolutional Neural Network with Convolutional Block Attention Module for Micro-Expression Recognition

A micro-expression is defined as an uncontrollable muscular movement shown on the face of humans when one is trying to conceal or repress his true emotions. Many researchers have applied the deep learning framework to micro-expression recognition in recent years. However, few have introduced the human visual attention mechanism to micro-expression recognition. In this study, we propose a three-dimensional (3D) spatiotemporal convolutional neural network with the convolutional block attention module (CBAM) for micro-expression recognition. First image sequences were input to a medium-sized convolutional neural network (CNN) to extract visual features. Afterwards, it learned to allocate the feature weights in an adaptive manner with the help of a convolutional block attention module. The method was testified in spontaneous micro-expression databases (Chinese Academy of Sciences Micro-expression II (CASME II), Spontaneous Micro-expression Database (SMIC)). The experimental results show that the 3D CNN with convolutional block attention module outperformed other algorithms in micro-expression recognition.