Saliency Attention Research Articles

Supporting Driver Attention Toward Potential Hazards During Takeover: A Preliminary Result

This study investigates the impact of a supportive system on takeover transitions in conditionally automated driving (SAE Level 3). The supportive system is designed to direct drivers’ attention toward potential hazards in the environment when a takeover request occurs. The study comprises two components: (a) identifying various types of potential hazards using naturalistic driving data, and (b) conducting a driving simulator study to develop and assess a gaze guidance system based on the N-SEEV model of visual attention. Results indicate that drivers using a highly salient attention guidance system were less likely to collide with a secondary hazard during takeover transitions. This suggests that gaze guidance support is an effective approach for assisting drivers during takeover transitions.

Multi-Source Image Fusion Based Regional Classification Method for Apple Diseases and Pests

Efficient diagnosis of apple diseases and pests is crucial to the healthy development of the apple industry. However, the existing single-source image-based classification methods have limitations due to the constraints of single-source input image information, resulting in low classification accuracy and poor stability. Therefore, a classification method for apple disease and pest areas based on multi-source image fusion is proposed in this paper. Firstly, RGB images and multispectral images are obtained using drones to construct an apple diseases and pests canopy multi-source image dataset. Secondly, a vegetation index selection method based on saliency attention is proposed, which uses a multi-label ReliefF feature selection algorithm to obtain the importance scores of vegetation indices, enabling the automatic selection of vegetation indices. Finally, an apple disease and pest area multi-label classification model named AMMFNet is constructed, which effectively combines the advantages of RGB and multispectral multi-source images, performs data-level fusion of multi-source image data, and combines channel attention mechanisms to exploit the complementary aspects between multi-source data. The experimental results demonstrated that the proposed AMMFNet achieves a significant subset accuracy of 92.92%, a sample accuracy of 85.43%, and an F1 value of 86.21% on the apple disease and pest multi-source image dataset, representing improvements of 8.93% and 10.9% compared to prediction methods using only RGB or multispectral images. The experimental results also proved that the proposed method can provide technical support for the coarse-grained positioning of diseases and pests in apple orchards and has good application potential in the apple planting industry.

TriCAFFNet: A Tri-Cross-Attention Transformer with a Multi-Feature Fusion Network for Facial Expression Recognition.

In recent years, significant progress has been made in facial expression recognition methods. However, tasks related to facial expression recognition in real environments still require further research. This paper proposes a tri-cross-attention transformer with a multi-feature fusion network (TriCAFFNet) to improve facial expression recognition performance under challenging conditions. By combining LBP (Local Binary Pattern) features, HOG (Histogram of Oriented Gradients) features, landmark features, and CNN (convolutional neural network) features from facial images, the model is provided with a rich input to improve its ability to discern subtle differences between images. Additionally, tri-cross-attention blocks are designed to facilitate information exchange between different features, enabling mutual guidance among different features to capture salient attention. Extensive experiments on several widely used datasets show that our TriCAFFNet achieves the SOTA performance on RAF-DB with 92.17%, AffectNet (7 cls) with 67.40%, and AffectNet (8 cls) with 63.49%, respectively.

Multispectral Pedestrian Detection Based on Prior-Saliency Attention and Image Fusion

Detecting pedestrians in varying illumination conditions poses a significant challenge, necessitating the development of innovative solutions. In response to this, we introduce Prior-AttentionNet, a pedestrian detection model featuring a Prior-Attention mechanism. This model leverages the stark contrast between thermal objects and their backgrounds in far-infrared (FIR) images by employing saliency attention derived from FIR images via UNet. However, extracting salient regions of diverse scales from FIR images poses a challenge for saliency attention. To address this, we integrate Simple Linear Iterative Clustering (SLIC) superpixel segmentation, embedding the segmentation feature map as prior knowledge into UNet’s decoding stage for comprehensive end-to-end training and detection. This integration enhances the extraction of focused attention regions, with the synergy of segmentation prior and saliency attention forming the core of Prior-AttentionNet. Moreover, to enrich pedestrian details and contour visibility in low-light conditions, we implement multispectral image fusion. Experimental evaluations were conducted on the KAIST and OTCBVS datasets. Applying Prior-Attention mode to FIR-RGB images significantly improves the delineation and focus on multi-scale pedestrians. Prior-AttentionNet’s general detector demonstrates the capability of detecting pedestrians with minimal computational resources. The ablation studies indicate that the FIR-RGB+ Prior-Attention mode markedly enhances detection robustness over other modes. When compared to conventional multispectral pedestrian detection models, Prior-AttentionNet consistently surpasses them by achieving higher mean average precision and lower miss rates in diverse scenarios, during both day and night.

Foreground and background separated image style transfer with a single text condition

Traditional image-based style transfer requires additional reference style images, making it less user-friendly. Text-based methods are more convenient but suffer from issues like slow generation, unclear content, and poor quality. In this work, we propose a new style transfer method SA2-CS (means Semantic-Aware and Salient Attention CLIPStyler), which is based on the Comparative Language Image Pretraining (CLIP) model and a salient object detection network. Masks obtained from the salient object detection network are utilized to guide the style transfer process, and various strategies are employed to optimize according to different masks. Adequate experiments with diverse content images and style text descriptions were conducted, demonstrating our method's advantages: the network is easily trainable and converges rapidly; it achieves stable, superior generation results compared to other methods. Our approach addresses over-stylization issues in the foreground, enhances foreground-background contrast, and enables precise control over style transfer in various semantic regions.

A study on the analysis and understanding of art works based on graph neural networks

Abstract With the development of the Internet, many art works are uploaded for people to enjoy. To make it more convenient for viewers to understand and analyze the content of art works. In this paper, we use the GNN graph neural network analysis model to extract the art style, creation period, color expression and other features of the works through the work feature saliency attention module, and classify them using the similarity metric. At the same time, we put forward the GNNMMVisRe model to explore the works from the visual multimodality. The analysis results concluded that the Impressionist style of art works had its heyday in the 18th-19th centuries, when the number of works reached 11,674. The Baroque classicism style reached its heyday in the mid-18th century with 5,921 pieces. In terms of color use, low saturation and low luminance color palettes were selected, with average values of 81.3% and 83.4%, respectively. Using the GNN model to categorize, analyze and study the art works improves the understanding and appreciation of the works. It is also significant for the further development of art.

S$^{2}$-Net:Semantic and Saliency Attention Network for Person Re-Identification

S$^{2}$-Net:Semantic and Saliency Attention Network for Person Re-Identification

Lifespan associations of resting-state brain functional networks with ADHD symptoms

SummaryAttention-deficit/hyperactivity disorder (ADHD) is increasingly being diagnosed in both children and adults, but the neural mechanisms that underlie its distinct symptoms and whether children and adults share the same mechanism remain poorly understood. Here, we used a nested-spectral partition approach to study resting-state brain networks of ADHD patients (n = 97) and healthy controls (HCs, n = 97) across the lifespan (7–50 years). Compared to the linear lifespan associations of brain segregation and integration with age in HCs, ADHD patients have a quadratic association in the whole-brain and in most functional systems, whereas the limbic system dominantly affected by ADHD has a linear association. Furthermore, the limbic system better predicts hyperactivity, and the salient attention system better predicts inattention. These predictions are shared in children and adults with ADHD. Our findings reveal a lifespan association of brain networks with ADHD and provide potential shared neural bases of distinct ADHD symptoms in children and adults.

Lightweight single image deraining algorithm incorporating visual saliency

There are still some challenges in the task of single image rain removal, such as artefact remnant, background over-smooth, and increasingly complex and heavy-weight network architecture. Especially too heavy-weight network to fit outdoor detection devices or mobile devices. To address the above challenges, we propose a lightweight single image Deraining algorithm incorporating visual attention saliency mechanisms (LDVS). The proposed network consists of five blocks and two convolution operations, where each block consists of a dilation convolution module and a convolutional block attention module (CBAM). Specifically, visual saliency module CBAM is used for accurate localization of rain streak, and further the combinations of dilated convolution with CBAM is used to extract feature maps of rain streaks faithfully, which is able to remove artefact remnant while maintaining background details. A good tradeoff is presented between the network's weight size and effect of rain removal. Specifically, with only 48,268 parameters, the proposed model can achieve a guaranteed performance. Extensive experiments on a few typical rainy scenarios on synthetic and real-world datasets have demonstrated that to achieve the same level of performance, the proposed method has far smaller size than most of the baselines under both qualitative and quantitative analyses.

Spatio–Temporal Attention-Based Deep Learning Framework for Mesoscale Eddy Trajectory Prediction

Accurate prediction of mesoscale eddy trajectories requires efficient models with large-size available data instances to capture the main eddy characteristics. However, there is a lack of salient attention mechanisms that can recognize the demand of extracting the aggregated features over multi-dimensional eddy data sources. Additionally, deep learning techniques are very important for eddy trajectory prediction to dynamically capture properties of mesoscale eddies in the South China Sea (SCS). In this paper, we propose a spatio-temporal attention-based deep learning framework that can orchestrate heterogeneous data integration and propagation trajectory forecast together. It consists of a novel autoencoder equipped with channel and spatial attention mechanisms (CSA-Encoder), and a gated recurrent unit (GRU) network with temporal attention layer (TA-GRU). CSA-Encoder compresses stereoscopic eddy data with convolutional layers and generates the small-scale and high-quality dataset as the input of TA-GRU. The finer-grained TA-GRU method is extended to accurately predict eddy trajectories with more valuable imagery information so that the temporal attention mechanism can automatically select relevant regions within the next 14 days. Our cross-validation results demonstrate that our framework averagely achieves a lower distance error (9 km) and 54% performance improvement over the baseline GRU technique in the next one day, and outperforms two state-of-the-art techniques of long short-term memory (LSTM) and recurrent neural network (RNN) by 54.9% and 65.6%, respectively.

Multi-modal approach to evaluate adaptive visual stimuli of remote operation system using gaze behavior

A multi-modal approach is proposed to evaluate the usability of Adaptive Visual Stimuli for User Interface (AVS4UI) of remote operation systems. This study focuses on the evaluation of AVS4UI for forklift work because the operation complexity includes driving and cargo handling, which typically requires multiple salient attention. Presenting this amount of information simultaneously on a User Interface (UI) tends to cause confusions to operators and reduces operation efficiency. AVS4UI can therefore be one of the promising solutions where the optimal visual stimuli are autonomously presented for different work conditions. However, evaluation of AVS4UI is challenging because operators may be disoriented by adaptive information and worked without safety considerations. Therefore, novel gaze metrics are proposed to evaluate responses of forklift operators to AVS4UI so that undesired behavior can be evaluated. The proposed metrics implicitly represent gaze pattern in terms of transition and distribution between UI elements, operation safety, and familiarity with the adaptive system. The ideal AVS4UI is expected to minimize the proposed gaze metrics and outperform the non-adaptive UI. More importantly, the results of these metrics are consistent with those of perceived workload defined by NASA-Task Load Index. We also propose a correlation model using stepwise linear regression that provides reasonable estimation of perceived workload. Such novel metrics and correlation model enable objective and online evaluation to minimize biases of subjective response. Therefore, online work support system can be developed to support workers.

Dongba Painting Few-Shot Classification Based on Graph Neural Network

<p indent="0mm">In view of the outstanding lines and distinguishing colors of the Naxi Dongba paintings, an end- to-end few-shot classification method based on graph neural network is proposed. Firstly, a multi-resolution and multi-scale image feature extraction network is designed, the image feature and edge feature are fused and embedded into the graph as nodes to construct classification graph neural network. Secondly, with edge- labeling framework, the 2-dimensional edge feature is used as the image classification basis, and the intra- class similarity and interclass dissimilarity required for node classification are preserved. Finally, a method combining self-attention and feature saliency attention mechanism is proposed to update the node features and enhance the feature correlation between nodes. The model is implemented by Python, and RTX 2080Ti is used to conduct classification experiments on the self-built Dongba painting few-shot dataset. The experimental results show that the method can better extract the features of Dongba painting image and more sufficiently retain local details and structural features required for image classification of Dongba painting. The accuracy and standard deviation of the classification are better than others on the few-shot dataset of Dongba painting image.

Optic Disc Detection Based on Saliency Detection and Attention Convolutional Neural Networks

Optic Disc Detection Based on Saliency Detection and Attention Convolutional Neural Networks

Visualization of Salient Object With Saliency Maps Using Residual Neural Networks

Visual saliency techniques based on Convolutional Neural Networks (CNNs) exhibit an excessive performance for saliency fixation in a scene, but it is harder to train a network in view of their complexity. The imparting Residual Network Model (ResNet) that is more capable to optimize features for predicting salient area in the form of saliency maps within the images. To get saliency maps, an amalgamated framework is presented that contains two streams of Residual Network Model (ResNet-50). Each stream of Reset-50 that is used to enhance the low-level and high-level semantics features and build a network of 99 layers at two different image scales for generating the normal saliency attention. This model is trained with transfer learning for initialization that is pretrained on ImageNet for object detection, and with some modifications to minimize prediction error. At the end, the two streams integrate the features by fusion at low and high scale dimensions of images. This model is fine-tuned on four commonly used datasets and examines both qualitative and quantitative evaluation metrics for state-of-the-art deep saliency model outcomes.

A Study of Two-Way Short- and Long-Term Memory Network Intelligent Computing IoT Model-Assisted Home Education Attention Mechanism.

This paper analyzes and collates the research on traditional homeschooling attention mechanism and homeschooling attention mechanism based on two-way short- and long-term memory network intelligent computing IoT model and finds the superiority of two-way short- and long-term memory network intelligent computing IoT model. The two-way short- and long-term memory network intelligent computing IoT model is improved and an improved deep neural network intelligent computing IoT is proposed, and the improved method is verified based on discrete signal homeschooling classification experiments, followed by focusing on the application research of the two-way short- and long-term memory network intelligent computing IoT model-assisted homeschooling attention mechanism. Learning based on neural network, human behavior recognition method combining spatiotemporal networks, a homeschooling method integrating bidirectional short- and long-term memory networks and attention mechanisms is designed. The visual attention mechanism is used to add weight information to the deep visual features extracted by the convolutional neural network, and a new feature sequence incorporating salient attention weights is output. This feature sequence is then decoded using an IndRNN independent recurrent neural network to finally classify and decide on the homeschooling category. Experiments on the UCF101 dataset demonstrate that the incorporation of the attention mechanism can improve the ability of the network to classify. The attention mechanism can help the intelligent computing IoT model discover key features, and the self-attention mechanism can effectively capture the internal features of homeschooling and optimize the feature vector. We propose the strategy of combining the self-attention mechanism with a bidirectional short- and long-term memory network to solve the family education classification problem and experimentally verify that the intelligent computing IoT model combined with the self-attention mechanism can more easily capture the interdependent features in family education, which can effectively solve the family education problem and further improve the family education classification accuracy.

Complemental Attention Multi-Feature Fusion Network for Fine-Grained Classification

Transformer-based architecture network has shown excellent performance in the coarse-grained image classification. However, it remains a challenge for the fine-grained image classification task, which needs more significant regional information. As one of the attention mechanisms, transformer pays attention to the most significant region while neglecting other sub-significant regions. To use more regional information, in this letter, we propose a complemental attention multi-feature fusion network (CAMF), which extracts multiple attention features to obtain more effective features. In CAMF, we propose two novel modules: (i) a complemental attention module (CAM) that extracts the most salient attention feature and the complemental attention feature. (ii) a multi-feature fusion module (MFM) that uses different branches to extract multiple regional discriminative features. Furthermore, a new feature similarity loss is proposed to measure the diversity of inter-class features. Experiments were conducted on four public fine-grained classification datasets. Our CAMF achieves 91.2%, 92.8%, 93.3%, 95.3% on CUB-200-2011, Stanford Dogs, FGVC-Aircraft, and Stanford Cars. The ablation study verified that CAM and MFM can focus on more local discriminative regions and improve fine-grained classification performance.

Attention-Enriched Deep Learning Model for Breast Tumor Segmentation in Ultrasound Images

Incorporating human domain knowledge for breast tumor diagnosis is challenging because shape, boundary, curvature, intensity or other common medical priors vary significantly across patients and cannot be employed. This work proposes a new approach to integrating visual saliency into a deep learning model for breast tumor segmentation in ultrasound images. Visual saliency refers to image maps containing regions that are more likely to attract radiologists’ visual attention. The proposed approach introduces attention blocks into a U-Net architecture and learns feature representations that prioritize spatial regions with high saliency levels. The validation results indicate increased accuracy for tumor segmentation relative to models without salient attention layers. The approach achieved a Dice similarity coefficient (DSC) of 90.5% on a data set of 510 images. The salient attention model has the potential to enhance accuracy and robustness in processing medical images of other organs, by providing a means to incorporate task-specific knowledge into deep learning architectures.

Spatial-temporal saliency action mask attention network for action recognition

Recently, video action recognition about two-stream network is still a popular research topic in computer vision. However, most of current two-stream-based methods have two redundancy issues, including: inter-frame redundancy and intra-frame redundancy. To solve the above problems, a Spatial-Temporal Saliency Action Mask Attention network (STSAMANet) is built for action recognition. First, this paper introduces a key-frame mechanism to eliminate inter-frame redundancy. This mechanism can compute key frames on each video sequence to get the greatest difference between frames. Then, Mask R-CNN detection technology is introduced to build a saliency attention layer to eliminate intra-frame redundancy. This layer is to focus on the saliency human body and objects for each action class. We experiment on two public video action datasets, i.e., the UCF101 dataset and Penn Action dataset to verify the effectiveness of our method in action recognition.

Predicting PET-derived demyelination from multimodal MRI using sketcher-refiner adversarial training for multiple sclerosis

Multiple sclerosis (MS) is the most common demyelinating disease. In MS, demyelination occurs in the white matter of the brain and in the spinal cord. It is thus essential to measure the tissue myelin content to understand the physiopathology of MS, track progression and assess treatment efficacy. Positron emission tomography (PET) with [11C]PIB is a reliable method to measure myelin content in vivo. However, the availability of PET in clinical centers is limited. Moreover, it is expensive to acquire and invasive due to the injection of a radioactive tracer. By contrast, MR imaging is non-invasive, less expensive and widely available, but conventional MRI sequences cannot provide a direct and reliable measure of myelin. In this work, we therefore propose, to the best of our knowledge for the first time, a method to predict the PET-derived myelin content map from multimodal MRI. To that purpose, we introduce a new approach called Sketcher-Refiner generative adversarial networks (GANs) with specifically designed adversarial loss functions. The first network (Sketcher) generates global anatomical and physiological information. The second network (Refiner) refines and generates the tissue myelin content. A visual attention saliency map is also proposed to interpret the attention of neural networks. Our approach is shown to outperform the state-of-the-art methods in terms of image quality and myelin content prediction. Particularly, our prediction results show similar results to the PET-derived gold standard at both global and voxel-wise levels indicating the potential for clinical management of patients with MS.

A Genre-Based Investigation of Inter/Intra-Lingual Relationships between Persian and English Academic Writings: Common Underlying Proficiency Oriented

Although L2 writing has attracted salient attention and monopolized many studies in EFL contexts, there is still no full image of its complicated nature. Trying to play a supplementary role in achieving that image, this study aimed at finding whether Persian and English argumentative and descriptive academic writings were inter/intra-lingually associated and if genre played a role in provoking any differences in the means of total scores obtained from the essays. In so doing, 50 Iranian senior MA TEFL students at the intermediate level were asked to write down two argumentative and two descriptive essays in Persian and English. Each essay was assessed against IELTS task 2 writing band descriptors. Then, the data were analyzed through the Pearson product-moment correlation and paired-samples t-test. The results revealed there were significant positive correlations between the total scores obtained from Persian and English argumentative and descriptive essays. The findings supported the existence of a common underlying proficiency (Cummins, 1980), out of which L1 and L2 writing skills emerged. The results also illustrated there were significant differences between the means of some of the total scores. It was supposed these differences might be due to the possible genre-sensitive nature of common underlying proficiency.