Attention-based Network Research Articles

Generalizable Underwater Acoustic Target Recognition Using Feature Extraction Module of Neural Network

The underwater acoustic target signal is affected by factors such as the underwater environment and the ship’s working conditions, causing the generalization of the recognition model is essential. This study is devoted to improving the generalization of recognition models, proposing a feature extraction module based on neural network and time-frequency analysis, and validating the feasibility of the model-based transfer learning method. A network-based filter based on one-dimensional convolution is built according to the calculation mode of the finite impulse response filter. An attention-based model is constructed using the convolution network components and full-connection components. The attention-based network utilizes convolution components to perform the Fourier transform and feeds back the optimization gradient of a specific task to the network-based filter. The network-based filter is designed to filter the observed signal for adaptive perception, and the attention-based model is constructed to extract the time-frequency features of the signal. In addition, model-based transfer learning is utilized to further improve the model’s performance. Experiments show that the model can perceive the frequency domain features of underwater acoustic targets, and the proposed method demonstrates competitive performance in various classification tasks on real data, especially those requiring high generalizability.

Corn Disease Recognition Based on Attention Mechanism Network

To extract more accurate and abundant features of corn disease and solve the problems of rough classification and low recognition accuracy, the attention mechanism is introduced into the field of corn disease recognition. The corn disease recognition model (AT-AlexNet) is proposed based on an attention mechanism. The network was based on AlexNet, and the new down-sampling attention module was constructed to enhance the foreground response of the disease; the Mish activation function was introduced to improve the nonlinear expression of the network; the new module of the full connection layer was designed to reduce the network parameters. In the experiment of the enhanced corn disease datasets, the average recognition accuracy of the attention-based network model AT-AlexNet is 99.35%. The recognition accuracy of using the Mish activation function is 0.65% higher than that of the ReLu activation function. The experiments show that compared with other identification methods, the proposed method has better classification performance for corn diseases.

Ayu-Characterization of healthy aging from neuroimaging data with deep learning and rsfMRI.

Estimating brain age and establishing functional biomarkers that are prescient of cognitive declines resulting from aging and different neurological diseases are still open research problems. Functional measures such as functional connectivity are gaining interest as potentially more subtle markers of neurodegeneration. However, brain functions are also affected by “normal” brain aging. More information is needed on how functional connectivity relates to aging, particularly in the absence of neurodegenerative disorders. Resting-state fMRI enables us to investigate functional brain networks and can potentially help us understand the processes of development as well as aging in terms of how functional connectivity (FC) matures during the early years and declines during the late years. We propose models for estimation of the chronological age of a healthy person from the resting state brain activation (rsfMRI). In this work, we utilized a dataset (N = 638, age-range 20–88) comprising rsfMRI images from the Cambridge Centre for Aging and Neuroscience (Cam-CAN) repository of a healthy population. We propose an age prediction pipeline Ayu which consists of data preprocessing, feature selection, and an attention-based model for deep learning architecture for brain age assessment. We extracted features from the static functional connectivity (sFC) to predict the subject's age and classified them into different age groups (young, middle, middle, and old ages). To the best of our knowledge, a classification accuracy of 72.619 % and a mean absolute error of 6.797, and an r2 of 0.754 reported by our Ayu pipeline establish competitive benchmark results as compared to the state-of-the-art-approach. Furthermore, it is vital to identify how different functional regions of the brain are correlated. We also analyzed how functional regions contribute differently across ages by applying attention-based networks and integrated gradients. We obtained well-known resting-state networks using the attention model, which maps to within the default mode network, visual network, ventral attention network, limbic network, frontoparietal network, and somatosensory network connected to aging. Our analysis of fMRI data in healthy elderly Age groups revealed that dynamic FC tends to slow down and becomes less complex and more random with increasing age.

Multilayer feature fusion and attention-based network for crops and weeds segmentation

Multilayer feature fusion and attention-based network for crops and weeds segmentation

Spatio-temporal aggregation of skeletal motion features for human motion prediction

This study proposes a human body motion prediction model that can adapt to the disorders in various human motion patterns and represent the kinematic constraints. In human motion prediction, the acquisition of features that capture inter-motion and inter-joint linkages is considered effective. To generate links that are adaptive to the reference time of dominant features and their crossing-over joints, we construct an attention-based network that aggregates motion sequences temporally and spatially. We evaluated the motion prediction results using the Human3.6M dataset with the indices of mean angle error and mean per joint position error and showed that our method outperforms other state-of-the-art methods.

Classification of Tetanus Severity in Intensive-Care Settings for Low-Income Countries Using Wearable Sensing.

Infectious diseases remain a common problem in low- and middle-income countries, including in Vietnam. Tetanus is a severe infectious disease characterized by muscle spasms and complicated by autonomic nervous system dysfunction in severe cases. Patients require careful monitoring using electrocardiograms (ECGs) to detect deterioration and the onset of autonomic nervous system dysfunction as early as possible. Machine learning analysis of ECG has been shown of extra value in predicting tetanus severity, however any additional ECG signal analysis places a high demand on time-limited hospital staff and requires specialist equipment. Therefore, we present a novel approach to tetanus monitoring from low-cost wearable sensors combined with a deep-learning-based automatic severity detection. This approach can automatically triage tetanus patients and reduce the burden on hospital staff. In this study, we propose a two-dimensional (2D) convolutional neural network with a channel-wise attention mechanism for the binary classification of ECG signals. According to the Ablett classification of tetanus severity, we define grades 1 and 2 as mild tetanus and grades 3 and 4 as severe tetanus. The one-dimensional ECG time series signals are transformed into 2D spectrograms. The 2D attention-based network is designed to extract the features from the input spectrograms. Experiments demonstrate a promising performance for the proposed method in tetanus classification with an F1 score of 0.79 ± 0.03, precision of 0.78 ± 0.08, recall of 0.82 ± 0.05, specificity of 0.85 ± 0.08, accuracy of 0.84 ± 0.04 and AUC of 0.84 ± 0.03.

FusionNet: A Convolution–Transformer Fusion Network for Hyperspectral Image Classification

In recent years, deep-learning-based hyperspectral image (HSI) classification networks have become one of the most dominant implementations in HSI classification tasks. Among these networks, convolutional neural networks (CNNs) and attention-based networks have prevailed over other HSI classification networks. While convolutional neural networks with perceptual fields can effectively extract local features in the spatial dimension of HSI, they are poor at capturing the global and sequential features of spectral–spatial information; networks based on attention mechanisms, for example, Transformer, usually have better ability to capture global features, but are relatively weak in discriminating local features. This paper proposes a fusion network of convolution and Transformer for HSI classification, known as FusionNet, in which convolution and Transformer are fused in both serial and parallel mechanisms to achieve the full utilization of HSI features. Experimental results demonstrate that the proposed network has superior classification results compared to previous similar networks, and performs relatively well even on a small amount of training data.

FA-Net: fused attention-based network for Hindi English code-mixed offensive text classification

FA-Net: fused attention-based network for Hindi English code-mixed offensive text classification

A COMPARATIVE STUDY FOR BUILDING SEGMENTATION IN REMOTE SENSING IMAGES USING DEEP NETWORKS: CSCRS ISTANBUL BUILDING DATASET AND RESULTS

Abstract. Building semantic segmentation is an exceedingly important issue in the field of remote sensing. A new building dataset as created consisting of very high-resolution optical satellite images provided by the Center for Satellite Communications and Remote Sensing (CSCRS). The imagery is obtained by Pleiades satellite and have a resolution of 0.5 meters. Segmentation results have been obtained using post-FCN architectures. Architectures examined in this work fall under one of few categories. The first category is Encoder-Decoder Network: an encoder that reduces the spatial resolution of the data and a decoder that recreates the lower resolution result of the encoder and upsamples it. The second category is Feature Pyramid Network, in this type of network scene information is aggregated across pyramid structures which produce more comprehensive results. The third category is Dilated Network, due to its atrous structure, which can calculate any layer at any desired resolution, with the presence of holes in the filter. The final category is Attention-Based Network, in these networks, certain aspects of the data are emphasized while other aspects are ignored. After this work, it can be seen that according to several metrics Dilated and Attention-Based Networks perform better than their counterparts. As a result of the training of 100 epochs with the data set in architectures belonging to Dilated and Attention-Based Networks, IoU values above 0.90 were obtained.

Comparative Analysis of Current Deep Learning Networks for Breast Lesion Segmentation in Ultrasound Images.

Automatic lesion segmentation in breast ultrasound (BUS) images aids in the diagnosis of breast cancer, the most common type of cancer in women. Accurate lesion segmentation in ultrasound images is a challenging task due to speckle noise, artifacts, shadows, and lesion variability in size and shape. Recently, convolutional neural networks have demonstrated impressive results in medical image segmentation tasks. However, the lack of public benchmarks and a standardized evaluation method hampers the networks' performance comparison. This work presents a benchmark of seven state-of-the-art methods for the automatic breast lesion segmentation task. The methods were evaluated on a multi-center BUS dataset composed of three public datasets. Specifically, the U-Net, Dynamic U-Net, Semantic Segmentation Deep Residual Network with Variational Autoencoder (SegResNetVAE), U-Net Transformers, Residual Feedback Network, Multiscale Dual Attention-Based Network, and Global Guidance Network (GG-Net) architectures were evaluated. The training was performed with a combination of the cross-entropy and Dice loss functions and the overall performance of the networks was assessed using the Dice coefficient, Jaccard index, accuracy, recall, specificity, and precision. Despite all networks having obtained Dice scores superior to 75%, the GG-Net and SegResNetVAE architectures outperform the remaining methods, achieving 82.56% and 81.90%, respectively. Clinical Relevance- The results of this study allowed to prove the potential of deep neural networks to be used in clinical practice for breast lesion segmentation also suggesting the best model choices.

DCPCR: Deep Compressed Point Cloud Registration in Large-Scale Outdoor Environments

Reliable and accurate registration of point clouds is a challenging problem in robotics as well as in the domain of autonomous driving. In this article, we address the task of aligning point clouds with low overlap, containing moving objects, and without prior information about the initial guess. We enhance classical ICP-based registration with neural feature-based matching to reliably find point correspondences. Our novel 3D convolutional and attention-based network is trained in an end-to-end fashion to learn features, which are well suited for matching and for rating the quality of the point correspondences. By utilizing a compression encoder, we can directly operate on a compressed map representation, making our approach well suited for operation under memory constraints. We evaluate our approach on point clouds obtained at completely different points in time, showing that our approach is able to register point clouds even under those challenging conditions reliably.

CQA-Face: Contrastive Quality-Aware Attentions for Face Recognition

Few existing face recognition (FR) models take local representations into account. Although some works achieved this by extracting features on cropped parts around face landmarks, landmark detection may be inaccurate or even fail in some extreme cases. Recently, without relying on landmarks, attention-based networks can focus on useful parts automatically. However, there are two issues: 1) It is noticed that these approaches focus on few facial parts, while missing other potentially discriminative regions. This can cause performance drops when emphasized facial parts are invisible under heavy occlusions (e.g. face masks) or large pose variations; 2) Different facial parts may appear at various quality caused by occlusion, blur, or illumination changes. In this paper, we propose contrastive quality-aware attentions, called CQA-Face, to address these two issues. First, a Contrastive Attention Learning (CAL) module is proposed, pushing models to explore comprehensive facial parts. Consequently, more useful parts can help identification if some facial parts are invisible. Second, a Quality-Aware Network (QAN) is developed to emphasize important regions and suppress noisy parts in a global scope. Thus, our CQA-Face model is developed by integrating the CAL with QAN, which extracts diverse quality-aware local representations. It outperforms the state-of-the-art methods on several benchmarks, demonstrating its effectiveness and usefulness.

Region Dual Attention-Based Video Emotion Recognition.

To solve the emotional differences between different regions of the video frame and make use of the interrelationship between different regions, a region dual attention-based video emotion recognition method (RDAM) is proposed. RDAM takes as input video frame sequences and learns a discriminatory video emotion representation that can make full use of the emotional differences of different regions and the interrelationship between regions. Specifically, we construct two parallel attention modules: one is the regional location attention module, which generates a weight value for each feature region to identify the relative importance of different regions. Based on the weight, the emotion feature that can perceive the emotional sensitive region is generated. The other is the regional relationship attention module, which generates a region relation matrix that represents the interrelationship of different regions of a video frame. Based on the region relation matrix, the emotion feature that can perceive interrelationship between different regions is generated. The outputs of these two attention modules are fused to produce the emotional features of video frames. Then, the features of video frame sequences are fused by attention-based fusion network, and the final emotion feature of the video is produced. The experimental results on the video emotion recognition data sets show that the proposed method outperforms the other related works.

Video summarization with a convolutional attentive adversarial network

With the explosive growth of video data, video summarization, which attempts to seek the minimum subset of frames while still conveying the main story, has become one of the hottest topics. Nowadays, substantial achievements have been made by supervised learning techniques, especially after the emergence of deep learning. However, it is extremely expensive and difficult to construct a large-scale video summarization dataset through human annotation. To address this problem, we propose a convolutional attentive adversarial network (CAAN), whose key idea is to build a deep summarizer in an unsupervised way. Upon the generative adversarial network, our overall framework consists of a generator and a discriminator. The former predicts importance scores for all the frames of a video while the latter tries to distinguish the score-weighted frame features from original frame features. To capture the global and local temporal relationship of video frames, the generator employs a fully convolutional sequence network to build global representation of a video, and an attention-based network to predict normalized importance scores. To optimize the parameters, our objective function is composed of three loss functions, which can guide the frame-level importance score prediction collaboratively. To validate this proposed method, we have conducted extensive experiments on two public benchmarks SumMe and TVSum. The results show the superiority of our proposed method against other state-of-the-art unsupervised approaches. Our method even outperforms some published supervised approaches.

Efficient and Highly Accurate Diagnosis of Malignant Hematological Diseases Based on Whole-Slide Images Using Deep Learning.

Hematopoietic disorders are serious diseases that threaten human health, and the diagnosis of these diseases is essential for treatment. However, traditional diagnosis methods rely on manual operation, which is time consuming and laborious, and examining entire slide is challenging. In this study, we developed a weakly supervised deep learning method for diagnosing malignant hematological diseases requiring only slide-level labels. The method improves efficiency by converting whole-slide image (WSI) patches into low-dimensional feature representations. Then the patch-level features of each WSI are aggregated into slide-level representations by an attention-based network. The model provides final diagnostic predictions based on these slide-level representations. By applying the proposed model to our collection of bone marrow WSIs at different magnifications, we found that an area under the receiver operating characteristic curve of 0.966 on an independent test set can be obtained at 10× magnification. Moreover, the performance on microscopy images can achieve an average accuracy of 94.2% on two publicly available datasets. In conclusion, we have developed a novel method that can achieve fast and accurate diagnosis in different scenarios of hematological disorders.

Resolving combinatorial ambiguities in dilepton tt¯ event topologies with neural networks

We study the potential of deep learning to resolve the combinatorial problem in SUSY-like events with two invisible particles at the LHC. As a concrete example, we focus on dileptonic $t \bar t$ events, where the combinatorial problem becomes an issue of binary classification: pairing the correct lepton with each $b$ quark coming from the decays of the tops. We investigate the performance of a number of machine learning algorithms, including attention-based networks, which have been used for a similar problem in the fully-hadronic channel of $t\bar t$ production; and the Lorentz Boost Network, which is motivated by physics principles. We then consider the general case when the underlying mass spectrum is unknown, and hence no kinematic endpoint information is available. Compared against existing methods based on kinematic variables, we demonstrate that the efficiency for selecting the correct pairing is greatly improved by utilizing deep learning techniques.

Chatter detection in milling of thin-walled parts using multi-channel feature fusion and temporal attention-based network

Chatter often occurs in the milling of thin-walled parts due to their high flexibility, which can seriously impair machining accuracy, surface quality, and production efficiency. In this paper, a chatter detection method in the milling of thin-walled parts based on deep learning is proposed. Multi-channel signal features in the time, frequency, and time–frequency domains are employed and their chatter sensitivities are evaluated. A novel temporal attention-based network is constructed. The self-attention mechanism, channel attention mechanism, and attentive statistics pooling are applied to capture the intra-feature correlation, assign the weights of different features, and calculate the variations and weights of features at multiple moments, respectively. The mutual information of distinct features is integrated into the channel attention mechanism to enhance the function of suppressing useless features and enhancing useful ones. The additive angular margin loss function is applied to strengthen intra-class aggregation and inter-class separation. Milling experiments on straight-wall parts are performed, and the classification performance of this model is evaluated in three scenarios. Afterward, milling experiments on curved thin-walled parts are conducted to verify the generalization performance. The results show that the proposed method can accurately detect chatter under varying machining conditions.

Research on Action Recognition Based on Deep Learning with Long Short-Term Memory Network and Attention Mechanism

Action recognition is a basic and challenging task in the field of computer vision. In this paper, a deep learning action recognition method based on attention mechanism is proposed and successfully applied to several public data sets, with outstanding performance. Firstly, the video frames are sampled based on the improved sampling algorithm, and the video data enhancement algorithm is proposed to preprocess the original data, which will reduce the overfitting probability of the recognition model and reduce the white noise of the data. Then, feature selection is carried out through attention-based residual network. Finally, we completed the action classification by LSTM model and softmax. In addition, a series of ablation experiments were designed to verify the validity of the proposed model. The results indicate that compared with the traditional action recognition model, the proposed method can effectively extract key features, reduce the overfitting caused by a small number of samples, reduce the interference of redundant information through the screening of low-information video frames, and complete the action recognition accurately, quickly, and efficiently.

High-dimensional Bayesian optimization of 23 hyperparameters over 100 iterations for an attention-based network to predict materials property: A case study on CrabNet using Ax platform and SAASBO

Expensive-to-train deep learning models can benefit from an optimization of the hyperparameters that determine the model architecture. We optimize 23 hyperparameters of a materials informatics model, Compositionally-Restricted Attention-Based Network (CrabNet), over 100 adaptive design iterations using two models within the Adaptive Experimentation (Ax) Platform. This includes a recently developed Bayesian optimization (BO) algorithm, sparse axis-aligned subspaces Bayesian optimization (SAASBO), which has shown exciting performance on high-dimensional optimization tasks. Using SAASBO to optimize CrabNet hyperparameters, we demonstrate a new state-of-the-art on the experimental band gap regression task within the materials informatics benchmarking platform, Matbench (∼4.5% decrease in mean absolute error (MAE) relative to incumbent). Characteristics of the adaptive design scheme as well as feature importances are described for each of the Ax models. SAASBO has great potential to both improve existing surrogate models, as shown in this work, and in future work, to efficiently discover new, high-performing materials in high-dimensional materials science search spaces.

A channel-spatial-temporal attention-based network for vibration-based damage detection

Structural health monitoring (SHM) is extremely vital for the diagnosis and prognosis of civil structures. As an important part of the SHM system, vibration-based damage detection (VBDD) methods have become a research hotspot with the development of sensor technologies. These methods are utilized to assess structural conditions or localize and classify damages. Recently end-to-end deep learning architectures have been widely used in VBDD tasks and achieved state-of-the-art results. However, there are seldom investigations on the attention mechanism in VBDD, which has been demonstrated as an effective module to extract features in other domains. In this paper, we propose a channel-spatial-temporal attention-based network to refine and enrich the discriminative sample-specific features in three dimensions, namely, channel, space, and time simultaneously. Specifically, the local and global block we designed is to extract the local and global spatial features adaptively, and the grouped self-attention is presented to extract the long- and short-term temporal features. Moreover, the squeeze-and-excitation block is selected to emphasize vital channels. Extensive experiments are conducted on three-span continuous rigid frame bridge scale model and IASC-ASCE benchmark datasets, and the results prove that the proposed method is superior to the existing state-of-the-art methods.