Attention Convolutional Network Research Articles

Research on Bearing Remaining Useful Life Anti-noise Prediction Based on Fusion of Color-Grayscale Time-Frequency Features

Abstract In contemporary industrial processes, vibration signals collected from bearings often contain significant noise, challenging the efficacy of conventional predictive models in extracting critical degradation features and accurately predicting the remaining useful life (RUL) of bearings. Addressing these challenges, this paper introduces a novel method for predicting bearing RUL under noisy conditions, leveraging a dual-branch multi-scale convolutional attention network (DMCSA) integrated with a dense residual feature fusion network (DRF). Initially, the method applies continuous wavelet trans-form (CWT) to vibration signals to extract color time-frequency image data, followed by grayscale processing to construct a comprehensive color-grayscale time-frequency image dataset, thereby augmenting the model's input features. Enhanced channel and spatial attention mechanisms, combined with multi-scale convolutions, facilitate supe-rior feature extraction and selection. The model's resilience to noise is fortified by in-corporating noise into the training dataset. Subsequently, selected color-gray time-frequency features undergo fusion and relearning through the DRF framework at the model's backend. The crayfish optimization algorithm (COA) is deployed for the astute determination of the model's critical hyperparameters. The proposed DMCSA-DRF model is then applied to predict the health indicator (MSCA-DRF-HI) of the test dataset, culminating in the accurate prediction of the bearings' RUL. Validation experiments demonstrate that our method surpasses comparative models in prediction accuracy un-der diverse noise interferences, signifying a substantial advancement in predictive performance.

LCASPMDA: a computational model for predicting potential microbe-drug associations based on learnable graph convolutional attention networks and self-paced iterative sampling ensemble

IntroductionNumerous studies show that microbes in the human body are very closely linked to the human host and can affect the human host by modulating the efficacy and toxicity of drugs. However, discovering potential microbe-drug associations through traditional wet labs is expensive and time-consuming, hence, it is important and necessary to develop effective computational models to detect possible microbe-drug associations.MethodsIn this manuscript, we proposed a new prediction model named LCASPMDA by combining the learnable graph convolutional attention network and the self-paced iterative sampling ensemble strategy to infer latent microbe-drug associations. In LCASPMDA, we first constructed a heterogeneous network based on newly downloaded known microbe-drug associations. Then, we adopted the learnable graph convolutional attention network to learn the hidden features of nodes in the heterogeneous network. After that, we utilized the self-paced iterative sampling ensemble strategy to select the most informative negative samples to train the Multi-Layer Perceptron classifier and put the newly-extracted hidden features into the trained MLP classifier to infer possible microbe-drug associations.Results and discussionIntensive experimental results on two different public databases including the MDAD and the aBiofilm showed that LCASPMDA could achieve better performance than state-of-the-art baseline methods in microbe-drug association prediction.

Multi-hop double-branch network for small sample hyperspectral image classification

ABSTRACT Neural networks have been extensively utilized in the classification of hyperspectral images. The study introduces a novel dual-branch structured network named MGACN for hyperspectral image classification under the condition of small sample sizes. This model a multi-hop graph network (MHGCN) and a graph attention convolutional network (GACNN). Graph Convolutional Network (GCN) integrates multi-hop mechanism to obtain and aggregate information between long-distance non-neighbor nodes. The Graph Attention Network (GAT) combines 2D-CNN by incorporating a noise reduction mechanism, as well as position and channel attention mechanisms. The classification accuracy reaches 98.43%, 99.02%, and 98.55% on the Indian Pines, Salinas, and Pavia University datasets, which is significantly better than that of the comparison model.

Incorporating the Third Law of Geography with Spatial Attention Module–Convolutional Neural Network–Transformer for Fine-Grained Non-Stationary Air Quality Predictive Learning

Accurate air quality prediction is paramount in safeguarding public health and addressing air pollution control. However, previous studies often ignore the geographic similarity among different monitoring stations and face challenges in dynamically capturing different spatial–temporal relationships between stations. To address this, an air quality predictive learning approach incorporating the Third Law of Geography with SAM–CNN–Transformer is proposed. Firstly, the Third Law of Geography is incorporated to fully consider the geographical similarity among stations via a variogram and spatial clustering. Subsequently, a spatial–temporal attention convolutional network that combines the spatial attention module (SAM) with the convolutional neural network (CNN) and Transformer is designed. The SAM is employed to extract spatial–temporal features from the input data. The CNN is utilized to capture local information and relationships among each input feature. The Transformer is applied to capture time dependencies across long-distance time series. Finally, Shapley’s analysis is employed to interpret the model factors. Numerous experiments with two typical air pollutants (PM2.5, PM10) in Haikou City show that the proposed approach has better comprehensive performance than baseline models. The proposed approach offers an effective and practical methodology for fine-grained non-stationary air quality predictive learning.

Genomic prediction for sugarcane diseases including hybrid Bayesian-machine learning approaches.

Sugarcane smut and Pachymetra root rots are two serious diseases of sugarcane, with susceptible infected crops losing over 30% of yield. A heritable component to both diseases has been demonstrated, suggesting selection could improve disease resistance. Genomic selection could accelerate gains even further, enabling early selection of resistant seedlings for breeding and clonal propagation. In this study we evaluated four types of algorithms for genomic predictions of clonal performance for disease resistance. These algorithms were: Genomic best linear unbiased prediction (GBLUP), including extensions to model dominance and epistasis, Bayesian methods including BayesC and BayesR, Machine learning methods including random forest, multilayer perceptron (MLP), modified convolutional neural network (CNN) and attention networks designed to capture epistasis across the genome-wide markers. Simple hybrid methods, that first used BayesR/GWAS to identify a subset of 1000 markers with moderate to large marginal additive effects, then used attention networks to derive predictions from these effects and their interactions, were also developed and evaluated. The hypothesis for this approach was that using a subset of markers more likely to have an effect would enable better estimation of interaction effects than when there were an extremely large number of possible interactions, especially with our limited data set size. To evaluate the methods, we applied both random five-fold cross-validation and a structured PCA based cross-validation that separated 4702 sugarcane clones (that had disease phenotypes and genotyped for 26k genome wide SNP markers) by genomic relationship. The Bayesian methods (BayesR and BayesC) gave the highest accuracy of prediction, followed closely by hybrid methods with attention networks. The hybrid methods with attention networks gave the lowest variation in accuracy of prediction across validation folds (and lowest MSE), which may be a criteria worth considering in practical breeding programs. This suggests that hybrid methods incorporating the attention mechanism could be useful for genomic prediction of clonal performance, particularly where non-additive effects may be important.

Predicting the Intensity of Tropical Cyclones over the Western North Pacific Using a Dual-Branch Spatiotemporal Attention Convolutional Network

Abstract This paper proposes a spatiotemporal attention convolutional network (STAC-Pred) that leverages deep learning techniques to model the spatiotemporal features of tropical cyclones (TCs) and enable real-time prediction of their intensity. The proposed model employs dual branches to concurrently extract and integrate features from intensity heatmaps and satellite cloud imagery. Additionally, a residual attention (RA) module is integrated into the three-channel cloud imagery convolution process to automatically respond to high wind speed regions. TC’s longitude, latitude, and radius of winds are injected into the multi-timepoint prediction model to assist in the prediction task. Furthermore, a rolling mechanism (RM) is employed to smooth the fluctuation of losses, achieving accurate prediction of TC intensity. We use several TC records to evaluate and validate the universality and effectiveness of the model. The results indicate that STAC-Pred achieves satisfactory performance. Specifically, the STAC-Pred model improves prediction performance by 47.69% and 28.26% compared to the baseline (official institutions) at 3- and 6-h intervals, respectively. Significance Statement Tropical cyclones are one of the most deadly and damaging natural disasters in coastal areas worldwide. Early prediction can significantly reduce casualties and property losses. This study innovatively conducts dimensionality augmentation on one-dimensional intensity numerical sequences and proposes a new network model for rolling forecast of their future intensity. The proposed prototype model (not yet incorporating any atmospheric conditions) shows promising results for 3- and 6-h advance forecasts, providing valuable guidance for forecasters regarding real-time operational predictions of short-term tropical cyclone intensity.

Flood Water Depth Prediction with Convolutional Temporal Attention Networks

Robust and accurate flood hazard maps are essential for early warning systems and flood risk management. Although physically based models are effective in estimating pluvial flooding, the computational burden makes them difficult to use for real-time flood prediction. In contrast, data-driven models can provide faster flood predictions if trained offline. While most studies have focused on predicting maximum water depth, in this study, we predict pixel-wise water depth maps for entire catchments at a lead time of 2 h. To that end, we propose a deep learning approach that uses a sequence encoding network with temporal self-attention. We also adapt the popular hydrological performance metric Nash–Sutcliffe efficiency (NSE) as our loss function. We test the effectiveness and generalizability of our method using a new dataset called SwissFlood, which consists of 100 catchments and 1500 rainfall events extracted from real observations in Switzerland. Our method produces 2 m spatial resolution flood maps with absolute error as low as 27 cm for water depth exceeding 1 m.

A multi-resolution convolutional attention network for efficient diabetic retinopathy classification

Diabetic retinopathy (DR) is a disorder commonly associated with individuals who have diabetes and is caused by excessive glucose levels in the blood, which induce retinal damage. Manual diagnosis of DR by medical experts is time-consuming; hence, a computer-aided diagnosis (CAD) solution for faster identification and improved efficiency is required. Diagnosis of DR is complex due to the prevalence of inter-class differences and small lesions, making it difficult for convolutional neural networks (CNNs) to identify discriminative regions effectively. The most critical challenge is locating complex patterns connected with visual differences and lesion locations. To tackle the aforementioned challenges, this work presents a novel multi-resolution convolutional attention network (MuR-CAN), which improves overall performance by emphasizing discriminative features. MuR-CAN includes a multi-dilation attention block (MDAB), which uses depth-wise convolution layers with various dilation rates to allow the model to prioritize relevant features and capture multi-scale spatial information. This method provides for simultaneously highlighting fine features and subtle visual patterns. Furthermore, the features extracted from the MuR-CAN are utilized to train the support vector machine (SVM) for DR classification. The integration of the SVM classifier with the proposed CNN architecture ensures improved accuracy. The experimental findings demonstrate that the proposed model outperforms the current state-of-the-art methods.

Effective Monoaural Speech Separation through Convolutional Top-Down Multi-View Network

Speech separation, sometimes known as the “cocktail party problem”, is the process of separating individual speech signals from an audio mixture that includes ambient noises and several speakers. The goal is to extract the target speech in this complicated sound scenario and either make it easier to understand or increase its quality so that it may be used in subsequent processing. Speech separation on overlapping audio data is important for many speech-processing tasks, including natural language processing, automatic speech recognition, and intelligent personal assistants. New speech separation algorithms are often built on a deep neural network (DNN) structure, which seeks to learn the complex relationship between the speech mixture and any specific speech source of interest. DNN-based speech separation algorithms outperform conventional statistics-based methods, although they typically need a lot of processing and/or a larger model size. This study presents a new end-to-end speech separation network called ESC-MASD-Net (effective speaker separation through convolutional multi-view attention and SuDoRM-RF network), which has relatively fewer model parameters compared with the state-of-the-art speech separation architectures. The network is partly inspired by the SuDoRM-RF++ network, which uses multiple time-resolution features with downsampling and resampling for effective speech separation. ESC-MASD-Net incorporates the multi-view attention and residual conformer modules into SuDoRM-RF++. Additionally, the U-Convolutional block in ESC-MASD-Net is refined with a conformer layer. Experiments conducted on the WHAM! dataset show that ESC-MASD-Net outperforms SuDoRM-RF++ significantly in the SI-SDRi metric. Furthermore, the use of the conformer layer has also improved the performance of ESC-MASD-Net.

Compound fault diagnosis of diesel engines by combining generative adversarial networks and transfer learning

In order to solve the problem of compound fault diagnosis of diesel engine fuel injection system under the condition of few samples, a comprehensive diagnosis method based on generative adversarial networks and transfer learning based multi label classification models is proposed in this paper, which is based on convolutional attention networks. The generative adversarial networks are used to enhance the original data, and the enhanced data is used as the source data for transfer learning to ensure its effectiveness. Design multi-dimensional labels for compound fault diagnosis to enable the model to diagnose compound faults and their individual faults. Pre train the convolutional attention network with enhanced data generated from adversarial networks. Subsequently, the original data is used to integrate the feature information in the network and perform secondary training on the classification module. Finally, a compound fault diagnosis is implemented based on the model after secondary training. The effectiveness of the comprehensive diagnostic method proposed in the article is verified using real fault data of the fuel injection system. The diagnostic accuracy on test data reached 80%. The effectiveness of the comprehensive diagnostic method proposed in the article was verified using real fault data of the fuel injection system. The diagnostic accuracy on test data reached 80%. Compared to the method of directly training convolutional attention networks using raw data and the method of directly training convolutional attention networks without transfer learning using GAN enhanced datasets, the accuracy is improved by 16% and 8%, respectively.

Asymmetric convolutional multi-level attention network for micro-lens segmentation

Tiny target recognition in automation is currently a hot research task that usually suffers from typical issues such as complex background, dim target, and slow detection speed. In the current study, a data-driven method is proposed to realize the posture recognition of micro-lens during optical device coupling to achieve accurate clamping of the gripper. First, we establish a pixel-by-pixel labeled optical micro-lens dataset named single-frame micro-lens target (SFMT), which provides data support for the subsequently proposed convolutional neural network. Subsequently, an asymmetric convolutional multi-level attention network (ACMANet) is proposed to realize accurate segmentation detection of micro-lenses by employing an embedded multi-scale asymmetric convolutional module (MACM) and a multi-level interactive attention module (MIAM). MACM achieves not only a reduction in computational complexity but also enhanced robustness for rotated image recognition through multi-scale asymmetric convolutional kernels. Furthermore, MIAM improves the accuracy of image segmentation by connecting the down-sampling and up-sampling stages and realizing the fusion of pixel position details and key channel features. Extensive experimental results based on our self-constructed image acquisition system demonstrate that the values of normalized intersection over union and dice are successively 91.41% and 95.50%, and the processing speed is 3.3 s/100 images, which shows the advance of ACMANet.

Fault diagnosis of pump truck waterproof valves using multi-sensor high-dimensional time-domain feature expansion map

The master cylinder of most pump trucks is equipped with a waterproof valve, whose purpose is to prevent water from the tank from entering the master cylinder. Once waterproof valve fails to failure, the waterproof valve at the main cylinder can only be supported by a BS seal (this seal is very easy to fail), which results in oil emulsification and pollution of the hydraulic system. Therefore, a fault diagnosis method combining a multi-sensor high-dimensional time-domain feature expansion map (MHTFEM) with an attentional convolutional capsule network (ACCN) is proposed. In this method, the raw vibration signals acquired by all sensors are first preprocessed to generate a high-dimensional feature matrix. Then the different high-dimensional feature matrices are stitched, expanded and generated into grayscale images, followed by randomly dividing the training set and the testing set. Finally, the training set is brought into the ACCN for training and the testing set is brought into the network model for fault type identification. A test bench was built to confirm the effectiveness of the method for waterproof valve fault diagnosis. This provides a method to achieve intelligent fault diagnosis of construction machinery to ensure its reliability.

Hierarchical Convolutional Attention Network for Depression Detection on Social Media and Its Impact During Pandemic.

People across the globe have felt and are still going through the impact of COVID-19. Some of them share their feelings and suffering online via different online social media networks such as Twitter. Due to strict restrictions to reduce the spread of the novel virus, many people are forced to stay at home, which significantly impacts people's mental health. It is mainly because the pandemic has directly affected the lives of the people who were not allowed to leave home due to strict government restrictions. Researchers must mine the related human-generated data and get insights from it to influence government policies and address people's needs. In this paper, we study social media data to understand how COVID-19 has impacted people's depression. We share a large-scale COVID-19 dataset that can be used to analyze depression. We also have modeled the tweets of depressed and non-depressed users before and after the start of the COVID-19 pandemic. To this end, we developed a new approach based on Hierarchical Convolutional Neural Network (HCN) that extracts fine-grained and relevant content on user historical posts. HCN considers the hierarchical structure of user tweets and contains an attention mechanism that can locate the crucial words and tweets in a user document while also considering the context. Our new approach is capable of detecting depressed users occurring within the COVID-19 time frame. Our results on benchmark datasets show that many non-depressed people became depressed during the COVID-19 pandemic.

Music emotion recognition based on temporal convolutional attention network using EEG.

Music is one of the primary ways to evoke human emotions. However, the feeling of music is subjective, making it difficult to determine which emotions music triggers in a given individual. In order to correctly identify emotional problems caused by different types of music, we first created an electroencephalogram (EEG) data set stimulated by four different types of music (fear, happiness, calm, and sadness). Secondly, the differential entropy features of EEG were extracted, and then the emotion recognition model CNN-SA-BiLSTM was established to extract the temporal features of EEG, and the recognition performance of the model was improved by using the global perception ability of the self-attention mechanism. The effectiveness of the model was further verified by the ablation experiment. The classification accuracy of this method in the valence and arousal dimensions is 93.45% and 96.36%, respectively. By applying our method to a publicly available EEG dataset DEAP, we evaluated the generalization and reliability of our method. In addition, we further investigate the effects of different EEG bands and multi-band combinations on music emotion recognition, and the results confirm relevant neuroscience studies. Compared with other representative music emotion recognition works, this method has better classification performance, and provides a promising framework for the future research of emotion recognition system based on brain computer interface.

Discriminant Decision Making of Cardiovascular Diseases Based on Cloud-Based Convolutional Attention Network

Cardiovascular diseases (CVDs) have become the number one killer affecting human health. In order to reduce the burden of medical workers, facilitate government screening of the population and enable patients to conduct their own health status checks, there is an urgent need for a complementary diagnostic system to predict the occurrence of CVD. In this study, a new cloud-based convolutional attention network (C-CAN) model is proposed for the discriminant decision making of CVD. In this model, the indicator data for discriminant decision making of CVD are trained using an improved one-dimensional convolutional neural network (1D CNN) model structure based on the correlation of factors influencing CVD given by decision-making trial and evaluation laboratory (DEMATEL) and cloud models. This 1D CNN model consists of a convolutional pooling module, an attention module and a fully connected module. The cloud model is used to process the original data based on the discriminating opinion of experts, so as to select the important factors that affect CVD. The attention mechanism is effective in augmenting attention to the essential elements of the data and reducing attention to the less important features. Both have similarities in that they are effective in augmenting the important features in the data and combine with each other to achieve better results. Moreover, the C-CAN is compared with decision tree (DT), [Formula: see text]-nearest neighbors (KNN), random forests (RF) and normal CNN according to the CVD dataset from the Kaggle platform. The results show that the classification accuracy, precision, recall and F1 value of C-CAN are all higher than that of all compared models. Further, the proposed model is further externally validated using other imbalanced datasets, and the results indicate that C-CAN has good resilience for imbalanced data. Our findings suggest that C-CAN represents a promising new approach that may somehow address the challenges associated with deep learning (DL) in the medical field.

PreSubLncR: Predicting Subcellular Localization of Long Non-Coding RNA Based on Multi-Scale Attention Convolutional Network and Bidirectional Long Short-Term Memory Network

The subcellular localization of long non-coding RNA (lncRNA) provides important insights and opportunities for an in-depth understanding of cell biology, revealing disease mechanisms, drug development, and innovation in the biomedical field. Although several computational methods have been proposed to identify the subcellular localization of lncRNA, it is difficult to accurately predict the subcellular localization of lncRNA effectively with these methods. In this study, a new deep-learning predictor called PreSubLncR has been proposed for accurately predicting the subcellular localization of lncRNA. This predictor firstly used the word embedding model word2vec to encode the RNA sequences, and then combined multi-scale one-dimensional convolutional neural networks with attention and bidirectional long short-term memory networks to capture the different characteristics of various RNA sequences. This study used multiple RNA subcellular localization datasets for experimental validation, and the results showed that our method has higher accuracy and robustness compared with other state-of-the-art methods. It is expected to provide more in-depth insights into cell function research.

Mixed Geometry Message and Trainable Convolutional Attention Network for Knowledge Graph Completion

Knowledge graph completion (KGC) aims to study the embedding representation to solve the incompleteness of knowledge graphs (KGs). Recently, graph convolutional networks (GCNs) and graph attention networks (GATs) have been widely used in KGC tasks by capturing neighbor information of entities. However, Both GCNs and GATs based KGC models have their limitations, and the best method is to analyze the neighbors of each entity (pre-validating), while this process is prohibitively expensive. Furthermore, the representation quality of the embeddings can affect the aggregation of neighbor information (message passing). To address the above limitations, we propose a novel knowledge graph completion model with mixed geometry message and trainable convolutional attention network named MGTCA. Concretely, the mixed geometry message function generates rich neighbor message by integrating spatially information in the hyperbolic space, hypersphere space and Euclidean space jointly. To complete the autonomous switching of graph neural networks (GNNs) and eliminate the necessity of pre-validating the local structure of KGs, a trainable convolutional attention network is proposed by comprising three types of GNNs in one trainable formulation. Furthermore, a mixed geometry scoring function is proposed, which calculates scores of triples by novel prediction function and similarity function based on different geometric spaces. Extensive experiments on three standard datasets confirm the effectiveness of our innovations, and the performance of MGTCA is significantly improved compared to the state-of-the-art approaches.

Locational Detection of False Data Injection Attacks in Smart Grids: A Graph Convolutional Attention Network Approach

Locational Detection of False Data Injection Attacks in Smart Grids: A Graph Convolutional Attention Network Approach

Convolutional Attention Network with edge supervision for void and spot segmentation in chip resistors based on X-ray images

The presence of voids inside solder joints in chip resistors can affect the stability and reliability of the product. Therefore, accurate void and spot segmentation is crucial for performance evaluation. In this study, we propose Convolutional Attention Network (CA-net) with edge supervision for void and spot segmentation based on X-ray images of chip resistors. First, CA-net has an encoder-decoder architecture and use skip connections to aggregate low-level features and high-level features. Second, the multi-scale convolutional attention modules (MSCA) are employed at each encoder and decoder to capture multi-scale feature maps of voids and spots. Finally, adding edge-supervised branch to the encoder allows the network to focus more on edge information, so that the response area of the network will be more focused on pads and empty areas. Experimental results show that CA-net outperforms other methods in void and spot segmentation, achieving 96.56 % MIOU and 97.44 % mean DIC. These results indicate that CA-net is a promising approach for accurate void and spot segmentation in chip resistors based on X-ray images.

MINDG: a drug-target interaction prediction method based on an integrated learning algorithm.

Drug-target interaction (DTI) prediction refers to the prediction of whether a given drug molecule will bind to a specific target and thus exert a targeted therapeutic effect. Although intelligent computational approaches for drug target prediction have received much attention and made many advances, they are still a challenging task that requires further research. The main challenges are manifested as follows: (i) most graph neural network-based methods only consider the information of the first-order neighboring nodes (drug and target) in the graph, without learning deeper and richer structural features from the higher-order neighboring nodes. (ii) Existing methods do not consider both the sequence and structural features of drugs and targets, and each method is independent of each other, and cannot combine the advantages of sequence and structural features to improve the interactive learning effect. To address the above challenges, a Multi-view Integrated learning Network that integrates Deep learning and Graph Learning (MINDG) is proposed in this study, which consists of the following parts: (i) a mixed deep network is used to extract sequence features of drugs and targets, (ii) a higher-order graph attention convolutional network is proposed to better extract and capture structural features, and (iii) a multi-view adaptive integrated decision module is used to improve and complement the initial prediction results of the above two networks to enhance the prediction performance. We evaluate MINDG on two dataset and show it improved DTI prediction performance compared to state-of-the-art baselines. https://github.com/jnuaipr/MINDG.