Attention-based Deep Neural Network Research Articles

Exposing video surveillance object forgery by combining TSF features and attention-based deep neural networks

Recently, forensics has encountered a new challenge with video surveillance object forgery. This type of forgery combines the characteristics of popular video copy-move and splicing forgeries, failing most existing video forgery detection schemes. In response to this new forgery challenge, this paper proposes a Video Surveillance Object Forgery Detection (VSOFD) method including three parts components: (i) The proposed method presents a special-combined extraction technique that incorporates Temporal-Spatial-Frequent (TSF) perspectives for TSF feature extraction. Furthermore, TSF features can effectively represent video information and benefit from feature dimension reduction, improving computational efficiency. (ii) The proposed method introduces a universal, extensible attention-based Convolutional Neural Network (CNN) baseline for feature processing. This CNN processing architecture is compatible with various series and parallel feed-forward CNN structures, considering these structures as processing backbones. Therefore, the proposed CNN architecture benefits from various state-of-the-art structures, leading to addressing each independent TSF feature. (iii) The method adopts an encoder-attention-decoder RNN framework for feature classification. By incorporating temporal characteristics, the framework can further identify the correlations between the adjacent frames to classify the forgery frames better. Finally, experimental results show that the proposed network can achieve the best F1 = 94.69 % score, increasing at least 5–12 % from the existing State-Of-The-Art (SOTA) VSOFD schemes and other video forensics.

Attention-Based Deep Spiking Neural Networks for Temporal Credit Assignment Problems.

The temporal credit assignment (TCA) problem, which aims to detect predictive features hidden in distracting background streams, remains a core challenge in biological and machine learning. Aggregate-label (AL) learning is proposed by researchers to resolve this problem by matching spikes with delayed feedback. However, the existing AL learning algorithms only consider the information of a single timestep, which is inconsistent with the real situation. Meanwhile, there is no quantitative evaluation method for TCA problems. To address these limitations, we propose a novel attention-based TCA (ATCA) algorithm and a minimum editing distance (MED)-based quantitative evaluation method. Specifically, we define a loss function based on the attention mechanism to deal with the information contained within the spike clusters and use MED to evaluate the similarity between the spike train and the target clue flow. Experimental results on musical instrument recognition (MedleyDB), speech recognition (TIDIGITS), and gesture recognition (DVS128-Gesture) show that the ATCA algorithm can reach the state-of-the-art (SOTA) level compared with other AL learning algorithms.

Evaluating User Engagement in Online News: A Deep Learning Approach Based on Attractiveness and Multiple Features

Online news platforms have become users’ primary information sources. However, they focus on attracting users to click on the news and ignore whether the news triggers a sense of engagement, which could potentially reduce users’ participation in public events. Therefore, this study constructs four indicators by assessing user engagement to build an intelligent system to help platforms optimize their publishing strategies. First, this study defines user engagement evaluation as a classification task that divides user engagement into four indicators and proposes an extended LDA model based on user click–comment behavior (UCCB), using which the attractiveness of words in news headlines and content can be effectively represented. Second, this study proposes a deep user engagement evaluation (DUEE) model that integrates news attractiveness and multiple features in an attention-based deep neural network for user engagement evaluation. The DUEE model considers various elements that collectively determine the ability of the news to attract clicks and engagement. Third, the proposed model is compared with the baseline and state-of-the-art techniques, showing that it outperforms all existing methods. This study provides new research contributions and ideas for improving user engagement in online news evaluation.

DeepCKID: A Multi-Head Attention-Based Deep Neural Network Model Leveraging Classwise Knowledge to Handle Imbalanced Textual Data

This paper presents DeepCKID, a Multi-Head Attention (MHA)-based deep learning model that exploits statistical and semantic knowledge corresponding to documents across different classes in the datasets to improve the model’s ability to detect minority class instances in imbalanced text classification. In this process, corresponding to each document, DeepCKID extracts — (i) word-level statistical and semantic knowledge, namely, class correlation and class similarity corresponding to each word, based on its association with different classes in the dataset, and (ii) class-level knowledge from the document using n-grams and relation triplets corresponding to classwise keywords present, identified using cosine similarity utilizing Transformers-based Pre-trained Language Models (PLMs). DeepCKID encodes the word-level and class-level features using deep convolutional networks, which can learn meaningful patterns from them. At first, DeepCKID combines the semantically meaningful Sentence-BERT document embeddings and word-level feature matrix to give the final document representation, which it further fuses to the different classwise encoded representations to strengthen feature propagation. DeepCKID then passes the encoded document representation and its different classwise representations through an MHA layer to identify the important features at different positions of the feature subspaces, resulting in a latent dense vector accentuating its association with a particular class. Finally, DeepCKID passes the latent vector to the softmax layer to learn the corresponding class label. We evaluate DeepCKID over six publicly available Amazon reviews datasets using four Transformers-based PLMs. We compare DeepCKID with three approaches and four ablation-like baselines. Our study suggests that in most cases, DeepCKID outperforms all the comparison approaches, including baselines.

Enhancing Water-Deficient Potato Plant Identification: Assessing Realistic Performance of Attention-Based Deep Neural Networks and Hyperspectral Imaging for Agricultural Applications.

Hyperspectral imaging has emerged as a pivotal technology in agricultural research, offering a powerful means to non-invasively monitor stress factors, such as drought, in crops like potato plants. In this context, the integration of attention-based deep learning models presents a promising avenue for enhancing the efficiency of stress detection, by enabling the identification of meaningful spectral channels. This study assesses the performance of deep learning models on two potato plant cultivars exposed to water-deficient conditions. It explores how various sampling strategies and biases impact the classification metrics by using a dual-sensor hyperspectral imaging systems (VNIR -Visible and Near-Infrared and SWIR-Short-Wave Infrared). Moreover, it focuses on pinpointing crucial wavelengths within the concatenated images indicative of water-deficient conditions. The proposed deep learning model yields encouraging results. In the context of binary classification, it achieved an area under the receiver operating characteristic curve (AUC-ROC-Area Under the Receiver Operating Characteristic Curve) of 0.74 (95% CI: 0.70, 0.78) and 0.64 (95% CI: 0.56, 0.69) for the KIS Krka and KIS Savinja varieties, respectively. Moreover, the corresponding F1 scores were 0.67 (95% CI: 0.64, 0.71) and 0.63 (95% CI: 0.56, 0.68). An evaluation of the performance of the datasets with deliberately introduced biases consistently demonstrated superior results in comparison to their non-biased equivalents. Notably, the ROC-AUC values exhibited significant improvements, registering a maximum increase of 10.8% for KIS Krka and 18.9% for KIS Savinja. The wavelengths of greatest significance were observed in the ranges of 475-580 nm, 660-730 nm, 940-970 nm, 1420-1510 nm, 1875-2040 nm, and 2350-2480 nm. These findings suggest that discerning between the two treatments is attainable, despite the absence of prominently manifested symptoms of drought stress in either cultivar through visual observation. The research outcomes carry significant implications for both precision agriculture and potato breeding. In precision agriculture, precise water monitoring enhances resource allocation, irrigation, yield, and loss prevention. Hyperspectral imaging holds potential to expedite drought-tolerant cultivar selection, thereby streamlining breeding for resilient potatoes adaptable to shifting climates.

Knee Angle Estimation from Surface EMG during Walking Using Attention-Based Deep Recurrent Neural Networks: Feasibility and Initial Demonstration in Cerebral Palsy.

Accurately estimating knee joint angle during walking from surface electromyography (sEMG) signals can enable more natural control of wearable robotics like exoskeletons. However, challenges exist due to variability across individuals and sessions. This study evaluates an attention-based deep recurrent neural network combining gated recurrent units (GRUs) and an attention mechanism (AM) for knee angle estimation. Three experiments were conducted. First, the GRU-AM model was tested on four healthy adolescents, demonstrating improved estimation compared to GRU alone. A sensitivity analysis revealed that the key contributing muscles were the knee flexor and extensors, highlighting the ability of the AM to focus on the most salient inputs. Second, transfer learning was shown by pretraining the model on an open source dataset before additional training and testing on the four adolescents. Third, the model was progressively adapted over three sessions for one child with cerebral palsy (CP). The GRU-AM model demonstrated robust knee angle estimation across participants with healthy participants (mean RMSE 7 degrees) and participants with CP (RMSE 37 degrees). Further, estimation accuracy improved by 14 degrees on average across successive sessions of walking in the child with CP. These results demonstrate the feasibility of using attention-based deep networks for joint angle estimation in adolescents and clinical populations and support their further development for deployment in wearable robotics.

TEE4EHR: Transformer event encoder for better representation learning in electronic health records

Irregular sampling of time series in electronic health records (EHRs) is one of the main challenges for developing machine learning models. Additionally, the pattern of missing values in certain clinical variables is not at random but depends on the decisions of clinicians and the state of the patient. Point process is a mathematical framework for analyzing event sequence data consistent with irregular sampling patterns. Our model, TEE4EHR, is a transformer event encoder (TEE) with point process loss that encodes the pattern of laboratory tests in EHRs. The utility of our TEE has been investigated in various benchmark event sequence datasets. Additionally, we conduct experiments on two real-world EHR databases to provide a more comprehensive evaluation of our model. Firstly, in a self-supervised learning approach, the TEE is jointly learned with an existing attention-based deep neural network, which gives superior performance in negative log-likelihood and future event prediction. Besides, we propose an algorithm for aggregating attention weights to reveal the events’ interactions. Secondly, we transfer and freeze the learned TEE to the downstream task for the outcome prediction, where it outperforms state-of-the-art models for handling irregularly sampled time series. Furthermore, our results demonstrate that our approach can improve representation learning in EHRs and be useful for clinical prediction tasks.

CBAR-UNet: A novel methodology for segmentation of cardiac magnetic resonance images using block attention-based deep residual neural network

CBAR-UNet: A novel methodology for segmentation of cardiac magnetic resonance images using block attention-based deep residual neural network

Aspect-Based Sentiment Analysis of Specific Targets

Different from traditional sentiment analysis tasks, Aspect-Based Sentiment Analysis (ABSA) aims to automatically identify the emotional polarity of different aspects in the same sentence, which helps to mine users' more delicate emotional expressions for different targets and has become a research hotspot in the field of natural language processing in recent years. Thanks to the rapid development of attention-based deep neural network models, the accuracy of aspect sentiment analysis has continuously made breakthroughs. However, existing works pay little attention to the performance bounds of different application scenarios, such as different text topics or text lengths. In response to the above issues, this paper conducts a comparative analysis of several classic end-to-end neural network models to explore topic-level sentiment polarity analysis. By quantitatively evaluating the accuracy and F1 score of each model at different fixed text lengths, this paper summarizes the strengths, weaknesses, and different aspects of each model's efficiency in use. In addition, this paper also discusses the existing problems with these models and puts forward suggestions for improvement, which can provide some new insights into the research of ABSA.

Unmanned aerial vehicle–human collaboration route planning for intelligent infrastructure inspection

AbstractMotivated by the strengths of unmanned aerial vehicle (UAV), the UAV–human collaboration route planning (UHCRP) for intelligent infrastructure inspection is a problem worthy of discussion to help reduce human costs and minimize the risk of noninspected infrastructures under limited resources. To facilitate UHCRP, this paper proposes a novel deep reinforcement learning (DRL)‐based approach to well handle multi‐source uncertain features and constraints at a fast speed. To begin with, UHCRP is mathematically described and reformulated as a dual interdependent deep reinforcement learning (diDRL) framework to reflect real‐world scenarios. Afterward, a novel policy network named the attention‐based deep neural network (A‐DNN) is introduced to learn the route planning decisions for the combinatorial optimization problem. In particular, A‐DNN is made up of an encoder and a dual decoder for UAV and human inspection, where the multi‐head attention mechanism is incorporated to generate richer representations for model performance improvement. Performance of the proposed dual multi‐head attention model (DAM) has been tested in simulations and a real‐world case study regarding wind farm inspection. Results indicate that DAM under the sampling decoding strategy can deliver a high‐quality path plan and show better generalizability for larger scale problem sizes compared to single‐head attention model (SAM), multi‐head attention model (AM), and two baseline models, namely OR‐Tools and genetic algorithm. Moreover, DAM trained by randomly generated data can be directly employed to solve the practical problem with standardization of inputs. Overall, DRL integrates decision‐making for inspection method selection and inspected infrastructure selection, providing adaptive and intelligent inspection path planning for UAV and human in complex and dynamic engineering environments.

Link-Based Xcorr Normalization and Attention Mechanism for Predicting the Threats over the Network Model

Sensor Networks (SNs) play an essential role in upcoming technologies like the Internet of Things (IoT), where technical services are highly prone to crucial vulnerability due to attacks. This research motivates to provide a mechanism to identify the link reliability of connected sensor nodes. The privacy-preserving keys are distributed among the corresponding network nodes. When the nodes suffer from an attack, it damages the linking nodes' community. It has the nature of healing itself when the attacks are identified over the network. The self-healing nature is not so complex, and it is termed a lightweight process. A novel link-based intrusion prediction mechanism uses attention-based Deep Neural Networks (-DNN) for lightweight linkage identification and labelling. This model helps predict basic network patterns using topological analysis with better generalization. The simulation is done with Python where the proposed -DNN model outperforms the five different conventional approaches with the adoption of a benchmark dataset (network traffic) for extensive analysis. The AUC is improved in an average manner with the adoption of -DNN. This model enhances the linkage connectivity to make different connectivity processes more efficient and reach the target non-convincing. It is sensed that the proposed -DNN outperforms the existing approaches by improving the network resilience by maintaining higher energy efficiency.

GADNN: A graph attention-based method for drug-drug association prediction considering the contribution rate of different types of drug-related features

The simultaneous use of multiple drugs, known as drug combinations, is increasingly common in the treatment of complex diseases such as cancer. However, drug-drug interactions (DDIs) caused by the simultaneous use of multiple drugs can lead to unexpected side effects and even death, resulting in high medical costs. Therefore, understanding potential DDI is an essential step in reducing the risk of adverse drug reactions before administering clinical drug compounds. Recently, many machine learning-based methods for DDI prediction have been introduced, but in these methods, no attention is paid to determining the contribution rate of each type of drug feature. Here, a Graph Attention-based Deep Neural Network method called GADNN is proposed to predict DDIs based on considering the influence or contribution rate of different drug-related features. The introduced method consists of two stages. In the first stage, four basic drug datasets, including target, enzyme, infrastructure, and pathway, are used as types of drug features. Using a graph neural network-based method, the embedding vector is generated for each drug based on each dataset separately. In the second stage, the contribution coefficient of each dataset is calculated dynamically through a new graph attention mechanism. Then, the weighted combination of the separate vectors is used to predict the probability of drug-drug interactions using a dense neural network. The experimental results confirmed that the presented innovation has been able to improve the accuracy compared to state-of-the-art methods.

CREaTor: zero-shot cis-regulatory pattern modeling with attention mechanisms

Linking cis-regulatory sequences to target genes has been a long-standing challenge. In this study, we introduce CREaTor, an attention-based deep neural network designed to model cis-regulatory patterns for genomic elements up to 2 Mb from target genes. Coupled with a training strategy that predicts gene expression from flanking candidate cis-regulatory elements (cCREs), CREaTor can model cell type-specific cis-regulatory patterns in new cell types without prior knowledge of cCRE-gene interactions or additional training. The zero-shot modeling capability, combined with the use of only RNA-seq and ChIP-seq data, allows for the ready generalization of CREaTor to a broad range of cell types.

High-precision cutterhead torque prediction for tunnel boring machines using an attention-based embedded LSTM neural network

For avoiding the serious accident caused by the cutterhead jamming of tunnel boring machines, high-precise cutterhead torque (CHT) prediction has important implications for adjusting operational parameters. For accurately predicting the CHT, a novel CHT prediction approach is presented. Firstly, correlation analysis based on cosine similarity are performed to select the monitoring data highly correlated to CHT. Secondly, a cell state extraction module (CSEM) is designed to replace the saturation-prone forget gate for reducing the saturation rate of forget gate. With CSEM, a novel embedded long short-term memory (ELSTM) network with dual-memory structure is proposed and its state transition updating formulas are derived. Lastly, an attention-based deep neural network is designed for fusing the sequential features learned by ELSTM at all instants into the predicted CHTs. Compared with the current approaches, the contrast experiments on an in-situ datasets show that the proposed method has stronger predictive ability under various geological conditions.

A Novel Discriminative Enhancement Method for Few-Shot Remote Sensing Image Scene Classification

Remote sensing image scene classification (RSISC) has garnered significant attention in recent years. Numerous methods have been put forward in an attempt to tackle this issue, particularly leveraging deep learning methods that have shown promising performance in classifying remote sensing image (RSI). However, it is widely recognized that deep learning methods typically require a substantial amount of labeled data to effectively converge. Acquiring a sufficient quantity of labeled data often necessitates significant human and material resources. Hence, few-shot RSISC has become highly meaningful. Fortunately, the recently proposed deep nearest neighbor neural network based on the attention mechanism (DN4AM) model incorporates episodic training and class-related attention mechanisms, effectively reducing the impact of background noise regions on classification results. Nevertheless, the DN4AM model does not address the problem of significant intra-class variability and substantial inter-class similarities observed in RSI scenes. Therefore, the discriminative enhanced attention-based deep nearest neighbor neural network (DEADN4) is proposed to address the few-shot RSISC task. Our method makes three contributions. Firstly, we introduce center loss to enhance the intra-class feature compactness. Secondly, we utilize the deep local-global descriptor (DLGD) to increase inter-class feature differentiation. Lastly, we modify the Softmax loss by incorporating cosine margin to amplify the inter-class feature dissimilarity. Experiments are conducted on three diverse RSI datasets to gauge the efficacy of our approach. Through comparative analysis with various cutting-edge methods including MatchingNet, RelationNet, MAML, Meta-SGD, DN4, and DN4AM, our approach showcases promising outcomes in the few-shot RSISC task.

AttenSyn: An Attention-Based Deep Graph Neural Network for Anticancer Synergistic Drug Combination Prediction.

Identifying synergistic drug combinations is fundamentally important to treat a variety of complex diseases while avoiding severe adverse drug-drug interactions. Although several computational methods have been proposed, they highly rely on handcrafted feature engineering and cannot learn better interactive information between drug pairs, easily resulting in relatively low performance. Recently, deep-learning methods, especially graph neural networks, have been widely developed in this area and demonstrated their ability to address complex biological problems. In this study, we proposed AttenSyn, an attention-based deep graph neural network for accurately predicting synergistic drug combinations. In particular, we adopted a graph neural network module to extract high-latent features based on the molecular graphs only and exploited the attention-based pooling module to learn interactive information between drug pairs to strengthen the representations of drug pairs. Comparative results on the benchmark datasets demonstrated that our AttenSyn performs better than the state-of-the-art methods in the prediction of anticancer synergistic drug combinations. Additionally, to provide good interpretability of our model, we explored and visualized some crucial substructures in drugs through attention mechanisms. Furthermore, we also verified the effectiveness of our proposed AttenSyn on two cell lines by visualizing the features of drug combinations learnt from our model, exhibiting satisfactory generalization ability.

Modification and Evaluation of Attention-Based Deep Neural Network for Structural Crack Detection.

Cracks are one of the safety-evaluation indicators for structures, providing a maintenance basis for the health and safety of structures in service. Most structural inspections rely on visual observation, while bridges rely on traditional methods such as bridge inspection vehicles, which are inefficient and pose safety risks. To alleviate the problem of low efficiency and the high cost of structural health monitoring, deep learning, as a new technology, is increasingly being applied to crack detection and recognition. Focusing on this, the current paper proposes an improved model based on the attention mechanism and the U-Net network for crack-identification research. First, the training results of the two original models, U-Net and lrassp, were compared in the experiment. The results showed that U-Net performed better than lrassp according to various indicators. Therefore, we improved the U-Net network with the attention mechanism. After experimenting with the improved network, we found that the proposed ECA-UNet network increased the Intersection over Union (IOU) and recall indicators compared to the original U-Net network by 0.016 and 0.131, respectively. In practical large-scale structural crack recognition, the proposed model had better recognition performance than the other two models, with almost no errors in identifying noise under the premise of accurately identifying cracks, demonstrating a stronger capacity for crack recognition.

Generating multi-level explanations for process outcome predictions

Process mining focuses on the analysis of event log data to build various process analytical capabilities. Predictive process analytics has emerged as one of such key capabilities and it uses machine learning techniques to construct process prediction models. In recent years, deep neural networks have gained increasing interest in process prediction since they can handle multi-dimensional sequential inputs with minimal information loss. However, they are considered black-box models and existing studies in explaining deep neural network-based process predictions rely on only event-level features for explanation. In this paper, we propose a new approach for generating explanations for process outcome predictions at multiple levels. The approach is underpinned by three different prediction models: a transparent model for generating global explanations based on case-level features, an attention-based deep neural network for generating local explanations based on event-level features, and a novel eXplainable Dual-learning Deep network (XD2-net) for generating local explanations based on case-level features. Using three publicly available datasets, we have tested the applicability of the approach and further examined the multi-level explanations generated by the approach through an elaborate case study. Unlike others, the design of our approach promotes the idea of leveraging the complementary capabilities of different models and utilizing their strengths, rather than focusing on model performance competition. This will contribute towards generating more comprehensive explanations that meet the needs of different end users and purposes in the future.

TransSleep: Transitioning-Aware Attention-Based Deep Neural Network for Sleep Staging.

Sleep staging is essential for sleep assessment and plays a vital role as a health indicator. Many recent studies have devised various machine/deep learning methods for sleep staging. However, two key challenges hinder the practical use of those methods: 1) effectively capturing salient waveforms in sleep signals and 2) correctly classifying confusing stages in transitioning epochs. In this study, we propose a novel deep neural-network structure, TransSleep, that captures distinctive local temporal patterns and distinguishes confusing stages using two auxiliary tasks. In particular, TransSleep captures salient waveforms in sleep signals by an attention-based multiscale feature extractor and correctly classifies confusing stages in transitioning epochs, while modeling contextual relationships with two auxiliary tasks. Results show that TransSleep achieves promising performance in automatic sleep staging. The validity of TransSleep is demonstrated by its state-of-the-art performance on two publicly available datasets: 1) Sleep-EDF and 2) MASS. Furthermore, we performed ablations to analyze our results from different perspectives. Based on our overall results, we believe that TransSleep has immense potential to provide new insights into deep-learning-based sleep staging.

Cascaded Temporal and Spatial Attention Network for solar adaptive optics image restoration

Context. Atmospheric turbulence severely degrades the quality of images observed through a ground-based telescope. An adaptive optics (AO) system only partially improves the image quality by correcting certain level wavefronts, making post-facto image processing necessary. Several deep learning-based methods have recently been applied in solar AO image post-processing. However, further research is still needed to get better images while enhancing model robustness and using inter-frame and intra-frame information. Aims. We propose an end-to-end network that can better handle solar adaptive image anisoplanatism by leveraging attention mechanisms, pixel-wise filters, and cascaded architecture. Methods. We developed a cascaded attention-based deep neural network named Cascaded Temporal and Spatial Attention Network (CTSAN) for solar AO image restoration. CTSAN consists of four modules: optical flow estimation PWC-Net for inter-frame explicit alignment, temporal and spatial attention for dynamic feature fusion, temporal sharpness prior for sharp feature extraction, and encoder-decoder architecture for feature reconstruction. We also used a hard example mining strategy to create a loss function in order to focus on the regions that are difficult to restore, and a cascaded architecture to further improve model stability. Results. CTSAN and the other two state-of-the-art (SOTA) supervised learning methods for solar AO image restoration are trained on real 705 nm photospheric and 656 nm chromospheric AO images supervised by corresponding Speckle images. Then all the methods are quantitatively and qualitatively evaluated on five real testing sets. Compared to the other two SOTA methods, CTSAN can restore clearer solar images, and shows better stability and generalization performance when restoring the lowest contrast AO image.