Attention-based Recurrent Neural Network Research Articles

ECG signal reconstruction from PPG using a hybrid attention-based deep learning network

Electrocardiography (ECG) and photoplethysmography (PPG) are non-invasive methods used to quantify signals originating from the cardiovascular system. Although there is a strong correlation between the cycles of the two measures, the correlation between the waveforms has received little attention in research. Measuring the PPG is significantly simpler and more convenient compared to the ECG measure. Recent research has demonstrated that PPG signals can be utilized to rebuild the ECG signals, suggesting that healthcare professionals might acquire a comprehensive comprehension of patients' cardiovascular well-being by only measuring PPG. With the advancement of artificial intelligence, the deep learning model has made significant progress in reconstructing ECG signals from PPG signals. However, the quality of the reconstructed ECG signal is still in need of more enhancement. In order to enhance the quality of the reconstructed ECG signal, we propose an innovative hybrid attention-based bidirectional recurrent neural network (Bi-RNN) that incorporates a dilated convolutional neural network technique for the purpose of reconstructing ECG signals from PPG signals. This addresses issues that occur when standard dilated CNN (DCNN) models neglect the link between circumstances and dispersion of gradients. The suggested approach maximizes the utilization of the DCNN and Bi-RNN unit design to provide fusion features. In order to ensure the model's resilience to deformation, it is imperative to initially extract spatial properties utilizing convolutional neural networks (CNNs). Once we have obtained geographic data, we employ BiLSTM to extract temporal features. The proposed model is so-called hybrid attention-based CNN and BiLSTM (HA-CNN-BiLSTM). BiLSTM mitigates the issues of gradient vanishing and exploding while maintaining accuracy. To showcase the advantages of the suggested model, we conduct a comparison between HA-CNN-BiLSTM and each separate state-of-the-art method. The simulation results indicated that the proposed method yielded superior root mean square error (RMSE) when reconstructing the ECG signal across different optimizers. The results also demonstrated that the proposed method results in the lowest RMSE with SGDM optimizer which is 0.031.

Two-staged attention-based identification of the porosity with the composite features of spatters during the laser powder bed fusion

Porosity is one of the most serious defects in laser powder bed fusion (LPBF). Reducing porosity is essential to improve the mechanical properties of parts in high-end applications. It is found that the spatter dynamic status is closely related to the porosity, giving an idea to identify. In this study, we propose a novel approach for in-situ identification of porosity in the LPBF with spatter features. To achieve efficient and accurate detection, segmentation, and motion tracking of spatters from captured high-speed images during the LPBF process, YOLO and DeepSORT algorithms are employed. Subsequently, a two-staged attention-based recurrent neural network (TARNN) method is proposed to realize the classification of porosity. The input to TARNN consists of both static and dynamic spatter features extracted from every 10 consecutive frames. Leveraging the RNN architecture enables us to effectively exploit temporal information. Moreover, we introduce an attention-aware linear layer and an attention-based RNN to enhance the extraction of representative features related to porosity characteristics. Through the analysis of the attention mechanism, we can explicitly assess the importance ranking of input features, providing a deeper understanding of spatter features. Experimental results demonstrate the superior performance of the proposed TARNN, with an accuracy of 99.50 % at an inference time of 6.5 milliseconds. The proposed method offers a promising avenue for advancing the understanding and characterization of porosity using spatter features in the LPBF process.

Reconstruction of reservoir rock using attention-based convolutional recurrent neural network

The digital reconstruction of reservoir rock or porous media is important as it enables us to visualize and explore their real internal structures. The reservoir rocks (such as sandstone and carbonate) contain both spatial and temporal characteristics, which pose a big challenge in their characterization through routine core analysis or x-ray microcomputer tomography. While x-ray micro-computed tomography gives us three-dimensional images of the porous media, it is often impossible to quantify the variability of the pore, grains, structure, and orientation experimentally. Recently, machine learning has successfully demonstrated the reconstruction ability of reservoir rock images or any porous media. These reservoir rock images are crucial for the digital characterization of the reservoir. We propose a novel algorithm consisting of the convolutional neural network, an attention mechanism, and a recurrent neural network for the reconstruction of reservoir rock or porous media images. The attention-based convolutional recurrent neural network (ACRNN) can reconstruct a representative sample of reservoir rocks. The reconstructed image quality was checked by comparing them with the original Parker sandstone, Leopard sandstone, carbonate shale, Berea sandstone, and sandy medium images. We evaluated the reconstruction by measuring pore and throat properties, two-point probability function, and structural similarity index. Results show that ACRNN can reconstruct reservoir rock or porous media of different scales with approximately the same geometrical, statistical, and topological parameters of the reservoir rock images. This deep learning method is computationally efficient, fast, and reliable for synthetic image realizations. The model was trained and validated on real images, and the reconstructed images showed excellent concordance with the real images having almost the same pore and grain structures. The deep learning-based digital rock reconstruction of reservoir rock or porous media images can aid in rapid image generation to better understand reservoir rock or subsurface formation.

Robust Stochastic Model Predictive Control for Autonomous Vehicle Motion Planning

This work presents a Robust Stochastic Model Predictive Control (RSMPC) framework for real-time motion planning autonomous vehicles, addressing the complex multi-modal vehicle interactions. The proposed framework involves adding expert policy from observations to the dataset and applying the Data Aggregation (DAgger) method to filter unsafe demonstrations and resolve expert conflicts. A Dual-Stage Attention-based Recurrent Neural Network (DA-RNN) model is integrated to predict dual class variables from the dataset, producing a set containing constraints collision-avoidance predicted to be active. The RSMPC framework enhances formulation optimization by eliminating irrelevant collision avoidance constraints, resulting in faster control signals. The framework is applied iteratively, continuously updating observations and solving the RSMPC optimization formulation in real-time. Evaluation of the DA-RNN model achieved a recall value of 0.97 and a high accuracy rate of 98.1% in predicting dual interactions, with a minimal false negative rate of 0.026, highlighting its effectiveness in capturing interaction intricacies. Validated through simulations of interactive traffic intersections, the proposed framework demonstrably excels, showing high feasibility of 99.84% and a 15-fold increase in response speed compared to the baseline. This approach ensures autonomous vehicles navigate safely and efficiently in complex traffic scenarios, paving the way for more reliable and scalable autonomous driving solutions.

Mutation-Based Multivariate Time-Series Anomaly Generation on Latent Space with an Attention-Based Variational Recurrent Neural Network for Robust Anomaly Detection in an Industrial Control System

Anomaly detection involves identifying data that deviates from normal patterns. Two primary strategies are used: one-class classification and binary classification. In Industrial Control Systems (ICS), where anomalies can cause significant damage, timely and accurate detection is essential, often requiring analysis of time-series data. One-class classification is commonly used but tends to have a high false alarm rate. To address this, binary classification is explored, which can better differentiate between normal and anomalous data, though it struggles with class imbalance in ICS datasets. This paper proposes a mutation-based technique for generating ICS time-series anomalies. The method maps ICS time-series data into a latent space using a variational recurrent autoencoder, applies mutation operations, and reconstructs the time-series, introducing plausible anomalies that reflect multivariate correlations. Evaluations of ICS datasets show that these synthetic anomalies are visually and statistically credible. Training a binary classifier on data augmented with these anomalies effectively mitigates the class imbalance problem.

AI-empowered consumer behavior analysis for trustworthy track recommendation over musical dance electronic products

—In the digital music era, accurate and trustworthy track recommendations for musical dance electronic products are becoming increasingly important to improve user experiences and attract more consumers. Consumer behavior modeling is critical in user interest learning and has been extensively used in recommender systems to improve recommendation accuracy. This paper proposes a novel AI-empowered consumer behavior analysis method for trustworthy track recommendations over musical dance electronic products. Specifically, we first model consumer behavior by integrating collaborative filtering and a hidden Markov model to capture the key interactive patterns between consumers and musical dance electronic products. Then, we develop a trustworthy track recommendation method based on multi-layer attention representation learning, which leverages scattering transform for audio preprocessing and attention-based independent recurrent neural networks for encoding user preferences and product features. Extensive experiments on real-world datasets demonstrate the superiority of our proposed method in terms of recommendation accuracy and trustworthiness.

RDGait: A mmWave Based Gait User Recognition System for Complex Indoor Environments Using Single-chip Radar

In this paper, we aim to study millimeter-wave-based gait recognition in complex indoor environments, focusing on dealing with multipath ghosts and supporting rapid deployment to new environments. We design a ghost detection algorithm based on velocity change patterns. This algorithm relies solely on velocity estimation, requiring no environmental priors or multipath modeling. Hence, it is suitable for single-chip millimeter-wave radar with low angular resolution and can be conveniently deployed in new indoor settings. In addition, we build a gait recognition network based on an attention-based Recurrent Neural Network (RNN) to extract spatiotemporal-velocity features from RD heatmaps. We have evaluated RDGait in two scenarios: a corridor scenario and a crowded office scenario, with 125 volunteers of different genders and ages ranging from 6 to 63. RDGait achieves a user recognition accuracy exceeding 95% among 125 candidates in both scenarios. We have further deployed RDGait in two additional scenarios using the pretrain-finetune approach. With minimal user registration data, RDGait could achieve satisfactory (> 90%) recognition accuracy in these new environments considering different radar placements, heights, and number of co-existing users.

ALRN-RCS: Advanced Approach to Network Intrusion Detection Using Attention Long-Term Recurrent Networks and Chaotic Optimization

Detecting intrusions within a network is essential for protecting digital environments; it encompasses the observation and analysis of network traffic to recognize and counteract unauthorized or malicious activities. Conventional methods in network intrusion detection face several challenges such as polymorphic and evasive attacks, scalability issues, and anomaly-based complexity. To address these complexities, this paper proposed a novel method named Attention Long-term Recurrent Network-based Random Chaotic Sine (ALRN-RCS) algorithm for network intrusion detection. In this study, Long Short-Term Memory (LSTM) is utilized to capture complex patterns in network traffic and identify anomalous activities. Also, the attention-based Recurrent Neural Network (RNN) is employed to focus on relevant features within network traffic and enable precise intrusion detection. In this paper, we have incorporated the Chaotic Chimp Sine Cosine optimization algorithm, employing a random update strategy, to optimize hyperparameters and improve the overall efficacy of the proposed approach and the study conducted experiments on the datasets namely the UNSW NB-15, Network Intrusion Detection dataset, and the Segmented Image-based Network Intrusion Detection (SIDD) dataset. Diverse assessment criteria, including accuracy, F1-score, recall, AUC-ROC, and precision, are employed to assess the effectiveness of the ALRN-RCS method and to draw comparisons with established methodologies. The experimental results depict the effectiveness of the ALRN-RCS method for addressing the dynamic and sophisticated nature of modern cyber threats.

Enhancing digital cryptocurrency trading price prediction with an attention-based convolutional and recurrent neural network approach: The case of Ethereum

To predict Ethereum price fluctuations, this study proposes a new two-stage Machine Learning approach using an improved convolutional neural network and a recurrent neural network framework, integrating an attention mechanism-based distribution function algorithm. We construct a dataset and perform model training, fitting, and forecasting. The results indicate that compared with traditional neural networks and time-series models such as GRU and ARIMA, respectively, this approach can effectively use the data information of digital cryptocurrency and improve the prediction accuracy and interpretability of attention-based allocation functions. This study contributes to the literature by offering a new approach for stakeholders.

Attention-based RNN with question-aware loss and multi-level copying mechanism for natural answer generation

Natural answer generation is in a very clear practical significance and strong application background, which can be widely used in the field of knowledge services such as community question answering and intelligent customer service. Traditional knowledge question answering is to provide precise answer entities and neglect the defects; namely, users hope to receive a complete natural answer. In this research, we propose a novel attention-based recurrent neural network for natural answer generation, which is enhanced with multi-level copying mechanisms and question-aware loss. To generate natural answers that conform to grammar, we leverage multi-level copying mechanisms and the prediction mechanism which can copy semantic units and predict common words. Moreover, considering the problem that the generated natural answer does not match the user question, question-aware loss is introduced to make the generated target answer sequences correspond to the question. Experiments on three response generation tasks show our model to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 0.727 BLEU on the SimpleQuestions response generation task, improving over the existing best results by over 0.007 BLEU. Our model has scored a significant enhancement on naturalness with up to 0.05 more than best performing baseline. The simulation results show that our method can generate grammatical and contextual natural answers according to user needs.

TA-RNN: an attention-based time-aware recurrent neural network architecture for electronic health records.

Electronic health records (EHRs) represent a comprehensive resource of a patient's medical history. EHRs are essential for utilizing advanced technologies such as deep learning (DL), enabling healthcare providers to analyze extensive data, extract valuable insights, and make precise and data-driven clinical decisions. DL methods such as recurrent neural networks (RNN) have been utilized to analyze EHR to model disease progression and predict diagnosis. However, these methods do not address some inherent irregularities in EHR data such as irregular time intervals between clinical visits. Furthermore, most DL models are not interpretable. In this study, we propose two interpretable DL architectures based on RNN, namely time-aware RNN (TA-RNN) and TA-RNN-autoencoder (TA-RNN-AE) to predict patient's clinical outcome in EHR at the next visit and multiple visits ahead, respectively. To mitigate the impact of irregular time intervals, we propose incorporating time embedding of the elapsed times between visits. For interpretability, we propose employing a dual-level attention mechanism that operates between visits and features within each visit. The results of the experiments conducted on Alzheimer's Disease Neuroimaging Initiative (ADNI) and National Alzheimer's Coordinating Center (NACC) datasets indicated the superior performance of proposed models for predicting Alzheimer's Disease (AD) compared to state-of-the-art and baseline approaches based on F2 and sensitivity. Additionally, TA-RNN showed superior performance on the Medical Information Mart for Intensive Care (MIMIC-III) dataset for mortality prediction. In our ablation study, we observed enhanced predictive performance by incorporating time embedding and attention mechanisms. Finally, investigating attention weights helped identify influential visits and features in predictions. https://github.com/bozdaglab/TA-RNN.

Knee Angle Estimation from Surface EMG during Walking Using Attention-Based Deep Recurrent Neural Networks: Feasibility and Initial Demonstration in Cerebral Palsy.

Accurately estimating knee joint angle during walking from surface electromyography (sEMG) signals can enable more natural control of wearable robotics like exoskeletons. However, challenges exist due to variability across individuals and sessions. This study evaluates an attention-based deep recurrent neural network combining gated recurrent units (GRUs) and an attention mechanism (AM) for knee angle estimation. Three experiments were conducted. First, the GRU-AM model was tested on four healthy adolescents, demonstrating improved estimation compared to GRU alone. A sensitivity analysis revealed that the key contributing muscles were the knee flexor and extensors, highlighting the ability of the AM to focus on the most salient inputs. Second, transfer learning was shown by pretraining the model on an open source dataset before additional training and testing on the four adolescents. Third, the model was progressively adapted over three sessions for one child with cerebral palsy (CP). The GRU-AM model demonstrated robust knee angle estimation across participants with healthy participants (mean RMSE 7 degrees) and participants with CP (RMSE 37 degrees). Further, estimation accuracy improved by 14 degrees on average across successive sessions of walking in the child with CP. These results demonstrate the feasibility of using attention-based deep networks for joint angle estimation in adolescents and clinical populations and support their further development for deployment in wearable robotics.

Urban energy transition in renewable energy management considering policy and law challenges and opportunities in the pursuit of clean and sustainable urban environments

This research presents a novel approach to enhance sustainable energy development based on wind turbine (WT) output power prediction through the development of a hybrid deep learning model, considering policy and law challenges associated with the problem. The proposed model integrates a Self-Supervised Generative Adversarial Network (GAN) with Attention-based Recurrent Neural Networks (RNNs). The Self-Supervised GAN contributes to improved feature representation learning, while Attention-based RNNs capture temporal dependencies for accurate WT output power forecasting considering political challenges. Additionally, a Modified Adaptive Flower Pollination Algorithm (AFPA) is introduced to optimize the selection of the GAN model structure in view of the legal and political objectives. This adaptive approach refines the architecture of the GAN, enhancing its ability to generate realistic and informative features for the subsequent Attention-based RNNs. The efficacy of the hybrid model is demonstrated through real-world applications in an urban area, emphasizing sensitivity and stability analyses. A thorough investigation into the sensitivity of key model parameters, including the number of hidden units in the Attention-based RNNs and GAN structures, is conducted to fine-tune the model's performance. Stability analysis is employed to assess the robustness of the proposed method under dynamic conditions, ensuring its reliability in practical scenarios. Results indicate that the hybrid model, guided by the Modified AFPA, outperforms existing approaches in WT output power prediction. The proposed methodology showcases the significance of combining state-of-the-art deep learning techniques with adaptive optimization algorithms for accurate and stable predictions in urban energy management applications.

Sentiment analysis of student feedback using attention-based RNN and transformer embedding

Sentiment analysis systems aim to assess people’s opinions across various domains by collecting and categorizing feedback and reviews. In our study, researchers put forward a sentiment analysis system that leverages three distinct embedding techniques: automatic, global vectors (GloVe) for word representation, and bidirectional encoder representations from transformers (BERT). This system features an attention layer, with the best model chosen through rigorous comparisons. In developing the sentiment analysis model, we employed a hybrid dataset comprising students’ feedback and comments. This dataset comprises 3,820 comments, including 2,773 from formal evaluations and 1,047 generated by ChatGPT and prompting engineering. Our main motivation for integrating generative AI was to balance both positive and negative comments. We also explored recurrent neural network (RNN), gated recurrent unit (GRU), long short-term memory (LSTM), and bidirectional long short-term memory (Bi-LSTM), with and without pre-trained GloVe embedding. These techniques produced F-scores ranging from 67% to 69%. On the other hand, the sentiment model based on BERT, particularly its KERAS implementation, achieved higher F-scores ranging from 83% to 87%. The Bi-LSTM architecture outperformed other models and the inclusion of an attention layer further enhanced the performance, resulting in F-scores of 89% and 88% from the Bi-LSTM-BERT sentiment models, respectively.

WIFI based human activity recognition using multi-head adaptive attention mechanism

Human Activity Recognition (HAR) utilizing Channel State Information (CSI) extracted from WiFi signals has garnered substantial interest across various domains and applications. This field’s potential paths and applications extend beyond CSI-based HAR and include smart homes, assisted living, security, gaming, surveillance, and context-aware computing. The ability of deep learning algorithms to effectively process and interpret CSI data opens up new possibilities for accurate and robust human activity recognition in real-world scenarios. However, traditional Recurrent Neural Networks (RNN) models, such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), rely solely on their internal memory cells to maintain information over time. Important details might be diluted or lost within the memory cells in complex CSI sequences. To address this limitation, we propose a lightweight approach that incorporates a multi-head adaptive attention weight mechanism MHAAM into the HAR framework. The multi-head attention mechanism allows the model to attend to different informative patterns within the CSI data simultaneously, capturing fine-grained temporal dependencies and improving the model’s ability to recognize complex activities. The implemented models effectively filter out noise and irrelevant information by assigning higher weights to informative CSI features, further enhancing activity classification accuracy. Experimental evaluations and comparative analyses of HAR for seven activities demonstrate that attention-based RNN models with multi-head attention consistently outperform traditional RNN models. The multi-head attention mechanism achieves improved generalization and testing for seven common human activities and environments, leading to a higher complex human activity classification accuracy of up to 98.5%.

A Temporal Fusion Transformer Model to Forecast Overflow from Sewer Manholes during Pluvial Flash Flood Events

This study employs a temporal fusion transformer (TFT) for predicting overflow from sewer manholes during heavy rainfall events. The TFT utilised is capable of forecasting overflow hydrographs at the manhole level and was tested on a sewer network with 975 manholes. As part of the investigations, the TFT was compared to other deep learning architectures to evaluate its predictive performance. In addition to precipitation measurements and forecasts, the issue of how the additional consideration of measurements in the sewer network as model inputs impacts forecast accuracy was investigated. A varying number of sensors and different measurement signals were compared. The results indicate high performance for the TFT compared to other model architectures like a long short-term memory (LSTM) network or a dual-stage attention-based recurrent neural network (DA-RNN). Additionally, results suggest that considering a single measuring point at the outlet of the sewer network instead of an entire measuring network yields better forecasts. One possible explanation is the high correlation between measurements, which increases model and training complexity without adding much value.

Harnessing Attention-Based Graph Recurrent Neural Networks for Enhanced Conversational Flow Prediction via Conversational Graph Construction

Conversational flow refers to the progression of a conversation, encompassing the arrangement of topics discussed and how responses are delivered. A smooth flow involves participants taking turns to speak and respond naturally and intuitively. Conversely, a more disjointed flow may entail prolonged pauses or difficulties establishing common ground. Numerous factors influence conversation flow, including the personalities of those involved, their familiarity with each other, and the contextual setting. A conversational graph pattern outlines how a conversation typically unfolds or the underlying structure it adheres to. It involves combining different sentence types, the sequential order of topics discussed, and the roles played by different individuals. Predicting subsequent sentences relies on predefined patterns, the context derived from prior conversation flow in the data, and the trained system. The accuracy of sentence predictions varies based on the probability of identifying sentences that fit the subsequent pattern. We employ the Graph Recurrent Neural Network with Attention (GRNNA) model to generate conversational graphs and perform next-sentence prediction. This model constructs a conversational graph using an adjacency matrix, node features (sentences), and edge features (semantic similarity between the sentences). The proposed approach leverages attention mechanisms, recurrent updates, and information aggregation from neighbouring nodes to predict the next node (sentence). The model achieves enhanced predictive capabilities by updating node representations through multiple iterations of message passing and recurrent updates. Experimental results using the conversation dataset demonstrate that the GRNNA model surpasses the Graph Neural Network (GNN) model in next-sentence prediction, achieving an impressive accuracy of 98.89%.

Deep learning assisted solar forecasting for battery swapping stations

ABSTRACT For the development of renewable energy generation as a viable alternative to fossil fuel-based generation, solar power has received widespread acceptance. An increase in solar power usage can mitigate the impact of climate change, energy demand, and cost-effective dispatch. To use solar power efficiently and ensure grid consistency, reliable and accurate forecasting of information becomes very crucial. Hence, the main purpose of this work is to get the best suitable solar irradiance forecasting model and to implement the forecasted solar power to the designed hybrid battery swapping stations. Each year, new techniques and approaches are used to improve the model accuracy with the critical aim of lowering the variability of predictions. This paper takes a close look at different deep-learning methods to predict solar irradiance for a certain period of time. The real-time time series data is used, and for analyzing the time series data, the statistical ARIMA model, LSTM-based RNN technique, and Dual Attention-based Recurrent Neural network have been used. These models are implemented using Jupyter Notebook with Python programming language. To analyze the efficiency and evaluate the performance of real-time data of the models, a comparative study has been done on the different error metrics. This paper reveals that the LSTM model is providing the best solar power forecasting with the least error metrics like MSE of .0091, MAE of .0525, RMSE of .0953, and MSLE of .0047.

Attention-based recurrent neural network for automatic behavior laying hen recognition

Attention-based recurrent neural network for automatic behavior laying hen recognition

Decomposition aided attention-based recurrent neural networks for multistep ahead time-series forecasting of renewable power generation.

Renewable energy plays an increasingly important role in our future. As fossil fuels become more difficult to extract and effectively process, renewables offer a solution to the ever-increasing energy demands of the world. However, the shift toward renewable energy is not without challenges. While fossil fuels offer a more reliable means of energy storage that can be converted into usable energy, renewables are more dependent on external factors used for generation. Efficient storage of renewables is more difficult often relying on batteries that have a limited number of charge cycles. A robust and efficient system for forecasting power generation from renewable sources can help alleviate some of the difficulties associated with the transition toward renewable energy. Therefore, this study proposes an attention-based recurrent neural network approach for forecasting power generated from renewable sources. To help networks make more accurate forecasts, decomposition techniques utilized applied the time series, and a modified metaheuristic is introduced to optimized hyperparameter values of the utilized networks. This approach has been tested on two real-world renewable energy datasets covering both solar and wind farms. The models generated by the introduced metaheuristics were compared with those produced by other state-of-the-art optimizers in terms of standard regression metrics and statistical analysis. Finally, the best-performing model was interpreted using SHapley Additive exPlanations.