Attention-based Recurrent Neural Network Research Articles

Sentiment analysis of student feedback using attention-based RNN and transformer embedding

Sentiment analysis systems aim to assess people’s opinions across various domains by collecting and categorizing feedback and reviews. In our study, researchers put forward a sentiment analysis system that leverages three distinct embedding techniques: automatic, global vectors (GloVe) for word representation, and bidirectional encoder representations from transformers (BERT). This system features an attention layer, with the best model chosen through rigorous comparisons. In developing the sentiment analysis model, we employed a hybrid dataset comprising students’ feedback and comments. This dataset comprises 3,820 comments, including 2,773 from formal evaluations and 1,047 generated by ChatGPT and prompting engineering. Our main motivation for integrating generative AI was to balance both positive and negative comments. We also explored recurrent neural network (RNN), gated recurrent unit (GRU), long short-term memory (LSTM), and bidirectional long short-term memory (Bi-LSTM), with and without pre-trained GloVe embedding. These techniques produced F-scores ranging from 67% to 69%. On the other hand, the sentiment model based on BERT, particularly its KERAS implementation, achieved higher F-scores ranging from 83% to 87%. The Bi-LSTM architecture outperformed other models and the inclusion of an attention layer further enhanced the performance, resulting in F-scores of 89% and 88% from the Bi-LSTM-BERT sentiment models, respectively.

WIFI based human activity recognition using multi-head adaptive attention mechanism

Human Activity Recognition (HAR) utilizing Channel State Information (CSI) extracted from WiFi signals has garnered substantial interest across various domains and applications. This field’s potential paths and applications extend beyond CSI-based HAR and include smart homes, assisted living, security, gaming, surveillance, and context-aware computing. The ability of deep learning algorithms to effectively process and interpret CSI data opens up new possibilities for accurate and robust human activity recognition in real-world scenarios. However, traditional Recurrent Neural Networks (RNN) models, such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), rely solely on their internal memory cells to maintain information over time. Important details might be diluted or lost within the memory cells in complex CSI sequences. To address this limitation, we propose a lightweight approach that incorporates a multi-head adaptive attention weight mechanism MHAAM into the HAR framework. The multi-head attention mechanism allows the model to attend to different informative patterns within the CSI data simultaneously, capturing fine-grained temporal dependencies and improving the model’s ability to recognize complex activities. The implemented models effectively filter out noise and irrelevant information by assigning higher weights to informative CSI features, further enhancing activity classification accuracy. Experimental evaluations and comparative analyses of HAR for seven activities demonstrate that attention-based RNN models with multi-head attention consistently outperform traditional RNN models. The multi-head attention mechanism achieves improved generalization and testing for seven common human activities and environments, leading to a higher complex human activity classification accuracy of up to 98.5%.

A Temporal Fusion Transformer Model to Forecast Overflow from Sewer Manholes during Pluvial Flash Flood Events

This study employs a temporal fusion transformer (TFT) for predicting overflow from sewer manholes during heavy rainfall events. The TFT utilised is capable of forecasting overflow hydrographs at the manhole level and was tested on a sewer network with 975 manholes. As part of the investigations, the TFT was compared to other deep learning architectures to evaluate its predictive performance. In addition to precipitation measurements and forecasts, the issue of how the additional consideration of measurements in the sewer network as model inputs impacts forecast accuracy was investigated. A varying number of sensors and different measurement signals were compared. The results indicate high performance for the TFT compared to other model architectures like a long short-term memory (LSTM) network or a dual-stage attention-based recurrent neural network (DA-RNN). Additionally, results suggest that considering a single measuring point at the outlet of the sewer network instead of an entire measuring network yields better forecasts. One possible explanation is the high correlation between measurements, which increases model and training complexity without adding much value.

Harnessing Attention-Based Graph Recurrent Neural Networks for Enhanced Conversational Flow Prediction via Conversational Graph Construction

Conversational flow refers to the progression of a conversation, encompassing the arrangement of topics discussed and how responses are delivered. A smooth flow involves participants taking turns to speak and respond naturally and intuitively. Conversely, a more disjointed flow may entail prolonged pauses or difficulties establishing common ground. Numerous factors influence conversation flow, including the personalities of those involved, their familiarity with each other, and the contextual setting. A conversational graph pattern outlines how a conversation typically unfolds or the underlying structure it adheres to. It involves combining different sentence types, the sequential order of topics discussed, and the roles played by different individuals. Predicting subsequent sentences relies on predefined patterns, the context derived from prior conversation flow in the data, and the trained system. The accuracy of sentence predictions varies based on the probability of identifying sentences that fit the subsequent pattern. We employ the Graph Recurrent Neural Network with Attention (GRNNA) model to generate conversational graphs and perform next-sentence prediction. This model constructs a conversational graph using an adjacency matrix, node features (sentences), and edge features (semantic similarity between the sentences). The proposed approach leverages attention mechanisms, recurrent updates, and information aggregation from neighbouring nodes to predict the next node (sentence). The model achieves enhanced predictive capabilities by updating node representations through multiple iterations of message passing and recurrent updates. Experimental results using the conversation dataset demonstrate that the GRNNA model surpasses the Graph Neural Network (GNN) model in next-sentence prediction, achieving an impressive accuracy of 98.89%.

Deep learning assisted solar forecasting for battery swapping stations

ABSTRACT For the development of renewable energy generation as a viable alternative to fossil fuel-based generation, solar power has received widespread acceptance. An increase in solar power usage can mitigate the impact of climate change, energy demand, and cost-effective dispatch. To use solar power efficiently and ensure grid consistency, reliable and accurate forecasting of information becomes very crucial. Hence, the main purpose of this work is to get the best suitable solar irradiance forecasting model and to implement the forecasted solar power to the designed hybrid battery swapping stations. Each year, new techniques and approaches are used to improve the model accuracy with the critical aim of lowering the variability of predictions. This paper takes a close look at different deep-learning methods to predict solar irradiance for a certain period of time. The real-time time series data is used, and for analyzing the time series data, the statistical ARIMA model, LSTM-based RNN technique, and Dual Attention-based Recurrent Neural network have been used. These models are implemented using Jupyter Notebook with Python programming language. To analyze the efficiency and evaluate the performance of real-time data of the models, a comparative study has been done on the different error metrics. This paper reveals that the LSTM model is providing the best solar power forecasting with the least error metrics like MSE of .0091, MAE of .0525, RMSE of .0953, and MSLE of .0047.

Attention-based recurrent neural network for automatic behavior laying hen recognition

Attention-based recurrent neural network for automatic behavior laying hen recognition

Decomposition aided attention-based recurrent neural networks for multistep ahead time-series forecasting of renewable power generation.

Renewable energy plays an increasingly important role in our future. As fossil fuels become more difficult to extract and effectively process, renewables offer a solution to the ever-increasing energy demands of the world. However, the shift toward renewable energy is not without challenges. While fossil fuels offer a more reliable means of energy storage that can be converted into usable energy, renewables are more dependent on external factors used for generation. Efficient storage of renewables is more difficult often relying on batteries that have a limited number of charge cycles. A robust and efficient system for forecasting power generation from renewable sources can help alleviate some of the difficulties associated with the transition toward renewable energy. Therefore, this study proposes an attention-based recurrent neural network approach for forecasting power generated from renewable sources. To help networks make more accurate forecasts, decomposition techniques utilized applied the time series, and a modified metaheuristic is introduced to optimized hyperparameter values of the utilized networks. This approach has been tested on two real-world renewable energy datasets covering both solar and wind farms. The models generated by the introduced metaheuristics were compared with those produced by other state-of-the-art optimizers in terms of standard regression metrics and statistical analysis. Finally, the best-performing model was interpreted using SHapley Additive exPlanations.

Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition.

Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network-time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31-50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

A neural network encoder-decoder for time series prediction: Application on 137Cs particulate concentrations in nuclearized rivers

Monitoring the impact of human activities on the environment is a major challenge as many pollutants can be found in the different ecosystems. This is the case of the caesium-137 that has been present in the environment for many decades as a result of atmospheric tests, accidents such as Chernobyl and release from nuclear industries. With the recent advance in data-driven models, this study evaluate the relevance of a deep learning tool for reconstructing caesium-137 chronics particulate concentration in rivers. An encoder-decoder neural network, “Hierarchical Attention-Based Recurrent Highway Networks”(HRHN), is proposed notably for its ability to extract the most relevant temporal and spatial information from the databases. Three monitoring stations were studied, one on the Rhône River and two on the Loire River, all of them downstream nuclear industries in these catchments affected by the global fallout and the accident of Chernobyl. The objective is to predict the future concentration from a set of variables providing past information on water discharge, washout flux and industrial radioactive releases. Once optimised, the model generates first results in agreement with the real concentration curves by correctly following the main trends, with a NSE of 0.89, 0.53 and 0.35 respectively for the Rhone station and the two stations on the Loire. The main reason of inaccuracies is due to the quantity of data available. The originality of this model is its capacity to make predictions on different catchment areas. In fact the training was conducted on the Rhône station as the range of the concentration was higher (from 265.4 to 2700.0 Bq/kg) and the testing on the two Loire station. Another encoder-decoder model DA-RNN (Dual-Stage Attention-Based Recurrent Neural Network) was also evaluate in order to compare the performance of an alternative architecture, without convolution layer. The conclusion is that HRHN remains more powerful in the predictions on the 3 systems. With these first interesting results for HRHN, further investigations should be taken into account for other pollutants than caesium-137 to better understand the robustness of the model.

Explainable Artificial Intelligence and Natural Language Processing for Unraveling Deceptive Contents

Deceptive content recognition in social media employing artificial intelligence (AI) includes the use of sophisticated techniques and machine learning (ML) methods to recognize deceptive or wrong data shared on numerous platforms. AI methods analyse textual as well as multimedia content, investigative patterns, linguistic cues, and contextual info to flag latent cases of deception. As a result of the use of natural language processing (NLP) and computer vision (CV), these systems identify subtle nuances, misrepresentation strategies, and anomalies in user-generated content. This active technique permits social media platforms, organizations, and consumers to recognize and diminish the spread of deceptive content, donates to a more reliable online atmosphere, and aids in fighting tasks modelled by misinformation and false news. This study offers a novel sine cosine algorithm with deep learning-based deceptive content detection on social media (SCADL-DCDSM) technique. The SCADL-DCDSM technique incorporates the ensemble learning process with a hyperparameter tuning strategy for classifying the sentiments. Primarily, the SCADL-DCDSM technique pre-processes the input data to change the input data into a valuable format. Moreover, the SCADL-DCDSM algorithm follows the BERT model for the word embedding process. Moreover, the SCADL-DCDSM technique involves an ensemble of three models for sentiment classification such as long short-term memory (LSTM), extreme learning machine (ELM), and attention-based recurrent neural network (ARNN). Finally, SCA can be executed for better hyperparameter choice of the DL models. The SCADL-DCDSM system integrates the explainable artificial intelligence (XAI) system LIME has been employed for a comprehensive explainability and understanding of the black-box process, enhancing correct deceptive content recognition. The simulation result analysis of the SCADL-DCDSM algorithm has been examined on a benchmark database. The simulation outcome illustrated that the SCADL-DCDSM methodology achieves optimum solution than other approaches in terms of different measures.

Research on workflow recognition for liver rupture repair surgery.

Liver rupture repair surgery serves as one tool to treat liver rupture, especially beneficial for cases of mild liver rupture hemorrhage. Liver rupture can catalyze critical conditions such as hemorrhage and shock. Surgical workflow recognition in liver rupture repair surgery videos presents a significant task aimed at reducing surgical mistakes and enhancing the quality of surgeries conducted by surgeons. A liver rupture repair simulation surgical dataset is proposed in this paper which consists of 45 videos collaboratively completed by nine surgeons. Furthermore, an end-to-end SA-RLNet, a self attention-based recurrent convolutional neural network, is introduced in this paper. The self-attention mechanism is used to automatically identify the importance of input features in various instances and associate the relationships between input features. The accuracy of the surgical phase classification of the SA-RLNet approach is 90.6%. The present study demonstrates that the SA-RLNet approach shows strong generalization capabilities on the dataset. SA-RLNet has proved to be advantageous in capturing subtle variations between surgical phases. The application of surgical workflow recognition has promising feasibility in liver rupture repair surgery.

Election-based optimization algorithm with deep learning-enabled false data injection attack detection in cyber-physical systems

<abstract> <p>Cyber-physical systems (CPSs) are affected by cyberattacks once they are more connected to cyberspace. Advanced CPSs are highly complex and susceptible to attacks such as false data injection attacks (FDIA) targeted to mislead the systems and make them unstable. Leveraging an integration of anomaly detection methods, real-time monitoring, and machine learning (ML) algorithms, research workers are developing robust frameworks to recognize and alleviate the effect of FDIA. These methods often scrutinize deviations from predictable system behavior, using statistical analysis and anomaly detection systems to determine abnormalities that can indicate malicious activities. This manuscript offers the design of an election-based optimization algorithm with a deep learning-enabled false data injection attack detection (EBODL-FDIAD) method in the CPS infrastructure. The purpose of the EBODL-FDIAD technique is to enhance security in the CPS environment via the detection of FDIAs. In the EBODL-FDIAD technique, the linear scaling normalization (LSN) approach can be used to scale the input data into valuable formats. Besides, the EBODL-FDIAD system performs ensemble learning classification comprising three classifiers, namely the kernel extreme learning machine (KELM), long short-term memory (LSTM), and attention-based bidirectional recurrent neural network (ABiRNN) model. For optimal hyperparameter selection of the ensemble classifiers, the EBO algorithm can be applied. To validate the enriched performance of the EBODL-FDIAD technique, wide-ranging simulations were involved. The extensive results highlighted that the EBODL-FDIAD algorithm performed well over other systems concerning numerous measures.</p> </abstract>

An Attention-Based Convolutional Recurrent Neural Networks for Scene Text Recognition

An Attention-Based Convolutional Recurrent Neural Networks for Scene Text Recognition

Factorized Recurrent Neural Network with Attention for Language Identification and Content Detection

Language identification and content detection are essential for ensuring effective digital communication, and content moderation. While extensive research has primarily focused on well-known and widely spoken languages, challenges persist when dealing with indigenous and resource-limited languages, especially between closely similar languages such as Ethiopian languages. This article aims to simultaneously identify the language of a given text and detect its content, and to achieve this, we propose a novel attention-based recurrent neural network framework. The proposed method has an attention-embedded Bidirectional-LSTM architecture with two classifiers that identify the language of a given text and content within the text. The two classifiers share a common feature space before they branched at their task-specific layers where both layers are assisted by attention mechanism. We use five different topics in Six Ethiopian Languages the dataset consists of nearly 22,624 sentences. We compared our result with the classical NLP techniques, the proposed method shortened the data prepossessing steps. We evaluated the model performance using the accuracy metric, achieving results of 98.88% for language identification and 96.5% for text content detection. The dataset, source code, and pretrained model are available at https://github.com/bdu-birhanu/LID_TCD .

Strategic Implicit Balancing With Energy Storage Systems via Stochastic Model Predictive Control

Battery Energy Storage Systems (BESS) may exploit the increasing price volatility in imbalance settlement mechanisms via inter-temporal arbitrage. However, participating in these markets requires a careful trade-off between expected profits, accounting for the impact of BESS actions on prevailing imbalance prices, the financial risks and the incurred battery degradation costs. This paper introduces a novel forecast-informed Model Predictive Control (MPC) methodology in which a strategic and potentially risk-averse BESS performs implicit balancing by taking out-of-balance positions in near-real time. Thereby it anticipates expected imbalance prices in a European-style balancing market, and takes into account state of charge-dependent battery degradation costs. To this end, an attention-based recurrent neural network forecasting technique is leveraged to predict the System Imbalance. The proposed methodology is tested on a real-life case study of the Belgian balancing market. Expected profits of a 2 MW/2 MWh BESS (21,784 MW/month) are shown to exceed those of different benchmarks available in the literature, including the profit associated with participating in the day-ahead energy market with perfect price foresight (7,082 /MW/month). From a system perspective, these implicit balancing actions performed by the BESS owner reduce the system imbalance in 75% of all cases, thus improving the cost-efficiency of power systems.

MMA-RNN: A multi-level multi-task attention-based recurrent neural network for discrimination and localization of atrial fibrillation

Background:Atrial fibrillation significantly impacts individual health, public wellness, and healthcare expenditure. The primary tool for its detection is the Electrocardiograph (ECG), a non-invasive test that generates abundant data requiring expert human analysis. This task becomes exceedingly complex when interpreting ECG signals with cyclical patterns, fluctuating lengths, and vulnerability to noise. Despite these challenges, there exists a significant gap in the research focusing on further differentiating persistent and paroxysmal atrial fibrillation, a task akin to multi-class discrimination, and pinpointing the exact start and end of abnormal episodes, a process central to localization. Methodology:We present the Multi-level Multi-task Attention-based Recurrent Neural Network (MMA-RNN), a solution founded on a hierarchical structure that leverages Bidirectional Long and Short-Term Memory Networks (Bi-LSTM) and attention layers to discern three-level sequential features. This framework enhances information synergy and reduces error accumulation by deploying a multi-head classifier that concurrently tackles both previously mentioned critical tasks, facilitating a more streamlined and accurate analysis. Results:We affirm the proficiency of our model through rigorous testing on the CPSC 2021 dataset, where it achieved 0.9437 out of 1 in discrimination and 1.3538 out of 1.6479 in localization. Our method also exhibited superior performance in line with official evaluation benchmarks. Conclusions:Our innovative algorithm holds the potential for real-time AF monitoring through wearable technology, promoting proactive healthcare and early intervention.

NRDL: Decentralized user preference learning for privacy-preserving next POI recommendation

Predicting where a user goes next in terms of his or her previously visited points of interest (POIs) is significant for facilitating users’ daily lives. Simultaneously, it must be acknowledged that the check-in and trajectory information of the user is absolutely disclosed to others in location-based social networks when recommending the next POIs. Therefore, how to achieve an accurate next POI recommendation on the premise of privacy preservation is a critical challenge. To address this challenge, we propose decentralized user preference learning for privacy-preserving next POI recommendation, called NRDL. First, to capture the user’s next POI preference, we model the user’s real-time demand representation by POI profile, POI category, absolute time, transition time and distance between previously visited POIs, which is input into an attention-based recurrent neural network (RNN) model for embedding. Second, to perform privacy preservation, we develop a decentralized learning framework that can achieve user preference learning by raw data on each user’s side. Learning on each user’s side can make the privacy data of the user not be revealed to platforms or others, and learning from raw data can guarantee the value of check-ins and further accuracy. Finally, we evaluate the proposed model on two widely used Gowalla and Foursquare datasets, and the improvements over the state-of-the-art model are 25.00% and 9.95% at recall@1 and NDCG@1 on Gowalla as well as 22.51% and 10.59% on Foursquare.

Decomposition strategy and attention-based long short-term memory network for multi-step ultra-short-term agricultural power load forecasting

Accurate forecasting of the agricultural power load is important for rural electricity networks’ safe and stable operation. But it is more uncertain and difficult to forecast than industrial, commercial, and residential loads. Multivariate time series prediction methods have problems such as poor refinement of weather services and difficulty in quantifying the influencing factors. Univariate time series prediction methods should be emphasized, and the characteristic information in the historical load data should be fully exploited. Therefore, this paper systematically investigates the decomposition strategy and attention-based recurrent neural network for univariate time-series multi-step prediction. The decomposition strategy is employed to generate derived variable sequences from the target sequence as inputs for the model, while the dual-stage attention mechanism is used for dynamically selecting features. A one-year real-world dataset of agricultural power loads in southern China is used for validation. The effectiveness of two different decomposition algorithms on dual-stage attention-based recurrent neural network (DARNN) is discussed, then the proposed ensemble empirical mode decomposition and dual-stage attention-based recurrent neural network (EEMD-DARNN) is compared with four univariate time series prediction models and three ablation models. The proposed model is optimal in four evaluation indicators, with R2, RMSE, MAPE, and MAE of 0.970, 12.298, 0.040, and 7.335 respectively in the shortest forecast horizon (15 min), and 0.941, 17.305, 0.065 and 11.347 respectively in the longest forecast horizon (1 h). It is a significant improvement over the existing models. In conclusion, EEMD-DARNN can achieve more accurate ultra-short-term agricultural power load forecasting.

State-of-health estimation of lithium-ion batteries using a novel dual-stage attention mechanism based recurrent neural network

Accurate estimation of the state of health (SOH) of lithium-ion batteries is an important guarantee to ensure safe and reliable operation of lithium-ion battery systems. However, the complex aging mechanism inside the battery makes it difficult to measure the battery SOH directly. In this paper, a SOH estimation method based on a novel dual-stage attention-based recurrent neural network (DARNN) and health feature (HF) extraction from time varying charging process is proposed. Firstly, the constant current charging time, the maximum temperature time, the isochronous voltage difference, and the isochronous current were extracted as lithium-ion battery HFs, and their correlations with SOH are verified by spearman correlation coefficient. Secondly, the DARNN is proposed to capture the time-dependent and temporal features of the input sequence and to accurately predict SOH. Finally, the proposed estimation method is validated on the NASA battery dataset. The results show that the method can accurately estimate SOH for lithium-ion batteries. The mean square error and the mean absolute percentage error of the method are <0.5 %.

Multi-aspect attentive text representations for simple question answering over knowledge base

With the deepening of knowledge base research and application, question answering over knowledge base, also called KBQA, has recently received more and more attention from researchers. Most previous KBQA models focus on mapping the input query and the fact in KBs into an embedding format. Then the similarity between the query vector and the fact vector is computed eventually. Based on the similarity, each query can obtain an answer representing a tuple (subject, predicate, object) from the KBs. However, the information about each word in the input question will lose inevitably during the process. To retain as much original information as possible, we introduce an attention-based recurrent neural network model with interactive similarity matrixes. It can extract more comprehensive information from the hierarchical structure of words among queries and tuples stored in the knowledge base. This work makes three main contributions: (1) A neural network-based question-answering model for the knowledge base is proposed to handle single relation questions. (2) An attentive module is designed to obtain information from multiple aspects to represent queries and data, which contributes to avoiding losing potentially valuable information. (3) Similarity matrixes are introduced to obtain the interaction information between queries and data from the knowledge base. Experimental results show that our proposed model performs better on simple questions than state-of-the-art in several effectiveness measures.