Layer Recurrent Neural Network Research Articles

Estimating the Health State of Lithium-Ion Batteries Using an Adaptive Gated Sequence Network and Hierarchical Feature Construction

State of health (SOH) estimation plays a vital role in ensuring the safe and stable operation of lithium-ion battery management systems (BMSs). Data-driven methods are widely used to estimate SOH; however, existing methods often suffer from fixed or excessively high feature dimensions, impacting the model’s adjustability and applicability. This study first proposed a layered knee point strategy based on the charging voltage curve, which reduced the complexity of feature extraction. Then, a new hybrid framework called the adaptive gated sequence network (AGSN) model was proposed. This model integrated independently recurrent neural network (IndRNN) layers, active state tracking long short-term memory (AST-LSTM) layers, and adaptive gating mechanism (AGM) layers. By integrating a multi-layered structure and an adaptive gating mechanism, the SOH prediction performance was significantly improved. Finally, batteries under different operating conditions were tested using the NASA battery dataset. The results show that the AGSN model demonstrated higher accuracy and robustness in battery SOH estimation, with estimation errors consistently within 1%.

KAZAKH SIGN LANGUAGE RECOGNITION BY USING MACHINE LEARNING METHODS

Pattern recognition involves identifying objects within images, where the objects are abstract and can vary in shape. We have chosen to explore the relevant area of gesture recognition. To recognize Kazakh Sign Language (KSL), it is essential first to learn the Kazakh Sign Alphabet (KSA). Training a neural network to recognize KSL requires collecting datasets in the form of images depicting hand gestures. Gesture recognition is a classification task, which is a subset of pattern recognition. The fundamental basis of recognition lies in the theory of pattern recognition.This publication focuses on the study of supervised learning methods, deep learning techniques, classification tasks, and gesture recognition trained on proprietary datasets (images obtained and divided into frames from video sequences captured by a webcam or mobile device). These methods significantly expand the range of tasks that can be effectively solved in real-time within the field of gesture recognition.The study presents results using metrics for testing deep learning models. The recognition accuracy of these models has been demonstrated based on precision, recall, F1-score, and support for each class, as well as overall accuracy and average scores. A hybrid architectural neural network model utilizing recurrent and convolutional neural network layers has been tested. Software has been developed to recognize gestures in real-time.

Maximum Power Point Tracking Controller of PV System Based on Two Hidden Layer Recurrent Neural Network

Maximum Power Point Tracking Controller of PV System Based on Two Hidden Layer Recurrent Neural Network

A neuro-enhanced Duhem model and parameter identification using a modified gorilla troops optimizer for generic piezoelectric hysteresis response

The positioning precision of piezoelectric actuator is closely related to the modeling of hysteresis nonlinearity and the parameter identification. In order to accurately characterize the hysteresis nonlinearity, an innovative scheme based on a modified gorilla troops optimizer identified by a neuro-enhanced Duhem model is proposed. The main contribution of this work lies in that a neuro-enhanced Duhem model combining a Duhem model with a layer recurrent neural network is established and also in that, to overcome the defects of traditional gorilla troops optimizer including slow convergence speed, low identification accuracy, ease to fall into local optima, the gorilla troops optimizer is modified to enhance its identification ability by introducing a series of new mechanisms. A new adaptive mechanism is designed to balance the exploration and exploitation; a Gaussian-Cauchy-t distribution hybrid strategy is proposed to generate new individuals; a Levy flight enhanced exploitation strategy is integrated to increase the diversity of the population; and a greedy retention strategy is employed to improve the convergence speed. The parameter sensitivity, rate-dependent modeling capability, identification results, robustness of the algorithm are rigorously evaluated through a series of experiments. The results reveal that, the rate-dependent and asymmetric hysteresis of piezoelectric systems can be accurately characterized by the neuro-enhanced Duhem model, and that the modified gorilla troops optimizer algorithm can find the model parameters with high accuracy, fast speed and high repeatability.

Optimising air quality prediction in smart cities with hybrid particle swarm optimization‐long‐short term memory‐recurrent neural network model

AbstractIn smart cities, air pollution is a critical issue that affects individual health and harms the environment. The air pollution prediction can supply important information to all relevant parties to take appropriate initiatives. Air quality prediction is a hot area of research. The existing research encounters several challenges that is, poor accuracy and incorrect real‐time updates. This research presents a hybrid model based on long‐short term memory (LSTM), recurrent neural network (RNN), and Curiosity‐based Motivation method. The proposed model extracts a feature set from the training dataset using an RNN layer and achieves sequencing learning by applying an LSTM layer. Also, to deal with the overfitting issues in LSTM, the proposed model utilises a dropout strategy. In the proposed model, input and recurrent connections can be dropped from activation and weight updates using the dropout regularisation approach, and it utilises a Curiosity‐based Motivation model to construct a novel motivational model, which helps in the reconstruction of long short‐term memory recurrent neural network. To minimise the prediction error, particle swarm optimisation is implemented to optimise the LSTM neural network's weights. The authors utilise an online Air Pollution Monitoring dataset from Salt Lake City, USA with five air quality indicators for comparison, that is, SO2, CO, O3, and NO2, to predict air quality. The proposed model is compared with existing Gradient Boosted Tree Regression, Existing LSTM, and Support Vector Machine based Regression Model. Experimental analysis shows that the proposed method has 0.0184 (Root Mean Square Error (RMSE)), 0.0082 (Mean Absolute Error), 2002*109 (Mean Absolute Percentage Error), and 0.122 (R2‐Score). The experimental findings demonstrate that the proposed LSTM model had RMSE performance in the prescribed dataset and statistically significant superior outcomes compared to existing methods.

Machine Learning Models for Real-Time Traffic Prediction: A Case Study in Urban Traffic Management

This study introduces and evaluates the Long-term Traffic Prediction Network (LTPN), a specialized machine learning framework designed for real-time traffic prediction in urban environments. Utilizing a unique combination of convolutional and recurrent neural network layers, the LTPN model consistently outperforms established predictive models across various metrics. It demonstrates significantly lower error rates in both short and long-term traffic forecasts, highlighting its superior accuracy and reliability. The effectiveness of the LTPN model is underscored by its robust performance under diverse traffic conditions, making it a promising tool for enhancing the efficiency and responsiveness of intelligent transportation systems (ITS). This paper details the model's architecture, training processes, and a comprehensive comparison of its predictive capabilities against traditional models, providing clear evidence of its advantages in real-world applications.

Predicting carbon dioxide emissions in the United States of America using machine learning algorithms.

Carbon dioxide (CO2) emissions result from human activities like burning fossil fuels. CO2 is a greenhouse gas, contributing to global warming and climate change. Efforts to reduce CO2 emissions include transitioning to renewable energy. Monitoring and reducing CO2 emissions are crucial for mitigating climate change. Strategies include energy efficiency and renewable energy adoption. In the past few decades, several nations have experienced air pollution and environmental difficulties because of carbon dioxide (CO2) emissions. One of the most crucial methods for regulating and maximizing CO2 emission reductions is precise forecasting. Four machine learning algorithms with high forecasting precision and low data requirements were developed in this study to estimate CO2 emissions in the United States (US). Data from a dataset covering the years 1973/01 to 2022/07 that included information on different energy sources that had an impact on CO2 emissions were examined. Then, four algorithms performed the CO2 emissions forecast from the layer recurrent neural network with 10 nodes (L-RNN), a feed-forward neural network with 10 nodes (FFNN), a convolutional neural network with two layers with 10 and 5 filters (CNN1), and convolutional neural network with two layers and with 50 and 25 filters (CNN2) models. Each algorithm's forecast accuracy was assessed using eight indicators. The three preprocessing techniques used are (1) without any processing techniques, (2) processed using max-min normalization technique, and (3) processed using max-min normalization technique and decomposed by variation mode decomposition (VMD) technique with 7 intrinsic mode functions and 1000 iterations. The latter with L-RNN algorithm gave a high accuracy between the forecasting and actual values. The results of CO2 emissions from 2011/05 to 2022/07 have been forecasted, and the L-RNN algorithm had the highest forecast accuracy. The L-RNN model has the lowest value of 1.187028078, 135.5668592, and 11.64331822 for MAPE, MSE, and RMSE, respectively. The L-RNN model provides precise and timely forecasts that can help formulate plans to reduce carbon emissions and contribute to a more sustainable future. Moreover, the results of this investigation can improve our comprehension of the dynamics of carbon dioxide emissions, resulting in better-informed environmental policies and initiatives targeted at lowering carbon emissions.

Data Representation for Deep Learning - Based Arabic Text Summarization Performance Using Python Results

A sequence-to-sequence model is used as the foundation for a suggested abstractive Arabic text summarizing system. Our goal is to create a sequence-to-sequence model by utilizing multiple deep artificial neural networks and determining which one performs the best. The encoder and decoder have been developed using several layers of recurrent neural networks, gated recurrent units, recursive neural networks, convolutional neural networks, long short-term memory, and bidirectional long short-term memory. We are re-implementing the fundamental summarization model in this study, which uses the sequence-to-sequence framework. Using a Google Colab Jupiter notebook that runs smoothly, we have constructed these models using the Keras library. The results further demonstrate that one of the key techniques that has led to breakthrough performance with deep neural networks is the use of Gensim for word embeddings over other text representations by abstractive summarization models, along with FastText, a library for efficient learning of word representations and sentence classification.

Enhancing Lip Reading: A Deep Learning Approach with CNN and RNN Integration

This research introduces an innovative approach to enhance lip reading-based text extraction and translation through the integration of a double Convolutional Neural Network (CNN) coupled with Recurrent Neural Network (RNN) architecture. The proposed model aims to leverage the strengths of both CNN and RNN to achieve superior accuracy in lip movement interpretation and subsequent text extraction. The methodology involves training the double CNN+RNN model on extensive datasets containing synchronized lip movements and corresponding linguistic expressions. The initial layers of the model utilize CNNs to effectively capture spatial features from the visual input of lip images. The extracted features are then fed into RNN layers, allowing the model to grasp temporal dependencies and contextual information crucial for accurate lip reading. The trained model showcases its proficiency in extracting textual content from spoken words, demonstrating an advanced capability to decipher nuances in lip gestures. Furthermore, the extracted text undergoes a translation process, enabling the conversion of spoken language into various target languages. This research not only contributes to the advancement of lip reading technologies but also establishes a robust foundation for real-world applications such as accessibility solutions for individuals with hearing impairments, real-time multilingual translation services, and improved communication in challenging acoustic environments. The abstract concludes with a discussion on the potential impact of the double CNN+RNN model in pushing the boundaries of human-computer interaction, emphasizing the synergy between deep learning, lip reading, and translation technologies

GRAformer: A gated residual attention transformer for multivariate time series forecasting

Recurrent Neural Networks (RNNs), particularly when equipped with output windows – a standard practice in contemporary time series forecasting – have shown proficiency in handling short-term dependencies. Nonetheless, RNNs can encounter challenges in maintaining hidden states over extended forecasting periods, particularly in longer-term predictions where increased hidden state sizes and extended look-back windows can lead to gradient instability. In contrast, Transformer-based models, with their distinctive architecture designed to encode complex contextual relationships and enable computations to be done in parallel, are emerging as a popular alternative in this field. However, current research has mainly focused on modifying attention mechanisms, overlooking opportunities to improve the feedforward layer, which could lead to efficiency limitations. Moreover, prevailing methods often assume absolute independence between channels, disregarding distinct features among variables and failing to fully leverage channel-specific information. To address these gaps, we propose an efficient transformer design for multivariate time series prediction. Our approach integrates two key components: (i) a gated residual attention unit that enhances predictive accuracy and computational efficiency, and (ii) a channel embedding technique that differentiates between series and boosts performance. Theoretically, we prove that our model has recurrent dynamics introduced by the RNN layer. Through extensive experiments on real-world data, we demonstrate that our proposed method achieves competitive predictive accuracy compared to prior approaches while exhibiting accelerated processing relative to state-of-the-art transformers. Our code, data, and trained models are available at https://github.com/MythosAd/GRAformer.

RNN-Based Time Series Analysis for Wind Turbine Energy Forecasting

One significant source of renewable energy is wind power, which has the potential to generate sustainable energy. However, wind turbines have many challenges, such as high initial investment costs, the dynamic nature of wind speed, and the challenge of locating wind-efficient energy regions. Wind power predicting is crucial for effective planning of wind power generation, optimization of power generation, grid integration, and security of supply. Therefore, highly accurate forecasts ensure the efficient and sustainable operation of the wind energy sector and contribute to energy security, economic stability, and environmental sustainability. This study proposes a deep learning (DL) approach based on recurrent neural networks (RNNs) for long-term wind power forecasting utilizing climatic data. The input data that forms the basis of this study is obtained directly from a wind turbine system operating under real-world conditions. The proposed model in this study is based on a multilayer back-propagation neural network (RNN) architecture specifically designed to effectively handle complex data sets and time-dependent series. The architecture of the model is built on an RNN consisting of four separate layers, each with 50 hidden neurons, carefully structured to increase its capacity to capture complex features. To improve the robustness of the model and avoid overlearning, each RNN layer is followed by a dropout (regularizing) layer that randomly deactivates 20% of the neurons to enhance the generalization ability of the network. To finalize the prediction capability of the model, a linear function was chosen in the last layer to directly match the actual values. Evaluating the model performance metrics, the proposed architecture achieved a prediction accuracy of 91% R2 on the test dataset. The findings indicate that proposed method based on multilayer RNN can successfully capture the relationships between the sequential data of the wind turbine.

Stage-Wise Magnitude-Based Pruning for Recurrent Neural Networks.

A recurrent neural network (RNN) has shown powerful performance in tackling various natural language processing (NLP) tasks, resulting in numerous powerful models containing both RNN neurons and feedforward neurons. On the other hand, the deep structure of RNN has heavily restricted its implementation on mobile devices, where quite a few applications involve NLP tasks. Magnitude-based pruning (MP) is a promising way to address such a challenge. However, the existing MP methods are mostly designed for feedforward neural networks that do not involve a recurrent structure, and, thus, have performed less satisfactorily on pruning models containing RNN layers. In this article, a novel stage-wise MP method is proposed by explicitly taking the featured recurrent structure of RNN into account, which can effectively prune feedforward layers and RNN layers, simultaneously. The connections of neural networks are first grouped into three types according to how they are intersected with recurrent neurons. Then, an optimization-based pruning method is applied to compress each group of connections, respectively. Empirical studies show that the proposed method performs significantly better than the commonly used RNN pruning methods; i.e., up to 96.84% connections are pruned with little or even no degradation of precision indicators on the testing datasets.

One-shot lip-based biometric authentication: Extending behavioral features with authentication phrase information

Lip-based biometric authentication (LBBA) is an authentication method based on a person's lip movements during speech in the form of video data. LBBA can utilize both physical and behavioral characteristics of lip movements without requiring any additional sensory equipment apart from an RGB camera. Current approaches employ deep siamese neural networks trained with one-shot learning to generate embedding vectors from lip movement features. However, most of these approaches don't discriminate against speech content which makes them vulnerable to video replay attacks. Moreover, there is a lack of comprehensive analysis regarding the impact of distinct lip characteristics or difficult dataset phrases with significant word overlap on the performance of authentication in one-shot approaches. To address this, we introduce the GRID-CCP dataset and train a siamese neural network using 3D convolutions and recurrent neural network layers to additionally discriminate against speech content. For loss calculation, we propose a custom triplet loss function for efficient and customizable batch-wise hard-negative mining. Our experimental results, using an open-set protocol, demonstrate a False Acceptance Rate (FAR) of 3.2% and a False Rejection Rate (FRR) of 3.8% on the test set of the GRID-CCP dataset. Finally, we conduct an analysis to assess the influence and discriminative power of behavioral and physical features in LBBA.

Enhancing Load Forecasting Accuracy in Smart Grids: A Novel Parallel Multichannel Network Approach Using 1D CNN and Bi‐LSTM Models

Load forecasting plays a pivotal role in the efficient energy management of smart grid. However, the complex, intermittent, and nonlinear smart grids and the complexity of large dataset handling pose difficulty in accurately forecasting loads. The important issue is considering the cyclic features, which have not yet been adequately addressed through the trigonometric transformations. Furthermore, using long short‐term memory (LSTM) or 1D convolution neural network (1D CNN) and existing hybrid models involve stacked CNN‐LSTM architectures, employing 1D convolutions as a preprocessing step to downsample sequences and extract high‐ and low‐level spatial features. However, these models often overlook temporal features, emphasizing higher‐level features processed by the subsequent recurrent neural network layer. Therefore, this study considers a novel approach to independently process features for spatial and temporal characteristics using a parallel multichannel network comprising 1D CNN and bidirectional‐LSTM (Bi‐LSTM) models. The proposed model evaluated the National Transmission and Dispatch Company (NTDC) load dataset, with additional assessment on two datasets, American Electric Power and Commonwealth Edison, to ensure its generalizability. Performance evaluation on the NTDC dataset yields results of 3.4% mean absolute percentage error (MAPE), 513.95 mean absolute error (MAE), and 623.78 root mean square error (RMSE) for day‐ahead forecasting, and 0.56% MAPE, 94.84 MAE, and 115.67 RMSE for hour‐ahead load forecast. The experimental results demonstrate that the proposed model outperforms stacked CNN‐LSTM models, particularly in forecasting hour‐ and day‐ahead loads. Moreover, a comparative analysis with previous studies reveals superior performance in reducing the error gap between predicted and actual values.

Virtual impedance based low-voltage distribution grid topology detection using double hidden layer recurrent neural network

Low-voltage distribution grid (LVDG) topology detection refers to detecting whether the topology connection between distribution grid nodes is correct. Accurate topology connection is critical for the normal operation and planning of LVDG. However, due to the incomplete measurement device, unknown line parameters, and rapid growth of renewable energy, the topology detection of LVDG becomes one of the most prominent challenges. This paper proposes an LVDG topology detection method based on virtual impedance, utilizing measurement data from nodes in the LVDG to achieve the detection of abnormal topological connections. Specifically, the electrical distances between nodes are analyzed to establish a topology detection model using virtual impedance. Then, the double hidden layer recurrent neural network is proposed to fit the mapping relationships between variables in the power flow constraints. The virtual impedance between nodes is solved. The value of virtual impedance is used to determine whether the topological connection between nodes is correct. Finally, the test results in the actual LVDG prove the effectiveness of the proposed method.

Adaptive Neural Backstepping Terminal Sliding Mode Control of a DC-DC Buck Converter.

In this paper, an adaptive backstepping terminal sliding mode control (ABTSMC) method based on a double hidden layer recurrent neural network (DHLRNN) is proposed for a DC-DC buck converter. The DHLRNN is utilized to approximate and compensate for the system uncertainty. On the basis of backstepping control, a terminal sliding mode control (TSMC) is introduced to ensure the finite-time convergence of the tracking error. The effectiveness of the composite control method is verified on a converter prototype in different test conditions. The experimental comparison results demonstrate the proposed control method has better steady-state performance and faster transient response.

The Modeling of the Rucklidge Chaotic System with Artificial Neural Networks

Chaotic systems are nonlinear systems that show sensitive dependence on initial conditions, and an immeasurably small change in initial value causes an immeasurably large change in the future state of the system. Besides, there is no randomness in chaotic systems and they have an order within themselves. Researchers use chaotic systems in many areas such as mixer systems that can make more homogeneous mixtures, encryption systems that can be used with high security, and artificial neural networks by taking the advantage of the order in this disorder. Differential equations in which chaotic systems are expressed mathematically are solved by numerical solution methods such as Heun, Euler, ODE45, RK4, RK5-Butcher and Dormand-Prince in the literature. In this research, Feed Forward Neural Network (FFNN), Layer Recurrent Neural Network (LRNN) and Cascade Forward Backpropogation Neural Network (CFNN) structures were used to model the Rucklidge chaotic system by making use of the MATLAB R2021A program Neural Network (NN) Toolbox. By comparing the results of different activation functions used in the modeling, the ANN structure that can best model the Rucklidge chaotic system has been determined. The training of the compared Artificial Neural Networks (ANNs) was carried out with the values obtained from the Euler numerical solution method, which can get satisfactory and fast results.

Differentiating brain states via multi-clip random fragment strategy-based interactive bidirectional recurrent neural network

EEG is widely adopted to study the brain and brain computer interface (BCI) for its non-invasiveness and low costs. Specifically EEG can be applied to differentiate brain states, which is important for better understanding the working mechanisms of the brain. Recurrent neural network (RNN)-based learning strategy has been widely utilized to differentiate brain states, because its optimization architectures improve the classification performance for differentiating brain states at the group level. However, present classification performance is still far from satisfactory. We have identified two major focal points for improvements: one is about organizing the input EEG signals, and the other is related to the design of the RNN architecture. To optimize the above-mentioned issues and achieve better brain state classification performance, we propose a novel multi-clip random fragment strategy-based interactive bidirectional recurrent neural network (McRFS-IBiRNN) model in this work. This model has two advantages over previous methods. First, the McRFS component is designed to re-organize the input EEG signals to make them more suitable for the RNN architecture. Second, the IBiRNN component is an innovative design to model the RNN layers with interaction connections to enhance the fusion of bidirectional features. By adopting the proposed model, promising brain states classification performances are obtained. For example, 96.97% and 99.34% of individual and group level four-category classification accuracies are successfully obtained on the EEG motor/imagery dataset, respectively. A 99.01% accuracy can be observed for four-category classification tasks with new subjects not seen before, which demonstrates the generalization of our proposed method. Compared with existing methods, our model outperforms them with superior results. Overall, the proposed McRFS-IBiRNN model demonstrates great superiority in differentiating brain states on EEG signals

Semi-supervised learning improves the performance of cardiac event detection in echocardiography

Detection of end-diastole (ED) and end-systole (ES) frames in echocardiography video is a critical step for assessment of cardiac function. A recently released large public dataset, i.e., EchoNet-Dynamic, could be used as a benchmark for cardiac event detection. However, only a pair of ED and ES frames are annotated in each echocardiography video and the annotated ED comes before ES in most cases. This means that only a few frames during systole in each video are utilizable for training, which makes it challenging to train a cardiac event detection model using the dataset. Semi-supervised learning (SSL) could alleviate the problems. An architecture combining convolutional neural network (CNN), recurrent neural network (RNN) and fully-connected layers (FC) is adopted. Experimental results indicate that SSL brings at least three benefits: faster convergence rate, performance improvement and more reasonable volume curves. The best mean absolute errors (MAEs) for ED and ES detection are 40.2 ms (2.1 frames) and 32.6 ms (1.7 frames), respectively. In addition, the results show that models trained on apical four-chamber (A4C) view could work well on other standard views, such as other apical views and parasternal short axis (PSAX) views.

Sentiment prediction based on neural network

Recent years have seen a surge in interest in deep learning, and numerous new neural network models have appeared. In this study, we use CNN and LSTM, two popular deep learning models, to predict sentiment. We use the bidirectional model for the LSTM model. We make predictions by integrating the feature vectors of both sides after receiving sentences inputted in the reverse order from both ends. The recurrent neural network layer is replaced with the CNN model, which defines numerous one-dimensional convolution kernels to conduct convolution operations on the input. The two models are assessed and compared based on how well they anticipate and analyze sentence emotion tones. By comparison, we discover that the BiLSTM model's prediction accuracy on the test set is 10% greater than CNN's, despite requiring more training time.