Traditional Recurrent Neural Networks Research Articles

Hierarchical Predictions of Fine-to-Coarse Time Span and Atmospheric Field Reconstruction for Typhoon Track Prediction

The prediction of typhoon tracks in the Northwest Pacific is key to reducing human casualties and property damage. Traditional numerical forecasting models often require substantial computational resources, are high-cost, and have significant limitations in prediction speed. This research is dedicated to using deep learning methods to address the shortcomings of traditional methods. Our method (AFR-SimVP) is based on a large-kernel convolutional spatio-temporal prediction network combined with multi-feature fusion for forecasting typhoon tracks in the Northwest Pacific. In order to more effectively suppress the effect of noise in the dataset to enhance the generalization ability of the model, we use a multi-branch structure, incorporate an atmospheric reconstruction subtask, and propose a second-order smoothing loss to further improve the prediction ability of the model. More importantly, we innovatively propose a multi-time-step typhoon prediction network (HTAFR-SimVP) that does not use the traditional recurrent neural network family of models at all. Instead, through fine-to-coarse hierarchical temporal feature extraction and dynamic self-distillation, multi-time-step prediction is achieved using only a single regression network. In addition, combined with atmospheric field reconstruction, the network achieves integrated prediction for multiple tasks, which greatly enhances the model’s range of applications. Experiments show that our proposed network achieves optimal performance in the 24 h typhoon track prediction task. Our regression network outperforms previous recurrent network-based typhoon prediction models in the multi-time-step prediction task and also performs well in multiple integration tasks.

Multi-scene application of intelligent inspection robot based on computer vision in power plant

As industries develop, the automation and intelligence level of power plants is constantly improving, and the application of patrol robots is also increasingly widespread. This research combines computer vision technology and particle swarm optimization algorithm to build an obstacle recognition model and obstacle avoidance model of an intelligent patrol robot in a power plant respectively. Firstly, the traditional convolutional recurrent neural network is optimized, and the obstacle recognition model of an intelligent patrol robot is built by combining the connection timing classification algorithm. Then, the artificial potential field method optimizes the traditional particle swarm optimization algorithm, and an obstacle avoidance model of an intelligent patrol robot is built. The performance of the two models was tested, and it was found that the highest precision, recall, and F1 values of the identification model were 0.978, 0.974, and 0.975. The highest precision, recall, and F1 values of the obstacle avoidance model were 0.97, 0.96, and 0.96 respectively. The two optimization models designed in this research have better performance. In conclusion, the two models in this study are superior to the traditional methods in recognition effect and obstacle avoidance efficiency, providing an effective technical scheme for intelligent patrol inspection of power plants.

Deterministic convergence analysis for regularized long short-term memory and its application to regression and multi-classification problems

Long short-term memory (LSTM) is a recurrent neural network (RNN) framework designed to solve the gradient disappearance and explosion problems of traditional RNNs. In recent years, LSTM has become a state-of-the-art model for solving various machine-learning problems. This paper propose a novel regularized LSTM based on the batch gradient method. Specifically, the L2 regularization is appended to the objective function as a systematic external force, effectively controlling the excessive growth of weights in the network and preventing the overfitting phenomenon. In addition, a rigorous convergence analysis of the proposed method is carried out, i.e., monotonicity, weak convergence, and strong convergence results are obtained. Finally, comparative simulations are conducted on the benchmark data set for regression and classification problems, and the simulation results verify the effectiveness of the method.

WIFI based human activity recognition using multi-head adaptive attention mechanism

Human Activity Recognition (HAR) utilizing Channel State Information (CSI) extracted from WiFi signals has garnered substantial interest across various domains and applications. This field’s potential paths and applications extend beyond CSI-based HAR and include smart homes, assisted living, security, gaming, surveillance, and context-aware computing. The ability of deep learning algorithms to effectively process and interpret CSI data opens up new possibilities for accurate and robust human activity recognition in real-world scenarios. However, traditional Recurrent Neural Networks (RNN) models, such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), rely solely on their internal memory cells to maintain information over time. Important details might be diluted or lost within the memory cells in complex CSI sequences. To address this limitation, we propose a lightweight approach that incorporates a multi-head adaptive attention weight mechanism MHAAM into the HAR framework. The multi-head attention mechanism allows the model to attend to different informative patterns within the CSI data simultaneously, capturing fine-grained temporal dependencies and improving the model’s ability to recognize complex activities. The implemented models effectively filter out noise and irrelevant information by assigning higher weights to informative CSI features, further enhancing activity classification accuracy. Experimental evaluations and comparative analyses of HAR for seven activities demonstrate that attention-based RNN models with multi-head attention consistently outperform traditional RNN models. The multi-head attention mechanism achieves improved generalization and testing for seven common human activities and environments, leading to a higher complex human activity classification accuracy of up to 98.5%.

Sensor-Based Indoor Fire Forecasting Using Transformer Encoder.

Indoor fires may cause casualties and property damage, so it is important to develop a system that predicts fires in advance. There have been studies to predict potential fires using sensor values, and they mostly exploited machine learning models or recurrent neural networks. In this paper, we propose a stack of Transformer encoders for fire prediction using multiple sensors. Our model takes the time-series values collected from the sensors as input, and predicts the potential fire based on the sequential patterns underlying the time-series data. We compared our model with traditional machine learning models and recurrent neural networks on two datasets. For a simple dataset, we found that the machine learning models are better than ours, whereas our model gave better performance for a complex dataset. This implies that our model has a greater potential for real-world applications that probably have complex patterns and scenarios.

Early fire detection technology based on improved transformers in aircraft cargo compartments

The implementation of early and accurate detection of aircraft cargo compartment fire is of great significance to ensure flight safety. The current airborne fire detection technology mostly relies on single-parameter smoke detection using infrared light. This often results in a high false alarm rate in complex air transportation environments. The traditional deep learning model struggles to effectively address the issue of long-term dependency in multivariate fire information. This paper proposes a multi-technology collaborative fire detection method based on an improved transformers model. Dual-wavelength optical sensors, flue gas analyzers, and other equipment are used to carry out multi-technology collaborative detection methods and characterize various feature dimensions of fire to improve detection accuracy. The improved Transformer model which integrates the self-attention mechanism and position encoding mechanism is applied to the problem of long-time series modeling of fire information from a global perspective, which effectively solves the problem of gradient disappearance and gradient explosion in traditional RNN (recurrent neural network) and CNN (convolutional neural network). Two different multi-head self-attention mechanisms are used to classify and model multivariate fire information, respectively, which solves the problem of confusing time series modeling and classification modeling in dealing with multivariate classification tasks by a single attention mechanism. Finally, the output results of the two models are fused through the gate mechanism. The research results show that, compared with the traditional single-feature detection technology, the multi-technology collaborative fire detection method can better capture fire information. Compared with the traditional deep learning model, the multivariate fire prediction model constructed by the improved Transformer can better detect fires, and the accuracy rate is 0.995.

Short-term wind power forecasting through stacked and bi directional LSTM techniques.

Computational intelligence (CI) based prediction models increase the efficient and effective utilization of resources for wind prediction. However, the traditional recurrent neural networks (RNN) are difficult to train on data having long-term temporal dependencies, thus susceptible to an inherent problem of vanishing gradient. This work proposed a method based on an advanced version of RNN known as long short-term memory (LSTM) architecture, which updates recurrent weights to overcome the vanishing gradient problem. This, in turn, improves training performance. The RNN model is developed based on stack LSTM and bidirectional LSTM. The parameters like mean absolute error (MAE), standard deviation error (SDE), and root mean squared error (RMSE) are utilized as performance measures for comparison with recent state-of-the-art techniques. Results showed that the proposed technique outperformed the existing techniques in terms of RMSE and MAE against all the used wind farm datasets. Whereas, a reduction in SDE is observed for larger wind farm datasets. The proposed RNN approach performed better than the existing models despite fewer parameters. In addition, the approach requires minimum processing power to achieve compatible results.

Universal approximation property of stochastic configuration networks for time series

For the purpose of processing sequential data, such as time series, and addressing the challenge of manually tuning the architecture of traditional recurrent neural networks (RNNs), this paper introduces a novel approach-the Recurrent Stochastic Configuration Network (RSCN). This network is constructed based on the random incremental algorithm of stochastic configuration networks. Leveraging the foundational structure of recurrent neural networks, our learning model commences with a modest-scale recurrent neural network featuring a single hidden layer and a solitary hidden node. Subsequently, the node parameters of the hidden layer undergo incremental augmentation through a random configuration process, with corresponding weights assigned structurally. This iterative expansion continues until the network satisfies predefined termination criteria. Noteworthy is the adaptability of this algorithm to handle time series data, exhibiting superior performance compared to traditional recurrent neural networks with similar architectures. The experimental results presented in this paper underscore the efficacy of the proposed RSCN for sequence data processing, showcasing its advantages over conventional recurrent neural networks in the context of the performed experiments.

Improved Hierarchical Temporal Memory for Online Prediction of Ocean Time Series Data

Time series prediction is an effective tool for marine scientific research. The Hierarchical Temporal Memory (HTM) model has advantages over traditional recurrent neural network (RNN)-based models due to its online learning and prediction capabilities. Given that the neuronal structure of HTM is ill-equipped for the complexity of long-term marine time series applications, this study proposes a new, improved HTM model, incorporating Gated Recurrent Units (GRUs) neurons into the temporal memory algorithm to overcome this limitation. The capacities and advantages of the proposed model were tested and evaluated on time series data collected from the Xiaoqushan Seafloor Observatory in the East China Sea. The improved HTM model both outperforms the original one in short-term and long-term predictions and presents results with lower errors and better model stability than the GRU model, which is proficient in long-term predictions. The findings allow for the conclusion that the mechanism of online learning has certain advantages in predicting ocean observation data.

Water Temperature Model: A Case Study in Alagoinhas-BA

In recent years, climate change has generated disturbances at both local and global scales, directly impacting the water temperature of rivers. Temperature fluctuations have become a common occurrence in river water, where after experiencing heating, the return to its natural temperature is gradual. Consequently, temperature changes affect various internal processes within rivers, including nutrient consumption, food availability, and dissolved oxygen concentration. These alterations subsequently impact the growth and distribution of aquatic organisms. This article focuses on the physical variables derived from a model characterized by limited computational complexity. The final structure of the model consists of a single ordinary differential equation, which is linearly dependent on air and water temperature, as well as flow rate. Additionally, to handle data sequences such as time series more efficiently than traditional Recurrent Neural Networks (RNNs), we employ a Long Short-Term Memory (LSTM) neural network.

Modeling and optimization of sporulation by Bacillus licheniformis BF-002 based on dynamics and recurrent neural networks

Bacillus licheniformis is widely utilized in disease prevention and environmental remediation. Spore quantity is a critical factor in determining the quality of microbiological agents containing vegetative cells. To improve the understanding of Bacillus licheniformis BF-002 strain culture, a hybrid model integrating traditional dynamic modeling and recurrent neural network was developed. This model enabled the optimization of carbon/nitrogen source feeding rates, pH, temperature and agitation speed using genetic algorithms. Carbon and nitrogen source consumption in the optimal duplicate batches showed no significant difference compared to the control batch. However, the spore quantity in the broth increased by 16.2% and 35.2% in the respective duplicate batches. Overall, the hybrid model outperformed the traditional dynamic model in accurately tracking the cultivation dynamics of Bacillus licheniformis, leading to increased spore production when used for optimizing cultivation conditions.

Water quality prediction based on sparse dataset using enhanced machine learning

Water quality in surface bodies remains a pressing issue worldwide. While some regions have rich water quality data, less attention is given to areas that lack sufficient data. Therefore, it is crucial to explore novel ways of managing source-oriented surface water pollution in scenarios with infrequent data collection such as weekly or monthly. Here we showed sparse-dataset-based prediction of water pollution using machine learning. We investigated the efficacy of a traditional Recurrent Neural Network alongside three Long Short-Term Memory (LSTM) models, integrated with the Load Estimator (LOADEST). The research was conducted at a river-lake confluence, an area with intricate hydrological patterns. We found that the Self-Attentive LSTM (SA-LSTM) model outperformed the other three machine learning models in predicting water quality, achieving Nash-Sutcliffe Efficiency (NSE) scores of 0.71 for CODMn and 0.57 for NH3N when utilizing LOADEST-augmented water quality data (referred to as the SA-LSTM-LOADEST model). The SA-LSTM-LOADEST model improved upon the standalone SA-LSTM model by reducing the Root Mean Square Error (RMSE) by 24.6% for CODMn and 21.3% for NH3N. Furthermore, the model maintained its predictive accuracy when data collection intervals were extended from weekly to monthly. Additionally, the SA-LSTM-LOADEST model demonstrated the capability to forecast pollution loads up to ten days in advance. This study shows promise for improving water quality modeling in regions with limited monitoring capabilities.

GSSTU: Generative Spatial Self-Attention Transformer Unit for Enhanced Video Prediction.

Future frame prediction is a challenging task in computer vision with practical applications in areas such as video generation, autonomous driving, and robotics. Traditional recurrent neural networks have limited effectiveness in capturing long-range dependencies between frames, and combining convolutional neural networks (CNNs) with recurrent networks has limitations in modeling complex dependencies. Generative adversarial networks have shown promising results, but they are computationally expensive and suffer from instability during training. In this article, we propose a novel approach for future frame prediction that combines the encoding capabilities of 3-D CNNs with the sequence modeling capabilities of Transformers. We also propose a spatial self-attention mechanism and a novel neighborhood pixel intensity loss to preserve structural information and local intensity, respectively. Our approach outperforms existing methods in terms of structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and learned perceptual image patch similarity (LPIPS) scores on five public datasets. More precisely, our model exhibited an average improvement of 4.64%, 18.5%, and 42% concerning SSIM, PSNR, and LPIPS for the second most proficient method correspondingly, across all datasets. The results demonstrate the effectiveness of our proposed method in generating high-quality predictions of future frames.

Real-time prediction of ROP based on GRU-Informer

Accurate ROP (rate of penetration) prediction contributes to better production task planning, ensuring efficient production line operation, and reducing production costs. ROP prediction is influenced by multiple factors, making accurate prediction challenging. Current research primarily relies on historical data for training and modeling, lacking methods for real-time ROP prediction. This paper introduces a GRU-Informer model for real-time ROP prediction. The model employs GRU (Gated Recurrent Unit) neural networks at the lower level to capture short-term correlations in drilling parameters and uses the Informer model at the top to address long-term dependencies among drilling parameters. Thus, the GRU-Informer can capture both short-term and long-term time dependencies, providing better ROP predictions. This paper constructs a dataset using historical data from a southwestern Chinese oil field for experimentation. RMSE (Root Mean Square Error), MAE (mean absolute error) and R2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${R}^{2}$$\\end{document} (Coefficient of Determination) are employed as evaluation metrics for the model. Experimental results demonstrate that the GRU-Informer outperforms traditional recurrent neural networks like LSTM (Long Short-Term Memory), GRU neural networks and Informer in real-time ROP prediction, indicating its practical value.

A time series driven model for early sepsis prediction based on transformer module

Sepsis remains a critical concern in intensive care units due to its high mortality rate. Early identification and intervention are paramount to improving patient outcomes. In this study, we have proposed predictive models for early sepsis prediction based on time-series data, utilizing both CNN-Transformer and LSTM-Transformer architectures. By collecting time-series data from patients at 4, 8, and 12 h prior to sepsis diagnosis and subjecting it to various network models for analysis and comparison. In contrast to traditional recurrent neural networks, our model exhibited a substantial improvement of approximately 20%. On average, our model demonstrated an accuracy of 0.964 (± 0.018), a precision of 0.956 (± 0.012), a recall of 0.967 (± 0.012), and an F1 score of 0.959 (± 0.014). Furthermore, by adjusting the time window, it was observed that the Transformer-based model demonstrated exceptional predictive capabilities, particularly within the earlier time window (i.e., 12 h before onset), thus holding significant promise for early clinical diagnosis and intervention. Besides, we employed the SHAP algorithm to visualize the weight distribution of different features, enhancing the interpretability of our model and facilitating early clinical diagnosis and intervention.

A RUL prediction method of rolling bearings based on degradation detection and deep BiLSTM

<abstract> <p>In order to grasp the degradation of rolling bearings and prevent the failure of mechanical equipment, a remaining useful life (RUL) prediction method of rolling bearings based on degradation detection and deep bidirectional long short-term memory networks (BiLSTM) was proposed, considering the incomplete degradation feature extraction and low prediction accuracy of existing methods. By extracting the characteristics of time domain, frequency domain, and time-frequency domain of the full-life bearing vibration signal, the monotonicity, trend, and robustness measurement indexes of each feature were calculated. The best feature set that can fully reflect the degradation information was constructed by ranking the weighted comprehensive indexes of the features. A degradation detection strategy was used to determine the degradation starting time for setting piecewise linear network label. The RUL prediction model based on deep BiLSTM was established and optimized through Dropout technology and piecewise learning rate. The model was verified by the full-life data set of rolling bearings. The results showed that compared with the support vector machine (SVM), the traditional recurrent neural network (RNN), the single-layer BiLSTM, and long short-term memory networks (LSTM) model without Dropout, the proposed method fitted the degradation trend best, and the root mean square error (RMSE) was the smallest and only 0.0281, which improved the accuracy of RUL prediction of rolling bearings, helped prevent bearing failure, and ensured the safe and reliable operation of rotating machinery.</p> </abstract>

Modified Noise-Immune Fuzzy Neural Network for Solving the Quadratic Programming With Equality Constraint Problem.

Quadratic programming with equality constraint (QPEC) problems have extensive applicability in many industries as a versatile nonlinear programming modeling tool. However, noise interference is inevitable when solving QPEC problems in complex environments, so research on noise interference suppression or elimination methods is of great interest. This article proposes a modified noise-immune fuzzy neural network (MNIFNN) model and use it to solve QPEC problems. Compared with the traditional gradient recurrent neural network (TGRNN) and traditional zeroing recurrent neural network (TZRNN) models, the MNIFNN model has the advantage of inherent noise tolerance ability and stronger robustness, which is achieved by combining proportional, integral, and differential elements. Furthermore, the design parameters of the MNIFNN model adopt two disparate fuzzy parameters generated by two fuzzy logic systems (FLSs) related to the residual and residual integral term, which can improve the adaptability of the MNIFNN model. Numerical simulations demonstrate the effectiveness of the MNIFNN model in noise tolerance.

Vehicle Trajectory-Based Traffic Volume Prediction on Urban Roads With Fast-Communication License Plate Recognition Data

Real-time traffic volume prediction is critical for proactive traffic control and road guidance via fast-communication networks. However, existing research relies on historical traffic volume and fails to consider vehicle travel route choices, leading to lower prediction accuracy and insufficient capture of real-time travel dynamics. To address this limitation, we propose a novel approach for real-time traffic volume prediction on urban links that incorporates vehicle travel trajectories extracted from recorded license plate recognition data using Radio Frequency Identification. A Recurrent Neural Network is used to identify route patterns and predict future routes, and link traffic volume is estimated by aggregating all predicted vehicle trajectories that pass through each base station. Our approach achieves a high prediction accuracy with a mean value of 85.19% and a standard deviation of 5.29%. Comparing our approach with a traditional Recurrent Neural Network based on historical traffic volume data, the vehicle trajectory-based approach yields an average accuracy improvement of 39.70%. These results highlight the superiority of our approach for predicting traffic volume in real-time and demonstrate its potential as a valuable tool for traffic control and road guidance.

Intelligent control of district heating system based on RDPG

Given the continuous expansion of heating areas in recent years, the design of a precise and dependable district heating system (DHS) has become increasingly crucial. Traditional control decisions are made based on real-time environmental temperature feedback, often leading to uneven heating on the user side and affecting residents' comfort. This paper proposes an intelligent control strategy based on the deep reinforcement learning recurrent deterministic policy gradient (RDPG) algorithm for DHSs. To explore the control performance of the RDPG algorithm on DHS, we have meticulously modeled the pivotal components of the DHSs, namely plate heat exchangers, secondary heating pipe networks, and heat users. Moreover, taking into account the periodic factors in heating regulation, the traditional recurrent neural network (RNN) in the recurrent deterministic policy gradient (RDPG) algorithm has been replaced with the long short-term memory (LSTM) network. The proposed algorithm was trained using actual data from a heat exchange station in Tianjin and compared with reinforcement learning algorithms such as TD3, DPPO, DDPG, and A3C in terms of training rewards, effectiveness, and training stability. The results of the models are evaluated and visualized. Experimental results show that the proposed control method based on the RDPG algorithm, compared to other control schemes, can achieve the highest training reward and the most stable control performance, with an indoor temperature fluctuation range of only 0.1 °C.

A Landslide Displacement Prediction Model Based on the ICEEMDAN Method and the TCN–BiLSTM Combined Neural Network

Influenced by autochthonous geological conditions and external environmental changes, the evolution of landslides is mostly nonlinear. This article proposes a combined neural network prediction model that combines a temporal convolutional neural network (TCN) and a bidirectional long short-term memory neural network (BiLSTM) to address the shortcomings of traditional recurrent neural networks in predicting displacement-fluctuation-type landslides. Based on the idea of time series decomposition, the improved complete ensemble empirical mode decomposition with an adaptive noise method (ICEEMDAN) was used to decompose displacement time series data into trend and fluctuation terms. Trend displacement is mainly influenced by the internal geological conditions of a landslide, and polynomial fitting is used to determine the future trend displacement; The displacement of the fluctuation term is mainly influenced by the external environment of landslides. This article selects three types of landslide-influencing factors: rainfall, groundwater level elevation, and the historical displacement of landslides. It uses a combination of gray correlation (GRG) and mutual information (MIC) correlation modules for feature screening. Then, TCN is used to extract landslide characteristic factors, and BiLSTM captures the relationship between features and displacement to achieve the prediction of wave term displacement. Finally, the trend term and fluctuation term displacement prediction values are reconstructed to obtain the total displacement prediction value. The results indicate that the ICEEMDAN–TCN–BiLSTM model proposed in this article can accurately predict landslide displacement and has high engineering application value, which is helpful for planning and constructing landslide disaster prevention projects.