Recurrent Neural Network Structure Research Articles

Particle Swarm Optimization-Based Model Abstraction and Explanation Generation for a Recurrent Neural Network

In text classifier models, the complexity of recurrent neural networks (RNNs) is very high because of the vast state space and uncertainty of transitions, which makes the RNN classifier’s explainability insufficient. It is almost impossible to explain the large-scale RNN directly. A feasible method is to generalize the rules undermining it, that is, model abstraction. To deal with the low efficiency and excessive information loss in existing model abstraction for RNNs, this work proposes a PSO (Particle Swarm Optimization)-based model abstraction and explanation generation method for RNNs. Firstly, the k-means clustering is applied to preliminarily partition the RNN decision process state. Secondly, a frequency prefix tree is constructed based on the traces, and a PSO algorithm is designed to implement state merging to address the problem of vast state space. Then, a PFA (probabilistic finite automata) is constructed to explain the RNN structure with preserving the origin RNN information as much as possible. Finally, the quantitative keywords are labeled as an explanation for classification results, which are automatically generated with the abstract model PFA. We demonstrate the feasibility and effectiveness of the proposed method in some cases.

Dynamic spatial-temporal model for carbon emission forecasting

Addressing the urgent need for accurate carbon emissions forecasting to support global emissions reduction goals, this paper introduces a novel approach for carbon emissions prediction that dynamically considers the spatial–temporal correlation of carbon emissions across forecasting targets. Traditional statistical and machine learning models have limitations, such as oversimplification and inefficiency in capturing evolving dynamics among multiple forecasting targets, alongside a focus on longer forecasting horizons like monthly or yearly. To tackle these, we propose an innovative Dynamic Spatial-Temporal Graph Convolutional Recurrent Network (DSTGCRN), a blend of graph convolutional and recurrent neural network structures customized for up-to-date multi-regional spatial–temporal predictions at a daily level. Comprehensive empirical analyses on datasets from China, the US, and the EU confirm DSTGCRN’s superior performance and robustness across diverse geographical contexts. It outperforms the second-best among 10 baseline models, achieving 40.6%, 24.6%, and 38.5% improvements in MAE, RMSE, and MAPE, respectively, across various horizons. The incorporation of environmental data, including temperature and Air Quality Index, as supplementary predictors in the DSTGCRN model, has demonstrated its efficacy in the context of the China dataset. The DSTGCRN model’s improved accuracy and daily multi-regional forecasting deepen emission dynamics understanding, supporting the development of informed and region-specific environmental policies that can respond promptly to changes.

Universal approximation property of stochastic configuration networks for time series

For the purpose of processing sequential data, such as time series, and addressing the challenge of manually tuning the architecture of traditional recurrent neural networks (RNNs), this paper introduces a novel approach-the Recurrent Stochastic Configuration Network (RSCN). This network is constructed based on the random incremental algorithm of stochastic configuration networks. Leveraging the foundational structure of recurrent neural networks, our learning model commences with a modest-scale recurrent neural network featuring a single hidden layer and a solitary hidden node. Subsequently, the node parameters of the hidden layer undergo incremental augmentation through a random configuration process, with corresponding weights assigned structurally. This iterative expansion continues until the network satisfies predefined termination criteria. Noteworthy is the adaptability of this algorithm to handle time series data, exhibiting superior performance compared to traditional recurrent neural networks with similar architectures. The experimental results presented in this paper underscore the efficacy of the proposed RSCN for sequence data processing, showcasing its advantages over conventional recurrent neural networks in the context of the performed experiments.

SRNN-RSA: a new method to solving time-dependent shortest path problems based on structural recurrent neural network and ripple spreading algorithm

AbstractInfluenced by external factors, the speed of vehicles in the traffic network is changing all the time, which makes the traditional static shortest route unable to meet the real logistics distribution needs. Considering that the existing research on time-dependent shortest path problems (TDSPP) do not include the topological information of the traffic network, it is unable to reflect the spatial and temporal dynamic characteristics of the traffic network during the vehicle travelling process and is unable to update to the changes of the vehicle speed in real time, and poor scalability. Therefore, we used the structural RNN (SRNN) model containing topological information of the road network is used to predict time-varying speeds in the traffic road network. We proposed an SRNN-RSA framework for solving the TDSPP problem, which achieves a synergistic evolution between the real-time vehicle speed change process and the RSA solving process, and the scalability of the proposed SRNN-RSA is demonstrated and validated using different real data. Compared with other algorithms, the results show that SRNN-RSA has the lowest error with the actual situation, which can balance the solution accuracy and calculation speed and is more consistent with the real traffic road network, with better stability and expandability.

Aggregating knowledge and collaborative information for sequential recommendation

Sequential recommendation aims to predict users’ future activities based on their historical interaction sequences. Various neural network architectures, such as Recurrent Neural Networks (RNN), Graph Neural Networks (GNN), and self-attention mechanisms, have been employed in the tasks, exploring multiple aspects of user preferences, including general interests, short-term interests, long-term interests, and item co-occurrence patterns. Despite achieving good performance, there are still limitations in capturing complex user preferences. Specifically, the current structures of RNN, GNN, etc., only capture item-level transition relations while neglecting attribute-level transition relations. Additionally, the explicit item relations are studied using item co-occurrence modules, but they cannot capture the implicit item-item relations. To address these issues, we propose a knowledge-augmented Gated Recurrent Unit (GRU) to improve the short-term user interest module and adopt a collaborative item aggregation method to enhance the item co-occurrence module. Additionally, our long-term interest module utilizes a bitwise gating mechanism to select historical item features significant to users’ current preferences. We extensively evaluate our model on three real-world datasets alongside competitive methods, demonstrating its effectiveness in top K sequential recommendation.

Stage-Wise Magnitude-Based Pruning for Recurrent Neural Networks.

A recurrent neural network (RNN) has shown powerful performance in tackling various natural language processing (NLP) tasks, resulting in numerous powerful models containing both RNN neurons and feedforward neurons. On the other hand, the deep structure of RNN has heavily restricted its implementation on mobile devices, where quite a few applications involve NLP tasks. Magnitude-based pruning (MP) is a promising way to address such a challenge. However, the existing MP methods are mostly designed for feedforward neural networks that do not involve a recurrent structure, and, thus, have performed less satisfactorily on pruning models containing RNN layers. In this article, a novel stage-wise MP method is proposed by explicitly taking the featured recurrent structure of RNN into account, which can effectively prune feedforward layers and RNN layers, simultaneously. The connections of neural networks are first grouped into three types according to how they are intersected with recurrent neurons. Then, an optimization-based pruning method is applied to compress each group of connections, respectively. Empirical studies show that the proposed method performs significantly better than the commonly used RNN pruning methods; i.e., up to 96.84% connections are pruned with little or even no degradation of precision indicators on the testing datasets.

Prediction of medial knee contact force using multisource fusion recurrent neural network and transfer learning.

Estimation of knee contact force (KCF) during gait provides essential information to evaluate knee joint function. Machine learning has been employed to estimate KCF because of the advantages of low computational cost and real-time. However, the existing machine learning models do not adequately consider gait-related data's temporal-dependent, multidimensional, and highly heterogeneous nature. This study is aimed at developing a multisource fusion recurrent neural network to predict the medial condyle KCF. First, a multisource fusion long short-term memory (MF-LSTM) model was established. Then, we developed a transfer learning strategy based on the MF-LSTM model for subject-specific medial KCF prediction. Four subjects with instrumented tibial prostheses were obtained from the literature. The results showed that the MF-LSTM model could predict medial KCF to a certain high level of accuracy (the mean of ρ = 0.970). The transfer learning model improved the prediction accuracy (the mean of ρ = 0.987). This study shows that the MF-LSTM model is a powerful and accurate computational tool for medial KCF prediction. Introducing transfer learning techniques could further improve the prediction performance for the target subject. This coupling strategy can help clinicians accurately estimate and track joint contact forces in real time.

STDAN: Deformable Attention Network for Space-Time Video Super-Resolution.

The target of space-time video super-resolution (STVSR) is to increase the spatial-temporal resolution of low-resolution (LR) and low-frame-rate (LFR) videos. Recent approaches based on deep learning have made significant improvements, but most of them only use two adjacent frames, that is, short-term features, to synthesize the missing frame embedding, which cannot fully explore the information flow of consecutive input LR frames. In addition, existing STVSR models hardly exploit the temporal contexts explicitly to assist high-resolution (HR) frame reconstruction. To address these issues, in this article, we propose a deformable attention network called STDAN for STVSR. First, we devise a long short-term feature interpolation (LSTFI) module that is capable of excavating abundant content from more neighboring input frames for the interpolation process through a bidirectional recurrent neural network (RNN) structure. Second, we put forward a spatial-temporal deformable feature aggregation (STDFA) module, in which spatial and temporal contexts in dynamic video frames are adaptively captured and aggregated to enhance SR reconstruction. Experimental results on several datasets demonstrate that our approach outperforms state-of-the-art STVSR methods. The code is available at https://github.com/littlewhitesea/STDAN.

A Seismic Inversion Method Based on Multi-Scale Super-Asymmetric Cycle-JNet Network

In order to improve the resolution and accuracy of seismic inversion, this study constructs a multi-scale super-asymmetric network (Cycle-JNet). In this model, wavelet analysis is used to capture the multi-scale data characteristics of well-seismic data, thereby improving the machine’s ability to learn details. Using the UNet neural network from Convolutional Neural Network (CNN), we modified the network structure by adding several convolution kernel layers at the output end to expand generated data, solving the problem of mismatched resolutions in well-seismic data, thus improving the resolution of seismic inversion and achieving the purpose of accurately identifying thin sandstone layers. Meanwhile, a cycle structure of Recurrent Neural Network (RNN) was designed for the secondary learning of the seismic data generated by JNet. By comparing the data transformed through inverse wavelet transform with the original data again, the accuracy of machine learning can be improved. After optimization, the Cycle-JNet model significantly outperforms traditional seismic inversion methods in terms of resolution and accuracy. This indicates that this method can provide more precise inversion results in more complex data environments, providing stronger support for seismic analysis.

A Deep Vision Learning-Based Intelligent Recognition Method for Dynamic Sports Gestures

Nowadays, artificial intelligence-assisted automatic technological analyses of sports matches have received a more general demand. Among them, the effective recognition of sports actions from videos or images acts as the most fundamental issue to be solved. As sports gestures are characterized by remarkable specialty and instantaneity, it is required to develop specific effective recognition algorithms for this purpose. As a result, this work takes the case of Taekwondo sports as an example, and introduces deep vision learning to develop a specific intelligent recognition method for dynamic sports gestures. After sampling the key frames from dynamic videos, vital joint points of limbs are first obtained as the basic bottom features. Then, gesture expertise of this domain is formed to construct a multiview feature extraction work flow to better analyze complex characteristics of each gesture. Finally, the extracted feature representation is input into a recurrent neural network structure to output the discriminative results. At last, experiments are conducted for assessment, whose results show that the proposal can reach an accuracy of 95.6% and can be well suitable for the investigated problem scenario.

A novel vision transformer network for rolling bearing remaining useful life prediction

The accurate predictions of remaining useful life (RUL) have become a key and extremely challenging problem. Due to the limitations of the classical convolutional neural network and recurrent neural network structure, the attention mechanism has been introduced to improve feature representation of the long-term bearing degradation data. Transformer network based on attention mechanism is successfully applied in many fields and recognized as an excellent creation for deep learning models. In this paper, a novel lightweight mobile vision transformer (MobileViT) architecture based on deep networks is proposed for the RUL predictions. This new network is named prognostics separable vision Transformer (ProgSViT), which combines the separable convolution and MobileViT. In ProgSViT network, the separable convolutions are first constructed for extracting local feature from the input vibration signal, and the new vision transformer architecture is proposed to learn the global feature representations. In improved MobileViT model, the loss function is optimized, and a new training strategy is provided. Finally, the obtained features are input to the global average pool layers and the full connection layers to perform RUL estimation. Experiment results present the proposed ProgSViT network surpasses the other models in RUL predicting, which possesses higher precision and computational efficiency.

Encoder-decoder models in sequence-to-sequence learning: A survey of RNN and LSTM approaches

In today's information age, from natural language processing to audio signal processing, from time series analysis to machine translation, the application of sequence data involves various fields. Encoder-decoder models, as a powerful approach to sequence modeling, have attracted extensive attention and research. This review paper aims to explore encoder-decoder models, focusing on the principles and operational steps of recurrent neural networks (RNN) and long short-term memory (LSTM), aiming to provide researchers and practitioners with a deep understanding of the fundamentals and applications of these models Condition. Through the analysis and summary of relevant literature, this study reveals the advantages of RNN and LSTM in sequence data processing; RNN structure is simple and effective, can process sequence input and output of any length, and can capture the time dependence in sequence data. But there is a problem of gradient disappearance, which makes it difficult to deal with long-term dependencies. To solve this problem, the LSTM model introduces different gating mechanisms, which effectively solve the gradient problem while better capturing long-term dependencies. Through the review of the principles and applications of the two, it provides a useful reference for further research and application.

Combined Fault Diagnosis Scheme Based on Recurrent Neural Network and H− Observer for Satellite Attitude Control System

This study proposes a combined fault diagnosis scheme based on a recurrent neural network (RNN) and an H− observer for satellite attitude control systems (ACSs) in the presence of model uncertainties, external disturbances, and measurement noise. The ACS is decoupled into multiple independent channels, such that the residuals generated by the observers respond to the corresponding faults. A novel multilayer adaptive Gaussian recurrent neural network (MAGRNN) structure is developed as an approximator to estimate the lumped disturbance; a robust term is introduced to improve accuracy. Considering the actuator fault in the finite frequency domain and the fault-sensitive index, a set of H− unknown input observers (UIOs) is designed using the output of the MAGRNN-based approximator as compensation. The existence conditions of the approximator and observer are proposed and proved. The fault diagnosis results for three cases verify that the proposed method can be used for small fault detection and isolation.

Optimal predictive neuro-navigator design for mobile robot navigation with moving obstacles.

Introduction: The challenge of navigating a Mobile robot in dynamic environments has grasped significant attention in recent years. Despite the available techniques, there is still a need for efficient and reliable approaches that can address the challenges of real-time near optimal navigation and collision avoidance. Methods: This paper proposes a novel Log-concave Model Predictive Controller (MPC) algorithm that addresses these challenges by utilizing a unique formulation of cost functions and dynamic constraints, as well as a convergence criterion based on Lyapunov stability theory. The proposed approach is mapped onto a novel recurrent neural network (RNN) structure and compared with the CVXOPT optimization tool. The key contribution of this study is the combination of neural networks with model predictive controller to solve optimal control problems locally near the robot, which offers several advantages, including computational efficiency and the ability to handle nonlinear and complex systems. Results: The major findings of this study include the successful implementation and evaluation of the proposed algorithm, which outperforms other methods such as RRT, A-Star, and LQ-MPC in terms of reliability and speed. This approach has the potential to facilitate real-time navigation of mobile robots in dynamic environments and ensure a feasible solution for the proposed constrained-optimization problem.

The Modeling of the Rucklidge Chaotic System with Artificial Neural Networks

Chaotic systems are nonlinear systems that show sensitive dependence on initial conditions, and an immeasurably small change in initial value causes an immeasurably large change in the future state of the system. Besides, there is no randomness in chaotic systems and they have an order within themselves. Researchers use chaotic systems in many areas such as mixer systems that can make more homogeneous mixtures, encryption systems that can be used with high security, and artificial neural networks by taking the advantage of the order in this disorder. Differential equations in which chaotic systems are expressed mathematically are solved by numerical solution methods such as Heun, Euler, ODE45, RK4, RK5-Butcher and Dormand-Prince in the literature. In this research, Feed Forward Neural Network (FFNN), Layer Recurrent Neural Network (LRNN) and Cascade Forward Backpropogation Neural Network (CFNN) structures were used to model the Rucklidge chaotic system by making use of the MATLAB R2021A program Neural Network (NN) Toolbox. By comparing the results of different activation functions used in the modeling, the ANN structure that can best model the Rucklidge chaotic system has been determined. The training of the compared Artificial Neural Networks (ANNs) was carried out with the values obtained from the Euler numerical solution method, which can get satisfactory and fast results.

Determination Of A Task Force For New Police Officers At The State Police School Using A Fully Recurrent Neural Network

Artificial Neural Networks (ANN) can be used to solve specific problems such as prediction, classification, processing data, and robotics. Based on the exposure, this study tried to develop a system by applying ANN models Fully Recurrent Neural Network (FRNN) to deal with the problems of classification determination unit for New Police Officer at NES Kupang include DitSabara, DitPolAir, and SatBrimob, which has been using the system manually, the ANN system Fully Recurrent Neural Networks can provide accurate information to the NES Kupang to determine the right decision. Fully Recurrent Neural Network structures have been presence of feedback that can make faster iteration thus making the update parameters and convergence speed become faster. The learning method used is Backpropagation Through Time . The system is implemented using the C# program. Input vectors used consisted of 7 variables.The results showed t the developed system will rapidly converge and able to achieve the most optimal error value (minimum error) when using one hidden layer with 17 units of the number of neurons . The best accuracy can be obtained using the LR of 0.001 , target of 0.1 and momentum 0.95, with 25 test data of data, the system accuracy reaches 96%, while the real data, the accuracy reached 83.33%.

Fault-tolerant quaternary belief propagation decoding based on a neural network

The article discusses the challenge of finding an efficient decoder for quantum error correction codes for fault-tolerant experiments in quantum computing. The study aims to develop a better decoding scheme based on the flag-bridge fault tolerance experiment. The research compares two decoding algorithms, a deep neural network decoding scheme and a simple decoder, and a recurrent neural network decoding scheme based on the belief propagation algorithm variant MBP4 algorithm. The study improved the syndrome extraction circuit based on the flag-bridge method to meet the requirements of fault-tolerant experiments better. Two decoding schemes were studied, a combination of a deep neural network and a simple decoder and a recurrent neural network structure based on the MBP4 algorithm. The first scheme used neural networks to assist simple decoders in determining whether additional logical corrections need to be added. The second scheme used a recurrent neural network structure designed through the variant MBP4 algorithm, along with a post-processing method to pinpoint the error qubit position for decoding. Experimental results showed that the decoding scheme developed in the study improved the pseudo-threshold by 39.52% compared to the minimum-weight perfect matching decoder. The two decoders had thresholds of approximately 15.8% and 16.4%, respectively. The study’s findings suggest that the proposed decoding schemes could improve quantum error correction and fault-tolerant experiments in quantum computing.

Intelligent auxiliary system for music performance under edge computing and long short-term recurrent neural networks.

Music performance action generation can be applied in multiple real-world scenarios as a research hotspot in computer vision and cross-sequence analysis. However, the current generation methods of music performance actions have consistently ignored the connection between music and performance actions, resulting in a strong sense of separation between visual and auditory content. This paper first analyzes the attention mechanism, Recurrent Neural Network (RNN), and long and short-term RNN. The long and short-term RNN is suitable for sequence data with a strong temporal correlation. Based on this, the current learning method is improved. A new model that combines attention mechanisms and long and short-term RNN is proposed, which can generate performance actions based on music beat sequences. In addition, image description generative models with attention mechanisms are adopted technically. Combined with the RNN abstract structure that does not consider recursion, the abstract network structure of RNN-Long Short-Term Memory (LSTM) is optimized. Through music beat recognition and dance movement extraction technology, data resources are allocated and adjusted in the edge server architecture. The metric for experimental results and evaluation is the model loss function value. The superiority of the proposed model is mainly reflected in the high accuracy and low consumption rate of dance movement recognition. The experimental results show that the result of the loss function of the model is at least 0.00026, and the video effect is the best when the number of layers of the LSTM module in the model is 3, the node value is 256, and the Lookback value is 15. The new model can generate harmonious and prosperous performance action sequences based on ensuring the stability of performance action generation compared with the other three models of cross-domain sequence analysis. The new model has an excellent performance in combining music and performance actions. This paper has practical reference value for promoting the application of edge computing technology in intelligent auxiliary systems for music performance.

Multiplexing working memory and time in the trajectories of neural networks.

Working memory (WM) and timing are generally considered distinct cognitive functions, but similar neural signatures have been implicated in both. To explore the hypothesis that WM and timing may rely on shared neural mechanisms, we used psychophysical tasks that contained either task-irrelevant timing or WM components. In both cases, the task-irrelevant component influenced performance. We then developed recurrent neural network (RNN) simulations that revealed that cue-specific neural sequences, which multiplexed WM and time, emerged as the dominant regime that captured the behavioural findings. During training, RNN dynamics transitioned from low-dimensional ramps to high-dimensional neural sequences, and depending on task requirements, steady-state or ramping activity was also observed. Analysis of RNN structure revealed that neural sequences relied primarily on inhibitory connections, and could survive the deletion of all excitatory-to-excitatory connections. Our results indicate that in some instances WM is encoded in time-varying neural activity because of the importance of predicting when WM will be used.

Status Quo and Influence of Martial Arts Fitness in Pursuit of Health Using LSTM Recurrent Neural Network Algorithm

With the acceleration of the socialist modernization process, people’s living standards continue to improve, and the modernization of cities is also developing rapidly. In the process of urban construction, a series of problems such as how to deal with the relationship between citizens and the surrounding environment, how to carry out citizens' health activities, and how to organize citizens to exercise healthily are becoming more and more obvious. With the introduction of the National Sports Outline, as an important sports activity, Chinese martial arts has a long history and profound cultural heritage, and it still plays an irreplaceable role in promoting the national spirit and building a harmonious society today. Therefore, this paper introduced the LSTM recurrent neural network algorithm to collect and extract the current data on martial arts fitness in the pursuit of health. By constructing the LSTM neuron structure and building the LSTM recurrent neural network structure, a state estimation based on LSTM feature extraction was proposed. The algorithm used the memory ability of LSTM to extract the relevant features of the data stream, which significantly improved the accuracy of the data stream collection. The experimental results showed that the time required by the method in this paper were 12.7, 10.2, 12.4 s and 11.8 min respectively, and the accuracy rates were 98.78, 98.26, 99.03 and 97.89%, respectively, which were greatly improved compared with the existing methods.