Recurrent Neural Network Learning Research Articles

Gradient-free training of recurrent neural networks using random perturbations.

Recurrent neural networks (RNNs) hold immense potential for computations due to their Turing completeness and sequential processing capabilities, yet existing methods for their training encounter efficiency challenges. Backpropagation through time (BPTT), the prevailing method, extends the backpropagation (BP) algorithm by unrolling the RNN over time. However, this approach suffers from significant drawbacks, including the need to interleave forward and backward phases and store exact gradient information. Furthermore, BPTT has been shown to struggle to propagate gradient information for long sequences, leading to vanishing gradients. An alternative strategy to using gradient-based methods like BPTT involves stochastically approximating gradients through perturbation-based methods. This learning approach is exceptionally simple, necessitating only forward passes in the network and a global reinforcement signal as feedback. Despite its simplicity, the random nature of its updates typically leads to inefficient optimization, limiting its effectiveness in training neural networks. In this study, we present a new approach to perturbation-based learning in RNNs whose performance is competitive with BPTT, while maintaining the inherent advantages over gradient-based learning. To this end, we extend the recently introduced activity-based node perturbation (ANP) method to operate in the time domain, leading to more efficient learning and generalization. We subsequently conduct a range of experiments to validate our approach. Our results show similar performance, convergence time and scalability when compared to BPTT, strongly outperforming standard node perturbation and weight perturbation methods. These findings suggest that perturbation-based learning methods offer a versatile alternative to gradient-based methods for training RNNs which can be ideally suited for neuromorphic computing applications.

RECURRENT NEURAL NETWORK LEARNING ALGORITHM-BASED EVENT-TRIGGERED OBSERVER OF THE PERMANENT MAGNET SYNCHRONOUS MOTOR

In this paper, we propose a novel method to estimate the electrical angular velocity, the electrical angle, and the currents of the permanent magnet synchronous motor. A recurrent neural network learning algorithm is first developed to estimate the states of the permanent magnet synchronous motor. Then, an event-triggered state observer is designed for the recurrent neural network. This state observer robustly estimates state variables of the permanent magnet synchronous motor. A sufficient condition in terms of a convex optimization problem for the existence of the event-triggered state observer is established. In contrast with the abundance of state estimation methods based on time-triggered state observers where the measurements are always continuously available, the ones in this paper are updated when an event-triggered condition holds. Therefore, it lessens the stress on communication resources while can still maintain an estimation performance. Simulation results are provided to demonstrate the merit of the proposed method.

RECURRENT NEURAL NETWORK LEARNING ALGORITHM-BASED EVENT-TRIGGERED OBSERVER OF THE PERMANENT MAGNET SYNCHRONOUS MOTOR

In this paper, we propose a novel method to estimate the electrical angular velocity, the electrical angle, and the currents of the permanent magnet synchronous motor. A recurrent neural network learning algorithm is first developed to estimate the states of the permanent magnet synchronous motor. Then, an event-triggered state observer is designed for the recurrent neural network. This state observer robustly estimates state variables of the permanent magnet synchronous motor. A sufficient condition in terms of a convex optimization problem for the existence of the event-triggered state observer is established. In contrast with the abundance of state estimation methods based on time-triggered state observers where the measurements are always continuously available, the ones in this paper are updated when an event-triggered condition holds. Therefore, it lessens the stress on communication resources while can still maintain an estimation performance. Simulation results are provided to demonstrate the merit of the proposed method.

Fuzzy Deep Learning Recurrent Neural Network Algorithm to Detect Corn Leaf Disease

Corn is a major commodity after rice in supporting food self-sufficiency in Indonesia. However, due to leaf disease, the quality and quantity of corn plants are greatly reduced. The problem with detecting corn leaf diseases is that the detection method is still manual, making it inefficient and ineffective. Therefore, in this study, disease detection on corn leaves was performed using the Fuzzy C-Means (FCM) and Long Short-Term Memory (LSTM) methods. First, oversampling was carried out to ensure an equal amount of data in all classes, then the corn leaf images were pre-processed before being input into the LSTM algorithm. After completing clustering process in the [Formula: see text] algorithm, the next step involved extracting texture features using the Gray Level Co-occurrence Matrix (GLCM) technique, followed by classification using LSTM. To assess their performance, both algorithms underwent evaluation using the k-fold cross-validation method, and their accuracy and speed were compared. The results of the k-fold cross-validation demonstrated that the [Formula: see text] algorithm achieved an accuracy of 63.53%, whereas the LSTM algorithm achieved an accuracy of 80.24%. In terms of the time required for training and prediction, the LSTM algorithm took 13[Formula: see text]min and 18[Formula: see text]s for training on corn leaf disease images, while the prediction process only took 1.59[Formula: see text]s. The training and prediction time required for the [Formula: see text] algorithm were 65[Formula: see text]min and 24[Formula: see text]s and 5[Formula: see text]min and 44[Formula: see text]s, respectively. The conclusion of this study is that the LSTM algorithm has better accuracy and time compared to [Formula: see text] on the dataset used in this study in terms of corn leaf disease detection.

Diabetic Drug ontology mapping for individual Diabetic Person and predict Insulin Dosage on Daily Basis

Managing diabetes requires a personalized treatment plan based on the patient's health records and drug usage effectively. In this paper, we have proposed accurate prediction of insulin dosage for diabetic patients daily by using ontology mapping with deep learning recurrent neural network models such as long short-term memory (LSTM), Gated recurrent unit( GRU), and Bidirectional long short-term memory(BiLSTM). The patient's information is arranged in a structured format. The structured ontology data provides a meaningful connection between patients' personal information, physical activity, insulin intake, carbohydrate intake and glucose level. The structured data is beneficial to take decisions effectively to recommend the insulin dosage level daily to avoid a sudden decrease in sugar level. The proposed LSTM model outperforms well on insulin prediction compared to GRU and BiLSTM models. The proposed model is very effective for diabetic personal care and improves patient outcomes and quality of life.

Prediction of medial knee contact force using multisource fusion recurrent neural network and transfer learning.

Estimation of knee contact force (KCF) during gait provides essential information to evaluate knee joint function. Machine learning has been employed to estimate KCF because of the advantages of low computational cost and real-time. However, the existing machine learning models do not adequately consider gait-related data's temporal-dependent, multidimensional, and highly heterogeneous nature. This study is aimed at developing a multisource fusion recurrent neural network to predict the medial condyle KCF. First, a multisource fusion long short-term memory (MF-LSTM) model was established. Then, we developed a transfer learning strategy based on the MF-LSTM model for subject-specific medial KCF prediction. Four subjects with instrumented tibial prostheses were obtained from the literature. The results showed that the MF-LSTM model could predict medial KCF to a certain high level of accuracy (the mean of ρ = 0.970). The transfer learning model improved the prediction accuracy (the mean of ρ = 0.987). This study shows that the MF-LSTM model is a powerful and accurate computational tool for medial KCF prediction. Introducing transfer learning techniques could further improve the prediction performance for the target subject. This coupling strategy can help clinicians accurately estimate and track joint contact forces in real time.

Designing of VehiNet Using Convolutional Neural Networks and Deep Learning Techniques

Traffic congestion has become a critical issue in metropolitan areas, inflicting significant hardships on individuals on a daily basis. This adverse situation poses multifaceted challenges, including increased air pollution, heightened fuel consumption, and compromised road safety. Deep learning algorithms, encompassing both Convolutional Neural Networks (CNN), and Recurrent Neural Networks (RNN) have showcased remarkable performance in areas such as computer vision, natural language processing, healthcare, finance, and autonomous systems. This research paper proposes a novel methodology to develop a vehicle detection model - VehiNet, which can subsequently be refined to create an automated traffic signal system. The study explores the effectiveness of various CNN, RNN, and deep learning algorithms and techniques in accomplishing this objective. The dataset picked for the study in addition to being compiled from internet resources, was also painstakingly assembled manually using a variety of methods to collect over 2000 photographs from various angles, lighting conditions, and locations within Bengaluru city. The main motive is to apply CNN, RNN, and other Deep Learning classification techniques for object detection and classification of Indian vehicles and see how well these deep learning algorithms and techniques affect prediction efficiency. This research stands out from others and makes its novelty clear thanks to the carefully collected dataset and the detailed analysis of the methodologies.

An Inexact Sequential Quadratic Programming Method for Learning and Control of Recurrent Neural Networks.

This article considers the two-stage approach to solving a partially observable Markov decision process (POMDP): the identification stage and the (optimal) control stage. We present an inexact sequential quadratic programming framework for recurrent neural network learning (iSQPRL) for solving the identification stage of the POMDP, in which the true system is approximated by a recurrent neural network (RNN) with dynamically consistent overshooting (DCRNN). We formulate the learning problem as a constrained optimization problem and study the quadratic programming (QP) subproblem with a convergence analysis under a restarted Krylov-subspace iterative scheme that implicitly exploits the structure of the associated Karush-Kuhn-Tucker (KKT) subsystem. In the control stage, where a feedforward neural network (FNN) controller is designed on top of the RNN model, we adapt a generalized Gauss-Newton (GGN) algorithm that exploits useful approximations to the curvature terms of the training data and selects its mini-batch step size using a known property of some regularization function. Simulation results are provided to demonstrate the effectiveness of our approach.

The Analysis of Deep Learning Recurrent Neural Network in English Grading Under the Internet of Things

The Analysis of Deep Learning Recurrent Neural Network in English Grading Under the Internet of Things

Retracted: Telemetry Data Compression Algorithm Using Balanced Recurrent Neural Network and Deep Learning.

[This retracts the article DOI: 10.1155/2022/4886586.].

Statistical Online Learning in Recurrent and Feedforward Quantum Neural Networks

Statistical Online Learning in Recurrent and Feedforward Quantum Neural Networks

Interpretability of simple RNN and GRU deep learning models used to map land susceptibility to gully erosion

Gully erosion possess a serious hazard to critical resources such as soil, water, and vegetation cover within watersheds. Therefore, spatial maps of gully erosion hazards can be instrumental in mitigating its negative consequences. Among the various methods used to explore and map gully erosion, advanced learning techniques, especially deep learning (DL) models, are highly capable of spatial mapping and can provide accurate predictions for generating spatial maps of gully erosion at different scales (e.g., local, regional, continental, and global). In this paper, we applied two DL models, namely a simple recurrent neural network (RNN) and a gated recurrent unit (GRU), to map land susceptibility to gully erosion in the Shamil-Minab plain, Hormozgan province, southern Iran. To address the inherent black box nature of DL models, we applied three novel interpretability methods consisting of SHaply Additive explanation (SHAP), ceteris paribus and partial dependence (CP-PD) profiles and permutation feature importance (PFI). Using the Boruta algorithm, we identified seven important features that control gully erosion: soil bulk density, clay content, elevation, land use type, vegetation cover, sand content, and silt content. These features, along with an inventory map of gully erosion (based on a 70 % training dataset and 30 % test dataset), were used to generate spatial maps of gully erosion using DL models. According to the Kolmogorov–Smirnov (KS) statistic performance assessment measure, the simple RNN model (with KS = 91.6) outperformed the GRU model (with KS = 66.6). Based on the results from the simple RNN model, 7.4 %, 14.5 %, 18.9 %, 31.2 % and 28 % of total area of the plain were classified as very-low, low, moderate, high and very-high hazard classes, respectively. According to SHAP plots, CP-PD profiles, and PFI measures, soil silt content, vegetation cover (NDVI) and land use type had the highest impact on the model's output. Overall, the DL modelling techniques and interpretation methods used in this study proved to be helpful in generating spatial maps of soil erosion hazard, especially gully erosion. Their interpretability can support watershed sustainable management.

Motion prediction using brain waves based on artificial intelligence deep learning recurrent neural network.

Electroencephalogram (EEG) research has gained widespread use in various research domains due to its valuable insights into human body movements. In this study, we investigated the optimization of motion discrimination prediction by employing an artificial intelligence deep learning recurrent neural network (gated recurrent unit, GRU) on unique EEG data generated from specific movement types among EEG signals. The experiment involved participants categorized into five difficulty levels of postural control, targeting gymnasts in their twenties and college students majoring in physical education (n=10). Machine learning techniques were applied to extract brain-motor patterns from the collected EEG data, which consisted of 32 channels. The EEG data underwent spectrum analysis using fast Fourier transform conversion, and the GRU model network was utilized for machine learning on each EEG frequency domain, thereby improving the performance index of the learning operation process. Through the development of the GRU network algorithm, the performance index achieved up to a 15.92% improvement compared to the accuracy of existing models, resulting in motion recognition accuracy ranging from a minimum of 94.67% to a maximum of 99.15% between actual and predicted values. These optimization outcomes are attributed to the enhanced accuracy and cost function of the GRU network algorithm's hidden layers. By implementing motion identification optimization based on artificial intelligence machine learning results from EEG signals, this study contributes to the emerging field of exercise rehabilitation, presenting an innovative paradigm that reveals the interconnectedness between the brain and the science of exercise.

On-site instrumental seismic intensity prediction for China via recurrent neural network and transfer learning

On-site instrumental seismic intensity prediction for China via recurrent neural network and transfer learning

Fault Identification and Classification of Asynchronous Motor Drive Using Optimization Approach with Improved Reliability

This article aims to provide a technique for identifying and categorizing interturn insulation problems in variable-speed motor drives by combining Salp Swarm Optimization (SSO) with Recurrent Neural Network (RNN). The goal of the proposed technique is to detect and classify Asynchronous Motor faults at their early stages, under both normal and abnormal operating conditions. The proposed technique uses a recurrent neural network in two phases to identify and label interturn insulation concerns, with the first phase being utilised to establish whether or not the motors are healthy. In the second step, it discovers and categorises potentially dangerous interturn errors. The SSO approach is used in the second phase of the recurrent neural network learning procedure, with the goal function of minimizing error in mind. The proposed CSSRN technique simplifies the system for detecting and categorizing the interturn insulation issue, resulting in increased system precision. In addition, the proposed model is implemented in the MATLAB/Simulink, where metrics such as accuracy, precision, recall, and specificity may be analysed. Similarly, existing methods such as Adaptive Neuro-Fuzzy Inference System (ANFIS), Recurrent Neural Network (RNN), and Salp Swarm Algorithm Artificial Neural Network (SSAANN) are utilised to evaluate metrics such as Root mean squared error (RMSE), Mean bias error (MBE), Mean absolute percentage error (MAPE), consumption, and execution time for comparative analysis.

LSTM and GRU based Accurate Water Quality Prediction for Smart Aquaculture

The fast and precise water quality prediction in smart aquaculture can assist the farmers in taking corrective actions before the ecological environment deteriorates. However, the empirical methods take high computation time and provide low accuracy in water quality prediction due to the non-linear and dynamic nature of water quality parameters. This research work proposes long short-term memory (LSTM) and gated recurrent unit (GRU) deep learning recurrent neural network (DL-RNN) models for aquaculture water quality prediction (A-WQP). This work also presents an extensive study about the impact of hyper-parameters (hp ) on the performance of the proposed DL-RNN models using two different water quality datasets. The experimental evaluations show that for the optimal sets of hyper-parameters, the proposed DL models offer superior prediction accuracy and computation efficiency.

Fog-cloud based intrusion detection system using Recurrent Neural Networks and feature selection for IoT networks

Deep learning (DL) techniques are being widely researched for their effectiveness in detecting cyber intrusions against the Internet of Things (IoT). Time sensitive Critical Infrastructures (CIs) that rely on IoT require rapid detection of cyber intrusions close to the constrained devices in order to prevent service delays. Deep learning techniques perform better in detecting attacks compared to shallow machine learning algorithms and can be used for intrusion detection. However, communication overheads due to large volume of IoT data and computation requirements for deep learning models prevents effective application of deep learning models closer to the constrained devices. Existing IDS techniques are either based on shallow learning algorithms or not trained on relevant IoT datasets and furthermore not designed for distributed fog-cloud deployment. To counter these issues, we propose a novel fog-cloud based IoT intrusion detection framework which incorporates a distributed processing by splitting the dataset according to attack class and a feature selection step on time-series IoT data. This is followed by a deep learning Recurrent Neural Network (SimpleRNN and Bi-directional Long Short-Term Memory (LSTM)) for attack detection. The effectiveness of the proposed approach was evaluated using the high-dimensional BoT-IoT dataset which contains large volumes of realistic IoT attack traffic. Results show that feature selection methods significantly reduced the dataset size by 90% under the computation requirements without compromising on the attack detection ability. The models built on reduced dataset achieved higher recall rate compared to models trained on full feature set without loosing class differentiation ability. The SimpleRNN and Bi-LSTM models also did not suffer any underfitting or overfitting with the reduced feature space. The proposed deep learning based IoT intrusion detection framework is suitable for fog-cloud based deployment and can scale well even with large volumes of IoT data.

Emergency Vehicle Driving Assistance System Using Recurrent Neural Network with Navigational Data Processing Method

Emergency vehicle transportation is important for responding to and transporting individuals during emergencies. This type of transportation faces several issues, such as road safety, navigation and communication, time-critical operations, resource utilisation, traffic congestion, data processing and analysis, and individual safety. Vehicle navigation and coordination is a critical aspect of emergency response that involves guiding emergency vehicles, such as ambulances, to the location of an emergency or medical centre as quickly and safely as possible. Therefore, it requires additional effort to reduce driving risks. The roadside units support emergency vehicles and infrastructure to decrease collisions and enhance optimal navigation routes. However, during the emergency vehicle’s data communication and navigation process, communication is interrupted due to vehicle outages. Therefore, this study proposes the Navigation Data Processing for Assisted Driving (NDP-AD) method to address the problem. The proposed approach assimilates infrastructure and neighbouring location information during driving. The integrated information is processed for distance and traffic during the previous displacement interval. The NDP-AD method employs a recurrent neural network learning approach to analyse opposing vehicle distance and traffic to provide accurate, independent guidance. This effective learning-based guidance process minimises false navigations and deviation in displacement. System efficiency is evaluated based on processing latency, displacement error, data utilisation, false rate, and accuracy metrics.

Multi-Window Identification of Landslide Hazards Based on InSAR Technology and Factors Predisposing to Disasters

Identification of potential landslide hazards is of great significance for disaster prevention and control. CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks) and many other deep learning methods have been used to identify landslide hazards. However, most samples are made with a fixed window size, which affects recognition accuracy to some extent. This paper presents a multi-window hidden danger identification CNN method according to the scale of the landslide in the experimental area. Firstly, the hidden danger area is preliminarily screened by InSAR deformation processing technology. Secondly, based on topography, geology, hydrology and human activities, a total of 15 disaster-prone factors are used to create factor datasets for in-depth learning. According to the general scale of the landslide, models with four window sizes of 48 × 48, 32 × 32, 16 × 16 and 8 × 8 are trained, respectively, and several window models with better recognition effect and suitable for the scale of landslide in the experimental area are selected for the accurate identification of landslide hazards. The results show that, among the four windows, 16 × 16 and 8 × 8 windows have the best model recognition effect. Then, according to the scale of the landslide, these optimal windows are pertinently selected, and the precision, recall rate and F-measure of the multi-window deep learning model are improved (82.86%, 78.75%, 80.75%). The research results prove that the multi-window identification method of landslide hazards combining InSAR technology and factors predisposing to disasters is effective, which can play an important role in regional disaster identification and enhance the scientific and technological support ability of geological disaster prevention and mitigation.

Deep evidential learning in diffusion convolutional recurrent neural network

<abstract><p>Graph neural networks (GNNs) is applied successfully in many graph tasks, but there still exists a limitation that many of GNNs model do not consider uncertainty quantification of its output predictions. For uncertainty quantification, there are mainly two types of methods which are frequentist and Bayesian. But both methods need to sampling to gradually approximate the real distribution, in contrast, evidential deep learning formulates learning as an evidence acquisition process, which could get uncertainty quantification by placing evidential priors over the original Gaussian likelihood function and training the NN to infer the hyperparameters of the evidential distribution without sampling. So evidential deep learning (EDL) has its own advantage in measuring uncertainty. We apply it with diffusion convolutional recurrent neural network (DCRNN), and do the experiment in spatiotemporal forecasting task in a real-world traffic dataset. And we choose mean interval scores (MIS), a good metric for uncertainty quantification. We summarized the advantages of each method.</p></abstract>