Recurrent Neural Network Model Research Articles

Reinforcement learning based MPC with neural dynamical models

This paper presents an end-to-end learning approach to developing a Nonlinear Model Predictive Control (NMPC) policy, which does not require an explicit first-principles model and assumes that the system dynamics are either unknown or partially known. The paper proposes the use of available measurements to identify a nominal Recurrent Neural Network (RNN) model to capture the nonlinear dynamics, which includes constraints on the state variables and inputs. To address the issue of suboptimal control policies resulting from simply fitting the model to the data, this paper uses Reinforcement learning (RL) to tune the NMPC scheme and generate an optimal policy for the real system. The approach’s novelty lies in the use of RL to overcome the limitations of the nominal RNN model and generate a more accurate control policy. The paper discusses the implementation aspects of initial state estimation for RNN models and integration of neural models in MPC. The presented method is demonstrated on a classic benchmark control problem: cascaded two tank system (CTS).

Emotion Recognition in Reddit Comments Using Recurrent Neural Networks

Background: Reddit comments are a valuable source of natural language data where emotion plays a key role in human communication. However, emotion recognition is a difficult task that requires understanding the context and sentiment of the texts. In this paper, we aim to compare the effectiveness of four recurrent neural network (RNN) models for classifying the emotions of Reddit comments. Methods: We use a small dataset of 4,922 comments labeled with four emotions: approval, disapproval, love, and annoyance. We also use pre-trained Glove.840B.300d embeddings as the input representation for all models. The models we compare are SimpleRNN, Long ShortTerm Memory (LSTM), bidirectional LSTM, and Gated Recurrent Unit (GRU). We experiment with different text preprocessing steps, such as removing stopwords and applying stemming, removing negation from stopwords, and the effect of setting the embedding layer as trainable on the models. Results: We find that GRU outperforms all other models, achieving an accuracy of 74%. Bidirectional LSTM and LSTM are close behind, while SimpleRNN performs the worst. We observe that the low accuracy is likely due to the presence of sarcasm, irony, and complexity in the texts. We also notice that setting the embedding layer as trainable improves the performance of LSTM but increases the computational cost and training time significantly. We analyze some examples of misclassified texts by GRU and identify the challenges and limitations of the dataset and the models Conclusion: In our study GRU was found to be the best model for emotion classification of Reddit comments among the four RNN models we compared. We also discuss some future directions for research to improve the emotion recognition task on Reddit comments. Furthermore, we provide an extensive discussion of the applications and methods behind each technique in the context of the paper.

Diagnosing OSA and Insomnia at Home Based Only on an Actigraphy Total Sleep Time and RIP Belts an Algorithm "Nox Body Sleep™".

The COVID-19 pandemic has influenced clinical sleep protocols with stricter hospital disinfection requirements. Facing these new rules, we tested if a new artificial intelligence (AI) algorithm: The Nox BodySleep™ (NBS) developed without airflow signals for the analysis of sleep might assess pertinently sleep in patients with Obstructive Sleep Apnea (OSA) and chronic insomnia (CI) as a control group, compared to polysomnography (PSG) manual scoring. NBS is a recurrent neural network model that estimates Wake, NREM, and REM states, given features extracted from activity and respiratory inductance plethysmography (RIP) belt signals (Nox A1 PSG). Sleep states from 139 PSG studies (CI N = 72; OSA N = 67) were analyzed by NBS and compared to manually scored PSG using positive percentage agreement, negative percentage agreement, and overall agreement metrics. Similarly, we compared common sleep parameters and OSA severity using sleep states estimated by NBS for each recording and compared to manual scoring using Bland-Altman analysis and intra-class correlation coefficient. For 127,170 sleep epochs, an overall agreement of 83% was reached for Wake, NREM and REM states (92% for REM states in CI patients) between NBS and manually scored PSG. Overall agreement for estimating OSA severity was 100% for moderate-severe OSA and 91% for minimal OSA. The absolute errors of the apnea-hypopnea index (AHI) and total sleep time (TST) were significantly lower for the NBS compared to no scoring of sleep. The intra-class correlation was higher for AHI and significantly higher for TST using the NBS compared to no scoring of sleep. NBS gives sleep states, parameters and AHI with a good positive and negative percentage agreement, compared with manually scored PSG.

Speech decoding from stereo-electroencephalography (sEEG) signals using advanced deep learning methods

Objective. Brain–computer interfaces (BCIs) are technologies that bypass damaged or disrupted neural pathways and directly decode brain signals to perform intended actions. BCIs for speech have the potential to restore communication by decoding the intended speech directly. Many studies have demonstrated promising results using invasive micro-electrode arrays and electrocorticography. However, the use of stereo-electroencephalography (sEEG) for speech decoding has not been fully recognized. Approach. In this research, recently released sEEG data were used to decode Dutch words spoken by epileptic participants. We decoded speech waveforms from sEEG data using advanced deep-learning methods. Three methods were implemented: a linear regression method, an recurrent neural network (RNN)-based sequence-to-sequence model (RNN), and a transformer model. Main results. Our RNN and transformer models outperformed the linear regression significantly, while no significant difference was found between the two deep-learning methods. Further investigation on individual electrodes showed that the same decoding result can be obtained using only a few of the electrodes. Significance. This study demonstrated that decoding speech from sEEG signals is possible, and the location of the electrodes is critical to the decoding performance.

Investigating the Impact of Data Normalization Methods on Predicting Electricity Consumption in a Building Using different Artificial Neural Network Models.

The study investigates the impact of data normalization on the prediction of electricity consumption in buildings using four multilayer Artificial Neural Networks (ANN) algorithms: Long Short-Term Memory Networks (LSTM), Levenberg-Marquardt Back-propagation (LMBP), Recurrent Neural Networks (RNN), and General Regression Neural Network (GRNN). Four data normalization approaches, Min-Max Scaling, Mean, Z-score, and Gaussian function were assessed on experimental datasets. The LSTM algorithm, when combined with Min-Max normalization, showed the most favorable predictive capabilities, with a low Coefficient of Variation of the Root Mean Square Error (CVRMSE) of 10.3 and Normalized Mean Bias Error (NMBE) of 0.6. The remaining three normalization approaches showed satisfactory concordance with empirical data, but with slight disparities in precision. The LMBP model, when using Z-score normalization, had favorable performance in forecasting electricity consumption, but the discrepancies across the models were not significant. The Recurrent Neural Network (RNN) model, when used with Gaussian normalization, exhibited the most favorable performance, with the lowest Coefficient of Variation of Root Mean Square Error (CVRMSE) at 11.8 and Normalized Mean Biased Error (NMBE) at 0.6. The Generalized Regression Neural Network (GRNN) model, trained on unprocessed data, exhibited superior performance, with the lowest Coefficient of Variation of Root Mean Square Error (CVRMSE) at 19.2 and NMBE at 1.0. In conclusion, the study highlights the significant influence of data normalization on the predictive capabilities of various ANN models, suggesting that careful use of data normalization techniques can significantly improve the accuracy of electricity consumption forecasting in buildings.

Real estate price forecasting utilizing recurrent neural networks incorporating genetic algorithms

This study aims to develop and examine the effectiveness of the Recurrent Neural Network (RNN) model incorporating Genetic Algorithm (GA) in forecasting real estate prices. Real estate prices have a significant impact on a country’s financial system. Therefore, the ability to accurately forecast its price is valuable. A set of data containing 5.4 million unique records of real estate with their prices is used in the study. The data set, which spans from 2018 to 2021, contains twelve independent variables and one dependent variable. We preprocessed the data set to reduce noise and outliers that could potentially lead to poor model performance. The RNN model was selected because (GA) optimises the hyperparameters in the Recurrent Neural Network (RNN) hidden layer to optimise the performance of the RNN model. Relative Root Mean Squared Error (RRMSE), Root Mean Square Error (RMSE), Mean Absolute Percentage Error (MAPE), and Mean Absolute Error (MAE) examine the effectiveness of the RNN-GA model.The results show that the RNN-GA outperforms RNN and the traditional statistical models used in the real estate industry. This study provides an understanding of the effectiveness of incorporating GA in optimizing the RNN model for real estate price forecasting, which could benefit stakeholders in the real estate industry and sustain the financial system. This study uses a novel hybrid RNN-GA model for predicting housing prices with a large dataset.

Comparison of machine and deep learning algorithms using Google Earth Engine and Python for land classifications

Classifying land use and land cover (LULC) is essential for various environmental monitoring and geospatial analysis applications. This research focuses on land classification in District Sukkur, Pakistan, employing the comparison between machine and deep learning models. Three satellite indices, namely, NDVI, MNDWI, and NDBI, were derived from Landsat-8 data and utilized to classify four primary categories: Built-up Area, Water Bodies, Barren Land, and Vegetation. The main objective of this study is to evaluate and compare the effectiveness of comparison of machine and deep learning models. The machine learning models including Random Forest achieved an overall accuracy of 91.3% and a Kappa coefficient of 0.90. It accurately classified 2.7% of the area as Built-up Area, 1.9% as Water Bodies, 54.8% as Barren Land, and 40.4% as Vegetation. While slightly less accurate, Decision Tree model provided reliable classifications. Deep learning models showed significant accuracy, of Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). The CNN model achieved an impressive overall accuracy of 97.3%, excelling in classifying Water Bodies with User and Producer Accuracy exceeding 99%. The RNN model, with an overall accuracy of 96.2%, demonstrated strong performance in categorizing Vegetation. These findings offer valuable insights into the potential applications of machine learning and deep learning models for perfect land classifications, with implications for environmental monitoring management and geospatial analysis. The rigorous validation and comparative analysis of these models contribute to advancing remote sensing techniques and their utilization in land classification tasks. This research presents a significant contribution to the field and underscores the importance of precise land classification in the context of sustainable land management and environmental conservation.

Prediction Model of E-commerce Users' Purchase Behavior Based on Deep Learning

This study mainly uses Deep Learning (DL) technology to build a prediction model of e-commerce users' purchasing behavior, and evaluates its application effect on actual e-commerce data. Methodologically, Recurrent Neural Network (RNN) is used to capture the temporal dependence of user behavior, and the performance of the model is improved through detailed data preprocessing and feature engineering. The research results show that the DL model based on RNN has achieved remarkable advantages in predicting the purchase behavior of e-commerce users. Compared with other methods, RNN model can capture the temporal dependence of user behavior more accurately, thus improving the prediction accuracy. This advantage is especially obvious when dealing with complex and dynamic user behavior data. It provides new ideas and methods for accurate marketing and personalized service of e-commerce industry.

Exploration–Exploitation Mechanisms in Recurrent Neural Networks and Human Learners in Restless Bandit Problems

A key feature of animal and human decision-making is to balance the exploration of unknown options for information gain (directed exploration) versus selecting known options for immediate reward (exploitation), which is often examined using restless bandit tasks. Recurrent neural network models (RNNs) have recently gained traction in both human and systems neuroscience work on reinforcement learning, due to their ability to show meta-learning of task domains. Here we comprehensively compared the performance of a range of RNN architectures as well as human learners on restless four-armed bandit problems. The best-performing architecture (LSTM network with computation noise) exhibited human-level performance. Computational modeling of behavior first revealed that both human and RNN behavioral data contain signatures of higher-order perseveration, i.e., perseveration beyond the last trial, but this effect was more pronounced in RNNs. In contrast, human learners, but not RNNs, exhibited a positive effect of uncertainty on choice probability (directed exploration). RNN hidden unit dynamics revealed that exploratory choices were associated with a disruption of choice predictive signals during states of low state value, resembling a win-stay-loose-shift strategy, and resonating with previous single unit recording findings in monkey prefrontal cortex. Our results highlight both similarities and differences between exploration behavior as it emerges in meta-learning RNNs, and computational mechanisms identified in cognitive and systems neuroscience work.

A Hybrid CNN-RNN Model for Enhanced Anemia Diagnosis: A Comparative Study of Machine Learning and Deep Learning Techniques

This study proposes a hybrid Convolutional Neural Network-Recurrent Neural Network (CNN-RNN) model for the accurate diagnosis of anemia types, leveraging the strengths of both architectures in capturing spatial and temporal patterns in Complete Blood Count (CBC) data. The research involves the development and evaluation of various models of single-architecture deep learning (DL) models, specifically Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Fully Convolutional Network (FCN). The models are trained and validated using stratified k-fold cross-validation to ensure robust performance. Key metrics such as test accuracy are utilized to provide a comprehensive assessment of each model's performance. The hybrid CNN-RNN model achieved the highest test accuracy of 90.27%, surpassing the CNN (89.88%), FCN (85.60%), MLP (79.77%), and RNN (73.54%) models. The hybrid model also demonstrated superior performance in cross-validation, with an accuracy of 87.31% ± 1.77%. Comparative analysis highlights the hybrid model's advantages over single-architecture DL models, particularly in handling imbalanced data and providing reliable classifications across all anemia types. The results underscore the potential of advanced DL architectures in medical diagnostics and suggest pathways for further refinements, such as incorporating attention mechanisms or additional feature engineering, to enhance model performance. This study contributes to the growing body of knowledge on AI-driven medical diagnostics and presents a viable tool for clinical decision support in anemia diagnosis

A Vision-Based Bolt Looseness Detection Method for a Multi-Bolt Connection

Many vision-based bolt looseness detection methods that directly observe the bolts have been developed. However, these methods have many limitations in terms of the conditions and processes of their implementation. To address these problems, this paper proposed a fully automated vision-based bolt looseness detection method for a rigid multi-bolt connection. The proposed method combines digital shearing speckle pattern interferometry (DSSPI) and recurrent neural network (RNN) and involves capturing speckle fringe patterns under various looseness cases using the DSSPI system and classifying these patterns with an RNN model to detect the loose bolts. The proposed method can detect all the bolts within the measured surface at one time, which is efficient. On the other hand, it eliminates the need for prior information such as the initial angle and position of each bolt. It can even detect unseen bolts in multi-bolt connections, making it applicable for connections in complex structures in which occlusion often occurs. Additionally, the method eliminates the complex process of distortion rectification. These features make the method achieve a single-judgment time (four bolts at one detection) of only 4.70 millisecond with a detection accuracy over 99%, which has potential for the real-time detection of loose bolts in multi-bolt connections.

Study on Nowcasting Method of Severe Convective Weather Based on SA-PredRNN++

Severe convective weather, characterized by short-term intense precipitation, thunderstorms, and strong winds, poses significant threats to human life and property. Therefore, accurate and efficient prediction of severe convective weather is crucial for disaster prevention. Currently, utilizing deep learning for radar echo extrapolation stands as the primary method for forecasting severe convective weather. We propose a predictive recurrent neural network model that integrates a self-attention mechanism, specifically designed for radar echo extrapolation in severe convective weather forecasting. The self-attention mechanism offers the advantage of being lightweight, as it does not substantially increase the model parameters. Additionally, it facilitates global attention extraction, thereby enhancing the model's accuracy to some extent. By utilizing radar echo images from the previous hour as input, the model undergoes self-learning to achieve the best forecast for radar echo extrapolation in the subsequent two hours. Research findings demonstrate that our model outperforms other models in accurately predicting severe convective weather within this two-hour timeframe.

A Study on Speech Recognition by a Neural Network Based on English Speech Feature Parameters

In this study, from the perspective of English speech feature parameters, two feature parameters, the mel-frequency cepstral coefficient (MFCC) and filter bank (Fbank), were selected to identify English speech. The algorithms used for recognition employed the classical back-propagation neural network (BPNN), recurrent neural network (RNN), and long short-term memory (LSTM) that were obtained by improving RNN. The three recognition algorithms were compared in the experiments, and the effects of the two feature parameters on the performance of the recognition algorithms were also compared. The LSTM model had the best identification performance among the three neural networks under different experimental environments; the neural network model using the MFCC feature parameter outperformed the neural network using the Fbank feature parameter; the LSTM model had the highest correct rate and the highest speed, while the RNN model ranked second, and the BPNN model ranked worst. The results confirm that the application of the LSTM model in combination with MFCC feature parameter extraction to English speech recognition can achieve higher speech recognition accuracy compared to other neural networks.

Forecasting Next-Time-Step Forex Market Stock Prices Using Neural Networks

Purpose: This study aims to predict the closing price of the EUR/JPY currency pair in the forex market using recurrent neural network (RNN) architectures, namely Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), with the incorporation of Bidirectional layers. Methods: The dataset comprises hourly price data obtained from Yahoo Finance and pre-processed accordingly. The data is divided into training and testing sets, and time series sequences are constructed for input into the models. The RNN, LSTM, and GRU models are trained using the Adam optimization algorithm with the mean squared error (MSE) loss metric. Results: Results indicate that the LSTM model, particularly when coupled with Bidirectional layers, exhibits superior predictive performance compared to the other models, as evidenced by lower MSE values. Conclusions: Therefore, the LSTM model with Bidirectional layers is the most effective in predicting the EUR/JPY currency pair's closing price in the forex market. These findings offer valuable insights for practitioners and researchers involved in financial market prediction and neural network modelling

MUSIC GENERATION USING NEURAL NETWORK

This study investigates the utilization of recurrent neural networks (RNNs) for generating music through MIDI files. By encoding musical data into sequences, RNN models are trained to learn patterns and structures inherent in compositions. Through the analysis of MIDI data and the evaluation of generated sequences, the effectiveness of RNNs in autonomously creating cohesive musical pieces is explored, advancing the frontier of AI-driven musical composition. Keywords—Music Generation, Long Short-Term Memory, Recurrent Neural Network, MIDI data .

A Comprehensive Review of Methods for Hydrological Forecasting Based on Deep Learning

Artificial intelligence has undergone rapid development in the last thirty years and has been widely used in the fields of materials, new energy, medicine, and engineering. Similarly, a growing area of research is the use of deep learning (DL) methods in connection with hydrological time series to better comprehend and expose the changing rules in these time series. Consequently, we provide a review of the latest advancements in employing DL techniques for hydrological forecasting. First, we examine the application of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) in hydrological forecasting, along with a comparison between them. Second, a comparison is made between the basic and enhanced long short-term memory (LSTM) methods for hydrological forecasting, analyzing their improvements, prediction accuracies, and computational costs. Third, the performance of GRUs, along with other models including generative adversarial networks (GANs), residual networks (ResNets), and graph neural networks (GNNs), is estimated for hydrological forecasting. Finally, this paper discusses the benefits and challenges associated with hydrological forecasting using DL techniques, including CNN, RNN, LSTM, GAN, ResNet, and GNN models. Additionally, it outlines the key issues that need to be addressed in the future.

Metaheuristic optimized electrocardiography time-series anomaly classification with recurrent and long-short term neural networks

This study explores the realm of time series forecasting, focusing on the utilization of Recurrent Neural Networks (RNN) to detect abnormal cardiovascular rhythms in Electrocardiogram (ECG) signals. The principal objective is to optimize RNN performance by finely tuning hyperparameters, a complex task with known NP-hard complexity. To address this challenge, the study employs metaheuristic algorithms, specialized problem-solving techniques crafted for navigating intricate and non-deterministic optimization landscapes. Additionally, a refined algorithm is introduced to overcome limitations inherent in the original approach. This modified algorithm exhibits significant improvements, surpassing its predecessor in identifying anomalous cardiovascular rhythms within ECG signals. The most successful optimized model achieves an accuracy of 99.26%, outperforming models optimized by other contemporary metaheuristics assessed in the study. Further experimentation extends the initial inquiry by exploring the capabilities of Long Short-Term Memory (LSTM) models augmented by attention layers. In this extension, the best models demonstrate an accuracy of 99.83%, surpassing the original RNN models. These findings underscore the crucial importance of refining machine learning models and emphasize the potential for substantial advancements in healthcare through innovative algorithmic approaches.

Machine learning-based determination of Mode II translaminar fracture toughness of composite laminates from simple V-notched shear tests

This paper presents a novel method for measuring the translaminar crack resistance curve of composite laminates under Mode II shear loading. A machine learning (ML)-based approach is utilized to extract the inapparent information of the crack resistance curve from the translaminar shear strength measurements obtained from simple V-notched shear tests. The entire campaign is built on the framework of the Finite Fracture Mechanics (FFM) combined with Finite Element Method (FEM). Special emphasis is made on the nonlinear mechanical behavior of composites under shear stress since the original FFM models are designed for quasi-brittle materials. With the well-trained recurrent neural network model, the Mode II R-curve of composite laminate can be obtained with un-notched and V-notched shear strength values as inputs. Experiments were conducted on carbon fiber-reinforced composites to validate the accuracy of the R-curve obtained by the proposed approach and that by the traditional compact shear test. The successful implementation of the method suggests a more convenient and low-cost way of obtaining this important damage-related parameter for composites.

An improved multi-scale convolutional neural network with gated recurrent neural network model for protein secondary structure prediction

An improved multi-scale convolutional neural network with gated recurrent neural network model for protein secondary structure prediction

Prediction of emission characteristics of diesel/n-hexanol/graphene oxide blended fuels based on fast outlier detection-sparrow search algorithm-bidirectional recurrent neural network

This study aims to utilize deep learning (DL) technology to address the issue of carbon soot emission in traditional internal combustion engines. Spraying and combustion experiments were conducted in a constant volume combustion (CVC) chamber using n-hexanol and graphene oxide nanoparticles as partial substitutes for diesel, with the goal of predicting their emission characteristics based on experimental data. Initially, spray and combustion data under different operating conditions were collected using the CVC experimental setup, and outliers were removed using the fast outlier detection (FOD) algorithm. Subsequently, the learning rate and the number of hidden layer neurons in the Bidirectional Recurrent Neural Network (BiRNN) model were optimized using the Sparrow Search Algorithm (SSA) to enhance the model's accuracy and generalization ability. Unlike previous research methods, the novel FOD-coupled SSA optimization method used in this study ensures the optimal parameter configuration of the speed prediction model while accelerating the convergence speed, which ultimately improves the stability of the model. In addition, the optimal output variables were determined based on the Spearman and Pearson correlation coefficient rules. Finally, the effectiveness of the predictive model was validated using evaluation metrics such as root mean square error (RMSE) and R-squared (R2). The results indicate that the KL coefficient for D80H20GO20 fuel in the CVC device is minimal. The FOD-SSA-BiRNN model achieved R2 values of 0.99409, 0.99529, and 0.99692 for D100, D80H20, and D80H20GO20 fuels, respectively. Therefore, this study provides an effective method for the online accurate prediction of carbon soot emissions in mixed fuels.