Variant Of Recurrent Neural Network Research Articles

SERT-StructNet: Protein secondary structure prediction method based on multi-factor hybrid deep model

Protein secondary structure prediction (PSSP) is a pivotal research endeavour that plays a crucial role in the comprehensive elucidation of protein functions and properties. Current prediction methodologies are focused on deep-learning techniques, particularly focusing on multi-factor features. Diverging from existing approaches, in this study, we placed special emphasis on the effects of amino acid properties and protein secondary structure propensity scores (SSPs) on secondary structure during the meticulous selection of multi-factor features. This differential feature-selection strategy results in a distinctive and effective amalgamation of the sequence and property features. To harness these multi-factor features optimally, we introduced a hybrid deep feature extraction model. The model initially employs mechanisms such as dilated convolution (D-Conv) and a channel attention network (SENet) for local feature extraction and targeted channel enhancement. Subsequently, a combination of recurrent neural network variants (BiGRU and BiLSTM), along with a transformer module, was employed to achieve global bidirectional information consideration and feature enhancement. This approach to multi-factor feature input and multi-level feature processing enabled a comprehensive exploration of intricate associations among amino acid residues in protein sequences, yielding a Q3 accuracy of 84.9% and an Sov score of 85.1%. The overall performance surpasses that of the comparable methods. This study introduces a novel and efficient method for determining the PSSP domain, which is poised to deepen our understanding of the practical applications of protein molecular structures.

Prediction of ground water quality in western regions of Tamilnadu using LSTM network

Assessing and safeguarding groundwater quality is critical for sustaining life in water-scarce regions like western Tamil Nadu. The motivation behind this study stems from the pressing need to address water quality challenges in a region grappling with scarcity. Despite existing efforts, a notable research gap exists in predictive tools that comprehensively capture the nuanced temporal variations and trends in groundwater quality. This is where the LSTM network steps in, showcasing exceptional accuracy in short-term predictions and discerning long-term trends. This research uses Long Short-Term Memory (LSTM) networks, a variant of recurrent neural networks, to predict groundwater quality in South Indian Regions, especially in Tamil Nadu. Extensive data, encompassing parameters such as pH, dissolved oxygen, turbidity, and various chemical constituents, were gathered over an extended timeframe. The LSTM model was then trained on this historical dataset, factoring in temporal dependencies and seasonality inherent in groundwater quality data. The validation process rigorously tests the LSTM model against actual groundwater quality measurements. The results were impressive, as the model demonstrated a remarkable ability to unravel the complex variations in groundwater quality.

Speaker identification under noisy conditions using hybrid convolutional neural network and gated recurrent unit

<p><span>Speaker identification is biometrics that classifies or identifies a person from other speakers based on speech characteristics. Recently, deep learning models outperformed conventional machine learning models in speaker identification. Spectrograms of the speech have been used as input in deep learning-based speaker identification using clean speech. However, the performance of speaker identification systems gets degraded under noisy conditions. Cochleograms have shown better results than spectrograms in deep learning-based speaker recognition under noisy and mismatched conditions. Moreover, hybrid convolutional neural network (CNN) and recurrent neural network (RNN) variants have shown better performance than CNN or RNN variants in recent studies. However, there is no attempt conducted to use a hybrid CNN and enhanced RNN variants in speaker identification using cochleogram input to enhance the performance under noisy and mismatched conditions. In this study, a speaker identification using hybrid CNN and the gated recurrent unit (GRU) is proposed for noisy conditions using cochleogram input. VoxCeleb1 audio dataset with real-world noises, white Gaussian noises (WGN) and without additive noises were employed for experiments. The experiment results and the comparison with existing works show that the proposed model performs better than other models in this study and existing works</span><span>.</span></p>

Nonlinear Spiking Neural Systems With Autapses for Predicting Chaotic Time Series.

Spiking neural P (SNP) systems are a class of distributed and parallel neural-like computing models that are inspired by the mechanism of spiking neurons and are 3rd-generation neural networks. Chaotic time series forecasting is one of the most challenging problems for machine learning models. To address this challenge, we first propose a nonlinear version of SNP systems, called nonlinear SNP systems with autapses (NSNP-AU systems). In addition to the nonlinear consumption and generation of spikes, the NSNP-AU systems have three nonlinear gate functions, which are related to the states and outputs of the neurons. Inspired by the spiking mechanisms of NSNP-AU systems, we develop a recurrent-type prediction model for chaotic time series, called the NSNP-AU model. As a new variant of recurrent neural networks (RNNs), the NSNP-AU model is implemented in a popular deep learning framework. Four datasets of chaotic time series are investigated using the proposed NSNP-AU model, five state-of-the-art models, and 28 baseline prediction models. The experimental results demonstrate the advantage of the proposed NSNP-AU model for chaotic time series forecasting.

Enhancing Electrical Power Demand Prediction Using LSTM-Based Deep Learning Models for Local Energy Communities

The pursuit of accurate electrical power demand forecasting has led to the application of deep learning algorithms, notably demonstrating promising outcomes despite the prerequisite of substantial training data. This study pioneers a new learning paradigm, employing sophisticated deep learning models specifically, Long Short-Term Memory networks and recurrent neural networks (RNNs). Leveraging historical load data, temperature, wind speed, and day-ahead predicted spot prices, this approach follows a structured flow involving data preprocessing, sequence generation, model training, and subsequent prediction of future load demand using LSTM-based RNN variants. The study’s paramount findings underscore the substantial advancement achieved by this proposed methodology over prevailing techniques. The method significantly improves prediction accuracy by over 11%, demonstrating the efficacy of deep learning models and a significant leap forward in forecasting precision. Beyond its superior predictive capabilities, this novel strategy serves as a catalyst for enhancing energy distribution management in local energy communities. Its effectiveness lies not only in its precision but also in enabling the optimization and cost-effective control of energy distribution, vital for sustainable energy management in these communities. Ultimately, this pioneering approach presents a robust solution poised to revolutionize the landscape of electrical power demand forecasting and its practical application in local energy systems.

Movie sentiment analysis based on Long Short-Term Memory Network

An important task in the study of Natural Language Processing (NLP) is the analysis of movie reviews. It finishes the task of classifying movie review texts into sentiment, such as positive, negative or neutral sentiment. Previous works mainly follow the pipeline of LSTM (Long Short-Term Memory Network). The network model is a variant of Recurrent Neural Network (RNN) and particularly suitable for processing natural language texts. Though existing LSTM-based works have improved the performance significantly, we argue that most of them deal with the problem of analyzing the sentiment of movie reviews while ignore analyze the model performance in different application scenarios, such as different lengths of the reviews and the frequency of sentiment adverbs in the reviews. To alleviate the above issue, in this paper, we constructed a simple LSTM model containing an embedding layer, a batch normalization layer, a dropout layer, a one-dimensional convolutional layer, a maximal pooling layer, a bi-directional LSTM layer and a fully connected layer. We used the existing IMDB movie review dataset to train the model, and selected two research scenarios of movie review length and frequency of occurrence of sentiment adverbs to test the model, respectively. From the experimental results, we proposed a model for the scenarios in which the LSTM model handles the problem of sentiment analysis with respect to the dataset construction, model stability and generalization ability, text fragment processing, data preprocessing and feature extraction, model optimization and improvement.

Multi-task learning using non-linear autoregressive models and recurrent neural networks for tide level forecasting

Tide level forecasting plays an important role in environmental management and development. Current tide level forecasting methods are usually implemented for solving single task problems, that is, a model built based on the tide level data at an individual location is only used to forecast tide level of the same location but is not used for tide forecasting at another location. This study proposes a new method for tide level prediction at multiple locations simultaneously. The method combines nonlinear autoregressive moving average with exogenous inputs (NARMAX) model and recurrent neural networks (RNNs), and incorporates them into a multi-task learning (MTL) framework. Experiments are designed and performed to compare single task learning (STL) and MTL with and without using non-linear autoregressive models. Three different RNN variants, namely, long short-term memory (LSTM), gated recurrent unit (GRU) and bidirectional LSTM (BiLSTM) are employed together with non-linear autoregressive models. A case study on tide level forecasting at many different geographical locations (5 to 11 locations) is conducted. Experimental results demonstrate that the proposed architectures outperform the classical single-task prediction methods.

A model-free method based on RDPG for fiber diameter steady control

Optical fibers have made outstanding contributions to optical instruments such as micro-channel plates (MCPs) and micro-pore optics (MPOs). As the stricter performance requirements for the MCP and MPO, traditional diameter control methods are incapable. Additionally, the partial observation issue of fiber drawing process further limits the outcome of the existing control approaches. Innovative control algorithms are vital for the diameter precision enhancement. We investigate the significance of recurrent deterministic policy gradient (RDPG) on diameter control. RDPG consists of deep deterministic policy gradient (DDPG) and recurrent neural network (RNN) where DDPG is a model-free reinforcement learning algorithm for continuous control and RNN for addressing partial observation. The impact of RNN variants selection on control performance is further discussed. Results illustrate that the application of RNN substantially solves the partial observation and varying RNN variants play a positive role in diameter control, but bidirectional gated recurrent unit (BiGRU) reaches the best accuracy. Comparison experiment results indicate that RDPG is able to learn reasonable control strategies and achieves significantly more accurate and robust diameter control in contrast with the existing methods.

Echo state network structure optimization algorithm based on correlation analysis

Echo State Network (ESN) is an effective variant of Recurrent Neural Network (RNN). However, it is difficult for traditional ESN to determine the reservoir size that matches a given task. In this paper, an ESN structure optimization pruning algorithm based on correlation analysis, called PCESN, is proposed to design the size of the reservoir automatically. First, a characteristic matrix is constructed utilizing probability theory and information theory to measure the correlation between each neuron in the reservoir and the output neuron. On this basis, a pruning criterion is proposed to achieve a sparse reservoir structure by dynamically removing neurons with low correlation. Second, in order to retain the sample information of the neurons in the removed reservoir during the network reduction, the input weights of the retained reservoir are updated by means of the average transverse propagation of the weights. Finally, the performance of PCESN is tested on multiple time series. Simulation results show that the proposed PCESN outperforms some fixed size ESN and other dynamic ESN in terms of prediction accuracy, generalization performance and model complexity.

Precipitation forecast using RNN variants by analyzing Optimizers and Hyperparameters for Time-series based Climatological Data

Flood is a significant problem in many regions of the world for the catastrophic damage it causes to both property and human lives; excessive precipitation being the major cause. The AI technologies, Deep Learning Neural Networks and Machine Learning algorithms attempt realistic solutions to numerous disaster management challenges. This paper works on RNN- based rainfall/ precipitation forecasting models by investigating the performances of various Recurrent Neural Network (RNN) architectures, Bidirectional RNN (BRNN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU) and ensemble models such as BRNN-GRU, BRNN-LSTM, LSTM-GRU, BRNN-LSTM-GRU using NASAPOWER datasets of Andhra Pradesh (AP) and Tamil Nadu (TN) in India. The different stages in the workflow of the methodology are Data collection, Data pre-processing, Data splitting, Defining hyperparameters, Model building and Performance evaluation. Experiments for identifying improved optimizers and hyperparameters for the time-series climatological data are investigated for accurate precipitation forecast. The metrics: Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Square Error (RMSE) and Root Mean Squared Logarithmic Error (RMSLE) values are used to compare the precipitation predictions of different models. The RNN variants and ensemble models, BRNN, LSTM, GRU, BRNN-GRU, BRNN-LSTM, LSTM-GRU, BRNN-LSTM-GRU produce predictions with RMSLE values of 2.448, 0.555, 0.255, 1.305, 1.383, 0.364, 1.740 for AP and 1.735, 0.663, 0.152, 0.889, 1.118, 0.379, 1.328 for TN respectively. The best performing RNN model, GRU when ensembled with the existing statistical model SARIMA produces an RMSLE value of 0.754 and 1.677 respectively for AP and TN.

Physical reservoirs based on MoS2-HZO integrated ferroelectric field-effect transistors for reservoir computing systems.

Reservoir computing (RC), a variant of recurrent neural networks (RNNs), is well-known for its reduced energy consumption through exclusive focus on training the output weight and its superior performance in handling spatiotemporal information. Implementing these networks in hardware requires devices with superior fading memory behavior. Unlike filament-based two-terminal devices, those relying on ferroelectric switching demonstrate improved voltage reliability, while three-terminal transistors provide additional active control. HfO2-based ferroelectric materials such as Hf0.5Zr0.5O2 (HZO), have garnered attention for their scalability and seamless integration with CMOS technology. This study implements a RC hardware based on MoS2-HZO integrated device structure with enhanced spontaneous polarization field. By adjusting the oxygen vacancy concentration, the devices exhibit consistent responses to both identical and nonidentical voltages, making them suitable for diverse RC applications. The high accuracy of MNIST handwritten digits recognition highlights the rich reservoir states of the traditional RC architecture. Additionally, the impact of masks on RC implementation is assessed, showcasing the device's capability for spatiotemporal signal analysis. This development paves the way for implementing energy-efficient and high-performance computing solutions.

Deep Learning-Based Bitcoin Price Prediction Using Long Short-Term Memory Networks

Cryptocurrencies, notably Bitcoin, have surged in popularity and importance, attracting widespread attention due to their volatile nature and potential for significant financial gains. Predicting Bitcoin prices accurately has emerged as a challenging yet indispensable task for investors, traders, and policymakers seeking to navigate the complexities of the cryptocurrency market. Traditional forecasting methods often struggle to capture the intricate temporal patterns inherent in this domain, prompting the exploration of advanced techniques such as deep learning. This paper presents a comprehensive study on employing Long Short-Term Memory (LSTM) networks for Bitcoin price prediction, a variant of Recurrent Neural Networks (RNNs). Our research encompasses the entire process, from data collection to model evaluation, focusing on addressing the unique challenges posed by cryptocurrency market data. We begin by discussing the methodology involved in data acquisition, preprocessing, and feature engineering, which are crucial steps for ensuring the quality and relevance of input data. Next, we delve into the intricacies of LSTM model selection, hyperparameter tuning, and training strategies, aiming to optimize predictive performance while mitigating overfitting. Through extensive experimentation on real-world Bitcoin price datasets spanning multiple periods, we rigorously evaluate the effectiveness of LSTM networks in capturing long-term dependencies and forecasting future price movements. Comparative analyses against traditional time-series forecasting methods underscore the superior predictive capabilities of deep learning approaches in this context. Furthermore, we investigate the interpretability of LSTM-based models, examining the insights gained from analyzing learned representations and hidden states. These analyses provide valuable perspectives for market participants seeking to understand the driving factors behind Bitcoin price dynamics. Our findings demonstrate the practical utility of LSTM networks in Bitcoin price prediction and contribute to advancing the broader discourse on deep learning applications in financial forecasting. By shedding light on the potential of deep learning techniques to enhance predictive accuracy and robustness in cryptocurrency markets, our research offers valuable insights for stakeholders navigating this rapidly evolving landscape.

A Multitask Integrated Deep-Learning Probabilistic Prediction for Load Forecasting

Spinning reserve without accurate load forecasting can lead to automatic disconnection of critical loads by under-frequency load shedding devices. Such a predicament poses a grave threat to the economic and social welfare of the affected region, and in extreme scenarios, can result in a debilitating grid collapse. However, existing models lack the deep feature extraction capability and cannot accurately predict the uncertainty of electricity demand. Thus, a multitask integrated deep-learning probabilistic prediction with multidimensional feature extraction based on granularity information and quantile regression is explored in this study to solve the insufficient feature extraction problem in probabilistic prediction. In the proposed scheme, we designed a multi-layer feature extraction framework, which information granularity based on fuzzy rough sets to extract time-domain features, multilayer Laplace operators to extract features globally, and quantile regression recurrent neural network variants to extract time context information. In addition, the Pareto integration method is extended to capture better individuals. The experimental results demonstrate that the proposed system can effectively improve deterministic forecasting and uncertainty analysis of electricity demand and manage daily fluctuations.

Estimating Counterfactual Treatment Outcomes Over Time in Complex Multiagent Scenarios.

Evaluation of intervention in a multiagent system, for example, when humans should intervene in autonomous driving systems and when a player should pass to teammates for a good shot, is challenging in various engineering and scientific fields. Estimating the individual treatment effect (ITE) using counterfactual long-term prediction is practical to evaluate such interventions. However, most of the conventional frameworks did not consider the time-varying complex structure of multiagent relationships and covariate counterfactual prediction. This may lead to erroneous assessments of ITE and difficulty in interpretation. Here, we propose an interpretable, counterfactual recurrent network in multiagent systems to estimate the effect of the intervention. Our model leverages graph variational recurrent neural networks (GVRNNs) and theory-based computation with domain knowledge for the ITE estimation framework based on long-term prediction of multiagent covariates and outcomes, which can confirm the circumstances under which the intervention is effective. On simulated models of an automated vehicle and biological agents with time-varying confounders, we show that our methods achieved lower estimation errors in counterfactual covariates and the most effective treatment timing than the baselines. Furthermore, using real basketball data, our methods performed realistic counterfactual predictions and evaluated the counterfactual passes in shot scenarios.

Depression Risk Prediction using Hybrid Deep Learning Algorithms

Abstract: This research endeavors to address the critical challenge of early prediction of depression, a pervasive mental health disorder that often eludes timely detection. Recognizing the substantial impact of late-stage diagnosis on treatment outcomes, this study introduces a robust machine learning model that leverages diverse data sources to predict the likelihood of an individual experiencing depression. The proposed model under- goes meticulous development, involving extensive data collection and pre-processing to curate a comprehensive dataset encompassing various aspects of an individual’s life. Machine learning algorithms are then applied to analyze the dataset, extracting patterns and features indicative of depressive tendencies. To enhance the model’s predictive performance and overall efficiency, the suggested system advocates the use of hybrid algorithms, specifically combining Convolutional Neural Network (ConvNet) and Recurrent Neural Network (RNN) variants. This hybrid approach brings forth several advantages, including spa- tial feature extraction and a hierarchy of features. The integration of RNN variants with ConvNet facilitates effective extraction of spatial features from diverse data types such as text, images, videos, and other spatially structured data. Additionally, the CNN layers in the hybrid model learn hierarchical representations of features, capturing both low-level and high-level spatial patterns. This unique capability enhances the model’s understanding of complex structures within the input data. The proposed model is meticulously trained and validated using a diverse set of metrics to ensure its reliability and generalizability. The anticipated outcome of this project holds significant potential to revolutionize early intervention strategies, facilitating timely support for individuals at risk of depression. By amalgamating advanced machine learning techniques with a holistic approach to data analysis, this study contributes to the ongoing efforts aimed at enhancing mental health outcomes and alleviating the societal burden associated with depression.

Deep Learning Approaches for Predicting Intraday Price Movements: An Evaluation of RNN Variants on High-Frequency Stock Data

This study discusses the comparison of four recurrent neural networks (RNN) models: Simple RNN, Gated Recurrent Unit (GRU), Long Short-Term Memory (LSTM), and Bidirectional RNN (BiRNN), in forecasting minute-level stock price time series data. The performance of these four models is evaluated using the Mean Absolute Percentage Error (MAPE) on a stock dataset from Bank Central Asia (BBCA.JK). The experimental results reveal that the GRU model exhibits the best performance with an average MAPE of 0.0255%, followed by the LSTM model with an average MAPE of 0.0377%. The BiRNN model also demonstrates good performance with an average MAPE of 0.0668%, while the Simple RNN has the highest average MAPE at 0.5118%. This suggests that more complex recurrent architectures like GRU and LSTM have better capabilities in capturing patterns in high-frequency time series data. This study can be expanded by exploring other models such as CNN, conducting tests on diverse datasets, and experimenting with a wider range of hyperparameter variations. Additional variables such as economic indicators, global market data, and social data can also offer a more comprehensive understanding of factors influencing stock prices.

Addressing posterior collapse by splitting decoders in variational recurrent autoencoders

Variational recurrent autoencoder model (VRAE) is an appealing technique for capturing the variabilities underlying complex sequential data, which is realized by introducing high-level latent random variables as hidden states. Existing models suffer from the well-known ‘posterior collapse’ problem, meaning that a powerful autoregressive decoder equipped in the model itself could capture all the variabilities and hence leave the latent variables learning nothing from the data. From the perspective of model training, the posterior collapse problem can result in a very low Kullback–Leibler divergence (KL-divergence) value, which means the posteriors of the latent variables tend to be just the priors. In this paper, we address this problem by proposing a Bayesian variational recurrent neural network (BVRNN) model, in which two additional decoders are added into the original VRAE. These extra decoders can force the latent variables to learn meaningful knowledge during the training process. We conduct experiments on MNIST and Fashion-MNIST dataset. The experimental results show that the proposed model outperforms several baseline models. We further adapt the proposed model to a very challenging task in natural language processing, namely Named-Entity Recognition (NER). Experimental results show that our model is competitive to the state-of-the-art models on NER.

Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling

Extracting the rules of real-world multi-agent behaviors is a current challenge in various scientific and engineering fields. Biological agents independently have limited observation and mechanical constraints; however, most of the conventional data-driven models ignore such assumptions, resulting in lack of biological plausibility and model interpretability for behavioral analyses. Here we propose sequential generative models with partial observation and mechanical constraints in a decentralized manner, which can model agents’ cognition and body dynamics, and predict biologically plausible behaviors. We formulate this as a decentralized multi-agent imitation-learning problem, leveraging binary partial observation and decentralized policy models based on hierarchical variational recurrent neural networks with physical and biomechanical penalties. Using real-world basketball and soccer datasets, we show the effectiveness of our method in terms of the constraint violations, long-term trajectory prediction, and partial observation. Our approach can be used as a multi-agent simulator to generate realistic trajectories using real-world data.

Bi-Directional Hybrid Deep Learning model for Speaker Iden-tification

Speaker identification is the process of automatically determining who is speaking from the known speakers by the model. It is crucial in voice-based authentication, forensic investigations, security and surveillance. In recent studies, the combination of convolutional neural network (CNN) and recurrent neural network (RNN) variants performed better than separate models of both. However, only limited studies are conducted in speaker identification using a combination of CNN and RNN variants. In this study, we proposed speaker identification using hybrid two-dimensional CNN (2DCNN) and bidirectional gated recurrent unit (BiGRU) to improve performance. The proposed model integrates the advantage of 2DCNN and BiGRU layers to improve the performance of the model. 2DCNN layers have the advantage of extracting short-term spatial features from input data and it has a limited number of parameters for computation. BiGRU layers have an advantage in extracting long-term temporal dependency between the features in both directions (i.e. backward and forward) and it is efficient in achieving convergence during training. Spectrograms of the speech were used as input in our proposed model because of the rich acoustic features of the speaker. To compare the performance of the proposed model, additional experiments are conducted using the models 2DCNN, CNN-LSTM, CNN-BiLSTM and CNN-GRU. The experiments were conducted on the VoxCeleb1 audio dataset, which consists of 153,516 utterances collected from the 1251 speakers. The accuracy, precision, recall and f1 score of the proposed model are 98.28%, 99.08%, 98.92% and 98.97% respectively. The proposed model was compared with the existing works to show the effectiveness of the proposed model. The experiment results and the comparison with the existing works show that the proposed model has higher performance than both existing works and other models experimented in this study.

Reservoir computing decoupling memory–nonlinearity trade-off

Reservoir computing (RC), a variant recurrent neural network, has very compact architecture and ability to efficiently reconstruct nonlinear dynamics by combining both memory capacity and nonlinear transformations. However, in the standard RC framework, there is a trade-off between memory capacity and nonlinear mapping, which limits its ability to handle complex tasks with long-term dependencies. To overcome this limitation, this paper proposes a new RC framework called neural delayed reservoir computing (ND-RC) with a chain structure reservoir that can decouple the memory capacity and nonlinearity, allowing for independent tuning of them, respectively. The proposed ND-RC model offers a promising solution to the memory–nonlinearity trade-off problem in RC and provides a more flexible and effective approach for modeling complex nonlinear systems with long-term dependencies. The proposed ND-RC framework is validated with typical benchmark nonlinear systems and is particularly successful in reconstructing and predicting the Mackey–Glass system with high time delays. The memory–nonlinearity decoupling ability is further confirmed by several standard tests.