Class Of Recurrent Neural Networks Research Articles

A multi-modal Parkinson’s disease diagnosis system from EEG signals and online handwritten tasks using grey wolf optimization based deep learning model

Parkinson’s disease (PD) is a neurodegenerative disorder characterized by the gradual deterioration of motor function, affecting speech, writing, muscle control, and mobility. The existing studies have not utilized both the electroencephalography (EEG) signals and online handwritten tasks together to diagnose PD. The studies have also not explored the EEG signals collected from specific brain regions like the substansia niagra (SN) and ventral tegmental area (VTA), crucial for dopamine production linked to PD. This article proposes a multi-modal PD diagnosis system from EEG signals (collected from SN and VTA regions of the brain), collected during performing online handwritten tasks, using grey wolf optimization (GWO) algorithm. Mel-frequency cepstral coefficients (MFCC) features have been generated from the EEG signals and optimized by the GWO algorithm. The classification (diagnosis) experiments on the optimal number of feature values, obtained from GWO algorithm, have been carried out using bidirectional long short-term memory (BLSTM) variant of recurrent neural network (RNN). The classification experiments have also been conducted using support vector machine (SVM), bagged random forest (BRF), and long short-term memory (LSTM) variant of RNN classifier to have a performance comparison with the proposed method. The effectiveness of the introduced PD diagnosis system has been analyzed on a self-generated dataset named EEG signal based on online handwriting (ESOH). A maximum classification accuracy of 99.30% has been achieved from the proposed PD diagnosis system. The experimental outcomes illustrate that the introduced PD diagnosis system outperforms the state-of-the-art PD diagnosis systems relying on the EEG signals for diagnosing the PD.

QMWT: Design of an Improved EEG Classification Model via Q-Learning Based Processing of Multispectral Wave Traits

Electroencephalogram (EEG) signals represent various wave patterns that assist in identification of normal & abnormal brain activities. These wave patterns consist of alpha waves, which indicate relaxation phases, beta waves, that indicate normal brain rhythms & can be disturbed due to cortical & other damages, delta waves, which are dominant in infants during sleep, & theta waves, which represent irregular metabolic & hydrocephalus activities in adults. A combination of these waves is capable of representing multiple brain conditions for both adolescents & adults. Various models have been proposed by researchers to analyze these signals, and most of them work on single or bi-domain features, which limits their classification performance. Most of these models are also static, and do not incorporate continuous feedback & incremental processes. Due to which their precision & recall performance is either constant or reduces w.r.t. newer evaluations. To overcome these limitations, this text proposes design of a novel EEG classification model that uses Q-Learning for classification of multispectral feature sets. The model extracts Mel Frequency Cepstral Coefficients (MFCC), and iVector features from raw EEG data, which assists in multispectral representation of these signals. The extracted features are classified via Q-Learning based Recurrent Neural Network (RNN) classifier, that combines Gated Recurrent Unit (GRU), and Long-Short-Term Memory (LSTM) based feature sets. Due to extraction of MFCC, the GRU & LSTM Models are able to identify power spectral variation, cepstral variations, spectrogram patterns, DCT (Discrete Cosine Transform) variations, etc. While, due to iVector, entropy variations are recognized and processed for better accuracy levels. Thus, both LSTM & GRU Models assist in augmenting extracted features, which improves feature variance for better classification performance. Results of these classifications are feedback into the training set via a correlation-based analysis layer, that assists in continuously improving precision & recall performance for different evaluations. Due to incorporation of this layer, the model is capable of improving precision by 8.5%, recall by 8.3%, and accuracy by 4.9% when compared with various state-of-the-art models. It was also observed that this performance incrementally improves w.r.t. number of evaluations, which assists in deploying the model for real-time applications.

Design and Development of Intrusive Detection Model based on Sequential Deep Recurrent Neural Network (SDRNN) Classifier

Intrusion detection systems (IDS) must defend advanced communication networks. The systems' major goal was to identify broken patterns, signatures, and rules. IDS is crucial to network cyber defense. IDS can now detect both regular and atypical patterns thanks to advances in artificial intelligence, notably machine learning and deep learning. This research discusses a revolutionary deep learning and machine learning approach to intelligent categorization in IDS model development. The main objective is to detect network traffic irregularities utilizing HIKARI-2021. With an optimized feature set for the dataset, a deep learning model achieved 99.78% accuracy for binary and multi-class classification tasks.

Leveraging hybrid 1D-CNN and RNN approach for classification of brain cancer gene expression

Leveraging deep learning (DL) approaches in genomics data has led to significant advances in cancer prediction. The continuous availability of gene expression datasets over the preceding years has made them one of the most accessible sources of genome-wide data, advancing cancer bioinformatics research and advanced prediction of cancer genomic data. To contribute to this topic, the proposed work is based on DL prediction in both convolutional neural network (CNN) and recurrent neural network (RNN) for five classes in brain cancer using gene expression data obtained from Curated Microarray Database (CuMiDa). This database is used for cancer classification and is publicly accessible on the official CuMiDa website. This paper implemented DL approaches using a One Dimensional-Convolutional Neural Network (1D-CNN) followed by an RNN classifier with and without Bayesian hyperparameter optimization (BO). The accuracy of this hybrid model combination of (BO + 1D-CNN + RNN) produced the highest classification accuracy of 100% instead of the 95% for the ML model in prior work and 90% for the (1D-CNN + RNN) algorithm considered in the paper. Therefore, the classification of brain cancer gene expression according to the hybrid model (BO + 1D-CNN + RNN) provides more accurate and useful assessments for patients with different types of brain cancers. Thus, gene expression data are used to create a DL classification-based- hybrid model that will hold senior promise in the treatment of brain cancer.

Mining software insights: uncovering the frequently occurring issues in low-rating software applications.

In today's digital world, app stores have become an essential part of software distribution, providing customers with a wide range of applications and opportunities for software developers to showcase their work. This study elaborates on the importance of end-user feedback for software evolution. However, in the literature, more emphasis has been given to high-rating & popular software apps while ignoring comparatively low-rating apps. Therefore, the proposed approach focuses on end-user reviews collected from 64 low-rated apps representing 14 categories in the Amazon App Store. We critically analyze feedback from low-rating apps and developed a grounded theory to identify various concepts important for software evolution and improving its quality including user interface (UI) and user experience (UX), functionality and features, compatibility and device-specific, performance and stability, customer support and responsiveness and security and privacy issues. Then, using a grounded theory and content analysis approach, a novel research dataset is curated to evaluate the performance of baseline machine learning (ML), and state-of-the-art deep learning (DL) algorithms in automatically classifying end-user feedback into frequently occurring issues. Various natural language processing and feature engineering techniques are utilized for improving and optimizing the performance of ML and DL classifiers. Also, an experimental study comparing various ML and DL algorithms, including multinomial naive Bayes (MNB), logistic regression (LR), random forest (RF), multi-layer perception (MLP), k-nearest neighbors (KNN), AdaBoost, Voting, convolutional neural network (CNN), long short-term memory (LSTM), bidirectional long short term memory (BiLSTM), gated recurrent unit (GRU), bidirectional gated recurrent unit (BiGRU), and recurrent neural network (RNN) classifiers, achieved satisfactory results in classifying end-user feedback to commonly occurring issues. Whereas, MLP, RF, BiGRU, GRU, CNN, LSTM, and Classifiers achieved average accuracies of 94%, 94%, 92%, 91%, 90%, 89%, and 89%, respectively. We employed the SHAP approach to identify the critical features associated with each issue type to enhance the explainability of the classifiers. This research sheds light on areas needing improvement in low-rated apps and opens up new avenues for developers to improve software quality based on user feedback.

Automatic diagnosis of epileptic seizures using entropy-based features and multimodel deep learning approaches

Epilepsy is one of the most common brain diseases, characterised by repeated seizures that occur on a regular basis. During a seizure, a patient's muscles flex uncontrollably, causing a loss of mobility and balance, which can be harmful or even fatal. Developing an automatic approach for warning patients of oncoming seizures necessitates substantial research. Analyzing the electroencephalogram (EEG) output from the human brain's scalp region can help predict seizures. EEG data were analyzed to extract time domain features such as Hurst exponent (Hur), Tsallis entropy (TsEn), enhanced permutation entropy (impe), and amplitude-aware permutation entropy (AAPE). In order to automatically diagnose epileptic seizure in children from normal children, this study conducted two sessions. In the first session, the extracted features from the EEG dataset were classified using three machine learning (ML)-based models, including support vector machine (SVM), K nearest neighbor (KNN), or decision tree (DT), and in the second session, the dataset was classified using three deep learning (DL)-based recurrent neural network (RNN) classifiers in The EEG dataset was obtained from the Neurology Clinic of the Ibn Rushd Training Hospital. In this regard, extensive explanations and research from the time domain and entropy characteristics demonstrate that employing GRU, LSTM, and BiLSTM RNN deep learning classifiers on the All−time−entropy fusion feature improves the final classification results.

Hybrid deep learning approach for sentiment analysis using text and emojis

ABSTRACT Sentiment Analysis (SA) is a technique for categorizing texts based on the sentimental polarity of people’s opinions. This paper introduces a sentiment analysis (SA) model with text and emojis. The two preprocessed data’s are data with text and emojis and text without emojis. Feature extraction consists text features and text with emojis features. The text features are features like N-grams, modified Term Frequency-Inverse Document Frequency (TF-IDF), and Bag-of-Words (BoW) features extracted from the text. In classification, CNN (Conventional Neural Network) and MLP (Multi-Layer Perception) use emojis and text-based SA. The CNN weight is optimized by a new Electric fish Customized Shark Smell Optimization (ECSSO) Algorithm. Similarly, the text-based SA is carried out by hybrid Long Short-Term Memory (LSTM) and Recurrent Neural Network (RNN) classifiers. The bagged data are given as input to the classification process via RNN and LSTM. Here, the weight of LSTM is optimized by the suggested ECSSO algorithm. Then, the mean of LSTM and RNN determines the final output. The specificity of the developed scheme is 29.01%, 42.75%, 23.88%,22.07%, 25.31%, 18.42%, 5.68%, 10.34%, 6.20%, 6.64%, and 6.84% better for 70% than other models. The efficiency of the proposed scheme is computed and evaluated.

Spike-based computation using classical recurrent neural networks

Spiking neural networks (SNNs) are a type of artificial neural networks in which communication between neurons is only made of events, also called spikes. This property allows neural networks to make asynchronous and sparse computations and therefore drastically decrease energy consumption when run on specialized hardware. However, training such networks is known to be difficult, mainly due to the non-differentiability of the spike activation, which prevents the use of classical backpropagation. This is because state-of-the-art SNNs are usually derived from biologically-inspired neuron models, to which are applied machine learning methods for training. Nowadays, research about SNNs focuses on the design of training algorithms whose goal is to obtain networks that compete with their non-spiking version on specific tasks. In this paper, we attempt the symmetrical approach: we modify the dynamics of a well-known, easily trainable type of recurrent neural network (RNN) to make it event-based. This new RNN cell, called the spiking recurrent cell, therefore communicates using events, i.e. spikes, while being completely differen-tiable. Vanilla backpropagation can thus be used to train any network made of such RNN cell. We show that this new network can achieve performance comparable to other types of spiking networks in the MNIST benchmark and its variants, the Fashion-MNIST and the Neuromorphic-MNIST. Moreover, we show that this new cell makes the training of deep spiking networks achievable.

A novel transformer-based DL model enhanced by position-sensitive attention and gated hierarchical LSTM for aero-engine RUL prediction

Accurate prediction of remaining useful life (RUL) for aircraft engines is essential for proactive maintenance and safety assurance. However, existing methods such as physics-based models, classical recurrent neural networks, and convolutional neural networks face limitations in capturing long-term dependencies and modeling complex degradation patterns. In this study, we propose a novel deep-learning model based on the Transformer architecture to address these limitations. Specifically, to address the issue of insensitivity to local context in the attention mechanism employed by the Transformer encoder, we introduce a position-sensitive self-attention (PSA) unit to enhance the model's ability to incorporate local context by attending to the positional relationships of the input data at each time step. Additionally, a gated hierarchical long short-term memory network (GHLSTM) is designed to perform regression prediction at different time scales on the latent features, thereby improving the accuracy of RUL estimation for mechanical equipment. Experiments on the C-MAPSS dataset demonstrate that the proposed model outperforms existing methods in RUL prediction, showcasing its effectiveness in modeling complex degradation patterns and long-term dependencies.

Exploring deep echo state networks for image classification: a multi-reservoir approach

Echo state networks (ESNs) belong to the class of recurrent neural networks and have demonstrated robust performance in time series prediction tasks. In this study, we investigate the capability of different ESN architectures to capture spatial relationships in images without transforming them into temporal sequences. We begin with three pre-existing ESN-based architectures and enhance their design by incorporating multiple output layers, customising them for a classification task. Our investigation involves an examination of the behaviour of these modified networks, coupled with a comprehensive performance comparison against the baseline vanilla ESN architecture. Our experiments on the MNIST data set reveal that a network with multiple independent reservoirs working in parallel outperforms other ESN-based architectures for this task, achieving a classification accuracy of 98.43%. This improvement on the classical ESN architecture is accompanied by reduced training times. While the accuracy of ESN-based architectures lags behind that of convolutional neural network-based architectures, the significantly lower training times of ESNs with multiple reservoirs operating in parallel make them a compelling choice for learning spatial relationships in scenarios prioritising energy efficiency and rapid training. This multi-reservoir ESN architecture overcomes standard ESN limitations regarding memory requirements and training times for large networks, providing more accurate predictions than other ESN-based models. These findings contribute to a deeper understanding of the potential of ESNs as a tool for image classification.

An adaptive Marine Predator Optimization Algorithm (MPOA) integrated Gated Recurrent Neural Network (GRNN) classifier model for arrhythmia detection

Cardiovascular disorders are typically diagnosed using an Electrocardiogram (ECG). It is a painless method that mimics the cyclical contraction and relaxation of the heart's muscles. By monitoring the electrical activity of the heart, an ECG can be used to identify irregular heartbeats, strokes, cardiac illnesses, or enlarged hearts. Numerous studies and analyses of ECG signals to identify heart problems have been conducted during the past few years. The disadvantages of the current and hybrid approaches include low accuracy, prolonged time, and a dearth of high-quality datasets. The proposed work aims to design a new detection tool for the diagnosis of Arrhythmia from ECG signals. In this study, the wavelet transformation technique is used for preprocessing after collecting the input ECG signal in order to create the quality-improved and normalized signals. The most pertinent characteristics are then selected from the preprocessed output using the cutting-edge Marine Predator Optimization Algorithm (MPOA), which accelerates classifier learning. In addition, the hyper tuned Gated Recurrent Neural Network (Hyp-GRNN) method is used to learn the most effective features and properly classify the normal and arrhythmia-affected signals. The Ebola Optimization Algorithm (EOSA) is used to execute the hyper parameter tweaking in order to enhance the accuracy and training time of the classifier. At the time of evaluation, performance is assessed and compared using a variety of metrics, including accuracy, precision, recall, F-measure and etc.

An Incremental Input-to-State Stability Condition for a Class of Recurrent Neural Networks

An Incremental Input-to-State Stability Condition for a Class of Recurrent Neural Networks

Optimizing dynamic keystroke pattern recognition with hybrid deep learning technique and multiple soft biometric factors

In this work, we propose an optimization approach for dynamic keystroke pattern recognition by leveraging a hybrid deep learning technique and multiple soft biometric factors. Our methodology begins with the introduction of a novel algorithm called dynamic drone squadron optimization (DDSO) to optimize the selection of optimal features from a pool of multiple keystroke features. We then present an enhanced version of the improved sperm swarm optimization (ISSO) algorithm, which effectively combines the optimal weight features derived from multiple biometric responses. Furthermore, we introduce the multi-stage recurrent neural network (MS-RNN) classifier to accurately recognize and classify keystroke patterns. The performance of our proposed ISSO+MS-RNN technique is evaluated using the benchmark KBOC dataset to validate its effectiveness. Comparative analysis is conducted against existing state-of-the-art techniques, employing various evaluation measures, to demonstrate the superior performance of proposed approach.`

Global inverse optimality for a class of recurrent neural networks with multiple proportional delays

This paper formulates two novel theoretical designs of input-to-state stabilizing control for a class of recurrent neural networks with multiple proportional delays. The analysis tool developed in this paper is based on Lyapunov function and inverse optimality method, which does not require solving Hamilton-Jacobi-Bellman equations. Two inverse optimal feedback laws are constructed via the dimensions of state and input, which ensure the input-state stability for the considered system. When the dimensions of state and input are different, we establish a scalar function and give one of the control laws by Sontag's formula. Furthermore, the designs of inverse optimal control reach both global inverse optimality and global asymptotic stability of the system for some meaningful cost functional. Four numerical examples are provided to show the effectiveness of the inverse optimal control.

LMI-Based Design of a Robust Model Predictive Controller for a Class of Recurrent Neural Networks With Guaranteed Properties

LMI-Based Design of a Robust Model Predictive Controller for a Class of Recurrent Neural Networks With Guaranteed Properties

Determination of railway switch state using a recurrent neural network classifier

Determination of railway switch state using a recurrent neural network classifier

First/second-order predefined-time convergent ZNN models for time-varying quadratic programming and robotic manipulator application

Zeroing neural network (ZNN) model, an important class of recurrent neural network, has been widely applied in the field of computation and optimization. In this paper, two ZNN models with predefined-time convergence are proposed for the time-varying quadratic programming (TVQP) problem. First, in the framework of the traditional ZNN model, the first-order predefined-time convergent ZNN (FPTZNN) model is proposed in combination with a predefined-time controller. Compared with the existing ZNN models, the proposed ZNN model is error vector combined with sliding mode control technique. Then, the FPTZNN model is further extended and the second-order predefined-time convergent ZNN (SPTZNN) model is developed. Combined with the Lyapunov method and the concept of predefined-time stability, it is shown that the proposed FPTZNN and SPTZNN models have the properties of predefined-time convergence, and their convergence time can be flexibly adjusted by predefined-time control parameters. Finally, the proposed FPTZNN and SPTZNN models are compared with the existing ZNN models for the TVQP problem in simulation experiment, and the simulation experiment results verify the effectiveness and superior performance of the proposed FPTZNN and SPTZNN models. In addition, the proposed FPTZNN model for robot motion planning problem is applied and successfully implemented to verify the practicality of the model.

Stable fixed points of combinatorial threshold-linear networks

Combinatorial threshold-linear networks (CTLNs) are a special class of recurrent neural networks whose dynamics are tightly controlled by an underlying directed graph. Recurrent networks have long been used as models for associative memory and pattern completion, with stable fixed points playing the role of stored memory patterns in the network. In prior work, we showed that target-free cliques of the graph correspond to stable fixed points of the dynamics, and we conjectured that these are the only stable fixed points possible [19,8]. In this paper, we prove that the conjecture holds in a variety of special cases, including for networks with very strong inhibition and graphs of size n≤4. We also provide further evidence for the conjecture by showing that sparse graphs and graphs that are nearly cliques can never support stable fixed points. Finally, we translate some results from extremal combinatorics to obtain an upper bound on the number of stable fixed points of CTLNs in cases where the conjecture holds.

Performance enhancement of wind power forecast model using novel pre‐processing strategy and hybrid optimization approach

SummaryDue to the energy crisis and environmental concerns, wind power has seen a considerable increase in use over the past 10 years as a source of renewable energy. Since wind is intermittent and variable, it is obvious that as penetration levels rise, the influence of wind power generation on the electric power system must be considered. Wind power forecasting is essential because large‐scale wind power integration will make it more difficult to plan, operate, and control the power system. An accurate forecast is an efficient way to deal with the operational problems brought on by wind variability. To better utilize wind energy resources, it is essential to increase prediction accuracy. Frequently, the noise in the dataset causes the ramp events to be misclassified or overestimated. The main emphasis of this study is the pre‐processing of wind power data that produces precise time‐series data while reducing noise or artifact content and maintaining the swing feature of the original data. For this task, the data augmentation approach is proposed, where the augmented data (synthetic data) will be added up with the training data, in such a way as to make the strategy useful for real‐time applications. As the next step, the significant features, such as higher‐order statistical features and lower‐order statistical features are extracted. The extracted features act as the input to the recurrent neural network (RNN) classifier, the weights of which are tuned using the proposed honey badger crow (HBCro) optimization algorithm. The proposed HBCro optimization algorithm acts as the major contribution of the proposed model, and it is modeled with the integration of the concepts of the crow search optimization (CSO) algorithm and the honey badger optimization algorithm (HBA). The proposed system is simulated in MATLAB and the effectiveness of the proposed method is validated by comparison with other conventional methods in terms of Error measures. Furthermore, the developed HBCro‐based RNN obtained efficient performance in terms of MSE, MAE, RMSE, RMPSE, MAPE, MARE, MSRE, and RMSRE of 0.1578, 0.0442, 0.2102, 123.238, 124.72, 0.9944, 1.1799, and 1.0732, respectively.

Finite time convergent recurrent neural network for variational inequality problems subject to equality constraints

This paper aims to present a new class of recurrent neural networks based on projection operators for solving variational inequalities and related optimization problems subject to linear equalities and bound constraints. In the proposed approach, the neural network structure is based on the Karush–Kuhn–Tucker (KKT) optimality conditions. Instead of commonly used activation functions, the KKT multipliers are considered control inputs and implemented with finite time stabilizing terms based on unit control. The output variables of the neural network are proven to be stable in the sense of Lyapunov and convergent to optimal solutions in finite time. In addition, a methodology based on the particle swarm optimization (PSO) algorithm is presented for the optimal selection of network design parameters. The main advantage of the proposed neural network is its fixed number of parameters regardless of problem dimension, enabling the network to be easily scalable from a lower to a higher dimension. Finally, simulation results of numerical examples are provided to illustrate the effectiveness and performance of the proposed neural network.