Recurrent Neural Network Research Articles

Wide-Range Operation of Microwave Photonic Sensor Using Recurrent Neural Network

Wide-Range Operation of Microwave Photonic Sensor Using Recurrent Neural Network

Multi-Radar Interference Mitigation in Photonics-Based Radar With Sliding Window LSTM Recurrent Neural Network

Multi-Radar Interference Mitigation in Photonics-Based Radar With Sliding Window LSTM Recurrent Neural Network

Forecasting research influence: a recurrent neural network approach to citation prediction

Forecasting research influence: a recurrent neural network approach to citation prediction

A Systematic Literature Review on The Role of LSTM Networks in Capturing Temporal Dependencies in Data Mining Algorithms

Data mining plays a crucial role in extracting meaningful insights from large and complex data sets, with broad applications in sectors like finance, health care, and market analysis. Traditional techniques—such as classification, clustering, association rule mining, regression, and anomaly detection—are effective for analyzing structured data but struggle with sequential data due to the challenges of modeling temporal dependencies. Long Short Term Memory (LSTM) networks, a specialized form of Recurrent Neural Networks (RNNs), provide a solution to these challenges. By incorporating memory cells and gating mechanisms, LSTM effectively manage long-term dependencies and address issues like vanishing and exploding gradients. This paper reviews the impact of LSTM networks on data mining, analyzing over 60 key publications. By synthesizing concepts and recent advancements, the review underscores how LSTMs enhance the ability of data mining algorithms to capture and predict temporal patterns, reflecting current research trends and innovations

Diabetes Prediction Using Parametric Swish-based Recurrent Neural Network

Diabetes Prediction Using Parametric Swish-based Recurrent Neural Network

A Secured Keystroke-Based Model for Preventing Social Engineering Attacks using Recurrent Neural Network

Among several authentication problems, preventing social engineering attacks using behavioural biometric approach has not received the required attention especially with focus on keystroke dynamics. This study aims to leverage the power of deep learning for more accurate and robust continuous authentication based on typing patterns. The proposed framework for this study utilized deep learning algorithm for behavioural biometrics authentication using Keystroke dynamics. The deep learning model was developed using Recurrent Neural Network (RNN) algorithm and was optimized was to obtain a better performance with Bayesian optimization which, eventually enhanced the model's accuracy. The dataset was split into training and testing in the model design phase and some hyperparameters such as dense, activation, batch size, sigmoid, filament, input size and epoch were used and optimized for building the deep learning algorithm. The RNN model is used to generate the evaluation metrics such as log loss, accuracy, precision and recall. The result presented the accuracy, precision, recall, and loss function as 100%, 100%, 100%, and 36% respectively for optimized model. The cost metrics yielded 0.0032, 0.0032, and 0.0006 MAE, MSE, and RMSE respectively. The developed KBB shows high level of social engineering attacks mitigation in comparison with the existing solution from the performance measure results.

Machine learning techniques for diagnosis of rare diseases from medical images

Introduction/Background: Rare diseases are life-threatening or chronically debilitating conditions that affect a small percentage of the population. Early and accurate diagnosis of rare diseases is challenging due to their complexity and limited understanding. Conventional diagnostic methods are often inadequate, time-consuming, and expensive. Machine learning (ML) has emerged as a promising technique for the analysis of medical images to support the diagnosis of various diseases including rare disorders. ML algorithms can learn complex patterns from large medical imaging datasets to aid clinicians in disease diagnosis, prognosis, and detection of complications. Materials and Methods: A structured search was conducted in electronic databases, including PubMed, Scopus, Web of Science, and Google Scholar to identify peer-reviewed articles published between 2013 to 2023 related to ML-based diagnosis of rare diseases from medical images. A total of 187 articles were identified after removing duplicates. The titles and abstracts of retrieved articles were screened to determine their relevance based on the research objective. Finally, 51 full-text articles were selected for the final review. The key data extracted from selected studies included diseases, imaging modalities, ML algorithms, performance metrics, datasets, and limitations. Results: Most studies evaluated deep learning-based ML techniques, with convolutional neural networks (CNN) being the most applied algorithm. CNNs were mainly used to classify rare diseases based on specific imaging features in X-rays, CT, and MRI scans. Diseases frequently investigated included cardiovascular, neurological, and rare genetic disorders. Publicly available datasets such as Deep Lesion, MSD, and ChestX-ray14 were commonly utilized. Most studies reported high classification accuracy ranging from 80-95% on test datasets. However, limited generalizability due to small private datasets and lack of external validation were limitations. Discussion: The results demonstrate the potential of deep learning for the image-based diagnosis of rare diseases. Features learned from large imaging data sets enabled CNNs to distinguish subtle abnormalities. Hybrid models combining CNNs with other techniques like recurrent neural networks further improved performance. Overall, ML showed promise as a decision support tool. However, more multi-center validation studies are needed using larger and diverse datasets before clinical adoption. Standardization of performance metrics and reporting is also required to establish reliability and generalizability. Conclusion: This comprehensive review provided insights into ongoing research applying ML to medical images for rare disease diagnosis. While initial results are encouraging, further work is still necessary before ML can be reliably used in clinical decision making. Larger collaborative efforts and open-source datasets are needed to advance the field. Standardization of ML frameworks can also promote reproducible research and comparison of different methodologies.

Comparative analysis of hybrid cognitive radar techniques for enhanced target detection and tracking: A performance evaluation perspective

This paper introduces an innovative hybrid algorithmic approach for cognitive radar systems, by integrating unique machine learning optimization techniques such as, YOLO (You Only Look Once), Mask R-CNN, and Recurrent Neural Networks (RNNs). Through extensive simulations, the integrated approach demonstrates notable enhancements in target detection, instance segmentation, and target tracking within radar systems. Leveraging deep learning models, the framework facilitates adaptive and intelligent processing of radar data, augmenting system performance in dynamic environments. By seamlessly integrating state-of-the-art techniques, the proposed framework showcases a comprehensive solution to the challenges faced by traditional radar systems. This research represents a significant stride forward in radar technology, promising transformative impacts across various domains, including surveillance, remote sensing, and autonomous navigation. As radar systems continue to evolve, the adoption of advanced deep learning techniques offers unprecedented opportunities for enhancing situational awareness and decision-making capabilities in complex operational scenarios.

Review of Convolutional Neural Network

Over the past decade, the domain of deep learning has emerged as a swiftly expanding area of interest among researchers globally. It offers substantial benefits over traditional shallow networks, particularly in the realms of feature extraction and model fitting. Deep learning excels at uncovering intricate, distributed features from raw data, which exhibit robust generalization capabilities. It has triumphantly addressed challenges that were once deemed intractable within the field of artificial intelligence. With the exponential growth in the volume of training data and a marked increase in computational power, deep learning has achieved remarkable strides and found applications across a spectrum of domains, including but not limited to object detection, computer vision, natural language processing, speech recognition, and semantic parsing, thereby accelerating the evolution of artificial intelligence. Deep learning encompasses a series of non-linear transformations organized hierarchically, with deep neural networks representing the predominant form of contemporary deep learning methodologies. Inspired by the neural connections found in the animal visual cortex, these networks are characterized by local connectivity, shared weights, and pooling operations. Such attributes not only simplify the network model and curtail the number of trainable parameters but also engender invariance to shifts, distortions, and scaling, thereby endowing the model with enhanced robustness and fault tolerance. This makes the training and optimization of the network structure more manageable. This paper commences with a retrospective on the evolution of convolutional neural networks (CNNs). Subsequently, it delineates the architectures of neuron models and multilayer perceptron. It proceeds to dissect the CNN architecture, which typically encompasses a sequence of convolutional and pooling layers, culminating in fully connected layers, each serving distinct functions. The paper also elaborates on various enhanced algorithms for CNNs, such as the “Network in Network” and “Spatial Transformer Networks,” while also introducing supervised and unsupervised learning methodologies pertinent to CNNs and highlighting several prevalent open-source tools. Further, the paper scrutinizes the application of CNNs in various fields, including image classification, facial recognition, audio retrieval, electrocardiogram analysis, and object detection. It posits the amalgamation of CNNs with recurrent neural networks as a potential alternative for training datasets. The research culminates in the design of CNN structures with varying parameters and depths, supported by experimental analysis that elucidates the interplay among these parameters and the ramifications of their configuration. Ultimately, the paper synthesizes the merits and lingering challenges associated with CNNs and their practical applications.

Water-to-air unmanned aerial vehicle (UAV) based rolling shutter optical camera communication (OCC) system with gated recurrent unit neural network (GRU-NN)

Underwater sensors and autonomous underwater vehicles (AUVs) are widely adopted in oceanic research activities. As the number of underwater sensors and AUVs is growing quickly, the bandwidth requirements are increasing accordingly. In this work, we put forward and demonstrate a large field-of-view (FOV) water-to-air unmanned aerial vehicle (UAV) based optical camera communication (OCC) system with gated recurrent unit neural network (GRU-NN) for the first time to the best of our knowledge. As the UAVs are embedded with complementary-metal-oxide-semiconductor (CMOS) cameras, there is no need to install OCC receivers (Rxs), reducing the deployment cost. Moreover, the large photo-sensitive area of the CMOS camera can support a large FOV OCC transmission without the need for precise optical alignment. Here, by utilizing the column matrix identification during the rolling shutter pattern decoding in the CMOS image sensor, the scintillation caused by water turbulence can be reduced. Besides, in the outdoor and windy environment, the UAV will experience significant movement caused by the wind making it very difficult to capture stable OCC frames in the CMOS image sensor. Here, we propose and demonstrate utilizing GRU-NN, which is a special realization of the recurrent neural network (RNN) with memory cells capable of learning the time-domain dependent signals. It is shown that the GRU-NN can learn effectively from successive image frames in time-domain and produce correct prediction even under the windy and unstable UAV flying environment. Experimental results reveal that the proposed GRU-NN can outperform the previous pixel-per-symbol labeling neural network (PPS-NN), and also can significantly reduce the computation time when compared with long-short-term-memory-neural-network (LSTM-NN). The proposed system can decode 4-level pulse-amplitude-modulation (PAM4) rolling shutter OCC patterns at data rates of 5.4 kbit/s and 3.0 kbit/s under clear and cloudy water, respectively, fulfilling the pre-forward error correction bit-error rate (pre-FEC BER = 3.8 × 10−3). We also demonstrate that the UAV based OCC system can support data rates of 5.4 kbit/s, 4.2 kbit/s, and 3.0 kbit/s at distances of 2.2 m, 3.2 m, and 4.2 m, respectively, at outdoor and windy environments.

Machine learning for air quality index (AQI) forecasting: shallow learning or deep learning?

In this study, several machine learning (ML) models consisting of shallow learning (SL) models (e.g., random forest (RF), K-nearest neighbor (KNN), weighted K-nearest neighbor (WKNN), support vector machine (SVM), artificial neural network (ANN), and deep learning (DL) models (e.g., long short-term memory (LSTM), gated recurrent unit (GRU), recurrent neural network (RNN), and convolutional neural network (CNN)) have been employed for predicting air pollution and its classification. The models were selected based on factors such as prediction accuracy, model generalization, model complexity, and training time. Our study focuses on analyzing and predicting the air quality index (AQI) using daily PM10 concentration as natural pollutants and nine meteorological parameters from March 2013 to February 2022 in Zabol. We also utilized the information gain (IG) method for feature selection. Several measures including accuracy, F1 score, precision, recall, and the area under the curve (AUC), are computed to assess model performance. This study demonstrates the efficacy of DL models, particularly CNN, in predicting the AQI with remarkable accuracy. Our findings reveal that all models effectively classify air quality levels, with an AUC of 0.95 for the good class in both DL and ANN models, significantly outperforming SL models. The AUC values for the hazardous and moderate classes of DL models were also impressive, at 0.90 and 0.83, respectively, underscoring their effectiveness in critical classifications. In terms of performance, CNN achieved an accuracy of 0.60, leading the models, while RF followed closely at 0.58. RNN, GRU, ANN, and SVM each reached an accuracy of 0.57, demonstrating a competitive edge. LSTM and WKNN recorded an accuracy of 0.55, and KNN was slightly lower at 0.53. These results highlight the superior capabilities of DL models in addressing complex air quality classifications, providing invaluable insights for policymakers. By leveraging these advanced techniques, stakeholders can implement more effective strategies to combat air pollution and safeguard public health. It is worth noting that irregular monitoring of air quality data may affect the robustness of our predictions, highlighting the need for more consistent data collection to ensure an accurate representation of pollution levels.

Enhancing IoT Security Using GA-HDLAD: A Hybrid Deep Learning Approach for Anomaly Detection

The adoption and use of the Internet of Things (IoT) have increased rapidly over recent years, and cyber threats in IoT devices have also become more common. Thus, the development of a system that can effectively identify malicious attacks and reduce security threats in IoT devices has become a topic of great importance. One of the most serious threats comes from botnets, which commonly attack IoT devices by interrupting the networks required for the devices to run. There are a number of methods that can be used to improve security by identifying unknown patterns in IoT networks, including deep learning and machine learning approaches. In this study, an algorithm named the genetic algorithm with hybrid deep learning-based anomaly detection (GA-HDLAD) is developed, with the aim of improving security by identifying botnets within the IoT environment. The GA-HDLAD technique addresses the problem of high dimensionality by using a genetic algorithm during feature selection. Hybrid deep learning is used to detect botnets; the approach is a combination of recurrent neural networks (RNNs), feature extraction techniques (FETs), and attention concepts. Botnet attacks commonly involve complex patterns that the hybrid deep learning (HDL) method can detect. Moreover, the use of FETs in the model ensures that features can be effectively extracted from spatial data, while temporal dependencies are captured by RNNs. Simulated annealing (SA) is utilized to select the hyperparameters necessary for the HDL approach. In this study, the GA-HDLAD system is experimentally assessed using a benchmark botnet dataset, and the findings reveal that the system provides superior results in comparison to existing detection methods.

Prediction of QT interval in ECGs through baseline wander removal and deep learning technique

Abstract Background In electrocardiograms (ECGs), the QT interval, calculated from the onset of the Q-wave to the end of the T-wave, reflects the repolarization status of the heart. Currently, the QT interval automatically measured in 12-lead electrocardiograms is inaccurately derived as the median value, rather than utilizing the more precise mean value of QT intervals from all leads. This approach fails to provide information on heterogeneous QT prolongation crucial for predicting arrhythmias, as it does not accurately represent the values of QT intervals in each lead. Purpose Traditional analysis of ECGs based on QT intervals necessitates direct interpretation by cardiologists, leading to potential errors in diagnosis. In response to this challenge, we present a model that integrates low-pass filtering and continuous wavelet transformation to eliminate baseline wander noise occurring during ECG measurements. Moreover, to account for temporal features, we incorporate Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory (Bi-LSTM) to estimate QT intervals. Methods The dataset used in this study was the QT Database, comprising ECG data for 105 patients recorded at a sampling rate of 250Hz over a duration of 15 minutes. Each ECG was associated with annotations for two leads and information regarding the P, QRS, and T waves. We structured the dataset into a ratio of approximately 8:1:1, resulting in 51 patients for training (15,490 samples), 6 patients for validation (1,564 samples), and 7 patients for testing (1,682 samples). Results For the evaluation of deep learning, we selected Accuracy and F1-Score as metrics. We experimented with two configurations: the proposed model with two Bi-LSTM layers and an alternative with a single Bi-LSTM layer of 100 units. We also tested a similar recurrent neural network, GRU, with the same unit configuration. The results indicated that the method with two Bi-LSTM layers outperformed others in both Accuracy and F1-Score. Conclusion By incorporating CNN and Bi-LSTM, we conducted QT interval estimation, leveraging temporal features. This methodology facilitated the elimination of diverse noise types during ECG measurements, resulting in enhanced accuracy for QT interval estimation through the consideration of temporal ECG characteristics. In our future research endeavors, we plan to deploy this model for predicting arrhythmias, specifically targeting conditions like long QT syndrome and drug-induced QT prolongation.Methods and result

Sentiment analysis of Algerian Arabic dialect on social media Using Bi-LSTM recurrent neural networks

This paper presents a sentiment analysis approach using Bidirectional Long Short-Term Memory (Bi-LSTM) Recurrent Neural Networks to train predictive models for sentiment analysis on social media, particularly focusing on Algerian Arabic Dialect. The method leverages word-to-vector embedding for word representation and incorporates natural language understanding of emojis to improve semantic interpretation. The model achieves a high accuracy of 94%, demonstrating its effectiveness in analyzing sentiments in online discussions. The originality lies in applying Bi-LSTM to handle multilingual challenges on social platforms. The findings have practical implications for business, policymaking, and public sentiment evaluation, while also contributing positively to fostering informed online discourse.

Multi-Attention Recurrent Neural Network for Multi-Step Prediction of Chlorophyll Concentration

Abstract: Chlorophyll prediction facilitates the comprehension of red tide characteristics and enables early warning. In practice, it is formulated as a multivariate time series forecasting problem aimed at forecasting future chlorophyll concentrations by considering both exogenous factors and chlorophyll. However, the multi-step prediction of chlorophyll concentration poses a formidable challenge due to the intricate interaction between factors and the long temporal dependence between input sequences. In this work, we propose a Multi-attention Recurrent Neural Network (MaRNN) for the multi-step prediction of chlorophyll concentration. The MaRNN comprises an encoder incorporating two-stage spatial attention and a decoder employing temporal attention. The encoder first learns the significance of exogenous factors for prediction in the first phase, and subsequently captures the spatial correlation between the exogenous sequence and chlorophyll sequence in the second phase. The decoder further excavates input sequences that exhibit a strong correlation with the task through temporal attention module, thereby enhancing the prediction accuracy of the model. Experiments conducted on two real-world datasets reveal that MaRNN not only surpasses state-of-the-art methods in performance, but also offers interpretability for chlorophyll prediction.

Artificial Neural Network to Predict Curvature Light Shelf Design Related Daylighting Optimization on Office Spaces

Energy Optimization in building design field now has been revolutionized due to AI and machine learning applications. Leveraging daylight to reduce artificial lighting consumption holds promise for significant energy savings, yet the nonlinear nature of daylight patterns poses challenges in prediction and optimization. This study proposes a novel approach to automated light shelf design using machine learning algorithms, specifically artificial neural networks (ANNs) such as recurrent neural networks (RNN) by long short term memory (LSTM), to accelerate daylighting simulation and optimization processes. The methodology employed two distinct approaches: Firstly, we employed the theoretical-analytical approach to explore methods for utilizing machine learning in natural lighting and light shelf parameters. Second, the practical and applied method involved creating a predictive model for designing the light shelf using appropriate AI and ML techniques. This model is based on an office geometry model at the El Arish weather file in Egypt, four-dimensional training room models with three internal zones-oriented south. Rhinoceros and Grasshopper, two parametric simulation tools, are used to normalize and optimize light shelf parameters like geometry cross-section, curvature surface, width, height position, depth, tilt angle, and reflectance materials. Then, the Galapagos plug-in and Colibri2 are used for dataset creation and optimization. The results demonstrate that automated light shelf operation has a significant impact on internal daylighting quality. RNNs enable rapid prediction of optimization models, reducing time consumption in the early design phase. ML facilitates decision-making by generating evaluative criteria from user-selected design options. RNNs were classified as good and bad and used LSTM to enhance prediction accuracy for efficient illuminance values metric in zones 1 and 2. Challenges include increasing the simulation procedure's efficiency. The results of model accuracy reached 99%. Hence, future research should prioritize resolving the previously identified concerns. In conclusion, this study underscores the importance of ML-driven approaches in early design phases to optimize building energy consumption and pave the way for sustainable architectural practices.

Sentiment Analysis of Twitter Users Towards the Kartu Prakerja Program Using the Naive Bayes Method

This study conducts a sentiment analysis of Twitter users regarding the Indonesian government’s Kartu Prakerja program, utilizing the Naive Bayes method for classification. Launched in 2020 to enhance employability skills amidst the COVID-19 pandemic, the program has garnered various public responses. A total of 836 tweets containing the keyword "Kartu Prakerja" were collected using the Twitter API and analyzed to determine sentiment distribution. Results indicate a predominance of neutral sentiment (800 tweets), with only 17 positive and 22 negative tweets. The Naive Bayes method achieved an accuracy of 95%, demonstrating its effectiveness in sentiment classification. However, comparisons with other methods, such as Support Vector Machine (SVM) and Recurrent Neural Network (RNN), reveal that these techniques yield higher accuracy rates (98.34% and 96%, respectively). This research highlights the importance of sentiment analysis in understanding public perceptions and informs policymakers about areas needing improvement. The findings underscore the potential of integrating advanced machine learning techniques to enhance sentiment analysis and provide insights into the effectiveness of government programs like Kartu Prakerja.

Enhancing Daily Crash Count Prediction using Deep Learning with Window Size Selection and Seasonality Predictor Integration

The escalating need for proactive safety measures, coupled with advancements in data collection and analytical techniques, has significantly refined the accuracy of crash count predictions, shifting from annual scales to finer daily or hourly estimates. This research places emphasis on recurrent neural networks (RNNs), specifically the long short-term memory (LSTM) model, acknowledged for effectively managing sequential data in time series predictions. Paramount considerations encompass the treatment of input data, including the decision to incorporate temporal features alongside endogenous historical target values, and the establishment of an optimal window size for data input. Despite the critical nature of these facets, exhaustive studies concurrently investigating both under controlled conditions are scarce. This research addresses this gap, assessing diverse scenarios featuring distinct temporal treatments and window sizes, and employing an LSTM model with uniform fine-tuned parameters to ensure a fair comparison. Findings indicate a significant variation in performance among models employing different window sizes and month predictor integration, under identical RNN structures and LSTM configurations. The best model outperformed the least effective by roughly 30% concerning root mean square error (RMSE) and ranking correlation between predicted and actual target crash counts in the test datasets. Unlike seasonality predictor treatment, diverse window size selections did not lead to statistically significant differences in model performance. Generally, a window size of 3 performed worst under most conditions, followed by a size of 14. Sizes of 7 and 28 performed best in nearly an equal numbers of cases.

Does a Fractional-Order Recurrent Neural Network Improve the Identification of Chaotic Dynamics?

This paper presents a quantitative study of the effects of using arbitrary-order operators in Neural Networks. It is based on a Recurrent Wavelet First-Order Neural Network (RWFONN), which can accurately identify several chaotic systems (measured by the mean square error and the coefficient of determination, also known as R-Squared, r2) under a fixed parameter scheme in the neural algorithm. Using fractional operators, we analyze whether the identification capabilities of the RWFONN are improved, and whether it can identify signals from fractional-order chaotic systems. The results presented in this paper show that using a fractional-order Neural Network does not bring significant advantages in the identification process, compared to an integer-order RWFONN. Nevertheless, the neural algorithm (modeled with an integer-order derivative) proved capable of identifying fractional-order dynamical systems, whose behavior ranges from periodic and multi-stable to chaotic oscillations. That is, the performances of the Neural Network model with an integer-order derivative and the fractional-order network are practically identical, making the use of fractional-order RWFONN-type networks meaningless. The results deepen the work previously published by the authors, and contribute to developing structures based on robust and generic neural algorithms to identify more than one chaotic oscillator without retraining the Neural Network.

Photometric redshift estimation for CSST Survey with LSTM Neural Networks

Abstract Accurate estimation of photometric redshifts (photo-zs) is crucial for cosmological surveys. Various methods have been developed for this purpose, such as template fitting methods and machine learning techniques, each with its own applications, advantages, and limitations. In this study, we propose a new approach that utilizes a deep learning model based on Recurrent Neural Networks (RNN) with Long Short-Term Memory (LSTM) to predict photo-z. Unlike many existing machine learning models, our method requires only flux measurements from different observed filters as input. The model can automatically learn the complex relationships between the flux data across different wavelengths, eliminating the need for manually extracted or derived input features, thereby providing precise photo-z estimates. The effectiveness of our proposed model is evaluated using simulated data from the Chinese Space Station Telescope (CSST) sourced from the Hubble Space Telescope Advanced Camera for Surveys (HST-ACS) and the COSMOS catalog, considering anticipated instrument effects of the future CSST. Results from experiments demonstrate that our LSTM model, compared to commonly used template fitting and machine learning approaches, requires minimal input parameters and achieves high precision in photo-z estimation. For instance, when trained on the same dataset and provided only with photometric fluxes as input features, the proposed LSTM model yields one-third of the outliers fout observed with a Multi-Layer Perceptron Neural Network (MLP) model, while the normalized median absolute deviation $\rm \sigma _{NMAD}$ is only two-thirds that of the MLP model. This study presents a novel approach to accurately estimate photo-zs of galaxies using photometric data from large-scale survey projects.