RNN Model Research Articles

Comparison of Convergence for Different Models and Loss Function: Evidence from Transformer, Diffusion and RNN

Abstract. With the development of science and technology, people are using machine learning as a method to make life easier. However, the performance corresponding to different models will produce different predictions and results for one thing. In order to improve the performance of the model, this paper investigates the influence of the learning rate hyperparameter on the model's performance, and demonstrates it through the convergence of the loss function. After experimental research, it has been found that different models exhibit significant differences in their performance when processing the same dataset. Meanwhile, different learning rates also have a significant impact on the performance of the model. Therefore, after selecting the correct model for machine learning, one should also adjust a relatively good hyperparameter to make the entire process smoother. Based on the analysis, one will gain a basic understanding of the optimal learning rates for transformer, diffusion, and RNN models when training MNIST. It is convenient for people to set better hyperparameters and obtain better prediction and decision-making results when using these three models, so that one can demonstrate better performance when using these three models.

Using DL Models in the Service Layer to Enhance the Fault Tolerance of IoT Networks

In an IoT network, the networked servers form a service layer, providing services to the users and the devices. The request to the service servers is routed through the gateway on one side of the services layer and the networked controllers on the other side. Data are transported from the sensors/devices through cluster heads en route to base stations and the controllers to the service servers, where the data are processed and sent for storage in the cloud through gateways. When any device is broken down or becomes non-operational, the inputs are not sensed, creating a gap in the data. The data transmitted from the devices would then become an incomplete flow; such data are not suitable for undertaking data analytics or predictions. The missing data must be first identified as the data flow and estimated or predicated to complete the data before they are transmitted through the cloud for storage and subsequent retrievals. This paper proposes a recurrent (RNN) neural network to predict the missing data. Two models are tested to predict the missing data: the multi-layer perceptron (MLP) model and a long short-term memory (LSTM)-based RNN model. The RNN-based model provides 99.66% accurate data prediction compared to other models.

Deep learning-based improved transformer model on android malware detection and classification in internet of vehicles

With the growing popularity of autonomous vehicles (AVs), confirming their safety has become a significant concern. Vehicle manufacturers have combined the Android operating system into AVs to improve consumer comfort. However, the diversity and weaknesses of the Android operating system pose substantial safety risks to AVs, as these factors can expose them to threats, namely Android malware. The advanced behaviour of multi-data source fusion in autonomous driving models has mitigated recognition accuracy and effectualness for Android malware. To efficiently counter new malware variants, novel techniques distinct from conventional methods must be utilized. Machine learning (ML) techniques cannot detect every new and complex malware variant. The deep learning (DL) model is an efficient tool for detecting various malware variants. This manuscript proposes a Deep Learning-Based Improved Transformer Model on Android Malware Detection (DLBITM-AMD) technique for Internet vehicles (IoVs). The main aim of the presented DLBITM-AMD approach is to detect Android malware effectually and accurately. The DLBITM-AMD method performs a Z-score normalization process to convert the raw data into a standard form. Then, the DLBITM-AMD approach utilizes the binary grey wolf optimization (BGWO) model to select optimum feature subsets. An improved transformer is integrated with the RNN model and softmax to enhance classification for Android malware recognition. Finally, the snake optimizer algorithm (SOA) method is employed to select the optimum parameter for the classification method. An extensive experiment of the DLBITM-AMD method is accomplished on a benchmark dataset. The performance validation of the DLBITM-AMD technique portrayed a superior accuracy value of 99.26% over existing Android malware recognition models.

Recurrent Neural Networks based Online Behavioural Malware Detection Techniques for Cloud Infrastructure

: With the increased basis of applications and services on Cloud, the dependence on cloud technologies has expanded rapidly. This transition eases some of the hardware management troubles, but it also exposes broad security threats, especially in regard to malware. Malware is a major threat to cloud computing systems mainly in IaaS dominated environments. In this paper we explore the usage of Recurrent Neural Networks, RNNs in malware detection in cloud VMs. In particular, we examine two of the most popular RNN models; Long short term Memory, LSTM and Bidirectional RNNs, BIDI, that try to understand malware patterns by watching the activity of systems and the use of CPU, Memory and Disk. On the dataset of 40,680 samples 40,680 malware and benign samples, the models were trained on process level features from real malware in a free and open cloud environment. Both models achieved detection rates of over 99% in average key performance metrics. Moreover, wealso investigate the impact of different representations of input data on the performance of the models. The results of the study clearly support the viability of RNNs for online malware detection in cloud infrastructure, and allow for high accuracy and real time detection.

A deep multi-filtering network based on dilated mask RCNN framework with self-attention RNN model for segmenting and classifying bone fracture

A deep multi-filtering network based on dilated mask RCNN framework with self-attention RNN model for segmenting and classifying bone fracture

AI-Based Network Traffic Analysis for Threat Hunting

The focus of this study is to explore the effectiveness of advanced models like CNN and RNN to detect cyber threats on our network systems that are posted to us with every passing day. By using the CIC-DDoS 2019 dataset we test the possibility of detecting threats on network traffic in a realistic environment. This study tests both the limitations and strengths of neural networks for our case study. Our aim is to make a system efficient enough to improve itself over time and mitigate the threats posed in future by predicting them beforehand. This involves difficult testing of the CNN and RNN models to identify any patterns or anomalies that are indicate DDos attacks. This will work with the integration of AI with Cybersecurity and working on the real-world data and improve the accuracy and speed of these models to work efficiently. This will help make our environment resistant to the threats that will emerge with the advent of technology. These findings are intended to inform the development of more robust and autonomous AI systems that safeguard Internet while respecting user privacy.

Prediction of carbon dioxide levels in the European Alps based on machine learning algorithms

With the rapid development of world industrialization, the continuous increase of global carbon dioxide emissions has led to the gradual deterioration of the ecological environment and the obvious aggravation of the greenhouse effect. In this paper, the carbon dioxide content in the air over the European Alps is taken as the research object, and the RNN and LSTM neural network prediction models are respectively used to compare and predict it. The results show that the fitting effect of LSTM is better than that of RNN, and the fitting effect of the prediction model will also improve with the increase of iteration times and sample size. Since carbon monoxide and methane in the air will cause changes in carbon dioxide content, this paper adds the two factors into the LSTM prediction model as influencing factors, and the goodness-of-fit can reach 0.95, which is higher than the prediction results when there are only one influencing factor or no influencing factor. In order to reduce the running time, XGBoost, LightGBM and random forest algorithms are respectively used in this paper and Bayesian optimization algorithm is used to predict the carbon dioxide content. The results show that the prediction effect is slightly lower than LSTM. Therefore, this paper takes the above three algorithms as the base model. Linear regression experiments are carried out for the meta-model's Stacking fusion algorithm. The goodness-of-fit can reach 0.92, which significantly improves the prediction effect compared with the base model. Finally, the sensitivity analysis of the Stacking fusion model is carried out in this paper. The experimental results show that the model has strong stability.

Remaining useful life prediction of lithium-ion batteries based on DBO[sbnd]CNN-DSformer

In order to improve the accuracy of predicting RUL of lithium-ion batteries, a lithium-ion battery RUL prediction method based on the DBOCNN-DSformer model is proposed. Firstly, the health characteristics of the battery are extracted and the local information of health features is mined using CNN. DSformer is utilized for global information, local information, and variable correlation learning of battery aging characteristics. The DBO is used to optimize the super-parameters of the CNN-DSformer model and build the DBOCNN-DSformer model. Finally, the battery aging data set was used for verification. The results show that DBOCNN-DSformer, which sets different prediction starting points, can extract sequence information from input data and establish long-term dependencies between sequences. The maximum average MRE error in the NASA data set was 0.05, the maximum average MAE was 0.018, and the maximum average AE error was within 5. The maximum average MRE error of the CALCE data set was 0.37, the maximum average MAE was 0.014, and the maximum average AE prediction error was within 10. Compared with LSTM, RNN, and Transformer models, it was found that DBOCNN-DSformer showed high prediction accuracy and good robustness.

Improving predictions of rock tunnel squeezing with ensemble Q-learning and online Markov chain

Predicting rock tunnel squeezing in underground projects is challenging due to its intricate and unpredictable nature. This study proposes an innovative approach to enhance the accuracy and reliability of tunnel squeezing prediction. The proposed method combines ensemble learning techniques with Q-learning and online Markov chain integration. A deep learning model is trained on a comprehensive database comprising tunnel parameters including diameter (D), burial depth (H), support stiffness (K), and tunneling quality index (Q). Multiple deep learning models are trained concurrently, leveraging ensemble learning to capture diverse patterns and improve prediction performance. Integration of the Q-learning-Online Markov Chain further refines predictions. The online Markov chain analyzes historical sequences of tunnel parameters and squeezing class transitions, establishing transition probabilities between different squeezing classes. The Q-learning algorithm optimizes decision-making by learning the optimal policy for transitioning between tunnel states. The proposed model is evaluated using a dataset from various tunnel construction projects, assessing performance through metrics like accuracy, precision, recall, and F1-score. Results demonstrate the efficiency of the ensemble deep learning model combined with Q-learning-Online Markov Chain in predicting surrounding rock tunnel squeezing. This approach offers insights into parameter interrelationships and dynamic squeezing characteristics, enabling proactive planning and support measures implementation to mitigate tunnel squeezing hazards and ensure underground structure safety. Experimental results show the model achieves a prediction accuracy of 98.11%, surpassing individual CNN and RNN models, with an AUC value of 0.98.

Computer-Assisted Online Learning of English Oral Pronunciation Based on DAE End-to-End Recurrent Neural Networks

With the development of globalization, learning a second language has received increasing attention from people. To improve English oral proficiency, a computer-aided online learning system for English oral pronunciation is studied. A denoising autoencoder is integrated into the system to create a simplified end-to-end recurrent neural network for pronunciation detection and diagnosis based on deep learning. The study first collected and preprocessed oral pronunciation data of English learners, including enhancing speech signals and reducing noise. Next, an RNN model with Long Short-Term Memory (LSTM) as the core was constructed to capture time series characteristics in pronunciation. And use DAE to extract features and reduce the influence of background noise to enhance the recognition of pronunciation features. At the same time, the study utilized web crawler technology to collect a large amount of oral pronunciation data from non-native English learners, and constructed an English oral corpus containing pronunciation errors. And in order to simulate real situations, white noise and pink noise were artificially added to the corpus in the study, and they were divided into training and testing sets in a ratio of 60% to 40%. The results showed that the classification accuracy of the system in the training and testing sets under white noise environment was 78.97% and 94.01%, respectively, and the classification accuracy in the pink noise environment was 76.19% and 94.03%, respectively. The system’s error detection accuracy in vowel and consonant pronunciation detection is 88.91% and 91.68%, respectively, and the error correction accuracy in vowel and consonant pronunciation detection is 90.67% and 91.96%, respectively. In summary, the research on computer-aided online learning of English oral pronunciation based on Denoising Auto Encoders end-to-end recurrent neural networks has effectively improved learning efficiency.

Passive earth pressure coefficients prediction using deep learning techniques

Predicting passive earth pressure coefficients is crucial for the design of retaining structures, as these coefficients, dependent on soil properties and wall geometry, they impact wall stability. In a practical context, the experimental and analytical approaches are often inaccurate and complex. In this paper, we have developed and tested three Recurrent Neural Networks: RNN, LSTM, and GRU models to predict passive earth pressure coefficients 〖(K〗_pγ, K_pq and K_(pc )) using datasets with 1147 observations, including key parameters such as the ratio of wall width to height (b⁄(h)), the soil-wall interface friction angle to internal friction angle ratio (δ⁄(ϕ)) and the soil friction angle (ϕ). The results demonstrate that our deep neural network models perform better in terms of accuracy and reliability when compared to previously models. As a result, the LSTM model outperformed the RNN, and GRU models in terms of prediction accuracy, exhibiting the best MSE performance (K_pγ = 0.0014, K_pq = 0.0020, K_pc = 0.0020).

A Comparative Study of the Effect of Weather Conditions on Solar Irradiance Forecasting Using Various Methods in Yogyakarta, Indonesia

Abstract The government’s plan to build utility-scale solar power plants for the next decade will impact the high penetration of solar energy into the grid. This problem is associated with the intermittent nature of solar electricity, which can disrupt the grid system’s stability and reliability if it is not tackled. Furthermore, the complicated weather conditions in Indonesia make this variable challenging to predict. Therefore, one of the solutions offered is the solar irradiance forecasting technique that helps the operator during planning. This paper will look at the differences in these weather characteristics in predicting solar irradiance using three forecasting methods: LSTM, RNN, and ARIMA. The data is taken from direct measurements installed in the center of the island of Java, namely Yogyakarta. The result of this work shows that among LSTM, RNN, and ARIMA forecasting models, the highest forecast accuracy has been performed by LSTM with RMSE, MAPE, MAE, and R2 are 33.01 W/m2, 7.72%, 24.51 W/m2 and 0.96 respectively. Subsequently, partially cloudy, cloudy, or rainy weather has a worse predicting performance than weather with a sunny or clear sky. However, they are still regarded as reasonable forecasts.

Rumor Detection With Hierarchical Representation on Bipartite Ad Hoc Event Trees.

The rapid growth of social media has caused tremendous effects on information propagation, raising extreme challenges in detecting rumors. Existing rumor detection methods typically exploit the reposting propagation of a rumor candidate for detection by regarding all reposts to a rumor candidate as a temporal sequence and learning semantics representations of the repost sequence. However, extracting informative support from the topological structure of propagation and the influence of reposting authors for debunking rumors is crucial, which generally has not been well addressed by existing methods. In this article, we organize a claim post in circulation as an ad hoc event tree, extract event elements, and convert it into bipartite ad hoc event trees in terms of both posts and authors, i.e., author tree and post tree. Accordingly, we propose a novel rumor detection model with hierarchical representation on the bipartite ad hoc event trees called BAET. Specifically, we introduce word embedding and feature encoder for the author and post tree, respectively, and design a root-aware attention module to perform node representation. Then we adopt the tree-like RNN model to capture the structural correlations and propose a tree-aware attention module to learn tree representation for the author tree and post tree, respectively. Extensive experimental results on two public Twitter datasets demonstrate the effectiveness of BAET in exploring and exploiting the rumor propagation structure and the superior detection performance of BAET over state-of-the-art baseline methods.

Short term road network Macroscopic Fundamental Diagram parameters and traffic state prediction based on LSTM

Macroscopic Fundamental Diagram (MFD) is widely used in traffic state evaluation due to its description of the macro level of urban road network. This study focuses on the discrimination and short-term prediction of macro traffic states in urban road networks, using MFD combined with FCM clustering for state partitioning to characterize different macro states of the road network. To predict the MFD state, this paper builds two LSTMs to perform short-term predictions on two important parameters in MFD: road network weighted flow qw and road network weighted density kw. The parameters of the first three statistical intervals and the predicted time period are used as inputs to output the MFD parameters for the predicted time period. To ensure that the two LSTM structures and hyper-parameters settings can achieve the best prediction performance for MFD, the parameter optimization process of both should be included in the same search framework for hyper-parameters search. Therefore, this paper uses GA algorithm combined with multi-objective particle swarm optimization algorithm as the solving algorithm, with the accuracy of solving two MFD parameters and the accuracy of MFD point positioning as the hyper-parameters solving objectives. The study was validated using actual road network data from Hong Kong, and the results showed that the method proposed in this paper has an MRE prediction error of less than 7.8% for the two parameters of MFD, and can predict the future temporal trend of the two parameters, demonstrating the feasibility of MFD related predictions. The model's prediction of the overall shape and change trajectory trend of MFD is consistent with reality, and some test sets in MFD state prediction show high accuracy, the overall accuracy is 81.45%. To verify the effectiveness of the multi-objective search algorithm, typical LSTM models and RNN models were used for comparison. The experiment proved that the model used in this study performed better in error control and state prediction. This study explores and practices a short-term prediction method for road network MFD parameters, MFD status, and their changing trends. Provided path reference for urban road network prediction, traffic control status, and MFD related research.

A Performance Analysis of RNN Algorithms for Accurate Hypoglycemia Prediction in Type-1 Diabetes

Diabetes is a chronic disease originating from preeminent blood glucose levels where there is an impairment in the ability of the human body to produce or effectively use insulin, therefore resulting in possible complications to various organ systems. Hypoglycemia can be defined as a critical condition in diabetes that is characterized by very low levels of blood sugar and can emanate from any imbalance between insulin, glucose, and external factors such as medication or physical activity. The prediction of hypoglycemia is a very important task in diabetes management, which encompasses sophisticated technologies and deep-learning systems, such as RNNs, in performing analyses on patient-specific data. These must provide timely warnings to avoid hazardous blood sugar dips. This paper takes a close look at the performance of RNN in the case of accurate hypoglycemia prediction among patients with Type-1 Diabetes, considering a dataset from Shanghai T1DM. In this work, three different RNN architectures are considered for performance evaluation: Long-Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Simple RNN. The main goal was to compare their predictive performances in forecasting hypoglycemic events, which is an issue of utmost relevance when it comes to proactive diabetes management. The results show generally variable performances by the RNN models. Overall, GRU performed with striking accuracy in hypoglycemia predictions, while LSTM had high specificity. These findings underline that various metrics should be considered for the comprehensive evaluation of predictive models in the management of diabetes. It will give an idea about various RNN algorithms, strengths, and weaknesses to develop more effective and personalized strategies related to hypoglycemia prediction in Type-1 Diabetes.

Exploring Deep Learning Models for Lyric Generation and Addressing Biases in Word Embeddings

The aim of this project is to explore the performance of different model architectures (such as RNN, LSTM, GRU) by generating lyrics using deep learning models, and to use the Word2Vec model for distributed semantic analysis to understand semantic phenomena and potential biases in word embedding models. The experimental results show that LSTM and GRU perform better than traditional RNN models when processing long sequence data. In addition, by analyzing word embeddings, we revealed potential gender and racial biases and proposed corresponding solutions.

Short-term building cooling load prediction based on AKNN and RNN models

Cooling load is the main energy consumption factor for commercial buildings during summer. This paper proposes a new approach to realize short-term building cooling load prediction by combining adaptive feature extraction and deep learning models. An autonomous kernel-based neural network (AKNN) is proposed to extract intrinsic patterns from sensor data by clustering kernels in a bottom-up way. Furthermore, a recurrent neural network-based sequence-to-sequence structure is proposed to make multi-step-ahead forecasting. Tested on an industrial park, the results indicate that the proposed model achieves the lowest RMSE of 1.79 MW for 1-hour-ahead prediction. For 24-hour-ahead predictions, the proposed model maintains superior accuracy with an RMSE of 2.85 MW, while ARIMA has an RMSE of 7.27 MW. Additionally, using a public dataset, the proposed model shows an 18.02% lower RMSE than ARIMA for 24-hour prediction, verifying its accurate and stable performance.

Seismic response prediction method of train-bridge coupled system based on convolutional neural network-bidirectional long short-term memory-attention modeling

Seismic response prediction is crucial for the safety analysis of train-bridge coupled systems. However, due to the complexity, suddenness, and high-risk nature of earthquakes, there are strong nonlinear relationships among different parts of bridges, making it challenging to express their spatial correlations using analytical models and traditional neural networks. To address this, this paper establishes a ballast track shaker scaling model and employs the grating monitoring measurement method to construct a spatial quasi-distributed monitoring system for the ballast track, thereby collecting seismic strain responses of the train-bridge coupled system under various seismic conditions. A hybrid neural network method is proposed for predicting the seismic responses of the train-bridge coupled system. This hybrid neural network integrates the features of a Convolutional Neural Network (CNN), Bidirectional Long Short-Term Memory Neural Network (BiLSTM), and the attention mechanism, thereby termed the CNN-BiLSTM-attention hybrid neural network. The model was validated using strain responses from 54 seismic scenarios. The results indicate that the model has a Mean Absolute Error (MAE) of 0.2349 and a coefficient of determination (R2) of 0.9446. Comparing the prediction results with those from RNN and LSTM models, it was found that the CNN effectively extracts features under various seismic parameters, while the BiLSTM better captures the temporal information of the strain responses, ensuring effective prediction regardless of the magnitude of strain responses. Therefore, the CNN-BiLSTM-attention hybrid neural network model is recommended for predicting seismic response.

IEV2Mol: Molecular Generative Model Considering Protein-Ligand Interaction Energy Vectors.

Generating drug candidates with desired protein-ligand interactions is a significant challenge in structure-based drug design. In this study, a new generative model, IEV2Mol, is proposed that incorporates interaction energy vectors (IEVs) between proteins and ligands obtained from docking simulations, which quantitatively capture the strength of each interaction type, such as hydrogen bonds, electrostatic interactions, and van der Waals forces. By integrating this IEV into an end-to-end variational autoencoder (VAE) framework that learns the chemical space from SMILES and minimizes the reconstruction error of the SMILES, the model can more accurately generate compounds with the desired interactions. To evaluate the effectiveness of IEV2Mol, we performed benchmark comparisons with randomly selected compounds, unconstrained VAE models (JT-VAE), and compounds generated by RNN models based on interaction fingerprints (IFP-RNN). The results show that the compounds generated by IEV2Mol retain a significantly greater percentage of the binding mode of the query structure than those of the other methods. Furthermore, IEV2Mol was able to generate compounds with interactions similar to those of the input compounds, regardless of structural similarity. The source code and trained models for IEV2Mol, JT-VAE, and IFP-RNN designed for generating compounds active against the DRD2, AA2AR, and AKT1, as well as the data sets (DM-QP-1M, active compounds to each protein, and ChEMBL33) utilized in this study, are released under the MIT License and available at https://github.com/sekijima-lab/IEV2Mol.

An End-to-End Scene Text Recognition for Bilingual Text

Text localization and recognition from natural scene images has gained a lot of attention recently due to its crucial role in various applications, such as autonomous driving and intelligent navigation. However, two significant gaps exist in this area: (1) prior research has primarily focused on recognizing English text, whereas Arabic text has been underrepresented, and (2) most prior research has adopted separate approaches for scene text localization and recognition, as opposed to one integrated framework. To address these gaps, we propose a novel bilingual end-to-end approach that localizes and recognizes both Arabic and English text within a single natural scene image. Specifically, our approach utilizes pre-trained CNN models (ResNet and EfficientNetV2) with kernel representation for localization text and RNN models (LSTM and BiLSTM) with an attention mechanism for text recognition. In addition, the AraElectra Arabic language model was incorporated to enhance Arabic text recognition. Experimental results on the EvArest, ICDAR2017, and ICDAR2019 datasets demonstrated that our model not only achieves superior performance in recognizing horizontally oriented text but also in recognizing multi-oriented and curved Arabic and English text in natural scene images.