RNN-based Model Research Articles

Using DL Models in the Service Layer to Enhance the Fault Tolerance of IoT Networks

In an IoT network, the networked servers form a service layer, providing services to the users and the devices. The request to the service servers is routed through the gateway on one side of the services layer and the networked controllers on the other side. Data are transported from the sensors/devices through cluster heads en route to base stations and the controllers to the service servers, where the data are processed and sent for storage in the cloud through gateways. When any device is broken down or becomes non-operational, the inputs are not sensed, creating a gap in the data. The data transmitted from the devices would then become an incomplete flow; such data are not suitable for undertaking data analytics or predictions. The missing data must be first identified as the data flow and estimated or predicated to complete the data before they are transmitted through the cloud for storage and subsequent retrievals. This paper proposes a recurrent (RNN) neural network to predict the missing data. Two models are tested to predict the missing data: the multi-layer perceptron (MLP) model and a long short-term memory (LSTM)-based RNN model. The RNN-based model provides 99.66% accurate data prediction compared to other models.

FlexSleepTransformer: a transformer-based sleep staging model with flexible input channel configurations

Clinical sleep diagnosis traditionally relies on polysomnography (PSG) and expert manual classification of sleep stages. Recent advancements in deep learning have shown promise in automating sleep stage classification using a single PSG channel. However, variations in PSG acquisition devices and environments mean that the number of PSG channels can differ across sleep centers. To integrate a sleep staging method into clinical practice effectively, it must accommodate a flexible number of PSG channels. In this paper, we proposed FlexSleepTransformer, a transformer-based model designed to handle varying number of input channels, making it adaptable to diverse sleep staging datasets. We evaluated FlexSleepTransformer using two distinct datasets: the public SleepEDF-78 dataset and the local SleepUHS dataset. Notably, FlexSleepTransformer is the first model capable of simultaneously training on datasets with differing number of PSG channels. Our experiments showed that FlexSleepTransformer trained on both datasets together achieved 98% of the accuracy compared to models trained on each dataset individually. Furthermore, it outperformed models trained exclusively on one dataset when tested on the other dataset. Additionally, FlexSleepTransformer surpassed state-of-the-art CNN and RNN-based models on both datasets. Due to its adaptability with varying channels numbers, FlexSleepTransformer holds significant potential for clinical adoption, especially when trained with data from a wide range of sleep centers.

A Stock Prediction Method Based on Multidimensional and Multilevel Feature Dynamic Fusion

Stock price prediction has long been a topic of interest in academia and the financial industry. Numerous factors influence stock prices, such as a company’s performance, industry development, national policies, and other macroeconomic factors. These factors are challenging to quantify, making predicting stock price movements difficult. This paper presents a novel deep neural network framework that leverages the dynamic fusion of multi-dimensional and multi-level features for stock price prediction, which means we utilize fundamental trading data and technical indicators as multi-dimensional data and local and global multi-level information. Firstly, the model dynamically assigns weights to multi-dimensional features of stocks to capture the impact of each feature on stock prices. Next, it applies the Fourier transform to the global features to capture the long-term trends of the global environment and dynamically fuses these with local and global features of the stocks to capture the overall market environment’s impact on individual stocks. Finally, temporal features are captured using an attention layer and an RNN-based model, which incorporates historical price data to forecast future prices. Experiments on stocks from various industries within the Chinese CSI 300 index reveal that the proposed model outperforms traditional methods and other deep learning approaches in terms of stock price prediction. This paper proposes a method that facilitates the dynamic integration of multi-dimensional and multi-level features in an efficient manner and experimental results show that it improves the accuracy of stock price predictions.

A Hybrid RNN-based Deep Learning Model for Lung Cancer and COPD Detection

A Hybrid RNN-based Deep Learning Model for Lung Cancer and COPD Detection

Reinforcement Learning for Improving Chemical Reaction Performance.

Deep learning (DL) methods have gained notable prominence in predictive and generative tasks in molecular space. However, their application in chemical reactions remains grossly underutilized. Chemical reactions are intrinsically complex: typically involving multiple molecules besides bond-breaking/forming events. In reaction discovery, one aims to maximize yield and/or selectivity that depends on a number of factors, mostly centered on reacting partners and reaction conditions. Herein, we introduce RE-EXPLORE, a novel approach that integrates deep reinforcement learning (RL) with an RNN-based deep generative model to identify prospective new reactants/catalysts, whose yield/selectivity is estimated using a pretrained regressor. Three chemical databases (ChEMBL, ZINC, and COCONUT containing half a million to one million unlabeled molecules) are independently used for pretraining the generators to enrich them with valuable information from diverse chemical space. Standard RL methods are found to be insufficient, as learners tend to prioritize exploitation for immediate gains, resulting in repetitive generation of same/similar molecules. Our engineered reward function includes a Tanimoto-based uniqueness factor within the RL loop that improved the exploration of the environment and has helped accrue larger returns. Integration of a user-defined core fragment into the generated molecules facilitated learning of specific reaction types. Together, RE-EXPLORE can navigate the reaction space toward practically meaningful regions and offers notable improvements across the three distinct reaction types considered in this study. It identifies high-yielding substrates and highly enantioselective chiral catalysts. This RL-based approach has the potential to expedite reaction discovery and aid in the synthesis planning of important compounds, including drugs and pharmaceuticals.

An open innovative inventory management based demand forecasting approach for the steel industry

This research focuses on accurately forecasting steel wire mesh demand to ensure timely order fulfillment. Various univariate time series forecasting methods were employed, including Prophet, Support Vector Regression (SVR), 1-Dimensional Convolutional Neural Network (1D-CNN), Recurrent Neural Network (RNN), Gated Recurrent Unit (GRU), and Long Short-Term Memory (LSTM). These RNN-based models were implemented to enhance the dynamics of open innovation. The study also employed the split test method to assess the impact of data size on model performance. Following a comprehensive comparative analysis, LSTM outperformed other models at 80:20 data split ratio, achieving the lowest RMSE of 51,473 and MAPE of 1.43. The robust performance of LSTM underscores its ability to expertly predict complex demand patterns. Moreover, forecast accuracy was enhanced using ensemble models, with RNN-LSTM delivering the best results (RMSE of 45,931 and MAPE of 1.20) at the same data split ratio. This accurate forecasting supports steel wire mesh companies in making informed decisions regarding inventory management. The study additionally demonstrated the integration of forecasting results into inventory strategies to improve decision-making for cost optimization.

Predicting temporal evacuation travel time in staircases between adjacent floors of super high-rise buildings by artificial neural networks

Predicting temporal evacuation travel time in staircases between adjacent floors is crucial for dynamic evacuation guidance to improve vertical evacuation efficiency during emergencies in super high-rise buildings. However, the prediction methods used in previous studies mainly focus on the total evacuation time in buildings and cannot predict the temporal evacuation travel time in every single part of the total evacuation route. This study proposes a novel method to predict temporal evacuation travel time in staircases between adjacent floors of super high-rise buildings by using artificial neural networks. Firstly, evacuation simulation data of a 71-storey office building are analyzed for feature engineering and dataset building. Secondly, three types of artificial neural networks, including convolutional neural networks, RNN-based model (the traditional recurrent neural networks, long-short term memory networks, and bidirectional long-short term memory networks), and Attention-based model (Attention model and Transformer model) are extended to predict the temporal evacuation travel time in staircases between adjacent floors based on the simulation data. Finally, the prediction results by different artificial neural networks are compared. The results show that the artificial neural networks can provide accurate prediction values for the temporal evacuation travel time in staircases between adjacent floors of super high-rise buildings.

Improved Urdu-English Neural Machine Translation with a fully Convolutional Neural Network Encoder

Neural machine translation (NMT) approaches driven by artificial intelligence (AI) has gained more and more attention in recent years, mainly due to their simplicity yet state-of-the-art performance. Despite NMT models with attention mechanism relying heavily on the accessibility of substantial parallel corpora, they have demonstrated efficacy even for languages with limited linguistic resources. The convolutional neural network (CNN) is frequently employed in tasks involving visual and speech recognition. Implementing CNN for MT is still challenging compared to the predominant approaches. Recent research has shown that the CNN-based NMT model cannot capture long-term dependencies present in the source sentence. The CNN-based model can only capture the word dependencies within the width of its filters. This unnatural character often causes a worse performance for CNN-based NMT than the RNN-based NMT models. This study introduces a simple method to improve neural translation of a low-resource language, specifically Urdu-English (UR-EN). In this paper, we use a Fully Convolutional Neural Network (FConv-NN) based NMT architecture to create a powerful MT encoder for UR-EN translation that can capture the long dependency of words in a sentence. Although the model is quite simple, it yields strong empirical results. Experimental results show that the FConv-NN model consistently outperforms the traditional CNN-based model with filters. On the Urdu-English Dataset, the FConv-NN model produces translation with a gain of 18.42 BLEU points. Moreover, the quantitative and comparative analysis shows that in a low-resource setting, FConv-NN-based NMT outperforms conventional CNN-based NMT models.

Sign Language Interpreting System Using Recursive Neural Networks

According to the World Health Organization (WHO), 5% of people around the world have hearing disabilities, which limits their capacity to communicate with others. Recently, scientists have proposed systems based on deep learning techniques to create a sign language-to-text translator, expecting this to help deaf people communicate; however, the performance of such systems is still low for practical scenarios. Furthermore, the proposed systems are language-oriented, which leads to particular problems related to the signs for each language. For this reason, to address this problem, in this paper, we propose a system based on a Recursive Neural Network (RNN) focused on Mexican Sign Language (MSL) that uses the spatial tracking of hands and facial expressions to predict the word that a person intends to communicate. To achieve this, we trained four RNN-based models using a dataset of 600 clips that were 30 s long; each word included 30 clips. We conducted two experiments; we tailored the first experiment to determine the most well-suited model for the target application and measure the accuracy of the resulting system in offline mode; in the second experiment, we measured the accuracy of the system in online mode. We assessed the system’s performance using the following metrics: the precision, recall, F1-score, and the number of errors during online scenarios, and the results computed indicate an accuracy of 0.93 in the offline mode and a higher performance for the online operating mode compared to previously proposed approaches. These results underscore the potential of the proposed scheme in scenarios such as teaching, learning, commercial transactions, and daily communications among deaf and non-deaf people.

Annealing Temporal–Spatial Contrastive Learning for multi-view Online Action Detection

In open-world scenarios, the analysis of action events from multiple viewpoints is crucial for achieving a holistic understanding, a concept that resonates with human perception. However, achieving a synthesis of information from multiple viewpoints presents a challenge, as the data can induce inter-view regularization, complicating the learning process. This paper is the first to delve into Online Action Detection (OAD) through a multi-view lens, underscoring the value of cross-observation in enriching view-level information. By harnessing the spatiotemporal dynamics inherent in multi-view video sequences, an Annealing Temporal–Spatial Contrastive Learning (ATSCL) consisting of Annealing Temporal Contrastive Learning (ATCL) and Spatial Contrastive Learning (SCL) is proposed, optimized for compatibility with RNN-based models. ATCL employs an annealing temporal loss to uncover the intrinsic video structures via a temporal annealing sampling mechanism. Concurrently, SCL utilizes a spatial loss to draw representations from various viewpoints closer together, mitigating the regularization effects. The ATSCL liberates training multi-view OAD from the stringent requirements of synchronized training videos, enabling the execution of OAD tasks asynchronously. Experiments demonstrate that the RNN-based models realize an average improvement of 5.92% on the DAHLIA dataset, 3.36% on the IKEA ASM dataset, 2.92% on the BREAKFAST dataset and 0.6% (mAP) on the THUMOS’14 dataset following the integration of the ATSCL framework, underscoring the ATSCL’s efficacy across different RNN structures.

A comparative analysis of single learning and ensemble learning approaches for the forecasting of electricity prices

Forecasting electricity prices accurately is imperative in deregulated power markets. Nonetheless, the intricate nature of electricity prices, characterized by high frequency and volatility, poses a challenge in building an effective forecasting model for policymakers and scientists. Precision in the electricity price prediction is crucial for providing valuable guidance to market participants, helping them maximize their benefits. In prior studies, various methods, including statistical models and artificial neural network models, have been used to forecast electricity prices. This study proposes three ensemble learning approaches - AdaBoost-LSTM, AdaBoost-BLSTM, and AdaBoost-GRU - which combine the AdaBoost algorithm with LSTM, Bi-LSTM, and GRU networks, respectively, to enhance the accuracy of electricity price predictions. The study aims to assess the effectiveness of the proposed models by comparing their predictive performance with that of single RNN-based models (LSTM, Bi-LSTM, GRU) using daily maximum electricity prices from 2004 to 2008. Notably, there has been no existing research that compares the effectiveness of these single and hybrid models. In the current literature, these single models are widely acknowledged as potent tools for improving forecasting accuracy. On the other hand, although the proposed ensemble learning approaches obtained using the AdaBoost boosting technique have been used in areas such as financial forecasting so far, they have never been used in electricity price forecasting. Accuracy assessment utilizing R-squared and MAPE clearly demonstrates that the AdaBoost-BLSTM approach performs very closely to, but better than, other boosting ensemble approaches, and significantly better than single models.

A Robust Recurrent Neural Networks-Based Surrogate Model for Thermal History and Melt Pool Characteristics in Directed Energy Deposition.

In directed energy deposition (DED), accurately controlling and predicting melt pool characteristics is essential for ensuring desired material qualities and geometric accuracies. This paper introduces a robust surrogate model based on recurrent neural network (RNN) architectures-Long Short-Term Memory (LSTM), Bidirectional LSTM (Bi-LSTM), and Gated Recurrent Unit (GRU). Leveraging a time series dataset from multi-physics simulations and a three-factor, three-level experimental design, the model accurately predicts melt pool peak temperatures, lengths, widths, and depths under varying conditions. RNN algorithms, particularly Bi-LSTM, demonstrate high predictive accuracy, with an R-square of 0.983 for melt pool peak temperatures. For melt pool geometry, the GRU-based model excels, achieving R-square values above 0.88 and reducing computation time by at least 29%, showcasing its accuracy and efficiency. The RNN-based surrogate model built in this research enhances understanding of melt pool dynamics and supports precise DED system setups.

Optimizing Recurrent Neural Networks: A Study on Gradient Normalization of Weights for Enhanced Training Efficiency

Recurrent Neural Networks (RNNs) are classical models for processing sequential data, demonstrating excellent performance in tasks such as natural language processing and time series prediction. However, during the training of RNNs, the issues of vanishing and exploding gradients often arise, significantly impacting the model’s performance and efficiency. In this paper, we investigate why RNNs are more prone to gradient problems compared to other common sequential networks. To address this issue and enhance network performance, we propose a method for gradient normalization of network weights. This method suppresses the occurrence of gradient problems by altering the statistical properties of RNN weights, thereby improving training effectiveness. Additionally, we analyze the impact of weight gradient normalization on the probability-distribution characteristics of model weights and validate the sensitivity of this method to hyperparameters such as learning rate. The experimental results demonstrate that gradient normalization enhances the stability of model training and reduces the frequency of gradient issues. On the Penn Treebank dataset, this method achieves a perplexity level of 110.89, representing an 11.48% improvement over conventional gradient descent methods. For prediction lengths of 24 and 96 on the ETTm1 dataset, Mean Absolute Error (MAE) values of 0.778 and 0.592 are attained, respectively, resulting in 3.00% and 6.77% improvement over conventional gradient descent methods. Moreover, selected subsets of the UCR dataset show an increase in accuracy ranging from 0.4% to 6.0%. The gradient normalization method enhances the ability of RNNs to learn from sequential and causal data, thereby holding significant implications for optimizing the training effectiveness of RNN-based models.

Research on maintenance cycle prediction for energy equipment with limited and sensitive data

Failures in natural gas compressor units can lead to significant production incidents that impact energy security, making intelligent preventative maintenance of this equipment critically important. Given the limited and sensitive data from critical energy equipment like natural gas compressor units, this paper conducts preventative maintenance research based on domain adaptation and federated learning (DFPM), and proposes a novel maintenance cycle prediction framework. The feasibility of the proposed method was validated through historical operational data from field natural gas compressor units, providing a data-oriented reference for maintenance decision-making. Verification results demonstrate that the proposed multi-level adversarial domain adaptation method possesses superior feature representation capability, enhancing the preservation of the original degradation trend for time series data compared to traditional DA models. The novel federated adversarial strategy presented in this paper can effectively transfer source domain knowledge to the target domain, further enhancing the cross-unit predictive performance of the target domain model with limited samples. Additionally, this strategy with the chaotic multi-mode predictive model further improves the limitations of low parallelism and efficiency in traditional RNN-based prediction models while ensuring data security. This research provides a new perspective for the intelligent operation and maintenance of large energy equipment, which is of significance in practical engineering.

Comprehensive Neural Cryptanalysis on Block Ciphers Using Different Encryption Methods

In this paper, we perform neural cryptanalysis on five block ciphers: Data Encryption Standard (DES), Simplified DES (SDES), Advanced Encryption Standard (AES), Simplified AES (SAES), and SPECK. The block ciphers are investigated on three different deep learning-based attacks, Encryption Emulation (EE), Plaintext Recovery (PR), Key Recovery (KR), and Ciphertext Classification (CC) attacks. The attacks attempt to break the block ciphers in various cases, such as different types of plaintexts (i.e., block-sized bit arrays and texts), different numbers of round functions and quantity of training data, different text encryption methods (i.e., Word-based Text Encryption (WTE) and Sentence-based Text Encryption (STE)), and different deep learning model architectures. As a result, the block ciphers can be vulnerable to EE and PR attacks using a large amount of training data, and STE can improve the strength of the block ciphers, unlike WTE, which shows almost the same classification accuracy as the plaintexts, especially in a CC attack. Moreover, especially in the KR attack, the Recurrent Neural Network (RNN)-based deep learning model shows higher average Bit Accuracy Probability than the fully connected-based deep learning model. Furthermore, the RNN-based deep learning model is more suitable than the transformer-based deep learning model in the CC attack. Besides, when the keys are the same as the plaintexts, the KR attack can perfectly break the block ciphers, even if the plaintexts are randomly generated. Additionally, we identify that DES and SPECK32/64 applying two round functions are more vulnerable than those applying the single round function by performing the KR attack with randomly generated keys and randomly generated single plaintext.

Chatbot Architecture based on NLP Techniques

Chatbots have emerged as a prominent technological innovation with a profound impact on various aspects of human life, including customer service, communication, and automation. This abstract provides an overview of the evolution, functionalities, and implications of chatbots in contemporary The potential of chatbots goes beyond commercial applications. They are being used in education to provide personalized learning experiences and in mental health support, offering immediate responses to those in need. In the workplace, chatbots can streamline administrative tasks, manage appointments, and assist with onboarding processes. Their adaptability and scalability have made them a valuable asset in various industries. The continued development of chatbot technology and a thoughtful approach to their use are essential for maximizing their benefits while mitigating potential drawbacks in the rapidly changing landscape of artificial intelligence and human-computer interaction. Key Words: Chatbot, NLP ,AI, Seq2Seq architecture, RNN-based models

Automated detection of steps in videos of strabismus surgery using deep learning

BackgroundLearning to perform strabismus surgery is an essential aspect of ophthalmologists’ surgical training. Automated classification strategy for surgical steps can improve the effectiveness of training curricula and the efficient evaluation of residents’ performance. To this end, we aimed to develop and validate a deep learning (DL) model for automated detecting strabismus surgery steps in the videos.MethodsIn this study, we gathered 479 strabismus surgery videos from Shanghai Children’s Hospital, affiliated to Shanghai Jiao Tong University School of Medicine, spanning July 2017 to October 2021. The videos were manually cut into 3345 clips of the eight strabismus surgical steps based on the International Council of Ophthalmology’s Ophthalmology Surgical Competency Assessment Rubrics (ICO-OSCAR: strabismus). The videos dataset was randomly split by eye-level into a training (60%), validation (20%) and testing dataset (20%). We evaluated two hybrid DL algorithms: a Recurrent Neural Network (RNN) based and a Transformer-based model. The evaluation metrics included: accuracy, area under the receiver operating characteristic curve, precision, recall and F1-score.ResultsDL models identified the steps in video clips of strabismus surgery achieved macro-average AUC of 1.00 (95% CI 1.00–1.00) with Transformer-based model and 0.98 (95% CI 0.97-1.00) with RNN-based model, respectively. The Transformer-based model yielded a higher accuracy compared with RNN-based models (0.96 vs. 0.83, p < 0.001). In detecting different steps of strabismus surgery, the predictive ability of the Transformer-based model was better than that of the RNN. Precision ranged between 0.90 and 1 for the Transformer-based model and 0.75 to 0.94 for the RNN-based model. The f1-score ranged between 0.93 and 1 for the Transformer-based model and 0.78 to 0.92 for the RNN-based model.ConclusionThe DL models can automate identify video steps of strabismus surgery with high accuracy and Transformer-based algorithms show excellent performance when modeling spatiotemporal features of video frames.

Graphical User Interface for Handwritten Mathematical Expression Employing RNN-based Encoder-Decoder Model

Graphical User Interface for Handwritten Mathematical Expression Employing RNN-based Encoder-Decoder Model

Enhanced earthquake impact analysis based on social media texts via large language model

Social media aids disaster response but suffers from noise, hindering accurate impact assessment and decision making for resilient cities, which few studies considered. To address the problem, this study proposes the first domain-specific LLM model and an integrated method for rapid earthquake impact assessment. First, a few categories are introduced to classify and filter microblogs considering their relationship to the physical and social impacts of earthquakes, and a dataset comprising 7282 earthquake-related microblogs from twenty earthquakes in different locations is developed as well. Then, with a systematic analysis of various influential factors, QuakeBERT, a domain-specific large language model (LLM), is developed and fine-tuned for accurate classification and filtering of microblogs. Meanwhile, an integrated method integrating public opinion trend analysis, sentiment analysis, and keyword-based physical impact quantification is introduced to assess both the physical and social impacts of earthquakes based on social media texts. Experiments show that data diversity and data volume dominate the performance of QuakeBERT and increase the macro average F1 score by 27 %, while the best classification model QuakeBERT outperforms the CNN- or RNN-based models by improving the macro average F1 score from 60.87 % to 84.33 %. Finally, the proposed approach is applied to assess two earthquakes with the same magnitude and focal depth. Results show that the proposed approach can effectively enhance the impact assessment process by accurate detection of noisy microblogs, which enables effective post-disaster emergency responses to create more resilient cities.

Heteroscedasticity effects as component to future stock market predictions using RNN-based models.

Heteroscedasticity effects are useful for forecasting future stock return volatility. Stock volatility forecasting provides business insight into the stock market, making it valuable information for investors and traders. Predicting stock volatility is a crucial task and challenging. This study proposes a hybrid model that predicts future stock volatility values by considering the heteroscedasticity element of the stock price. The proposed model is a combination of Generalized Autoregressive Conditional Heteroskedasticity (GARCH) and a well-known Recurrent Neural Network (RNN) algorithm Long Short-Term Memory (LSTM). This proposed model is referred to as GARCH-LSTM model. The proposed model is expected to improve prediction accuracy by considering heteroscedasticity elements. First, the GARCH model is employed to estimate the model parameters. After that, the ARCH effect test is used to test the residuals obtained from the model. Any untrained heteroscedasticity element must be found using this step. The hypothesis of the ARCH test yielded a p-value less than 0.05 indicating there is valuable information remaining in the residual, known as heteroscedasticity element. Next, the dataset with heteroscedasticity is then modelled using an LSTM-based RNN algorithm. Experimental results revealed that hybrid GARCH-LSTM had the lowest MAE (7.961), RMSE (10.466), MAPE (0.516) and HMAE (0.005) values compared with a single LSTM. The accuracy of forecasting was also significantly improved by 15% and 13% with hybrid GARCH-LSTM in comparison to single LSTMs. Furthermore, the results reveal that hybrid GARCH-LSTM fully exploits the heteroscedasticity element, which is not captured by the GARCH model estimation, outperforming GARCH models on their own. This finding from this study confirmed that hybrid GARCH-LSTM models are effective forecasting tools for predicting stock price movements. In addition, the proposed model can assist investors in making informed decisions regarding stock prices since it is capable of closely predicting and imitating the observed pattern and trend of KLSE stock prices.