Long Short-term Memory Recurrent Neural Network Research Articles

A study on the evaluation of tokenizer performance in natural language processing

ABSTRACT The present study aims to compare and analyze the performance of two tokenizers, Mecab-Ko and SentencePiece, in the context of natural language processing for sentiment analysis. The study adopts a comparative approach, employing five algorithms - Naive Bayes (NB), k-Nearest Neighbor (kNN), Support Vector Machine (SVM), Artificial Neural Networks (ANN), and Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) - to evaluate the performance of each tokenizer. The performance was assessed based on four widely used metrics in the field, accuracy, precision, recall, and F1-score. The results indicated that SentencePiece performed better than Mecab-Ko. To ensure the validity of the results, paired t-tests were conducted on the evaluation outcomes. The study concludes that SentencePiece demonstrated superior classification performance, especially in the context of ANN and LSTM-RNN, when used to interpret customer sentiment based on Korean online reviews. Furthermore, SentencePiece can assign specific meanings to short words or jargon commonly used in product evaluations but not defined beforehand.

Time series radiomics for the prediction of prostate cancer progression in patients on active surveillance

Serial MRI is an essential assessment tool in prostate cancer (PCa) patients enrolled on active surveillance (AS). However, it has only moderate sensitivity for predicting histopathological tumour progression at follow-up, which is in part due to the subjective nature of its clinical reporting and variation among centres and readers. In this study, we used a long short-term memory (LSTM) recurrent neural network (RNN) to develop a time series radiomics (TSR) predictive model that analysed longitudinal changes in tumour-derived radiomic features across 297 scans from 76 AS patients, 28 with histopathological PCa progression and 48 with stable disease. Using leave-one-out cross-validation (LOOCV), we found that an LSTM-based model combining TSR and serial PSA density (AUC 0.86 [95% CI: 0.78–0.94]) significantly outperformed a model combining conventional delta-radiomics and delta-PSA density (0.75 [0.64–0.87]; p = 0.048) and achieved comparable performance to expert-performed serial MRI analysis using the Prostate Cancer Radiologic Estimation of Change in Sequential Evaluation (PRECISE) scoring system (0.84 [0.76–0.93]; p = 0.710). The proposed TSR framework, therefore, offers a feasible quantitative tool for standardising serial MRI assessment in PCa AS. It also presents a novel methodological approach to serial image analysis that can be used to support clinical decision-making in multiple scenarios, from continuous disease monitoring to treatment response evaluation.Key Points•LSTM RNN can be used to predict the outcome of PCa AS using time series changes in tumour-derived radiomic features and PSA density.•Using all available TSR features and serial PSA density yields a significantly better predictive performance compared to using just two time points within the delta-radiomics framework.•The concept of TSR can be applied to other clinical scenarios involving serial imaging, setting out a new field in AI-driven radiology research.

RETRACTED: A BERT encoding with Recurrent Neural Network and Long-Short Term Memory for breast cancer image classification

Computer-aided diagnostic systems developed with Artificial Intelligence (AI) are a major advancement in healthcare analytics, assisting radiologists with a second opinion on cancer diagnosis. In this study, we explore two modalities of breast images, ultrasound, and histology, for their classification into cancerous and non-cancerous categories. Traditionally Convolution Neural Networks (CNN) have done a commendable job of extracting features using convolution kernels from images with great accuracy. Also, the Bidirectional Encoder Representations from Transformers (BERT) has been widely used in Natural Language Processing (NLP) for feature encoding and downstream tasks like segmentation and classification. We extract the power of Vision Transformers (ViT) and implement transfer learning using BERT pre-training of Image Transformers (BEiT) as a feature encoding technique. We use the encoded features for classification with Recurrent Neural Network – Long short Term Memory (RNN-LSTM). The classification is performed on two modalities of breast image datasets: BUSI1311 and breast histopathological images. Both modalities yielded competitive accuracies. The BUSI1311 dataset produced 99 percent accuracy compared to 91 percent accuracy for breast histology images.

Online State of Health Estimation of Lithium-Ion Batteries Based on Charging Process and Long Short-Term Memory Recurrent Neural Network

Accurate state of health (SOH) estimation is critical to the operation, maintenance, and replacement of lithium-ion batteries (LIBs), which have penetrated almost every aspect of our life. This paper introduces a new approach to accurately estimate the SOH for rechargeable lithium-ion batteries based on the corresponding charging process and long short-term memory recurrent neural network (LSTM-RNN). In order to learn the mapping function without employing battery models and filtering techniques, the LSTM-RNN is initially fed into the health indicators (HIs) extracted from the charging process and trained to encode the dependencies of the related data sequence. Subsequently, the trained LSTM-RNN can properly estimate online SOHs of LIBs using extracted HIs. We experiment on two public datasets for model construction, validation, and comparison. Conclusively, the trained LSTM-RNN achieves an overall root mean square error (RMSE) lower than 1% on the cases with the same discharging current rate and an RMSE of 1.1198% above 80% SOH on another testing case that underwent a different discharging current rate.

A Soft Sensor to Estimate the Opening of Greenhouse Vents Based on an LSTM-RNN Neural Network.

In greenhouses, sensors are needed to measure the variables of interest. They help farmers and allow automatic controllers to determine control actions to regulate the environmental conditions that favor crop growth. This paper focuses on the problem of the lack of monitoring and control systems in traditional Mediterranean greenhouses. In such greenhouses, most farmers manually operate the opening of the vents to regulate the temperature during the daytime. Therefore, the state of vent opening is not recorded because control systems are not usually installed due to economic reasons. The solution presented in this paper consists of developing a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) as a soft sensor to estimate vent opening using the measurements of different inside and outside greenhouse climate variables as input data. A dataset from a traditional greenhouse located in Almería (Spain) was used. The data were processed and analyzed to study the relationships between the measured climate variables and the state of vent opening, both statistically (using correlation coefficients) and graphically (with regression analysis). The dataset (with 81 recorded days) was then used to train, validate, and test a set of candidate LSTM-based networks for the soft sensor. The results show that the developed soft sensor can estimate the actual opening of the vents with a mean absolute error of 4.45%, which encourages integrating the soft sensor as part of decision support systems for farmers and using it to calculate other essential variables, such as greenhouse ventilation rate.

Channel estimation using hybrid optimizer based recurrent neural network long short term memory for MIMO communications in 5G network

In the fifth-generation (5G) networks, multiple input multiple-output (MIMO) systems are further developed to enhance transmission reliability. However, channel estimation is one of the major challenges which needs focus for improved data transmission in MIMO. Although efficient estimation techniques have been recently proposed, estimation accuracy needs to be upgraded further. Hence, an optimized Recurrent Neural Network-Long Short-Term Memory (RNN-LSTM) network is presented in this paper for channel estimation. At first, history of channel responses of pilot block is collected or estimated using Least Square (LS) channel estimation method. Using these collected channel responses, the proposed RNN-LSTM is trained where weight parameters are chosen optimally using hybrid Particle Swarm Optimization (PSO)-Adam optimizer. Using the trained PSO-Adam optimizer based RNN-LSTM, the current channel response is predicted or estimated. The performance of the proposed channel estimation scheme is analysed by varying pilot sequence length and number of antennas to evaluate the metrics Bit Error Rate (BER) and Mean Square Error (MSE). Complexity analysis of the proposed scheme is compared with standard estimators like LS and Minimum Mean-Square Error (MMSE).

Prediction of continuous and discrete kinetic parameters in horses from inertial measurement units data using recurrent artificial neural networks

Vertical ground reaction force (GRFz) measurements are the best tool for assessing horses' weight-bearing lameness. However, collection of these data is often impractical for clinical use. This study evaluates GRFz predicted using data from body-mounted IMUs and long short-term memory recurrent neural networks (LSTM-RNN). Twenty-four clinically sound horses, equipped with IMUs on the upper-body (UB) and each limb, walked and trotted on a GRFz measuring treadmill (TiF). Both systems were time-synchronised. Data from randomly selected 16, 4, and 4 horses formed training, validation, and test datasets, respectively. LSTM-RNN with different input sets (All, Limbs, UB, Sacrum, or Withers) were trained to predict GRFz curves or peak-GRFz. Our models could predict GRFz shapes at both gaits with RMSE below 0.40 N.kg−1. The best peak-GRFz values were obtained when extracted from the predicted curves by the all dataset. For both GRFz curves and peak-GRFz values, predictions made with the All or UB datasets were systematically better than with the Limbs dataset, showing the importance of including upper-body kinematic information for kinetic parameters predictions. More data should be gathered to confirm the usability of LSTM-RNN for GRFz predictions, as they highly depend on factors like speed, gait, and the presence of weight-bearing lameness.

RNN LSTM-based emotion recognition using EEG signals

Emotion, a representation of the human state of mind, plays an important role in day-to-day human life and helps one make good decisions. A typical way to understand human emotion is by observing a person's facial expressions and modulation of speech, and it can be categorized as sad, angry, happy, fearful, and so on. Emotion recognition using Brain Computer Interface (BCI) systems is beneficial for patients suffering from paralysis, autism, and mental retardation who cannot express their emotions like regular people. In this paper, after analyzing several data mining algorithms and various Neural Network models such as Convolution Neural Networks (CNN), Recurrent Neural Networks (RNN), and the Bi-directional RNN it has been proposed that Recurrent Neural Network-Long Short-Term Memory (RNN-LSTM) based emotion recognition using Electroencephalography (EEG) signals provides a better result. The main purpose of this paper is to introduce models which can work better than the existing ones on the K-EmoCon dataset. The metrics used in this paper are valence and arousal. The proposed RNN-LSTM model achieves a valence accuracy of 69.85% and an arousal accuracy of 45.07%. This model improves the accuracy of emotion detection on the K-EmoCon dataset. This approach achieves 4% more accuracy when compared to existing models such as the Convolution-augmented Transformer.

Predictive Maintenance and Anomaly Detection of Wind Turbines Based on Bladed Simulator Models

This paper proposes a novel data-driven scheme for condition monitoring and detecting anomalies in the wind turbine critical components based on advanced deep learning algorithms. The proposed method employs time-series data taken from a Bladed simulator model for the 5MW wind turbine. To emulate the characteristic behavior of essential wind turbine components, we develop supervised and unsupervised deep learning models using self-organizing map (SOM), long short-term memory auto encoder (LSTM-AE), and long short-term memory recurrent neural network (LSTM-RNN). Statistical process control(SPC) charts are used to evaluate the anomalous behavior predicted by the developed data-driven models. The proposed method is tested on a Bladed 5MW wind turbine model with 24 m/sec wind speed for validating its accuracy and applicability.

Prediction of Power Generation of a Photovoltaic Power Plant Based on Neural Networks

Photovoltaic energy production is an important factor for increasing the electricity supply. The ability to predict the electric power production (EPP) of a photovoltaic (PV) farm supports from the management process of the power grid to the trade in the energy market and much more. Also, by predicting the production of PV power (PVP), it is possible to monitor the lifetime of the solar cells that form the backbone of any solar PV system. As a critical result, sudden failures of the PV plant can be avoided. Using a long short-term memory recurrent neural network (LSTM-RNN) model, this work evaluates the prediction accuracy of two forecasting strategies: the recursive strategy and the non-recursive Multiple-Input and Multiple-Output, respectively. The dataset consists of 5-years in-filed production data measurements collected from the CETATEA photovoltaic power plant, a research site facility for renewable energies located in Cluj-Napoca, Romania. The high granularity of the electric power production dataset values recorded each 1 hour guarantees the overall prediction accuracy of the system. The impact of the dataset size, the number of previous observations, and the forecast horizon on the neural network prediction accuracy is evaluated for each strategy. The performance metrics used to evaluate the prediction accuracy are the root mean square error, the mean bias error, and the mean average error. The results analysis demonstrates the ability of the implemented machine learning models to predict electric power production, as well as their importance in the energy loss management process.

Social Engineering Attack Classifications on Social Media Using Deep燣earning

In defense-in-depth, humans have always been the weakest link in cybersecurity. However, unlike common threats, social engineering poses vulnerabilities not directly quantifiable in penetration testing. Most skilled social engineers trick users into giving up information voluntarily through attacks like phishing and adware. Social Engineering (SE) in social media is structurally similar to regular posts but contains malicious intrinsic meaning within the sentence semantic. In this paper, a novel SE model is trained using a Recurrent Neural Network Long Short Term Memory (RNN-LSTM) to identify well-disguised SE threats in social media posts. We use a custom dataset crawled from hundreds of corporate and personal Facebook posts. First, the social engineering attack detection pipeline (SEAD) is designed to filter out social posts with malicious intents using domain heuristics. Next, each social media post is tokenized into sentences and then analyzed with a sentiment analyzer before being labelled as an anomaly or normal training data. Then, we train an RNN-LSTM model to detect five types of social engineering attacks that potentially contain signs of information gathering. The experimental result showed that the Social Engineering Attack (SEA) model achieves 0.84 in classification precision and 0.81 in recall compared to the ground truth labeled by network experts. The experimental results showed that the semantics and linguistics similarities are an effective indicator for early detection of SEA.

Wave condition preview assisted real-time nonlinear predictive control of point-absorbing wave energy converter based on long short-term memory recurrent neural identification

A real-time nonlinear model predictive controller is presented for point-absorbing wave energy converter (PA-WEC) to maintain the optimal wave energy extraction by tracking the reference PA-WEC velocity. In order to reduce computational cost, the predictive controller is designed using wave condition preview and the Taylor series expansion based nonlinear optimization to determine the q-axis current control signals while the constraints on both the device velocity and q-axis current are respected. The dynamics of the PA-WEC is modeled and details of the nonlinear model predictive controller are presented. A pragmatic method is proposed to obtain the optimal reference PA-WEC velocity by observing the time instants when the excitation force passes a threshold. A long short-term memory recurrent neural network (LSTM-RNN) identifier is designed to identify the wave condition term for implementing the controller. The overall stability of the closed control system including the predictive control and the LSTM-RNN identifier is proved. The proposed nonlinear predictive controller is validated under realistic wave conditions. The results indicate that the proposed control allows higher PA-WEC power production than the conventional control under representative irregular wave conditions.

Fostering deep learning approaches to evaluate the impact of urbanization on vegetation and future prospects

Vegetation is an essential component of our global ecosystem and an important indicator of the dynamics and productivity of land cover. Vegetation forecasting research has been accelerated using several deep learning (DL) algorithms through remote sensing (RS) data. In this context, we used artificial intelligence (AI) and the long-short-term memory recurrent neural network (LSTM-RNN) method to explore and forecast future urban–rural vegetation disparities (ΔEVI, where EVI is the enhanced vegetation index) in Pakistan’s six megacities using MODIS EVI data. The forecast results revealed that ΔEVI is decreasing in all cities. The Root Mean Square Error (RMSE) and the Mean Absolute Percentage Error (MAPE) were used to evaluate LSTM-RNN. RSME values were recorded as 0.03, 0.07, 0.02, 0.03, 0.05, and 0.06 for Faisalabad, Gujranwala, Rawalpindi, Lahore, Multan, and Sialkot, respectively. MAPE was estimated as 0.12, 0.55, 0.24, 0.18, 0.28, and 0.47 for Faisalabad, Gujranwala, Rawalpindi, Lahore, Multan, and Sialkot, respectively. This situation indicates that LSTM-RNN can be used as a new reliable AI technique for forecasting. The results suggested that the average of forecasted ΔEVI for the next 10 years is −0.23, −0.21, −0.09, −0.13, −0.22, and −0.11 for Faisalabad, Gujranwala, Rawalpindi, Lahore, Multan, and Sialkot, respectively. The findings of this study will help evaluate the impact of urbanization on EVI by leveraging DL techniques along with implementing an urbanization policy for urban development and environmental protection for long-term urban sustainability.

Trajectory Propagation of Symmetrical Quasi-classical Dynamics with Meyer-Miller Mapping Hamiltonian Using Machine Learning

The long short-term memory recurrent neural network (LSTM-RNN) approach is applied to realize the trajectory-based nonadiabatic dynamics within the framework of the symmetrical quasi-classical dynamics method based on the Meyer-Miller mapping Hamiltonian (MM-SQC). After construction, the LSTM-RNN model allows us to propagate the entire trajectory evolutions of all involved degrees of freedoms (DOFs) from initial conditions. The proposed idea is proven to be reliable and accurate in the simulations of the dynamics of several site-exciton electron-phonon coupling models and three Tully's scattering models. It indicates that the LSTM-RNN model perfectly captures the dynamical information on the trajectory evolution in the MM-SQC dynamics. Our work proposes a novel machine learning approach in the simulation of trajectory-based nonadiabatic dynamic of complex systems with a large number of DOFs.

Forecasting Detrended Volatility Risk and Financial Price Series Using LSTM Neural Networks and XGBoost Regressor

It is common practice to employ returns, price differences or log returns for financial risk estimation and time series forecasting. In De Prado’s 2018 book, it was argued that by using returns we lose memory of time series. In order to verify this statement, we examined the differences between fractional differencing and logarithmic transformations and their impact on data memory. We employed LSTM (long short-term memory) recurrent neural networks and an XGBoost regressor on the data using those transformations. We forecasted risk (volatility) and price value and compared the results of all models using original, unmodified prices. From the results, models showed that, on average, a logarithmic transformation achieved better volatility predictions in terms of mean squared error and accuracy. Logarithmic transformation was the most promising transformation in terms of profitability. Our results were controversial to Marco Lopez de Prado’s suggestion, as we managed to achieve the most accurate volatility predictions in terms of mean squared error and accuracy using logarithmic transformation instead of fractional differencing. This transformation was also most promising in terms of profitability.

Machine Learning Approach for Fake News Detection

A vital role is played by the media in the circulation of public information about the happenings. A fast spread of information using social media and social networks or websites is possible due to the quick development of the internet. With no worry about the reliability of the information, the misinformation is spread via social websites or networks and reaches multiple users. It is a very big challenge for society to deal with the spread of misinformation. The present research interest is the analysis of instinctive credibility of the news articles. For linguistic modeling, LSTM (Long Short-Term Memory), RNN (Recurrent Neural Network), etc. are widely utilized deep learning models. This paper presents a Bidirectional LSTM (BiLSTM) based sequential model for the detection of fake news. The proposed model uses word embeddings for the representation of words in high dimensional vector format. These word embeddings and Bidirectional LSTM tries to incorporate more semantic information of the content than that of RNN and unidirectional LSTM. 3 different publicly available datasets from Kaggle dataset repository are combined together for training, testing and validation of the proposed model. Along with this, the performance of different word embeddings is analyzed in this paper.

Predicting stress-strain behavior of normal weight and lightweight aggregate concrete exposed to high temperature using LSTM recurrent neural network

Lightweight aggregate is commonly used to reduce the self-weight of concrete elements which can efficiently mitigate the environmental impact. This study investigates the mechanical performance of normal weight aggregate concrete (NWAC) and Lightweight aggregate concrete (LWAC) exposed to high temperature, focusing on developing a predictive model for the stress-strain relationship of the LWAC based on the property of cement, silica fume, light weight aggregate and water/cement ratio. Mechanical properties of 30 different mixtures, including uniaxial compressive stress–strain relationship, compressive strength, modulus of elasticity, and failure strain are determined before and after high temperature exposures at 250, 500, and 750 °C. Results show that the modulus of elasticity and compressive strength of LWAC increased with exposure temperature. In particular, mixture S4 was found to outperform other mixtures, which retained 96, 75, and 46 % of compressive strength following exposures to 250, 500, and 750 °C, respectively. At all the temperatures investigated, the specimens of this mixture were able to achieve satisfactory modulus of elasticity and maximum strain. The specimens prepared with mixtures containing 75 % LECA (S23) showed a higher retention compressive strength at 750 °C. This study also utilizes the long short-term memory (LSTM) neural network to predict the stress-strain relationship of NWAC and LWAC mixtures after exposure to high temperatures. The results show that the LSTM model could adequately predict the stress-strain relationship of both LWAC and NWAC mixtures in terms of compressive strength, modulus of elasticity, and failure strain at elevated temperatures.

Modeling of municipal waste disposal behaviors related to meteorological seasons using recurrent neural network LSTM models

The literature suggests that long short-term memory (LSTM) paired with recurrent neural network (RNN) can better express long- and short-term reliance of a data set. The study objectives are to quantify mixed waste disposal (MWD) behaviors at a Canadian landfill from 2013 to 2021, and develop separate RNN-LSTM models to predict MWD rates under four meteorological seasons. Seasonal variations are clearly presented in the historical disposal data, with higher MWD of 417.8 t/month in summer and about 289.7 t/month in winter. The variabilities of MWD are also different among the seasons. Winter experienced the least variation, probably due to similarities in inhabitants' lifestyles. All seasonal sets are negatively skewed, and the highest skewness is observed in summer. The overall model performance using the entire data range is generally satisfactory, with R2 values between 0.72–0.86. Meteorological seasons appear to be a significant factor in waste disposal rate modeling. The model performances are less reliable for smaller disposal rates <200 t/day, with 0.01 < R2 < 0.59. The results suggest the disposal behaviors on a quiet day can be quite different. The use of distinct time series related to seasons on MWD modeling is original. The proposed analytical approach provides an alternative waste modeling approach accounting for both short term (seasonal) and longer term (annual) effects.

Prediction of COVID-19 Spreading using LSTM Recurrent Neural Networks in Southeast Asia

The world has come to a standstill due to the coronavirus. The number of confrmed cases and the number of deaths are increasing day by day. Now accurate and real-time predictions are very important. Currently, a lot of work is being done on the spread prediction of coronavirus. If we can predict the number of corona patients in a country in advance, then it will be benefcial for the government of that country to take action in advance. In this paper, we experiment with Bangladesh and India data using the deep learning method. We have used LSTM (LONG SHORT-TERM MEMORY) neural network which is a type of recurrent neural network. The LSTM method can work very well even with very little data. Since we do not currently have much data on covid19, LSTM neural networks may be a suitable model. We have got very good prediction results using this method even with very little data. The prediction of various parameters (number of confrmed cases per day, number of deaths per day, number of total confrmed cases, and number of total deaths). We also calculate MAPE, MAE, and RMSE values and compare them among different models. Our modeloutperforms others’ models. We think this research will be a benefcial tool for administrators and health offcials. GUB JOURNAL OF SCIENCE AND ENGINEERING, Vol 8, Dec 2021 P 1-6

An Accurate Line-of-Sight Rate Estimation Method Based on LSTM Recurrent Neural Network for Strapdown Imaging Seeker

Obtaining accurate line-of-sight (LOS) rates is a challenging task for strapdown imaging seekers. The traditional Kalman filter and its improved algorithm are difficult to deal with the non-Gaussian noise. To this end, we design a model-driven and data-driven filter using long short-term memory (LSTM) recurrent neural network. A decoupling algorithm is proposed to isolate LOS information from the attitude angle and generate one-step predictions. Predictions and observations are used as input for supervised learning, which solves the problem of difficult to describe noise features. Finally, we design a simulation experiment to compare the proposed LSTM model with extended Kalman filter (EKF), unscented Kalman filtering (UKF) and particle filter (PF) on the task of estimating the LOS angle and LOS rate. The experimental results show that the proposed LSTM model is better than the other three methods in terms of accuracy.