Performance Of Long Short-term Memory Research Articles

Cloud-based email phishing attack using machine and deep learning algorithm

Cloud computing refers to the on-demand availability of personal computer system assets, specifically data storage and processing power, without the client's input. Emails are commonly used to send and receive data for individuals or groups. Financial data, credit reports, and other sensitive data are often sent via the Internet. Phishing is a fraudster's technique used to get sensitive data from users by seeming to come from trusted sources. The sender can persuade you to give secret data by misdirecting in a phished email. The main problem is email phishing attacks while sending and receiving the email. The attacker sends spam data using email and receives your data when you open and read the email. In recent years, it has been a big problem for everyone. This paper uses different legitimate and phishing data sizes, detects new emails, and uses different features and algorithms for classification. A modified dataset is created after measuring the existing approaches. We created a feature extracted comma-separated values (CSV) file and label file, applied the support vector machine (SVM), Naive Bayes (NB), and long short-term memory (LSTM) algorithm. This experimentation considers the recognition of a phished email as a classification issue. According to the comparison and implementation, SVM, NB and LSTM performance is better and more accurate to detect email phishing attacks. The classification of email attacks using SVM, NB, and LSTM classifiers achieve the highest accuracy of 99.62%, 97% and 98%, respectively.

Front-End Model of Wireless Network Combined with Artificial Intelligence in Computer Information Management System

To address the security problem of computer information management, an artificial intelligence- (AI-) based information intrusion detection model is built in combination with wireless network. Firstly, the background and characteristics of wireless local area network (WLAN) technology are analyzed, and the relationship between AI technology and deep learning is introduced. Secondly, an intrusion detection model on account of long short-term memory (LSTM) neural network and gated recursive unit (GRU) is constructed after analysis of different neural network models. The L2 weight attenuation and dropout regularization strategies are combined with the neural network model. Finally, an intrusion detection front-end model combining wireless network and AI is established. From the comparison of intrusion detection experiments, the generalization ability of the model can be improved by using L2 weight attenuation and dropout regularization strategies. Nevertheless, the performance improvement is only slight, so the early stop method is adopted instead of the regularization strategy. Compared with the existing classification models, the overall performance of LSTM and GRU models is improved by about 17%. The performance of GRU model is not much different from that of LSTM model, but the amount of computation is reduced. Therefore, GRU model is the optimal choice to construct intrusion detection system. The intrusion detection models in WLAN and GRU can improve the security performance of computer information management system. To sum up, this work provides reference for the development of computer management system.

Impact of spatial distribution information of rainfall in runoff simulation using deep learning method

Abstract. Rainfall-runoff modeling is of great importance for flood forecast and water management. Hydrological modeling is the traditional and commonly used approach for rainfall-runoff modeling. In recent years, with the development of artificial intelligence technology, deep learning models, such as the long short-term memory (LSTM) model, are increasingly applied to rainfall-runoff modeling. However, current works do not consider the effect of rainfall spatial distribution information on the results. Focusing on 10 catchments from the Catchment Attributes and Meteorology for Large-Sample Studies (CAMELS) dataset, this study compared the performance of LSTM with different look-back windows (7, 15, 30, 180, 365 d) for future 1 d discharges and for future multi-day simulations (7, 15 d). Secondly, the differences between LSTMs as individual models trained independently in each catchment and LSTMs as regional models were also compared across 10 catchments. All models are driven by catchment mean rainfall data and spatially distributed rainfall data, respectively. The results demonstrate that regardless of whether LSTMs are trained independently in each catchment or trained as regional models, rainfall data with spatial information improves the performance of LSTMs compared to models driven by mean rainfall data. The LSTM as a regional model did not obtain better results than LSTM as an individual model in our study. However, we found that using spatially distributed rainfall data can reduce the difference between LSTM as a regional model and LSTM as an individual model. In summary, (a) adding information about the spatial distribution of the data is another way to improve the performance of LSTM where long-term rainfall records are absent, and (b) understanding and utilizing the spatial distribution information can help improve the performance of deep learning models in runoff simulations.

Real-Time Encrypted Traffic Classification with Deep Learning

Confidentiality requirements of individuals and companies led to the dominance of encrypted payloads in the overall Internet traffic. Hence, traffic classification on a network became increasingly difficult as it must rely on only the packet headers. Many vital tasks such as differential pricing, providing a safe Internet for children, and eliminating malicious connections require traffic classification, even if the payload contents are encrypted. Encrypted traffic is harder to classify as packet content becomes unreadable. In this work, we aim to provide an insight into traffic classification using encrypted packets in terms of both accuracy and packet processing time. LSTM (Long Short-Term Memory) architecture is a good candidate for this problem as it can handle sequences. Each flow can be modeled as a sequence and patterns of the sequences can provide valuable information. We compare the performance of LSTM with other methods in both real-time and offline experiments. Compared to a machine learning method both online and offline LSTM excelled with precision and recall differences up to 50%. Average accuracy with LSTM was measured as 97.77% offline and 91.7% in real-time. Average packet processing time in real-time was recorded as 0.593 msec which is 5 times faster than a recent work that uses LSTM method.

LSTM-based Multivariate Time-Series Analysis: A Case of Journal Visitors Forecasting

Forecasting is the process of predicting something in the future based on previous patterns. Forecasting will never be 100% accurate because the future has a problem of uncertainty. However, using the right method can make forecasting have a low error rate value to provide a good forecast for the future. This study aims to determine the effect of increasing the number of hidden layers and neurons on the performance of the long short-term memory (LSTM) forecasting method. LSTM performance measurement is done by root mean square error (RMSE) in various architectural scenarios. The LSTM algorithm is considered capable of handling long-term dependencies on its input and can predict data for a relatively long time. Based on research conducted from all models, the best results were obtained with an RMSE value of 0.699 obtained in model 1 with the number of hidden layers 2 and 64 neurons. Adding the number of hidden layers can significantly affect the RMSE results using neurons 16 and 32 in Model 1.

Effects of Training Data on the Learning Performance of LSTM Network for Runoff Simulation

With the rapid development of Artificial Intelligence (AI) technology, the Long Short-Term Memory (LSTM) network has been widely used for forecasting hydrological process. To evaluate the effect of training data amount on the performance of LSTM, the study proposed an experiment scheme. First, K-Nearest Neighbour (KNN) algorithm is employed for generating the meteorological data series of 130 years based on the observed data, and the Soil and Water Assessment Tool (SWAT) model is used to obtain the corresponding runoff series with the generated meteorological data series. Then, the 130 years of rainfall and runoff data is divided into two parts: the first 80 years of data for model training and the remaining 50 years of data for model verification. Finally, the LSTM models are developed and evaluated, with the first 5-year, 10-year, 20-year, 40-year and 80-year data series as training data respectively. The results obtained in Yalong River, Minjiang River and Jialing River show that increasing the training data amount can effectively reduce the over-fittings of LSTM network and improve the prediction accuracy and stability of LSTM network.

Improving performance of deep learning predictive models for COVID-19 by incorporating environmental parameters

The Coronavirus disease 2019 (COVID-19) pandemic has severely crippled the economy on a global scale. Effective and accurate forecasting models are essential for proper management and preparedness of the healthcare system and resources, eventually aiding in preventing the rapid spread of the disease. With the intention to provide better forecasting tools for the management of the pandemic, the current research work analyzes the effect of the inclusion of environmental parameters in the forecasting of daily COVID-19 cases. Three univariate variants of the long short-term memory (LSTM) model (basic/vanilla, stacked, and bi-directional) were employed for the prediction of daily cases in 9 cities across 3 countries with varying climatic zones (tropical, sub-tropical, and frigid), namely India (New Delhi and Nagpur), USA (Yuma and Los Angeles) and Sweden (Stockholm, Skane, Uppsala and Vastra Gotaland). The results were compared to a basic multivariate LSTM model with environmental parameters (temperature (T) and relative humidity (RH)) as additional inputs. Periods with no or minimal lockdown were chosen specifically in these cities to observe the uninhibited spread of COVID-19 and explore its dependence on daily environmental parameters. The multivariate LSTM model showed the best overall performance; the mean absolute percentage error (MAPE) showed an average of 64% improvement from other univariate models upon the inclusion of the above environmental parameters. Correlation with temperature was generally positive for the cold regions and negative for the warm regions. RH showed mixed correlations, most likely driven by its temperature dependence and effect of allied local factors. The results suggest that the inclusion of environmental parameters could significantly improve the performance of LSTMs for predicting daily cases of COVID-19, although other positive and negative confounding factors can affect the forecasting power.

Guest editorial: Smart communications and networking: architecture, applications, and future challenges

Guest editorial: Smart communications and networking: architecture, applications, and future challenges

LogLS: Research on System Log Anomaly Detection Method Based on Dual LSTM

System logs record the status and important events of the system at different time periods. They are important resources for administrators to understand and manage the system. Detecting anomalies in logs is critical to identifying system faults in time. However, with the increasing size and complexity of today’s software systems, the number of logs has exploded. In many cases, the traditional manual log-checking method becomes impractical and time-consuming. On the other hand, existing automatic log anomaly detection methods are error-prone and often use indices or log templates. In this work, we propose LogLS, a system log anomaly detection method based on dual long short-term memory (LSTM) with symmetric structure, which regarded the system log as a natural-language sequence and modeled the log according to the preorder relationship and postorder relationship. LogLS is optimized based on the DeepLog method to solve the problem of poor prediction performance of LSTM on long sequences. By providing a feedback mechanism, it implements the prediction of logs that do not appear. To evaluate LogLS, we conducted experiments on two real datasets, and the experimental results demonstrate the effectiveness of our proposed method in log anomaly detection.

TypeNet: Deep Learning Keystroke Biometrics

We study the performance of Long Short-Term Memory networks for keystroke biometric authentication at large scale in free-text scenarios. For this we explore the performance of Long Short-Term Memory (LSTMs) networks trained with a moderate number of keystrokes per identity and evaluated under different scenarios including: i) three learning approaches depending on the loss function (softmax, contrastive, and triplet loss); ii) different number of training samples and lengths of keystroke sequences; iii) four databases based on two device types (physical vs touchscreen keyboard); and iv) comparison with existing approaches based on both traditional statistical methods and deep learning architectures. Our approach called TypeNet achieves state-of-the-art keystroke biometric authentication performance with an Equal Error Rate of 2.2% and 9.2% for physical and touchscreen keyboards, respectively, significantly outperforming previous approaches. Our experiments demonstrate a moderate increase in error with up to 100,000 subjects, demonstrating the potential of TypeNet to operate at an Internet scale. To the best of our knowledge, the databases used in this work are the largest existing free-text keystroke databases available for research with more than 136 million keystrokes from 168,000 subjects in physical keyboards, and 60,000 subjects with more than 63 million keystrokes acquired on mobile touchscreens.

Performance evaluation of LSTM neural networks for consumption prediction

Energy consumption and energy efficiency are topics that have attracted the attention of researchers in recent years, in order to seek sustainable solutions for energy production and reduction of costs, aiming to provide a balance between development and protection of natural resources. One of the alternatives that have obtained satisfactory results is the use of technologies based on Internet of Things (IoT) and Deep Learning systems. Based on this, we assessed the performance of Long Short-Term Memory (LSTM) neural networks in time series electric energy consumption prediction, for a forecasting module of an IoT system. Three time series were used and we compared LSTM to the algorithms Extreme Boost Gradient and Random Forest. Computational results indicate that the LSTM model showed a tendency of better RMSE performance in the first data set, and statistically significant better results in other two data sets, according to the Kruskal-Wallis test (p < 0.0001 in both cases). Thus, the proposed model was implemented and validated from experiments, presenting accurate prediction for monitoring and estimating power consumption, being applicable to Energy Efficiency and decision making.

An LSTM-PINN Hybrid Method to Estimate Lithium-Ion Battery Pack Temperature

Physics-based models for battery temperature prediction are often not suitable for online applications due to the large number of fitted parameters, low fidelity results from parameter inaccuracy and unaccounted model dynamics, limited high quality experimental data, and slow convergence of predictions from unknown initial conditions. On the other hand, data-driven models require much less computation power but a large dataset to learn the time dependent behavior. This paper proposes a physics-informed neural network (PINN) to take full advantage of both physics-based and data-driven models. Four comparative studies were performed to investigate the effectiveness of including chamber temperature with two different activation functions, the optimal number of neurons and hidden layers, the dependance on reversible heat generation, and the performance of long short-term memory (LSTM)-PINN model. The results show that the LSTM-PINN with chamber temperature as one of the inputs delivers better prediction accuracy. The LSTM-PINN with an exponential activation function for the chamber temperature has a more accurate prediction for the direct current fast charge (DCFC) test profile and similar prediction accuracy for the grade load (GL) 100 test profile. The root mean square errors (RMSEs) of the LSTM-PINN are 0.57°C for DCFC and 0.52°C for GL 100, respectively. In addition, having 55 neurons and 4 hidden layers gives the lowest prediction error. Furthermore, the improvement by having reversible heat generation is negligible. At last, the LSTM-PINN model has less prediction error than the LSTM model, especially when the battery temperature range is enormous.

ChiNet: Deep Recurrent Convolutional Learning for Multimodal Spacecraft Pose Estimation

This paper presents an innovative deep learning pipeline which estimates the relative pose of a spacecraft by incorporating the temporal information from a rendezvous sequence. It leverages the performance of long short-term memory (LSTM) units in modelling sequences of data for the processing of features extracted by a convolutional neural network (CNN) backbone. Three distinct training strategies, which follow a coarse-to-fine funnelled approach, are combined to facilitate feature learning and improve end-to-end pose estimation by regression. The capability of CNNs to autonomously ascertain feature representations from images is exploited to fuse thermal infrared data with red-green-blue (RGB) inputs, thus mitigating the effects of artefacts from imaging space objects in the visible wavelength. Each contribution of the proposed framework, dubbed ChiNet, is demonstrated on a synthetic dataset, and the complete pipeline is validated on experimental data.

Depolarization Current Prediction of Transformers OPI System Affected From Detrapped Charge Using LSTM

The depolarization current is critical for the computation of insulation-sensitive parameters of the transformer. Prolonged polarization&#x2013;depolarization current (PDC) measurement times, on the other hand, can increase the likelihood of obtaining distorted original data as a result of environmental factors and electromagnetic noise. Therefore, techniques for measuring PDC data over a short period of time, such as forecasting depolarization current, are required. Additionally, unlike polarization current, which is easily predictable, depolarization current is affected by charging duration and detrapping current. Consequently, an extremely accurate prediction method is required. This paper proposes a multivariate recurrent neural network model based on long short-term memory (LSTM) network for depolarization current prediction using only polarization data. It reduces measurement time and treats the model with time-varying parameters (MTVP) elements as features. Therefore, the influence of the detrapped charge effect while forecasting depolarization current is also considered. The performance of LSTM is compared to that of Linear, Multi-dense, and 1D-Conv (one-dimensional convolutional neural network) to determine the optimum model for forecasting depolarization current. Thereafter, a permutation-based explainability method is used in python to describe model behavior by assessing the variable importance through drop-out loss. The presented results demonstrate that forecasting with the LSTM model produces the lowest mean absolute error and maintains prediction consistency throughout the testing period.

LSTM and Statistical Learning for Dynamic Inferential Modeling with Applications to a 660MW Boiler

Statistical learning methods have been widely studied and practiced in the past for inferential modeling. In recent years, deep learning methods have been implemented for inferential sensor modeling. As a popular deep learning model, the long short-term memory (LSTM) network is capable of handling data nonlinearity and dynamics and is therefore applied for dynamic inferential modeling. In this paper, we analyze and compare LSTM with other statistical learning methods for the dynamic NOx emission prediction of a 660 MW industrial boiler. Support vector regression (SVR), partial least squares (PLS), and Least absolute shrinkage and selection operator (Lasso) with embedded dynamics are compared with LSTM for dynamic inferential modeling. The experimental results indicate that SVR, PLS, and Lasso outperform LSTM. By disabling the LSTM gates to realize a simple memory structure, the LSTM performance is signifcantly improved. The main goal of the paper is to demonstrate that a deep neural network that is effective in other domains requires close scrutiny and detailed study to show its superiority in process applications.

Deep-Learning-Based Production Decline Curve Analysis in the Gas Reservoir through Sequence Learning Models

Production performance prediction of tight gas reservoirs is crucial to the estimation of ultimate recovery, which has an important impact on gas field development planning and economic evaluation. Owing to the model’s simplicity, the decline curve analysis method has been widely used to predict production performance. The advancement of deep-learning methods provides an intelligent way of analyzing production performance in tight gas reservoirs. In this paper, a sequence learning method to improve the accuracy and efficiency of tight gas production forecasting is proposed. The sequence learning methods used in production performance analysis herein include the recurrent neural network (RNN), long short-term memory (LSTM) neural network, and gated recurrent unit (GRU) neural network, and their performance in the tight gas reservoir production prediction is investigated and compared. To further improve the performance of the sequence learning method, the hyperparameters in the sequence learning methods are optimized through a particle swarm optimization algorithm, which can greatly simplify the optimization process of the neural network model in an automated manner. Results show that the optimized GRU and RNN models have more compact neural network structures than the LSTM model and that the GRU is more efficiently trained. The predictive performance of LSTM and GRU is similar, and both are better than the RNN and the decline curve analysis model and thus can be used to predict tight gas production.

Meteorological influence on forecasting urban pollutants: Long-term predictability versus extreme events in a spatially heterogeneous urban ecosystem

With the escalation of heat- and pollution-related threats in cities across the globe, timely counteractions and emergency procedures are vital, which calls for accurate co-prediction of urban heat and air quality under both standard conditions and under extreme events. In this study, we used historical hourly data recorded at 9 sites across the Sydney metropolitan area to test the performance of long short-term memory (LSTM) forecasting architectures in predicting 5 urban pollutants based on different combinations of meteorological inputs and considering standard, bushfire, and pandemic lockdown conditions. We demonstrate that, in most cases and even in a fast-growing city, there is no significant benefit achieved by including extra predictors to temperature and humidity, when adequate forecasting techniques capable of learning long-term dependencies are used. Further, in agreement with previous studies, we provide evidence of ozone's higher responsiveness to all weather parameters and thus enhanced predictability and PM10's lower predictability as compared to all other considered urban pollutants. The prediction accuracy tends to be comparable between standard conditions and bushfire events. However, the predictability significantly declines under anomalies in anthropogenic patterns and urban metabolic rates as those recorded during the pandemic. The inclusion of local emission sources and anthropogenic factors in the input dataset is considered necessary for NO and PM10 to properly predict urban air quality, especially under human-related extreme conditions.

Temporal prediction of algal parameters in Three Gorges Reservoir based on highly time-resolved monitoring and long short-term memory network

Many dammed rivers throughout the world have experienced frequent harmful algal blooms (HABs) in the context of climate change and anthropogenic activities. Accurate forecasting of algal parameters (i.e., algal cell density and microcystin concentration) has great practical significance for taking precautions against HABs risks. Long short-term memory (LSTM) networks have recently shown potential in predicting water quality parameters. However, there is still little known about the robustness of the LSTM in forecasting highly time-resolved measurement of algal parameters. This study developed a hybrid deep-learning architecture (XG-LSTM) composed of one XGBoost module and two parallel LSTM models to predict algal cell density and microcystin concentration in the Three Gorges Reservoir (TGR). The proposed model was validated by in situ multi-sensor-system monitoring data at four bloom-impacted tributaries in the TGR. Each modelling process utilized the antecedent information of the algal parameters and the corresponding environmental variables as inputs for forecasting the algal parameters for the coming hours and days. As expected, the presented model achieved better performance than those without special feature extraction procedures, providing that the use of selected environmental parameters can improve LSTM performance. In addition, the hybrid XG-LSTM model successfully captured the time-series patterns of both algal cell density and microcystin concentration compared with other data-driven models, further suggesting the reliable utilization of this model in early warnings of bloom toxicity. Thus, the results presented demonstrate the potential of deep learning technology for real-time prediction of algal parameters in the TGR, and possibly for rapid detection of developing HABs in other aquatic ecosystems.

Evaluation of Deep Learning Models for Smoking Recognition with Smartwatch and Smartphone Sensors

Smartwatches and smartphones are extensively used in human activity recognition, particularly for step counting and daily sports applications, thanks to the motion sensors integrated into these devices. Machine learning algorithms are often utilized to process sensor data and classify the activities. There are many studies that explore the use of traditional classification algorithms in activity recognition, however, recently, deep learning approaches are also receiving attention. In this paper, we use a dataset that particularly consists of smoking-related activities and explores the recognition performance of three deep learning architectures, namely Long-Short Term Memory (LSTM)}, Recurrent Neural Networks (RNN) and Convolutional Neural Networks (CNN). We evaluate their performances according to different hyperparameters, different sensor types and device types. The results show that the performance of LSTM is much higher than that of CNN and RNN. Moreover, the use of magnetometer and gyroscope together with accelerometer data improves the performance. Use of data from smartphone sensors also enhances the performance results and the final accuracy with the best parameter combinations is observed to be 98%.

Comparing LSTM and GRU Models to Predict the Condition of a Pulp Paper Press

The accuracy of a predictive system is critical for predictive maintenance and to support the right decisions at the right times. Statistical models, such as ARIMA and SARIMA, are unable to describe the stochastic nature of the data. Neural networks, such as long short-term memory (LSTM) and the gated recurrent unit (GRU), are good predictors for univariate and multivariate data. The present paper describes a case study where the performances of long short-term memory and gated recurrent units are compared, based on different hyperparameters. In general, gated recurrent units exhibit better performance, based on a case study on pulp paper presses. The final result demonstrates that, to maximize the equipment availability, gated recurrent units, as demonstrated in the paper, are the best options.