Deep Long Short-term Memory Model Research Articles

An intelligent water drop algorithm with deep learning driven vehicle detection and classification

<abstract> <p>Vehicle detection in Remote Sensing Images (RSI) is a specific application of object recognition like satellite or aerial imagery. This application is highly beneficial in different fields like defense, traffic monitoring, and urban planning. However, complex particulars about the vehicles and the surrounding background, delivered by the RSIs, need sophisticated investigation techniques depending on large data models. This is crucial though the amount of reliable and labelled training datasets is still a constraint. The challenges involved in vehicle detection from the RSIs include variations in vehicle orientations, appearances, and sizes due to dissimilar imaging conditions, weather, and terrain. Both specific architecture and hyperparameters of the Deep Learning (DL) algorithm must be tailored to the features of RS data and the nature of vehicle detection tasks. Therefore, the current study proposes the Intelligent Water Drop Algorithm with Deep Learning-Driven Vehicle Detection and Classification (IWDADL-VDC) methodology to be applied upon the Remote Sensing Images. The IWDADL-VDC technique exploits a hyperparameter-tuned DL model for both recognition and classification of the vehicles. In order to accomplish this, the IWDADL-VDC technique follows two major stages, namely vehicle detection and classification. For vehicle detection process, the IWDADL-VDC method uses the improved YOLO-v7 model. After the vehicles are detected, the next stage of classification is performed with the help of Deep Long Short-Term Memory (DLSTM) approach. In order to enhance the classification outcomes of the DLSTM model, the IWDA-based hyperparameter tuning process has been employed in this study. The experimental validation of the model was conducted using a benchmark dataset and the results attained by the IWDADL-VDC technique were promising over other recent approaches.</p> </abstract>

A Deep Learning Approach to Real-time Electricity Load Forecasting

In light of the increasing importance of accurate and real-time electrical demand forecasting, this research presents a deep learning model with the goal of dramatically improving predictive accuracy. Conventional methods of forecasting, such as linear regression, have trouble capturing the complex patterns included in data about electricity usage. Standard machine learning methods are shown to be wanting when compared to the suggested deep Long Short-Term Memory (LSTM) model. Mean Absolute Error (MAE) of 5.454 and Mean Squared Error (MSE) of 18.243 demonstrate the deep LSTM model's proficiency in tackling this problem. The linear regression, on the other hand, achieved a MAE of 47.352 and an MSE of 65.606, which is lower than the proposed model. Because of its greater predictive precision and reliability, the deep LSTM model is a viable option for accurate, real-time prediction of electricity demand.

Modelling and forecasting high-frequency data with jumps based on a hybrid nonparametric regression and LSTM model

High-frequency financial data is more difficult to predict than low-frequency data because it possesses nonlinearity, nonstationarity, higher volatility, and long memory and is frequently accompanied by the jump phenomena. In this paper, the nonparametric regression (NR) model based on kernel function is used to fit the nonlinear relationship between the nonstationary series Yt and its lagging series to model the trend of high frequency financial time series. Furthermore, the deep learning LSTM (long short-term memory) model is applied to capture the high volatility and frequent jumps of high frequency financial data and to improve the forecasting accuracy of the residual series. The results demonstrate that the hybrid NR and LSTM model has greatly improved the forecasting accuracy in several evaluation criteria. In comparison to NR, support vector machine (SVM), LSTM, ARIMA and NR-SVM models, the mean absolute error (MAE) of NR-LSTM has reduced by 89.78%, 97.85%, 86.48%, 32.47% and 89%, respectively. In addition, we have constructed the trading strategy for the Shanghai-Shenzhen 300 index by using the NR-LSTM model. The NR-LSTM model can continue to provide good returns even during a bear market, which can serve as a guide for investors. Furthermore, the NR-LSTM model also exhibits the best forecasting effect when we model the high-frequency data of Ping An bank in China, the FTSE 100 index in the UK, and the S&P 500 index in the US.

Air quality particulate-pollution prediction applying GAN network and the Neural Turing Machine

Urban areas in many countries face significant concerns over the presence of aerosol particles and their effects on human health. These particles, which range in size from 1 nanometer to 100 micrometers, can easily penetrate organic matter and transport toxic gas compounds and mineral substances such as carbon monoxide, ozone, nitrogen dioxide, and sulfur dioxide. High concentrations of airborne particles pose serious challenges to health, the economy, the environment, and society, and it is crucial to investigate ways to improve the quality of life for people. The main objective of this study is to develop a framework that accurately predicts aerosol and air quality index (AQI) values in advance by estimating missing data and predicting future data. To achieve this, we propose the DAerosol.GAN.NTM framework, which combines a neural Turing machine with a generative adversarial network (GAN) to address the limitations of previous studies. Our framework outperforms previous methods, including DAerosol.NTM (without GAN) and four baseline studies using Multilayer Perceptron (MLP), Deep Neural Networks (DNN), Long-short Term Memory (LSTM), and Deep LSTM (DLSTM) models, in terms of accuracy, precision, root mean square error (RMSE), mean absolute percentage error (MAPE), and the prediction of aerosol pollution surge hours in advance according to the Time Interval Before and After the Aerosol Event (TIBAAE) criterion.

Forecasting of Ionospheric Total Electron Content Data Using Multivariate Deep LSTM Model for Different Latitudes and Solar Activity

The ionospheric state is becoming increasingly important to forecast for the reliable operation of terrestrial and space-based radio-communication systems which are influenced by ionospheric space weather. In this study, we have investigated and tested a multivariate long short-term memory (LSTM) deep learning model for its forecasting accuracy over different latitudinal regions during the solar quiet and solar active years. We also tested its prediction capability during the occurrence of a geomagnetic storm. Four stations qaq1 (60.7°N, 46.04°W), baie (49.18°N, 68.26°W), mas1 (27.76°N, 15.63°W), and bogt (4.64°N, 74.08°W) in the northern hemisphere were used in this study. To optimize the feature extraction process, we used heat map to find the correlation between TEC and the various exogenous parameters and finally nine correlated parameters were used as inputs to train the LSTM model. The performance of the LSTM model was validated by comparing it with the multilayer perceptron (MLP) machine learning algorithm using root mean square error (RMSE) and mean absolute error (MAE) as evaluation indices. The results showed an accuracy improvement of 70% and 64% over MLP during the solar quiet and active years, respectively. The prediction accuracy of our LSTM model was also 74% better than MLP during the geomagnetic storm event. These findings demonstrate the effectiveness of the developed LSTM model and the right selection of the exogenous parameters in estimating TEC, and suggest that this LSTM model can be used for short-term TEC forecasting.

Detection of Abnormal Network Traffic Using Bidirectional Long Short-Term Memory

Nowadays, web systems and servers are constantly at great risk from cyberattacks. This paper proposes a novel approach to detecting abnormal network traffic using a bidirectional long short-term memory (LSTM) network in combination with the ensemble learning technique. First, the binary classification module was used to detect the current abnormal flow. Then, the abnormal flows were fed into the multilayer classification module to identify the specific type of flow. In this research, a deep learning bidirectional LSTM model, in combination with the convolutional neural network and attention technique, was deployed to identify a specific attack. To solve the real-time intrusion-detecting problem, a stacking ensemble-learning model was deployed to detect abnormal intrusion before being transferred to the attack classification module. The class-weight technique was applied to overcome the data imbalance between the attack layers. The results showed that our approach gained good performance and the F1 accuracy on the CICIDS2017 data set reached 99.97%, which is higher than the results obtained in other research.

A Deep Stacked Bidirectional LSTM (SBiLSTM) Model for Petroleum Production Forecasting

Deep learning models have demonstrated outstanding forecasting effects and are extensively used in forecasting problems in numerous scenarios. The Long short-term memory neural network (LSTM), a deep learning variant, has huge potential in forecasting the forthcoming values of given patterns using historical time series data. Lately, the forecasting task has captivated the focus of researchers in deep learning areas to handle the constraints of traditional statistical approaches. Owing to the dire need for accurate forecasting and also due to technological advancements, the collection of time series data has become more accessible, and this has paved the way for deep forecasting models. In this paper, we propose the Stacked Bi-LSTM (SBiLSTM) architecture, an adaptation of the traditional deep long-short term memory (TDLM). The approach is evaluated using two oilfield production time series. The performance of the proposed SBiLSTM model is compared with the recurrent neural network-RNNs, multi-layer RNNs, deep gated recurrent unit (DGRU), and deep long-short term memory (DLSTM). Using different measurement criteria, the empirical results show that the proposed DLSTM model outperforms other standard approaches. It is observed that the proposed SBiLSTM model acquires long and short-range interdependent features of univariate time series data without the large memory requirement.

Recognising Cattle Behaviour with Deep Residual Bidirectional LSTM Model Using a Wearable Movement Monitoring Collar

Cattle behaviour is a significant indicator of cattle welfare. With the advancements in electronic equipment, monitoring and classifying multiple cattle behaviour patterns is becoming increasingly important in precision livestock management. The aim of this study was to detect important cattle physiological states using a neural network model and wearable electronic sensors. A novel long short-term memory (LSTM) recurrent neural network model that uses two-way information was developed to accurately classify cattle behaviour and compared with baseline LSTM. Deep residual bidirectional LSTM and baseline LSTM were used to classify six behavioural patterns of cows with window sizes of 64, 128 and 256 (6.4 s, 12.8 s and 25.6 s, respectively). The results showed that when using deep residual bidirectional LSTM with window size 128, four classification performance indicators, namely, accuracy, precision, recall, and F1-score, achieved the best results of 94.9%, 95.1%, 94.9%, and 94.9%, respectively. The results showed that the deep residual bidirectional LSTM model can be used to classify time-series data collected from twelve cows using inertial measurement unit collars. Six aim cattle behaviour patterns can be classified with high accuracy. This method can be used to quickly detect whether a cow is suffering from bovine dermatomycosis. Furthermore, this method can be used to implement automated and precise cattle behaviour classification techniques for precision livestock farming.

Interpretable deep learning LSTM model for intelligent economic decision-making

For sustainable economic growth, information about economic activities and prospects is critical to decision-makers such as governments, central banks, and financial markets. However, accurate predictions have been challenging due to the complexity and uncertainty of financial and economic systems amid repeated changes in economic environments. This study provides two approaches for better economic prediction and decision-making. We present a deep learning model based on the long short-term memory (LSTM) network architecture to predict economic growth rates and crises by capturing sequential dependencies within the economic cycle. In addition, we provide an interpretable machine learning model that derives economic patterns of growth and crisis through efficient use of the eXplainable AI (XAI) framework. For major G20 countries from 1990 to 2019, our LSTM model outperformed other traditional predictive models, especially in emerging countries. Moreover, in our model, private debt in developed economies and government debt in emerging economies emerged as major factors that limit future economic growth. Regarding the economic impact of COVID-19, we found that sharply reduced interest rates and expansion of government debt increased the probability of a crisis in some emerging economies in the future.

Making the hospital smart: using a deep long short-term memory model to predict hospital performance metrics

PurposeAbundant studies of outpatient visits apply traditional recurrent neural network (RNN) approaches; more recent methods, such as the deep long short-term memory (DLSTM) model, have yet to be implemented in efforts to forecast key hospital data. Therefore, the current study aims to reports on an application of the DLSTM model to forecast multiple streams of healthcare data.Design/methodology/approachAs the most advanced machine learning (ML) method, static and dynamic DLSTM models aim to forecast time-series data, such as daily patient visits. With a comparative analysis conducted in a high-level, urban Chinese hospital, this study tests the proposed DLSTM model against several widely used time-series analyses as reference models.FindingsThe empirical results show that the static DLSTM approach outperforms seasonal autoregressive integrated moving averages (SARIMA), single and multiple RNN, deep gated recurrent units (DGRU), traditional long short-term memory (LSTM) and dynamic DLSTM, with smaller mean absolute, root mean square, mean absolute percentage and root mean square percentage errors (RMSPE). In particular, static DLSTM outperforms all other models for predicting daily patient visits, the number of daily medical examinations and prescriptions.Practical implicationsWith these results, hospitals can achieve more precise predictions of outpatient visits, medical examinations and prescriptions, which can inform hospitals' construction plans and increase the efficiency with which the hospitals manage relevant information.Originality/valueTo address a persistent gap in smart hospital and ML literature, this study offers evidence of the best forecasting models with a comparative analysis. The study extends predictive methods for forecasting patient visits, medical examinations and prescriptions and advances insights into smart hospitals by testing a state-of-the-art, deep learning neural network method.

Spontaneous Speech Emotion Recognition Using Multiscale Deep Convolutional LSTM

Recently, emotion recognition in real sceneries such as in the wild has attracted extensive attention in affective computing, because existing spontaneous emotions in real sceneries are more challenging and difficult to identify than other emotions. Motivated by the diverse effects of different lengths of audio spectrograms on emotion identification, this paper proposes a multiscale deep convolutional long short-term memory (LSTM) framework for spontaneous speech emotion recognition. Initially, a deep convolutional neural network (CNN) model is used to learn deep segment-level features on the basis of the created image-like three channels of spectrograms. Then, a deep LSTM model is adopted on the basis of the learned segment-level CNN features to capture the temporal dependency among all divided segments in an utterance for utterance-level emotion recognition. Finally, different emotion recognition results, obtained by combining CNN with LSTM at multiple lengths of segment-level spectrograms, are integrated by using a score-level fusion strategy. Experimental results on two challenging spontaneous emotional datasets, i.e., the AFEW5.0 and BAUM-1s databases, demonstrate the promising performance of the proposed method, outperforming state-of-the-art methods.

Insider Threat Detection Based on NLP Word Embedding and Machine Learning

The growth of edge computing, the Internet of Things (IoT), and cloud computing have been accompanied by new security issues evolving in the information security infrastructure. Recent studies suggest that the cost of insider attacks is higher than the external threats, making it an essential aspect of information security for organizations. Efficient insider threat detection requires state-of-the-art Artificial Intelligence models and utility. Although significant have been made to detect insider threats for more than a decade, there are many limitations, including a lack of real data, low accuracy, and a relatively low false alarm, which are major concerns needing further investigation. In this paper, an attempt to fulfill these gaps by detecting insider threats with the novelties of the present investigation first developed two deep learning hybrid LSTM models integrated with Google's Word2vec LSTM (Long Short-Term Memory) GLoVe (Global Vectors for Word Representation) LSTM. Secondly, the performance of two hybrid DL models was compared with the state-of-the-art ML models such as XGBoost, AdaBoost, RF (Random Forest), KNN (K-Nearest Neighbor) and LR (Logistics Regression). Thirdly, the present investigation bridges the gaps of using a real dataset, high accuracy, and significantly lower false alarm rate. It was found that ML-based models outperformed the DL-based ones. The results were evaluated based on earlier studies and deemed efficient at detecting insider threats using the real dataset.

Modeling for the Prediction of Soil Moisture in Litchi Orchard with Deep Long Short-Term Memory

Soil moisture is an important factor determining yield. With the increasing demand for agricultural irrigation water resources, evaluating soil moisture in advance to create a reasonable irrigation schedule would help improve water resource utilization. This paper established a continuous system for collecting meteorological information and soil moisture data from a litchi orchard. With the acquired data, a time series model called Deep Long Short-Term Memory (Deep-LSTM) is proposed in this paper. The Deep-LSTM model has five layers with the fused time series data to predict the soil moisture of a litchi orchard in four different growth seasons. To optimize the data quality of the soil moisture sensor, the Symlet wavelet denoising algorithm was applied in the data preprocessing section. The threshold of the wavelets was determined based on the unbiased risk estimation method to obtain better sensor data that would help with the model learning. The results showed that the root mean square error (RMSE) values of the Deep-LSTM model were 0.36, 0.52, 0.32, and 0.48%, and the mean absolute percentage error (MAPE) values were 2.12, 2.35, 1.35, and 3.13%, respectively, in flowering, fruiting, autumn shoots, and flower bud differentiation stages. The determination coefficients (R2) were 0.94, 0.95, 0.93, and 0.94, respectively, in the four different stages. The results indicate that the proposed model was effective at predicting time series soil moisture data from a litchi orchard. This research was meaningful with regards to acquiring the soil moisture characteristics in advance and thereby providing a valuable reference for the litchi orchard’s irrigation schedule.

A DeepLSTM Model for Personality Traits Classification Using EEG Signals

A Deep Long Short-Term Memory (DeepLSTM) Deep Learning model for the classification of personality traits using Electroencephalogram (EEG) signals is developed in this study. The objective is to assess the efficiency of the DeepLSTM model in classification. The publicly available ASCERTAIN EEG dataset is used, which uses the Big Five Factor model for predicting personality. We also evaluated DeepLSTM model performance to that of existing state-of-the-art based machine learning classifiers such as Multilayer Perceptron (MLP), K-Nearest Neighbors (KNN), Support Vector Machine (SVM), and LibSVM. The proposed model outperforms the existing classifiers for the 70-30 partitioning approach, with a maximum classification accuracy of 90.32%.

EEG-Based Personality Prediction Using Fast Fourier Transform and DeepLSTM Model.

In this paper, a deep long short term memory (DeepLSTM) network to classify personality traits using the electroencephalogram (EEG) signals is implemented. For this research, the Myers–Briggs Type Indicator (MBTI) model for predicting personality is used. There are four groups in MBTI, and each group consists of two traits versus each other; i.e., out of these two traits, every individual will have one personality trait in them. We have collected EEG data using a single NeuroSky MindWave Mobile 2 dry electrode unit. For data collection, 40 Hindi and English video clips were included in a standard database. All clips provoke various emotions, and data collection is focused on these emotions, as the clips include targeted, inductive scenes of personality. Fifty participants engaged in this research and willingly agreed to provide brain signals. We compared the performance of our deep learning DeepLSTM model with other state-of-the-art-based machine learning classifiers such as artificial neural network (ANN), K-nearest neighbors (KNN), LibSVM, and hybrid genetic programming (HGP). The analysis shows that, for the 10-fold partitioning method, the DeepLSTM model surpasses the other state-of-the-art models and offers a maximum classification accuracy of 96.94%. The proposed DeepLSTM model was also applied to the publicly available ASCERTAIN EEG dataset and showed an improvement over the state-of-the-art methods.

Integration of complete ensemble empirical mode decomposition with deep long short-term memory model for particulate matter concentration prediction.

The atmospheric particulate matter (PM) with a diameter of 2.5 μm or less (PM2.5) is one of the key indicators of air pollutants. Accurate prediction of PM2.5 concentration is very important for air pollution monitoring and public health management. However, the presence of noise in PM2.5 data series is a major challenge of its accurate prediction. A novel hybrid PM2.5 concentration prediction model is proposed in this study by combining complete ensemble empirical mode decomposition (CEEMD) method, Pearson's correlation analysis, and a deep long short-term memory (LSTM) method. CEEMD was employed to decompose historical PM2.5 concentration data to different frequencies in order to enhance the timing characteristics of data. Pearson's correlation was used to screen the different frequency intrinsic-mode functions of decomposed data. Finally, the filtered enhancement data were inputted to a deep LSTM network with multiple hidden layers for training and prediction. The results evidenced the potential of the CEEMD-LSTM hybrid model with a prediction accuracy of approximately 80% and model convergence after 700 training epochs. The secondary screening of Pearson's correlation test improved the model (CEEMD-Pearson) accuracy up to 87% but model convergence after 800 epochs. The hybrid model combining CEEMD-Pearson with the deep LSTM neural network showed a prediction accuracy of nearly 90% and model convergence after 650 interactions. The results provide a clear indication of higher prediction accuracy of PM2.5 with less computation time through hybridization of CEEMD-Pearson with deep LSTM models and its potential to be employed for air pollution monitoring.

Classification of thermoluminescence features of CaCO3 with long short-term memory model.

Calcium carbonate (CaCO3 ), a mineral commonly found in the Earth's crust, is mainly in the forms of calcite and aragonite. Calcite has the most stable type of thermodynamics at room temperature and ambient pressure. It has wide band gap structure and is of great interest for a wide-range technical and industrial applications due to its physical properties and suitability. In this study, a new method based on the long short-term memory (LSTM) model of deep learning is proposed to classify the thermoluminescence properties such as fading, cycle of measurement, heating rate, and dose-response of CaCO3 . Therefore the thermoluminescence properties of calcite was investigated as a suitable band structure and its coherent data were used to classify the features using a deep learning LSTM model. Experiments were carried out on a dataset consisting of four classes. The accuracy, precision, and sensitivity values of the proposed model obtained were 98.34, 97.90, and 98.56%, respectively. The classification process of the results obtained from the designed model showed good performance.

Predicting the geopolymerization process of fly ash-based geopolymer using deep long short-term memory and machine learning

In this study, the geopolymerization process of fly ash-based geopolymer was estimated using the deep long short-term memory (LSTM) and machine learning models. The k-Nearest Neighbors (kNN) and support vector regression (SVR) approaches were used in the machine learning models. The percentages of major chemical components in fly ash, the alkaline solution concentration, the mole ratio, the liquid-to-fly ash mass ratio, and the curing temperature were selected as input variables in the models. The geopolymerization peak time, geopolymerization peak heat, dissolution peak time, and dissolution peak heat obtained from the calorimetric curve of fly ash-based geopolymer were used as the output variables. The deep LSTM, SVR, and kNN models estimated the geopolymerization peak time with 99.55%, 98.83%, and 91.62% accuracy, respectively. The geopolymerization peak heat was estimated by the deep LSTM, SVR, and kNN models with 99.69%, 98.91%, and 88.36% accuracy, respectively. The deep LSTM, SVR, and kNN models predicted the dissolution peak time with 99.49%, 99.43%, and 92.86% accuracy, respectively. The dissolution peak heat was estimated using the deep LSTM, SVR, and kNN models with 99.67%, 99.38%, and 90.60% accuracy, respectively. This study found that the deep LSTM model can be estimated the geopolymerization process of fly ash-based geopolymer with higher accuracy than the SVR and kNN models.

The Deep Learning LSTM and MTD Models Best Predict Acute Respiratory Infection among Under-Five-Year Old Children in Somaliland

The most effective techniques for predicting time series patterns include machine learning and classical time series methods. The aim of this study is to search for the best artificial intelligence and classical forecasting techniques that can predict the spread of acute respiratory infection (ARI) and pneumonia among under-five-year old children in Somaliland. The techniques used in the study include seasonal autoregressive integrated moving averages (SARIMA), mixture transitions distribution (MTD), and long short term memory (LSTM) deep learning. The data used in the study were monthly observations collected from five regions in Somaliland from 2011–2014. Prediction results from the three best competing models are compared by using root mean square error (RMSE) and absolute mean deviation (MAD) accuracy measures. Results have shown that the deep learning LSTM and MTD models slightly outperformed the classical SARIMA model in predicting ARI values.

Deep LSTM-Based Transfer Learning Approach for Coherent Forecasts in Hierarchical Time Series.

Hierarchical time series is a set of data sequences organized by aggregation constraints to represent many real-world applications in research and the industry. Forecasting of hierarchical time series is a challenging and time-consuming problem owing to ensuring the forecasting consistency among the hierarchy levels based on their dimensional features. The excellent empirical performance of our Deep Long Short-Term Memory (DLSTM) approach on various forecasting tasks motivated us to extend it to solve the forecasting problem through hierarchical architectures. Toward this target, we develop the DLSTM model in auto-encoder (AE) fashion and take full advantage of the hierarchical architecture for better time series forecasting. DLSTM-AE works as an alternative approach to traditional and machine learning approaches that have been used to manipulate hierarchical forecasting. However, training a DLSTM in hierarchical architectures requires updating the weight vectors for each LSTM cell, which is time-consuming and requires a large amount of data through several dimensions. Transfer learning can mitigate this problem by training first the time series at the bottom level of the hierarchy using the proposed DLSTM-AE approach. Then, we transfer the learned features to perform synchronous training for the time series of the upper levels of the hierarchy. To demonstrate the efficiency of the proposed approach, we compare its performance with existing approaches using two case studies related to the energy and tourism domains. An evaluation of all approaches was based on two criteria, namely, the forecasting accuracy and the ability to produce coherent forecasts through through the hierarchy. In both case studies, the proposed approach attained the highest accuracy results among all counterparts and produced more coherent forecasts.