Ensemble Of Long Short-term Memory Research Articles

Reduced‐Order Probabilistic Emulation of Physics‐Based Ring Current Models: Application to RAM‐SCB Particle Flux

AbstractIn this work, we address the computational challenge of large‐scale physics‐based simulation models for the ring current. Reduced computational cost allows for significantly faster than real‐time forecasting, enhancing our ability to predict and respond to dynamic changes in the ring current, valuable for space weather monitoring and mitigation efforts. Additionally, it can also be used for a comprehensive investigation of the system. Thus, we aim to create an emulator for the Ring current‐Atmosphere interactions Model with Self‐Consistent magnetic field (RAM‐SCB) particle flux that not only improves efficiency but also facilitates forecasting with reliable estimates of prediction uncertainties. The probabilistic emulator is built upon the methodology developed by Licata and Mehta (2023), https://doi.org/10.1029/2022sw003345. A novel discrete sampling is used to identify 30 simulation periods over 20 years of solar and geomagnetic activity. Focusing on a subset of particle flux, we use Principal Component Analysis for dimensionality reduction and Long Short‐Term Memory (LSTM) neural networks to perform dynamic modeling. Hyperparameter space was explored extensively resulting in about 5% median symmetric accuracy across all data sets for one‐step dynamic prediction. Using a hierarchical ensemble of LSTMs, we have developed a reduced‐order probabilistic emulator (ROPE) tailored for time‐series forecasting of particle flux in the ring current. This ROPE offers accurate predictions of omnidirectional flux at a single energy with no pitch angle information, providing robust predictions on the test set with an error score below 11% and calibration scores under 8% with bias under 2% providing a significant speed up as compared to the full RAM‐SCB run.

Intelligent intrusion detection through deep autoencoder and stacked long short-term memory

In the realm of network intrusion detection, the escalating complexity and diversity of cyber threats necessitate innovative approaches to enhance detection accuracy. This study introduces an integrated solution leveraging deep learning techniques for improved intrusion detection. The proposed framework consists on a deep autoencoder for feature extraction, and a stacked long short-term memory (LSTM) network ensemble for classification. The deep autoencoder compresses raw network data, extracting salient features and mitigating noise. Subsequently, the stacked LSTM ensemble captures intricate temporal dependencies, correcting anomaly detection precision. Experiments conducted on the UNSW-NB15 dataset, and a benchmark in intrusion detection validate the effectiveness of the approach. The solution achieves an accuracy of 90.59%, with precision, recall, and F1-Score metrics reaching 90.65, 90.59, and 90.57, respectively. Notably, the framework outperforms standalone models and demonstrates the advantage of synergizing deep autoencoder-driven feature extraction with the stacked LSTM ensemble. Furthermore, a binary classification experiment attains an accuracy of about 90.59%, surpassing the multiclass classification and affirming the model's potential for binary threat identification. Comparative analyses highlight the pivotal role of feature extraction, while experimentation illustrates the enhancement achieved by incorporating the synergistic deep autoencoder-Stacked LSTM approach.

Early detection of chronic kidney disease using eurygasters optimization algorithm with ensemble deep learning approach

Chronic kidney disease (CKD) emerges as a global health problem with high morbidity and mortality rates, and it induces other diseases. Patients often fail to notice these diseases because there are no obvious symptoms during the earlier stage of CKD. Earlier diagnosis of CKD allows them to receive early medical intervention to improve the disease progression and outcome. Earlier stratification of CKD can affect the health care provided to patients through various options, such as kidney transportation, hemodialysis, or pharmacological care in milder cases. Recently, deep learning (DL) and machine learning (ML) approaches have gained significance in the domain of medical diagnoses due to their high prediction accuracy. DL and ML approaches can successfully aid physicians in achieving these goals due to their accurate and fast detection performance. The performance of the proposed technique mainly depends on selecting the suitable algorithms and appropriate features. Thus, the study introduces an Eurygasters Optimization Algorithm with an Ensemble DL CKD detection (EOAEDL-CKDD) method. The EOAEDL-CKDD method aims to detect and classify the presence of CKD using feature selection and hyperparameter tuning strategies. The EOAEDL-CKDD method applies a min-max scalar to convert the data into a uniform format to achieve this. The EOAEDL-CKDD technique exploits EOA to select features. This is followed by an ensemble of long short-term memory (LSTM), bidirectional gated recurrent unit (BiGRU), and bidirectional LSTM (BiLSTM) models that are used for the CKD detection process. Finally, a shuffled frog leap algorithm (SFLA) based hyperparameter selection process is carried out to choose the ensemble models' hyperparameter values optimally. The empirical assessment of the EOAEDL-CKDD method is tested on the benchmark CKD dataset. The experimental values highlighted that the EOAEDL-CKDD technique gains an optimal detection rate compared to existing models.

Explainable machine learning for labquake prediction using catalog-driven features

Recently, Machine learning (ML) has been widely utilized for laboratory earthquake (labquake) prediction using various types of data. This study pioneers in time to failure (TTF) prediction based on ML using acoustic emission (AE) records from three laboratory stick-slip experiments performed on Westerly granite samples with naturally fractured rough faults, more similar to the heterogeneous fault structures in the nature. 47 catalog-driven seismo-mechanical and statistical features are extracted introducing some new features based on focal mechanism. A regression voting ensemble of Long-Short Term Memory (LSTM) networks predicts TTF with a coefficient of determination (R2) of 70% on the test dataset. Feature importance analysis revealed that AE rate, correlation integral, event proximity, and focal mechanism-based features are the most important features for TTF prediction. Results reveal that the network uses all information among the features for prediction, including general trends in high correlated features as well as fine details about local variations and fault evolution involved in low correlated features. Therefore, some highly correlated and physically meaningful features may be considered less important for TTF prediction due to their correlation with other important features. Our study provides a ground for applying catalog-driven to constrain TTF of complex heterogeneous rough faults, which is capable to be developed for real application.

Computer aided progression detection model based on optimized deep LSTM ensemble model and the fusion of multivariate time series data

Alzheimer’s disease (AD) is the most common form of dementia. Early and accurate detection of AD is crucial to plan for disease modifying therapies that could prevent or delay the conversion to sever stages of the disease. As a chronic disease, patient’s multivariate time series data including neuroimaging, genetics, cognitive scores, and neuropsychological battery provides a complete profile about patient’s status. This data has been used to build machine learning and deep learning (DL) models for the early detection of the disease. However, these models still have limited performance and are not stable enough to be trusted in real medical settings. Literature shows that DL models outperform classical machine learning models, but ensemble learning has proven to achieve better results than standalone models. This study proposes a novel deep stacking framework which combines multiple DL models to accurately predict AD at an early stage. The study uses long short-term memory (LSTM) models as base models over patient’s multivariate time series data to learn the deep longitudinal features. Each base LSTM classifier has been optimized using the Bayesian optimizer using different feature sets. As a result, the final optimized ensembled model employed heterogeneous base models that are trained on heterogeneous data. The performance of the resulting ensemble model has been explored using a cohort of 685 patients from the University of Washington's National Alzheimer’s Coordinating Center dataset. Compared to the classical machine learning models and base LSTM classifiers, the proposed ensemble model achieves the highest testing results (i.e., 82.02, 82.25, 82.02, and 82.12 for accuracy, precision, recall, and F1-score, respectively). The resulting model enhances the performance of the state-of-the-art literature, and it could be used to build an accurate clinical decision support tool that can assist domain experts for AD progression detection.

An ensemble deep learning based IDS for IoT using Lambda architecture

The Internet of Things (IoT) has revolutionized our world today by providing greater levels of accessibility, connectivity and ease to our everyday lives. It enables massive amounts of data to be traversed across multiple heterogeneous devices that are all interconnected. This phenomenon makes IoT networks vulnerable to various network attacks and intrusions. Building an Intrusion Detection System (IDS) for IoT networks is challenging as they enable a massive amount of data to be aggregated, which is difficult to handle and analyze in real time mainly because of the heterogeneous nature of IoT devices. This inefficient, traditional IDS approach accentuates the need to develop advanced IDS techniques by employing Machine or Deep Learning. This paper presents a deep ensemble-based IDS using Lambda architecture by following a multi-pronged classification approach. Binary classification uses Long Short Term Memory (LSTM) to differentiate between malicious and benign traffic, while the multi-class classifier uses an ensemble of LSTM, Convolutional Neural Network and Artificial Neural Network classifiers to detect the type of attacks. The model training is performed in the batch layer, while real-time evaluation is carried out through model inferences in the speed layer of the Lambda architecture. The proposed approach gives high accuracy of over 99.93% and saves useful processing time due to the multi-pronged classification strategy and using the lambda architecture.

Dimensionality reduction and ensemble of LSTMs for antimicrobial resistance prediction

Bacterial resistance to antibiotics has been rapidly increasing, resulting in low antibiotic effectiveness even treating common infections. The presence of resistant pathogens in environments such as a hospital Intensive Care Unit (ICU) exacerbates the critical admission-acquired infections. This work focuses on the prediction of antibiotic resistance in Pseudomonas aeruginosa nosocomial infections at the ICU, using Long Short-Term Memory (LSTM) artificial neural networks as the predictive method. The analyzed data were extracted from the Electronic Health Records (EHR) of patients admitted to the University Hospital of Fuenlabrada from 2004 to 2019 and were modeled as Multivariate Time Series. A data-driven dimensionality reduction method is built by adapting three feature importance techniques from the literature to the considered data and proposing an algorithm for selecting the most appropriate number of features. This is done using LSTM sequential capabilities so that the temporal aspect of features is taken into account. Furthermore, an ensemble of LSTMs is used to reduce the variance in performance. Our results indicate that the patient’s admission information, the antibiotics administered during the ICU stay, and the previous antimicrobial resistance are the most important risk factors. Compared to other conventional dimensionality reduction schemes, our approach is able to improve performance while reducing the number of features for most of the experiments. In essence, the proposed framework achieve, in a computationally cost-efficient manner, promising results for supporting decisions in this clinical task, characterized by high dimensionality, data scarcity, and concept drift.

Unscrambling Customer Recommendations: A Novel LSTM Ensemble Approach in Airline Recommendation Prediction Using Online Reviews

Customer feedback is an essential criterion for upcoming customers to learn from their experience with a company’s products. Customer reviews and ratings also help companies improve performance and figure out new methodologies to provide better services. This research concentrates on customer reviews and ratings to investigate which product a customer evaluates and its association with its recommendations. This work predicts the user recommendations in two modules. The first module performs sentiment analysis of customer reviews using the long short-term memory (LSTM) model, which estimates the probability of the customer’s sentiment about the airline’s services. The second module experimented over only various service aspect ratings on different airline services provided by customers. These two modules ensemble together to determine the predictive recommendations of the airlines. The obtained results reinforce the essential theoretical contribution to the literature on service appraisal, online review, and recommendations. In addition, our proposed ensemble approach will be helpful to those practitioners who wish to use any proposal that will provide a quick and essential vision by bringing together customer-generated reviews and ratings, thereby helping them in strategy designing, service improvement, and post-purchases planning. Also, forthcoming travelers may benefit from this proposed approach by assimilating an aggregating view of service quality.

Computationally expedient Photovoltaic power Forecasting: A LSTM ensemble method augmented with adaptive weighting and data segmentation technique

Photovoltaics (PVs) hold the promise of sustainable electricity production. However, PV output is significantly influenced by variations in terrestrial solar radiation and other weather factors, causing problems in unit commitment, economic power dispatch and reliable electricity distribution from grid-connected, hybrid and off-grid PV plants. Thus, accurate and consistent PV output prediction is crucial for utilities companies and researchers. Current initiatives enlist a plethora of forecasting techniques, including: statistical models, physical models, artificial neural networks and deep learning algorithms. Nevertheless, most of these approaches suffer from computational costs, complex models and uncertainties. Hence, this research employed an ensemble-based Long Short-Term Memory (LSTM) algorithm comprising 10 component LSTM models. The method compares the effects of different data segmentations (three-months to one-day) and is based on varying time-horizons (14-days to 5-mins) in order to compare the effects of seasonal and periodic variations on time-series data and PV output forecast. The comparison is then used to minimise uncertainty by implementing grid search technique. Subsequently, the model’s performance was evaluated using MAPE analysis for two cases involving erratic weather conditions and PV output in two specific days of the year 2020. From the evaluation, the best prediction was found to be for the two-week dataset with MAPE of 6.02. This approach combined with online data acquisition from the Yulara Solar System power plant in central Australia leads to a more practical, computationally economical and robust PV power generation forecasting technique.

Personal Identification Using an Ensemble Approach of 1D-LSTM and 2D-CNN with Electrocardiogram Signals

Conventional personal identification methods (ID, password, authorization certificate, etc.) entail various issues, including forgery or loss. Technological advances and the diffusion across industries have enhanced convenience; however, privacy risks due to security attacks are increasing. Hence, personal identification based on biometrics such as the face, iris, fingerprints, and veins has been used widely. However, biometric information including faces and fingerprints is difficult to apply in industries requiring high-level security, owing to tampering or forgery risks and recognition errors. This paper proposes a personal identification technique based on an ensemble of long short-term memory (LSTM) and convolutional neural network (CNN) that uses electrocardiograms (ECGs). An ECG uses internal biometric information, representing the heart rate in signals using microcurrents and thereby including noises during measurements. This noise is removed using filters in a preprocessing step, and the signals are divided into cycles with respect to R-peaks for extracting features. LSTM is used to perform personal identification using ECG signals; 1D ECG signals are transformed into the time–frequency domain using STFT, scalogram, FSST, and WSST; and a 2D-CNN is used to perform personal identification. This ensemble of two models is used to attain higher performances than LSTM or 2D-CNN. Results reveal a performance improvement of 1.06–3.75%.

An Enhanced Ensemble of Long Short-Term Memory and Vector AutoRegression for Energy Consumption Forecasting

Objectives: To design and develop an enhanced ensemble model for residential energy consumption prediction using time series analysis. Methods: The system is consistent of an ensemble made from two time series models, which are later combined to produce a final prediction through the use of bagging techniques. The energy profile is built for this study using the data collected from a single-phase Minion Energy Monitor, installed in residential buildings, which is developed by Minion Labs India Private Limited. Samples are collected for a time of 7 days with a two second interval. This data is then restructured and normalized for it to fit the enhanced ensemble model of Long Short-Term Memory (LSTM) and Vector Autoregression (VAR) using bagging techniques and weighted average to obtain the predictions.Findings: The proposed model has produced an enhanced R2 score of 98.99% when compared to LSTM (74.85%), SVM (62.41%), VAR (82.914%) and ARIMA (93.152%) standalone models. It makes use of an analyzed lag variable to reduce computation complexity and resource utilization. Further, an ensemble technique is used to pick out the strengths of two models. An analysis performed showed that this architecture is 6.08% better than the algorithms for LSTM and VAR individually. Novelty: A data driven solution is proposed in this study through the enhancement of existing models to create an ensemble and thereby creating a stable structure that predicts values using a weighted average. The weighted average ensures that precise outputs are obtained by giving more importance to predictions that are closer to the actual values. The use of a lag variable further increases the efficiency, learning rate, dealing with non-linear features, less error and faster training of the proposed architecture. All these factors also aid in improving the accuracy for a time series data prediction. Keywords: Energy Consumption Forecast; LSTM; VAR; Ensemble; Bagging; Artificial Neural Networks; Support Vector Machine

Forecasting Daily Tourism Demand for Tourist Attractions with Big Data: An Ensemble Deep Learning Method

Accurate forecasting of daily tourism demand is a meaningful and challenging task, but studies on this issue are scarce. To address this issue, multisource time series data, relating to past tourist volumes, web search information, daily weather conditions, and the dates of public holidays, are selected as the forecasting variables. To fully capture the relationship between these forecasting variables and actual tourism demand automatically, an ensemble of long short-term memory (LSTM) networks is proposed with a correlation-based predictor selection (CPS) algorithm. The effectiveness of the proposed method is verified in daily tourism demand forecasting for the Huangshan Mountain Area, benchmarked against 11 forecasting methods. This study contributes to the literature by (1) introducing the use of big data in daily tourism demand forecasting, (2) proposing an ensemble of LSTM networks for daily tourism demand forecasting, and (3) providing an effective predictor selection algorithm in ensemble learning.

Differential radial basis function network for sequence modelling

We propose a differential radial basis function (RBF) network termed RBF-DiffNet—whose hidden layer blocks are partial differential equations (PDEs) linear in terms of the RBF—to make the baseline RBF network robust to noise in sequential data. Assuming that the sequential data derives from the discretisation of the solution to an underlying PDE, the differential RBF network learns constant linear coefficients of the PDE, consequently regularising the RBF network by following modified backward-Euler updates. We experimentally validate the differential RBF network on the logistic map chaotic timeseries as well as on 30 real-world timeseries provided by Walmart in the M5 forecasting competition. The proposed model is compared with the normalised and unnormalised RBF networks, ARIMA, and ensembles of multilayer perceptrons (MLPs) and recurrent networks with long short-term memory (LSTM) blocks. From the experimental results, RBF-DiffNet consistently shows a marked reduction in the prediction error over the baseline RBF network (e.g., 41% reduction in the root mean squared scaled error on the M5 dataset, and 53% reduction in the mean absolute error on the logistic map); RBF-DiffNet also shows a comparable performance to the LSTM ensemble but requires 99% less computational time. Our proposed network consequently enables more accurate predictions—in the presence of observational noise—in sequence modelling tasks such as timeseries forecasting that leverage the model interpretability, fast training, and function approximation properties of the RBF network.

Combining LSTM Network Ensemble via Adaptive Weighting for Improved Time Series Forecasting

Time series forecasting is essential for various engineering applications in finance, geology, and information technology, etc. Long Short-Term Memory (LSTM) networks are nowadays gaining renewed interest and they are replacing many practical implementations of the time series forecasting systems. This paper presents a novel LSTM ensemble forecasting algorithm that effectively combines multiple forecast (prediction) results from a set of individual LSTM networks. The main advantages of our LSTM ensemble method over other state-of-the-art ensemble techniques are summarized as follows: (1) we develop a novel way of dynamically adjusting the combining weights that are used for combining multiple LSTM models to produce the composite prediction output; for this, our method is devised for updating combining weights at each time step in an adaptive and recursive way by using both past prediction errors and forgetting weight factor; (2) our method is capable of well capturing nonlinear statistical properties in the time series, which considerably improves the forecasting accuracy; (3) our method is straightforward to implement and computationally efficient when it comes to runtime performance because it does not require the complex optimization in the process of finding combining weights. Comparative experiments demonstrate that our proposed LSTM ensemble method achieves state-of-the-art forecasting performance on four real-life time series datasets publicly available.

An Ensemble of LSTM Neural Networks for High-Frequency Stock Market Classification

We propose an ensemble of Long-Short Term Memory (LSTM) Neural Networks for intraday stock predictions, using a large variety of Technical Analysis indicators as network inputs. The proposed ensemble operates in an online way, weighting the individual models proportionally to their recent performance, which allows us to deal with possible non-stationarities in an innovative way. The performance of the models is measured by Area Under the Curve of the Receiver Operating Characteristic. We evaluate the predictive power of our model on several US large-cap stocks and benchmark it against Lasso and Ridge logistic classifiers. The proposed model is found to perform better than the benchmark models or equally weighted ensembles.

Deep learning for automatic stereotypical motor movement detection using wearable sensors in autism spectrum disorders

Autism Spectrum Disorders are associated with atypical movements, of which stereotypical motor movements (SMMs) interfere with learning and social interaction. The automatic SMM detection using inertial measurement units (IMU) remains complex due to the strong intra and inter-subject variability, especially when handcrafted features are extracted from the signal. We propose a new application of the deep learning to facilitate automatic SMM detection using multi-axis IMUs. We use a convolutional neural network (CNN) to learn a discriminative feature space from raw data. We show how the CNN can be used for parameter transfer learning to enhance the detection rate on longitudinal data. We also combine the long short-term memory (LSTM) with CNN to model the temporal patterns in a sequence of multi-axis signals. Further, we employ ensemble learning to combine multiple LSTM learners into a more robust SMM detector. Our results show that: (1) feature learning outperforms handcrafted features; (2) parameter transfer learning is beneficial in longitudinal settings; (3) using LSTM to learn the temporal dynamic of signals enhances the detection rate especially for skewed training data; (4) an ensemble of LSTMs provides more accurate and stable detectors. These findings provide a significant step toward accurate SMM detection in real-time scenarios.