Long Short-term Memory Module Research Articles

Physics-guided TL-LSTM network for early-stage degradation trajectory prediction of lithium-ion batteries

Predicting early degradation trajectories of lithium-ion batteries is crucial in enhancing system reliability and promoting battery technology advancement. Existing data-driven methods often require large amounts of historical data and similar cycle information, which are not easily accessible in real-world applications. To this end, a physics-guided TL-LSTM network is proposed for degradation trajectory prediction, which integrates the physical degradation property into a transfer learning framework. Specifically, we first build a backbone network that combines a Long Short-Term Memory (LSTM) module and a Fully Connected (FC) module to perform fine-grained model training on the source domain. Then, the well-trained LSTM module is frozen and transferred to the target domain, and the FC module is fine-tuned using 30% of its early-cycle data. The entire network optimization process is guided by two inherent properties of battery degradation. Extensive comparative analysis demonstrates that our proposed approach outperforms existing methods in predicting early degradation trajectories.

A groundwater level spatiotemporal prediction model based on graph convolutional networks with a long short-term memory

ABSTRACT The performance of regional groundwater level (GWL) prediction model hinges on understanding intricate spatiotemporal correlations among monitoring wells. In this study, a graph convolutional network (GCN) with a long short-term memory (LSTM) (GCN–LSTM) model is introduced for GWL prediction utilizing data from 16 wells located in the northeastern Xiangtan City, China. This model is designed to account for both the hybrid temporal dependencies and spatial autocorrelations among wells. It consists of two parts: the spatial part employs GCNs to extract spatial characteristics from a spatial self-similarity weight matrix and an attribute self-similarity weight matrix among wells; the temporal part utilizes a LSTM module to capture the temporal patterns of GWL sequences, along with monthly precipitation and temperature data. This model dynamically predicts changes in groundwater levels, achieving higher accuracy on average compared to single-well predictions using LSTM. By incorporating both temporal dependencies and spatial autocorrelations, the GCN–LSTM model demonstrated an average improvement in goodness-of-fit of approximately 11.21% over the LSTM-based model for individual wells. Its application holds significant reference value for the sustainable utilization and development of groundwater resources in Xiangtan City.

Using acoustic emission technique for structural health monitoring of laminate composite: A novel CNN-LSTM framework

This research has developed a real-time end-to-end deep learning model for structural health monitoring (SHM) method for composite impact damage diagnosis based on Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM). The acoustic emission (AE) signals collected under low-velocity impacts by means of piezoelectric sensors on composite materials are used for training deep learning networks. Based on the impact load curves, specimens are categorized into minor failure, intermediate failure, and severe failure. The convolved signals are segmented and reconstructed at a given length for the following LSTM module. The average accuracies for basic CNN, CNN– Recurrent Neural Network (RNN), CNN-LSTM, and CNN– Gated Recurrent Unit (GRU) are respectively 88.7 %, 92.6 %, 98 %, and 95.4 %. A sensitivity analysis on sub-signal length was conducted on the CNN-LSTM model, revealing that the model achieved its best performance when the sub-signal length was set at 16. The model attained prediction accuracies of 97.4 %, 100 %, and 100 %, respectively, for minor failure, intermediate failure, and severe failure cases.

Explore Bayesian analysis in Cognitive-aware Key–Value Memory Networks for knowledge tracing in online learning

Knowledge Tracing is a crucial aspect of personalized learning that aims to track the evolving knowledge states of students with respect to one or more concepts. However, existing literature either models knowledge state for each predefined concept separately, potentially overlooking the underlying interrelationships between concepts, or lacks the capability to uncover long-term dependencies within a historical sequence of questions. Furthermore, they have limitations in precisely identifying the mastery level of individual learners for specific concepts. In order to address these problems, we propose a novel deep learning-based knowledge tracing model, named Bayesian Cognitive-aware Key–Value Memory Network (BCKVMN). BCKVMN combines the strengths of Dynamic Key–Value Memory Network (DKVMN), Long Short-Term Memory (LSTM), embedding layer, and Bayesian theory. To capture long-term dependencies in question sequences and contextual information, we extend DKVMN by incorporating a standard LSTM module that integrates cognitive processes observed in human learning. Additionally, we employ an embedding layer to uncover the influence of question difficulty on final predictions. Furthermore, we introduce Bayesian theory to enhance parameter interpretation for the model. To investigate complex representation of human learning, our approach considers not only three distinct learning states during the learning process but also the relationships between underlying concepts and students’ mastery levels for each concept. Experimental evaluations are conducted on the three real-world datasets Algebra200, ASSISTMENTS2009 and ASSISTMENTS2017. The results have demonstrated that the proposed model outperforms state-of-the-art methods (achieving nearly 7% improvement in AUC in some cases) while achieving a better balance between performance and interpretability.

S[formula omitted]T-Net: A novel electroencephalogram signals-oriented emotion recognition model

In this paper, a novel skipping spatial–spectral–temporal network (S3T-Net) is developed to handle intra-individual differences in electroencephalogram (EEG) signals for accurate, robust, and generalized emotion recognition. In particular, aiming at the 4D features extracted from the raw EEG signals, a multi-branch architecture is proposed to learn spatial–spectral cross-domain representations, which benefits enhancing the model generalization ability. Time dependency among different spatial–spectral features is further captured via a bi-directional long-short term memory module, which employs an attention mechanism to integrate context information. Moreover, a skip-change unit is designed to add another auxiliary pathway for updating model parameters, which alleviates the vanishing gradient problem in complex spatial–temporal network. Evaluation results show that the proposed S3T-Net outperforms other advanced models in terms of the emotion recognition accuracy, which yields an performance improvement of 0.23% , 0.13%, and 0.43% as compared to the sub-optimal model in three test scenes, respectively. In addition, the effectiveness and superiority of the key components of S3T-Net are demonstrated from various experiments. As a reliable and competent emotion recognition model, the proposed S3T-Net contributes to the development of intelligent sentiment analysis in human–computer interaction (HCI) realm.

An Improved CNN-BILSTM Model for Power Load Prediction in Uncertain Power Systems

Power load prediction is fundamental for ensuring the reliability of power grid operation and the accuracy of power demand forecasting. However, the uncertainties stemming from power generation, such as wind speed and water flow, along with variations in electricity demand, present new challenges to existing power load prediction methods. In this paper, we propose an improved Convolutional Neural Network–Bidirectional Long Short-Term Memory (CNN-BILSTM) model for analyzing power load in systems affected by uncertain power conditions. Initially, we delineate the uncertainty characteristics inherent in real-world power systems and establish a data-driven power load model based on fluctuations in power source loads. Building upon this foundation, we design the CNN-BILSTM model, which comprises a convolutional neural network (CNN) module for extracting features from power data, along with a forward Long Short-Term Memory (LSTM) module and a reverse LSTM module. The two LSTM modules account for factors influencing forward and reverse power load timings in the entire power load data, thus enhancing model performance and data utilization efficiency. We further conduct comparative experiments to evaluate the effectiveness of the proposed CNN-BILSTM model. The experimental results demonstrate that CNN-BILSTM can effectively and more accurately predict power loads within power systems characterized by uncertain power generation and electricity demand. Consequently, it exhibits promising prospects for industrial applications.

Research on lung sound classification model based on dual-channel CNN-LSTM algorithm

ulmonary diseases have a significant impact on human health and life safety, and abnormalities in the lungs are a direct response to lung diseases. Establishing an effective lung sound classification model that can assist in diagnosis is of great significance for electronic auscultation.In addressing the issue of lung sound signal classification, this study introduces a deep learning classification model based on a dual-channel CNN-LSTM algorithm. Initially, Mel-scale Frequency Cepstral Coefficients (MFCC) are employed for feature extraction from the dataset, transforming lung sound signals into Mel spectrograms. On this foundation, a dual-channel algorithm classification model is constructed, with parallel Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) modules. The CNN module is designed to capture spatial dimension features of the input data, while the LSTM module focuses on temporal dimension features. These two feature sets are fused together, enabling the model to classify lung sounds and thereby assisting in diagnosing pulmonary diseases for healthcare practitioners. This experiment used the ICBHI2017 Challenge Lungs dataset and obtained 5054 pieces of data through data augmentation and sampling techniques.The results show that the accuracy, recall, and F1 score of this model reach 99.01%, 99.13%, and 0.9915, respectively, significantly superior to other models, highlighting its practical application value.

MetaKernel: Learning Variational Random Features With Limited Labels.

Few-shot learning deals with the fundamental and challenging problem of learning from a few annotated samples, while being able to generalize well on new tasks. The crux of few-shot learning is to extract prior knowledge from related tasks to enable fast adaptation to a new task with a limited amount of data. In this paper, we propose meta-learning kernels with random Fourier features for few-shot learning, we call MetaKernel. Specifically, we propose learning variational random features in a data-driven manner to obtain task-specific kernels by leveraging the shared knowledge provided by related tasks in a meta-learning setting. We treat the random feature basis as the latent variable, which is estimated by variational inference. The shared knowledge from related tasks is incorporated into a context inference of the posterior, which we achieve via a long-short term memory module. To establish more expressive kernels, we deploy conditional normalizing flows based on coupling layers to achieve a richer posterior distribution over random Fourier bases. The resultant kernels are more informative and discriminative, which further improves the few-shot learning. To evaluate our method, we conduct extensive experiments on both few-shot image classification and regression tasks. A thorough ablation study demonstrates that the effectiveness of each introduced component in our method. The benchmark results on fourteen datasets demonstrate MetaKernel consistently delivers at least comparable and often better performance than state-of-the-art alternatives.

Flood prediction using nonlinear instantaneous unit hydrograph and deep learning: A MATLAB program

In this study, we developed a MATLAB program for flood prediction in a watershed. The program consists of three modules. The instantaneous unit hydrograph (IUH) generation module utilizes a power-law based interpolation method to generate IUHs. The generated IUH is a function of the rainfall excess intensity and therefore considers nonlinearity. The long short-term memory (LSTM) module employs “lstmLayer” from the MATLAB deep learning toolbox to predict total rainfall excess; this is then used to estimate the curve number (CN) value for each flood event. The LSTM module uses a land surface modeling dataset and rainfall-runoff data as inputs. The flood hydrograph generation module calculates effective rainfall hyetographs and then predicts flood hydrographs using a convolution integration. A detailed description of the program is provided along with an application example for real watersheds. The application results demonstrated that our program can be effectively used for flood prediction in practice, particularly for large flood events.

A hybrid cloud load balancing and host utilization prediction method using deep learning and optimization techniques

Virtual machine (VM) integration methods have effectively proven an optimized load balancing in cloud data centers. The main challenge with VM integration methods is the trade-off among cost effectiveness, quality of service, performance, optimal resource utilization and compliance with service level agreement violations. Deep Learning methods are widely used in existing research on cloud load balancing. However, there is still a problem with acquiring noisy multilayered fluctuations in workload due to the limited resource-level provisioning. The long short-term memory (LSTM) model plays a vital role in the prediction of server load and workload provisioning. This research presents a hybrid model using deep learning with Particle Swarm Intelligence and Genetic Algorithm (“DPSO-GA”) for dynamic workload provisioning in cloud computing. The proposed model works in two phases. The first phase utilizes a hybrid PSO-GA approach to address the prediction challenge by combining the benefits of these two methods in fine-tuning the Hyperparameters. In the second phase, CNN-LSTM is utilized. Before using the CNN-LSTM approach to forecast the consumption of resources, a hybrid approach, PSO-GA, is used for training it. In the proposed framework, a one-dimensional CNN and LSTM are used to forecast the cloud resource utilization at various subsequent time steps. The LSTM module simulates temporal information that predicts the upcoming VM workload, while a CNN module extracts complicated distinguishing features gathered from VM workload statistics. The proposed model simultaneously integrates the resource utilization in a multi-resource utilization, which helps overcome the load balancing and over-provisioning issues. Comprehensive simulations are carried out utilizing the Google cluster traces benchmarks dataset to verify the efficiency of the proposed DPSO-GA technique in enhancing the distribution of resources and load balancing for the cloud. The proposed model achieves outstanding results in terms of better precision, accuracy and load allocation.

Fusion Neural Network for Gas Concentration Prediction in Mixed Gas Environments

Due to the inherent complexity and nonlinearity of mixed gas data, existing pattern recognition algorithms utilized in electronic noses often encounter difficulties in accurately predicting gas concentrations. Addressing this issue, we propose a fusion neural network that merges Long Short-Term Memory (LSTM) and Temporal Convolutional Network (TCN), which we denote as the LSTM-TCN fusion model. The LSTM module effectively captures long-term dependencies in time-series data, while the TCN targets local correlations, thereby enhancing the prediction accuracy for complex gas concentrations. Experimental validation was conducted using a mixed gas dataset comprising ethylene and carbon monoxide. When compared with traditional models, including LSTM, TCN, and GRU, the proposed LSTM-TCN model demonstrated superior performance, achieving an R2 value as high as 0.9922. This research holds considerable practical significance and shows promising application prospects, contributing novel insights and methods to the study and application of electronic nose technology.

An Expressway ETC Missing Data Restoration Model Considering Multi-Attribute Features

Electronic toll collection (ETC) data mining has become one of the hotspots in the research of intelligent expressway extension applications. Ensuring the integrity of ETC data stands as a critical measure in upholding data quality. ETC data are typical structured data, and although deep learning holds great potential in the ETC data restoration field, its applications in structured data are still in the early stages. To address these issues, we propose an expressway ETC missing transaction data restoration model considering multi-attribute features (MAF). Initially, we employ an entity embedding neural network (EENN) to automatically learn the representation of categorical features in multi-dimensional space, subsequently obtaining embedding vectors from networks that have been adequately trained. Then, we use long short-term memory (LSTM) neural networks to extract the changing patterns of vehicle speeds across several continuous sections. Ultimately, we merge the processed features with other features as input, using a three-layer multilayer perceptron (MLP) to complete the ETC data restoration. To validate the effectiveness of the proposed method, we conducted extensive tests using real ETC datasets and compared it with methods commonly used for structured data restoration. The experimental results demonstrate that the proposed method significantly outperforms others in restoration accuracy on two different datasets. Specifically, our sample data size reached around 400,000 entries. Compared to the currently best method, our method improved the restoration accuracy by 19.06% on non-holiday ETC datasets. The MAE and RMSE values reached optimal levels of 12.394 and 23.815, respectively. The fitting degree of the model to the dataset also reached its peak (R2 = 0.993). Meanwhile, the restoration stability of our method on holiday datasets increased by 5.82%. An ablation experiment showed that the EENN and LSTM modules contributed 7.60% and 9% to the restoration accuracy, as well as 4.68% and 7.29% to the restoration stability. This study indicates that the proposed method not only significantly improves the quality of ETC data but also meets the timeliness requirements of big data mining analysis.

A novel deep time‐index framework with limited data and feature extraction for battery capacity degradation prediction

AbstractAccurate capacity estimation and degradation trajectory predictions are crucial part of effectively monitoring the current and future health status of lithium‐ion batteries, which ensures their optimal performance and longevity. However, it is hard to acquire precise predictions under various application conditions when relying on limited historical data. In this paper, we propose a novel deep time‐index framework based on meta‐learning to achieve online capacity estimation and degradation trajectory predictions with only a few monitoring data from early cycles. Highly correlated health feature from lithium‐ion battery degradation data is extracted for online capacity estimation. The deep time‐index framework consists of three parts, Implicit Neural Representations (INRs), Long Short‐Term Memory (LSTM) and Ridge regressor. Cycle numbers are selected as time feature and fed into the INRs to obtain informative representations. A LSTM module is employed to process the temporal representations and capture the long‐term dependencies to Ridge regressor to generate a direct multi‐step predictions. Three distinct lithium‐ion battery degradation datasets are applied to validate the efficacy of the proposed approach. The experimental results validate that within the deep time‐index framework, the short‐term predictions show remarkable performance. This high level of accuracy makes the proposed deep time‐index framework suitable for a wide range of real‐world scenarios, as it demonstrates precise predictions even under varying charging and discharging conditions with limited historical data.

Improved Deep Reinforcement Learning for Intelligent Traffic Signal Control Using ECA_LSTM Network

Reinforcement learning is one of the most widely used methods for traffic signal control, but the method experiences issues with state information explosion, inadequate adaptability to special scenarios, and low security. Therefore, this paper proposes a traffic signal control method based on the efficient channel attention mechanism (ECA-NET), long short-term memory (LSTM), and double Dueling deep Q-network (D3QN), which is EL_D3QN. Firstly, the ECA-NET and LSTM module are included in order to lessen the state space’s design complexity, improve the model’s robustness, and adapt to various emergent scenarios. As a result, the cumulative reward is improved by 27.9%, and the average queue length, average waiting time, and CO2 emissions are decreased by 15.8%, 22.6%, and 4.1%, respectively. Next, the dynamic phase interval tgap is employed to enable the model to handle more traffic conditions. Its cumulative reward is increased by 34.2%, and the average queue length, average waiting time, and CO2 emissions are reduced by 19.8%, 30.1%, and 5.6%. Finally, experiments are carried out using various vehicle circumstances and unique scenarios. In a complex environment, EL_D3QN reduces the average queue length, average waiting time, and CO2 emissions by at least 13.2%, 20.2%, and 3.2% compared to the four existing methods. EL_D3QN also exhibits good generalization and control performance when exposed to the traffic scenarios of unequal stability and equal stability. Furthermore, even when dealing with unique events like a traffic surge, EL_D3QN maintains significant robustness.

Remaining useful life prediction method combining the life variation laws of aero-turbofan engine and auto-expandable cascaded LSTM model

The real-time status monitoring and health management of aero-turbofan engines (ATEs) can effectively reduce the risk of engine failure and ensure aircraft flight safety. Accurate prediction of the remaining useful life (RUL) of ATE is a vital tool for effectively monitoring the engine operating condition, in which long-short term memory (LSTM) networks are often applied to RUL prediction. However, because of the complex mechanical structure and operation mode of the aero-engine, the prediction accuracy of the LSTM network is not enough to meet the actual demand. This paper proposes an auto-expandable cascaded long-short term memory (ACLSTM) prediction model that incorporates the lifetime variation laws of ATE, which is mainly applied for the degradation assessment of ATEs and the accurate prediction of RUL. The ACLSTM model adopts the network structure of multiple LSTM modules connected step by step to continuously set the prediction error of the previous module as the training outputs of the latter module. This data processing method transforms the prediction process of the original data into that of the output error, effectively reducing the prediction error and improving the prediction effect. In addition, to further improve the prediction accuracy, this paper comprehensively proposes several empirical formulas for further correction of the prediction effect obtained by the ACLSTM model. In the experimental part, the prediction effectiveness of the proposed method is tested based on four subsets of the C-MAPSS dataset published by the National Aeronautics and Space Administration. The experimental results on the four datasets show that the root mean square error (RMSE) of the ACLSTM prediction model decreases by 95.44% on average compared to the traditional LSTM network. In addition, the RMSE of the model decreases by 96.48% on average after incorporating the empirical formula. The proposed method has the lowest RMSE compared to other methods with the highest prediction accuracy. The experiments thoroughly verify that the ACLSTM model and the proposed empirical formula are feasible and effective for improving the prediction accuracy of the RUL of ATE.

MTDN: Learning Multiple Temporal Dynamics Representation for Emotional Valence Classification with EEG.

Emotion recognition from electroencephalogram (EEG) requires computational models to capture the crucial features of the emotional response to external stimulation. Spatial, spectral, and temporal information are relevant features for emotion recognition. However, learning temporal dynamics is a challenging task, and there is a lack of efficient approaches to capture such information. In this work, we present a deep learning framework called MTDN that is designed to capture spectral features with a filterbank module and to learn spatial features with a spatial convolution block. Multiple temporal dynamics are jointly learned with parallel long short-term memory (LSTM) embedding and self-attention modules. The LSTM module is used to embed the time segments, and then the self-attention is utilized to learn the temporal dynamics by intercorrelating every embedded time segment. Multiple temporal dynamics representations are then aggregated to form the final extracted features for classification. We experiment on a publicly available dataset, DEAP, to evaluate the performance of our proposed framework and compare MTDN with existing published results. The results demonstrate improvement over the current state-of-the-art methods on the valence dimension of the DEAP dataset.

Supervised Contrastive Learned Deep Model for Question Continuation Evaluation

Question continuation evaluation (QCE) is a branch task of dialogue act prediction (DAP) in the natural language processing area, which is aimed at predicting whether each question in a dialogue is worthy of being followed-up under a specific context. QCE is important for communication, education, and even entertainment. Regrettably, QCE has always been disregarded as an auxiliary task for conversational machine reading comprehension. QCE involves more information and relationships than the original DAP task, making it more complex. Moreover, the classification of QCE inherently renders the samples confusing. In this article, a transformer long short-term memory (LSTM)-based supervised contrastive learned model for QCE is proposed to automatically distribute QCE labels. This model is mainly constructed with transformer encoder blocks and LSTM modules, and supervised contrastive learning (SCL) is innovatively introduced to the training process. This model is good at extracting both information about corpora and the relationships among corpora, and SCL alleviates any confusion. With the only applicable dataset, i.e., Question Answering in Context (QuAC), experiments are conducted. This model is proven to perform well and is robust to missing data. The performance is 2.3% (accuracy) and 12.2% (macro-F1 score) higher than baselines from QuAC and only decreases by approximately 2.3% when 10% data remain.

Unified Framework for Identity and Imagined Action Recognition From EEG Patterns

We present a unified deep learning framework for the recognition of user identity and the recognition of imagined actions, based on electroencephalography (EEG) signals, for application as a brain–computer interface. Our solution exploits a novel shifted subsampling preprocessing step as a form of data augmentation, and a matrix representation to encode the inherent local spatial relationships of multielectrode EEG signals. The resulting image-like data are then fed to a convolutional neural network to process the local spatial dependencies, and eventually analyzed through a bidirectional long-short term memory module to focus on temporal relationships. Our solution is compared against several methods in the state of the art, showing comparable or superior performance on different tasks. Specifically, we achieve accuracy levels above 90% both for action and user classification tasks. In terms of user identification, we reach 0.39% equal error rate in the case of known users and gestures, and 6.16% in the more challenging case of unknown users and gestures. Preliminary experiments are also conducted in order to direct future works toward everyday applications relying on a reduced set of EEG electrodes.

Development and Experimental Verification of an Intelligent Isolation System Based on Long Short-Term Memory Module Model for Ground Motion Characteristics Prediction

Semiactive seismic control requires appropriate control laws that dictate the behavior of seismic suppression devices based on measurements and feedback during an earthquake. Optimal control parameters should be determined in real time to achieve high performance. The initial ground motion characteristics of an earthquake substantially affect control performance. In this study, a genetic algorithm (GA)-optimized long short-term memory (LSTM)-based intelligent control system (hereafter denoted GA-LSTM system) for obtaining optimal control parameters for a semiactive variable-stiffness isolation system was proposed. First, the LSTM module classifies earthquakes as near-fault or far-field events to determine the optimal control strategy. Second, earthquakes from a global database are analyzed to determine a fuzzy inference surface for the optimal parameters of the earthquake suppression system. Both numerical simulations and shaking table experimental results indicated that the proposed GA-LSTM system exhibited superior isolation displacement and superstructure acceleration suppression for both near-fault and far-field earthquakes when compared with other control techniques. The proposed intelligent control system is highly efficient and could reliably protect structures from earthquakes with any ground motion characteristics.

A Deep Learning Model for Ship Trajectory Prediction Using Automatic Identification System (AIS) Data

The rapid growth of ship traffic leads to traffic congestion, which causes maritime accidents. Accurate ship trajectory prediction can improve the efficiency of navigation and maritime traffic safety. Previous studies have focused on developing a ship trajectory prediction model using a deep learning approach, such as a long short-term memory (LSTM) network. However, a convolutional neural network (CNN) has rarely been applied to extract the potential correlation among different variables (e.g., longitude, latitude, speed, course over ground, etc.). Therefore, this study proposes a deep-learning-based ship trajectory prediction model (namely, CNN-LSTM-SE) that considers the potential correlation of variables and temporal characteristics. This model integrates a CNN module, an LSTM module and a squeeze-and-excitation (SE) module. The CNN module is utilized to extract data on the relationship among different variables (e.g., longitude, latitude, speed and course over ground), the LSTM module is applied to capture temporal dependencies, and the SE module is introduced to adaptively adjust the importance of channel features and focus on the more significant ones. Comparison experiments of two cargo ships at a time interval of 10 s show that the proposed CNN-LSTM-SE model can obtain the best prediction performance compared with other models on evaluation indexes of average root mean squared error (ARMSE), average mean absolute percentage error (AMAPE), average Euclidean distance (AED), average ground distance (AGD) and Fréchet distance (FD).