Long Short Time Memory Research Articles

DualLSTM: A novel key-quality prediction for a hierarchical cone thickener

Due to the inaccuracy and significant disturbance of the complex and harsh environment in real industrial processes, the traditional sensor devices cannot meet the high-performance requirement of measuring key quality variables. However, in practical industrial thickener cone systems, the underflow concentration is hard to measure and has a high cost and significant time delay. Furthermore, the higher encoder representation often causes information loss from the process variables. This paper presents a novel efficient dual long short-time memory (LSTM) method for concentration prediction in the deep cone thickener system. To this end, dual feedforward and inverse bidirectional long-short time memory are proposed for feature learning and long temporal prediction. The proposed framework introduces an averaging moving filtering to pass through features, therefore the performance of dual LSTM is increased by a large margin. In addition, the feedforward and reverse bidirectional LSTM are employed to learn the robust information without loss. At last, experimental verification of the performance of an industrial deep cone thickener demonstrates the proposed dual LSTM method outperforms other state-of-the-art methods.

A Series of Models based on Long Short Time Memory for Temperature Prediction

Given the increasingly serious climate problems that have been appearing around the world in recent years, as part of the climate research, finding ways of smart, accurate machine learning to carry out the temperature prediction is of great significance both for human activities and the earth's ecosystem. In this study, 3 kinds of the Long Short Time Memory (LSTM) models, which differs on the way how the data is input and output and the network structure, respectively Directly-Multiple-Output (DMO), Single-Step-Scrolling (SSS), and Convolutional Neural Networks Plus Long Short Time Memory (CNN+LSTM) are built and trained by Pytorch on the Jena Climate dataset to compare their predicted performance. The training loss of these 3 models are 0.0053, 0.1568 and 0. 0079; testing loss are 0.0048, 0.1764 and 0.0096; Mean Absolute Percentage Error (MAPE) are 3.42%,6.58% and 3.30%. The result turns out that CNN+LSTM is the best model in comprehensive consideration with the least convergence time, low loss and MAPE, followed by the DMO model with the least loss and low MAPE but longer convergence time. SSS performs worst with increasing high loss. In general, CNN+LSTM is suitable for temperature prediction while DMO is also good when short convergence time is not needed. SSS is not recommended for temperature prediction.

Deep learning assisted surrogate modeling of large-scale power grids

The model order reduction (MOR) enterprise has shown unprecedented applications in the area of power systems by allowing tractable and realistic simulations of complex power grids. Nearly all model reduction techniques rely on constructing a linear reduced trial subspace followed by a Galerkin projection to restrict the evolution of the latent dynamics onto the reduced subspace. However, power system models often exhibit a slow decay of singular values and, as such, are often approximated poorly due to the truncation of higher-order modes. Furthermore, employing a linear projection for such models places a fundamental limit on the accuracy of the reduced order models (ROMs). This paper uses techniques from machine learning to develop a dimensionality reduction framework for large-scale power grid models to overcome these limitations effectively. In particular, we use autoencoder (AE) network to learn low-dimensional nonlinear trial manifold to avoid the limitation of a linear trial subspace. This is followed by the use of long short-time memory (LSTM) networks to obtain the evolution of the reduced dynamics in a non-intrusive manner. The resulting framework yields ROMs that are compact and are several orders of magnitude more accurate than the linear projection-based methods. We demonstrate the proposed technique on the standard IEEE 118-Bus system, which has 54 generators, and the European high-voltage system with 260 generators.

Application of Market Cycle Analysis and LSTM in Prediction of Stock Price Movements

The stock market prediction has been carried out by several ways in data science using deep learning approaches to capture profitable trading opportunities and making the trading plans. However, it is widely believed there are two main issues involved in it, i.e., efficient market hypothesis and low information noise ratio. Therefore, a prediction based model will be affected by noises thus hard to produce a prediction. In this paper, two methods will be presented for forecasting stock future performance. To be specific, LSTM (long-short time memory) and cycle analysis are implemented to predict the future period that gives a higher return than average times. According to the analysis, introducing the time analysis as a variable to input could significantly increase the accuracy of predicting the return for the next few weeks. These results shed light on guiding further exploration of the different ways of extracting periodic behaviors of the market and marking predictions based on the analysis.

Trajectory prediction in pipeline form for intercepting hypersonic gliding vehicles based on LSTM

The interception problem of Hypersonic Gliding Vehicles (HGVs) has been an important aspect of missile defense systems. In order to provide interceptors with accurate information of target trajectory, a model based on an improved Long Short-Time Memory (LSTM) network for trajectory prediction pipeline is proposed for the interception of a skip gliding hypersonic target. Firstly, for trajectory prediction required by intercepting guidance laws, the altitude, velocity and velocity direction of the target are formulated in the form of analytic functions, consisting of linear decay terms and amplitude decay sinusoidal terms. Then, the dynamic characteristics of the model parameters are analyzed, and the target trajectory prediction pipeline is proposed with the prediction error considered. Finally, an improved LSTM network is designed to estimate parameters in a dynamically-updated manner, and estimation results are used for the calculation of the final trajectory prediction pipeline. The proposed prediction algorithm provides information on the velocity vector for midcourse guidance with the effect of prediction errors on interception taken into account. Simulation is conducted and the results show the high accuracy of the algorithm in HGVs’ trajectory prediction which is conducive to increasing the interception success rate.

An Intelligent Internet-of-Things-Aided Financial Crisis Prediction Model in FinTech

Financial technology (FinTech) has become a hot research topic recently, moreover, an area of major investment for most financial institutions. As the FinTech unleashed many new strategic solutions for major financial problems. One of those pivotal strategies of FinTech is financial crisis prediction (FCP) that dictates the financial status of an institution. Also, the rise of the Internet-of-Things (IoT) technology has paved a new way for interaction between humans and the physical world. Therefore, IoT can feasibly be incorporated into the FCP model to obtain a real-time analysis of the financial data from the clients. With this perspective, we propose an intelligent IoT-aided FCP model using metaheuristic algorithms. The proposed FCP method comprises data acquisition, preprocessing, feature selection (FS), and classification. First, the financial data of the enterprises are collected using IoT devices, such as smartphones, laptops, etc. Next, the quantum artificial butterfly optimization (QABO) approach for FS is applied to choose an optimal set of features. Afterward, long short-time memory (LSTM) with recurrent neural network (RNN) model is employed to classify the collected financial data. An exhaustive experimental validation process is carried out to ensure the performance of the proposed QABO-LSTM-RNN model. The simulation results accredited the efficacy of the proposed model contrasted with other baseline methods in terms of best cost, sensitivity, specificity, accuracy, F-score, kappa, and Mathew correlation coefficient (MCC).

Prediction of the skid-resistance deterioration in asphalt pavement based on peephole–LSTM neural network

ABSTRACT Poor pavement skid resistance contributes to frequent traffic accidents, with the asphalt surface rapidly deteriorating over time. Accurate prediction of skid resistance behavior is essential for road safety. This study introduces a pavement accelerated polishing system, collecting data from 11 stages using a British pendulum tester and a high-precision three-dimensional laser scanner. It develops a predictive model, FrictionNet-II, based on a peephole long-short time memory (peephole-LSTM) neural network. The model uses data from the first eight polishing stages as input and the last three as output, allowing for predictions of pavement skid resistance after 160,000, 180,000, and 200,000 cycles based on the friction deterioration curve of the initial stages (0 to 140,000 cycles). FrictionNet-II accurately forecasts skid resistance after 160,000, 180,000, and 200,000 cycles, achieving R2 values of 88%, 90%, and 90%, respectively. This research demonstrates the potential of deep learning in predicting asphalt pavement skid resistance, laying the foundation for intelligent pavement preservation and improved skid-resistance performance.

Classifying animal behavior from accelerometry data via recurrent neural networks

We study the classification of animal behavior from accelerometry data using various recurrent neural network (RNN) models. RNNs have extensively been employed to classify time-series data in various applications. However, their utilization for classifying animal behavior from wearable sensor data, particularly accelerometry data, as well as the underlying accuracy-complexity trade-offs is rather under-explored. We use four triaxial accelerometry datasets acquired from grazing cattle via collar tags or ear tags to evaluate the classification performance and complexity of the considered RNN models, which feature long short-time memory (LSTM) or gated recurrent unit (GRU) architectures with varying depths and widths. In the evaluations, we also include two state-of-the-art convolutional neural network (CNN)-based time-series classification models. The results show that the considered RNN-based models can achieve similar or higher animal behavior classification accuracy compared to the CNN-based models while having smaller computational and memory requirements. We also observe that the GRU-based models generally outperform the LSTM-based ones in terms of classification accuracy despite being less complex. A single-layer uni-directional GRU model with 64 hidden units appears to offer a good balance between accuracy and complexity making it suitable for implementation on edge/embedded devices.

DEEP REINFORCEMENT LEARNING WITH HIDDEN MARKOV MODEL FOR SPEECH RECOGNITION

Nowadays, many applications uses speech recognition especially the field of computer science and electronics, Speech Recognition (SR) is the interpretation of words spoken into a text. It is also known as Speech-To-Text (STT) or Automatic-Speech-Recognition(ASR), or just Word-Recognition(WR). The Hidden-Markov-Model (HMM) is a type of Markov model, which means that the future state of the model depends on the current state, not on the entire history of the system and the goal of HMM is to learn a sequence of hidden states from a set of known states. The Long-Short-Time-Memory (LSTM) network is a type of Recurrent Neural Network (RNN) that can learn long-term dependencies between time steps of sequence data. The LSTM network is trained by the network in order to predict the values of subsequent time steps in a series-to-series regression. Deep Neural Network (DNN) models are better classifiers than Gaussian Mixture Models (GMMs), they can generalize much better with a smaller number of parameters over complex distributions. They model distributions of different classes jointly, called “distributed” learning, or, more properly “tied” learning. This work is aimed at developing a speech recognition model that will predict isolated speech of some selected fruits in Hausa, Igbo and Yoruba language by using the predicting power of Mel-Frequency-Cepstral-Coefficient (MFCC), LSTM and HMM algorithms. The findings of the study would improve the development of better automatic speech applications systems and would benefit the academic and research community in the field of Natural Language Processing.

Abnormal Dynamic Recognition of Space Targets From ISAR Image Sequences With SSAE-LSTM Network

Abnormal dynamics awareness of space targets is critical for space surveillance. With the intrinsic range-Doppler projection mechanism, an inverse synthetic aperture radar (ISAR) image sequence naturally has crucial potential for abnormal dynamics interpretation on uncooperative targets. In this paper, we develop an automated deep neural network architecture for abnormal dynamic recognition of space targets from ISAR image sequences. With the accommodation of the stacked sparse autoencoder (SSAE) network joint sparsity constraint, the geometrical feature flow of an ISAR image sequence is learned to represent the target dynamics concisely. Then, the abnormal dynamic recognition is turned into a sequential classification task by exploiting the encoded feature flow through the long-short time memory (LSTM) network. Extensive experiment results confirm the superiority of the proposal in both precision and efficiency aspects.

A Novel Time-Series Memory Auto-Encoder With Sequentially Updated Reconstructions for Remaining Useful Life Prediction.

One of the significant tasks in remaining useful life (RUL) prediction is to find a good health indicator (HI) that can effectively represent the degradation process of a system. However, it is difficult for traditional data-driven methods to construct accurate HIs due to their incomprehensive consideration of temporal dependencies within the monitoring data, especially for aeroengines working under nonstationary operating conditions (OCs). Aiming at this problem, this article develops a novel unsupervised deep neural network, the so-called times series memory auto-encoder with sequentially updated reconstructions (SUR-TSMAE) to improve the accuracy of extracted HIs, which directly takes the multidimensional time series as input to simultaneously achieve feature extraction from both feature-dimension and time-dimension. Further, to make full use of the temporal dependencies, a novel long-short time memory with sequentially updated reconstructions (SUR-LSTM), which uses the errors not only from the current memory cell but also from subsequent memory cells to update the output layer's weight of the current memory cell, is developed to act as the reconstructed layer in the SUR-TSMAE. The use of SUR-LSTM can help the SUR-TSMAE rapidly reconstruct the input time series with higher precision. Experimental results on a public dataset demonstrate the outstanding performance of SUR-TSMAE in comparison with some existing methods.

A self-supervised temporal temperature prediction method based on dilated contrastive learning

Due to the scarcity of the labeled data, traditional supervised learning methods have a limited application scope, which caused the supervised-based model performance will greatly be decreased. In this paper, we propose a promising model based on self-supervised learning. To update the weight and the contrastive relation in the features, a new self-supervised loss, is introduced. First, the convolution neural network is used in the proposed network to extract the deep feature in the first processing. Second, the self-supervised long–short time memory (LSTM) sequential is constructed for further deal. At last, the teacher net and student net have coordinately fine-tuned the credibility of the temperature prediction. By the experimental comparison, our proposed CNN-SSDLSTM is competitive with other supervised and semi-supervised methods. The evaluation experiments achieve state-of-the-art performance in aluminum electrolysis temperature prediction applications.

Performance Evaluation of LSTM and RNN Models in the Detection of Email Spam Messages

Email spam is an unwanted bulk message that is sent to a recipient’s email address without explicit consent from the recipient. This is usually considered a means of advertising and maximizing profit, especially with the increase in the usage of the internet for social networking, but can also be very frustrating and annoying to the recipients of these messages. Recent research has shown that about 14.7 billion spam messages are sent out every single day of which more than 45% of these messages are promotional sales content that the recipient did not specifically opt-in. This has gotten the attention of many researchers in the area of natural language processing. In this paper, we used the Long Short-Time Memory (LSTM) for classification tasks between spam and Ham messages. The performance of LSTM is compared with that of a Recurrent Neural Network( RNN) which can also be used for a classification task of this nature but suffers from short-time memory and tends to leave out important information from earlier time steps to later ones in terms of prediction. The evaluation of the result shows that LSTM achieved 97% accuracy with both Adams and RMSprop optimizers compared to RNN with an accuracy of 94% with RMSprop and 87% accuracy with Adams optimizer.

Predicting mine water inrush accidents based on water level anomalies of borehole groups using long short-term memory and isolation forest

Water level variation of explorational boreholes in mining sites is one of the most direct representations of water inrush risk. Despite recent efforts on mine water inrush accident prediction based on various types of observation data including water level of boreholes using a wide range of machine learning models, the accuracy and timeliness of the prediction for major accidents are still unsatisfactory. In this study, a novel prediction method based on the analysis of water level variation anomalies in borehole groups is proposed using the coupled model of long short-time memory (LSTM) and Isolation forest (IForest). Multi-variate LSTM algorithm is firstly used to model the correlated time sequence water level data in the borehole group, followed by the full extraction of the data variation feature based on the combination of variables including the prediction error being used as inputs of the IForest algorithm. As a result, anomalous variations of the data are detected as possible precursors of water inrush accident. Finally, a set of water inrush risk level grading rule is established based on the number of simultaneous anomalous data instances being detected within a borehole group. The results obtained for the water inrush accident in Xingdong coal mine demonstrate extraordinary timeliness and high accuracy as high water-inrush risk warning is issued for more than 2 days earlier than the time of actual sighting of the accident with very few false alarms being caused throughout the period of the testing dataset. This study provides an innovative solution to the low prediction accuracy and slow responding speed during major water inrush accidents of existing methods. The major parameters of the proposed coupled model of LSTM + IForest can also be easily calibrated based on traditional empirical methods for the best performance, allowing the method to be widely applied in various mining conditions.

Space Target Dynamic Identification by Exploiting Geometrical Feature Flow From ISAR Image Sequences

Dynamic identification is crucial for space target investigation and awareness, which is especially challenging for uncooperative and uncontrolled targets. This article presents a dynamic identification algorithm using the time-variant geometric feature flow extracted from an inverse synthetic aperture radar (ISAR) image sequence. The range-Doppler (RD) projection, along with the sequential radar light of slight of the target component, intrinsically relies on the targets’ dynamic states. With the accommodation of a long short-time memory (LSTM) network, reliable dynamic identification of space targets can be implemented by exploiting the geometrical feature flow from the ISAR sequence. The superiority of the proposed method is demonstrated through comparison experiments.

Autonomous piloting and berthing based on Long Short Time Memory neural networks and nonlinear model predictive control algorithm

Autonomous berthing of large underactuated ships is one of challenge tasks in the development of intelligent and autonomous ships, especially under hybrid disturbances of wind and currents, both trajectory prediction and tracking control are difficult issues due to the complexity of the ship motion. In this paper, a low-speed underactuated ship-handling and motion model which fully consider the disturbances of wind and currents is proposed to support the trajectory prediction and tracking control. In addition, a novel trajectory prediction model base on Long Short Time Memory (LSTM) neural network is presented by exploiting the time series features of the berthing trajectories, in which the proposed accurate ship motion model is used to create teaching data that help improve the data consistent and the accuracy of the trajectory prediction. Furthermore, nonlinear Model Predictive Control (MPC) model based on accurate ship motion model is employed to control the trajectory tracking, which efficiently improves the capability of the anti-disturbance. Experiments were designed to validate the reliability and accuracy of the proposed approach under various disturbances of wind and currents. Experimental results have demonstrated that the proposed approach has outstanding performance and effectiveness under the hybrid disturbances of wind and currents.

Wavelet Transform-Based 3D Landscape Design and Optimization for Digital Cities

As a hot concept, the digital city has developed rapidly in recent years. The digital city uses information technology to realize all the contents of the past, present, and future of the city on the network, and can build a three-dimensional visual landscape. However, the traditional information fusion model has the problems of noise susceptibility and low efficiency in landscape design. In order to solve the above problems, this article proposes a wavelet transform-based 3D landscape design and optimization method for digital cities, which removes the noise influenced by wavelet change and builds an information fusion model based on neural network to complete the design optimization of the three-dimensional landscape of the digital city. Firstly, for the wavelet change denoising problem, an effective denoising algorithm for natural noise and abnormal noise is proposed by combining convolutional neural network and wavelet transform. The algorithm extracts mixed feature information of local long path and local short path based on the information retention module, and decomposes the information by combining wavelet transform, inputs the different components obtained from the decomposition into the network for training, and removes the noise by subsequent feature screening of the network structure. Then, aiming at the optimization of 3D landscape design, an information fusion model based on long-short time memory network and radial basis backpropagation network is proposed for fusing multiple sources of information in the digital city to evaluate the landscape. The method collects digital city feature information at the information layer and preprocesses the feature information by a rate detection algorithm. Then, LSTM and RBF-BP neural networks in deep learning are used in the feature layer for adaptive learning of multiple feature signals, and finally, fuzzy logic is used to control the system decision output to improve the efficiency of 3D landscape design. Finally, the simulation experimental results show that the proposed denoising method in this article can better retain the texture details in the images and the denoised images have better visual effects; the proposed information fusion model has higher accuracy compared with the traditional methods. Combining this method to design and optimize 3D landscapes in digital cities can improve the efficiency of landscape design.

The evaluation of automobile interior acceleration sound fused with physiological signal using a hybrid deep neural network

The evaluation of automobile sound quality is not only related to the inherent properties of the sound, but also to the psychological and physiological state of the evaluator, which makes the evaluation of sound quality become an interdisciplinary research. However, there are some deficiencies regarding the existing studies as follows: (1) it is imprecise enough to visualize the preference of evaluators for sound on the scale; (2) the evaluators without acoustic experience are prone to deviations in their evaluations, resulting in the universally and unapplicable results; (3) the automobile sound quality cannot be fully reflected only by physical and psychological parameters, and most of the existing evaluation models of sound quality are only characterized by shallow architectures. To alleviate the above flaws, a hybrid deep neural network (HDNN) is constructed to achieve the evaluation of automobile interior acceleration sound fused with physiological signals in this paper. An EEG test paradigm under the stimulation of automobile sound is meanwhile designed for a feasibility confirmation of the proposed method. In addition, the acquired EEG datum are respectively analyzed from two perspectives: the manual extraction of typical EEG features and the adaptive extraction of EEG features based on HDNN model. Furthermore, the performance of the proposed HDNN is also validated by comparing with conventional convolutional neural network (CNN) and long short time memory (LSTM). The results indicate that the adaptively extracted EEG features are superior to the manually extracted EEG features, and the proposed HDNN outperforms the CNN and LSTM in terms of accuracy, sensitivity as well as precision, and the optimal accuracy of HDNN is up to 88.1%. Thus, the constructed HDNN contributes to achieving accurate evaluation of automobile sound quality with EEG signal.

The Role of CNN and RNN in the Classification of Audio Music Genres

This study aims at determining how various types of neural networks can be used to categorize music files. We used the GTZAN dataset that contains several genres of traditional music. Every genre has some common traditions that can be referred to as features. The task of classifying music genres based on features is challenging. Deep neural architectures such as Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) have been considered for music analysis. However, it has been observed that neural architectures are data-intensive and face the problem of overfitting. To address this issue, we present a framework containing CNN and RNN with Long Short Time Memory (LSTM) having multiple layers to categorize the music genres and handle the problem of overfitting. Our experiments also revealed the strengths and limitations of deep learning. Finally, we found CNN to be best among other state-of-the-art models and achieved the training and test accuracies of 86.53 % and 81.90 % respectively.

Deep learning-based multi-model ensemble method for classification of PQDs in a hydrogen energy-based microgrid using modified weighted majority algorithm

In this study, new multiple deep classifiers with a modified Weighted Majority Voting (WMV)-based method are proposed to identify power quality disturbances (PQDs) in a hydrogen energy-based microgrid. In the proposed approach, closed-loop deep LSTM (Long Short Time Memory), deep CNN (Convolutional Neural Network), and hybrid (CNN-LSTM) models are used for automatic feature extraction and classification. Then, a modified WMV method is employed to ensemble the outputs of the three deep learning (DL) classifier models. The enhanced WMV mechanism performs an automatically updated weighting based on the validation results of the DL classification models, unlike voting methods in the literature. The improved WMV mechanism eliminates the challenges of using multiple DL classifiers in the voting system. The mathematical data results in LabVIEW, simulation results in Matlab/Simulink, and real data results in the laboratory show that the proposed method shows superior performance in accuracy and noise immunity to state-of-the-art methods.