Bidirectional Long Short-term Memory Neural Network Research Articles

A novel hybrid water quality forecast model based on real-time data decomposition and error correction

Accurate forecast for water quality is of great importance because it can support water resource management with the future information. In this research, we propose a novel hybrid model by using data decomposition, error correction, and machine learning. In our method, first, the initial forecast is obtained by a prediction model that uses improved complete ensemble empirical mode decomposition with adaptive noise and bidirectional long short-term memory (BLSTM) neural network. Next, a novel error correction framework, which is built by variational mode decomposition and BLSTM neural network, is used to improve forecast accuracy by correcting the initial forecast error. Water quality data of Poyang Lake, China is used to evaluate our model. Results indicate that our model shows highly accurate forecast performance for all of the 9 water quality datasets (the average of mean absolute percentage error (MAPE) of 7 day-ahead forecast is 2.12%; 30 day-ahead forecast is 4.06%). In addition, our model outperforms the competitor models, particularly, compared to the prediction model without error correction framework, the average of MAPE is reduced by 33.33% for 7 day-ahead forecast; 30.48% for 30 day-ahead forecast. This research demonstrates that the proposed error correction framework is an effective tool to improve forecast accuracy for water quality.

Wind Power Forecasting Method Based on Bidirectional Long Short-Term Memory Neural Network and Error Correction

With the improvement of penetration rate of wind power in the power system, its volatility and intermittence bring new problems to the power grid. The accurate forecasting of wind power is an effective way to alleviate the impact on the power grid. In this paper, a novel wind power forecasting method is proposed. Firstly, a wind power forecasting model based on the bidirectional long short-term memory (BiLSTM) neural network is established to forecast the wind power according to the wind speed of numerical weather forecasting (NWF). Secondly, a wind power forecasting error time series model based on empirical mode decomposition (EMD) is established to decrease the forecasting error. Finally, this paper uses the real data to simulate and verify the proposed method. Evaluating by the root mean square error (RMSE), symmetric mean absolute percentage error (SMAPE), and Theil inequality coefficient (TIC), the simulation results show that the forecasting accuracy of the BiLSTM neural network model are 10.25%, 6.71% and 12.18% higher than LSTM model respectively. After correcting wind power forecasting error using the proposed time series model based on EMD, the accuracy of wind power forecasting is further improved.

A wind power forecasting method based on optimized decomposition prediction and error correction

To reduce the effect of nonlinearity and volatility in the wind power time sequence, a two-stage short-term wind power forecasting method based on optimized decomposition prediction and error correction is proposed. In the first stage, in order to improve the decomposition effect of variational mode decomposition (VMD), the decomposition loss is defined as the evaluation criterion to guide the parameter setting of VMD, and flower pollination algorithm (FPA) is utilized to automatically optimize the parameters of VMD. Then the complex wind power sequence is decomposed into simple intrinsic mode functions (IMFs). Besides, bi-directional long short-term memory (BiLSTM) neural network is built for each IMF to explore the deep time-series features of wind power in both past and future directions. In the second stage, to reduce the correlation among meteorological factors, principal component analysis (PCA) is employed to convert the multi-dimensional meteorological factors into low-dimensional principal components. Then, with the input of IMFs and principal component, an error correction model based on BiLSTM neural network is established to reduce the inherent error of the model. The experimental results show that the proposed method has higher prediction accuracy than the traditional methods in single-step and multi-step ahead forecasting.

Reflection Coefficients Inversion Based on the Bidirectional Long Short-Term Memory Network

Improving the vertical resolution of seismic data to satisfy the demands for detailed characterization of reservoirs is an important part in seismic data processing. The sparse spike inversion(SSI) technique greatly enhances the seismic resolution by assuming that the reflection coefficients follow the sparse distributions. However, its characterization for the spatial structure of thin and thin interbed reservoirs is still limited. In this letter, we propose a novel approach for enhancing seismic resolution using a bidirectional long short-term memory(BiLSTM) neural network by extracting reflection coefficients directly from seismic data. By designing the network architectures, multiple seismic samples map to the specific reflection coefficient. Compared with the SSI method, the BiLSTM neural network provides higher resolution inversion results on model data, which demonstrates the effectiveness of the proposed approach for thin structure characterizations. The convergence speed of the proposed method is fast and the training process is stable. In addition, we conduct the high-resolution seismic data processing on the field data based on the transfer learning technology. High-resolution processing results illustrate the generalization ability and adaptability of the BiLSTM neural network.

Predictions of Deep Excavation Responses Considering Model Uncertainty: Integrating BiLSTM Neural Networks with Bayesian Updating

Predictions of excavation responses frequently differ from monitoring data due to geotechnical uncertainties. This paper proposes an efficient Bayesian updating approach for excavation responses that considers the uncertainties of soil properties and the calculation model. To evaluate the depth-dependent characteristic of model uncertainty, the model factor is quantified by a constant part and a trending component. Bidirectional long short-term memory (BiLSTM) neural networks are constructed to act as a substitute for the finite-element method to achieve higher computational efficiency. An excavation project in Taipei, Taiwan, is used in this study to illustrate the proposed approach. The results demonstrate that the BiLSTM successfully learns the mapping between soil parameters and deflection responses. The uncertainties of key soil parameters and model factors are significantly reduced when observed deflections at multiple points are incorporated on a stage-by-stage basis. The trending component of the model factor plays an essential role in the early stages, but its impact decreases as the excavation progresses. The prediction intervals using the updated parameters generally cover the monitoring data. The proposed method can rapidly update and improve the predictions of subsequent responses once the monitoring data is obtained. This means early remedial actions can be taken and construction safety ensured.

Deep BiLSTM neural network model for emotion detection using cross-dataset approach

The purpose of this research is to use a cross-dataset approach to construct an EEG-based emotion recognition system. So far, numerous modeling strategies for emotion recognition have been revealed using the same dataset and subject-dependent and independent criteria. We propose EEG-based cross-dataset emotion classification in this study, where the datasets for training and testing are completely distinct. The research is carried out using two benchmark datasets, DEAP and SEED, as well as our own IDEA dataset. The three datasets differ in a variety of technical factors, including electroencephalography (EEG) devices, stimuli, methodology, subject, country, culture, and so on. Multilayer perceptron (MLP), support vector machine (SVM), k-nearest neighbors (k-NN), and deep RNNs model based on bidirectional long short-term memory (BiLSTM) network were trained in this study using features namely PSD, Hjorth parameters, DE, and LF-DE. When the DEAP dataset is used to train the model and the SEED dataset is used to test it, the recognition accuracy improves by 8.2 %, and when the model is the SEED dataset-trained and the DEAP dataset-tested, the recognition accuracy improves by 1.5 % when compared to the previous result. It has been revealed that LF-DE with BiLSTM outperforms other features and classifiers for the same input data. A deep neural network-BiLSTM gives deep features from the lowest level to the highest level from large datasets. The results of the experiments reveal that the optimization of deep neural network parameters can improve the performance of the emotion recognition system to a positive extent.

Fault detection and diagnosis with a novel source-aware autoencoder and deep residual neural network

The capability of deep learning (DL) techniques for dealing with non-linear, dynamic and correlated data has paved the way for DL-based fault detection and diagnosis (FDD). Among them, autoencoders (AEs) have shown their potential to serve as the fault detection network. However, misclassifying faulty samples that share similar patterns to normal samples is a common drawback of AEs. In this work, a source-aware autoencoder (SAAE) is proposed as an extension of AEs to incorporate faulty samples in the training stage. In SAAE, flexibility in tuning recall and precision trade-off, ability to detect unseen faults and applicability in imbalanced data sets are achieved. Bidirectional long short-term memory (BiLSTM) with skip connections SAAE is designed as the structure of the fault detection network. Further, a deep network with BiLSTM and residual neural network (ResNet) is proposed for the subsequent fault diagnosis step to avoid randomness imposed by the order of the input features. A framework for combining fault detection and fault diagnosis networks is also presented without the assumption of having a perfect fault detection network. A comprehensive comparison among relevant existing techniques in the literature and SAAE-ResNet is also conducted on the Tennessee-Eastman process, which shows the superiority of the proposed FDD method.

The Application of Deep Learning Algorithms for PPG Signal Processing and Classification

Photoplethysmography (PPG) is widely used in wearable devices due to its conveniency and cost-effective nature. From this signal, several biomarkers can be collected, such as heart and respiration rate. For the usual acquisition scenarios, PPG is an artefact-ridden signal, which mandates the need for the designated classification algorithms to be able to reduce the noise component effect on the classification. Within the selected classification algorithm, the hyperparameters’ adjustment is of utmost importance. This study aimed to develop a deep learning model for robust PPG wave detection, which includes finding each beat’s temporal limits, from which the peak can be determined. A study database consisting of 1100 records was created from experimental PPG measurements performed in 47 participants. Different deep learning models were implemented to classify the PPG: Long Short-Term Memory (LSTM), Bidirectional LSTM, and Convolutional Neural Network (CNN). The Bidirectional LSTM and the CNN-LSTM were investigated, using the PPG Synchrosqueezed Fourier Transform (SSFT) as the models’ input. Accuracy, precision, recall, and F1-score were evaluated for all models. The CNN-LSTM algorithm, with an SSFT input, was the best performing model with accuracy, precision, and recall of 0.894, 0.923, and 0.914, respectively. This model has shown to be competent in PPG detection and delineation tasks, under noise-corrupted signals, which justifies the use of this innovative approach.

A Deep Learning Approach for Predicting Antigenic Variation of Influenza A H3N2.

Modeling antigenic variation in influenza (flu) virus A H3N2 using amino acid sequences is a promising approach for improving the prediction accuracy of immune efficacy of vaccines and increasing the efficiency of vaccine screening. Antigenic drift and antigenic jump/shift, which arise from the accumulation of mutations with small or moderate effects and from a major, abrupt change with large effects on the surface antigen hemagglutinin (HA), respectively, are two types of antigenic variation that facilitate immune evasion of flu virus A and make it challenging to predict the antigenic properties of new viral strains. Despite considerable progress in modeling antigenic variation based on the amino acid sequences, few studies focus on the deep learning framework which could be most suitable to be applied to this task. Here, we propose a novel deep learning approach that incorporates a convolutional neural network (CNN) and bidirectional long-short-term memory (BLSTM) neural network to predict antigenic variation. In this approach, CNN extracts the complex local contexts of amino acids while the BLSTM neural network captures the long-distance sequence information. When compared to the existing methods, our deep learning approach achieves the overall highest prediction performance on the validation dataset, and more encouragingly, it achieves prediction agreements of 99.20% and 96.46% for the strains in the forthcoming year and in the next two years included in an existing set of chronological amino acid sequences, respectively. These results indicate that our deep learning approach is promising to be applied to antigenic variation prediction of flu virus A H3N2.

Natural Language Processing Algorithms for Normalizing Expressions of Synonymous Symptoms in Traditional Chinese Medicine.

Background The modernization of traditional Chinese medicine (TCM) demands systematic data mining using medical records. However, this process is hindered by the fact that many TCM symptoms have the same meaning but different literal expressions (i.e., TCM synonymous symptoms). This problem can be solved by using natural language processing algorithms to construct a high-quality TCM symptom normalization model for normalizing TCM synonymous symptoms to unified literal expressions. Methods Four types of TCM symptom normalization models, based on natural language processing, were constructed to find a high-quality one: (1) a text sequence generation model based on a bidirectional long short-term memory (Bi-LSTM) neural network with an encoder-decoder structure; (2) a text classification model based on a Bi-LSTM neural network and sigmoid function; (3) a text sequence generation model based on bidirectional encoder representation from transformers (BERT) with sequence-to-sequence training method of unified language model (BERT-UniLM); (4) a text classification model based on BERT and sigmoid function (BERT-Classification). The performance of the models was compared using four metrics: accuracy, recall, precision, and F1-score. Results The BERT-Classification model outperformed the models based on Bi-LSTM and BERT-UniLM with respect to the four metrics. Conclusions The BERT-Classification model has superior performance in normalizing expressions of TCM synonymous symptoms.

Prediction of land surface temperature of major coastal cities of India using bidirectional LSTM neural networks

Abstract Surface Temperature (ST) is important in terms of surface energy and terrestrial water balances affecting urban ecosystems. In this study, to process the nonlinear changes of climatological variables by leveraging the distinct advantages of Long Short-Term Memory (LSTM) and Bidirectional Long Short-Term Memory (BiLSTM), we propose an LSTM-BiLSTM hybrid deep learning model which extracts multi-dimension features of inputs, i.e., backward (future to past) or forward (past to future) to predict ST. This study assessed the climatological variables, i.e., wind speed, wind direction, relative humidity, dew point temperature, and atmospheric pressure impact on ST using five major coastal cities of India: Chennai, Mangalore, Visakhapatnam, Cuddalore, and Cochin. The Recurrent Neural Networks (RNN) and hybrid LSTM-BiLSTM models have effectively predicted ST and outperformed the standalone Artificial Neural Networks (ANN), LSTM, and BiLSTM models. The RNN and LSTM-BiLSTM models have performed better in predicting ST for Mangalore (Nash-Sutcliffe efficiency (NSE)=0.91), followed by Cochin (NSE=0.89), Chennai (NSE=0.88), Cuddalore (NSE=0.88), and Vishakhapatnam (NSE=0.81). The hybrid data-driven modeling framework indicated that coupling the LSTM and BiLSTM models was proven effective in predicting the ST of coastal cities.

Short-Term Load Forecasting Based on Deep Learning Bidirectional LSTM Neural Network

Accurate load forecasting guarantees the stable and economic operation of power systems. With the increasing integration of distributed generations and electrical vehicles, the variability and randomness characteristics of individual loads and the distributed generation has increased the complexity of power loads in power systems. Hence, accurate and robust load forecasting results are becoming increasingly important in modern power systems. The paper presents a multi-layer stacked bidirectional long short-term memory (LSTM)-based short-term load forecasting framework; the method includes neural network architecture, model training, and bootstrapping. In the proposed method, reverse computing is combined with forward computing, and a feedback calculation mechanism is designed to solve the coupling of before and after time-series information of the power load. In order to improve the convergence of the algorithm, deep learning training is introduced to mine the correlation between historical loads, and the multi-layer stacked style of the network is established to manage the power load information. Finally, actual data are applied to test the proposed method, and a comparison of the results of the proposed method with different methods shows that the proposed method can extract dynamic features from the data as well as make accurate predictions, and the availability of the proposed method is verified with real operational data.

Modelling customers credit card behaviour using bidirectional LSTM neural networks

With the rapid growth of consumer credit and the huge amount of financial data developing effective credit scoring models is very crucial. Researchers have developed complex credit scoring models using statistical and artificial intelligence (AI) techniques to help banks and financial institutions to support their financial decisions. Neural networks are considered as a mostly wide used technique in finance and business applications. Thus, the main aim of this paper is to help bank management in scoring credit card clients using machine learning by modelling and predicting the consumer behaviour with respect to two aspects: the probability of single and consecutive missed payments for credit card customers. The proposed model is based on the bidirectional Long-Short Term Memory (LSTM) model to give the probability of a missed payment during the next month for each customer. The model was trained on a real credit card dataset and the customer behavioural scores are analysed using classical measures such as accuracy, Area Under the Curve, Brier score, Kolmogorov–Smirnov test, and H-measure. Calibration analysis of the LSTM model scores showed that they can be considered as probabilities of missed payments. The LSTM model was compared to four traditional machine learning algorithms: support vector machine, random forest, multi-layer perceptron neural network, and logistic regression. Experimental results show that, compared with traditional methods, the consumer credit scoring method based on the LSTM neural network has significantly improved consumer credit scoring.

Deep Learning for Sentiment Analysis of Tunisian Dialect

Automatic sentiment analysis has become one of the fastest growing research areas in the Natural Language Processing (NLP) field. Despite its importance, this is the first work towards sentiment analysis at both aspect and sentence levels for the Tunisian Dialect in the field of Tunisian supermarkets. Therefore, we experimentally evaluate, in this paper, three deep learning methods, namely convolution neural networks (CNN), long short-term memory (LSTM), and bi-directional long-short-term-memory (Bi-LSTM). Both LSTM and Bi-LSTM constitute two major types of Recurrent Neural Networks (RNN). Towards this end, we gathered a corpus containing comments posted on the official Facebook pages of Tunisian supermarkets. To conduct our experiments, this corpus was annotated on the basis of five criteria (very positive/ positive/ neutral/ negative/ very negative) and other twenty categories of aspects. In this evaluation, we show that the gathered features can lead to very encouraging performances through the use of CNN and Bi-LSTM neural networks.

Steganalysis of Low Embedding Rate CNV-QIM in Speech

To address the difficulty of detecting low embedding rate and high-concealment CNV-QIM (complementary neighbor vertices-quantization index modulation) steganography in low bit-rate speech codec, the code-word correlation model based on a BiLSTM (bi-directional long short-term memory) neural network is built to obtain the correlation features of the LPC codewords in speech codec in this paper. Then, softmax is used to classify and effectively detect low embedding rate CNV-QIM steganography in VoIP streams. The experimental results show that for speech steganography of short samples with low embedding rate, the BiLSTM method in this paper has a superior detection accuracy than state-of-the-art methods of the RNN-SM (recurrent neural network-steganalysis model) and SS-QCCN (simplest strong quantization codeword correlation network). At an embedding rate of 20% and a duration of 3 s, the detection accuracy of BiLSTM method reaches 75.7%, which is higher than that of RNN-SM by 11.7%. Furthermore, the average testing time of samples (100% embedding) is 0.3 s, which shows that the method can realize real-time steganography detection of VoIP streams.

Batteries State of Health Estimation via Efficient Neural Networks With Multiple Channel Charging Profiles

The prognostics and health management (PHM) plays the main role to handle the risk of failure before its occurrence. Next, it has a broad spectrum of applications including utility networks, energy storage systems (ESS), etc. However, an accurate capacity estimation of batteries in ESS is mandatory for their safe operations and decision making policy. ESS comprises of different storage mechanisms such as batteries, capacitors, etc. Consequently, the measurement of different charging profiles (CPs) has a strong relation to battery capacity. These profiles include temperature (T), voltage (V), and current (I) where the CPs patterns vary as the battery ages with cycles. Consequently, estimating a battery capacity, the conventional methods practice single channel charging profile (SCCP) and hop multiple channel CPs (MCCPs) that cause incorrect battery health estimation. To tackle these issues, this article proposes MCCPs based battery management system (BMS) to estimate batteries health/capacity through the deep learning (DL) concept where the patterns in these CPs are changed as the battery ages with time and cycles. Thus, we deeply investigate both machine learning (ML) and DL based methods to provide a concrete comparative analysis of our method. The adaptive boosting (AB) and support vector regression (SVR) are widely compared with long short-term memory (LSTM), multi-layer perceptron (MLP), bi-directional LSTM (BiLSTM), and convolutional neural network (CNN) to attain the appropriate approach for battery capacity and state of health (SOH) estimation. These approaches have a high learning capability of inter-relation between the battery capacity and variation in CPs patterns. To validate and verify the proposed technique, we use NASA battery dataset and experimentally prove that BiLSTM outperforms all the approaches and obtains the smallest error values for MAE, MSE, RMSE, and MAPE using MCCPs compared to SCCP.

EEG and Deep Learning Based Brain Cognitive Function Classification

Electroencephalogram signals are used to assess neurodegenerative diseases and develop sophisticated brain machine interfaces for rehabilitation and gaming. Most of the applications use only motor imagery or evoked potentials. Here, a deep learning network based on a sensory motor paradigm (auditory, olfactory, movement, and motor-imagery) that employs a subject-agnostic Bidirectional Long Short-Term Memory (BLSTM) Network is developed to assess cognitive functions and identify its relationship with brain signal features, which is hypothesized to consistently indicate cognitive decline. Testing occurred with healthy subjects of age 20–40, 40–60, and >60, and mildly cognitive impaired subjects. Auditory and olfactory stimuli were presented to the subjects and the subjects imagined and conducted movement of each arm during which Electroencephalogram (EEG)/Electromyogram (EMG) signals were recorded. A deep BLSTM Neural Network is trained with Principal Component features from evoked signals and assesses their corresponding pathways. Wavelet analysis is used to decompose evoked signals and calculate the band power of component frequency bands. This deep learning system performs better than conventional deep neural networks in detecting MCI. Most features studied peaked at the age range 40–60 and were lower for the MCI group than for any other group tested. Detection accuracy of left-hand motor imagery signals best indicated cognitive aging (p = 0.0012); here, the mean classification accuracy per age group declined from 91.93% to 81.64%, and is 69.53% for MCI subjects. Motor-imagery-evoked band power, particularly in gamma bands, best indicated (p = 0.007) cognitive aging. Although the classification accuracy of the potentials effectively distinguished cognitive aging from MCI (p < 0.05), followed by gamma-band power.

Short-term Power Load Forecast of an Electrically Heated House in St. John’s, Newfoundland, Canada

A highly efficient deep learning method for short-term power load forecasting has been developed recently. It is a challenge to improve forecasting accuracy, as power consumption data at the individual household level is erratic for variable weather conditions and random human behaviour. In this paper, a robust short-term power load forecasting method is developed based on a Bidirectional long short-term memory (Bi-LSTM) and long short-term memory (LSTM) neural network with stationary wavelet transform (SWT). The actual power load data is classified according to seasonal power usage behaviour. For each load classification, short-term power load forecasting is performed using the developed method. A set of lagged power load data vectors is generated from the historical power load data, and SWT decomposes the vectors into sub-components. A Bi-LSTM neural network layer extracts features from the sub-components, and an LSTM layer is used to forecast the power load from each extracted feature. A dropout layer with fixed probability is added after the Bi-LSTM and LSTM layers to bolster the forecasting accuracy. In order to evaluate the accuracy of the proposed model, it is compared against other developed short-term load forecasting models which are subjected to two seasonal load classifications.

Short-term Load Forecasting Model Based on IBFO-BILSTM

Short-term power load forecasting has always been a very important position in power grid operation. Improving prediction accuracy has great significance for improving the power company’s dispatch efficiency and economic benefits. In view of the feed-forward neural network cannot learn the time series relationship between load data, a hybrid depth model based on improved bacterial foraging algorithm (IBFO) and long short-term memory (LSTM) network is proposed for short-term power load forecasting. Using historical load data and its associated temperature data and date information as input, the initial weights and bias are optimized by improved bacterial foraging algorithm, and the multi-layer bidirectional LSTM neural network is used to mine the hidden time series relationship between input data, and finally output prediction. The proposed prediction model was verified by using the historical load data of Nantong. The data shows that the proposed model has higher prediction accuracy than other models.

Ship Motion Attitude Prediction Based on an Adaptive Dynamic Particle Swarm Optimization Algorithm and Bidirectional LSTM Neural Network

A new neural network prediction model is proposed for predicting ship motion attitude with high accuracy. This prediction model is based on an adaptive dynamic particle swarm optimization algorithm (ADPSO) and bidirectional long short-term memory (BiLSTM) neural network, which is to optimize the hyperparameters of BiLSTM neural network by the proposed ADPSO algorithm. The ADPSO algorithm introduces dynamic search space strategy into the classical particle swarm optimization algorithm and adjusts the learning factor adaptively to balance the global and local search ability, so as to improve the optimization performance and improve its optimization effect in BiLSTM parameter optimization process. The results show that the model can obtain higher prediction accuracy and faster convergence speed, and has better prediction performance in the prediction of ship motion attitude.