Traditional LSTM Research Articles

Meta-learning prediction of physical and chemical properties of magnetized water and fertilizer based on LSTM

BackgroundDue to the high cost of data collection for magnetization detection of media, the sample size is limited, it is not suitable to use deep learning method to predict its change trend. The prediction of physical and chemical properties of magnetized water and fertilizer (PCPMWF) by meta-learning can help to explore the effects of magnetized water and fertilizer irrigation on crops.MethodIn this article, we propose a meta-learning optimization model based on the meta-learner LSTM in the field of regression prediction of PCPMWF. In meta-learning, LSTM is used to replace MAML’s gradient descent optimizer for regression tasks, enables the meta-learner to learn the update rules of the LSTM, and apply it to update the parameters of the model. The proposed method is compared with the experimental results of MAML and LSTM to verify the feasibility and correctness.ResultsThe average absolute percentage error of the meta-learning optimization model of meta-learner LSTM is reduced by 0.37% compared with the MAML model, and by 4.16% compared with the LSTM model. The loss value of the meta-learning optimization model in the iterative process drops the fastest and steadily compared to the MAML model and the LSTM model. In cross-domain experiments, the average accuracy of the meta-learning optimized model can still reach 0.833.ConclusionsIn the case of few sample, the proposed model is superior to the traditional LSTM model and the basic MAML model. And in the training of cross-domain datasets, this model performs best.

Development of deep learning artificial neural networks models to predict temperature and power demand variation for demand response application in cold storage

Food warehouses and cold rooms have a significant potential for Demand Response (DR) application (stopping or reducing the power of fans and compressors of the refrigeration system) due to thermal inertia of food products. However, as air and food temperature might increase beyond acceptable limits during DR periods, DR needs to be carefully applied in order to respect the food temperature regulation and to maintain quality and safety of the products. It is thus important to predict the system behaviour in case of DR application in order to evaluate its potential impacts and to decide if DR can be performed or not. Four deep learning artificial neural networks (ANN) models, traditional Long Short-Term Memory LSTM, stacked LSTM, bidirectional LSTM and convolutional LSTM, were developed to predict future temperature and power demand perturbations due to the application of DR in cold storage. The aims of this work are: first, to assess the performance of those models in predicting the system behaviours, in particular the sudden variations during and after DR applications, and second, to identify the impact of data availability (number of sensors, their positions) and data characteristic (quality, quantity and DR patterns) on the prediction performance. The results have shown the high potential of deep learning ANN models in supporting DR application in cold storage.

Speech enhancement from fused features based on deep neural network and gated recurrent unit network

Speech is easily interfered by external environment in reality, which results in the loss of important features. Deep learning has become a popular speech enhancement method because of its superior potential in solving nonlinear mapping problems for complex features. However, the deficiency of traditional deep learning methods is the weak learning capability of important information from previous time steps and long-term event dependencies between the time-series data. To overcome this problem, we propose a novel speech enhancement method based on the fused features of deep neural networks (DNNs) and gated recurrent unit (GRU). The proposed method uses GRU to reduce the number of parameters of DNNs and acquire the context information of the speech, which improves the enhanced speech quality and intelligibility. Firstly, DNN with multiple hidden layers is used to learn the mapping relationship between the logarithmic power spectrum (LPS) features of noisy speech and clean speech. Secondly, the LPS feature of the deep neural network is fused with the noisy speech as the input of GRU network to compensate the missing context information. Finally, GRU network is performed to learn the mapping relationship between LPS features and log power spectrum features of clean speech spectrum. The proposed model is experimentally compared with traditional speech enhancement models, including DNN, CNN, LSTM and GRU. Experimental results demonstrate that the PESQ, SSNR and STOI of the proposed algorithm are improved by 30.72%, 39.84% and 5.53%, respectively, compared with the noise signal under the condition of matched noise. Under the condition of unmatched noise, the PESQ and STOI of the algorithm are improved by 23.8% and 37.36%, respectively. The advantage of the proposed method is that it uses the key information of features to suppress noise in both matched and unmatched noise cases and the proposed method outperforms other common methods in speech enhancement.

Developing C-LSTM model for evaluating moisture content of carrot slices during drying

Moisture content (MC) is an important indicator for evaluating the quality of agricultural products during drying. The rapid detection of MC helps to realize optimization of the configuration and online adjustment of drying process parameters, ensuring the quality of the final product and achieving green and efficient drying. As an effective tool for rapid and nondestructive detection of agricultural product quality, the use of hyperspectral/multispectral image (HSI/MSI) for MC evaluation of agricultural products/food during drying has been extensively studied. However, how to extract effective features from HSI/MSI and build a function relationship between these features and MC is still a challenging problem. The main work of this study is to develop a convolutional – long short term memory(C-LSTM) prediction model for MC evaluation of drying agricultural products based on HSI/MSI. The proposed C-LSTM model uses the convolution module to automatically extract the features of the HSI/MSI, and applies LSTM to explore the correlation information between the features extracted from each wavelength and the MC, thus finally realize the accurate prediction of MC. The carrot slices were dried with different drying time by using two different drying equipment (electric thermostatic drying oven drier and radio frequency- hot air drier (RF-HAD), and MSI of dried samples, covering the spectral range of 675–975 nm, were collected using multispectral imaging system based on a single shot. The best prediction results were achieved by C-LSTM models, with Rp of 0.966, RMSEP of 0.079(%) and RPD of 3.734 for oven drier, and Rp of 0.965, RMSEP of 0.085(%), and RPD of 3.604 for RF-HAD, compared to other four traditional models and LSTM model. The experimental results indicated that C-LSTM model is a potential modeling method for MC detection of agricultural products based on HSI/MSI.

Dynamic Evolution Mechanism of Digital Entrepreneurship Ecosystem Based on Text Sentiment Computing Analysis.

To solve the limitations of the current entrepreneurial ecosystem, the research on the digital entrepreneurial ecosystem is more meaningful. This article aims to study the dynamic evolution mechanism of the digital entrepreneurship ecosystem based on text sentiment computing analysis. It proposes an improved Bi-directional long short-term memory (Bi-LSTM) model, which uses a multilayer neural network to deal with classification problems. It has a higher accuracy rate, recall rate, and F1 value than the traditional LSTM model and can better perform sentiment analysis on text. The algorithm uses the optimized Naive Bayes algorithm, which is based on Euclidean distance weighting and can assign different weights to the final classification results according to different attributes. Compared with the general Bayes algorithm, it improves the calculation efficiency and can better match the digital entrepreneurial ecosystem, which is evolving dynamically, predicting and analyzing its future development. The experimental results in this article show that the improved Bi-LSTM is better than the traditional Bi-LSTM model in terms of accuracy and F1 value. The accuracy rate is increased by 1.1%, the F1 value is increased by 0.6%, and the recall rate is only <0.2%. Running on the Spark platform, although 3% accuracy is sacrificed, the running time is increased by 320%. Compared with the traditional cellular neural network (CNN) algorithm, the accuracy rate is increased by 4%, the recall rate is increased by 14%, and the F1 value is increased by 9%, which proves that it has a strong non-linear fitting ability. The performance improvement brought by the huge data set is very huge, which fully proves the feasibility of the digital entrepreneurship ecosystem.

A simple approach for short-term wind speed interval prediction based on independently recurrent neural networks and error probability distribution

Improving the quality of Wind Speed Interval prediction is important to maximize the usage of integrated wind energy as well as to reduce the adverse effects of the uncertainties, introduced by the random fluctuations of wind, to the power systems. This paper utilizes independently recurrent neural network to propose two new interval prediction frameworks. This network possesses the ability to retain memory at different lengths, which is helpful in capturing temporal features, especially for multi-horizon forecasts where the local dynamics get quite involved. In the first approach, we integrated a quantile regression loss function into this network to generate the intervals. This framework however, require to train different regressors to generate the conditional quantiles. Removing this limitation, a new simple and intuitive approach, is proposed which estimates the prediction intervals using a Gaussian function centered on the prediction and estimated error by a point prediction model and an error prediction model respectively. In our computational experiments, which involve two different wind fields contributing to eight different cases, an improvement of 43% and 12%, in average coverage width criterion index, over traditional models and LSTM based model respectively is remarkable. Thus, the proposed framework is able to produce high quality PIs while simultaneously reducing the computational cost.

Li-ion battery state of health Prediction based on Long Short-Term Memory Recurrent Neural Network

The state of health (SOH) prediction of lithium-ion battery is essential for the health management of batteries. At present, the prediction method combined with the extraction of health indicators in charge-discharge process has received extensive attention, however, many studies ignored that the extraction of battery discharge data will be affected by the actual operating conditions, which will affect the effectiveness of health indicators extraction. In this work, a type of recurrent neural network (RNN), which is long short-term memory-RNN(LSTM-RNN), is proposed to prediction the SOH of Li-ion batteries through the data of charging process and capacity. Because the different choice of network parameters will also affect the performance of the model, particle swarm optimization (PSO) algorithm is used to optimize LSTM model. The test results show that this method can effectively predict SOH of battery, and the maximum RMSE is less than 0.01. Compared with the traditional LSTM algorithm, it has higher accuracy.

Research on the recommendation of aerobics music adaptation based on computer aided design software

Aerobics is full of charm, and music plays an inestimable role in it. With the penetration of music in aerobics, the “sound” of music art is introduced into the “shape” of aerobics movements, and the visual art and visual experience are perfectly combined, which greatly expands the extension and extension of aerobics. This paper proposes an aerobics music adaptation recommendation algorithm that combines classification and collaborative filtering. First, by calculating the similarity of the user context information, the collaborative filtering algorithm obtains the initial annihilation grass music recommendation list; then the classification model is trained by the machine learning algorithm to obtain the user’s aerobics music type preference in a specific context; finally, collaborative filtering The obtained recommendation list is integrated with the aerobics music preference obtained by the classification model to provide personalized aerobics music adaptation recommendations for users in specific situations. In the specific aerobics music adaptation recommendation, the algorithm is implemented by a deep neural network composed of an independent cyclic neural network algorithm and an attention mechanism. In the data preprocessing stage, the audio of the user’s listening history is preprocessed by scattering transformation. The audio features of the user’s listening history are extracted by scattering transformation, and then this feature is combined with the user’s portrait to obtain a recommendation list through an independent recurrent neural network with a hybrid attention mechanism. The experimental results show that this method can effectively improve the performance of the personalized music recommendation system. Compared with the traditional single algorithm IndRNN and LSTM, the recommendation accuracy is improved by 7.8% and 20.9%, respectively.

Expression-EEG Bimodal Fusion Emotion Recognition Method Based on Deep Learning.

As one of the key issues in the field of emotional computing, emotion recognition has rich application scenarios and important research value. However, the single biometric recognition in the actual scene has the problem of low accuracy of emotion recognition classification due to its own limitations. In response to this problem, this paper combines deep neural networks to propose a deep learning-based expression-EEG bimodal fusion emotion recognition method. This method is based on the improved VGG-FACE network model to realize the rapid extraction of facial expression features and shorten the training time of the network model. The wavelet soft threshold algorithm is used to remove artifacts from EEG signals to extract high-quality EEG signal features. Then, based on the long- and short-term memory network models and the decision fusion method, the model is built and trained using the signal feature data extracted under the expression-EEG bimodality to realize the final bimodal fusion emotion classification and identification research. Finally, the proposed method is verified based on the MAHNOB-HCI data set. Experimental results show that the proposed model can achieve a high recognition accuracy of 0.89, which can increase the accuracy of 8.51% compared with the traditional LSTM model. In terms of the running time of the identification method, the proposed method can effectively be shortened by about 20 s compared with the traditional method.

A Fuzzy Seasonal Long Short-Term Memory Network for Wind Power Forecasting

To protect the environment and achieve the Sustainable Development Goals (SDGs), reducing greenhouse gas emissions has been actively promoted by global governments. Thus, clean energy, such as wind power, has become a very important topic among global governments. However, accurately forecasting wind power output is not a straightforward task. The present study attempts to develop a fuzzy seasonal long short-term memory network (FSLSTM) that includes the fuzzy decomposition method and long short-term memory network (LSTM) to forecast a monthly wind power output dataset. LSTM technology has been successfully applied to forecasting problems, especially time series problems. This study first adopts the fuzzy seasonal index into the fuzzy LSTM model, which effectively extends the traditional LSTM technology. The FSLSTM, LSTM, autoregressive integrated moving average (ARIMA), generalized regression neural network (GRNN), back propagation neural network (BPNN), least square support vector regression (LSSVR), and seasonal autoregressive integrated moving average (SARIMA) models are then used to forecast monthly wind power output datasets in Taiwan. The empirical results indicate that FSLSTM can obtain better performance in terms of forecasting accuracy than the other methods. Therefore, FSLSTM can efficiently provide credible prediction values for Taiwan’s wind power output datasets.

CloudLSTM: A Recurrent Neural Model for Spatiotemporal Point-cloud Stream Forecasting

This paper introduces CloudLSTM, a new branch of recurrent neural models tailored to forecasting over data streams generated by geospatial point-cloud sources. We design a Dynamic Point-cloud Convolution (DConv) operator as the core component of CloudLSTMs, which performs convolution directly over point-clouds and extracts local spatial features from sets of neighboring points that surround different elements of the input. This operator maintains the permutation invariance of sequence-to-sequence learning frameworks, while representing neighboring correlations at each time step -- an important aspect in spatiotemporal predictive learning. The DConv operator resolves the grid-structural data requirements of existing spatiotemporal forecasting models and can be easily plugged into traditional LSTM architectures with sequence-to-sequence learning and attention mechanisms. We apply our proposed architecture to two representative, practical use cases that involve point-cloud streams, i.e. mobile service traffic forecasting and air quality indicator forecasting. Our results, obtained with real-world datasets collected in diverse scenarios for each use case, show that CloudLSTM delivers accurate long-term predictions, outperforming a variety of competitor neural network models.

Improved LSTM-Based Abnormal Stream Data Detection and Correction System for Internet of Things

The Internet of Things (IoT) is the integration of all information and Internet technology in the information age, which can realize the collection and transmission of intelligent information. A large number of sensors are producing and collecting data involving various industries every day. The amount of stream data generated is huge, and a large number of abnormal data are also generated in the process. Due to the demands of business and life quality improvement, the application of IoT technology to real-time monitoring and correction of massive stream data, especially the correction of abnormal data, is a very valuable research direction, and also the key to ensure the credibility and fidelity of IoT data. This article proposes a recurrent neural network model based on long- and short-term memory network (LSTM) and LSTM+. LSTM+ model not only reduces the regression error compared with the traditional LSTM model, but also can detect abnormal data collected by IoT terminal nodes, and can correct the abnormal data in real time, so as to ensure that the network prediction can have good stability and robustness.

Stock Price Movement Prediction Based on a Deep Factorization Machine and the Attention Mechanism

The prediction of stock price movement is a popular area of research in academic and industrial fields due to the dynamic, highly sensitive, nonlinear and chaotic nature of stock prices. In this paper, we constructed a convolutional neural network model based on a deep factorization machine and attention mechanism (FA-CNN) to improve the prediction accuracy of stock price movement via enhanced feature learning. Unlike most previous studies, which focus only on the temporal features of financial time series data, our model also extracts intraday interactions among input features. Further, in data representation, we used the sub-industry index as supplementary information for the current state of the stock, since there exists stock price co-movement between individual stocks and their industry index. The experiments were carried on the individual stocks in three industries. The results showed that the additional inputs of (a) the intraday interactions among input features and (b) the sub-industry index information effectively improved the prediction accuracy. The highest prediction accuracy of the proposed FA-CNN model is 64.81%. It is 7.38% higher than that of traditional LSTM, and 3.71% higher than that of the model without sub-industry index as additional input features.

Temperature monitoring and prediction under different transmission modes

With the increasing maturity of Internet of Things technology, temperature sensors based on Zigbee, LoRa and other transmission modes have successively appeared. However, due to factors such as outdoor environment and transmission distance, the temperature measurement data under different transmission methods may be inaccurate and the update of background data may be delayed. To master the temperature measurement system parameters in different transmission modes in detail, Zigbee and LoRa transmission modes are used to design temperature sensors respectively. At the same time, the prediction accuracy and running time of the line temperature under several existing neural networks are analyzed and compared, then based on the LSTM network with the highest accuracy, a temperature prediction method based on Long Short Time Memory and Extreme Learning Machine(LSTM-ELM) network is proposed to realize the rapid prediction of line temperature. Simulation results show that the network prediction accuracy based on LSTM-ELM reaches 93.32%, and the prediction time is greatly reduced to 875.75 s, which is nearly 2000% higher than the traditional LSTM network, and can provide a reliable basis for background management in engineering practice.

Long Short‐Term Memory Neural Network for Ionospheric Total Electron Content Forecasting Over China

AbstractAn increasing number of terrestrial‐ and space‐based radio‐communication systems are influenced by the ionospheric space weather, making the ionospheric state increasingly important to forecast. In this study, a novel extended encoder‐decoder long short‐term memory extended (ED‐LSTME) neural network, which can predict ionospheric total electron content (TEC) is proposed. Useful inherent features were automatically extracted from the historical TEC by LSTM layers, and the performance of the proposed model was enhanced by considering solar flux and geomagnetic activity data. The proposed ED‐LSTME model was validated using 15‐min TEC values from GPS measurements over one solar cycle (from January 2006 to July 2018) collected at 15 GPS stations in China. Different assessment experiments were conducted in different geographical locations and seasons as well as under varying geomagnetic activities, to comprehensively evaluate the model's performance. These comparative experiments were conducted using an ED‐LSTM, a traditional LSTM, a deep neural network, autoregressive integrated moving average, and the 2016 International Reference Ionosphere models. The results indicated that the ED‐LSTME model is superior to the other statistical models, with R2 and root mean square error values of 0.89 and 12.09 TECU, respectively. In addition, TEC was adequately predicted under different ionospheric conditions, and satisfactory results were obtained even under geomagnetically disturbed conditions. These results suggest that the prediction performance could be significantly improved by utilizing auxiliary data. These observations confirm that the proposed model outperforms several state‐of‐the‐art models in making predictions at different times and under diverse conditions.

Improve Visual Question Answering Based On Text Feature Extraction

VQA requires the machine to be able to answer questions related to the image based on the image. VQA is mainly divided into four modules: text extraction module, image extraction module, text-image feature fusion module and answer prediction module. In this paper, the most mainstream VQA2.0[1] data set is selected for experiment, and the text extraction module is improved on the basis of a baseline model. The Glove vector is used to preprocess the problem text data, and the GRU network replaces the traditional LSTM network encoding and processing text vector. Experimental results show that the improvement of the text extraction module improves by 2.42% on the basis of the original model, and the GRU network can also accelerate the training speed.

A novel prediction model for the inbound passenger flow of urban rail transit

High-precision short-term inbound passenger flow prediction is of great significance to the daily crowd management and line rescheduling in urban rail systems. Although current models have been applied to prediction, most methods need optimization to meet refined passenger flow management demand. In order to better predict the passenger flow, a novel Wave-LSTM model, based on long short-term memory network (LSTM) and wavelet, is introduced in this paper. In an empirical study with practical passenger flow data of Dongzhimen Station in the Beijing Subway system, the hybrid model exhibited more effective performance in terms of prediction accuracy than the existing algorithms, e.g., autoregressive integrated moving average (ARIMA), nonlinear regression (NAR), and traditional LSTM model. The study illustrates that our newly adopted model is a promising approach for predicting high-precision short-term inbound passenger flow.

Research on health assessment of electric power information system based on deep belief networks and cluster analysis

It has been a hot issue for a long time to effectively ensure the safe and stable operation of the power service information system and make a reasonable evaluation of the healthy operation of the power service information system. Without relying too much on the subjective experience and scoring of experts, this paper introduces a health assessment method based on deep belief networks and cluster analysis for the power service information system. Firstly, cluster analysis is used to obtain corresponding cluster centers and evaluation index sets based on the selected evaluation indexes of the information system. Secondly, the entropy weight method is adopted to score different clustering categories and obtain the corresponding health evaluation categories. Finally, based on the score of the healthy operation, the samples are supervised and trained by the deep belief network model, and then the objective comprehensive evaluation of the system is completed. Experiment results show that this model is more accurate than traditional SVM, KNN, random forest and LSTM. The proposed model has theoretical significance and practical value for the healthy operation evaluation of the power service information system.

Soft sensor based on eXtreme gradient boosting and bidirectional converted gates long short-term memory self-attention network

In this paper, a new soft sensor that combines eXtreme Gradient Boosting (Xgboost) decision trees and a bidirectional, converted gate long short-term memory (BiCG-LSTMs) self-attention (SEA) mechanism network is proposed. Xgboost is first utilized to select relevant input variables according to their importance. It then acts as an encoder to weigh the selected input variables based on their importance scores. The encoded input variables are normalized and then sent to the bidirectional converted gates LSTM (BiCG-LSTMs) to extract dynamic information hidden in the process data. The BiCG-LSTMs is designed to avoid multiple gates function, a characteristic of traditional LSTM units in bidirectional LSTM that consumes additional calculation time. Next, a regularization method by smoothing dynamic features based on self-attention weights is utilized to denoise and alleviate the overfitting of the regression once new features are added. In addition, self-attention takes into account the internal dependence of input variables regardless how far the distance between input variables. Finally, the effectiveness of the proposed Xgboost-BiCG-LSTM-SEA soft sensor framework is demonstrated by an application to the prediction of melt intrinsic viscosity of the polyester polymerization process.

Adaptive particle swarm optimization algorithm based long short-term memory networks for sentiment analysis

Text Sentiment analysis is the process of predicting whether a segment of text has opinionated or objective content and analyzing the polarity of the text’s sentiment. Understanding the needs and behavior of the target customer plays a vital role in the success of the business so the sentiment analysis process would help the marketer to improve the quality of the product as well as a shopper to buy the correct product. Due to its automatic learning capability, deep learning is the current research interest in Natural language processing. Skip-gram architecture is used in the proposed model for better extraction of the semantic relationships as well as contextual information of words. However, the main contribution of this work is Adaptive Particle Swarm Optimization (APSO) algorithm based LSTM for sentiment analysis. LSTM is used in the proposed model for understanding complex patterns in textual data. To improve the performance of the LSTM, weight parameters are enhanced by presenting the Adaptive PSO algorithm. Opposition based learning (OBL) method combined with PSO algorithm becomes the Adaptive Particle Swarm Optimization (APSO) classifier which assists LSTM in selecting optimal weight for the environment in less number of iterations. So APSO - LSTM ‘s ability in adjusting the attributes such as optimal weights and learning rates combined with the good hyper parameter choices leads to improved accuracy and reduces losses. Extensive experiments were conducted on four datasets proved that our proposed APSO-LSTM model secured higher accuracy over the classical methods such as traditional LSTM, ANN, and SVM. According to simulation results, the proposed model is outperforming other existing models.