Term Memory Network Research Articles

Research on prediction method of expressway section traffic flow considering anomaly detection probability

AbstractReal‐time and accurate short‐term traffic flow prediction can provide a scientific basis for decision making by travellers and traffic management, and alleviate traffic congestion to a certain extent. The existing traffic flow prediction methods often encounter limitations in real‐time performance and accuracy due to the post‐processing required to rectify anomaly detection results during data pre‐processing. This paper presents a novel traffic flow prediction model, termed the With‐Anomaly Detection Probability (ADP) Attention‐Bidirectional Long‐Short Term Memory (BiLSTM) model. This model takes the probability of anomaly detection into consideration, integrating the anomaly detection outcomes as an inherent parameter into the traffic flow prediction framework. Additionally, the model incorporates an attention mechanism within the long‐short term memory network. Through a comprehensive simulation study utilizing actual measured traffic flow data from the Shanghai‐Chongqing Expressway, the effectiveness of the proposed model is rigorously evaluated. The prediction results show that the model proposed in this paper is a real‐time and accurate short‐time traffic flow prediction model compared with the basic models.

Short‐term wind power prediction based on the combination of firefly optimization and LSTM

AbstractWith the development of social resources, people's consumption of energy is huge, so renewable energy, such as wind energy, has been widely concerned and developed. Although there has been sufficient development of wind power generation, its output has some problems such as uncertainty, which leads to insufficient utilization of wind energy resources and uneven power output quality level, which brings great challenges to the grid connection. To solve this problem, a short‐term wind power prediction model combining firefly algorithm and long term memory network is proposed. The main motivation of the research is to improve the accuracy of wind power prediction and thus improve the utilization of wind energy resources. Compared with the existing methods, the innovation of FA‐LSTM model lies in the integration of the two algorithms, making full use of the advantages of FA in global search optimization and LSTM in time series data processing, and improving the accuracy and stability of prediction. During the experiment, we used different wind farm data to train and test the model. The results show that the FA‐LSTM model can improve the optimal fitness by more than 50% compared with other algorithms, and the iterative prediction error is smaller. Standard root mean square error (RMSE) and mean absolute error (MAE) were used to evaluate the model. The accuracy of RMSE and MAE reached over 97% and 98% respectively. When the test data is highly volatile, the data accuracy of FA‐LSTM model reaches 92% and 94%, and the FA‐LSTM model drops to the stable value faster. FA‐LSTM model has the best fitting degree with the true value curve, and the fitting degree reaches more than 90%. Comparing the actual power and predicted power of different units, the actual power of Unit 1 is 34.875, and the predicted power obtained by FA‐LSTM model is 34.935, with an error of only 0.06. The key finding of this study is that the prediction model combining FA and LSTM has high accuracy and stability in wind power prediction, and can effectively deal with the uncertainty and volatility of wind energy resource utilization. FA‐LSTM model provides an effective solution for wind power prediction, which is helpful to improve the utilization rate of wind energy resources.

Comparison of deep learning approaches for forecasting urban short-term water demand a Greater Sydney Region case study

Accurate water demand forecasting is essential for water utilities for effective water planning and pump scheduling. To conduct effective water planning and pump scheduling, Sydney Water needs a reliable method for predicting water demand. Short-term water demand forecasting deep learning- based models have been applied to different datasets collected from various water plants in the Greater Sydney region. This paper uses correlation analysis, the attention mechanism and a deep convolutional neural network to identify and study the importance of n number of factors that inform water prediction in the Greater Sydney region. The aim of this study is to develop, evaluate and compare multi- variate input deep learning-based models to predict water demand in various locations in the Greater Sydney region. This paper investigates five models — the recurrent neural network (RNN), long short- term memory networks (LSTM), bidirectional long short-term memory networks (BiLSTM), gated recurrent unit (GRU), and the deep convolutional neural network (CNN). The results of this study demonstrate that all the proposed models achieved a superior forecasting performance with accuracy hits of 97% using water consumption and climate factors as input data.

Combining bidirectional long short‐term memory and self‐attention mechanism for code search

SummaryWith the wide application of deep learning in code search, especially the proposed code search model based on attention mechanism, the accuracy of code search has been greatly improved. However, the attention mechanism only captures the attention weight relationship between two words in the code fragment, without considering the contextual semantic relationship that exists between words in the code fragment, which can help improve the accuracy of code search. To address this problem, this paper proposes a model that combining bidirectional long short‐term memory and self‐attention mechanisms for code search (CBLSAM‐CS). The model first captures the contextual semantic relationship of each word in the code fragment by long‐short term memory network, and then uses the self‐attention mechanism to extract deep‐level features of the sequence. In order to verify the effectiveness of the proposed model, the paper has been conducted an experimental comparison with three other baseline models, CODEnn, CARLCS‐CNN, and SAN‐CS, on the basis of a public dataset containing 18 million code fragments. The experimental results show that the proposed model in this paper achieves 92.24% and 93.55% in mean reciprocal rank value and normalized discounted cumulative gain metrics, respectively, which are better than the baseline model. Therefore, it shows that the CBLSAM‐CS model proposed in this paper can effectively improve the accuracy and efficiency of code search.

A hybrid deep learning model based on feature capture of water level influencing factors and prediction error correction for water level prediction of cascade hydropower stations under multiple time scales

The operating conditions of large cascade hydropower stations are complex. Improving the water level prediction accuracy of large cascade hydropower stations is significant for flood control, shipping, irrigation, etc. A new hybrid model based on a refined deep residual shrinkage network and an optimized gated recurrent unit – long term memory network (GRU-LSTM) model is proposed for water level prediction at different time scales. First, to make the input data in the form of a feature map so the model can capture the weight characteristics of each influencing factor when the water level changes, the water level, flow rate, and hydroelectric power plant output data are constructed as high-dimensional feature inputs. To reduce the prediction error caused by downstream tributary backwater jacking, downstream tributary flow is added to the input data. At the same time, a semisoft threshold function with an adjustable function is inserted into the model to improve feature recognition accuracy, reduce deviation, and eliminate the noise of the original hydrological data. Second, an error weight correction function is used to adjust the error between the predicted water level and the observed water level so that the model can automatically adjust the weights of each influencing factor according to the evaluation results of water level prediction. In this process, the parameters of LSTM are optimized using the Archimedes optimization algorithm, and the output results of the LSTM and GRU networks are weighted to obtain a more accurate water level prediction. The proposed model achieves good accuracy and efficiency in downstream water level prediction of the Xiangjiaba Hydropower Station, outperforms existing hydrodynamic and artificial intelligence methods, and has robust scalability in flood forecasting and urban rainwater forecasting.

Limits to ecological forecasting: Estimating uncertainty for critical transitions with deep learning

Abstract In the current age of a rapidly changing environment, it is becoming increasingly important to study critical transitions and how to best anticipate them. Critical transitions pose extremely challenging forecasting problems, which necessitate informative uncertainty estimation rather than point forecasts. In this study, we apply some of the most cutting edge deep learning methods for probabilistic time series forecasting to several classic ecological models that examine critical transitions. Our analysis focuses on three different simulated examples of critical transitions: a Hopf bifurcation, a saddle‐node bifurcation and a stochastic transition. For each scenario, we compare the forecasts from four deep learning models, long‐short term memory networks, gated recurrent unit networks, lock recurrent neural networks and transformers, to forecasts from an ARIMA model and a MCMC estimated model that is given the true transition dynamics. We found that the deep learning models were able to perform comparably to the idealized MCMC model on the stochastic transition case, and generally in between the MCMC and ARIMA models on the Hopf and saddle‐node bifurcation examples. Our results establish that deep learning methods warrant further exploration on the challenging class of critical transition forecasting problems.

Koopman operator for time-dependent reliability analysis

Time-dependent structural reliability analysis of nonlinear dynamical systems is non-trivial; subsequently, scope of most of the structural reliability analysis methods is limited to time-independent reliability analysis only. In this work, we propose a Koopman operator based approach for time-dependent reliability analysis of nonlinear dynamical systems. Since the Koopman representations can transform any nonlinear dynamical system into a linear dynamical system, the time evolution of dynamical systems can be obtained by Koopman operators seamlessly regardless of the nonlinear or chaotic behavior. Despite the fact that the Koopman theory has been in vogue a long time back, identifying intrinsic coordinates is a challenging task; to address this, we propose an end-to-end deep learning architecture that learns the Koopman observables and then use it for time marching the dynamical response. Unlike purely data-driven approaches, the proposed approach is robust even in the presence of uncertainties; this renders the proposed approach suitable for time-dependent reliability analysis. We propose two architectures; one suitable for time-dependent reliability analysis when the system is subjected to random initial condition and the other suitable when the underlying system have uncertainties in system parameters. The proposed approach is robust and generalizes to unseen environment (out-of-distribution prediction). Efficacy of the proposed approached is illustrated using four numerical examples. Results obtained indicate supremacy of the proposed approach as compared to purely data-driven auto-regressive neural network, long short- term memory network, and DeepONet.

Malware classification using word embeddings algorithms and long‐short term memory networks

AbstractThe number of malicious software applications, or malware programs, increases every year. Their development becomes more sophisticated as new techniques are used to bypass program scanning software applications, such as antiviruses. Thereby, deep learning‐based methods emerge as a new promising way to identify these threats. Our main purpose and contribution in this work is proposing and implementing a successful approach to tackle both binary and multiclass malware classification problems. We used unsupervised word embedding algorithms for representing software applications to be analyzed and long‐short term memory for classifying the software applications. For evaluating our pipeline, we introduce a new dataset for binary and multiclass malware classification because we could not find large datasets containing sufficient samples of cleanware and the various malware types for multiclass classification that could be used to evaluate classification models. Our experimental results reached an accuracy of 88.94% for binary classification and 75.13% for multiclass classification. These results suggest that the proposed dataset is challenging, and using it can help in the training of better malware classifiers, improving security.

Abnormal Event Detection in Videos using Spatiotemporal AutoEncoder

The detection of abnormal events from surveillance video input is more difficult due to the small number of such occurrences that may occur over time. Through learning a supervised model, this can be achieved to detect such events from video sequences. The proposed system used Deep learning based Convolutional Long Short- Term Memory (Conv-LSTM) networks to learn video sequences as supervised models and to compute the regularity score from the learned frames. Any divergences on the computed scores represents the detection of events as the learning model has represented it through reconstruction of errors. The learned model can be chosen as the best model based on the detection performed through regularity scores. The learning videos of abnormal sequences are very less in real time video and which has to be determined through the model accurately. Thus, the proposed work is more challenging to get the event detection identification at a particular frame accurately. Auto encoder and decoders used in the proposed model. The model is also experimented on Avenue Dataset for detection of abnormal events. The Detection of Anomalous Occurrences from video sequences is accurately detected in experimental results. Key Words: Deep learning, Convolutional Neural Networks - Long Short-Term Memory (CNN-LSTM), Supervised learning, Regulatory scores

The Cultivation Quality Evaluation of International Talents Based on Deep Learning

The average quality of Chinese talents, especially international talents, is falling behind the international level. For the introduction of high-level international talents, the evaluation of the cultivation quality of high-level international talents is an important factor, and this work is a basis of the said matter. However, most of the existing works related to this topic are just about the definitions of high-level international talents, their features, and importance, or discussions of other aspects. This paper aims to study the evaluation of the cultivation quality of international talents based on deep learning. At first, the paper gave a framework for the ideas of international talent cultivation, established an Evaluation Index System (EIS) containing 4 basic elements for such evaluation, and emphasized the importance of the training and testing of professional business English. At first, the paper gave a framework for the ideas of international talent cultivation, and established an Evaluation Index System (EIS) containing 4 basic elements for such evaluation. Then, this paper proposed the evaluation flow, and adopted an Empirical Mode Decomposition (EMD) algorithm to the analysis of the quality change during the cultivation process of international talents. After that, for the change trend of the cultivation quality of international talents at different time scales, this paper applied a Long Short Term Memory (LSTM) network for training and obtained the corresponding predicted quality value. At last, experimental results verified the effectiveness of the proposed model in predicting the cultivation quality evaluation of international talents.

When Patients Recover From COVID-19: Data-Driven Insights From Wearable Technologies.

Coronavirus disease 2019 (COVID-19) is known as a contagious disease and caused an overwhelming of hospital resources worldwide. Therefore, deciding on hospitalizing COVID-19 patients or quarantining them at home becomes a crucial solution to manage an extremely big number of patients in a short time. This paper proposes a model which combines Long-short Term Memory (LSTM) and Deep Neural Network (DNN) to early and accurately classify disease stages of the patients to address the problem at a low cost. In this model, the LSTM component will exploit temporal features while the DNN component extracts attributed features to enhance the model's classification performance. Our experimental results demonstrate that the proposed model achieves substantially better prediction accuracy than existing state-of-art methods. Moreover, we explore the importance of different vital indicators to help patients and doctors identify the critical factors at different COVID-19 stages. Finally, we create case studies demonstrating the differences between severe and mild patients and show the signs of recovery from COVID-19 disease by extracting shape patterns based on temporal features of patients. In summary, by identifying the disease stages, this research will help patients understand their current disease situation. Furthermore, it will also help doctors to provide patients with an immediate treatment plan remotely that addresses their specific disease stages, thus optimizing their usage of limited medical resources.

A novel global-local block spatial-spectral fusion attention model for hyperspectral image classification

ABSTRACT Deep learning brought a new method for hyperspectral image (HSI) classification, in which images are usually pre-processed by reducing their dimensions before being packaged into pieces to be input to the deep network for feature extraction. However, the learning capability of convolutional kernels of fixed dimensions is usually limited, and thus they are inclined to cause losses of feature details. In this paper, a new global-local block spatial-spectral fusion attention (GBSFA) model is proposed. An improved Inception structure is designed to extract the feature information of the global block, and the self-attention mechanism and spatial pyramid pooling (SPP) are applied to focus on the interclass edge feature information of the local block. Combined with long-short term memory (LSTM) networks, the effective information of the spectral dimension is extracted. Finally, the features extracted from the spatial dimension and the spectral dimension are conveyed in the full connection layer for classification training. Experimental results show that the classification accuracy of the proposed approach is higher than that of other comparative methods using small training sets.

A New Way of Airline Traffic Prediction Based on GCN-LSTM.

With the development of society and the improvement of people's material level, more and more people like to travel by airplane. If we can predict the passenger flow of an airline in advance, it can be used as an important decision-making basis for its flight route planning, crew scheduling planning and ticket price formulation in the process of management and operation. However, due to the high complexity of aviation network, the existing traffic prediction methods generally have the problem of low prediction accuracy. In order to overcome this problem, this paper makes full use of graph convolutional neural network and long—short memory network to construct a prediction system with short—term prediction ability. Specifically, this paper uses the graph convolutional neural network as a feature extraction tool to extract the key features of air traffic data, and solves the problem of long term and short term dependence between data through the long term memory network, then we build a high-precision air traffic prediction system based on it. Finally, we design a comparison experiment to compare the algorithm with the traditional algorithms. The results show that the algorithm we proposed in this paper has a higher accuracy in air flow prediction according to the lower loss function value.

A prison term prediction model based on fact descriptions by capturing long historical information

The legal judgments are always based on the description of the case, the legal document. However, retrieving and understanding large numbers of relevant legal documents is a time-consuming task for legal workers. The legal judgment prediction (LJP) focus on applying artificial intelligence technology to provide decision support for legal workers. The prison term prediction(PTP) is an important task in LJP which aims to predict the term of penalty utilizing machine learning methods, thus supporting the judgement. Long-Short Term Memory(LSTM) Networks are a special type of Recurrent Neural Networks(RNN) that are capable of handling long term dependencies without being affected by an unstable gradient. Mainstream RNN models such as LSTM and GRU can capture long-distance correlation but training is time-consuming, while traditional CNN can be trained in parallel but pay more attention to local information. Both have shortcomings in case description prediction. This paper proposes a prison term prediction model for legal documents. The model adds causal expansion convolution in general TextCNN to make the model not only limited to the most important keyword segment, but also focus on the text near the key segments and the corresponding logical relationship of this paragraph, thereby improving the predicting effect and the accuracy on the data set. The causal TextCNN in this paper can understand the causal logical relationship in the text, especially the relationship between the legal text and the prison term. Since the model uses all CNN convolutions, compared with traditional sequence models such as GRU and LSTM, it can be trained in parallel to improve the training speed and can handling long term. So causal convolution can make up for the shortcomings of TextCNN and RNN models. In summary, the PTP model based on causality is a good solution to this problem. In addition, the case description is usually longer than traditional natural language sentences and the key information related to the prison term is not limited to local words. Therefore, it is crucial to capture substantially longer memory for LJP domains where a long history is required. In this paper, we propose a Causality CNN-based Prison Term Prediction model based on fact descriptions, in which the Causal TextCNN method is applied to build long effective history sizes (i.e., the ability for the networks to look very far into the past to make a prediction) using a combination of very deep networks (augmented with residual layers) and dilated convolutions. The experimental results on a public data show that the proposed model outperforms several CNN and RNN based baselines.

Deep Learning for seasonality modelling in Inflation-Indexed Swap pricing

An Inflation-Indexed Swap (IIS) is a derivative in which, at every payment date, the counterparties swap an inflation rate with a fixed rate. For the calculation of the Inflation Leg cash flows it is necessary to build a mathematical model suitable for the Consumer Price Index (CPI) projection. For this purpose, quants typically start by using market quotes for the Zero-Coupon swaps in order to derive the future trend of the inflation index, together with a seasonality model for capturing the typical periodical effects. In this study, we propose a forecasting model for inflation seasonality based on a Long Short Term Memory (LSTM) network: a deep learning methodology particularly useful for forecasting purposes. The CPI predictions are conducted using a FinTech paradigm, but in respect of the traditional quantitative finance theory developed in this research field. The paper is structured according to the following sections: the first two parts illustrate the pricing methodologies for the most popular IIS: the Zero Coupon Inflation-Indexed Swap (ZCIIS) and the Year-on-Year Inflation-Indexed Swap (YYIIS); section 3 deals with the traditional standard method for the forecast of CPI values (trend + seasonality), while section 4 describes the LSTM architecture, and section 5 focuses on CPI projections, also called inflation bootstrap. Then section 6 describes a robust check, implementing a traditional SARIMA model in order to improve the interpretation of the LSTM outputs; finally, section 7 concludes with a real market case, where the two methodologies are used for computing the fair-value for a YYIIS and the model risk is quantified.

On the Evaluation and Implementation of LSTM Model for Speech Emotion Recognition using MFCC

Speech Emotion Recognition is an emerging research field and is expected to benefit many application domains by providing effective Human Computer Interface. Researchers are extensively working towards decoding of human emotions through speech signal in order to achieve effective interface and smart response by computers. The perfection of speech emotion recognition greatly depends upon the types of features used and also on the classifier employed for recognition. The contribution of this paper is to evaluate twelve different Long Short Term Memory (LSTM) networks models as classifier based on Mel-Frequency Cepstrum Coefficients (MFCC) feature. The paper presents performance evaluation in terms of important parameters such as: precision, recall, F-measure and accuracy for four emotions like happy, neutral, sad and angry using the emotional speech databases namely Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). The measurement accuracy obtained is 89% which is 9.5% more than reported in recent literature. The suitable LSTM model is further successfully implemented on Raspberry PI board creating standalone Speech Emotion Recognition system.

Real time monitoring of water Quality using IoT and Deep learning

Access to safe drinking water is one of the most pressing issues facing many developing countries. Water must meet Environmental Protection Agency (E.P.A.) requirements. The normal method of measuring physico-chemical parameters is to take samples manually and send them to the laboratory to check the water quality. In this paper, we proposed a new intelligent design of a real-time water quality monitoring system using Deep Learning technology. This system is composed of several sensors that allow us to measure water parameters (physico-chemical parameters), bacteriological parameters and organoleptic parameters) and to detect the presence of certain substances (undesirable substances, toxic substances) and of a single-board/mobile computer module, Internet and other accessories. Water parameters are automatically detected by the single-board computer. Raspberry Pi3 model B. The single board computer receives the data from the sensors and this data is sent to the web server using the Internet module. It is able to detect the water quality situation worldwide. The data will be analysed in real time. The application of deep learning to these areas has been an important research topic. The Long-Short Term Memory (LSTM) network has been shown to be well suited for processing and predicting large events with long intervals and delays in the time series. LSTM networks have the ability to retain long-term memory.

Implementation of Deep Learning Based Sentiment Classification and Product Aspect Analysis

With the increase in E-Commerce businesses in the last decade,the sentiment analysis of product reviews has gained a lot of attention in linguistic research. In literature, the survey depicts the majority of the research done emphasizes on mere polarity identification of the reviews. The proposed system emphasized on classifying the sentiment polarity and the product aspect identification from the reviews. Proposed work experimented with traditional machine learning techniques as well as deep neural networks such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and Long Short Term Memory(LSTM) Networks. The proposed system gives a better understanding of these algorithms by comparing the outcomes. The Deep Learning approach in the proposed work successfully provides a mechanism which identifies the review polarity and intensity of the reviews and also analyses the short form words used by people in the reviews. The experimental results in this work, applied on amazon product dataset, shows that the LSTM model works the best for sentiment analysis and intensity of reviews with 93% accuracy. This research work also predicts polarity for short-form word reviews which is the common trend these days while writing the reviews.

249 Predicting body weight of finishing pigs using machine and deep learning algorithms

Abstract Understanding and exploiting feeding patterns in swine could allow a reduced feed waste and minimized sorting losses. The objectives of this study were to evaluate the ability to predict whether a pig reached a target weight at finishing by using several algorithms and to compare the prediction using varying amounts of data during the growing period. Data were collected on 655 pigs from 75 to 166 days of age. Pigs were housed with 8 to 15 pigs and a Feed Intake Recording Equipment in each pen. Feed consumption, occupation time, and body weight per visit were recorded when a pig visited the feeder. Lasso Regression (LS), a machine learning algorithm: Random Forest (RF), and a deep learning algorithm: Long-short Term Memory (LSTM) network, were used to forecast whether pigs can reach 129 kg at the finishing stage (159–166 d). Times of visits, a sum of feed consumption, a sum of occupation time in the feeder every day, and age were used as predictors. Data were split into 6 slices by 14 days and used to calibrate the models and their predictive ability was tested with data corresponding to the last 8 days of the study period. The greatest correlation coefficients were 0.799, 0.828, and 0.868 using slice 6 (145–158 d) to train the LS, RF, and LSTM, respectively. The LS and LSTM algorithms had a smaller root mean squared error, 0.863 and 0.895 compared to the RF with 1.375 in the prediction. Overall, LS and LSTM performed best. Predictions using data closest to the finishing stage proved better. This study connects the dynamics of feeding behavior and feed intake data to growth using prediction methods that will hopefully accelerate the mainstream application of electronic feeders in pig production systems.

Enhancing Streamflow Forecast and Extracting Insights Using Long‐Short Term Memory Networks With Data Integration at Continental Scales

AbstractRecent observations with varied schedules and types (moving average, snapshot, or regularly spaced) can help to improve streamflow forecasts, but it is challenging to integrate them effectively. Based on a long short‐term memory (LSTM) streamflow model, we tested multiple versions of a flexible procedure we call data integration (DI) to leverage recent discharge measurements to improve forecasts. DI accepts lagged inputs either directly or through a convolutional neural network unit. DI ubiquitously elevated streamflow forecast performance to unseen levels, reaching a record continental‐scale median Nash‐Sutcliffe Efficiency coefficient value of 0.86. Integrating moving‐average discharge, discharge from the last few days, or even average discharge from the previous calendar month could all improve daily forecasts. Directly using lagged observations as inputs was comparable in performance to using the convolutional neural network unit. Importantly, we obtained valuable insights regarding hydrologic processes impacting LSTM and DI performance. Before applying DI, the base LSTM model worked well in mountainous or snow‐dominated regions, but less well in regions with low discharge volumes (due to either low precipitation or high precipitation‐energy synchronicity) and large interannual storage variability. DI was most beneficial in regions with high flow autocorrelation: it greatly reduced baseflow bias in groundwater‐dominated western basins and also improved peak prediction for basins with dynamical surface water storage, such as the Prairie Potholes or Great Lakes regions. However, even DI cannot elevate performance in high‐aridity basins with 1‐day flash peaks. Despite this limitation, there is much promise for a deep‐learning‐based forecast paradigm due to its performance, automation, efficiency, and flexibility.