Recurrent Neural Network Approach Research Articles

Learning and Subjective Expectation Formation: A Recurrent Neural Network Approach

I propose a flexible non-parametric method using Recurrent Neural Networks (RNN) to estimate a generalized model of expectation formation. This approach does not rely on restrictive assumptions of functional forms and parametric methods yet nests the standard approaches of empirical studies on expectation formation. Applying this approach to data on macroeconomic expectations from the Michigan Survey of Consumers (MSC) and a rich set of signals available to U.S. households, I document three novel findings: (1) agents' expectations about the future economic condition have asymmetric and non-linear responses to signals; (2) agents' attentions shift from signals about the current state to signals about the future: they behave as adaptive learners in ordinary periods and become forward-looking as the state of economy gets worse; (3) the content of signals on economic conditions, rather than the amount of news coverage on these signals, plays the most important role in creating the attention-shift. Double Machine Learning approach is then used to obtain statistical inferences of these empirical findings. Finally, I show these stylized facts can be generated by a model with rational inattention, in which information endogenously becomes more valuable when economic status worsens.

An augmented delays-dependent region partitioning approach for recurrent neural networks with multiple time-varying delays

This paper investigates the global asymptotic stability problem of recurrent neural networks with multiple time-varying delays. First, to make full use of the relationship between all delayed states x(t-τi)(i=1,…,N) and current state x(t), an augmented delays-dependent region partitioning (ADRP) approach is proposed. Then combining Wirtinger-based integral inequality and the reciprocally convex approach, two delay-dependent stability criteria with less conservatism are developed. Finally, two numerical simulations are shown to verify the effectiveness and the feasibility.

Residue buildup predictive modeling for stencil cleaning profile decision-making using recurrent neural network

This research proposes a novel framework to control the stencil cleaning profile selection in the stencil printing process (SPP). The SPP is a major contributor to yield loss in surface mount technology (SMT). Enhancement in SPP performance is critical to improving the printed circuit board (PCB) assembly line. The selection of a solvent-based or a dry-based cleaning profile is challenging, but the choice determines the effectiveness and efficiency of the stencil cleaning operation. The amount of residue buildup under the stencil is the main criterion used to decide the appropriate cleaning profile in SPP. In this research, a multi-dimensional temporal recurrent neural network (RNN) approach is used to accurately predict the amount of residue buildup on the underneath surface of the stencil in real-time. Specifically, the long short-term memory (LSTM) architecture is trained using actual residue buildup data. The proposed LSTM prediction model is compared with other state-of-the-art regression models such as multilayer perceptron (MLP) and ensemble learning models. Experimental results show the proposed LSTM model outperforms the state-of-the-art regression models and accurately predicts the stencil status. The proposed research aids decision-makers in the SPP line to select the appropriate stencil cleaning profile adaptively and in real-time. As a result, the overall SPP performance is improved.

Survey on RNN and CRF models for de-identification of medical free text

The increasing reliance on electronic health record (EHR) in areas such as medical research should be addressed by using ample safeguards for patient privacy. These records often tend to be big data, and given that a significant portion is stored as free (unstructured) text, we decided to examine relevant work on automated free text de-identification with recurrent neural network (RNN) and conditional random field (CRF) approaches. Both methods involve machine learning and are widely used for the removal of protected health information (PHI) from free text. The outcome of our survey work produced several informative findings. Firstly, RNN models, particularly long short-term memory (LSTM) algorithms, generally outperformed CRF models and also other systems, namely rule-based algorithms. Secondly, hybrid or ensemble systems containing joint LSTM-CRF models showed no advantage over individual LSTM and CRF models. Thirdly, overfitting may be an issue when customized de-identification datasets are used during model training. Finally, statistical validation of performance scores and diversity during experimentation were largely ignored. In our comprehensive survey, we also identify major research gaps that should be considered for future work.

Application of integrated recurrent neural network with multivariate adaptive regression splines on SPC-EPC process

The integration of statistical process control and engineering process control has been reported as an effective way to monitor and control the autocorrelated process. However, because engineering process control compensates for the effects of underlying disturbances, the disturbance patterns become very hard to recognize, especially when various abnormal control chart patterns are mixed and co-existed in the engineering process. In this study, a new control chart pattern recognition model which integrates multivariate adaptive regression splines and recurrent neural network is proposed to not only address the problem of feature selection (i.e., lagged process measurements) but also improve the pattern recognition accuracy. The performance of the proposed method is evaluated by comparing the recognition results of multivariate adaptive regression splines and recurrent neural network with the results of four competing approaches (multivariate adaptive regression splines-extreme learning machine, multivariate adaptive regression splines-random forest, single recurrent neural network, and single random forest) on the simulated individual process data. The experimental study shows that the proposed multivariate adaptive regression splines and recurrent neural network approach can not only solve the problem of variable selection but also outperform other competing models. Moreover, according to the lagged process measurements selected by the proposed approach, lagged observations that exerted significant impact on the construction of the control chart pattern recognition model can be identified successfully. This study has significant implications for research and practice in production management and provides a valuable reference for manufacturing process managers to better understand and develop strategies for control chart pattern recognition.

Player Identification in Hockey Broadcast Videos

We present a deep recurrent convolutional neural network (CNN) approach to solve the problem of hockey player identification in NHL broadcast videos. Player identification is a difficult computer vision problem mainly because of the players’ similar appearance, occlusion, and blurry facial and physical features. However, we can observe players’ jersey numbers over time by processing variable length image sequences of players (aka ‘tracklets’). We propose an end-to-end trainable ResNet+LSTM network, with a residual network (ResNet) base and a long short-term memory (LSTM) layer, to discover spatio-temporal features of jersey numbers over time and learn long-term dependencies. Additionally, we employ a secondary 1-dimensional convolutional neural network classifier as a late score-level fusion method to classify the output of the ResNet+LSTM network. For this work, we created a new hockey player tracklet dataset that contains sequences of hockey player bounding boxes. This achieves an overall player identification accuracy score over 87% on the test split of our new dataset.

Cryptocurrency malware hunting: A deep Recurrent Neural Network approach

In recent years, cryptocurrency trades have increased dramatically, and this trend has attracted cyber-threat actors to exploit the existing vulnerabilities and infect their targets. The malicious actors use cryptocurrency malware to perform complex computational tasks using infected devices. Since cryptocurrency malware threats perform a legal process, it is a challenging task to detect this type of threat by a manual or heuristic method. In this paper, we propose a novel deep Recurrent Neural Network (RNN) learning model for hunting cryptocurrency malware threats. Specifically, our proposed model utilizes the RNN to analyze Windows applications’ operation codes (Opcodes) as a case study. We collect a real-world dataset that comprises of 500 cryptocurrency malware and 200 benign-ware samples, respectively. The proposed model trains with five different Long Short-Term Memory (LSTM) structures and is evaluated by a 10-fold cross-validation (CV) technique. The obtained results prove that a 3-layer configuration model gains 98% of detection accuracy, which is the highest rate among other current configurations. We also applied traditional machine learning (ML) classifiers to show the applicability of deep learners (LSTM) versus traditional models in dealing with cryptocurrency malware.

Modelling Taxi Drivers’ Behaviour for the Next Destination Prediction

In this paper, we study how to model taxi drivers’ behavior and geographical information for an interesting and challenging task: the next destination prediction in a taxi journey. Predicting the next location is a well-studied problem in human mobility, which finds several applications in real-world scenarios, from optimizing the efficiency of electronic dispatching systems to predicting and reducing the traffic jam. This task is normally modeled as a multiclass classification problem, where the goal is to select, among a set of already known locations, the next taxi destination. We present a Recurrent Neural Network (RNN) approach that models the taxi drivers’ behavior and encodes the semantics of visited locations by using geographical information from Location-Based Social Networks (LBSNs). In particular, the RNNs are trained to predict the exact coordinates of the next destination, overcoming the problem of producing, in output, a limited set of locations, seen during the training phase. The proposed approach was tested on the ECML/PKDD Discovery Challenge 2015 dataset—based on the city of Porto—, obtaining better results with respect to the competition winner, whilst using less information, and on Manhattan and San Francisco datasets.

Long short-term memory embedded nudging schemes for nonlinear data assimilation of geophysical flows

Reduced rank nonlinear filters are increasingly utilized in data assimilation of geophysical flows but often require a set of ensemble forward simulations to estimate forecast covariance. On the other hand, predictor–corrector type nudging approaches are still attractive due to their simplicity of implementation when more complex methods need to be avoided. However, optimal estimate of the nudging gain matrix might be cumbersome. In this paper, we put forth a fully nonintrusive recurrent neural network approach based on a long short-term memory (LSTM) embedding architecture to estimate the nudging term, which plays a role not only to force the state trajectories to the observations but also acts as a stabilizer. Furthermore, our approach relies on the power of archival data, and the trained model can be retrained effectively due to the power of transfer learning in any neural network applications. In order to verify the feasibility of the proposed approach, we perform twin experiments using the Lorenz 96 system. Our results demonstrate that the proposed LSTM nudging approach yields more accurate estimates than both the extended Kalman filter (EKF) and ensemble Kalman filter (EnKF) when only sparse observations are available. With the availability of emerging artificial intelligence friendly and modular hardware technologies and heterogeneous computing platforms, we articulate that our simplistic nudging framework turns out to be computationally more efficient than either the EKF or EnKF approaches.

A semi-supervised recurrent neural network for video salient object detection

A semi-supervised, one-dimensional recurrent neural network (RNN) approach called RVS has been proposed in this paper for video salient object detection. The proposed RVS approach involves the processing of each frame independently without explicitly considering temporal information. The RNN is trained using one-dimensional superpixel features to classify the salient object regions into salient foreground and non-salient background superpixels. Deep learning algorithms generally exhibit heavy dependence on training data size and often take extremely long time for training. On the contrary, the proposed RVS approach involves the training of an RNN using a small data which results in significant reduction in training time. The RVS approach has been extensively evaluated and its results are compared with those of several state-of-the-art methods using the public-domain VideoSeg, SegTrack v1 and SegTrack v2 benchmark video datasets. Further, the RVS approach has been tested using the authors’ own video dataset and the complex DAVIS and video object segmentation datasets to evaluate the impact of motion and blur on its performance. The RVS approach delivers results superior to those of several approaches that strongly rely upon spatio-temporal features in detecting the salient objects from the video sequences.

Distributed long-term hourly streamflow predictions using deep learning – A case study for State of Iowa

Accurate streamflow forecasting has always been a challenge. Although there were many novel approaches using deep learning models, accuracy of these models is often limited to a short lead time. This study proposes a new deep recurrent neural network approach, Neural Runoff Model (NRM), which has been applied on 125 USGS streamflow gages in the State of Iowa for predicting the next 120 h. We use a semi-distributed model structure with observation and forecast data from the model output of upstream stations as additional input for downstream gages. The proposed model outperforms the streamflow persistence, ridge regression and random forest regression on majority of the gages. Our model has shown strong predictive power and can be used for long-term streamflow predictions. This study also shows that the semi-distributed structure in NRM can improve the streamflow predictions by integrating water level data from upstream stream gauges.

Deep Sentiment Classification and Topic Discovery on Novel Coronavirus or COVID-19 Online Discussions: NLP Using LSTM Recurrent Neural Network Approach.

Internet forums and public social media, such as online healthcare forums, provide a convenient channel for users (people/patients) concerned about health issues to discuss and share information with each other. In late December 2019, an outbreak of a novel coronavirus (infection from which results in the disease named COVID-19) was reported, and, due to the rapid spread of the virus in other parts of the world, the World Health Organization declared a state of emergency. In this paper, we used automated extraction of COVID-19-related discussions from social media and a natural language process (NLP) method based on topic modeling to uncover various issues related to COVID-19 from public opinions. Moreover, we also investigate how to use LSTM recurrent neural network for sentiment classification of COVID-19 comments. Our findings shed light on the importance of using public opinions and suitable computational techniques to understand issues surrounding COVID-19 and to guide related decision-making. In addition, experiments demonstrated that the research model achieved an accuracy of 81.15% - a higher accuracy than that of several other well-known machine-learning algorithms for COVID-19-Sentiment Classification.

A recurrent neural network for urban long-term traffic flow forecasting

This paper investigates the use of recurrent neural network to predict urban long-term traffic flows. A representation of the long-term flows with related weather and contextual information is first introduced. A recurrent neural network approach, named RNN-LF, is then proposed to predict the long-term of flows from multiple data sources. Moreover, a parallel implementation on GPU of the proposed solution is developed (GRNN-LF), which allows to boost the performance of RNN-LF. Several experiments have been carried out on real traffic flow including a small city (Odense, Denmark) and a very big city (Beijing). The results reveal that the sequential version (RNN-LF) is capable of dealing effectively with traffic of small cities. They also confirm the scalability of GRNN-LF compared to the most competitive GPU-based software tools when dealing with big traffic flow such as Beijing urban data.

Deep learning with knowledge transfer for explainable anomaly prediction in wind turbines

AbstractThe last decade has witnessed an increased interest in applying machine learning techniques to predict faults and anomalies in the operation of wind turbines. These efforts have lately been dominated by deep learning techniques which, as in other fields, tend to outperform traditional machine learning algorithms given sufficient amounts of training data. An important shortcoming of deep learning models is their lack of transparency—they operate as black boxes and typically do not provide rationales for their predictions, which can lead to a lack of trust in predicted outputs. In this article, a novel hybrid model for anomaly prediction in wind farms is proposed, which combines a recurrent neural network approach for accurate classification with an XGBoost decision tree classifier for transparent outputs. Experiments with an offshore wind turbine show that our model achieves a classification accuracy of up to 97%. The model is further able to generate detailed feature importance analyses for any detected anomalies, identifying exactly those components in a wind turbine that contribute to an anomaly. Finally, the feasibility of transfer learning is demonstrated for the wind domain by porting our “offshore” model to an unseen dataset from an onshore wind farm. The latter model achieves an accuracy of 65% and is able to detect 85% of anomalies in the unseen domain. These results are encouraging for application to wind farms for which no training data are available, for example, because they have not been in operation for long.

A Novel Framework of Real-Time Regional Collision Risk Prediction Based on the RNN Approach

Regional collision risk identification and prediction is important for traffic surveillance in maritime transportation. This study proposes a framework of real-time prediction for regional collision risk by combining Density-Based Spatial Clustering of Applications with Noise (DBSCAN) technique, Shapley value method and Recurrent Neural Network (RNN). Firstly, the DBSCAN technique is applied to cluster vessels in specific sea area. Then the regional collision risk is quantified by calculating the contribution of each vessel and each cluster with Shapley value method. Afterwards, the optimized RNN method is employed to predict the regional collision risk of specific seas in short time. As a result, the framework is able to determine and forecast the regional collision risk precisely. At last, a case study is carried out with actual Automatic Identification System (AIS) data, the results show that the proposed framework is an effective tool for regional collision risk identification and prediction.

Improving the Reliability of Probabilistic Multi-Step-Ahead Flood Forecasting by Fusing Unscented Kalman Filter with Recurrent Neural Network

It is fundamentally challenging to quantify the uncertainty of data-driven flood forecasting. This study introduces a general framework for probabilistic flood forecasting conditional on point forecasts. We adopt an unscented Kalman filter (UKF) post-processing technique to model the point forecasts made by a recurrent neural network and their corresponding observations. The methodology is tested by using a long-term 6-h timescale inflow series of the Three Gorges Reservoir in China. The main merits of the proposed approach lie in: first, overcoming the under-prediction phenomena in data-driven flood forecasting; second, alleviating the uncertainty encountered in data-driven flood forecasting. Two commonly used artificial neural networks, a recurrent and a static neural network, were used to make the point forecasts. Then the UKF approach driven by the point forecasts demonstrated its competency in increasing the reliability of probabilistic flood forecasts significantly, where predictive distributions encountered in multi-step-ahead flood forecasts were effectively reduced to small ranges. The results demonstrated that the UKF plus recurrent neural network approach could suitably extract the complex non-linear dependence structure between the model’s outputs and observed inflows and overcome the systematic error so that model reliability as well as forecast accuracy for future horizons could be significantly improved.

A recurrent neural network approach to predicting hemoglobin trajectories in patients with End-Stage Renal Disease.

The most severe form of kidney disease, End-Stage Renal Disease (ESRD) is treated with various forms of dialysis - artificial blood cleansing. Dialysis patients suffer many health burdens including high mortality and hospitalization rates, and symptomatic anemia: a low red blood cell count as indicated by a low hemoglobin (Hgb) level. ESRD-induced anemia is treated, with variable patient response, by erythropoiesis stimulating agents (ESAs): expensive injectable medications typically administered during dialysis sessions. The dosing protocol is typically a population level protocol based on original clinical trials, the use of which often results in Hgb cycling. This cycling phenomenon occurs primarily due to the mismatch in the time between dosing decisions and the time it takes for the effects of a dosing change to be fully realized. In this paper we develop a recurrent neural network approach that uses historic data together with future ESA and iron dosing data to predict the 1, 2, and 3 month Hgb levels of patients with ESRD-induced anemia. The results of extensive experimentation indicate that this approach generates predictions that are clinically relevant: the mean absolute error of the predictions is comparable to estimates of the intra-individual variability of the laboratory test for Hgb.

Bidirectional Molecule Generation with Recurrent Neural Networks.

Recurrent neural networks (RNNs) are able to generate de novo molecular designs using simplified molecular input line entry systems (SMILES) string representations of the chemical structure. RNN-based structure generation is usually performed unidirectionally, by growing SMILES strings from left to right. However, there is no natural start or end of a small molecule, and SMILES strings are intrinsically nonunivocal representations of molecular graphs. These properties motivate bidirectional structure generation. Here, bidirectional generative RNNs for SMILES-based molecule design are introduced. To this end, two established bidirectional methods were implemented, and a new method for SMILES string generation and data augmentation is introduced-the bidirectional molecule design by alternate learning (BIMODAL). These three bidirectional strategies were compared to the unidirectional forward RNN approach for SMILES string generation, in terms of the (i) novelty, (ii) scaffold diversity, and (iii) chemical-biological relevance of the computer-generated molecules. The results positively advocate bidirectional strategies for SMILES-based molecular de novo design, with BIMODAL showing superior results to the unidirectional forward RNN for most of the criteria in the tested conditions. The code of the methods and the pretrained models can be found at URL https://github.com/ETHmodlab/BIMODAL.

Basic Food Basket Monthly Price of Southern Bahia Cities: A Time Series Forecasting with Deep Learning Using a Recurrent Neural Network Approach

The goal of time series analysis is extract non-trivial information from chronological sorted points. Time series data creation it is not a difficult task, however the same cannot be said about its analysis and predictions, with these processes being considered in 2006 one of the ten main challenges of data mining research field. The time series utilized in this research was an economic time series with the monthly prices of the basic food basket in Ilheus and Itabuna cities. When analyzing time series, different methods can be applied, but recently the methods related to Deep Learning are becoming more popular, with recurrent neural networks (RNNs) being the most popular among them. This paper experimented an RNN architecture in three different experiments, predicting in different time terms the prices of the basic food basket time series of 2019 and 2020. All experiments presented a satisfactory prediction result and sensitivity to the real values series behavior. These results suggest that the RNN architecture can generalize well the time series to periods yet not seen by the network and that the NN can have an even better performance in a more data abundant environment.

Feedback Based Adaptive Recurrent Neural Network for Cancer Detection using Gene Data Pattern

Cancer detecting technology plays a vital role in the medical community. Researches have shown that patients that are affected by cancer carry same type of genetic patterns in their DNA. With this in mind, this research work concentrates on analysing gene pattern for detecting cancer using deep learning algorithms. The Feedback based Adaptive Recurrent Neural Network (FA-RNN) approach is designed to classify and analyse the gene pattern recognition. The data augmentation is done to improve the quality of the input data from COSMIC dataset which includes the detection of missing values, removing the noise present in input using multiple imputations and reducing higher base value can be done using dimensionality reduction process. After obtaining the improved dataset, the training phase begins by estimating the exact weight value of feedback layer using feedback weight loop calculation technique to lessen number of repetition during training. Moreover, the error calculation is done to evaluate the exact weight values of feedback layer used for classification. Finally the classification is done by selecting the next appropriate hidden neuron using the neuron selection activation function. The performance of the Feedback based Adaptive Recurrent Neural Network technique can be analysed using the evaluation metrics accuracy, computation time and Root Mean Square Error (RMSE) and the attained results are compared with the Recursive Neural Network(RNN) and Convolutional Neural Network(CNN) algorithms. The obtained results such as higher accuracy, reduced RMSE and less computation time in Feedback based Adaptive Recurrent Neural Network indicates that it performs the enhanced operation than CNN and RNN.