Long Short-term Memory Recurrent Neural Network Research Articles

Modeling dynamic construction work template from existing scheduling records via sequential machine learning

Today, construction planning and scheduling is almost always performed manually, by experienced practitioners. The knowledge of those individuals is materialized, maintained, and propagated through master schedules and look-ahead plans. While historical project schedules are available, manually mining their embedded knowledge to create generic work templates for future projects or revising look-ahead schedules is very difficult, time-consuming and error-prone. The rigid work templates from prior research are also not scalable to cover the inter and intra-class variability in historical schedule activities. This paper aims at fulfilling these needs via a new method to automatically learn construction knowledge from historical project planning and scheduling records and digitize such knowledge in a flexible and generalizable data schema. Specifically, we present Dynamic Process Templates (DPTs) based on a novel vector representation for construction activities where the sequencing knowledge is modeled with generative Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs). Our machine learning models are exhaustively tested and validated on a diverse dataset of 32 schedules obtained from real-world projects. The experimental results show our method is capable of learning planning and sequencing knowledge at high accuracy across different projects. The benefits for automated project planning and scheduling, schedule quality control, and automated generation of project look-aheads are discussed in detail.

A Classification Method for Power-Quality Disturbances Using Hilbert–Huang Transform and LSTM Recurrent Neural Networks

Power quality disturbances are one of the main problems in an electric power system, where deviations in the voltage and current signals can be evidenced. These sudden changes are potential causes of malfunctions and could affect equipment performance at different demand locations. For this reason, a classification strategy is essential to provide relevant information related to the occurrence of the disturbance. Nevertheless, traditional data extraction and detection methods have failed to carry out the classification process with the performance required, in terms of accuracy and efficiency, due to the presence of a non-stationary and non-linear dynamics, specific of these signals. This paper proposes a hybrid approach that involves the implementation of the Hilbert–Huang Transform (HHT) and long short-term memory (LSTM), recurrent neural networks (RNN) to detect and classify power quality disturbances. Nine types of synthetic signals were reproduced and pre-processed taking into account the mathematical models and their specifications established in the IEEE 1159 standard. In order to eliminate the presence of mode mixing, the ensemble empirical decomposition (EEMD) and masking signal methods were implemented. Additionally, based on the successful benefits of LSTM RNNs reported in the literature, associated to the high accuracy rates achieved at learning long short-term dependencies, this classification technique is implemented to analyze the sequences obtained from the HHT. Based on the experimental results, it is possible to show that the ensemble recognition approach using the EEMD yields a better classification accuracy rate (98.85%) compared with the masking signal and the traditional HHT approach.

A new approach of ECG steganography and prediction using deep learning

In this paper, a new approach of ECG steganography of hiding patient’s confidential information is proposed. As steganography results in distortion within the ECG signal which hampers the clinical features, in this work, encryption was performed within TP-segment of ECG. Additionally, segment classification and feature extraction were used for data concealing within normal TP-segments, while keeping abnormal segments unaffected. To reduce the computational complexity and execution time, encryption was performed in time domain signal, using a new approach. Finally, after decryption of hidden data, to predict original sample values of modified TP-segments, a long short-term memory recurrent neural network (LSTM-RNN) was used which efficiently reduced the error between the original and predicted signal. This algorithm was successfully implemented on mitdb, ptbdb and European ST-T database, available in physionet and percent root mean square difference (PRD), PRD normalized (PRDN) were obtained less than 1% along with signal to noise ratio (SNR) and peak SNR (PSNR) more than 80 dB. It was observed that this algorithm provided better result among other frequency domain techniques and recently published works.

A Recurrent Neural Network based deep learning model for offline signature verification and recognition system

With the recent advancement in information technology field, the demand to develop a person authentication system through verifying their offline signatures is gradually increasing. This type of system may be used to verify various official documents through verifying the signatures of the concerned persons present in the documents. This article proposes a Recurrent Neural Network (RNN), a deep learning network, based method to verify and recognize offline signatures of different persons. Various structural and directional features have been extracted locally from each signature sample and the generated feature vectors have been studied using two different models of RNN—long-short term memory (LSTM) and bidirectional long–short term memory (BLSTM). The performance of the proposed system has been tested on six widely used public signature databases—GPDS synthetic, GPDS-300, MCYT-75, CEDAR, BHSig260 Hindi, and BHSig260 Bengali. Experiment has also been performed using Convolutional Neural Network (CNN) to have a comparison with RNN based results. Experimental results demonstrate that the proposed RNN based signature verification and recognition system is superior over CNN and also outperforms the existing state-of-the-art results in this regard.

WITHDRAWN: A proposal of bidirectional grid long short term memory based model for user intention identification in on-line search query using text

Abstract Whenever any search query is entered in to a search engine by the user, the set of web pages which are deemed to be relevant for the query being made are identified with help of an index and some algorithms to rank the web pages order at the time of display. Any result set of search query would be effective if it matches exactly with the user requirements and for this to happen understanding user intention behind the search query entered plays vital role. Research in Machine learning algorithms has shown tremendous promising results in identifying user intention. In this paper authors have done a brief survey of work related to working and application of machine learning methods in identifying user intention behind query entered and based on observations, the authors propose a Bi Directional Grid Long-Short Term Memory RNN frame work for enhancing the efficiency of user intention identification.

Lifetime Prediction of Ultraviolet Light-Emitting Diodes Using a Long Short-Term Memory Recurrent Neural Network

Ultraviolet light-emitting diodes (UV LEDs) play an important role in inactivating novel coronavirus pneumonia, but the lack of rapid lifetime prediction can easily cause untimely failure detection, long product development cycles, and high costs. This study predicts the lifetime of UV LEDs based on the long short-term memory (LSTM) recurrent neural network (RNN). First, the equipment setup was designed to conduct an aging test to obtain a predicted length of life for the UV LED samples using a Weibull distribution. Next, LSTM RNN was employed to predict the lifetime of the UV LEDs based on the radiation power degradation. The results were then compared with those from nonlinear least squares (NLS) regression recommended by the IESNA TM-21 industry standard. Finally, the robustness of the two methods was analyzed by changing the starting times of the predictions. The results showed that the LSTM RNN proposed in this letter reveals not only a 29.7% lower lifetime prediction error compared with the NLS regression, but also a more stable robustness. Thus, the LSTM RNN method is found to be more accurate and more robust in predicting the lifetime of UV LEDs.

Circuit Health State Estimation via An Integrated Deep Neural Network

accelerating the industry to become more integrated and intricated. It is almost inevitable for a system to encounter failures during its whole life span. Thus, it is imperative to monitor the operating system from a system-level perspective to avoid potential catastrophes. Intuitively, inclusive prior knowledge is required for prognostics and health management (PHM). However, due to time-varying parameters and external conditions, the system is usually too complex to neatly fit into a prior-built model. This paper presents a novel pragmatic method, encompassing the convolutional autoencoder (CAE) and long short-term memory recurrent neural network (LSTM-RNN), to track the health state of a circuit. Briefly, the proposed method can be divided into two steps. First, degradation characteristics are extracted by using the time-domain features and CAE to prepare for the later health state estimation step. Then, the LSTM-RNN is used to finish the predictive process, i.e., to map the extracted abstract features to the health state. In addition, the degradation of a practical circuit considering the angular distance is discussed to quantify the health state of the circuit system. Furthermore, a case study based on that prognostics scheme is conducted to verify the proposed method. The comparison with other existing popular methods indicates the superiority of the proposed methodology.

Predicting Vacant Parking Space Availability: A Long Short-Term Memory Approach

The accurate prediction of vacant parking space availability is becoming increasingly essential for assisting drivers to determine where to park in advance. It helps ease traffic pressure and reduce gas emission and pollution. This article proposes a novel multistep long short-term memory recurrent neural network (LSTM-NN) model to predict the number of the vacant parking spaces on the basis of historical parking availability information. The key parameters of the model are deeply optimized. The prediction model is fully benchmarked with five well-known machine learning models&#x2014;i.e. a gated recurrent units neural network, a stacked autoencoder, a support vector regression, a back propagation neural network, and a <i>k</i>-nearest neighbor algorithm&#x2014;whose key parameters are sufficiently optimized as well. Adequate experiments with practical data collected from two parking lots with various capacities and traffic flows were conducted to evaluate the models&#x2019; performances on short- and long-term predictions. Experimental results show that the proposed multistep LSTM-NN model outperforms all the benchmark models, especially in a commercial parking lot with heavy traffic flow.

Reduced order framework for convection dominant and pure diffusive problems based on combination of deep long short‐term memory and proper orthogonal decomposition/dynamic mode decomposition methods

AbstractIn many real‐world applications, mathematical models are highly complex, and numerical simulations in high‐dimensional systems are challenging. Model order reduction is a useful method to obtain a reasonable approximation by significantly reducing the computational cost of such problems. Deep learning technology is a recent improvement in artificial neural networks that can find more hidden information from the data. Deep learning has the advantage of processing data in its raw form and trains the nonlinear system with different levels of representation and predicts the data. In this article, a reduced order model framework based on a combination of deep learning [long short‐term memory (LSTM)] and proper orthogonal decomposition/dynamic mode decomposition (POD/DMD) modes is presented. Due to the robustness and stability of the LSTM recurrent neural network in predicting chaotic dynamical systems, we consider LSTM architecture to develop our data‐driven reduced order modeling (ROM). We investigate the proposed method performance by solving two well‐known canonical cases: a steady shear flow exhibiting the Kelvin‐Helmholtz instability, and two‐dimensional and unsteady mass diffusion equation. The focus of this article is to use LSTM deep recursive neural network to learn the time dynamics and POD/DMD to generate the order reduction model. The results show that the proposed method is very accurate in predicting time dynamics and input reconstruction.

Influence of medical domain knowledge on deep learning for Alzheimer's disease prediction

Background and objectiveAlzheimer's disease (AD) is the most common type of dementia that can seriously affect a person's ability to perform daily activities. Estimates indicate that AD may rank third as a cause of death for older people, after heart disease and cancer. Identification of individuals at risk for developing AD is imperative for testing therapeutic interventions. The objective of the study was to determine could diagnostics of AD from EMR data alone (without relying on diagnostic imaging) be significantly improved by applying clinical domain knowledge in data preprocessing and positive dataset selection rather than setting naïve filters. MethodsData were extracted from the repository of heterogeneous ambulatory EMR data, collected from primary care medical offices all over the U.S. Medical domain knowledge was applied to build a positive dataset from data relevant to AD. Selected Clinically Relevant Positive (SCRP) datasets were used as inputs to a Long-Short-Term Memory (LSTM) Recurrent Neural Network (RNN) deep learning model to predict will the patient develop AD. ResultsRisk scores prediction of AD using the drugs domain information in an SCRP AD dataset of 2,324 patients achieved high out-of-sample score - 0.98-0.99 Area Under the Precision-Recall Curve (AUPRC) when using 90% of SCRP dataset for training. AUPRC dropped to 0.89 when training the model using less than 1,500 cases from the SCRP dataset. The model was still significantly better than when using naïve dataset selection. ConclusionThe LSTM RNN method that used data relevant to AD performed significantly better when learning from the SCRP dataset than when datasets were selected naïvely. The integration of qualitative medical knowledge for dataset selection and deep learning technology provided a mechanism for significant improvement of AD prediction.Accurate and early prediction of AD is significant in the identification of patients for clinical trials, which can possibly result in the discovery of new drugs for treatments of AD. Also, the contribution of the proposed predictions of AD is a better selection of patients who need imaging diagnostics for differential diagnosis of AD from other degenerative brain disorders.

Gamma ray log generation from drilling parameters using deep learning

Lithology identification plays a vital role in defining the petroleum reservoir. Although well logging represents the traditional means of obtaining petrophysical data for lithology identification, there could be cases where logging while drilling instruments may fail during drilling. This paper presents an approach that utilizes drilling parameters obtained from mud logging and measurement while drilling (MWD) for real-time prediction of gamma ray log which is used as a lithology identifier. In this work, several machine learning methodologies, such as simple recurrent neural network (RNN), long short-term memory recurrent neural network (LSTM-RNN), temporal convolution network (TCN), gated recurrent unit (GRU) network, nonlinear autoregressive network with exogenous inputs (NARX) and simple artificial neural network (ANN) were tested for their ability to capture the relationship between hydro-mechanical specific energy (computed from drilling parameters) and gamma ray log. A recently drilled exploration gas well in the tertiary deltaic system of the Niger delta basin was used as the case study. Based on the field data, the results show the TCN and simple RNN performed best. The receptive field of the TCN played a significant role in its performance, and therefore, the LSTM-RNN can be made to perform comparable to that of TCN/simple RNN if the LSTM-RNN is manually made to work with an optimal window of input data points for each output data point. The size and nature of the data (volume, velocity, variety, and veracity) is likely a significant factor in the performance of the machine learning methodologies. Thus, it is recommended to focus on obtaining the best receptive field of the data during the machine learning development phase.

State of Charge Estimation Method for Lithium-Ion Batteries in All-Electric Ships Based on LSTM Neural Network

Abstract All-electric ships (AES) are considered an effective solution for reducing greenhouse gas emissions as they are a platform to use clean energy sources such as lithium-ion batteries, fuel cells and solar cells instead of fossil fuel. Even though these batteries are a promising alternative, the accuracy of the battery state of charge (SOC) estimation is a critical factor for their safe and reliable operation. The SOC is a key indicator of battery residual capacity. Its estimation can effectively prevent battery over-discharge and over-charge. Next, this enables reliable estimation of the operation time of fully electric ferries, where little time is spent at the harbour, with limited time available for charging. Thus, battery management systems are essential. This paper presents a neural network model of battery SOC estimation, using a long short-term memory (LSTM) recurrent neural network (RNN) as a method for accurate estimation of the SOC in lithium-ion batteries. The current, voltage and surface temperature of the batteries are used as the inputs of the neural network. The influence of different numbers of neurons in the neural network’s hidden layer on the estimation error is analysed, and the estimation error of the neural network under different training times is compared. In addition, the hidden layer is varied from 1 to 3 layers of the LSTM nucleus and the SOC estimation error is analysed. The results show that the maximum absolute SOC estimation error of the LSTM RNN is 1.96% and the root mean square error is 0.986%, which validates the feasibility of the method.

Distracted driver detection by combining in-vehicle and image data using deep learning

Distracted driving is among the most important reasons for traffic accidents today. Recently, there is an increasing interest in building driver assistance systems that detect the actions of the drivers and help them drive safer. In these studies, although some distinct data types such as the physical conditions of the driver, audio and visual features, car information are used; the main data source is the images of the driver that include the face, arms, and hands taken with a camera placed inside the car. In this work, we propose to integrate sensor data into the vision-based distracted driver detection model to improve the generalization ability of the system. With this purpose, we created a new data set that includes driver images and sensor data collected from real-world drives. Then, we constructed a two-stage distracted driving detection system to detect nine distracted behaviors. In the first stage, vision-based Convolutional Neural Network (CNN) models were created by transfer learning and fine-tuning methods. In the second stage, Long-Short Term Memory-Recurrent Neural Network (LSTM-RNN) models were created using sensor and image data together. We evaluate our system by two different fusion techniques and show that integrating sensor data to image-based driver detection significantly increases the overall performance with both of the fusion techniques. We also show that the accuracy of the vision-based model increases by fine-tuning the pre-trained CNN model using a related public dataset.

Nucleation and growth of cholesteric collagen tactoids: A time-series statistical analysis based on integration of direct numerical simulation (DNS) and long short-term memory recurrent neural network (LSTM-RNN)

HypothesisLiquid-crystalline phase separation by nucleation and growth (NG) is a crucial step in the formation of collagen-based biomaterials. However, the fundamental mechanisms are not completely understood for chiral lyotropic colloidal mesogens such as collagen. MethodologyTo capture the dynamics of NG under a quenching process into the biphasic equilibrium zone, we use direct numerical simulation based on the time-dependent Ginzburg-Landau model allowing minimization of the total free energy comprised of five key contributions: phase separation (Flory-Huggins), ordering (Landau-de Gennes), chiral orientational elasticity (Frank-Oseen-Mermin), interfacial and coupling effects. LSTM-RNN is applied as a surrogate model to greatly enrich the results. Significant correlations are established using Symbolic Regression. FindingsWe quantify the NG boundaries existing in the collagen phase diagram that has recently been developed and validated by our thermodynamic model (Khadem and Rey, 2019 [1]). We characterize the three NG stages (induction, nucleation, and coarsening) in terms of tactoids’ shape, morphology, growth laws, and population across the NG zone. Wide-range generic correlations are developed, revealing the quench depth dependence of NG characteristics and connecting the sequential NG stages. We confirm experimental observations on time-dependent growth law exponent changes from an initial n≈0.5 for the mass transfer limited regime to n≈1 for the volume-driven phase ordering regime upon increasing quench depth during the nucleation period and having exclusively a value of n≈0.5 for the coarsening period regardless of quench depth. We lastly uncover the underlying physics behind the NG phenomena.

Toward Packet Routing With Fully Distributed Multiagent Deep Reinforcement Learning

Packet routing is one of the fundamental problems in computer networks in which a router determines the next-hop of each packet in the queue to get it as quickly as possible to its destination. Reinforcement learning (RL) has been introduced to design autonomous packet routing policies with local information of stochastic packet arrival and service. However, the curse of dimensionality of RL prohibits the more comprehensive representation of dynamic network states, thus limiting its potential benefit. In this article, we propose a novel packet routing framework based on <i>multiagent</i> deep RL (DRL) in which each router possess an <i>independent</i> long short term memory (LSTM) recurrent neural network (RNN) for training and decision making in a <i>fully distributed</i> environment. The LSTM RNN extracts routing features from rich information regarding backlogged packets and past actions, and effectively approximates the value function of Q-learning. We further allow each route to communicate periodically with direct neighbors so that a broader view of network state can be incorporated. The experimental results manifest that our multiagent DRL policy can strike the delicate balance between congestion-aware and shortest routes, and significantly reduce the packet delivery time in general network topologies compared with its counterparts.

A LSTM-RNN-Assisted Vector Tracking Loop for Signal Outage Bridging

Global Navigation Satellite System (GNSS) has been the most popular tool for providing positioning, navigation, and timing (PNT) information. Some methods have been developed for enhancing the GNSS performance in signal challenging environments (urban canyon, dense foliage, signal blockage, multipath, and none-line-of-sight signals). Vector Tracking Loop (VTL) was recognized as the most promising and prospective one among these technologies, since VTL realized mutual aiding between channels. However, momentary signal blockage from part of the tracking channels affected the VTL operation and the navigation solution estimation. Moreover, insufficient available satellites employed would lead to the navigation solution errors diverging quickly over time. Short-time or temporary signal blockage was common in urban areas. Aiming to improve the VTL performance during the signal outage, in this paper, the deep learning method was employed for assisting the VTL navigation solution estimation; more specifically, a Long Short-Term Memory-Recurrent Neural Network (LSTM-RNN) was employed to aid the VTL navigation filter (navigation filter was usually a Kalman filter). LSTM-RNN obtained excellent performance in time-series data processing; therefore, in this paper, the LSTM-RNN was employed to predict the navigation filter innovative sequence values during the signal outage, and then, the predicted innovative values were employed to aid the navigation filter for navigation solution estimation. The LSTM-RNN was well trained while the signal was normal, and the past innovative sequence was employed as the input of the LSTM-RNN. A simulation was designed and conducted based on an open-source Matlab GNSS software receiver; a dynamic trajectory with several temporary signal outages was designed for testing the proposed method. Compared with the conventional VTL, the LSTM-RNN-assisted VTL could keep the horizontal positioning errors within 50 meters during a signal outage. Also, conventional Support Vector Machine (SVM) and radial basis function neural network (RBF-NN) were compared with the LSTM-RNN method; LSTM-RNN-assisted VTL could maintain the positioning errors less than 20 meters during the outages, which demonstrated LSTM-RNN was superior to the SVM and RBF-NN in these applications.

RETRACTED: Forecasting biofuel production using adaptive integrated optimization network model

Abstract Recently, biofuel production has been increasing all over the world over the past few decades owing to the relative changes in energy price and policy intervention. Several research studies are focussed on producing optimal biofuel production using numerous approaches. However, those approaches have few drawbacks such as computational complexity, less accuracy, high error value rate, etc. So to overcome such shortcomings, this paper aims in developing a novel Adaptive Integrated Optimization Network (AION approach) to attain optimal biofuel production with high accuracy and minimum error value rate. Moreover, the proposed AION approach comprises four significant phases namely Pre-processing of data, Re-construction of component, Prediction of an individual and Ensemble forecasting. The initial process utilizes an empirical mode decomposition (EMD) approach to divide a raw complicated data. The reconstruction phase employs the Fine to coarse (FTC) reconstructing approach that rebuilds the IMF into two different t-testing components namely the low-frequency component and the high-frequency component. Moreover, Gray Wolf Optimization (GWO) based Adaptive Wavelet Neural Network (AWNN) and Long Short Term Memory-Recurrent Neural Network (LSTM-RNN) approaches are applied to predict the low and the high-frequency components. Then the forecasted results are summed by using an ADD operator so as to obtain absolute forecasting results. The experimental analysis reveals that the proposed AION approach provides an optimal biofuel production with enhanced accuracy, minimum error value rate, and complexity issues.

PREDICT URBAN AIR POLLUTION IN SURABAYA USING RECURRENT NEURAL NETWORK – LONG SHORT TERM MEMORY

Air is one of the primary needs of living things. If the condition of air is polluted, then the lives of humans and other living things will be disrupted. So it is needed to perform special handling to maintain air quality. One way to facilitate the prevention of air pollution is to make air pollution forecasting by utilizing past data. Through the Environmental Office, the Surabaya City Government has monitored air quality in Surabaya every 30 minutes for various air quality parameters including CO, NO, NO2, NOx, PM10, SO2 and meteorological data such as wind direction, wind direction, wind speed, wind speed, global radiation, humidity, and air temperature. These data are very useful to build a prediction model for the forecast of air pollution in the future. With the large amount and variance of data generated from monitoring air quality in Surabaya city, a qualified algorithm is needed to process it. One algorithm that can be used is Recurrent Neural Network - Long Short Term Memory (RNN-LSTM). RNN-LSTM is built for sequential data processing such as time-series data. In this study, several analyses are performed. There are trend analysis, correlation analysis of pollutant values to meteorological data, and predictions of carbon monoxide pollutants using the Recurrent Neural Network - LSTM in the city of Surabaya correlated with meteorological data. The results of this study indicate that the best prediction model using RNN-LSTM with RMSE calculation gets an error of 1,880 with the number of hidden layer 2 and epoch 50 scenarios. The predicted results built can be used as a reference in determining the policy of the city government to deal with air pollution going forward.

Time Series Prediction Method Based on Variant LSTM Recurrent Neural Network

Time series prediction problems are a difficult type of predictive modeling problem. In this paper, we propose a time series prediction method based on a variant long short-term memory (LSTM) recurrent neural network. In the proposed method, we firstly improve the memory module of the LSTM recurrent neural network by merging its forget gate and input gate into one update gate, and using Sigmoid layer to control information update. Using improved LSTM recurrent neural network, we develop a time series prediction model. In the proposed model, the parameter migration method is used model update to ensure the model has good predictive ability after predicting multi-step sequences. Experimental results show, compared with several typical time series prediction models, the proposed method have better performance for long-sequence data prediction.

Deep and reinforcement learning for automated task scheduling in large‐scale cloud computing systems

SummaryCloud computing is undeniably becoming the main computing and storage platform for today's major workloads. From Internet of things and Industry 4.0 workloads to big data analytics and decision‐making jobs, cloud systems daily receive a massive number of tasks that need to be simultaneously and efficiently mapped onto the cloud resources. Therefore, deriving an appropriate task scheduling mechanism that can both minimize tasks' execution delay and cloud resources utilization is of prime importance. Recently, the concept of cloud automation has emerged to reduce the manual intervention and improve the resource management in large‐scale cloud computing workloads. In this article, we capitalize on this concept and propose four deep and reinforcement learning‐based scheduling approaches to automate the process of scheduling large‐scale workloads onto cloud computing resources, while reducing both the resource consumption and task waiting time. These approaches are: reinforcement learning (RL), deep Q networks, recurrent neural network long short‐term memory (RNN‐LSTM), and deep reinforcement learning combined with LSTM (DRL‐LSTM). Experiments conducted using real‐world datasets from Google Cloud Platform revealed that DRL‐LSTM outperforms the other three approaches. The experiments also showed that DRL‐LSTM minimizes the CPU usage cost up to 67% compared with the shortest job first (SJF), and up to 35% compared with both the round robin (RR) and improved particle swarm optimization (PSO) approaches. Moreover, our DRL‐LSTM solution decreases the RAM memory usage cost up to 72% compared with the SJF, up to 65% compared with the RR, and up to 31.25% compared with the improved PSO.