Short-term Memory-based Recurrent Neural Networks Research Articles

Operational wind and turbulence nowcasting capability for advanced air mobility

The present study introduces “WindAware”, a wind and turbulence prediction system that provides nowcasts of wind and turbulence parameters every 5 min up to 6 h over a predetermined airway over Chicago, Illinois, USA, based on 100 m high-resolution simulations (HRSs). This system is a long short-term memory-based recurrent neural network (LSTM-RNN) that uses existing ground-based wind data to provide nowcasts (forecasts up to 6 h every 5 min) of wind speed, wind direction, wind gust, and eddy dissipation rate to support the Uncrewed Aircraft Systems (UASs) safe integration into the National Airspace System (NAS). These HRSs are validated using both ground-based measurements over airports and upper-air radiosonde observations and their skill is illustrated during lake-breeze events. A reasonable agreement is found between measured and simulated winds especially when the boundary layer is convective, but the timing and inland penetration of lake-breeze events are overall slightly misrepresented. The WindAware model is compared with the classic multilayer perceptron (MLP) and the eXtreme Gradient Boosting (XGBoost) models. It is demonstrated by comparison to high-resolution simulations that WindAware provides more accurate predictions than the MLP over the 6 h lead times and has almost similar performance as the XGBoost model although the XGBoost’s training is the fastest using its parallelized implementation. WindAware also has higher prediction errors when validated against lake-breeze events data due to their under-representation in the training dataset.

Deep spatiotemporal fusion network for vision-based robotic inspection of structures

The convolutional neural networks commonly deployed for semantic understanding of visual inspection data can, in general, learn robust spatial features. However, they lack the ability to capture temporal dependencies that characterize the video data collected by various robotic inspection systems. As a result, they are found lacking in dealing with various challenges arising from cross-view illumination variation, perspective difference, scale change, background clutter, and occlusion. Their performance is further deteriorated by motion blur and other distortions induced by rapid camera movement. This study aims to address this challenge by extending the task of visual scene understanding from the still image domain to the video domain by incorporating cross-frame information fusion. A deep end-to-end network is developed by integrating an encoder–decoder-based convolutional neural network with a long short-term memory-based recurrent neural network for pixel-level semantic labeling of sequential visual inspection data. The proposed multishot architecture can jointly learn discriminative fusion features leading to a rich understanding of the complex spatiotemporal dynamics. The proposed approach is validated with two case studies involving automatic structural element segmentation in robotic building and bridge inspection videos. Two different multishot fusion techniques are suggested leveraging sequence-to-one and sequence-to-sequence architectures. Additionally, two different fusion schemes based on the sum-of-scores and Bayesian updating rules are examined to aggregate multiple label maps produced at each time step by an overlapping sliding window-based inference scheme. A comprehensive performance evaluation indicated that multishot fusion could enhance the intersection over union (IoU) score by 4.6% and 13.3% for building and bridge component segmentation tasks, respectively, compared to a baseline single-shot approach.

A Molecular Generative Model of COVID-19 Main Protease Inhibitors Using Long Short-Term Memory-Based Recurrent Neural Network.

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has caused a serious threat to public health and prompted researchers to find anti-coronavirus 2019 (COVID-19) compounds. In this study, the long short-term memory-based recurrent neural network was used to generate new inhibitors for the coronavirus. First, the model was trained to generate drug compounds in the form of valid simplified molecular-input line-entry system strings. Then, the structures of COVID-19 main protease inhibitors were applied to fine-tune the model. After fine-tuning, the network could generate new molecular structures as novel SARS-CoV-2 main protease inhibitors. Molecular docking exhibited that some generated compounds have the proper affinity to the active site of the protease. Molecular Dynamics simulations explored binding free energies of the compounds over simulation trajectories. In addition, in silico absorption, distribution, metabolism, and excretion studies showed that some novel compounds could be formulated as orally active agents. Based on molecular docking and molecular dynamics simulation studies, compound AADH possessed significant binding affinity and presumably inhibition against the SARS-CoV-2 main protease enzyme. Therefore, the proposed deep learning-based model was capable of generating promising anti-COVID-19 drugs.

Fault detection with confidence level evaluation for perception module of autonomous vehicles based on long short term memory and Gaussian Mixture Model

The reliability of perception is crucial for developing higher levels of autonomy to enhance the safety and performance of fully autonomous vehicles. As a result, fault detection in the perception module is an essential function for autonomous vehicles. While many approaches rely on hardware redundancy, necessitating additional sensors, achieving analytical redundancy in perception is a pivotal factor in reducing the cost and complexity of the autonomous system. This paper utilizes a motion predictor for surrounding vehicles and a Gaussian Mixture Model (GMM) to evaluate the confidence level of the perception module without requiring additional sensor installations. The motion predictor is designed based on a long short-term memory-based recurrent neural network. The error between the motion prediction and detection results is leveraged to estimate the confidence level of the perception module. In essence, the motion prediction results are treated as redundant sensor measurements. The error distribution is modeled using a GMM, and the cumulative probability density of the GMM is employed to assess the confidence level. The proposed algorithm's effectiveness is evaluated through a driving data-based simulation study with fault injection. This study shows an enhanced sensitivity to injected faults and a reduced time for fault detection. Furthermore, the proposed algorithm provides a quantitative estimate of the performance degradation level of the perception module. This estimation can serve as an indicator of uncertainty that the motion planner should account for.

Named Entity Recognition for English Language Using Deep Learning Based Bi Directional LSTM-RNN

The NER has been important in different applications like data Retrieval and Extraction, Text Summarization, Machine Translation, Question Answering (Q-A), etc. While several investigations have been carried out for NER in English, a high-accuracy tool still must be designed per the Literature Survey. This paper suggests an English Named Entities Recognition methodology using NLP algorithms called Bi-Directional Long short-term memory-based recurrent neural network (LSTM-RNN). Most English Language NER systems use detailed features and handcrafted algorithms with gazetteers. The proposed model is language-independent and has no domain-specific features or handcrafted algorithms. Also, it depends on semantic knowledge from word vectors realized by an unsupervised learning algorithm on an unannotated corpus. It achieved state-of-the-art performance in English without the use of any morphological research or without using gazetteers of any sort. A little database group of 200 sentences includes 3080 words. The features selection and generations are presented to catch the Name Entity. The proposed work is desired to forecast the Name Entity of the focus words in a sentence with high accuracy with the benefit of practical knowledge acquisition techniques.

Optimized hybrid RNN model for human activity recognition in untrimmed video

Human activity recognition is a field of video processing that requires restricted temporal analysis of video sequences for estimating the existence of different human actions. Designing an efficient human activity model requires credible implementations of keyframe extraction, preprocessing, feature extraction and selection, classification, and pattern recognition methods. In the real-time video, sequences are untrimmed and do not have any activity endpoints for effective recognition. Thus, we propose a hybrid gated recurrent unit and long short-term memory-based recurrent neural network model for high-efficiency human action recognition in untrimmed video datasets. The proposed model is tested on the TRECVID dataset, along with other online datasets, and is observed to have an accuracy of over 91% for untrimmed video-based activity recognition. This accuracy is compared with various state-of-the-art models and is found to be higher when evaluated on multiple datasets.

Three-dimensional deep learning-based reduced order model for unsteady flow dynamics with variable Reynolds number

In this article, we present a deep learning-based reduced order model (DL-ROM) for predicting the fluid forces and unsteady vortex shedding patterns. We consider the flow past a sphere to examine the accuracy of our DL-ROM predictions. The proposed DL-ROM methodology relies on a three-dimensional convolutional recurrent autoencoder network (3D CRAN) to extract the low-dimensional flow features from the full-order snapshots in an unsupervised manner. The low-dimensional features are evolved in time using a long short-term memory-based recurrent neural network and reconstructed back to the full-order as flow voxels. These flow voxels are introduced as static and uniform query probes in the point cloud domain to reduce the unstructured mesh complexity while providing convenience in the 3D CRAN training. We introduce a novel procedure to recover the interface description and the instantaneous force quantities from these 3D flow voxels. To evaluate the 3D flow reconstruction and inference, the 3D CRAN methodology is first applied to an external flow past a static sphere at the single Reynolds number of Re = 300. We provide an assessment of the computing requirements in terms of the memory usage, training, and testing cost of the 3D CRAN framework. Subsequently, variable Re-based flow information is infused in one 3D CRAN to learn a symmetry-breaking flow regime (280 ≤ Re ≤ 460) for the flow past a sphere. Effects of transfer learning are analyzed for training this complex 3D flow regime on a relatively smaller time series dataset. The 3D CRAN framework learns the flow regime nearly 20 times faster than the parallel full-order model and predicts this flow regime in time with a reasonable accuracy. Based on the predicted flow fields, the network demonstrates an R2 accuracy of 98.58% for the drag and 76.43% for the lift over the sphere in this flow regime. The proposed framework aligns with the development of a digital twin for 3D unsteady flow field and instantaneous force predictions with variable Re-based effects.

Upper Body Posture Recognition Using Inertial Sensors and Recurrent Neural Networks

Inadequate sitting posture can cause imbalanced loading on the spine and result in abnormal spinal pressure, which serves as the main risk factor contributing to irreversible and chronic spinal deformity. Therefore, sitting posture recognition is important for understanding people’s sitting behaviors and for correcting inadequate postures. Recently, wearable devices embedded with microelectromechanical systems (MEMs) sensors, such as inertial measurement units (IMUs), have received increased attention in human activity recognition. In this study, a wearable device embedded with IMUs and a machine learning algorithm were developed to classify seven static sitting postures: upright, slump, lean, right and left bending, and right and left twisting. Four 9-axis IMUs were uniformly distributed between thoracic and lumbar regions (T1-L5) and aligned on a sagittal plane to acquire kinematic information about subjects’ backs during static-dynamic alternating motions. Time-domain features served as inputs to a signal-based classification model that was developed using long short-term memory-based recurrent neural network (LSTM-RNN) architecture, and the model’s classification performance was used to evaluate the relevance between sensor signals and sitting postures. Overall results from performance evaluation tests indicate that this IMU-based measurement and LSTM-RNN structural scheme was appropriate for sitting posture recognition.

A hybrid partitioned deep learning methodology for moving interface and fluid–structure interaction

In this work, we present a hybrid partitioned deep learning framework for the reduced-order modeling of moving interfaces and predicting fluid–structure interaction. Using the discretized Navier–Stokes in the arbitrary Lagrangian–Eulerian reference frame, we generate the full-order flow snapshots and point cloud displacements as target physical data for the learning and inference of coupled fluid–structure dynamics. The hybrid operation of this methodology comes by combining two separate data-driven models for fluid and solid subdomains via deep learning-based reduced-order models (DL-ROMs). The proposed multi-level framework comprises the partitioned data-driven drivers for unsteady flow and the moving point cloud displacements. At the fluid–structure interface, the force information is exchanged synchronously between the two partitioned subdomain solvers. The first component of our proposed framework relies on the proper orthogonal decomposition-based recurrent neural network (POD-RNN) as a DL-ROM procedure to infer the point cloud with a moving interface. This model utilizes the POD basis modes to reduce dimensionality and evolve them in time via long short-term memory-based recurrent neural networks (LSTM-RNNs). The second component employs the convolution-based recurrent autoencoder network (CRAN) as a self-supervised DL-ROM procedure to infer the nonlinear flow dynamics at static Eulerian probes. We introduce these probes as spatially structured query nodes in the moving point cloud to treat the Lagrangian-to-Eulerian conflict together with convenience in training the CRAN driver. To determine these Eulerian probes, we construct a novel snapshot-field transfer and load recovery algorithm. They are chosen in such a way that the two components (i.e., POD-RNN and CRAN) are constrained at the interface to recover the bulk force quantities. These DL-ROM-based data-driven drivers rely on the LSTM-RNNs to evolve the low-dimensional states. A popular prototypical fluid–structure interaction problem of flow past a freely oscillating cylinder is considered to assess the efficacy of the proposed methodology for a different set of reduced velocities that lead to vortex-induced vibrations. The proposed framework tracks the interface description with acceptable accuracy and predicts the nonlinear wake dynamics over the chosen test data range. The proposed framework aligns with the development of partitioned digital twin of engineering systems, especially those involving moving boundaries and fluid–structure interactions.

Predicting the Time Until a Vehicle Changes the Lane Using LSTM-Based Recurrent Neural Networks

To plan safe and comfortable trajectories for automated vehicles on highways, accurate predictions of traffic situations are needed. So far, a lot of research effort has been spent on detecting lane change maneuvers rather than on estimating the point in time a lane change actually happens. In practice, however, this temporal information might be even more useful. This paper deals with the development of a system that accurately predicts the time to the next lane change of surrounding vehicles on highways using long short-term memory-based recurrent neural networks. An extensive evaluation based on a large real-world data set shows that our approach is able to make reliable predictions, even in the most challenging situations, with a root mean squared error around 0.7 seconds. Already 3.5 seconds prior to lane changes the predictions become highly accurate, showing a median error of less than 0.25 seconds. In summary, this article forms a fundamental step towards downstreamed highly accurate position predictions.

Regression modeling for enterprise electricity consumption: A comparison of recurrent neural network and its variants

Effective electricity consumption forecasting is extremely significant for enterprises’ electricity planning which can provide data support for production decision, thus improving the level of enterprises' clean production. In recent years, recurrent neural network (RNN) and its variants have led to extensive research for time series forecasting. However, the performance and selection of these models in enterprise electricity forecasting have not been reported. With this study, we attempted to back some of these solutions with experimental results. This paper focused on a comparison for daily enterprise electricity consumption forecasting using different RNN models, i.e, standard RNN, long short-term memory-based RNN (LSTM), and gated recurrent unit-based RNN (GRU). To test their regression performance, three Chinese enterprises with different scales of electricity consumption are investigated. The comparison results show that the LSTM and the GRU models are slightly better than that of the RNN in terms of normalized root-mean-square error, mean absolute percentage error and threshold statistic. Moreover, the GRU model with the simplest structure is significantly different from the RNN, but not from LSTM in terms of Friedman testing. Hence the GRU model can be regarded as the first candidate for the enterprise electricity consumption forecasting in the future work.

An Energy-Efficient and Noise-Tolerant Recurrent Neural Network Using Stochastic Computing

Recurrent neural networks (RNNs) are widely used to solve a large class of recognition problems, including prediction, machine translation, and speech recognition. The hardware implementation of RNNs is, however, challenging due to the high area and energy consumption of these networks. Recently, stochastic computing (SC) has been considered for implementing neural networks and reducing the hardware consumption. In this paper, we propose an energy-efficient and noise-tolerant long short-term memory-based RNN using SC. In this SC-RNN, a hybrid structure is developed by utilizing SC designs and binary circuits to improve the hardware efficiency without significant loss of accuracy. The area and energy consumption of the proposed design are between 1.6%–2.3% and 6.5%–11.2%, respectively, of a 32-bit floating-point (FP) implementation. The SC-RNN requires significantly smaller area and lower energy consumption in most cases compared to an 8-bit fixed point implementation. The proposed design achieves a higher noise tolerance compared to binary implementations. The inference accuracy is from 10% to 13% higher than an FP design when the noise level is high in the computation process.

Long short-term memory-based recurrent neural networks for nonlinear target tracking

Nonlinear target tracking is essentially to estimate the target state from observations where the system model or observation model undergoes nonlinearities. Among the traditional methods, the particle filter can handle this problem well, though its statistical effectiveness and computational efficiency should be balanced in implementation. In this paper, deep neural network-based methods are proposed to resolve this problem because of their strong capabilities of fitting any mapping as long as the training data is sufficient. Specifically, the long short-term memory-based recurrent neural networks are proposed to take in the observations and output the true states in a sequential manner. Simulation results show that the proposed networks can obtain better estimation accuracy with shorter computational time compared to traditional methods. The newly proposed structure is shown to be able to provide an estimate of the uncertainty related to the target state online and automatically. Besides, nearly the same estimation accuracy can be provided by the proposed methods even when the exact initial prior of the target state is considered unknown.

Forecasting hotel reservations with long short-term memory-based recurrent neural networks

Hotel reservations tie up a much large portion of a hotels annual revenue. It is of great benefit for hotel managers to accurately forecast the numbers of reservations each day so that they can make better operational and tactical decisions. In this study, we review three types of forecasting methods commonly used in practice and briefly illustrate the concepts of neural networks. We then propose two long short-term memory (LSTM) models based on recurrent neural networks. Actual reservations of four hotels in the USA are used to estimate and test the proposed two models. To measure the relative performance, six machine learning (ML) models of decision tree, multilayer perceptron, lasso, linear regression, random forest, and ridge are also estimated and tested against the same datasets. The empirical results show that, on average, the forecasting accuracy of using LSTM models has been improved about 3.0% over that of using the best of the ML models.

Real-time prediction of online shoppers’ purchasing intention using multilayer perceptron and LSTM recurrent neural networks

In this paper, we propose a real-time online shopper behavior analysis system consisting of two modules which simultaneously predicts the visitor’s shopping intent and Web site abandonment likelihood. In the first module, we predict the purchasing intention of the visitor using aggregated pageview data kept track during the visit along with some session and user information. The extracted features are fed to random forest (RF), support vector machines (SVMs), and multilayer perceptron (MLP) classifiers as input. We use oversampling and feature selection preprocessing steps to improve the performance and scalability of the classifiers. The results show that MLP that is calculated using resilient backpropagation algorithm with weight backtracking produces significantly higher accuracy and F1 Score than RF and SVM. Another finding is that although clickstream data obtained from the navigation path followed during the online visit convey important information about the purchasing intention of the visitor, combining them with session information-based features that possess unique information about the purchasing interest improves the success rate of the system. In the second module, using only sequential clickstream data, we train a long short-term memory-based recurrent neural network that generates a sigmoid output showing the probability estimate of visitor’s intention to leave the site without finalizing the transaction in a prediction horizon. The modules are used together to determine the visitors which have purchasing intention but are likely to leave the site in the prediction horizon and take actions accordingly to improve the Web site abandonment and purchase conversion rates. Our findings support the feasibility of accurate and scalable purchasing intention prediction for virtual shopping environment using clickstream and session information data.