Encoder-decoder Recurrent Neural Network Research Articles

A Regression Framework for Energy Consumption in Smart Cities with Encoder-Decoder Recurrent Neural Networks

Currently, a smart city should ideally be environmentally friendly and sustainable, and energy management is one method to monitor sustainable use. This research project investigates the potential for a “smart city” to improve energy management by enabling the adoption of various types of intelligent technology to improve the energy sustainability of a city’s infrastructure and operational efficiency. In addition, the South Korean smart city region of Songdo serves as the inspiration for this case study. In the first module of the proposed framework, we place a strong emphasis on the data capabilities necessary to generate energy statistics for each of the numerous structures. In the second phase of the procedure, we employ the collected data to conduct a data analysis of the energy behavior within the microcities, from which we derive characteristics. In the third module, we construct baseline regressors to assess the proposed model’s varying degrees of efficacy. Finally, we present a method for building an energy prediction model using a deep learning regression model to solve the problem of 48-hour-ahead energy consumption forecasting. The recommended model is preferable to other models in terms of R2, MAE, and RMSE, according to the study’s findings.

Forecasting of breathing events during nocturnal sleep using encoder-decoder recurrent neural network based on a sensors data of consumer smartwatches.

Respiratory disorders during nocturnal sleep are the states of abnormal and difficult breathing, including snoring, hypopnea and different apnea types. Some of them have a negligible effect on health, while others can lead to a serious consequences. Therefore, the development of low-cost, portable, user-friendly devices and corresponding algorithms for diagnosis and forecasting of such events is of particular importance. In the current paper, an encoder-decoder recurrent neural network was developed for respiratory pattern forecasting. The system is based on a physiological sensors (accelerometer and photoplethysmography) data gathered from the consumer smartwatches during nocturnal sleep. The influence of the length of time series in the encoder part (available history for forecasting), and the length of time series at the output of decoder (forecasting length) is studied. The average achieved f1 score and Cohen's Kappa agreement of the proposed model varies in the range from 0.35 to 0.5 and from 0.25 to 0.4, respectively, depending on forecasting length. The efficiency of the forecasting largely depends on the model complexity, presence or absence of respiration events in the encoder part, and forecasting length.Clinical Relevance- Results of the current paper may be used for the development of the respiration events screening tool based on a wearable devices sensors data.

Building the Waray-waray Neural Language Model using Recurrent Neural Network

In the Philippines, language modeling is challenging since most of its languages are low-resourced. Tagalog and Cebuano are the only languages present in machine translation platforms like Google Translate; Winaray, a language spoken in the Eastern Visayas region, is inexistent. Hence, this study developed a Winaray language model that could be used in any natural language processing-related tasks. The text corpus used in creating the model was scrapped from the web (religious and local news websites, and Wikipedia) containing Winaray sentences. The model was trained using an encoder-decoder recurrent neural network with four sequential layers and 100 hidden neurons. The text prediction accuracy of the model reached 76.17%. The model was manually evaluated based on its text-generated sentences using linguistic quality dimensions such as grammaticality, non-redundancy, focus, structure and coherence. Results of manual evaluation showed a promising result as the linguistic quality reached 3.66 (acceptable); however, training data must be improved in terms of size with the addition of texts in various text genres.

Causal speech enhancement using dynamical-weighted loss and attention encoder-decoder recurrent neural network.

Speech enhancement (SE) reduces background noise signals in target speech and is applied at the front end in various real-world applications, including robust ASRs and real-time processing in mobile phone communications. SE systems are commonly integrated into mobile phones to increase quality and intelligibility. As a result, a low-latency system is required to operate in real-world applications. On the other hand, these systems need efficient optimization. This research focuses on the single-microphone SE operating in real-time systems with better optimization. We propose a causal data-driven model that uses attention encoder-decoder long short-term memory (LSTM) to estimate the time-frequency mask from a noisy speech in order to make a clean speech for real-time applications that need low-latency causal processing. The encoder-decoder LSTM and a causal attention mechanism are used in the proposed model. Furthermore, a dynamical-weighted (DW) loss function is proposed to improve model learning by varying the weight loss values. Experiments demonstrated that the proposed model consistently improves voice quality, intelligibility, and noise suppression. In the causal processing mode, the LSTM-based estimated suppression time-frequency mask outperforms the baseline model for unseen noise types. The proposed SE improved the STOI by 2.64% (baseline LSTM-IRM), 6.6% (LSTM-KF), 4.18% (DeepXi-KF), and 3.58% (DeepResGRU-KF). In addition, we examine word error rates (WERs) using Google's Automatic Speech Recognition (ASR). The ASR results show that error rates decreased from 46.33% (noisy signals) to 13.11% (proposed) 15.73% (LSTM), and 14.97% (LSTM-KF).

Time-Series Forecasting of Chlorophyll-a in Coastal Areas Using LSTM, GRU and Attention-Based RNN Models

The chlorophyll-a (Chl-a) concentration is commonly considered as the main indicator of phytoplankton biomass in coastal waters. Forecasting and understanding the status of Chl-a is beneficial to coastal ecosystem management and is an important emergency management measure for algae blooms. To obtain accurate predictions, the long short-term memory neural network (LSTM) and gated recurrent unit neural network (GRU) were implemented for Chl-a forecasting, and based on the LSTM and GRU units, two simplified attention-based encoder-decoder recurrent neural network (AEDRNN) models were also developed for time series predictions. The performance of the proposed models was compared with that of the auto-regressive integrated moving average (ARIMA), multilayer perceptron (MLP) and Elman recurrent neural network (ERNN) models by experimentally generating multi-step-ahead predictions using a dataset in the Zhejiang coastal areas of China. The results demonstrated that the LSTM, GRU and AEDRNN models significantly outperformed the ARIMA, MLP and ERNN models according to multiple statistical indicators. Moreover, the AEDRNN models were superior to the LSTM and GRU models, especially for middle-term predictions. In addition, the AEDRNN model with LSTM units was more robust than the AEDRNN model with GRU units in terms of accuracy and stability; therefore, it was considered to be the best model for Chl-a forecasting.

Parameter estimation for WMTI-Watson model of white matter using encoder-decoder recurrent neural network.

Biophysical modeling of the diffusion MRI (dMRI) signal provides estimates of specific microstructural tissue properties. Although non-linear least squares (NLLS) is the most widespread fitting method, it suffers from local minima and high computational cost. Deep learning approaches are steadily replacing NLLS, but come with the limitation that the model needs to be retrained for each acquisition protocol and noise level. In this study, a novel fitting approach was proposed based on the encoder-decoder recurrent neural network (RNN) to accelerate model estimation with good generalization to various datasets. The white matter tract integrity (WMTI)-Watson model as an implementation of the Standard Model of diffusion in white matter derives its parameters indirectly from the diffusion and kurtosis tensors (DKI). The RNN-based solver, which estimates the WMTI-Watson model from DKI, is therefore more readily translatable to various data, irrespective of acquisition protocols as long as the DKI was pre-computed from the signal. An embedding approach was also used to render the model insensitive to potential differences in distributions between training data and experimental data. The analytical solution, NLLS, RNN-, and a multilayer perceptron (MLP)-based methods were evaluated on synthetic and in vivo datasets of rat and human brain. The proposed RNN solver showed highly reduced computation time over the analytical solution and NLLS, with similar accuracy but improved robustness, and superior generalizability over MLP. The RNN estimator can be easily applied to various datasets without retraining, which shows great potential for a widespread use.

Artificial intelligence enabled conversational agent for mental healthcare

Conversational agents are a software program that can converse with users in the manner of a real-world conversation. Artificial intelligence (AI) is not complete without conversation modeling. The most difficult artificial intelligence endeavor since its start has been developing an effective chatbot application. Despite chatbots may do a variety of tasks, their main duty is to accurately understand human speech and respond appropriately. Previously, manual patterns and instructions or simple statistical methods were used to create chatbots architectures. Due to its improved capacity for training, end-to-end AI has replaced these models since 2015. The most popular technique for conversation simulation at the moment is the encoder-decoder recurrent neural network (RNN). The realm of language comprehension served as inspiration for this design. Until recently, a number of additions and changes dramatically enhanced chatbot conversational abilities. In this paper, we outline our research results into creating an interactive digital chatbot that may provide patients with psychological assistance. To build and train the chatbot, we used resources such Rasa Natural Language Processing (NLU) technology, which employs natural language processing (NLP) methods. The results of the investigation showed that selecting proper responses while conversing with patients had a more than 70% predictive performance.

Attention based Recurrent Neural Network for Nepali Text Summarization

Automatic text summarization has been a challenging topic in natural language processing (NLP) as it demands preserving important information while summarizing the large text into a summary. Extractive and abstractive text summarization are widely investigated approaches for text summarization. In extractive summarization, the important sentence from the large text is extracted and combined to create a summary whereas abstractive summarization creates a summary that is more focused on meaning, rather than content. Therefore, abstractive summarization gained more attention from researchers in the recent past. However, text summarization is still an untouched topic in the Nepali language. To this end, we proposed an abstractive text summarization for Nepali text. Here, we, first, create a Nepali text dataset by scraping Nepali news from the online news portals. Second, we design a deep learning-based text summarization model based on an encoder-decoder recurrent neural network with attention. More precisely, Long Short-Term Memory (LSTM) cells are used in the encoder and decoder layer. Third, we build nine different models by selecting various hyper-parameters such as the number of hidden layers and the number of nodes. Finally, we report the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score for each model to evaluate their performance. Among nine different models created by adjusting different numbers of layers and hidden states, the model with a single-layer encoder and 256 hidden states outperformed all other models with F-Score values of 15.74, 3.29, and 15.21 for ROUGE-1 ROUGE-2 and ROUGE-L, respectively.

Software Reliability Prediction through Encoder-Decoder Recurrent Neural Networks

With the growing demand for high reliability and safety software, software reliability prediction has attracted more and more attention to identifying potential faults in software. Software reliability growth models (SRGMs) are the most commonly used prediction models in practical software reliability engineering. However, their unrealistic assumptions and environment-dependent applicability restrict their development. Recurrent neural networks (RNNs), such as the long short-term memory (LSTM), provide an end-to-end learning method, have shown a remarkable ability in time-series forecasting and can be used to solve the above problem for software reliability prediction. In this paper, we present an attention-based encoder-decoder RNN called EDRNN to predict the number of failures in the software. More specifically, the encoder-decoder RNN estimates the cumulative faults with the fault detection time as input. The attention mechanism improves the prediction accuracy in the encoder-decoder architecture. Experimental results demonstrate that our proposed model outperforms other traditional SRGMs and neural network-based models in terms of accuracy.

Joint decision-making of parallel machine scheduling restricted in job-machine release time and preventive maintenance with remaining useful life constraints

The machine remaining useful life (RUL), the job-machine release time and the correlation between the maintenance duration and the machine enlistment age are, in this paper, collectively emphasized at the parallel machine scheduling problem. Based on this, a corresponding mixed integer programming model is constructed to minimize the makespan and the processing loss beyond the machine RUL threshold, where a discrete teaching and learning based optimization algorithm is applied to solve this NP-hard problem, and a fault mode-assisted gated recurrent unit (FGRU) life prediction method is used to guide the predictive maintenance initiation time of all machines. In addition, this paper demonstrates that the FGRU method is more accurate than three common methods (Encoder-Decoder Recurrent Neural Network, Bidirectional Long Short-Term Memory and GRU) through two actual bearing degradation cases, and shows through three benchmark cases that the joint decision-making can effectively reduce the time cost of manufacturing enterprises.

Towards Automatic Job Description Generation with Capability-Aware Neural Networks

A job description shows the responsibilities of the job position and the skill requirements for the job. An effective job description will help employers to identify the right talents for the job, and give a clear understanding to candidates of what their duties and qualifications for a particular position would be. In this paper, we investigate how to automate the process to generate job descriptions with less human intervention. We propose an end-to-end capability-aware neural job description generation framework, namely Cajon, to facilitate the writing of job description. Specifically, we first propose a novel capability-aware neural topic model to distill the various capability information from the larger-scale recruitment data. Also, an encoder-decoder recurrent neural network is designed for enabling the job description generation. In particular, the capability-aware attention and copy mechanisms are proposed to guide the generation process to ensure the generated job descriptions can comprehensively cover relevant and representative capability requirements for the job. Moreover, we propose a capability-aware policy gradient training algorithm to further enhance the rationality of the generated job description. Finally, extensive experiments on real-world recruitment data clearly show our Cajon framework can help to generate more effective job descriptions in an interpretable way

Planning for Autonomous Driving via Interaction-Aware Probabilistic Action Policies

Devising planning algorithms for autonomous driving is non-trivial due to the presence of complex and uncertain interaction dynamics between road users. In this paper, we introduce a planning framework encompassing multiple action policies that are learned jointly from episodes of human-human interactions in naturalistic driving. The policy model is composed of encoder-decoder recurrent neural networks for modeling the sequential nature of interactions and mixture density networks for characterizing the probability distributions over driver actions. The model is used to simultaneously generate a finite set of context-dependent candidate plans for an autonomous agent and to anticipate the probable future plans of human drivers. This is followed by an evaluation stage to select the agent plan with the highest expected utility for execution. Our approach leverages rapid sampling of action distributions in parallel on a graphic processing unit, offering fast computation even when modeling the interactions among multiple vehicles and over several time steps. We present ablation experiments and comparison with two existing baseline methods to highlight several design choices that we found to be essential to our model’s success.We test the proposed planning approach in a simulated highway driving environment, showing that by using the model, the agent can plan actions that mimic the interactive behavior of humans.

Deep Learning Methods for Vessel Trajectory Prediction Based on Recurrent Neural Networks

Data-driven methods open up unprecedented possibilities for maritime surveillance using Automatic Identification System (AIS) data. In this work, we explore deep learning strategies using historical AIS observations to address the problem of predicting future vessel trajectories with a prediction horizon of several hours. We propose novel sequence-to-sequence vessel trajectory prediction models based on encoder-decoder recurrent neural networks (RNNs) that are trained on historical trajectory data to predict future trajectory samples given previous observations. The proposed architecture combines Long Short-Term Memory (LSTM) RNNs for sequence modeling to encode the observed data and generate future predictions with different intermediate aggregation layers to capture space-time dependencies in sequential data. Experimental results on vessel trajectories from an AIS dataset made freely available by the Danish Maritime Authority show the effectiveness of deep-learning methods for trajectory prediction based on sequence-to-sequence neural networks, which achieve better performance than baseline approaches based on linear regression or on the Multi-Layer Perceptron (MLP) architecture. The comparative evaluation of results shows: i) the superiority of attention pooling over static pooling for the specific application, and ii) the remarkable performance improvement that can be obtained with labeled trajectories, i.e., when predictions are conditioned on a low-level context representation encoded from the sequence of past observations, as well as on additional inputs (e.g., port of departure or arrival) about the vessel's high-level intention, which may be available from AIS.

Model predictive control strategy using encoder-decoder recurrent neural networks for smart control of thermal environment

The development of deep learning (DL) technology provides an opportunity for accurate prediction and effective control of complex building systems. However, despite the high prediction performance of DL models, few studies integrate DL with the model predictive control (MPC) algorithm for improving building management. In this study, a DL-based MPC framework using encoder-decoder recurrent neural network is developed for real-time control of building thermal environment to exploit the advantage of both DL and MPC algorithms. In addition, to simulate the dynamic interaction between indoor airflow and HVAC systems, a co-simulation platform by integrating HVAC simulator and CFD model is developed to evaluate the proposed building control strategy. Two case studies including a confined space with mixing ventilation and an office room subjected to solar radiation are used for validation of the proposed method. The performance of the DL-based MPC algorithm for building thermal environment control is compared with the traditional proportion-integration-differentiation (PID) controller and the adaptive PID controller. Approximately 4% and 7% energy savings are achieved on average through DL-based MPC compared with adaptive and conventional PID control, respectively. The proposed DL-based MPC framework shows a promising application prospect for building automationby taking the advantage of both deep learning and MPC algorithms.

Clustering and Recognition of Spatiotemporal Features Through Interpretable Embedding of Sequence to Sequence Recurrent Neural Networks.

Encoder-decoder recurrent neural network models (RNN Seq2Seq) have achieved success in ubiquitous areas of computation and applications. They were shown to be effective in modeling data with both temporal and spatial dependencies for translation or prediction tasks. In this study, we propose an embedding approach to visualize and interpret the representation of data by these models. Furthermore, we show that the embedding is an effective method for unsupervised learning and can be utilized to estimate the optimality of model training. In particular, we demonstrate that embedding space projections of the decoder states of RNN Seq2Seq model trained on sequences prediction are organized in clusters capturing similarities and differences in the dynamics of these sequences. Such performance corresponds to an unsupervised clustering of any spatio-temporal features and can be employed for time-dependent problems such as temporal segmentation, clustering of dynamic activity, self-supervised classification, action recognition, failure prediction, etc. We test and demonstrate the application of the embedding methodology to time-sequences of 3D human body poses. We show that the methodology provides a high-quality unsupervised categorization of movements. The source code with examples is available in a Github repository1.

Cross-lingual Unified Medical Language System entity linking in online health communities.

ObjectiveIn Hebrew online health communities, participants commonly write medical terms that appear as transliterated forms of a source term in English. Such transliterations introduce high variability in text and challenge text-analytics methods. To reduce their variability, medical terms must be normalized, such as linking them to Unified Medical Language System (UMLS) concepts. We present a method to identify both transliterated and translated Hebrew medical terms and link them with UMLS entities.Materials and MethodsWe investigate the effect of linking terms in Camoni, a popular Israeli online health community in Hebrew. Our method, MDTEL (Medical Deep Transliteration Entity Linking), includes (1) an attention-based recurrent neural network encoder-decoder to transliterate words and mapping UMLS from English to Hebrew, (2) an unsupervised method for creating a transliteration dataset in any language without manually labeled data, and (3) an efficient way to identify and link medical entities in the Hebrew corpus to UMLS concepts, by producing a high-recall list of candidate medical terms in the corpus, and then filtering the candidates to relevant medical terms.ResultsWe carry out experiments on 3 disease-specific communities: diabetes, multiple sclerosis, and depression. MDTEL tagging and normalizing on Camoni posts achieved 99% accuracy, 92% recall, and 87% precision. When tagging and normalizing terms in queries from the Camoni search logs, UMLS-normalized queries improved search results in 46% of the cases.ConclusionsCross-lingual UMLS entity linking from Hebrew is possible and improves search performance across communities. Annotated datasets, annotation guidelines, and code are made available online (https://github.com/yonatanbitton/mdtel).

Smart alarm based on sleep stages prediction.

A sleep inertia after waking up strongly affects the overall mental recovery after sleep. The sleep inertia depends not as much on overall sleep quality but also on the sleep stage in the waking up moment. The fix-time alarming system results in waking up at random sleep stage, which results in frequent sleep inertia. The widely used flow-time alarming systems based on motion detection (actigraphy) reduce but do not eliminate sleep inertia. Such systems do not wake up users in Deep sleep stage, but may instead wake them up in Wake, Light, or REM (Rapid Eyes Movement) stages. Moreover, frequent waking up in the REM stage results in serious psychological issues. We present a smartwatch alarm system that predicts sleep stages and thus produces an alarm call at an easy waking up moment with minimal sleep inertia effect. The sleep stages are predicted using an Encoder-Decoder Recurrent Neural Network model. The rationale of the prediction is that each sleep stages cycling pattern is a continuous quasi-periodic process. Experimental results from over 138 nocturnal sleep periods from 92 respondents show that our system provides 66-70% accuracy for Deep, Light, Wake, REM sleep stages and 71-77% accuracy for 2-classes (Deep/REM vs. Light/Wake stages) prediction classifications. The proposed alarm system wakes up the user at the moment when Easy Wake (Wake/Light) stage is the most probable.

English to Bangla Machine Translation Using Recurrent Neural Network

The applications of recurrent neural networks in machine translation are increasing in natural language processing. Besides other languages, Bangla language contains a large amount of vocabulary. Improvement of English to Bangla machine translation would be a significant contribution to Bangla Language processing. This paper describes an architecture of English to Bangla machine translation system. The system has been implemented with the encoder-decoder recurrent neural network. The model uses a knowledge-based context vector for the mapping of English and Bangla words. Performances of the model based on activation functions are measured here. The best performance is achieved for the linear activation function in encoder layer and the tanh activation function in decoder layer. From the execution of GRU and LSTM layer, GRU performed better than LSTM. The attention layers are enacted with softmax and sigmoid activation function. The approach of the model outperforms the previous state-of-the-art systems in terms of cross-entropy loss metrics. The reader can easily find out the structure of the machine translation of English to Bangla and the efficient activation functions from the paper.

Online sales prediction via trend alignment-based multitask recurrent neural networks

While business trends are constantly evolving, the timely prediction of sales volume offers precious information for companies to achieve a healthy balance between supply and demand. In practice, sales prediction is formulated as a time series prediction problem which aims to predict the future sales volume for different products with the observation of various influential factors (e.g. brand, season, discount, etc.) and corresponding historical sales records. To perform accurate sales prediction under the offline setting, we gain insights from the encoder–decoder recurrent neural network (RNN) structure and have proposed a novel framework named TADA (Chen et al., in: ICDM, 2018) to carry out trend alignment with dual-attention, multitask RNNs for sales prediction. However, the sales data accumulates at a fast rate and is updated on a regular basis, rendering it difficult for the trained model to maintain the prediction accuracy with new data. In this light, we further extend the model into TADA$$^+$$, which is enhanced by an online learning module based on our innovative similarity-based reservoir. To construct the data reservoir for model retraining, different from most existing random sampling-based reservoir, our similarity-based reservoir selects data samples that are “hard” for the model to mine apparent dynamic patterns. The experimental results on two real-world datasets comprehensively show the superiority of TADA and TADA$$^+$$ in both online and offline sales prediction tasks against other state-of-the-art competitors.

Time-series forecasting of coal-fired power plant reheater metal temperatures using encoder-decoder recurrent neural networks

With the increase in renewable energy penetration of electrical grids, coal power stations will be required to operate flexibly rather than functioning as baseload units. During flexible operation of conventional coal-fired stations, thermal stresses are induced in reheaters which could lead to tube ruptures and unplanned plant downtime. The current study sets out to develop a data-driven sequence-to-sequence recurrent neural network model capable of predicting future reheater metal temperatures using plant operational data. The best-performing network and training algorithm configuration was found by implementing a coarse grid search of hyperparameter combinations. The proposed model architecture uses stacked encoder and decoder sections with GRU cells and 512 hidden units per layer. An input sequence length of 8 min was used to predict an output sequence of 5 min, with sequence intervals of 1 min. The results indicate that the encoder-decoder GRU network has adequate accuracy. The mean absolute percentage error for the test dataset was below 1% which corresponds to a root-mean-squared error in predicted metal temperatures of 6.2 °C.