Recurrent Highway Networks Research Articles

A neural network encoder-decoder for time series prediction: Application on 137Cs particulate concentrations in nuclearized rivers

Monitoring the impact of human activities on the environment is a major challenge as many pollutants can be found in the different ecosystems. This is the case of the caesium-137 that has been present in the environment for many decades as a result of atmospheric tests, accidents such as Chernobyl and release from nuclear industries. With the recent advance in data-driven models, this study evaluate the relevance of a deep learning tool for reconstructing caesium-137 chronics particulate concentration in rivers. An encoder-decoder neural network, “Hierarchical Attention-Based Recurrent Highway Networks”(HRHN), is proposed notably for its ability to extract the most relevant temporal and spatial information from the databases. Three monitoring stations were studied, one on the Rhône River and two on the Loire River, all of them downstream nuclear industries in these catchments affected by the global fallout and the accident of Chernobyl. The objective is to predict the future concentration from a set of variables providing past information on water discharge, washout flux and industrial radioactive releases. Once optimised, the model generates first results in agreement with the real concentration curves by correctly following the main trends, with a NSE of 0.89, 0.53 and 0.35 respectively for the Rhone station and the two stations on the Loire. The main reason of inaccuracies is due to the quantity of data available. The originality of this model is its capacity to make predictions on different catchment areas. In fact the training was conducted on the Rhône station as the range of the concentration was higher (from 265.4 to 2700.0 Bq/kg) and the testing on the two Loire station. Another encoder-decoder model DA-RNN (Dual-Stage Attention-Based Recurrent Neural Network) was also evaluate in order to compare the performance of an alternative architecture, without convolution layer. The conclusion is that HRHN remains more powerful in the predictions on the 3 systems. With these first interesting results for HRHN, further investigations should be taken into account for other pollutants than caesium-137 to better understand the robustness of the model.

Enhancing local citation recommendation with recurrent highway networks and SciBERT-based embedding

When writing academic papers, referencing statements, claims and previous studies is always an important activity. However, it is considered challenging for scientists to find relevant and appropriate scientific articles which are closely related to their current works in order to reference them in their research. As a consequence of the rapid growth in scientific papers being published every year, researchers might easily get overwhelmed within a huge number of resources. One way to help them find the desired references more easily is to use context-aware citation recommendation. The citation recommender system can automatically provide a list of suitable papers as references based on specified inputs which reflect the researchers’ interests. Among the outstanding achievements of deep learning and natural language processing in recent years, the utilization of deep neural learning architectures have supported to address the problem of citation recommendation. As the result, the neural citation recommendation area has received much attention from the academic community, with the aim of enhancing the precision and correctness of the results of existing citation recommendation systems. Following this research direction, in our paper we present a novel context-aware citation recommendation model, called RHN-DualLCR (Recurrent Highway Networks – Dual Local Citation Recommendation), which integrates Recurrent Highway Networks (RHN), an improved model of the original Bidirectional Long Short-Term Memory (BiLSTM) architecture, and uses a SciBERT-based (Science text of Bidirectional Encoder Representations from Transformers) embedding layer to build up the efficiency of the state-of-the-art local citation recommendation model, which enriches context representation with global information. Our research demonstrates its originality and relevance because we used one of the latest achievements of deep models (RHN model) and natural language processing (SciBERT) applied to the citation recommendation problem. We have conducted experiments on the RHN-DualLCR model on 3 widely known datasets for the citation recommendation problem: ACL-200 (Association for Computational Linguistics), ACL-600 and RefSeer, and also used 2 common evaluation standards Mean Reciprocal Rank (MRR) and the Recall@K (R@K for short) to evaluate the performance of our model. Experimental results show that our proposed model is 3% to 16% better than the original models or state-of-the-art models.

Use of machine learning and deep learning to predict particulate 137Cs concentrations in a nuclearized river

Cesium-137, discharged by nuclear installations under normal operations and deposited in watersheds following atmospheric testing and accidents (i.e. Chernobyl, Fukushima …), has been studied for decades. Thus, modelling of 137Cs concentration in rivers have been developed based on geochemical approaches and equilibrium assumptions (solid/liquid ratio) as this radionuclide has moved into rivers and oceans due to soil erosion. Recently a new approach is possible to model these concentrations with the popularization of data-driven models based on data acquired in the environment by monitoring networks. In this study, the concentrations of particulate cesium-137 measured near the mouth of the Rhône River (France), a highly nuclearized river, are simulated using two data-driven models, a Hierarchical Attention-Based Recurrent Highway Networks (HRHN) and a Random Forest Regressor (RF). The data-driven predictions were done using only hydrological data (water discharge and suspended solid fluxes) and industrial input of 137Cs. Although the data-driven models provided a better prediction than a recent empirical model, the best prediction (R2 = 0.71) was obtained with HRHN, a model that considers the temporal aspect of the monitoring data. The most important predictors were the hydrological data at the monitoring station and of the tributary that generate the most sediment flux (Durance River). In fact, the concentration of 137Cs in the perimeter of this study was more related to hydrology than to nuclear release, as there were few events with high 137Cs concentrations (concomitant nuclear release and low water discharge). However, the HRHN approach, which is more complex to implement than RF, can predict the concentrations of such events correctly despite their low representation of these events. The results of this study demonstrate the usefulness of data-driven models to assist monitoring programs by filling in gaps or helping to understand observed concentrations.

Story Point Estimation Using Issue Reports With Deep Attention Neural Network

Background: Estimating the effort required for software engineering tasks is incredibly tricky, but it is critical for project planning. Issue reports are frequently used in the agile community to describe tasks, and story points are used to estimate task effort. Aim: This paper proposes a machine learning regression model for estimating the number of story points needed to solve a task. The system can be trained from raw input data to predict outcomes without the need for manual feature engineering. Method: Hierarchical attention networks are used in the proposed model. It has two levels of attention mechanisms implemented at word and sentence levels. The model gradually constructs a document vector by grouping significant words into sentence vectors and then merging significant sentence vectors to create document vectors. Then, the document vectors are fed into a shallow neural network to predict the story point. Results: The experiments show that the proposed approach outperforms the state-of-the-art technique Deep-S which uses Recurrent Highway Networks. The proposed model has improved Mean Absolute Error (MAE) by an average of 16.6% and has improved Median Absolute Error (MdAE) by an average of 53%. Conclusion: An empirical evaluation shows that the proposed approach outperforms the previous work.

Partially Recurrent Network With Highway Connections

It is difficult to train deep recurrent neural networks (RNNs) to learn the complex dependencies in sequential data, because of gradient problems and learning conflicts. To address these learning problems, this article proposes a partially recurrent network (PR-Net), which consists of the partially recurrent (PR) layers with highway connections. The proposed PR layer explicitly arranges the neurons into a memory module and an output module, which are implemented by highway networks. The memory module places memory on the information highway, to maintain long short-term memory (LSTM). The output module places external input on the information highway, to approximate the complex transition function from one step to the next. Because of the highway connections in the output module, the PR layer can pass gradient information across layers easily, to allow the training of the PR-Net with multiple stacked PR layers. Furthermore, the explicit separation of the memory module and output module can reduce the learning burden of RNNs that which neurons are responsible for memory and which are for output. With a comparable number of parameters, sublayers in the recurrent layer, and stacked recurrent layers, the proposed PR-Net outperformed the recurrent highway network and LSTM on three sequence learning benchmarks, significantly.

Efficient and effective training of sparse recurrent neural networks

Recurrent neural networks (RNNs) have achieved state-of-the-art performances on various applications. However, RNNs are prone to be memory-bandwidth limited in practical applications and need both long periods of training and inference time. The aforementioned problems are at odds with training and deploying RNNs on resource-limited devices where the memory and floating-point operations (FLOPs) budget are strictly constrained. To address this problem, conventional model compression techniques usually focus on reducing inference costs, operating on a costly pre-trained model. Recently, dynamic sparse training has been proposed to accelerate the training process by directly training sparse neural networks from scratch. However, previous sparse training techniques are mainly designed for convolutional neural networks and multi-layer perceptron. In this paper, we introduce a method to train intrinsically sparse RNN models with a fixed number of parameters and floating-point operations (FLOPs) during training. We demonstrate state-of-the-art sparse performance with long short-term memory and recurrent highway networks on widely used tasks, language modeling, and text classification. We simply use the results to advocate that, contrary to the general belief that training a sparse neural network from scratch leads to worse performance than dense networks, sparse training with adaptive connectivity can usually achieve better performance than dense models for RNNs.

The effect of coefficients on the continuous attractors in coupled Highway Neural Networks

Inspired by the recurrent highway networks, we study the coupled networks as a modular behavior of the networks, which can better improve the performance of the system. For a neural network, the non-zero continuous attractors play an important role to decode the information. The emphasis of this paper is to study the conditions for the generation of non-zero continuous attractors in the coupled new system theoretically. By exploring the relationship between network parameters, the conditions for generating non-zero continuous attractors in the new model are successfully determined. In order to verify the correctness and validity of the proposed conclusions, the results of this paper are further verified by simulation.

Multitime Resolution Hierarchical Attention-Based Recurrent Highway Networks for Taxi Demand Prediction

Taxi demand forecasting is an important consideration in building up smart cities. However, complex nonlinear spatiotemporal relationships in demand data make it difficult to construct an accurate prediction model. Considering that a single time resolution may not enable accurate learning of the time pattern of taxi demand, we expand the time series prediction model in our proposed multitime resolution hierarchical attention-based recurrent highway network (MTR-HRHN) model, using three time resolutions to model temporal closeness, period, and trend properties of demand data to capture a more comprehensive time pattern. We evaluate the MTR-HRHN on a taxi trip record dataset and the results show that the forecasting performance of the MTR-HRHN exceeds that of eight well-known methods in the short-term demand prediction in some high-demand regions.

Recurrent Highway Networks With Grouped Auxiliary Memory

Recurrent neural networks (RNNs) are challenging to train, let alone those with deep spatial structures. Architectures built upon highway connections such as Recurrent Highway Network (RHN) were developed to allow larger step-to-step transition depth, leading to more expressive models. However, problems that require capturing long-term dependencies still cannot be well addressed by these models. Moreover, the ability to keep long-term memories tends to diminish when the spatial depth increases, since deeper structure may accelerate gradient vanishing. In this paper, we address these issues by proposing a novel RNN architecture based on RHN, namely the Recurrent Highway Network with Grouped Auxiliary Memory (GAM-RHN). The proposed architecture interconnects the RHN with a set of auxiliary memory units specifically for storing long-term information via reading and writing operations, which is analogous to Memory Augmented Neural Networks (MANNs). Experimental results on artificial long time lag tasks show that GAM-RHNs can be trained efficiently while being deep in both time and space. We also evaluate the proposed architecture on a variety of tasks, including language modeling, sequential image classification, and financial market forecasting. The potential of our approach is demonstrated by achieving state-of-the-art results on these tasks.

A Deep Learning Model for Estimating Story Points

Although there has been substantial research in software analytics for effort estimation in traditional software projects, little work has been done for estimation in agile projects, especially estimating the effort required for completing user stories or issues. Story points are the most common unit of measure used for estimating the effort involved in completing a user story or resolving an issue. In this paper, we propose a prediction model for estimating story points based on a novel combination of two powerful deep learning architectures: long short-term memory and recurrent highway network. Our prediction system is end-to-end trainable from raw input data to prediction outcomes without any manual feature engineering. We offer a comprehensive dataset for story points-based estimation that contains 23,313 issues from 16 open source projects. An empirical evaluation demonstrates that our approach consistently outperforms three common baselines (Random Guessing, Mean, and Median methods) and six alternatives (e.g., using Doc2Vec and Random Forests) in Mean Absolute Error, Median Absolute Error, and the Standardized Accuracy.

Hierarchical recurrent highway networks

A contemporary approach for acquiring the computational gains of depth in recurrent neural networks (RNNs) is to hierarchically stack multiple recurrent layers. However, such performance gains come with the cost of challenging optimization of hierarchal RNNs (HRNNs) which are deep both hierarchically and temporally. Though the researchers have exclusively highlighted the significance of using highways (or direct shortcuts) for learning deep hierarchical representations and deep temporal dependencies. However, no significant efforts are made to unify these finding into a single framework for learning deep HRNNs. We propose hierarchical recurrent highway network (HRHN) that contains highway within the hierarchical and temporal structure of the network for unimpeded information propagation across both dimensions, thus alleviating gradient vanishing problem. The proposed HRHN contain significantly reduced data-dependent parameters as compared to related methods. The experiments on language modeling (LM) tasks have demonstrated that the proposed architecture leads to design effective models. On character-level LM using Hutter prize dataset, the model achieved entropy of 2.4 bits per character. The attained perplexity on world-level LM using Penn TreeBank is 68.1. The HRHN outperformed other baseline and related models that we tested.

Residual Recurrent Highway Networks for Learning Deep Sequence Prediction Models

A contemporary approach for acquiring the computational gains of depth in recurrent neural networks (RNNs) is to hierarchically stack multiple recurrent layers. However, such performance gains come with the cost of challenging optimization of hierarchal RNNs (HRNNs) which are deep both hierarchically and temporally. The researchers have exclusively highlighted the significance of using shortcuts for learning deep hierarchical representations and deep temporal dependencies. However, no significant efforts are made to unify these finding into a single framework for learning deep HRNNs. We propose residual recurrent highway network (R2HN) that contains highways within temporal structure of the network for unimpeded information propagation, thus alleviating gradient vanishing problem. The hierarchical structure learning is posed as residual learning framework to prevent performance degradation problem. The proposed R2HN contain significantly reduced data-dependent parameters as compared to related methods. The experiments on language modeling (LM) tasks have demonstrated that the proposed architecture leads to design effective models. On LM experiments with Penn TreeBank, the model achieved 60.3 perplexity and outperformed baseline and related models that we tested.

Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling

Recurrent neural networks have shown remarkable success in modeling sequences. However low resource situations still adversely affect the generalizability of these models. We introduce a new family of models, called Lattice Recurrent Units (LRU), to address the challenge of learning deep multi-layer recurrent models with limited resources. LRU models achieve this goal by creating distinct (but coupled) flow of information inside the units: a first flow along time dimension and a second flow along depth dimension. It also offers a symmetry in how information can flow horizontally and vertically. We analyze the effects of decoupling three different components of our LRU model: Reset Gate, Update Gate and Projected State. We evaluate this family of new LRU models on computational convergence rates and statistical efficiency.Our experiments are performed on four publicly-available datasets, comparing with Grid-LSTM and Recurrent Highway networks. Our results show that LRU has better empirical computational convergence rates and statistical efficiency values, along with learning more accurate language models.