Recurrent Layers Research Articles

Partially Recurrent Network With Highway Connections

It is difficult to train deep recurrent neural networks (RNNs) to learn the complex dependencies in sequential data, because of gradient problems and learning conflicts. To address these learning problems, this article proposes a partially recurrent network (PR-Net), which consists of the partially recurrent (PR) layers with highway connections. The proposed PR layer explicitly arranges the neurons into a memory module and an output module, which are implemented by highway networks. The memory module places memory on the information highway, to maintain long short-term memory (LSTM). The output module places external input on the information highway, to approximate the complex transition function from one step to the next. Because of the highway connections in the output module, the PR layer can pass gradient information across layers easily, to allow the training of the PR-Net with multiple stacked PR layers. Furthermore, the explicit separation of the memory module and output module can reduce the learning burden of RNNs that which neurons are responsible for memory and which are for output. With a comparable number of parameters, sublayers in the recurrent layer, and stacked recurrent layers, the proposed PR-Net outperformed the recurrent highway network and LSTM on three sequence learning benchmarks, significantly.

Deep State Inference: Toward Behavioral Model Inference of Black-Box Software Systems

Many software engineering tasks, such as testing, debugging, and anomaly detection can benefit from the ability to infer a behavioral model of the software. Most existing inference approaches assume access to code to collect execution sequences. In this paper, we investigate a black-box scenario, where the system under analysis cannot be instrumented in this fashion. This scenario is particularly common when it comes to the analysis of control system logs, which often take the form of continuous signals. In this situation, an execution trace amounts to a multivariate time-series of input and output signals, where different states of the system correspond to different “phases” in the time-series. From an inference perspective, the challenge is to detect when these phase changes take place. Unfortunately, most existing solutions are either univariate, make assumptions about the data distribution, or have limited learning power. In this paper we propose a hybrid deep neural network that accepts as input a multivariate time series and applies a set of convolutional and recurrent layers to learn the non-linear correlations between signals and the patterns over time. We show how this approach can be used to accurately detect state changes, and how the inferred models can be successfully applied to transfer-learning scenarios, to accurately process traces from different products with similar execution characteristics. Our experimental results on two UAV autopilot case studies (one industrial and one open-source) indicate that our approach is highly accurate (over 90% F1 score for state classification) and significantly improves baselines (by up to 102% for change point detection). Using transfer learning we also show that up to 90% of the maximum achievable F1 scores in the open-source case study can be achieved by reusing the trained models from the industrial case and only fine tuning them using as low as 5 labeled samples, which reduces the manual labeling effort by 98%.

Modelling the galaxy–halo connection with semi-recurrent neural networks

ABSTRACT We present an artificial neural network design in which past and present-day properties of dark matter haloes and their local environment are used to predict time-resolved star formation histories and stellar metallicity histories of central and satellite galaxies. Using data from the IllustrisTNG simulations, we train a tensorflow-based neural network with two inputs: a standard layer with static properties of the dark matter halo, such as halo mass and starting time; and a recurrent layer with variables such as overdensity and halo mass accretion rate, evaluated at multiple time steps from 0 ≤ z ≲ 20. The model successfully reproduces key features of the galaxy halo connection, such as the stellar-to-halo mass relation, downsizing, and colour bimodality, for both central and satellite galaxies. We identify mass accretion history as crucial in determining the geometry of the star formation history and trends with halo mass such as downsizing, while environmental variables are important indicators of chemical enrichment. We use these outputs to compute optical spectral energy distributions, and find that they are well matched to the equivalent results in IllustrisTNG, recovering observational statistics such as colour bimodality and mass–magnitude diagrams.

Improved action proposals using fine-grained proposal features with recurrent attention models

Recent models for the temporal action proposal task show that local properties can be an alternative to the region proposal network (RPN) for generating good proposal candidates on untrimmed videos. In this study, we devise an RPN model with a new two-stage pipeline and a new joint scoring function for temporal proposals. The evaluation of local properties is integrated into our RPN model to search for the best proposal candidates that can be distinguished mainly in fine details of proposal regions. Our network models proposals in multiple scales using two recurrent neural network layers with attention mechanisms. We observe that joint training of the RPN with local clues and multi-scale modeling of proposals with recurrent attention mechanisms improve the performance of the proposal generation task. Our model yields state-of-the-art results on the THUMOS-14 and comparable results on the ActivityNet-1.3 datasets.

Real-Time Monitoring of Parameters and Diagnostics of the Technical Condition of Small Unmanned Aerial Vehicle’s (UAV) Units Based on Deep BiGRU-CNN Models

The paper describes an original technique for the real-time monitoring of parameters and technical diagnostics of small unmanned aerial vehicle (UAV) units using neural network models with the proposed CompactNeuroUAV architecture. As input data, the operation parameter values for a certain period preceding the current and actual control actions on the UAV actuators are used. A reference parameter set model is trained based on historical data. CompactNeuroUAV is a combined neural network consisting of convolutional layers to compact data and recurrent layers with gated recurrent units to encode the time dependence of parameters. Processing provides the expected parameter value and estimates the deviation of the actual value of the parameter or a set of parameters from the reference model. Faults that have led to the deviation threshold crossing are then classified. A smart classifier is used here to detect the failed UAV unit and the fault or pre-failure condition cause and type. The paper also provides the results of experimental validation of the proposed approach to diagnosing faults and pre-failure conditions of fixed-wing type UAVs for the ALFA dataset. Models have been built to detect conditions such as engine thrust loss, full left or right rudder fault, elevator fault in a horizontal position, loss of control over left, right, or both ailerons in a horizontal position, loss of control over the rudder and ailerons stuck in a horizontal position. The results of estimating the developed model accuracy on a test dataset are also provided.

An adaptive active power rolling dispatch strategy for high proportion of renewable energy based on distributed deep reinforcement learning

In this article, an adaptive active power rolling dispatch strategy based on distributed deep reinforcement learning is proposed to deal with the uncertainty of high-proportioned renewable energy. For each agent, by using recurrent neural network layers and graph attention layers in its network structure, we aim to improve the generalization ability of the multiple agents in active power flow control. Furthermore, a regional graph attention network algorithm, which can effectively help agents aggregate the regional information of their neighborhood, is proposed to improve the information capture ability of agents. We adopt the structure of ‘centralized training, distributed execution’ to help agents improve the effectiveness of proposed methods in dynamic environments. The case studies demonstrate that the proposed algorithm can help multi-agents learn effective active power control strategies. Each agent has a strong generalization ability in terms of time granularity and network topology. We expect that such an approach can improve the practicability and adaptability of distributed AI method on power system control issues.

IMU-Based Classification of Locomotion Modes, Transitions, and Gait Phases with Convolutional Recurrent Neural Networks.

This paper focuses on the classification of seven locomotion modes (sitting, standing, level ground walking, ramp ascent and descent, stair ascent and descent), the transitions among these modes, and the gait phases within each mode, by only using data in the frequency domain from one or two inertial measurement units. Different deep neural network configurations are investigated and compared by combining convolutional and recurrent layers. The results show that a system composed of a convolutional neural network followed by a long short-term memory network is able to classify with a mean F1-score of 0.89 and 0.91 for ten healthy subjects, and of 0.92 and 0.95 for one osseointegrated transfemoral amputee subject (excluding the gait phases because they are not labeled in the data-set), using one and two inertial measurement units, respectively, with a 5-fold cross-validation. The promising results obtained in this study pave the way for using deep learning for the control of transfemoral prostheses with a minimum number of inertial measurement units.

DeepCLD: An Efficient Sequence-Based Predictor of Intrinsically Disordered Proteins.

Intrinsic disorder is common in proteins, plays important roles in protein functionality, and is commonly associated with various human diseases. To have an accurate tool for the annotation of intrinsic disorder in proteins, this paper proposes a novel algorithm, DeepCLD, for sequence-based prediction of intrinsically disordered proteins. This algorithm uses amino acid position specific scoring matrix (PSSM) to capture the intrinsic variability characteristic of sequence patterns, ResNet to preserve feature space structure, and bidirectional CudnnLSTM as recurrent layer to further improve the efficiency. Futhermore, DeepCLD also utilized the attention mechanism to solve the problem of gradient disappearing in deep network. Comparative analyses show that DeepCLD has faster training speed and higher prediction accuracy than comparable methods.

Qualifying and raising anti-money laundering alarms with deep learning

We propose a deep learning approach to qualify and raise anti-money laundering alarms in banks. The motivating idea is to replace predefined rules with latent features automatically extracted from transaction sequences. To this end, we experiment with recurrent and Transformer encoder layers. We test the approach with a large data set from Spar Nord Bank. Our best model employs gated recurrent units and self-attention. When used to qualify alarms raised by a traditional AML system, the model is able to reduce the number of false positives by more than 33.3% while retaining 98.8% of all true positives. In a small experiment, we also use the model to raise 75 new alarms on clients with high risk scores but with no traditional alarm inquiries in the last 12 months. After expert review, this prompted 26 clients to be reported to Danish authorities.

Combining recurrent and Graph Neural Networks to predict the next place’s category

Graph Neural Networks (GNNs) have a growing potential for helping solve problems in different contexts by working as a secondary technique that assists traditional methods. In the context of human mobility, it is expected that Recurrent Neural Networks (RNNs) are used to make predictions about individuals’ routine/mobility. In this work, we present POI-RGNN (Points of Interest-Recurrent and Graph-based Neural Network), a neural network for predicting the category of the next PoI that an individual will visit. Our proposal leverages Recurrent Neural Networks and Graph Neural Networks and combines them in a novel architecture. POI-RGNN explores new types of inputs sent to recurrent and graph layers. We evaluate the solution on a well-known and labeled dataset and a raw GPS-based dataset. For the latter, we propose a novel neural network model named Prediction of General Categories (PGC) for predicting a wide variety of PoI categories (e.g., Home, Work, Shopping, Nightlife, among others). We apply transfer learning with a GNN model that fuses different perspectives of the users’ mobility. As a result, we show that POI-RGNN leads to significant improvements compared to the state-of-the-art by combining RNNs and GNNs. Moreover, PGC can assign categories of places in an offline approach that requires minimum data.

EVALUATING DEEP LEARNING MODELS FOR MUSIC EMOTION RECOGNITION

Music listening helps people not only for entertainment, but also to reduce emotional stress in their daily lives. People nowadays tend to use online music streaming services such as Spotify, Amazon Music, Google Play Music, etc. rather than storing the songs on their devices. The songs in these streaming services are categorized into different emotional labels such as happy, sad, romantic, devotional, etc. In the music streaming applications, the songs are manually tagged with their emotional categories for music recommendation. Considering the growth of music on different social media platforms and the internet, the need for automatic tagging will increase in coming time. The work presented deals with training the deep learning model for automatic emotional tagging. It covers implementation of two different deep learning architectures for classifying the audio files using the Mel-spectrogram of music audio. The first architecture proposed is Convolutional Recurrent Model (CRNN) and the second architecture is a Parallel Convolutional Recurrent Model (Parallel CNN). Both the architectures exploit the combined features of Convolutional and Recurrent layers. This combination is used to extract features from time and frequency domains. The results with accuracies in the range of 51 to 54 % are promising for both models for a small dataset of 138 songs, considering the large datasets required for training deep learning models.

Investigating (re)current state-of-the-art in human activity recognition datasets

Many human activities consist of physical gestures that tend to be performed in certain sequences. Wearable inertial sensor data have as a consequence been employed to automatically detect human activities, lately predominantly with deep learning methods. This article focuses on the necessity of recurrent layers—more specifically Long Short-Term Memory (LSTM) layers—in common Deep Learning architectures for Human Activity Recognition (HAR). Our experimental pipeline investigates the effects of employing none, one, or two LSTM layers, as well as different layers' sizes, within the popular DeepConvLSTM architecture. We evaluate the architecture's performance on five well-known activity recognition datasets and provide an in-depth analysis of the per-class results, showing trends which type of activities or datasets profit the most from the removal of LSTM layers. For 4 out of 5 datasets, an altered architecture with one LSTM layer produces the best prediction results. In our previous work we already investigated the impact of a 2-layered LSTM when dealing with sequential activity data. Extending upon this, we now propose a metric, rGP, which aims to measure the effectiveness of learned temporal patterns for a dataset and can be used as a decision metric whether to include recurrent layers into a network at all. Even for datasets including activities without explicit temporal processes, the rGP can be high, suggesting that temporal patterns were learned, and consequently convolutional networks are being outperformed by networks including recurrent layers. We conclude this article by putting forward the question to what degree popular HAR datasets contain unwanted temporal dependencies, which if not taken care of, can benefit networks in achieving high benchmark scores and give a false sense of overall generability to a real-world setting.

Toward improved lumped groundwater level predictions at catchment scale: Mutual integration of water balance mechanism and deep learning method

Model development in groundwater simulation and physics informed deep learning (DL) has been advancing separately with limited integration. This study develops a general hybrid model for groundwater level (GWL) simulations, wherein water balance-based groundwater processes are embedded as physics constrained recurrent neural layers into prevalent DL architectures. Because of the automatic parameterizing process, physics-informed deep learning algorithm (DLA) equips the hybrid model with enhanced abilities of inferring geological structures of catchment and unobserved groundwater-related processes implicitly. The main purposes of this study are: 1) to explore an optimized data-driven method as alternative to complicated groundwater models; 2) to improve the awareness of hydrological knowledge of DL model for lumped GWL simulation; and 3) to explore the lumped data-driven groundwater models for cross-region applications. The 91 illustrative cases of GWL modeling across the middle eastern continental United States (CONUS) demonstrate that the hybrid model outperforms the pure DL models in terms of prediction accuracy, generality, and robustness. More specifically, the hybrid model outperforms the pure DL models in 78 % of catchments with the improved Δ NSE = 0.129. Meanwhile, the hybrid model simulates more stably with different input strategies. This study reveals the superiority and powerful simulation ability of the DL model with physical constraints, which increases trust in data-driven approaches on groundwater modellings.

Fully automated segmentation of prostatic urethra for MR-guided radiation therapy.

Accurate delineation of the urethra is a prerequisite for urethral dose reduction in prostate radiotherapy. However, even in magnetic resonance-guided radiation therapy (MRgRT), consistent delineation of the urethra is challenging, particularly in online adaptive radiotherapy. This paper presented a fully automatic MRgRT-based prostatic urethra segmentation framework. Twenty-eight prostate cancer patients were included in this study. In-house 3D half fourier single-shot turbo spin-echo (HASTE) and turbo spin echo (TSE) sequences were used to image the Foley-free urethra on a 0.35T MRgRT system. The segmentation pipeline uses 3D nnU-Net as the base and innovatively combines ground truth and its corresponding radial distance (RD) map during training supervision. Additionally, we evaluate the benefit of incorporating a convolutional long short term memory (LSTM-Conv) layer and spatial recurrent convolution layer (RCL) into nnU-Net. A novel slice-by-slice simple exponential smoothing (SEPS) method specifically for tubular structures was used to post-process the segmentation results. The experimental results show that nnU-Net trained using a combination of Dice, cross-entropy and RD achieved a Dice score of 77.1±2.3% in the testing dataset. With SEPS, Hausdorff distance (HD) and 95% HD were reduced to 2.95±0.17mm and 1.84±0.11mm, respectively. LSTM-Conv and RCL layers only minimally improved the segmentation precision. We present the first Foley-free MRgRT-based automated urethra segmentation study. Our method is built on a data-driven neural network with novel cost functions and a post-processing step designed for tubular structures. The performance is consistent with the need for online and offline urethra dose reduction in prostate radiotherapy.

Gated spatial-temporal graph neural network based short-term load forecasting for wide-area multiple buses

Existing short-term bus load forecasting methods mostly use temporal domain features, such as historical loads, to forecast and do not fully consider the influence of unstructured spatial–temporal coupling correlations among multiple bus loads in a wide spatial area on the forecasting results. In this paper, a wide-area multiple bus loads forecasting model is proposed. First, take the rapid computation of the maximal information coefficient (RapidMIC) value between bus loads as edge features, and combine the multiple node feature set to construct a similar-weighted spatial–temporal graph. The constructed similar-weighted spatial–temporal graph is not constrained by geography and grid topology, and can directly reflect the degree of spatial–temporal coupling correlation between bus loads. Second, the neighboring node features of each node in the graph are extracted by the spatial convolution layer (SCL) to achieve full-domain node feature enhancement. Finally, the features extracted were formed into temporal series and input to the gated recurrent unit layer (GRUL) to achieve a wide-area multiple buses short-term load forecast. The research was carried out by selecting the measured load data (15 busbars). The corresponding symmetric mean absolute percentage error (SMAPE) and mean absolute error (MAE) are 3.19 % to 5.89 % and 1.83 megawatts (MW) to 9.8 MW, respectively. Compared with the SMAPE of other methods, the worst evaluation metrics in the test set are improved by 1.82 % to 5.94 %, and it has better robustness in the scene with abnormal load data.

Influence-aware memory architectures for deep reinforcement learning in POMDPs

AbstractDue to its perceptual limitations, an agent may have too little information about the environment to act optimally. In such cases, it is important to keep track of the action-observation history to uncover hidden state information. Recent deep reinforcement learning methods use recurrent neural networks (RNN) to memorize past observations. However, these models are expensive to train and have convergence difficulties, especially when dealing with high dimensional data. In this paper, we propose influence-aware memory, a theoretically inspired memory architecture that alleviates the training difficulties by restricting the input of the recurrent layers to those variables that influence the hidden state information. Moreover, as opposed to standard RNNs, in which every piece of information used for estimating Q values is inevitably fed back into the network for the next prediction, our model allows information to flow without being necessarily stored in the RNN’s internal memory. Results indicate that, by letting the recurrent layers focus on a small fraction of the observation variables while processing the rest of the information with a feedforward neural network, we can outperform standard recurrent architectures both in training speed and policy performance. This approach also reduces runtime and obtains better scores than methods that stack multiple observations to remove partial observability.

Estimating the danger of snow avalanches with a machine learning approach using a comprehensive snow cover model

The avalanche warning services use the regional avalanche danger level and activity as key metrics. In this study a dataset of a ski resort from 2012 to 2020 was used. The local avalanche danger level, the success of artificial avalanche release by snow groomer or blasting, and the decision to try an avalanche release by blasting are target variables. Five meteorological recordings and twenty five variables of the modeled snowpack with an hourly resolution were used as input for the machine learning approach. An artificial neural network consists of recurrent layers and convolutional layers for merging the temporal and spatial data. Support vector machines were adapted to calculate the probabilities for the target variables. A logistic regression was used as ensemble method as stacked generalization.The accuracy of the models, i.e., the precision for predicting the local avalanche danger level was 0.73 and 0.52 for the train and test data, respectively. The precisions for artificial avalanche release by snow groomer were 0.94 and 0.85, for artificial avalanche release by blasting 0.75 and 0.72, and for the decision to try an avalanche release by blasting 0.75 and 0.70.Unlike previous studies, which used regional avalanche danger levels as target variable, we used local warning levels and additional target variables. The comparison shows that despite differences similar accuracy can be achieved. We further identified opportunities to optimize the model by examining the relevance of the input variables and performing a sensitivity analysis.

Study on the Application of Improved Audio Recognition Technology Based on Deep Learning in Vocal Music Teaching

As one of the hotspots in music information extraction research, music recognition has received extensive attention from scholars in recent years. Most of the current research methods are based on traditional signal processing methods, and there is still a lot of room for improvement in recognition accuracy and recognition efficiency. There are few research studies on music recognition based on deep neural networks. This paper expounds on the basic principles of deep learning and the basic structure and training methods of neural networks. For two kinds of commonly used deep networks, convolutional neural network and recurrent neural network, their typical structures, training methods, advantages, and disadvantages are analyzed. At the same time, a variety of platforms and tools for training deep neural networks are introduced, and their advantages and disadvantages are compared. TensorFlow and Keras frameworks are selected from them, and the practice related to neural network research is carried out. Training lays the foundation. Results show that through the development and experimental demonstration of the prototype system, as well as the comparison with other researchers in the field of humming recognition, it is proved that the deep-learning method can be applied to the humming recognition problem, which can effectively improve the accuracy of humming recognition and improve the recognition time. A convolutional recurrent neural network is designed and implemented, combining the local feature extraction of the convolutional layer and the ability of the recurrent layer to summarize the sequence features, to learn the features of the humming signal, so as to obtain audio features with a higher degree of abstraction and complexity and improve the performance of the humming signal. The ability of neural networks to learn the features of audio signals lays the foundation for an efficient and accurate humming recognition process.

Growing deep echo state network with supervised learning for time series prediction

Multilayer echo state networks (ESNs) are powerful on learning hierarchical temporal representation. However, how to determine the depth of multilayer ESNs is still an open issue. In this paper, we propose a novel approach to automatically determine the depth of a multilayer ESN, named growing deep ESN (GD-ESN). First, an incremental hierarchical structure is proposed, where the recurrent layers and the pre-trained feedforward layers are alternately added to the network one by one. Then, a control scheme is designed for the growth of the network based on the newly defined averaged mutual information and the full rank criterion. Finally, the proposed GD-ESN is evaluated on both benchmark datasets and real-world applications. The experimental results show the effectiveness of the proposed method.

Deep learning for predicting refractive error from multiple photorefraction images

BackgroundRefractive error detection is a significant factor in preventing the development of myopia. To improve the efficiency and accuracy of refractive error detection, a refractive error detection network (REDNet) is proposed that combines the advantages of a convolutional neural network (CNN) and a recurrent neural network (RNN). It not only extracts the features of each image, but also fully utilizes the sequential relationship between images. In this article, we develop a system to predict the spherical power, cylindrical power, and spherical equivalent in multiple eccentric photorefraction images.ApproachFirst, images of the pupil area are extracted from multiple eccentric photorefraction images; then, the features of each pupil image are extracted using the REDNet convolution layers. Finally, the features are fused by the recurrent layers in REDNet to predict the spherical power, cylindrical power, and spherical equivalent.ResultsThe results show that the mean absolute error (MAE) values of the spherical power, cylindrical power, and spherical equivalent can reach 0.1740 D (diopters), 0.0702 D, and 0.1835 D, respectively.SignificanceThis method demonstrates a much higher accuracy than those of current state-of-the-art deep-learning methods. Moreover, it is effective and practical.