Deep Recurrent Network Research Articles

Spatio-Temporal Deep Residual Network with Hierarchical Attentions for Video Event Recognition

Event recognition in surveillance video has gained extensive attention from the computer vision community. This process still faces enormous challenges due to the tiny inter-class variations that are caused by various facets, such as severe occlusion, cluttered backgrounds, and so forth. To address these issues, we propose a spatio-temporal deep residual network with hierarchical attentions (STDRN-HA) for video event recognition. In the first attention layer, the ResNet fully connected feature guides the Faster R-CNN feature to generate object-based attention (O-attention) for target objects. In the second attention layer, the O-attention further guides the ResNet convolutional feature to yield the holistic attention (H-attention) in order to perceive more details of the occluded objects and the global background. In the third attention layer, the attention maps use the deep features to obtain the attention-enhanced features. Then, the attention-enhanced features are input into a deep residual recurrent network, which is used to mine more event clues from videos. Furthermore, an optimized loss function named softmax-RC is designed, which embeds the residual block regularization and center loss to solve the vanishing gradient in a deep network and enlarge the distance between inter-classes. We also build a temporal branch to exploit the long- and short-term motion information. The final results are obtained by fusing the outputs of the spatial and temporal streams. Experiments on the four realistic video datasets, CCV, VIRAT 1.0, VIRAT 2.0, and HMDB51, demonstrate that the proposed method has good performance and achieves state-of-the-art results.

DRAN: Deep recurrent adversarial network for automated pancreassegmentation

Automated pancreas segmentation in abdominal computed tomography (CT) scans is of high clinical relevance (i.e. pancreas cancer diagnosis and prognosis), but extremely difficult because the pancreas is a soft, small, and flexible abdominal organ with high anatomical variability, which causes the previous segmentation methods to result in low precision. In this study, the authors present a new deep recurrent adversarial network (DRAN) to tackle this challenge. DRAN contains three steps: (i) preserving global resolution of CT scans and modifying the receptive field of kernel adaptively through a dilated convolution autoencoder module; (ii) modelling contextual spatial correlation between neighbouring CT scan patches benefits from a specially designed local long short-term memory module; and (iii) improving the performance and generalisation by leveraging an adversarial module, which can constrain the spatial smoothness consistency between continuous CT scans based on the long-range spatial interaction. The system is evaluated on a dataset of 80 manually segmented CT volumes, using four-fold cross-validation. Its performance surpasses other state-of-the-art methods, with the Dice similarity coefficient of 89.87 ± 3.17 % and pixel-wise accuracy of 95.85 ± 3.04 % . Also, they perform a qualitative evaluation by an expert further revealing the effectiveness and potential of their DRAN as a clinical segmentation tool.

Embedding Encoder-Decoder With Attention Mechanism for Monaural Speech Enhancement

The auditory selection framework with attention and memory (ASAM), which has an attention mechanism, embedding generator, generated embedding array, and life-long memory, is used to deal with mixed speech. When ASAM is applied to speech enhancement, the discrepancy between the voice and noise feature memories is huge and the separability of noise and voice is increased. However, ASAM cannot achieve desirable performance in terms of speech enhancement because it fails to utilize the time-frequency dependence of the embedding vectors to generate a corresponding mask unit. This work proposes a novel embedding encoder-decoder (EED), and a convolutional neural network (CNN) is used as decoder. The CNN structure is good at detecting local patterns, which can be exploited to extract correlation embedding data from the embedding array to generate the target spectrogram. This work evaluates a similar ASAM, EED with an LSTM encoder and a CNN decoder (RC-EED), RC-EED with an attention mechanism (RC-AEED), other similar EED structures and baseline models. Experiment results show that RC-EED and RC-AEED networks have good performance on speech enhancement task at low signal-to-noise ratio conditions. In addition, RC-AEED exhibits superior speech enhancement performance over ASAM and achieves better speech quality than do deep recurrent network and convolutional recurrent network.

Modelling of pollutants and particulate matter in air using auto-tuned deep recurrent networks

Atmospheric pollutants and Particulate Matter of size less than 10µm (PM10) are becoming dominant in the atmosphere due to human activities and natural calamities. To address their associated problems on human health, the interactions between pollutants and PM10 have to be envisaged. Machine learning techniques like Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) were successfully employed in establishing the interactions between various factors at play. However, these techniques are denounced for following a heuristic approach for determining network hyper-parameters. We propose a novel evolutionary multi-objective optimization algorithm which can optimally determine the hyper-parameters in deep recurrent neural networks. We test the algorithm to build optimal RNNs and LSTMs for modelling and forecasting the pollutants and PM10 data generated in northern Taiwan region during the year 2015. A state-of-the-art network training algorithm, Truncated Back Propagation Through Time was used in our study and single variable regression was done for CO, NOx, SO2, and PM10. Except for SO2 with RNN, model developed with the proposed algorithm gave high R2 values. LSTM was found to be superior than RNN in all the cases with R2 going as high as 0.9584 for PM10, while that attained by RNN is 0.93.

Variational Conditioning of Deep Recurrent Networks for Modeling Complex Motion Dynamics

This work introduces stochastic models to address the problem of complex motion generation. Long-term motion generation is the primary task in several fields; however, less work has happened in this direction and is restricted to single-person activities. Looking forward, this work strives to solve the problem of two-person 3D motion generation. Single-person motion generation is comparatively simpler than two-person, where the complexity is significantly higher. Error-propagation during motion generation is a key challenge. Current approaches often fail to keep the predicted skeletons on the manifold of valid poses. Another challenge is when the model restricts its learning to the training distribution. This leads to stagnation in generated moves. To this end, an end-to-end hierarchical auto-regressive model is proposed for efficient long-term motion generation. It is further constrained by an alignment network to reduce prediction errors. Proposed approach shows comparable results on prediction tasks, while outperforming the state-of-the-art on long-term motion generation.

PoSeg: Pose-Aware Refinement Network for Human Instance Segmentation

Human instance segmentation is a core problem for human-centric scene understanding and segmenting human instances poses a unique challenge to vision systems due to large intra-class variations in both appearance and shape, and complicated occlusion patterns. In this paper, we propose a new pose-aware human instance segmentation method. Compared to the previous pose-aware methods which first predict bottom-up poses and then estimate instance segmentation on top of predicted poses, our method integrates both top-down and bottom-up cues for an instance: it adopts detection results as human proposals and jointly estimates human pose and instance segmentation for each proposal. We develop a modular recurrent deep network that utilizes pose estimation to refine instance segmentation in an iterative manner. Our refinement modules exploit pose cues in two levels: as a coarse shape prior and local part attention. We evaluate our approach on two public multi-person benchmarks: OCHuman dataset and COCOPersons dataset. The proposed method surpasses the state-of-the-art methods on OCHuman dataset by 3.0 mAP and on COCOPersons by 6.4 mAP, demonstrating the effectiveness of our approach.

Unsupervised Pre-training of a Deep LSTM-based Stacked Autoencoder for Multivariate Time Series Forecasting Problems

Currently, most real-world time series datasets are multivariate and are rich in dynamical information of the underlying system. Such datasets are attracting much attention; therefore, the need for accurate modelling of such high-dimensional datasets is increasing. Recently, the deep architecture of the recurrent neural network (RNN) and its variant long short-term memory (LSTM) have been proven to be more accurate than traditional statistical methods in modelling time series data. Despite the reported advantages of the deep LSTM model, its performance in modelling multivariate time series (MTS) data has not been satisfactory, particularly when attempting to process highly non-linear and long-interval MTS datasets. The reason is that the supervised learning approach initializes the neurons randomly in such recurrent networks, disabling the neurons that ultimately must properly learn the latent features of the correlated variables included in the MTS dataset. In this paper, we propose a pre-trained LSTM-based stacked autoencoder (LSTM-SAE) approach in an unsupervised learning fashion to replace the random weight initialization strategy adopted in deep LSTM recurrent networks. For evaluation purposes, two different case studies that include real-world datasets are investigated, where the performance of the proposed approach compares favourably with the deep LSTM approach. In addition, the proposed approach outperforms several reference models investigating the same case studies. Overall, the experimental results clearly show that the unsupervised pre-training approach improves the performance of deep LSTM and leads to better and faster convergence than other models.

Deep Recurrent Network for Fast and Full-Resolution Light Field Deblurring

Restoring a sharp light field image from its blurry input has become essential due to the increasing popularity of parallax-based image processing. State-of-the-art blind light field deblurring methods suffer from several issues such as slow processing, reduced spatial size, and a limited motion blur model. In this work, we address these challenging problems by generating a complex blurry light field dataset and proposing a learning-based deblurring approach. In particular, we model the full 6-degree of freedom (6-DOF) light field camera motion, which is used to create the blurry dataset using a combination of real light fields captured with a Lytro Illum camera, and synthetic light field renderings of 3D scenes. Furthermore, we propose a light field deblurring network that is built with the capability of large receptive fields. We also introduce a simple strategy of angular sampling to train on the large-scale blurry light field effectively. We evaluate our method through both quantitative and qualitative measurements and demonstrate superior performance compared to the state-of-the-art method with a massive speedup in execution time. Our method is about 16K times faster than Srinivasan et. al. [22] and can deblur a full-resolution light field in less than 2 seconds.

Deep Progressive Hashing for Image Retrieval

Hashing is a widely adopted method based on an approximate nearest neighbor search and is used in large-scale image retrieval tasks. Conventional learning-based hashing algorithms employ end-to-end representation learning, which is a one-off technique. Because of the tradeoff between efficiency and performance, conventional learning-based hashing methods must sacrifice code length to improve performance, which increases their computational complexity. To improve the efficiency of binary codes, motivated by the “nonsalient-to-salient” attention scheme of humans, we propose a recursive hashing mechanism that maps progressively expanded salient regions to a series of binary codes. These salient regions are generated by a conventional saliency model based on bottom-up saliency-driven attention and a semantic-guided saliency model based on top-down task-driven attention. After obtaining a series of salient regions, we perform long-range temporal modeling of salient regions using a graph-based recurrent deep network to obtain more refined representative features. The later output nodes inherit aggregated information from all previous nodes and extract discriminative features from more salient regions. Therefore, this network possesses more significant information and satisfactory scalability. The proposed recursive hashing neural network, optimized by a triplet ranking loss, is end-to-end trainable. Extensive experimental results from several image retrieval benchmarks show the scalability of our method and demonstrate its strong performance compared with state-of-the-art methods.

Accurate breast lesion segmentation by exploiting spatio-temporal information with deep recurrent and convolutional network

Breast lesion segmentation in dynamic contrast enhanced magnetic resonance imaging (DCE-MRI) is a challenging task since DCE-MRI includes not only spatial information within slices but also temporal correlation between different sequences. Previous works only focused on one-side of the spatio-temporal information, therefore could not obtain accurate segmentation results. In this paper, we propose an automatic breast lesion segmentation framework with much less parameters utilizing both spatial and temporal features. Specifically, a breast region extraction is applied to eliminate unconcerned tissues. After that, the segmentation result can be obtained by a spatio-temporal segmentation network, where we exploit a multi-pathway structure and a fusion operation to unearth more spatio-temporal features. The proposed framework is validated on our real dataset and obtains a more accurate segmentation result compared to other approaches.

Deep Multi-Scale Recurrent Network for Synthetic Aperture Radar Images Despeckling

For the existence of speckles, many standard optical image processing methods, such as classification, segmentation, and registration, are restricted to synthetic aperture radar (SAR) images. In this work, an end-to-end deep multi-scale recurrent network (MSR-net) for SAR image despeckling is proposed. The multi-scale recurrent and weights sharing strategies are introduced to increase network capacity without multiplying the number of weights parameters. A convolutional long short-term memory (convLSTM) unit is embedded to capture useful information and helps with despeckling across scales. Meanwhile, the sub-pixel unit is utilized to improve the network efficiency. Besides, two criteria, edge feature keep ratio (EFKR) and feature point keep ratio (FPKR), are proposed to evaluate the performance of despeckling capacity for SAR, which can assess the retention ability of the despeckling algorithm to edge and feature information more effectively. Experimental results show that our proposed network can remove speckle noise while preserving the edge and texture information of images with low computational costs, especially in the low signal noise ratio scenarios. The peak signal to noise ratio (PSNR) of MSR-net can outperform traditional despeckling methods SAR-BM3D (Block-Matching and 3D filtering) by more than 2 dB for the simulated image. Furthermore, the adaptability of optical image processing methods to real SAR images can be enhanced after despeckling.

Recurrence is required to capture the representational dynamics of the human visual system

The human visual system is an intricate network of brain regions that enables us to recognize the world around us. Despite its abundant lateral and feedback connections, object processing is commonly viewed and studied as a feedforward process. Here, we measure and model the rapid representational dynamics across multiple stages of the human ventral stream using time-resolved brain imaging and deep learning. We observe substantial representational transformations during the first 300 ms of processing within and across ventral-stream regions. Categorical divisions emerge in sequence, cascading forward and in reverse across regions, and Granger causality analysis suggests bidirectional information flow between regions. Finally, recurrent deep neural network models clearly outperform parameter-matched feedforward models in terms of their ability to capture the multiregion cortical dynamics. Targeted virtual cooling experiments on the recurrent deep network models further substantiate the importance of their lateral and top-down connections. These results establish that recurrent models are required to understand information processing in the human ventral stream.

Recurrent Deep Network Models for Clinical NLP Tasks: Use Case with Sentence Boundary Disambiguation.

Although a number of foundational natural language processing (NLP) tasks like text segmentation are considered a simple problem in the general English domain dominated by well-formed text, complexities of clinical documentation lead to poor performance of existing solutions designed for the general English domain. We present an alternative solution that relies on a convolutional neural network layer followed by a bidirectional long short-term memory layer (CNN-Bi-LSTM) for the task of sentence boundary disambiguation and describe an ensemble approach for domain adaptation using two training corpora. Implementations using the Keras neural-networks API are available at https://github.com/NLPIE/clinical-sentences.

Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expressions

Recently, the recognition task of spontaneous facial micro-expressions has attracted much attention with its various real-world applications. Plenty of handcrafted or learned features have been employed for a variety of classifiers and achieved promising performances for recognizing micro-expressions. However, the micro-expression recognition is still challenging due to the subtle spatiotemporal changes of micro-expressions. To exploit the merits of deep learning, we propose a novel deep recurrent convolutional networks based micro-expression recognition approach, capturing the spatial-temporal deformations of micro-expression sequence. Specifically, the proposed deep model is constituted of several recurrent convolutional layers for extracting visual features and a classificatory layer for recognition. It is optimized by an end-to-end manner and obviates manual feature design. To handle sequential data, we exploit two types of extending the connectivity of convolutional networks across temporal domain, in which the spatiotemporal deformations are modeled in views of facial appearance and geometry separately. Besides, to overcome the shortcomings of limited and imbalanced training samples, temporal data augmentation strategies as well as a balanced loss are jointly used for our deep network. By performing the experiments on three spontaneous micro-expression datasets, we verify the effectiveness of our proposed micro-expression recognition approach compared to the state-of-the-art methods.

Combo-Action: Training Agent For FPS Game with Auxiliary Tasks

Deep reinforcement learning (DRL) has achieved surpassing human performance on Atari games, using raw pixels and rewards to learn everything. However, first-person-shooter (FPS) games in 3D environments contain higher levels of human concepts (enemy, weapon, spatial structure, etc.) and a large action space. In this paper, we explore a novel method which can plan on temporally-extended action sequences, which we refer as Combo-Action to compress the action space. We further train a deep recurrent Q-learning network model as a high-level controller, called supervisory network, to manage the Combo-Actions. Our method can be boosted with auxiliary tasks (enemy detection and depth prediction), which enable the agent to extract high-level concepts in the FPS games. Extensive experiments show that our method is efficient in training process and outperforms previous stateof-the-art approaches by a large margin. Ablation study experiments also indicate that our method can boost the performance of the FPS agent in a reasonable way.

Playing a FPS Doom Video Game with Deep Visual Reinforcement Learning

Because of the advancement in Deep visual reinforcement learning, now autonomous game agents are allowed to perform well which often leave behind human beings by using only the raw screen pixels for making their actions or decisions. In this paper, we propose Deep Q-Network (DQN) and a Deep Recurrent Q-Learning Network (DRQN) implementation by playing the Doom video game. Our findings are based on a publication from Lample and Chaplot (2016). Deep Q-learning under two variants (DQN and DRQN) applied is presented first, then how we build an implementation of a testbed for such algorithms is described. we presented our results on a simplified game scenario(s) by showing the predicted enemy positions (game features) with the difference in performance of DQN and DRQN. Finally, unlike other existing works, we show that our proposed architecture performs better with an accuracy of almost 72% in predicting the enemy positions.

Efficient neural decoding of self-location with a deep recurrent network.

Place cells in the mammalian hippocampus signal self-location with sparse spatially stable firing fields. Based on observation of place cell activity it is possible to accurately decode an animal’s location. The precision of this decoding sets a lower bound for the amount of information that the hippocampal population conveys about the location of the animal. In this work we use a novel recurrent neural network (RNN) decoder to infer the location of freely moving rats from single unit hippocampal recordings. RNNs are biologically plausible models of neural circuits that learn to incorporate relevant temporal context without the need to make complicated assumptions about the use of prior information to predict the current state. When decoding animal position from spike counts in 1D and 2D-environments, we show that the RNN consistently outperforms a standard Bayesian approach with either flat priors or with memory. In addition, we also conducted a set of sensitivity analysis on the RNN decoder to determine which neurons and sections of firing fields were the most influential. We found that the application of RNNs to neural data allowed flexible integration of temporal context, yielding improved accuracy relative to the more commonly used Bayesian approaches and opens new avenues for exploration of the neural code.

Quantum Entanglement in Deep Learning Architectures.

Modern deep learning has enabled unprecedented achievements in various domains. Nonetheless, employment of machine learning for wave function representations is focused on more traditional architectures such as restricted Boltzmann machines (RBMs) and fully connected neural networks. In this Letter, we establish that contemporary deep learning architectures, in the form of deep convolutional and recurrent networks, can efficiently represent highly entangled quantum systems. By constructing tensor network equivalents of these architectures, we identify an inherent reuse of information in the network operation as a key trait which distinguishes them from standard tensor network-based representations, and which enhances their entanglement capacity. Our results show that such architectures can support volume-law entanglement scaling, polynomially more efficiently than presently employed RBMs. Thus, beyond a quantification of the entanglement capacity of leading deep learning architectures, our analysis formally motivates a shift of trending neural-network-based wave function representations closer to the state-of-the-art in machine learning.

DRFN: Deep Recurrent Fusion Network for Single-Image Super-Resolution With Large Factors

Recently, single-image super-resolution has made great progress owing to the development of deep convolutional neural networks (CNNs). The vast majority of CNN-based models use a pre-defined upsampling operator, such as bicubic interpolation, to upscale input low-resolution images to the desired size and learn non-linear mapping between the interpolated image and ground truth high-resolution (HR) image. However, interpolation processing can lead to visual artifacts as details are over-smoothed, particularly when the super-resolution factor is high. In this paper, we propose a Deep Recurrent Fusion Network (DRFN), which utilizes transposed convolution instead of bicubic interpolation for upsampling and integrates different-level features extracted from recurrent residual blocks to reconstruct the final HR images. We adopt a deep recurrence learning strategy and thus have a larger receptive field, which is conducive to reconstructing an image more accurately. Furthermore, we show that the multi-level fusion structure is suitable for dealing with image super-resolution problems. Extensive benchmark evaluations demonstrate that the proposed DRFN performs better than most current deep learning methods in terms of accuracy and visual effects, especially for large-scale images, while using fewer parameters.

A Novel Deep Recurrent Belief Network Model for Trend Prediction of Transformer DGA Data

Oil chromatography data together with its variation trend provide the key basis for the evaluation of the transformer health state. The existing studies on deep belief network (DBN) and support vector machine (SVM) have reported a few results in the field of oil chromatography data prediction. However, the above-mentioned methods are proposed for the classification problem, so there is no theoretical basis for applying the above-mentioned methods to the time-series prediction. The wrong usage limits the accuracy of the predicted results, which is observed as the obvious “time-shift” error in the prediction curve, leading to the predicted state inconsistent with the actual situation. To fill the gap, a deep recurrent belief network (DRBN) model for the transformer state prediction was proposed based on the time-series theory and oil chromatography data characteristics. In this model, the construction shortage of DBN in time-series prediction was analyzed as a prototype pattern to structure the stereoscopic mapping relation. The self-adaptive delay network based on the principle of autocorrelation, together with the corresponding error feedback network, realized the stereoscopic flow of data in multi-dimensional space and, thus, ensured the model's valid expression of the time-domain correlation. In addition, the cross-entropy loss function, based on the Kullback-Leibler divergence and the Weibull distribution with an obvious characteristic of DGA, was constructed to eliminate the uncertainty of initialization process and also effectively control the direction and step size of the error gradient. The examples in the field were used to verify the method to find that the model proposed in this paper can availably overcome the “time-shift” error, and its prediction accuracy can reach more than 95.16%.