Convolutional Recurrent Network Research Articles

Video Super-Resolution Using Non-Simultaneous Fully Recurrent Convolutional Network.

Video super-resolution (SR) aims at restoring fine details and enhancing visual experience for low-resolution (LR) videos. In this paper, we propose a very deep non-simultaneous fully recurrent convolutional network for video SR. To make full use of temporal information, we employ motion compensation, very deep fully recurrent convolutional layers and late fusion in our system. Residual connection is also employed in our recurrent structure for more accurate SR. Finally a new model ensemble strategy is used to combine our method with single-image SR method. Experimental results demonstrate that the proposed method is better than state-of-the-art SR methods on quantitative visual quality assessment.

Deep recurrent convolutional networks for inferring user interests from social media

Online social media services, such as Facebook and Twitter, have recently increased in popularity. Although determining the subjects of individual posts is important for extracting users’ interests from social media, this task is nontrivial because posts are highly contextualized, informal, and limited in length. To address this problem, we propose a deep-neural-network-based approach for predicting user interests in social media. In our framework, a word-embedding technique is used to map the words in social media content into vectors. These vectors are used as input to a bidirectional gated recurrent unit (biGRU). Then, the output of the biGRU and the word-embedding vectors are used to construct a sentence matrix. The sentence matrix is then used as input to a convolutional neural network (CNN) model to predict a user’s interests. Experimental results show that our proposed method combining biGRU and CNN models outperforms existing methods for identifying users’ interests from social media. In addition, posts in social media are sensitive to trends and change with time. Here, we collected posts from two different social media platforms at different time intervals, and trained the proposed model with one set of social media data and tested it with another set of social media data. The experimental results showed that our proposed model can predict users’ interests from the independent data set with high accuracies.

Sequential Video VLAD: Training the Aggregation Locally and Temporally.

As characterizing videos simultaneously from spatial and temporal cues has been shown crucial for the video analysis, the combination of convolutional neural networks and recurrent neural networks, i.e., recurrent convolution networks (RCNs), should be a native framework for learning the spatio-temporal video features. In this paper, we develop a novel sequential vector of locally aggregated descriptor (VLAD) layer, named SeqVLAD, to combine a trainable VLAD encoding process and the RCNs architecture into a whole framework. In particular, sequential convolutional feature maps extracted from successive video frames are fed into the RCNs to learn soft spatio-temporal assignment parameters, so as to aggregate not only detailed spatial information in separate video frames but also fine motion information in successive video frames. Moreover, we improve the gated recurrent unit (GRU) of RCNs by sharing the input-to-hidden parameters and propose an improved GRU-RCN architecture named shared GRU-RCN (SGRU-RCN). Thus, our SGRU-RCN has a fewer parameters and a less possibility of overfitting. In experiments, we evaluate SeqVLAD with the tasks of video captioning and video action recognition. Experimental results on Microsoft Research Video Description Corpus, Montreal Video Annotation Dataset, UCF101, and HMDB51 demonstrate the effectiveness and good performance of our method.

Towards occupant activity driven smart buildings via WiFi-enabled IoT devices and deep learning

With the unprecedented advancement of Internet of Things (IoT), automatic occupant activity recognition is becoming realizable for a myriad of emerging applications in smart buildings for energy efficiency and user experience enhancement. Existing activity recognition approaches require either the deployment of extra infrastructure or the cooperation of occupants to carry dedicated devices, which are expensive, intrusive and inconvenient for pervasive implementation. In this paper, we propose DeepHare, a deep learning-based human activity recognition scheme that can automatically identify common activities using only commodity WiFi-enabled IoT devices. We design a novel OpenWrt-based IoT platform to collect Channel State Information (CSI) measurements from commercial IoT devices. Moreover, an innovative deep learning framework, Autoencoder Long-term Recurrent Convolutional Network (AE-LRCN), is proposed. We developed dedicated neural network architectures to sanitize the noise in raw CSI data, extract high-level salient features and reveal the inherent temporal dependencies among data for accurate human activity recognition. All the parameters in AE-LRCN are fine-tuned end-to-end automatically. Extensive experiments are conducted in typical three indoor environments and the experimental results demonstrate that DeepHare outperforms existing methods and achieves a 97.6% activity recognition accuracy without human intervention.

A Deep Spatial Contextual Long-Term Recurrent Convolutional Network for Saliency Detection.

Traditional saliency models usually adopt hand-crafted image features and human-designed mechanisms to calculate local or global contrast. In this paper, we propose a novel computational saliency model, i.e., deep spatial contextual long-term recurrent convolutional network (DSCLRCN), to predict where people look in natural scenes. DSCLRCN first automatically learns saliency related local features on each image location in parallel. Then, in contrast with most other deep network based saliency models which infer saliency in local contexts, DSCLRCN can mimic the cortical lateral inhibition mechanisms in human visual system to incorporate global contexts to assess the saliency of each image location by leveraging the deep spatial long short-term memory (DSLSTM) model. Moreover, we also integrate scene context modulation in DSLSTM for saliency inference, leading to a novel deep spatial contextual LSTM (DSCLSTM) model. The whole network can be trained end-to-end and works efficiently when testing. Experimental results on two benchmark datasets show that DSCLRCN can achieve state-of-the-art performance on saliency detection. Furthermore, the proposed DSCLSTM model can significantly boost the saliency detection performance by incorporating both global spatial interconnections and scene context modulation, which may uncover novel inspirations for studies on them in computational saliency models.

Video representation by dense trajectories motion map applied to human activity recognition

ABSTRACTThis paper introduces an efficient video representation method based on the dense trajectory motion map (DTM). We utilize the salient features of dense trajectories and motion descriptor to integrate the discriminative information of a video into a map. Firstly, we extract the dense trajectories features by using dense optical flow then multiple descriptors are computed along trajectories to capture appearance and motion information. This result is then integrated into frames difference to integrate entire discriminative information and motion energy to get our first motion map. For the final DTM each generated motion map will be integrated with the absolute frame difference of next two frames till the end of entire video. Finally, we process the resultant DTM by exploring the efficient long-term recurrent convolutional network module for encoding and action label generation. The developed approach is shown better and had comparable recognition results over the existing methods when applied to the publically available human action datasets.

Co-Saliency-Enhanced Deep Recurrent Convolutional Networks for Human Fall Detection in E-Healthcare.

This paper addresses the issue of fall detection from videos for e-healthcare and assisted-living. Instead of using conventional hand-crafted features from videos, we propose a fall detection scheme based on co-saliency-enhanced recurrent convolutional network (RCN) architecture for fall detection from videos. In the proposed scheme, a deep learning method RCN is realized by a set of Convolutional Neural Networks (CNNs) in segment-levels followed by a Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), to handle the time-dependent video frames. The co-saliency-based method enhances salient human activity regions hence further improves the deep learning performance. The main contributions of the paper include: (a) propose a recurrent convolutional network (RCN) architecture that is dedicated to the tasks of human fall detection in videos; (b) integrate a co-saliency enhancement to the deep learning scheme for further improving the deep learning performance; (c) extensive empirical tests for performance analysis and evaluation under different network settings and data partitioning. Experiments using the proposed scheme were conducted on an open dataset containing multicamera videos from different view angles, results have shown very good performance (test accuracy 98.96%). Comparisons with two existing methods have provided further support to the proposed scheme.

SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network.

We propose an analysis of surgical videos that is based on a novel recurrent convolutional network (SV-RCNet), specifically for automatic workflow recognition from surgical videos online, which is a key component for developing the context-aware computer-assisted intervention systems. Different from previous methods which harness visual and temporal information separately, the proposed SV-RCNet seamlessly integrates a convolutional neural network (CNN) and a recurrent neural network (RNN) to form a novel recurrent convolutional architecture in order to take full advantages of the complementary information of visual and temporal features learned from surgical videos. We effectively train the SV-RCNet in an end-to-end manner so that the visual representations and sequential dynamics can be jointly optimized in the learning process. In order to produce more discriminative spatio-temporal features, we exploit a deep residual network (ResNet) and a long short term memory (LSTM) network, to extract visual features and temporal dependencies, respectively, and integrate them into the SV-RCNet. Moreover, based on the phase transition-sensitive predictions from the SV-RCNet, we propose a simple yet effective inference scheme, namely the prior knowledge inference (PKI), by leveraging the natural characteristic of surgical video. Such a strategy further improves the consistency of results and largely boosts the recognition performance. Extensive experiments have been conducted with the MICCAI 2016 Modeling and Monitoring of Computer Assisted Interventions Workflow Challenge dataset and Cholec80 dataset to validate SV-RCNet. Our approach not only achieves superior performance on these two datasets but also outperforms the state-of-the-art methods by a significant margin.

Multi-Rate Gated Recurrent Convolutional Networks for Video-Based Pedestrian Re-Identification

Matching pedestrians across multiple camera views has attracted lots of recent research attention due to its apparent importance in surveillance and security applications.While most existing works address this problem in a still-image setting, we consider the more informative and challenging video-based person re-identification problem, where a video of a pedestrian as seen in one camera needs to be matched to a gallery of videos captured by other non-overlapping cameras. We employ a convolutional network to extract the appearance and motion features from raw video sequences, and then feed them into a multi-rate recurrent network to exploit the temporal correlations, and more importantly, to take into account the fact that pedestrians, sometimes even the same pedestrian, move in different speeds across different camera views. The combined network is trained in an end-to-end fashion, and we further propose an initialization strategy via context reconstruction to largely improve the performance. We conduct extensive experiments on the iLIDS-VID and PRID-2011 datasets, and our experimental results confirm the effectiveness and the generalization ability of our model.

Learning Differences Between Visual Scanning Patterns Can Disambiguate Bipolar and Unipolar Patients

Bipolar Disorder (BD) and Major Depressive Disorder (MDD) are two common and debilitating mood disorders. Misdiagnosing BD as MDD is relatively common and the introduction of markers to improve diagnostic accuracy early in the course of the illness has been identified as one of the top unmet needs in the field. In this paper, we present novel methods to differentiate between BD and MDD patients. The methods use deep learning techniques to quantify differences between visual scanning patterns of BD and MDD patients. In the methods, visual scanning patterns that are described by ordered sequences of fixations on emotional faces are encoded into a lower dimensional space and are fed into a long-short term memory recurrent neural network (RNN). Fixation sequences are encoded by three different methods: 1) using semantic regions of interests (RoIs) that are manually defined by experts, 2) using semi-automatically defined grids of RoIs, or 3) using a convolutional neural network (CNN) to automatically extract visual features from saliency maps. Using data from 47 patients with MDD and 26 patients with BD we showed that using semantic RoIs, the RNN improved the performance of a baseline classifier from an AUC of 0.603 to an AUC of 0.878. Similarly using grid RoIs, the RNN improved the performance of a baseline classifier from an AUC of 0.450 to an AUC of 0.828. The classifier that automatically extracted visual features from saliency maps (a long recurrent convolutional network that is fully data-driven) had an AUC of 0.879. The results of the study suggest that by using RNNs to learn differences between fixation sequences the diagnosis of individual patients with BD or MDD can be disambiguated with high accuracy. Moreover, by using saliency maps and CNN to encode the fixation sequences the method can be fully automated and achieve high accuracy without relying on user expertise and/or manual labelling. When compared with other markers, the performance of the class of classifiers that was introduced in this paper is better than that of detectors that use differences in neural structures, neural activity or cortical hemodynamics to differentiate between BD and MDD patients. The novel use of RNNs to quantify differences between fixation sequences of patients with mood disorders can be easily generalized to studies of other neuropsychological disorders and to other fields such as psychology and advertising.

Recurrent convolutional network for video-based smoke detection

Video-based smoke detection plays an important role in the fire detection community. Such interesting topic, however, always suffers from great challenge due to the large variances of smoke texture, shape and color in the real applications. To effectively exploiting the long-range motion context, we propose a novel video-based smoke detection method via Recurrent Neural Networks (RNNs). More concretely, the proposed method first captures the space and motion context information by using deep convolutional motion-space networks. Then a temporal pooling layer and RNNs are used to effectively train the smoke model. Finally, to promote further research and evaluation of video-based smoke models, we also construct a new large database of 3000 challenging smoke video clips that cover large variations in illuminance and weather conditions. Experimental results demonstrate that our proposed method is capable of achieving state-of-the-art performance on all public benchmarks.

Interactive Temporal Recurrent Convolution Network for Traffic Prediction in Data Centers

Accurately predicting future service traffic would be of great help for load balancing and resource allocation, which plays a key role in guaranteeing the quality of service (QoS) in cloud computing. With the rapid development of data center, the large-scale network traffic prediction requires more suitable methods to deal with the complex properties (e.g., high-dimension, long-range dependence, non-linearity, and so on). However, due to the limitations of traditional methods (e.g., strong theoretical assumptions and simple implementation), few research works could predict the large-scale network traffic efficiently and accurately. More importantly, most of the studies took only the temporal features but without the services’ communications into consideration, which may weaken the QoS of applications in the data center. To this end, we applied the gated recurrent unit (GRU) model and the interactive temporal recurrent convolution network (ITRCN) to single-service traffic prediction and interactive network traffic prediction, respectively. Especially, ITRCN takes the communications between services as a whole and directly predicts the interactive traffic in large-scale network. Within the ITRCN model, the convolution neural network (CNN) part learns network traffic as images to capture the network-wide services’ correlations, and the GRU part learns the temporal features to help the interactive network traffic prediction. We conducted comprehensive experiments based on the Yahoo! data sets, and the results show that the proposed novel method outperforms the conventional GRU and CNN method by an improvement of 14.3% and 13.0% in root mean square error, respectively.

Intelligent intrusion detection system featuring a virtual fence, active intruder detection, classification, tracking, and action recognition

An intrusion detection system (IDS) is primarily used to protect nuclear power plants from external threats, such as sabotage and malicious attacks. However, earlier versions of IDSs are configured to detect an intrusion from visual inspection by an operator. This has the disadvantages of requiring standby human resources and relying on operator capabilities. In this paper, therefore, we propose an image-based intelligent intrusion detection system (IIDS) with a virtual fence, active intruder detection, classification, and tracking, and motion recognition to solve these limitations. An integrated acquisition device was manufactured combining optical and thermal cameras to compensate for the disadvantages of optical cameras, which have difficulty detecting an intrusion at night, under adverse weather conditions, and when the intruder is camouflaged. The virtual fence has a function to set the boundary between surveillance and external areas in a graphical user interface, and to define an early pre-alarm area if necessary. The background model is designed to detect moving objects, and detected objects are segmented into bounding boxes. We implemented a network model based on a convolutional neural network (CNN) to classify moving objects as either intruders or wild animals. If an intruder is detected in real time and is crossing the virtual fence, the alarm tile blinks with the associated color. Five types of intruder behavior patterns are recognized by optimizing a long-term recurrent convolutional network (LRCN) model. The proposed IIDS meets the physical protection requirements recommended in the nuclear regulatory guidelines, and can be used as an unmanned surveillance system. It is expected to perform more active and reliable intrusion detection in combination with existing sensors, such as microwaves, electric fields, and fence disturbance sensors in a nuclear power plant.

Video super-resolution based on spatial-temporal recurrent residual networks

In this paper, we propose a new video Super-Resolution (SR) method by jointly modeling intra-frame redundancy and inter-frame motion context in a unified deep network. Different from conventional methods, the proposed Spatial-Temporal Recurrent Residual Network (STR-ResNet) investigates both spatial and temporal residues, which are represented by the difference between a high resolution (HR) frame and its corresponding low resolution (LR) frame and the difference between adjacent HR frames, respectively. This spatial-temporal residual learning model is then utilized to connect the intra-frame and inter-frame redundancies within video sequences in a recurrent convolutional network and to predict HR temporal residues in the penultimate layer as guidance to benefit estimating the spatial residue for video SR. Extensive experiments have demonstrated that the proposed STR-ResNet is able to efficiently reconstruct videos with diversified contents and complex motions, which outperforms the existing video SR approaches and offers new state-of-the-art performances on benchmark datasets.

Superpixel-based 3D deep neural networks for hyperspectral image classification

This paper presents a novel hyperspectral image (HSI) classification method to effectively exploit the 3D spectral-spatial information via superpixel-based 3D deep neural networks (3D DNNs). Superpixel can represent the structure of HSI with adaptive sizes and shapes, and therefore, it is incorporated into 3D DNNs to improve the classification performance, especially for noisy classification and boundary misclassification. First, a spatial feature image via superpixel is constructed to increase the spectral-spatial similarity and diversity. Second, a 3D superpixel-based sample filling method is designed to solve the misclassification problem of boundaries. Third, a 3D recurrent convolutional networks (3D RCNNs) are designed to further exploit spatial continuity and suppress noisy prediction. Experimental results on real HSI datasets demonstrate the superiority of the proposed method over several well-known methods in both visual appearance and classification accuracy.

Learning Spatial-Semantic Context with Fully Convolutional Recurrent Network for Online Handwritten Chinese Text Recognition

Online handwritten Chinese text recognition (OHCTR) is a challenging problem as it involves a large-scale character set, ambiguous segmentation, and variable-length input sequences. In this paper, we exploit the outstanding capability of path signature to translate online pen-tip trajectories into informative signature feature maps, successfully capturing the analytic and geometric properties of pen strokes with strong local invariance and robustness. A multi-spatial-context fully convolutional recurrent network (MC-FCRN) is proposed to exploit the multiple spatial contexts from the signature feature maps and generate a prediction sequence while completely avoiding the difficult segmentation problem. Furthermore, an implicit language model is developed to make predictions based on semantic context within a predicting feature sequence, providing a new perspective for incorporating lexicon constraints and prior knowledge about a certain language in the recognition procedure. Experiments on two standard benchmarks, Dataset-CASIA and Dataset-ICDAR, yielded outstanding results, with correct rates of 97.50 and 96.58 percent, respectively, which are significantly better than the best result reported thus far in the literature.

Adult content detection in videos with convolutional and recurrent neural networks

The amount of adult content on the Internet grows daily. Much of the pornographic content is unconstrained and freely-available for all users, requiring parents to make use of parental control strategies for protecting their children. Current parental control devices depend on human intervention, and hence there is the need of computational approaches for automatically detecting and blocking pornographic content. Toward that goal, this paper proposes ACORDE, a novel deep learning architecture that comprises both convolutional neural networks and LSTM recurrent networks for adult content detection in videos. Experiments over the freely-available NPDI dataset show that ACORDE significantly outperforms the previous state-of-the-art approaches for this task, decreasing by half the number of false positives and by a third the number of false negatives.

Spatiotemporal Recurrent Convolutional Networks for Traffic Prediction in Transportation Networks.

Predicting large-scale transportation network traffic has become an important and challenging topic in recent decades. Inspired by the domain knowledge of motion prediction, in which the future motion of an object can be predicted based on previous scenes, we propose a network grid representation method that can retain the fine-scale structure of a transportation network. Network-wide traffic speeds are converted into a series of static images and input into a novel deep architecture, namely, spatiotemporal recurrent convolutional networks (SRCNs), for traffic forecasting. The proposed SRCNs inherit the advantages of deep convolutional neural networks (DCNNs) and long short-term memory (LSTM) neural networks. The spatial dependencies of network-wide traffic can be captured by DCNNs, and the temporal dynamics can be learned by LSTMs. An experiment on a Beijing transportation network with 278 links demonstrates that SRCNs outperform other deep learning-based algorithms in both short-term and long-term traffic prediction.

Video Super-Resolution via Bidirectional Recurrent Convolutional Networks.

Super resolving a low-resolution video, namely video super-resolution (SR), is usually handled by either single-image SR or multi-frame SR. Single-Image SR deals with each video frame independently, and ignores intrinsic temporal dependency of video frames which actually plays a very important role in video SR. Multi-Frame SR generally extracts motion information, e.g., optical flow, to model the temporal dependency, but often shows high computational cost. Considering that recurrent neural networks (RNNs) can model long-term temporal dependency of video sequences well, we propose a fully convolutional RNN named bidirectional recurrent convolutional network for efficient multi-frame SR. Different from vanilla RNNs, 1) the commonly-used full feedforward and recurrent connections are replaced with weight-sharing convolutional connections. So they can greatly reduce the large number of network parameters and well model the temporal dependency in a finer level, i.e., patch-based rather than frame-based, and 2) connections from input layers at previous timesteps to the current hidden layer are added by 3D feedforward convolutions, which aim to capture discriminate spatio-temporal patterns for short-term fast-varying motions in local adjacent frames. Due to the cheap convolutional operations, our model has a low computational complexity and runs orders of magnitude faster than other multi-frame SR methods. With the powerful temporal dependency modeling, our model can super resolve videos with complex motions and achieve well performance.

Retrieval of Sentence Sequences for an Image Stream via Coherence Recurrent Convolutional Networks.

We propose an approach for retrieving a sequence of natural sentences for an image stream. Since general users often take a series of pictures on their experiences, much online visual information exists in the form of image streams, for which it would better take into consideration of the whole image stream to produce natural language descriptions. While almost all previous studies have dealt with the relation between a single image and a single natural sentence, our work extends both input and output dimension to a sequence of images and a sequence of sentences. For retrieving a coherent flow of multiple sentences for a photo stream, we propose a multimodal neural architecture called coherence recurrent convolutional network (CRCN), which consists of convolutional neural networks, bidirectional long short-term memory (LSTM) networks, and an entity-based local coherence model. Our approach directly learns from vast user-generated resource of blog posts as text-image parallel training data. We collect more than 22 K unique blog posts with 170 K associated images for the travel topics of NYC, Disneyland , Australia, and Hawaii. We demonstrate that our approach outperforms other state-of-the-art image captioning methods for text sequence generation, using both quantitative measures and user studies via Amazon Mechanical Turk.