Convolutional Feature Maps Research Articles

Health indicator construction of machinery based on end-to-end trainable convolution recurrent neural networks

In the field of prognostic and health management of engineered systems, health indicator construction of bearings is one of the most significant and challenging problems. Many data-driven approaches centered on deep learning have been proposed recently in the context of smart manufacturing, where massive condition monitoring data could be collected. Among them, there are two representative methods, i.e., the convolution neural networks (CNN) based method and the recurrent neural networks (RNN) based method. However, there are some problems with them. The former has small receptive field size and cannot encode time-series information that is crucial for determining the bearings degradation degree, while the latter need hand-crafted features with prior knowledge of experts. Aimed at these problems, an intelligent and end-to-end health indicator construction approach is proposed. It combines structural advantages of previous two methods. It firstly converts the original input data into a series of local features that maintain chronological order in the convolution feature map. Then the sequential local features are elegantly connected by a recurrent neural network, which makes the extracted features in the recurrent layer contain global semantic information with time series. The bearing experiment under two different operating conditions demonstrates that the proposed method is reliable and effective in establishing bearing health indicator and characterizes the nonlinear degradation trend of bearings into approximately linear process over time. The experimental results also show that the proposed method achieves better results concerning trendability and monotonicity, compared with the CNN-based method and the RNN-based method.

Convolutional Networks With Channel and STIPs Attention Model for Action Recognition in Videos

With the help of convolutional neural networks (CNNs), video-based human action recognition has made significant progress. CNN features that are spatial and channel-wise can provide rich information for powerful image description. However, CNNs lack the ability to process the long-term temporal dependency of an entire video and further cannot well focus on the informative motion regions of actions. Aiming at the two problems, we propose a novel video-based action recognition framework in this paper. We first represent videos with dynamic image sequences (DISs), which effectively describe videos by modeling the local spatial-temporal dynamics and dependencies. Then a channel and spatial-temporal interest points (STIPs) attention model (CSAM) based on CNNs is proposed to focus on the discriminative channels in networks and the informative spatial motion regions of human actions. Specifically, channel attention (CA) is implemented by automatically learning channel-wise convolutional features and assigning different weights for different channels. STIPs attention (SA) is encoded by projecting the detected STIPs on frames of dynamic image sequences into the corresponding convolutional feature map space. The proposed CSAM is embedded after CNN convolutional layers to refine the feature maps, followed by global average pooling to produce effective feature representations for videos. Finally frame-level video representations are fed into an LSTM to capture the temporal dependencies and make classification. Experiments on three challenging RGB-D datasets show that our method has better performance and outperforms the state-of-the-art approaches with only depth data.

Pseudo-3D Attention Transfer Network with Content-aware Strategy for Image Captioning

In this article, we propose a novel Pseudo-3D Attention Transfer network with Content-aware Strategy (P3DAT-CAS) for the image captioning task. Our model is composed of three parts: the Pseudo-3D Attention (P3DA) network, the P3DA-based Transfer (P3DAT) network, and the Content-aware Strategy (CAS). First, we propose P3DA to take full advantage of three-dimensional (3D) information in convolutional feature maps and capture more details. Most existing attention-based models only extract the 2D spatial representation from convolutional feature maps to decide which area should be paid more attention to. However, convolutional feature maps are 3D and different channel features can detect diverse semantic attributes associated with images. P3DA is proposed to combine 2D spatial maps with 1D semantic-channel attributes and generate more informative captions. Second, we design the transfer network to maintain and transfer the key previous attention information. The traditional attention-based approaches only utilize the current attention information to predict words directly, whereas transfer network is able to learn long-term attention dependencies and explore global modeling pattern. Finally, we present CAS to provide a more relevant and distinct caption for each image. The captioning model trained by maximum likelihood estimation may generate the captions that have a weak correlation with image contents, resulting in the cross-modal gap between vision and linguistics. However, CAS is helpful to convey the meaningful visual contents accurately. P3DAT-CAS is evaluated on Flickr30k and MSCOCO, and it achieves very competitive performance among the state-of-the-art models.

Semantic–geometric visual place recognition: a new perspective for reconciling opposing views

Human drivers are capable of recognizing places from a previous journey even when viewing them from the opposite direction during the return trip under radically different environmental conditions, without needing to look back or employ a camera or LIDAR sensor. Such navigation capabilities are attributed in large part to the robust semantic scene understanding capabilities of humans. However, for an autonomous robot or vehicle, achieving such human-like visual place recognition capability presents three major challenges: (1) dealing with a limited amount of commonly observable visual content when viewing the same place from the opposite direction; (2) dealing with significant lateral viewpoint changes caused by opposing directions of travel taking place on opposite sides of the road; and (3) dealing with a radically changed scene appearance due to environmental conditions such as time of day, season, and weather. Current state-of-the-art place recognition systems have only addressed these three challenges in isolation or in pairs, typically relying on appearance-based, deep-learnt place representations. In this paper, we present a novel, semantics-based system that for the first time solves all three challenges simultaneously. We propose a hybrid image descriptor that semantically aggregates salient visual information, complemented by appearance-based description, and augment a conventional coarse-to-fine recognition pipeline with keypoint correspondences extracted from within the convolutional feature maps of a pre-trained network. Finally, we introduce descriptor normalization and local score enhancement strategies for improving the robustness of the system. Using both existing benchmark datasets and extensive new datasets that for the first time combine the three challenges of opposing viewpoints, lateral viewpoint shifts, and extreme appearance change, we show that our system can achieve practical place recognition performance where existing state-of-the-art methods fail.

Deep Metadata Fusion for Traffic Light to Lane Assignment

We present a deep metadata fusion approach that connects image data and heterogeneous metadata inside a Convolutional Neural Network (CNN). This approach enables us to assign all relevant traffic lights to their associated lanes. To achieve this, a common CNN topology is trained by down-sampled and transformed input images to predict an indication vector. The indication vector contains the column positions of all the relevant traffic lights that are associated with lanes. In parallel, we fuse prepared and adaptively weighted Metadata Feature Maps (MFM) with the convolutional feature map input of a selected convolutional layer. The results are compared to rule-based, only-metadata, and only-vision approaches. In addition, human performance of the traffic light to ego-vehicle lane assignment has been measured by a subjective test. The proposed approach outperforms all other approaches. It achieves about 93.0% average precision for a real-world dataset. In a more complex dataset, 87.1% average precision is achieved. In particular, the new approach reaches significantly higher results with 93.7% to 91.0% average accuracy for a real-world dataset in contrast to lower human performance.

Traffic sign detection method based on Faster R-CNN

Traffic sign detection is the pivotal technology of the traffic sign recognition system. In this article, a traffic sign detection method comes up based on Faster R-CNN deep learning framework. In this method, a convolution neural network is devoted to extract traffic sign image features automatically, and the extracted convolution feature map is sent into a Region Proposal Network (RPN) for foreground objects filtration and regression of bounding boxes. Then the proposed regions are mapped to the feature map, and the fixed-size proposal boxes via Region of Interest pooling layer (RoI). After that, we use the classification network to perform specific classification tasks and further compute the bounding box regression. The experiments performs on the German Traffic Sign Detection Benchmark (GTSDB) and the experimental results show that the method has effectiveness and robustness to different light, block, and motion.

Pose detection in complex classroom environment based on improved Faster R‐CNN

Pose detection of small targets in poor imaging conditions like heavy occlusion and low resolution is still an open and challenging task in computer vision. For instance, detection of students' poses in classrooms that are even indistinguishable to human eyes remains a rather difficult task. Motivated by the success of convolutional feature merging and locality preserving, the authors propose a pose detection framework combining merged region of interest (ROI) pooling and locality preserving learning. Unlike usual object detection algorithms which use general top-level convolutional features as inputs, their method uses a merged ROI pooling structure to merge semantic feature and high-resolution feature from the last two levels of convolutional feature maps, so that this merged feature is made more expressive than the single-level feature. In addition, the locality feature-preserving learning is used in the last fully-connected layer. Through locality preserving learning, features belonging to the same class would be forced to be closer in the feature space, which enables the model with stronger classification ability. Experimental results show that the proposed method outperforms the state-of-the-art methods.

Feature Boosting Network For 3D Pose Estimation.

In this paper, a feature boosting network is proposed for estimating 3D hand pose and 3D body pose from a single RGB image. In this method, the features learned by the convolutional layers are boosted with a new long short-term dependence-aware (LSTD) module, which enables the intermediate convolutional feature maps to perceive the graphical long short-term dependency among different hand (or body) parts using the designed Graphical ConvLSTM. Learning a set of features that are reliable and discriminatively representative of the pose of a hand (or body) part is difficult due to the ambiguities, texture and illumination variation, and self-occlusion in the real application of 3D pose estimation. To improve the reliability of the features for representing each body part and enhance the LSTD module, we further introduce a context consistency gate (CCG) in this paper, with which the convolutional feature maps are modulated according to their consistency with the context representations. We evaluate the proposed method on challenging benchmark datasets for 3D hand pose estimation and 3D full body pose estimation. Experimental results show the effectiveness of our method that achieves state-of-the-art performance on both of the tasks.

A convolutional feature map-based deep network targeted towards traffic detection and classification

Vehicle detection and classification is an important task for street surveillance and scene perception for robot navigation or autonomous vehicles. This research work focuses on traffic detection for real time applications using three components. The first component includes designing convolutional feature map-based classifier based on multimodal optical flow features. The second component is to utilize an effective adaptive learning rate technique to deal with saddle points; and to propose an average covariance matrix based pre-conditioning approach. The third component is to separately train multimodal model using blur data which caters blur effect of real time data. Extensive experimental results with different learning rates, architectures are reported using benchmark datasets such as Apollo, KITTI, Cityscapes, Berkeley, Caltech, PASCAL VOC and self created. Experimental results demonstrate that in comparison to fully connected network based classifier, Network on Convolutional (NoC) feature map classifier provided approximately 10% hike in classification accuracy without data per-processing, and almost 18% improvement with pre-processed data. The blur model enhances accuracy by almost 15% on blurred data as compared to normal RGB data. Moreover, multimodal features provide 12% and 2% higher accuracy while using standard classifiers and NoC classifiers respectively.

Learning to Fuse Multiscale Features for Visual Place Recognition

Efficient and robust visual place recognition is of great importance to autonomous mobile robots. Recent work has shown that features learned from convolutional neural networks achieve impressed performance with efficient feature size, where most of them are pooled or aggregated from a convolutional feature map. However, convolutional filters only capture the appearance of their perceptive fields, which lack the considerations on how to combine the multiscale appearance for place recognition. In this paper, we propose a novel method to build a multiscale feature pyramid and present two approaches to use the pyramid to augment the place recognition capability. The first approach fuses the pyramid to obtain a new feature map, which has an awareness of both the local and semi-global appearance, and the second approach learns an attention model from the feature pyramid to weight the spatial grids on the original feature map. Both approaches combine the multiscale features in the pyramid to suppress the confusing local features while tackling the problem in two different ways. Extensive experiments have been conducted on benchmark datasets with varying degrees of appearance and viewpoint variations. The results show that the proposed approaches achieve superior performance over the networks without the multiscale feature fusion and the multiscale attention components. Analyses on the performance of using different feature pyramids are also provided.

Video Copy Detection Using Spatio-Temporal CNN Features

To protect the copyright of digital videos, video copy detection has become a hot topic in the field of digital copyright protection. Since a video sequence generally contains a large amount of data, to achieve efficient and effective copy detection, the key issue is to extract compact and discriminative video features. To this end, we propose a video copy detection scheme using spatio-temporal convolutional neural network (CNN) features. First, we divide each video sequence into multiple video clips and sample the frames of each video clip. Second, the sampled frames of each video clip are fed into a pre-trained CNN model to generate the corresponding convolutional feature maps (CFMs). Third, based on the generated CFMs, we extract the CNN features on the spatial and temporal domains of each video clip, i.e., the spatio-temporal CNN features. Finally, video copy detection is efficiently and effectively implemented based on the extracted spatio-temporal CNN features. The experiments on the commonly used video dataset, i.e., TRECVID 2008, demonstrate that the proposed method performs well in aspects of both accuracy and efficiency and shows superiority to several other copy detection methods using the state-of-the-art features.

Weakly-Supervised Learning of a Deep Convolutional Neural Networks for Semantic Segmentation

Deep convolutional neural networks (DCNNs) trained on the pixel-wise annotated images have dramatically improved the state-of-the-art in semantic segmentation. However, due to the high cost of labeling training data, its application has great limitation. In this paper, we propose a DCNNs model for generating the pixel-level labels using the image-level annotation. The model consists of an encoder-decoder, a feature decomposer, and a multi-label classifier. The encoder extracts the deep convolutional feature maps of the input image. The feature decomposer can decompose the convolutional feature extracted by the encoder into feature components of different semantics. The decomposer is based on the orthogonal non-negative matrix factorization (NMF) technology. The function of the decoder is to map the feature components of different semantics to the input resolution images. The decoder uses the position index of maximum pooling provided by the corresponding encoder to perform non-linear up-sampling, which eliminates the learning requirement of up-sampling. Since the image reconstruction is conducted according to the semantic categories, image regions of different semantics are restored to different images. Then, the regions of different semantics can be segmented through the posted-processing algorithms. The experimental results on open data set show that the proposed model outperforms some recently developed methods.

MsRi-CCF: Multi-Scale and Rotation-Insensitive Convolutional Channel Features for Geospatial Object Detection

Geospatial object detection is a fundamental but challenging problem in the remote sensing community. Although deep learning has shown its power in extracting discriminative features, there is still room for improvement in its detection performance, particularly for objects with large ranges of variations in scale and direction. To this end, a novel approach, entitled multi-scale and rotation-insensitive convolutional channel features (MsRi-CCF), is proposed for geospatial object detection by integrating robust low-level feature generation, classifier generation with outlier removal, and detection with a power law. The low-level feature generation step consists of rotation-insensitive and multi-scale convolutional channel features, which were obtained by learning a regularized convolutional neural network (CNN) and integrating multi-scaled convolutional feature maps, followed by the fine-tuning of high-level connections in the CNN, respectively. Then, these generated features were fed into AdaBoost (chosen due to its lower computation and storage costs) with outlier removal to construct an object detection framework that facilitates robust classifier training. In the test phase, we adopted a log-space sampling approach instead of fine-scale sampling by using the fast feature pyramid strategy based on a computable power law. Extensive experimental results demonstrate that compared with several state-of-the-art baselines, the proposed MsRi-CCF approach yields better detection results, with 90.19% precision with the satellite dataset and 81.44% average precision with the NWPU VHR-10 datasets. Importantly, MsRi-CCF incurs no additional computational cost, which is only 0.92 s and 0.7 s per test image on the two datasets. Furthermore, we determined that most previous methods fail to gain an acceptable detection performance, particularly when they face several obstacles, such as deformations in objects (e.g., rotation, illumination, and scaling). Yet, these factors are effectively addressed by MsRi-CCF, yielding a robust geospatial object detection method.

Boundary-Guided Feature Aggregation Network for Salient Object Detection

Fully convolutional networks (FCN) has significantly improved the performance of many pixel-labeling tasks, such as semantic segmentation and depth estimation. However, it still remains non-trivial to thoroughly utilize the multi-level convolutional feature maps and boundary information for salient object detection. In this paper, we propose a novel FCN framework to integrate multi-level convolutional features recurrently with the guidance of object boundary information. First, a deep convolutional network is used to extract multi-level feature maps and separately aggregate them into multiple resolutions, which can be used to generate coarse saliency maps. Meanwhile, another boundary information extraction branch is proposed to generate boundary features. Finally, an attention-based feature fusion module is designed to fuse boundary information into salient regions to achieve accurate boundary inference and semantic enhancement. The final saliency maps are the combination of the predicted boundary maps and integrated saliency maps, which are more closer to the ground truths. Experiments and analysis on four large-scale benchmarks verify that our framework achieves new state-of-the-art results.

DCF-BoW: Build Match Graph Using Bag of Deep Convolutional Features for Structure From Motion

Matching a large number of images is quite time-consuming for structure from motion (SfM) due to the image matching by comparing features between all image pairs. In this letter, a bag of deep convolutional features (DCF-BoW) model is proposed to create match graph to reduce the number of matches. First, the convolutional feature map of an image is extracted using the VGG-16 convolutional neural network trained on ImageNet. Then, each local region in the original image can be represented by a feature vector in the feature map. The feature vectors are normalized and used to construct a bag of words model, which could convert each image into a DCF-BoW representation. Finally, the match graph is constructed by selecting the top 10 images with the highest similarities, which are calculated by computing the distances between those DCF-BoW representations. The experiment results show that the proposed DCF-BoW can create the match graph effectively in short time and find the potential overlapping image pairs. The match graph created by the proposed DCF-BoW is better than those built by DBoW3 and VocabTree, which is clearly showed in precision-recall curve on the Urban data set. The results of the SfM reconstruction based on the match graph created by the proposed DCF-BoW are slightly worse than those of the exhaustive matching, while the number of matches is reduced by 97.4% and 92.1%, respectively, on the Urban and South Building data sets.

Learning channel-aware deep regression for object tracking

Deep regression tracker assigns soft Gaussian labels to the dense samples drawn on convolution feature map. It has become a popular tracking framework as its intrinsic end-to-end learning characteristics. In this paper, we analyze the channel-wise information of convolution features in deep regression tracking and propose an effective channel-aware learning algorithm. The proposed algorithm can automatically select the most representative feature channels during the learning of deep regression tracker. We show that this channel selection procedure can be formulated as multi-task learning (MTL) and solved via simple gradient descent. Extensive experiments on OTB-2013 and OTB-2015 datasets validate the effectiveness of the proposed method.

Global Temporal Representation Based CNNs for Infrared Action Recognition

Infrared human action recognition has many advantages, i.e., it is insensitive to illumination change, appearance variability, and shadows. Existing methods for infrared action recognition are either based on spatial or local temporal information, however, the global temporal information, which can better describe the movements of body parts across the whole video, is not considered. In this letter, we propose a novel global temporal representation named optical-flow stacked difference image (OFSDI) and extract robust and discriminative feature from the infrared action data by considering the local, global, and spatial temporal information together. Due to the small size of the infrared action dataset, we first apply convolutional neural networks on local, spatial, and global temporal stream respectively to obtain efficient convolutional feature maps from the raw data rather than train a classifier directly. Then these convolutional feature maps are aggregated into effective descriptors named three-stream trajectory-pooled deep-convolutional descriptors by trajectory-constrained pooling. Furthermore, we improve the robustness of these features by using the locality-constrained linear coding (LLC) method. With these features, a linear support vector machine (SVM) is adopted to classify the action data in our scheme. We conduct the experiments on infrared action recognition datasets InfAR and NTU RGB+D. The experimental results show that the proposed approach outperforms the representative state-of-the-art handcrafted features and deep learning features based methods for the infrared action recognition.

Pushing the Limits of Deep CNNs for Pedestrian Detection

Compared with other applications in computer vision, convolutional neural networks (CNNs) have underperformed on pedestrian detection. A breakthrough was made very recently using sophisticated deep CNN (DCNN) models, with a number of handcrafted features or explicit occlusion handling mechanism. In this paper, we show that by reusing the convolutional feature maps of a DCNN model as image features to train an ensemble of boosted decision models, we are able to achieve the best reported accuracy without using specially designed learning algorithms. We empirically identify and disclose important implementation details. We also show that pixel labeling may be simply combined with a detector to boost the detection performance. By adding complementary handcrafted features such as optical flow, the DCNN-based detector can be further improved. We advance the state-of-the-art results by lowering the log-average miss rate from 11.7% to 8.9% on the Caltech data set and from 11.2% to 8.6% on the Inria data set. We also achieve a comparable result to state-of-the-art approaches on the KITTI data set.

A Deep Network Solution for Attention and Aesthetics Aware Photo Cropping.

We study the problem of photo cropping, which aims to find a cropping window of an input image to preserve as much as possible its important parts while being aesthetically pleasant. Seeking a deep learning-based solution, we design a neural network that has two branches for attention box prediction (ABP) and aesthetics assessment (AA), respectively. Given the input image, the ABP network predicts an attention bounding box as an initial minimum cropping window, around which a set of cropping candidates are generated with little loss of important information. Then, the AA network is employed to select the final cropping window with the best aesthetic quality among the candidates. The two sub-networks are designed to share the same full-image convolutional feature map, and thus are computationally efficient. By leveraging attention prediction and aesthetics assessment, the cropping model produces high-quality cropping results, even with the limited availability of training data for photo cropping. The experimental results on benchmark datasets clearly validate the effectiveness of the proposed approach. In addition, our approach runs at 5 fps, outperforming most previous solutions. The code and results are available at: https://github.com/shenjianbing/DeepCropping.

Modeling Temporal Tonal Relations in Polyphonic Music Through Deep Networks With a Novel Image-Based Representation

We propose an end-to-end approach for modeling polyphonic music with a novel graphical representation, based on music theory, in a deep neural network. Despite the success of deep learning in various applications, it remains a challenge to incorporate existing domain knowledge in a network without affecting its training routines. In this paper we present a novel approach for predictive music modeling and music generation that incorporates domain knowledge in its representation. In this work, music is transformed into a 2D representation, inspired by tonnetz from music theory, which graphically encodes musical relationships between pitches. This representation is incorporated in a deep network structure consisting of multilayered convolutional neural networks (CNN, for learning an efficient abstract encoding of the representation) and recurrent neural networks with long short-term memory cells (LSTM, for capturing temporal dependencies in music sequences). We empirically evaluate the nature and the effectiveness of the network by using a dataset of classical music from various composers. We investigate the effect of parameters including the number of convolution feature maps, pooling strategies, and three configurations of the network: LSTM without CNN, LSTM with CNN (pre-trained vs. not pre-trained). Visualizations of the feature maps and filters in the CNN are explored, and a comparison is made between the proposed tonnetz-inspired representation and pianoroll, a commonly used representation of music in computational systems. Experimental results show that the tonnetz representation produces musical sequences that are more tonally stable and contain more repeated patterns than sequences generated by pianoroll-based models, a finding that is directly useful for tackling current challenges in music and AI such as smart music generation.