Intermediate Feature Maps Research Articles

Fine-Grained Age Estimation With Multi-Attention Network

Human age estimation from a single image is a quite challenging task due to the subtle appearance change in the slow aging process. In this article, we propose a compact multi-attention deep network for age estimation based on the idea of fine-grained learning and visual attention mechanism. Concerning the problem that age estimation is a fine-grained visual classification problem, it relies on not only the global features of the face image, but also the fine-grained feature representations from age-sensitive local regions. Therefore, accurate age estimation benefits from multi-scale features and their fusion. Therefore, in this article, a multi-attention model built on a complementary two-stream compact network is proposed for age estimation. For a given intermediate feature map from the network, spatial attentions and channel attentions can be inferred in both self-attention and mutual-attention way. To emphasize crucial features from age-sensitive regions, the multi-attention maps are then multiplied to the input feature map for adaptive feature refinement. Finally, the refined feature maps at multiple layers are aggregated as the fine-grained feature for age estimation. Compared to bulky models, our model is compact and end-to-end. However, the performance of our model is competitive compared with those state-of-the-art methods.

Cascaded Attention Guidance Network for Single Rainy Image Restoration.

Restoring a rainy image with raindrops or rainstreaks of varying scales, directions, and densities is an extremely challenging task. Recent approaches attempt to leverage the rain distribution (e.g., location) as prior to generate satisfactory results. However, concatenation of a single distribution map with the rainy image or with intermediate feature maps is too simplistic to fully exploit the advantages of such priors. To further explore this valuable information, an advanced cascaded attention guidance network, dubbed as CAG-Net, is formulated and designed as a three-stage model. In the first stage, a multitask learning network is constructed for producing the attention map and coarse de-raining results simultaneously. Subsequently, the coarse results and the rain distribution map are concatenated and fed to the second stage for results refinement. In this stage, the attention map generation network from the first stage is used to formulate a novel semantic consistency loss for better detail recovery. In the third stage, a novel pyramidal "whereand- how" learning mechanism is formulated. At each pyramid level, a two-branch network is designed to take the features from previous stages as inputs to generate better attention-guidance features and de-raining features, which are then combined via a gating scheme to produce the final de-raining results. Moreover, the uncertainty maps are also generated in this stage for more accurate pixel-wise loss calculation. Extensive experiments are carried out for removing raindrops or rainstreaks from both synthetic and real rainy images, and CAG-Net is demonstrated to produce significantly better results than state-of-the-art models. Code will be publicly available after paper acceptance.

SEnDec: An Improved Image to Image CNN for Foreground Localization

Although it is not immediately intuitive that Deep Convolutional Neural Networks (DCNNs) can yield adequate feature representation for a Foreground Localization (FGL) task, recent architectural and algorithmic advancements in Deep Learning (DL) have shown that the DCNNs have become forefront methodology for this pixel-level classification problem. In FGL, the DCNNs face an inherent trade-off between moving objects, i.e., the foreground (FG) and the non-static background (BG) scenes, through learning from local- and global-level features. Driven by the latest success of the innovative structures for image classification and semantic segmentation, this work introduces a novel architecture, called Slow Encoder-Decoder (sEnDec) that aims to improve the learning capacity of a traditional image-to-image DCNN. The proposed model subsumes two subnets for contraction (encoding) and expansion (decoding), wherein both phases, it employs an intermediate feature map up-sampling and residual connections. In this way, the lost structural details due to spatial subsampling are recovered. It helps to get a more delineated FG region. The experimental study is carried out with two variants of the proposed model: one with strided convolution (conv) and the other with max pooling for spatial subsampling. A comparative analysis on sixteen benchmark video sequences, including baseline, dynamic background, camera jitter, shadow effects, intermittent object motion, night videos, and bad weather show that the proposed sEnDec model performs very competitively against the prior- and state-of-the-art approaches.

Residual attention unit for action recognition

3D CNNs are powerful tools for action recognition that can intuitively extract spatio-temporal features from raw videos. However, most of the existing 3D CNNs have not fully considered the disadvantageous effects of the background motion that frequently appears in videos. The background motion is usually misclassified as a part of human action, which may undermine modeling the dynamic pattern of the action. In this paper, we propose the residual attention unit (RAU) to address this problem. RAU aims to suppress the background motion by upweighting the values associated with the foreground region in the feature maps. Specifically, RAU contains two separate submodules in parallel, i.e., spatial attention as well as channel-wise attention. Given an intermediate feature map, the spatial attention works in a bottom-up top-down manner to generate the attention mask, while the channel-wise attention recalibrates the feature responses of all channels automatically. As applying the attention mechanism directly to the input features may lead to the loss of discriminative information, we design a bypass to preserve the integrity of the original features by a shortcut connection between the input and output of the attention module. Notably, our RAU can be embedded into 3D CNNs easily and enables end-to-end training along with the networks. The experimental results on UCF101 and HMDB51 demonstrate the validity of our RAU.

A Real-Time Convolutional Neural Network for Super-Resolution on FPGA With Applications to 4K UHD 60 fps Video Services

In this paper, we present a novel hardware-friendly super-resolution (SR) method based on a convolutional neural network (CNN) and its dedicated hardware (HW) on field programmable gate array (FPGA). Although CNN-based SR methods have shown very promising results for SR, their computational complexities are prohibitive for hardware implementation. To the best of our knowledge, we are the first to implement a real-time CNN-based SR HW that upscales 2K full high-definition video to 4K ultra high-definition (UHD) video at 60 frames per second (fps). In our dedicated CNN-based SR HW, low-resolution input frames are processed line-by-line, and the number of convolutional filter parameters is reduced significantly by incorporating depth-wise separable convolutions with a residual connection. Our CNN-based SR HW incorporates a cascade of 1D convolutions having large receptive fields along horizontal lines while keeping vertical receptive fields minimal, which allows us to save required line memory space in achieving comparable SR performance against full 2D convolution operations. For efficient HW implementation, we use a simple and effective quantization method with little peak signal-to-noise ratio (PSNR) degradation. Also, we propose a compression method to efficiently store intermediate feature map data to reduce the number of line memories used in HW. Our HW implementation on the FPGA generates 4K UHD frames of higher PSNR values at 60 fps and shows better visual quality, compared with conventional CNN-based SR methods that are trained and tested in software.

3D RANs: 3D Residual Attention Networks for action recognition

In this work, we propose 3D Residual Attention Networks (3D RANs) for action recognition, which can learn spatiotemporal representation from videos. The proposed network consists of attention mechanism and 3D ResNets architecture, and it can capture spatiotemporal information in an end-to-end manner. Specifically, we separately add the attention mechanism along channel and spatial domain to each block of 3D ResNets. For each sliced tensor of an intermediate feature map, we sequentially infer channel and spatial attention maps by channel and spatial attention mechanism submodules in each residual unit block, and the attention maps are multiplied to the input feature map to reweight the key features. We validate our network through extensive experiments in UCF-101, HMDB-51 and Kinetics datasets. Our experiments show that the proposed 3D RANs are superior to the state-of-the-art approaches for action recognition, demonstrating the effectiveness of our networks.

Low Complexity CNN Structure for Automatic Bleeding Zone Detection in Wireless Capsule Endoscopy Imaging.

Wireless capsule endoscopy (WCE) is a swallowable device used for screening different parts of the human digestive system. Automatic WCE image analysis methods reduce the duration of the screening procedure and alleviate the burden of manual screening by medical experts. Recent studies widely employ convolutional neural networks (CNNs) for automatic analysis of WCE images; however, these studies do not consider CNN's structural and computational complexities. In this paper, we address the problem of simplifying the CNN's structure. A low complexity CNN structure for bleeding zone detection is proposed which takes a single patch as input and then outputs a segmented patch of the same size. The proposed network is inspired by the FCN paradigm with a simplified structure. Since it is based on image patches, the resulting network benefits from moderate-sized intermediate feature maps. Moreover, the problem of redundant computations in patch-based methods is circumvented by non-overlapping patch processing. The proposed method is evaluated using the publicly available KID dataset for WCE image analysis. Experimental results show that the proposed network has better accuracy and AUC than previous structures while requiring less computational operations.

Selective Multi-Scale Feature Learning by Discriminative Local Representation

In the computer vision community, the general trend has been to capture and select discriminative features in order to yield significantly better performance. Recent advances in attention mechanism proposed several attention blocks to adaptively recalibrate the feature response. However, most of them overlooked the context information at a multi-scale level. In this paper, we propose a simple yet effective building block for ResNeXt-style backbones, namely discriminative local representation (DLR) module, which allows discriminative local representation learning for multi-scale feature information across multi-parallel branches. Our DLR module contains two sub-modules: channel selective module (CSM) and spatial selective module (SSM). Given an intermediate feature map, the CSM first selectively generates the channel-wise attention maps and recalibrates the response from different branches according to the weight vector calculated by softmax layer. And then, the SSM further captures the spatial discriminative information at different scales respectively and emphasizes the interdependent channel maps. Besides, we place a high-order item during the process of multi-branch fusion and residual connection to enhance the intensity of structure nonlinearity. Various DLR modules can be stacked to a deep convolution network named DLRNet. To validate our DLRNet, we conduct comprehensive experiments on classification benchmarks (i.e. CIFAR10, CIFAR100 and ImageNet-1K), as well as two publicly available fine-grained datasets (i.e. CUB-200-2011 and Stanford Dogs). The experiments show consistent improvement gains over previous baseline models with reasonable overhead, and demonstrate the capability of our proposed method for discriminative local representation.

Mixture separability loss in a deep convolutional network for image classification

In machine learning, the cost function is crucial because it measures how good or bad a system is. In image classification, well-known networks only consider modifying the network structures and applying cross-entropy loss at the end of the network. However, using only cross-entropy loss causes a network to stop updating weights when all training images are correctly classified. This is the problem of early saturation. This study proposes a novel cost function, called mixture separability loss (MSL), which updates the weights of the network even when most of the training images are accurately predicted. MSL consists of between-class and within-class loss. Between-class loss maximises the differences between inter-class images, whereas within-class loss minimises the similarities between intra-class images. They designed the proposed loss function to attach to different convolutional layers in the network in order to utilise intermediate feature maps. Experiments show that a network with MSL deepens the learning process and obtains promising results with some public datasets, such as Street View House Number, Canadian Institute for Advanced Research, and the authors' self-collected Inha Computer Vision Lab gender dataset.

Rank pooling dynamic network: Learning end-to-end dynamic characteristic for action recognition

Abstract In video recognition, rank-pooling operators are a type of models for sorting video sequences, which act on either the raw inputs or the intermediate feature maps of convolutional neural network (CNN). However, such models are currently restricted in the optimization of the linear ranking function by Rank-SVM and Rank-SVR. In this paper, we first propose a CNN architecture called RGB Rank Pooling Dynamic Network (RGB-RPDN), mapping a video to multiple frame-level dynamic spaces with the same size as the input. Importantly, a classical classification (e.g. FC, CNN) advanced in 2D image can be jointly positioned behind the generated representation for action classification, thus the joint architecture can be trained in an end-to-end manner. Second, we analyze how the flow-level evolution can be modeled by the hand-crafted rank-pooling machine, and extend the dynamic space of frame-level to that of flow-level by the Flow Rank Pooling Dynamic Network (Flow-RPDN). Third, equivalence relations between hand-crafted rank-pooling and RPDN are formulated, further the comparison of computing cost are qualitatively analyzed. Finally, the frame-level and flow-level pipelines are combined to achieve the final prediction by the late fusion. Specifically, with the models pre-trained on the large-scale Kinetics dataset, we train the two-stream RPDN on the UCF101 and HMDB51, where the parameters are initialized by the pre-trained models above. Experimental results demonstrate that the RPDN significantly improves the hand-crafted rank-pooling machines by a large margin of promotion, and achieves the correct rate of more excellent classification in action recognition.