Features Of Different Levels Research Articles

A Hierarchical CNN-RNN Approach for Visual Emotion Classification

Visual emotion classification is predicting emotional reactions of people for the given visual content. Psychological studies show that human emotions are affected by various visual stimuli from low level to high level, including contrast, color, texture, scene, object, and association, among others. Traditional approaches regarded different levels of stimuli as independent components and ignored to effectively fuse different stimuli. This article proposes a hierarchical convolutional neural network (CNN)-recurrent neural network (RNN) approach to predict the emotion based on the fused stimuli by exploiting the dependency among different-level features. First, we introduce a dual CNN to extract different levels of visual stimulus, where two related loss functions are designed to learn the stimuli representation under a multi-task learning structure. Further, to model the dependency between the low- and high-level stimulus, a stacked bi-directional RNN is proposed to fuse the preceding learned features from the dual CNN. Comparison experiments on one large-scale and three small scale datasets show that the proposed approach brings significant improvement. Ablation experiments demonstrate the effectiveness of different modules from our model.

Multi-label learning for concept-oriented labels of product image data

In the designing field, designers usually retrieve the images for reference according to product attributes when designing new proposals. To obtain the attributes of the product, the designers take lots of time and effort to collect product images and annotate them with multiple labels. However, the labels of product images represent the concept of subjective perception, which makes the multi-label learning more challenging to imitate the human aesthetic rather than discriminate the appearance. In this paper, a Feature Correlation Learning (FCL) network is proposed to solve this problem by exploiting the potential feature correlations of product images. Given a product image, the FCL network calculates the features of different levels and their correlations via gram matrices. The FCL is aggregated with the DenseNet to predict the labels of the input product image. The proposed method is compared with several outstanding multi-label learning methods, as well as DenseNet. Experimental results demonstrate that the proposed method outperforms the state-of-the-arts for multi-label learning problem of product image data.

ATTENTION BASED CONVOLUTIONAL NEURAL NETWORK FOR BUILDING EXTRACTION FROM VERY HIGH RESOLUTION REMOTE SENSING IMAGE

Abstract. Buildings are a major element in the formation of cities and are essential for urban mapping. The precise extraction of buildings from remote sensing data has become a significant topic and has received much attention in recent years. The recently developed convolutional neural networks have shown effective and superior performance to perform well on learning high-level and discriminative features in extracting buildings because of the outstanding feature learning and end-to-end pixel labelling abilities. However, it is difficult to use the features of different levels with a certain degree of importance that is appropriate to deep learning networks. To tackle this problem, a network based on U-Nets and the attention mechanism block was proposed. The network contains an encoder part and a decoder part and a spatial attention module. The special architecture presented in this article enhances the propagation of features and effectively utilizes the features at various levels to reduce errors. The other remarkable thing is that attention module blocks only lead to a minimal increase in model complexity. We effectively demonstrate an improvement of building extraction accuracy on challenging Potsdam and Vaihingen benchmark datasets. The results of this paper show that the proposed architecture improves building extraction in very high resolution remote sensing images compared to previous models.

ML–ResNet: A novel network to detect and locate myocardial infarction using 12 leads ECG

Background and objectiveMyocardial infarction (MI) is one of the most threatening cardiovascular diseases for human beings, which can be diagnosed by electrocardiogram (ECG). Automated detection methods based on ECG focus on extracting handcrafted features. However, limited by the performance of traditional methods and individual differences between patients, it's difficult for predesigned features to detect MI with high performance. MethodsThe paper presents a novel method to detect and locate MI combining a multi-lead residual neural network (ML–ResNet) structure with three residual blocks and feature fusion via 12 leads ECG records. Specifically, single lead feature branch network is trained to automatically learn the representative features of different levels between different layers, which exploits local characteristics of ECG to characterize the spatial information representation. Then all the lead features are fused together as global features. To evaluate the generalization of proposed method and clinical utility, two schemes including the intra-patient scheme and inter-patient scheme are all employed. ResultsExperimental results based on PTB (Physikalisch-Technische Bundesanstalt) database shows that our model achieves superior results with the accuracy of 95.49%, the sensitivity of 94.85%, the specificity of 97.37%, and the F1 score of 96.92% for MI detection under the inter-patient scheme compared to the state-of-the-art. By contrast, the accuracy is 99.92% and the F1 score is 99.94% based on 5-fold cross validation under the intra-patient scheme. As for five types of MI location, the proposed method also yields an average accuracy of 99.72% and F1 of 99.67% in the intra-patient scheme. ConclusionsThe proposed method for MI detection and location has achieved superior results compared to other detection methods. However, further promotion of the performance based on MI location for the inter-patient scheme still depends significantly on the mass data and the novel model which reflects spatial location information of different leads subtly.

Multifractal Spectrum Identification of Partial Discharge in GIS Based on Optical Detection

Partial discharge (PD) pattern recognition is an effective means to accurately obtain the correspondence between PD features and insulation defects. Optical detection for PD has strong anti-interference ability. Therefore, this paper carried out research on the pattern recognition of PD optical signal. In this paper, the PD optical signal under four typical defects of gas-insulated switchgears (GIS) are obtained by fluorescent fiber sensing system, and the four kinds of grayscale images are established. The multifractal spectrum features of the grayscale images of the optical signals are extracted. By analysing the physical meaning of each feature information in the multifractal spectrum, the feature components for PD pattern recognition are extracted, and the improved conjugate gradient back propagation (BP) neural network algorithm is selected as the classifier. This paper show that the multifractal spectrum can effectively describe the degree of unevenness of PD grayscale images and the geometric features of different level. The correct recognition rate of all kinds of defects by the multifractal spectrum is more than 87%, which is better than the identification of the box dimension and information dimension as the feature components. It lays the foundation for the future application of the optical detection.

Contextualized CNN for Scene-Aware Depth Estimation From Single RGB Image

Directly benefited from deep learning techniques, depth estimation from single image has gained great momentum in recent years. However, most of the existing approaches treat depth prediction as an isolated problem without taking into consideration high-level semantic context information, which results in inefficient utilization of training dataset and unavoidably requires a large number of captured depth data during the training phase. To ameliorate, this paper develops a novel scene-aware contextualized convolution neural network (CCNN), which characterizes the semantic context relationship at the class-level and refines depth at the pixel-level. Our newly-proposed CCNN is built upon the intrinsic exploitation of context-dependent depth association, including inner-object continuous depth and inter-object depth change priors nearby. Specifically, rather than conducting regression on depth in single CNN, we make the first attempt to integrate both class-level and pixel-level conditional random fields (CRFs) based probabilistic graphical model into the powerful CNN framework to simultaneously learn different-level features within the same CNN layer. With our CCNN, the former model will guide the latter one to learn the contextualized RGB-Depth mapping. Hence, CCNN has desirable properties in both class-level integrity and pixel-level discrimination, which makes it ideal to share such two-level convolutional features in parallel during the end-to-end training with the commonly-used back-propagation algorithm. We conduct extensive experiments and comprehensive evaluations on public benchmarks involving various indoor and outdoor scenes, and all the experiments confirm that, our method outperforms the state-of-the-art depth estimation methods, especially for the cases where only small-scale training data are readily available.

GoogLeNet-Based Ensemble FCNet Classifier for Focal Liver Lesion Diagnosis.

Transfer learning techniques are recently preferred for the computer aided diagnosis (CAD) of variety of diseases, as it makes the classification feasible from limited training dataset. In this work, an ensemble FCNet classifier is proposed to classify hepatic lesions from the deep features extracted using GoogleNet-LReLU transfer learning approachs. In the existing GoogLeNet architecture three modifications are done: ReLU activation functions in the inception modules are replaced by leaky ReLU activation function; a stack of three fully connected layers are included before the classification layer; and deep features of different level of abstraction extracted from the output of every inception layer given as classifier input in order to significantly enhance the classifier performance. The performance of the proposed classifier by the virtue of the above mentioned modifications is tested on six classes of liver CT images namely normal, hepatocellular carcinoma, hemangioma, cyst, abscess and liver metastasis. The results presented in this work demonstrate the efficacy of the proposed classifier design in achieving better classification accuracy.

Extracting Crop Spatial Distribution from Gaofen 2 Imagery Using a Convolutional Neural Network

Using satellite remote sensing has become a mainstream approach for extracting crop spatial distribution. Making edges finer is a challenge, while simultaneously extracting crop spatial distribution information from high-resolution remote sensing images using a convolutional neural network (CNN). Based on the characteristics of the crop area in the Gaofen 2 (GF-2) images, this paper proposes an improved CNN to extract fine crop areas. The CNN comprises a feature extractor and a classifier. The feature extractor employs a spectral feature extraction unit to generate spectral features, and five coding-decoding-pair units to generate five level features. A linear model is used to fuse features of different levels, and the fusion results are up-sampled to obtain a feature map consistent with the structure of the input image. This feature map is used by the classifier to perform pixel-by-pixel classification. In this study, the SegNet and RefineNet models and 21 GF-2 images of Feicheng County, Shandong Province, China, were chosen for comparison experiment. Our approach had an accuracy of 93.26%, which is higher than those of the existing SegNet (78.12%) and RefineNet (86.54%) models. This demonstrates the superiority of the proposed method in extracting crop spatial distribution information from GF-2 remote sensing images.

Finer Resolution Mapping of Marine Aquaculture Areas Using WorldView-2 Imagery and a Hierarchical Cascade Convolutional Neural Network

Marine aquaculture plays an important role in seafood supplement, economic development, and coastal ecosystem service provision. The precise delineation of marine aquaculture areas from high spatial resolution (HSR) imagery is vital for the sustainable development and management of coastal marine resources. However, various sizes and detailed structures of marine objects make it difficult for accurate mapping from HSR images by using conventional methods. Therefore, this study attempts to extract marine aquaculture areas by using an automatic labeling method based on the convolutional neural network (CNN), i.e., an end-to-end hierarchical cascade network (HCNet). Specifically, for marine objects of various sizes, we propose to improve the classification performance by utilizing multi-scale contextual information. Technically, based on the output of a CNN encoder, we employ atrous convolutions to capture multi-scale contextual information and aggregate them in a hierarchical cascade way. Meanwhile, for marine objects with detailed structures, we propose to refine the detailed information gradually by using a series of long-span connections with fine resolution features from the shallow layers. In addition, to decrease the semantic gaps between features in different levels, we propose to refine the feature space (i.e., channel and spatial dimensions) using an attention-based module. Experimental results show that our proposed HCNet can effectively identify and distinguish different kinds of marine aquaculture, with 98% of overall accuracy. It also achieves better classification performance compared with object-based support vector machine and state-of-the-art CNN-based methods, such as FCN-32s, U-Net, and DeeplabV2. Our developed method lays a solid foundation for the intelligent monitoring and management of coastal marine resources.

Facial Expression Recognition Based on a Hybrid Model Combining Deep and Shallow Features

Facial expression recognition plays an important role in the field involving human-computer interactions. Given the wide use of convolutional neural networks or other neural network models in automatic image classification systems, high-level features can be automatically learned by hierarchical neural networks. However, the training of CNNs requires large amounts of training data to permit adequate generalization. The traditional scale-invariant feature transform (SIFT) does not need large learning samples to obtain features. In this paper, we proposed a feature extraction method for use in the facial expressions recognition from a single image frame. The hybrid features use a combination of SIFT and deep learning features of different levels extracted from a CNN model. The combined features are adopted to classify expressions using support vector machines. The performance of proposed method is tested using the publicly available extended Cohn-Kanade (CK+) database. To evaluate the generalization ability of our method, several experiments are designed and carried out in a cross-database environment. Compared with the 76.57% accuracy obtained using SIFT-bag of features (BoF) features and the 92.87% accuracy obtained using CNN features, we achieve a FER accuracy of 94.82% using the proposed hybrid SIFT-CNN features. The results of additional cross-database experiments also demonstrate the considerable potential of combining shallow features with deep learning features, and these results are more promising than state-of-the-art models. Combining shallow and deep learning features is effective when the training data are not sufficient to obtain a deep model with considerable generalization ability.

DRFN: Deep Recurrent Fusion Network for Single-Image Super-Resolution With Large Factors

Recently, single-image super-resolution has made great progress owing to the development of deep convolutional neural networks (CNNs). The vast majority of CNN-based models use a pre-defined upsampling operator, such as bicubic interpolation, to upscale input low-resolution images to the desired size and learn non-linear mapping between the interpolated image and ground truth high-resolution (HR) image. However, interpolation processing can lead to visual artifacts as details are over-smoothed, particularly when the super-resolution factor is high. In this paper, we propose a Deep Recurrent Fusion Network (DRFN), which utilizes transposed convolution instead of bicubic interpolation for upsampling and integrates different-level features extracted from recurrent residual blocks to reconstruct the final HR images. We adopt a deep recurrence learning strategy and thus have a larger receptive field, which is conducive to reconstructing an image more accurately. Furthermore, we show that the multi-level fusion structure is suitable for dealing with image super-resolution problems. Extensive benchmark evaluations demonstrate that the proposed DRFN performs better than most current deep learning methods in terms of accuracy and visual effects, especially for large-scale images, while using fewer parameters.

Separable-spectral convolution and inception network for hyperspectral image super-resolution

Due to the limitation of the imaging system, it is hard to get Hyperspectral Image (HSI) with very high spatial resolution. Super-Resolution (SR) is a handling missing data technology to restore high-frequency information from the low-resolution image, can be used to solve this problem. Recently, Deep Learning (DL) has achieved great performance in computer vision, including SR. However, most DL-based HSI SR methods neglect the spectral disorder caused by normal 2D convolution. This paper proposes a novel end–end deep learning-based network named Separable-Spectral and Inception Network (SSIN) for HSI SR. In SSIN, the feature extraction module independently extracts features of each band image, and then these features are fused together to further exploit residual image by using feature fusion module. In reconstruction module, a multi-path connection is built to obtain features of different levels to restore high spatial resolution image in a coarse-to-fine manner. Experiments are implemented on two datasets include both indoor and airborne HSIs, and the performances of SSIN are evaluated in different conditions. Experimental results show that adding several separable spectral convolutions and multi-path connection in a deep network can greatly improve the SR performance, and SSIN achieves higher accuracy and better visualization compare with other methods.

Hierarchical Multimodal Fusion for Ground-Based Cloud Classification in Weather Station Networks

Recently, the multimodal information is taken into consideration for ground-based cloud classification in weather station networks, but intrinsic correlations between the multimodal information and the visual information cannot be mined sufficiently. We propose a novel approach called hierarchical multimodal fusion (HMF) for ground-based cloud classification in weather station networks, which fuses the deep multimodal features and the deep visual features in different levels, i.e., low-level fusion and high-level fusion. The low-level fusion directly fuses the heterogeneous features, which focuses on the modality-specific fusion. The high-level fusion integrates the output of low-level fusion with deep visual features and deep multimodal features, which could learn complex correlations among them owing to the deep fusion structure. We employ one loss function to train the overall framework of the HMF so as to improve the discrimination of cloud representations. The experimental results on the MGCD dataset indicate that our method outperforms other methods, which verifies the effectiveness of the HMF in ground-based cloud classification.

Multi-layer attention for person re-identification

Person re-identification has been a significant application in the field of video surveillance analysis, yet it remains a challenging work to recognize the person of interest across disjoint cameras of different viewpoints. The factors affecting the identification results include the variation in background, different illumination conditions and the changes of human body poses. Existing person re-identification methods mainly focus on the feature extraction of the whole frame and metric learning functions. However, most of those algorithms treat different areas without distinction. It is worth emphasizing that different local regions make different contributions to image representaion, which exactly conforms to the attention mechanism. In this paper, we introduce a novel attention network which explores spatial attention in a convolutional neural network. Our algorithm learns the visual attention in multi-layer feature maps. The proposed model not only pays attention to the spatial probabilities of local regions, but also takes the features in different levels into consideration. We evaluate this multi-layer spatial attention model on three benchmark person re-identification datasets: Market-1501, CUHK03, and DukeMTMC-reID. The experiment results validate the advances of our adopted network by comparing with state-of-the-art baselines.

Adversarially Regularized U-Net-based GANs for Facial Attribute Modification and Generation

Modifying and generating facial images with desired attributes are two important and highly related tasks in the field of computer vision. Some current methods can take advantage of their relationship and use a unified model to handle them simultaneously. However, producing high visual quality images on both tasks is still a challenge. To tackle this issue, we propose a novel model called adversarially regularized U-net (ARU-net)-based generative adversarial networks (ARU-GANs). The ARU-net is the major part of the ARU-GAN and is inspired by the design principle of U-net. It uses skip connections to pass different-level features from encoder to decoder, which preserves sufficient attribute-independent details for the modification task. Besides, this U-net-like architecture employs an adversarial regularization term to guide the distribution of latent representation to match the prior distribution, which guarantees to generate meaningful faces from this prior. We also propose a joint training technique for the ARU-GAN, which enables the facial attribute modification and generation tasks to learn together during training. We perform experiments on celebfaces attributes (CelebA) dataset and make visual analysis and quantitative evaluation on both tasks, which demonstrates that our model can successfully produce high visual quality facial images. Also, the results show that learning two tasks jointly can lead to performance improvement compared with learning them individually. At last, we further validate the effectiveness of our method by making an ablation study and experimenting on another dataset.

Deep gated attention networks for large-scale street-level scene segmentation

Street-level scene segmentation aims to label each pixel of street-view images into specific semantic categories. It has been attracting growing interest due to various real-world applications, especially in the area of autonomous driving. However, this pixel-wise labeling task is very challenging under the complex street-level scenes and large-scale object categories. Motivated by the scene layout of street-view images, in this work we propose a novel Spatial Gated Attention (SGA) module, which automatically highlights the attentive regions for pixel-wise labeling, resulting in effective street-level scene segmentation. The proposed module takes as input the multi-scale feature maps based on a Fully Convolutional Network (FCN) backbone, and produces the corresponding attention mask for each feature map. The learned attention masks can neatly highlight the regions of interest while suppress background clutter. Furthermore, we propose an efficient multi-scale feature interaction mechanism which is able to adaptively aggregate the hierarchical features. Based on the proposed mechanism, the features of different levels are adaptively re-weighted according to the local spatial structure and the surrounding contextual information. Consequently, the proposed modules are able to boost standard FCN architectures and result in an enhanced pixel-wise segmentation for street-level scene images. Extensive experiments on three public available street-level benchmarks demonstrate that the proposed Gated Attention Network (GANet) approach achieves consistently superior performance and outperforms the very recent state-of-the-art methods.

Improved Convolutional Neural Networks for Acoustic Event Classification

To further exploit the potential performance of convolutional neural networks in acoustic event classification, an improved convolutional neural network called AecNet (Acoustic event classification net) is proposed. For traditional convolutional neural network lacks the representation of low-level features, the proposed model includes more feature layers to reserve the information of low-level and high-level features of the input. In order to extract the features of different level effectively, 1 × 1 convolutions are adopted to compress the feature maps of all convolutional layers except the top convolutional layer. Then the condensed features are concatenated into one layer, which contains all features in different levels. So, the feature learning is enhanced and multi-scale convolutional neural network is constructed. In order to extract the dynamic features of the sound clip better, multi-channels spectrogram features comprised of mel-spectrogram, its first order delta along frequency and time, second order delta along frequency and time are adopted. In experiment, point of FFT, number of mel-bands and type of mel-spectrogram deltas are detailedly discussed and reasonable choice are suggested in practice. Experiments results on datasets ESC-10, ESC-50 and DCASE show that the proposed method yields improvements of recognition accuracy in various degrees compared with some state-of-art results on standard benchmark.

Identification of fact-implied implicit sentiment based on multi-level semantic fused representation

Sentiment can be expressed in an explicit or implicit manner. Most of the current studies on sentiment analysis focus on the identification of explicit sentiment but ignore the implicit. According to our statistics during data labeling in previous work, nearly a third of subjective sentences contain implicit sentiment, and 72% of the implicit sentiment sentences are fact-implied ones. We analyze the characteristics of the sentences that express fact-implied implicit sentiment and consider that fact-implied implicit sentiment is usually affected by its sentiment target, context semantic background and its own sentence structure. This paper focuses on the recognition of fact-implied implicit sentiment at the sentence level. A multi-level semantic fusion method is proposed to learn the features for identification based on representation learning. Three features in different levels are learned from the corpus, namely, sentiment target representation at the word level, structure embedded representation at the sentence level and context semantic background representation at the document level. We manually construct a fact-implied implicit sentiment corpus in Chinese, and experiments on the datasets show that the proposed method can effectively recognize fact-implied implicit sentiment sentences.

Multi attention module for visual tracking

We propose a new visual tracking algorithm leveraging multi-level visual attention to take full use of the information during tracking. Visual attention has been widely applied in many visual tasks, such as image captioning and question answering. However, most existing attention models only focus on one or two aspects, ignoring the other useful information in visual tracking. Here, we think there are four main attentional aspects in the tracking task and propose a unified network to leverage multi-level visual attention, which includes layer-wise attention, temporal attention, spatial attention and channel-wise attention. Considering that deep features of different levels may be suitable for different scenarios, we propose to train an attention network in the off-line stage to facilitate feature selection in online tracking. To better exploit the temporal consistency assumption of visual tracking, we implement the attention network with long short term memory (LSTM) units, which are capable of capturing the historical context information to perform more reliable inference at the current time step. Different from the image classification task, background clutter is more complicated in the tracking task. Thus, we purify the features by spatial attention and channel-wise attention to effectively suppress the background noise and highlight the target region. In addition, we also enforce deep feature sharing across target candidates using Region of Interest pooling, allowing the features of all candidates to be extracted in only one forward pass of the DNN. To further improve tracking accuracy, a promoting strategy for trackers with detection results of a generic object detector is proposed, reducing the risk of tracking drifts. The proposed tracking algorithm compares favorably against state-of-the-art methods on three popular benchmark datasets. Extensive experimental evaluations demonstrate the effectiveness of the proposed techniques.

Coordinate CNNs and LSTMs to categorize scene images with multi-views and multi-levels of abstraction

Due to complexities of scene images, scene categorization is a challenging task in the computer vision community. To categorize scene images effectively, in this paper, we propose to coordinate Convolutional Neural Networks (CNNs) and Long Short-Term Memory networks (LSTMs) to perform scene categorization with multi-views and multi-levels of abstraction. Specifically, to utilize the complementary properties of features of different levels of abstraction, we employ CNNs to extract features of multi-levels of abstraction based on its hierarchical structure. Furthermore, in order to deal with variations in scene image contents, we represent each image with multiple views, and in order to take correlation between image views into consideration, we treat image view features from the same image as a sequence and employ Long Short-Term Memory networks (LSTMs) to perform classification. Based on the proposed method, information of multi-views and multi-levels of abstraction can be made full use of in a single framework. We evaluate the proposed method on two challenging scene datasets, MIT indoor scene 67 and SUN 397. Obtained results demonstrate the effectiveness of utilizing CNNs and LSTMs to categorize scene images with multi-views and multi-levels of abstraction. Experiments on comparison to state-of-the-art methods show that the proposed method outperforms all the other methods used for comparison.