Discriminative Deep Features Research Articles

Zero-shot Fine-grained Classification by Deep Feature Learning with Semantics

Fine-grained image classification, which aims to distinguish images with subtle distinctions, is a challenging task for two main reasons: lack of sufficient training data for every class and difficulty in learning discriminative features for representation. In this paper, to address the two issues, we propose a two-phase framework for recognizing images from unseen fine-grained classes, i.e., zero-shot fine-grained classification. In the first feature learning phase, we finetune deep convolutional neural networks using hierarchical semantic structure among fine-grained classes to extract discriminative deep visual features. Meanwhile, a domain adaptation structure is induced into deep convolutional neural networks to avoid domain shift from training data to test data. In the second label inference phase, a semantic directed graph is constructed over attributes of fine-grained classes. Based on this graph, we develop a label propagation algorithm to infer the labels of images in the unseen classes. Experimental results on two benchmark datasets demonstrate that our model outperforms the state-of-the-art zero-shot learning models. In addition, the features obtained by our feature learning model also yield significant gains when they are used by other zero-shot learning models, which shows the flexility of our model in zero-shot fine-grained classification.

A Multi-Feature Fusion Based on Transfer Learning for Chicken Embryo Eggs Classification

The fertility detection of Specific Pathogen Free (SPF) chicken embryo eggs in vaccine preparation is a challenging task due to the high similarity among six kinds of hatching embryos (weak, hemolytic, crack, infected, infertile, and fertile). This paper firstly analyzes two classification difficulties of feature similarity with subtle variations on six kinds of five- to seven-day embryos, and proposes a novel multi-feature fusion based on Deep Convolutional Neural Network (DCNN) architecture in a small dataset. To avoid overfitting, data augmentation is employed to generate enough training images after the Region of Interest (ROI) of original images are cropped. Then, all the augmented ROI images are fed into pretrained AlexNet and GoogLeNet to learn the discriminative deep features by transfer learning, respectively. After the local features of Speeded Up Robust Feature (SURF) and Histogram of Oriented Gradient (HOG) are extracted, the multi-feature fusion with deep features and local features is implemented. Finally, the Support Vector Machine (SVM) is trained with the fused features. The verified experiments show that this proposed method achieves an average classification accuracy rate of 98.4%, and that the proposed transfer learning has superior generalization and better classification performance for small-scale agricultural image samples.

Myocardium Detection by Deep SSAE Feature and Within-Class Neighborhood Preserved Support Vector Classifier and Regressor.

Automatic detection of left ventricle myocardium is essential to subsequent cardiac image registration and tissue segmentation. However, it is considered challenging mainly because of the complex and varying shape of the myocardium and surrounding tissues across slices and phases. In this study, a hybrid model is proposed to detect myocardium in cardiac magnetic resonance (MR) images combining region proposal and deep feature classification and regression. The model firstly generates candidate regions using new structural similarity-enhanced supervoxel over-segmentation plus hierarchical clustering. Then it adopts a deep stacked sparse autoencoder (SSAE) network to learn the discriminative deep feature to represent the regions. Finally, the features are fed to train a novel nonlinear within-class neighborhood preserved soft margin support vector (C-SVC) classifier and multiple-output support vector (-SVR) regressor for refining the location of myocardium. To improve the stability and generalization, the model also takes hard negative sample mining strategy to fine-tune the SSAE and the classifier. The proposed model with impacts of different components were extensively evaluated and compared to related methods on public cardiac data set. Experimental results verified the effectiveness of proposed integrated components, and demonstrated that it was robust in myocardium localization and outperformed the state-of-the-art methods in terms of typical metrics. This study would be beneficial in some cardiac image processing such as region-of-interest cropping and left ventricle volume measurement.

Practical Device-Free Gesture Recognition Using WiFi Signals Based on Metalearning

Device-free gesture recognition (DFGR) is a promising sensing technique, which can recognize a gesture by analyzing its influence on surrounding wireless signals. Most of the DFGR systems are designed based on machine learning. However, the recognition performance will drop dramatically when the testing condition is different with the training one. Inspired by the transferrable knowledge learning ability of humans, this paper develops a practical DFGR system based on metalearning to solve the aforementioned problem. Specifically, we design a deep network which could not only learn discriminative deep features, but also learn a transferrable similarity evaluation ability from the training set and apply the learned knowledge to the new testing conditions. Extensive experiments conducted by four users in two scenarios demonstrate that the proposed system could recognize new types of gestures, or gestures performed in new conditions, with an accuracy of more than 90%, using very few number of new samples.

DeepStar: Detecting Starring Characters in Movies

Recent advances in the film industry have given rise to exponential growth in movie/drama production and adaptation of the Big Data concept. Automatic identification and classification of movie characters have received tremendous attention from researchers due to its applications in video semantic analysis, video summarization, and personalized video retrieval for which several methods have been recently presented. However, these methods cannot detect main characters properly due to their variation in pose and style in different scenes of a movie. To address this problem we present DeepStar, a novel framework for starring character identification based on deep high-level robust features. The proposed framework is threefold: the extraction of shots with clear faces from the input video; face clustering using discriminative deep features; and the occurrence matrix generation, helping in the selection of starring characters. The promising results obtained using representative Hollywood movies demonstrate the effectiveness of our method in detecting starring characters over the state-of-the-art methods.

A jointly learned deep embedding for person re-identification

Person re-identification (ReID) is an importance study issue in the modern video surveillance area. However, it is very challenging due to the large variations of intra-class and the small variations of inter-class. To solve the problem, most deep ReID methods usually learn a feature embedding based on verification or identification. However, both learning approaches are complementary and neither of them can well address this issue. Motivated by this, we propose a deep joint learning framework based on verification-identification to achieve a discriminative deep feature embedding. Specifically, for verification, a triplet network is adopted and an improved triplet loss is proposed by adding a novel penalty item in the framework. For identification, an identity classifier with the softmax loss is attached to the top of the triplet network at the same time. Thus, driven by the improved triplet loss and the softmax loss simultaneously, a more discriminative and compact feature embedding will be learned for ReID. Extensive experiments over several popular benchmarks achieve state-of-the-art results which well demonstrate the effectiveness of the proposed method for ReID.

Geometric discriminative deep features for traffic image analysis

Traffic image analysis is an important application in intelligent transportation. For local features’ robustness to image variances, such as scale changes and occlusions, they are widely used in image classification. However, how to integrate these local features for modeling traffic images optimally is still a crucial challenge. In this paper, a novel deep learning method, geometric discriminative feature fusion (GDFF), is proposed to tackle this problem. First, we use a variety of data sets to train the general convolutional neural network (CNN), which is used to extract the features of the training and test set after deep level. Deep architecture makes it possible for people to learn more abstract and internal features that are robust to changes in viewpoint and illumination. It can fuse image geometric related local features, such as local regions’ RGB histograms, into high level discriminative features, which can be used for better classifying complex scene images. Our framework’s central task is to build a structural kernel, called discriminative topological kernel. Firstly, we segment the traffic images into several regions and use a region connected graph (RCG) to model regions location relationships. We use frequent sub graph mining algorithm to mine all frequent sub structures (topologies) occurs in all training RCGs. And a selection algorithm is designed to select the k qualified topologies from the entire mined frequent topologies. We call these selected topologies geometric feature fusers, which are both high discriminative and low redundant structures in all training RCGs. Finally, given a pair of RCGs and to each geometric fuser, we extract all pairs of sub graphs sharing the same topology and calculate distance between them. All k distances are accumulated for the final kernel. The experimental result demonstrates the effectiveness of our method.

Learning head pose-insensitive and discriminative deep features for smile detection

Smile detection plays an important role in human emotion analysis and has wide applications. However, there is still a gap between the performance of the current smile detection algorithms and real-world applications, due to variations of head pose and environment noise. We propose a robust framework based on convolutional neural networks (CNNs) for smile detection. To alleviate the influence of head pose variations and improve performance, the proposed framework customizes two-feature learning layers such as (1) smile feature extraction layer is constructed by hidden factor analysis for learning head pose-insensitive smile features; (2) smile feature discrimination layer is constructed by marginal Fisher analysis, and it is used to learn discriminative features for further enhancing the discrimination between smile and nonsmile. The two layers both work as fully connected layers, and they are connected layer by layer to a backbone CNN network. Experiments have been performed on two publicly available datasets, and the results show that the proposed framework delivers promising performance (95.45% on GENKI4K and 93.62% on labeled faces in the wild attribute) and outperforms the state of the art.

Cross-View Discriminative Feature Learning for Person Re-Identification

The viewpoint variability across a network of non-overlapping cameras is a challenging problem affecting person re-identification performance. In this paper, we investigate how to mitigate the cross-view ambiguity by learning highly discriminative deep features under the supervision of a novel loss function. The proposed objective is made up of two terms, the steering meta center term and the enhancing centers dispersion term, that steer the training process to mining effective intra-class and inter-class relationships in the feature domain of the identities. The effect of our loss supervision is to generate a more expanded feature space of compact classes where the overall level of the inter-identities' interference is reduced. Compared with the existing metric learning techniques, this approach has the advantage of achieving a better optimization because it jointly learns the embedding and the metric contextually. Our technique, by dismissing side-sources of performance gain, proves to enhance the CNN invariance to viewpoint without incurring increased training complexity (like in Siamese or triplet networks) and outperforms many related state-of-the-art techniques on Market-1501 and CUHK03.

Embedding Visual Hierarchy with Deep Networks for Large-Scale Visual Recognition.

In this paper, a layer-wise mixture model (LMM) is developed to support hierarchical visual recognition, where a Bayesian approach is used to automatically adapt the visual hierarchy to the progressive improvements of the deep network along the time. Our LMM algorithm can provide an end-to-end approach for jointly learning: (a) the deep network for achieving more discriminative deep representations for object classes and their inter-class visual similarities; (b) the tree classifier for recognizing large numbers of object classes hierarchically; and (c) the visual hierarchy adaptation for achieving more accurate assignment and organization of large numbers of object classes. By learning the tree classifier, the deep network and the visual hierarchy adaptation jointly in an end-to-end manner, our LMM algorithm can achieve higher accuracy rates on hierarchical visual recognition. Our experiments are carried on ImageNet1K and ImageNet10K image sets, which have demonstrated that our LMM algorithm can achieve very competitive results on the accuracy rates as compared with the baseline methods.

Deep feature learning via structured graph Laplacian embedding for person re-identification

Learning the distance metric between pairs of examples is of great importance for visual recognition, especially for person re-identification (Re-Id). Recently, the contrastive and triplet loss are proposed to enhance the discriminative power of the deeply learned features, and have achieved remarkable success. As can be seen, either the contrastive or triplet loss is just one special case of the Euclidean distance relationships among these training samples. Therefore, we propose a structured graph Laplacian embedding algorithm, which can formulate all these structured distance relationships into the graph Laplacian form. The proposed method can take full advantages of the structured distance relationships among these training samples, with the constructed complete graph. Besides, this formulation makes our method easy-to-implement and super-effective. When embedding the proposed algorithm with the softmax loss for the CNN training, our method can obtain much more robust and discriminative deep features with inter-personal dispersion and intra-personal compactness, which is essential to person Re-Id. We did experiments on top of three popular networks, namely AlexNet [1], DGDNet [2] and ResNet50 [3], on recent four widely used Re-Id benchmark datasets, and it shows that the proposed structure graph Laplacian embedding is very effective.

Leveraging Content Sensitiveness and User Trustworthiness to Recommend Fine-Grained Privacy Settings for Social Image Sharing

To configure successful privacy settings for social image sharing, two issues are inseparable: 1) content sensitiveness of the images being shared; and 2) trustworthiness of the users being granted to see the images. This paper aims to consider these two inseparable issues simultaneously to recommend fine-grained privacy settings for social image sharing. For achieving more compact representation of image content sensitiveness (privacy), two approaches are developed: 1) a deep network is adapted to extract 1024-D discriminative deep features; and 2) a deep multiple instance learning algorithm is adopted to identify 280 privacy-sensitive object classes and events. Second, users on the social network are clustered into a set of representative social groups to generate a discriminative dictionary for user trustworthiness characterization. Finally, both the image content sensitiveness and the user trustworthiness are integrated to train a tree classifier to recommend fine-grained privacy settings for social image sharing. Our experimental studies have demonstrated both the efficiency and the effectiveness of our proposed algorithms.

Deep Salient Feature Based Anti-Noise Transfer Network for Scene Classification of Remote Sensing Imagery

Remote sensing (RS) scene classification is important for RS imagery semantic interpretation. Although tremendous strides have been made in RS scene classification, one of the remaining open challenges is recognizing RS scenes in low quality variance (e.g., various scales and noises). This paper proposes a deep salient feature based anti-noise transfer network (DSFATN) method that effectively enhances and explores the high-level features for RS scene classification in different scales and noise conditions. In DSFATN, a novel discriminative deep salient feature (DSF) is introduced by saliency-guided DSF extraction, which conducts a patch-based visual saliency (PBVS) algorithm using “visual attention” mechanisms to guide pre-trained CNNs for producing the discriminative high-level features. Then, an anti-noise network is proposed to learn and enhance the robust and anti-noise structure information of RS scene by directly propagating the label information to fully-connected layers. A joint loss is used to minimize the anti-noise network by integrating anti-noise constraint and a softmax classification loss. The proposed network architecture can be easily trained with a limited amount of training data. The experiments conducted on three different scale RS scene datasets show that the DSFATN method has achieved excellent performance and great robustness in different scales and noise conditions. It obtains classification accuracy of 98.25%, 98.46%, and 98.80%, respectively, on the UC Merced Land Use Dataset (UCM), the Google image dataset of SIRI-WHU, and the SAT-6 dataset, advancing the state-of-the-art substantially.

Multiscale deep features learning for land-use scene recognition

The features extracted from deep convolutional neural networks (CNNs) have shown their promise as generic descriptors for land-use scene recognition. However, most of the work directly adopts the deep features for the classification of remote sensing images, and does not encode the deep features for improving their discriminative power, which can affect the performance of deep feature representations. To address this issue, we propose an effective framework, LASC-CNN, obtained by locality-constrained affine subspace coding (LASC) pooling of a CNN filter bank. LASC-CNN obtains more discriminative deep features than directly extracted from CNNs. Furthermore, LASC-CNN builds on the top convolutional layers of CNNs, which can incorporate multiscale information and regions of arbitrary resolution and sizes. Our experiments have been conducted using two widely used remote sensing image databases, and the results show that the proposed method significantly improves the performance when compared to other state-of-the-art methods.

Multiscale blur detection by learning discriminative deep features

Abstract Blur detection is an important and challenging task in computer vision. An essential problem in blur detection is how to choose effective features to distinguish blurred and non-blurred image regions. Unlike previous methods relying on various handcrafted features, we propose to learn discriminative blur features via deep convolutional neural networks (CNNs). We design a simple yet effective 6-layer CNN model, with 5 layers for feature extraction and 1 for binary classification, which can faithfully produce patch-level blur likelihood. We empirically show that, by going to deeper layers, our model is able to generate more effective features with refined discriminative power. We apply this network at three coarse-to-fine scales and optimally fuse multiscale blur likelihood maps to generate better blur detection. Extensive experiments on challenging benchmark dataset validate that our model can significantly improve the performance of blur detection. We also show that the accuracy of saliency detection can always be improved by treating regional blur likelihood as saliency prior.

Minutiae-Based Weighting Aggregation of Deep Convolutional Features for Vein Recognition

Deep convolutional neural network (DCNN) has achieved an outstanding performance in large-scale image recognition task because of its discriminative feature representation ability, and pre-trained DCNN models trained for one task have also been applied to domains that are different from their original purposes. Inspired by this idea, a novel hand-dorsa vein recognition model is constructed by adopting DCNN pre-trained on a large-scale database as a universal feature descriptor. However, due to the sparse distribution property of vein information, it is difficult to employ pre-trained DCNN model to extract discriminative deep convolutional features. Therefore, to obtain useful and discriminative deep convolutional features, a novel minutiae-based weighting aggregation (MWA) method is proposed. In specific, the proposed global max-pooling of preserving spatial position information is applied on the feature maps of convolutional layer to localize the minutiae of vein information, and then the minutiae feature of vein image is regarded as the mask that is named as minutiae feature mask, to select deep convolutional features that contain minutiae feature information of vein image. The final feature representation is formed by concatenating each selected deep convolutional feature that is generated by each minutiae feature mask. Series rigorous experiments on the lab-made database are conducted to evidence the effectiveness and feasibility of the proposed MWA for vein recognition. What’s more, an additional experiment with subset of PolyU database illustrates its generalization ability and robustness.

Discriminative deep feature learning method by fusing linear discriminant analysis for image recognition

Discriminative deep feature learning method by fusing linear discriminant analysis for image recognition

Discriminative Deep Feature Learning for Semantic-Based Image Retrieval

Semantic-based image retrieval plays an important role in many practical applications, which aims to look for images with similar contents. Extracting discriminative representations of images is the real crux of this task. Directly utilizing the results of fully connected layers of convolutional neural networks (CNNs) is one of the best feature extraction methods. However, the fully connected layer is only supervised by the softmax loss, which only aims to maximize the accuracy of object classification and hardly pays attention to the spatial distribution of features, especially the intra-class and inter-class feature distances which are of great importance in semantic-based image retrieval. To compensate the performance degradation due to this reason, we try to address the spatial distribution of the features by two different loss functions. The first loss function jointly combines the softmax loss and the center loss in the training of the CNNs, which simultaneously ensures that the features of images with different contents are separable and the features of images in the same class are close. The second loss function is defined as an improved center loss which not only penalizes the distance between the obtained deep feature and the feature center of its own class, but also encourages a far distance between the feature and the feature centers of any other classes. We have conducted experiments on both ILSVRC data set and Caltech256 data set, demonstrated that the deep features got by our approaches can achieve better performance than other methods in semantic-based image retrieval, and the improved center loss can further outperform the joint supervision of the softmax loss and the center loss.

Pedestrian search in surveillance videos by learning discriminative deep features

Abstract Searching for a target person in videos captured by many non-overlapped cameras is an important yet challenging problem in the fields of intelligent video surveillance. Person re-identification is a key technique in the person searching task. In this paper, we propose a discriminative objective function to learn deep CNN features for person re-id. Specially, the proposed objective function reduces distances between instances belonging to the same person, and enlarges distances between instances belonging to different persons at the same time. With the goal of inter-class dispersion and intra-class compactness, the obtained deep features can be more discriminative than many traditional training objectives, e.g. softmax, contrastive and triplet objective functions. Extensive experiments on person re-id benchmarks validated the superiority of the proposed objective function. Based on the proposed person re-identification algorithm, we implement a pedestrian search system by integrating three components: pedestrian detection, multi-person tracking and person re-identification together. The whole system is evaluated on a Cross-Camera Pedestrian Search Challenge and achieves superior performances on the evaluation set.

Encoding Spectral and Spatial Context Information for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is a popular yet challenging research topic in the remote sensing community. This letter attempts to encode both spectral and spatial information into deep features for HSI classification. We first propose a semisupervised method for training the stacked autoencoder to obtain discriminative deep features. A batch training scheme is introduced to constrain the label consistency on a neighborhood region. Second, a mean pooling procedure is suggested to further fuse the spectral and local spatial information for deep feature generation. The experimental results on two hyperspectral scenes show that the proposed method achieves promising classification performance.