Middle-level Features Research Articles

Cross-Modality Person Re-Identification via Modality-aware Collaborative Ensemble Learning.

Visible thermal person re-identification (VT-ReID) is a challenging cross-modality pedestrian retrieval problem due to the large intra-class variations and modality discrepancy across different cameras. Existing VT-ReID methods mainly focus on learning cross-modality sharable feature representations by handling the modality-discrepancy in feature level. However, the modality difference in classifier level has received much less attention, resulting in limited discriminability. In this paper, we propose a novel modality-aware collaborative ensemble (MACE) learning method with middle-level sharable two-stream network (MSTN) for VT-ReID, which handles the modality-discrepancy in both feature level and classifier level. In feature level, MSTN achieves much better performance than existing methods by capturing sharable discriminative middlelevel features in convolutional layers. In classifier level, we introduce both modality-specific and modality-sharable identity classifiers for two modalities to handle the modality discrepancy. To utilize the complementary information among different classifiers, we propose an ensemble learning scheme to incorporate the modality sharable classifier and the modality specific classifiers. In addition, we introduce a collaborative learning strategy, which regularizes modality-specific identity predictions and the ensemble outputs. Extensive experiments on two cross-modality datasets demonstrate that the proposed method outperforms current state-of-the-art by a large margin, achieving rank- 1/mAP accuracy 51.64%/50.11% on the SYSU-MM01 dataset, and 72.37%/69.09% on the RegDB dataset.

A new intelligent pattern classifier based on deep-thinking

A new intelligent pattern classifier based on the human being’s thinking logics is developed in this paper, aiming to approximate the optimal design process and avoid the matrix inverse computation in conventional classifier designs. It is seen that the proposed classifier has no parameters to be determined via mathematical optimization. Instead, it is built by using the correlation principles to construct the clusters at first. The middle-level feature vectors can then be extracted from the statistical information of the correlations between the input data and the ones in each pattern cluster. For accurate classification purpose, the advanced feature vectors are generated with the moments’ information of the middle-level feature vectors. After that, Bayesian inference is implemented to make decisions from the weighted sum of the advanced feature components. In addition, a real-time fine-tuning loop (layer) is designed to adaptively “widen” the border of each pattern clustering region such that the input data can be directly classified once they are located in one of the clustering regions. An experiment for the classification of the handwritten digit images from the MNIST database is performed to show the excellent performance and effectiveness of the proposed intelligent pattern classifier.

Convolutional neural network for spectral–spatial classification of hyperspectral images

Hyperspectral images (HSIs) have great potential in military reconnaissance, land use, marine monitoring and many other fields. In recent years, the convolutional neural network (CNN) has been successfully used to classify hyperspectral data and achieved remarkable performance. However, the limited labeled samples of HSI lead to the small sample size problem, which remains the major challenge for CNN-based HSI classification. Besides, most CNN models have large number of parameters needed to be learned, which cause high computational cost. To address the aforementioned two issues, a novel CNN-based HSI classification method is proposed. The proposed classification method has several distinguishing characteristics. First, the proposed method can robustly extract spectral and spatial features of the HSI simultaneously. Second, in the proposed CNN architecture, all convolution layers are 1 × 1 convolution layer except the first one, which can greatly reduce the number of network parameters, thus accelerating the training and testing process. Third, a small convolution and feature reuse (SC-FR) module is developed. The SC-FR module is composed of two composite layers and each composite layer consists of two cascaded 1 × 1 convolution layers. Through cross-layer connecting, the input and output features of each composite layer are concatenated and passed to the next convolution layer, thus achieving feature reuse mechanism. Cross-layer connection increases information flow and the utilization rate of middle-level features, which enhances the generalization performance of CNN effectively. Experimental results on three benchmark HSIs demonstrate the competitive superiority of the proposed method over several state-of-the-art HSI classification methods, especially when training samples are limited.

Human action recognition based on HOIRM feature fusion and AP clustering BOW

In this paper, we propose a human action recognition method using HOIRM (histogram of oriented interest region motion) feature fusion and a BOW (bag of words) model based on AP (affinity propagation) clustering. First, a HOIRM feature extraction method based on spatiotemporal interest points ROI is proposed. HOIRM can be regarded as a middle-level feature between local and global features. Then, HOIRM is fused with 3D HOG and 3D HOF local features using a cumulative histogram. The method further improves the robustness of local features to camera view angle and distance variations in complex scenes, which in turn improves the correct rate of action recognition. Finally, a BOW model based on AP clustering is proposed and applied to action classification. It obtains the appropriate visual dictionary capacity and achieves better clustering effect for the joint description of a variety of features. The experimental results demonstrate that by using the fused features with the proposed BOW model, the average recognition rate is 95.75% in the KTH database, and 88.25% in the UCF database, which are both higher than those by using only 3D HOG+3D HOF or HOIRM features. Moreover, the average recognition rate achieved by the proposed method in the two databases is higher than that obtained by other methods.

A Multiscale Deep Middle-level Feature Fusion Network for Hyperspectral Classification

Recently, networks consider spectral-spatial information in multiscale inputs less, even though there are some networks that consider this factor, however these networks cannot guarantee to get optimal features, which are extracted from each scale input. Furthermore, these networks do not consider the complementary and related information among different scale features. To address these issues, a multiscale deep middle-level feature fusion network (MMFN) is proposed in this paper for hyperspectral classification. In MMFN, the network fully fuses the strong complementary and related information among different scale features to extract more discriminative features. The training of network contains two stages: the first stage obtains the optimal models corresponding to different scale inputs and extracts the middle-level features under the corresponding scale model. It can guarantee the multiscale middle-level features are optimal. The second stage fuses the optimal multiscale middle-level features in the convolutional layer, and the subsequent residual blocks can learn the complementary and related information among different scale middle-level features. Moreover, the idea of identity mapping in residual learning can help the network obtain a higher accuracy when the network is deeper. The effectiveness of our method is proved on four HSI data sets and the experimental results show that our method outperforms the other state-of-the-art methods especially with small training samples.

Two-Way Affective Modeling for Hidden Movie Highlights' Extraction.

Movie highlights are composed of video segments that induce a steady increase of the audience’s excitement. Automatic movie highlights’ extraction plays an important role in content analysis, ranking, indexing, and trailer production. To address this challenging problem, previous work suggested a direct mapping from low-level features to high-level perceptual categories. However, they only considered the highlight as intense scenes, like fighting, shooting, and explosions. Many hidden highlights are ignored because their low-level features’ values are too low. Driven by cognitive psychology analysis, combined top-down and bottom-up processing is utilized to derive the proposed two-way excitement model. Under the criteria of global sensitivity and local abnormality, middle-level features are extracted in excitement modeling to bridge the gap between the feature space and the high-level perceptual space. To validate the proposed approach, a group of well-known movies covering several typical types is employed. Quantitative assessment using the determined excitement levels has indicated that the proposed method produces promising results in movie highlights’ extraction, even if the response in the low-level audio-visual feature space is low.

Unsupervised Scene Change Detection via Latent Dirichlet Allocation and Multivariate Alteration Detection

Scene change detection is the process of identifying the differences between the multitemporal image scenes, which has significant potential in the application of urban development and land management at the semantic level. Traditional scene change detection methods are based on the supervised scene classification, and then directly compare the independent classification results without considering the temporal correlation between the unchanged regions. However, few studies have focused on detecting the semantic changes of multitemporal image scenes with unsupervised methods. In this paper, we propose a novel unsupervised scene change detection method by using latent Dirichlet allocation (LDA) and multivariate alteration detection (MAD). First, the scene is represented by the bag-of-visual-words model, and adopt the union dictionary to ensure the consistency of dictionary space. Then, LDA is used to achieve the middle-level feature dimension reduction, and generate the common topic space of the two multitemporal image scene datasets. And finally, the MAD method was applied to detect the semantic changes of corresponding multitemporal image scenes. Two experiments with high-resolution remote sensing image scene datasets demonstrated that our proposed approach can get a better performance in unsupervised scene change detection without prior knowledge.

On the Behavior of Convolutional Nets for Feature Extraction

Deep neural networks are representation learning techniques. During training, a deep net is capable of generating a descriptive language of unprecedented size and detail in machine learning. Extracting the descriptive language coded within a trained CNN model (in the case of image data), and reusing it for other purposes is a field of interest, as it provides access to the visual descriptors previously learnt by the CNN after processing millions of images, without requiring an expensive training phase. Contributions to this field (commonly known as feature representation transfer or transfer learning) have been purely empirical so far, extracting all CNN features from a single layer close to the output and testing their performance by feeding them to a classifier. This approach has provided consistent results, although its relevance is limited to classification tasks. In a completely different approach, in this paper we statistically measure the discriminative power of every single feature found within a deep CNN, when used for characterizing every class of 11 datasets. We seek to provide new insights into the behavior of CNN features, particularly the ones from convolutional layers, as this can be relevant for their application to knowledge representation and reasoning. Our results confirm that low and middle level features may behave differently to high level features, but only under certain conditions. We find that all CNN features can be used for knowledge representation purposes both by their presence or by their absence, doubling the information a single CNN feature may provide. We also study how much noise these features may include, and propose a thresholding approach to discard most of it. All these insights have a direct application to the generation of CNN embedding spaces.

Research on improved algorithm of object detection based on feature pyramid

To solve the low detection accuracy of SSD for the small size object, this paper proposed an improved algorithm of SSD object detection based on the feature pyramid (FP-SSD). In the deep convolutional neural network, the high-level features contain well semantic information but are not sensitive to the translations. The low-level features have high resolutions but could not represent the features well. The feature pyramid structure contains multi-scale features. To combine the high and low-level features of the pyramid, the algorithm of this paper applied the deconvolution network to the high-level features of the feature pyramid to get the semantic information, dilated convolution network to learn the position information of the low-level features and used convolution for the middle level features to reduce the feature channels, then used convolution to fuse the features. After using the algorithm, a multi-scale detection structure is constructed. FP-SSD achieves a mean accuracy of 79% on PASCAL VOC2007, and 47% on MSCOCO, which has a great improve compared with SSD. We compared the detection accuracy and results with all kinds of scales by experiments, compared with SSD, the accuracy of FP-SSD is higher, which has more accurate location and higher recognition confidence.

Multifeature Hyperspectral Image Classification With Local and Nonlocal Spatial Information via Markov Random Field in Semantic Space

Hyperspectral images (HSIs) provide invaluable information in both spectral and spatial domains for image classification tasks. In this paper, we use semantic representation as a middle-level feature to describe image pixels’ characteristics. Deriving effective semantic representation is critical for achieving good classification performance. Since different image descriptors depict characteristics from different perspectives, combining multiple features in the same semantic space makes semantic representation more meaningful. First, a probabilistic support vector machine is used to generate semantic representation-based multifeatures. In order to derive better semantic representation, we introduce a new adaptive spatial regularizer that well exploits the local spatial information, while a nonlocal regularizer is also used to search for global patch-pair similarities in the whole image. We combine multiple features with local and nonlocal spatial constraints using an extended Markov random field model in the semantic space. Experimental results on three hyperspectral data sets show that the proposed method provides better performance than several state-of-the-art techniques in terms of region uniformity, overall accuracy, average accuracy, and Kappa statistics.

Real-time enhancement of sparse 3D maps using a parallel segmentation scheme based on superpixels

In this work, we focus on the problem of feature-based 3D mapping of environments with large textureless regions, which generates sparse 3D maps that may not represent well the mapped scene. To deal with this problem, based on our previous work (Cruz Martinez et al. in 2016 IEEE international symposium on mixed and augmented reality (ISMAR) adjunct proceedings, IEEE, 2016), we propose to enhance sparse 3D maps by using a superpixel-based segmentation with the aim of generating denser 3D maps of the scene in real time. Superpixels are middle-level features, which represent similar regions in an image, which can be connected in order to segment textureless areas. We propose a graphics processor unit architecture for (1) superpixel extraction considering chromatic and depth information, (2) superpixel-based segmentation, generation of connectivity matrix to compute the connected components algorithm and (3) mapping of segmented regions to 3D points. We use the ORB-SLAM system (Mur-Artal et al. in IEEE Trans Robot 31(5):1147–1163, 2015) to generate a sparse 3D map and to project the textureless segments onto it at 27 frames per second. We assessed our approach in terms of segmentation and map quality. Regarding the latter, covered area by the generated map, depth accuracy, and computational performance are reported.

A deep feature based framework for breast masses classification

Characteristic classification of mass plays a role of vital importance in diagnosis of breast cancer. The existing computer aided diagnosis (CAD) methods used to benefit a lot from low-level or middle-level features which are not that good at the simulation of real diagnostic processes, adding difficulties in improving the classification performance. In this paper, we design a deep feature based framework for breast mass classification task. It mainly contains a convolutional neural network (CNN) and a decision mechanism. Combining intensity information and deep features automatically extracted by the trained CNN from the original image, our proposed method could better simulate the diagnostic procedure operated by doctors and achieved state-of-art performance. In this framework, doctors׳ global and local impressions left by mass images were represented by deep features extracted from two different layers called high-level and middle-level features. Meanwhile, the original images were regarded as detailed descriptions of the breast mass. Then, classifiers based on features above were used in combination to predict classes of test images. And outcomes of classifiers based on different features were analyzed jointly to determine the types of test images. With the help of two kinds of feature visualization methods, deep features extracted from different layers illustrate effective in classification performance and diagnosis simulation. In addition, our method was applied to DDSM dataset and achieved high accuracy under two objective evaluation measures.

Multi-level feature representations for video semantic concept detection

Video semantic concept detection is a fundamental problem with many practical applications such as concept-based video retrieval. The major challenge of concept detection lies in the existence of the well-known semantic gap between the low-level visual features and the user׳s semantic interpretation of visual data. To bridge the semantic gap, in this paper we propose to promote low-level visual features to middle-level representations, expecting that the underlying latent semantic aspects of image data can be discovered, and such aspects can better model the semantic of images. Specifically, we employ latent Dirichlet allocation (LDA) approach to cluster the image data into semantic topics and the distributions of image low-level features on such topics are used as the middle-level feature vectors of images. Meanwhile, a recently developed more efficient probabilistic representation of low-level features, i.e., Fisher Vector is used to complement the LDA representation for video concept detection. The experimental results on the TRECVID 2013 Semantic Indexing dataset have demonstrated the effectiveness of the proposed approach.

Object Detection in Optical Remote Sensing Images Based on Weakly Supervised Learning and High-Level Feature Learning

The abundant spatial and contextual information provided by the advanced remote sensing technology has facilitated subsequent automatic interpretation of the optical remote sensing images (RSIs). In this paper, a novel and effective geospatial object detection framework is proposed by combining the weakly supervised learning (WSL) and high-level feature learning. First, deep Boltzmann machine is adopted to infer the spatial and structural information encoded in the low-level and middle-level features to effectively describe objects in optical RSIs. Then, a novel WSL approach is presented to object detection where the training sets require only binary labels indicating whether an image contains the target object or not. Based on the learnt high-level features, it jointly integrates saliency, intraclass compactness, and interclass separability in a Bayesian framework to initialize a set of training examples from weakly labeled images and start iterative learning of the object detector. A novel evaluation criterion is also developed to detect model drift and cease the iterative learning. Comprehensive experiments on three optical RSI data sets have demonstrated the efficacy of the proposed approach in benchmarking with several state-of-the-art supervised-learning-based object detection approaches.

Fine-tuning regression forests votes for object alignment in the wild.

In this paper, we propose a object alignment method that detects the landmarks of an object in 2D images. In the regression forests (RFs) framework, observations (patches) that are extracted at several image locations cast votes for the localization of several landmarks. We propose to refine the votes before accumulating them into the Hough space, by sieving and/or aggregating. In order to filter out false positive votes, we pass them through several sieves, each associated with a discrete or continuous latent variable. The sieves filter out votes that are not consistent with the latent variable in question, something that implicitly enforces global constraints. In order to aggregate the votes when necessary, we adjusts on-the-fly a proximity threshold by applying a classifier on middle-level features extracted from voting maps for the object landmark in question. Moreover, our method is able to predict the unreliability of an individual object landmark. This information can be useful for subsequent object analysis like object recognition. Our contributions are validated for two object alignment tasks, face alignment and car alignment, on data sets with challenging images collected in the wild, i.e., the Labeled Face in the Wild, the Annotated Facial Landmarks in the Wild, and the street scene car data set. We show that with the proposed approach, and without explicitly introducing shape models, we obtain performance superior or close to the state of the art for both tasks.

Exploiting high level feature for dynamic textures recognition

In this paper, a novel framework is proposed for dynamic textures (DTs) recognition by learning a high level feature using deep neural network (DNN). The insight behind the method is that a DT appearing in different videos should share similar features, which can be learned for better recognition performance. Unlike many prior works only focus on low level or middle level features, we propose a novel high level feature learning method using DNN. Our goal is to construct a compact and discriminative semantic feature. The conventional bag of features approach using k-means is not semantically meaningful since the clustering criterion is based on appearance similarity. The proposed framework can effectively overcome the problem by capturing the semantic relations of the middle level by DNN. Extensive experiments with qualitative and quantitative results demonstrate the efficacy of our approach.

Pre-release sales forecasting: A model-driven context feature extraction approach

Social media can be a potential data source in new product pre-release sales forecasting. However, the relationship is not straightforward because sales volume is the result of the interaction between different context entities. To reveal the important hidden dimensions from the structural relationships between context entities, we propose a model-driven feature extraction approach. Based on the semantic entity relationship model, middle-level features can be automatically generated through predefined primitive features with respect to typical structural relationships. Then, spatio-temporal analytics is applied in feature selection for sales forecasting models. The proposed technical approach is verified in movie box-office forecasting for a leading cinema chain in China.

Enhanced Biologically Inspired Model for Object Recognition

The biologically inspired model (BIM) proposed by Serre presents a promising solution to object categorization. It emulates the process of object recognition in primates' visual cortex by constructing a set of scale- and position-tolerant features whose properties are similar to those of the cells along the ventral stream of visual cortex. However, BIM has potential to be further improved in two aspects: mismatch by dense input and randomly feature selection due to the feedforward framework. To solve or alleviate these limitations, we develop an enhanced BIM (EBIM) in terms of the following two aspects: 1) removing uninformative inputs by imposing sparsity constraints, 2) apply a feedback loop to middle level feature selection. Each aspect is motivated by relevant psychophysical research findings. To show the effectiveness of the EBIM, we apply it to object categorization and conduct empirical studies on four computer vision data sets. Experimental results demonstrate that the EBIM outperforms the BIM and is comparable to state-of-the-art approaches in terms of accuracy. Moreover, the new system is about 20 times faster than the BIM.

Web image concept annotation with better understanding of tags and visual features

This paper focuses on improving the semi-manual method for web image concept annotation. By sufficiently studying the characteristics of tag and visual feature, we propose the Grouping-Based-Precision & Recall-Aided (GBPRA) feature selection strategy for concept annotation. Specifically, for visual features, we construct a more robust middle level feature by concatenating the k-NN results for each type of visual feature. For tag, we construct a concept-tag co-occurrence matrix, based on which the probability of an image belonging to certain concept can be calculated. By understanding the tags’ quality and groupings’ semantic depth, we propose a grouping based feature selection method; by studying the tags’ distribution, we adopt Precision and Recall as a complementary indicator for feature selection. In this way, the advantages of both tags and visual features are boosted. Experimental results show our method can achieve very high Average Precision, which greatly facilitates the annotation of large-scale web image dataset.

Multi Features Combination for Pedestrian Detection

In this paper, we propose a new approach for pedestrian detection in crowded scene from static images. The method is based on hybrid features, one type of middle-level features, which compose of multi features include gradient features, Edgelet features and haar-like features, three low-level feature sets. The gradient features focus on the local point information, the Edgelet features focus on the local edge information and the haar-like features focus on the local region information of the image. We use two stages of Adaboost to train the final classifier. In the first stage, the whole image is divided into many small windows which all include numerous low-level features. Adaboost is used in each window to get one middle-level feature which composes of some best features including gradient features, Edgelet features and haar-like features in this window. Secondly, from all middle-level features, Adaboost is used again to get the final classifier. Experiment results on common datasets and comparisons with some previous methods are given.