Discriminative Image Representation Research Articles

Transformer-based feature interactor for person re-identification with margin self-punishment loss

Person re-identification aims to retrieve specific pedestrians from different cameras and scenes, in which extracting robust and discriminative features is crucial for this task. To explore the potential interactions among images and learn more robust representations, this paper proposes Transformer-based Feature Interactor(TFI) and improved Margin Self-punishment Softmax loss(MS-Softamx). The Transformer-based Feature Interactor consists of Group Channel Pyramid Attention(GCPA) and Neighbor Interaction Modeling(NIM). Firstly, the Group Channel Pyramid Attention module provides prior information for high-level semantics via low-level semantics. The attention information is gradually stacked from coarse to fine to obtain enhanced hierarchical multi-scale features. Then, Neighbor Interaction Modeling effectively model the input and similar neighbors to produce a more robust and discriminative image representation. To make TFI more focused on intra-class embedding learning, we also propose Margin Self-punishment Softmax guide deep network learning, which obtains a tighter custom classification boundary by pushing the inter-class threshold and minimizing the intra-class variance. The proposed method is verified on four datasets, and this achieves 92.8%/95.6% mAP/Rank-1 on Market1501, 86.1%/90.8% mAP/Rank-1 on DukeMTMC, 64.4%/ 81.2% mAP/Rank-1 on MSMT17, 79.7%/80.8% mAP/Rank-1 on CUHK03-detected and 81.8%/81.9% mAP/Rank-1 on CUHK03-labeled. Extensive experiments demonstrate that the proposed method achieves competitive performance with other state-of-the-art methods.

Generative Reasoning Integrated Label Noise Robust Deep Image Representation Learning.

The development of deep learning based image representation learning (IRL) methods has attracted great attention for various image understanding problems. Most of these methods require the availability of a set of high quantity and quality of annotated training images, which can be time-consuming, complex and costly to gather. To reduce labeling costs, crowdsourced data, automatic labeling procedures or citizen science projects can be considered. However, such approaches increase the risk of including label noise in training data. It may result in overfitting on noisy labels when discriminative reasoning is employed as in most of the existing methods. This leads to sub-optimal learning procedures, and thus inaccurate characterization of images. To address this issue, in this paper, we introduce a generative reasoning integrated label noise robust deep representation learning (GRID) approach. The proposed GRID approach aims to model the complementary characteristics of discriminative and generative reasoning for IRL under noisy labels. To this end, we first integrate generative reasoning into discriminative reasoning through a supervised variational autoencoder. This allows the proposed GRID approach to automatically detect training samples with noisy labels. Then, through our label noise robust hybrid representation learning strategy, GRID adjusts the whole learning procedure for IRL of these samples through generative reasoning and that of the other samples through discriminative reasoning. Our approach learns discriminative image representations while preventing interference of noisy labels during training independently from the IRL method being selected. Thus, unlike the existing label noise robust methods, GRID does not depend on the type of annotation, label noise, neural network architecture, loss function or learning task, and thus can be directly utilized for various image understanding problems. Experimental results show the effectiveness of the proposed GRID approach compared to the state-of-the-art methods. The code of the proposed approach is publicly available at https://github.com/gencersumbul/GRID.

Robust Image Hashing With Saliency Map And Sparse Model

Abstract Image hashing is an effective technology for extensive image applications, such as retrieval, authentication and copy detection. This paper designs a new image hashing scheme based on saliency map and sparse model. The major contributions are twofold. The first contribution is the construction of a weighted image representation by combining a visual attention model called Itti model and the matrix of color vector angle (CVA). Since the Itti model can efficiently detect saliency map and CVA fully captures color information of image, they contribute to a visually robust and discriminative image representation. The second contribution is the hash extraction from the weighted image representation via sparse model. A classical sparse model called robust principal component analysis is exploited to decompose the weighted image representation into a low-rank component and a sparse component. As the low-rank component can describe intrinsic structure of image, hash calculation with low-rank component can achieve good discrimination. The efficiencies of the proposed scheme are validated by extensive experiments with open databases. The results demonstrate that the proposed scheme is superior to some state-of-the-art schemes in terms of classification performance between robustness and discrimination.

Seeing Crucial Parts: Vehicle Model Verification via a Discriminative Representation Model

Widely used surveillance cameras have promoted large amounts of street scene data, which contains one important but long-neglected object: the vehicle. Here we focus on the challenging problem of vehicle model verification. Most previous works usually employ global features (e.g., fully connected features) to further perform vehicle-level deep metric learning (e.g., triplet-based network). However, we argue that it is noteworthy to investigate the distinctiveness of local features and consider vehicle-part-level metric learning by reducing the intra-class variance as much as possible. In this article, we introduce a simple yet powerful deep model—the enforced intra-class alignment network (EIA-Net)—which can learn a more discriminative image representation by localizing key vehicle parts and jointly incorporating two distance metrics: vehicle-level embedding and vehicle-part-sensitive embedding. For learning features, we propose an effective feature extraction module that is composed of two components: the regional proposal network (RPN)-based network and part-based CNN. The RPN is used to define key vehicle regions and aggregate local features on these regions, whereas part-based CNN offers supplementary global features for the RPN-based network. The fusion features learned by feature extraction module are cast into the deep metric learning module. Especially, we derived an enforced intra-class alignment loss by re-utilizing key vehicle part information to enhance reducing intra-class variance. Furthermore, we modify the coupled cluster loss to model the vehicle-level embedding by enlarging the inter-class variance while shortening intra-class variance. Extensive experiments over benchmark datasets VehicleID and CompCars have shown that the proposed EIA-Net significantly outperforms the state-of-the-art approaches for vehicle model verification. Furthermore, we also conduct comprehensive experiments on vehicle re-identification datasets (i.e., VehicleID and VeRi776) to validate the generalization ability effectiveness of our proposed method.

Global Correlative Network for Person re-identification

Person re-identification (re-id) aims to retrieve the image which belongs to the same person from a set of images according to a few clues. The dataset images are usually from multiple cameras of different locations. Recently, local feature extraction has been one of the mainstream to handle this task. Recent methods show that exploration of part-level characteristics in person re-id can be helpful for its ability in capturing discriminative features. However, those works ignore the association among image regions, and it will lead to wrong retrieval results when not considering such clues. To address this limitation, we propose an effective Global Correlative Network (GCN) to jointly exploit the potential relationships between the features from semantic regions for more discriminative image representations. Specifically, we integrate the association between local areas and the global for discriminative region-aggregated features. With a relatively concise structure, a novel relation learning framework is proposed to enhance the representation with the contextual relations between the local and global perspectives. Experiments on Market-1501, DukeMTMC-reID, and CUHK03 show the effectiveness of our proposed method. Specifically, our method achieves a state-of-the-art rank-1 accuracy of 90.0%, 83.7%, 78.5% on the DukeMTMC-reID, CUHK03(Labeled), and CUHK03(Detected) datasets, so it sets a new state-of-the-art.

Learning Intact Features by Erasing-Inpainting for Few-shot Classification

Few-shot classification aims to categorize the samples from unseen classes with only few labeled samples. To address such a challenge, many methods exploit a base set consisting of massive labeled samples to learn an instance embedding function, i.e., image feature extractor, and it is expected to possess good transferability among different tasks. Such characteristics of few-shot learning are essentially different from that of traditional image classification only pursuing to get discriminative image representations. In this paper, we propose to learn intact features by erasing-inpainting for few-shot classification. Specifically, we argue that extracting intact features of target objects is more transferable, and then propose a novel cross-set erasing-inpainting (CSEI) method. CSEI processes the images in the support set using erasing and inpainting, and then uses them to augment the query set of the same task. Consequently, the feature embedding produced by our proposed method can contain more complete information of target objects. In addition, we propose task-specific feature modulation to make the features adaptive to the current task. The extensive experiments on two widely used benchmarks well demonstrates the effectiveness of our proposed method, which can consistently get considerable performance gains for different baseline methods.

Robust and discriminative image representation: fractional-order Jacobi-Fourier moments

Robust and discriminative image representation is a long-lasting battle in the computer vision and pattern recognition. Moment-based image representation model is effective in satisfying the core conditions of semantic description, due to its geometric invariance and independence. However, moment-based descriptors suffer from a contradiction between the robustness and discriminability, which limits the further improvement of description quality. In this paper, a set of generic moments along with a novel representation framework are proposed to mitigate this troublesome contradiction. We first define a new set of orthogonal moments, named Fractional-order Jacobi-Fourier Moments (FJFM), which is characterized by the generic nature and time-frequency analysis capability. We then develop a new framework to improve both the robustness and discriminability of image representation, called Mixed Low-order Moment Feature (MLMF), by fully exploiting the time-frequency property of FJFM. Extensive experimental results and a real-world application are provided to demonstrate the superior performance of our FJFM-based MLMF, with respect to robustness and discriminability.

Learning Category-Specific Sharable and Exemplary Visual Elements for Image Classification

This paper presents a novel method to determine the discriminative image representation via visual dictionary learning framework for image classification task. Visual dictionary learning has the capacity to represent input image using an over-complete element set. Sparsity restrains distractors and prevents over-fitting. The two main characteristics benefit the classification solution. However, one shortcoming of existing dictionary learning is that it neglects to exploit the potential correlations across visual elements, especially from the category-specific feature space. To address this problem, we first propose to learn multiple discriminative category-specific dictionaries (DCSD) from all categories. The DCSD can explore the visual elements from each category in terms of sharable property. For this reason, these learned category-specific visual elements encourage image features from the same class to have the similar feature representations. In addition, exemplary data reflect the main characteristic of whole dataset and can improve the performance of algorithm that employs them. Therefore, we further propose a representative pattern dictionary (RPD) model to discover the exemplary visual elements for promoting the discriminative capability of feature representation. These exemplary visual elements are essentially a subset of over-complete visual elements and can represent the whole sample data effectively. Finally, we design a novel strategy that integrates the merits of object proposals and deep features jointly to strengthen the semantic information of image-level feature. Experimental results on benchmark datasets demonstrate the effectiveness of our method, which is shown to be superior to the recently competing dictionary learning and deep learning based image classification approaches.

Anatomical Attention Guided Deep Networks for ROI Segmentation of Brain MR Images.

Brain region-of-interest (ROI) segmentation based on structural magnetic resonance imaging (MRI) scans is an essential step for many computer-aid medical image analysis applications. Due to low intensity contrast around ROI boundary and large inter-subject variance, it has been remaining a challenging task to effectively segment brain ROIs from structural MR images. Even though several deep learning methods for brain MR image segmentation have been developed, most of them do not incorporate shape priors to take advantage of the regularity of brain structures, thus leading to sub-optimal performance. To address this issue, we propose an anatomical attention guided deep learning framework for brain ROI segmentation of structural MR images, containing two subnetworks. The first one is a segmentation subnetwork, used to simultaneously extract discriminative image representation and segment ROIs for each input MR image. The second one is an anatomical attention subnetwork, designed to capture the anatomical structure information of the brain from a set of labeled atlases. To utilize the anatomical attention knowledge learned from atlases, we develop an anatomical gate architecture to fuse feature maps derived from a set of atlas label maps and those from the to-be-segmented image for brain ROI segmentation. In this way, the anatomical prior learned from atlases can be explicitly employed to guide the segmentation process for performance improvement. Within this framework, we develop two anatomical attention guided segmentation models, denoted as anatomical gated fully convolutional network (AG-FCN) and anatomical gated U-Net (AG-UNet), respectively. Experimental results on both ADNI and LONI-LPBA40 datasets suggest that the proposed AG-FCN and AG-UNet methods achieve superior performance in ROI segmentation of brain MR images, compared with several state-of-the-art methods.

Image Representations with Spatial Object-to-Object Relations for RGB-D Scene Recognition.

Scene recognition is challenging due to the intra-class diversity and inter-class similarity. Previous works recognize scenes either with global representations or with the intermediate representations of objects. In contrast, we investigate more discriminative image representations of object-to-object relations for scene recognition, which are based on the triplets of <object, relation, object> obtained with detection techniques. Particularly, two types of representations, including co-occurring frequency of object-to-object relation (denoted as COOR) and sequential representation of object-to-object relation (denoted as SOOR), are proposed to describe objects and their relative relations in different forms. COOR is represented as the intermediate representation of co-occurring frequency of objects and their relations, with a three order tensor that can be fed to scene classifier without further embedding. SOOR is represented in a more explicit and freer form that sequentially describe image contents with local captions. And a sequence encoding model (e.g., recurrent neural network (RNN)) is implemented to encode SOOR to the features for feeding the classifiers. In order to better capture the spatial information, the proposed COOR and SOOR are adapted to RGB-D data, where a RGB-D proposal fusion method is proposed for RGB-D object detection. With the proposed approaches COOR and SOOR, we obtain the state-of-the-art results of RGB-D scene recognition on SUN RGB-D and NYUD2 datasets.

Computer-aided detection and visualization of pulmonary embolism using a novel, compact, and discriminative image representation.

Diagnosing pulmonary embolism (PE) and excluding disorders that may clinically and radiologically simulate PE poses a challenging task for both human and machine perception. In this paper, we propose a novel vessel-oriented image representation (VOIR) that can improve the machine perception of PE through a consistent, compact, and discriminative image representation, and can also improve radiologists' diagnostic capabilities for PE assessment by serving as the backbone of an effective PE visualization system. Specifically, our image representation can be used to train more effective convolutional neural networks for distinguishing PE from PE mimics, and also allows radiologists to inspect the vessel lumen from multiple perspectives, so that they can report filling defects (PE), if any, with confidence. Our image representation offers four advantages: (1) Efficiency and compactness-concisely summarizing the 3D contextual information around an embolus in only three image channels, (2) consistency-automatically aligning the embolus in the 3-channel images according to the orientation of the affected vessel, (3) expandability-naturally supporting data augmentation for training CNNs, and (4) multi-view visualization-maximally revealing filling defects. To evaluate the effectiveness of VOIR for PE diagnosis, we use 121 CTPA datasets with a total of 326 emboli. We first compare VOIR with two other compact alternatives using six CNN architectures of varying depths and under varying amounts of labeled training data. Our experiments demonstrate that VOIR enables faster training of a higher-performing model compared to the other compact representations, even in the absence of deep architectures and large labeled training sets. Our experiments comparing VOIR with the 3D image representation further demonstrate that the 2D CNN trained with VOIR achieves a significant performance gain over the 3D CNNs. Our robustness analyses also show that the suggested PE CAD is robust to the choice of CT scanner machines and the physical size of crops used for training. Finally, our PE CAD is ranked second at the PE challenge in the category of 0mm localization error.

Semisupervised Scene Classification for Remote Sensing Images: A Method Based on Convolutional Neural Networks and Ensemble Learning

The scarcity of labeled samples has been the main obstacle to the development of scene classification for remote sensing images. To alleviate this problem, the efforts have been dedicated to semisupervised classification which exploits both labeled and unlabeled samples for training classifiers. In this letter, we propose a novel semisupervised method that utilizes the effective residual convolutional neural network (ResNet) to extract preliminary image features. Moreover, the strategy of ensemble learning (EL) is adopted to establish discriminative image representations by exploring the intrinsic information of all available data. Finally, supervised learning is performed for scene classification. To verify the effectiveness of the proposed method, it is further compared with several state-of-the-art feature representation and semisupervised classification approaches. The experimental results show that by combining ResNet features with EL, the proposed method can obtain more effective image representations and achieve superior results.

A novel discriminant multiscale representation for ear recognition

This paper proposes a novel representation for ear recognition. It introduces a new alternative of binarised statistical image features based on multiscale framework. The proposed representation allows capturing the image content at multiple resolutions. The recognition accuracy can be enhanced by the following steps. First, for a given ear image, a set of multiscale response images are derived from the bank of binarised statistical image features (B-BSIF) filter. Second, the obtained response images are summarised by concatenating their histograms, which are obtained at each scale. Finally, a discriminative ear image representation is build by projecting the above mentioned histograms into a linear discriminant analysis subspace. The proposed representation is applied on three public databases: IIT Delhi-1, IIT Delhi-2 and USTB. The obtained recognition accuracy confirms its performance than the recent existing methods.

Boosting VLAD with weighted fusion of local descriptors for image retrieval

In the last decade, many efforts have been developed for discriminative image representations. Among these works, vector of locally aggregated descriptors (VLAD) has been demonstrated to be an effective one. However, most VLAD-based methods generally employ detected SIFT descriptors and contain limited content information, in which the representation ability is deteriorated. In this work, we propose a novel framework to boost VLAD with weighted fusion of local descriptors (WF-VLAD), which encodes more discriminative clues and maintains higher performance. Toward a preferable image representation that contains sufficient details, our approach fuses SIFT sampled densely (dense SIFT) and detected from the interest points (detected SIFT) in the aggregation. Furthermore, we assign each detected SIFT corresponding weight that measured by saliency analysis to make the salient descriptors with relatively high importance. The proposed method can include sufficient image content information and highlight the important image regions. Finally, experiments on publicly available datasets demonstrate that our approach shows competitive performance in retrieval tasks.

Weighted two-step aggregated VLAD for image retrieval

The vector of locally aggregated descriptor (VLAD) has been demonstrated to be efficient and effective in image retrieval and classification tasks. Due to the small-size codebook adopted by the method, the feature space division is coarse and the discriminative power is limited. Toward a discriminative and compact image representation for visual search, we develop a novel aggregating method to build VLAD, called two-step aggregated VLAD. Firstly, we propose the bidirectional quantization from both views of descriptors and visual words, for getting finer division of feature space. Secondly, we impose the probabilistic inverse document frequency to weight the local descriptors, for highlighting the discriminative ones. Experimental results on extensive datasets show that our method yields significant improvement and is competitive with the state-of-the-art methods.

Adaptive features selection for expert datasets: A cultural heritage application

Image Retrieval is still a very active field of image processing as the number of available image datasets continuously increases. One of the principal objectives of Content-Based Image Retrieval (CBIR) is to return to user the most similar images to a given query with respect to their visual content. Our work fits in a very specific application context: indexing small expert image dataset, e.g. cultural heritage images, with no prior knowledge on the images. Because of the image complexity, one of our contributions is the choice of effective descriptors from literature placed in direct competition. Two strategies are used to combine features: a psycho-visual one and a statistical one. In this context, we propose an automatic and adaptive framework based on the well-known bags of visual words and phrases models that select relevant visual descriptors for each keypoint to construct a more discriminative image representation. Experiment results show the adaptiveness and the performance of our framework on “generic” benchmark datasets and on two cultural heritage datasets.

Deep hashing with top similarity preserving for image retrieval

Hashing has drawn more and more attention in image retrieval due to its high search speed and low storage cost. Traditional hashing methods project the high-dimensional hand-crafted visual features to compact binary codes by linear or non-linear hashing functions. Deep hashing methods, which integrate image representation learning and hash functions learning into a unified framework, have shown more superior performance. Most of existing supervised deep hashing methods mainly consider the semantic similarities among images by using pair-wise or triplet-wise constraints as supervision information. However, as a kind of crucial information, the rankings of the retrieval results, are neglected. Consequently, the produced hash codes may be suboptimal. In this paper, a new Deep Hashing with Top Similarity Preserving (DHTSP) method is proposed to optimize the quality of hash codes for image retrieval. Specifically, we utilize AlexNet to extract discriminative image representations directly from the raw image pixels and learn hash functions simultaneously. Then a top similarity preserving loss function is designed to preserve the similarity of returned images at the top of the ranking list. Experimental results on three benchmark datasets show that our proposed method outperforms most of state-of-the-art deep hashing methods and traditional hashing methods.

Discriminative Image Representation based on Multi-Cues for Computational Advertising

Discriminative Image Representation based on Multi-Cues for Computational Advertising

Direct and simultaneous estimation of cardiac four chamber volumes by multioutput sparse regression.

Cardiac four-chamber volume estimation serves as a fundamental and crucial role in clinical quantitative analysis of whole heart functions. It is a challenging task due to the huge complexity of the four chambers including great appearance variations, huge shape deformation and interference between chambers. Direct estimation has recently emerged as an effective and convenient tool for cardiac ventricular volume estimation. However, existing direct estimation methods were specifically developed for one single ventricle, i.e., left ventricle (LV), or bi-ventricles; they can not be directly used for four chamber volume estimation due to the great combinatorial variability and highly complex anatomical interdependency of the four chambers. In this paper, we propose a new, general framework for direct and simultaneous four chamber volume estimation. We have addressed two key issues, i.e., cardiac image representation and simultaneous four chamber volume estimation, which enables accurate and efficient four-chamber volume estimation. We generate compact and discriminative image representations by supervised descriptor learning (SDL) which can remove irrelevant information and extract discriminative features. We propose direct and simultaneous four-chamber volume estimation by the multioutput sparse latent regression (MSLR), which enables jointly modeling nonlinear input-output relationships and capturing four-chamber interdependence. The proposed method is highly generalized, independent of imaging modalities, which provides a general regression framework that can be extensively used for clinical data prediction to achieve automated diagnosis. Experiments on both MR and CT images show that our method achieves high performance with a correlation coefficient of up to 0.921 with ground truth obtained manually by human experts, which is clinically significant and enables more accurate, convenient and comprehensive assessment of cardiac functions.

Improving content-based image retrieval with compact global and local multi-features

The accuracy of content-based image retrieval (CBIR) systems is significantly affected by the discriminatory power of image features and distance measures. This paper performs an investigation towards finding the best local and global features and distance measures for content-based image retrieval. It provides insights into the trade-offs regarding computational costs, memory utilization and accuracy on several standard datasets which include MIRFLICKR, Corel, Holidays and ZuBuD. First, low-dimensional global and local features are extracted individually to generate a bank of small image features. Second, multilevel descriptor forms are utilized to produce highly discriminative image representations based on multi-features aggregation scheme. The relationship is highlighted between features (local and global) and other retrieval factors such as quantization approaches, visual codebooks, distance measures, vectorization methods, memory and retrieval speed. The resulting composite image representations are compact, i.e., only 32–64 vector dimension and 32–128 codebook size, and preserve high discriminative levels which further boost the retrieval accuracy and performance. The experimental results show that the presented multi-features image representations are efficient and outperform many competitive methods of the state-of-the-art.