Fine-grained Image Classification Research Articles

TOAN: Target-Oriented Alignment Network for Fine-Grained Image Categorization With Few Labeled Samples

The challenges of high intra-class variance yet low inter-class fluctuations in fine-grained visual categorization are more severe with few labeled samples, \textit{i.e.,} Fine-Grained categorization problems under the Few-Shot setting (FGFS). High-order features are usually developed to uncover subtle differences between sub-categories in FGFS, but they are less effective in handling the high intra-class variance. In this paper, we propose a Target-Oriented Alignment Network (TOAN) to investigate the fine-grained relation between the target query image and support classes. The feature of each support image is transformed to match the query ones in the embedding feature space, which reduces the disparity explicitly within each category. Moreover, different from existing FGFS approaches devise the high-order features over the global image with less explicit consideration of discriminative parts, we generate discriminative fine-grained features by integrating compositional concept representations to global second-order pooling. Extensive experiments are conducted on four fine-grained benchmarks to demonstrate the effectiveness of TOAN compared with the state-of-the-art models.

Channel Interaction Networks for Fine-Grained Image Categorization

Fine-grained image categorization is challenging due to the subtle inter-class differences. We posit that exploiting the rich relationships between channels can help capture such differences since different channels correspond to different semantics. In this paper, we propose a channel interaction network (CIN), which models the channel-wise interplay both within an image and across images. For a single image, a self-channel interaction (SCI) module is proposed to explore channel-wise correlation within the image. This allows the model to learn the complementary features from the correlated channels, yielding stronger fine-grained features. Furthermore, given an image pair, we introduce a contrastive channel interaction (CCI) module to model the cross-sample channel interaction with a metric learning framework, allowing the CIN to distinguish the subtle visual differences between images. Our model can be trained efficiently in an end-to-end fashion without the need of multi-stage training and testing. Finally, comprehensive experiments are conducted on three publicly available benchmarks, where the proposed method consistently outperforms the state-of-the-art approaches, such as DFL-CNN(Wang, Morariu, and Davis 2018) and NTS(Yang et al. 2018).

Graph-Propagation Based Correlation Learning for Weakly Supervised Fine-Grained Image Classification

The key of Weakly Supervised Fine-grained Image Classification (WFGIC) is how to pick out the discriminative regions and learn the discriminative features from them. However, most recent WFGIC methods pick out the discriminative regions independently and utilize their features directly, while neglecting the facts that regions' features are mutually semantic correlated and region groups can be more discriminative. To address these issues, we propose an end-to-end Graph-propagation based Correlation Learning (GCL) model to fully mine and exploit the discriminative potentials of region correlations for WFGIC. Specifically, in discriminative region localization phase, a Criss-cross Graph Propagation (CGP) sub-network is proposed to learn region correlations, which establishes correlation between regions and then enhances each region by weighted aggregating other regions in a criss-cross way. By this means each region's representation encodes the global image-level context and local spatial context simultaneously, thus the network is guided to implicitly discover the more powerful discriminative region groups for WFGIC. In discriminative feature representation phase, the Correlation Feature Strengthening (CFS) sub-network is proposed to explore the internal semantic correlation among discriminative patches' feature vectors, to improve their discriminative power by iteratively enhancing informative elements while suppressing the useless ones. Extensive experiments demonstrate the effectiveness of proposed CGP and CFS sub-networks, and show that the GCL model achieves better performance both in accuracy and efficiency.

Attentional Kernel Encoding Networks for Fine-Grained Visual Categorization

Fine-grained visual categorization aims to recognize objects from different sub-ordinate categories, which is a challenging task due to subtle visual differences between images. It is highly desired to identify discriminative regions while achieving highly non-linear compact representation for fine-grained visual categorization. However, existing methods either rely on manually defined part-based annotations to indicate the distinctive regions or operate on longitudinal vectors to capture the non-linear information, which may lose important spatial layout information. In this paper, we propose the Attentional Kernel Encoding Networks (AKEN) for fine-grained visual categorization. Specifically, the AKEN aggregates feature maps from the last convolutional layer of ConvNets to obtain a holistic feature representation. By Fourier embedding, it encodes features from both the longitudinal and transverse directions, which largely retains the spatial layout information. Moreover, we incorporate a Cascaded Attention (Cas-Attention) module to highlight local regions that distinguish among subordinate categories, enabling the AKEN to extract the most discriminative features. Working in conjunction with the attention mechanism, the proposed AKEN combines the strengths of ConvNets and kernels for non-linear feature learning, which can establish discriminative and descriptive feature representations for fine-grained image categorization. Experiments on three benchmark datasets show that the proposed AKEN delivers highly competitive performance, surpassing most existed methods and achieving state-of-the-art results.

Few-Shot Learning for Domain-Specific Fine-Grained Image Classification

Learning to recognize novel visual categories from a few examples is a challenging task for machines in real-world industrial applications. In contrast, humans have the ability to discriminate even similar objects with little supervision. This article attempts to address the few-shot fine-grained image classification problem. We propose a feature fusion model to explore discriminative features by focusing on key regions. The model utilizes the focus-area location mechanism to discover the perceptually similar regions among objects. High-order integration is employed to capture the interaction information among intraparts. We also design a center neighbor loss to form robust embedding space distributions. Furthermore, we build a typical fine-grained and few-shot learning dataset miniPPlankton from the real-world application in the area of marine ecological environments. Extensive experiments are carried out to validate the performance of our method. The results demonstrate that our model achieves competitive performance compared with state-of-the-art models. Our work is a valuable complement to the model domain-specific industrial applications.

Fine-Grained Butterfly Classification in Ecological Images Using Squeeze-And-Excitation and Spatial Attention Modules

Most butterfly larvae are agricultural pests and forest pests, but butterflies have important ornamental value and the ability to sense and respond to changes in the ecological environment. There are many types of butterflies, and the research on classification of butterfly species is of great significance in practical work such as environmental protection and control of agricultural and forest pests. Butterfly classification is a fine-grained image classification problem that is more difficult than generic image classification. Common butterfly photos are mostly specimen photos (indoor photos) and ecological photos (outdoor photos/natural images). At present, research on butterfly classification is more based on specimen photos. Compared with specimen photos, classification based on ecological photos is relatively difficult. This paper mainly takes ecological photos as the research object, and presents a new classification network that combines the dilated residual network, squeeze-and-excitation (SE) module, and spatial attention (SA) module. The SA module can make better use of the long-range dependencies in the images, while the SE module takes advantage of global information to enhance useful information features and suppress less useful features. The results show that the integrated model achieves higher recall, precision, accuracy, and f1-score than the state-of-the-art methods on the introduced butterfly dataset.

Progressive learning for weakly supervised fine-grained classification

Despite fine-grained image classification has made considerable progress, it still remains a challenging task due to the difficulty of finding subtle distinctions. Most existing methods solve this problem by selecting the top-N highest scores’ discriminative patches from candidate patches at one time. However, since the classification network often highlights small and sparse regions, the selected patches with the lower rank may contain noise information.To address this problem and ensure the diversity of fine-grained features, we propose a progressive patch localization module (PPL) to find the discriminative patches more accurately. Specifically, this work employs the classification model to find first most discriminative patch, then removes the most salient region to help the localization of the next most discriminative patch, and the top-K discriminative patches can be found by repeating this procedure. In addition, in order to further improve the representational power of patch-level features, we propose a feature calibration module (FCM). This module employs the global information to selectively emphasize discriminative features and suppress useless information, which can obtain more robust and discriminative local feature representations and then help classification network achieve better performance. Extensive experiments are conducted to show the substantial improvements of our method on three benchmark datasets.

Adversarial erasing attention for fine-grained image classification

Recognizing fine-grained subcategories is a challenging task due to the large intra-class diversities and small inter-class variances of the fine-grained images. The common thought is to find out the parts that can distinguish similar subcategories efficiently. Most previous works rely on the manual annotations or attention technologies to localize the discriminative parts and have achieved great progress. However, these manual annotations are demanding in practical applications and some complicated constrains on the loss functions have to be adopted to localize the discriminative parts for building multi-view feature representations. To handle the challenges above, the strategy of adversarial erasing is applied on the attention module in this paper, which learns to localize different discriminative parts by erasing the most one from the image. Without the complicated loss functions, the proposed attention module can localize the discriminative parts more efficiently. Different from many part based methods, the classification network which consists of three subnetworks is introduced, and the subnetworks are trained by the original image and two discriminative parts respectively. Moreover, features learned from the three subnetworks are then fused in a more efficiently way to build better feature representations. Four mostly used datasets of CUB-200-2011, Stanford Dogs, Stanford Cars and FGVC-Aircraft are utilized to evaluate the proposed method and experimental results show that it can outperform some state-of-the-art methods without using the manual annotations.

A New Benchmark for Instance-Level Image Classification

Although fine-grained image classification is able to classify more fine-grained sub-categories compared to its coarse-grained counterpart, it often fails to identify individual instances. Therefore, we propose a new instance-level image classification task which further refines the granularity of fine-grained classification in order to identify unique instances rather than a sub-category containing multiple instances. In addition, we introduce an instance-level image classification dataset, AircraftCarrier, which contains 20 global aircraft carrier classes, as the first publically available dataset for instance-level image classification. The classification of instance-level aircraft carriers can prove to be a challenging task due to large intra-category differences as well as variations in the camera view, illumination, scale, and the presence of complex backgrounds. The AircraftCarrier dataset put forward here has the potential to improve the development of instance-level image classification. At the same time, we provide a Simple Classification Head (SCH) technique for the classification of aircraft carriers, with classical convolutional neural network models as the backbone network. The SCH has better performance than a direct classification head, and these results provide a benchmark performance result for researchers. Furthermore, we evaluate several fine-grained image classification methods and give their benchmark results. Finally, we present the challenges of instance-level classification and discuss further directions. This study provides the first publicly available instance-level image classification dataset and a performance benchmark for further research. The dataset and codes can be downloaded at https://github.com/tsingqsu/AircraftCarrier_Dataset/.

The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification.

The key to solving fine-grained image categorization is finding discriminate and local regions that correspond to subtle visual traits. Great strides have been made, with complex networks designed specifically to learn part-level discriminate feature representations. In this paper, we show that it is possible to cultivate subtle details without the need for overly complicated network designs or training mechanisms - a single loss is all it takes. The main trick lies with how we delve into individual feature channels early on, as opposed to the convention of starting from a consolidated feature map. The proposed loss function, termed as mutual-channel loss (MC-Loss), consists of two channel-specific components: a discriminality component and a diversity component. The discriminality component forces all feature channels belonging to the same class to be discriminative, through a novel channel-wise attention mechanism. The diversity component additionally constraints channels so that they become mutually exclusive across the spatial dimension. The end result is therefore a set of feature channels, each of which reflects different locally discriminative regions for a specific class. The MC-Loss can be trained end-to-end, without the need for any bounding-box/part annotations, and yields highly discriminative regions during inference. Experimental results show our MC-Loss when implemented on top of common base networks can achieve state-of-the-art performance on all four fine-grained categorization datasets (CUB-Birds, FGVC-Aircraft, Flowers-102, and Stanford Cars). Ablative studies further demonstrate the superiority of the MC-Loss when compared with other recently proposed general-purpose losses for visual classification, on two different base networks.

多尺度特征融合的细粒度图像分类

多尺度特征融合的细粒度图像分类

基于Grad-CAM与B-CNN的细粒度图像分类方法研究

基于Grad-CAM与B-CNN的细粒度图像分类方法研究

面向细粒度图像分类的双线性残差注意力网络

面向细粒度图像分类的双线性残差注意力网络

A Novel Semantics-Preserving Hashing for Fine-Grained Image Retrieval

With the advent of the era of big data, the storage and retrieval of data have become a research hotspot. Hashing methods that transform high-dimensional data into compact binary codes have received increasing attention. Recently, with the successful application of convolutional neural networks in computer vision, deep hashing methods utilize an end-to-end framework to learn feature representations and hash codes mutually, which achieve better retrieval performance than conventional hashing methods. However, deep hashing methods still face some challenges in image retrieval. Firstly, most existing deep hashing methods preserve similarity between original data space and hash coding space using loss functions with high time complexity, which cannot get a win-win situation in time and accuracy. Secondly, few existing deep hashing methods are designed for fine-grained image retrieval, which is necessary in practice. In this study, we propose a novel semantics-preserving hashing method which solves the above problems. We add a hash layer before the classification layer as a feature switch layer to guide the classification. At the same time, we replace the complicated loss with the simple classification loss, combining with quantization loss and bit balance loss to generate high-quality hash codes. Besides, we incorporate feature extractor designed for fine-grained image classification into our network for better representation learning. The results on three widely-used fine-grained image datasets show that our method is superior to other state-of-the-art image retrieval methods.

Learning Semantically Enhanced Feature for Fine-Grained Image Classification

We aim to provide a computationally cheap yet effective approach for fine-grained image classification (FGIC) in this letter. Unlike previous methods that rely on complex part localization modules, our approach learns fine-grained features by enhancing the semantics of sub-features of a global feature. Specifically, we first achieve the sub-feature semantic by arranging feature channels of a CNN into different groups through channel permutation. Meanwhile, to enhance the discriminability of sub-features, the groups are guided to be activated on object parts with strong discriminability by a weighted combination regularization. Our approach is parameter parsimonious and can be easily integrated into the backbone model as a plug-and-play module for end-to-end training with only image-level supervision. Experiments verified the effectiveness of our approach and validated its comparable performance to the state-of-the-art methods. Code is available at https://github.com/cswluo/SEF .

Fine-grained image classification with factorized deep user click feature

The advantages of user click data greatly inspire its wide application in fine-grained image classification tasks. In previous click data based image classification approaches, each image is represented as a click frequency vector on a pre-defined query/word dictionary. However, this approach not only introduces high-dimensional issues, but also ignores the part of speech (POS) of a specific word as well as the word correlations. To address these issues, we devise the factorized deep click features to represent images. We first represent images as the factorized TF-IDF click feature vectors to discover word correlation, wherein several word dictionaries of different POS are constructed. Afterwards, we learn an end-to-end deep neural network on click feature tensors built on these factorized TF-IDF vectors. We evaluate our approach on the public Clickture-Dog dataset. It shows that: 1) the deep click feature learned on click tensor performs much better than traditional click frequency vectors; and 2) compared with many state-of-the-art textual representations, the proposed deep click feature is more discriminative and with higher classification accuracies.

Convolutional neural network: a review of models, methodologies and applications to object detection

Deep learning has developed as an effective machine learning method that takes in numerous layers of features or representation of the data and provides state-of-the-art results. The application of deep learning has shown impressive performance in various application areas, particularly in image classification, segmentation and object detection. Recent advances of deep learning techniques bring encouraging performance to fine-grained image classification which aims to distinguish subordinate-level categories. This task is extremely challenging due to high intra-class and low inter-class variance. In this paper, we provide a detailed review of various deep architectures and model highlighting characteristics of particular model. Firstly, we described the functioning of CNN architectures and its components followed by detailed description of various CNN models starting with classical LeNet model to AlexNet, ZFNet, GoogleNet, VGGNet, ResNet, ResNeXt, SENet, DenseNet, Xception, PNAS/ENAS. We mainly focus on the application of deep learning architectures to three major applications, namely (i) wild animal detection, (ii) small arm detection and (iii) human being detection. A detailed review summary including the systems, database, application and accuracy claimed is also provided for each model to serve as guidelines for future work in the above application areas.

Learning Cascade Attention for fine-grained image classification

Fine-grained image classification is a challenging task due to the large inter-class difference and small intra-class difference. In this paper, we propose a novel Cascade Attention Model using the Deep Convolutional Neural Network to address this problem. Our method first leverages the Spatial Confusion Attention to identify ambiguous areas of the input image. Two constraint loss functions are proposed: the Spatial Mask loss and the Spatial And loss; Second, the Cross-network Attention, applying different pre-train parameters to the two stream architecture. Also, two novel loss functions called Cross-network Similarity loss and Satisfied Rank loss are proposed to make the two-stream networks reinforce each other and get better results. Finally, the Network Fusion Attention merges intermediate results with the novel entropy add strategy to obtain the final predictions. All of these modules can work together and can be trained end to end. Besides, different from previous works, our model is fully weak-supervised and fully paralleled, which leads to easier generalization and faster computation. We obtain the state-of-the-art performance on three challenge benchmark datasets (CUB-200-2011, FGVC-Aircraft and Flower 102) with results of 90.8%, 92.1%, and 98.5%, respectively. The model will be publicly available at https://github.com/billzyx/LCA-CNN.

A Survey of Fine-Grained Image Classification Based on Deep Learning

The deep learning technology has shown impressive performance in various vision tasks such as image classification and object detection. Recent advances of deep learning techniques bring encouraging performance to fine-grained image classification which aims to distinguish subordinate-level categories. This paper reviews the major deep learning concepts pertinent to fine-grained image analysis and summarizes to the field, briefly introduces the representative method of fine-grained image classification based on deep learning. At present, the field of fine classification has developed a variety of methods including multi-network learning, target part detection and visual attention. Each method is to obtain a distinguishing area in the image, which help the network learn more effective features to complete the classification and recognition of fine-grained images.

Attention Bilinear Pooling for Fine-Grained Classification

Fine-grained image classification is a challenging problem because of its large intra-class differences and low inter-class variance. Bilinear pooling based models have been shown to be effective at fine-grained classification, while most previous approaches neglect the fact that distinctive features or modeling distinguishing regions usually have an important role in solving the fine-grained problem. In this paper, we propose a novel convolutional neural network framework, i.e., attention bilinear pooling, for fine-grained classification with attention. This framework can learn the distinctive feature information from the channel or spatial attention. Specifically, the channel and spatial attention allows the network to better focus on where the key targets are in the image. This paper embeds spatial attention and channel attention in the underlying network architecture to better represent image features. To further explore the differences between channels and spatial attention, we propose channel attention bilinear pooling (CAB), spatial attention bilinear pooling (SAB), channel spatial attention bilinear pooling (CSAB), and spatial channel attention bilinear pooling (SCAB) as four alternative frames. A variety of experiments on several datasets show that our proposed method has a very impressive performance compared to other methods based on bilinear pooling.