Fine-grained Image Research Articles

Attention-based supervised contrastive learning on fine-grained image classification

Attention-based supervised contrastive learning on fine-grained image classification

Deep Learning-Based Fine-Grained Image Recognition

Fine-grained image recognition (FGIR) is an important part of today's vision field. Unlike basic image classification, FGIR aims at recognizing multiple different dependent categories of meta-category objects. However, early FGIR faced many challenges, such as the fact that the differences between fine-grained images are subtle and the background is often noisy, which greatly affects the accuracy of the recognition performance. Now, with the cconstant development of deep learning in the field of optical recognition, FGIR has made large accomplishments and progress. In this paper, we make a more broad survey of deep learning-based FGIR methods in recent years and categorize these new methods. Then they are comparatively analyzed through recognized datasets in recent years to summarize the contribution of different methods to the current challenges. Eventually, the current state of research is analyzed and several directions for future research are put forward.

Lychee cultivar fine-grained image classification method based on improved ResNet-34 residual network

Lychee, a key economic crop in southern China, has numerous similar-looking varieties. Classifying these can aid farmers in understanding each variety's growth and market demand, enhancing agricultural efficiency. However, existing classification techniques are subjective, complex, and costly. This paper proposes a lychee classification method using an improved ResNet-34 residual network for six common varieties. We enhance the CBAM attention mechanism by replacing the large receptive field in the SAM module with a smaller one. Attention mechanisms are added at key network stages, focusing on crucial image information. Transfer learning is employed to apply ImageNet-trained model weights to this task. Test set evaluations demonstrate that our improved ResNet-34 network surpasses the original, achieving a recognition accuracy of 95.8442%, a 5.58 percentage point improvement.

Fine-Grained Few-Shot Image Classification Based on Feature Dual Reconstruction

Fine-grained few-shot image classification is a popular research area in deep learning. The main goal is to identify subcategories within a broader category using a limited number of samples. The challenge stems from the high intra-class variability and low inter-class variability of fine-grained images, which often hamper classification performance. To overcome this, we propose a fine-grained few-shot image classification algorithm based on bidirectional feature reconstruction. This algorithm introduces a Mixed Residual Attention Block (MRA Block), combining channel attention and window-based self-attention to capture local details in images. Additionally, the Dual Reconstruction Feature Fusion (DRFF) module is designed to enhance the model’s adaptability to both inter-class and intra-class variations by integrating features of different scales across layers. Cosine similarity networks are employed for similarity measurement, enabling precise predictions. The experiments demonstrate that the proposed method achieves classification accuracies of 96.99%, 98.53%, and 89.78% on the CUB-200-2011, Stanford Cars, and Stanford Dogs datasets, respectively, confirming the method’s efficacy in fine-grained classification tasks.

DACBN: Dual attention convolutional broad network for fine-grained visual recognition

Fine-grained visual classification (FGVC) is a challenging task due to its small inter-class differences and large intra-class differences. Most existing methods rely on manual labeling of key identification areas, which requires high labor costs. In addition, existing methods also tend to ignore the differences effect of different feature channels in the feature map, which has a certain impact on the model classification accuracy. To solve the above problems, this paper proposes a dual attention convolutional broad network. Firstly, a new dual attention mechanism is designed to suppress the background noise of fine-grained images and give greater weight to the discriminative feature regions and channels. Secondly, the ensemble broad learning system framework is used to further enhance the dual attention features, so that the discriminative features can further improve the recognition ability of the model. Finally, by multiple comparative experiments, it is reported that the method proposed in this article has achieved excellent recognition results on three commonly used datasets.

Adversarially attack feature similarity for fine-grained visual classification

Fine-grained visual classification (FGVC) strives to distinguish images from distinct sub-classes within the same overarching meta-class, which is significant in various practical applications. Existing works mainly employ attention mechanisms to learn discriminative feature representations of objects under weakly supervised learning. In this paper, we argue that this likehood-based attention learning manner often outputs an inadequate feature representation since the available image-level labels fail to provide an explicit supervisory signal for attention learning, especially when the fine-grained images share a small and inconsistent inter-class variance. To alleviate this issue, we consider this challenging task from the perspective of attacking the feature representation between similar sub-classes to maximize the feature discriminativeness via learning adversarial examples, and propose an Adversarial-Aware Fine-Grained Visual Classification Network (A2Net). Specifically, we first propose an adversarial attack module based on projected gradient descent, which appends multiple-scale adversarial perturbations to simulate subclass examples with different similarities. Then, we introduce an adversarial attention generation module that estimates the effect of attention learned on adversarial examples and legitimate examples for the final class prediction through causal inference. The adversarial attention generation module is encouraged to maximize the effect, which provides powerful supervision to capture more attention indicating the discriminative parts. We further propose an adversarial-aware module to learn the feature-level differences between legitimate and adversarial examples, which helps enhance the semantic boundaries of class-specific features for accurate FGVC. The extensive array of experiments conducted serves to underscore the efficacy of the proposed A2Net, outperforming state-of-the-art FGVC methods on CUB-200–2011, FGVC-Aircraft, Stanford Cars, Stanford Dogs, and NABirds benchmarks.

A simple scheme to amplify inter-class discrepancy for improving few-shot fine-grained image classification

Few-shot image classification is a challenging topic in pattern recognition and computer vision. Few-shot fine-grained image classification is even more challenging, due to not only the few shots of labelled samples but also the subtle differences to distinguish subcategories in fine-grained images. A recent method called task discrepancy maximisation (TDM) can be embedded into the feature map reconstruction network (FRN) to generate discriminative features, by preserving the appearance details through reconstructing the query image and then assigning higher weights to more discriminative channels, producing the state-of-the-art performance for few-shot fine-grained image classification. However, due to the small inter-class discrepancy in fine-grained images and the small training set in few-shot learning, the training of FRN+TDM can result in excessively flexible boundaries between subcategories and hence overfitting. To resolve this problem, we propose a simple scheme to amplify inter-class discrepancy and thus improve FRN+TDM. To achieve this aim, instead of developing new modules, our scheme only involves two simple amendments to FRN+TDM: relaxing the inter-class score in TDM, and adding a centre loss to FRN. Extensive experiments on five benchmark datasets showcase that, although embarrassingly simple, our scheme is quite effective to improve the performance of few-shot fine-grained image classification. The code is available at https://github.com/Airgods/AFRN.git.

An Optimization Method for Lightweight Rock Classification Models: Transferred Rich Fine-Grained Knowledge.

Rock image classification represents a challenging fine-grained image classification task characterized by subtle differences among closely related rock categories. Current contrastive learning methods prevalently utilized in fine-grained image classification restrict the model's capacity to discern critical features contrastively from image pairs, and are typically too large for deployment on mobile devices used for in situ rock identification. In this work, we introduce an innovative and compact model generation framework anchored by the design of a Feature Positioning Comparison Network (FPCN). The FPCN facilitates interaction between feature vectors from localized regions within image pairs, capturing both shared and distinctive features. Further, it accommodates the variable scales of objects depicted in images, which correspond to differing quantities of inherent object information, directing the network's attention to additional contextual details based on object size variability. Leveraging knowledge distillation, the architecture is streamlined, with a focus on nuanced information at activation boundaries to master the precise fine-grained decision boundaries, thereby enhancing the small model's accuracy. Empirical evidence demonstrates that our proposed method based on FPCN improves the classification accuracy mobile lightweight models by nearly 2% while maintaining the same time and space consumption.

Multi-FusNet: fusion mapping of features for fine-grained image retrieval networks.

As the diversity and volume of images continue to grow, the demand for efficient fine-grained image retrieval has surged across numerous fields. However, the current deep learning-based approaches to fine-grained image retrieval often concentrate solely on the top-layer features, neglecting the relevant information carried in the middle layer, even though these information contains more fine-grained identification content. Moreover, these methods typically employ a uniform weighting strategy during hash code mapping, risking the loss of critical region mapping-an irreversible detriment to fine-grained retrieval tasks. To address the above problems, we propose a novel method for fine-grained image retrieval that leverage feature fusion and hash mapping techniques. Our approach harnesses a multi-level feature cascade, emphasizing not just top-layer but also intermediate-layer image features, and integrates a feature fusion module at each level to enhance the extraction of discriminative information. In addition, we introduce an agent self-attention architecture, marking its first application in this context, which steers the model to prioritize on long-range features, further avoiding the loss of critical regions of the mapping. Finally, our proposed model significantly outperforms existing state-of-the-art, improving the retrieval accuracy by an average of 40% for the 12-bit dataset, 22% for the 24-bit dataset, 16% for the 32-bit dataset, and 11% for the 48-bit dataset across five publicly available fine-grained datasets. We also validate the generalization ability and performance stability of our proposed method by another five datasets and statistical significance tests. Our code can be downloaded from https://github.com/BJFU-CS2012/MuiltNet.git.

Enhancing Fine-Grained Image Recognition with Multi-Channel Self-Attention Mechanisms: A Focus on Fruit Fly Species Classification

Enhancing Fine-Grained Image Recognition with Multi-Channel Self-Attention Mechanisms: A Focus on Fruit Fly Species Classification

Multi-level information fusion Transformer with background filter for fine-grained image recognition

Multi-level information fusion Transformer with background filter for fine-grained image recognition

A Fine-Grained Image Classification Method Built on MobileViT

With a rapid development of artificial intelligence technology, fine-grained image classification has gained widespread application. For mobile terminals, this paper introduces an image classification method built on MobileViT, and it can apply into fine-grained image classification. The original MobileViT model has been optimized in three ways. Initially, the h-swish activation function is used to enhance the network performance. Second, the cross-entropy loss function is used to further realize the parameter optimization and model accuracy improvement. Finally, a dropout layer is joined before the fully connected layer can effectively decrease the model recognition time and prevent over-fitting. Experimental data on public tomato disease datasets demonstrate that the improved fine-grained image classification method put forward in this paper exhibits higher classification accuracy, better stability and network generalization ability than other models.

Prognostic prediction of ovarian cancer based on hierarchical sampling & fine-grained recognition convolution neural network

Ovarian cancer ranks among the deadliest gynecological malignancies, with high-grade serous adenocarcinoma (HGSA) constituting 75 % of ovarian cancer cases and accounting for 80 %–90 % of ovarian cancer-related fatalities. Accurate prognosis prediction for ovarian HGSA is of critical clinical significance. However, existing prognostic analysis methods exhibit suboptimal performance in identifying prognostically relevant pathological markers, making it challenging, even for experienced pathologists, to forecast the prognosis of ovarian HGSA patients accurately. Deep learning holds promise in enhancing prognostic prediction accuracy. However, a significant challenge in this field arises from the impracticality of directly inputting histopathology whole-slide images with millions to billions of pixels into existing deep learning networks for training and inference. To address this issue, we propose a prognostic analysis network for ovarian cancer based on hierarchical sampling and fine-grained recognition. This network comprises a two-stage hierarchical sampling sub-network, a fine-grained image recognition sub-network, and a prognostic analysis sub-network. We assess the system's performance using a pathological dataset of 450 cases of ovarian HGSA diagnosed and treated at the Pathology Department of the West China Second University Hospital of Sichuan University. Results indicate that: (1) The proposed prognostic analysis network based on two-stage hierarchical sampling sub-network can effectively analyze the histopathology whole-slide image; (2) the use of fine-grained image recognition and the introduction of clinical information can improve the performance of HGSA prognosis analysis method, with an improvement range of 4.77–10.66%; and (3) the proposed method can be used for the model analysis of pathological datasets of HGSA. Moreover, this method can be used to explore effective characteristics from the multi-modal dataset through automatic learning with prediction recall, accuracy, and precision rates of 80.0%, 81.1%, and 81.8%, respectively, underscoring its clinical potential. This study reveals the reliability and effectiveness of the proposed prognosis evaluation method of HGCA. Conclusions can help clinicians precisely evaluate the recurrence risk of patients, take the initiative to master the diagnosis and treatment, and increase the long-term survival rate of patients.

Content-Aware Rectified Activation for Zero-Shot Fine-Grained Image Retrieval.

Fine-grained image retrieval mainly focuses on learning salient features from the seen subcategories as discriminative embedding while neglecting the problems behind zero-shot settings. We argue that retrieving fine-grained objects from unseen subcategories may rely on more diverse clues, which are easily restrained by the salient features learnt from seen subcategories. To address this issue, we propose a novel Content-aware Rectified Activation model, which enables this model to suppress the activation on salient regions while preserving their discrimination, and spread activation to adjacent non-salient regions, thus mining more diverse discriminative features for retrieving unseen subcategories. Specifically, we construct a content-aware rectified prototype (CARP) by perceiving semantics of salient regions. CARP acts as a channel-wise non-destructive activation upper bound and can be selectively used to suppress salient regions for obtaining the rectified features. Moreover, two regularizations are proposed: 1) a semantic coherency constraint that imposes a restriction on semantic coherency of CARP and salient regions, aiming at propagating the discriminative ability of salient regions to CARP, 2) a feature-navigated constraint to further guide the model to adaptively balance the discrimination power of rectified features and the suppression power of salient features. Experimental results on fine-grained and product retrieval benchmarks demonstrate that our method consistently outperforms the state-of-the-art methods.

Deep learning-based differentiation of ventricular septal defect from tetralogy of Fallot in fetal echocardiography images.

Congenital heart disease (CHD) seriously affects children's health and quality of life, and early detection of CHD can reduce its impact on children's health. Tetralogy of Fallot (TOF) and ventricular septal defect (VSD) are two types of CHD that have similarities in echocardiography. However, TOF has worse diagnosis and higher morality than VSD. Accurate differentiation between VSD and TOF is highly important for administrative property treatment and improving affected factors' diagnoses. TOF and VSD were differentiated using convolutional neural network (CNN) models that classified fetal echocardiography images. We collected 105 fetal echocardiography images of TOF and 96 images of VSD. Four CNN models, namely, VGG19, ResNet50, NTS-Net, and the weakly supervised data augmentation network (WSDAN), were used to differentiate the two congenital heart diseases. The performance of these four models was compared based on sensitivity, accuracy, specificity, and AUC. VGG19 and ResNet50 performed similarly, with AUCs of 0.799 and 0.802, respectively. A superior performance was observed with NTS-Net and WSDAN specific for fine-grained image categorization tasks, with AUCs of 0.823 and 0.873, respectively. WSDAN had the best performance among all models tested. WSDAN exhibited the best performance in differentiating between TOF and VSD and is worthy of further clinical popularization.

Attribute guided fusion network for obtaining fine-grained image captions

Attribute guided fusion network for obtaining fine-grained image captions

Posture-guided part learning for fine-grained image categorization

Posture-guided part learning for fine-grained image categorization

Classification of Rainfall Intensity and Cloud Type from Dash Cam Images Using Feature Removal by Masking

Weather Report is an initiative from Weathernews Inc. to obtain sky images and current weather conditions from the users of its weather app. This approach can provide supplementary weather information to radar observations and can potentially improve the accuracy of forecasts However, since the time and location of the contributed images are limited, gathering data from different sources is also necessary. This study proposes a system that automatically submits weather reports using a dash cam with communication capabilities and image recognition technology. This system aims to provide detailed weather information by classifying rainfall intensities and cloud formations from images captured via dash cams. In models for fine-grained image classification tasks, there are very subtle differences between some classes and only a few samples per class. Therefore, they tend to include irrelevant details, such as the background, during training, leading to bias. One solution is to remove useless features from images by masking them using semantic segmentation, and then train each masked dataset using EfficientNet, evaluating the resulting accuracy. In the classification of rainfall intensity, the model utilizing the features of the entire image achieved up to 92.61% accuracy, which is 2.84% higher compared to the model trained specifically on road features. This outcome suggests the significance of considering information from the whole image to determine rainfall intensity. Furthermore, analysis using the Grad-CAM visualization technique revealed that classifiers trained on masked dash cam images particularly focused on car headlights when classifying the rainfall intensity. For cloud type classification, the model focusing solely on the sky region attained an accuracy of 68.61%, which is 3.16% higher than that of the model trained on the entire image. This indicates that concentrating on the features of clouds and the sky enables more accurate classification and that eliminating irrelevant areas reduces misclassifications.

Graph Representation and Prototype Learning for webly supervised fine-grained image recognition

Webly supervised fine-grained image recognition (FGIR) learns to distinguish sub-ordinate categories based on webly-retrieved data, which can dramatically alleviate the dependency on manually annotated labels. This is quite a challenging task due to the heavy noise labels and the inherent dilemma of small inter-class variance and large intra-class variance. Current webly supervised algorithms learn holistic category prototypes to help correct noisy labels but ignore local features that can distinguish different sub-ordinate categories. In this work, we propose a Graph Representation and Prototype Learning (GRPL) framework to automatically mine discriminative local regions and their interactions with holistic image to learn instance graph representation both category graph prototype to help correct noisy labels and retrieve out-of-distribution (OOD) samples. Specifically, an attention-focused module is designed to extract the discriminative regions and then build a structured graph to correlate them with the holistic image for each instance and an identical graph to model holistic-local correlations for each category. Next, we apply two stacked graph convolution networks to explore holistic-local interaction within each graph and across two graphs to learn graph representation for each instance and graph prototype for each category. Finally, the similarities between the instance-level and prototype-level graph representation are learned to help correct noisy labels and exclude OOD samples. Extensive experiments conducted on several datasets show the proposed approach achieves superior performance compared with current leading algorithms.

Fine-grained image classification method based on hybrid attention module.

To efficiently capture feature information in tasks of fine-grained image classification, this study introduces a new network model for fine-grained image classification, which utilizes a hybrid attention approach. The model is built upon a hybrid attention module (MA), and with the assistance of the attention erasure module (EA), it can adaptively enhance the prominent areas in the image and capture more detailed image information. Specifically, for tasks involving fine-grained image classification, this study designs an attention module capable of applying the attention mechanism to both the channel and spatial dimensions. This highlights the important regions and key feature channels in the image, allowing for the extraction of distinct local features. Furthermore, this study presents an attention erasure module (EA) that can remove significant areas in the image based on the features identified; thus, shifting focus to additional feature details within the image and improving the diversity and completeness of the features. Moreover, this study enhances the pooling layer of ResNet50 to augment the perceptual region and the capability to extract features from the network's less deep layers. For the objective of fine-grained image classification, this study extracts a variety of features and merges them effectively to create the final feature representation. To assess the effectiveness of the proposed model, experiments were conducted on three publicly available fine-grained image classification datasets: Stanford Cars, FGVC-Aircraft, and CUB-200-2011. The method achieved classification accuracies of 92.8, 94.0, and 88.2% on these datasets, respectively. In comparison with existing approaches, the efficiency of this method has significantly improved, demonstrating higher accuracy and robustness.