Fine-grained Image Research Articles

Manifold and patch-based unsupervised deep metric learning for fine-grained image retrieval

Manifold and patch-based unsupervised deep metric learning for fine-grained image retrieval

A cross-granularity feature fusion method for fine-grained image recognition

A cross-granularity feature fusion method for fine-grained image recognition

TPTE: Text-Guided Patch Token Exploitation for Unsupervised Fine-Grained Representation Learning

Recent advances in pre-trained vision-language models have successfully boosted the performance of unsupervised image representation in many vision tasks. Most of existing works focus on learning global visual features with Transformers and neglect detailed local cues, leading to suboptimal performance in fine-grained vision tasks. In this article, we propose a text-guided patch token exploitation framework to enhance the discriminative power of unsupervised representation by exploiting more detailed local features. Our text-guided decoder extracts local features with the guidance of texts or learned prompts describing discriminative object parts. We hence introduce a local-global relation distillation loss to promote the joint optimization of local and global features. The proposed method allows to flexibly extract either global or global-local features as the image representation. It significantly outperforms previous methods in fine-grained image retrieval and base-to-new fine-grained classification tasks. For instance, our Recall@1 metric surpasses the recent unsupervised retrieval method STML by 6.0% on the SOP dataset. The code is publicly available at https://github.com/maosnhehe/TPTE .

The scope and limits of fine-grained image and category information in the ventral visual pathway.

Humans can easily abstract incoming visual information into discrete semantic categories. Previous research employing functional MRI (fMRI) in humans has identified cortical organizing principles that allow not only for coarse-scale distinctions such as animate versus inanimate objects but also more fine-grained distinctions at the level of individual objects. This suggests that fMRI carries rather fine-grained information about individual objects. However, most previous work investigating fine-grained category representations either additionally included coarse-scale category comparisons of objects, which confounds fine-grained and coarse-scale distinctions, or only used a single exemplar of each object, which confounds visual and semantic information. To address these challenges, here we used multisession human fMRI (female and male) paired with a broad yet homogenous stimulus class of 48 terrestrial mammals, with 2 exemplars per mammal. Multivariate decoding and representational similarity analysis (RSA) revealed high image-specific reliability in low- and high-level visual regions, indicating stable representational patterns at the image level. In contrast, analyses across exemplars of the same animal yielded only small effects in the lateral occipital complex (LOC), indicating rather subtle category effects in this region. Variance partitioning with a deep neural network and shape model showed that across exemplar effects in EVC were largely explained by low-level visual appearance, while representations in LOC appeared to also contain higher category-specific information. These results suggest that representations typically measured with fMRI are dominated by image-specific visual or coarse-grained category information but indicate that commonly employed fMRI protocols may reveal subtle yet reliable distinctions between individual objects.Significance Statement While it has been suggested that functional MRI (fMRI) responses in ventral visual cortex carry fine-grained information about individual objects, much previous research has confounded fine-grained with coarse-scale category information or only used individual visual exemplars, which potentially confounds semantic and visual object information. Here we address these challenges in a multisession fMRI study where participants viewed a highly homogenous stimulus set of 48 land mammals with 2 exemplars per animal. Our results reveal a strong dominance of image-specific effects and additionally indicate subtle yet reliable category-specific effects in lateral occipital complex, underscoring the capacity of commonly employed fMRI protocols to uncover fine-grained visual information.

Fine-grained recognition algorithm of crop pests based on cross-layer bilinear aggregation and multi-task learning

Fine-grained recognition of crop pests is a crucial concern in the field of agriculture, as the accuracy of recognition and generalization ability directly affect the yield and quality of crops. Aiming at the characteristics of crop pests with a wide variety of species, small inter-class and large intra-class differences in external morphology, as well as the problems of uneven sample distribution and noisy labels in fine-grained image datasets under complex environments, we propose a fine-grained recognition model of crop pests (MT-MACLBPHSNet) based on cross-layer bilinear aggregation and multi-task learning, which consists of three key modules: the backbone network module, the cross-layer bilinear aggregation module, and the multi-task learning module. A new union loss function is designed in the primary task of the multi-task learning module, which is used to alleviate the two problems existing in the model training fine-grained image datasets. The experimental results show that the model effectively balances the model complexity and recognition accuracy in a comparative analysis with several existing excellent network models on the IP102-CP13 dataset, with the recognition accuracy reaching 75.37%, which is 7.06% higher than the Baseline model, and the F1-score reaching 67.06%. Additionally, the generalization of the model is also verified on the IP102-VP16 dataset, and the model outperforms most of the models in terms of recognition accuracy and generalization ability, which can provide an effective reference for fine-grained recognition of crop pests.

Georecon: a coarse-to-fine visual 3D reconstruction approach for high-resolution images with neural matching priors

ABSTRACT Visual 3D reconstruction enables rebuilding 3D scenes from captured images, serving as a fundamental data source for digital earth modeling and intelligent cities. In the foundational step, recent methods leverage learning-based descriptors for image registration and achieve tremendous advances in precision and robustness. However, these methods inevitably execute down sampling towards high-resolution images to fit the needs of neural networks, which leads to precision degradation of feature localization and matching. Thus, we propose GeoRecon: a novel coarse-to-fine visual 3D reconstruction method that optimally utilizes high-resolution images for high-quality visual 3D reconstruction. Firstly, the coarse stage conducts coarse reconstruction from downsampled images by performing neural matching with geometric priors. Secondly, we define the fine-grained stage, proposing a GPU-based algorithm for generating image-patch correspondences based on the neural matching priors to perform fine-grained image registration. Finally, based on the optimized camera poses under this coarse-to-fine paradigm, progressive dense reconstruction leveraging efficient neural radiance fields is proposed to accomplish the high-quality MVS reconstruction. Comparative experiments across various scenarios demonstrate the proposed method’s superior precision, robustness, and reconstruction quality.

MT-DSNet: Mix-mask teacher–student strategies and dual dynamic selection plug-in module for fine-grained image recognition

The fine-grained image recognition (FGIR) task aims to classify and distinguish subtle differences between subcategories with visually similar appearances, such as bird species and the makes or models of vehicles. However, subtle interclass differences and significant intraclass variances lead to poor model recognition performance. To address these challenges, we developed a mixed-mask teacher–student cooperative training strategy. A mixed masked image is generated and embedded into a knowledge distillation network by replacing one image’s visible marker with another’s masked marker. Collaborative reinforcement between teachers and students is used to improve the recognition performance of the network. We chose the classic transformer architecture as a baseline to better explore the contextual relationships between features. Additionally, we suggest a dual dynamic selection plug-in for choosing features with discriminative capabilities in the spatial and channel dimensions and filter out irrelevant interference information to efficiently handle background and noise features in fine-grained images. The proposed feature suppression module is used to enhance the differences between different features, thereby motivating the network to mine more discriminative features. We validated our method using two datasets: CUB-200-2011 and Stanford Cars. The experimental results show that the proposed MT-DSNet can significantly improve the feature representation for FGIR tasks. Moreover, by applying it to different fine-grained networks, the FGIR accuracy can be improved without changing the original network structure. We hope that this work provides a promising approach for improving the feature representation of networks in the future.

Imbalanced image classification algorithm based on fine-grained analysis

Fine-grained attribute analysis and data imbalance have always been research hotspots in the field of computer vision. Due to the complexity and diversity of fine-grained attribute images, traditional image classification methods have shortcomings in paying attention to fine-grained attributes of images and perform poorly when dealing with imbalanced data sets. To overcome these problems, this study proposes a fine-grained image threshold classification algorithm based on deep metric learning. By introducing a metric learning method, the focus on fine-grained attributes of images is enhanced. At the same time, by applying pairwise loss and proxy loss, the classification accuracy of the model is improved and the model convergence speed is accelerated. In order to deal with the problem of data imbalance, a classifier based on threshold analysis is designed. The classifier uses threshold analysis technology to achieve multi-level classification of fine-grained images, thereby improving the problem of low classification accuracy of a few categories in imbalanced data sets. Experimental results show that the proposed fine-grained image threshold classification algorithm based on deep metric learning is significantly superior to other methods in terms of classification accuracy.

A Hierarchically Discriminative Loss with Group Regularization for Fine-Grained Image Classification

Fine-grained visual classification targets the discrimination of subordinate categories within broader classes, such as avian species, aerial crafts, and notably, in medical diagnostics like breast cancer. In the domain of breast cancer classification, there has been an emergence of numerous fine-grained models leveraging pathological images. Elevating the interpretability of the model’s discriminative processes among these nuanced categories holds promise in enhancing the transparency of AI decision-making. However, the effective utilization of hierarchical label structures within medical data remains a crucial consideration. In this work, we explore how the label hierarchy can be used to better learn subtle feature embeddings. We observe that the semantic relationships between fine-grained categories can help to analyze the misclassified samples. To this end, with the label hierarchy, we introduce two novel losses to cultivate subtle feature representations, coordinate with feature learning, and align with the principles of Explainable AI (XAI). First, we propose hierarchically discriminative loss to enhance the interactions between fine- and coarse-level features, which can help reinforce the discriminability of fine-grained features for reducing false predictions belonging to the out-group relation. Second, we introduce the in-group regularization loss to establish the interactions between the target class and those confusing classes belonging to the in-group relation. Treating another confusing class as a distraction can regularize feature learning of the target class. Thus, it allows the network to discover more discriminative features and reduces in-group false predictions. Five commonly used datasets for fine-grained classification are extensively evaluated. Our experimental results validate the effectiveness of our proposed novel losses when compared to state-of-the-art methods that utilize hierarchical multi-granularity labels.

A fine-grained image classification algorithm based on self-supervised learning and multi-feature fusion of blood cells

Leukemia is a prevalent and widespread blood disease, and its early diagnosis is crucial for effective patient treatment. Diagnosing leukemia types heavily relies on pathologists’ morphological examination of blood cell images. However, this process is tedious and time-consuming, and the diagnostic results are subjective, leading to potential misdiagnosis and underdiagnosis. This paper proposes a blood cell image classification method that combines MAE with an enhanced Vision Transformer to tackle these challenges. Initially, pre-training occurs on two datasets, TMAMD and Red4, using the MAE self-supervised learning algorithm. Subsequently, the pre-training weights are transferred to our improved model.This paper introduces feature fusion of the outputs from each layer of the Transformer encoder to maximize the utilization of features extracted from lower layers, such as color, contour, and texture of blood cells, along with deeper semantic features. Furthermore, the dynamic margins for the subcenter Arcface Loss function are employed to enhance the model’s fine-grained feature representation by achieving inter-class dispersion and intra-class aggregation. Models trained using our method achieved state-of-the-art results on both the TMAMD dataset and Red4 dataset, with classification accuracies of 93.51% and 81.41%, respectively. This achievement is expected to be a valuable reference for physicians in their clinical diagnoses.

Intensity grading of kidney biopsy direct immunofluorescence IgG image via semantically enhanced feature network

The intensity of direct immunofluorescence IgG imaging for kidney biopsy is an important reference index for the diagnosis of common chronic kidney diseases (CKD) such as membranous nephropathy (MN). Although this index is widely used in diagnosing membranous nephropathy, the intensity levels are currently judged only by the subjective perception of fluorescence luminance in pathological images through physicians’ naked eyes. Therefore, consistent intensity level classification cannot be achieved. To address this issue, this paper proposes a Preprocessing Semantically Enhanced Feature image classification method. The segmentation network is used to segment the original kidney biopsy direct immunofluorescence IgG images to isolate the diseased glomeruli. These segmented glomeruli images are then compared with the original glomeruli images to standardize the kidney biopsy direct immunofluorescence images. Simultaneously, considering the characteristics of small inter-class differences and large intra-class differences in fine-grained images, a semantically enhanced feature image classification model is employed to automatically classify the intensity of the standardized images. The method was evaluated on a constructed real clinical dataset, and the experimental results demonstrated that the proposed method achieved high classification accuracy on this dataset. The source code is made available on https://github.com/SnowRain510/PSEF-NET

Improved fine-grained image classification in few-shot learning based on channel-spatial attention and grouped bilinear convolution

Improved fine-grained image classification in few-shot learning based on channel-spatial attention and grouped bilinear convolution

CSDNet: cross-sketch with dual gated attention for fine-grained image captioning network

CSDNet: cross-sketch with dual gated attention for fine-grained image captioning network

Enhancing representation in radiography-reports foundation model: a granular alignment algorithm using masked contrastive learning

Recently, multi-modal vision-language foundation models have gained significant attention in the medical field. While these models offer great opportunities, they still face crucial challenges, such as the requirement for fine-grained knowledge understanding in computer-aided diagnosis and the capability of utilizing very limited or even no task-specific labeled data in real-world clinical applications. In this study, we present MaCo, a masked contrastive chest X-ray foundation model that tackles these challenges. MaCo explores masked contrastive learning to simultaneously achieve fine-grained image understanding and zero-shot learning for a variety of medical imaging tasks. It designs a correlation weighting mechanism to adjust the correlation between masked chest X-ray image patches and their corresponding reports, thereby enhancing the model’s representation learning capabilities. To evaluate the performance of MaCo, we conducted extensive experiments using 6 well-known open-source X-ray datasets. The experimental results demonstrate the superiority of MaCo over 10 state-of-the-art approaches across tasks such as classification, segmentation, detection, and phrase grounding. These findings highlight the significant potential of MaCo in advancing a wide range of medical image analysis tasks.

Bi-Directional Ensemble Feature Reconstruction Network for Few-Shot Fine-Grained Classification.

The main challenge for fine-grained few-shot image classification is to learn feature representations with higher inter-class and lower intra-class variations, with a mere few labelled samples. Conventional few-shot learning methods however cannot be naively adopted for this fine-grained setting - a quick pilot study reveals that they in fact push for the opposite (i.e., lower inter-class variations and higher intra-class variations). To alleviate this problem, prior works predominately use a support set to reconstruct the query image and then utilize metric learning to determine its category. Upon careful inspection, we further reveal that such unidirectional reconstruction methods only help to increase inter-class variations and are not effective in tackling intra-class variations. In this paper, we introduce a bi-reconstruction mechanism that can simultaneously accommodate for inter-class and intra-class variations. In addition to using the support set to reconstruct the query set for increasing inter-class variations, we further use the query set to reconstruct the support set for reducing intra-class variations. This design effectively helps the model to explore more subtle and discriminative features which is key for the fine-grained problem in hand. Furthermore, we also construct a self-reconstruction module to work alongside the bi-directional module to make the features even more discriminative. We introduce the snapshot ensemble method in the episodic learning strategy - a simple trick to further improve model performance without increasing training costs. Experimental results on three widely used fine-grained image classification datasets, as well as general and cross-domain few-shot image datasets, consistently show considerable improvements compared with other methods.

AI-Vision: A Three-Layer Accessible Image Exploration System for People with Visual Impairments in China

Accessible image exploration systems are able to help people with visual impairments (PVI) to understand image content by providing different types of interactions. With the development of computer vision technologies, image exploration systems are supporting more fine-grained image content processing, including image segmentation, description and object recognition. However, in developing countries like China, it is still rare for PVI to widely rely on such accessible system. To better understand the usage situation of accessible image exploration system in China and improve the image understanding of PVI in China, we developed AI-Vision, an Android based hierarchical accessible image exploration system supporting the generations of image general description, local object description and metadata information. Our 7-day diary study with 10 PVI verified the usability of AI-Vision and also revealed a series of design implications for improving accessible image exploration systems similar to AI-Vision.

DKA-RG: Disease-Knowledge-Enhanced Fine-Grained Image–Text Alignment for Automatic Radiology Report Generation

Automatic radiology report generation is a task that combines artificial intelligence and medical information processing, and it fully relies on computer vision and natural language processing techniques. Nowadays, automatic radiology report generation is still a very challenging task because it requires semantically adequate alignment of data from two modalities: radiology images and text. Existing approaches tend to focus on coarse-grained alignment at the global level and do not take into account the disease characteristics of radiology images at fine-grained semantics, which results in the generated reports potentially omitting key disease diagnostic descriptions. In this work, we propose a new approach, disease-knowledge-enhanced fine-grained image–text alignment for automatic radiology report generation (DKA-RG). The method combines global and disease-level alignment, thus facilitating the extraction of fine-grained disease features by the model. Our approach also introduces a knowledge graph to inject medical domain expertise into the model. Our proposed DKA-RG consists of two training steps: the image–report alignment stage and the image-to-report generation stage. In the alignment stage, we use global contrastive learning to align images and texts from a high level and also augment disease contrastive learning with medical knowledge to enhance the disease detection capability. In the report generation stage, the report text generated from the images is more accurate in describing the disease information thanks to sufficient alignment. Through extensive quantitative and qualitative experiments on two widely used datasets, we validate the effectiveness of our DKA-RG on the task of radiology report generation. Our DKA-RG achieves superior performance on multiple types of metrics (natural language generation and clinical efficacy metrics) compared to existing methods, demonstrating that the method can improve the reliability and accuracy of automatic radiology report generation systems.

Attribute-Driven Filtering: A new attributes predicting approach for fine-grained image captioning

Fine-grained image captioning with attribute information has garnered significant attention in the realms of computer vision and natural language processing, demanding precise and contextually relevant descriptions of visual content. While previous attribute-driven image captioning models have shown improvements, challenges remain, such as the independence of attribute predictors and caption generators and the semantic gap between images and attributes. Another common issue is the inclusion of all attributes at every time step, despite most attributes being irrelevant to the word currently being generated. This can divert the model’s attention toward erroneous semantic details, resulting in a performance decline. To address these issues, we propose a novel Attribute-Driven Filtering (ADF) captioning network designed to provide rich and nuanced descriptions. This model incorporates a unique Attribute Predictor Module (APM) that dynamically predicts the most pertinent attributes in accordance with the textual context, utilizing different attributes at various time steps. The novelty of this approach lies in recognizing that not all attributes hold equal relevance at each time step, and the APM filters out irrelevant attributes to generate precise and contextually relevant captions. Furthermore, this model features a fusion mechanism that integrates visual information from a conventional attention module with attribute information predicted by the APM, aiming to reduce the visual semantic gap between images and attributes. Extensive experimentation demonstrates that the ADF model outperforms advanced models, achieving impressive CIDEr-D scores of 72.0 (Flickr30K) and 123.3 (MS-COCO) through reinforcement learning optimization. It consistently surpasses baseline models across diverse evaluation metrics, highlighting its effectiveness and robustness.

Orchid2024: A cultivar-level dataset and methodology for fine-grained classification of Chinese Cymbidium Orchids

BackgroundChinese Cymbidium orchids, cherished for their deep-rooted cultural significance and significant economic value in China, have spawned a rich tapestry of cultivars. However, these orchid cultivars are facing challenges from insufficient cultivation practices and antiquated techniques, including cultivar misclassification, complex identification, and the proliferation of counterfeit products. Current commercial techniques and academic research primarily emphasize species identification of orchids, rather than delving into that of orchid cultivars within species.ResultsTo bridge this gap, the authors dedicated over a year to collecting a cultivar image dataset for Chinese Cymbidium orchids named Orchid2024. This dataset contains over 150,000 images spanning 1,275 different categories, involving visits to 20 cities across 12 provincial administrative regions in China to gather pertinent data. Subsequently, we introduced various visual parameter-efficient fine-tuning (PEFT) methods to expedite model development, achieving the highest top-1 accuracy of 86.14% and top-5 accuracy of 95.44%.ConclusionExperimental results demonstrate the complexity of the dataset while highlighting the considerable promise of PEFT methods within flower image classification. We believe that our work not only provides a practical tool for orchid researchers, growers and market participants, but also provides a unique and valuable resource for further exploring fine-grained image classification tasks. The dataset and code are available at https://github.com/pengyingshu/Orchid2024.

Ocular Disease Detection with Deep Learning (Fine-Grained Image Categorization) Applied to Ocular B-Scan Ultrasound Images.

The aim of this work is to develop a deep learning (DL) system for rapidly and accurately screening for intraocular tumor (IOT), retinal detachment (RD), vitreous hemorrhage (VH), and posterior scleral staphyloma (PSS) using ocular B-scan ultrasound images. Ultrasound images from five clinically confirmed categories, including vitreous hemorrhage, retinal detachment, intraocular tumor, posterior scleral staphyloma, and normal eyes, were used to develop and evaluate a fine-grained classification system (the Dual-Path Lesion Attention Network, DPLA-Net). Images were derived from five centers scanned by different sonographers and divided into training, validation, and test sets in a ratio of 7:1:2. Two senior ophthalmologists and four junior ophthalmologists were recruited to evaluate the system's performance. This multi-center cross-sectional study was conducted in six hospitals in China. A total of 6054 ultrasound images were collected; 4758 images were used for the training and validation of the system, and 1296 images were used as a testing set. DPLA-Net achieved a mean accuracy of 0.943 in the testing set, and the area under the curve was 0.988 for IOT, 0.997 for RD, 0.994 for PSS, 0.988 for VH, and 0.993 for normal. With the help of DPLA-Net, the accuracy of the four junior ophthalmologists improved from 0.696 (95% confidence interval [CI] 0.684-0.707) to 0.919 (95% CI 0.912-0.926, p < 0.001), and the time used for classifying each image reduced from 16.84 ± 2.34s to 10.09 ± 1.79s. The proposed DPLA-Net showed high accuracy for screening and classifying multiple ophthalmic diseases using B-scan ultrasound images across mutiple centers. Moreover, the system can promote the efficiency of classification by ophthalmologists.