Fine-grained Image Research Articles

Associating multiple vision transformer layers for fine-grained image representation

- Accurate discriminative region proposal has an important effect for fine-grained image recognition. The vision transformer (ViT) brings about a striking effect in computer vision due to its innate multi-head self-attention mechanism. However, the attention maps are gradually similar after certain layers, and since ViT used a classification token to achieve classification, it is unable to effectively select discriminative image patches for fine-grained image classification. To accurately detect discriminative regions, we propose a novel network AMTrans, which efficiently increases layers to learn diverse features and utilizes integrated raw attention maps to capture more salient features. Specifically, we employ DeepViT as backbone to solve the attention collapse issue. Then, we fuse each head attention weight within each layer to produce an attention weight map. After that, we alternatively use recurrent residual refinement blocks to promote salient feature and then utilize the semantic grouping method to propose the discriminative feature region. A lot of experiments prove that AMTrans acquires the SOTA performance on four widely used fine-grained datasets under the same settings, involving Stanford-Cars, Stanford-Dogs, CUB-200-2011, and ImageNet.

Chest X-Ray Quality Assessment Method With Medical Domain Knowledge Fusion

Digital X-ray radiography is widely used in clinical diagnosis. High quality chest X-ray is conducive to the accurate diagnosis of diseases by clinicians. However, the quality assessment of the chest X-ray images mainly depends on the subjective evaluation of doctors, and the results are influenced by the skill level and experience of the evaluators, involving many issues such as heavy workload and various uncertain factors in such subjective judgment. In this paper, we propose a chest X-ray quality assessment method that combines image-text contrastive learning with medical domain knowledge fusion. Based on pretraining the model from contrastive text-image pairs, large-scale real clinical chest X-ray and diagnostic report text information are fused, and the model is fine-tuned to achieve cross domain transfer learning. While improving the prediction accuracy of the algorithm, the cost of massive sample data annotation is avoided. The local visual patch features of the X-ray images are aligned with multiple text features to ensure that the visual features contain more fine-grained image information. Theoretical analysis and experimental results show that the contrastive learning algorithm based on the fusion of triplet information in medical knowledge graph and chest X-ray multi-modal data has achieved good performance in terms of accuracy. In addition, the method proposed in this paper can be easily extended to complete other tasks such as medical image multi-lesion segmentation and disease progression prediction.

EFM-Net: An Essential Feature Mining Network for Target Fine-Grained Classification in Optical Remote Sensing Images

Target fine-grained classification has been the research hotspot in remote sensing image interpretation, which has received general attention in application fields. One challenge of the fine-grained classification task is to learn the most discriminative feature using the deep convolutional neural network (DCNN). At present, many works of fine-grained image classification obtain target features by optimizing the feature extraction and enhancement, which are not accurate enough in remote sensing images. In this paper, we propose an essential feature mining network (EFM-net for short) based on DCNN to address this issue. Its major motivation is to obtain the essential feature which is fine enough to distinguish between similar instances. The proposed pipeline includes the Miner for purifying the essential feature and the Refiner for data augmentation. These two modules can work in a mutually reinforcing way, and extract the essential feature of targets. We evaluate EFM-Net on two public fine-grained classification datasets in remote sensing, FGSC-23 and FGSCR-42, and our Aircraft-16. The results show that the proposed method consistently outperforms existing alternatives. We have released our source code in Github https://github.com/JACYI/EFM-Net-Pytorch.git.

The problem of automatic classification of pictures using an intelligent decision-making system based on the knowledge graph and fine-grained image analysis

In order to prevent the illegal export of paintings abroad, a museum examination using various methods for studying a work of art is carried out. At the same time, an analysis is also made of historical, art history, financial and other information and documents confirming the painting’s authenticity — provenance. Automation of such examination is hampered by the need to take into account numerical values of visual features, quality indicators, and verbal descriptions from provenance. In this paper, we consider the problem of automatic multi-task classification of paintings for museum expertise. A system architecture is proposed that checks provenance, implements a fine-grained image analysis (FGIA) of visual image features, and automatically classifies a painting by authorship, genre, and time of creation. Provenance is contained in a knowledge graph; for its vectorization, it is proposed to use a graph2vec type encoder with an attention mechanism. Fine-grained image analysis is proposed to be performed using searching discriminative regions (SDR) and learning discriminative regions (LDR) allocated by convolutional neural networks. To train the classifier, a generalized loss function is proposed. A data set is also proposed, including provenance and images of paintings by European and Ukrainian artists.

Fine-Grained Image Classification Based on Cross-Attention Network

Due to the high similarity of fine-grained image subclasses, small inter-class changes and large intra-class changes are caused, which leads to the difficulty of fine-grained image classification task. However, existing convolutional neural networks have been unable to effectively solve this problem. Aiming at the above-mentioned fine-grained image classification problem, this paper proposes a multi-scale and multi-level ViT model. First, through data augmentation techniques, the accuracy of fine-grained image classification can be effectively improved. Secondly, the small-scale input and large-scale input of the model make the input image have more feature ex-pressions. The subsequent multi-layeredness effectively utilizes the results of the previous layer of ViT, so that the data of the previous layer can be more effectively used in the next layer of ViT. Finally, cross-attention allows the results of two scale inputs to be fused in a reasonable way. The proposed model is competitive with current mainstream state-of-the-art methods on multiple datasets.

DRFENet: An Improved Deep Learning Neural Network via Dilated Skip Convolution for Image Denoising Application

Deep learning technology dominates current research in image denoising. However, denoising performance is limited by target noise feature loss from information propagation in association with the depth of the network. This paper proposes a Dense Residual Feature Extraction Network (DRFENet) combined with a Dense Enhancement Block (DEB), a Residual Dilated Block (RDB), a Feature Enhancement Block (FEB), and a Simultaneous Iterative Reconstruction Block (SIRB). The DEB uses our proposed interval transmission strategy to enhance the extraction of noise features in the initial stage of the network. The RDB module uses a combination strategy of concatenated dilated convolution and a skip connection, and the local features are amplified through different perceptual dimensions. The FEB enhances local feature information. The SIRB uses an attention block to learn the noise distribution while using residual learning (RL) technology to reconstruct a denoised image. The combination strategy in DRFENet makes the neural network deeper to obtain higher fine-grained image information. We respectively examined the performance of DRFENet in gray image denoising on datasets BSD68 and SET12 and color image denoising on datasets McMaster, Kodak24, and CBSD68. The experimental results showed that the denoising accuracy of DRFENet is better than most existing image-denoising methods under PSNR and SSIM evaluation indicators.

Webly Supervised Fine-Grained Image Recognition with Graph Representation and Metric Learning

The aim of webly supervised fine-grained image recognition (FGIR) is to distinguish sub-ordinate categories based on data retrieved from the Internet, which can significantly mitigate the dependence of deep learning on manually annotated labels. Most current fine-grained image recognition algorithms use a large-scale data-driven deep learning paradigm, which relies heavily on manually annotated labels. However, there is a large amount of weakly labeled free data on the Internet. To utilize fine-grained web data effectively, this paper proposes a Graph Representation and Metric Learning (GRML) framework to learn discriminative and effective holistic–local features by graph representation for web fine-grained images and to handle noisy labels simultaneously, thus effectively using webly supervised data for training. Specifically, we first design an attention-focused module to locate the most discriminative region with different spatial aspects and sizes. Next, a structured instance graph is constructed to correlate holistic and local features to model the holistic–local information interaction, while a graph prototype that contains both holistic and local information for each category is introduced to learn category-level graph representation to assist in processing the noisy labels. Finally, a graph matching module is further employed to explore the holistic–local information interaction through intra-graph node information propagation as well as to evaluate the similarity score between each instance graph and its corresponding category-level graph prototype through inter-graph node information propagation. Extensive experiments were conducted on three webly supervised FGIR benchmark datasets, Web-Bird, Web-Aircraft and Web-Car, with classification accuracy of 76.62%, 85.79% and 82.99%, respectively. In comparison with Peer-learning, the classification accuracies of the three datasets separately improved 2.47%, 4.72% and 1.59%.

Fine-Grained Image Analysis With Deep Learning: A Survey

Fine-grained image analysis (FGIA) is a longstanding and fundamental problem in computer vision and pattern recognition, and underpins a diverse set of real-world applications. The task of FGIA targets analyzing visual objects from subordinate categories, e.g., species of birds or models of cars. The small inter-class and large intra-class variation inherent to fine-grained image analysis makes it a challenging problem. Capitalizing on advances in deep learning, in recent years we have witnessed remarkable progress in deep learning powered FGIA. In this paper we present a systematic survey of these advances, where we attempt to re-define and broaden the field of FGIA by consolidating two fundamental fine-grained research areas - fine-grained image recognition and fine-grained image retrieval. In addition, we also review other key issues of FGIA, such as publicly available benchmark datasets and related domain-specific applications. We conclude by highlighting several research directions and open problems which need further exploration from the community.

Fine-grained Vehicle Classification Based on Feature Augmentation

Fine-grained vehicle classification is a hot research topic in the field of computer vision, which aims to recognize different models of vehicles belonging to the same brand. Limited by the large appearance differences of the same vehicle (due to different attitudes and perspectives) and small appearance differences between different vehicles, the vehicle’s reorganization performance is not high and can’t meet the real application requirements. In this paper, a Fine-grained Vehicle Classification is proposed based on Feature Augmentation (FAFCC), which focuses on mining the key discriminative parts of the object. Specifically, the FAFCC first improves the weakly supervised data augmentation network (WSDAN) by using multi-layer feature extraction and fusion instead of single-layer features. Further, the FAFCC assigns local feature weights to select key features, which can solve the problem that the discriminant features in the channel are not prominent enough and realize the high-quality compact fine-grained vehicle recognition task. The performance of the FAFCC on the fine-grained image classification tasks for vehicles has been evaluated through algorithm comparison experiments and ablation experiments. Through visual analysis, it can be seen that the introduction of each new module has optimized the loss function value and convergence speed of the model. In general, our FAFCC algorithm has a certain improvement effect compared with the original algorithm.

Fine-grained image classification of microscopic insect pest species: Western Flower thrips and Plague thrips

Accurate identification of insect pests is essential in crop management as they are one of the primary causes of yield losses. However, differences between insect species demand different pest control strategies. Hence, research on new technology for fine-grained classification of insect pests is potentially important. Morphologically similar microscopic pest species classification has received little attention in the literature, and is targeted by this study as a means to address the need for agricultural pest management. We propose a novel computational method for deep learning-based, fine-grained classification of microscopic insects using the Vision Transform (ViT) architecture. This architecture employs an attention mechanism motivated by domain knowledge. The proposed approach consists of two main modules, a Data Preprocessing Module to segment relevant insect features and split the insect into body segments to inform identification, and a Domain Knowledge-Driven Stacked Model based on ViT to generate the prediction from each body segment and to fuse predictions for each segment into an accurate species-level classification. We tested the approach using an image dataset of two economically devastating thrip species – Western Flower thrips (Frankliniella occidentalis) and Plague thrips (Thrips imaginis). These insects are small (∼1mm), exhibit minute inter-species differences, and require different pest control strategies. We compared our model with the original ViT model, RestNet101, and RestNet50. Experimental results achieve an F1-score of 0.978, a 3.27% improvement over the baselines. This is important in the horticultural context given the yield losses that these pest insects are known to cause if their populations remain incorrectly quantified.

Tool Wear Monitoring in Milling Based on Fine-Grained Image Classification of Machined Surface Images.

Cutting tool wear state assessment during the manufacturing process is extremely significant. The primary purpose of this study is to monitor tool wear to ensure timely tool change and avoid excessive tool wear or sudden tool breakage, which causes workpiece waste and could even damage the machine. Therefore, an intelligent system, that is efficient and precise, needs to be designed for addressing these problems. In our study, an end-to-end improved fine-grained image classification method is employed for workpiece surface-based tool wear monitoring, which is named efficient channel attention destruction and construction learning (ECADCL). The proposed method uses a feature extraction module to extract features from the input image and its corrupted images, and adversarial learning is used to avoid learning noise from corrupted images while extracting semantic features by reconstructing the corrupted images. Finally, a decision module predicts the label based on the learned features. Moreover, the feature extraction module combines a local cross-channel interaction attention mechanism without dimensionality reduction to characterize representative information. A milling dataset is conducted based on the machined surface images for monitoring tool wear conditions. The experimental results indicated that the proposed system can effectively assess the wear state of the tool.

Automated Bone Age Assessment: A New Three-Stage Assessment Method from Coarse to Fine.

Bone age assessment (BAA) based on X-ray imaging of the left hand and wrist can accurately reflect the degree of the body's physiological development and physical condition. However, the traditional manual evaluation method relies too much on inefficient specialist labor. In this paper, to propose automatic BAA, we introduce a hierarchical convolutional neural network to detect the regions of interest (ROI) and classify the bone grade. Firstly, we establish a dataset of children's BAA containing 2518 left hand X-rays. Then, we use the fine-grained classification to obtain the grade of the region of interest via object detection. Specifically, fine-grained classifiers are based on context-aware attention pooling (CAP). Finally, we perform the model assessment of bone age using the third version of the Tanner-Whitehouse (TW3) methodology. The end-to-end BAA system provides bone age values, the detection results of 13 ROIs, and the bone maturity of the ROIs, which are convenient for doctors to obtain information for operation. Experimental results on the public dataset and clinical dataset show that the performance of the proposed method is competitive. The accuracy of bone grading is 86.93%, and the mean absolute error (MAE) of bone age is 7.68 months on the clinical dataset. On public dataset, the MAE is 6.53 months. The proposed method achieves good performance in bone age assessment and is superior to existing fine-grained image classification methods.

Fine-grained image recognition via trusted multi-granularity information fusion

Fine-grained image recognition via trusted multi-granularity information fusion

Fish image recognition method based on multi-layer feature fusion convolutional network

Solving the problem of fish image classification is important to conserve fish diversity. This conundrum can be addressed by developing a new fish image classification method based on deep learning by training data with complex backgrounds. To this end, this paper proposes a fusion model, referred to as Tripmix-Net. The backbone network of the proposed model primarily consists of multiscale parallel and improved residual networks that are connected in an alternate manner, and network fusion is used to integrate the information that is extracted from shallow and deep layers. Experiments conducted on the 15-category WildFish fish image dataset verified the efficacy of the proposed Tripmix-Net for classifying same-genus fish images with complex backgrounds. The model achieved an accuracy of 95.31%, which is a significant improvement over traditional methods. The proposed approach serves as a new concept for the fine-grained image classification of fish against complex backgrounds.

Robust fine-grained image classification with noisy labels

Robust fine-grained image classification with noisy labels

Graph-based discriminative features learning for fine-grained image retrieval

Fine-grained image retrieval has gradually become a hot topic in computer vision , which aims to retrieve images with the same subcategories from general visual categories. Though fine-grained image retrieval has made a breakthrough with the development of convolutional neural networks, its performance is still limited by the low discriminative feature embedding. To solve this problem, most prior works focus on mining more discriminative features with various strategies. In this paper, we propose a novel graph-based discriminative features learning network for fine-grained image retrieval (GDF-Net). We first design a global fine-grained feature aggregation module, which reconstructs the discriminative features through capturing context correlation based on a K-Nearest Neighbor graph. To reduce storage overhead and speed up retrieval, we further design a semantic hash encoding module, which generates a semantically compact hash code under the guidance of Cauchy quantization loss and bit balance loss. Validated by extensive experiments and ablation studies, our method consistently outperforms state-of-the-art generic retrieval methods as well as fine-grained retrieval methods on three datasets, e.g., CUB Birds, Stanford Dogs and Stanford Cars.

Fine-grained Image Recognition Method using Discriminative Region-based Data Augmentation

In order to reduce the complexity of the network model and improve the accuracy of image recognition, a fine-grained image recognition method using discriminative region-based data augmentation is proposed. The method obtains the discriminative regions of the image through the attention mechanism, and then performs diversity data augmentation based on the discriminative regions, including region crop, region drop and region mix, and then uses the generated augmented samples to train the network model, the backbone network of the model is ResNet50. The proposed method is tested on 4 commonly used fine-grained image datasets CUB-200-2011, Stanford Cars, FGVC Aircraft and Stanford Dogs, and achieves high accuracy. The experimental results show that the proposed method can improve the localization ability and feature extraction ability of the model for discriminative regions, and it is more lightweight and easier to implement.

Few-Shot Fine-Grained Image Classification via GNN.

Traditional deep learning methods such as convolutional neural networks (CNN) have a high requirement for the number of labeled samples. In some cases, the cost of obtaining labeled samples is too high to obtain enough samples. To solve this problem, few-shot learning (FSL) is used. Currently, typical FSL methods work well on coarse-grained image data, but not as well on fine-grained image classification work, as they cannot properly assess the in-class similarity and inter-class difference of fine-grained images. In this work, an FSL framework based on graph neural network (GNN) is proposed for fine-grained image classification. Particularly, we use the information transmission of GNN to represent subtle differences between different images. Moreover, feature extraction is optimized by the method of meta-learning to improve the classification. The experiments on three datasets (CIFAR-100, CUB, and DOGS) have shown that the proposed method yields better performances. This indicates that the proposed method is a feasible solution for fine-grained image classification with FSL.

Learning enhanced features and inferring twice for fine-grained image classification

Fine-Grained Visual Categorization (FGVC) aims to distinguish between extremely similar subordinate-level categories within the same basic-level category. Existing research has proven the great importance of the discriminative features in FGVC but ignored the contributions for correct classification from other features, and the extracted features always contain more information about the obvious regions but less about subtle regions. In this paper, firstly, a novel module named forcing module is proposed to force the network to extract more diverse features for FGVC, which generates a suppression mask based on the class activation maps to suppress the most distinguishable regions, so as to force the network to extract other secondary distinguishable features as the final features. The forcing module consists of the original branch and the forcing branch. The original branch focuses on the primary discriminative regions while the forcing branch focuses on secondary discriminative regions. Secondly, in order to solve the problem that information of small-scale distinguishable features is lost seriously after multi-layer down-sampling, according to the class activation maps of the first prediction, the object is cropped and scaled as the second input. To reduce the prediction error, the first and second prediction probabilities are fused as the final prediction result. Experimental results indicate that the proposed method not only outperforms the baseline model by a large margin (3.7%, 5.9%, 3.1% respectively) on CUB-200-2011, Stanford-Cars, and FGVC-Aircraft, but also achieves state-of-the-art performance on FGVC-Aircraft.

Fine-grained image retrieval by combining attention mechanism and context information

Fine-grained image retrieval by combining attention mechanism and context information