Fine-grained Image Research Articles

Image Recognition of Mine Water Inrush Based on Bilinear Convolutional Neural Network with Few-Shot Learning.

With the increasingly widespread application of deep learning technology in the field of coal mines, the image recognition of mine water inrush has become a hot research topic. Underground environments are complex, and images have a high noise and low brightness. Additionally, mine water inrush is accidental, and few actual image samples are available. Therefore, this paper proposes an algorithm that recognizes mine water inrush images based on few-shot deep learning. According to the characteristics of images with coal wall water seepage, a bilinear neural network was used to extract the image features and enhance the network's fine-grained image recognition. First, features were extracted using a bilinear convolutional neural network. Second, the network was pre-trained based on cosine similarity. Finally, the network was fine-tuned for the predicted image. For single-line feature extraction, the method is compared with big data and few-shot learning. According to the experimental results, the recognition rate reaches 95.2% for few-shot learning based on a bilinear neural network, thus demonstrating its effectiveness.

GAN-based adaptive cost learning for enhanced image steganography security

A well-defined cost function is a key issue for image steganography to minimize the embedding distortion. In recent years, deep learning has been introduced into image steganography to automatically learn embedding costs and improve steganographic security. For most existing generative adversarial network (GAN) based cost learning works, the generator usually adopts an encoder–decoder architecture. However, due to repeated encoding and decoding operations, this architecture is prone to information loss, making the generator difficult to well capture fine-grained image features. In this work, we propose a novel GAN-based image steganography work that improves the cost function by learning better embedding probability maps. Specifically, we design an attention mechanism to be integrated into the U-Net architecture, which enables the generator to concentrate on texture-rich regions of input images. Moreover, an extra input stream, namely the enhanced image, is introduced into the generator, improving the generator’s ability to learn structural features from input images. Different skip connections are used for different input streams to facilitate information flow between different layers. Extensive experimental results demonstrate that the proposed approach effectively learns the embedding probability maps and achieves superior security against various steganalysis attacks.

Toward Automatic Typography Analysis: Serif Classification and Font Similarities

Whether a document is of historical or contemporary significance, typography plays a crucial role in its composition. From the early days of modern printing, typographic techniques have evolved and transformed, resulting in changes to the features of typography. By analyzing these features, we can gain insights into specific time periods, geographical locations, and messages conveyed through typography. Therefore, in this paper, we aim to investigate the feasibility of training a model to classify serif typeswithout knowledge of the font and character. We also investigate how to train a vectorial-based image model able to group together fonts with similar features. Specifically, we compare the use of state-of-theart image classification methods, such as the EfficientNet-B2 and the Vision Transformer Base model with different patch sizes, and the state-of-the-art fine-grained image classification method, TransFG, on the serif classification task. We also evaluate the use of the DeepSVG model to learn to group fonts with similar features. Our investigation reveals that fine-grained image classification methods are better suited for the serif classification tasks and that leveraging the character labels helps to learn more meaningful font similarities.This repository contains: - Paper published in the Journal of data mining and digital humanities:WasimEtAl_Toward_Automatic_Typography_Analysis__Serif_Classification_and_Font_Similarities.pdf - Two datasets: The first for serif classification consisting of 126666 training and 2914 font-independent testing images in raster format. The second dataset for svg based similarity learning consists of 124010 training and 2914 font-independent testing images. The images have been categorized into sans-serif, linear-serif, slab-serif, and triangular serif by a designer at the EPFL+ECAL Labfont_serif_dataset.zipfont_svg_dataset.zip - model weights:serif_B_16_long_checkpoint.bin: refers to TransFG ViT B/16 Backendserif_B_32_long_checkpoint.bin: refers to TransFG ViT B/32 BackendViT-B_16.npz ViT-B_32.npz are publicly available at: https://github.com/TACJu/TransFGand were used for training TransFG ViT B/16 Backend and TransFG ViT B/32 Backend - code:https://github.com/TalalWasim/GEST-Serif

Learning Relationship-Enhanced Semantic Graph for Fine-Grained Image-Text Matching.

Image-text matching of natural scenes has been a popular research topic in both computer vision and natural language processing communities. Recently, fine-grained image-text matching has shown its significant advance in inferring the high-level semantic correspondence by aggregating pairwise region-word similarity, but it remains challenging mainly due to insufficient representation of high-order semantic concepts and their explicit connections in one modality as its matched in another modality. To tackle this issue, we propose a relationship-enhanced semantic graph (ReSG) model, which can improve the image-text representations by learning their locally discriminative semantic concepts and then organizing their relationships in a contextual order. To be specific, two tailored graph encoders, visual relationship-enhanced graph (VReG) and textual relationship-enhanced graph (TReG), are respectively exploited to encode the high-level semantic concepts of corresponding instances and their semantic relationships. Meanwhile, the representations of each graph node are optimized by aggregating semantically contextual information to enhance the node-level semantic correspondence. Further, the hard-negative triplet ranking loss, center hinge loss, and positive-negative margin loss are jointly leveraged to learn the fine-grained correspondence between the ReSG representations of image and text, whereby the discriminative cross-modal embeddings can be explicitly obtained to benefit various image-text matching tasks in a more interpretable way. Extensive experiments verify the advantages of the proposed fine-grained graph matching approach, by achieving the state-of-the-art image-text matching results on public benchmark datasets.

ACANet: A Fine-grained Image Classification Optimization Method Based on Convolution and Attention Fusion

<p>The key to solve the problem of fine-grained image classification is to find the differentiation regions related to fine-grained features. In this paper, we try to add new network components to the original network and adjust various parameters to try to propose a new fine-grained image classification network. We propose a fine-grained image classification network based on the fusion of asymmetric convolution, convolution and self-attention mechanisms. Firstly, an enhanced module using asymmetric convolution to assist classical convolution proposed to help convolution learn deep features. Secondly, according to the common points of convolution and self-attention mechanism, we invented a fusion module of convolution and self-attention mechanism to improve the learning ability of the network.We integrate these two modules into the residual network and invent a new residual network .Finally, according to the experience, we design a new downsampling layer to adapt to the new component of the attention mechanism and improve the performance of the model. The experiment test on three publicly available datasets, and three methods for comparison. The results show that the new structure can effectively complete the task of fine-grained image classification, and the classification accuracy of different methods and different datasets are significantly improved.</p> <p> </p>

Duet of ViT and CNN: multi-scale dual-branch network for fine-grained image classification of marine organisms

Fine-grained image classification of marine organisms involves dividing subcategories within a larger category. For instance, this could mean distinguishing specific species of fish or types of algae. This type of classification is more intricate than regular image classification, as the minor feature differences between subcategories are often concentrated in one or a few specific areas. Therefore, accurately identifying these critical regions and effectively using local features are crucial in improving the accuracy of fine-grained image classification. Existing methods for fine-grained image classification primarily rely on single-branch models based on convolutional neural networks (CNNs) or vision transformers (ViTs). Consequently, merging them allows for a more comprehensive understanding of marine organism images. In addition, marine organism images are affected by the distance and angle of the shot, making it challenging to capture detailed local nuances at a single scale. To address these challenges, we propose a multi-scale dual-branch network (MSDBN) that combines the strengths of ViT and CNN for fine-grained image classification of marine organisms. Our model uses a novel two-stage selection module to select discriminative regions from the ViT branch. Following this, the CNN branch executes a more detailed feature extraction on the local regions. To effectively utilise the multi-scale information of marine organisms, we introduce our designed multi-scale shift-window self-attention, specifically for the ViT branch. MSDBN demonstrates improved performance compared to existing classical methods and the best-performing dual-branch methods on three marine datasets. Our code is released publicly at https://github.com/Xiaosigz/MSDBN.

IremulbNet: Rethinking the inverted residual architecture for image recognition

An increasing need of running Convolutional Neural Network (CNN) models on mobile devices encourages the studies on efficient and lightweight neural network model. In this paper, an Inverse Residual Multi-Branch Network named IremulbNet is proposed to solve the problem of insufficient classification accuracy in existing lightweight network models. The core module of this model is to reconstruct an inverse residual structure, in which a special feature fusion method, multi-branch feature extraction, and depthwise separable convolution techniques are used to improve the classification accuracy. The performance of model is tested using image databases. Experimental results show that for the fine-grained image dataset Imagenet-woof, IremulbNet achieved 10.9%, 12.2%, and 15.3% higher accuracy than that of MobileNet V3, ShuffleNet V2, and PeleeNet, respectively. Moreover, it can reduce inference time (GPU) about 42.09% and 75.56% compared to classic ResNet50 and DenseNet121.

Pyramid hybrid pooling quantization for efficient fine-grained image retrieval

Deep hashing approaches, including deep quantization and deep binary hashing, have become a common solution to large-scale image retrieval due to their high computation and storage efficiency. Most existing hashing methods cannot produce satisfactory results for fine-grained retrieval, because they usually adopt the outputs of the last CNN layer to generate binary codes. Since deeper layers tend to summarize visual clues, e.g., texture, into abstract semantics, e.g., dogs and cats, the feature produced by the last CNN layer is less effective in capturing subtle but discriminative visual details that mostly exist in shallow layers. To improve fine-grained image hashing, we propose Pyramid Hybrid Pooling Quantization (PHPQ). Specifically, we propose a Pyramid Hybrid Pooling (PHP) module to capture and preserve fine-grained semantic information from multi-level features, which emphasizes the subtle discrimination of different sub-categories. Besides, we propose a learnable quantization module with a partial codebook attention mechanism, which helps to optimize the most relevant codewords and improves the quantization. Comprehensive experiments on two widely-used public benchmarks, i.e., CUB-200-2011 and Stanford Dogs, demonstrate that PHPQ outperforms state-of-the-art methods.

Gradient aggregation based fine-grained image retrieval: A unified viewpoint for CNN and Transformer

The gradients of CNN are traditionally utilized for optimization and visualization. In this paper, we find that a discriminative representation hides in the gradients of convolution filters. Based on this, we propose a corresponding feature extraction and aggregation method for fine-grained image retrieval (FGIR). Firstly, we propose a metric to evaluate manually-designed loss functions and design a loss function originating from Grad-CAM in the testing phase based on it to extract the gradients of the convolution filters. Secondly, we take the gradients as the new features and design a succinct approach to aggregate them into a compact vector, which is named as Convolution Filters Gradient Aggregation (CFGA) feature. CFGA features can be extracted from pre-trained and fine-tuned CNN models. Extensive experiments are conducted on FGIR to verify the effectiveness of our proposed CFGA approach, compared with five supervised state-of-the-art methods and two unsupervised methods on two standard fine-grained retrieval datasets. Moreover, we generalize the CFGA method designed for CNN to Swin Transformer, and propose the Transformer parameter gradients aggregation (TPGA) method, which proves the applicability of the core idea of CFGA/TPGA to mainstream feature extraction models. We achieve state-of-the-art FGIR performance on CUB-200-2011 dataset and CARS196 dataset.

Deep Progressive Asymmetric Quantization Based on Causal Intervention for Fine-Grained Image Retrieval

In the field of computer vision, fine-grained image retrieval is an extremely challenging task due to the inherently subtle intra-class object variations. In addition, the high-dimensional real-valued features extracted from large-scale fine-grained image datasets slow the retrieval speed and increase the storage cost. To solve above issues, existing fine-grained image retrieval methods mainly focus on finding more discriminative local regions for generating discriminative and compact hash codes, which achieve limited fine-grained image retrieval performance due to the large quantization errors and the confounding granularities and context of discriminative parts, i.e., the correct recognition of fine-grained objects mainly attribute to the discriminative parts and their context. To learn robust causal features and reduce the quantization errors, we propose a deep progressive asymmetric quantization (DPAQ) method based on causal intervention to learn compact and robust descriptions for fine-grained image retrieval task. Specifically, we introduce a structural causal model to learn robust casual features via causal intervention for fine-grained visual recognition. Subsequently, we design a progressive asymmetric quantization layer in the feature embedding space, which can preserve the semantic information and reduce the quantization errors sufficiently. Finally, we incorporate both the fine-grained image classification and retrieval tasks into an end-to-end deep learning architecture for generating robust and compact descriptions. Experimental results on several fine-grained image retrieval datasets demonstrate that the proposed DPAQ method performs the best for fine-grained image retrieval task and surpasses the state-of-the art fine-grained hashing methods by a large margin.

Fine-grained Recognition with Learnable Semantic Data Augmentation.

Fine-grained image recognition is a longstanding computer vision challenge that focuses on differentiating objects belonging to multiple subordinate categories within the same meta-category. Since images belonging to the same meta-category usually share similar visual appearances, mining discriminative visual cues is the key to distinguishing fine-grained categories. Although commonly used image-level data augmentation techniques have achieved great success in generic image classification problems, they are rarely applied in fine-grained scenarios, because their random editing-region behavior is prone to destroy the discriminative visual cues residing in the subtle regions. In this paper, we propose diversifying the training data at the feature-level to alleviate the discriminative region loss problem. Specifically, we produce diversified augmented samples by translating image features along semantically meaningful directions. The semantic directions are estimated with a covariance prediction network, which predicts a sample-wise covariance matrix to adapt to the large intra-class variation inherent in fine-grained images. Furthermore, the covariance prediction network is jointly optimized with the classification network in a meta-learning manner to alleviate the degenerate solution problem. Experiments on four competitive fine-grained recognition benchmarks (CUB-200-2011, Stanford Cars, FGVC Aircrafts, NABirds) demonstrate that our method significantly improves the generalization performance on several popular classification networks (e.g., ResNets, DenseNets, EfficientNets, RegNets and ViT). Combined with a recently proposed method, our semantic data augmentation approach achieves state-of-the-art performance on the CUB-200-2011 dataset. The source code will be released.

The Image Data and Backbone in Weakly Supervised Fine-Grained Visual Categorization: A Revisit and Further Thinking

Weakly-supervised fine-grained visual categorization (FGVC) aims to achieve subclass classification within the same large class using only label information. Compared to general images, fine-grained images have similar appearances and features, and are often affected by disturbances such as viewpoint, lighting, and occlusion during data collection, resulting in significant intra-class variance and small inter-class variance. To achieve FGVC, carefully designed models are often needed to explore the locally discriminative regions of the image. This paper revisits high-quality FGVC publications based on deep learning and analyzes from two new perspective: fine-grained image data and backbone. We address two ignored but interesting problems in FGVC. First, we argue that the reasons for exacerbating intra-class variance are not the same in data of animal, plant, and commodity types, and it is necessary to consider the effects of posture, covariate shift, and structural changes. Additionally, the “soft boundary” between subclasses intensifies the difficulty of classification. Second, we highlight that convolutional networks and self-attention networks have different receptive fields and shape biases, leading to performance differences when processing different types of fine-grained data. Overall, our analysis provides new insights into recent advances, challenges, and future directions for FGVC based on deep learning, which can help researchers develop more effective models for FGVC.

Alleviating Over-Fitting in Hashing-Based Fine-Grained Image Retrieval: From Causal Feature Learning to Binary-Injected Hash Learning

Alleviating Over-Fitting in Hashing-Based Fine-Grained Image Retrieval: From Causal Feature Learning to Binary-Injected Hash Learning

Autonomous novel class discovery for vision-based recognition in non-interactive environments

Visual recognition with deep learning has recently been shown to be effective in robotic vision. However, these algorithms tend to be build under fixed and structured environment, which is rarely the case in real life. When facing unknown objects, avoidance or human interactions are required, which may miss critical objects or be prohibitively costly to obtain on robots in the real world. We consider a practical problem setting that aims to allow robots to automatically discover novel classes with only labelled known class samples in hand, defined as open-set clustering (OSC). To address the OSC problem, we propose a framework combining three approaches: 1) using selfsupervised vision transformers to mitigate the discard of information needed for clustering unknown classes; 2) adaptive weighting for image patches to prioritize patches with richer textures; and 3) incorporating a temperature scaling strategy to generate more separable feature embeddings for clustering. We demonstrate the efficacy of our approach in six fine-grained image datasets.

Robust Fine-Grained Visual Recognition With Neighbor-Attention Label Correction.

Existing deep learning methods for fine-grained visual recognition often rely on large-scale, well-annotated training data. Obtaining fine-grained annotations in the wild typically requires concentration and expertise, such as fine category annotation for species recognition, instance annotation for person re-identification (re-id) and dense annotation for segmentation, which inevitably leads to label noise. This paper aims to tackle label noise in deep model training for fine-grained visual recognition. We propose a Neighbor-Attention Label Correction (NALC) model to correct labels during the training stage. NALC samples a training batch and a validation batch from the training set. It hence leverages a meta-learning framework to correct labels in the training batch based on the validation batch. To enhance the optimization efficiency, we introduce a novel nested optimization algorithm for the meta-learning framework. The proposed training procedure consistently improves label accuracy in the training batch, consequently enhancing the learned image representation. Experimental results demonstrate that our method significantly increases label accuracy from 70% to over 98% and outperforms recent approaches by up to 13.4% in mean Average Precision (mAP) on various fine-grained image retrieval (FGIR) tasks, including instance retrieval on CUB200 and person re-id on Market1501. We also demonstrate the efficacy of NALC on noisy semantic segmentation datasets generated from Cityscapes, where it achieves a significant 7.8% improvement in mIOU score. NALC also exhibits robustness to different types of noise, including simulated noise such as Asymmetric, Pair-Flip, and Pattern noise, as well as practical noisy labels generated by tracklets and clustering.

Angular Isotonic Loss Guided Multi-Layer Integration for Few-Shot Fine-Grained Image Classification.

Recent research on few-shot fine-grained image classification (FSFG) has predominantly focused on extracting discriminative features. The limited attention paid to the role of loss functions has resulted in weaker preservation of similarity relationships between query and support instances, thereby potentially limiting the performance of FSFG. In this regard, we analyze the limitations of widely adopted cross-entropy loss and introduce a novel Angular ISotonic (AIS) loss. The AIS loss introduces an angular margin to constrain the prototypes to maintain a certain distance from a pre-set threshold. It guides the model to converge more stably, learn clearer boundaries among highly similar classes, and achieve higher accuracy faster with limited instances. Moreover, to better accommodate the feature requirements of the AIS loss and fully exploit its potential in FSFG, we propose a Multi-Layer Integration (MLI) network that captures object features from multiple perspectives to provide more comprehensive and informative representations of the input images. Extensive experiments demonstrate the effectiveness of our proposed method on four standard fine-grained benchmarks. Codes are available at: https://github.com/Legenddddd/AIS-MLI.

Improving the Conditional Fine-Grained Image Generation With Part Perception

Improving the Conditional Fine-Grained Image Generation With Part Perception

Logit Variated Product Quantization Based on Parts Interaction and Metric Learning With Knowledge Distillation for Fine-Grained Image Retrieval

Logit Variated Product Quantization Based on Parts Interaction and Metric Learning With Knowledge Distillation for Fine-Grained Image Retrieval

Fine-Grained Image Recognition Methods and Their Applications in Remote Sensing Images: A Review

Fine-Grained Image Recognition Methods and Their Applications in Remote Sensing Images: A Review

Local Patch AutoAugment With Multi-Agent Collaboration

Data augmentation (DA) plays a critical role in improving the generalization of deep learning models. Recent works on automatically searching for DA policies from data have achieved great success. However, existing automated DA methods generally perform the search at the image level, which limits the exploration of diversity in local regions. In this paper, we propose a more fine-grained automated DA approach, dubbed Patch AutoAugment, to divide an image into a grid of patches and search for the joint optimal augmentation policies for the patches. We formulate it as a multi-agent reinforcement learning (MARL) problem, where each agent learns an augmentation policy for each patch based on its content together with the semantics of the whole image. The agents cooperate with each other to achieve the optimal augmentation effect of the entire image by sharing a team reward. We show the effectiveness of our method on multiple benchmark datasets of image classification, fine-grained image recognition and object detection ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</i> . <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">g</i> ., CIFAR-10, CIFAR-100, ImageNet, CUB-200-2011, Stanford Cars, FGVC-Aircraft and Pascal VOC 2007). Extensive experiments demonstrate that our method outperforms the state-of-the-art DA methods while requiring fewer computational resources. Automatic Data Augmentation. Multi-Agent Reinforcement Learning.