Fine-grained Classification Task Research Articles

Fine-grained Vehicle Classification Based on Feature Augmentation

Fine-grained vehicle classification is a hot research topic in the field of computer vision, which aims to recognize different models of vehicles belonging to the same brand. Limited by the large appearance differences of the same vehicle (due to different attitudes and perspectives) and small appearance differences between different vehicles, the vehicle’s reorganization performance is not high and can’t meet the real application requirements. In this paper, a Fine-grained Vehicle Classification is proposed based on Feature Augmentation (FAFCC), which focuses on mining the key discriminative parts of the object. Specifically, the FAFCC first improves the weakly supervised data augmentation network (WSDAN) by using multi-layer feature extraction and fusion instead of single-layer features. Further, the FAFCC assigns local feature weights to select key features, which can solve the problem that the discriminant features in the channel are not prominent enough and realize the high-quality compact fine-grained vehicle recognition task. The performance of the FAFCC on the fine-grained image classification tasks for vehicles has been evaluated through algorithm comparison experiments and ablation experiments. Through visual analysis, it can be seen that the introduction of each new module has optimized the loss function value and convergence speed of the model. In general, our FAFCC algorithm has a certain improvement effect compared with the original algorithm.

Image local structure information learning for fine-grained visual classification

Learning discriminative visual patterns from image local salient regions is widely used for fine-grained visual classification (FGVC) tasks such as plant or animal species classification. A large number of complex networks have been designed for learning discriminative feature representations. In this paper, we propose a novel local structure information (LSI) learning method for FGVC. Firstly, we indicate that the existing FGVC methods have not properly considered how to extract LSI from an input image for FGVC. Then an LSI extraction technique is introduced which has the ability to properly depict the properties of different local structure features in images. Secondly, a novel LSI learning module is proposed to be added into a given backbone network for enhancing the ability of the network to find salient regions. Thirdly, extensive experiments show that our proposed method achieves better performance on six image datasets. Particularly, the proposed method performs far better on datasets with a limited number of images.

A Fine-Grained Image Classification and Detection Method Based on Convolutional Neural Network Fused with Attention Mechanism.

Due to the existence of attention system, people pay attention to the distinguishable area of the image, rather than directly receiving and processing the information of the whole image. This natural advantage makes attention mechanism widely used in fine-grained image classification. The research goal of fine-grained image classification task often is to differentiate subclass objects belonging to the same basic category. The difficulty of classification is that there are only slight local differences between different categories, but there may be large feature differences within the same category. At the same time, complex background features also bring interference factors to image recognition. In order to further extract discriminant regional features, this paper proposes a fine-grained image classification method WSFF-BCNN based on weak supervision feature fusion from two aspects: the improvement of the loss function in the training process of convolution neural network and the refinement of fine-grained image feature extraction. It uses the mixed attention of channel domain and spatial domain to obtain the detailed description information in the feature to highlight the response of the corresponding channel and spatial location in the feature map and pay attention to the attention characteristics of different dimensions. The original images of different sizes are input into the improved bilinear model to obtain multi-scale features. The large-scale features can represent the spatial location information of key areas, and the small-scale features represent the low-level features of the image. The backbone network of bilinear network uses ResNet50 to extract features and sample and zoom and uses bilinear pooling to fuse features of different scales to obtain a rich image feature representation.

ConvNeXt-Based Fine-Grained Image Classification and Bilinear Attention Mechanism Model

Thus far, few studies have been conducted on fine-grained classification tasks for the latest convolutional neural network ConvNeXt, and no effective optimization method has been made available. To achieve more accurate fine-grained classification, this paper proposes two attention embedding methods based on ConvNeXt network and designs a new bilinear CBAM; simultaneously, a multiscale, multi-perspective and all-around attention framework is proposed, which is then applied in ConvNeXt. Experimental verification shows that the accuracy rate of the improved ConvNeXt for fine-grained image classification reaches 87.8%, 91.2%, and 93.2% on fine-grained classification datasets CUB-200-2011, Stanford Cars, and FGVC Aircraft, respectively, showing increases of 2.7%, 0.3% and 0.4%, respectively, compared to those of the original network without optimization, and increases of 3.7%, 8.0% and 2.0%, respectively, compared to those of the traditional BCNN. In addition, ablation experiments are set up to verify the effectiveness of the proposed attention framework.

Application of Efficient Channel Attention Residual Mechanism in Blast Furnace Tuyere Image Anomaly Detection

In the steelmaking industry, the state of the blast furnace tuyere is an important basis for obtaining the internal information of the blast furnace. Traditional detections mainly rely on manual experience judgment, which is a time-consuming and tiring procedure for a human. In order to improve the efficiency of the detection, this paper is devoted to applying artificial intelligence methods to blast furnace anomaly detection. However, because of the low imaging degree of the abnormal state monitoring of the furnace mouth, the difference in the abnormal category is inconspicuous, and it is difficulty to extract the features with the existing intelligent models. To solve these problems, a novel and stable method is proposed in this paper to classify the image recognition of the abnormal state of the tuyere into one category; this is a new architecture that combines multiple technologies. For the fine-grained image classification task, an improved abnormal state recognition algorithm of the blast furnace tuyere based on the channel attention residual mechanism is proposed. In the model, the dataset is augmented by rotating it at random angles to balance the amount of data in each category; then, the residual module is used to integrate high- and low-order feature information and optimize the network; then, the multi-layer channel attention module is added based on the channel attention residual mechanism, and it obtains the optimal parameter combination of the model through k-fold cross-validation. Moreover, the number of channels was reduced by half after channel fusion, which could effectively reduce the model parameters and model complexity. It is shown in our experiments that the proposed method has an accuracy rate of 97.10% in identifying the abnormal state of the tuyere in our collection of blast furnace tuyere datasets. In order to test the performance of the proposed method, some existing models, such as SERNet, ResNeXt, and repVGG, are involved for comparison, and the proposed method has a better classification effect in comparison to them.

Embedding Pose Information for Multiview Vehicle Model Recognition

Vehicle model recognition is a typical fine-grained classification task that has a wide range of application prospects in safe cities and constitutes a research hotspot in the field of computer vision. Vehicles in images can appear at various angles, resulting in large differences in appearance. The existence of “multiviews” renders vehicle model recognition challenging. Recent research on vehicle model recognition has not fully explored the pose information of vehicles in different images, resulting in low model performance. In this study, we use vehicle pose information to solve the multiview vehicle model recognition (MV-VMR) problem and design a convolutional neural network (CNN) model with embedded vehicle pose information, known as the embedding pose CNN (EP-CNN). The proposed model includes two subnetworks: the pose estimation subnetwork (PE-SubNet) and vehicle model classification subnetwork (VMC-SubNet). PE-SubNet extracts the vehicle pose information, including the pose features and vehicle viewpoint. In VMC-SubNet, considering the scale variation of vehicles, an improved squeeze-and-excitation (SE) block, named the MultiSE block is implemented. We embed the vehicle viewpoint into the MultiSE block, which reweighs each channel such that the extracted features elicit different responses to different viewpoints. Subsequently, the pose features and classification features are integrated for classification. Experiments are conducted on the benchmark CompCars web-nature and Stanford Cars datasets. The results demonstrate that the proposed EP-CNN method can achieve higher recognition accuracy than most classic CNN models and several state-of-the-art fine-grained vehicle model classification algorithms. Code has been made available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/HFUT-CV/EP-CNN</uri> .

Towards an Effective Orthogonal Dictionary Convolution Strategy

Orthogonality regularization has proven effective in improving the precision, convergence speed and the training stability of CNNs. Here, we propose a novel Orthogonal Dictionary Convolution Strategy (ODCS) on CNNs to improve orthogonality effect by optimizing the network architecture and changing the regularized object. Specifically, we remove the nonlinear layer in typical convolution block “Conv(BN) + Nonlinear + Pointwise Conv(BN)”, and only impose orthogonal regularization on the front Conv. The structure, “Conv(BN) + Pointwise Conv(BN)”, is then equivalent to a pair of dictionary and encoding, defined in sparse dictionary learning. Thanks to the exact and efficient representation of signal with dictionaries in low-dimensional projections, our strategy could reduce the superfluous information in dictionary Conv kernels. Meanwhile, the proposed strategy relieves the too strict orthogonality regularization in training, which makes hyper-parameters tuning of model to be more flexible. In addition, our ODCS can modify the state-of-the-art models easily without any extra consumption in inference phase. We evaluate it on a variety of CNNs in small-scale (CIFAR), large-scale (ImageNet) and fine-grained (CUB-200-2011) image classification tasks, respectively. The experimental results show that our method achieve a stable and superior improvement.

Graph convolutional networks with hierarchical multi-head attention for aspect-level sentiment classification.

Aspect-level sentiment classification has been widely used by researchers as a fine-grained sentiment classification task to predict the sentiment polarity of specific aspect words in a given sentence. Previous studies have shown relatively good experimental results using graph convolutional networks, so more and more approaches are beginning to exploit sentence structure information for this task. However, these methods do not link aspect word and context well. To address this problem, we propose a method that utilizes a hierarchical multi-head attention mechanism and a graph convolutional network (MHAGCN). It fully considers syntactic dependencies and combines semantic information to achieve interaction between aspect words and context. To fully validate the effectiveness of the method proposed in this paper, we conduct extensive experiments on three benchmark datasets, which, according to the experimental results, show that the method outperforms current methods.

A Spatial Feature-Enhanced Attention Neural Network with High-Order Pooling Representation for Application in Pest and Disease Recognition

With the development of advanced information and intelligence technologies, precision agriculture has become an effective solution to monitor and prevent crop pests and diseases. However, pest and disease recognition in precision agriculture applications is essentially the fine-grained image classification task, which aims to learn effective discriminative features that can identify the subtle differences among similar visual samples. It is still challenging to solve for existing standard models troubled by oversized parameters and low accuracy performance. Therefore, in this paper, we propose a feature-enhanced attention neural network (Fe-Net) to handle the fine-grained image recognition of crop pests and diseases in innovative agronomy practices. This model is established based on an improved CSP-stage backbone network, which offers massive channel-shuffled features in various dimensions and sizes. Then, a spatial feature-enhanced attention module is added to exploit the spatial interrelationship between different semantic regions. Finally, the proposed Fe-Net employs a higher-order pooling module to mine more highly representative features by computing the square root of the covariance matrix of elements. The whole architecture is efficiently trained in an end-to-end way without additional manipulation. With comparative experiments on the CropDP-181 Dataset, the proposed Fe-Net achieves Top-1 Accuracy up to 85.29% with an average recognition time of only 71 ms, outperforming other existing methods. More experimental evidence demonstrates that our approach obtains a balance between the model’s performance and parameters, which is suitable for its practical deployment in precision agriculture art applications.

Transformer-Based Attention Network for Vehicle Re-Identification

Vehicle re-identification (ReID) focuses on searching for images of the same vehicle across different cameras and can be considered as the most fine-grained ID-level classification task. It is fundamentally challenging due to the significant differences in appearance presented by a vehicle with the same ID (especially from different viewpoints) coupled with the subtle differences between vehicles with different IDs. Spatial attention mechanisms that have been proven to be effective in computer vision tasks also play an important role in vehicle ReID. However, they often require expensive key-point labels or suffer from noisy attention masks when trained without key-point labels. In this work, we propose a transformer-based attention network (TAN) for learning spatial attention information and hence for facilitating learning of discriminative features for vehicle ReID. Specifically, in contrast to previous studies that adopted a transformer network, we designed the attention network as an independent branch that can be flexibly utilized in various tasks. Moreover, we combined the TAN with two other branches: one to extract global features that define the image-level structures, and the other to extract the auxiliary side-attribute features that are invariant to viewpoint, such as color, car type, etc. To validate the proposed approach, experiments were conducted on two vehicle datasets (the VeRi-776 and VehicleID datasets) and a person dataset (Market-1501). The experimental results demonstrated that the proposed TAN is effective in improving the performance of both the vehicle and person ReID tasks, and the proposed method achieves state-of-the-art (SOTA) perfomance.

Study of chrysanthemum image phenotype on-line classification based on transfer learning and bilinear convolutional neural network

As one of the largest families of flowering plants, chrysanthemum has a wide variety and variation. There exists small phenotypic differences between classes, leading to the fact that classifying the class of chrysanthemum is a challenging fine-grained image classification problem. The fine-grained image classification task based on deep learning often concerns a large number of parameters and requires a long training time. For tackling these issues, this paper proposes a chrysanthemum image phenotype classification framework based on transfer learning and bilinear Convolutional Neural Network. After pre-processing the images, the symmetric VGG16 network is adopted as a feature extractor. The pre-trained parameters are then transferred to the proposed framework, which is divided into two stages for training the full connection layer and fine tuning all layers respectively. The phenotypic features of chrysanthemum output from the two networks are transposed and multiplied. Last the global features are input into the classification layer for classification. In this paper, a total of eight methods, including other bilinear network models, non-transfer learning and non-bilinear models, are compared with our proposal. The experimental result proves that our proposal can achieve better performance and lower loss, compared with other network models, by reaching at an accuracy rate of 0.9815 and a recall rate of 0.9800.

Fine-grained Image Classification via Multi-scale Selective Hierarchical Biquadratic Pooling

How to extract distinctive features greatly challenges the fine-grained image classification tasks. In previous models, bilinear pooling has been frequently adopted to address this problem. However, most bilinear pooling models neglect either intra or inter layer feature interaction. This insufficient interaction brings in the loss of discriminative information. In this article, we devise a novel fine-grained image classification approach named M ulti-scale S elective H ierarchical bi Q uadratic P ooling (MSHQP). The proposed biquadratic pooling simultaneously models intra and inter layer feature interactions and enhances part response by integrating multi-layer features. The subsequent coarse-to-fine multi-scale interaction structure captures the complementary information within features. Finally, the active interaction selection module adaptively learns the optimal interaction subset for a specific dataset. Consequently, we obtain a robust image representation with coarse-to-fine semantics. We conduct experiments on five benchmark datasets. The experimental results demonstrate that MSHQP achieves competitive or even match the state-of-the-art methods in terms of both accuracy and computational efficiency, with 89.0%, 94.9%, 93.4%, 90.4%, and 91.5% top-1 classification accuracy on CUB-200-2011, Stanford-Cars, FGVC-Aircraft, Stanford-Dog, and VegFru, respectively.

Bin similarity‐based domain adaptation for fine‐grained image classification

Fine-grained classification tasks are challenging because fine-grained data sets are quite scarce. Thus, we utilized the domain adaptation method to migrate knowledge from large, labeled data sets to fine-grained target data sets. We employed the bin similarity (BS) algorithm to measure and select the approximate domains from large-scale data sets to the fine-grained target domains. Source domain feature space was divided into multiple bins and the features of the target domains were sampled to fill the bins. The most similar domains were selected based on the similarity statistics of the sample features. We implemented the BS algorithm combined with the popular convolutional neural networks, pretrained the network on the selected similar subdata sets, and subsequently fine-tuned it on the fine-grained data sets. We evaluated the BS classification model on Stanford Dogs and Oxford Flower data sets, and the results showed improved BS classification performance compared with the state-of-the-art domain adaptation methods, earth mover's distance, selective joint fine-tuning, L2 with starting point, and domain similarity for transfer learning. Furthermore, BS is a pluggable module that boosts the performance of domain adaptation.

Dual-Concentrated Network With Morphological Features for Tree Species Classification Using Hyperspectral Image

At present, deep learning is a hot topic in the field of the classification of hyperspectral image (HSI), and it has aroused wide attention. However, in fine-grained classification tasks, such as tree species classification, the uncertain spectrum remains the major factor restraining the classification performance. In order to solve the dilemma of forest tree species classification, a Dual-concentrated Network with Morphological Features (DNMF) is proposed. Firstly, mathematical morphology is used to extract the morphological features of HSI. Then, coarse-grained information is extracted from the original hyperspectral data, and fine-grained information is extracted from morphological features. After that, both morphological representations and spectral inputs are fed into DNMF, and the overall evaluation index and visual image are obtained. The advantage of DNMF is that it decouples the spatial and spectral information, and a multi-source information fusion process is then simulated. Accordingly, DNMF obtains high tree species classification accuracy. In order to verify the superiority of DNMF, we choose Gaofeng State-owned Forest Farm (GSFF) in Guangxi Province and the Belgium dataset which was collected near the western part of Belgium as the research area. Related experiments demonstrate that the DNMF model achieves clearly better classification performance over other competitive baselines.

Improving Deep Subdomain Adaptation by Dual-Branch Network Embedding Attention Module for SAR Ship Classification

This study aims at improving fine-grained ship classification performance under the condition that there is no labeled samples available in SAR domain (target domain) by transferring the knowledge from optical remote sensing (ORS) domain (source domain) which has rich labeled samples. The proposed method improves the original deep subdomain adaptation network (DSAN) by designing a dual-branch network (DBN) embedding attention module to extract more discriminative deep transferable features, thereby improving the performance of the subdomain adaptation. Specifically, we utilized a deep base network (ResNet-50) and a shallow base network (ResNet-18) to build the DBN, and embedded the convolutional block attention module (CBAM) after the first and the last convolutional layer of each branch. Extensive experiments demonstrate that the proposed method, which is termed as DSAN++, is feasible and achieves remarkable improvement than the state-of-the-art methods on the task of fine-grained ship classification.

Remote Sensing Fine-Grained Ship Data Augmentation Pipeline With Local-Aware Progressive Image-to-Image Translation

Remote sensing image ship fine-grained classification is a challenging vision problem due to factors such as inter-class similarity, real-world image scarcity, and class imbalance. Data augmentation aims to solve these problems from the data perspective. Most of these methods cannot work effectively in complex scenes, which restricts their practical application. Here, we propose a novel data augmentation pipeline based on local-aware image translation to achieve the representation mapping between cross-domain corresponding instances. Our pipeline contains three modules: Imaging Simulation System (ISS), Local-aware Progressive Image-to-Image Translation (LoPIT), and Image Harmonization (IH) modules. The ISS module generates simulated images with correct appearance and diverse features based on the input requirement information. To tackle the domain gap between simulated image and real-world image, we propose the Local-aware CycleGAN in the LoPIT module to achieve mapping based on local-aware learning and apply two sub-modules to progressively complete the remote sensing image global cross-domain translation. The IH module uses image harmonization technology to coordinate the visual appearance between the foreground and background to generate sufficient remote sensing ship images with precise representation, photorealistic style, and harmonious features. Moreover, we present a mixed dataset including real-world images and our synthetic images for remote sensing image fine-grained ship classification. Our dataset named RSSA-12 contains 12 categories of ship targets in 3831 images with high-quality annotated category labels, effectively alleviating the long-tail problem of existing datasets. Experimental results demonstrate that our progressive pipeline outperforms the state-of-the-art data augmentation method on the remote sensing fine-grained ship classification task.

RALR: Random Amplify Learning Rates for Training Neural Networks

It has been shown that the learning rate is one of the most critical hyper-parameters for the overall performance of deep neural networks. In this paper, we propose a new method for setting the global learning rate, named random amplify learning rates (RALR), to improve the performance of any optimizer in training deep neural networks. Instead of monotonically decreasing the learning rate, we expect to escape saddle points or local minima by amplifying the learning rate between reasonable boundary values based on a given probability. Training with RALR rather than conventionally decreasing the learning rate achieves further improvement on networks’ performance without extra consumption. Remarkably, the RALR is complementary with state-of-the-art data augmentation and regularization methods. Besides, we empirically study its performance on image classification tasks, fine-grained classification tasks, object detection tasks, and machine translation tasks. Experiments demonstrate that RALR can bring a notable improvement while preventing overfitting when training deep neural networks. For example, the classification accuracy of ResNet-110 trained on the CIFAR-100 dataset using RALR achieves a 1.34% gain compared with ResNet-110 trained traditionally.

SCWC: Structured channel weight sharing to compress convolutional neural networks

Convolutional neural networks (CNNs) have surpassed humans in many computer vision areas. However, the redundancy of CNNs inhibits its application in the embedded device. In this paper, a simple shared channel weight convolution (SCWC) approach is proposed to reduce the number of parameters. Multiplication is much more complex than accumulation, so reducing multiplication is more significant than reducing accumulation in CNNs. The proposed SCWC can reduce the number of multiplications by the distributive property due to the structured channel parameter sharing. Furthermore, the fully trained CNN can be directly compressed without tedious training from scratch. To evaluate the performance of the proposed SCWC, five competitive benchmark datasets for image classification and object detection, CIFAR-10, CIFAR-100, ImageNet, CUB-200–2011 and PASCAL VOC are adopted. Experiments demonstrate that the proposed SCWC can reduce about 50% of parameters and multiplications for ResNet50 with only 0.13% accuracy loss, and reduce about 70% of parameters and multiplications for VGG16 with 1.64% accuracy loss on ImageNet, which is better than many other pruning methods for CNNs. The accuracy loss is very small because of the soft parameter sharing. Moreover, the proposed SCWC is also effective for object detection and fine-grained image classification tasks.

Rumour detection based on deep hybrid structural and sequential representation networks

The rapid development of online social networks significantly facilitates the interaction of people and dramatically expands the diffusion sphere of information. Rumours, however, are not excluded from the list of beneficiaries. The widespread of rumours has lengthened the psychological distance and caused tremendous economic losses, and rumour detection has become an inescapable and challenging task of great practical importance. In this work, we propose a novel neural network architecture for rumour classification and early rumour detection of fine-grained categories. Unlike using tree-like modules for structural feature extraction, we build an information stream network and employ graph convolutional networks to explore the relations among the hierarchical nodes in the network. To enhance the sequential representation learning, the module of deep bilateral gated recurrent units is further incorporated to reveal the crucial features hiding behind the information flow. Moreover, to organically fuse the learned high-dimensionally structural and sequential features, attention mechanism is applied to automatically adjust the trainable weights. Comparative experiments on two real-world datasets demonstrate that our proposed model outperforms the state-of-the-art methods in the task of fine-grained rumour classification and is capable of identifying rumours at an early stage.

An Automatic Light Stress Grading Architecture Based on Feature Optimization and Convolutional Neural Network

The identification of light stress is crucial for light control in plant factories. Image-based lighting classification of leafy vegetables has exhibited remarkable performance with high convenience and economy. Convolutional Neural Network (CNN) has been widely used for crop image analysis because of its architecture, high accuracy and efficiency. Among them, large intra-class differences and small inter-class differences are important factors affecting crop identification and a critical challenge for fine-grained classification tasks based on CNN. To address this problem, we took the Lettuce (Lactuca sativa L.) widely grown in plant factories as the research object and constructed a leaf image set containing four stress levels. Then a light stress grading model combined with classic pre-trained CNN and Triplet loss function is constructed, which is named Tr-CNN. The model uses the Triplet loss function to constrain the distance of images in the feature space, which can reduce the Euclidean distance of the samples from the same class and increase the heterogeneous Euclidean distance. Multiple sets of experimental results indicate that the model proposed in this paper (Tr-CNN) has obvious advantages in light stress grading dataset and generalized dataset.