Image Classification Datasets Research Articles

Parameter-Efficient Model Adaptation for Vision Transformers

In computer vision, it has achieved great transfer learning performance via adapting large-scale pretrained vision models (e.g., vision transformers) to downstream tasks. Common approaches for model adaptation either update all model parameters or leverage linear probes. In this paper, we aim to study parameter-efficient model adaptation strategies for vision transformers on the image classification task. We formulate efficient model adaptation as a subspace training problem and perform a comprehensive benchmarking over different efficient adaptation methods. We conduct an empirical study on each efficient model adaptation method focusing on its performance alongside parameter cost. Furthermore, we propose a parameter-efficient model adaptation framework, which first selects submodules by measuring local intrinsic dimensions and then projects them into subspace for further decomposition via a novel Kronecker Adaptation method. We analyze and compare our method with a diverse set of baseline model adaptation methods (including state-of-the-art methods for pretrained language models). Our method performs the best in terms of the tradeoff between accuracy and parameter efficiency across 20 datasets under the few-shot setting and 7 image classification datasets under the full-shot setting.

CALIP: Zero-Shot Enhancement of CLIP with Parameter-Free Attention

Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with promising zero-shot performance. To further improve its downstream accuracy, existing works propose additional learnable modules upon CLIP and fine-tune them by few-shot training sets. However, the resulting extra training cost and data requirement severely hinder the efficiency for model deployment and knowledge transfer. In this paper, we introduce a free-lunch enhancement method, CALIP, to boost CLIP's zero-shot performance via a parameter-free attention module. Specifically, we guide visual and textual representations to interact with each other and explore cross-modal informative features via attention. As the pre-training has largely reduced the embedding distances between two modalities, we discard all learnable parameters in the attention and bidirectionally update the multi-modal features, enabling the whole process to be parameter-free and training-free. In this way, the images are blended with textual-aware signals and the text representations become visual-guided for better adaptive zero-shot alignment. We evaluate CALIP on various benchmarks of 14 datasets for both 2D image and 3D point cloud few-shot classification, showing consistent zero-shot performance improvement over CLIP. Based on that, we further insert a small number of linear layers in CALIP's attention module and verify our robustness under the few-shot settings, which also achieves leading performance compared to existing methods. Those extensive experiments demonstrate the superiority of our approach for efficient enhancement of CLIP. Code is available at https://github.com/ZiyuGuo99/CALIP.

Bi-directional Feature Reconstruction Network for Fine-Grained Few-Shot Image Classification

The main challenge for fine-grained few-shot image classification is to learn feature representations with higher inter-class and lower intra-class variations, with a mere few labelled samples. Conventional few-shot learning methods however cannot be naively adopted for this fine-grained setting -- a quick pilot study reveals that they in fact push for the opposite (i.e., lower inter-class variations and higher intra-class variations). To alleviate this problem, prior works predominately use a support set to reconstruct the query image and then utilize metric learning to determine its category. Upon careful inspection, we further reveal that such unidirectional reconstruction methods only help to increase inter-class variations and are not effective in tackling intra-class variations. In this paper, we for the first time introduce a bi-reconstruction mechanism that can simultaneously accommodate for inter-class and intra-class variations. In addition to using the support set to reconstruct the query set for increasing inter-class variations, we further use the query set to reconstruct the support set for reducing intra-class variations. This design effectively helps the model to explore more subtle and discriminative features which is key for the fine-grained problem in hand. Furthermore, we also construct a self-reconstruction module to work alongside the bi-directional module to make the features even more discriminative. Experimental results on three widely used fine-grained image classification datasets consistently show considerable improvements compared with other methods. Codes are available at: https://github.com/PRIS-CV/Bi-FRN.

Probability Guided Loss for Long-Tailed Multi-Label Image Classification

Long-tailed learning has attracted increasing attention in very recent years. Long-tailed multi-label image classification is one subtask and remains challenging and poorly researched. In this paper, we provide a fresh perspective from probability to tackle this problem. More specifically, we find that existing cost-sensitive learning methods for long-tailed multi-label classification will affect the predicted probability of positive and negative labels in varying degrees during training, and different processes of probability will affect the final performance in turn. We thus propose a probability guided loss which contains two components to control this process. One is the probability re-balancing which can flexibly adjust the process of training probability. And the other is the adaptive probability-aware focal which can further reduce the probability gap between positive and negative labels. We conduct extensive experiments on two long-tailed multi-label image classification datasets: VOC-LT and COCO-LT. The results demonstrate the rationality and superiority of our strategy.

Dropout Is NOT All You Need to Prevent Gradient Leakage

Gradient inversion attacks on federated learning systems reconstruct client training data from exchanged gradient information. To defend against such attacks, a variety of defense mechanisms were proposed. However, they usually lead to an unacceptable trade-off between privacy and model utility. Recent observations suggest that dropout could mitigate gradient leakage and improve model utility if added to neural networks. Unfortunately, this phenomenon has not been systematically researched yet. In this work, we thoroughly analyze the effect of dropout on iterative gradient inversion attacks. We find that state of the art attacks are not able to reconstruct the client data due to the stochasticity induced by dropout during model training. Nonetheless, we argue that dropout does not offer reliable protection if the dropout induced stochasticity is adequately modeled during attack optimization. Consequently, we propose a novel Dropout Inversion Attack (DIA) that jointly optimizes for client data and dropout masks to approximate the stochastic client model. We conduct an extensive systematic evaluation of our attack on four seminal model architectures and three image classification datasets of increasing complexity. We find that our proposed attack bypasses the protection seemingly induced by dropout and reconstructs client data with high fidelity. Our work demonstrates that privacy inducing changes to model architectures alone cannot be assumed to reliably protect from gradient leakage and therefore should be combined with complementary defense mechanisms.

POPNASv3: A pareto-optimal neural architecture search solution for image and time series classification

The growing demand for machine learning applications in industry has created a need for fast and efficient methods to develop accurate machine learning models. Automated Machine Learning (AutoML) algorithms have emerged as a promising solution to this problem, designing models without the need for human expertise. Given the effectiveness of neural network models, Neural Architecture Search (NAS) specialises in designing their architectures autonomously, with results that rival the most advanced hand-crafted models. However, this approach requires significant computational resources and hardware investment, making it less attractive for real-world applications. This article presents the third version of Pareto-Optimal Progressive Neural Architecture Search (POPNASv3), a new NAS algorithm that employs Sequential Model-Based Optimisation and Pareto optimality. This choice makes POPNASv3 flexible to different hardware environments, computational budgets and tasks, as the algorithm can efficiently explore user-defined search spaces of varying complexity. Pareto optimality extracts the architectures that achieve the best trade-off with respect to the metrics considered, reducing the number of models sampled during the search and dramatically improving time efficiency without sacrificing accuracy. The experiments performed on image and time series classification datasets provide evidence that POPNASv3 can explore a large set of different operators and converge to optimal architectures suited to the type of data provided under different scenarios.11The algorithm is open source and available on Zenodo at https://zenodo.org/record/7439022. The repository contains the code, the URIs of the datasets used in the experiments, and the models stored in ONNX format.

A federated learning algorithm using parallel-ensemble method on non-IID datasets

Traditional federated learning algorithms suffer from considerable performance reduction with non-identically and independently distributed datasets. This paper proposes a federated learning algorithm based on parallel-ensemble learning, which improves performance for image classification on these datasets. The training process of this algorithm includes basic federation learning and meta federation learning. First, several basic models are trained through cross-validation of federated learning, and then the meta-model is trained using the prediction results of the validation sets. In the training process, the training of different basic models is parallel. In prediction, meta-model is used to aggregate the output of the basic models to get the final prediction results. Our algorithm can achieve higher accuracy than traditional federated learning when using non-independent identically distributed datasets for image classification. Our algorithm aggregates different models through federated learning based on parallel-ensemble method, and improves the image classification performance of federated learning on non-independent identically distributed datasets.

Local pseudo-attributes for long-tailed recognition

Existing long-tailed recognition methods focus on learning global image representation by re-weighing, re-sampling, or global representation learning. However, we observe that solving real-world long-tailed recognition problems requires a fine-grained understanding of local parts within the image in order to avoid confusion among images with similar global configurations. We propose a novel self-supervised learning framework based on local pseudo-attributes (LPA) that are learned via clustering of local features without any human annotations. Such pseudo-attributes are often more balanced compared to image-level class labels. Our method outperforms the state-of-the-art on various long-tailed image classification datasets, such as CIFAR100-LT, iNaturalist, and ImageNet-LT.

Improvement accuracy in deep learning: An increasing neurons distance approach with the penalty term of loss function

The increasing use of neural networks for solving complex tasks has emphasized the need to optimize their performance. In recent years, the development of neural networks has enabled them to tackle increasingly challenging problems for applications in various fields. Existing methods, such as L1 and L2 penalty terms, improve the neural network performance by reducing the neurons' weight and removing some neurons' output. However, these algorithms reduce the number of active neurons and the theoretical maximum capacity of the neural network. A loss function penalty term can improve the network performance by guiding the gradient descent direction of neurons during the training process. Moreover, it does not need to alter the network structure, increase the data, or change the training process. In this study, a novel algorithm is proposed to improve the performance of the network by adding a new loss function penalty term. The algorithm proposed in this study enhances the accuracy and anti-interference capabilities of neural networks by increasing the number of active neurons based on the finding that identical neurons extract identical features from the data, while distinct neurons extract unique features. The neurons are treated as hyperspace points and the distance between them is converted into repulsive forces to increase the distance between these points. To verify the validity of the proposed algorithm, we tested it on image classification datasets (CIFAR-100 and CALTECH-256), text datasets (SST-2 and THUCNews), and an image detection dataset (PASCAL VOC). The results show that the proposed algorithm can improve the network performance for a variety of neural network structures, data types, and task types. This algorithm has demonstrated its effectiveness in robustness against data noise and compatibility with pre-trained models such as Faster R-CNN, YOLO, and RetinaNet.

Task guided representation learning using compositional models for zero-shot domain adaptation

Zero-shot domain adaptation (ZDA) methods aim to transfer knowledge about a task learned in a source domain to a target domain, while task-relevant data from target domain are not available. In this work, we address learning feature representations which are invariant to and shared among different domains considering task characteristics for ZDA. To this end, we propose a method for task-guided ZDA (TG-ZDA) which employs multi-branch deep neural networks to learn feature representations exploiting their domain invariance and shareability properties. The proposed TG-ZDA models can be trained end-to-end without requiring synthetic tasks and data generated from estimated representations of target domains. The proposed TG-ZDA has been examined using benchmark ZDA tasks on image classification datasets. Experimental results show that our proposed TG-ZDA outperforms state-of-the-art ZDA methods for different domains and tasks.

DE-DARTS: Neural architecture search with dynamic exploration

Neural architecture search (NAS) methods automatically find optimal neural networks without human assistance. Numerous algorithms for NAS have been studied to find architectures with gradient-based search. Differentiable architecture search (DARTS), one of the key papers of gradient-based search, dramatically reduced search cost, and showed outstanding performance through continuous relaxation and meta-learning based approximation. However, one of the issues with DARTS is that the gradient-based search process is biased due to the nested bi-level optimization structure, and the greedy behavior of the gradient descent. As a result, there is a problem that search spaces are limited to a limited set of architectures. To overcome the bias of the gradient-based search in network architecture search (NAS), we used a dynamic search method. This technique allows gradient-based search to have exploration. In this paper, we present a novel approach, namely, Dynamic-Exploration DARTS (DE-DARTS). For effective exploration, we use dynamic attention networks (DANs) in DE-DARTS, which change model architectures based on input data. As our DANs are activated early in the search, more diverse architectures are considered, depending on input data at the beginning of search. Our algorithm is evaluated in multiple image classification datasets including CIFAR-10, and ImageNet, and shows improved performance.

Classification and Discrimination of Birds and Small Drones Using Radar Micro-Doppler Spectrogram Images

This paper investigates the use of micro-Doppler spectrogram signatures of flying targets, such as drones and birds, to aid in their remote classification. Using a custom-designed 10-GHz continuous wave (CW) radar system, measurements from different scenarios on a variety of targets were recorded to create datasets for image classification. Time/velocity spectrograms generated for micro-Doppler analysis of multiple drones and birds were used for target identification and movement classification using TensorFlow. Using support vector machines (SVMs), the results showed an accuracy of about 90% for drone size classification, about 96% for drone vs. bird classification, and about 85% for individual drone and bird distinction between five classes. Different characteristics of target detection were explored, including the landscape and behavior of the target.

Deep learning based active learning technique for data annotation and improve the overall performance of classification models

Deep learning (DL) models achieved a lot of success due to the availability of labeled training data. In contrast, labeling a huge amount of data by a human is a time-consuming and expensive solution. Active Learning (AL) efficiently addresses the issue of labeled data collection at a low cost by picking the most useful samples from a large number of unlabeled datasets. However, current AL techniques largely depend on regular human involvement to annotate the most uncertain/informative samples in the collection. Therefore, we introduce a Deep Learning Based Active Learning (DLBAL) method that may incrementally learn from a small number of annotated training samples to build an effective classifier with optimum feature representation. The proposed DLBAL method improves the contemporary AL approaches in two steps. First, EfficientNet-B0 networks are integrated into AL. The incrementally annotated useful samples may be used to update both the representation of the feature and the classification algorithm concurrently. Second, we provide an effective technique for sample selection that improves the performance of the classifier with fewer manual annotations at a lower cost. In contrast to previous approaches that emphasize only uncertain instances with low prediction confidence, we obtain a significant number of instances with high-confidence from the unannotated set for pattern learning. Particularly, these instances are randomly picked and assigned labels to them continuously. Detailed experiments are conducted to show that the proposed DLBAL system obtained reasonable results on two difficult image classification datasets, namely Cross-Age Celebrity face recognition Dataset (CACD) and Caltech-256.

Fine coordinate attention for surface defect detection

Surface defect detection remains a challenging task due to issues such as inconspicuous targets, significant variations among identical defects, and minimal differences between distinct defects. To address these challenges, a Fine Coordinate Attention (FCA) block is proposed in this paper, which encodes both average and salient information in two coordinate directions, so that the spatial dependence can be captured and the long-range interaction can be achieved. And such localization-friendly information is crucial for industrial surface defect images with subtle targets. Specifically, the FCA block can recalibrate feature maps of a surface defect image through three steps: coordinate information aggregation, cross-dimension interaction, and attention generation. It can be embedded into any convolutional neural network (CNN) structure to improve performance. Additionally, two resistance spot welding (RSW) surface defect datasets are published in this paper: an image classification dataset RSW-C and an object detection dataset RSW-D. Experimental results for image classification and object detection demonstrate that the FCA block outperforms existing attention mechanisms. The code is available at , while the two RSW datasets can be found at www.kaggle.com/datasets/alfredzimmer/rswdatasets.

Multilevel Attention-Based Sample Correlations for Knowledge Distillation

Recently, model compression has been widely used for the deployment of cumbersome deep models on resource-limited edge devices in the performance-demanding industrial Internet of Things (IoT) scenarios. As a simple yet effective model compression technique, knowledge distillation (KD) aims to transfer the knowledge (e.g., sample relationships as the relational knowledge) from a large teacher model to a small student model. However, existing relational knowledge distillation methods usually build sample correlations directly from the feature maps at a certain middle layer in deep neural networks, which tends to overfit the feature maps of the teacher model and fails to address the most important sample regions. Inspired by this, we argue that the characteristics of important regions are of great importance and thus introduce attention maps to construct sample correlations for knowledge distillation. Specifically, with attention maps from multiple middle layers, attention-based sample correlations are newly built upon the most informative sample regions, and can be used as an effective and novel relational knowledge for knowledge distillation. We refer to the proposed method as multi-level attention-based sample correlations for knowledge distillation (or MASCKD). We perform extensive experiments on popular knowledge distillation datasets for image classification, image retrieval, and person re-identification, where the experimental results demonstrate the effectiveness of the proposed method for relational knowledge distillation.

Multitask Feature Learning as Multiobjective Optimization: A New Genetic Programming Approach to Image Classification.

Feature learning is a promising approach to image classification. However, it is difficult due to high image variations. When the training data are small, it becomes even more challenging, due to the risk of overfitting. Multitask feature learning has shown the potential for improving generalization. However, existing methods are not effective for handling the case that multiple tasks are partially conflicting. Therefore, for the first time, this article proposes to solve a multitask feature learning problem as a multiobjective optimization problem by developing a genetic programming approach with a new representation to image classification. In the new approach, all the tasks share the same solution space and each solution is evaluated on multiple tasks so that the objectives of all the tasks can be optimized simultaneously using a single population. To learn effective features, a new and compact program representation is developed to allow the new approach to evolving solutions shared across tasks. The new approach can automatically find a diverse set of nondominated solutions that achieve good tradeoffs between different tasks. To further reduce the risk of overfitting, an ensemble is created by selecting nondominated solutions to solve each image classification task. The results show that the new approach significantly outperforms a large number of benchmark methods on six problems consisting of 15 image classification datasets of varying difficulty. Further analysis shows that these new designs are effective for improving the performance. The detailed analysis clearly reveals the benefits of solving multitask feature learning as multiobjective optimization in improving the generalization.

Class-Incremental Learning: Survey and Performance Evaluation on Image Classification.

For future learning systems, incremental learning is desirable because it allows for: efficient resource usage by eliminating the need to retrain from scratch at the arrival of new data; reduced memory usage by preventing or limiting the amount of data required to be stored - also important when privacy limitations are imposed; and learning that more closely resembles human learning. The main challenge for incremental learning is catastrophic forgetting, which refers to the precipitous drop in performance on previously learned tasks after learning a new one. Incremental learning of deep neural networks has seen explosive growth in recent years. Initial work focused on task-incremental learning, where a task-ID is provided at inference time. Recently, we have seen a shift towards class-incremental learning where the learner must discriminate at inference time between all classes seen in previous tasks without recourse to a task-ID. In this paper, we provide a complete survey of existing class-incremental learning methods for image classification, and in particular, we perform an extensive experimental evaluation on thirteen class-incremental methods. We consider several new experimental scenarios, including a comparison of class-incremental methods on multiple large-scale image classification datasets, an investigation into small and large domain shifts, and a comparison of various network architectures.

One-shot Federated Learning without server-side training

Federated Learning (FL) has recently made significant progress as a new machine learning paradigm for privacy protection. Due to the high communication cost of traditional FL, one-shot federated learning is gaining popularity as a way to reduce communication cost between clients and the server. Most of the existing one-shot FL methods are based on Knowledge Distillation; however, distillation based approach requires an extra training phase and depends on publicly available data sets or generated pseudo samples. In this work, we consider a novel and challenging cross-silo setting: performing a single round of parameter aggregation on the local models without server-side training. In this setting, we propose an effective algorithm for Model Aggregation via Exploring Common Harmonized Optima (MA-Echo), which iteratively updates the parameters of all local models to bring them close to a common low-loss area on the loss surface, without harming performance on their own data sets at the same time. Compared to the existing methods, MA-Echo can work well even in extremely non-identical data distribution settings where the support categories of each local model have no overlapped labels with those of the others. We conduct extensive experiments on two popular image classification data sets to compare the proposed method with existing methods and demonstrate the effectiveness of MA-Echo, which clearly outperforms the state-of-the-arts. The source code can be accessed in https://github.com/FudanVI/MAEcho.

Urban street tree dataset for image classification and instance segmentation

Tree species identification and tree organ segmentation using images are challenging problems that are useful in many forestry-related tasks. In this paper, the urban street tree dataset is proposed as a comprehensive, publicly available dataset covering 50 tree species that contains 41,467 high-resolution classification images (22872 annotated images) from 10 city scenes. Our dataset includes leaf, tree, trunk, branch and trunk (hereafter referred to as branch), flower and fruit subdatasets that were captured under various light intensity, seasonal and shooting conditions. Annotations were performed in a fine-grained manner by using polygons to outline individual objects. We assessed the performance of various vision algorithms on different classification and segmentation tasks, including tree species identification and instance segmentation. Details on the urban street tree dataset are available at https://ytt917251944.github.io/dataset_jekyll.

Privileged information learning with weak labels

Privileged information learning is proposed to construct the classifier by incorporating privileged knowledge. At present, most of the privileged information learning methods assume that the instance is accurately labeled. However, in real-world applications, an instance may be weakly labeled. In this paper, we propose a novel privileged information learning method with weak labels (PLWB). The hypothesis of our work is that an instance may be annotated by a number of labelers and different labelers may give different labels to this instance due to distinct professional knowledge and subjective factors. It leads to ambiguous labels of instances, namely weak labels. To solve this problem, our methodology is to give each labeler a weight and incorporate these weights into a privileged information learning model. Our technique is to employ a heuristic framework to optimize the labeler weights and the privileged information learning model jointly. The existing privileged information learning methods do not consider the weak label problem, and assign an equal or random weight to each labeler. Our work is different from these methods. The novelty and theoretical contribution is that this is the first work to deal with the weak label problem in privileged information learning. The merit is that we assign an unknown weight to each labeler and solve the optimal values of these weights in the optimization process, such that the performance of the learning model can be improved with the optimal labeler weights. In the experiments, the tool that we use is MATLAB, in which we implement our algorithm. The experimental datasets include one handwritten categorization dataset, two image classification datasets (i.e., Animals-with-Attributes dataset and Caltech-101 dataset), and one disease diagnosis dataset (i.e., Alzheimer’s Disease Neuroimaging Initiative dataset), in which the number of instances used is 2000, 6180, 8677 and 202, respectively. The obtained results are that: (1) by optimizing the labeler weights, the proposed PLWB method obtains explicitly higher classification accuracy than the existing privileged information learning methods; (2) PLWB has relatively higher training time since it needs to solve the labeler weights in the optimization process.