Image Classification Datasets Research Articles

Multi-Objective Evolutionary Neural Architecture Search with Weight-Sharing Supernet

Deep neural networks have played a crucial role in the field of deep learning, achieving significant success in practical applications. The architecture of neural networks is key to their performance. In the past few years, these architectures have been manually designed by experts with rich domain knowledge. Additionally, the optimal neural network architecture can vary depending on specific tasks and data distributions. Neural Architecture Search (NAS) is a class of techniques aimed at automatically searching for and designing neural network architectures according to the given tasks and data. Specifically, evolutionary-computation-based NAS methods are known for their strong global search capability and have aroused widespread interest in recent years. Although evolutionary-computation-based NAS has achieved success in a wide range of research and applications, it still faces bottlenecks in training and evaluating a large number of individuals during optimization. In this study, we first devise a multi-objective evolutionary NAS framework based on a weight-sharing supernet to improve the search efficiency of traditional evolutionary-computation-based NAS. This framework combines the population optimization characteristic of evolutionary algorithms with the weight-sharing ideas in one-shot models. We then design a bi-population MOEA/D algorithm based on the proposed framework to effectively solve the NAS problem. By constructing two sub-populations with different optimization objectives, the algorithm can effectively explore network architectures of various sizes in complex search spaces. An inter-population communication mechanism further enhances the algorithm’s exploratory capability, enabling it to find network architectures with uniform distribution and high diversity. Finally, we conduct performance comparison experiments on image classification datasets of different scales and complexities. Experimental results demonstrate the effectiveness of the proposed multi-objective evolutionary NAS framework and the practicality and transferability of the introduced bi-population MOEA/D-based NAS method compared to existing state-of-the-art NAS methods.

ConVision Benchmark: A Contemporary Framework to Benchmark CNN and ViT Models

Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have shown remarkable performance in computer vision tasks, including object detection and image recognition. These models have evolved significantly in architecture, efficiency, and versatility. Concurrently, deep-learning frameworks have diversified, with versions that often complicate reproducibility and unified benchmarking. We propose ConVision Benchmark, a comprehensive framework in PyTorch, to standardize the implementation and evaluation of state-of-the-art CNN and ViT models. This framework addresses common challenges such as version mismatches and inconsistent validation metrics. As a proof of concept, we performed an extensive benchmark analysis on a COVID-19 dataset, encompassing nearly 200 CNN and ViT models in which DenseNet-161 and MaxViT-Tiny achieved exceptional accuracy with a peak performance of around 95%. Although we primarily used the COVID-19 dataset for image classification, the framework is adaptable to a variety of datasets, enhancing its applicability across different domains. Our methodology includes rigorous performance evaluations, highlighting metrics such as accuracy, precision, recall, F1 score, and computational efficiency (FLOPs, MACs, CPU, and GPU latency). The ConVision Benchmark facilitates a comprehensive understanding of model efficacy, aiding researchers in deploying high-performance models for diverse applications.

Modified osprey algorithm for optimizing capsule neural network in leukemia image recognition

The diagnosis of leukemia is a serious matter that requires immediate and accurate attention. This research presents a revolutionary method for diagnosing leukemia using a Capsule Neural Network (CapsNet) with an optimized design. CapsNet is a cutting-edge neural network that effectively captures complex features and spatial relationships within images. To improve the CapsNet's performance, a Modified Version of Osprey Optimization Algorithm (MOA) has been utilized. Thesuggested approach has been tested on the ALL-IDB database, a widely recognized dataset for leukemia image classification. Comparative analysis with various machine learning techniques, including Combined combine MobilenetV2 and ResNet18 (MBV2/Res) network, Depth-wise convolution model, a hybrid model that combines a genetic algorithm with ResNet-50V2 (ResNet/GA), and SVM/JAYA demonstrated the superiority of our method in different terms. As a result, the proposed method is a robust and powerful tool for diagnosing leukemia from medical images.

Multiple Contrastive Experts for long-tailed image classification

Real-world image classification data usually exhibits a challenging long-tailed distribution, attributed to the inherent difficulty in data collection. Existing ensemble approaches predominantly prioritize the empirical diversification of the ensemble model, sidelining its critical aspects of representation ability. A noticeable gap also exists in theoretical analysis elucidating the intricate relationship between ensemble model diversity and its generalization efficacy. This paper introduces a loosely coupled ensemble framework, Multiple Contrastive Experts (MCE) tailored for long-tailed image classification, aiming to bolster image representation while ensuring diversity within the proposed MCE. Leveraging skill-diverse classification losses, the experts in the ensemble model are able to specialize in different kinds of classes in the long-tailed distribution. An adapted supervised contrastive learning (SCL) loss is introduced to guide training for each feature learning branch to enhance the representation ability of MCE. Through the effective integration of the different losses from all experts, each expert model can be optimized coordinately. Moreover, the relationship between the generalization ability of MCE and its diversity is theoretically revealed against a single classification model. At last, extensive experiments were made over five widely used long-tailed image classification datasets. The results show that our proposed MCE is very competitive performance against the state-of-the-art methods, while maintaining a relatively lower computational cost. Besides, MCE has also exhibited superior performance with the increasing number of experts.

An Improved Reacceleration Optimization Algorithm Based on the Momentum Method for Image Recognition

The optimization algorithm plays a crucial role in image recognition by neural networks. However, it is challenging to accelerate the model’s convergence and maintain high precision. As a commonly used stochastic gradient descent optimization algorithm, the momentum method requires many epochs to find the optimal parameters during model training. The velocity of its gradient descent depends solely on the historical gradients and is not subject to random fluctuations. To address this issue, an optimization algorithm to enhance the gradient descent velocity, i.e., the momentum reacceleration gradient descent (MRGD), is proposed. The algorithm utilizes the point division of the current momentum and the gradient relationship, multiplying it with the gradient. It can adjust the update rate and step size of the parameters based on the gradient descent state, so as to achieve faster convergence and higher precision in training the deep learning model. The effectiveness of this method is further proven by applying the reacceleration mechanism to the Adam optimizer, resulting in the MRGDAdam algorithm. We verify both algorithms using multiple image classification datasets, and the experimental results show that the proposed optimization algorithm enables the model to achieve higher recognition accuracy over a small number of training epochs, as well as speeding up model implementation. This study provides new ideas and expansions for future optimizer research.

Few-shot image classification via hybrid representation

Few-shot image classification aims to learn an embedding model on the base datasets and design a base learner to recognize novel categories. The few-shot image classification framework is a two-phase process. First, the pre-train phase utilizes the base data to train a CNN-based feature extractor. Next, in the meta-test phase, the frozen feature extractor is applied to novel data with categories different from the base data. A base learner is then designed for recognition. Several simple base learners, including nearest neighbor, support vector machine, and logistic regression classifiers, have been recently introduced for few-shot learning tasks. However, these base learners are separately designed to consider specific representations (e.g., the class center) or shared representations (e.g., the boundaries). This paper mainly focuses on exploring the representation-residual base learners, which aim to represent a query sample with the support set and predict the query sample’s label based on the minimal residual error. We first introduce two representation-residual base learners: a specific representation base learner and a shared representation base learner. Then, we propose a novel hybrid representation base learner that combines both base learners to generate competitive representation. Additionally, we extend our approach by incorporating a self-training framework to utilize the query data fully. We evaluate our proposed method on several benchmark few-shot image classification datasets, such as miniImageNet, tieredImageNet, CIFAR-FS, FC100, and CUB datasets. The experimental results indicate that our proposed approach shows a significant performance improvement.

Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images

Recent advancements in diffusion models have enabled the generation of realistic deepfakes from textual prompts in natural language. While these models have numerous benefits across various sectors, they have also raised concerns about the potential misuse of fake images and cast new pressures on fake image detection. In this work, we pioneer a systematic study on deepfake detection generated by state-of-the-art diffusion models. Firstly, we conduct a comprehensive analysis of the performance of contrastive and classification-based visual features, respectively extracted from CLIP-based models and ResNet or ViT-based architectures trained on image classification datasets. Our results demonstrate that fake images share common low-level cues, which render them easily recognizable. Further, we devise a multimodal setting wherein fake images are synthesized by different textual captions, which are used as seeds for a generator. Under this setting, we quantify the performance of fake detection strategies and introduce a contrastive-based disentangling method that lets us analyze the role of the semantics of textual descriptions and low-level perceptual cues. Finally, we release a new dataset, called COCOFake, containing about 1.2M images generated from the original COCO image-caption pairs using two recent text-to-image diffusion models, namely Stable Diffusion v1.4 and v2.0.

Eigenspectrum regularisation reverse neighbourhood discriminative learning

AbstractLinear discriminant analysis is a classical method for solving problems of dimensional reduction and pattern classification. Although it has been extensively developed, however, it still suffers from various common problems, such as the Small Sample Size (SSS) and the multimodal problem. Neighbourhood linear discriminant analysis (nLDA) was recently proposed to solve the problem of multimodal class caused by the contravention of independently and identically distributed samples. However, due to the existence of many small‐scale practical applications, nLDA still has to face the SSS problem, which leads to instability and poor generalisation caused by the singularity of the within‐neighbourhood scatter matrix. The authors exploit the eigenspectrum regularisation techniques to circumvent the singularity of the within‐neighbourhood scatter matrix of nLDA, which is called Eigenspectrum Regularisation Reverse Neighbourhood Discriminative Learning (ERRNDL). The algorithm of nLDA is reformulated as a framework by searching two projection matrices. Three eigenspectrum regularisation models are introduced to our framework to evaluate the performance. Experiments are conducted on the University of California, Irvine machine learning repository and six image classification datasets. The proposed ERRNDL‐based methods achieve considerable performance.

On the fusion of soft-decision-trees and concept-based models

In the field of eXplainable Artificial Intelligence (XAI), the generation of interpretable models that are able to match the performance of state-of-the-art deep learning methods is one of the main challenges. In this work, we present a novel interpretable model for image classification that combines the power of deep convolutional networks and the transparency of decision trees. We explore different training techniques where convolutional networks and decision trees can be trained together using gradient-based optimization methods as usually done in deep learning environments. All of this results in a transparent model in which a soft decision tree makes the final classification based on human-understandable concepts that are extracted by a convolutional neural network. We tested the proposed solution on two challenge image classification datasets and compared them with the state-of-the-art approaches, achieving competitive results.

Fine-grained image classification method based on hybrid attention module.

To efficiently capture feature information in tasks of fine-grained image classification, this study introduces a new network model for fine-grained image classification, which utilizes a hybrid attention approach. The model is built upon a hybrid attention module (MA), and with the assistance of the attention erasure module (EA), it can adaptively enhance the prominent areas in the image and capture more detailed image information. Specifically, for tasks involving fine-grained image classification, this study designs an attention module capable of applying the attention mechanism to both the channel and spatial dimensions. This highlights the important regions and key feature channels in the image, allowing for the extraction of distinct local features. Furthermore, this study presents an attention erasure module (EA) that can remove significant areas in the image based on the features identified; thus, shifting focus to additional feature details within the image and improving the diversity and completeness of the features. Moreover, this study enhances the pooling layer of ResNet50 to augment the perceptual region and the capability to extract features from the network's less deep layers. For the objective of fine-grained image classification, this study extracts a variety of features and merges them effectively to create the final feature representation. To assess the effectiveness of the proposed model, experiments were conducted on three publicly available fine-grained image classification datasets: Stanford Cars, FGVC-Aircraft, and CUB-200-2011. The method achieved classification accuracies of 92.8, 94.0, and 88.2% on these datasets, respectively. In comparison with existing approaches, the efficiency of this method has significantly improved, demonstrating higher accuracy and robustness.

Ta-Adapter: Enhancing few-shot CLIP with task-aware encoders

Contrastive Language-Image Pre-training (CLIP) has shown impressive zero-shot transfer capabilities, but its potential for specific downstream tasks is not fully utilized. To further enhance CLIP’s few-shot capability for specific datasets, some subsequent works have been proposed, such as methods based on lightweight adapters and prompt learning. However, since CLIP is pretrained on a diverse collection of image and text pairs sourced from the internet, it is difficult to sufficiently tune models to specific datasets using only lightweight adaptions. In this paper, we argue that largely modifying the internal representations within CLIP’s encoders can yield better results on downstream datasets. In this work, we introduce Ta-Adapter, a method that equips both the visual and textual encoders of CLIP with task-specific prompts. These prompts are generated using a collaborative prompt learning approach, which allows the encoders to produce representations that are better aligned with specific downstream datasets. Then, we initialize an adapter module using the optimized features generated by the task-aware visual encoder for further feature alignment, and this module can also be further fine-tuned. Our extensive experiments on image classification datasets show that compared to the state-of-the-art few-shot methods Tip-Adapter-F and MaPLe, our model exhibits good performance and obtains an average absolute gain of 2.04% and 1.62% on 11 different image recognition datasets, respectively. In conclusion, this work presents a unique and effective approach to unlocking the full potential of CLIP’s few-shot learning capabilities.

Disentangle Then Calibrate With Gradient Guidance: A Unified Framework for Common and Rare Disease Diagnosis.

The computer-aided diagnosis (CAD) for rare diseases using medical imaging poses a significant challenge due to the requirement of large volumes of labeled training data, which is particularly difficult to collect for rare diseases. Although Few-shot learning (FSL) methods have been developed for this task, these methods focus solely on rare disease diagnosis, failing to preserve the performance in common disease diagnosis. To address this issue, we propose the Disentangle then Calibrate with Gradient Guidance (DCGG) framework under the setting of generalized few-shot learning, i.e., using one model to diagnose both common and rare diseases. The DCGG framework consists of a network backbone, a gradient-guided network disentanglement (GND) module, and a gradient-induced feature calibration (GFC) module. The GND module disentangles the network into a disease-shared component and a disease-specific component based on gradient guidance, and devises independent optimization strategies for both components, respectively, when learning from rare diseases. The GFC module transfers only the disease-shared channels of common-disease features to rare diseases, and incorporates the optimal transport theory to identify the best transport scheme based on the semantic relationship among different diseases. Based on the best transport scheme, the GFC module calibrates the distribution of rare-disease features at the disease-shared channels, deriving more informative rare-disease features for better diagnosis. The proposed DCGG framework has been evaluated on three public medical image classification datasets. Our results suggest that the DCGG framework achieves state-of-the-art performance in diagnosing both common and rare diseases.

Polymorphic Clustering and Approximate Masking Framework for Fine-Grained Insect Image Classification

Insect diversity monitoring is crucial for biological pest control in agriculture and forestry. Modern monitoring of insect species relies heavily on fine-grained image classification models. Fine-grained image classification faces challenges such as small inter-class differences and large intra-class variances, which are even more pronounced in insect scenes where insect species often exhibit significant morphological differences across multiple life stages. To address these challenges, we introduce segmentation and clustering operations into the image classification task and design a novel network model training framework for fine-grained classification of insect images using multi-modality clustering and approximate mask methods, named PCAM-Frame. In the first stage of the framework, we adopt the Polymorphic Clustering Module, and segmentation and clustering operations are employed to distinguish various morphologies of insects at different life stages, allowing the model to differentiate between samples at different life stages during training. The second stage consists of a feature extraction network, called Basenet, which can be any mainstream network that performs well in fine-grained image classification tasks, aiming to provide pre-classification confidence for the next stage. In the third stage, we apply the Approximate Masking Module to mask the common attention regions of the most likely classes and continuously adjust the convergence direction of the model during training using a Deviation Loss function. We apply PCAM-Frame with multiple classification networks as the Basenet in the second stage and conduct extensive experiments on the Insecta dataset of iNaturalist 2017 and IP102 dataset, achieving improvements of 2.2% and 1.4%, respectively. Generalization experiments on other fine-grained image classification datasets such as CUB200-2011 and Stanford Dogs also demonstrate positive effects. These experiments validate the pertinence and effectiveness of our framework PCAM-Frame in fine-grained image classification tasks under complex conditions, particularly in insect scenes.

On the effects of recursive convolutional layers in convolutional neural networks

The Recursive Convolutional Layer (RCL) is a module that wraps a recursive feedback loop around a convolutional layer (CL). The RCL has been proposed to address some of the shortcomings of Convolutional Neural Networks (CNNs), as its unfolding increases the depth of a network without increasing the number of weights. We investigated the “naïve” substitution of CL with RCL on three base models: a 4-CL model, ResNet, DenseNet and their RCL-ized versions: C-FRPN, R-ResNet, and R-DenseNet using five image classification datasets. We find that this one-to-one replacement significantly improves the performances of the 4-CL model, but not those of ResNet or DenseNet. This led us to investigate the implication of the RCL substitution on the 4-CL model which reveals, among a number of properties, that RCLs are particularly efficient in shallow CNNs. We proceeded to re-visit the first set of experiments by gradually transforming the 4-CL model and the C-FRPN into respectively ResNet and R-ResNet, and find that the performance improvement is largely driven by the training regime whereas any depth increase negatively impacts the RCL-ized version. We conclude that the replacement of CLs by RCLs shows great potential in designing high-performance shallow CNNs.

Enhancing performance of vision transformers on small datasets through local inductive bias incorporation

Vision transformers (ViTs) achieve remarkable performance on large datasets, but tend to perform worse than convolutional neural networks (CNNs) when trained from scratch on smaller datasets, possibly due to a lack of local inductive bias in the architecture. Recent studies have therefore added locality to the architecture and demonstrated that it can help ViTs achieve performance comparable to CNNs in the small-size dataset regime. Existing methods, however, are architecture-specific or have higher computational and memory costs. Thus, we propose a module called Local InFormation Enhancer (LIFE) that extracts patch-level local information and incorporates it into the embeddings used in the self-attention block of ViTs. Our proposed module is memory and computation efficient, as well as flexible enough to process auxiliary tokens such as the classification and distillation tokens. Empirical results show that the addition of the LIFE module improves the performance of ViTs on small image classification datasets. We further demonstrate how the effect can be extended to downstream tasks, such as object detection and semantic segmentation. In addition, we introduce a new visualization method, Dense Attention Roll-Out, specifically designed for dense prediction tasks, allowing the generation of class-specific attention maps utilizing the attention maps of all tokens. The code for this project is available on Github (https://github.com/NeurAI-Lab/LIFEhttps://github.com/NeurAI-Lab/LIFE).

Advancing Attribution-Based Neural Network Explainability through Relative Absolute Magnitude Layer-Wise Relevance Propagation and Multi-Component Evaluation

Recent advancement in deep-neural network performance led to the development of new state-of-the-art approaches in numerous areas. However, the black-box nature of neural networks often prohibits their use in areas where model explainability and model transparency are crucial. Over the years, researchers proposed many algorithms to aid neural network understanding and provide additional information to the human expert. One of the most popular methods being Layer-Wise Relevance Propagation (LRP). This method assigns local relevance based on the pixel-wise decomposition of nonlinear classifiers. With the rise of attribution method research, there has emerged a pressing need to assess and evaluate their performance. Numerous metrics have been proposed, each assessing an individual property of attribution methods such as faithfulness, robustness, or localization. Unfortunately, no single metric is deemed optimal for every case, and researchers often use several metrics to test the quality of the attribution maps. In this work, we address the shortcomings of the current LRP formulations and introduce a novel method for determining the relevance of input neurons through layer-wise relevance propagation. Furthermore, we apply this approach to the recently developed Vision Transformer architecture and evaluate its performance against existing methods on two image classification datasets, namely ImageNet and PascalVOC. Our results clearly demonstrate the advantage of our proposed method. Furthermore, we discuss the insufficiencies of current evaluation metrics for attribution-based explainability and propose a new evaluation metric that combines the notions of faithfulness, robustness, and contrastiveness. We utilize this new metric to evaluate the performance of various attribution-based methods. Our code is available at: https://github.com/davor10105/relative-absolute-magnitude-propagation

Coordinate Attention Guided Dual-Teacher Adaptive Knowledge Distillation for image classification

Knowledge distillation (KD) refers to transferring the knowledge learned from a teacher network with complex architecture and strong learning ability to another student network with light-weight and weak learning ability through a specific distillation strategy. However, most existing KD approaches to image classification often employ a single teacher network to guide the training of the student network. When the teacher network makes an erroneous prediction, the transferred knowledge will deteriorate the performance of the student network. To address or mitigate the above issue, we develop a novel KD approach called Coordinate Attention Guided Dual-Teacher Adaptive Knowledge Distillation (CAG-DAKD), to deliver more discriminative and comprehensive knowledge obtained from two teacher networks to a compact student network. Specifically, we integrate the positive prediction distribution of two teacher networks according to whether the two teacher networks predict correctly and the magnitude of the cross-entropy to deliver better output distribution to guide the student network. Furthermore, to distill the most valuable knowledge from the first teacher network that has a similar architecture to the student network, a coordinate attention mechanism is introduced into different layers of the first teacher network so that the student network can effectively learn more discriminative feature representations. We conduct extensive experiments on three standard image classification datasets: CIFAR10, CIFAR100, and ImageNet to verify the superiority of the proposed method over other state-of-the-art competitors. Code will be available at https://github.com/mdt1219/CAG-DAKD.git/

Learning to Double-Check Model Prediction From a Causal Perspective.

The present machine learning schema typically uses a one-pass model inference (e.g., forward propagation) to make predictions in the testing phase. It is inherently different from human students who double-check the answer during examinations especially when the confidence is low. To bridge this gap, we propose a learning to double-check (L2D) framework, which formulates double check as a learnable procedure with two core operations: recognizing unreliable predictions and revising predictions. To judge the correctness of a prediction, we resort to counterfactual faithfulness in causal theory and design a contrastive faithfulness measure. In particular, L2D generates counterfactual features by imagining: "what would the sample features be if its label was the predicted class" and judges the prediction by the faithfulness of the counterfactual features. Furthermore, we design a simple and effective revision module to revise the original model prediction according to the faithfulness. We apply the L2D framework to three classification models and conduct experiments on two public datasets for image classification, validating the effectiveness of L2D in prediction correctness judgment and revision.

Positional normalization-based mixed-image data augmentation and ensemble self-distillation algorithm

The prior approach to data augmentation using positional normalization necessitated manual determination of weight parameters for blending two images. However, theoretically, these parameters cannot cater to all samples. We propose a novel data augmentation approach called pnMix (Positional normalization-based data mixing) based on data blending and positional normalization to address these issues. Moreover, we have improved the teacher model of an existing network structure based on self-distillation technology using an ensembling approach called ESKD. Our method enhances performance across multiple datasets and tasks. We outperform previous methods in complex real-world datasets such as imagenet-a and imagenet-o, which we believe are highly meaningful for practical applications. Specifically, we obtain good results on the CIFAR100 and Imagenet image classification datasets, the Pascal VOC and COCO object detection datasets, and the COCO and Cityscapes image segmentation datasets. Furthermore, we analyze the pnMix method's advantages compared to other data augmentation methods through visualization techniques, demonstrating its effectiveness and interpretability. Our source code and pre-trained models are available at https://github.com/sydney72380/pnMix.

Algorithm-Driven Tele-otoscope for Remote Care for Patients With Otitis Media.

The COVID-19 pandemic has spurred a growing demand for telemedicine. Artificial intelligence and image processing systems with wireless transmission functionalities can facilitate remote care for otitis media (OM). Accordingly, this study developed and validated an algorithm-driven tele-otoscope system equipped with Wi-Fi transmission and a cloud-based automatic OM diagnostic algorithm. Prospective, cross-sectional, diagnostic study. Tertiary Academic Medical Center. We designed a tele-otoscope (Otiscan, SyncVision Technology Corp) equipped with digital imaging and processing modules, Wi-Fi transmission capabilities, and an automatic OM diagnostic algorithm. A total of 1137 otoscopic images, comprising 987 images of normal cases and 150 images of cases of acute OM and OM with effusion, were used as the dataset for image classification. Two convolutional neural network models, trained using our dataset, were used for raw image segmentation and OM classification. The tele-otoscope delivered images with a resolution of 1280 × 720 pixels. Our tele-otoscope effectively differentiated OM from normal images, achieving a classification accuracy rate of up to 94% (sensitivity, 80%; specificity, 96%). Our study demonstrated that the developed tele-otoscope has acceptable accuracy in diagnosing OM. This system can assist health care professionals in early detection and continuous remote monitoring, thus mitigating the consequences of OM.