CIFAR-10 Datasets Research Articles

BR-FEEL: A backdoor resilient approach for federated edge learning with fragment-sharing

In the resource-constrained federated edge learning (FEEL) systems, fragment-sharing is an efficient approach for multiple clients to cooperatively train a giant model with billions of parameters. Compared with the classical federated learning schemes where the local model is fully trained and exchanged by each client, the fragment-sharing only requires each client to optionally choose a parameter-fragment to train and share, according to its storage, computing, and networking abilities. However, when the full model is no longer delivered in fragment-sharing, the backdoor attacks hidden behind the fragments become harder to be detected, which introduces formidable challenge for the security of FEEL systems. In this paper, we firstly show that the existing fragment-sharing works suffer a lot from the backdoor attacks. Then, a Backdoor-Resilient approach, named BR-FEEL, is introduced to defend against the potential backdoor attacks. Specifically, a twin model is built by each benign client to integrate the parameter-fragments from others. A knowledge distillation process is designed on each client to transfer the clean knowledge from its twin model to local model. With the twin model and knowledge distillation process, our BR-FEEL approach makes sure that the local models of the benign clients will not be backdoored. Experiments on CIFAR-10 and GTSRB datasets with MobileNetV2 and ResNet-34 are conducted. The numerical results demonstrate the efficacy of BR-FEEL on reducing attack success rates by over 90% compared to other baselines under various attack methods.

A Multi-Dimensional Reverse Auction Mechanism for Volatile Federated Learning in the Mobile Edge Computing Systems

Federated learning (FL) can break the problem of data silos and allow multiple data owners to collaboratively train shared machine learning models without disclosing local data in mobile edge computing. However, how to incentivize these clients to actively participate in training and ensure efficient convergence and high test accuracy of the model has become an important issue. Traditional methods often use a reverse auction framework but ignore the consideration of client volatility. This paper proposes a multi-dimensional reverse auction mechanism (MRATR) that considers the uncertainty of client training time and reputation. First, we introduce reputation to objectively reflect the data quality and training stability of the client. Then, we transform the goal of maximizing social welfare into an optimization problem, which is proven to be NP-hard. Then, we propose a multi-dimensional auction mechanism MRATR that can find the optimal client selection and task allocation strategy considering clients’ volatility and data quality differences. The computational complexity of this mechanism is polynomial, which can promote the rapid convergence of FL task models while ensuring near-optimal social welfare maximization and achieving high test accuracy. Finally, the effectiveness of this mechanism is verified through simulation experiments. Compared with a series of other mechanisms, the MRATR mechanism has faster convergence speed and higher testing accuracy on both the CIFAR-10 and IMAGE-100 datasets.

A Gastrointestinal Image Classification Method Based on Improved Adam Algorithm

In this study, a gastrointestinal image classification method based on the improved Adam algorithm is proposed. Gastrointestinal image classification is of great significance in the field of medical image analysis, but it presents numerous challenges, including slow convergence, susceptibility to local minima, and the complexity and imbalance of medical image data. Although the Adam algorithm is widely used in stochastic gradient descent, it tends to suffer from overfitting and gradient explosion issues when dealing with complex data. To address these problems, this paper proposes an improved Adam algorithm, AdamW_AGC, which combines the weight decay and Adaptive Gradient Clipping (AGC) strategies. Weight decay is a common regularization technique used to prevent machine learning models from overfitting. Adaptive gradient clipping avoids the gradient explosion problem by restricting the gradient to a suitable range and helps accelerate the convergence of the optimization process. In order to verify the effectiveness of the proposed algorithm, we conducted experiments on the HyperKvasir dataset and validation experiments on the MNIST and CIFAR10 standard datasets. Experimental results on the HyperKvasir dataset demonstrate that the improved algorithm achieved a classification accuracy of 75.8%, compared to 74.2% for the traditional Adam algorithm, representing an improvement of 1.6%. Furthermore, validation experiments on the MNIST and CIFAR10 datasets resulted in classification accuracies of 98.69% and 71.7%, respectively. These results indicate that the AdamW_AGC algorithm has advantages in handling complex, high-dimensional medical image classification tasks, effectively improving both classification accuracy and training stability. This study provides new ideas and expansions for future optimizer research.

BayesianSpikeFusion: accelerating spiking neural network inference via Bayesian fusion of early prediction.

Spiking neural networks (SNNs) have garnered significant attention due to their notable energy efficiency. However, conventional SNNs rely on spike firing frequency to encode information, necessitating a fixed sampling time and leaving room for further optimization. This study presents a novel approach to reduce sampling time and conserve energy by extracting early prediction results from the intermediate layer of the network and integrating them with the final layer's predictions in a Bayesian fashion. Experimental evaluations conducted on image classification tasks using MNIST, CIFAR-10, and CIFAR-100 datasets demonstrate the efficacy of our proposed method when applied to VGGNets and ResNets models. Results indicate a substantial energy reduction of 38.8% in VGGNets and 48.0% in ResNets, illustrating the potential for achieving significant efficiency gains in spiking neural networks. These findings contribute to the ongoing research in enhancing the performance of SNNs, facilitating their deployment in resource-constrained environments. Our code is available on GitHub: https://github.com/hanebarla/BayesianSpikeFusion.

Multi-head co-training: An uncertainty-aware and robust semi-supervised learning framework

Co-training, an advanced form of self-training, allows multiple base models to learn collaboratively, leading to superior performance in semi-supervised learning tasks. However, its widespread adoption is hindered by high computational costs and intricate design choices. To address these challenges, we present Multi-Head Co-Training, a streamlined and efficient framework that consolidates individual models into a multi-head structure, adding minimal extra parameters. Each classification head in this unified model collaborates with others via a “Weak and Strong Augmentation” strategy, with diversity organically introduced through robust data augmentation. Consequently, our approach implicitly promotes diversity while incurring only a minor increase in computational overhead, making co-training more accessible. We validate the effectiveness of Multi-Head Co-Training through an empirical study on standard semi-supervised learning benchmarks. For example, our method achieves up to a 3.1% accuracy improvement on the semi-supervised CIFAR dataset compared to recent methods.Recognizing the necessity for more practical performance metrics beyond accuracy, we assess our framework from three additional perspectives: robust generalization, uncertainty, and computational efficiency. To evaluate robust generalization, we expand the conventional SSL experimental setting to a more comprehensive open-set semi-supervised learning scenario. For uncertainty assessment, we conduct experiments on model calibration and selective classification benchmarks. For example, our method achieves up to a 4.3% accuracy improvement on the open-set semi-supervised CIFAR dataset. Our extensive experiments confirm that our proposed framework better captures prediction confidence and uncertainty, rendering it more suitable for SSL deployment in open environments. The code is available at https://github.com/chenmc1996/Multi-Head-Co-Training.

Probabilistic Attention Based on Gaussian Processes for Deep Multiple Instance Learning.

Multiple instance learning (MIL) is a weakly supervised learning paradigm that is becoming increasingly popular because it requires less labeling effort than fully supervised methods. This is especially interesting for areas where the creation of large annotated datasets remains challenging, as in medicine. Although recent deep learning MIL approaches have obtained state-of-the-art results, they are fully deterministic and do not provide uncertainty estimations for the predictions. In this work, we introduce the attention Gaussian process (AGP) model, a novel probabilistic attention mechanism based on Gaussian processes (GPs) for deep MIL. AGP provides accurate bag-level predictions as well as instance-level explainability and can be trained end-to-end. Moreover, its probabilistic nature guarantees robustness to overfit on small datasets and uncertainty estimations for the predictions. The latter is especially important in medical applications, where decisions have a direct impact on the patient's health. The proposed model is validated experimentally as follows. First, its behavior is illustrated in two synthetic MIL experiments based on the well-known MNIST and CIFAR-10 datasets, respectively. Then, it is evaluated in three different real-world cancer detection experiments. AGP outperforms state-of-the-art MIL approaches, including deterministic deep learning ones. It shows a strong performance even on a small dataset with less than 100 labels and generalizes better than competing methods on an external test set. Moreover, we experimentally show that predictive uncertainty correlates with the risk of wrong predictions, and therefore it is a good indicator of reliability in practice. Our code is publicly available.

Rock lithology classification algorithm based on improved self-attention mechanism VIT

Abstract When applying VIT to image classification, although good accuracy has been achieved, there is a limitation on the number of stacked layers. The similarity between different tokens increases as the model deepens, leading to low accuracy in shallow VIT models and wasting computational resources in deep VIT models. To address this issue, a multi-valued self-attention mechanism is proposed. By introducing an additional clue “V,” that does not vary arbitrarily, the dot product similarity observes each position in the token sequence more, reducing the depth of the attention mechanism network and avoiding the problem of similarity disappearing between deep tokens. Additionally, the loss function is improved by combining the Focal Loss function with Label Smoothing, enhancing the model’s handling of uncertain images while retaining the category focus of Focal Loss. Experiments are conducted on a rock lithology classification dataset, showing that compared to the VGG model and ResNet-18, the accuracy of the ResNet-50 model improves by 17.6%, 12%, and 4.9%, respectively. This paper also demonstrates the effectiveness of the multi-valued self-attention mechanism, which significantly reduces depth and parameter count while maintaining comparable accuracy. Finally, generalization experiments on the CIFAR-10 dataset further validate these conclusions.

Addressing unreliable local models in federated learning through unlearning

Federated unlearning (FUL) is a promising solution for removing negative influences from the global model. However, ensuring the reliability of local models in FL systems remains challenging. Existing FUL studies mainly focus on eliminating bad data influences and neglecting scenarios where other factors, such as adversarial attacks and communication constraints, also contribute to negative influences that require mitigation. In this paper, we introduce Local Model Refining (LMR), a FUL method designed to address the negative impacts of bad data as well as other factors on the global model. LMR consists of three components: (i) Identifying and categorizing unreliable local models into two classes based on their influence source: bad data or other factors. (ii) Bad Data Influence Unlearning (BDIU): BDIU is a client-side algorithm that identifies affected layers in unreliable models and employs gradient ascent to mitigate bad data influences. Boosting training is applied when necessary under specific conditions. (iii) Other Influence Unlearning (OIU): OIU is a server-side algorithm that identifies unaffected parameters in the unreliable local model and combines them with corresponding parameters of the previous global model to construct the updated local model. Finally, LMR aggregates updated local models with remaining local models to produce the unlearned global model. Extensive evaluation shows LMR enhances accuracy and accelerates average unlearning speed by 5x compared to comparison methods on MNIST, FMNIST, CIFAR-10, and CelebA datasets.

Adaptive Perturbation for Adversarial Attack.

In recent years, the security of deep learning models achieves more and more attentions with the rapid development of neural networks, which are vulnerable to adversarial examples. Almost all existing gradient-based attack methods use the sign function in the generation to meet the requirement of perturbation budget on L∞ norm. However, we find that the sign function may be improper for generating adversarial examples since it modifies the exact gradient direction. Instead of using the sign function, we propose to directly utilize the exact gradient direction with a scaling factor for generating adversarial perturbations, which improves the attack success rates of adversarial examples even with fewer perturbations. At the same time, we also theoretically prove that this method can achieve better black-box transferability. Moreover, considering that the best scaling factor varies across different images, we propose an adaptive scaling factor generator to seek an appropriate scaling factor for each image, which avoids the computational cost for manually searching the scaling factor. Our method can be integrated with almost all existing gradient-based attack methods to further improve their attack success rates. Extensive experiments on the CIFAR10 and ImageNet datasets show that our method exhibits higher transferability and outperforms the state-of-the-art methods.

Comparative analysis of federated learning algorithms under extreme non-IID conditions for computer vision

Data heterogeneity presents enormous challenges for Federated Learning (FL) where data is characterized by non-independent and identically distributed (non-IID) patterns across participating clients. Although a prominent issue, there is a lack of comprehensive studies regarding the efficacy of FL algorithms under different configurations of hyperparameters and models. Three state-of-the-art algorithms were chosen to be the focus of this paper FedAvg, FedProx, and SCAFFOLD. Experiments of the paper study varying degrees of data imbalance and number of local training steps to see their effects on model training results. The algorithms and hyperparameters were evaluated using ResNet50, AlexNet, and DenseNet121 on the CIFAR-10 dataset. Experimental results reveal that FedAvg and SCAFFOLD generally outperform FedProx, except for the latter excelling in the scenario of extreme non-IID data distributions in combination with minimal local training. It was also observed that models with complex architectures like ResNet50 are more susceptible to data imbalances, while simpler models such as AlexNet prove to be more robust. This study provides valuable insights to how different configurations of hyperparameters affect FL in non-IID scenarios, contributing to future development of more efficient and robust FL algorithms.

Multi-Dimensional Fusion Attention Mechanism with Vim-like Structure for Mobile Network Design

Recent advancements in mobile neural networks, such as the squeeze-and-excitation (SE) attention mechanism, have significantly improved model performance. However, they often overlook the crucial interaction between location information and channels. The interaction of multiple dimensions in feature engineering is of paramount importance for achieving high-quality results. The Transformer model and its successors, such as Mamba and Vision Mamba, have effectively combined features and linked location information. This approach has transitioned from NLP (natural language processing) to CV (computer vision). This paper introduces a novel attention mechanism for mobile neural networks inspired by the structure of Vim (Vision Mamba). It adopts a “1 + 3” architecture to embed multi-dimensional information into channel attention, termed ”Multi-Dimensional Vim-like Attention Mechanism”. The proposed method splits the input into two major branches: the left branch retains the original information for subsequent feature screening, while the right branch divides the channel attention into three one-dimensional feature encoding processes. These processes aggregate features along one channel direction and two spatial directions, simultaneously capturing remote dependencies and preserving precise location information. The resulting feature maps are then combined with the left branch to produce direction-aware, location-sensitive, and channel-aware attention maps. The multi-dimensional Vim-like attention module is simple and can be seamlessly integrated into classical mobile neural networks such as MobileNetV2 and ShuffleNetV2 with minimal computational overhead. Experimental results demonstrate that this attention module adapts well to mobile neural networks with a low parameter count, delivering excellent performance on the CIFAR-100 and MS COCO datasets.

FedNor: A robust training framework for federated learning based on normal aggregation

Addressing data security and data silo issues in Edge Intelligence, this paper proposes a Byzantine-resilient framework (FedNor) for Federated Learning (FL). FedNor integrates robust statistical methods with personalized FL strategies to enhance resilience against malicious updates while maintaining model generalization capabilities. The framework comprises two key components: the Robust Normal Aggregation (RN) module and the Personalized Fusion (PF) module. The RN module employs normality tests to identify and rectify anomalous updates, thereby ensuring the integrity and quality of model updates. Concurrently, the PF module incorporates data distribution considerations when integrating global and local models to optimize model security and accuracy. Experimental results demonstrate FedNor's effectiveness in mitigating eight distinct poisoning attacks on the MNIST datasets, with minimal accuracy degradation ranging from 0.42% to 1.96%. Furthermore, FedNor limits the backdoor attack success rate on the CIFAR-10 datasets to below 20%, while maintaining accuracy comparable to personalized FL schemes.

A geometric approach for accelerating neural networks designed for classification problems

This paper proposes a geometric-based technique for compressing convolutional neural networks to accelerate computations and improve generalization by eliminating non-informative components. The technique utilizes a geometric index called separation index to evaluate the functionality of network elements such as layers and filters. By applying this index along with center-based separation index, a systematic algorithm is proposed that optimally compresses convolutional and fully connected layers. The algorithm excludes layers with low performance, selects the best subset of filters in the filtering layers, and tunes the parameters of fully connected layers using center-based separation index. An illustrative example of classifying CIFAR-10 dataset is presented to explain the algorithm step-by-step. The proposed method achieves impressive pruning results on networks trained by CIFAR-10 and ImageNet datasets, with 87.5%, 77.6%, and 78.8% of VGG16, GoogLeNet, and DenseNet parameters pruned, respectively. Comparisons with state-of-the-art works are provided to demonstrate the effectiveness of the proposed method.

Vehicle Classification Algorithm Based on Improved Vision Transformer

Vehicle classification technology is one of the foundations in the field of automatic driving. With the development of deep learning technology, visual transformer structures based on attention mechanisms can represent global information quickly and effectively. However, due to direct image segmentation, local feature details and information will be lost. To solve this problem, we propose an improved vision transformer vehicle classification network (IND-ViT). Specifically, we first design a CNN-In D branch module to extract local features before image segmentation to make up for the loss of detail information in the vision transformer. Then, in order to solve the problem of misdetection caused by the large similarity of some vehicles, we propose a sparse attention module, which can screen out the discernible regions in the image and further improve the detailed feature representation ability of the model. Finally, this paper uses the contrast loss function to further increase the intra-class consistency and inter-class difference of classification features and improve the accuracy of vehicle classification recognition. Experimental results show that the accuracy of the proposed model on the datasets of vehicle classification BIT-Vehicles, CIFAR-10, Oxford Flower-102, and Caltech-101 is higher than that of the original vision transformer model. Respectively, it increased by 1.3%, 1.21%, 7.54%, and 3.60%; at the same time, it also met a certain real-time requirement to achieve a balance of accuracy and real time.

Structured segment rescaling with Gaussian processes for parameter efficient ConvNets

We introduce a novel mechanism for structured pruning on ConvNet blocks and channels. Our mechanism, Structured Segment Rescaling (SSR) down-samples a ConvNet’s dimensions using depth and width modifiers that respectively remove whole blocks and channels. SSR is a systematic approach for constructing ConvNets that can replace arbitrary design heuristics. The SSR modifiers rescale logical partitions (segments) of a ConvNet with grouped layers. Different modifiers on segments yield many different architectures with unique rescales for their blocks. This diversity of architectures is then systemically explored using a Gaussian Process (GP) that optimizes for modifiers that maintain accuracy and reduce parameters. We analyze SSR in the context of resource constrained environments using ResNets trained on the CIFAR datasets. An initial set of depth and width modifiers explore extreme rescales of ResNet segments, where we find up to 70% parameter reduction. The GP then generalizes on these initial rescales by being trained on them and then predicts the accuracy of other rescaled ConvNet given their segment modifiers. SSR produces over 105 ConvNets that can be trained selectively based on their GP predicted accuracy. The GP enabled SSR pushes compression to over 80% with minimal accuracy impact. While both depth and width modifiers can reduce parameters, we show reducing blocks is better for reducing latency with up to 80% faster ConvNets. Using our mechanism, we can efficiently customize ConvNets using their parameter-accuracy trade-offs. SSR only requires 101 GPU hours and modest engineering to yield efficient new ConvNets that can facilitate edge inference.

Comparative analysis of transformer and GoogleNet models in image classification based on the CIFAR dataset

Comparative analysis of transformer and GoogleNet models in image classification based on the CIFAR dataset

An 8b-Precison 16-Kb FDSOI 8T SRAM CIM macro based on time-domain for energy-efficient edge AI devices

Compute-in-memory has been increasingly appreciated by researchers as a well-suited hardware accelerator in convolutional neural networks (CNNs), because it can achieve low power consumption and high inference accuracy. This work presents a novel TD-CIM structure using:1) A Capacitor Charging scheme that uses Compact 8T Model for multiply-and-accumulate (MAC) Operations with serials inputs in Time Domain Level; 2) a new replicated bit-line time-domain converter (RBL-TDC) to achieve the quantization of the multiply-accumulate operations with high accuracy; 3) A 22 nm FD-SOI 16 Kb TD-CIM macro fabricated using foundry provided compact 8T-SRAM cells, which achieves normalized energy efficiency(EF) of 5816.5 TOPS/W, normalized area efficiency(64TOPS/mm2), and 8-bit weight for 8-bit serials inputs with 64 accumulations per cycle, as well as output precision(14b) in the MAC operation. This work also obtains an inference accuracy of 92.57 % on the VGG-16 network using the Cifar10 dataset over PVT variations.

Promoting the Shift From Pixel-Level Correlations to Object Semantics Learning by Rethinking Computer Vision Benchmark Data Sets.

In computer vision research, convolutional neural networks (CNNs) have demonstrated remarkable capabilities at extracting patterns from raw pixel data, achieving state-of-the-art recognition accuracy. However, they significantly differ from human visual perception, prioritizing pixel-level correlations and statistical patterns, often overlooking object semantics. To explore this difference, we propose an approach that isolates core visual features crucial for human perception and object recognition: color, texture, and shape. In experiments on three benchmarks-Fruits 360, CIFAR-10, and Fashion MNIST-each visual feature is individually input into a neural network. Results reveal data set-dependent variations in classification accuracy, highlighting that deep learning models tend to learn pixel-level correlations instead of fundamental visual features. To validate this observation, we used various combinations of concatenated visual features as input for a neural network on the CIFAR-10 data set. CNNs excel at learning statistical patterns in images, achieving exceptional performance when training and test data share similar distributions. To substantiate this point, we trained a CNN on CIFAR-10 data set and evaluated its performance on the "dog" class from CIFAR-10 and on an equivalent number of examples from the Stanford Dogs data set. The CNN poor performance on Stanford Dogs images underlines the disparity between deep learning and human visual perception, highlighting the need for models that learn object semantics. Specialized benchmark data sets with controlled variations hold promise for aligning learned representations with human cognition in computer vision research.

Stabilized GAN models training with kernel-histogram transformation and probability mass function distance

Image generation using generative adversarial networks (GANs) has been extensively researched in recent years. Despite active developments, the chronic issue of training instability in GANs remains unresolved. To alleviate this problem, this study proposes a model named probability mass function GANs (PMF-GAN), which handles the inherent limitation of GANs. The PMF-GAN framework employs kernels, histogram transformation, and probability mass function (PMF) distance for distribution learning. The configuration of PMF-GAN kernel and PMF distance offers flexibility, allowing for optimal settings tailored to datasets and experimental environments. In this study, experiments were conducted using the gaussian kernel across five different distances. The experiments demonstrated that PMF-GAN outperforms the baselines in terms of visual quality and evaluation metrics, such as Inception score and Frechet Inception distance (FID). For example, in the CIFAR-10 dataset, Euclidean-based PMF-GAN applying with 3 bins showed a 21.5 % and 32.8 % improvement in Inception score and FID, respectively, compared to conventional WGAN-GP. Similarly, in the AFHQ dataset with the same settings, the improvements were 56.9 % and 61.5 %. As a result, this study presents the potential to achieve stable training processes in GAN models with modified loss function structures. The flexibility of the proposed model allows for simultaneous application to various models, contributing to the overall improvement of generative model training processes in the future.

Spatial-frequency gradient fusion based model augmentation for high transferability adversarial attack

In recent years, deep learning has gained widespread application across diverse fields, including image classification and machine translation. Nevertheless, the emergence of adversarial examples has revealed a vulnerability of deep learning techniques to potential attacks. Despite the introduction of diverse adversarial attack methods, they are still constrained by certain limitations. Specifically, global perturbations are easily discernible by humans, resulting in poor imperceptibility. Additionally, current methods encounter limited transferability due to their reliance on attacking specific models. To address these challenges, this paper proposed a spatial-frequency gradient fusion based model augmentation for adversarial attack. First, we utilize a Gaussian convolution kernel to pinpoint regions in images that exhibit significant pixel variation, aiming to generate locally imperceptible perturbations undetectable by humans. These areas, which we consider as complex texture regions, are ideal for adding perturbations. Then, we design a perceptual similarity constraint to regulate the generation of perturbations in smooth texture regions. Subsequently, to further enhance the transferability of our method, we propose a spatial-frequency gradient fusion based model augmentation, applying random spectral transformation to shift into the frequency domain for narrowing the differences between models. Additionally, we design complex region scaling transformations in the spatial domain, aimed at capturing common features shared across models. Finally, we integrate the gradients from both the spatial and frequency domains, leveraging the strengths of both to empower attack models in effectively simulating the target model. Extensive experiments conducted on ImageNet and CIFAR-10 datasets have shown that our method attains a remarkable black-box attack success rate of up to 93.1%, with a perceptual loss reduction of approximately 8.39%, while also exhibiting stronger robustness.