CIFAR-10 Datasets Research Articles

Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

PurposeThe purpose of this study is to explore the potential of trainable activation functions to enhance the performance of deep neural networks, specifically ResNet architectures, in the task of image classification. By introducing activation functions that adapt during training, the authors aim to determine whether such flexibility can lead to improved learning outcomes and generalization capabilities compared to static activation functions like ReLU. This research seeks to provide insights into how dynamic nonlinearities might influence deep learning models' efficiency and accuracy in handling complex image data sets.Design/methodology/approachThis research integrates three novel trainable activation functions – CosLU, DELU and ReLUN – into various ResNet-n architectures, where “n” denotes the number of convolutional layers. Using CIFAR-10 and CIFAR-100 data sets, the authors conducted a comparative study to assess the impact of these functions on image classification accuracy. The approach included modifying the traditional ResNet models by replacing their static activation functions with the trainable variants, allowing for dynamic adaptation during training. The performance was evaluated based on accuracy metrics and loss profiles across different network depths.FindingsThe findings indicate that trainable activation functions, particularly CosLU, can significantly enhance the performance of deep learning models, outperforming the traditional ReLU in deeper network configurations on the CIFAR-10 data set. CosLU showed the highest improvement in accuracy, whereas DELU and ReLUN offered varying levels of performance enhancements. These functions also demonstrated potential in reducing overfitting and improving model generalization across more complex data sets like CIFAR-100, suggesting that the adaptability of activation functions plays a crucial role in the training dynamics of deep neural networks.Originality/valueThis study contributes to the field of deep learning by introducing and evaluating the impact of three novel trainable activation functions within widely used ResNet architectures. Unlike previous works that primarily focused on static activation functions, this research demonstrates that incorporating trainable nonlinearities can lead to significant improvements in model performance and adaptability. The introduction of CosLU, DELU and ReLUN provides a new pathway for enhancing the flexibility and efficiency of neural networks, potentially setting a new standard for future deep learning applications in image classification and beyond.

Harnessing Neuron Stability to Improve DNN Verification

Deep Neural Networks (DNN) have emerged as an effective approach to tackling real-world problems. However, like human-written software, DNNs are susceptible to bugs and attacks. This has generated significant interest in developing effective and scalable DNN verification techniques and tools. Recent developments in DNN verification have highlighted the potential of constraint-solving approaches that combine abstraction techniques with SAT solving. Abstraction approaches are effective at precisely encoding neuron behavior when it is linear, but they lead to overapproximation and combinatorial scaling when behavior is non-linear. SAT approaches in DNN verification have incorporated standard DPLL techniques, but have overlooked important optimizations found in modern SAT solvers that help them scale on industrial benchmarks. In this paper, we present VeriStable, a novel extension of the recently proposed DPLL-based constraint DNN verification approach. VeriStable leverages the insight that while neuron behavior may be non-linear across the entire DNN input space, at intermediate states computed during verification many neurons may be constrained to have linear behavior – these neurons are stable. Efficiently detecting stable neurons reduces combinatorial complexity without compromising the precision of abstractions. Moreover, the structure of clauses arising in DNN verification problems shares important characteristics with industrial SAT benchmarks. We adapt and incorporate multi-threading and restart optimizations targeting those characteristics to further optimize DPLL-based DNN verification. We evaluate the effectiveness of VeriStable across a range of challenging benchmarks including fully- connected feedforward networks (FNNs), convolutional neural networks (CNNs) and residual networks (ResNets) applied to the standard MNIST and CIFAR datasets. Preliminary results show that VeriStable is competitive and outperforms state-of-the-art DNN verification tools, including α-β-CROWN and MN-BaB, the first and second performers of the VNN-COMP, respectively.

Predictive Power of a Bayesian Effective Action for Fully Connected One Hidden Layer Neural Networks in the Proportional Limit.

We perform accurate numerical experiments with fully connected one hidden layer neural networks trained with a discretized Langevin dynamics on the MNIST and CIFAR10 datasets. Our goal is to empirically determine the regimes of validity of a recently derived Bayesian effective action for shallow architectures in the proportional limit. We explore the predictive power of the theory as a function of the parameters (the temperature T, the magnitude of the Gaussian priors λ_{1}, λ_{0}, the size of the hidden layer N_{1}, and the size of the training set P) by comparing the experimental and predicted generalization error. The very good agreement between the effective theory and the experiments represents an indication that global rescaling of the infinite-width kernel is a main physical mechanism for kernel renormalization in fully connected Bayesian standard-scaled shallow networks.

Application research of image classification based on Pytorch and Convolutional neural network

The image classification problem of convolutional neural network (CNN) on CIFAR-10 dataset is studied, and the model is implemented and experimented with PyTorch. Through the training and testing of the model, we analyze the performance of the model and explore the possibility of improvement. The experimental results show that the model achieves a certain accuracy in the image classification task, but there is space for improvement in some categories.

Structural prior-driven feature extraction with gradient-momentum combined optimization for convolutional neural network image classification

Recent image classification efforts have achieved certain success by incorporating prior information such as labels and logical rules to learn discriminative features. However, these methods overlook the variability of features, resulting in feature inconsistency and fluctuations in model parameter updates, which further contribute to decreased image classification accuracy and model instability. To address this issue, this paper proposes a novel method combining structural prior-driven feature extraction with gradient-momentum (SPGM), from the perspectives of consistent feature learning and precise parameter updates, to enhance the accuracy and stability of image classification. Specifically, SPGM leverages a structural prior-driven feature extraction (SPFE) approach to calculate gradients of multi-level features and original images to construct structural information, which is then transformed into prior knowledge to drive the network to learn features consistent with the original images. Additionally, an optimization strategy integrating gradients and momentum (GMO) is introduced, dynamically adjusting the direction and step size of parameter updates based on the angle and norm of the sum of gradients and momentum, enabling precise model parameter updates. Extensive experiments on CIFAR10 and CIFAR100 datasets demonstrate that the SPGM method significantly reduces the top-1 error rate in image classification, enhances the classification performance, and outperforms state-of-the-art methods.

A CNN pruning approach using constrained binary particle swarm optimization with a reduced search space for image classification

Deep convolutional neural networks (CNNs) have exhibited exceptional performance in a range of computer vision tasks. However, these deep CNNs typically demand significant computational resources, which not only hinders their practical deployment but also contributes to a considerable carbon footprint. To tackle this issue, several filter pruning methods based on evolutionary algorithms have been proposed to provide significant memory and energy savings during CNN inference. However, due to the curse of high dimensionality in the structure of deep CNNs, the search space expands dramatically, presenting significant challenges for these methods. This paper proposes a novel algorithm called BPSO-FPruner for CNN filter pruning. BPSO-FPruner utilizes a constrained binary particle swarm optimization algorithm for filter pruning, incorporating a new initialization strategy based on filter weighting information and a reduced search space strategy. Extensive validation using VGG, ResNet, DenseNet, and MobileNetv2 architectures on the CIFAR-10, CIFAR-100, and Tiny ImageNet datasets demonstrates the effectiveness of BPSO-FPruner in reducing model computational costs and carbon footprint emissions while maintaining or improving performance.

A High-Performance and Ultra-Low-Power Accelerator Design for Advanced Deep Learning Algorithms on an FPGA

This article addresses the growing need in resource-constrained edge computing scenarios for energy-efficient convolutional neural network (CNN) accelerators on mobile Field-Programmable Gate Array (FPGA) systems. In particular, we concentrate on register transfer level (RTL) design flow optimization to improve programming speed and power efficiency. We present a re-configurable accelerator design optimized for CNN-based object-detection applications, especially suitable for mobile FPGA platforms like the Xilinx PYNQ-Z2. By not only optimizing the MAC module using Enhanced clock gating (ECG), the accelerator can also use low-power techniques such as Local explicit clock gating (LECG) and Local explicit clock enable (LECE) in memory modules to efficiently minimize data access and memory utilization. The evaluation using ResNet-20 trained on the CIFAR-10 dataset demonstrated significant improvements in power efficiency consumption (up to 22%) and performance. The findings highlight the importance of using different optimization techniques across multiple hardware modules to achieve better results in real-world applications.

Efficient and lightweight convolutional neural network architecture search methods for object classification

Determining the architecture of deep learning models is a complex task. Several automated search techniques have been proposed, but these methods typically require high-performance graphics processing units (GPUs), manual parameter adjustments, and specific training approaches. This study introduces an efficient, lightweight convolutional neural network architecture search approach tailored for object classification. It features an optimized search space design and a novel controller design.This study introduces a refined search space design incorporating optimizations in both spatial and operational aspects. The focus is on the synergistic integration of convolutional units, dimension reduction units, and the stacking of Convolutional Neural Network (CNN) architectures. To enhance the search space, ShuffleNet modules are integrated, reducing the number of parameters and training time. Additionally, BlurPool is implemented in the dimension reduction unit operation to achieve translational invariance, alleviate the gradient vanishing problem, and optimize unit compositions. Moreover, an innovative controller model, Stage LSTM, is proposed based on Long Short-Term Memory (LSTM) to generate lightweight architectural sequences. In conclusion, the refined search space design and the Stage LSTM controller model are synergistically combined to establish an efficient and lightweight architecture search technique termed Stage and Lightweight Network Architecture Search (SLNAS).The experimental results highlight the superior performance of the optimized search space design, primarily when implemented with the Stage LSTM controller model. This approach shows significantly improved accuracy and stability compared to random, traditional LSTM, and Genetic Algorithm (GA) controller models, with statistically significant differences. Notably, the Stage LSTM controller excels in accuracy while producing models with fewer parameters within the expanded architecture search space.The study adopts the Stage LSTM controller model due to its ability to approximate optimal sequence structures, particularly when combined with the optimized search space design, referred to as SLNAS. SLNAS's performance is evaluated through experiments and comparisons with other Neural Architecture Search (NAS) and object classification methods from different researchers. These experiments consider model parameters, hardware resources, model stability, and multiple datasets. The results show that SLNAS achieves a low error rate of 2.86 % on the CIFAR-10 dataset after just 0.2 days of architecture search, matching the performance of manually designed models but using only 2 % of the parameters. SLNAS consistently demonstrates robust performance across various image classification domains, with an approximate parameter count 700,000.To summarize, SLNAS emerges as a highly effective automated network architecture search method tailored for image classification. It streamlines the model design process, making it accessible to researchers without specialized knowledge in deep learning. Optimizing this method unlocks the full potential of deep learning across diverse research areas. Interested parties can publicly access the source code and pre-trained models through the following link: https://github.com/huanyu-chen/LNASG-and-SLNAS-model.

LDD: High-Precision Training of Deep Spiking Neural Network Transformers Guided by an Artificial Neural Network.

The rise of large-scale Transformers has led to challenges regarding computational costs and energy consumption. In this context, spiking neural networks (SNNs) offer potential solutions due to their energy efficiency and processing speed. However, the inaccuracy of surrogate gradients and feature space quantization pose challenges for directly training deep SNN Transformers. To tackle these challenges, we propose a method (called LDD) to align ANN and SNN features across different abstraction levels in a Transformer network. LDD incorporates structured feature knowledge from ANNs to guide SNN training, ensuring the preservation of crucial information and addressing inaccuracies in surrogate gradients through designing layer-wise distillation losses. The proposed approach outperforms existing methods on the CIFAR10 (96.1%), CIFAR100 (82.3%), and ImageNet (80.9%) datasets, and enables training of the deepest SNN Transformer network using ImageNet.

Federated learning for spiking neural networks by hint-layer knowledge distillation

In recent years, research on the federated Spiking Neural Network (SNN) has attracted increasing attention because of its advantages of low-power consumption and privacy security. However, existing federated SNN researches rely primarily on the Federated Average (FedAvg) strategy, transferring full network parameters between the server and clients and incurring substantial communication. To address this issue, we propose a Hint-layer Distillation-based Spiking Federated Learning (HDSFL) framework that reduces the communication cost by transferring knowledge and losslessly compressing the spiking tensor. To compensate for the information loss due to the binary representation of spikes in knowledge distillation, we introduce the hint-layer information instead of just soft-label distribution. To aggregate the knowledge effectively across clients on the server, we process the weighted knowledge aggregation on the spiking knowledge representation based on local performance. The experimental results on four classical and large-scale benchmarks show that our method reduces the communication cost by about 1–2 orders of magnitude compared to conventional methods, while achieving comparable accuracies. Especially on the CIFAR-10 dataset, our method achieves the same accuracy as FedAvg using only 20% of the communications, and consumes only 5.8% communications of FedAvg in a single round.

MobileDenseNeXt: Investigations on biomedical image classification

Background and objectiveWe are living in the information era. Therefore, intelligence-based researchers are hot-topic such as artificial intelligence. In the artificial intelligence research area, machine learning and deep learning models have frequently used to create intelligence assistants and deep learning is the shining star of the AI. Specifically, in the computer vision, numerous deep learning models have been proposed, leading to a competition between transformers and convolutional neural networks (CNNs). Since the introduction of Vision Transformers (ViT), many transformer models have been advocated for computer vision, often overshadowing CNNs. Therefore, it is crucial to propose CNNs to showcase their prowess in image classification. This research introduces a lightweight CNN named MobileDenseNeXt. Materials and methodsThe proposed MobileDenseNeXt comprises four main blocks: (i) input, (ii) main, (iii) average pooling-based downsampling, and (iv) output. This research also incorporates convolution-based residual blocks and uses a depth concatenation layer to increase the number of filters. For downsampling, an average pooling operation has been employed, similar to the original DenseNet. Furthermore, the swish activation function is utilized in the presented CNN. MobileDenseNeXt has approximately 1.4 million learnable parameters, categorizing it as a lightweight CNN model. Additionally, a deep feature engineering approach has been developed using MobileDenseNeXt, incorporating two feature extractors with global average pooling and dropout layers, along with 10 feature selectors, to demonstrate the transfer learning capabilities of MobileDenseNeXt. ResultsThe recommended models achieved over 95% test classification accuracy on the used three datasets, unequivocally demonstrating the high image classification proficiency of the proposed MobileDenseNeXt. Moreover, to show general classification ability of the proposed model, MobileDenseNeXt was trained on the CIFAR10 dataset and reached 98.62% accuracy. ConclusionsThis research not only highlights the efficiency and effectiveness of MobileDenseNeXt in biomedical image classification but also highlights the competitive potential of this model for computer vision.

Asynchronous SGD with stale gradient dynamic adjustment for deep learning training

Asynchronous stochastic gradient descent (ASGD) is a computationally efficient algorithm, which speeds up deep learning training and plays an important role in distributed deep learning. However, ASGD suffers from the stale gradient problem, i.e., the gradient of worker may mismatch the weight of parameter server. This problem seriously affects the model performance and even causes the divergence. To address this issue, this paper designs a dynamic adjustment scheme via the momentum algorithm, which uses both stale penalty and stale compensation, i.e., stale penalty is to reduce the trust in stale gradient, stale compensation is to compensate the hurt of stale gradient. Based on this dynamic adjustment scheme, this paper proposes a dynamic asynchronous stochastic gradient descent algorithm (DASGD), which dynamically adjusts the compensation factor and the penalty factor via stale size. Moreover, we prove that DASGD is convergent under some mild assumptions. Finally, we build a real distributed training cluster to evaluate our DASGD on Cifar10 and ImageNet datasets. Compared with four SOTA baselines, experiment results confirm the superior performance of DASGD. More specifically, our DASGD has nearly the same test accuracy as SGD on Cifar10 and ImageNet, and only uses around 27.6% and 40.8% training time that of SGD, respectively.

Reweighted Alternating Direction Method of Multipliers for DNN weight pruning

As Deep Neural Networks (DNNs) continue to grow in complexity and size, leading to a substantial computational burden, weight pruning techniques have emerged as an effective solution. This paper presents a novel method for dynamic regularization-based pruning, which incorporates the Alternating Direction Method of Multipliers (ADMM). Unlike conventional methods that employ simple and abrupt threshold processing, the proposed method introduces a reweighting mechanism to assign importance to the weights in DNNs. Compared to other ADMM-based methods, the new method not only achieves higher accuracy but also saves considerable time thanks to the reduced number of necessary hyperparameters. The method is evaluated on multiple architectures, including LeNet-5, ResNet-32, ResNet-56, and ResNet-50, using the MNIST, CIFAR-10, and ImageNet datasets, respectively. Experimental results demonstrate its superior performance in terms of compression ratios and accuracy compared to state-of-the-art pruning methods. In particular, on the LeNet-5 model for the MNIST dataset, it achieves compression ratios of 355.9× with a slight improvement in accuracy; on the ResNet-50 model trained with the ImageNet dataset, it achieves compression ratios of 4.24× without sacrificing accuracy.

An attention mechanism network based on the winner-take-all

In the field of neurology, neurons compete for brain attention in winner-take-all (WTA) competitions. Inspired by this, we propose a new WTA-based attention network (called ANW), which can be extended to general neural networks. The ANW network simulates WTA behavior in biological neurons; that is, the winner neuron itself becomes excited and sends its own spike signal. Furthermore, the winner will output an inhibition signal, which will be input into the cell synapses of other neurons. The ANW network regards the update of neuron weights after backpropagation as a stimulus input to neurons. Combined with the WTA function, the neural network will continue to update until it reaches a stable state. In this way, the ANW network can find the most active neurons in the network and generate the winners among these neurons to improve the ability of the neural network to extract features. As the ANW is a small and universal network, it can be seamlessly inserted into any neural network architecture, with negligible overhead, and can be trained together with a basic CNN. We verified the neural network with ANW by using the ImageNet-100, CIFAR-100 and MS COCO detection datasets for the experiments. The experimental results show that, compared with those of the baseline network, the top-1 err gains of 0.63% and 3.85% can be achieved for ANW, with classical ResNet-50 as the backbone network, respectively, demonstrating that the method can improve the performance of the model in image classification and target detection with wide applicability. The proposed method provides an effective, and almost consumption-neutral solution, for performance improvement in the field of computer vision target detection.

Image Processing Technology Based on Machine Learning

Machine learning is a relatively new field. With the deepening of people's research in this field, the application of machine learning is increasingly extensive. On the other hand, with the development of science and technology, image has become an indispensable medium of information transmission, and image processing technology is also booming. This paper introduces machine learning into image processing, and studies the image processing technology based on machine learning. This paper summarizes the current popular image processing technology, compares various image technology in detail, and explains the limitations of each image processing method. In addition, on the basis of image processing, this paper introduces machine learning algorithm, applies convolution neural network to feature extraction of image processing, and carries out simulation test. In the test, we select voc2007 dataset for image segmentation, Imagenet dataset for target detection, cifar100 dataset for image classification, and ROC curve for performance evaluation. The results show that the algorithm based on deep learning can achieve high accuracy in image segmentation, classification and target detection. The accuracy of image segmentation is 0.984, the accuracy of image classification is 0.987, and the accuracy of target detection is 0.986. Image processing based on machine learning has great advantages.

Image Classification Using CNN with CIFAR-10 Dataset

Abstract: Image classification is a very important issue in digital image analysis. CNN is type of Deep Neural Networks (DNN) that contains multiple layerssuch as Conv layers, integration layer and fully integrated layer. Convolutional neural networks (CNNs) are widely used in pattern-and picture- recognition problems as they have a number of advantages compared to others strategies. CIFAR-10is a very popular computer vision database. Thisdatabase is well read in it many types of in-depth study of object recognition. This database contains 60,000 images separated by 10 target classes, each a section containing 6000 images of 32 * 32 shapes. This database contains images of low-resolution (32 * 32), which allows researchers to experiment with new algorithms. Image classification is a fundamentaltask in computer vision that involves assigning a labelor category to an image. Convolutional Neural Networks (CNNs) have become the state-of-the-art method for image classification due to their ability toautomatically learn features from raw pixel values. Inthis project, we aim to build a CNN model that can accurately classify images in the CIFAR-10 dataset.

Matching the Ideal Pruning Method with Knowledge Distillation for Optimal Compression

In recent years, model compression techniques have gained significant attention as a means to reduce the computational and memory requirements of deep neural networks. Knowledge distillation and pruning are two prominent approaches in this domain, each offering unique advantages in achieving model efficiency. This paper investigates the combined effects of knowledge distillation and two pruning strategies, weight pruning and channel pruning, on enhancing compression efficiency and model performance. The study introduces a metric called “Performance Efficiency” to evaluate the impact of these pruning strategies on model compression and performance. Our research is conducted on the popular datasets CIFAR-10 and CIFAR-100. We compared diverse model architectures, including ResNet, DenseNet, EfficientNet, and MobileNet. The results emphasize the efficacy of both weight and channel pruning in achieving model compression. However, a significant distinction emerges, with weight pruning showing superior performance across all four architecture types. We realized that the weight pruning method better adapts to knowledge distillation than channel pruning. Pruned models show a significant reduction in parameters without a significant reduction in accuracy.

GLNAS: Greedy Layer-wise Network Architecture Search for low cost and fast network generation

The process of applying machine learning algorithms to practical problems can be a challenging and tedious task for non-experts. Previous research has sought to alleviate this burden by introducing automated machine learning techniques, including Network Architecture Search (NAS) and Differentiable Architecture Search (DARTS). However, these methods use a fixed number of layers and predefined skip connections which impose limitations on the generation of an optimal network architecture. In this paper, we propose a novel approach called Greedy Layer-wise Network Architecture Search (GLNAS), which trains network layers one after another and evaluates the network’s performance after each layer is added. GLNAS also assesses the effectiveness of skip connections between layers by testing various outputs of previous layers as an input to the current layer. Our experiment results demonstrate that the network generated by GLNAS requires fewer parameters (i.e., 3.5 millions in both CIFAR-10 and CIFAR-100 datasets) and GPU resources during the searching phase (i.e., 0.17 and 0.24 GPU days in CIFAR-10 and CIFAR-100 datasets respectively) than many existing methods.

DualAD: Dual adversarial network for image anomaly detection⋆

AbstractAnomaly Detection, also known as outlier detection, is critical in domains such as network security, intrusion detection, and fraud detection. One popular approach to anomaly detection is using autoencoders, which are trained to reconstruct input by minimising reconstruction error with the neural network. However, these methods usually suffer from the trade‐off between normal reconstruction fidelity and abnormal reconstruction distinguishability, which damages the performance. The authors find that the above trade‐off can be better mitigated by imposing constraints on the latent space of images. To this end, the authors propose a new Dual Adversarial Network (DualAD) that consists of a Feature Constraint (FC) module and a reconstruction module. The method incorporates the FC module during the reconstruction training process to impose constraints on the latent space of images, thereby yielding feature representations more conducive to anomaly detection. Additionally, the authors employ dual adversarial learning to model the distribution of normal data. On the one hand, adversarial learning was implemented during the reconstruction process to obtain higher‐quality reconstruction samples, thereby preventing the effects of blurred image reconstructions on model performance. On the other hand, the authors utilise adversarial training of the FC module and the reconstruction module to achieve superior feature representation, making anomalies more distinguishable at the feature level. During the inference phase, the authors perform anomaly detection simultaneously in the pixel and latent spaces to identify abnormal patterns more comprehensively. Experiments on three data sets CIFAR10, MNIST, and FashionMNIST demonstrate the validity of the authors’ work. Results show that constraints on the latent space and adversarial learning can improve detection performance.

The educational resource management based on image data visualization and deep learning

In order to address issues such as inaccurate education resource positioning and inefficient resource utilization, this study optimizes the Educational Resource Management System (ERMS) by combining image data visualization techniques with convolutional neural networks (CNNs) technology in deep learning. Firstly, the crucial role of ERMS in education and teaching is analyzed. Secondly, the application of image data visualization techniques and CNNs in the system is explained, along with the associated challenges. Finally, by optimizing the CNNs model and system architecture and validating with experimental data, the rationality of the proposed model is confirmed. Experimental results indicate a significant improvement in various performance metrics compared to traditional models. The recognition accuracy on the Mnist dataset reaches 98.1 %, and notably, on the cifar-10 dataset, the optimized model achieves an accuracy close to 98.3 % with improved runtime reduced to only 640.4 s. Additionally, through systematic simulation experiments, the designed system is shown to fully meet the earlier requirements for system functionality, validating the feasibility and rationality of the model and system in this study. Therefore, this study holds high practical value for optimizing ERMS and provides meaningful insights into image data visualization techniques and CNNs optimization.