ImageNet Dataset Research Articles

How deep is your art: An experimental study on the limits of artistic understanding in a single-task, single-modality neural network.

Computational modeling of artwork meaning is complex and difficult. This is because art interpretation is multidimensional and highly subjective. This paper experimentally investigated the degree to which a state-of-the-art Deep Convolutional Neural Network (DCNN), a popular Machine Learning approach, can correctly distinguish modern conceptual art work into the galleries devised by art curators. Two hypotheses were proposed to state that the DCNN model uses Exhibited Properties for classification, like shape and color, but not Non-Exhibited Properties, such as historical context and artist intention. The two hypotheses were experimentally validated using a methodology designed for this purpose. VGG-11 DCNN pre-trained on ImageNet dataset and discriminatively fine-tuned was trained on handcrafted datasets designed from real-world conceptual photography galleries. Experimental results supported the two hypotheses showing that the DCNN model ignores Non-Exhibited Properties and uses only Exhibited Properties for artwork classification. This work points to current DCNN limitations, which should be addressed by future DNN models.

Continual compression model for online continual learning

Task-Free Continual Learning (TFCL) presents a notably demanding but realistic ongoing learning concept, aiming to address catastrophic forgetting in sequential learning systems. In this paper, we tackle catastrophic forgetting by introducing an innovative dynamic expansion framework designed to adaptively enhance the model’s capacity for novel data learning while also remembering the information learnt in the past, by using a minimal-size processing architecture. Our proposed framework incorporates three key mechanisms to mitigate model’ forgetting: (1) by employing a Maximum Mean Discrepancy (MMD)-based expansion mechanism that assesses the disparity between previously acquired knowledge and that from the new training data, serving as a signal for the model’s architecture expansion; (2) a component discarding mechanism that eliminates components characterized by redundant information, thereby optimizing the model size while fostering knowledge diversity; (3) a novel training sample selection strategy that leads to the diversity of the training data for each task. We conduct a series of TFCL experiments that demonstrate the superiority of the proposed framework over all baselines while utilizing fewer components than alternative dynamic expansion models. The results on the Split Mini ImageNet dataset, after splitting the original dataset into multiple tasks, are improved by more than 2% when compared to the closest baseline.

A Strip Steel Surface Defect Salient Object Detection Based on Channel, Spatial and Self-Attention Mechanisms

Strip steel is extensively utilized in industries such as automotive manufacturing and aerospace due to its superior machinability, economic benefits, and adaptability. However, defects on the surface of steel strips, such as inclusions, patches, and scratches, significantly affect the performance and service life of the product. Therefore, the salient object detection of surface defects on strip steel is crucial to ensure the quality of the final product. Many factors, such as the low contrast of surface defects on strip steel, the diversity of defect types, complex texture structures, and irregular defect distribution, hinder existing detection technologies from accurately identifying and segmenting defect areas against complex backgrounds. To address the above problems, we propose a novel detector called S3D-SOD for the salient object detection of strip steel surface defects. For the encoding stage, a residual self-attention block is proposed to explore semantic information cues of high-level features to locate and guide low-level feature information. In addition, we apply a general residual channel and spatial attention to low-level features, enabling the model to adaptively focus on the key channels and spatial areas of feature maps with high resolutions, thereby enhancing the encoder features and accelerating the convergence of the model. For the decoding stage, a simple residual decoder block with an upsampling operation is proposed to realize the integration and interaction of feature information between different layers. Here, the simple residual decoder block is used for feature integration due to the following observation: backbone networks like ResNet and the Swin Transformer, after being pretrained on the large dataset ImageNet and then fine-tuned on a smaller dataset for strip steel surface defects, are capable of extracting feature maps that contain both general image features and the specific characteristics required for the salient object detection of strip steel surface defects. The experimental results on the SD-saliency-900 dataset show that S3D-SOD is better than advanced methods, and it has strong generalization ability and robustness.

SSAT: Active Authorization Control and User’s Fingerprint Tracking Framework for DNN IP Protection

As training a high-performance deep neural network (DNN) model requires a large amount of data, powerful computing resources and expert knowledge, protecting well-trained DNN models from intellectual property (IP) infringement has raised serious concerns in recent years. Most existing methods using DNN watermarks to verify the ownership of the models after IP infringement occurs, which is reactive in the sense that they cannot prevent unauthorized users from using the model in the first place. Different from these methods, in this article, we propose an active authorization control and user’s fingerprint tracking method for the IP protection of DNN models by utilizing sample-specific backdoor attack. The proposed method inversely and multiplely exploits sample-specific trigger as the key to implement authorization control for DNN model, in which the generated triggers are imperceptible and sample-specific for clean images. Specifically, a U-Net model is used to generate backdoor instances. Then, the target model is trained on the clean images and backdoor instances, which are inversely labeled as wrong classes and correct classes, respectively. Only authorized users can use the target model normally by pre-processing the clean images through the U-Net model. Moreover, the images processed by the U-Net model will contain unique fingerprint that can be extracted to verify and track the corresponding user’s identity. This article is the first work that utilizes the sample-specific backdoor attack to implement active authorization control and user’s fingerprint management for DNN model under black-box scenarios. Extensive experimental results on ImageNet dataset and YouTube Aligned Face dataset demonstrate that the proposed method is effective in protecting the DNN model from unauthorized usage. Specifically, the protected model has a low inference accuracy (1.00%) for unauthorized users, while maintaining a normal inference accuracy (97.67%) for authorized users. Besides, the proposed method can achieve 100% fingerprint tracking success rates on both the ImageNet and YouTube Aligned Face datasets. Moreover, it is demonstrated that the proposed method is robust against fine-tuning attack, pruning attack, pruning attack with retraining, reverse-engineering attack, adaptive attack, and JPEG compression attack. The code is available at https://github.com/nuaaaisec/SSAT .

Illuminating Salient Contributions in Neuron Activation with Attribution Equilibrium.

With the remarkable success of deep neural networks, there is a growing interest in research aimed at providing clear interpretations of their decision-making processes. In this paper, we introduce Attribution Equilibrium, a novel method to decompose output predictions into fine-grained attributions, balancing positive and negative relevance for clearer visualization of the evidence behind a network decision. We carefully analyze conventional approaches to decision explanation and present a different perspective on the conservation of evidence. We define the evidence as a gap between positive and negative influences among gradient-derived initial contribution maps. Then, we incorporate antagonistic elements and a user-defined criterion for the degree of positive attribution during propagation. Additionally, we consider the role of inactivated neurons in the propagation rule, thereby enhancing the discernment of less relevant elements such as the background. We conduct various assessments in a verified experimental environment with PASCAL VOC 2007, MS COCO 2014, and ImageNet datasets. The results demonstrate that our method outperforms existing attribution methods both qualitatively and quantitatively in identifying the key input features that influence model decisions.

LDF-BNN: A Real-Time and High-Accuracy Binary Neural Network Accelerator Based on the Improved BNext.

Significant progress has been made in industrial defect detection due to the powerful feature extraction capabilities of deep neural networks (DNNs). However, the high computational cost and memory requirement of DNNs pose a great challenge to the deployment of industrial edge-side devices. Although traditional binary neural networks (BNNs) have the advantages of small storage space requirements, high parallel computing capability, and low power consumption, the problem of significant accuracy degradation cannot be ignored. To tackle these challenges, this paper constructs a BNN with layered data fusion mechanism (LDF-BNN) based on BNext. By introducing the above mechanism, it strives to minimize the bandwidth pressure while reducing the loss of accuracy. Furthermore, we have designed an efficient hardware accelerator architecture based on this mechanism, enhancing the performance of high-accuracy BNN models with complex network structures. Additionally, the introduction of multi-storage parallelism alleviates the limitations imposed by the internal transfer rate, thus improving the overall computational efficiency. The experimental results show that our proposed LDF-BNN outperforms other methods in the comprehensive comparison, achieving a high accuracy of 72.23%, an image processing rate of 72.6 frames per second (FPS), and 1826 giga operations per second (GOPs) on the ImageNet dataset. Meanwhile, LDF-BNN can also be well applied to defect detection dataset Mixed WM-38, achieving a high accuracy of 98.70%.

Transfer learning-assisted modulation classification and OSNR estimation using constellation diagrams

Abstract A cost-effective deep transfer learning (TL) learned with characteristics derived from the constellation diagram is proposed for executing modulation classification and optical signal-to-noise ratio (OSNR) estimation for the future generation of elastic optical networks. The information acquired with the ImageNet dataset is transferred using pre-trained TL versions including VGG19, ResNet152V2, InceptionV3, and ResNet50 to identify various modulation formats and their corresponding OSNR. Adam optimizer is employed to ascertain the appropriate settings for the hyperparameters. Numerical results demonstrate that the proposed VGG19 method achieves the highest level of accuracy (100%) in the recognition of various modulation forms. To fulfil the requirements of actual use, OSNR monitoring is also explored, with a total precision equal to 95.2%. Furthermore, a thorough investigation of the impact of training data dimensions, on TL performance is conducted. The outcomes of the proposed method demonstrate that the suggested TL-based algorithms are more accurate and need much less training and testing time than non-TL approaches.

Adversarial sample attack method based on loss smoothing

Deep neural networks (DNNs) are vulnerable to adversarial examples.Although the existing momentum-based adversarial example generation method can achieve a close 100%white-box attack success rate, it is still not ideal when attacking other models, and the black- box attack success rate is low. To address this, an adversarial example attack method based on loss smoothing is proposed to improve the transferability of adversarial examples. In the iterative process of calculating the gradient at each step, the current gradient is not used directly, but the local average gradient is used to accumulate momentum, so as to suppress the local oscillation phenomenon on the loss function surface, thereby stabilizing the update direction and escaping the local extreme point. A large number of experimental results on the ImageNet dataset show that compared with the existing momentum-based method, the average black-box attack success rate of the proposed method in single model attack experiments is improved by 38.07%and 27.77%, and the average black-box attack success rate in integrated model attack experiments is improved by and.Rising32.50%and28.63%

Pathologist-level diagnosis of ulcerative colitis inflammatory activity level using an automated histological grading method

Background and AimsInflammatory bowel disease (IBD) is a global disease that is evolving with increasing incidence. However, there are few works on computationally assisted diagnosis of IBD based on pathological images. Therefore, based on the UK and Chinese IBD diagnostic guidelines, our study established an artificial intelligence-assisted diagnostic system for histologic grading of inflammatory activity in ulcerative colitis (UC). MethodsWe proposed an efficient deep-learning (DL) method for grading inflammatory activity in whole-slide images (WSIs) of UC pathology. Our model was constructed using 603 UC WSIs from Nanjing Drum Tower Hospital for model train set and internal test set. We collected 212 UC WSIs from Zhujiang Hospital as an external test set. Initially, the pre-trained ResNet50 model on the ImageNet dataset was employed to extract image patch features from UC patients. Subsequently, a multi-instance learning (MIL) approach with embedded self-attention was utilized to aggregate tissue image patch features, representing the entire WSI. Finally, the model was trained based on the aggregated features and WSI annotations provided by senior gastrointestinal pathologists to predict the level of inflammatory activity in UC WSIs. ResultsIn the task of distinguishing the presence or absence of inflammatory activity, the Area Under Curve (AUC) value in the internal test set is 0.863 (95% confidence interval [CI] 0.829, 0.898), with a sensitivity of 0.913 (95% [CI] 0.866, 0.961), and specificity of 0.816 (95% [CI] 0.771, 0.861). The AUC in the external test set is 0.947 (95% confidence interval [CI] 0.939, 0.955), with a sensitivity of 0.889 (905% [CI] 0.837, 0.940), and specificity of 0.858 (95% [CI] 0.777, 0.939). For distinguishing different levels of inflammatory activity in UC, the average Macro-AUC in the internal test set and the external test set are 0.827 (95% [CI] 0.803, 0.850) and 0.908 (95% [CI] 0.882, 0.935). the average Micro-AUC in the internal test set and the external test set are 0.816 (95% [CI] 0.792, 0.840) and 0.898 (95% [CI] 0.869, 0.926). ConclusionsComparative analysis with diagnoses made by pathologists at different expertise levels revealed that the algorithm reached a proficiency comparable to the pathologist with 5 years of experience. Furthermore, our algorithm performed superior to other MIL algorithms.

Accelerating one-shot neural architecture search via constructing a sparse search space

Neural Architecture Search (NAS) has garnered significant attention for its ability to automatically design high-quality deep neural networks (DNNs) tailored to various hardware platforms. The major challenge for NAS is the time-consuming network estimation process required to select optimal networks from a large pool of candidates. Rather than training each candidate from scratch, recent one-shot NAS methods accelerate the estimation process by only training a supernet and sampling sub-networks from it, inheriting partial network architectures and weights. Despite significant acceleration, the supernet training with a large search space (i.e., the number of candidate sub-networks) still requires thousands of GPU hours to support high-quality sub-network sampling. In this work, we propose SparseNAS, an approach for one-shot NAS acceleration by reducing the redundancy of the search space. We observe that many sub-networks in the space are underperforming, with significant performance disparity to high-performance sub-networks. Crucially, this disparity can be observed early in the beginning of the supernet training. Therefore, we train an early predictor to learn this disparity and filter out high-quality networks in advance. Then, the supernet training will be conducted in this space sub-space. Compared to the state-of-the-art one-shot NAS, our SparseNAS reports a 3.1× training speedup with comparable network performance on the ImageNet dataset. Compared to the state-of-the-art acceleration method, SparseNAS reports a maximum of 1.5% higher Top-1 accuracy and 28% training cost reduction with a 7× bigger search space. Extensive experiment results demonstrated that SparseNAS achieves better trade-offs between efficiency and performance than state-of-the-art one-shot NAS.

A Knowledge Distillation Method Based on Evidence Theory to Prevent Catastrophic Forgetting

Abstract Incremental learning is an emerging machine-learning approach designed to prevent catastrophic forgetting while learning a new task with knowledge retained from previous tasks. However, existing methods often must fully utilize the distributional information of the old task model’s outputs. To alleviate this, we propose a class incremental learning method grounded in evidence theory to leverage the distributional information of outputs from the old task model, which integrates uncertainty estimation into the knowledge distillation process to ensure the new task model effectively learns from the old task model’s distribution information. Specifically, we incorporate uncertainty estimation based on evidence theory to calculate knowledge distillation loss during training. We compute the uncertainty estimates of outputs from both the new and old task models, and then they are used in the knowledge distillation loss calculation. We propose a novel classification strategy that considers outputs’ probability and uncertainty estimates, which could determine the sample category without requiring an additional training phase or supplementary models. Experiments on the CIFAR-100 and ImageNet datasets demonstrate the effectiveness of the proposed method, showing that it outperforms existing methods by improving accuracy by 1%-2%. Results suggest that leveraging the distribution information of model outputs can effectively mitigate catastrophic forgetting in deep learning models within incremental learning scenarios.

Lossy Image Compression with Stochastic Quantization

Introduction. Lossy image compression algorithms play a crucial role in various domains, including graphics, and image processing. As image information density increases, so do the resources required for processing and transmission. One of the most prominent approaches to address this challenge is color quantization, proposed by Orchard et al. (1991). This technique optimally maps each pixel of an image to a color from a limited palette, maintaining image resolution while significantly reducing information content. Color quantization can be interpreted as a clustering problem (Krishna et al. (1997), Wan (2019)), where image pixels are represented in a three-dimensional space, with each axis corresponding to the intensity of an RGB channel. The purpose of the paper. Scaling of traditional algorithms like K-Means can be challenging for large data, such as modern images with millions of colors. This paper reframes color quantization as a three-dimensional stochastic transportation problem between the set of image pixels and an optimal color palette, where the number of colors is a predefined hyperparameter. We employ Stochastic Quantization (SQ) with a seeding technique proposed by Arthur et al. (2007) to enhance the scalability of color quantization. This method introduces a probabilistic element to the quantization process, potentially improving efficiency and adaptability to diverse image characteristics. Results. To demonstrate the efficiency of our approach, we present experimental results using images from the ImageNet dataset. These experiments illustrate the performance of our Stochastic Quantization method in terms of compression quality, computational efficiency, and scalability compared to traditional color quantization techniques. Conclusions. This study introduces a scalable algorithm for solving the color quantization problem without memory constraints, demonstrating its efficiency on a subset of images from the ImageNet dataset. The convergence speed of the algorithm can be further enhanced by modifying the update rule with alternative methods to Stochastic Gradient Descent (SGD) that incorporate adaptive learning rates. Moreover, the stochastic nature of the proposed solution enables the utilization of parallelization techniques to simultaneously update the positions of multiple quants, potentially leading to significant performance improvements. This aspect of parallelization and its impact on algorithm efficiency presents a topic for future research. The proposed method not only addresses the limitations of existing color quantization techniques but also opens up new possibilities for optimizing image compression algorithms in resource-constrained environments. Keywords: non-convex optimization, stochastic optimization, stochastic quantization, color quantization, lossy compression.

Quantum Transfer Learning with Adversarial Robustness for Classification of High‐Resolution Image Datasets

AbstractThe application of quantum machine learning to large‐scale high‐resolution image datasets is not yet possible due to the limited number of qubits and relatively high level of noise in the current generation of quantum devices. In this work, this challenge is addressed by proposing a quantum transfer learning (QTL) architecture that integrates quantum variational circuits with a classical machine learning network pre‐trained on ImageNet dataset. Through a systematic set of simulations over a variety of image datasets such as Ants & Bees, CIFAR‐10, and Road Sign Detection, the superior performance of the QTL approach over classical and quantum machine learning without involving transfer learning is demonstrated. Furthermore, the adversarial robustness of QTL architecture with and without adversarial training is evaluated, confirming that our QTL method is adversarially robust against data manipulation attacks and outperforms classical methods.

Detection and diagnosis of diabetic eye diseases using two phase transfer learning approach.

Early diagnosis and treatment of diabetic eye disease (DED) improve prognosis and lessen the possibility of permanent vision loss. Screening of retinal fundus images is a significant process widely employed for diagnosing patients with DED or other eye problems. However, considerable time and effort are required to detect these images manually. Deep learning approaches in machine learning have attained superior performance for the binary classification of healthy and pathological retinal fundus images. In contrast, multi-class retinal eye disease classification is still a difficult task. Therefore, a two-phase transfer learning approach is developed in this research for automated classification and segmentation of multi-class DED pathologies. In the first step, a Modified ResNet-50 model pre-trained on the ImageNet dataset was transferred and learned to classify normal diabetic macular edema (DME), diabetic retinopathy, glaucoma, and cataracts. In the second step, the defective region of multiple eye diseases is segmented using the transfer learning-based DenseUNet model. From the publicly accessible dataset, the suggested model is assessed using several retinal fundus images. Our proposed model for multi-class classification achieves a maximum specificity of 99.73%, a sensitivity of 99.54%, and an accuracy of 99.67%.

Two-step deep-learning identification of heel keypoints from video-recorded gait.

Accurate and fast extraction of step parameters from video recordings of gait allows for richer information to be obtained from clinical tests such as Timed Up and Go. Current deep-learning methods are promising, but lack in accuracy for many clinical use cases. Extracting step parameters will often depend on extracted landmarks (keypoints) on the feet. We hypothesize that such keypoints can be determined with an accuracy relevant for clinical practice from video recordings by combining an existing general-purpose pose estimation method (OpenPose) with custom convolutional neural networks (convnets) specifically trained to identify keypoints on the heel. The combined method finds keypoints on the posterior and lateral aspects of the heel of the foot in side-view and frontal-view images from which step length and step width can be determined for calibrated cameras. Six different candidate convnets were evaluated, combining three different standard architectures as networks for feature extraction (backbone), and with two different networks for predicting keypoints on the heel (head networks). Using transfer learning, the backbone networks were pre-trained on the ImageNet dataset, and the combined networks (backbone + head) were fine-tuned on data from 184 trials of older, unimpaired adults. The data was recorded at three different locations and consisted of 193 k side-view images and 110 k frontal-view images. We evaluated the six different models using the absolute distance on the floor between predicted keypoints and manually labelled keypoints. For the best-performing convnet, the median error was 0.55 cm and the 75% quartile was below 1.26 cm using data from the side-view camera. The predictions are overall accurate, but show some outliers. The results indicate potential for future clinical use by automating a key step in marker-less gait parameter extraction.

CNN-based transfer learning for forest aboveground biomass prediction from ALS point cloud tomography

ABSTRACT This study presents a new approach for predicting forest aboveground biomass (AGB) from airborne laser scanning (ALS) data: AGB is predicted from sequences of images depicting vertical cross-sections through the ALS point clouds. A 3D version of the VGG16 convolutional neural network (CNN) with initial weights transferred from pre-training on the ImageNet dataset was used. The approach was tested on datasets from Canada, Poland, and the Czech Republic. To analyse the effect of training sample size on model performance, different-sized samples ranging from 10 to 375 ground plots were used. The CNNs were compared with random forest models (RFs) trained on point cloud metrics. At the maximum number of training samples, the difference in RMSE between observed and predicted AGB of CNNs and RFs ranged from −2 t/ha to 5 t/ha, and the difference in squared Pearson correlation coefficient ranged from −0.05 to 0.06. Additional pre-training on synthetic data derived from virtual laser scanning of simulated forest stands could only improve the prediction performance of the CNNs when only a few real training samples (10–40) were available. While 3D CNNs trained on cross-section images derived from real data showed promising results, RFs remain a competitive alternative.

80 Empowering informed choices: How computer vision can assist consumers in making decisions about meat quality

Abstract Consumer perception of meat quality is highly influenced by meat tenderness and juiciness. Artificial intelligence technology, such as computer vision systems (CVS), can be a powerful tool to predict such meat attributes in retail markets and help consumers make informed decisions. This study aimed to develop a CVS using smartphone images to classify beef steaks and pork chops tenderness (1), predicting shear-force (SF) and intramuscular fat (IMF) content (2), and performing a comparative evaluation between consumer assessments and the CVS (3). The dataset for beef steak consisted of 915 images (one image per steak), with 691 in the training set, 82 in the validation set, and 142 in the testing set. For pork, there were 514 images (one image per chop), distributed as 377 in the training set, 80 in the validation set, and 57 in the testing set. We trained deep neural networks (Xception), pretrained on the ImageNet dataset, for image classification and regression. The tenderness of the steaks was categorized based on Warner-Bratzler shear-force (N) as tender, intermediate, and tough. To achieve the third object, 1,000 pairwise comparisons were drawn from the 142 testing set images, where both consumers and the CVS performed predictions on tender and tough meat. The option with the least SF in each pair was identified as the most tender meat, with ground truth established using SF measurements. The same pairs were presented to meat consumers through a survey. For classifying beef as tender and tough, the algorithms demonstrated F1-scores of 68.1% and 70.8%, respectively. However, intermediate meat yielded worse results, with an F1-score of 36.4%. The dataset was subsequently re-categorized into two classes: tender and tough, with all intermediate steaks classified as tough, and resulted in an F1-score of 76.6% for tough meat. For classifying pork chops as tender and tough, the algorithms demonstrated F1-scores of 81.4% and 85.7%, respectively. The intermediate class in pork chops also exhibited worse results, with an F1-score of 24.0%. After re-categorizing the dataset into two classes, the F1-score for classifying pork chops as tough was 83.9%. The regression model predicted SF in beef steak and pork chops with R2 of 0.64 and 0.76, and RMSEP of 16.9 N and 9.15 N, respectively. The regression model, after analyzing with 1,000 pairwise comparisons, accurately predicted the tenderest steak with an accuracy of 76.5%. In contrast, human recognition of meat tenderness achieved an average accuracy of 46.7% for beef steak. To predict IMF in beef steaks and pork chops the model achieved R2 values of 0.61 and 0.54, and RMSEP of 2.6% and 1.22%, respectively. These findings suggest that CVS can provide a more objective method for evaluating meat tenderness and IMF before purchase, potentially enhancing consumer satisfaction.

A Clustering Pruning Method Based on Multidimensional Channel Information

Pruning convolutional neural networks offers a promising solution to mitigate the computational complexity challenges encountered during application deployment. However, prevalent pruning techniques primarily concentrate on model parameters or feature mapping analysis to devise static pruning strategies, often overlooking the underlying feature extraction capacity of convolutional kernels. To address this, the study first quantitatively expresses the feature extraction capability of convolutional channels from three aspects: global features, distribution metrics, and directional metrics. It explores the multi-dimensional information of the channels, calculates the overall expectation, variance, and cosine distance from the unit vector as the quantitative results of the channels. Subsequently, a clustering algorithm is employed to categorize the multidimensional information. This approach ensures that convolutional channels grouped within each cluster possess similar feature extraction capabilities. An enhanced differential evolutionary algorithm is utilized to optimize the number of clustering centers across all convolutional layers, ensuring optimal grouping. The final step involves achieving channel sparsification through the calculation of crowding distances for each sample within its designated cluster. This preserves a diverse subset of channels that are critical for maintaining model accuracy. Extensive empirical evaluations conducted on three benchmark image classification datasets demonstrate the efficacy of this method. For instance, on the ImageNet dataset, the ResNet-50 model experiences a substantial reduction in FLOPs by 58.43% while incurring a minimal decrease in TOP-1 accuracy of only 1.15%.

Layer‐parallel training of residual networks with auxiliary variable networks

AbstractGradient‐based methods for training residual networks (ResNets) typically require a forward pass of input data, followed by back‐propagating the error gradient to update model parameters, which becomes time‐consuming as the network structure goes deeper. To break the algorithmic locking and exploit synchronous module parallelism in both forward and backward modes, auxiliary‐variable methods have emerged but suffer from communication overhead and a lack of data augmentation. By trading off the recomputation and storage of auxiliary variables, a joint learning framework is proposed in this work for training realistic ResNets across multiple compute devices. Specifically, the input data of each processor is generated from its low‐capacity auxiliary network (AuxNet), which permits the use of data augmentation and realizes forward unlocking. Backward passes are then executed in parallel, each with a local loss function derived from the penalty or augmented Lagrangian (AL) method. Finally, the AuxNet is adjusted to reproduce updated auxiliary variables through an end‐to‐end training process. We demonstrate the effectiveness of our method on ResNets and WideResNets across CIFAR‐10, CIFAR‐100, and ImageNet datasets, achieving speedup over the traditional layer‐serial training approach while maintaining comparable testing accuracy.

Vision transformers inference acceleration based on adaptive layer normalization

Over the past decade, deep neural networks (DNNs) have been widely used due to their remarkable accuracy in real-world applications. However, this increase in accuracy often results in computationally expensive models and high memory usage, leading to longer prediction latencies and exorbitant release costs. In this paper, we propose a new adaptive layer normalization (ALN) algorithm for transformer models, which tackles the computational, and memory problems encountered by traditional layer normalization (LN) techniques. The proposed method computes and stores statistical moments during the training and uses them directly during the inference phase, allowing the normalization layer to be merged with the nearest linear layer. The result is a significant acceleration in inference time, by up to 29%. In classification tasks, our evaluations on the ImageNet dataset show an improvement in accuracy of 0.1%, while maintaining comparable accuracy in object detection tasks on the COCO reference dataset. The proposed ALN algorithm is a simple and effective solution for improving the inference time of pre-trained transformer models, making it a valuable tool for natural language processing and computer vision tasks.