Degradation In Classification Accuracy Research Articles

Towards practical superconducting accelerators for machine learning using U-SFQ

Most popular superconducting circuits operate on information carried by ps-wide, \(\boldsymbol{\mu}\) V-tall, single flux quantum (SFQ) pulses. These circuits can operate at frequencies of hundreds of GHz with orders of magnitude lower switching energy than complementary-metal-oxide-semiconductors (CMOS). However, under the stringent area constraints of modern superconductor technologies, fully-fledged, CMOS-inspired superconducting architectures cannot be fabricated at large scales. Unary SFQ (U-SFQ) is an alternative computing paradigm that can address these area constraints. In U-SFQ, information is mapped to a combination of streams of SFQ pulses and in the temporal domain. In this work, we extend U-SFQ to introduce novel building blocks such as a multiplier and an accumulator. These blocks reduce area and power consumption by 2 \(\times\) and 4 \(\times\) compared with previously-proposed U-SFQ building blocks, and yield at least 97% area savings compared with binary approaches. Using these multiplier and adder, we propose a U-SFQ Convolutional Neural Network (CNN) hardware accelerator capable of comparable peak performance with state-of-the-art superconducting binary approach (B-SFQ) in 32 \(\times\) less area. CNNs can operate with 5-8 bits of resolution with no significant degradation in classification accuracy. For 5 bits of resolution, our proposed accelerator yields 5 \(\times\) -63 \(\times\) better performance than CMOS and 15 \(\times\) -173 \(\times\) better area efficiency than B-SFQ.

CPRNC: Channels pruning via reverse neuron crowding for model compression

Channel pruning is an efficient technique for model compression, removing redundant parts of a convolutional neural network with minor degradation in classification accuracy. Previous criteria of channel pruning ignore neurons’ intrinsic relationship and the high correlation with input samples. Inspired by the visual crowding phenomenon in neuroscience, this paper presents a novel channel pruning method via reverse neuron crowding, dubbed CPRNC, to address this issue. First, CPRNC involves a neuron crowding degree measure (NCDM) module, which builds the relationship model among all artificial neurons by observing their crowding behaviors. Subsequently, each channel’s importance is evaluated by the crowding degree of corresponding channels. Considering that the channel importance is affected by the characteristic of input samples, CPRNC designs a neuron crowding degree recalibrate (NCDR) module. NCDR emphasizes discriminative samples to recalibrate the channel priority list generated by NCDM, further enhancing the precision of the pruning criterion. Experimental results show that CPRNC achieves performance that competes with state-of-the-art pruning methods, including dynamic channel pruning and learning-based pruning. For example, we prune ResNet-50 with 56.7% FLOPs on the large-scale dataset ImageNet1K with only a 0.19% decrease in accuracy. At low pruning rates, CPRNC achieves lossless compression, e.g., the pruned ResNet-56 on CIFAR-10 increases accuracy by 0.13% over the baseline model at 56.3% FLOPs reduction.

Radiation-Tolerant Deep Learning Processor Unit (DPU)-Based Platform Using Xilinx 20-nm Kintex UltraScale FPGA

This paper presents a platform and design approach for enabling radiation-tolerant deep learning acceleration on SRAM-based 20nm Kintex UltraScale™ FPGAs, for terrestrial and high-radiation environments. The presented platform is suitable for deep neural network (DNN) implementations with an emphasis on image classification and includes solutions to mitigate both radiation-induced Single Event Functional Interrupts (SEFIs) and network datapath corruptions. The radiation-tolerant deep learning platform combines Xilinx’s Deep Learning Processing Unit (DPU) IP, Triple Modular Redundancy (TMR) MicroBlaze soft processor IP and Soft Error Mitigation (SEM)-IP to mitigate SEFIs. Furthermore, a technique known as Fault Aware Training (FAT) was applied to effectively mitigate single event effects in the datapath. Test results from a high-energy proton beam (> 60 MeV) experiment using the ResNet-18 Convolutional Neural Network (CNN) for image classification are presented. The Single Event Upset (SEU) rate, system-level SEFI rate and neural network classification/datapath performance are compared between the radiation-tolerant platform and a standard, non-mitigated approach. Results show that datapath classification errors dominate the system response (90%) vs. SEFIs (10%). When compared to standard non-mitigated training techniques, the radiation-tolerant platform using fault aware training methods shows dramatic improvements in overall system response: the overall single event cross-section was reduced by half and 40% reduction in misclassification errors were observed. Also, datapath events with classification accuracy degradation larger than 5% were completely mitigated. The SEFI rate was reduced by 100X with implemented solutions and can be further reduced by optimizing the physical separation between TMR modules.

A Credit Card Fraud Model Prediction Method Based on Penalty Factor Optimization AWTadaboost

With the popularity of online payment, how to perform credit card fraud detection more accurately has also become a hot issue. And with the emergence of the adaptive boosting algorithm (Adaboost), credit card fraud detection has started to use this method in large numbers, but the traditional Adaboost is prone to overfitting in the presence of noisy samples. Therefore, in order to alleviate this phenomenon, this paper proposes a new idea: using the number of consecutive sample misclassifications to determine the noisy samples, while constructing a penalty factor to reconstruct the sample weight assignment. Firstly, the theoretical analysis shows that the traditional Adaboost method is overfitting in a noisy training set, which leads to the degradation of classification accuracy. To this end, the penalty factor constructed by the number of consecutive misclassifications of samples is used to reconstruct the sample weight assignment to prevent the classifier from over-focusing on noisy samples, and its reasonableness is demonstrated. Then, by comparing the penalty strength of the three different penalty factors proposed in this paper, a more reasonable penalty factor is selected. Meanwhile, in order to make the constructed model more in line with the actual requirements on training time consumption, the Adaboost algorithm with adaptive weight trimming (AWTAdaboost) is used in this paper, so the penalty factor-based AWTAdaboost (PF_AWTAdaboost) is finally obtained. Finally, PF_AWTAdaboost is experimentally validated against other traditional machine learning algorithms on credit card fraud datasets and other datasets. The results show that the PF_AWTAdaboost method has better performance, including detection accuracy, model recall and robustness, than other methods on the credit card fraud dataset. And the PF_AWTAdaboost method also shows excellent generalization performance on other datasets. From the experimental results, it is shown that the PF_AWTAdaboost algorithm has better classification performance.

A Transformer-based Multi-task Learning Framework for Myoelectric Pattern Recognition Supporting Muscle Force Estimation.

Simultaneous implementation of myoelectric pattern recognition and muscle force estimation is highly demanded in building natural gestural interfaces but a challenging task due to the gesture classification accuracy degradation under varying muscle strengths. To address this problem, a novel method using transformer-based multi-task learning (MTL-Transformer) for the prediction of both myoelectric patterns and corresponding muscle strengths was proposed to describe the inherent characteristics of an individual gesture pattern under different force conditions, thereby improving the accuracy of myoelectric pattern recognition. In addition, the transformer model enabled the characterization of long-term temporal correlations to ensure precise and smooth estimation of the muscle force. The performance of the proposed MTL-Transformer framework was evaluated via experiments of classifying eleven hand gestures and estimating the corresponding muscle force simultaneously, using high-density surface electromyogram (HD-sEMG) recordings from forearm flexor muscles of eleven intact-limbed subjects. The MTL-Transformer framework yielded high classification accuracy (98.70±1.21%) and low root mean square deviation (12.59±2.76%), and outperformed other two common temporally modelling methods significantly (p < 0.05) in terms of both improved gesture recognition accuracies and reduced muscle force estimation errors. The MTL-Transformer framework is demonstrated as an effective solution for simultaneous implementation of myoelectric pattern recognition and muscle force estimation. This study promotes the development of robust and smooth myoelectric control systems, with wide applications in gestural interfaces, prosthetic and orthotic control.

String Current Compensation Method in VNAND Flash for Hardware-Based BNNs

This work proposes a method to prevent unwanted string current degradation in multistacks vertical NAND (VNAND) flash memory for hardware-based binary neural networks (BNNs). Simulations investigate the effect of string resistance on the accuracy of a multilayer BNNs, considering the measured cell string characteristics. When the weights obtained from off-chip training are transferred to the BNN without compensation cells, the classification accuracy of the hardware-based BNN is reduced by ~80.3% in 23-layer network. However, the proposed compensation cell model shows a degradation of classification accuracy of 1.4% in the 23-layer BNN compared to the software-based BNN.

Speeding-up one-versus-all training for extreme classification via mean-separating initialization

In this paper, we show that a simple, data dependent way of setting the initial vector can be used to substantially speed up the training of linear one-versus-all classifiers in extreme multi-label classification (XMC). We discuss the problem of choosing the initial weights from the perspective of three goals. We want to start in a region of weight space (a) with low loss value, (b) that is favourable for second-order optimization, and (c) where the conjugate-gradient (CG) calculations can be performed quickly. For margin losses, such an initialization is achieved by selecting the initial vector such that it separates the mean of all positive (relevant for a label) instances from the mean of all negatives – two quantities that can be calculated quickly for the highly imbalanced binary problems occurring in XMC. We demonstrate a training speedup of up to 5times on Amazon-670K dataset with 670,000 labels. This comes in part from the reduced number of iterations that need to be performed due to starting closer to the solution, and in part from an implicit negative-mining effect that allows to ignore easy negatives in the CG step. Because of the convex nature of the optimization problem, the speedup is achieved without any degradation in classification accuracy. The implementation can be found at https://github.com/xmc-aalto/dismecpp.

Fault Injection Attacks in Spiking Neural Networks and Countermeasures

Spiking Neural Networks (SNN) are fast emerging as an alternative option to Deep Neural Networks (DNN). They are computationally more powerful and provide higher energy-efficiency than DNNs. While exciting at first glance, SNNs contain security-sensitive assets (e.g., neuron threshold voltage) and vulnerabilities (e.g., sensitivity of classification accuracy to neuron threshold voltage change) that can be exploited by the adversaries. We explore global fault injection attacks using external power supply and laser-induced local power glitches on SNN designed using common analog neurons to corrupt critical training parameters such as spike amplitude and neuron’s membrane threshold potential. We also analyze the impact of power-based attacks on the SNN for digit classification task and observe a worst-case classification accuracy degradation of −85.65%. We explore the impact of various design parameters of SNN (e.g., learning rate, spike trace decay constant, and number of neurons) and identify design choices for robust implementation of SNN. We recover classification accuracy degradation by 30–47% for a subset of power-based attacks by modifying SNN training parameters such as learning rate, trace decay constant, and neurons per layer. We also propose hardware-level defenses, e.g., a robust current driver design that is immune to power-oriented attacks, improved circuit sizing of neuron components to reduce/recover the adversarial accuracy degradation at the cost of negligible area, and 25% power overhead. We also propose a dummy neuron-based detection of voltage fault injection at ∼1% power and area overhead each.

SiO2 Fin-Based Flash Synaptic Cells in AND Array Architecture for Binary Neural Networks

An oxide fin-based AND flash memory synaptic device is proposed and fabricated using a spacer patterning technology for a hardware-based binary neural network (BNN). A fin-like curved channel structure provides local electric field enhancement, which improves programming efficiency compared to planar-type flash synaptic devices. The fin-based AND flash cell exhibits a high on/off current ratio (>10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">5</sup> ) with sub-pA off current, and a low programming voltage (< 9 V) is used to achieve a sufficient dynamic range of synaptic weights (>10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ) for BNNs. Furthermore, a hardware-based BNN using novel two cell-based synaptic devices arranged in AND array architecture is proposed to implement parallel XNOR operation and bit-counting. Proposed BNN using the synapse model with measured dynamic range and retention property shows only < 0.5 % degradation of classification accuracy compared to the baseline accuracy, which is suitable to perform off-chip event-driven computation using parallel read-out operations.

ResNet Autoencoders for Unsupervised Feature Learning From High-Dimensional Data: Deep Models Resistant to Performance Degradation

Efficient modeling of high-dimensional data requires extracting only relevant dimensions through feature learning. Unsupervised feature learning has gained tremendous attention due to its unbiased approach, no need for prior knowledge or expensive manual processing, and ability to handle exponential data growth. Deep Autoencoder (AE) is a state-of-the-art deep neural network for unsupervised feature learning, which learns embedded-representations using a series of stacked layers. However, as the AE network gets deeper, these learned embedded-representations can deteriorate due to vanishing gradient, leading to performance degradation. This article presents ResNet Autoencoder (RAE) and its convolutional version (C-RAE) for unsupervised feature learning. The advantage of RAE and C-RAE is that it enables the user to add residual connections for increased network capacity without incurring the cost of degradation for unsupervised feature learning compared to standard AEs. While RAE and C-RAE inherit all the advantages of AEs, such as automated non-linear feature extraction and unsupervised learning, they also allow users to design larger networks without adverse effects on feature learning performance. We performed classification on learned embedded-representation to evaluate RAE and C-RAE. RAE and C-RAE were compared against AEs on MNIST, Fashion MNIST, and CIFAR10 datasets. When increasing the number of layers, C-RAE outperformed AE by showing significantly lower performance degradation of classification accuracy (less than 3%) compared to AE (33% to 65%). Further, C-RAE exhibited higher mean accuracy and lower variance of accuracy than standard AE. When comparing RAE and C-RAE with widely used feature learning methods (Convolutional AE, PCA, ICA, LLE, Factor Analysis, and SVD), C-RAE showed the highest accuracy.

MixedNet: Network Design Strategies for Cost-Effective Quantized CNNs

This paper proposes design strategies for a low-cost quantized neural network. To prevent the classification accuracy from being degraded by quantization, a structure-design strategy that utilizes a large number of channels rather than deep layers is proposed. In addition, a squeeze-and-excitation (SE) layer is adopted to enhance the performance of the quantized network. Through a quantitative analysis and simulations of the quantized key convolution layers of ResNet and MobileNets, a low-cost layer-design strategy for use when building a neural network is proposed. With this strategy, a low-cost network referred to as a MixedNet is constructed. A 4-bit quantized MixedNet example achieves an on-chip memory size reduction of 60% and fewer memory access by 53% with negligible classification accuracy degradation in comparison with conventional networks while also showing classification accuracy rates of approximately 73% for Cifar-100 and 93% for Cifar-10.

Reducing SRAM Reading Power With Column Data Segment and Weights Correlation Enhancement for CNN Processing

Convolutional neural network (CNN) has been widely deployed in various processors for intelligent visual signal processing. However, the large amount of activations and weights in CNN causes huge power consumption on SRAM access. Data-adaptive SRAM design is a widely studied method to reduce SRAM reading power based on the utilization of data patterns, while current designs only exploit data patterns in a coarse granularity, and have no advantages when faced with randomly distributed weight data. In this article, we propose a hardware–software co-design scheme to reduce SRAM reading power for CNN processing. First, we propose a reconfigurable data-adaptive SRAM architecture with column data segmentation (CDS-RSRAM) to utilize data patterns. Data in one column is partitioned into several segments, and finer-grained data patterns are exploited within each segment for further reading power reduction. Then, a novel training method—minimum segmented neighbor difference (miniSND)—is proposed for enhancing the correlation of weights. MiniSND improves the similarity of weights without classification accuracy degradation, thus weights could benefit from CDS-RSRAM and be read out with less power consumption. Simulation results demonstrate that the co-design scheme saves up to 66%(8b)/89%(2b) power consumption compared with 8T SRAM.

AirNN: A Featherweight Framework for Dynamic Input-Dependent Approximation of CNNs

In this work, we propose AirNN, a novel framework which enables dynamic approximation of an already-trained convolutional neural network (CNN) in hardware during inference. AirNN enables input-dependent approximation of the CNN to achieve energy saving without much degradation in its classification accuracy at runtime. For each input, AirNN uses only a fraction of the CNN’s weights based on that input (with the rest remaining 0) to conduct the inference. Consequently, energy saving is possible due to fewer number of fetches from off-chip memory as well as fewer multiplications for majority of the inputs. To achieve per-input approximation, we propose a clustering algorithm that groups similar weights in the CNN based on their importance, and design an iterative framework that decides dynamically how many clusters of weights should be fetched from off-chip memory for each individual input. We also propose new hardware structures to implement our framework on top of a recently proposed FPGA-based CNN accelerator. In our experiments with popular CNNs, we, on average, show 49% energy saving with less than 3% degradation in classification accuracy due to doing inference with only a fraction of the weights for the majority of the inputs. We also propose a greedy interleaving scheme, implemented in hardware, in order to improve the performance of the iterative procedure and compensate for its latency overhead.

Clinically Relevant Vulnerabilities of Deep Machine Learning Systems for Skin Cancer Diagnosis

Clinically Relevant Vulnerabilities of Deep Machine Learning Systems for Skin Cancer Diagnosis

Evaluating Author Attribution on Emirati Tweets

Author Attribution (AA) is a critical stylometry problem that tries to deduce the identity of the authors of electronic texts (e-texts) by only examining the texts. AA is essential for enhancing various application domains, such as recommender systems and forensics. Nevertheless, existing techniques in AA have not been assessed with Emirati social media e-texts. The reason is that no suitable dataset exists for evaluating AA techniques in this context. This paper introduces the Khonji-Iraqi Emirati Tweets Author Identification (AID) dataset with 30 authors (KIT-30), and detailed evaluations. Compound grams, a new definition of grams, are introduced, which allows us to achieve higher classification accuracy. Also, when the number of suspect authors increases, the classification accuracy degradation is not as severe as previously reported, when using suitable data representation. Furthermore, in order to work towards addressing the lack of conveniently-available implementations of stylometry methods, we have developed an extensive e-text feature extraction library, namely Fextractor, with a highly intuitive API. The library generalizes all existing n-gram-based feature extraction methods under the at least l-frequent, dir-directed, k-skipped n-grams, and allows grams to be diversely defined, including definitions that are based on high-level grammatical aspects, such as Part of Speech (POS) tags, as well as lower-level ones, such as the distribution of function words and word shapes.

Memory Optimization for Energy-Efficient Differentially Private Deep Learning

With the advent of Internet of Things (IoT) technologies and availability of a large amount of data, deep learning has been applied in a variety of artificial intelligence (AI) applications. However, sharing personal data using IoT edge devices carries inherent risks to individual privacy. Meanwhile, the energy and memory resources needed during the inference process become a constraint to the resource-limited IoT edge devices. This article brings memory hardware optimization to meet the tight power budget in IoT edge devices by considering the privacy, accuracy, and power efficiency tradeoff in differentially efficient deep learning systems. Based on a detailed analysis on these characteristics, an integer linear programs (ILP) model is developed to minimize mean square error (MSE), thereby enabling optimal input data memory design. Our simulation results in 45-nm CMOS technology show that the proposed technique can enable near-threshold energy-efficient memory operation for different privacy requirements, with less than 1% degradation in classification accuracy.

Resource-Efficient Pet Dog Sound Events Classification Using LSTM-FCN Based on Time-Series Data.

The use of IoT (Internet of Things) technology for the management of pet dogs left alone at home is increasing. This includes tasks such as automatic feeding, operation of play equipment, and location detection. Classification of the vocalizations of pet dogs using information from a sound sensor is an important method to analyze the behavior or emotions of dogs that are left alone. These sounds should be acquired by attaching the IoT sound sensor to the dog, and then classifying the sound events (e.g., barking, growling, howling, and whining). However, sound sensors tend to transmit large amounts of data and consume considerable amounts of power, which presents issues in the case of resource-constrained IoT sensor devices. In this paper, we propose a way to classify pet dog sound events and improve resource efficiency without significant degradation of accuracy. To achieve this, we only acquire the intensity data of sounds by using a relatively resource-efficient noise sensor. This presents issues as well, since it is difficult to achieve sufficient classification accuracy using only intensity data due to the loss of information from the sound events. To address this problem and avoid significant degradation of classification accuracy, we apply long short-term memory-fully convolutional network (LSTM-FCN), which is a deep learning method, to analyze time-series data, and exploit bicubic interpolation. Based on experimental results, the proposed method based on noise sensors (i.e., Shapelet and LSTM-FCN for time-series) was found to improve energy efficiency by 10 times without significant degradation of accuracy compared to typical methods based on sound sensors (i.e., mel-frequency cepstrum coefficient (MFCC), spectrogram, and mel-spectrum for feature extraction, and support vector machine (SVM) and k-nearest neighbor (K-NN) for classification).

Supervised Classification High-Resolution Remote-Sensing Image Based on Interval Type-2 Fuzzy Membership Function

Because of the degradation of classification accuracy that is caused by the uncertainty of pixel class and classification decisions of high-resolution remote-sensing images, we proposed a supervised classification method that is based on an interval type-2 fuzzy membership function for high-resolution remote-sensing images. We analyze the data features of a high-resolution remote-sensing image and construct a type-1 membership function model in a homogenous region by supervised sampling in order to characterize the uncertainty of the pixel class. On the basis of the fuzzy membership function model in the homogeneous region and in accordance with the 3&sigma; criterion of normal distribution, we proposed a method for modeling three types of interval type-2 membership functions and analyze the different types of functions to improve the uncertainty of pixel class expressed by the type-1 fuzzy membership function and to enhance the accuracy of classification decision. According to the principle that importance will increase with a decrease in the distance between the original, upper, and lower fuzzy membership of the training data and the corresponding frequency value in the histogram, we use the weighted average sum of three types of fuzzy membership as the new fuzzy membership of the pixel to be classified and then integrated into the neighborhood pixel relations, constructing a classification decision model. We use the proposed method to classify real high-resolution remote-sensing images and synthetic images. Additionally, we qualitatively and quantitatively evaluate the test results. The results show that a higher classification accuracy can be achieved with the proposed algorithm.

Spiking Neural Network Using Synaptic Transistors and Neuron Circuits for Pattern Recognition With Noisy Images

We demonstrate the hardware implementation of spiking neural network (SNN) with synaptic transistors and neuron circuits. The method of conversion from software fully-connected network (FCN) to hardware SNN with little degradation is discussed. The degradation of classification accuracy is analyzed in terms of device variation and noisy images. In addition, the accuracy degradation is significantly improved by stacking denoising autoencoder (DAE) layer. FCN-SNN conversion with very little performance drop is demonstrated using weight normalization, and SNN with DAE layer shows a great tolerance to input image noise.

Ristretto: A Framework for Empirical Study of Resource-Efficient Inference in Convolutional Neural Networks.

Convolutional neural networks (CNNs) have led to remarkable progress in a number of key pattern recognition tasks, such as visual scene understanding and speech recognition, that potentially enable numerous applications. Consequently, there is a significant need to deploy trained CNNs to resource-constrained embedded systems. Inference using pretrained modern deep CNNs, however, requires significant system resources, including computation, energy, and memory space. To enable efficient implementation of trained CNNs, a viable approach is to approximate the network with an implementation-friendly model with only negligible degradation in classification accuracy. We present Ristretto, a CNN approximation framework that enables empirical investigation of the tradeoff between various number representation and word width choices and the classification accuracy of the model. Specifically, Ristretto analyzes a given CNN with respect to numerical range required to represent weights, activations, and intermediate results of convolutional and fully connected layers, and subsequently, it simulates the impact of reduced word width or lower precision arithmetic operators on the model accuracy. Moreover, Ristretto can fine-tune a quantized network to further improve its classification accuracy under a given number representation and word width configuration. Given a maximum classification accuracy degradation tolerance of 1%, we use Ristretto to demonstrate that three ImageNet networks can be condensed to use 8-bit dynamic fixed point for network weights and activations. Ristretto is available as a popular open-source software project and has already been viewed over 1,000 times on Github as of the submission of this brief.