Drop In Accuracy Research Articles

Radial Basis Function Network with Differential Privacy

Differential privacy (DP) remains a potent solution to what is arguably the defining issue in machine learning: balancing user privacy with an ever-increasing need for data. Practitioners must respect privacy, especially in sensitive healthcare domains. DP strives towards this aim by adding noise to training data to occlude its origin and nature, and is ideal for multiple Neural Network (NN) types. This includes deep varieties that utilise multiple hidden layers, and shallow ones with single hidden layers such as the Radial Base Function Network (RBFN). The work herein explores DP within this context by devising a model that leverages Gaussian RBF parameters to add privacy during training. Our model’s efficacy is examined against two real and three synthetic datasets, with results showing reasonable trade-offs between accuracy and privacy. With high intra-class variation, we retained 100% accuracy for two synthetic datasets and a drop of only 1.72% for another. If privacy is prioritised with low intra-class variation, we achieved accuracy drops of 8%–23% with an inherited epsilon that never exceeds one, indicating a good privacy guarantee. We also show that timely training is achievable on a high-dimensional dataset consisting of 2M records and 170 features.

An unsupervised domain adaptation approach for change detection and its application to deforestation mapping in tropical biomes

Changes in environmental conditions, geographical variability and different sensor properties typically make it almost impossible to employ previously trained classifiers for new data without a significant drop in classification accuracy. Domain adaptation (DA) techniques been proven useful to alleviate that problem. In particular, appearance adaptation techniques may be used to adapt images from a specific dataset in such a way that the generated images have a style that is similar to the images from another dataset. Such techniques are, however, prone to creating artifacts that hinder proper classification of the adapted images. In this work we propose an unsupervised DA approach for change detection tasks, which is based on a particular appearance adaptation method: the Cycle-Consistent Generative Adversarial Network (CycleGAN). Specifically, we extend that method by introducing additional constraints in the training phase of the model components, which make it preserve the semantic structure and class transitions in the adapted images. We evaluate the proposed approach on a deforestation detection application, considering different sites in the Amazon rain-forest and in the Brazilian Cerrado (savanna) using Landsat-8 images. In the experiments, each site corresponds to a domain, and the accuracy of a classifier trained with images and references from one (source) domain is measured in the classification of another (target) domain. The results show that the proposed approach is successful in producing artifact-free adapted images, which can be satisfactory classified by the pre-trained source classifiers. On average, the accuracies achieved in the classification of the adapted images outperformed the baselines (when no adaptation was made) by 7.1% in terms of mean average precision, and 9.1% in terms of F1-Score. To the best of our knowledge, the proposed method is the first unsupervised domain adaptation approach devised for change detection.

REHASH

Battery-free sensing devices harvest energy from their surrounding environment to perform sensing, computation, and communication. This enables previously impossible applications in the Internet-of-Things. A core challenge for these devices is maintaining usefulness despite erratic, random or irregular energy availability; which causes inconsistent execution, loss of service and power failures. Adapting execution (degrading or upgrading) seems promising as a way to stave off power failures, meet deadlines, or increase throughput. However, because of constrained resources and limited local information, it is a challenge to decide when would be the best time to adapt, and how exactly to adapt execution. In this paper, we systematically explore the fundamental mechanisms of energy-aware adaptation, and propose heuristic adaptation as a method for modulating the performance of tasks to enable higher sensor coverage, completion rates, or throughput, depending on the application. We build a task based adaptive runtime system for intermittently powered sensors embodying this concept. We complement this runtime with a user facing simulator that enables programmers to conceptualize the tradeoffs they make when choosing what tasks to adapt, and how, relative to real world energy harvesting environment traces. While we target battery-free, intermittently powered sensors, we see general application to all energy harvesting devices. We explore heuristic adaptation with varied energy harvesting modalities and diverse applications: machine learning, activity recognition, and greenhouse monitoring, and find that the adaptive version of our ML app performs up to 46% more classifications with only a 5% drop in accuracy; the activity recognition app captures 76% more classifications with only nominal down-sampling; and find that heuristic adaptation leads to higher throughput versus non-adaptive in all cases.

Detection of COVID-19 from Chest X-ray Images Using Deep Convolutional Neural Networks.

The COVID-19 global pandemic has wreaked havoc on every aspect of our lives. More specifically, healthcare systems were greatly stretched to their limits and beyond. Advances in artificial intelligence have enabled the implementation of sophisticated applications that can meet clinical accuracy requirements. In this study, customized and pre-trained deep learning models based on convolutional neural networks were used to detect pneumonia caused by COVID-19 respiratory complications. Chest X-ray images from 368 confirmed COVID-19 patients were collected locally. In addition, data from three publicly available datasets were used. The performance was evaluated in four ways. First, the public dataset was used for training and testing. Second, data from the local and public sources were combined and used to train and test the models. Third, the public dataset was used to train the model and the local data were used for testing only. This approach adds greater credibility to the detection models and tests their ability to generalize to new data without overfitting the model to specific samples. Fourth, the combined data were used for training and the local dataset was used for testing. The results show a high detection accuracy of 98.7% with the combined dataset, and most models handled new data with an insignificant drop in accuracy.

Enhancing long tail item recommendation in collaborative filtering: An econophysics-inspired approach

• A novel viewpoint of econophysics principles is formulated to address long tail effect. • Rating injection strategy inspired from econophysics principles is proposed. • Performance of the proposed method is evaluated on long tail and non-long tail items. • Proposed approach immensely improved the number of long tail items in recommendations. • The proposed principles can be applied as an add-on to state-of-the-art recommendation technique. Recommender systems have been immensely successful in overcoming information overload problem through personalized suggestions to consumers. Traditional recommendation algorithms tend to recommend more popular items. A significant number of items in an enterprise are non-popular (long tail items) due to lack of visibility in recommendations. These long tail items are left unsold and result in a significant loss to the business. The consumers on the other end are deprived of receiving relevant item recommendations. In this paper, we propose two approaches inspired from econophysics to recommend long tail items. The proposed approaches selectively inject ratings to the long tail items to diminish the bias towards the popular items by utilizing the existing rating information. Subsequently, the injected rating datasets are used to provide recommendations. The results on real-world datasets show that the proposed approaches outperform the existing techniques in mitigating long tail effect with little or no drop in accuracy.

American Sign Language Alphabet Recognition by Extracting Feature from Hand Pose Estimation.

Sign language is designed to assist the deaf and hard of hearing community to convey messages and connect with society. Sign language recognition has been an important domain of research for a long time. Previously, sensor-based approaches have obtained higher accuracy than vision-based approaches. Due to the cost-effectiveness of vision-based approaches, researchers have been conducted here also despite the accuracy drop. The purpose of this research is to recognize American sign characters using hand images obtained from a web camera. In this work, the media-pipe hands algorithm was used for estimating hand joints from RGB images of hands obtained from a web camera and two types of features were generated from the estimated coordinates of the joints obtained for classification: one is the distances between the joint points and the other one is the angles between vectors and 3D axes. The classifiers utilized to classify the characters were support vector machine (SVM) and light gradient boosting machine (GBM). Three character datasets were used for recognition: the ASL Alphabet dataset, the Massey dataset, and the finger spelling A dataset. The results obtained were 99.39% for the Massey dataset, 87.60% for the ASL Alphabet dataset, and 98.45% for Finger Spelling A dataset. The proposed design for automatic American sign language recognition is cost-effective, computationally inexpensive, does not require any special sensors or devices, and has outperformed previous studies.

Mental Workload Estimation Based on Physiological Features for Pilot-UAV Teaming Applications.

Manned-Unmanned Teaming (MUM-T) can be defined as the teaming of aerial robots (artificial agents) along with a human pilot (natural agent), in which the human agent is not an authoritative controller but rather a cooperative team player. To our knowledge, no study has yet evaluated the impact of MUM-T scenarios on operators' mental workload (MW) using a neuroergonomic approach (i.e., using physiological measures), nor provided a MW estimation through classification applied on those measures. Moreover, the impact of the non-stationarity of the physiological signal is seldom taken into account in classification pipelines, particularly regarding the validation design. Therefore this study was designed with two goals: (i) to characterize and estimate MW in a MUM-T setting based on physiological signals; (ii) to assess the impact of the validation procedure on classification accuracy. In this context, a search and rescue (S&R) scenario was developed in which 14 participants played the role of a pilot cooperating with three UAVs (Unmanned Aerial Vehicles). Missions were designed to induce high and low MW levels, which were evaluated using self-reported, behavioral and physiological measures (i.e., cerebral, cardiac, and oculomotor features). Supervised classification pipelines based on various combinations of these physiological features were benchmarked, and two validation procedures were compared (i.e., a traditional one that does not take time into account vs. an ecological one that does). The main results are: (i) a significant impact of MW on all measures, (ii) a higher intra-subject classification accuracy (75%) reached using ECG features alone or in combination with EEG and ET ones with the Adaboost, Linear Discriminant Analysis or the Support Vector Machine classifiers. However this was only true with the traditional validation. There was a significant drop in classification accuracy using the ecological one. Interestingly, inter-subject classification with ecological validation (59.8%) surpassed both intra-subject with ecological and inter-subject with traditional validation. These results highlight the need for further developments to perform MW monitoring in such operational contexts.

Oil palm fresh fruit bunch ripeness classification on mobile devices using deep learning approaches

The implementations of deep learning combined with other methods such as transfer learning and data augmentation in oil palm fresh fruit bunch (FFB) ripeness classification have been researched throughout the years. However, most of the methods require devices with high computational resources which could not be implemented in mobile applications. To overcome this problem, this research would focus on creating a mobile application to classify the ripeness levels of oil palm FFB using lightweight Convolutional Neural Network (CNN). In this research, we implemented ImageNet transfer learning on 4 lightweight CNN models with a novel data augmentation method named “9-angle crop”, which would be further optimized using post-training quantization. Transfer learning with 3 unfrozen convolution blocks and 9 angle crop successfully increased the classification accuracy on MobileNetV1, and when it was compared to other lightweight models, EfficientNetB0 performed best with 0.898 test accuracy on Keras. Float16 quantization also proved to be the most suitable post-training quantization method for this model, halving the size of EfficientNetB0 with the least increase in image classification time and an accuracy drop of only 0.005 after the model was converted to TensorFlow Lite interpreter. In conclusion, the best model created in this research is EfficientNetB0, trained with the combination of transfer learning, 9 angle crop, and float16 quantization, which enabled it to achieve an overall test accuracy of 0.893 on TensorFlow Lite with 96 ms classification time per image, far surpassing the other 3 compared models with the second-best model being MobileNetV1 with 0.811 accuracy. The model itself was able to achieve similar results when it was implemented on an Android application to classify the ripeness levels of oil palm FFB images obtained through live camera input.

A Slight Increase in Reverberation Time in the Classroom Affects Performance and Behavioral Listening Effort.

The purpose of this study was to investigate the effect of a small change in reverberation time (from 0.57 to 0.69 s) in a classroom on children's performance and listening effort. Aiming for ecological listening conditions, the change in reverberation time was combined with the presence or absence of classroom noise. In three academic tasks, the study examined whether the effect of reverberation was modulated by the presence of noise and depended on the children's age. A total of 302 children (aged 11-13 years, grades 6-8) with normal hearing participated in the study. Three typical tasks of daily classroom activities (speech perception, sentence comprehension, and mental calculation) were administered to groups of children in two listening conditions (quiet and classroom noise). The experiment was conducted inside real classrooms, where reverberation time was controlled. The outcomes considered were task accuracy and response times (RTs), the latter taken as a behavioral proxy for listening effort. Participants were also assessed on reading comprehension and math fluency. To investigate the impact of noise and/or reverberation, these two scores were entered in the statistical model to control for individual child's general academic abilities. While the longer reverberation time did not significantly affect accuracy or RTs under the quiet condition, it had several effects when in combination with classroom noise, depending on the task measured. A significant drop in accuracy with a longer reverberation time emerged for the speech perception task, but only for the grade 6 children. The effect on accuracy of a longer reverberation time was nonsignificant for sentence comprehension (always at ceiling), and depended on the children's age in the mental calculation task. RTs were longer for moderate than for short reverberation times in the speech perception and sentence comprehension tasks, while there was no significant effect of the different reverberation times on RTs in the mental calculation task. The results indicate small, but statistically significant, effects of a small change in reverberation time on listening effort as well as accuracy for children aged 11 to 13 performing typical tasks of daily classroom activities. Thus, the results extend previous findings in adults to children as well. The findings also contribute to a better understanding of the practical implications and importance of optimal ranges of reverberation time in classrooms. A comparison with previous studies underscored the importance of early reflections as well as reverberation times in classrooms.

Designing an encoder for StyleGAN image manipulation

Recently, there has been a surge of diverse methods for performing image editing by employing pre-trained unconditional generators. Applying these methods on real images, however, remains a challenge, as it necessarily requires the inversion of the images into their latent space. To successfully invert a real image, one needs to find a latent code that reconstructs the input image accurately, and more importantly, allows for its meaningful manipulation. In this paper, we carefully study the latent space of StyleGAN, the state-of-the-art unconditional generator. We identify and analyze the existence of a distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space. We then suggest two principles for designing encoders in a manner that allows one to control the proximity of the inversions to regions that StyleGAN was originally trained on. We present an encoder based on our two principles that is specifically designed for facilitating editing on real images by balancing these tradeoffs. By evaluating its performance qualitatively and quantitatively on numerous challenging domains, including cars and horses, we show that our inversion method, followed by common editing techniques, achieves superior real-image editing quality, with only a small reconstruction accuracy drop.

Improving motor imagery classification during induced motor perturbations

Objective. Motor imagery is the mental simulation of movements. It is a common paradigm to design brain-computer interfaces (BCIs) that elicits the modulation of brain oscillatory activity similar to real, passive and induced movements. In this study, we used peripheral stimulation to provoke movements of one limb during the performance of motor imagery tasks. Unlike other works, in which induced movements are used to support the BCI operation, our goal was to test and improve the robustness of motor imagery based BCI systems to perturbations caused by artificially generated movements. Approach. We performed a BCI session with ten participants who carried out motor imagery of three limbs. In some of the trials, one of the arms was moved by neuromuscular stimulation. We analysed 2-class motor imagery classifications with and without movement perturbations. We investigated the performance decrease produced by these disturbances and designed different computational strategies to attenuate the observed classification accuracy drop. Main results. When the movement was induced in a limb not coincident with the motor imagery classes, extracting oscillatory sources of the movement imagination tasks resulted in BCI performance being similar to the control (undisturbed) condition; when the movement was induced in a limb also involved in the motor imagery tasks, the performance drop was significantly alleviated by spatially filtering out the neural noise caused by the stimulation. We also show that the loss of BCI accuracy was accompanied by weaker power of the sensorimotor rhythm. Importantly, this residual power could be used to predict whether a BCI user will perform with sufficient accuracy under the movement disturbances. Significance. We provide methods to ameliorate and even eliminate motor related afferent disturbances during the performance of motor imagery tasks. This can help improving the reliability of current motor imagery based BCI systems.

Designing an encoder for StyleGAN image manipulation

Recently, there has been a surge of diverse methods for performing image editing by employing pre-trained unconditional generators. Applying these methods on real images, however, remains a challenge, as it necessarily requires the inversion of the images into their latent space. To successfully invert a real image, one needs to find a latent code that reconstructs the input image accurately, and more importantly, allows for its meaningful manipulation. In this paper, we carefully study the latent space of StyleGAN, the state-of-the-art unconditional generator. We identify and analyze the existence of a distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space. We then suggest two principles for designing encoders in a manner that allows one to control the proximity of the inversions to regions that StyleGAN was originally trained on. We present an encoder based on our two principles that is specifically designed for facilitating editing on real images by balancing these tradeoffs. By evaluating its performance qualitatively and quantitatively on numerous challenging domains, including cars and horses, we show that our inversion method, followed by common editing techniques, achieves superior real-image editing quality, with only a small reconstruction accuracy drop.

Individualized prediction of three- and six-year outcomes of psychosis in a longitudinal multicenter study: a machine learning approach

Schizophrenia and related disorders have heterogeneous outcomes. Individualized prediction of long-term outcomes may be helpful in improving treatment decisions. Utilizing extensive baseline data of 523 patients with a psychotic disorder and variable illness duration, we predicted symptomatic and global outcomes at 3-year and 6-year follow-ups. We classified outcomes as (1) symptomatic: in remission or not in remission, and (2) global outcome, using the Global Assessment of Functioning (GAF) scale, divided into good (GAF ≥ 65) and poor (GAF < 65). Aiming for a robust and interpretable prediction model, we employed a linear support vector machine and recursive feature elimination within a nested cross-validation design to obtain a lean set of predictors. Generalization to out-of-study samples was estimated using leave-one-site-out cross-validation. Prediction accuracies were above chance and ranged from 62.2% to 64.7% (symptomatic outcome), and 63.5–67.6% (global outcome). Leave-one-site-out cross-validation demonstrated the robustness of our models, with a minor drop in predictive accuracies of 2.3% on average. Important predictors included GAF scores, psychotic symptoms, quality of life, antipsychotics use, psychosocial needs, and depressive symptoms. These robust, albeit modestly accurate, long-term prognostic predictions based on lean predictor sets indicate the potential of machine learning models complementing clinical judgment and decision-making. Future model development may benefit from studies scoping patient’s and clinicians' needs in prognostication.

Reducing explicit word vectors dimensions using BPSO-based labeling algorithm and voting method

Interpretability of word vector components is very important for obtaining conceptual relations. Word vectors derived from counting models are interpretable but suffer from the high-dimensionality problem. Our goal in this study is to obtain interpretable low-dimensional word vectors in such a way that the least accuracy loss occurs. To achieve this goal, we propose an approach to reduce the dimensions of word vectors using a labeling method based on the BPSO algorithm and a voting method for selecting final context words. In this approach, we define several different base models to solve the labeling problem using different data and different objective functions. Then, we train each base model and select 3 of the best solutions for each model. We create the target word vectors of the dictionary based on the context words labeled 1. Next, we use the three best solutions of each base model to build the ensemble. After creating the ensemble, we use the voting method to assign the final label to the primary context words and select N final context words. In this study, we use the corpus ukWaC to construct word vectors. We evaluate the resulting word vectors on the MEN, RG-65, and SimLex-999 test sets. The evaluation results show that by reducing the word vectors dimensions from 5000 to 1507, the Spearman correlation coefficient of the proposed approach has been reduced to a lesser extent compared to each base model. Therefore, the accuracy drop of the proposed approach is justified after reducing the dimensions from 5000 to 1507. It is not a large penalty because the resulting word vectors are low-dimensional and interpretable.

Intraoperative (Contrast-Enhanced) Ultrasound Has the Highest Diagnostic Accuracy of Any Imaging Modality in Resection of Colorectal Liver Metastases

AimDefining sensitivity, specificity, diagnostic accuracy for detection of colorectal liver metastases in imaging compared to intraoperative assessment. Defining a cutoff, where accuracy of detection is impaired. MethodsProspective single-institution clinical trial (clinicaltrials.gov: NCT01522209). Patients underwent CEUS, MDCT, and 3 Tesla EOB-MRI within 2 weeks preoperatively. Intraoperative palpation, IOUS, and CEIOUS were performed. A patient and lesion-based database was analyzed for accuracy of detection of CEUS, CT, MRI, and Palp/IOUS/CEIOUS combined read. Histology was standard of reference. ResultsForty-seven high tumor load (mean 5, 4 lesions) patients were analyzed. Histopathology confirmed 264 lesions (245 malignant: 19 benign). Accuracy for detection of all lesions: CEUS 63%, CT 71%, MRI 92%, and PALP/IOUS/CEIOUS 98%. ROC analysis for lesion size showed severe impairment of accuracy in lesion detection smaller than 5mm. Intraoperative imaging was not impaired by lesion size. Patient-based analysis revealed a change of resection plan after IOUS/CEIOUS in 35% of patients. ConclusionAt 5-mm lesion size, preoperative imaging shows a drop in accuracy of detection. In patients with multiple lesions, addition of MRI to MDCT seems useful. Accuracy of intraoperative ultrasound is not impacted by lesion size and should be mandatory. CEIOUS can improve intraoperative decision-making. Trial RegistrationStudy registered with clinicaltrials.gov: NCT01522209

Dynamically Optimizing Network Structure Based on Synaptic Pruning in the Brain.

Most neural networks need to predefine the network architecture empirically, which may cause over-fitting or under-fitting. Besides, a large number of parameters in a fully connected network leads to the prohibitively expensive computational cost and storage overhead, which makes the model hard to be deployed on mobile devices. Dynamically optimizing the network architecture by pruning unused synapses is a promising technique for solving this problem. Most existing pruning methods focus on reducing the redundancy of deep convolutional neural networks by pruning unimportant filters or weights, at the cost of accuracy drop. In this paper, we propose an effective brain-inspired synaptic pruning method to dynamically modulate the network architecture and simultaneously improve network performance. The proposed model is biologically inspired as it dynamically eliminates redundant connections based on the synaptic pruning rules used during the brain's development. Connections are pruned if they are not activated or less activated multiple times consecutively. Extensive experiments demonstrate the effectiveness of our method on classification tasks of different complexity with the MNIST, Fashion MNIST, and CIFAR-10 datasets. Experimental results reveal that even for a compact network, the proposed method can also remove up to 59–90% of the connections, with relative improvement in learning speed and accuracy.

System-Technology Codesign of 3-D NAND Flash-Based Compute-in-Memory Inference Engine

Due to its ultrahigh density and commercially matured fabrication technology, 3-D NAND flash memory has been proposed as an attractive candidate of inference engine for deep neural network (DNN) workloads. However, the peripheral circuits require to be modified with conventional 3-D NAND flash to enable compute-in-memory (CIM), and the chip architectures need to be redesigned for an optimized dataflow. In this work, we present a design of 3-D NAND-CIM accelerator based on the macro parameters from an industry-grade prototype chip. The DNN inference performance is evaluated using the DNN+NeuroSim framework. To exploit the ultrahigh density of 3-D NAND flash, both inputs and weights mapping strategies are introduced to improve the throughput. The benchmarking on the VGG network was performed across the technological candidates for CIM, including SRAM, resistive random access memory (RRAM), and 3-D NAND. Compared to the similar designs with SRAM or RRAM, the result shows that the 3-D NAND-based CIM design can achieve not only 17%–24% chip size but also 1.9–2.7 times more competitive energy efficiency for 8-bit precision inference. Inference accuracy drop induced by 3-D NAND string current drift and variation is also investigated. No accuracy degradation by current variation was observed with the proposed input mapping scheme, while accuracy drops sensitive to the current drift, which implies that some compensation schemes are needed to maintain the inference accuracy.

Deep neural network compression through interpretability-based filter pruning

This paper proposes a method to compress deep neural networks (DNNs) based on interpretability. For a trained DNN model, the activation maximization technique is first used to visualize every filter of the DNN model. Then, a single-layer filter pruning approach is introduced from what is learned by visualization. The entire DNN model is compressed layer by layer by using the single-layer filter pruning method in which the compression of the current layer is based on the compression of the preceding layers. Importantly, in addition to effective compression, the proposed method renders a better interpretation of the deep learning process. With a 60% compression rate of the VGG-16, our method achieves 0.8429 Top-1 accuracy under CIFAR-10, with a slight accuracy drop of only 0.0322, and the storage space of the model can be compressed to 9.42 Mb. For a modern DNN model such as ResNet50, our visualization-based filter pruning method is significantly better than other pruning strategies in different convolutional layers under different compression rates and the larger ImageNet dataset. After pruning, the computation cost and storage requirement of the DNN can be significantly reduced, which means that complex DNN models can be easily implemented in small mobile devices, thus enabling the efficient use of DNNs in the Internet of Things technologies.

One-class anomaly detection via novelty normalization

Anomaly detection is an important task in many real-world applications, such as within cybersecurity and surveillance. As with most data these days, the size and dimensionality of the data within these fields are constantly growing, which makes it essential to develop an approach that can both accurately and efficiently identify anomalies within these datasets. In this paper, we address the problem of one-class anomaly detection, where after training on a singular class, we try to determine whether or not inputs belong to that said class. Most of the currently existing methods have limitations in which the criterion of the novel class relies solely on the reconstruction error term. We attempt to break away from this restriction by proposing the use of an autoencoder network with a normalization term. We pair this with an additive novelty scoring module during the training procedure as a way to determine the difference between a given image and our determined normal class, therefore improving the efficiency of our model. We evaluate our model on MNIST, CIFAR-10, and Fashion-MNIST, three popular datasets for image classification, and compare the results against other various state-of-the-art models to determine the efficacy of our efforts. Our model not only outperforms the existing methods, but it also gives us a narrower range of AUCs for the tested classes, suggesting a stark improvement in both accuracy and precision. Moreover, we discover that introducing this “Novelty Normalization” concept into our model allows us to expand its usage into multiclass scenarios without a steep drop in accuracy.

Zero-Keep Filter Pruning for Energy/Power Efficient Deep Neural Networks

Recent deep learning models succeed in achieving high accuracy and fast inference time, but they require high-performance computing resources because they have a large number of parameters. However, not all systems have high-performance hardware. Sometimes, a deep learning model needs to be run on edge devices such as IoT devices or smartphones. On edge devices, however, limited computing resources are available and the amount of computation must be reduced to launch the deep learning models. Pruning is one of the well-known approaches for deriving light-weight models by eliminating weights, channels or filters. In this work, we propose “zero-keep filter pruning” for energy-efficient deep neural networks. The proposed method maximizes the number of zero elements in filters by replacing small values with zero and pruning the filter that has the lowest number of zeros. In the conventional approach, the filters that have the highest number of zeros are generally pruned. As a result, through this zero-keep filter pruning, we can have the filters that have many zeros in a model. We compared the results of the proposed method with the random filter pruning and proved that our method shows better performance with many fewer non-zero elements with a marginal drop in accuracy. Finally, we discuss a possible multiplier architecture, zero-skip multiplier circuit, which skips the multiplications with zero to accelerate and reduce energy consumption.