Accuracy Of Deep Neural Network Research Articles

The Limits of SEMA on Distinguishing Similar Activation Functions of Embedded Deep Neural Networks

Artificial intelligence (AI) is progressing rapidly, and in this trend, edge AI has been researched intensively. However, much less work has been performed around the security of edge AI. Machine learning models are a mass of intellectual property, and an optimized network is very valuable. Trained machine learning models need to be black boxes as well because they may give away information about the training data to the outside world. As selecting the appropriate activation functions to enable fast training of accurate deep neural networks is an active area of research, it is important to conceal the information of the activation functions used in a neural network architecture as well. There has been research on the use of physical attacks such as the side-channel attack (SCA) in areas other than cryptography. The SCA is highly effective against edge artificial intelligence due to its property of the device computing close to the user. We studied a previously proposed method to retrieve the activation functions of a black box neural network implemented on an edge device by using simple electromagnetic analysis (SEMA) and improved the signal processing procedure for further noisy measurements. The SEMA attack identifies activation functions by directly observing distinctive electromagnetic (EM) traces that correspond to the operations in the activation function. This method requires few executions and inputs and also has little implementation dependency on the activation functions. We distinguished eight similar activation functions with EM measurements and examined the versatility and limits of this attack. In this work, the machine learning architecture is a multilayer perceptron, evaluated on an Arduino Uno.

An Evaporation Duct Height Estimation Algorithm Based on Deep Neural Networks

In order to solve the problem of the evaporation duct height (EDH) estimation, focusing on the propagation loss (PL) features with different EDHs, this paper proposes an evaporation duct height estimation method based on Deep Neural Networks (DNN). The theoretical basis is Universal Approximation Theorem. DNN computes the EDH value with the high-dimensional features and posterior probability of PL. A large amount of PL simulation datasets with EDH=15-25m are used for model training, and datasets with EDH=25-35m are used to verify the ability of the network. Experiments show that the model in this paper has good feature extraction capabilities for simulation datasets. Compared with the naive Bayes algorithm, the accuracy and anti-interference ability of DNN are greatly improved.

BEOL-Compatible Superlattice FEFET Analog Synapse With Improved Linearity and Symmetry of Weight Update

Pseudo-crossbar arrays using ferroelectric field effect transistor (FEFET) mitigates weight movement and allows <i>in situ</i> vector&#x2013;matrix multiplication (VMM), which can significantly accelerate online training of deep neural networks (DNNs). However, the training accuracy of DNNs using conventional FEFETs is low because of the non-idealities, such as nonlinearity, asymmetry, limited bit precision, and limited dynamic range of the weight updates. The limited endurance of these devices degrades the training accuracy further. Here, we show a novel approach for designing the gate-stack of an FEFET analog synapse using a superlattice (SL) of ferroelectric (FE)/dielectric (DE)/FE. The partial polarization states are stabilized by harnessing the depolarization field from the DE spacer, which mitigates the weight update non-idealities. We demonstrate a 7-bit SL-FEFET analog synapse with improved weight update profile, resulting in 94.1&#x0025; online training accuracy for MNIST handwritten digit classification task. The device uses an indium&#x2013;tungsten&#x2013;oxide (IWO) channel and back-end-of line (BEOL)-compatible process flow. The absence of low-<inline-formula> <tex-math notation="LaTeX">${k}$ </tex-math></inline-formula> interlayer (IL) results in high endurance (&#x003E;10¹⁰ cycles), while the BEOL compatibility paves the way to high-density integration of pseudo-crossbar arrays and flexibility for neuromorphic circuit design.

Automated Ensemble for Deep Learning Inference on Edge Computing Platforms

Advances in deep learning (DL) have triggered an explosion of mobile intelligence, posing a soaring demand for computing resources that cannot be satisfied by mobile devices. In this article, we employ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">edge computing</i> to deliver better DL inference services to end users. The key is to leverage deep neural network (DNN) ensemble techniques that provide state-of-the-art performance for many machine learning applications in terms of inference accuracy and robustness. Compared to end devices, the edge computing platform is endowed with more powerful computing resources, making it feasible to implement DNN ensembles for DL inferences. However, due to the constrained computing capacity of edge servers and the possible service response deadline, an edge server can only use a limited number of DNNs to construct DNN ensembles. This poses a unique problem, namely, DNN ensemble selection, for identifying the best-fit DNN ensembles. We propose a novel algorithm called automated DNN ensemble selection (AES) algorithm to solve this problem. Because DNNs exhibit performance variations over different distributions of input data, AES adaptively determines a DNN ensemble according to the features of admitted inference tasks. AES is an online learning algorithm that learns DNNs’ in-use performance over time. An ensemble selection rule is further designed as a subroutine of AES to recruit members into the DNN ensemble based on the accuracy and diversity of DNNs. In particular, we theoretically prove that AES can achieve asymptotic optimality. We carry out experiments on real-world data sets. The results show that using the DNN ensemble technique on edge computing platforms dramatically improves the DL inference quality, and AES outperforms other benchmark schemes.

Accuracy and Resiliency of Analog Compute-in-Memory Inference Engines

Recently, analog compute-in-memory (CIM) architectures based on emerging analog non-volatile memory (NVM) technologies have been explored for deep neural networks (DNNs) to improve scalability, speed, and energy efficiency. Such architectures, however, leverage charge conservation, an operation with infinite resolution, and thus are susceptible to errors. Thus, the inherent stochasticity in any analog NVM used to execute DNNs, will compromise performance. Several reports have demonstrated the use of analog NVM for CIM in a limited scale. It is unclear whether the uncertainties in computations will prohibit large-scale DNNs. To explore this critical issue of scalability, this article first presents a simulation framework to evaluate the feasibility of large-scale DNNs based on CIM architecture and analog NVM. Simulation results show that DNNs trained for high-precision digital computing engines are not resilient against the uncertainty of the analog NVM devices. To avoid such catastrophic failures, this article introduces the analog bi-scale representation for the DNN, and the Hessian-aware Stochastic Gradient Descent training algorithm to enhance the inference accuracy of trained DNNs. As a result of such enhancements, DNNs such as Wide ResNets for CIFAR-100 image recognition problem are demonstrated to have significant performance improvements in accuracy without adding cost to the inference hardware .

Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications

Deep neural networks (DNNs) have achieved unprecedented success in the field of artificial intelligence (AI), including computer vision, natural language processing, and speech recognition. However, their superior performance comes at the considerable cost of computational complexity, which greatly hinders their applications in many resource-constrained devices, such as mobile phones and Internet of Things (IoT) devices. Therefore, methods and techniques that are able to lift the efficiency bottleneck while preserving the high accuracy of DNNs are in great demand to enable numerous edge AI applications. This article provides an overview of efficient deep learning methods, systems, and applications. We start from introducing popular model compression methods, including pruning, factorization, quantization, as well as compact model design. To reduce the large design cost of these manual solutions, we discuss the AutoML framework for each of them, such as neural architecture search (NAS) and automated pruning and quantization. We then cover efficient on-device training to enable user customization based on the local data on mobile devices. Apart from general acceleration techniques, we also showcase several task-specific accelerations for point cloud, video, and natural language processing by exploiting their spatial sparsity and temporal/token redundancy. Finally, to support all these algorithmic advancements, we introduce the efficient deep learning system design from both software and hardware perspectives.

A regularization method to improve adversarial robustness of neural networks for ECG signal classification

With the advancement of machine leaning technologies, Deep Neural Networks (DNNs) have been utilized for automated interpretation of Electrocardiogram (ECG) signals to identify potential abnormalities in a patient's heart within a second. Studies have shown that the accuracy of DNNs for ECG signal classification could reach human-expert cardiologist level if a sufficiently large training dataset is available. However, it is known that, in the field of computer vision, DNNs are not robust to adversarial noises that may cause DNNs to make wrong class-label predictions. In this work, we confirm that DNNs are not robust to adversarial noises in ECG signal classification applications, and we propose a novel regularization method to improve DNN robustness by minimizing the noise-to-signal ratio. Our method is evaluated on two public datasets: the MIT-BIH dataset and the CPSC2018 dataset, and the evaluation results show that our method can significantly enhance DNN robustness against adversarial noises generated by Projected Gradient Descent (PGD) and Smooth Adversarial Perturbation (SAP) adversarial attacks, with a minimal reduction of accuracy on clean data. Our method may serve as the baseline for designing new methods to defend against adversarial attacks for life-critical applications depending on ECG interpretation. The code of this work is publicly available at github.com/SarielMa/Robust_DNN_for_ECG.

Impact of the Radar Image Resolution of Military Objects on the Accuracy of their Classification by a Deep Convolutional Neural Network

Introduction. Deep convolutional neural networks are considered as one of the most promising tools for classifying small-sized objects on radar images. However, no systemic study has been reported so far on the dependence between the classification accuracy achieved by convolutional neural networks and such an important image characteristic as resolution.Aim. Determination of a dependence between of the accuracy of classifying military objects by a deep convolutional neural network and the resolution of their radar images.Materials and methods. An eight-layer convolutional neural network was designed, trained and tested using the Keras library and Tensorflow 2.0 framework. For training and testing, the open part of the standard MSTAR dataset comprising ten classes of military objects radar images was used. The initial weight values of the MobileNetV1 and Xception networks used for a comparative assessment of the achieved classification accuracy were obtained from the training results on the Imagenet.Results. The accuracy of classifying military objects decreases rapidly along with a deterioration in resolution, amounting to 97.91, 90.22, 79.13, 52.2 and 23.68 % at a resolution of 0.3, 0.6, 0.9, 1.5 and 3 m, respectively. It is shown that the use of pretrained MobileNetV1 and Xception networks does not lead to an improvement in the classification accuracy compared to a simple VGG-type network.Conclusion. Effective recognition of military objects at a resolution worse than one meter is practically impossible. The classification accuracy of deep neural networks depends significantly on the difference in the image resolution of the training and test sets. A significant increase in the resistance of the classification accuracy to changes in the resolution can be achieved by training on a set of images with different resolutions.

Deep Learning Neural Modelling as a Precise Method in the Assessment of the Chronological Age of Children and Adolescents Using Tooth and Bone Parameters.

Dental age is one of the most reliable methods for determining a patient’s age. The timing of teething, the period of tooth replacement, or the degree of tooth attrition is an important diagnostic factor in the assessment of an individual’s developmental age. It is used in orthodontics, pediatric dentistry, endocrinology, forensic medicine, and pathomorphology, but also in scenarios regarding international adoptions and illegal immigrants. The methods used to date are time-consuming and not very precise. For this reason, artificial intelligence methods are increasingly used to estimate the age of a patient. The present work is a continuation of the work of Zaborowicz et al. In the presented research, a set of 21 original indicators was used to create deep neural network models. The aim of this study was to verify the ability to generate a more accurate deep neural network model compared to models produced previously. The quality parameters of the produced models were as follows. The MAE error of the produced models, depending on the learning set used, was between 2.34 and 4.61 months, while the RMSE error was between 5.58 and 7.49 months. The correlation coefficient R2 ranged from 0.92 to 0.96.

Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering

Deep neural networks (DNNs) have demonstrated superior performance over classical machine learning to support many features in safety-critical systems. Although DNNs are now widely used in such systems (e.g., self driving cars), there is limited progress regarding automated support for functional safety analysis in DNN-based systems. For example, the identification of root causes of errors, to enable both risk analysis and DNN retraining, remains an open problem. In this paper, we propose SAFE, a black-box approach to automatically characterize the root causes of DNN errors. SAFE relies on a transfer learning model pre-trained on ImageNet to extract the features from error-inducing images. It then applies a density-based clustering algorithm to detect arbitrary shaped clusters of images modeling plausible causes of error. Last, clusters are used to effectively retrain and improve the DNN. The black-box nature of SAFE is motivated by our objective not to require changes or even access to the DNN internals to facilitate adoption. Experimental results show the superior ability of SAFE in identifying different root causes of DNN errors based on case studies in the automotive domain. It also yields significant improvements in DNN accuracy after retraining, while saving significant execution time and memory when compared to alternatives.

CMD: controllable matrix decomposition with global optimization for deep neural network compression

The compression and acceleration of Deep neural networks (DNNs) are necessary steps to deploy sophisticated networks into resource-constrained hardware systems. Due to the weight matrix tends to be low-rank and sparse, several low-rank and sparse compression schemes are leveraged to reduce the overwhelmed weight parameters of DNNs. In these previous schemes, how to make the most of the low-rank and sparse components of weight matrices and how to globally decompose the weight matrix of different layers for efficient compression need to be further investigated. In this paper, in order to effectively utilize the low-rank and sparse characteristics of the weight matrix, we first introduce a sparse coefficient to dynamically control the allocation between the low-rank and sparse components, and an efficient reconstructed network is designed to reduce the inference time. Secondly, since the results of low-rank decomposition can affect the compression ratio and accuracy of DNNs, we establish an optimization problem to automatically select the optimal hyperparameters of the compressed network and achieve global compression for all the layers of network synchronously. Finally, to solve the optimization problem, we present a decomposition-searching algorithm to search the optimal solution. The algorithm can dynamically keep the balance between the compression ratio and accuracy. Extensive experiments of AlexNet, VGG-16 and ResNet-18 on CIFAR-10 and ImageNet are employed to evaluate the effectiveness of the proposed approach. After slight fine-tuning, compressed networks have gained \(1.2\times\) to \(11.3\times\) speedup and our method reduces the size of different networks by \(1.4\times\) to \(14.6\times\).

TaxoNERD: Deep neural models for the recognition of taxonomic entities in the ecological and evolutionary literature

Abstract Easy access to multi‐taxa information (e.g. distribution, traits, diet) in the scientific literature is essential to understand, map and predict all‐inclusive biodiversity. Tools are needed to automatically extract useful information from the ever‐growing corpus of ecological texts and feed this information to open data repositories. A prerequisite is the ability to recognise mentions of taxa in text, a special case of named entity recognition (NER). In recent years, deep learning‐based NER systems have become ubiquitous, yielding state‐of‐the‐art results in the general and biomedical domains. However, no such tool is available to ecologists wishing to extract information from the biodiversity literature. We propose a new tool called TaxoNERD that provides deep neural network (DNN) models to recognise taxon mentions in ecological documents. To achieve high performance, these models usually need to be trained on a large corpus of manually annotated text. Creating such a corpus is a laborious and costly process, with the result that manually annotated corpora in the ecological domain tend to be too small to learn an accurate DNN model from scratch. To address this issue, we leverage existing models pretrained on large biomedical corpora using transfer learning. The performance of our models is evaluated on four corpora and compared to the most popular taxonomic entity recognition tools. Our experiments suggest that existing taxonomic NER tools are not suited to the extraction of ecological information from text as they performed poorly on ecologically oriented corpora, either because they do not take account of the variability of taxon naming practices or because they do not generalise well to the ecological domain. Conversely, a domain‐specific DNN‐based tool like TaxoNERD outperformed the other approaches on an ecological information extraction task. Efforts are needed to raise ecological information extraction to the same level of performance as its biomedical counterpart. One promising direction is to leverage the huge corpus of unlabelled ecological texts to learn a language representation model that could benefit downstream tasks. These efforts could be highly beneficial to ecologists on the long term.

A cascade model-aware generative adversarial example detection method

Deep Neural Networks (DNNs) are demonstrated to be vulnerable to adversarial examples, which are elaborately crafted to fool learning models. Since the accuracy and robustness of DNNs are at odds for the adversarial training method, the adversarial example detection algorithms check whether the specific example is adversarial, which is promising to solve the issue of the adversarial example. However, among the existing methods, model-aware detection methods do not generalize well, while the detection accuracies of the generative-based methods are lower compared to the model-aware methods. In this paper, we propose a cascade model-aware generative adversarial example detection method, namely CMAG. CMAG consists of two first-order reconstructors and a second-order reconstructor, which can illustrate what the model sees to the human by reconstructing the logit and feature maps of the last convolution layer. Experimental results demonstrate that our method is effective and is more interpretable compared to some state-of-the-art methods.

Deep Neural Network for Speaker Identification Using Static and Dynamic Prosodic Feature for Spontaneous and Dictated Data

We can recognize a person by his voice alone. In principle, the sound has a tone (pitch) that is different for each person. This study aims to measure a Deep Neural Network (DNN) performance with static and dynamic prosodic features. Prosodic is information about sound related to tone, intonation, pressure, duration, and rhythm of a person's pronunciation. The data used is dictated and spontaneous voice data that taken from YouTube. The data used consists of three male voices and one female voice. The data is segmented into various duration, 3 seconds, 5 seconds, and 10 seconds. After the data has been segmented, the static prosodic features with 103 dimensions will be extracted and the dynamic prosodic features with 13 dimensions will be extracted too. Each feature and feature combination will be trained and tested using DNN with a ratio of 90:10. The result shows that the 10 seconds segmented data has higher accuracy than the others. Accuracy of static prosodic features is better than dynamic prosodic features. The average accuracy of DNN for static prosodic features is 87.02%. The average accuracy of DNN for dynamic prosodic features is 72.97%. The average accuracy of DNN for combined static and dynamic prosodic features is 87.72%.

Deep Neural Network for Speaker Identification Using Static and Dynamic Prosodic Feature for Spontaneous and Dictated Data

We can recognize a person by his voice alone. In principle, the sound has a tone (pitch) that is different for each person. This study aims to measure a Deep Neural Network (DNN) performance with static and dynamic prosodic features. Prosodic is information about sound related to tone, intonation, pressure, duration, and rhythm of a person's pronunciation. The data used is dictated and spontaneous voice data that taken from YouTube. The data used consists of three male voices and one female voice. The data is segmented into various duration, 3 seconds, 5 seconds, and 10 seconds. After the data has been segmented, the static prosodic features with 103 dimensions will be extracted and the dynamic prosodic features with 13 dimensions will be extracted too. Each feature and feature combination will be trained and tested using DNN with a ratio of 90:10. The result shows that the 10 seconds segmented data has higher accuracy than the others. Accuracy of static prosodic features is better than dynamic prosodic features. The average accuracy of DNN for static prosodic features is 87.02%. The average accuracy of DNN for dynamic prosodic features is 72.97%. The average accuracy of DNN for combined static and dynamic prosodic features is 87.72%.

A Smoothed LASSO-Based DNN Sparsification Technique

Deep Neural Networks (DNNs) are increasingly being used in a variety of applications. However, DNNs have huge computational and memory requirements. One way to reduce these requirements is to sparsify DNNs by using smoothed LASSO (Least Absolute Shrinkage and Selection Operator) functions. In this paper, we show that irrespective of error profile, the sparsity values obtained using various smoothed LASSO functions are similar, provided the maximum error of these functions with respect to the LASSO function is the same. We also propose a layer-wise DNN pruning algorithm, where the layers are pruned based on their individual allocated accuracy loss budget, determined by estimates of the reduction in number of multiply-accumulate operations (in convolutional layers) and weights (in fully connected layers). Further, the structured LASSO variants in both convolutional and fully connected layers are explored within the smoothed LASSO framework and the tradeoffs involved are discussed. The efficacy of proposed algorithm in enhancing the sparsity within the allowed degradation in DNN accuracy and results obtained on structured LASSO variants are shown on MNIST, SVHN, CIFAR-10, and Imagenette datasets and on larger networks such as ResNet-50 and Mobilenet.

Implementation of Lung Cancer Diagnosis based on DNN in Healthcare System

Cancer represents a kind of disease that is widespread throughout the world. Actually, there are several kinds of cancer. However, lung cancer represents the most prevalent cancer form and can lead to death with late healthcare. Therefore, it is essential to initialize therapy via diagnosing lung cancer for decreasing the death chance. Classification is one of the fundamental issues in the knowledge discovery fields and scientific decisions. There are many types of techniques used for constructing classifiers and cancer diagnosis. Recently, deep learning becomes a powerful and popular classification technique for many areas of medical data diagnosis in the healthcare systems. In this paper, an effective and accurate deep neural network (DNN) based lung cancer diagnosis implemented in the healthcare system has been proposed which includes three main phases; pre-processing, generating strong rules, and classification. The input data are pre-processed in the first phase. Because these data are entered from databases, so there are missing data that should be replaced with zero values. Then, data are normalized for speeding up the learning phase. After that, the class association rule is used to enhance the classification performance by generating frequent patterns inducible from the dataset which include features that are significant to the class attribute. Finally, DNN is utilized in the process of classification for obtaining a sample diagnosis estimate. DNN based diagnosis system was tested and evaluated on the lung cancer dataset which has 25 attributes and 1000 instances. The obtained results demonstrated that the proposed system achieved a high performance compared to other existing lung cancer diagnosis systems with 95% accuracy, 97% specificity, and 95% sensitivity.

FLASH: F ast Neura l A rchitecture S earch with H ardware Optimization

Neural architecture search (NAS) is a promising technique to design efficient and high-performance deep neural networks (DNNs). As the performance requirements of ML applications grow continuously, the hardware accelerators start playing a central role in DNN design. This trend makes NAS even more complicated and time-consuming for most real applications. This paper proposes FLASH, a very fast NAS methodology that co-optimizes the DNN accuracy and performance on a real hardware platform. As the main theoretical contribution, we first propose the NN-Degree, an analytical metric to quantify the topological characteristics of DNNs with skip connections (e.g., DenseNets, ResNets, Wide-ResNets, and MobileNets). The newly proposed NN-Degree allows us to do training-free NAS within one second and build an accuracy predictor by training as few as 25 samples out of a vast search space with more than 63 billion configurations. Second, by performing inference on the target hardware, we fine-tune and validate our analytical models to estimate the latency, area, and energy consumption of various DNN architectures while executing standard ML datasets. Third, we construct a hierarchical algorithm based on simplicial homology global optimization (SHGO) to optimize the model-architecture co-design process, while considering the area, latency, and energy consumption of the target hardware. We demonstrate that, compared to the state-of-the-art NAS approaches, our proposed hierarchical SHGO-based algorithm enables more than four orders of magnitude speedup (specifically, the execution time of the proposed algorithm is about 0.1 seconds). Finally, our experimental evaluations show that FLASH is easily transferable to different hardware architectures, thus enabling us to do NAS on a Raspberry Pi-3B processor in less than 3 seconds.

Evolutionary trajectory hybridization for improving deep learning accuracy in medical data prediction system

Medical prediction systems recognize as a significant AI field of study. A huge number of methodologies have been produced in the last decades to solve different kinds of prediction system problems in medical fields, in order to assist physicians in their diagnoses. Deep neural network (DNN) considers an extraordinary method in the medical prediction area. The purpose of this study is to enhance DNN accuracy using a meta-heuristic approach by propose a hybrid model formulated of two meta-heuristic algorithms evolutionary as population-based solution and trajectory as a single-based solution to improve (DNN) to facilitate exploration and exploitation balance and consequently enhance medical dataset prediction. The complex structure of DNN to achieve a best classification accuracy would be useful for meta-heuristic algorithm efficiency for searching the global optimum. To verify the proposal method, the study enhanced DNN with four meta-heuristic algorithms to test the selected medical benchmarks separately. The meta-heuristic algorithms included two population-based algorithms, namely, genetic algorithms GA and differential evolution DE, and two single-based algorithms, namely, simulated annealing SA and Tabu search TS. The proposed methodology registered a significant enhancement in anemia classification comparing with four other meta-heuristic algorithms; the anemia data was composed of a real dataset gathered from Iraqi blood laboratories to detect anemia diseases. Meanwhile, the proposal applied two benchmarks from the University of California repository (UCI), namely, Pima Indian diabetes data and liver disorder diseases. The proposed method attained remarkable results.

(AD)[formula omitted]: Adversarial domain adaptation to defense with adversarial perturbation removal

Deep Neural Networks (DNNs) are demonstrated to be vulnerable to adversarial examples, which are crafted by adding adversarial perturbations to the legitimate examples. To address this issue, some defense methods have been proposed. Among them, the adversarial training (AT) is a popular method to improve the robustness of DNNs. However, theory analysis has shown that in the adversarial training framework, the improvement of the robustness will lead to a decline of standard accuracy. In this paper, we propose a modularized defense framework, namely Adversarial Domain Adaptation to Defense ((AD)2). Different from all adversarial training methods, (AD)2 detects adversarial example using a generative algorithm and applies the adversarial domain adaptation method to remove adversarial perturbation. Experimental results show that (AD)2 is effective to remove the adversarial perturbation and mitigate the odds between the robustness and standard accuracy for DNNs.