Accuracy Of Deep Neural Network Research Articles

Benchmarking Test-Time DNN Adaptation at Edge with Compute-In-Memory

The prediction accuracy of deep neural networks (DNNs) deployed at the edge can deteriorate over time due to shifts in the data distribution. For heightened robustness, it’s crucial for DNNs to continually refine and improve their predictive capabilities. However, adaptation in resource-limited edge environments is fraught with challenges: (i) new labeled data might be unavailable; (ii) on-device adaptation is a necessity as cloud connections may be inaccessible; and (iii) the adaptation procedure should prioritize speed, memory efficiency, and energy conservation. Compute-In-Memory (CIM) has recently garnered attention for its computational efficacy and superior operational bandwidth. Additionally, emerging lightweight unsupervised DNN adaptation techniques during test-time have showcased promising results in enhancing model accuracy for data with noise. This article pioneers a holistic benchmarking exploration of these methods, assessing their performance and energy efficacy across diverse CIM architectures in edge and autonomous systems. Our findings reveal that the proposed adaptation strategies can adapt to both environment shifts and inherent hardware noise. Engaging in a thorough cross-layer algorithm-hardware-technology co-design space exploration, we highlight pivotal trade-offs among accuracy, performance, and energy for various DNN adaptation techniques and CIM configurations.

A depth analysis of recent innovations in non-invasive techniques using artificial intelligence approach for cancer prediction.

The fight against cancer, a relentless global health crisis, emphasizes the urgency for efficient and automated early detection methods. To address this critical need, this review assesses recent advances in non-invasive cancer prediction techniques, comparing conventional machine learning (CML) and deep neural networks (DNNs). Focusing on these seven major cancers, we analyze 310 publications spanning the years 2018 to 2024, focusing on detection accuracy as the key metric to identify the most effective predictive models, highlighting critical gaps in current methodologies, and suggesting directions for future research. We further delved into factors like datasets, features, and modalities to gain a comprehensive understanding of each approach's performance. Separate review tables for each cancer type and approach facilitated comparisons between top performers (accuracy exceeding 99%) and low performers (65.83 to 85.8%). Our exploration of public databases and commonly used classifiers revealed that optimal combinations of features, datasets, and models can achieve up to 100% accuracy for both CML and DNN. However, significant variations in accuracy (up to 35%) were observed, particularly when optimization was lacking. Notably, colorectal cancer exhibited the lowest accuracy (DNN 69%, CML 65.83%). A five-point comparative analysis (best/worst models, performance gap, average accuracy, and research trends) revealed that while DNN research is gaining momentum, CML approaches remain competitive, even outperforming DNN in some cases. This study presents an in-depth comparative analysis of CML and DNN techniques for cancer detection. This knowledge can inform future research directions and contribute to the development of increasingly accurate and reliable cancer detection tools.

On the choice of physical constraints in artificial neural networks for predicting flow fields

The application of Artificial Neural Networks (ANNs) has been extensively investigated for fluid dynamic problems. A specific form of ANNs are Physics-Informed Neural Networks (PINNs). They incorporate physical laws in the training and have increasingly been explored in the last few years. In this work, the prediction accuracy of PINNs is compared with that of conventional Deep Neural Networks (DNNs). The accuracy of a DNN depends on the amount of data provided for training. The change in prediction accuracy of PINNs and DNNs is assessed using a varying amount of training data. To ensure the correctness of the training data, they are obtained from analytical and numerical solutions of classical problems in fluid mechanics. The objective of this work is to quantify the fraction of training data relative to the maximum number of data points available in the computational domain, such that the accuracy gained with PINNs justifies the increased computational cost. Furthermore, the effects of the location of sampling points in the computational domain and noise in training data are analyzed. In the considered problems, it is found that PINNs outperform DNNs when the sampling points are positioned in the Regions of Interest. PINNs for predicting potential flow around a Rankine oval have shown a better robustness against noise in training data compared to DNNs. Both models show higher prediction accuracy when sampling points are randomly positioned in the flow domain as compared to a prescribed distribution of sampling points. The findings reveal new insights on the strategies to massively improve the prediction capabilities of PINNs with respect to DNNs.

Thermal Effects on Monolithic 3D Ferroelectric Transistors for Deep Neural Networks Performance

Monolithic three‐dimensional (M3D) integration advances integrated circuits by enhancing density and energy efficiency. Ferroelectric thin‐film transistors (Fe‐TFTs) attract attention for neuromorphic computing and back‐end‐of‐the‐line (BEOL) compatibility. However, M3D faces challenges like increased runtime temperatures due to limited heat dissipation, impacting system reliability. This work demonstrates the effect of temperature impact on single‐gate (SG) Fe‐TFT reliability. SG Fe‐TFTs have limitations such as read‐disturbance and small memory windows, constraining their use. To mitigate these, dual‐gate (DG) Fe‐TFTs are modeled using technology computer‐aided design, comparing their performance. Compute‐in‐memory (CIM) architectures with SG and DG Fe‐TFTs are investigated for deep neural networks (DNN) accelerators, revealing heat's detrimental effect on reliability and inference accuracy. DG Fe‐TFTs exhibit about 4.6x higher throughput than SG Fe‐TFTs. Additionally, thermal effects within the simulated M3D architecture are analyzed, noting reduced DNN accuracy to 81.11% and 67.85% for SG and DG Fe‐TFTs, respectively. Furthermore, various cooling methods and their impact on CIM system temperature are demonstrated, offering insights for efficient thermal management strategies.

Projection-based reduced order modeling of multi-species mixing and combustion

High-fidelity simulations of mixing and combustion processes are computationally demanding and time-consuming, hindering their wide application in industrial design and optimization. This study proposes projection-based reduced order models (ROMs) to predict spatial distributions of physical fields for multi-species mixing and combustion problems in a fast and accurate manner. The developed ROMs explore the suitability of various regression methods, including kriging, multivariate polynomial regression (MPR), k-nearest neighbors (KNN), deep neural network (DNN), and support vector regression (SVR), for the functional mapping between input parameters and reduced model coefficients of mixing and combustion problems. The ROMs are systematically examined using two distinct configurations: steam-diluted hydrogen-enriched oxy-combustion from a triple-coaxial nozzle and fuel-flexible combustion in a practical gas-turbine combustor. The projected low-dimensional manifolds are capable of capturing important combustion physics, and the response surfaces of reduced model coefficients present pronounced nonlinear characteristics of the flowfields with varying input parameters. The ROMs with kriging present a superior performance of establishing the input–output mapping to predict almost all physical fields, such as temperature, velocity magnitude, and combustion products for both test problems. The accuracy of DNN is less encouraging owing to the stringent requirement on the size of training database. KNN performs well in the region near the design points but its effectiveness diminishes when the test points are distant from the sampling points, whereas SVR and MPR exhibit large prediction errors. For the spatial prediction at unseen design points, the ROMs achieve a prediction time of up to eight orders of magnitude faster than conventional numerical simulations, rendering an efficient tool for the fast prediction of mixing and combustion fields and potentially an alternative of a full-order numerical solver.

Introducing innovative deep neural networks to predict natural frequency of the nanocomposites reinforced concrete structures under external loading

The dynamic response of concrete structures reinforced with nanocomposites to supersonic airflow represents a critical aspect in aerospace and defense applications, necessitating accurate predictive models for enhanced structural integrity and performance. In this study, we introduce innovative deep neural networks (DNNs) as a novel approach to predict the dynamic behavior of such structures under supersonic airflow conditions. Traditional modeling techniques often face challenges in capturing the intricate interactions between material properties, structural geometry, and airflow dynamics, particularly in the presence of nanocomposite reinforcements. DNNs offer a promising solution by leveraging their ability to learn complex patterns and nonlinear relationships from extensive datasets. This paper presents a comprehensive framework for developing and deploying DNN-based models for dynamic prediction, encompassing network architecture design, training strategies, and data preprocessing techniques tailored to the unique characteristics of nanocomposite-reinforced concrete structures. Through a series of case studies and comparative analyses, we demonstrate the effectiveness and accuracy of DNNs in capturing the dynamic response of such structures to supersonic airflow, including phenomena such as vibration, flutter, and aerodynamic instability. Furthermore, we discuss the potential advantages and challenges associated with the adoption of DNNs in aerospace and defense applications, including model interpretability, computational efficiency, and data requirements. Finally, we outline future research directions and opportunities for further advancing the application of DNNs in addressing dynamic challenges in aerospace engineering and beyond.

AraDQ: an automated digital phenotyping software for quantifying disease symptoms of flood-inoculated Arabidopsis seedlings

BackgroundPlant scientists have largely relied on pathogen growth assays and/or transcript analysis of stress-responsive genes for quantification of disease severity and susceptibility. These methods are destructive to plants, labor-intensive, and time-consuming, thereby limiting their application in real-time, large-scale studies. Image-based plant phenotyping is an alternative approach that enables automated measurement of various symptoms. However, most of the currently available plant image analysis tools require specific hardware platform and vendor specific software packages, and thus, are not suited for researchers who are not primarily focused on plant phenotyping. In this study, we aimed to develop a digital phenotyping tool to enhance the speed, accuracy, and reliability of disease quantification in Arabidopsis.ResultsHere, we present the Arabidopsis Disease Quantification (AraDQ) image analysis tool for examination of flood-inoculated Arabidopsis seedlings grown on plates containing plant growth media. It is a cross-platform application program with a user-friendly graphical interface that contains highly accurate deep neural networks for object detection and segmentation. The only prerequisite is that the input image should contain a fixed-sized 24-color balance card placed next to the objects of interest on a white background to ensure reliable and reproducible results, regardless of the image acquisition method. The image processing pipeline automatically calculates 10 different colors and morphological parameters for individual seedlings in the given image, and disease-associated phenotypic changes can be easily assessed by comparing plant images captured before and after infection. We conducted two case studies involving bacterial and plant mutants with reduced virulence and disease resistance capabilities, respectively, and thereby demonstrated that AraDQ can capture subtle changes in plant color and morphology with a high level of sensitivity.ConclusionsAraDQ offers a simple, fast, and accurate approach for image-based quantification of plant disease symptoms using various parameters. Its fully automated pipeline neither requires prior image processing nor costly hardware setups, allowing easy implementation of the software by researchers interested in digital phenotyping of diseased plants.

Separating hard clean samples from noisy samples with samples’ learning risk for DNN when learning with noisy labels

Learning with Noisy Labels (LNL) methods aim to improve the accuracy of Deep Neural Networks (DNNs) when the training set contains samples with noisy or incorrect labels, and have become popular in recent years. Existing popular LNL methods frequently regard samples with high learning difficulty (high-loss and low prediction probability) as noisy samples; however, irregular feature patterns from hard clean samples can also cause high learning difficulty, which can lead to the misclassification of hard clean samples as noisy samples. To address this insufficiency, we propose the Samples’ Learning Risk-based Learning with Noisy Labels (SLRLNL) method. Specifically, we propose to separate noisy samples from hard clean samples using samples’ learning risk, which represents samples’ influence on DNN’s accuracy . We show that samples’ learning risk is comprehensively determined by samples’ learning difficulty as well as samples’ feature similarity to other samples, and thus, compared to existing LNL methods that solely rely on the learning difficulty, our method can better separate hard clean samples from noisy samples, since the former frequently possess irregular feature patterns. Moreover, to extract more useful information from samples with irregular feature patterns (i.e., hard samples), we further propose the Relabeling-based Label Augmentation (RLA) process to prevent the memorization of hard noisy samples and better learn the hard clean samples, thus enhancing the learning for hard samples. Empirical studies show that samples’ learning risk can identify noisy samples more accurately, and the RLA process can enhance the learning for hard samples. To evaluate the effectiveness of our method, we compare it with popular existing LNL methods on CIFAR-10, CIFAR-100, Animal-10N, Clothing1M, and Docred. The experimental results indicate that our method outperforms other existing methods. The source code for SLRLNL can be found at https://github.com/yangbo1973/SLRLNL.

CardioDefense: Defending against adversarial attack in ECG classification with adversarial distillation training

In clinics, doctors rely on electrocardiograms (ECGs) to assess severe cardiac disorders. Owing to the development of technology and the increase in health awareness, ECG signals are currently obtained by using medical and commercial devices. Deep neural networks (DNNs) can be used to analyze these signals because of their high accuracy rate. However, researchers have found that adversarial attacks can significantly reduce the accuracy of DNNs. Studies have been conducted to defend ECG-based DNNs against traditional adversarial attacks, such as projected gradient descent (PGD), and smooth adversarial perturbation (SAP) which targets ECG classification; however, to the best of our knowledge, no study has completely explored the defense against adversarial attacks targeting ECG classification. Thus, we did different experiments to explore the effects of defense methods against white-box adversarial attack and black-box adversarial attack targeting ECG classification, and we found that some common defense methods performed well against these attacks. Besides, we proposed a new defense method based on adversarial distillation training (named CardioDefense) which comes from defensive distillation and can effectively improve the generalization performance of DNNs. The results show that our method performed more effectively against adversarial attacks targeting on ECG classification than the other baseline methods, namely, adversarial training, defensive distillation, Jacob regularization, and noise-to-signal ratio regularization. Furthermore, we found that our method performed better against PGD attacks with low noise levels, which means that our method has stronger robustness.

Neural Network Compression Based on Tensor Ring Decomposition.

Deep neural networks (DNNs) have made great breakthroughs and seen applications in many domains. However, the incomparable accuracy of DNNs is achieved with the cost of considerable memory consumption and high computational complexity, which restricts their deployment on conventional desktops and portable devices. To address this issue, low-rank factorization, which decomposes the neural network parameters into smaller sized matrices or tensors, has emerged as a promising technique for network compression. In this article, we propose leveraging the emerging tensor ring (TR) factorization to compress the neural network. We investigate the impact of both parameter tensor reshaping and TR decomposition (TRD) on the total number of compressed parameters. To achieve the maximal parameter compression, we propose an algorithm based on prime factorization that simultaneously identifies the optimal tensor reshaping and TRD. In addition, we discover that different execution orders of the core tensors result in varying computational complexities. To identify the optimal execution order, we construct a novel tree structure. Based on this structure, we propose a top-to-bottom splitting algorithm to schedule the execution of core tensors, thereby minimizing computational complexity. We have performed extensive experiments using three kinds of neural networks with three different datasets. The experimental results demonstrate that, compared with the three state-of-the-art algorithms for low-rank factorization, our algorithm can achieve better performance with much lower memory consumption and lower computational complexity.

Partial Density of States Representation for Accurate Deep Neural Network Predictions of X-ray Spectra

Partial Density of States Representation for Accurate Deep Neural Network Predictions of X-ray Spectra

An Accurate Deep Neural Network Based Inversion Approach for Multichannel Microwave Interferometer Characterization of Plasma Wake Electron Density Distribution

An Accurate Deep Neural Network Based Inversion Approach for Multichannel Microwave Interferometer Characterization of Plasma Wake Electron Density Distribution

Joint Protection Scheme for Deep Neural Network Hardware Accelerators and Models

Deep neural networks (DNNs) are utilized in numerous image processing, object detection, and video analysis tasks and need to be implemented using hardware accelerators to achieve practical speed. Logic locking is one of the most popular methods for preventing chip counterfeiting. Nevertheless, existing logic-locking schemes need to sacrifice the number of input patterns leading to wrong output under incorrect keys to resist the powerful satisfiability (SAT)-attack. Furthermore, DNN model inference is fault-tolerant. Hence, using a wrong key for those SAT-resistant logic-locking schemes may not affect the accuracy of DNNs. This makes the previous SAT-resistant logic-locking scheme ineffective on protecting DNN accelerators. Besides, to prevent DNN models from being illegally used, the models need to be obfuscated by the designers before they are provided to end-users. Previous obfuscation methods either require long time to retrain the model or leak information about the model. This paper proposes a joint protection scheme for DNN hardware accelerators and models. The DNN accelerator is modified using a hardware key (Hkey) and a model key (Mkey). Different from previous logic locking, the Hkey, which is used to protect the accelerator, does not affect the output when it is wrong. As a result, the SAT attack can be effectively resisted. On the other hand, a wrong Hkey leads to substantial increase in memory accesses, inference time, and energy consumption and makes the accelerator unusable. A correct Mkey can recover the DNN model that is obfuscated by the proposed method. Compared to previous model obfuscation schemes, our proposed method avoids model retraining and does not leak model information.

A Bayesian deep learning framework for reliable fault diagnosis in wind turbine gearboxes under various operating conditions

Vibration-based fault diagnostics combined with deep learning approaches has promising applications in detecting and diagnosing faults in wind turbine gearboxes. Specifically when time series vibration data is transformed to a 2-dimensional cyclic spectral coherence maps, the accuracy of deep neural networks in classifying faults increases. Nevertheless, standard deep learning techniques are vulnerable to inaccurate predictions when tested with new data originating from unseen faults or unusual operating conditions. To address some of these shortcomings in the context of wind turbine gearboxes, this paper investigates fault diagnostics using Bayesian convolutional neural network which provide accurate results with uncertainty bounds reducing wrong overconfident classifications. The performance of Bayesian and standard neural networks is compared using a simulation-based dataset of acceleration signals generated from a multibody dynamic model of a 5 MW wind turbine. The framework proposed in this paper has relevance to fault detection and diagnosis in other rotating machinery applications.

A White-Box Testing for Deep Neural Networks Based on Neuron Coverage.

With the introduction of neuron coverage as a testing criterion for deep neural networks (DNNs), covering more neurons to detect more internal logic of DNNs became the main goal of many research studies. While some works had made progress, some new challenges for testing methods based on neuron coverage had been proposed, mainly as establishing better neuron selection and activation strategies influenced not only obtaining higher neuron coverage, but also more testing efficiency, validating testing results automatically, labeling generated test cases to extricate manual work, and so on. In this article, we put forward Test4Deep, an effective white-box testing DNN approach based on neuron coverage. It is based on a differential testing framework to automatically verify inconsistent DNNs' behavior. We designed a strategy that can track inactive neurons and constantly triggered them in each iteration to maximize neuron coverage. Furthermore, we devised an optimization function that guided the DNN under testing to deviate predictions between the original input and generated test data and dominated unobservable generation perturbations to avoid manually checking test oracles. We conducted comparative experiments with two state-of-the-art white-box testing methods DLFuzz and DeepXplore. Empirical results on three popular datasets with nine DNNs demonstrated that compared to DLFuzz and DeepXplore, Test4Deep, on average, exceeded by 32.87% and 35.69% in neuron coverage, while reducing 58.37% and 53.24% testing time, respectively. In the meantime, Test4Deep also produced 58.37% and 53.24% more test cases with 23.81% and 98.40% fewer perturbations. Even compared with the two highest neuron coverage strategies of DLFuzz, Test4Deep still enhanced neuron coverage by 4.34% and 23.23% and achieved 94.48% and 85.67% higher generation time efficiency. Furthermore, Test4Deep could improve the accuracy and robustness of DNNs by merging generated test cases and retraining.

CRAFT: Criticality-Aware Fault-Tolerance Enhancement Techniques for Emerging Memories-Based Deep Neural Networks

Deep Neural Networks (DNNs) have emerged as the most effective programming paradigm for computer vision and natural language processing applications. With the rapid development of DNNs, efficient hardware architectures for deploying DNN-based applications on edge devices have been extensively studied. Emerging Non-Volatile Memories (NVMs), with their better scalability, non-volatility and good read performance, are found to be promising candidates for deploying DNNs. However, despite the promise, emerging NVMs often suffer from reliability issues such as stuck-at faults, which decrease the chip yield/memory lifetime and severely impact the accuracy of DNNs. A stuck-at cell can be read but not reprogrammed, thus, stuck-at faults in NVMs may or may not result in errors depending on the data to be stored. By reducing the number of errors caused by stuck-at faults, the reliability of a DNN-based system can be enhanced. This paper proposes CRAFT, i.e., Criticality-Aware Fault-Tolerance Enhancement Techniques to enhance the reliability of NVM-based DNNs in the presence of stuck-at faults. A data block remapping technique is used to reduce the impact of stuck-at faults on DNNs accuracy. Additionally, by performing bit-level criticality analysis on various DNNs, the critical-bit positions in network parameters that can significantly impact the accuracy are identified. Based on this analysis, we propose an encoding method which effectively swaps the critical bit positions with that of non-critical bits when more errors (due to stuck-at faults) are present in the critical bits.

Exploration of Bitflip’s Effect on Deep Neural Network Accuracy in Plaintext and Ciphertext

Neural Networks (NNs) are increasingly deployed to solve complex classification problems and produce accurate results on reliable systems. However, its accuracy quickly degrades in the presence of bit flips from memory errors or targeted attacks on DRAM main memory. Prior work has shown that a few bit errors significantly reduce NN accuracies, but it is unclear which bits have an outsized impact on network accuracy and why. This paper first investigates the relationship of the number representation for NN parameters with the impacts of bit flips on NN accuracy. We then explore the Bit Flip Detection (BFD) framework— four software-based error detectors that detect bit flips independent of NN topology. We discuss exciting findings and evaluate the various detectors’ efficacy, characteristics, and trade-offs.

A Framework for Identifying Essential Proteins with Hybridizing Deep Neural Network and Ordinary Least Squares

Essential proteins are vital for maintaining life activities and play a crucial role in biological processes. Identifying essential proteins is of utmost importance as it helps in understanding the minimal requirements for cell life, discovering pathogenic genes and drug targets, diagnosing diseases, and comprehending the mechanism of biological evolution. The latest research suggests that integrating protein–protein interaction (PPI) networks and relevant biological sequence features can enhance the accuracy and robustness of essential protein identification. In this paper, a deep neural network (DNN) method was used to identify a yeast essential protein, which was named IYEPDNN. The method combines gene expression profiles, PPI networks, and orthology as input features to improve the accuracy of DNN while reducing computational complexity. To enhance the robustness of the yeast dataset, the common least squares method is used to supplement absenting data. The correctness and effectiveness of the IYEPDNN method are verified using the DIP and GAVIN databases. Our experimental results demonstrate that IYEPDNN achieves an accuracy of 84%, and it outperforms state-of-the-art methods (WDC, PeC, OGN, ETBUPPI, RWAMVL, etc.) in terms of the number of essential proteins identified. The findings of this study demonstrate that the correlation between features plays a crucial role in enhancing the accuracy of essential protein prediction. Additionally, selecting the appropriate training data can effectively address the issue of imbalanced training data in essential protein identification.

Neural dynamics for improving optimiser in deep learning with noise considered

AbstractAs deep learning evolves, neural network structures become increasingly sophisticated, bringing a series of new optimisation challenges. For example, deep neural networks (DNNs) are vulnerable to a variety of attacks. Training neural networks under privacy constraints is a method to alleviate privacy leakage, and one way to do this is to add noise to the gradient. However, the existing optimisers suffer from weak convergence in the presence of increased noise during training, which leads to a low robustness of the optimiser. To stabilise and improve the convergence of DNNs, the authors propose a neural dynamics (ND) optimiser, which is inspired by the zeroing neural dynamics originated from zeroing neural networks. The authors first analyse the relationship between DNNs and control systems. Then, the authors construct the ND optimiser to update network parameters. Moreover, the proposed ND optimiser alleviates the non‐convergence problem that may be suffered by adding noise to the gradient from different scenarios. Furthermore, experiments are conducted on different neural network structures, including ResNet18, ResNet34, Inception‐v3, MobileNet, and long and short‐term memory network. Comparative results using CIFAR, YouTube Faces, and R8 datasets demonstrate that the ND optimiser improves the accuracy and stability of DNNs under noise‐free and noise‐polluted conditions. The source code is publicly available at https://github.com/LongJin‐lab/ND.

Feature-based deep neural network approach for predicting mortality risk in patients with COVID-19

In this study, we integrate deep neural network (DNN) with hybrid approaches (feature selection and instance clustering) to build prediction models for predicting mortality risk in patients with COVID-19. Besides, we use cross-validation methods to evaluate the performance of these prediction models, including feature based DNN, cluster-based DNN, DNN, and neural network (multi-layer perceptron). The COVID-19 dataset with 12,020 instances and 10 cross-validation methods are used to evaluate the prediction models. The experimental results showed that the proposed feature based DNN model, holding Recall (98.62%), F1-score (91.99%), Accuracy (91.41%), and False Negative Rate (1.38%), outperforms than original prediction model (neural network) in the prediction performance. Furthermore, the proposed approach uses the Top 5 features to build a DNN prediction model with high prediction performance, exhibiting the well prediction as the model built by all features (57 features). The novelty of this study is that we integrate feature selection, instance clustering, and DNN techniques to improve prediction performance. Moreover, the proposed approach which is built with fewer features performs much better than the original prediction models in many metrics and can still remain high prediction performance.