Accuracy Of Deep Neural Network Research Articles

Prediction of Mixing Uniformity of Hydrogen Injection inNatural Gas Pipeline Based on a Deep Learning Model

It is economical and efficient to use existing natural gas pipelines to transport hydrogen. The fast and accurate prediction of mixing uniformity of hydrogen injection in natural gas pipelines is important for the safety of pipeline transportation and downstream end users. In this study, the computational fluid dynamics (CFD) method was used to investigate the hydrogen injection process in a T-junction natural gas pipeline. The coefficient of variation (COV) of a hydrogen concentration on a pipeline cross section was used to quantitatively characterize the mixing uniformity of hydrogen and natural gas. To quickly and accurately predict the COV, a deep neural network (DNN) model was constructed based on CFD simulation data, and the main influencing factors of the COV including flow velocity, hydrogen blending ratio, gas temperature, flow distance, and pipeline diameter ratio were taken as input nodes of the DNN model. In the model training process, the effects of various parameters on the prediction accuracy of the DNN model were studied, and an accurate DNN architecture was constructed with an average error of 4.53% for predicting the COV. The computational efficiency of the established DNN model was also at least two orders of magnitude faster than that of the CFD simulations for predicting the COV.

Exploring Model Stability of Deep Neural Networks for Reliable RRAM-Based In-Memory Acceleration

RRAM-based in-memory computing (IMC) effectively accelerates deep neural networks (DNNs). Furthermore, model compression techniques, such as quantization and pruning, are necessary to improve algorithm mapping and hardware performance. However, in the presence of RRAM device variations, low-precision and sparse DNNs suffer from severe post-mapping accuracy loss. To address this, in this work, we investigate a new metric, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">model stability</i> , from the loss landscape to help shed light on accuracy loss under variations and model compression, which guides an algorithmic solution to maximize model stability and mitigate accuracy loss. Based on statistical data from a CMOS/RRAM 1T1R test chip at 65nm, we characterize wafer-level RRAM variations and develop a cross-layer benchmark tool that incorporates quantization, pruning, device variations, model stability, and IMC architecture parameters to assess post-mapping accuracy and hardware performance. Leveraging this tool, we show that a loss-landscape-based DNN model selection for stability effectively tolerates device variations and achieves a post-mapping accuracy higher than that with 50% lower RRAM variations. Moreover, we quantitatively interpret why model pruning increases the sensitivity to variations, while a lower-precision model has better tolerance to variations. Finally, we propose a novel variation-aware training method to improve model stability, in which there exists the most stable model for the best post-mapping accuracy of compressed DNNs. Experimental evaluation of the method shows up to 19%, 21%, and 11% post-mapping accuracy improvement for our 65nm RRAM device, across various precision and sparsity, on CIFAR-10, CIFAR-100, and SVHN datasets, respectively.

Energy-Efficient DNN Inference on Approximate Accelerators Through Formal Property Exploration

Deep neural networks (DNNs) are being heavily utilized in modern applications, putting energy-constraint devices to the test. To bypass high energy consumption issues, approximate computing has been employed in DNN accelerators to balance out the accuracy-energy reduction trade-off. However, the approximation-induced accuracy loss can be very high and drastically degrade the performance of the DNN. Therefore, there is a need for a fine-grain mechanism that would assign specific DNN operations to approximation to maintain acceptable DNN accuracy, while achieving low energy consumption. We present an automated framework for weight-to-approximation mapping through formal property exploration for approximate DNN accelerators. At the MAC unit level, our experimental evaluation surpassed already energy-efficient mappings by more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times 2$ </tex-math></inline-formula> in terms of energy gains, while supporting a fine-grain control over the introduced approximation.

A Framework for Neural Network Architecture and Compile Co-optimization

The efficiency of deep neural network (DNN) solutions on real hardware devices are mainly decided by the DNN architecture and the compiler-level scheduling strategy on the hardware. When we try to fully exploit the underlying hardware and obtain the optimal tradeoff between DNN accuracy and runtime performance, we discovered that the two optimization goals of DNN architecture and scheduling policy are intimately related to each other. However, current hardware-aware Neural Architecture Search (NAS) methods primarily focus on the DNN architecture search process, ignoring the effects of various compiler-level scheduling strategies (e.g., graph-level optimization, loop transformations, parallelization, etc.) on network candidates being evaluated in the search process. As a result, they may overlook the true-optimal DNN implementations on hardware, which can only be discovered by trying-out different combinations of scheduling strategies and DNN architectures. This work proposes a NAS framework (CHaNAS) that searches for not only the network architecture but also the dedicated compiler-level scheduling policy, as the optimal co-design solution on the target hardware. We propose to use a block-based pre-scheduling methodology to reduce the co-design search space and enable the automatic generation of the optimal co-design, including the network architecture and the tensor programs that practice the scheduling policy. Further, we introduce a new search objective function based on the generalization gap to prevent the selection of architectures that are prone to overfitting. We evaluate CHaNAS on Imagenet on different hardware back-ends against the state-of-the-art hardware-aware search method based on the MobileNet-v3 search space. Experimental results show that the co-design solutions obtained by ChaNAS show up to 1.6×, 1.9×, and 1.7×, 24 performance boost on NVIDIA P100 GPU, Intel Xeon 8163 CPU, and Samsung Note 10 Mobile, respectively, over the baselines of the same-level accuracy.

Can a proposed double branch multimodality-contribution-aware TripNet improve the prediction performance of the microvascular invasion of hepatocellular carcinoma based on small samples?

To evaluate the potential improvement of prediction performance of a proposed double branch multimodality-contribution-aware TripNet (MCAT) in microvascular invasion (MVI) of hepatocellular carcinoma (HCC) based on a small sample. In this retrospective study, 121 HCCs from 103 consecutive patients were included, with 44 MVI positive and 77 MVI negative, respectively. A MCAT model aiming to improve the accuracy of deep neural network and alleviate the negative effect of small sample size was proposed and the improvement of MCAT model was verified among comparisons between MCAT and other used deep neural networks including 2DCNN (two-dimentional convolutional neural network), ResNet (residual neural network) and SENet (squeeze-and-excitation network), respectively. Through validation, the AUC value of MCAT is significantly higher than 2DCNN based on CT, MRI, and both imaging (P < 0.001 for all). The AUC value of model with single branch pretraining based on small samples is significantly higher than model with end-to-end training in CT branch and double branch (0.62 vs 0.69, p=0.016, 0.65 vs 0.83, p=0.010, respectively). The AUC value of the double branch MCAT based on both CT and MRI imaging (0.83) was significantly higher than that of the CT branch MCAT (0.69) and MRI branch MCAT (0.73) (P < 0.001, P = 0.03, respectively), which was also significantly higher than common-used ReNet (0.67) and SENet (0.70) model (P < 0.001, P = 0.005, respectively). A proposed Double branch MCAT model based on a small sample can improve the effectiveness in comparison to other deep neural networks or single branch MCAT model, providing a potential solution for scenarios such as small-sample deep learning and fusion of multiple imaging modalities.

Accurate interatomic potential for the nucleation in liquid Ti-Al binary alloy developed by deep neural network learning method

For the traditional empirical potentials, the accuracy of chemical bonding in a wide temperature range is regrettable, and thereby they are not an appropriate choice to explore the nucleation process of crystals. As a developed superalloy in aerospace, the liquid structure and nucleation mechanism of Ti-Al alloy desire to be investigated to comprehend the solidification process deeply. This work develops an accurate deep neural network (DNN) potential for Ti-Al binary system by employing a machine-learning method. Then, the local structure and the nucleation process of liquid Ti-48 %Al and Ti-52 %Al (atomic percentage) alloys from the normal to the undercooled state are performed by molecular dynamics simulation with the DNN potential. It is found that the formation of the locally ordered structures depends on the Ti-Al atom pair during the solidification. The Honeycutt-Andersen indices of 1431,1541 and 1551 are dominant, and the liquid structure is mainly characterized by perfect and distorted icosahedral clusters. By Voronoi Polyhedron (VP) analysis, Ti-centered VPs favor the formation of body-centered-cubic (BCC) structure, while Al-centered VPs have the main contribution to the icosahedral structure. Besides, liquid Ti-52 %Al alloy prefers to form face-centered-cube (FCC) nuclei than BCC nuclei in a high undercooling, which is closely related to the ability to capture Ti-Al bonds. The favor of the nucleation of the γ phase causes the nuclei with non-equilibrium composition, comprised of Ti-rich BCC and Al-rich FCC coordinated atoms, respectively. Novelty, the FCC nuclei in liquid Ti-Al alloys are independently generated without the precursor of BCC coordinated atoms. High undercooling can transform the nucleation mechanism from two steps to one step.

A deep neural network and Bayesian method based framework for all-terminal network reliability estimation considering degradation

Most research on network reliability has considered links to have a binary state, i.e., functioning or failed, whereas nodes are considered flawless. In a more realistic scenario, both links and nodes might fail or may exhibit degradation behavior before failing. This study develops a framework to estimate the all-terminal reliability of a network that considers the degradation and probability of failure of all nodes and links in a network. Unlike previous works on network reliability that considered constant reliability for links, this paper considers the reliability of links, nodes, and the network as functions of time. In the proposed framework, the Bayesian methods (BM) are employed to estimate the reliability of links and nodes as functions of time considering degradation data. Due to the complexity of the all-terminal reliability problem, and to get fast estimations of the reliability of a network, an integration of Monte Carlo (MC) and Deep Neural Networks (DNNs) is proposed. The proposed MC algorithm can estimate the network reliability for given nodes and links reliability values. To speed up the calculation, a DNN model is integrated into the framework, thus enabling accurate and fast estimation of network reliability for given link and node reliability values. The DNN accuracy, based on the RMSE (0.01460), outperforms previous traditional artificial neural network (ANN) approaches. Moreover, the DNN model takes 0.3 ms to compute the reliability for any given links and reliability values. The proposed framework can provide not only reliability point estimates but also credible intervals. Finally, we take advantage of Bayesian methods to integrate new data into the framework as they become available. The framework uses the new data to refine and further update the degradation model parameters and the prediction of the reliability of links, nodes, and the network. The proposed methodology has been demonstrated with the real-world network topology Ion (125 nodes, 150 links) with actual degradation data.

High Conductivity Update Linearity MoS2 Memtransistors Array Based on Lithium‐Ion Modulation

AbstractThe typical nonlinear and asymmetric response of synaptic memristors based on 2D layered materials to positive and negative electrical pulses makes the implementation of accurate deep neural networks very challenging. Here, it has been constructed an integrated circuit (IC) process‐compatible array of modulatable neuromorphic devices based on MoS2 three‐terminal memristors and show that the switching characteristics of this memristor become more gradual and symmetric under electrical excitation, yielding up to 98% linearity of conductance updates in optimized simulations of biological synaptic behavior, achieving long‐term plasticity of 150 s. The linear weight update has better predictability and distinguishability. And all‐solid lithium cobaltate is introduced as an independent modulation channel to obtain a modulatable molybdenum disulfide bandgap by controlling the embedding and disembedding of lithium ions, which effectively reduces the contact potential barrier and more flexibly modulates the artificial synaptic properties based on multiple coupling of light, electricity, and ions. This work lays the foundation for the realization of high‐density and energy‐efficient neuromorphic computing chips.

Improve Robustness and Accuracy of Deep Neural Network with L2,∞ Normalization

Improve Robustness and Accuracy of Deep Neural Network with L2,∞ Normalization

SCANN: Synthesis of Compact and Accurate Neural Networks

Deep neural networks (DNNs) have become the driving force behind recent artificial intelligence (AI) research. With the help of a vast amount of training data, neural networks can perform better than traditional machine learning algorithms in many applications. An important problem with implementing a neural network is the design of its architecture. Typically, such an architecture is obtained manually by exploring its hyperparameter space and kept fixed during training. This approach is both time consuming and inefficient. Another issue is that modern neural networks often contain millions of parameters, whereas many applications require small inference models due to imposed resource constraints, such as energy constraints on battery-operated devices. However, efforts to migrate DNNs to such devices typically entail a significant loss of classification accuracy. To address these challenges, we propose a two-step neural network synthesis methodology, called DR+SCANN, that combines two complementary approaches to design compact and accurate DNNs. At the core of our framework is the SCANN methodology that uses three basic architecture-changing operations, namely, connection growth, neuron growth, and connection pruning, to synthesize feedforward architectures with arbitrary structure. These neural networks are not limited to the multilayer perceptron structure. SCANN encapsulates three synthesis methodologies that apply a repeated grow-and-prune paradigm to three architectural starting points. DR+SCANN combines the SCANN methodology with dataset dimensionality reduction to alleviate the curse of dimensionality. We demonstrate the efficacy of SCANN and DR+SCANN on various image and nonimage datasets. We evaluate SCANN on MNIST, CIFAR-10, and ImageNet benchmarks. Without any loss in accuracy, SCANN generates a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$46.3\times $ </tex-math></inline-formula> smaller network than the LeNet-5 Caffe model. We also compare SCANN-synthesized networks with a state-of-the-art fully connected (FC) feedforward model for MNIST, and show <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$20\times $ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$19.9\times $ </tex-math></inline-formula> ) reduction in the number of parameters (floating-point operations) with little drop in accuracy. For the CIFAR-10 dataset, we target AlexNet and VGG-16 baseline architectures. SCANN reduces the number of parameters in AlexNet by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$10.1\times $ </tex-math></inline-formula> without any drop in accuracy. It reduces the number of parameters in the FC layers of VGG-16 to only 2.5k while increasing accuracy by 1.05%. On the ImageNet dataset, for the VGG-16 and MobileNetV2 architectures, we reduce network parameters by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.0\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.3\times $ </tex-math></inline-formula> , respectively, with a similar or improved performance over their respective baselines. We also evaluate the efficacy of using dimensionality reduction alongside SCANN (DR+SCANN) on nine small-to-medium-size datasets. Using this methodology enables us to reduce the number of connections in the network by up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5078.7\times $ </tex-math></inline-formula> (geometric mean: <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$82.1\times $ </tex-math></inline-formula> ), with little to no drop in accuracy. On seven out of nine datasets, we show 0.41%–10.09% accuracy improvements over the FC baseline models. We also show that our synthesis methodology yields neural networks that are much better at navigating the accuracy versus energy efficiency space. This can enable neural network-based inference even on Internet-of-Things sensors.

Training Data Subset Search With Ensemble Active Learning

Deep Neural Networks (DNNs) often rely on vast datasets for training. Given the large size of such datasets, it is conceivable that they contain specific samples that either do not contribute or negatively impact the DNN’s optimization. Modifying the training distribution to exclude such samples could provide an effective solution to improve performance and reduce training time. This paper proposes to scale up ensemble Active Learning (AL) methods to perform acquisition at a large scale (10k to 500k samples at a time). We do this with ensembles of hundreds of models, obtained at a minimal computational cost by reusing intermediate training checkpoints. This allows us to automatically and efficiently perform a training data subset search for large labeled datasets. We observe that our approach obtains favorable subsets of training data, which can be used to train more accurate DNNs than training with the entire dataset. We perform an extensive experimental study of this phenomenon on three image classification benchmarks (CIFAR-10, CIFAR-100, and ImageNet), as well as an internal object detection benchmark for prototyping perception models for autonomous driving. Unlike existing studies, our experiments on object detection are at the scale required for production-ready autonomous driving systems. We provide insights on the impact of different initialization schemes, acquisition functions, and ensemble configurations at this scale. Our results provide strong empirical evidence that optimizing the training data distribution can significantly benefit large-scale vision tasks.

Contrastive label correction for noisy label learning

Noisy label learning is an important process that facilitates the collection of noisy label data for training accurate deep neural networks. The latest label correction methods are effective approaches that focus on identifying label errors in datasets for training. However, existing approaches to identify noise generally rely on model predictions that are not scalable and are error-prone. In this study, a new noisy label learning framework is proposed by leveraging supervised contrastive learning for enhanced representation and improved label correction. Specifically, the proposed framework consists of a class-balanced prototype queue, a prototype-based label correction algorithm, and a supervised representation learning module. The proposed framework produces closely aligned representations and model predictions for instances from the same classes and facilitates label correction by aggregating noisy and prototype labels. Furthermore, a theoretical analysis of the proposed framework is provided from the perspective of an expectation–maximization (EM) algorithm. To demonstrate the efficacy of the proposed framework, experiments were performed on synthetic datasets with various noise patterns and levels. The experimental results demonstrate that the proposed framework achieves superior classification accuracy compared with other label correction frameworks.

TALIPOT: Energy-Efficient DNN Booster Employing Hybrid Bit Parallel-Serial Processing in MSB-First Fashion

We propose a novel hybrid bit parallel-serial processing technique called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TALIPOT</i> in order to reduce energy consumption of deep neural networks (DNNs). <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TALIPOT</i> works as computation booster and has inborn ability of quality adjusting tradeoff between accuracy and energy consumption of DNNs. The core principal of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TALIPOT</i> is keeping the serial number of bits same in the entire computing process, so the energy is consumed effectively without extra waiting times between inputs and outputs of the computing blocks (adders, multipliers, and activation blocks). To achieve this, we implement activation rounding which scales down the accumulation of parallel bits at the output of the hidden layers of DNN. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TALIPOT</i> utilizes most significant bit first (MSB-first) fashion, so we obtain the most valuable bit information first, between the layers of DNN, which ensures the activation rounding process done accurately and efficiently. Thanks to this method, we optimize operating accuracy/energy point by cutting off bits at the output whenever we obtain the desired accuracy. Simulations using the MNIST and CIFAR-10 datasets show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TALIPOT</i> outperforms the state-of-the-art computation techniques in terms of energy consumption. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TALIPOT</i> performs the MNIST classification with the energy efficiency of 25.3 TOPS/W and the accuracy of 98.2% in ASIC environment using 40 nm CMOS process.

Evaluation of LAI Estimation of Mangrove Communities Using DLR and ELR Algorithms With UAV, Hyperspectral, and SAR Images

The high-precision estimation of mangrove leaf area index (LAI) using a deep learning regression algorithm (DLR) always requires a large amount of training sample data. However, it is difficult for LAI field measurements to collect a sufficient amount of sample data in mangrove wetlands. To tackle this challenge, this paper proposed an approach for expanding training samples and quantitatively evaluated the performance of estimating LAI for mangrove communities using Deep Neural Networks (DNN) and Transformer algorithms. This study also explored the effects of unmanned aerial vehicle (UAV) and Sentinel-2A multispectral, orbital hyper spectral (OHS), and GF-3 SAR images on LAI estimation of different mangrove communities. Finally, this paper evaluated the LAI estimation ability of mangrove communities using ensemble learning regression (ELR) and DLR algorithms. The results showed that: (1) the UAV images achieved the better LAI estimation of different mangrove communities (R2 = 0.5974–0.6186), and GF-3 SAR images were better for LAI estimation of Avicennia marina with high coverage (R2 = 0.567). The optimal spectral range for estimating LAI for mangroves in the optical images was between 650–680 nm. (2) The ELR model outperformed single base model, and produced the high-accuracy LAI estimation (R2 = 0.5266–0.713) for different mangrove communities. (3) The average accuracy (R2) of the ELR model was higher by 0.0019–0.149 than the DLR models, which demonstrated that the ELR model had a better capability (R2 = 0.5865–0.6416) in LAI estimation. The Transformer-based LAI estimation of A. marina (R2 = 0.6355) was better than the DNN model, while the DNN model produced higher accuracy for Kandelia candel (KC) (R2 = 0.5577). (4) With the increase in the expansion ratio of the training sample (10–50%), the LAI estimation accuracy (R2) of DNN and Transformer models for different mangrove communities increased by 0.1166–0.2037 and 0.1037–0.1644, respectively. Under the same estimation accuracy, the sample enhancement method in this paper could reduce the number of filed measurements by 20–40%.

Amplification trojan network: Attack deep neural networks by amplifying their inherent weakness

Recent works found that deep neural networks (DNNs) can be fooled by adversarial examples, which are crafted by adding adversarial noise on clean inputs. The accuracy of DNNs on adversarial examples will decrease as the magnitude of the adversarial noise increase. In this study, we show that DNNs can be also fooled when the noise is very small under certain circumstances. This new type of attack is called Amplification Trojan Attack (ATAttack). Specifically, we use a trojan network to transform the inputs before sending them to the target DNN. This trojan network serves as an amplifier to amplify the inherent weakness of the target DNN. The target DNN, which is infected by the trojan network, performs normally on clean data while being more vulnerable to adversarial examples. Since it only transforms the inputs, the trojan network can hide in DNN-based pipelines, e.g. by infecting the pre-processing procedure of the inputs before sending them to the DNNs. This new type of threat should be considered in developing safe DNNs.

On-line transfer learning for multi-fidelity data fusion with ensemble of deep neural networks

Deep Neural Network (DNN) is widely used in engineering applications for its ability to handle problems with almost any nonlinearities. However, it is generally difficult to obtain sufficient high-fidelity (HF) sample points for expensive optimization tasks, which may affect the generalization performance of DNN and result in inaccurate predictions. To solve this problem and improve the prediction accuracy of DNN, this paper proposes an on-line transfer learning based multi-fidelity data fusion (OTL-MFDF) method including two parts. In the first part, the ensemble of DNNs is established. Firstly, a large number of low-fidelity sample points and a few HF sample points are generated, which are used as the source dataset and target dataset, respectively. Then, the Bayesian Optimization (BO) is utilized to obtain several groups of hyperparameters, based on which DNNs are pre-trained using the source dataset. Next, these pre-trained DNNs are re-trained by fine-tuning on the target dataset, and the ensemble of DNNs is established by assigning different weights to each pre-trained DNN. In the second part, the on-line learning system is developed for adaptive updating of the ensemble of DNNs. To evaluate the uncertainty error of the predicted values of DNN and determine the location of the updated HF sample point, the query-by-committee strategy based on the ensemble of DNNs is developed. The Covariance Matrix Adaptation Evolutionary Strategies is employed as the optimizer to find out the location where the maximal disagreement is achieved by the ensemble of DNNs. The design space is partitioned by the Voronoi diagram method, and then the selected point is moved to its nearest Voronoi cell boundary to avoid clustering between the updated point and the existing sample points. Three different types of test problems and an engineering example are adopted to illustrate the effectiveness of the OTL-MFDF method. Results verify the outstanding efficiency, global prediction accuracy and applicability of the OTL-MFDF method.

A Universal RRAM-Based DNN Accelerator With Programmable Crossbars Beyond MVM Operator

Resistive-RAM (RRAM)-based deep neural network (DNN) accelerator has shown a great potential as it is good at the matrix–vector multiplication (MVM) operator. However, it does not benefit non-MVM operators, such as transcendental activation or elementwise operations, which often require customized CMOS circuits in conventional DNN accelerator designs. In this article, we propose a new RRAM-based DNN inference accelerator, which leverages the proposed RRAM-CORDIC and RRAM-MLP algorithms to make the transcendental and elementwise operators calculable in the RRAM crossbar just like MVM. Both algorithms can exploit the higher multiply-and-accumulation (MAC) parallelism that is traditionally expensive in CMOS but now efficient in the RRAM crossbar. Then, we further propose an intercrossbar pipelining scheme, which can balance the number of crossbars for MVM and non-MVM operations and orchestrate them in pursuing higher DNN computing throughput. The experimental results show that both algorithms can sustain a high arithmetic accuracy and deliver less than 1% DNN accuracy loss on typical inference workloads. The elimination of expensive CMOS circuits, in turn, can trade more crossbar resources in the same area to speed up the performance by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.16\times $ </tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.33\times $ </tex-math></inline-formula> . With the extended operators, the RRAM-based DNN accelerator can switch crossbar functions at will, and apply for a diverse of DNN models in a unified in-memory accelerator architecture.

Data-informed deep optimization.

Motivated by the impressive success of deep learning in a wide range of scientific and industrial applications, we explore in this work the application of deep learning into a specific class of optimization problems lacking explicit formulas for both objective function and constraints. Such optimization problems exist in many design problems, e.g., rotor profile design, in which objective and constraint values are available only through experiment or simulation. They are especially challenging when design parameters are high-dimensional due to the curse of dimensionality. In this work, we propose a data-informed deep optimization (DiDo) approach emphasizing on the adaptive fitting of the the feasible region as follows. First, we propose a deep neural network (DNN) based adaptive fitting approach to learn an accurate DNN classifier of the feasible region. Second, we use the DNN classifier to efficiently sample feasible points and train a DNN surrogate of the objective function. Finally, we find optimal points of the DNN surrogate optimization problem by gradient descent. To demonstrate the effectiveness of our DiDo approach, we consider a practical design case in industry, in which our approach yields good solutions using limited size of training data. We further use a 100-dimension toy example to show the effectiveness of our approach for higher dimensional problems. Our results indicate that, by properly dealing with the difficulty in fitting the feasible region, a DNN-based method like our DiDo approach is flexible and promising for solving high-dimensional design problems with implicit objective and constraints.

Balance label correction using contrastive loss

The training of accurate deep neural networks using noisy labels is attracting tremendous attention as of late. Several studies have been devoted to noisy-label-based learning using label correction methods. The latest label correction methods provide a theoretical guarantee and avoid the accumulation of irreversible errors. However, these methods often encounter class-biased problems: some classes (“easy”) are easier to correct than others (“hard”). It has been observed that the noisy samples with a lower class confidence are harder to correct, and the class confidence is measured by the average of the model predictions, resulting from the correctly predicted samples across all classes. To address this class-biased problem, we provide a theoretical analysis of the correction procedure by investigating model predictions. Furthermore, we propose a new balance label correction method by employing the statistics of contrastive loss as a dynamic balancing factor that focuses more on the lower confidence hard classes. To demonstrate the efficacy of the proposed method, we performed experiments on synthetic and real-world datasets with various noise patterns and levels. The experimental results demonstrated that the proposed method achieved a superior classification accuracy compared with other state-of-the-art noisy label learning frameworks.

D-NAT: Data-Driven Non-Ideality Aware Training Framework for Fabricated Computing-In-Memory Macros

To enable energy-efficient computation for deep neural networks (DNNs) at edge, computing-in-memory (CIM) is proposed to reduce the energy costs during intense off-chip memory access. However, CIM is prone to multiply-accumulate (MAC) errors due to non-idealities of memory crossbars and peripheral circuits, which severely degrade the accuracy of DNNs. In this work, we propose a Data-Driven Non-ideality Aware Training (D-NAT) framework to compensate for the accuracy degradation. The proposed D-NAT framework has the following contributions: 1) We measured a fabricated SRAM-based CIM macro to obtain a data-driven MAC error model (D-MAC-EM). Based on the derived D-MAC-EM, we analyze the impact of the non-idealities on DNN’s accuracy. 2) To make DNNs robust to the non-idealities of CIM macros, we incorporate the measured D-MAC-EM into DNN’s training procedure. Moreover, we propose a statistical training mechanism to better estimate the gradients of the discrete D-MAC-EM. 3) We investigate trade-offs between quantization range and quantization errors. To mitigate the quantization errors in activations, we introduce extended PACT (E-PACT) that adaptively learns the upper and lower bounds of input activations for each layer. Simulation results show that our proposed D-NAT improves the accuracy of ResNet20, VGG8, ResNet34, and VGG16 by 78.98%, 71.8%, 72.04%, and 57.85%, respectively, which reaches the ideal upper bound of the quantized model. Lastly, the D-NAT framework is validated on an FPGA platform with the fabricated SRAM-based CIM macro chip. Based on the measurement results, D-NAT successfully recovers the accuracy under non-idealities of a real SRAM-based CIM macro.