Large-scale Deep Neural Networks Research Articles

Towards Verifying the Geometric Robustness of Large-Scale Neural Networks

Deep neural networks (DNNs) are known to be vulnerable to adversarial geometric transformation. This paper aims to verify the robustness of large-scale DNNs against the combination of multiple geometric transformations with a provable guarantee. Given a set of transformations (e.g., rotation, scaling, etc.), we develop GeoRobust, a black-box robustness analyser built upon a novel global optimisation strategy, for locating the worst-case combination of transformations that affect and even alter a network's output. GeoRobust can provide provable guarantees on finding the worst-case combination based on recent advances in Lipschitzian theory. Due to its black-box nature, GeoRobust can be deployed on large-scale DNNs regardless of their architectures, activation functions, and the number of neurons. In practice, GeoRobust can locate the worst-case geometric transformation with high precision for the ResNet50 model on ImageNet in a few seconds on average. We examined 18 ImageNet classifiers, including the ResNet family and vision transformers, and found a positive correlation between the geometric robustness of the networks and the parameter numbers. We also observe that increasing the depth of DNN is more beneficial than increasing its width in terms of improving its geometric robustness. Our tool GeoRobust is available at https://github.com/TrustAI/GeoRobust.

PDAS: Improving network pruning based on Progressive Differentiable Architecture Search for DNNs

Network pruning is an efficient approach to adapting large-scale deep neural networks (DNNs) to resource-constrained systems; the networks are pruned using the predefined pruning criteria or a flexible network structure is explored with the help of neural architecture search, (NAS). However, the former crucially relies on the human expert knowledge, while the latter usually requires one to make many simplifications to ensure the efficiency of the search, resulting in limited performance. This paper presents a new pruning approach called Progressive Differentiable Architecture Search (PDAS) that realizes a better balance between computation efficiency and model performance. First, a joint search-update scheme for search optimization is presented; it constantly refines the candidate number of channels in each layer by performing differentiable searching and evolutionary updating alternately. The latter can provide new high-probability candidates continuously to avoid local minimum point. Second, a two-stage constrained progressive search strategy is presented for some complex nonlinear networks (such as ResNet) that are more difficult to prune for existing approaches; it effectively avoids the over-fitting problem caused by the excessive search space and largely reduces the consumption of 1x1 convolution in the skip connections of the residual blocks with little accuracy loss. Extensive experiments on some representative datasets (such as CIFAR-10, CIFAR-100, and ImageNet) approve the superior performance of PDAS compared to most existing network pruning algorithms available. Notably, compared to the state-of-the-art LFPC, PDAS can even prune about 8% more FLOPs on ResNet-110 (on CIFAR-10) and ResNet-50 (on ImageNet), respectively, while coming with almost identical and ignorable accuracy losses.

Low-Latency Federated Learning With DNN Partition in Distributed Industrial IoT Networks

Federated Learning (FL) empowers Industrial Internet of Things (IIoT) with distributed intelligence of industrial automation thanks to its capability of distributed machine learning without any raw data exchange. However, it is rather challenging for lightweight IIoT devices to perform computation-intensive local model training over large-scale deep neural networks (DNNs). Driven by this issue, we develop a communication-computation efficient FL framework for resource-limited IIoT networks that integrates DNN partition technique into the standard FL mechanism, wherein IIoT devices perform local model training over the bottom layers of the objective DNN, and offload the top layers to the edge gateway side. Considering imbalanced data distribution, we derive the device-specific participation rate to involve the devices with better data distribution in more communication rounds. Upon deriving the device-specific participation rate, we propose to minimize the training delay under the constraints of device-specific participation rate, energy consumption and memory usage. To this end, we formulate a joint optimization problem of device scheduling and resource allocation (i.e. DNN partition point, channel assignment, transmit power, and computation frequency), and solve the long-term min-max mixed integer non-linear programming based on the Lyapunov technique. In particular, the proposed dynamic device scheduling and resource allocation (DDSRA) algorithm can achieve a trade-off to balance the training delay minimization and FL performance. We also provide the FL convergence bound for the DDSRA algorithm with both convex and non-convex settings. Experimental results demonstrate the derived device-specific participation rate in terms of feasibility, and show that the DDSRA algorithm outperforms baselines in terms of test accuracy and convergence time.

Design of a Quantization-Based DNN Delta Compression Framework for Model Snapshots and Federated Learning

Deep neural networks (DNNs) have achieved remarkable success in many fields. However, large-scale DNNs also bring storage costs when storing snapshots for preventing clusters’ frequent failures or incur significant communication overheads when transmitting DNNs in the Federated Learning (FL). Recently, several approaches, such as Delta-DNN and LC-Checkpoint, aim to reduce the size of DNNs’ snapshot storage by compressing the difference between two neighboring versions of the DNNs (a.k.a., delta). However, we observe that existing approaches, applying traditional global lossy quantization techniques in DNN's delta compression, can not fully exploit the data similarity since the parameters’ value ranges vary among layers. To fully explore the similarity of the delta model and improve the compression ratio, we propose a quantization-based local-sensitive delta compression approach, named QD-Compressor, by developing a layer-based local-sensitive quantization scheme and error feedback mechanism. Specifically, the quantizers and number of quantization bits are adaptive among layers based on the value distribution and weighted entropy of the delta's parameters. To avoid quantization error degrading the performance of the restored model, an alternative error feedback mechanism is designed to dynamically correct the quantization error during the training process. Experiments on multiple popular DNNs and datasets show that QD-Compressor obtains a higher 7×-40× compression ratio in the model snapshot compression scenario than the state-of-the-art approaches. Additionally, QD-Compressor achieves an 11×-15× compression ratio to the residual model of the Federated Learning compression scenario.

Energy-Efficient DNN Partitioning and Offloading for Task Completion Rate Maximization in Multiuser Edge Intelligence

Deep Neural Network (DNN) has become an essential technology for edge intelligence. Due to significant resource and energy requirements for large-scale DNNs’ inference, executing them directly on energy-constrained Internet of Things (IoT) devices is impractical. DNN partitioning provides a feasible solution for this problem by offloading some DNN layers to execute on the edge server. However, the resources of edge servers are also typically limited. An energy-constrained and resource-constrained optimization problem is generated in such a realistic environment. Motivated by this, we investigate an optimization problem of DNN partitioning and offloading in a multiuser resource-constrained environment, which is considered an intractable Mixed-Integer Nonlinear Problem (MINLP). We decompose the problem into two subproblems and propose an Energy-Efficient DNN Partitioning and Offloading (EEDPO) strategy to solve it in polynomial time based on the minimum cut/maximum flow theorem and dynamic programming. Finally, we test the impact of energy constraint, DNN type, and device number on the performance of EEDPO. Simulation results on realistic DNN models demonstrate that the proposed strategy can significantly improve the DNN inference task completion rate compared to other methods.

A bidirectional DNN partition mechanism for efficient pipeline parallel training in cloud

Recently, deep neural networks (DNNs) have shown great promise in many fields while their parameter sizes are rapidly expanding. To break through the computation and memory limitation of a single machine, pipeline model parallelism is proposed for large-scale DNN training by fully utilizing the computation and storage power of the distributed cluster. Cloud data centers can also provide sufficient computing, storage and bandwidth resources. However, most existing approaches apply layer-wise partitioning, which is difficult to obtain an even model partition result because of the large computational overhead discrepancy between DNN layers, resulting in degraded efficiency. To tackle this issue, we propose “Bi-Partition”, a novel partitioning method based on bidirectional partitioning for forward propagation (FP) and backward propagation (BP), which improves the efficiency of the pipeline model parallelism system. By deliberated designing distinct cut positions for FP and BP of DNN training, workers in the pipeline get nearly equal computational loads, and the balanced pipeline fully utilizes the computing resources. Experiments on various DNN models and datasets validate the efficiency of our mechanism, e.g., the training efficiency achieving up to 1.9times faster than the state-of-the-art method PipeDream.

Deep Neural Network Training With Distributed K-FAC

Scaling deep neural network training to more processors and larger batch sizes is key to reducing end-to-end training time; yet, maintaining comparable convergence and hardware utilization at larger scales is challenging. Increases in training scales have enabled natural gradient optimization methods as a reasonable alternative to stochastic gradient descent and variants thereof. Kronecker-factored Approximate Curvature (K-FAC), a natural gradient method, preconditions gradients with an efficient approximation of the Fisher Information Matrix to improve per-iteration progress when optimizing an objective function. Here we propose a scalable K-FAC algorithm and investigate K-FAC’s applicability in large-scale deep neural network training. Specifically, we explore layer-wise distribution strategies, inverse-free second-order gradient evaluation, and dynamic K-FAC update decoupling, with the goal of preserving convergence while minimizing training time. We evaluate the convergence and scaling properties of our K-FAC gradient preconditioner, for image classification, object detection, and language modeling applications. In all applications, our implementation converges to baseline performance targets in 9–25% less time than the standard first-order optimizers on GPU clusters across a variety of scales.

GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices Based on Fine-Grained Structured Weight Sparsity.

It is appealing but challenging to achieve real-time deep neural network (DNN) inference on mobile devices, because even the powerful modern mobile devices are considered as "resource-constrained" when executing large-scale DNNs. It necessitates the sparse model inference via weight pruning, i.e., DNN weight sparsity, and it is desirable to design a new DNN weight sparsity scheme that can facilitate real-time inference on mobile devices while preserving a high sparse model accuracy. This paper designs a novel mobile inference acceleration framework GRIM that is General to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) and that achieves Real-time execution and high accuracy, leveraging fine-grained structured sparse model Inference and compiler optimizations for Mobiles. We start by proposing a new fine-grained structured sparsity scheme through the Block-based Column-Row (BCR) pruning. Based on this new fine-grained structured sparsity, our GRIM framework consists of two parts: (a) the compiler optimization and code generation for real-time mobile inference; and (b) the BCR pruning optimizations for determining pruning hyperparameters and performing weight pruning. We compare GRIM with Alibaba MNN, TVM, TensorFlow-Lite, a sparse implementation based on CSR, PatDNN, and ESE (a representative FPGA inference acceleration framework for RNNs), and achieve up to 14.08× speedup.

Towards efficient full 8-bit integer DNN online training on resource-limited devices without batch normalization

Huge computational costs brought by convolution and batch normalization (BN) have caused great challenges for the online training and corresponding applications of deep neural networks (DNNs), especially in resource-limited devices. Existing works only focus on the convolution or BN acceleration and no solution can alleviate both problems with satisfactory performance. Online training has gradually become a trend in resource-limited devices like mobile phones while there is still no complete technical scheme with acceptable model performance, processing speed, and computational cost. In this research, an efficient online-training quantization framework termed EOQ for abbreviation is proposed by combining Fixup initialization and a novel quantization scheme for the online training in resource-limited devices. Based on the proposed framework, we have successfully realized full 8-bit integer network training and removed BN in large-scale DNNs. Especially, weight updates are quantized to 8-bit integers for the first time. Theoretical analyses of EOQ utilizing Fixup initialization for removing BN have been further given using a novel Block Dynamical Isometry theory with weaker assumptions. Benefiting from rational quantization strategies and the absence of BN, the full 8-bit networks based on EOQ can achieve state-of-the-art accuracy and immense advantages in computational cost and processing speed. Experiments show that the 8-bit EOQ networks achieve 2.78%, 3.85%, and 4.31% accuracy improvements compared with existing full 8-bit integer networks in ResNet-18/34/50. At the same time, the 8-bit EOQ networks can improve the computing speed greatly, and decrease the power consumption and circuit area by about an order of magnitude compared with 32-bit floating-point vanilla networks. In addition to the huge advantages brought by quantization in convolution operations, 8-bit networks based on EOQ without BN can realize >66× lower in power, >13× faster in the processing speed compared with the traditional 32-bit floating-point BN in the inference process. What’s more, the design of deep learning chips can be profoundly simplified in the absence of unfriendly square root operations in BN. Beyond this, EOQ has been evidenced to be more advantageous in small-batch online training with fewer batch samples. In summary, the EOQ framework is specially designed for reducing the high cost of convolution and BN in network training, demonstrating a broad application prospect of online training in resource-limited devices.

A neural network compression method based on knowledge-distillation and parameter quantization for the bearing fault diagnosis

Condition monitoring and fault diagnosis have been critical for the optimal scheduling of machines, improving the system reliability and the reducing maintenance cost. In recent years, various of methods based on the deep learning method have made the great progress in the field of the mechanical fault diagnosis. However, there is a conflict between the massive parameters of the fault diagnosis networks and the limited computing resource of the embedded platforms. It is difficult to deploy the trained network on the small scale embedded platforms (like field programmable gate array (FPGA)) in the actual industrial situations. This seriously hinders the practical process of the intelligent fault diagnosis method. To address this problem, a new neural network compression method based on knowledge-distillation (K-D) and parameter quantization is proposed in this paper. In the proposed method, a large scale deep neural network with multiple convolutional layers and fully-connected layers is designed and trained as the teacher network. Then a small scale network with just one convolutional layer and one fully-connected layer is designed as the student network. When training the student network, the K-D process is conducted to improve the accuracy of the student network. After the training process, the parameter quantization is conducted to further compress the scale of the student network. Experimental results on the field programmable gate array (FPGA) are presented to demonstrate the effectiveness of the proposed method. The results show that the proposed method can greatly compress the scales of the fault diagnosis networks for over 10 times at the cost of the minimal loss of the accuracy.

StructADMM: Achieving Ultrahigh Efficiency in Structured Pruning for DNNs

Weight pruning methods of deep neural networks (DNNs) have been demonstrated to achieve a good model pruning rate without loss of accuracy, thereby alleviating the significant computation/storage requirements of large-scale DNNs. Structured weight pruning methods have been proposed to overcome the limitation of irregular network structure and demonstrated actual GPU acceleration. However, in prior work, the pruning rate (degree of sparsity) and GPU acceleration are limited (to less than 50%) when accuracy needs to be maintained. In this work, we overcome these limitations by proposing a unified, systematic framework of structured weight pruning for DNNs. It is a framework that can be used to induce different types of structured sparsity, such as filterwise, channelwise, and shapewise sparsity, as well as nonstructured sparsity. The proposed framework incorporates stochastic gradient descent (SGD; or ADAM) with alternating direction method of multipliers (ADMM) and can be understood as a dynamic regularization method in which the regularization target is analytically updated in each iteration. Leveraging special characteristics of ADMM, we further propose a progressive, multistep weight pruning framework and a network purification and unused path removal procedure, in order to achieve higher pruning rate without accuracy loss. Without loss of accuracy on the AlexNet model, we achieve 2.58× and 3.65× average measured speedup on two GPUs, clearly outperforming the prior work. The average speedups reach 3.15× and 8.52× when allowing a moderate accuracy loss of 2%. In this case, the model compression for convolutional layers is 15.0× , corresponding to 11.93× measured CPU speedup. As another example, for the ResNet-18 model on the CIFAR-10 data set, we achieve an unprecedented 54.2× structured pruning rate on CONV layers. This is 32× higher pruning rate compared with recent work and can further translate into 7.6× inference time speedup on the Adreno 640 mobile GPU compared with the original, unpruned DNN model. We share our codes and models at the link http://bit.ly/2M0V7DO.

Accuracy and Resiliency of Analog Compute-in-Memory Inference Engines

Recently, analog compute-in-memory (CIM) architectures based on emerging analog non-volatile memory (NVM) technologies have been explored for deep neural networks (DNNs) to improve scalability, speed, and energy efficiency. Such architectures, however, leverage charge conservation, an operation with infinite resolution, and thus are susceptible to errors. Thus, the inherent stochasticity in any analog NVM used to execute DNNs, will compromise performance. Several reports have demonstrated the use of analog NVM for CIM in a limited scale. It is unclear whether the uncertainties in computations will prohibit large-scale DNNs. To explore this critical issue of scalability, this article first presents a simulation framework to evaluate the feasibility of large-scale DNNs based on CIM architecture and analog NVM. Simulation results show that DNNs trained for high-precision digital computing engines are not resilient against the uncertainty of the analog NVM devices. To avoid such catastrophic failures, this article introduces the analog bi-scale representation for the DNN, and the Hessian-aware Stochastic Gradient Descent training algorithm to enhance the inference accuracy of trained DNNs. As a result of such enhancements, DNNs such as Wide ResNets for CIFAR-100 image recognition problem are demonstrated to have significant performance improvements in accuracy without adding cost to the inference hardware .

Energy-Efficient Offloading for DNN-Based Smart IoT Systems in Cloud-Edge Environments

Deep Neural Networks (DNNs) have become an essential and important supporting technology for smart Internet-of-Things (IoT) systems. Due to the high computational costs of large-scale DNNs, it might be infeasible to directly deploy them in energy-constrained IoT devices. Through offloading computation-intensive tasks to the cloud or edges, the computation offloading technology offers a feasible solution to execute DNNs. However, energy-efficient offloading for DNN based smart IoT systems with deadline constraints in the cloud-edge environments is still an open challenge. To address this challenge, we first design a new system energy consumption model, which takes into account the runtime, switching, and computing energy consumption of all participating servers (from both the cloud and edge) and IoT devices. Next, a novel energy-efficient offloading strategy based on a Self-adaptive Particle Swarm Optimization algorithm using the Genetic Algorithm operators (SPSO-GA) is proposed. This new strategy can efficiently make offloading decisions for DNN layers with layer partition operations, which can lessen the encoding dimension and improve the execution time of SPSO-GA. Simulation results demonstrate that the proposed strategy can significantly reduce energy consumption compared to other classic methods.

DDPQN: An Efficient DNN Offloading Strategy in Local-Edge-Cloud Collaborative Environments

With the rapid development of the Internet of Things (IoT) and communication technology, Deep Neural Network (DNN) applications like computer vision, can now be widely used in IoT devices. However, due to the insufficient memory, low computing capacity, and low battery capacity of IoT devices, it is difficult to support the high-efficiency DNN inference and meet users’ requirements for Quality of Service (QoS). Worse still, offloading failures may occur during the massive DNN data transmission due to the intermittent wireless connectivity between IoT devices and the cloud. In order to fill this gap, we consider the partitioning and offloading of the DNN model, and design a novel optimization method for parallel offloading of large-scale DNN models in a local-edge-cloud collaborative environment with limited resources. Combined with the coupling coordination degree and node balance degree, an improved Double Dueling Prioritized deep Q-Network (DDPQN) algorithm is proposed to obtain the DNN offloading strategy. Compared with existing algorithms, the DDPQN algorithm can obtain an efficient DNN offloading strategy with low delay, low energy consumption, and low cost under the premise of ensuring “delay-energy-cost” coordination and reasonable allocation of computing resources in a local-edge-cloud collaborative environment.

EosDNN: An Efficient Offloading Scheme for DNN Inference Acceleration in Local-Edge-Cloud Collaborative Environments

With the popularity of mobile devices, intelligent applications, e.g., face recognition, intelligent voice assistant, and gesture recognition, have been widely used in our daily lives. However, due to the lack of computing capacities, it is difficult for mobile devices to support complex Deep Neural Network (DNN) inference. To alleviate the pressure on these devices, traditional methods usually upload part of the DNN model to a cloud server and perform a DNN query after uploading an entire DNN model. To achieve real-time DNN query, we consider the collaboration between local, edge and cloud, and perform DNN query when uploading DNN partitions. In this paper, we propose an Efficient offloading scheme for DNN Inference Acceleration (EosDNN) in a local-edge-cloud collaborative environment, where the DNN inference acceleration is mainly embodied in the optimization of migration delay and realization of real-time DNN query. EosDNN comprehensively considers the migration plan and uploading plan, where for the former, a Particle Swarm Optimization with Genetic Algorithm (PSO-GA) is applied to obtain the distribution of DNN layers under the server with the lowest migration delay, and for the latter, a Layer Merge Uploading Algorithm (LMU) is proposed to obtain DNN partitions and their upload order with efficient DNN query performance. Experimental results demonstrate that EosDNN can be applied to large-scale DNN model migration, which can achieve an ideal migration delay and obtain a more fine-grained DNN partition uploading plan, thereby optimizing DNN query performance.

Homecare-Oriented ECG Diagnosis With Large-Scale Deep Neural Network for Continuous Monitoring on Embedded Devices

The accurate electrocardiogram (ECG) interpretation is important for several potentially life-threatening cardiac diseases. Recently developed deep learning methods show their ability to distinguish some severe heart diseases. However, since deep neural network requires a high cost on memory consumption and computation, implementation scenarios of these interpretation methods are constrained to nonportable devices. Few commercial portable devices only have heartbeat detection ability, and therefore, only a few simple cardiac diseases can be diagnosed. In this article, to achieve diagnosing a wide range of cardiac diseases and continuous monitoring, a homecare-oriented ECG diagnosis platform is designed based on a large-scale multilabel deep conventional neural network. The accuracy of the proposed neural network model is guaranteed by our constructed large-scale ECG dataset, which is comprised of 206 468 standard 12-lead ECG recordings from 89 488 patients, with respect to 26 types of most common heart rhythms and conduction abnormalities. Meanwhile, targeting lightweight homecare or wearable applications, algorithm-hardware co-optimization is conducted to accelerate the model computation on an embedded platform with field-programmable gate array (FPGA) for continuous monitoring. Channel-level pruning and parameters quantization strategy are employed to optimize the network, and a reconfigurable accelerator hardware architecture is designed to accelerate the convolution computation on FPGA. The final quantified model achieved a promising <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F_{1}$ </tex-math></inline-formula> score of 0.913% and 86.7% exact match ratio, in which parameters and floating-point operations per second (FLOPs) are significantly penalized compared to the original large-scale model. Real-time analysis is performed. Specifically, the average processing time for each ECG record is 2.895 s, and it can be applied to homecare or portable ECG diagnosis devices for continuous monitoring.

SHAT: A Novel Asynchronous Training Algorithm That Provides Fast Model Convergence in Distributed Deep Learning

The recent unprecedented success of deep learning (DL) in various fields is underlied by its use of large-scale data and models. Training a large-scale deep neural network (DNN) model with large-scale data, however, is time-consuming. To speed up the training of massive DNN models, data-parallel distributed training based on the parameter server (PS) has been widely applied. In general, a synchronous PS-based training suffers from the synchronization overhead, especially in heterogeneous environments. To reduce the synchronization overhead, asynchronous PS-based training employs the asynchronous communication between PS and workers so that PS processes the request of each worker independently without waiting. Despite the performance improvement of asynchronous training, however, it inevitably incurs the difference among the local models of workers, where such a difference among workers may cause slower model convergence. Fro addressing this problem, in this work, we propose a novel asynchronous PS-based training algorithm, SHAT that considers (1) the scale of distributed training and (2) the heterogeneity among workers for successfully reducing the difference among the local models of workers. The extensive empirical evaluation demonstrates that (1) the model trained by SHAT converges to the higher accuracy up to 5.22% than state-of-the-art algorithms, and (2) the model convergence of SHAT is robust under various heterogeneous environments.

Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks* *This article is an updated version of: Karakida R and Osawa K 2020 Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks Advances in Neural Information Processing Systems vol 33 ed H Larochelle, M Ranzato, R Hadsell, M F

Natural gradient descent (NGD) helps to accelerate the convergence of gradient descent dynamics, but it requires approximations in large-scale deep neural networks because of its high computational cost. Empirical studies have confirmed that some NGD methods with approximate Fisher information converge sufficiently fast in practice. Nevertheless, it remains unclear from the theoretical perspective why and under what conditions such heuristic approximations work well. In this work, we reveal that, under specific conditions, NGD with approximate Fisher information achieves the same fast convergence to global minima as exact NGD. We consider deep neural networks in the infinite-width limit, and analyze the asymptotic training dynamics of NGD in function space via the neural tangent kernel. In the function space, the training dynamics with the approximate Fisher information are identical to those with the exact Fisher information, and they converge quickly. The fast convergence holds in layer-wise approximations; for instance, in block diagonal approximation where each block corresponds to a layer as well as in block tri-diagonal and K-FAC approximations. We also find that a unit-wise approximation achieves the same fast convergence under some assumptions. All of these different approximations have an isotropic gradient in the function space, and this plays a fundamental role in achieving the same convergence properties in training. Thus, the current study gives a novel and unified theoretical foundation with which to understand NGD methods in deep learning.

A Lego-Based Neural Network Design Methodology With Flexible NoC

Deep Neural Networks (DNNs) have shown superiority in solving the problems of classification and recognition in recent years. However, DNN hardware implementation is challenging due to the high computational complexity and diverse dataflow in different DNN models. To mitigate this design challenge, a large body of research has focused on accelerating specific DNN models or layers and proposed dedicated designs. However, dedicated designs for specific DNN models or layers limit the design flexibility. In this work, we take advantage of the similarity among different DNN models and propose a novel Lego-based Deep Neural Network on a Chip (DNNoC) design methodology. We work on common neural computing units (e.g., multiply-accumulation and pooling) and create some neuron computing units called NeuLego processing elements (NeuLegoPEs). These NeuLegoPEs are then interconnected using a flexible Network-on-Chip (NoC), allowing to construct different DNN models. To support large-scale DNN models, we enhance the reusability of each NeuLegoPE by proposing a Lego placement method. The proposed design methodology allows leveraging different DNN model implementations, helping to reduce implementation cost and time-to-market. Compared with the conventional approaches, the proposed approach can improve the average throughput by 2,802% for given DNN models. Besides, the corresponding hardware is implemented to validate the proposed design methodology, showing on average 12,523% hardware efficiency improvement by considering the throughput and area overhead simultaneously.

COMET

Deep neural networks (DNNs) are becoming increasingly deeper, wider, and non-linear due to the growing demands on prediction accuracy and analysis quality. Training wide and deep neural networks require large amounts of storage resources such as memory because the intermediate activation data must be saved in the memory during forward propagation and then restored for backward propagation. However, state-of-the-art accelerators such as GPUs are only equipped with very limited memory capacities due to hardware design constraints, which significantly limits the maximum batch size and hence performance speedup when training large-scale DNNs. Traditional memory saving techniques either suffer from performance overhead or are constrained by limited interconnect bandwidth or specific interconnect technology. In this paper, we propose a novel memory-efficient CNN training framework (called COMET) that leverages error-bounded lossy compression to significantly reduce the memory requirement for training in order to allow training larger models or to accelerate training. Our framework purposely adopts error-bounded lossy compression with a strict error-controlling mechanism. Specifically, we perform a theoretical analysis on the compression error propagation from the altered activation data to the gradients, and empirically investigate the impact of altered gradients over the training process. Based on these analyses, we optimize the error-bounded lossy compression and propose an adaptive error-bound control scheme for activation data compression. Experiments demonstrate that our proposed framework can significantly reduce the training memory consumption by up to 13.5X over the baseline training and 1.8X over another state-of-the-art compression-based framework, respectively, with little or no accuracy loss.