Binary Deep Neural Network Research Articles

Intelligent Fault Diagnosis for Large-Scale Rotating Machines Using Binarized Deep Neural Networks and Random Forests

Recently, deep neural network (DNN) models work incredibly well, and edge computing has achieved great success in real-world scenarios, such as fault diagnosis for large-scale rotational machinery. However, DNN training takes a long time due to its complex calculation, which makes it difficult to optimize and retrain models. To address such an issue, this work proposes a novel fault diagnosis model by combining binarized DNNs (BDNNs) with improved random forests (RFs). First, a BDNN-based feature extraction method with binary weights and activations in a training process is designed to reduce the model runtime without losing the accuracy of feature extraction. Its generated features are used to train an RF-based fault classifier to relieve the information loss caused by binarization. Second, considering the possible classification accuracy reduction resulting from those very similar binarized features of two instances with different classes, we replace a Gini index with ReliefF as the attribute evaluation measure in training RFs to further enhance the separability of fault features extracted by BDNN and accordingly improve the fault identification accuracy. Third, an edge computing-based fault diagnosis mode is proposed to increase diagnostic efficiency, where our diagnosis model is deployed distributedly on a number of edge nodes close to the end rotational machines in distinct locations. Extensive experiments are conducted to validate the proposed method on the data sets from rolling element bearings, and the results demonstrate that, in almost all cases, its diagnostic accuracy is competitive to the state-of-the-art DNNs and even higher due to a form of regularization in some cases. Benefited from the relatively lower computing and storage requirements of BDNNs, it is easy to be deployed on edge nodes to realize real-time fault diagnosis concurrently. <i>Note to Practitioners</i>&#x2014;Rotating machines, such as engines and motors, are the cornerstones of the modern industry. Edge computing is an emerging computing paradigm where computation is performed on the edges of networks rather than on the central cloud, thereby reducing system response time, transmission overhead, storage space, and computation resources of the cloud. Motivated by the high demand on computation for deploying DNN models and lower computation complexity for running BDNN models and easiness for large-scale deployment of BDNNs, an edge computing-based method for real-time fault diagnosis of rotating machines is proposed. First, we design a BDNN-based feature extractor to decrease the amount of computation and speed up a diagnosis processes. Then, the resulting binary features are fed to train an RF-based classifier, where we use ReliefF instead of Gini index when training a random forest model to further improve the proposed method&#x2019;s diagnostic accuracy. Finally, a novel cloud-edge collaborative computing-based fault diagnostic mode is presented, where the model trained from the central cloud is deployed on the edge computing devices distributed in large-scale scenarios to realize real-time fault diagnosis. Experiment results show that the proposed method can maintain the desired accuracy but greatly enhance the diagnosis speed when deployed on the edge nodes near end physical machines. It is easily extended and used for fault detection in many industrial sectors.

Modularized Multi-Stage Binarized Deep-Model Compression and Performance Recovery

Despite the approaching Internet of Things (IoT) era, most smart devices cannot directly execute complicated deep neural networks (DNNs) due to their limited memory sizes and computing power. Even with model compression, compressing and executing a non-binarized DNN model often needs a codebook for weight calculation, which typically requires acceleration from a Graphics Processing Unit (GPU). Parameter binarization is one commonly used method to reduce computing time and space, but it often seriously compromises the accuracy of the binarized DNN (BNN), which tends to adopt single-stage non-modularized design. To address these issues, we propose a modularized multi-stage BNN model compression with flexible combination of parameter and structure compression to minimize model size while maintaining maximum accuracy possible. The compressed BNNs do not need any codebooks so they better fit into embedded platforms without any GPU acceleration. From our evaluations, the size of a BNN model can be reduced to 1/57 size at the cost of only about a 5% decrease in accuracy, which outperforms the state-of-the-art single-stage compression on BNN models. Furthermore, the proposed method can recover the accuracy of a BNN by 5% even after its size has been further compressed to approximately half its size, demonstrating its potentials in leveraging edge intelligence.

Semi‐ and fully supervised quantification techniques to improve population estimates from machine classifiers

AbstractModern in situ digital imaging systems collect vast numbers of images of marine organisms and suspended particles. Automated methods to classify objects in these images – largely supervised machine learning techniques – are now used to deal with this onslaught of biological data. Though such techniques can minimize the human cost of analyzing the data, they also have important limitations. In training automated classifiers, we implicitly program them with an inflexible understanding of the environment they are observing. When the relationship between the classifier and the population changes, the computer's performance degrades, potentially decreasing the accuracy of the estimate of community composition. This limitation of automated classifiers is known as “dataset shift.” Here, we describe techniques for addressing dataset shift. We then apply them to the output of a binary deep neural network searching for diatom chains in data generated by the Scripps Plankton Camera System (SPCS) on the Scripps Pier. In particular, we describe a supervised quantification approach to adjust a classifier's output using a small number of human corrected images to estimate the system error in a time frame of interest. This method yielded an 80% improvement in mean absolute error over the raw classifier output on a set of 41 independent samples from the SPCS. The technique can be extended to adjust the output of multi‐category classifiers and other in situ observing systems.

High-Throughput In-Memory Computing for Binary Deep Neural Networks With Monolithically Integrated RRAM and 90-nm CMOS

Deep learning hardware designs have been bottlenecked by conventional memories such as SRAM due to density, leakage and parallel computing challenges. Resistive devices can address the density and volatility issues, but have been limited by peripheral circuit integration. In this work, we demonstrate a scalable RRAM based in-memory computing design, termed XNOR-RRAM, which is fabricated in a 90nm CMOS technology with monolithic integration of RRAM devices between metal 1 and 2. We integrated a 128x64 RRAM array with CMOS peripheral circuits including row/column decoders and flash analog-to-digital converters (ADCs), which collectively become a core component for scalable RRAM-based in-memory computing towards large deep neural networks (DNNs). To maximize the parallelism of in-memory computing, we assert all 128 wordlines of the RRAM array simultaneously, perform analog computing along the bitlines, and digitize the bitline voltages using ADCs. The resistance distribution of low resistance states is tightened by write-verify scheme, and the ADC offset is calibrated. Prototype chip measurements show that the proposed design achieves high binary DNN accuracy of 98.5% for MNIST and 83.5% for CIFAR-10 datasets, respectively, with energy efficiency of 24 TOPS/W and 158 GOPS throughput. This represents 5.6X, 3.2X, 14.1X improvements in throughput, energy-delay product (EDP), and energy-delay-squared product (ED2P), respectively, compared to the state-of-the-art literature. The proposed XNOR-RRAM can enable intelligent functionalities for area-/energy-constrained edge computing devices.

Hybrid Fixed-Point/Binary Deep Neural Network Design Methodology for Low-Power Object Detection

Suffering from both high computational complexity and high memory bandwidth is the major challenge in realizing the deep neural network in low-power for real-time applications. Binarizing the feature maps as well as the filter coefficients in deep neural network is an efficient way to reduce the high power consumption in deep learning object detection, however, it greatly scarifies the detection accuracy when reducing the bit-width of a 32-bit word to a binary bit in a floating-point deep neural network. This paper proposes a hybrid fixed point/binary deep neural network design methodology for object detection to achieve low-power consumption by taking advantage of both the fixed-point and binary deep neural networks, which allocates enough bit-width to design the hardware datapath in different layers of deep neural network. The proposed methodology combines dynamic fixed-point quantization and binarization techniques together to extremely compress the object detection model to result in a compact hybrid fixed-point/binary detection neural network, which achieves lower bandwidth and lower computational complexity. An automation tool based on the proposed methodology is also developed to train a hybrid deep neural network under a specified quality loss range. Taking MobileNet-SSD as an example, using the proposed methodology, the resulted model achieves 91% model size reduction and 75.8% memory bandwidth reduction at the cost of less than 1% mAP quality degradation. The proposed design methodology for hybrid fixed-point/binary deep neural networks achieves a good balance on detection accuracy, model size compression ratio and feature map reduction for low-power deep learning object detection applications.

C3SRAM: In-Memory-Computing SRAM Macro Based on Capacitive-Coupling Computing

This letter presents C3SRAM, an in-memory-computing SRAM macro, which utilizes analog-mixed-signal capacitive-coupling computing to perform XNOR-and-accumulate operations for binary deep neural networks. The 256 × 64 C3SRAM macro asserts all 256 rows simultaneously and equips one ADC per column, realizing fully parallel vector-matrix multiplication in one cycle. C3SRAM demonstrates 672 TOPS/W and 1638 GOPS, and achieves 98.3% accuracy for MNIST and 85.5% for CIFAR-10 dataset. It achieves 3975× smaller energy-delay product than conventional digital processors.

Compact Hash Code Learning With Binary Deep Neural Network

Learning compact binary codes for image retrieval problem using deep neural networks has recently attracted increasing attention. However, training deep hashing networks is challenging due to the binary constraints on the hash codes. In this paper, we propose deep network models and learning algorithms for learning binary hash codes given image representations under both unsupervised and supervised manners. The novelty of our network design is that we constrain one hidden layer to directly output the binary codes. This design has overcome a challenging problem in some previous works: optimizing non-smooth objective functions because of binarization. In addition, we propose to incorporate independence and balance properties in the direct and strict forms into the learning schemes. We also include a similarity preserving property in our objective functions. The resulting optimizations involving these binary, independence, and balance constraints are difficult to solve. To tackle this difficulty, we propose to learn the networks with alternating optimization and careful relaxation. Furthermore, by leveraging the powerful capacity of convolutional neural networks, we propose an end-to-end architecture that jointly learns to extract visual features and produce binary hash codes. Experimental results for the benchmark datasets show that the proposed methods compare favorably or outperform the state of the art.

Recursive Binary Neural Network Training Model for Efficient Usage of On-Chip Memory

We present a novel deep learning model for a neural network that reduces both computation and data storage overhead. To do so, the proposed model proposes and combines a binary-weight neural network (BNN) training, a storage reuse technique, and an incremental training scheme. The storage requirements can be tuned to meet the desired classification accuracy, storing more parameters on an on-chip memory, and thereby reducing off-chip data storage accesses. Our experiments show 4–6 $\times $ reduction in weight storage footprint when training binary deep neural network models. On the FPGA platform, this results in a reduced amount of off-chip accesses, enabling our model to train a neural network in $14\times $ shorter latency, as compared to the conventional BNN training method.

Leveraging Stochasticity for In Situ Learning in Binarized Deep Neural Networks

A recent thrust in deep neural network (DNN) research has been toward binary approaches for compact and energy-sparing neuromorphic architectures utilizing emerging devices. However, approaches to deal with device process variations and the realization of stochastic behavior intrinsically within neural circuits remain underexplored. Herein, we leverage a novel probabilistic spintronic device for low-energy recognition operations that improves DNN performance through active in situ learning via the mitigation of device reliability challenges.

Binary neural networks for speech recognition

Recently, deep neural networks (DNNs) significantly outperform Gaussian mixture models in acoustic modeling for speech recognition. However, the substantial increase in computational load during the inference stage makes deep models difficult to directly deploy on low-power embedded devices. To alleviate this issue, structure sparseness and low precision fixed-point quantization have been applied widely. In this work, binary neural networks for speech recognition are developed to reduce the computational cost during the inference stage. A fast implementation of binary matrix multiplication is introduced. On modern central processing unit (CPU) and graphics processing unit (GPU) architectures, a 5–7 times speedup compared with full precision floatingpoint matrix multiplication can be achieved in real applications. Several kinds of binary neural networks and related model optimization algorithms are developed for large vocabulary continuous speech recognition acoustic modeling. In addition, to improve the accuracy of binary models, knowledge distillation from the normal full precision floating-point model to the compressed binary model is explored. Experiments on the standard Switchboard speech recognition task show that the proposed binary neural networks can deliver 3–4 times speedup over the normal full precision deep models. With the knowledge distillation from the normal floating-point models, the binary DNNs or binary convolutional neural networks (CNNs) can restrict the word error rate (WER) degradation to within 15.0%, compared to the normal full precision floating-point DNNs or CNNs, respectively. Particularly for the binary CNN with binarization only on the convolutional layers, the WER degradation is very small and is almost negligible with the proposed approach.