Binary Neural Network Research Articles

Lightweighted Hyperspectral Image Classification Network by Progressive Bi-Quantization

Convolutional neural network (CNN) has shown its powerful ability for hyperspectral image (HSI) classification, which however, is difficult to deploy on resource-limited or low-latency platforms due to its parameter and computation redundancy. Though binary neural network (BNN) has attracted attention for its extreme compressing and speeding up ability by binarizing both weights and activations, it has rarely been explored for HSI classification. In this study, we elaborately design a BNN with good performance for HSI classification task. Specifically, an adaptive gradient scale module is proposed to flexibly modify the gradient during training stage to better optimize the BNN and does not add any extra computation for inference. Furthermore, a curriculum learning-based progressive binarization strategy is utilized to improve the performance. Compared with the existing BNN works, our method can increase the HSI classification accuracy by a large margin while maintaining the compressing ratio. Abundant experiments on three datasets demonstrate the effectiveness of the proposed method.

E2FIF: Push the limit of Binarized Deep Imagery Super-resolution using End-to-end Full-precision Information Flow.

Binary neural network (BNN) provides a promising solution to deploy parameter-intensive deep single image super-resolution (SISR) models onto real devices with limited storage and computational resources. To achieve comparable performance with the full-precision counterpart, most existing BNNs for SISR mainly focus on compensating for the information loss incurred by binarizing weights and activations in the network through better approximations to the binarized convolution. In this study, we revisit the difference between BNNs and their full-precision counterparts and argue that the key to good generalization performance of BNNs lies on preserving a complete full-precision information flow along with an accurate gradient flow passing through each binarized convolution layer. Inspired by this, we propose to introduce a full-precision skip connection, or a variant thereof, over each binarized convolution layer across the entire network, which can increase the forward expressive capability and the accuracy of back-propagated gradient, thus enhancing the generalization performance. More importantly, such a scheme can be applied to any existing BNN backbones for SISR without introducing any additional computation cost. To validate the efficacy of the proposed approach, we evaluate it using four different backbones for SISR on four benchmark datasets and report obviously superior performance over existing BNNs and even some 4-bit competitors.

Binarized Neural Network With Parameterized Weight Clipping and Quantization Gap Minimization for Online Knowledge Distillation

As the applications for artificial intelligence are growing rapidly, numerous network compression algorithms have been developed to restrict computing resources such as smartphones, edge, and IoT devices. Knowledge distillation (KD) leverages soft labels derived from a teacher model to a less parameterized model achieving high accuracy with reduced computational burden. Moreover, online KD provides parallel computing through collaborative learning between teacher and student networks, thus enhancing the training speed. A binarized neural network (BNN) offers an intriguing opportunity to facilitate aggressive compression at the expense of drastically degraded accuracy. In this study, two performance improvements are proposed for online KD when a BNN is applied as a student network: 1) parameterized weight clipping (PWC) to reduce dead weights in the student network and 2) quantization gap-aware adaptive temperature scheduling between the teacher and student networks. In contrast to constant weight clipping (CWC), PWC demonstrates a 3.78% top-1 test accuracy enhancement with trainable weight clipping by decreasing the gradient mismatch with CIFAR-10 dataset. Furthermore, the quantization gap-aware temperature scheduling increases the top-1 test accuracy by 0.08% over online KD at a constant temperature. By aggregating both methodologies, the top-1 test accuracy for CIFAR-10 dataset was 94.60%, and that for Tiny-ImageNet dataset was comparable to that of the 32-bit full-precision neural network.

Hybrid MTJ/CNTFET-Based Binary Synapse and Neuron for Process-in-Memory Architecture

This paper proposes a reliable integrated binary synapse and neurons for hardware implementation of binary neural networks. Thanks to the nonvolatile nature of the magnetic tunnel junction ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MTJ</b> ) and the unique features of the carbon nanotube field-effect transistor ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CNTFET</b> ), the proposed design in this paper does not require external memory to store weights and also consumes low static power. Also, due to the proposed circuit structure that did not use sequential parts, the proposed circuit is immune to soft error. Because in binary neural networks, weights are limited to two values of ‘-1’ and ‘1’, the occurrence of soft errors dramatically reduces the accuracy of the network. Simulation results indicate that the proposed design in this paper consumes at least 9% lower power, occupies 34% lower area, and offers a 49% lower power delay area product ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PDAP</b> ). Also, Monte-Carlo simulations have been performed to study the effect of the process variation on the network. The result of the Monte-Carlo simulations shows that the proposed neuron has no logical error in 10000 simulations. Consequently, the accuracy of the network utilizing by the proposed neuron is equal to the software-implemented network and does not decrease even in the presence of process variations.

FlexBNN: Fast Private Binary Neural Network Inference With Flexible Bit-Width

Advancements in deep learning enable neural network (NN) inference to be a service, but service providers and clients want to keep their inputs secret for privacy protection. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Private Inference</i> is the task of evaluating NN without leaking private inputs. Existing secure multiparty computation (MPC)-based solutions mainly focus on fixed bit-width methodology, such as 32 and 64 bits. Binary Neural Network (BNN) is efficient when evaluated in MPC and has achieved reasonable accuracy for commonly used datasets, but prior private BNN inference solutions, which focus on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Boolean Circuits</i> , are still costly in communication and run-time. In this paper, we introduce FLEXBNN, a fast private BNN inference framework using three-party computation (3PC) in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Arithmetic Circuits</i> against semi-honest adversaries with honest-majority. In FLEXBNN, we propose to employ flexible and small bit-width equipped with a seamless bit-width conversion method and design several specific optimizations towards the basic operations: i) We propose bit-width determination methods for Matrix Multiplication and Sign-based Activation function. ii) We integrate Batch Normalization and Max-Pooling into the Sign-based Activation function for better efficiency. iii) More importantly, we achieve seamless bit-width conversion within the Sign-based Activation function with no additional cost. Extensive experiments illustrate that FLEXBNN outperforms state-of-the-art solutions in communication, run-time, and scalability. On average, FLEXBNN is 11× faster than XONN (USENIX Security’ 19) in LAN, 46× (resp. 9.3×) faster than QUOTIENT (ACM CCS’19) in LAN (resp. WAN), 10× faster than BANNERS (ACM IH&MMSec’21) in LAN, and 1.1-2.9× (resp. 1.5-2.7×) faster than FALCON (semi-honest, PoPETs’21) in LAN (resp. WAN), and improves the respective communication by 500×, 127×, and 1.3-1.5× compared to XONN, BANNERS, and FALCON.

Cellular Binary Neural Network for Accurate Image Classification and Semantic Segmentation

This paper presents the Cellular Binary Neural Network (CBNN), which is an efficient deep neural network with binary weights and activations. To address the challenge of performance drop caused by low-precision representation, the CBNN adopts multiple subnets which are connected via learnable global lateral paths. The introduced lateral connections are assumed to be sparse and grouped with respect to different source layers. The inter-network lateral connections and inner-network parameters are simultaneously optimized by the distributional loss, classification loss and the group sparse regularization term. Experiments on the CIFAR-10 and ImageNet datasets showed that, by incorporating optimized group-sparse lateral paths, the CBNN outperformed many state-of-the-art binary neural networks in terms of classification accuracy. Besides, to verify the generalization of the proposed binary model, we extended the CBNN on semantic segmentation task. CBNN takes advantage of the multiple subnets to derive the more informative feature maps which are computed by the parallel aggregation in the last convolution block. Experiments on PASCAL VOC segmentation dataset demonstrated that, under the same segmentation settings, the proposed method achieved the superior performance over other compared networks and even the full-precision counterpart.

An Optimized Hardware Inference of SABiNN: Shift-Accumulate Binarized Neural Network for Sleep Apnea Detection

This paper presents the design of an optimized hardware-based neural network (NN) called a Shift-Accumulate Binarized Neural Network (SABiNN). SABiNN is employed in detecting respiratory-related diseases such as sleep apnea (SA) among adults. Initially, a 3-hidden layer-based NN model was trained, validated, and tested with open-source apnea PSG datasets collected from the PhysioNET databank. Single lead ECG and pulse oximeter data were collected, pre-processed, and digitized for network training. The NN was later transformed into SABiNN, demonstrating model accuracy of 81.5% (CI: ± 3.5) with an energy efficiency of 5mJ on reprogrammable hardware. The precision rate of the model was further increased by re-designing the XNOR gate of the multiply-accumulate (MAC) operation with NAND gate-based XNOR. This re-design process significantly improved the overall model’s classification and precision. Further expansion of SABiNN was carried out to achieve a higher accuracy rate (over 88%) which was designed on the CMOS platform using a 130 nm open-source process design kit (PDK) developed by Google and Skywater. The proposed model on the CMOS platform utilized a chip area of 0.16 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> and showcased an energy efficiency of 1pJ. A total of ~11k CMOS cells with two 16-bit input and one 1-bit output pins were utilized to realize the SABiNN on CMOS. The success of this design technique can be leveraged in developing a fully system-on-a-chip (SoC) integrated wearable system for sleep apnea detection.

Permutation binary neural networks: Analysis of periodic orbits and its applications

&lt;p style='text-indent:20px;'&gt;This paper presents a permutation binary neural network characterized by local binary connection, global permutation connection, and the signum activation function. The dynamics is described by a difference equation of binary state variables. Depending on the connection, the network generates various periodic orbits of binary vectors. The binary/permutation connection brings benefits to precise analysis and to FPGA based hardware implementation. In order to consider the periodic orbits, we introduce three tools: a composition return map for visualization of the dynamics, two feature quantities for classification of periodic orbits, and an FPGA based hardware prototype for engineering applications. Using the tools, we have analyzed all the 6-dimensional networks. Typical periodic orbits are confirmed experimentally.&lt;/p&gt;

Relaxed Training Procedure for a Binary Neural Network

Relaxed Training Procedure for a Binary Neural Network

A variety of globally stable periodic orbits in permutation binary neural networks

The permutation binary neural networks are characterized by global permutation connections and local binary connections. Although the parameter space is not large, the networks exhibit various binary periodic orbits. Since analysis of all the periodic orbits is not easy, we focus on globally stable binary periodic orbits such that almost all initial points fall into the orbits. For efficient analysis, we define the standard permutation connection that represents multiple equivalent permutation connections. Applying the brute force attack to 7-dimensional networks, we present the main result: a list of standard permutation connections for all the globally stable periodic orbits. These results will be developed into detailed analysis of the networks and its engineering applications.

An efficient real-time accelerator for high-accuracy DNN-based optical flow estimation in FPGA

Recently, accelerator architectures of deep neural networks (DNNs) have been designed to accelerate computer vision tasks, gaining the advantages of both accuracy and speed. However, DNNs-based optical flow estimation accelerators have not been successfully deployed due to the resource constraint. Existing hardware accelerators for optical flow estimation are all designed for non-DNN methods and generally perform poorly in estimated accuracy. In this paper, we present FlowAcc, a dedicated hardware accelerator for DNN-based optical flow estimation, adopting a pipelined hardware design for the real-time processing of image streams. We design an efficient multiplexing binary neural network (BNN) architecture for pyramidal feature extraction to significantly reduce the hardware cost and make it independent of the pyramid-level number. Furthermore, efficient designs such as hamming distance calculation and competent flow regularization are used for hierarchical optical flow estimation to greatly improve the system’s efficiency. Comprehensive experimental results demonstrate that FlowAcc achieves state-of-the-art estimation accuracy and real-time performance on the Middlebury dataset when compared with the existing optical flow accelerators.

Energy efficient ternary computation unit using differential spin Hall effect MRAM

Recent advances in quantized neural networks (QNNs) have paved the way for energy efficient hardware architectures for machine learning tasks. Binary and ternary QNNs are suitable for image classification and recognition applications on highly resource constrained hardware. Binary neural networks have low precision thus suffer a significant accuracy loss for dense networks and large datasets. This issue can be resolved through ternary neural networks (TNNs) with higher weight precision and better resource utilization. TNN implementation using conventional complementary metal oxide semiconductor and memristive devices show limited improvement in area and energy efficiency. Spintronics based magnetic random access memory (MRAM) devices are the most prominent choice amongst the various non-volatile memories for neural networks. This work presents the implementation of differential spin Hall effect (DSHE) MRAM-based two and three input ternary computation units (TCUs) for TNN. Furthermore, a multilayer perceptron architecture with synaptic crossbar array using the proposed TCU is implemented for Modified National Institute of Standards and Technology data classification. The results show that DSHE-based TCU is 30% more energy efficient as compared with spin-transfer torque (STT)-MRAM based design. Furthermore, DSHE-MRAM based TNN shows improvement in energy and area by 82% and 9%, respectively, when compared to STT-based TNN.

An Information-Reserved and Deviation-Controllable Binary Neural Network for Object Detection

Object detection is a fundamental task in computer vision, which is usually based on convolutional neural networks (CNNs). While it is difficult to be deployed in embedded devices due to the huge storage and computing consumptions, binary neural networks (BNNs) can execute object detection with limited resources. However, the extreme quantification in BNN causes diversity of feature representation loss, which eventually influences the object detection performance. In this paper, we propose a method balancing Information Retention and Deviation Control to achieve effective object detection, named IR-DC Net. On the one hand, we introduce the KL-Divergence to compose multiple entropy for maximizing the available information. On the other hand, we design a lightweight convolutional module to generate scale factors dynamically for minimizing the deviation between binary and real convolution. The experiments on PASCAL VOC, COCO2014, KITTI, and VisDrone datasets show that our method improved the accuracy in comparison with previous binary neural networks.

S[formula omitted]NN: Time step reduction of spiking surrogate gradients for training energy efficient single-step spiking neural networks

As the scales of neural networks increase, techniques that enable them to run with low computational cost and energy efficiency are required. From such demands, various efficient neural network paradigms, such as spiking neural networks (SNNs) or binary neural networks (BNNs), have been proposed. However, they have sticky drawbacks, such as degraded inference accuracy and latency. To solve these problems, we propose a single-step spiking neural network (S3NN), an energy-efficient neural network with low computational cost and high precision. The proposed S3NN processes the information between hidden layers by spikes as SNNs. Nevertheless, it has no temporal dimension so that there is no latency within training and inference phases as BNNs. Thus, the proposed S3NN has a lower computational cost than SNNs that require time-series processing. However, S3NN cannot adopt naïve backpropagation algorithms due to the non-differentiability nature of spikes. We deduce a suitable neuron model by reducing the surrogate gradient for multi-time step SNNs to a single-time step. We experimentally demonstrated that the obtained surrogate gradient allows S3NN to be trained appropriately. We also showed that the proposed S3NN could achieve comparable accuracy to full-precision networks while being highly energy-efficient.

String Current Compensation Method in VNAND Flash for Hardware-Based BNNs

This work proposes a method to prevent unwanted string current degradation in multistacks vertical NAND (VNAND) flash memory for hardware-based binary neural networks (BNNs). Simulations investigate the effect of string resistance on the accuracy of a multilayer BNNs, considering the measured cell string characteristics. When the weights obtained from off-chip training are transferred to the BNN without compensation cells, the classification accuracy of the hardware-based BNN is reduced by ~80.3% in 23-layer network. However, the proposed compensation cell model shows a degradation of classification accuracy of 1.4% in the 23-layer BNN compared to the software-based BNN.

Binarized Neural Network Accelerator Macro Using Ultralow-Voltage Retention SRAM for Energy Minimum-Point Operation

A binarized neural network (BNN) accelerator based on a processing-in-memory (PIM)/ computing-in-memory (CIM) architecture using ultralow-voltage retention static random access memory (ULVR-SRAM) is proposed for the energy minimum-point (EMP) operation. The BNN accelerator (BNA) macro is designed to perform stable inference operations at EMP and substantive power-gating (PG) using ULVR at an ultralow voltage (< EMP), which can be applied to fully connected layers (FCLs) with arbitrary shapes and sizes. The EMP operation of the BNA macro, which is enabled by applying the ULVR-SRAM to the macro, can dramatically improve the energy efficiency (TOPS/W) and significantly enlarge the number of parallelized multiply–accumulate (MAC) operations. In addition, the ULVR mode of the BNA macro, which also benefits from the usage of ULVR-SRAM, is effective at reducing the standby power. The proposed BNA macro can show a high energy efficiency of 65 TOPS/W for FCLs. This BNA macro concept using the ULVR-SRAM can be expanded to convolution layers, where the EMP operation is also expected to enhance the energy efficiency of convolution layers.

Lite-Stereo: A Resource-Efficient Hardware Accelerator for Real-Time High-Quality Stereo Estimation Using Binary Neural Network

Stereo estimation plays a key role in many autonomous systems, such as robotics and self-driving cars. Recent work on StereoEngine, an FPGA-based accelerator for deep neural network (DNN)-based stereo estimation, has been demonstrated as a promising solution to achieve both real-time and high accuracy performance for depth sensing. However, this solution still suffers from over-utilizing the hardware resource of FPGAs. In this article, we present Lite-Stereo, a resource-efficient DNN-based stereo vision accelerator to improve the hardware efficiency for StereoEngine running on a resource-constrained FPGA. To achieve this, we design a set of optimized hardware architectures for resource-demanding bottleneck modules. In order to balance the gap between the processing speed and resource efficiency, the process elements in binary neural network modules are shared within and across modules. In addition, we provide reusing strategies on path aggregation and neighbor calculation to improve the resource efficiency of the semi-global matching module. Evaluation results demonstrate that Lite-Stereo reduces the hardware cost of ALUTs and RAM bits by 60% and 29%, respectively, without compromising the accuracy and energy efficiency compared with StereoEngine.

Multi-Model Inference Accelerator for Binary Convolutional Neural Networks

Binary convolutional neural networks (BCNN) have shown good accuracy for small to medium neural network models. Their extreme quantization of weights and activations reduces off-chip data transfer and greatly reduces the computational complexity of convolutions. Further reduction in the complexity of a BCNN model for fast execution can be achieved with model size reduction at the cost of network accuracy. In this paper, a multi-model inference technique is proposed to reduce the execution time of the binarized inference process without accuracy reduction. The technique considers a cascade of neural network models with different computation/accuracy ratios. A parameterizable binarized neural network with different trade-offs between complexity and accuracy is used to obtain multiple network models. We also propose a hardware accelerator to run multi-model inference throughput in embedded systems. The multi-model inference accelerator is demonstrated on low-density Zynq-7010 and Zynq-7020 FPGA devices, classifying images from the CIFAR-10 dataset. The proposed accelerator improves the frame rate per number of LUTs by 7.2× those of previous solutions on a ZYNQ7020 FPGA with similar accuracy. This shows the effectiveness of the multi-model inference technique and the efficiency of the proposed hardware accelerator.

Pattern Classification Using Quantized Neural Networks for FPGA-Based Low-Power IoT Devices.

With the recent growth of the Internet of Things (IoT) and the demand for faster computation, quantized neural networks (QNNs) or QNN-enabled IoT can offer better performance than conventional convolution neural networks (CNNs). With the aim of reducing memory access costs and increasing the computation efficiency, QNN-enabled devices are expected to transform numerous industrial applications with lower processing latency and power consumption. Another form of QNN is the binarized neural network (BNN), which has 2 bits of quantized levels. In this paper, CNN-, QNN-, and BNN-based pattern recognition techniques are implemented and analyzed on an FPGA. The FPGA hardware acts as an IoT device due to connectivity with the cloud, and QNN and BNN are considered to offer better performance in terms of low power and low resource use on hardware platforms. The CNN and QNN implementation and their comparative analysis are analyzed based on their accuracy, weight bit error, RoC curve, and execution speed. The paper also discusses various approaches that can be deployed for optimizing various CNN and QNN models with additionally available tools. The work is performed on the Xilinx Zynq 7020 series Pynq Z2 board, which serves as our FPGA-based low-power IoT device. The MNIST and CIFAR-10 databases are considered for simulation and experimentation. The work shows that the accuracy is 95.5% and 79.22% for the MNIST and CIFAR-10 databases, respectively, for full precision (32-bit), and the execution time is 5.8 ms and 18 ms for the MNIST and CIFAR-10 databases, respectively, for full precision (32-bit).

Bimodal-Distributed Binarized Neural Networks

Binary neural networks (BNNs) are an extremely promising method for reducing deep neural networks’ complexity and power consumption significantly. Binarization techniques, however, suffer from ineligible performance degradation compared to their full-precision counterparts. Prior work mainly focused on strategies for sign function approximation during the forward and backward phases to reduce the quantization error during the binarization process. In this work, we propose a bimodal-distributed binarization method (BD-BNN). The newly proposed technique aims to impose a bimodal distribution of the network weights by kurtosis regularization. The proposed method consists of a teacher–trainer training scheme termed weight distribution mimicking (WDM), which efficiently imitates the full-precision network weight distribution to their binary counterpart. Preserving this distribution during binarization-aware training creates robust and informative binary feature maps and thus it can significantly reduce the generalization error of the BNN. Extensive evaluations on CIFAR-10 and ImageNet demonstrate that our newly proposed BD-BNN outperforms current state-of-the-art schemes.