Binary Neural Network Research Articles

WDP-BNN: Efficient wafer defect pattern classification via binarized neural network

Wafer map defect pattern classification using convolutional neural network (CNN) has gained a lot of attention in recent years but it demands huge computation and memory cost. Therefore, a WDP-BNN framework based on the binarized neural network is proposed to reduce memory requirement by 29.70× and speed up by 1.66×. To overcome the imbalance problem and performance loss due to binarization of network, advanced data augmentation methods including (Chip Reverse, Chip Translate, Chip Combine) along with random under-sampling method have incorporated in the framework. Experimental results on the WM-811K dataset have demonstrated that the WDP-BNN model has outperformed the state-of-the-art works with the highest classification accuracy of 94.83% and the memory reduction of 1.10-25.93×.

Resilient Biomedical Systems Design Under Noise Using Logic-Based Machine Learning

Increased reliance on electronic health records and plethora of new sensor technologies has enabled the use of machine learning (ML) in medical diagnosis. This has opened up promising opportunities for faster and automated decision making, particularly in early and repetitive diagnostic routines. Nevertheless, there are also increased possibilities of data aberrance arising from environmentally induced noise. It is vital to create ML models that are resilient in the presence of data noise to minimize erroneous classifications that could be crucial. This study uses a recently proposed ML algorithm called the Tsetlin machine (TM) to study the robustness against noise-injected medical data. We test two different feature extraction methods, in conjunction with the TM, to explore how feature engineering can mitigate the impact of noise corruption. Our results show the TM is capable of effective classification even with a signal-to-noise ratio (SNR) of −15dB as its training parameters remain resilient to noise injection. We show that high testing data sensitivity can still be possible at very low SNRs through a balance of feature distribution–based discretization and a rule mining algorithm used as a noise filtering encoding method. Through this method we show how a smaller number of core features can be extracted from a noisy problem space resulting in reduced ML model complexity and memory footprint—in some cases up to 6x fewer training parameters while retaining equal or better performance. In addition, we investigate the cost of noise resilience in terms of energy when compared with recently proposed binarized neural networks.

Commodity Bit-Cell Sponsored MRAM Interaction Design for Binary Neural Network

Commodity Bit-Cell Sponsored MRAM Interaction Design for Binary Neural Network

A storage-efficient ensemble classification using filter sharing on binarized convolutional neural networks.

This paper proposes a storage-efficient ensemble classification to overcome the low inference accuracy of binary neural networks (BNNs). When external power is enough in a dynamic powered system, classification results can be enhanced by aggregating outputs of multiple BNN classifiers. However, memory requirements for storing multiple classifiers are a significant burden in the lightweight system. The proposed scheme shares the filters from a trained convolutional neural network (CNN) model to reduce storage requirements in the binarized CNNs instead of adopting the fully independent classifier. While several filters are shared, the proposed method only trains unfrozen learnable parameters in the retraining step. We compare and analyze the performances of the proposed ensemble-based systems depending on various ensemble types and BNN structures on CIFAR datasets. Our experiments conclude that the proposed method using the filter sharing can be scalable with the number of classifiers and effective in enhancing classification accuracy. With binarized ResNet-20 and ReActNet-10 on the CIFAR-100 dataset, the proposed scheme can achieve 56.74% and 70.29% Top-1 accuracies with 10 BNN classifiers, which enhances performance by 7.6% and 3.6% compared with that using a single BNN classifier.

Accelerating In-Vehicle Network Intrusion Detection System Using Binarized Neural Network

<div class="section abstract"><div class="htmlview paragraph">Controller Area Network (CAN), the de facto standard for in-vehicle networks, has insufficient security features and thus is inherently vulnerable to various attacks. To protect CAN bus from attacks, intrusion detection systems (IDSs) based on advanced deep learning methods, such as Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), have been proposed to detect intrusions. However, those models generally introduce high latency, require considerable memory space, and often result in high energy consumption. To accelerate intrusion detection and also reduce memory requests, we exploit the use of Binarized Neural Network (BNN) and hardware-based acceleration for intrusion detection in in-vehicle networks. As BNN uses binary values for activations and weights rather than full precision values, it usually results in faster computation, smaller memory cost, and lower energy consumption than full precision models. Moreover, unlike other deep learning methods, BNN can be further accelerated by leveraging Field-Programmable Grid Arrays (FPGAs) since BNN cuts down the hardware consumption. We design our BNN model to suit CAN traffic data and exploit sequential features of the CAN traffic instead of individual messages. We evaluate the proposed IDS with four different real vehicle datasets. Our experimental results show that the proposed BNN-based IDS reduces the detection latency on the same CPU (3 times faster) while maintaining acceptable detection rates compared to full precision models. We also implement the proposed IDS using FPGA hardware to reduce latency further and accelerate intrusion detection. Our experiments on multiple platforms demonstrate that using the FPGAs dramatically reduces the detection latency (128 times faster) with lower power consumption in comparison with an embedded CPU.</div></div>

Early Warning of Regional Landslide Disaster and Development of Rural Ecological Industrialization Based on IoT Sensor

Regional landslide disaster is actually the instability caused by the movement of ground or slopes, etc. If a landslide occurs in a habitat area where people live, it can cause a great deal of damage. So, in order to improve the early warning effect of regional geological landslide disaster, the study abandoned the conventional geological probe data, directly used the tilt photography data provided by the fixed camera, introduced the camera clock synchronization control system supported by the high-sensitivity atomic clock timing function, and used the data warehouse hardware and computing host hardware. Combined with the spatial convolution neural network, fuzzy neural network (logarithmic depth iterative regression neural network, polynomial depth iterative regression neural network, transfinite learning machine, and binary neural network) and other technologies finally realize high geological disaster early warning sensitivity and long early warning advance. The system will become an alternative to the previous geological disaster early warning system based on the direct data of geological embedded probes.

IE-Net: Information-Enhanced Binary Neural Networks for Accurate Classification

Binary neural networks (BNNs) have been proposed to reduce the heavy memory and computation burdens in deep neural networks. However, the binarized weights and activations in BNNs cause huge information loss, which leads to a severe accuracy decrease, and hinders the real-world applications of BNNs. To solve this problem, in this paper, we propose the information-enhanced network (IE-Net) to improve the performance of BNNs. Firstly, we design an information-enhanced binary convolution (IE-BC), which enriches the information of binary activations and boosts the representational power of the binary convolution. Secondly, we propose an information-enhanced estimator (IEE) to gradually approximate the sign function, which not only reduces the information loss caused by quantization error, but also retains the information of binary weights. Furthermore, by reducing the information loss in binary representations, the novel binary convolution and estimator gain large information compared with the previous work. The experimental results show that the IE-Net achieves accuracies of 88.5% (ResNet-20) and 61.4% (ResNet-18) on CIFAR-10 and ImageNet datasets respectively, which outperforms other SOTA methods. In conclusion, the performance of BNNs could be improved significantly with information enhancement on both weights and activations.

AAPFE: Aligned Assembly Pre-Training Function Embedding for Malware Analysis

The use of natural language processing to analyze binary data is a popular research topic in malware analysis. Embedding binary code into a vector is an important basis for building a binary analysis neural network model. Current solutions focus on embedding instructions or basic block sequences into vectors with recurrent neural network models or utilizing a graph algorithm on control flow graphs or annotated control flow graphs to generate binary representation vectors. In malware analysis, most of these studies only focus on the single structural information of the binary and rely on one corpus. It is difficult for vectors to effectively represent the semantics and functionality of binary code. Therefore, this study proposes aligned assembly pre-training function embedding, a function embedding scheme based on a pre-training aligned assembly. The scheme creatively applies data augmentation and a triplet network structure to the embedding model training. Each sub-network extracts instruction sequence information using the self-attention mechanism and basic block graph structure information with the graph convolution network model. An embedding model is pre-trained with the produced aligned assembly triplet function dataset and is subsequently evaluated against a series of comparative experiments and application evaluations. The results show that the model is superior to the state-of-the-art methods in terms of precision, precision ranking at top N (p@N), and the area under the curve, verifying the effectiveness of the aligned assembly pre-training and multi-level information extraction methods.

BWPT: Binary weight partial‐sum table for BNN/BWN acceleration

This letter proposes a novel architecture used in processing elements (PEs) to accelerate binary neural networks (BNNs) and binary weight networks (BWNs). We use the partial-sum value of weights with matching input activation to form a lookup table, named binary weight partial-sum table (BWPT). Following that, the constructed BWPT PE improves the computing efficiency in a variety of data flows. The proposed design has a 3× lower power delay product than the baseline PE and reduces the power consumption by 11.2%, 7.1% and 5.4% compared to 3-bit activation lookup table-based PE (LBPE3), 3-bit reduction tree PE and baseline PE, respectively.

An Efficient Channel-Aware Sparse Binarized Neural Networks Inference Accelerator

The binarized neural network (BNN) inference accelerators show great promise in cost- and power-restricted domains. However, the performances of these accelerators are still severely limited by the significant redundancies in BNNs inference. In this brief, we introduce channel-aware sparse accelerator (CAA) to alleviate the performance degradations induced by the redundancies in BNNs while maintaining original accuracies. First, motivated by the observation that the convolution processes of our rebuilt rectangle kernels contain many redundant operations which can be skipped by exploiting the BNN-specific property, we convert the entire original XNOR-popcount convolutions of each neuron into channel-aware-popcount (CAP) operations for all binarized convolutional and fully-connected layers in CAA by employing rectangle kernel simplification strategy and eliminate the unnecessary operations. Meanwhile, these CAP operations can be implemented to directly gain the final output without any extra steps. Furthermore, inspired by our new observations on two specific kinds of properties of the CAP operations, we adopt group pruning approach to save the remaining redundant CAP operations. Experimental results show that our design evaluated on an embedded FPGA achieves 4.2- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$6.6{\times }$ </tex-math></inline-formula> inference-speedup, 3.6- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.5{\times }$ </tex-math></inline-formula> energy-efficiency enhancement, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.35{\times }$ </tex-math></inline-formula> resource-efficiency improvement compared with state-of-the-art works.

Self-distribution binary neural networks

In this work, we study the binary neural networks (BNNs) of which both the weights and activations are binary (i.e., 1-bit representation). Feature representation is critical for deep neural networks, while in BNNs, the features only differ in signs. Prior work introduces scaling factors into binary weights and activations to reduce the quantization error and effectively improves the classification accuracy of BNNs. However, the scaling factors not only increase the computational complexity of networks, but also make no sense to the signs of binary features. To this end, Self-Distribution Binary Neural Network (SD-BNN) is proposed. Firstly, we utilize Activation Self Distribution (ASD) to adaptively adjust the sign distribution of activations, thereby improve the sign differences of the outputs of the convolution. Secondly, we adjust the sign distribution of weights through Weight Self Distribution (WSD) and then fine-tune the sign distribution of the outputs of the convolution. Extensive experiments on CIFAR-10 and ImageNet datasets with various network structures show that the proposed SD-BNN consistently outperforms the state-of-the-art (SOTA) BNNs (e.g., achieves 92.5% on CIFAR-10 and 66.5% on ImageNet with ResNet-18) with less computation cost. Code is available at https://github.com/ pingxue-hfut/SD-BNN.

Facial Makeup Detection Using Multi-Scale Local Binary Patterns and Convolutional Neural Network Fusion

This study presents a novel facial makeup detection scheme using fusion of multi-scale Local Binary Patterns (ms-LBP) texture methods and convolutional neural network (CNN) deep learning extractor. Facial makeups affect the accuracy of face recognition systems due to appearance alteration of individuals. In order to fuse the facial features, the proposed scheme first considers concatenation of extracted global histogram of a whole image using three different LBP operators. The LBP scales are used to extract and then concatenate the local histogram of overlapping and nonoverlapping blocks of images. This way utilizes the advantages of microtexton information of local primitives besides the extracted global textures. Finally, the extracted features using CNN feature extractor are combined with global and local multi-scale textures. In general, CNN deep learning extractor learns high-level discriminative characteristics of images and therefore the proposed system improves the facial makeup detection rate by involving both microtexton and discriminative information of facial images. In addition, the proposed method attempts to select the optimized subset of facial features to increase the detection performance of system by applying Particle Swarm Optimization (PSO) technique after feature level fusion. The paper uses Support Vector Machine (SVM) classifier for classifying the facial vectors into makeup or no-makeup classes. The proposed scheme is then evaluated using YMU, VMU and MIW facial makeup databases with consideration of light, medium and heavy makeups on several datasets. Experimental results analysis clarifies the effectiveness of proposed facial makeup detection framework of this study.

Implementation of Binarized Neural Networks in All-Programmable System-on-Chip Platforms

The Binarized Neural Network (BNN) is a Convolutional Neural Network (CNN) consisting of binary weights and activation rather than real-value weights. Smaller models are used, allowing for inference effectively on mobile or embedded devices with limited power and computing capabilities. Nevertheless, binarization results in lower-entropy feature maps and gradient vanishing, which leads to a loss in accuracy compared to real-value networks. Previous research has addressed these issues with various approaches. However, those approaches significantly increase the algorithm’s time and space complexity, which puts a heavy burden on those embedded devices. Therefore, a novel approach for BNN implementation on embedded systems with multi-scale BNN topology is proposed in this paper, from two optimization perspectives: hardware structure and BNN topology, that retains more low-level features throughout the feed-forward process with few operations. Experiments on the CIFAR-10 dataset indicate that the proposed method outperforms a number of current BNN designs in terms of efficiency and accuracy. Additionally, the proposed BNN was implemented on the All Programmable System on Chip (APSoC) with 4.4 W power consumption using the hardware accelerator.

Long-Term Accuracy Enhancement of Binary Neural Networks Based on Optimized Three-Dimensional Memristor Array.

In embedded neuromorphic Internet of Things (IoT) systems, it is critical to improve the efficiency of neural network (NN) edge devices in inferring a pretrained NN. Meanwhile, in the paradigm of edge computing, device integration, data retention characteristics and power consumption are particularly important. In this paper, the self-selected device (SSD), which is the base cell for building the densest three-dimensional (3D) architecture, is used to store non-volatile weights in binary neural networks (BNN) for embedded NN applications. Considering that the prevailing issues in written data retention on the device can affect the energy efficiency of the system’s operation, the data loss mechanism of the self-selected cell is elucidated. On this basis, we introduce an optimized method to retain oxygen ions and prevent their diffusion toward the switching layer by introducing a titanium interfacial layer. By using this optimization, the recombination probability of Vo and oxygen ions is reduced, effectively improving the retention characteristics of the device. The optimization effect is verified using a simulation after mapping the BNN weights to the 3D VRRAM array constructed by the SSD before and after optimization. The simulation results showed that the long-term recognition accuracy (greater than 105 s) of the pre-trained BNN was improved by 24% and that the energy consumption of the system during training can be reduced 25,000-fold while ensuring the same accuracy. This work provides high storage density and a non-volatile solution to meet the low power consumption and miniaturization requirements of embedded neuromorphic applications.

SOT-MRAM-based Binary Neural Networks Demonstration for Single Character Recognition

With the substantial increase in the amount of data, the mismatch between the processing speed of the hardware and the software results in the ‘Memory Wall’ problem. Processing-in-memory (PIM), in which the compute and memory units are integrated, can avoid frequent data transmission. Binary neural network (BNN) uses binary weights and activations instead of full-precision weights and activations in the convolutional neural network, which reduces computational complexity with minor influence on accuracy. In this paper, we used a one-step operation to write a pair of SOT-MRAM cells and verified the two basic operations in BNN: XNOR and bitcount. Then, we employed an external control circuit with FPGA and accomplished ‘I’ single-character recognition based on vector-matrix multiplication in the SOT-MRAM array.

BiRe-ID: Binary Neural Network for Efficient Person Re-ID

Person re-identification (Re-ID) has been promoted by the significant success of convolutional neural networks (CNNs). However, the application of such CNN-based Re-ID methods depends on the tremendous consumption of computation and memory resources, which affects its development on resource-limited devices such as next generation AI chips. As a result, CNN binarization has attracted increasing attention, which leads to binary neural networks (BNNs). In this article, we propose a new BNN-based framework for efficient person Re-ID (BiRe-ID). In this work, we discover that the significant performance drop of binarized models for Re-ID task is caused by the degraded representation capacity of kernels and features. To address the issues, we propose the kernel and feature refinement based on generative adversarial learning (KR-GAL and FR-GAL) to enhance the representation capacity of BNNs. We first introduce an adversarial attention mechanism to refine the binarized kernels based on their real-valued counterparts. Specifically, we introduce a scale factor to restore the scale of 1-bit convolution. And we employ an effective generative adversarial learning method to train the attention-aware scale factor. Furthermore, we introduce a self-supervised generative adversarial network to refine the low-level features using the corresponding high-level semantic information. Extensive experiments demonstrate that our BiRe-ID can be effectively implemented on various mainstream backbones for the Re-ID task. In terms of the performance, our BiRe-ID surpasses existing binarization methods by significant margins, at the level even comparable with the real-valued counterparts. For example, on Market-1501, BiRe-ID achieves 64.0% mAP on ResNet-18 backbone, with an impressive 12.51× speedup in theory and 11.75× storage saving. In particular, the KR-GAL and FR-GAL methods show strong generalization on multiple tasks such as Re-ID, image classification, object detection, and 3D point cloud processing.

A 15-Gbps BiCMOS XNOR gate for fast recognition of COVID-19 in binarized neural networks

The COVID-19 pandemic is spreading around the world causing more than 177 million cases and over 3.8 million deaths according to the European Centre for Disease Prevention and Control. The virus has devastating effects on economies, health, and well-being of worldwide population. Due to the high increase in daily cases, the available number of COVID-19 test kits in under-developed countries is scarce. Hence, it is vital to implement an effective screening method of patients using chest radiography since the equipment already exists. With the presence of automatic detection systems, any abnormalities in chest radiography that characterizes COVID-19 can be identified. Several artificial-intelligence algorithms have been proposed to detect the virus. However, neural networks training is considered to be time-consuming. Since computations in training neural networks are spent on floating-point multiplications, high computational power is required. Multipliers consume the most space and power among all arithmetic operators in deep neural networks. This paper proposes a 15 Gbps high-speed bipolar-complementary-metal-oxide-semiconductor (BiCMOS) exclusive-nor (XNOR) gate to replace multipliers in binarized neural networks. The proposed gate can be implemented on BiCMOS-based field-programmable gate arrays (FPGAs). This will significantly improve the response time in identifying chest abnormalities in CT scans and X-rays.

Hardware Platform-Aware Binarized Neural Network Model Optimization

Deep Neural Networks (DNNs) have shown superior accuracy at the expense of high memory and computation requirements. Optimizing DNN models regarding energy and hardware resource requirements is extremely important for applications with resource-constrained embedded environments. Although using binary neural networks (BNNs), one of the recent promising approaches, significantly reduces the design’s complexity, accuracy degradation is inevitable when reducing the precision of parameters and output activations. To balance between implementation cost and accuracy, in addition to proposing specialized hardware accelerators for corresponding specific network models, most recent software binary neural networks have been optimized based on generalized metrics, such as FLOPs or MAC operation requirements. However, with the wide range of hardware available today, independently evaluating software network structures is not good enough to determine the final network model for typical devices. In this paper, an architecture search algorithm based on estimating the hardware performance at the design time is proposed to achieve the best binary neural network models for hardware implementation on target platforms. With the XNOR-net used as a base architecture and target platforms, including Field Programmable Gate Array (FPGA), Graphic Processing Unit (GPU), and Resistive Random Access Memory (RRAM), the proposed algorithm shows its efficiency by giving more accurate estimation for the hardware performance at the design time than FLOPs or MAC operations.

Random and Systematic Variation in Nanoscale Hf0.5Zr0.5O2 Ferroelectric FinFETs: Physical Origin and Neuromorphic Circuit Implications

This work presents 2-bits/cell operation in deeply scaled ferroelectric finFETs (Fe-finFET) with a 1 µs write pulse of maximum ±5 V amplitude and WRITE endurance above 109 cycles. Fe-finFET devices with single and multiple fins have been fabricated on an SOI wafer using a gate first process, with gate lengths down to 70 nm and fin width 20 nm. Extrapolated retention above 10 years also ensures stable inference operation for 10 years without any need for re-training. Statistical modeling of device-to-device and cycle-to-cycle variation is performed based on measured data and applied to neural network simulations using the CIMulator software platform. Stochastic device-to-device variation is mainly compensated during online training and has virtually no impact on training accuracy. On the other hand, stochastic cycle-to-cycle threshold voltage variation up to 400 mV can be tolerated for MNIST handwritten digits recognition. A substantial inference accuracy drop with systematic retention degradation was observed in analog neural networks. However, quaternary neural networks (QNNs) and binary neural networks (BNNs) with Fe-finFETs as synaptic devices demonstrated excellent immunity toward the cumulative impact of stochastic and systematic variations.

Dual-configuration in-memory computing bitcells using SiOx RRAM for binary neural networks

Conventional DNN (deep neural network) implementations rely on networks with sizes in the order of MBs (megabytes) and computation complexity of the order of Tera FLOPs (floating point operations per second). However, implementing such networks in the context of edge-AI (artificial intelligence) poses limitations due to the requirement of high precision computation blocks, large memory requirement, and memory wall. To address this, low-precision DNN implementations based on IMC (in-memory computing) approaches utilizing NVM (non-volatile memory) devices have been explored recently. In this work, we experimentally demonstrate a dual-configuration XNOR (exclusive NOR) IMC bitcell. The bitcell is realized using fabricated 1T-1R SiOx RRAM (resistive random access memory) arrays. We have analyzed the trade-off in terms of circuit-overhead, energy, and latency for both IMC bitcell configurations. Furthermore, we demonstrate the functionality of the proposed IMC bitcells with mobilenet architecture based BNNs (binarized neural networks). The network is trained on VWW (visual wake words) and CIFAR-10 datasets, leading to an inference accuracy of ≈80.3% and ≈84.9%, respectively. Additionally, the impact of simulated BER (bit error rate) on the BNN accuracy is also analyzed.