Nonzero Bits Research Articles

Network Pruning for Bit-Serial Accelerators

Bit-serial architectures (BSAs) are becoming increasingly popular in low-power neural network processor (NNP) designs for edge scenarios. However, the performance and energy efficiency of state-of-the-art BSA NNPs heavily depends on both the proportion and distribution of ineffectual weight bits in neural networks (NNs). To boost the performance of typical BSA accelerators, we present Bit-Pruner, a software approach to learn BSA-favored NNs without resorting to hardware modifications. Bit-Pruner not only progressively prunes but also restructures the non-zero bits in weights so that the number of non-zero bits in the model can be reduced and the corresponding computing can be load-balanced to suit the target BSA accelerators. On top of Bit-Pruner, we further propose a Pareto frontier optimization algorithm to adjust the bit-pruning rate across network layers and fulfill diverse NN processing requirements in terms of performance and accuracy for various edge scenarios. However, an aggressive Bit-Pruner can lead to non-trivial accuracy loss especially for lightweight NNs and complex tasks. To this end, alternating direction method of multipliers (ADMM) is adapted to the retraining phase in Bit-Pruner to smooth the abrupt disturbance due to bit-pruning and enhance the resulting model accuracy. According to the experiments, Bit-Pruner increases the bit-sparsity up to 94.4% with negligible accuracy degradation and achieves optimized trade-off between NN accuracy and energy efficiency even under very-aggressive performance constraints. When pruned models are deployed onto typical BSA accelerators, the average performance is 2.1X and 1.6X higher than the baseline networks without pruning and those with classical weight pruning, respectively.

Ant-Colony-Algorithm-Based Intelligent Transmission Network Planning

Background: The efficiency of wireless sensor networks is limited by limitations in energy supply. Efficient routing strategies should be designed to save and balance the energy consumption of each node in a wireless sensor network. Aim: In this study, a transmission network based on an ant colony algorithm was proposed to meet the power load demands of a city. Objective: Based on the chaos ant colony algorithm, using a combination of wireless sensor network and node residual energy factors, a neighbor selection strategy was proposed. Results: The optimal result was 1896, and additional lines were: N14-15= 2, N4-16= 1, N5-12= 2, N7-13= L, N6-14= 1, N7-8. The coding method of solving transmission network planning based on multi-stage and multi-dimensional control variables was employed to decompose each control variable into two variables. The sum of total weight and non-zero bits was transformed into high-dimensional variables in state transition probability. Conclusion: The key analysis showed that the ant colony algorithm, as a simulated evolutionary algorithm, is an efficient internal heuristic method.

Structured Term Pruning for Computational Efficient Neural Networks Inference

The state-of-the-art convolutional neural network accelerators are showing a growing interest in exploiting the bit-level sparsity and eliminating the ineffectual computations of zero bits. However, the excessive redundancy and the irregular distribution of nonzero bits limit the real speedup in the accelerators. To address this, we propose an algorithm-architecture codesign, named structured term pruning (STP), to boost the computation efficiency of neural networks inference. Specifically, we enhance the bit sparsity by guiding the weights toward the value with fewer power-of-two terms. Then, we structure the terms with layer-wise group budgets. Retraining is adopted to recover the accuracy drop. We also design the hardware of the group processing element and the fast signed-digital encoder for efficient implementation of STP networks. The system design of STP is realized with some easy alterations on an input stationary systolic array design. Extensive evaluation results demonstrate that STP can reduce significant inference computation costs, and achieve <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.35\times $ </tex-math></inline-formula> computational energy saving for the ResNet18 network on the ImageNet dataset.

Products of Integers with Few Nonzero Digits

Abstract Let s(n) be the number of nonzero bits in the binary digital expansion of the integer n. We study, for fixed k, ℓ, m, the Diophantine system s(ab)= k, s(a)= ℓ, and s(b)= m in odd integer variables a, b.When k =2 or k = 3, we establish a bound on ab in terms of ℓ and m. While such a bound does not exist in the case of k =4, we give an upper bound for min{a, b} in terms of ℓ and m.

Hadamard Aperiodic Interval Codes for Parallel-Transmission 2D and 3D Synthetic Aperture Ultrasound Imaging

We present a new set of near orthogonal codes which we call Hadamard Aperiodic Interval (HAPI) codes and demonstrate their utility for parallel multi-transmitter synthetic aperture imaging. The codes are tri-state and sparse. Locations of non-zero bits are based on marks in a sequence of aperiodic intervals, also known as a Golomb ruler. The values of non-zero bits are selected from Hadamard sequences that are mutually orthogonal. This ensures that cross-correlation sidelobe magnitudes between differing codes are bounded by unity while the autocorrelation approaches a delta function with mainlobe-to-sidelobe levels scaling with the number of non-zero bits. We use simulations to demonstrate the potential of the codes for synthetic aperture imaging. A multiplicity of transmitter elements is used to transmit codes simultaneously, with a different code for each element. Echo signals are received from a multiplicity of transducer elements in parallel. Channel data from each receiver element are cross-correlated with respective HAPI codes to estimate the transmit–receive signature associated with each transmitter–receiver pair while minimizing crosstalk. This estimate of the full transmit–receive synthetic aperture dataset is then used to form high-quality images demonstrating image quality and signal-to-noise ratio improvements over multiple flash angle imaging and synthetic aperture imaging methods for linear arrays. We also demonstrate simulated full volume synthetic aperture imaging with random sparse arrays, possible with one extended HAPI code-set transmit event.

An Efficient FIR Filter Based on Hardware Sharing Architecture Using CSD Coefficient Grouping for Wireless Application

FIR filter is an essential part of digital signal processing that is extensively used in many areas such as wireless applications and digital processing system. The FIR filter design is inherently stable and has a linear phase characteristic under symmetric conditions, but its implementation often involves complexity and a large filter length to achieve specific design requirements. In this paper, the complexity of the FIR filter is reduced by eliminating the repeated subexpression in a canonic signed digit (CSD) number system based filter operation. A new grouping method has been proposed for the CSD number system-based filter coefficient to minimize the number of unpaired nonzero bits in the filter coefficient. The statistical analysis of the proposed grouping method is performed and compared with other existing schemes. The number of unpaired nonzero bits in the proposed grouping scheme is reduced by an average of 24.11\(\%\) as compared to other existing schemes. Further, an efficient FIR filter with hardware sharing architecture is designed and implemented to achieve a 14.65\(\%\) reduction in average power consumption, and the average operation speed is increased by 10.1\(\%\) compared to the other existing filter structures.

Deeper Weight Pruning Without Accuracy Loss in Deep Neural Networks: Signed-Digit Representation-Based Approach

In addition to the word-level weight pruning, which excludes the 0-value weights from the neural network inference computation, it is recently demonstrated that the bit-level weight pruning, which excludes the 0-bits in the weight value representation regardless of whether the weight values are zero or not, is very effective to further accelerate the neural network computation without accuracy loss. This work overcomes the inherent limitation of the bit-level weight pruning, that is, the maximal computation speedup is bounded by the total number of nonzero bits of the weights and the bound is invariably considered &#x201C;uncontrollable&#x201D; (i.e., constant) for the neural network to be pruned. Precisely, this work, based on the signed-digit encoding 1) proposes a transformation technique which converts the two&#x2019;s complement representation of every weight into a set of signed-digit representations of the minimal number of essential (i.e., nonzero) bits; 2) formulates the problem of selecting signed-digit representations of weights that maximize the parallelism of bit-level multiplication on the weights into a objective shortest path problem to achieve a maximal digit-index by digit-index (i.e., columnwise) compression for the weights and solves it efficiently using an approximation algorithm; 3) proposes a supporting novel acceleration architecture (<i>DWP</i>) with no additional inclusion of nontrivial hardware; and 4) proposes a variant of <i>DWP</i> to support bit-level parallel multiplication with the capability of predicting a tight worst-case latency of the parallel processing. Through experiments on several representative models using the ImageNet dataset, it is shown that our proposed approach is able to reduce the number of essential bits by 69&#x0025; on AlexNet, 74&#x0025; on VGG-16, and 68&#x0025; on ResNet-152, by which our accelerator is able to reduce the inference computation time by up to <inline-formula> <tex-math notation="LaTeX">$3.57\times $ </tex-math></inline-formula> over the conventional bit-level weight pruning.

Fast Ray-Scene Intersection for Interactive Shadow Rendering with Thousands of Dynamic Lights.

We present a method for the fast computation of the intersection between a ray and the geometry of a scene. The scene geometry is simplified with a 2D array of voxelizations computed from different directions, sampling the space of all possible directions. The 2D array of voxelizations is compressed using a vector quantization approach. The ray-scene intersection is approximated using the voxelization whose rows are most closely aligned with the ray. The voxelization row that contains the ray is looked up, the row is truncated to the extent of the ray using bit operations, and a truncated row with non-zero bits indicates that the ray intersects the scene. We support dynamic scenes with rigidly moving objects by building a separate 2D array of voxelizations for each type of object, and by using the same 2D array of voxelizations for all instances of an object type. We support complex dynamic scenes and scenes with deforming geometry by computing and rotating a single voxelization on the fly. We demonstrate the benefits of our method in the context of interactive rendering of scenes with thousands of moving lights, where we compare our method to ray tracing, to conventional shadow mapping, and to imperfect shadow maps.

Reconfigurable Low Pass FIR Filter Design using Canonic Signed Digit for Audio Applications

Objectives: This paper highlights the design of multiplier-less FIR filter. The binary coefficients are replaced by Canonic Signed Digit representation which reduces the complexity of the design. Methods/Statistical Analysis: In the current scenario more research is going on the optimization of Finite Impulse Response filters with less complex hardware design. The FIR filters performance depends on number of coefficient multipliers. The multipliers are expensive in terms delay area and power. In the CSD based filter, the number of non-zero bits is reduced. This proposed filter is designed in MATLAB, simulated in ISE environment and implemented on FPGA. Findings: The proposed filter is implemented on three FPGA devices, Xilinx's Spartan-3E, xc3s500e-4fg320, Virtex 2P, 2vp30ff1152-5 and Virtex 5P xc5v1x50t-3ff1136. Improvements: The designed structure uses reduced number of hardware components like slices, look up tables (LUTs) and flip-flops as compared to different structures and offers better performance.

Design of FRM-based MDFT filter banks in the canonic signed digit space using modified meta-heuristic algorithms

An optimal totally multiplier-less sharp transition width Modified Discrete Fourier Transform (MDFT) filter bank is proposed in this work. Since, here, the aim is to design a low complexity sharp transition width MDFT filter bank, the filters of the filter bank are designed using Frequency Response Masking technique to reduce the number of multipliers and hence the complexity. To replace multipliers with adders and shifters, the filter coefficients are converted into the Canonic Signed Digit space. In order to reduce the number of adders and shifters in the multiplier-less implementation, restricted number of non-zero bits is used in the conversion. This may degrade the performances of the MDFT filter bank. The performances are improved using modified meta-heuristic algorithms. A new hybrid algorithm which combines the qualities of Harmony Search Algorithm and Gravitational Search Algorithm is proposed in this work to optimise the performance of the MDFT filter bank.

희소한 부호 자리수 계수를 갖는 FIR 필터 설계

광대역 통신 모뎀이나 초고해상도 비디오 코덱 등과 같이 높은 데이터율을 갖는 시스템을 하드웨어로 구현할 때에는 디지털 필터의 고속 구현이 필수적이다. 디지털 필터의 임계경로는 대부분 MAC (multiplication and accumulation) 연산 회로이므로 필터 계수의 0이 아닌 비트의 갯수가 희소하다면 하드웨어 비용이 적은 덧셈기로도 디지털 필터를 고속으로 구현할 수 있다. 압축센싱은 신호의 희소 표현이나 희소 신호의 복원에 우수한 성능을 보임이 최근 연구에서 보고되고 있다. 본 논문에서는 압축센싱에 기반한 디지털 FIR 필터의 CSD (canonic signed digit) 계수를 찾는 방법을 제안한다. 주어진 주파수 응답과의 오차를 최소하면서 탐욕적 방법으로 희소한 0이 아닌 부호자리수를 찾고 잘못 선택되었던 부호자리수는 제거하는 과정을 반복한다. 설계 예를 통해 제안된 방법으로 희소한 0이 아닌 CSD 계수의 FIR 필터를 설계할 수 있음을 보인다. High speed implementation of digital filters is required in high data rate applications such as hard-wired wide band modem and high resolution video codec. Since the critical path of the digital filter is the MAC (multiplication and accumulation) circuit, the filter coefficient with sparse non-zero bits enables high speed implementation with adders of low hardware cost. Compressive sensing has been reported to be very successful in sparse representation and sparse signal recovery. In this paper a filter design method for digital FIR filters with CSD (canonic signed digit) coefficients using compressive sensing technique is proposed. The sparse non-zero signed bits are selected in the greedy fashion while pruning the mistakenly selected digits. A few design examples show that the proposed method can be utilized for designing sparse CSD coefficient digital FIR filters approximating the desired frequency response.

Canonic Signed Digit Recoding based RISC Processor Design

Objective: The objective of this work is to design a Reduced Instruction Set Computing (RISC) processor with Canonic Signed Digit (CSD) recoding. Methods: The incorporation of the CSD recording reduces the number of non-zero bits in the constant word length coefficient. The processor has a dedicated processing unit for the manipulation of floating point numbers. It uses CSD recoded number for the execution of the arithmetic operations such as multiplication. This novel technique reduces the switching activity and in turn resulting in reduction in the power consumed by the processor. Findings: The simulation was carried out using XILINX 14.3 and CADENCE NCLAUNCH. The RISC processor was synthesized with and without the CSD recoding. Although there is a slight increase in the area overhead, the use of ternary number representation in the processor design brought in a power reduction of 56.23%. Conclusions: The CSD recoding was found to be effective in terms of power consumption, making the RISC processor power efficient.

Design of low complexity sharp MDFT filter banks with perfect reconstruction using hybrid harmony-gravitational search algorithm

The design of low complexity sharp transition width Modified Discrete Fourier Transform (MDFT) filter bank with perfect reconstruction (PR) is proposed in this work. The current trends in technology require high data rates and speedy processing along with reduced power consumption, implementation complexity and chip area. Filters with sharp transition width are required for various applications in wireless communication. Frequency response masking (FRM) technique is used to reduce the implementation complexity of sharp MDFT filter banks with PR. Further, to reduce the implementation complexity, the continuous coefficients of the filters in the MDFT filter banks are represented in discrete space using canonic signed digit (CSD). The multipliers in the filters are replaced by shifters and adders. The number of non-zero bits is reduced in the conversion process to minimize the number of adders and shifters required for the filter implementation. Hence the performances of the MDFT filter bank with PR may degrade. In this work, the performances of the MDFT filter banks with PR are improved using a hybrid Harmony-Gravitational search algorithm.

Information--bit error rate and false positives in an MDS code

In this paper, a refinement of the weight distribution in an MDS code is computed. Concretely, the number of codewords with a fixed amount of nonzero bits in both information and redundancy parts is obtained. This refinement improves the theoretical approximation of the information-bit and -symbol error rate, in terms of the channel bit-error rate, in a block transmission through a discrete memoryless channel. Since a bounded distance reproducing encoder is assumed, the computation of the here-called false positive (a decoding failure with no information-symbol error) is provided. As a consequence, a new performance analysis of an MDS code is proposed.

Minimizing Design Costs of an FIR Filter Using a Novel Coefficient Optimization Algorithm

This work presents a novel coefficient optimization algorithm to reduce the area and improve the performance of finite impulse response (FIR) filter designs. Two basic architectures are commonly used in filters—direct and transposed. The coefficients of a filter can be encoded in the fewest possible nonzero bits using canonic signed digit (CSD) expressions. The proposed optimization algorithm can share common subexpressions (CS) and reduce the number of replicate operations that involve the CSD coefficients of filters with a transposed architecture. The effectiveness of the algorithm is confirmed by using filters with the collision detection multiple access (CDMA) standard, the 121-tap high-pass band, and 105- and 325-tap low-pass bands as benchmarks. For example, the proposed algorithm used in the optimization of 105-tap filter has a 30.44% smaller combinational logic area and a 16.69% better throughput/area than those of the best design that has been developed to date. Experimental results reveal that the proposed algorithm outperforms earlier designs.

A Novel Hybrid Monotonic Local Search Algorithm for FIR Filter Coefficients Optimization

This work presents a formulation of the FIR filter problem with sum of power-of-two (POT) coefficients as a mixed integer linear problem and solves it heuristically. The optimization problem is formulated to minimize the number of nonzero bits in each coefficient without violating the filter specifications within the pass and stop bands. A novel fast and efficient local search optimization algorithm for the filter coefficients is proposed. The algorithm called POTx does not use a tree structure in contrast to conventional MILP algorithms and offers fast computation because of a presorted search space, a monotonic dedicated search space, and the use of abort conditions. The proposed approach achieves comparable reductions to nonheuristic approaches because of a hybrid allocation scheme and multiple optimization iterations. The usefulness of the proposed algorithm for low power design of FIR filters is shown through the evaluation of several benchmark filters.

Optimized Architecture Using a Novel Subexpression Elimination on Loeffler Algorithm for DCT‐Based Image Compression

The canonical signed digit (CSD) representation of constant coefficients is a unique signed data representation containing the fewest number of nonzero bits. Consequently, for constant multipliers, the number of additions and subtractions is minimized by CSD representation of constant coefficients. This technique is mainly used for finite impulse response (FIR) filter by reducing the number of partial products. In this paper, we use CSD with a novel common subexpression elimination (CSE) scheme on the optimal Loeffler algorithm for the computation of discrete cosine transform (DCT). To meet the challenges of low‐power and high‐speed processing, we present an optimized image compression scheme based on two‐dimensional DCT. Finally, a novel and a simple reconfigurable quantization method combined with DCT computation is presented to effectively save the computational complexity. We present here a new DCT architecture based on the proposed technique. From the experimental results obtained from the FPGA prototype we find that the proposed design has several advantages in terms of power reduction, speed performance, and saving of silicon area along with PSNR improvement over the existing designs as well as theXilinx core.

Efficient Logarithmic Converters for Digital Signal Processing Applications

The hardware computation of the logarithm function is required in a multitude of applications. This brief investigates logarithmic converters based on piecewise linear approximations. This brief presents a rigorous technique, based on mixed-integer linear programming, to obtain optimal coefficients' values, which minimize the maximum relative approximation error while using a reduced number of nonzero bits for the coefficients. The proposed method results in a sensible reduction of the relative approximation error, as compared with previously published results. The hardware implementation realizes the multiplication by a few shifts and additions, avoiding the use of full multipliers. Implementation details and synthesis results in a 90-nm CMOS technology are also described in this brief.

On “A New Common Subexpression Elimination Algorithm for Realizing Low-Complexity Higher Order Digital Filters”

A thorough analysis of the paper above revealed several controversial arguments about the superiority of binary representation over canonical signed digits (CSD) for common subexpression elimination (CSE). It was improper to model the number of logic operators (LO) required after CSE as a linear sum of independently weighted numbers of nonzero bits, common subexpressions and unpaired bits. The logic depth (LD) penalty of binary CSE had been deemphasized by the errors in the reported LD. This comment corrects the LD of contention resolution algorithm, and points out some contradictions with reference to the latest experimentation of binary, CSD and minimal signed digit number representations for CSE. Upon correcting the error in the reported filter lengths for different stopband attenuations of digital advanced mobile phone system specification, the LO and LD data of the CSE algorithms compared in the above paper are recalculated using the corrected filter coefficient sets.

On the Computational Sequence of Scalar Multiplication with Left-to-Right Recoded NAF and Sliding Window Technique

Simple power analysis (SPA) can be employed in examining the power consumption trace of elliptic curve scalar multiplioation to retrieve the computational sequence. However, SPA cannot distinguish point addition from point subtraction. The attacker still requires an exhaustive search to recover the private key when it is recoded in NAF or recoded by the 2-bit sliding window method. The average Hamming weight of an n-bit NAF recoded scalar is n/3, and an exhaustive search among the 2 n/3 candidates is required. This paper shows that in a left-to-right NAF recoded or a left-to-right 2-bit sliding window manipulated scalar the relative position of nonzero bits will reveal their values. Our analysis skill reduces the number of candidates of the scalar from the naive search of 2 n/3 to 2 2n/9 and 2 0.19n respectively for the cases of NAF and sliding window method.