Gradient Mismatch Research Articles

BinVPR: Binary Neural Networks towards Real-Valued for Visual Place Recognition.

Visual Place Recognition (VPR) aims to determine whether a robot or visual navigation system locates in a previously visited place using visual information. It is an essential technology and challenging problem in computer vision and robotic communities. Recently, numerous works have demonstrated that the performance of Convolutional Neural Network (CNN)-based VPR is superior to that of traditional methods. However, with a huge number of parameters, large memory storage is necessary for these CNN models. It is a great challenge for mobile robot platforms equipped with limited resources. Fortunately, Binary Neural Networks (BNNs) can reduce memory consumption by converting weights and activation values from 32-bit into 1-bit. But current BNNs always suffer from gradients vanishing and a marked drop in accuracy. Therefore, this work proposed a BinVPR model to handle this issue. The solution is twofold. Firstly, a feature restoration strategy was explored to add features into the latter convolutional layers to further solve the gradient-vanishing problem during the training process. Moreover, we identified two principles to address gradient vanishing: restoring basic features and restoring basic features from higher to lower layers. Secondly, considering the marked drop in accuracy results from gradient mismatch during backpropagation, this work optimized the combination of binarized activation and binarized weight functions in the Larq framework, and the best combination was obtained. The performance of BinVPR was validated on public datasets. The experimental results show that it outperforms state-of-the-art BNN-based approaches and full-precision networks of AlexNet and ResNet in terms of both recognition accuracy and model size. It is worth mentioning that BinVPR achieves the same accuracy with only 1% and 4.6% model sizes of AlexNet and ResNet.

Diluted binary neural network

Binary neural networks (BNNs) are promising on resource-constrained devices because they reduce memory consumption and accelerate inference effectively. However, they are still potential on performance improvement. Prior studies attribute performance degradation of BNNs to limited representation ability and gradient mismatch. In this paper, we find that it also results from the mandatory representation of small full-precision auxiliary weights to large values. To tackle with this issue, we propose an approach dubbed as Diluted Binary Neural Network (DBNN). Besides avoiding mandatory representation effectively, the proposed DBNN also alleviates sign flip problem to a large extent. For activations, we jointly minimize quantization error and maximize information entropy to develop the binarization scheme. Compared with existing sparsity-binarization approaches, DBNN trains network from scratch without other procedures and achieves larger sparsity. Experiments on several datasets with various networks demonstrate the superiority of our approach.

Regularizing Binary Neural Networks via Ensembling for Efficient Person Re-Identification

This study aims to leverage Binary Neural Networks (BNN) to learn binary hash codes for efficient person re-identification (ReID). BNNs, which use binary weights and activations, show promise in speeding up the inference time in deep models. However, BNNs typically suffer from performance degradation mainly due to the discontinuity of the binarization operation. Proxy functions have been proposed to calculate the gradients in the backward propagation, but they lead to the gradient mismatch problem. In this study, we propose to address the gradient mismatch problem by designing a multi-branch ensemble model consisting of many weak hash code learners. Specifically, our design aggregates the gradients from multiple branches, which allows a better approximation of the gradients and regularizes the network. Our model adds little computational cost to the baseline BNN since a vast amount of network parameters are shared between the weak learners. Combining the efficiency of the BNNs and hash code learning, we obtain an effective ensemble model which is efficient both in feature extraction and ranking phases. Our experiments demonstrate that the proposed model outperforms a single BNN by more than %20 using nearly the same amount of floating point operations. Moreover, the proposed model outperforms a conventional ensemble of BNN by more than %7 while being nearly 10x and 2x more efficient in terms of CPU consumption and memory footprint, respectively. We explore the performance of BNNs for efficient person ReID as one of the first systems available in the literature. Moreover, we adopt the proposed ensemble model for further validation of the image classification task and observe that our method effectively regularizes BNNs, providing robustness to hyperparameter selection and producing more consistent results under different settings.

Binarized Neural Network With Parameterized Weight Clipping and Quantization Gap Minimization for Online Knowledge Distillation

As the applications for artificial intelligence are growing rapidly, numerous network compression algorithms have been developed to restrict computing resources such as smartphones, edge, and IoT devices. Knowledge distillation (KD) leverages soft labels derived from a teacher model to a less parameterized model achieving high accuracy with reduced computational burden. Moreover, online KD provides parallel computing through collaborative learning between teacher and student networks, thus enhancing the training speed. A binarized neural network (BNN) offers an intriguing opportunity to facilitate aggressive compression at the expense of drastically degraded accuracy. In this study, two performance improvements are proposed for online KD when a BNN is applied as a student network: 1) parameterized weight clipping (PWC) to reduce dead weights in the student network and 2) quantization gap-aware adaptive temperature scheduling between the teacher and student networks. In contrast to constant weight clipping (CWC), PWC demonstrates a 3.78% top-1 test accuracy enhancement with trainable weight clipping by decreasing the gradient mismatch with CIFAR-10 dataset. Furthermore, the quantization gap-aware temperature scheduling increases the top-1 test accuracy by 0.08% over online KD at a constant temperature. By aggregating both methodologies, the top-1 test accuracy for CIFAR-10 dataset was 94.60%, and that for Tiny-ImageNet dataset was comparable to that of the 32-bit full-precision neural network.

Deep Network Quantization via Error Compensation.

For portable devices with limited resources, it is often difficult to deploy deep networks due to the prohibitive computational overhead. Numerous approaches have been proposed to quantize weights and/or activations to speed up the inference. Loss-aware quantization has been proposed to directly formulate the impact of weight quantization on the model's final loss. However, we discover that, under certain circumstances, such a method may not converge and end up oscillating. To tackle this issue, we introduce a novel loss-aware quantization algorithm to efficiently compress deep networks with low bit-width model weights. We provide a more accurate estimation of gradients by leveraging the Taylor expansion to compensate for the quantization error, which leads to better convergence behavior. Our theoretical analysis indicates that the gradient mismatch issue can be fixed by the newly introduced quantization error compensation term. Experimental results for both linear models and convolutional networks verify the effectiveness of our proposed method.

Error Correction Coding for One-Bit Quantization With CNN-Based AutoEncoder

Recently, a deep learning based error correction coding scheme is proposed to compensate for the severe distortion due to one-bit quantization. However, the fully-connected (FC) layers aided autoencoder is too heavy, resulting in high storage cost. In this letter, a novel convolutional autoencoder named ECCNet is introduced to lighten the scheme. Additionally, the soft quantization function is introduced to overcome the gradient mismatch. The squeeze and excitation (SE) block is applied for further performance boosting. Simulations show that the BER performance of the proposed ECCNet outperforms the previous state-of-the-art method under 16-QAM modulation with fewer parameters required. Furthermore, the proposed autoencoder design has impressive robustness in near Gaussian multi-path fading channels.

A Lightweight Method for Vehicle Classification Based on Improved Binarized Convolutional Neural Network

Vehicle classification is an important part of intelligent transportation. Owing to the development of deep learning, better vehicle classification can be achieved compared to traditional methods. Contemporary deep network models have huge computational scales and require a large number of parameters. Binarized convolutional neural networks (CNNs) can effectively reduce model computational size and the number of parameters. Most contemporary lightweight networks are binarized directly on a full-precision model, leading to shortcomings such as gradient mismatch or serious accuracy degradation. To addresses the inherent defects of binarization networks, herein, we adjust and improve residual blocks and propose a new pooling method, which is called absolute value maximum pooling (Abs-MaxPooling). The information entropy after weight binary quantization is used to propose a weight distribution binary quantization method. A binarized CNN-based vehicle classification model is constructed, and the weights and activation values of the model are quantified to 1 bit, which saves data storage space and improves classification accuracy. The proposed binarized model performs well on the BIT-Vehicle dataset and outperforms some full-precision models.

Step-Ahead Error Feedback for Distributed Training with Compressed Gradient

Although the distributed machine learning methods can speed up the training of large deep neural networks, the communication cost has become the non-negligible bottleneck to constrain the performance. To address this challenge, the gradient compression based communication-efficient distributed learning methods were designed to reduce the communication cost, and more recently the local error feedback was incorporated to compensate for the corresponding performance loss. However, in this paper, we will show that a new "gradient mismatch" problem is raised by the local error feedback in centralized distributed training and can lead to degraded performance compared with full-precision training. To solve this critical problem, we propose two novel techniques, 1) step ahead and 2) error averaging, with rigorous theoretical analysis. Both our theoretical and empirical results show that our new methods can handle the "gradient mismatch" problem. The experimental results show that we can even train faster with common gradient compression schemes than both the full-precision training and local error feedback regarding the training epochs and without performance loss.

A Novel Low-Bit Quantization Strategy for Compressing Deep Neural Networks.

The increase in sophistication of neural network models in recent years has exponentially expanded memory consumption and computational cost, thereby hindering their applications on ASIC, FPGA, and other mobile devices. Therefore, compressing and accelerating the neural networks are necessary. In this study, we introduce a novel strategy to train low-bit networks with weights and activations quantized by several bits and address two corresponding fundamental issues. One is to approximate activations through low-bit discretization for decreasing network computational cost and dot-product memory. The other is to specify weight quantization and update mechanism for discrete weights to avoid gradient mismatch. With quantized low-bit weights and activations, the costly full-precision operation will be replaced by shift operation. We evaluate the proposed method on common datasets, and results show that this method can dramatically compress the neural network with slight accuracy loss.

A 75.3-dB SNDR 24-MS/s Ring Amplifier-Based Pipelined ADC Using Averaging Correlated Level Shifting and Reference Swapping for Reducing Errors From Finite Opamp Gain and Capacitor Mismatch

This paper proposes averaging correlated level shifting (ACLS) and reference swapping (RS) techniques for simultaneously reducing errors from the finite opamp gain and capacitor mismatch in a pipelined analog-to-digital converter (ADC). The ACLS technique reduces the sensitivity of ADC accuracy to the opamp gain by averaging the finite opamp gain errors in two amplifying phases, where the error in the second amplifying phase is designed to have the opposite polarity to the one in the first amplifying phase. Meanwhile, ACLS also decreases the opamp’s thermal noise. In addition, the RS utilizes the averaging operation to reduce capacitor random mismatch error and combines a simple capacitor layout arrangement to decrease capacitor gradient mismatch error. Using the ACLS and RS techniques, a 16-bit ring amplifier-based pipelined ADC without calibration is realized in a 90-nm CMOS technology. Operating at 24 MS/s for a 10-MHz sine wave input, the proposed ADC achieves a 74.3-dB signal-to-noise-and-distortion ratio (SNDR) and 85.5-dB spurious free dynamic range (SFDR), and consumes 5.1 mW, yielding Walden and Schreier figure of merits of 50.1 fJ/conversion-step and 168 dB, respectively.

A mechanism assessment for the anti-corrosion of zirconia coating under the condition of subcritical water corrosion

Natural grown ZrO2 cannot effectively protect Zr alloy due to the poor phase stability. In this work, a pre-defected ZrO2 coating was fabricated on Zr alloy and further bombarded by Ar8+. A spontaneous martensitic transformation in ZrO2 coating was triggered by lattice distortion and gradient mismatch on the interface. Additionally, Ar8+ bombardment induced the formation of interstitial-vacancy pairs in ZrO2. DFT modeling proved that the interstitial-vacancy pairs were superior in stabilizing T-phase ZrO2, compared with native oxygen vacancies. A mechanism for enhanced corrosion resistance was proposed based on the unique interface and defect state.

A 12-Bit 500-MS/s Current Steering CMOS DAC for High-Speed PLC Modems

A 12-bit 500-MS/s current steering digital-to-analog converter (DAC) for high-speed power line communication (PLC) modems is presented in this paper. The performance of current steering DAC is limited by the current cell mismatches and glitch problems caused by switching timing errors. In this paper, the current cell design procedure is presented to minimize random mismatches. Then, a new data-weighted averaging (DWA) technique with fewer glitches and low hardware complexity is proposed to compensate for the gradient mismatch. Spurious-free dynamic range (SFDR) improvement and low complexity are effectively achieved by employing both a row–column structure and a (CSA) structure as the floor plan of the proposed DAC. The proposed DAC is implemented in a standard 0.18-[Formula: see text]m CMOS process with an active area of 2.445[Formula: see text]mm2, which achieves a differential non linearity (DNL) of 0.25[Formula: see text]LSB and an integral non-linearity (INL) of 0.19[Formula: see text]LSB. Additionally, the SFDR increases by 13.2[Formula: see text]dB (on average) when employing the proposed DWA technique. The total power consumption of the proposed DAC is 176[Formula: see text]mW from a 1.8-V supply voltage.

A 10-bit 200-kS/s SAR ADC IP core for a touch screen SoC

Based on a 5 MSBs (most-significant-bits)-plus-5 LSBs (least-significant-bits) C-R hybrid D/A conversion and low-offset pseudo-differential comparison approach, with capacitor array axially symmetric layout topology and resistor string low gradient mismatch placement method, an 8-channel 10-bit 200-kS/s SAR ADC (successive-approximation-register analog-to-digital converter) IP core for a touch screen SoC (system-on-chip) is implemented in a 0.18 μm 1P5M CMOS logic process. Design considerations for the touch screen SAR ADC are included. With a 1.8 V power supply, the DNL (differential non-linearity) and INL (integral non-linearity) of this converter are measured to be about 0.32 LSB and 0.81 LSB respectively. With an input frequency of 91 kHz at 200-kS/s sampling rate, the spurious-free dynamic range and effective-number-of-bits are measured to be 63.2 dB and 9.15 bits respectively, and the power is about 136 μW. This converter occupies an area of about 0.08 mm2. The design results show that it is very suitable for touch screen SoC applications.

A Low-Power Second-Order Two-Channel Time-Interleaved .SIGMA..DELTA. Modulator for Broadband Applications

Time-interleaving is an efficient approach to increase the effective sampling rate of the ΣΔ modulators, but time-interleaved (TI) ΣΔ modulators are sensitive to channel mismatch, which causes the quantization noise folded back into the band of interest. To reduce the folded noise caused by the channel mismatch of two-channel TI ΣΔ modulators, a low-power second-order two-channel TI ΣΔ modulator is proposed. The noise transfer function (NTF) of the modulator is a band-pass filter. By using this band-pass NTF, the folded noised can be reduced. The entire modulator can be implemented by employing three op-amps, which is beneficial for power consumption. The circuit of implementation for the proposed modulator is designed in 0.18µm COMS technology. The proposed modulator can achieve a SNDR of 78.9dB with a channel mismatch of 0.5% and a linear gradient mismatch of 0.4% for unity sampling capacitors. Monte Carlo simulation is done with a random Gaussian mismatch of 0.4% standard deviation for all capacitors, resulting in an average SNDR of 80.5dB. It is indicated that the proposed TI modulator is insensitive to the channel mismatch. The total power consumption is 19.5mW from a 1.8V supply.

Thermal and Ballistic Dynamic Analyses for Gun Ceramic Nozzles

The primary concern for the design and test of the ceramic gun nozzle is the combination of the thermal stress and ballistic dynamic stress. On the thermal analysis basis, a transient sequentially coupled finite element model (FEM) is performed to investigate the thermal stresses due to large temperature gradient and coefficient mismatch of thermal expansion between the ceramic nozzle and the steel holder. The fully coupled thermal stress analysis is conducted for verification. A DYNA3D model is used for interior ballistic analysis for the ceramic nozzle. Three candidate ceramic materials, SN47, STK4, and ZRO2, are investigated and compared with the 4340 steel nozzle. All the stress components from both thermal and dynamic loads are determined. These predictions are significant to the selected ceramic materials for the gun nozzle design.

Hemodynamic function of the standard St. Jude bileaflet disc valve has no clinical impact 10 years after aortic valve replacement

Objectives: Size mismatch and impaired left ventricular function have been shown to determine the hemodynamic function of the standard St. Jude bileaflet disc valve early after aortic valve replacement (AVR). We aimed to analyse St. Jude valve hemodynamic function and its clinical impact in the survivors of a prospective series 10 years after AVR for aortic stenosis. Design: Forty-three survivors aged 32–90 years from a prospective series attended a follow-up study with Doppler echo and radionuclide cardiography 10 years after AVR for aortic stenosis. Six patients with significant left sided valve regurgitation were excluded from further analysis: they had significantly lower St. Jude valve gradient and left ventricular ejection fraction (LVEF) and larger mass index (LVMi) than 37 without. Results: In the 37 patients without left sided valve regurgitation peak and mean gradients were inversely related to St. Jude valve geometric orifice area (GOA) indexed for either body surface area or left ventricular end-diastolic dimension (LVEDD). The gradients correlated directly with LVEDD but not with LVEF or LVMi. Eleven patients with hypertension had higher peak gradients (31±13 versus 22±8 mmHg, p < 0.05), lower LVEF, and higher LVEDD and LVMi than 26 without. Peak gradient was greater than 35 mmHg in five hypertensive patients with normal LVEF but lesser than 30 mmHg in six with impaired LVEF. Supranormal LVEF and severe size mismatch identified the remaining patients (N = 3) with peak gradient above 35 mmHg. In a multilinear regression analysis GOA indexed for LVEDD, hypertension, and LVEF were independently related to peak gradient. Conclusion: High gradients of the standard St. Jude bileaflet disc valve 10 years after AVR was primarily related to systemic hypertension and mismatch between valve and left ventricular cavity size. Hypertension and left sided valve regurgitation, but not St. Jude valve gradient or size mismatch, were the dominant determinants of left ventricular hypertrophy and impaired function.

Mutation detection in the breast cancer gene BRCA1 using the protein truncation test.

About 400 distinct mutations have been defined in the BRCA1 gene, and these are spread fairly evenly through the 5592 bp of coding DNA. This circumstance presents a formidable challenge for mutation screening. Apart from total direct sequencing, the preferred screening method has been single-strand conformation polymorphism (SSCP) gels, with a smaller input from constant denaturant gradient electrophoresis (CDGE), heteroduplex (HD) analysis, and mismatch cleavage. The protein truncation test (PTT) was used early in BRCA1 mutation screening but has not been widely adopted, perhaps because a straightforward analysis of the whole BRCA1 gene requires working with RNA and all its perceived problems. The present work was undertaken to assess the practicality of using the PTT under routine conditions for the screening of long genes such as BRCA1 that are not highly expressed in lymphocytes. We conclude that, provided RNA preparation is carried out effectively and consistently, the PTT approach has significant advantages over other methodologies such as SSCP gels.