NSVQ: Noise Substitution in Vector Quantization for Machine Learning

Mohammad Hassan Vali,Tom Backstrom

doi:10.1109/access.2022.3147670

Mohammad Hassan Vali, Tom Backstrom

Open Access

https://doi.org/10.1109/access.2022.3147670

Copy DOI

Abstract

Machine learning algorithms have been shown to be highly effective in solving optimization problems in a wide range of applications. Such algorithms typically use gradient descent with backpropagation and the chain rule. Hence, the backpropagation fails if intermediate gradients are zero for some functions in the computational graph, because it causes the gradients to collapse when multiplying with zero. Vector quantization is one of those challenging functions for machine learning algorithms, since it is a piece-wise constant function and its gradient is zero almost everywhere. A typical solution is to apply the straight through estimator which simply copies the gradients over the vector quantization function in the backpropagation. Other solutions are based on smooth or stochastic approximation. This study proposes a vector quantization technique called NSVQ, which approximates the vector quantization behavior by substituting a multiplicative noise so that it can be used for machine learning problems. Specifically, the vector quantization error is replaced by product of the original error and a normalized noise vector, the samples of which are drawn from a zero-mean, unit-variance normal distribution. We test our proposed NSVQ in three scenarios with various types of applications. Based on the experiments, the proposed NSVQ achieves more accuracy and faster convergence in comparison to the straight through estimator, exponential moving averages, and the MiniBatchKmeans approaches.

Highlights

M ACHINE learning is one of the most significant and potent technological advancements in recent years [1], [2]
We introduce a novel vector quantization method called noise substitution in vector quantization (NSVQ), in which the vector quantization error is simulated with the product of the original quantization error magnitude with a normalized noise vector, the components of which are drawn from a zero-mean, unit-variance normal distribution
As explained in section IV, we analyzed the performance of our proposed noise substitution in vector quantization (NSVQ) and straight through estimator (STE) for three different scenarios

Summary

Introduction

M ACHINE learning is one of the most significant and potent technological advancements in recent years [1], [2]. Especially those based on neural networks, have been shown to be highly efficient and successful in a wide range of realworld applications such as speech enhancement [3], speech recognition [4], [5], natural language processing [6], [7], and computer vision [8]–[11]. With this great potential entailed by these applications, we can expect that machine learning can be used to improve efficiency in a wide range of future applications. In other words, learning is not feasible if there are functions with zero (or undefined) gradient in the computational graph, since this would cause the gradients to collapse when multiplying them with zero (or None) based on the chain rule gradient calculation

Objectives

Methods

Results

Conclusion