Minimal Complexity Machines Under Weight Quantization

Abstract

Implementing machine learning models on resource-constrained platforms such as hardware devices requires sparse models that can generalize well. This article analyzes the effect of parameter (or weight) quantization on the performance, number of support vectors, model size in bits, $L_2$ L 2 norm and training time on various Support Vector Machines (SVM) and Minimal Complexity Machhines (MCM)-based kernel methods. We show that, Empirical Feature Space (EFS) and hinge loss-based MCM algorithms result in comparable accuracy, (8–190)x smaller model size in bits and (10k–16k)x smaller $L_2$ L 2 norm at full precision compared with LIBSVM. The Least Squares (LS) variants of MCM based methods results in $\approx$ ≈ 2% improvement in accuracy, upto 16x reduction in model size and upto 3x reduction in $L_2$ L 2 norm at full precision compared with its state-of-the-art counterpart Sparse Fixed Size variant of LS-SVM (SFS-LS-SVM). We quantize the weights of the compared variants post-training and demonstrate that our methods can retain their accuracies even with 7 bits as opposed to 10 and 14 bits used by LIBSVM and SFS-LS-SVM, respectively. Our experiments illustrate that quantization further improves upon the model sizes used by our methods by upto 300x and 30x compared with the LIBSVM and SFS-LS-SVM. This has significant implications for implementation in Internet of Things (IoT) devices, which benefit from model sparsity and good generalization.