A stochastic approximation approach to simultaneous feature weighting and selection for nearest neighbour learners

Abstract

Nearest neighbour (NN) learners are some of the most popular methods in supervised machine learning with feature selection and weighting being two fundamental tools for improving their performance. In this study, we introduce a novel methodology for simultaneous feature selection and weighting for NN learners based on simultaneous perturbation stochastic approximation (SPSA). This is a pseudo-gradient descent optimisation algorithm that approximates gradient information from noisy objective function measurements without a need for an explicit functional form of the objective function nor its derivatives. In particular, we show how the process of simultaneous feature selection and weighting can be optimised within a stochastic approximation framework with repeated cross-validation (CV) performance as the objective function, which we call SPSA-FWS. We provide extensive computational experiments for assessment of this approach and we compare performance of SPSA-FWS to other feature weighting methods. Our results indicate that SPSA-FWS outperforms existing feature weighting algorithms for the most part and it stands as a competitive new method for this task. Specifically, when compared against its unweighted counterpart and feature weighting alone with 5-repeated 5-fold CV accuracy being the performance metric, SPSA-FWS provides improvements of up to 177.88% with a mean improvement of 20.52% for classification tasks and a mean improvement of 0.19 in the R-Squared metric for regression tasks respectively. In addition to its superior performance, SPSA-FWS has two attractive features: (1) it can be used in conjunction with any performance metric and any variant of nearest neighbour learners, and (2) it can be hybridised with other feature weighting methods.