Gaussian process approach for metric learning

Abstract

Learning appropriate distance metric from data can significantly improve the performance of machine learning tasks under investigation. In terms of the distance metric representation forms in the models, distance metric learning (DML) approaches can be generally divided into two categories: parametric and non-parametric. The first category needs to make parametric assumption on the distance metric and learns the parameters, easily leading to overfitting and limiting model flexibility. The second category abandons the above assumption and instead, directly learns a non-parametric distance metric whose complexity can be adjusted according to the number of available training data, and makes the model representation relatively flexible. In this paper we follow the idea of the latter category and develop a non-parametric DML approach. The main challenge of our work concerns the formulation and learning of non-parametric distance metric. To meet this, we use Gaussian Process (GP) to extend the bilinear similarity into a non-parametric metric (here we abuse the concept of metric) and then learn this metric for specific task. As a result, our approach learns not only nonlinear metric that inherits the flexibility of GP but also representative features for the follow-up tasks. Compared with the existing GP-based feature learning approaches, our approach can provide accurate similarity prediction in the new feature space. To the best of our knowledge, this is the first work that directly uses GP as non-parametric metric. In the experiments, we compare our approach with related GP-based feature learning approaches and DML approaches respectively. The results demonstrate the superior performance of our approach.