A Generalized Meta-loss Function for Distillation Based Learning Using Privileged Information for Classification and Regression

Abstract

Learning using privileged information (LUPI) is a powerful heterogeneous feature space machine learning framework that allows models to learn from highly informative (privileged) features which are available during training only. These models then generate test predictions using input space features which are available during both training and testing. LUPI can significantly improve prediction performance in a variety of machine learning problems. However, existing large margin and neural network implementations of learning using privileged information are mostly designed for classification tasks. In this work, we propose a simple yet effective formulation that allows general application of LUPI to classification, regression, and other related problems. We have verified the correctness, applicability, and effectiveness of our method on regression and classification problems over different synthetic and real-world problems. To test the usefulness of the proposed model in real-world problems, we have further evaluated our method on the problem of protein binding affinity prediction where the proposed scheme has shown to outperform the current state-of-the-art predictor.