Easy-But-Effective Domain Sub-Similarity Learning for Transfer Regression

Abstract

Transfer covariance function, which can model domain similarity and adaptively control the knowledge transfer across domains, is widely used in transfer learning. In this paper, we concentrate on Gaussian process ( <i>GP</i> ) models using a transfer covariance function for regression problems in a black-box learning scenario. Precisely, we investigate a family of rather general transfer covariance functions, <inline-formula><tex-math notation="LaTeX">${T}_{*}$</tex-math></inline-formula> , that can model the heterogeneous sub-similarities of domains through multiple kernel learning. A necessary and sufficient condition to obtain valid <i>GP</i> s using <inline-formula><tex-math notation="LaTeX">${T}_{*}$</tex-math></inline-formula> ( <inline-formula><tex-math notation="LaTeX">$GP_{T_{*}}$</tex-math></inline-formula> ) for any data is given. This condition becomes specially handy for practical applications as (i) it enables semantic interpretations of the sub-similarities and (ii) it can readily be used for model learning. In particular, we propose a computationally inexpensive model learning rule that can explicitly capture different sub-similarities of domains. We propose two instantiations of <inline-formula><tex-math notation="LaTeX">$GP_{T_{*}}$</tex-math></inline-formula> , one with a set of predefined constant base kernels and one with a set of learnable parametric base kernels. Extensive experiments on 36 synthetic transfer tasks and 12 real-world transfer tasks demonstrate the effectiveness of <inline-formula><tex-math notation="LaTeX">$GP_{T_{*}}$</tex-math></inline-formula> on the sub-similarity capture and the transfer performance.