Attention Mechanism Based Mixture of Gaussian Processes

Abstract

• We connect the attention mechanism and mixture of Gaussian processes • We design two novel mixture of Gaussian processes models based on the attention mechanism. • The proposed models are applicable to the case that the input variable lies on a manifold or a graph. The mixture of Gaussian processes (MGP) is a powerful model, which is able to characterize data generated by a general stochastic process. However, conventional MGPs assume the input variable obeys certain probabilistic distribution, thus cannot effectively handle the case where the input variable lies on a general manifold or a graph. In this paper, we first clarify the relationship between the MGP prediction strategy and the attention mechanism. Based on the attention mechanism, we further design two novel mixture models of Gaussian processes, which do not rely on probabilistic assumptions on the input domain, thus overcoming the difficulty of extending MGP models to manifold or graph. Experimental results on real-world datasets demonstrate the effectiveness of the proposed methods.