DistVAE: Distributed Variational Autoencoder for sequential recommendation

Abstract

Recommender systems (RS) play a vital role in daily life due to their practical significance. As a branch of RS, the sequential recommendation has attracted much attention because of its effectiveness in helping people find items of interest. Recently, the generative methods based on Variational Autoencoder (VAE) have shown promising performance in modeling temporal dependencies among items in user sequences. These methods require collecting all users’ raw data, which incurs high costs and privacy issues. Therefore, this paper makes efforts to implement a distributed sequential VAE method. However, the distributed version of VAE-based sequential recommendation faces the following challenges. Firstly, in a distributed setting, there is a small amount of data on a single device, increasing the randomness of the gradients used to update the global model. Meanwhile, users in recommender systems usually have significantly different preference patterns. It may magnify the above randomness and make the global model update unstable. Secondly, existing generative methods usually adopt GRU to model latent variable dependencies, but GRU performs poorly in modeling the association in long sequences. This limitation makes these methods fail to model the long-term dependence in the inference model. To solve the problems, this paper proposes Distributed Variational AutoEncoder for the sequential recommendation (DistVAE), where a trusted server coordinates thousands of clients to train an insightful generative model without pulling their raw data together. Specifically, the clustering algorithm is integrated into the distributed sequential recommendation framework to dynamically find peers with similar preferences, and a sequential VAE is adopted to wrap a masked attention layer to model the long-term dependence. Experiments on four real-world datasets show that DistVAE outperforms other state-of-the-art centralized methods under the distributed training framework.