Multimodal Representation Learning for Recommendation in Internet of Things

Abstract

The recommender system has recently drawn a lot of attention to the communities of information services and mobile applications. Many deep learning-based recommendation models have been proposed to learn the feature representations from items. However, in Internet of Things (IoT), items’ description information are typically heterogeneous and multimodal, posing a challenge to items’ representation learning of recommendation models. To address this challenge and to improve the recommendation effectiveness in IoT, a novel multimodal representation learning-based model (MRLM) has been proposed. In MRLM, two closely related modules were trained simultaneously; they are global feature representation learning and multimodal feature representation learning. The former was designed to learn to accurately represent the global features of items and users through simultaneous training on three tasks: 1) triplet metric learning; 2) softmax classification; and 3) microscopic verification. The latter was proposed to refine items’ global features and to generate the final multimodal features by using items’ multimodal description information. After MRLM converged, items’ multimodal features and users’ global features could be used to calculate users’ preferences on items via cosine similarity. Through extensive experiments on two real-world datasets, MRLM remarkably improved the recommendation effectiveness in IoT.