Is the suggested food your desired?: Multi-modal recipe recommendation with demand-based knowledge graph

Abstract

Personalized recipe recommender systems help users mine certain dishes they want to find and even really desire, which play a significant role in matching dishes, balancing nutrients, and preventing non-communicable diseases. Generally, customer’s preferences or needs vary from person to person, and people are often reluctant to accept recommended food without reasonable explanation, especially when their demands are not explicitly addressed. In this paper, we are devoted to providing recipe suggestions accompanied by rational interpretations generated from images or videos. First, we construct a recipe knowledge graph (RcpKG) through the use of multi-modality and hierarchical thought, which focuses on the underlying demands of users and the consideration of multiple fine-grained factors. On this basis, a novel multi-modal recipe recommendation method via the knowledge graph (RcpMKR) is proposed, which represents nodes in multiple aspects and performs multi-relational graph structure extraction of the RcpKG. It not only takes into account local associations within the graph but also global information, and incorporates user concerns at different levels. Then, we adopt BERT-based multi-modal models and generative adversarial networks to generate interpretations. Additionally, dynamic convolution and random synthetic attention are utilized in our work to discriminate among features. Experimental results show that the proposed method and BERT-based fusion models improve recipe recommendation performance and explanation generation. Specifically, the precision of the RcpMKR method through RcpKG, user concerns and graph convolutional network improves by 7.82%, and the viExpCBTBERT method via 2D&3D convolutional neural networks for developing text interpretations enhances the F1-score by 10% compared with the baseline.