Reinforcement Learning for Logic Recipe Generation: Bridging Gaps From Images to Plans

Abstract

It is a challenging task to produce recipes from images, due to the difficulty in bridging the gap from intuitive, static images to sequential, dynamic recipes. In this paper, we propose a novel recipe generation system for producing effective recipes from images. As medium steps, ingredient generation is introduced to guide recipe generation in our system. With potential information in ingredient lists, ingredient selection and ingredient sequence, the system is taught to generate effective recipes. For information representation, a hierarchical attention mechanism is designed to extract effective features for ingredient production and recipe generation. In order to guarantee the comprehensiveness and logic in recipes, a specific and explicit criterion around ingredients is designed under the framework of reinforcement learning. In ingredient generation, the system is required to generate ingredients with correct sequence in cooking procedures. And in recipe generation, ingredients in recipes are required to be consistent with produced ingredients. In experiments, the proposed method is compared with state-of-the-art methods to evaluate the feasibility. The results indicate that the proposed system achieves a better performance than other methods on both aspects of producing proper ingredients and effective recipes.