Common Sense Knowledge Infusion for Visual Understanding and Reasoning: Approaches, Challenges, and Applications

Abstract

Visual understanding involves detecting objects in a scene and investigating rich semantic relationships between the objects, which is required for downstream visual reasoning tasks. The scene graph is widely used for structured scene representation, however, the performance of the scene graph generation for visual reasoning is limited due to challenges posed by imbalanced datasets and insufficient attention toward common sense knowledge infusion. Most of the existing approaches use statistical or language priors for knowledge infusion. Common Sense knowledge infusion using heterogeneous knowledge graphs can help in improving the accuracy, robustness, and generalizability of the scene graph generation and enable explainable higher level reasoning by providing rich and diverse background and factual knowledge about the concepts in visual scenes. In this article, we present the background and applications of the scene graph generation and the initial approaches and key challenges in common sense knowledge infusion using heterogeneous knowledge graphs for visual understanding and reasoning.