HL-Net: Heterophily Learning Network for Scene Graph Generation

Abstract

Scene graph generation (SGG) aims to detect objects and predict their pairwise relationships within an image. Current SGG methods typically utilize graph neural net-works (GNNs) to acquire context information between ob-jects/relationships. Despite their effectiveness, however, current SGG methods only assume scene graph homophily while ignoring heterophily. Accordingly, in this paper, we propose a novel Heterophily Learning Network (HL-Net) to comprehensively explore the homophily and heterophily be-tween objects/relationships in scene graphs. More specif-ically, HL-Net comprises the following 1) an adaptive reweighting transformer module, which adaptively inte-grates the information from different layers to exploit both the heterophily and homophily in objects; 2) a relation-ship feature propagation module that efficiently explores the connections between relationships by considering het-erophily in order to refine the relationship representation; 3) a heterophily-aware message-passing scheme to fur-ther distinguish the heterophily and homophily between ob-jects/relationships, thereby facilitating improved message passing in graphs. We conducted extensive experiments on two public datasets: Visual Genome (VG) and Open Images (OI). The experimental results demonstrate the superiority of our proposed HL-Net over existing state-of-the-art approaches. In more detail, HL-Net outperforms the second-best competitors by 2.1% on the VG datasetfor scene graph classification and 1.2% on the IO dataset for the final score. Code is available at https://github.com/simI3/HL-Net.