Improving human–object interaction with auxiliary semantic information and enhanced instance representation

Abstract

Human–Object Interaction (HOI) detection has garnered considerable attention among computer vision researchers as it involves identifying and describing actions between humans and objects. Numerous approaches, such as sequential and end-to-end methods, have been proposed to tackle this problem, with a recent focus on exploring end-to-end systems. This study presents an enhanced end-to-end transformer-based human–object detector based on HOTR, which introduces three improvements. The proposed model improves instance representation through a simple yet effective mechanism, utilizes semantic information to provide contextual understanding and additional knowledge, and incorporates a cross-attention mechanism for fusing multi-level high-level feature maps within the Transformer architecture. Experimental results demonstrate significant performance gains over the baseline HOTR model, making it competitive with other state-of-the-art models on two widely-used datasets: V-COCO and HICO-DET.