Prompt guidance query with cascaded constraint decoders for human–object interaction detection

Abstract

AbstractHuman–object interaction (HOI) detection, which localises and recognises interactions between human and object, requires high‐level image and scene understanding. Recent methods for HOI detection typically utilise transformer‐based architecture to build unified future representation. However, these methods use random initial queries to predict interactive human–object pairs, leading to a lack of prior knowledge. Furthermore, most methods provide unified features to forecast interactions using conventional decoder structures, but they lack the ability to build efficient multi‐task representations. To address these problems, we propose a novel two‐stage HOI detector called PGCD, mainly consisting of prompt guidance query and cascaded constraint decoders. Firstly, the authors propose a novel prompt guidance query generation module (PGQ) to introduce the guidance‐semantic features. In PGQ, the authors build visual‐semantic transfer to obtain fuller semantic representations. In addition, a cascaded constraint decoder architecture (CD) with random masks is designed to build fine‐grained interaction features and improve the model's generalisation performance. Experimental results demonstrate that the authors’ proposed approach obtains significant performance on the two widely used benchmarks, that is, HICO‐DET and V‐COCO.