Human–object interaction detection with missing objects

Abstract

Human–object interaction (HOI) detection is an important vision task that requires the detection of individual object instances and reasoning of their relations. Despite encouraging advancement in recent years, past methods are still limited to relatively simple images where the human and object instances can be detected without difficulties. HOI in the wild should work even when the objects that a person is interacting with are not visible or hard to detect in the image. In this paper, we formulate HOI with missing objects (HOI-MO) as a research problem, and show that it is indeed critical as many such instances can be found even in the commonly-used public HOI detection datasets. We label these to compose new test sets for the proposed method. To our knowledge, we introduce the first method for such challenging HOI detection that incorporates global scene information. The effectiveness and superiority of the proposed method are demonstrated through extensive experiments and comparisons.