Third-Person View Attention Prediction in Natural Scenarios with Weak Information Dependency and Human-Scene Interaction Mechanism

Abstract

First-person view attention has been widely studied in computer science domain since 1990s while third-person view attention in natural scenarios begins to gain the intensive interest until 2015. This paper focuses on the problem of third-person view attention prediction in natural scenarios where a human freely performs daily activities without constraints. To handle the two insuffiencies of existing methods: (i) assuming some extra information (except for input images) are given in advance and (ii) ignoring the importance of human-scene interaction, this paper proposes a model with weak information dependency, which helps to alleviate annotation costs. In addition, a transformer-based human-scene interaction mechanism is proposed to explore the global and long-dependency contexts between the human and scene. The pipeline of the proposed model is firstly extracting human and scene features, then inferring human attention probability map by fusing human and scene features via a transformer-based network, and finally predicting human attention object based on human attention probability map and object detection. The experiments on two public datasets validate the effectiveness of our model.