Gaze Estimation via the Joint Modeling of Multiple Cues

Abstract

How to automatically predict people’s gaze has attracted attention in the field of computer vision and machine learning. Previous studies on this topic set many constraints, such as restricted scenarios and strict and complex inputs. To mitigate these constraints to predict the gaze of people in more general scenarios, we propose a three-pathway network (TPNet) to estimate gaze via the joint modeling of multiple cues. Specifically, we first design a human-centric relationship inference (HCRI) module to learn the object-level relationship between the target person and the surrounding persons/objects in a scene. To the best of our knowledge, this is the first time that the object-level relationship is introduced into the gaze estimation task. Then, we construct a novel deep network with three pathways to fuse multiple cues, including scene saliency, object-level relationships and head information, to predict the gaze target. In addition, to extract the multilevel features during network training, we build and embed a micropyramid module in TPNet. The performance of TPNet is evaluated on two gaze estimation datasets: GazeFollow and DLGaze. A large number of quantitative and qualitative experimental results verify that TPNet can obtain robust results and significantly outperform the existing state-of-the-art gaze estimation methods. The code of TPNet will be released later.