Detecting human-object interactions in videos by modeling the trajectory of objects and human skeleton

Abstract

This article focuses on the task of detecting human-object interactions (HOI) in videos, with the goal of identifying objects interacting with humans and predicting human-object interaction classes. Two frameworks are proposed which detect human-object interactions in videos by modeling the trajectory of objects and human skeleton. The first framework (knowledge-based spatial–temporal HOI) treats the entire scene to be a HOI graph made up of the human skeleton and objects. It has fewer parameters and a higher possibility for knowledge embedding. The second framework (hierarchical spatial–temporal HOI) constructs a HOI graph after obtaining the feature of the human skeleton and objects. It outperforms the competition in terms of performance and generalization. Experimental results in CAD-120 dataset and SYSU-HOI dataset show that the proposed frameworks are more advanced than the state-of-the-art methods, with smaller parameters and shorter inference time. Such results confirm that the proposed frameworks effectively reduce parameters and inference time while maintaining detection accuracy in HOI videos.