Participants-based Synchronous Optimization Network for skeleton-based action recognition

Abstract

Nowadays, graph convolutional networks are widely used in skeleton-based action recognition. However, these methods ignore the difference between main participant and subordinate participant, as well as the consistency and causality reasoning in human–human interactive actions. In this paper, we construct a novel Participants-based Synchronous Optimization Network (PSONet). Firstly, we construct main participant branch, subordination participant branch and relative movements branch for the individual and interactive information of participants. Secondly, in the training process, Participants-based Synchronous Response (PSR) loss is constructed to optimize our network. Online mutual response mechanism in PSR regulates the consistency and captures the causality between the main participant action and the overall interactive action. Joint cross-entropy loss in PSR is used to constrain the action instances with individual and interactive action information. Finally, Representative Temporal Enhanced (RTE) block is proposed to complement representative temporal aggregation features and enhance the spatial modeling of representative temporal frames. Experiments have been conducted on the NTU RGB+D 60 dataset and the NTU RGB+D 120 dataset, which have verified that PSONet outperforms state-of-the-art methods.