Video-based social behavior recognition based on kernel relevance analysis

Abstract

This paper presents a kernel-based relevance analysis for video data to support social behavior recognition. Our approach, termed KRAV, is twofold: (i) A feature ranking based on centered kernel alignment (CKA) is carried out to match social semantic features with the output labels (individual and group behaviors). The employed method is an extension of the conventional CKA to mitigate the imbalance effect of unusual human behaviors. (ii) A classification stage to perform the behavior prediction. For concrete testing, the Israel Institute of Technology social behavior database is employed to assess the KRAV under a tenfold cross-validation scheme. Attained results show that the proposed approach for the individual recognition task obtains 0.5925 $$F_1$$ measure using 50 relevant features. Likewise, for the group recognition task obtains 0.8094 $$F_1$$ measure using 12 relevant features, which in both cases outperforms state-of-the-art results concerning the classification performance and number of employed features. Also, our video-based approach would assist further social behavior analysis from the set of features selected regarding the recognition of individual profiles and group behaviors.