Multiperson interaction recognition in images: A body keypoint based feature image analysis

Abstract

AbstractMost interaction recognition approaches have been limited to single‐person action classification in videos. However, for still images where motion information is not available, the task becomes more complex. Aiming to this point, we propose an approach for multiperson human interaction recognition in images with keypoint‐based feature image analysis. Proposed method is a three‐stage framework. In the first stage, we propose feature‐based neural network (FCNN) for action recognition trained with feature images. Feature images are body features, that is, effective distances between a set of body part pairs and angular relation between body part triplets, rearranged in 2D gray‐scale image to learn effective representation of complex actions. In the later stage, we propose a voting‐based method for direction encoding to anticipate probable motion in steady images. Finally, our multiperson interaction recognition algorithm identifies which human pairs are interacting with each other using an interaction parameter. We evaluate our approach on two real‐world data sets, that is, UT‐interaction and SBU kinect interaction. The empirical experiments show that results are better than the state‐of‐the‐art methods with recognition accuracy of 95.83% on UT‐I set 1, 92.5% on UT‐I set 2, and 94.28% on SBU clean data set.