Classroom teacher action recognition based on spatio-temporal dual-branch feature fusion

Abstract

The classroom teaching action recognition task refers to recognizing and understanding teacher action through video temporal and spatial information. Due to complex backgrounds and significant occlusions, recognizing teacher action in the classroom environment poses substantial challenges. In this study, we propose a classroom teacher action recognition approach based on a spatio-temporal dual-branch feature fusion architecture, where the core task involves utilizing continuous human keypoint heatmap information and single-frame image information. Specifically, we fuse features from two modalities to propose a method combining image spatial information with temporal human keypoint heatmap information for teacher action recognition. Our approach ensures recognition accuracy while reducing the model’s parameters and computational complexity. Additionally, we constructed a teacher action dataset (CTA) in a real classroom environment, comprising 12 action categories, 13k+ video segments, and a total duration exceeding 15 h. The experimental results on the CTA dataset validate the effectiveness of our proposed method. Our research explores action recognition tasks in real complex classroom environments, providing a technical framework for classroom teaching intelligent analysis.