An Application of Machine Learning and Image Processing to Automatically Detect Teachers’ Gestures

Abstract

Providing teachers with detailed feedback about their gesticulation in class requires either one-on-one expert coaching, or highly trained observers to hand code classroom recordings. These methods are time consuming, expensive and require considerable human expertise, making them very difficult to scale to large numbers of teachers. Applying Machine Learning and Image processing we develop a non-invasive detector of teachers’ gestures. We use a multi-stage approach for the spotting task. Lessons recorded with a standard camera are processed offline with the OpenPose software. Next, using a gesture classifier trained on a previous training set with Machine Learning, we found that on new lessons the precision rate is between 54 and 78%. The accuracy depends on the training and testing datasets that are used. Thus, we found that using an accessible, non-invasive and inexpensive automatic gesture recognition methodology, an automatic lesson observation tool can be implemented that will detect possible teachers’ gestures. Combined with other technologies, like speech recognition and text mining of the teacher discourse, a powerful and practical tool can be offered to provide private and timely feedback to teachers about communication features of their teaching practices.