BabyPose: Real-Time Decoding of Baby’s Non-Verbal Communication Using 2D Video-Based Pose Estimation

Abstract

People use various communication methods to interact with each other, such as speech and body movements. However, communication methods in babies differ according to their developmental periods. Babies may not be able to express themselves verbally, but they do have their own communication methods. Naturally, it becomes important for parents to understand these signs/poses. Human pose estimation is extensively employed in different applications such as video surveillance, sports analysis and medical support/aid. The goal of this research is to assist new parents by addressing pose-based real-time body movements of babies (such as arching back, head banging, kicking legs, rubbing eyes, stretching, sucking fingers) and making sense of their activities. This is the first study to estimate poses on babies. In this paper, a baby pose dataset is created from 156 video clips through online video sharing platforms. Key-points are obtained from three different pose estimators - OpenPose, AlphaPose, and KAPAO. Different LSTM models are used to recognize the babies’ activities and different performance metrics are used to compare these models. The best model has 99% accuracy and 0.0712 loss ratio. Also, babies are tracked in real-time via DeepSORT algorithm. Experimental results show that the proposed system is very promising and filling a gap in making sense of baby poses and monitoring them.