A Joint Network Based CNN for Yoga Pose Classification and Scoring

Abstract

Comparing to traditional rehabilitation, rehabilitation at home becomes a need during pandemic. The technique brought up in this paper allows patients and yoga fans exercise at home with low cost and comfort while can also evaluate their postures. Previous works focus either on classifying poses or scoring on the sameness between the two input branches of patients’ poses and normative poses, but they ignore the combination of them in one single network. In this study, a residual block based Siamese CNN network with classification and scoring modules is proposed, aiming at providing accurate pose matching scores and classify pose types on yoga postures simultaneously. The Siamese network takes two inputs of learner’s pose and standard pose, which are preprocessed skeleton images by OpenPose. With the addition of residual block on the first convolutional module, back propagation is facilitated, which boosts up the process of updating parameters and optimization. The model calculates total loss by summing up cosine embedding loss and cross entropy loss in which the weight parameter lambda could be modified based on need. As for the scoring module, cosine similarity is used to calculate pose resemblance on batch level. The improvement in model performance is obvious when comparing the loss and accuracy between the Siamese network with residual block and VGG-16. Experimental results indicate that the residual block based Siamese network achieves competitive performance compared to the VGG-16 and can provide scoring feedback to learner’s yoga poses.