3DPalsyNet: A Facial Palsy Grading and Motion Recognition Framework Using Fully 3D Convolutional Neural Networks

Gary Storey,Shelagh Keogh,Richard Jiang,Chang-Tsun Li,Ahmed Bouridane

doi:10.1109/access.2019.2937285

Abstract

The capability to perform facial analysis from video sequences has significant potential to positively impact in many areas of life. One such area relates to the medical domain to specifically aid in the diagnosis and rehabilitation of patients with facial palsy. With this application in mind, this paper presents an end-to-end framework, named 3DPalsyNet, for the tasks of mouth motion recognition and facial palsy grading. 3DPalsyNet utilizes a 3D CNN architecture with a ResNet backbone for the prediction of these dynamic tasks. Leveraging transfer learning from a 3D CNNs pre-trained on the Kinetics data set for general action recognition, the model is modified to apply joint supervised learning using center and softmax loss concepts. 3DPalsyNet is evaluated on a test set consisting of individuals with varying ranges of facial palsy and mouth motions and the results have shown an attractive level of classification accuracy in these tasks of 82% and 86% respectively. The frame duration and the loss function affect was studied in terms of the predictive qualities of the proposed 3DPalsyNet, where it was found shorter frame duration's of 8 performed best for this specific task. Centre loss and softmax have shown improvements in spatio-temporal feature learning than softmax loss alone, this is in agreement with earlier work involving the spatial domain.

Highlights

T HE task of action recognition is a computer vision problem that has been subject to a significant amount of research for varying actions types
RELATED WORK The task of action recognition is well established within the field of computer vision, with applications ranging from identifying sports based upon the movement of the participants [16] to human facial emotion recognition [17]
Unlike the methods applied in object detection which deals with only the spatial domain, the learning of discriminative temporal domain features from motion data across n frames of a video sequence adds further challenges to action recognition task

Summary

INTRODUCTION

T HE task of action recognition is a computer vision problem that has been subject to a significant amount of research for varying actions types. While the C3D method uses 3D convolutional layers to learn spatio-temporal features and has demonstrated good performance accuracy on the sport action data set, it does not generalise well to other more complex recognition tasks [2] This is mainly due to the relatively small video data sets available for optimising the large number of parameters in 3D CNNs. In addition the C3D network is shallow in comparison to the state-ofthe-art architectures used in image based recognition tasks where deeper networks have generally performed better. The research team developed a new multi-task framework for joint face detection and facial landmarks locating, namely Integrated Deep Model (IDM), which has been demonstrated with robust performance on face and landmark detection Based on this initial work, a further novel framework 3DPalsyNet, for facial palsy diagnosis is proposed, where the IDM is cascaded with two further specific 3D ResNet components that are designed to detect mouth motion and carry out palsy level grading, respectively.

RELATED WORK

EXPERIMENTAL EVALUATION

MOUTH MOTION RECOGNITION

Findings

CONCLUSIONS