Automatic Degree Evaluation of Facial Nerve Paralysis Based on Triple-stream Long Short Term Memory

Abstract

Facial nerve paralysis adversely affects both mental and physical health of patients. Several degree evaluation methods for facial paralysis have been put forward based on the static facial asymmetry and computer vision technologies. However, traditional methods still suffer from two drawbacks: (1) the facial movement information is ignored which could be important for facial paralysis analysis; (2) the shallow machine learning models used by them may not well extract effective facial features for facial paralysis of different grades. In this paper, we present Triple-stream Long Short Term Memory (LSTM) to evaluate the severity degree of facial paralysis automatically. Hence, the Triple-stream LSTM employs two LSTM sub-networks to extract local movement features from the two involved facial regions and one LSTM sub-network to extract global features from the combination of the two regions. Then, the extracted high level representations from the three LSTMs are fused for final severity degree evaluation. By extracting both local and global movement features from the involved regions in an optimal sense, the final fused features could express the facial features for different degrees of facial nerve paralysis more effectively. The experimental results have verified the effectiveness of Triple-stream LSTM for evaluating the severity of facial paralysis.