Effective Facial Expression Recognition Through Multimodal Imaging for Traumatic Brain Injured Patient’s Rehabilitation

Abstract

This article presents the issues related to applying computer vision techniques to identify facial expressions and recognize the mood of Traumatic Brain Injured (TBI) patients in real life scenarios. Many TBI patients face serious problems in communication and activities of daily living. These are due to restricted movement of muscles or paralysis with lesser facial expression along with non-cooperative behaviour, and inappropriate reasoning and reactions. All these aforementioned attributes contribute towards the complexity of the system for the automatic understanding of their emotional expressions. Existing systems for facial expression recognition are highly accurate when tested on healthy people in controlled conditions. However, their performance is not yet verified on the TBI patients in the real environment. In order to test this, we devised a special arrangement to collect data from these patients. Unlike the controlled environment, it was very challenging because these patients have large pose variations, poor attention and concentration with impulsive behaviours. In order to acquire high-quality facial images from videos for facial expression analysis, effective techniques of data preprocessing are applied. The extracted images are then fed to a deep learning architecture based on Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM) network to exploit the spatiotemporal information with 3D face frontalization. RGB and thermal imaging modalities are used and the experimental results show that better quality of facial images and larger database enhance the system performance in facial expressions and mood recognition of TBI patients under natural challenging conditions. The proposed approach hopefully facilitates the physiotherapists, trainers and caregivers to deploy fast rehabilitation activities by knowing the positive mood of the patients.