Facial emotion detection using thermal and visual images based on deep learning techniques

Abstract

ABSTRACT The main objectives of this study are (i) to determine the different emotions, such as happy, anger, neutral, and sad, based on the visual and thermal images for facial expression recognition of healthy individuals; (ii) to train the modified pre-trained models, such as DenseNet-121, ResNet-50, and VGG-19 based on transfer learning technique, using the visual and thermal image dataset, and (iii) to compare the three different customize model with that of three pre-trained models in terms of accuracy. The customized CNN models were proposed to classify the four classes of emotions with high efficiency. Among thermal and visual image dataset, the thermal images have produced highest classification accuracy of 95.8% using the Customized Net-3 model and have outperformed the three modified pre-trained models while classifying the four facial expressions. Thus, the proposed customized CNN model is proven effective for facial emotion recognition based on thermal imaging using deep learning techniques.