Siamese Architecture-Based 3D DenseNet with Person-Specific Normalization Using Neutral Expression for Spontaneous and Posed Smile Classification

Kunyoung Lee,Eui Chul Lee

doi:10.3390/s20247184

Abstract

Clinical studies have demonstrated that spontaneous and posed smiles have spatiotemporal differences in facial muscle movements, such as laterally asymmetric movements, which use different facial muscles. In this study, a model was developed in which video classification of the two types of smile was performed using a 3D convolutional neural network (CNN) applying a Siamese network, and using a neutral expression as reference input. The proposed model makes the following contributions. First, the developed model solves the problem caused by the differences in appearance between individuals, because it learns the spatiotemporal differences between the neutral expression of an individual and spontaneous and posed smiles. Second, using a neutral expression as an anchor improves the model accuracy, when compared to that of the conventional method using genuine and imposter pairs. Third, by using a neutral expression as an anchor image, it is possible to develop a fully automated classification system for spontaneous and posed smiles. In addition, visualizations were designed for the Siamese architecture-based 3D CNN to analyze the accuracy improvement, and to compare the proposed and conventional methods through feature analysis, using principal component analysis (PCA).

Highlights

As indicators of human emotions, facial expressions are the most informative nonverbal behavior.as autonomic nervous system responses, facial movements are autonomously expressed, even when unintended
We investigated the use of a 3D convolutional neural network (CNN) model to classify posed and spontaneous smiles
We proposed a Siamese architecture-based 3D DenseNet using a neutral expression as an anchor for spontaneous and posed smile classification

Summary

Introduction

As indicators of human emotions, facial expressions are the most informative nonverbal behavior.as autonomic nervous system responses, facial movements are autonomously expressed, even when unintended. As indicators of human emotions, facial expressions are the most informative nonverbal behavior. Facial expressions are powerful visual indicators of the emotions and psychological state of individuals [1]. We investigated spontaneous and posed smile classification, which is one way of inferring human psychological states through facial expression analysis. It is a smile that is expressed autonomously by emotional stimulation, such as watching a comedy video. A neutral expression is a face in a non-expressive state, in which the emotional state is not revealed [5]. The use of the neutral expression as a standard for facial expression measurement is a well-known method in face

Methods

Results

Discussion

Conclusion