How Many Frames Does Facial Expression Recognition Require?

Abstract

Facial expression analysis is essential to enable socially intelligent processing of multimedia video content. Most facial expression recognition algorithms generally analyze the whole image sequence of an expression to exploit its temporal characteristics. However, it is seldom studied whether it is necessary to utilize all the frames of a sequence, since human beings are able to capture the dynamics of facial expressions from very short sequences (even only one frame). In this paper, we investigate the impact of the number of frames in a facial expression sequence on facial expression recognition accuracy. In particular, we develop a key frame selection method through key point based frame representation. Experimental results on the popular CK facial expression dataset indicate that recognition accuracy achieved with half of the sequence frames is comparable to that of utilizing all the sequence frames. Our key frame selection method can further reduce the number of frames without clearly compromising recognition accuracy.