Optical Flow Fusion Synthesis Based on Adversarial Learning from Videos for Facial Action Unit Detection

Abstract

AbstractHuman expression often happens simultaneously with head posture in real-time video, facial Action Units (AUs) is the key factor in facial expression detection. Optical flow can effectively capture weak motion displacements, that is, it can capture facial AUs caused by facial expressions. But the optical flow would produce a lot of noise, which would adverse the detection performance. To achieve a better facial AU detection performance, we propose a novel Optical Flow Synthesis Generative Adversarial Network (OFS-GAN). Firstly, we calculate the optical flow vector of the source frame and the target frame pair that randomly selected from video clips of facial expressions to promote the robustness of OFS-GAN. Secondly, in the generator of OFS-GAN, representation input into the encoder is yield by the optical flow vector concatenated with the source frame. The feature map output from the encoder is fed into the decoder to synthesize a target frame named generated target frame. In the end, through the comparison of the generated target frame and the target frame randomly selected from the target stage, OFS-GAN learns discriminative and robust facial action feature for facial AU detection following the principle of adversarial learning. Our novel OFS-GAN has been tested on DISFA+ and CK+ dataset with LOSO evaluation method. Qualitative results of our experiment demonstrate that OFS-GAN approaches or exceeds existing optical flow or deep learning algorithms.KeywordsAdversarial learningOptical flowGANAction unit detectionFacial expression recognsition