Exploring Domain Knowledge for Facial Expression-Assisted Action Unit Activation Recognition

Abstract

Current works on facial action unit (AU) activation recognition typically include supervised training using AU-annotated training images. Compared to facial expression labeling, AU annotation is a time-consuming, expensive, and error-prone process. Domain knowledge refers to the strong probabilistic dependencies between facial expressions and AUs, as well as dependencies among AUs. To take advantage of this, we avoid the time-consuming process of AU annotation and introduce a new AU activation recognition method that learns AU classifiers from domain knowledge, and requires only expression-annotated facial images. Specifically, we first generate pseudo AU labels according to the probabilistic dependencies between expressions and AUs as well as correlations among AUs summarized from domain knowledge. Then, we propose to use a Restricted Boltzmann Machine to model AU label prior distribution from the generated pseudo AU data. After that, we train AU classifiers from expression-annotated facial images and the learned prior model by maximizing the log likelihood of AU classifiers with regard to the learned AU label prior. The proposed AU activation recognition can also be extended to semi-supervised learning scenarios with partially AU-annotated facial images. Experimental results on four benchmark databases demonstrate the effectiveness of the proposed approach in learning AU classifiers from domain knowledge.