Facial Action Unit Recognition Augmented by Their Dependencies

Abstract

Due to the underlying anatomic mechanism that govern facial muscular interactions, there exist inherent dependencies between facial action units (AU). Such dependencies carry crucial information for AU recognition, yet have not been thoroughly exploited. Therefore, in this paper, we propose a novel AU recognition method with a three-layer hybrid Bayesian network, whose top two layers consist of a latent regression Bayesian network (LRBN), and the bottom two layers are Bayesian networks. The LRBN is a directed graphical model consisting of one latent layer and one visible layer. Specifically, the visible nodes of LRBN represent the ground-truth AU labels. Due to the explaining away effect in Bayesian networks, LRBN is able to capture both the dependencies among the latent variables given the observation and the dependencies among visible variables. Such dependencies successfully and faithfully represent relations among multiple AUs. The bottom two layers are two node Bayesian networks, connecting the ground truth AU labels and their measurements. Efficient learning and inference algorithms are also proposed. Furthermore, we extend the proposed hybrid Bayesian network model for facial expression-assisted AU recognition, since AU relations are influenced by expressions. By introducing facial expression nodes in the middle visible layer, facial expressions, which are only required during training, facilitate the estimation of label dependencies among AUs. Experimental results on three benchmark databases, i.e. the CK+ database, the SEMAINE database, and the BP4D database, demonstrate that the proposed approaches can successfully capture complex AU relationships, and the expression labels available only during training are benefit for AU recognition during testing.