Facial tracking and animation for digital social system

Abstract

Avatar expression appearing in the virtual social space is one of the key technologies to convey people's emotions and facilitate the social interactions effectively via the virtual social system. Aiming at lack of feasible solutions for synchronized facial expressions in current commercial virtual social systems, this paper presented a virtual social system with the focus on real-time avatar facial expressions. Firstly, cascaded pose regression was adopted to train a dynamic expression model to infer the expression coefficients from 2D video frames, and the facial landmarks in regression were extracted by supervised descent method instead of 2D cascaded pose regression to achieve better robustness and fault tolerance in facial tracking and animation. Secondly, we proposed a multi-scale adaptive expression coding technology for expression-voice data synchronization and striking balance between real-time and richness of facial expressions in varied complex network situations. The experimental results show that the proposed facial tracking and animation system is practical and feasible, and could produce a high degree of realistic emotional cues in virtual social system.