Abstract
This paper proposes a framework that allows 3D freeform manipulation of a face in live video. Unlike existing approaches, the proposed framework provides natural 3D manipulation of a face without background distortion and interactive face editing by a user’s input, which leads to freeform manipulation without any limitation of range or shape. To achieve these features, a 3D morphable face model is fitted to a face region in a video frame and is deformed by the user’s input. The video frame is then mapped as a texture to the deformed model, and the model is rendered on the video frame. Because of the high computational cost, parallelization and acceleration schemes are also adopted for real-time performance. Performance evaluation and comparison results show that the proposed framework is promising for 3D face editing in live video.
Highlights
Social media has played a significant role on the way people can communicate and interact with others
Facial landmark detection was implemented in Python using Theano, a deep learning framework, and other functions such as face model fitting and mesh deformation were implemented in C++ and invoked via Python bindings
Partial parallel processing with OpenMP was applied where extensive computation was required, as for example in visibility checking of all vertices of the face model to search for corresponding 3D landmarks on the model of 2D facial landmarks along the boundary of the face
Summary
Social media has played a significant role on the way people can communicate and interact with others. Face Warp 2 and Snow have the advantage of being able to edit both photos and live videos of human faces, but the editable parts and range are limited to a few templates Most such face editing applications are based on 2D image processing, and so incur background distortion. Manipulation has been implemented in various ways including copy and paste, geometric transformation, and isotropic or anisotropic scaling [8,9,10,11]; deformation to an arbitrary shape [7,8]; and physically based deformation [6,11] Even though such methods allow image or video editing based on 3D geometry of the target, they are unsuitable for face manipulation because the human face’s range of expressive deformations cannot be modeled by a rigid body or a simple physical models as in the above methods. This study details performance evaluation and comparison with existing applications
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
