SCIFI: 3D face reconstruction via smartphone screen lighting

Abstract

3D face reconstruction based on smartphones has a wide range of applications, such as face recognition, liveness detection, face animation, etc. To realize this function, these smartphones have been deployed with specialized hardware modules (infrared dot projector or depth sensor). However, it will inevitably increase the production cost of smartphones and the operation difficulty of users. In this article, we propose a smartphone screen illumination-based face reconstructIon (SCIFI) framework, which only relies on the front camera and screen lighting. Specifically, we have investigated the calibrated planar lighting to achieve the fine-grained textures in Lambertian-based reconstruction. Further, we introduce the face landmark to align multiple photographs, which aims to adjust the position mismatch caused by hand jitter. Moreover, we propose two different methods to eliminate outlier normals based on the characteristics of human face. Extensive experiments based on different environments (dark and bright), different lighting pattern (4-zones and 9-zones), and different testing subjects have validated the effectiveness and robustness of SCIFI in reconstructing the 3D face surface with a favorable surface shape as well as micro facial texture.