Abstract
Recovery and rehabilitation of facial mimics need enhanced decision support with multimodal bio-feedbacks from 3D real-time biomechanical head animation. Kinect V2.0 can detect and track 3-D high-definition face features (FFs) but the end of production can lead to difficult deployment of the developed solutions. Deep Neural Network (DNN)-based methods were employed, but the detected features were in 2-D or not accurate in 3-D. So, we developed a novel stereo-fusion scheme for enhancing the accuracy of 3-D features and generating biomechanical heads. Four stereo cameras were employed for detecting 2-D FFs based on DNN-based models. Stereo-triangulated 3-D FFs were fused using the Kalman filter. A head, skull, and muscle network were generated from the fused FFs. We validated the method with 1,000 virtual subjects and 5 Computed Tomography-based subjects. The in-silico trial errors (Mean±SD) were 2.27±0.29mm, 3.15±0.23mm, 1.72±0.13mm, and 3.08±0.39mm for the facial head, facial skull, muscle insertion point, and muscle attachment point regions. The experimental errors were 1.8384±0.1451mm, 2.6937±0.0575mm, 1.8271±0.1242mm, and 3.1428±0.2407mm. The errors were compatible with those using the Kinect V2.0 sensor and smaller than those using monovision-based 3-D feature detectors. This study has four contributions: (1) a stereo-fusion scheme for reconstructing 3-D FFs from 2-D FFs, (2) an enhancement accuracy for 3D DNN-based FF detection, (3) a biomechanical head generation from stereo-fusion cameras, and (4) a full validation procedure for 3-D FF detection. The method will be validated with facial palsy patients. Soft-tissue deformation will be integrated with mixed reality technology toward a next-generation of face decision-support system.
Published Version
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call