3D Facial Information Research Articles

Feasibility of using 3D facial information to predict phase angle for complications in hospitalized patients

Feasibility of using 3D facial information to predict phase angle for complications in hospitalized patients

Using 3D facial information to predict malnutrition and consequent complications.

Phase angle (PhA) correlates with body composition and could predict the nutrition status of patients and disease prognosis. We aimed to explore the feasibility of predicting PhA-diagnosed malnutrition using facial image information based on deep learning (DL). From August 2021 to April 2022, inpatients were enrolled from surgery, gastroenterology, and oncology departments in a tertiary hospital. Subjective global assessment was used as the gold standard of malnutrition diagnosis. The highest Youden index value was selected as the PhA cutoff point. We developed a multimodal DL framework to automatically analyze the three-dimensional (3D) facial data and accurately determine patients' PhA categories. The framework was trained and validated using a cross-validation approach and tested on an independent dataset. Four hundred eighty-two patients were included in the final dataset, including 176 with malnourishment. In male patients, the PhA value with the highest Youden index was 5.55°, and the area under the receiver operating characteristic curve (AUC) = 0.68; in female patients, the PhA value with the highest Youden index was 4.88°, and AUC = 0.69. Inpatients with low PhA had higher incidence of infectious complications during the hospital stay (P = 0.003). The DL model trained with 4096 points extracted from 3D facial data had the best performance. The algorithm showed fair performance in predicting PhA, with an AUC of 0.77 and an accuracy of 0.74. Predicting the PhA of inpatients from facial images is feasible and can be used for malnutrition assessment and prognostic prediction.

Development and evaluation of a deep learning framework for the diagnosis of malnutrition using a 3D facial points cloud: A cross-sectional study.

The feasibility of diagnosing malnutrition using facial features has been validated. A tool to integrate all facial features associated with malnutrition for disease screening is still demanded. This work aims to develop and evaluate a deep learning (DL) framework to accurately determine malnutrition based on a 3D facial points cloud. A group of 482 patients were studied in this perspective work. The 3D facial data were obtained using a 3D camera and represented as a 3D facial points cloud. A DL model, PointNet++, was trained and evaluated using the points cloud as inputs and classified the malnutrition states. The performance was evaluated with the area under the receiver operating characteristic curve, accuracy, specificity, sensitivity, and F1 score. Among the 482 patients, 150 patients (31.1%) were diagnosed as having moderate malnutrition and 54 patients (11.2%) as having severe malnutrition. The DL model achieved the performance with an area under the receiver operating characteristic curve of 0.7240 ± 0.0416. The DL model achieved encouraging performance in accurately classifying nutrition states based on a points cloud of 3D facial information of patients with malnutrition.

3D facial animation driven by speech-video dual-modal signals

In recent years, the applications of digital humans have become increasingly widespread. One of the most challenging core technologies is the generation of highly realistic and automated 3D facial animation that combines facial movements and speech. The single-modal 3D facial animation driven by speech typically ignores the weak correlation between speech and upper facial movements as well as head posture. In contrast, the video-driven approach can perfectly solve the posture problem while obtaining natural expressions. However, mapping 2D facial information to 3D facial information may lead to information loss, which make lip synchronization generated by video-driven methods is not as good as the speech-driven methods trained on 4D facial data. Therefore, this paper proposes a dual-modal generation method that uses speech and video information to generate more natural and vivid 3D facial animation. Specifically, the lip movements related to speech are generated by speech-video information, while speech-uncorrelated postures and expressions are generated solely by video information. The speech-driven module is used to extract speech features, and its output lip animation is then used as the foundation for facial animation. The expression and pose module is used to extract temporal visual features for regressing expression and head posture parameters. We fuse speech and video features to obtain chin posture parameters related to lip movements, and use these parameters to fine-tune the lip animation generated form the speech-driven module. This paper introduces multiple consistency losses to enhance the network’s capability to generate expressions and postures. Experiments conducted on the LRS3, TCD-TIMIT and MEAD datasets show that the proposed method achieves better performance on evaluation metrics such as CER, WER, VER and VWER than the current state-of-the-art methods. In addition, a perceptual user study show that over 77% and 70% of cases believe that this paper’s method is better than the comparative algorithms EMOCA and SPECTRE in terms of realism. In terms of lip synchronization, it received over 79% and 66% of cases support, respectively. Both evaluation methods demonstrate the effectiveness of the proposed method.

Enhancing Face Recognition with Detachable Self-Supervised Bypass Networks.

Attributed to the development of deep networks and abundant data, automatic face recognition (FR) has quickly reached human-level capacity in the past few years. However, the FR problem is not perfectly solved in case of large poses and uncontrolled occlusions. In this paper, we propose a novel bypass enhanced representation learning (BERL) method to improve face recognition under unconstrained scenarios. The proposed method integrates self-supervised learning and supervised learning together by attaching two auxiliary bypasses, a 3D reconstruction bypass and a blind inpainting bypass, to assist robust feature learning for face recognition. Among them, the 3D reconstruction bypass enforces the face recognition network to encode pose independent 3D facial information, which enhances the robustness to various poses. The blind inpainting bypass enforces the face recognition network to capture more facial context information for face inpainting, which enhances the robustness to occlusions. The whole framework is trained in end-to-end manner with two self-supervised tasks above and the classic supervised face identification task. During inference, the two auxiliary bypasses can be detached from the face recognition network, avoiding any additional computational overhead. Extensive experimental results on various face recognition benchmarks show that, without any cost of extra annotations and computations, our method outperforms state-of-the-art methods. Moreover, the learnt representations can also well generalize to other face-related downstream tasks such as the facial attribute recognition with limited labeled data.

Deep Portrait Lighting Enhancement with 3D Guidance

AbstractDespite recent breakthroughs in deep learning methods for image lighting enhancement, they are inferior when applied to portraits because 3D facial information is ignored in their models. To address this, we present a novel deep learning framework for portrait lighting enhancement based on 3D facial guidance. Our framework consists of two stages. In the first stage, corrected lighting parameters are predicted by a network from the input bad lighting image, with the assistance of a 3D morphable model and a differentiable renderer. Given the predicted lighting parameter, the differentiable renderer renders a face image with corrected shading and texture, which serves as the 3D guidance for learning image lighting enhancement in the second stage. To better exploit the long‐range correlations between the input and the guidance, in the second stage, we design an image‐to‐image translation network with a novel transformer architecture, which automatically produces a lighting‐enhanced result. Experimental results on the FFHQ dataset and in‐the‐wild images show that the proposed method outperforms state‐of‐the‐art methods in terms of both quantitative metrics and visual quality.

A novel 2D and 3D multimodal approach for in-the-wild facial expression recognition

This study proposes a novel deep learning approach for the fusion of 2D and 3D modalities in in-the-wild facial expression recognition (FER). Different from other studies, we exploit the 3D facial information in in-the-wild FER. In particular, in-the-wild 3D FER dataset is not widely available; therefore, 3D facial data are constructed from available 2D datasets thanks to recent advances in 3D face reconstruction. The 3D facial geometry features are then extracted by deep learning technique to exploit the mid-level details, which provides meaningful expression for the recognition. In addition, to demonstrate the potential of 3D data on FER, the 2D projected images of 3D faces are taken as additional input to FER. These features are then jointly fused with 2D features obtained from the original input. The fused features are then classified by support vector machines (SVMs). The results show that the proposed approach achieves state-of-the-art recognition performances on Real-World Affective Faces (RAF) and Static Facial Expressions in the Wild (SFEW 2.0), and AffectNet dataset. This approach is also applied to a 3D FER dataset, i.e. BU-3DFE, to compare the effectiveness of reconstructed and available 3D face data for FER. This is the first time such a deep learning combination of 3D and 2D facial modalities is presented in the context of in-the-wild FER.

PCA-based image recombination for multimodal 2D + 3D face recognition

Most of the existing approaches of multimodal 2D + 3D face recognition exploit the 2D and 3D information at the feature or score level. They do not fully benefit from the dependency between modalities. Exploiting this dependency at the early stage is more effective than the later stage. Early fusion data contains richer information about the input biometric than the compressed features or matching scores. We propose an image recombination for face recognition that explores the dependency between modalities at the image level. Facial cues from the 2D and 3D images are recombined into a more independent and discriminating data by finding transformation axes that account for the maximal amount of variances in the images. We also introduce a complete framework of multimodal 2D + 3D face recognition that utilizes the 2D and 3D facial information at the enrollment, image and score levels. Experimental results based on NTU-CSP and Bosphorus 3D face databases show that our face recognition system using image recombination outperforms other face recognition systems based on the pixel- or score-level fusion.

A Passive 3D Face Recognition System and Its Performance Evaluation

This paper proposes a three-dimensional (3D) face recognition system using passive stereo vision. So far, the reported 3D face recognition techniques have used active 3D measurement methods to capture high-quality 3D facial information. However, active methods employ structured illumination (structure projection, phase shift, moire topography, etc.) or laser scanning, which is not desirable in many human recognition applications. Addressing this problem, we propose a face recognition system that uses (i) passive stereo vision to capture 3D facial information and (ii) 3D matching using an ICP (Iterative Closest Point) algorithm with its improvement techniques. Experimental evaluation demonstrates efficient recognition performance of the proposed system compared with an active 3D face recognition system and a passive 3D face recognition system employing the original ICP algorithm.