Facial Depth Map Research Articles

A Robust Light-Weight Fused-Feature Encoder-Decoder Model for Monocular Facial Depth Estimation From Single Images Trained on Synthetic Data

Due to the real-time acquisition and reasonable cost of consumer cameras, monocular depth maps have been employed in a variety of visual applications. Regarding ongoing research in depth estimation, they continue to suffer from low accuracy and enormous sensor noise. To improve the prediction of depth maps, this paper proposed a lightweight neural facial depth estimation model based on single image frames. Following a basic encoder-decoder network design, the features are extracted by initializing the encoder with a high-performance pre-trained network and reconstructing high-quality facial depth maps with a simple decoder. The model can employ pixel representations and recover full details in terms of facial features and boundaries by employing a feature fusion module. When tested and evaluated across four public facial depth datasets, the suggested network provides more reliable and state-of-the-art results, with significantly less computational complexity and a reduced number of parameters. The training procedure is primarily based on the use of synthetic human facial images, which provide a consistent ground truth depth map, and the employment of an appropriate loss function leads to higher performance. Numerous experiments have been performed to validate and demonstrate the usefulness of the proposed approach. Finally, the model performs better than existing comparative facial depth networks in terms of generalization ability and robustness across different test datasets, setting a new baseline method for facial depth maps.

MFFNet: Single facial depth map refinement using multi-level feature fusion

Depth maps have been used in many vision tasks due to the real-time acquisition and low cost of consumer depth cameras. However, they still suffer from low precision and severe sensor noise, even with the significant research in depth enhancement. We propose a novel multi-level feature fusion convolutional neural network (CNN) for facial depth map refinement named MFFNet. It is a multi-stage network, where each stage is a local multi-level feature fusion (LMLF) block. For smoothing the noise as well as boosting detailed facial structure, a hierarchical fusion strategy is adopted to fully fuse multi-level features, i.e., an LMLF block fuses multi-level features locally in each stage, while inter-stage skip connections are employed to reach a global multi-level feature fusion. Moreover, the inter-stage skip connections can also ease the training through shortening the information propagation paths. We introduce an effective data augmentation method to synthesize noisy facial depth maps of various poses. Training with these synthetic data improves the robustness of the proposed method to face poses. The proposed method is evaluated with a synthetic facial depth map dataset, a real Kinect V2 facial depth map dataset and the Middlebury Stereo Dataset. Experimental results show that our method produces refined depth maps with high quality and outperforms several state-of-the-art methods.

Towards Monocular Neural Facial Depth Estimation: Past, Present, and Future

This article contains all of the information needed to conduct a study on monocular facial depth estimation problems. A brief literature review and applications on facial depth map research were offered first, followed by a comprehensive evaluation of publicly available facial depth datasets and widely used loss functions. The key properties and characteristics of each facial depth map dataset are described and evaluated. Furthermore, facial depth maps loss functions are briefly discussed, which will make it easier to train neural facial depth models on a variety of datasets for both short- and long-range depth maps. The network’s design and components are essential, but its effectiveness is largely determined by how it is trained, which necessitates a large dataset and a suitable loss function. Implementation details of how neural depth networks work and their corresponding evaluation matrices are presented and explained. In addition, an SoA neural model for facial depth estimation is proposed, along with a detailed comparison evaluation and, where feasible, direct comparison of facial depth estimation methods to serve as a foundation for a proposed model that is utilized. The model employed shows better performance compared with current state-of-the-art methods when tested across four datasets. The new loss function used in the proposed method helps the network to learn the facial regions resulting in an accurate depth prediction. The network is trained on synthetic human facial depth datasets whereas for validation purposes real as well as synthetic facial images are used. The results prove that the trained network outperforms current state-of-the-art networks performances, thus setting up a new baseline method for facial depth estimations.

Machine‐Learned Light‐Field Camera that Reads Facial Expression from High‐Contrast and Illumination Invariant 3D Facial Images

Facial expression conveys nonverbal communication information to help humans better perceive physical or psychophysical situations. Accurate 3D imaging provides stable topographic changes for reading facial expression. In particular, light‐field cameras (LFCs) have high potential for constructing depth maps, thanks to a simple configuration of microlens arrays and an objective lens. Herein, machine‐learned NIR‐based LFCs (NIR‐LFCs) for facial expression reading by extracting Euclidean distances of 3D facial landmarks in pairwise fashion are reported. The NIR‐LFC contains microlens arrays with asymmetric Fabry−Perot filter and NIR bandpass filter on CMOS image sensor, fully packaged with two vertical‐cavity surface‐emitting lasers. The NIR‐LFC not only increases the image contrast by 2.1 times compared with conventional LFCs, but also reduces the reconstruction errors by up to 54%, regardless of ambient illumination conditions. A multilayer perceptron (MLP) classifies input vectors, consisting of 78 pairwise distances on the facial depth map of happiness, anger, sadness, and disgust, and also exhibits exceptional average accuracy of 0.85 (p<0.05). This LFC provides a new platform for quantitatively labeling facial expression and emotion in point‐of‐care biomedical, social perception, or human−machine interaction applications.

Facial depth descend: A generation paradigm for facial depth map

Over the past decades, three-dimensional(3D) face recognition has developed rapidly due to its intrinsic invariance to pose and illumination changes. Yet despite this, deep learning is rarely used in 3D face recognition due to lack of training dat a. Noting that the majority of the recognition processes are based on facial depth map instead of real 3D data, we focus on the generation of facial depth map to surmount the shortage of training data. In this paper, we present a novel paradigm, Facial Depth Descend, which generates facial depth map from existing 3D face data. The key to this generation paradigm is to recombine facial components from existing faces and thereafter generate brand new faces based on them. We then propose a learning framework RPC to generate recognition-friendly faces. First, it extracts three facial components (eyes, nose and mouth) from the 3D data of real faces. Then, with these components as input, a Relative Location Estimator (RLE) is used to predict the relative location among them so that they can be composed in a reasonable manner. Finally, the method feeds a Pix2Pix network with these composed facial components to extrapolate areas surrounding them and output a full facial depth map. We here enforce an identity preserving loss on the generation network to make the facial depth map more discriminative and favorable for recognition tasks. Since we can choose different identities’ different components as input, RPC is theoretically able to generate a vast number of novel 2.5D faces. More specifically, RPC can generate an enlarged face dataset as large as N3 identities, where N is the number of identities in the original one. Experiments show that, adding the generated depth maps to the training dataset can help improve the recognition rate in 2.5D face recognition.

Edge‐guided single facial depth map super‐resolution using CNN

In recent years, consumer depth cameras have been widely used in digital entertainment and human-machine interaction due to the advantages of real-time performance and low cost. Facial depth maps have shown great potential in 3D-face-related studies. However, disadvantages of low resolution and precision limit its further applications. In this work, the authors propose an edge-guided convolutional neural network for single facial depth map super-resolution. It consists of two parts: an edge prediction sub-network and a depth reconstruction sub-network. The edge prediction sub-network generates an edge guidance map to guide the depth reconstruction sub-network to recover sharp edges and fine structures. Effective data augmentation methods are proposed as well. The network is patch-based and able to cope with any size of the input depth maps. In addition, it is insensitive to the face pose since the synthetic training dataset they generated covers a wide range of face poses. The proposed method is validated with three datasets including a synthetic facial depth data set, a real Kinect V2 facial depth data set and Middlebury Stereo Data set. Experimental results show that it outperforms the state-of-the-art methods on all the three data sets.

Joint-Feature Guided Depth Map Super-Resolution With Face Priors.

In this paper, we present a novel method to super-resolve and recover the facial depth map nicely. The key idea is to exploit the exemplar-based method to obtain the reliable face priors from high-quality facial depth map to improve the depth image. Specifically, a new neighbor embedding (NE) framework is designed for face prior learning and depth map reconstruction. First, face components are decomposed to form specialized dictionaries and then reconstructed, respectively. Joint features, i.e., low-level depth, intensity cues and high-level position cues, are put forward for robust patch similarity measurement. The NE results are used to obtain the face priors of facial structures and smooth maps, which are then combined in an uniform optimization framework to recover high-quality facial depth maps. Finally, an edge enhancement process is implemented to estimate the final high resolution depth map. Experimental results demonstrate the superiority of our method compared to state-of-the-art depth map super-resolution techniques on both synthetic data and real-world data from Kinect.

Novel descriptors for geometrical 3D face analysis

3D face was recently investigated for various applications, including biometrics and diagnosis. Describing facial surface, i.e. how it bends and which kinds of patches is composed by, is the aim of studies of Face Analysis, whose ultimate goal is to identify which features could be extracted from three-dimensional faces depending on the application. In this study, we propose 105 novel geometrical descriptors for Face Analysis. They are generated by composing primary geometrical descriptors such as mean, Gaussian, principal curvatures, shape index, curvedness, and the coefficients of the fundamental forms, and by applying standard functions such as sine, cosine, and logarithm to them. The new descriptors were mapped on 217 facial depth maps and analysed in terms of descriptiveness of facial shape and exploitability for localizing landmark points. Automatic landmark extraction stands as the final aim of this analysis. Results showed that some newly generated descriptors were sounder than the primary ones, meaning that their local behaviours in correspondence to a landmark position is thoroughly specific and can be registered with high similarity on every face of our dataset.

A unified 3D face authentication framework based on robust local mesh SIFT feature

In this paper, we design a unified 3D face authentication system for practical use. First, we propose a facial depth recovery method to construct a facial depth map from stereoscopic videos. It effectively utilize prior facial information and incorporate the visibility term to classify static and dynamic pixels for robust depth estimation. Secondly, in order to make 3D face authentication more accurate and consistent, we present an intrinsic scale feature detection for interesting points on 3D facial mesh regions. Then, a novel feature descriptor is proposed, called Local Mesh Scale-Invariant Feature Transform (LMSIFT) to reflect the different face recognition abilities in different facial regions. Finally, the sparse optimization problem of visual codebook is used to 3D face learning. We evaluate our approach on publicly available 3D face databases and self-collected realistic scene databases. We also develop an interactive education system to investigate its performance in practice, which demonstrates the high performance of the proposed approach for accurate 3D face authentication. Compared with previous popular approaches, our system has consistently better performance in terms of effectiveness, robustness and universality.

3D facial depth map recognition based on manifold learning-LOGMAP algorithm

3D facial depth map recognition based on manifold learning-LOGMAP algorithm