Near-infrared Face Research Articles

Face anti-spoofing with cross-stage relation enhancement and spoof material perception

Face Anti-Spoofing (FAS) seeks to protect face recognition systems from spoofing attacks, which is applied extensively in scenarios such as access control, electronic payment, and security surveillance systems. Face anti-spoofing requires the integration of local details and global semantic information. Existing CNN-based methods rely on small stride or image patch-based feature extraction structures, which struggle to capture spatial and cross-layer feature correlations effectively. Meanwhile, Transformer-based methods have limitations in extracting discriminative detailed features. To address the aforementioned issues, we introduce a multi-stage CNN-Transformer-based framework, which extracts local features through the convolutional layer and long-distance feature relationships via self-attention. Based on this, we proposed a cross-attention multi-stage feature fusion, employing semantically high-stage features to query task-relevant features in low-stage features for further cross-stage feature fusion. To enhance the discrimination of local features for subtle differences, we design pixel-wise material classification supervision and add a auxiliary branch in the intermediate layers of the model. Moreover, to address the limitations of a single acquisition environment and scarcity of acquisition devices in the existing Near-Infrared dataset, we create a large-scale Near-Infrared Face Anti-Spoofing dataset with 380k pictures of 1040 identities. The proposed method could achieve the state-of-the-art in OULU-NPU and our proposed Near-Infrared dataset at just 1.3GFlops and 3.2M parameter numbers, which demonstrate the effective of the proposed method.

Monitoring respiratory state from near-infrared face video images

In this paper, we propose an algorithm for estimating respiratory state using near-infrared facial video images. Estimation of respiratory state is an important indicator for early detection of respiratory diseases. In particular, there is a demand for monitoring respiratory state during the night. One method of monitoring respiratory state is to use contact-type sensors. However, this method requires the installation of many sensors and a visit to a hospital, which place a burden on patients. Therefore, we propose to acquire respiratory-induced features from near-infrared face video images and investigate their similarity to measurements obtained with a respirometer for non-contact monitoring of respiratory state in the dark. Respiratory-induced features were obtained from pulse wave signals extracted from the face video images. The results showed correlations in several respiratory states. This study opens some perspectives in non-contact monitoring of respiratory states.

Noise-Robust Pulse Wave Estimation from Near-Infrared Face Video Images Using the Wiener Estimation Method.

In this paper, we propose a noise-robust pulse wave estimation method from near-infrared face video images. Pulse wave estimation in a near-infrared environment is expected to be applied to non-contact monitoring in dark areas. The conventional method cannot consider noise when performing estimation. As a result, the accuracy of pulse wave estimation in noisy environments is not very high. This may adversely affect the accuracy of heart rate data and other data obtained from pulse wave signals. Therefore, the objective of this study is to perform pulse wave estimation robust to noise. The Wiener estimation method, which is a simple linear computation that can consider noise, was used in this study. Experimental results showed that the combination of the proposed method and signal processing (detrending and bandpass filtering) increased the SNR (signal to noise ratio) by more than 2.5 dB compared to the conventional method and signal processing. The correlation coefficient between the pulse wave signal measured using a pulse wave meter and the estimated pulse wave signal was 0.30 larger on average for the proposed method. Furthermore, the AER (absolute error rate) between the heart rate measured with the pulse wave meter was 0.82% on average for the proposed method, which was lower than the value of the conventional method (12.53% on average). These results show that the proposed method is more robust to noise than the conventional method for pulse wave estimation.

ForgeryNIR: Deep Face Forgery and Detection in Near-Infrared Scenario

Deep face forgery and detection is an emerging topic due to the development of GANs. Face forgery detection relies greatly on existing databases for evaluation and adequate training examples for data-hungry machine learning algorithms. However, considering the wide application of face recognition in near-infrared scenarios, there is no publicly available face forgery database that includes near-infrared modality currently. In this paper, we present an attempt at constructing a large-scale dataset for face forgery detection in the near-infrared modality and propose a new forgery detection method based on knowledge distillation named cross-modality knowledge distillation aiming to use a teacher model which is pre-trained on the visible light-based (VIS) big data to guide the student model with a small amount of near-infrared (NIR) data. The proposed near-infrared face forgery dataset, named ForgeryNIR, contains a total of over 50,000 real and fake identities. A number of perturbations are applied to help simulate real-world scenarios. All source images in ForgeryNIR are collected from CASIA NIR-VIS 2.0, and fake images are generated via multiple GAN techniques. The proposed dataset fills the gap of face forgery detection research in the near-infrared modality. A comprehensive study on six representative detection baselines is conducted to evaluate the performance of face forgery detection algorithms in the NIR domain. We further construct a hard testing set, named ForgeryNIR+, which contains forged images that have bypassed existing face forgery detection methods. The proposed datasets will be publicly available and aim to help boost further research on face forgery detection, as well as NIR face detection and recognition.

Homomorphic Filtering and Phase-Based Matching for Cross-Spectral Cross-Distance Face Recognition.

Facial recognition has a significant application for security, especially in surveillance technologies. In surveillance systems, recognizing faces captured far away from the camera under various lighting conditions, such as in the daytime and nighttime, is a challenging task. A system capable of recognizing face images in both daytime and nighttime and at various distances is called Cross-Spectral Cross Distance (CSCD) face recognition. In this paper, we proposed a phase-based CSCD face recognition approach. We employed Homomorphic filtering as photometric normalization and Band Limited Phase Only Correlation (BLPOC) for image matching. Different from the state-of-the-art methods, we directly utilized the phase component from an image, without the need for a feature extraction process. The experiment was conducted using the Long-Distance Heterogeneous Face Database (LDHF-DB). The proposed method was evaluated in three scenarios: (i) cross-spectral face verification at 1m, (ii) cross-spectral face verification at 60m, and (iii) cross-spectral face verification where the probe images (near-infrared (NIR) face images) were captured at 1m and the gallery data (face images) was captured at 60 m. The proposed CSCD method resulted in the best recognition performance among the CSCD baseline approaches, with an Equal Error Rate (EER) of 5.34% and a Genuine Acceptance Rate (GAR) of 93%.

SWU-NIRPV: a near-infrared pose variation face database and pose-invariant face recognition

Near-infrared (NIR) face recognition (FR) has demonstrated robustness against changes in ambient illumination, which makes it suitable for surveillance even under weak illumination conditions. However, the existing database for NIR FR only contains frontal face images, and the impact of pose variation on the robustness of NIR FR remains unascertained. We developed an NIR face database with 57 pose variations in a dark environment, which can be used in pose-invariant FR research. Convolutional neural networks (CNNs) were designed and tested in comparison to the traditional method in the database. The experimental results showed that a difference of even 10 deg between the gallery and testing sets can dramatically reduce the recognition performance. Additionally, an average accuracy of 90.58% was obtained for pose-invariant recognition by employing more pose variations in the gallery set using the CNN-based method.

Linear Cross-Modal Hash Encoding (LCMHE) for Visual and Near-Infrared Face Recognition

Face recognition in practical situations often encounters the change of acquisition sensors, i.e., the gallery images are captured in near-infrared and probe images captured in the visual domain or vice versa. Thus, for robust face recognition under such circumstance, a modality invariant representation is required. This letter proposes a hash-encoding-based descriptor to address this modality gap for face recognition problem, namely linear cross-modal hash encoding (LCMHE). LCMHE encodes the pixels of the face images based on a pretrained hash dictionary. The encoding highlights the reflectance part of the face and enables accurate recognition. We further explored the possibility of LCMHE by combining the deep-learning model to get better recognition. We experimented with publicly available CASIA NIR-VIS 2.0 databases and verified the effectiveness of our proposed letter.

A DenseUnet generative adversarial network for near-infrared face image colorization

A DenseUnet Generative Adversarial Network (GAN) is proposed to colorize near-infrared (NIR) face images. The GAN generator incorporates the advantages of both DenseNet and Unet structures. The DenseNet extracts facial features effectively by increasing the network depth, and the Unet keeps important facial details through skip-connection. The generator further integrates an optimized facial loss function designed by considering pixel loss, color loss and feature loss. The proposed network is evaluated against state-of-the-art grayscale image colorization GANs over two separate datasets. Both qualitative and quantitative comparisons demonstrate that with improved network structures and loss constrains the DenseUnet GAN can colorize NIR face images with natural color, minimal face shape distortion, and rich facial details.

Spatial-Spectral-Temporal Fusion for Remote Heart Rate Estimation

In this paper, we propose a novel heart rate (HR) estimation method using simultaneously recorded RGB and near-infrared (NIR) face videos to improve the robustness of camera-based remote HR estimation against illumination fluctuations and head motions. The key to robust HR estimation is constructing the histogram of HRs for a considered time window by voting candidate HRs that are estimated using different spatial face patches, spectral modalities (i.e., RGB and NIR), and temporal short-time sub-windows. The histogram voting is performed only for the candidate HRs that pass through a reliability check of HR estimation. The final HR estimate for the considered time window is then obtained by detecting the most frequently voted HR bin and performing parabola fitting using its neighboring bins. By spatially, spectrally, and temporally fusing the candidate HRs for majority voting, our method can automatically exploit suitable video sub-regions less affected by illumination fluctuations and head motions to enable robust HR estimation. Through the experiments on 168 RGB-NIR video recordings, we demonstrate that our fusion-based method achieves improved HR estimation accuracy compared with existing methods.

A DCNN-Based Fast NIR Face Recognition System Robust to Reflected Light From Eyeglasses

Due to an increasing need for face recognition under poor lighting conditions, near infrared (NIR) face recognition based on deep convolutional neural networks (DCNN) has become an active area of research. However, in NIR face images of eyeglasses wearers, reflected light is generated around the eyes due to active NIR light sources, and it is one of the main contributors to performance degradation in NIR face recognition. In addition, there have to date been no attempts to lighten DCNN models for NIR face recognition. To solve these problems, we propose a DCNN-based fast NIR face recognition system which is robust to reflected light. This work has two main contributions: 1) We generated synthetic face images of individuals with and without eyeglasses using our proposed CycleGAN-based Glasses2Non-glasses (G2NG) data augmentation. We then constructed an augmented training database by adding the synthetic images, and the database helps to make the NIR face recognition system robust against reflected light. 2) A lightweight NIR FaceNet (LiNFNet) architecture was developed to reduce the computational complexity of the proposed system by adapting the depthwise separable convolutions and linear bottlenecks to VGGNet 16. The proposed architecture reduces the computation required, while improving the performance of NIR face recognition. Through the experiments reported in this paper, we verified that the proposed G2NG data augmentation improved the face recognition validation rate by 99.09% for NIR face images which have the reflected light from eyeglasses. Also, LiNFNet reduces the number of multiplication operations by 4.4×10 9 compared with VGGNet 16.

NIR Reflection Augmentation for DeepLearning-Based NIR Face Recognition

Face recognition using a near-infrared (NIR) sensor is widely applied to practical applications such as mobile unlocking or access control. However, unlike RGB sensors, few deep learning approaches have studied NIR face recognition. We conducted comparative experiments for the application of deep learning to NIR face recognition. To accomplish this, we gathered five public databases and trained two deep learning architectures. In our experiments, we found that simple architecture could have a competitive performance on the NIR face databases that are mostly composed of frontal face images. Furthermore, we propose a data augmentation method to train the architectures to improve recognition of users who wear glasses. With this augmented training set, the recognition rate for users who wear glasses increased by up to 16%. This result implies that the recognition of those who wear glasses can be overcome using this simple method without constructing an additional training set. Furthermore, the model that uses augmented data has symmetry with those trained with real glasses-wearing data regarding the recognition of people who wear glasses.

Remote Heart Rate Measurement from RGB-NIR Video Based on Spatial and Spectral Face Patch Selection.

In this paper, we propose a novel heart rate (HR) estimation method using simultaneously recorded RGB and near-infrared (NIR) face videos. The key idea of our method is to automatically select suitable face patches for HR estimation in both spatial and spectral domains. The spatial and spectral face patch selection enables us to robustly estimate HR under various situations, including scenes under which existing RGB camera-based methods fail to accurately estimate HR. For a challenging scene in low light and with light fluctuations, our method can successfully estimate HR for all 20 subjects $( \pm 3$ beats per minute), while the RGB camera-based methods succeed only for 25% of the subjects.

Frankenstein: Learning Deep Face Representations Using Small Data.

Deep convolutional neural networks have recently proven extremely effective for difficult face recognition problems in uncontrolled settings. To train such networks, very large training sets are needed with millions of labeled images. For some applications, such as near-infrared (NIR) face recognition, such large training data sets are not publicly available and difficult to collect. In this paper, we propose a method to generate very large training data sets of synthetic images by compositing real face images in a given data set. We show that this method enables to learn models from as few as 10 000 training images, which perform on par with models trained from 500 000 images. Using our approach, we also obtain state-of-the-art results on the CASIA NIR-VIS2.0 heterogeneous face recognition data set.

Local SVD based NIR face retrieval

From last decade, local descriptor such as Local Binary Pattern (LBP) is accepted as a very prominent feature descriptor for characterizing the images such as faces. The performance of such descriptors depends upon the local relationship of the image. The local relationship of the image can be utilized in more discriminative and robust way after some preprocessing as compared to the original image. The preprocessed images in the form of 4 sub-bands (i.e. S, U, V, and D sub-bands) are obtained by applying the Singular Value Decomposition (SVD) over the original image. The local descriptors are computed over these sub-bands (mainly S sub-band) and termed as the SVD based local descriptors. The performance of four local descriptors over SVD sub-bands are tested for near-infrared face retrieval using PolyU-NIR and CASIA-NIR face databases, and compared with the results obtained using descriptors without SVD sub-band. The experimental results confirm the superiority of using S sub-band of SVD in terms of performance of the local descriptors over NIR face databases.

NIRFaceNet: A Convolutional Neural Network for Near-Infrared Face Identification

Near-infrared (NIR) face recognition has attracted increasing attention because of its advantage of illumination invariance. However, traditional face recognition methods based on NIR are designed for and tested in cooperative-user applications. In this paper, we present a convolutional neural network (CNN) for NIR face recognition (specifically face identification) in non-cooperative-user applications. The proposed NIRFaceNet is modified from GoogLeNet, but has a more compact structure designed specifically for the Chinese Academy of Sciences Institute of Automation (CASIA) NIR database and can achieve higher identification rates with less training time and less processing time. The experimental results demonstrate that NIRFaceNet has an overall advantage compared to other methods in the NIR face recognition domain when image blur and noise are present. The performance suggests that the proposed NIRFaceNet method may be more suitable for non-cooperative-user applications.

Visible-light and near-infrared face recognition at a distance

A method to solve the problem of face recognition at a distance (FRAD) under the visible-light (VIS) and the near-infrared (NIR) spectra is presented in this work. For images taken under visible light at day time, we perform the coarse-scale alignment/enhancement to eliminate a set of unlikely candidates at the first stage. Then, the fine-scale alignment/enhancement steps are conducted to refine the candidate list furthermore iteratively at the second stage. To address the additional challenge associated with NIR images captured at night time, we incorporate a restoration mechanism that reconstructs low-quality patches through a locally linear embedding (LLE) process with a local constraint. It is shown by experimental results that our FRAD solution outperforms state-of-the-art methods on both VIS and NIR images.

Structured detail enhancement for cross-modality face synthesis

Transformation, matching, and retrieval of face images in different modalities are often encountered in many applications. Cross-modality face synthesis, which aims to transform face images between modalities, can serve as an intermediate representation for these tasks, and remains an active yet challenging topic in low level vision. Most existing cross-modality face synthesis methods are effective in recovering global features for a given photo, but fail to capture fine-scale details or introduce some extra noise in the synthesis results. In this paper, we propose a two-step algorithm to tackle this problem by enhancing structured details after initial synthesis. In the first step, a combination of quasi-smooth area synthesis and facial component synthesis is proposed to generate more plausible facial components. In the second step, an MAP-based model is constructed to introduce more facial details into the initial synthesis, meanwhile eliminate noise. Experiments on face sketch synthesis and near-infrared face synthesis demonstrate the effectiveness of the proposed method in enhancing fine-scale facial details while reducing noise.

Near-infrared Face Recognition by Fusion of E-GV-LBP and FKNN

To solve the problem of face recognition with complex changes and further improve the efficiency, a new near-infrared face recognition algorithm which fuses E-GV-LBP and FKNN algorithm is proposed. Firstly, it transforms near infrared face image by Gabor wavelet. Then, it extracts LBP coding feature that contains space, scale and direction information. Finally, this paper introduces an improved FKNN algorithm which is based on spatial domain. The proposed approach has brought face recognition more quickly and accurately. The experiment results show that the new algorithm has improved the recognition accuracy and computing time under the near-infrared light and other complex changes. In addition, this method can be used for face recognition under visible light as well.

A Near-Infrared Face Detection and Recognition System Using ASM and PCA+LDA

Near-infrared (NIR) images have a lot of advantages, which can make up the shortages of visible light (VL) images. A novel NIR detection and recognition system is presented in this paper. First, a NIR imaging system is developed to provide good illumination conditions for subsequent face detection and recognition. Then, Active Shape Model (ASM) method is applied to detect the features and the low-dimensional Gabor features are extracted for recognition by combining Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). The recognition rate of the system is up to 90% in a few milliseconds. Finally, by using such real-time NIR face recognition system, comparative results are provided, from that we can seen that this system can work robustly and effectively

Rotation and noise invariant near-infrared face recognition by means of Zernike moments and spectral regression discriminant analysis

Face recognition is a rapidly growing research area, which is based heavily on the methods of machine learning, computer vision, and image processing. We propose a rotation and noise invariant near-infrared face-recognition system using an orthogonal invariant moment, namely, Zernike moments (ZMs) as a feature extractor in the near-infrared domain and spectral regression discriminant analysis (SRDA) as an efficient algorithm to decrease the computational complexity of the system, enhance the discrimination power of features, and solve the “small sample size” problem simultaneously. Experimental results based on the CASIA NIR database show the noise robustness and rotation invariance of the proposed approach. Further analysis shows that SRDA as a sophisticated technique, improves the accuracy and time complexity of the system compared with other data reduction methods such as linear discriminant analysis.