Visible Face Images Research Articles

Laser-Visible Face Image Translation and Recognition Based on CycleGAN and Spectral Normalization.

The range-gated laser imaging instrument can capture face images in a dark environment, which provides a new idea for long-distance face recognition at night. However, the laser image has low contrast, low SNR and no color information, which affects observation and recognition. Therefore, it becomes important to convert laser images into visible images and then identify them. For image translation, we propose a laser-visible face image translation model combined with spectral normalization (SN-CycleGAN). We add spectral normalization layers to the discriminator to solve the problem of low image translation quality caused by the difficulty of training the generative adversarial network. The content reconstruction loss function based on the Y channel is added to reduce the error mapping. The face generated by the improved model on the self-built laser-visible face image dataset has better visual quality, which reduces the error mapping and basically retains the structural features of the target compared with other models. The FID value of evaluation index is 36.845, which is 16.902, 13.781, 10.056, 57.722, 62.598 and 0.761 lower than the CycleGAN, Pix2Pix, UNIT, UGATIT, StarGAN and DCLGAN models, respectively. For the face recognition of translated images, we propose a laser-visible face recognition model based on feature retention. The shallow feature maps with identity information are directly connected to the decoder to solve the problem of identity information loss in network transmission. The domain loss function based on triplet loss is added to constrain the style between domains. We use pre-trained FaceNet to recognize generated visible face images and obtain the recognition accuracy of Rank-1. The recognition accuracy of the images generated by the improved model reaches 76.9%, which is greatly improved compared with the above models and 19.2% higher than that of laser face recognition.

Utilizing Alignment Loss to Advance Eye Center Detection and Face Recognition in the LWIR Band

Geometric normalization is an integral part of most of the face recognition (FR) systems. To geometrically normalize a face, it is essential to detect the eye centers, since one way to align the face images is to make the line joining the eye centers horizontal. This paper proposes a novel approach to detect eye centers in the challenging Long-Wave Infrared (LWIR) spectrum (8-14 μm). While using thermal band images for face recognition is a feasible approach in low-light and nighttime conditions, where visible face images cannot be used, there are not many thermal or dual band (visible and thermal) face datasets available to train and test new eye center detection models. This work takes advantage of the available deep learning based eye center detection algorithms in the visible band to detect the eye centers in thermal face images through image synthesis. While we empirically evaluate different image synthesis models, we determine that StarGAN2 yields the highest eye center detection accuracy, when compared to the other state-of-the-art models. We incorporate alignment loss that captures the normalized error between the detected and actual eye centers as an additional loss term during training (using the generated images during training, ground truth annotations, and an eye center detection model), so that the model learns to align the images to minimize this error. During test phase, visible images are generated from the thermal images using the trained model. Then, the available landmark detection algorithms in the visible band, namely, MT-CNN and HR-Net are used to detect the eye centers. Next, these eye centers are used to geometrically normalize the source thermal face images before performing same-spectral (thermal-to-thermal) face recognition. The proposed method improved the eye center detection accuracy by 60% over the baseline model, and by 14% over training only the StarGAN2 model without the alignment loss. The proposed approach also reports the highest improvement in the face recognition accuracy by 36% and 3% over the baseline and original StarGAN2 models, respectively, when using deep learning based face recognition models, namely, Facenet, ArcFace, and VGG-Face. We also perform experiments by augmenting the train and test datasets with images rotated in-plane to further demonstrate the efficiency of the proposed approach. When CycleGAN (another unpaired image translation network) is used to generate images before incorporating the alignment loss, it failed to preserve the alignment at the slightest, therefore the eye center detection accuracy was extremely low. With the alignment loss, the accuracy increased by 20%, 50%, and 80% when the normalized error (e) ≤ 0.05, 0.10 and 0.25 respectively.

Learning Domain and Pose Invariance for Thermal-to-Visible Face Recognition

Interest in thermal to visible face recognition has grown significantly over the last decade due to advancements in thermal infrared cameras and analytics beyond the visible spectrum. Despite large discrepancies between thermal and visible spectra, existing approaches bridge domain gaps by either synthesizing visible faces from thermal faces or by learning the cross-spectrum image representations. These approaches typically work well with frontal facial imagery collected at varying ranges and expressions, but exhibit significantly reduced performance when matching thermal faces with varying poses to frontal visible faces. We propose a novel Domain and Pose Invariant Framework that simultaneously learns domain and pose invariant representations. Our proposed framework is composed of modified networks for extracting the most correlated intermediate representations from off-pose thermal and frontal visible face imagery, a sub-network to jointly bridge domain and pose gaps, and a joint-loss function comprised of cross-spectrum and pose-correction losses. We demonstrate efficacy and advantages of the proposed method by evaluating on three thermal-visible datasets: ARL Visible-to-Thermal Face, ARL Multimodal Face, and Tufts Face. Although DPIF focuses on learning to match off-pose thermal to frontal visible faces, we also show that DPIF enhances performance when matching frontal thermal face images to frontal visible face images.

Cross-Spectrum Thermal Face Pattern Generator

Conversion of a visible face image into a thermal face image (V2T), or one thermal face image into another one given a different target temperature (T2T), is required in applications such as thermography, human body thermal pattern analysis, and surveillance using cross-spectral imaging. In this work, we propose to use conditional generative adversarial networks (cGAN) with cGAN loss, perceptual loss, and temperature loss to solve the conversion tasks. In our experiment, we used Carl and SpeakingFaces Databases. Frèchet Inception Distance (FID) is used to evaluate the generated images. As well, face recognition was applied to assess the performance of our models. For the V2T task, the FID of the generated thermal images reached a low value of 57.3. For the T2T task, we achieved a rank-1 face recognition rate of 91.0% which indicates that the generated thermal images preserve the majority of the identity information.

A Novel Infrared and Visible Image Fusion Approach Based on Adversarial Neural Network.

The presence of fake pictures affects the reliability of visible face images under specific circumstances. This paper presents a novel adversarial neural network designed named as the FTSGAN for infrared and visible image fusion and we utilize FTSGAN model to fuse the face image features of infrared and visible image to improve the effect of face recognition. In FTSGAN model design, the Frobenius norm (F), total variation norm (TV), and structural similarity index measure (SSIM) are employed. The F and TV are used to limit the gray level and the gradient of the image, while the SSIM is used to limit the image structure. The FTSGAN fuses infrared and visible face images that contains bio-information for heterogeneous face recognition tasks. Experiments based on the FTSGAN using hundreds of face images demonstrate its excellent performance. The principal component analysis (PCA) and linear discrimination analysis (LDA) are involved in face recognition. The face recognition performance after fusion improved by 1.9% compared to that before fusion, and the final face recognition rate was 94.4%. This proposed method has better quality, faster rate, and is more robust than the methods that only use visible images for face recognition.

A registration strategy from scale adjustment to fine for thermal-visible face images

The analysis of multispectral facial features is rising as one of the new emerging tools in medical diagnosis. The dynamic infrared thermography and visible image data fusion of a human face can manifest several physiological behaviours used in medical diagnosis. However, an essential but challenging step is the registration of thermal and visible images. This paper presents a two-step strategy to register visible face image into the same coordinate of the thermal image. First, to adjust the large-scale difference, a pre-transformation is performed, in which the thermal and visible images are coarsely aligned by automatic selection of common keypoints in the image pair using a calibration rig. Then, the fine registration is performed by extracting phase-congruent feature points in the image pair for more precise registration. The particle swarm optimization (PSO) based fast sample consensus (PSOFSC) algorithm is also introduced for selecting correct inlier correspondence pairs. The proposed method was validated using several thermal-visible face image pairs with various resolutions from different cameras, acquired in different scenarios. The results are encouraging when compared to many state-of-the-art registration methods for thermal-visible image pairs.

LAMP-HQ: A Large-Scale Multi-pose High-Quality Database and Benchmark for NIR-VIS Face Recognition

Near-infrared-visible (NIR-VIS) heterogeneous face recognition matches NIR to corresponding VIS face images. However, due to the sensing gap, NIR images often lose some identity information so that the NIR-VIS recognition issue is more difficult than conventional VIS face recognition. Recently, NIR-VIS heterogeneous face recognition has attracted considerable attention in the computer vision community because of its convenience and adaptability in practical applications. Various deep learning-based methods have been proposed and substantially increased the recognition performance, but the lack of NIR-VIS training samples leads to the difficulty of the model training process. In this paper, we propose a new $$\mathbf{L} {} \mathbf{a} $$ rge-Scale $$\mathbf{M} $$ ulti- $$\mathbf{P} $$ ose $$\mathbf{H} $$ igh- $$\mathbf{Q} $$ uality NIR-VIS database ‘ $$\mathbf{LAMP}-HQ $$ ’ containing 56,788 NIR and 16,828 VIS images of 573 subjects with large diversities in pose, illumination, attribute, scene and accessory. We furnish a benchmark along with the protocol for NIR-VIS face recognition via generation on LAMP-HQ, including Pixel2-Pixel, CycleGAN, ADFL, PCFH, and PACH. Furthermore, we propose a novel exemplar-based variational spectral attention network to produce high-fidelity VIS images from NIR data. A spectral conditional attention module is introduced to reduce the domain gap between NIR and VIS data and then improve the performance of NIR-VIS heterogeneous face recognition on various databases including the LAMP-HQ.

Adversarial Cross-Spectral Face Completion for NIR-VIS Face Recognition

Near infrared-visible (NIR-VIS) heterogeneous face recognition refers to the process of matching NIR to VIS face images. Current heterogeneous methods try to extend VIS face recognition methods to the NIR spectrum by synthesizing VIS images from NIR images. However, due to the self-occlusion and sensing gap, NIR face images lose some visible lighting contents so that they are always incomplete compared to VIS face images. This paper models high-resolution heterogeneous face synthesis as a complementary combination of two components: a texture inpainting component and a pose correction component. The inpainting component synthesizes and inpaints VIS image textures from NIR image textures. The correction component maps any pose in NIR images to a frontal pose in VIS images, resulting in paired NIR and VIS textures. A warping procedure is developed to integrate the two components into an end-to-end deep network. A fine-grained discriminator and a wavelet-based discriminator are designed to improve visual quality. A novel 3D-based pose correction loss, two adversarial losses, and a pixel loss are imposed to ensure synthesis results. We demonstrate that by attaching the correction component, we can simplify heterogeneous face synthesis from one-to-many unpaired image translation to one-to-one paired image translation, and minimize the spectral and pose discrepancy during heterogeneous recognition. Extensive experimental results show that our network not only generates high-resolution VIS face images but also facilitates the accuracy improvement of heterogeneous face recognition.

Infrared and visible face fusion recognition based on extended sparse representation classification and local binary patterns for the single sample problem

While near infrared and visible fusion recognition has been actively researched in recent years, most theoretical results and algorithms concentrate on the sufficient training samples setting. This paper focuses on the general fusion method when there are insufficient training samples with one pair of near-infrared and visible face images. Compared with existing methods, the proposed method requires neither sufficient samples nor the training step. To get a robust and time-efficient fusion model for unconstrained face recognition in the single sample situation, two models are proposed to fuse the local binary pattern based descriptors and the sparse representation based classification: the first fusion model directly fuses the representation error, while the second fusion model is an accelerated version that learns from a cross-spectral dictionary. Experiments are performed on the HITSZ LAB2 database, and the experiment results showed that the proposed fusion model extracted the complementary features of near-infrared and visible-light images. The fusion face recognition method had superior performance to state of the art fusion methods.

Human authentication based on fusion of thermal and visible face images

In recent past, considerable amount of research has been done to increase the performance of a face authentication system in uncontrolled environment such as illumination. However, the performance has not been improved significantly since visible face images are dependent of illumination. To overcome the limitation of visible face images, researchers are using infrared (IR) face images. However, it also does not completely independent of illumination. Image fusion of visible and thermal face images is an alternative in research community nowadays. In this work, a fusion method is introduced to fuse visible and IR images for face authentication. The proposed fusion method relies on translation invariant A-trous wavelet transform and fractal dimension using differential box counting method. Five popular fusion metrics namely, ratio of spatial frequency error, normalized mutual information, edge information, universal image quality index, extended frequency comparison index are considered to measure the effectiveness of the proposed fusion algorithm quantitatively over four state-of-the-art methods. A new similarity measure is also proposed to check how close a fused face image from others are. All the experiments are performed on three databases namely, IRIS benchmark face database, UGC-JU face database and SCface face database. All the results depict that the proposed fusion method along with similarity measure for face authentication outperforms all the four state-of-the-art methods in terms of accuracy, precision and recall.

À‐trous wavelet transform‐based hybrid image fusion for face recognition using region classifiers

AbstractThis paper presents a new hybrid fusion framework based on thermal and visible face images. Fusion of information is done here in two phases, first at the pixel level and then at the decision level. For the pixel level fusion process, à‐trous wavelet transform is applied on both the thermal and visible face images. In decision level fusion, 34 region classifiers, each concentrating on a specified region of the face image, are tested individually for their ability to identify a person from the face image. The region classifiers, which contribute significantly in recognizing the face image, are considered for decision level fusion using majority voting. All experiments have been conducted on the UGC‐JU face database and IRIS benchmark face database. The maximum recognition rate is about 97.22% for both the databases whereas decisions of 17 region classifiers among 34 are considered. Experimental results and comparative study show that the proposed fusion method provides a framework for recognition of face images in uncontrolled environments such as variations in illumination conditions, pose, and facial expressions.

Bridging the spectral gap using image synthesis: a study on matching visible to passive infrared face images

We propose an approach that bridges the gap between the visible and IR band of the electromagnetic spectrum, namely the mid-wave infrared or MWIR (3---5 $$\upmu \hbox {m}$$μm) and the long-wave infrared or LWIR (8---14 $$\upmu \hbox {m}$$μm) bands. Specifically, we investigate the benefits and limitations of using synthesized visible face images from thermal and vice versa, in cross-spectral face recognition systems when utilizing canonical correlation analysis and manifold learning dimensionality reduction. There are four primary contributions of this work. First, we assemble a database of frontal face images composed of paired VIS-MWIR and VIS-LWIR face images (using different methods for pre-processing and registration). Second, we formulate a image synthesis framework and post-synthesis restoration methodology, to improve face recognition accuracy. Third, we explore cohort-specific matching (per gender) instead of blind-based matching (when all images in the gallery are matched against all in the probe set). Finally, by conducting an extensive experimental study, we establish that the proposed scheme increases system performance in terms of rank-1 identification rate. Experimental results suggest that matching visible images against images acquired with passive infrared spectrum, and vice-versa, are feasible with promising results.

Kernel-aligned multi-view canonical correlation analysis for image recognition

Existing kernel-based correlation analysis methods mainly adopt a single kernel in each view. However, only a single kernel is usually insufficient to characterize nonlinear distribution information of a view. To solve the problem, we transform each original feature vector into a 2-dimensional feature matrix by means of kernel alignment, and then propose a novel kernel-aligned multi-view canonical correlation analysis (KAMCCA) method on the basis of the feature matrices. Our proposed method can simultaneously employ multiple kernels to better capture the nonlinear distribution information of each view, so that correlation features learned by KAMCCA can have well discriminating power in real-world image recognition. Extensive experiments are designed on five real-world image datasets, including NIR face images, thermal face images, visible face images, handwritten digit images, and object images. Promising experimental results on the datasets have manifested the effectiveness of our proposed method.

이질적 얼굴인식을 위한 심층 정준상관분석을 이용한 지역적 얼굴 특징 학습 방법

Face recognition has received a great deal of attention for the wide range of applications in real-world scenario. In this scenario, mismatches (so called heterogeneity) in terms of resolution and illumination between gallery and test face images are inevitable due to the different capturing conditions. In order to deal with the mismatch problem, we propose a local feature learning method using deep canonical correlation analysis (DCCA) for heterogeneous face recognition. By the DCCA, we can effectively reduce the mismatch between the gallery and the test face images. Furthermore, the proposed local feature learned by the DCCA is able to enhance the discriminative power by using facial local structure information. Through the experiments on two different scenarios (i.e., matching near-infrared to visible face images and matching low-resolution to high-resolution face images), we could validate the effectiveness of the proposed method in terms of recognition accuracy using publicly available databases.

Matching thermal to visible face images using hidden factor analysis in a cascaded subspace learning framework

Matching thermal (THM) face images against visible (VIS) face images poses a significant challenge to automated face recognition systems. In this work, we introduce a Heterogeneous Face Recognition (HFR) matching framework, which uses multiple sets of subspaces generated by sampling patches from VIS and THM face images and subjecting them to a sequence of transformations. In the training phase of the proposed scheme, face images from VIS and THM are subjected to three different filters separately and then tessellated into patches. Each patch is represented by either a Pyramid Scale Invariant Feature Transform (PSIFT) or Histograms of Principal Oriented Gradients (HPOG). Then, a cascaded subspace learning process consisting of whitening transformation, factor analysis, and common discriminant analysis is used to construct multiple common subspaces between VIS and THM facial images. During the testing phase, the projected feature vectors from individual subspaces are concatenated to form a final feature vector. Nearest Neighbor (NN) classifier is used to compare feature vectors and the resulting scores corresponding to three filtered images are combined via the sum-rule. The proposed face matching algorithm is evaluated on two multispectral face datasets and is shown to achieve very promising results.

A novel template protection scheme for multibiometrics based on fuzzy commitment and chaotic system

In recent years, biometrics template protection has been extensively studied and lots of schemes have been proposed. However, most of them have not considered the forgery, large difference of intra-class and the security of unimodal biometrics leakage. And there is no multibiometrics template scheme based on the fusion of dual iris, thermal and visible face images. In this paper, a novel multibiometrics template protection scheme based on fuzzy commitment and chaotic system, and the security analysis approach for unimodal biometrics leakage are proposed. Firstly, the thermal face images are captured to overcome the forgery. Then, the fuzzy commitment is generated from the corporation of error correcting code (ECC) and the fusion binary features. Additionally, the dual iris feature vectors are encrypted via the chaotic system, and the score level fusion based on Aczel-Alsina triangular-norm (AA T-norm) is implemented to acquire the final verification performance. Finally, the entropy of both mutlibiometrics and unimodal information leakage is analyzed to show the security of the proposed approach. The experimental tests are conducted on a virtual multibiometrics database, which merges the challenging CASIA-Iris-Thousand and the NVIE face database. The verification performance decreases from EER of \(3 \times 10^{-2}\) to \(1.163 \times 10^{-1}\) %, but the multibiometrics template security is enhanced from 80.53 to 167.80 bits based on BCH ECC (1,023, 123, 170).

Adaptive polar transform and fusion for human face image processing and evaluation

AbstractHuman face processing and evaluation is a problem due to variations in orientation, size, illumination, expression, and disguise. The goal of this work is threefold. First, we aim to show that the variant of polar transformation can be used to register face images against changes in pose and size. Second, implementation of fusion of thermal and visual face images in the wavelet domain to handle illumination and disguise and third, principal component analysis is applied in order to tackle changes due to expressions up to a particular extent of degrees. Finally, a multilayer perceptron has been used to classify the face image. Several techniques have been implemented here to depict an idea about improvement of results. Methods started from the simplest design, without registration; only combination of PCA and MLP as a method for dimensionality reduction and classification respectively to the range of adaptive polar registration, fusion in wavelet transform domain and final classification using MLP. A consistent increase in recognition performance has been observed. Experiments were conducted on two separate databases and results yielded are very much satisfactory for adaptive polar registration along with fusion of thermal and visual images in the wavelet domain.

Matching NIR Face to VIS Face Using Transduction

Visual versus near infrared (VIS-NIR) face image matching uses an NIR face image as the probe and conventional VIS face images as enrollment. It takes advantage of the NIR face technology in tackling illumination changes and low-light condition and can cater for more applications where the enrollment is done using VIS face images such as ID card photos. Existing VIS-NIR techniques assume that during classifier learning, the VIS images of each target people have their NIR counterparts. However, since corresponding VIS-NIR image pairs of the same people are not always available, which is often the case, so those methods cannot be applied. To address this problem, we propose a transductive method named transductive heterogeneous face matching (THFM) to adapt the VIS-NIR matching learned from training with available image pairs to all people in the target set. In addition, we propose a simple feature representation for effective VIS-NIR matching, which can be computed in three steps, namely Log-DoG filtering, local encoding, and uniform feature normalization, to reduce heterogeneities between VIS and NIR images. The transduction approach can reduce the domain difference due to heterogeneous data and learn the discriminative model for target people simultaneously. To the best of our knowledge, it is the first attempt to formulate the VIS-NIR matching using transduction to address the generalization problem for matching. Experimental results validate the effectiveness of our proposed method on the heterogeneous face biometric databases.

NEI Facial Expressions for Identification

This paper presents North East Indian (NEI) facial expression analysis using visual face images. A short review on relative works has been illustrated. Visual face images have been collected from tribes and non-tribes of North East Indian people. Six basic facial expressions i.e. anger, happy, sad, surprise, fear and disgust have been captured in NEI face database. The facial feature changes occurred during different expressions have been analyzed. This analysis shows that captured facial expressions meet the standard facial expressions. These face images can be taken for face identification purpose. Automatic facial expression identification system may be generated using the NEI facial expression database.

Regularized Transfer Boosting for Face Detection Across Spectrum

This letter addresses the problem of face detection in multispectral illuminations. Face detection in visible images has been well addressed based on the large scale training samples. For the recently emerging multispectral face biometrics, however, the face data is scarce and expensive to collect, and it is usually short of face samples to train an accurate face detector. In this letter, we propose to tackle the issue of multispectral face detection by combining existing large scale visible face images and a few multispectral face images. We cast the problem of face detection across spectrum into the transfer learning framework and try to learn the robust multispectral face detector by exploring relevant knowledge from visible data domain. Specifically, a novel Regularized Transfer Boosting algorithm named R-TrBoost is proposed, with features of weighted loss objective and manifold regularization. Experiments are performed with face images of two spectrums, 850 nm and 365 nm, and the results show significant improvement on multispectral face detection using the proposed algorithm.