Infrared Face Research Articles

Application of Visual Transformer in Low-resolution Thermal Infrared Image Recognition

Abstract Addressing the challenges of inadequate accuracy and limited robustness exhibited by current lightweight object detection networks specifically tailored for low-resolution thermal infrared face detection scenarios, this paper delves into developing an ultra-lightweight thermal infrared face detection algorithm that leverages visual attention mechanisms. To ascertain the optimal neural network complexity, a series of comparative experiments are meticulously conducted. With Yolo-FastestDet serving as the benchmark, this study endeavors to compress the backbone network, striking a delicate balance between network depth and detection speed. Additionally, to bolster the network’s capacity for profound feature extraction and precise discrimination of target edges and small objects, a Transformer-Encoder-based visual attention module is seamlessly integrated. Consequently, a lightweight face detection algorithm, enriched with attention mechanisms, is formulated. Furthermore, to mitigate the scarcity of low-resolution infrared face image data, a self-constructed visible-infrared face dataset is employed for training and evaluation purposes. The experimental outcomes reveal that the proposed algorithm attains an impressive mAP@0.5 score of 0.953 on the test dataset while satisfying the stringent real-time detection criterion of 30 frames per second (FPS) when deployed on an embedded Raspberry Pi CPU.

Dual Modal Face Recognition Based on Visible Light and Thermal Infrared

Abstract In recent years, face recognition has received widespread attention, in the use of visible light for face recognition, there are a series of problems that require auxiliary light sources, may be attacked by photos, and camouflage is difficult to penetrate. Compared with visible light recognition, thermal infrared face recognition shows usability, and it can still be stable in low light or dark extreme environmental conditions, but due to low resolution, it can not capture rich face feature information. In view of this paper proposes that dual mode face recognition can effectively integrate two single mode information and make up for the deficiency of single feature. Using a Bimodal Face network model based on visible light and thermal infrared, the idea of feature decoupling is introduced into image fusion. The cross-modal information is decomposed into common information and unique information, and the common information and unique information of different modes are fused respectively according to the feature fusion method. It is tested on the self-built dual modal dataset that this fusion strategy can achieve the optimal effect, and the accuracy rate can reach 97.08%, which is better than other five face recognition network models.

FeverNet: Enabling accurate and robust remote fever screening

Remote human fever screening via thermal infrared imaging helps reduce the risk of respiratory disease transmission and plays an important role in public health monitoring. However, the accuracy of such systems often falls prey to variations in measurement distance and environment temperature. Most previous methods tend to employ sensors to overcome these variations, which are expensive schemes and have limited performance improvement. To address above problems, this paper presents a novel and robust remote fever screening framework named FeverNet. Specifically, FeverNet introduces depth estimation network and temperature distribution constraints across time periods to reduce the influence of distance variations and environment temperature changes. The fever attention module is thus proposed to enhance feature representation and expand the difference between fever faces and normal ones. In addition, we provide the Extended Thermal Infrared Face dataset (ETIF), which further gives visible images (paired with thermal infrared images) for depth estimation and improve the fever face generated method based on the maximum temperature of the face. Extensive experiments on ETIF demonstrate the advantages of our FeverNet over the state-of-the-art methods.

Convergent validity of video-based observer rating of drowsiness, against subjective, behavioral, and physiological measures.

Driver drowsiness is a widely recognized cause of motor vehicle accidents. Therefore, a reduction in drowsy driving crashes is required. Many studies evaluating the crash risk of drowsy driving and developing drowsiness detection systems, have used observer rating of drowsiness (ORD) as a reference standard (i.e. ground truth) of drowsiness. ORD is a method of human raters evaluating the levels of driver drowsiness, by visually observing a driver. Despite the widespread use of ORD, concerns remain regarding its convergent validity, which is supported by the relationship between ORD and other drowsiness measures. The objective of the present study was to validate video-based ORD, by examining correlations between ORD levels and other drowsiness measures. Seventeen participants performed eight sessions of a simulated driving task, verbally responding to Karolinska sleepiness scale (KSS), while infra-red face video, lateral position of the participant's car, eye closure, electrooculography (EOG), and electroencephalography (EEG) were recorded. Three experienced raters evaluated the ORD levels by observing facial videos. The results showed significant positive correlations between the ORD levels and all other drowsiness measures (i.e., KSS, standard deviation of the lateral position of the car, percentage of time occupied by slow eye movement calculated from EOG, EEG alpha power, and EEG theta power). The results support the convergent validity of video-based ORD as a measure of driver drowsiness. This suggests that ORD might be suitable as a ground truth for drowsiness.

Dynamic Group Difference Coding Based on Thermal Infrared Face Image for Fever Screening

Recently, noncontact temperature measurement methods based on infrared face perception have received widely attentions since fever screening plays an important role in the early prediction of respiratory infections, such as SARS, H1N1, and COVID-19. However, the performance of these methods always significantly degrades when facing the changes of environment. Thus, the majority of these methods leverage the block-body and sensors to reduce the influence of environment changes. It is a pity that the increased instrument complexity leads to higher costs and failure rate. To address the aforementioned issues, this article presents a novel fever screening method, named dynamic group difference coding (DGDC), which is based on the analysis about the influencing factors. The key idea of DGDC is to compute the temperature differences between the target person and the recently passed crowd (dynamic group). Specifically, we develop the face temperature encoder (FTE) to describe the face temperature and thus construct the difference matrix of the embedding feature between the target person and the dynamic group. Multilayer perceptions (MLP) are employed to capture the intrinsic information by characterizing the difference matrix in vertical and horizontal directions, respectively. Finally, we provide a dataset of thermal infrared face (TIF) images and conduct extensive experiments to demonstrate the advantages of the proposed method over the competing methods.

Charlotte-ThermalFace: A Fully Annotated Thermal Infrared Face Dataset with Various Environmental Conditions and Distances

Facial thermography is one of the most popular research areas in infrared thermal imaging, with diverse applications in medical, surveillance, and environmental monitoring. However, in contrast to facial imagery in the visual spectrum, the lack of public datasets on facial thermal images is an obstacle to research improvement in this area. Thermal face imagery is still a relatively new research area to be evaluated and studied in different domains.The current thermal face datasets are limited in regards to the subjects’ distance from the camera, the ambient temperature variation, and facial landmarks’ localization. We address these gaps by presenting a new facial thermography dataset.This article makes two main contributions to the body of knowledge. First, it presents a comprehensive review and comparison of current public datasets in facial thermography. Second, it introduces and studies a novel public dataset on facial thermography, which we call it Charlotte-ThermalFace. Charlotte-ThermalFace contains more than10000 infrared thermal images in varying thermal conditions, several distances from the camera, and different head positions. The data is fully annotated with the facial landmarks, ambient temperature, relative humidity, the air speed of the room, distance to the camera, and subject thermal sensation at the time of capturing each image. Our dataset is the first publicly available thermal dataset annotated with the thermal sensation of each subject in different thermal conditions and one of the few datasets in raw 16-bit format. Finally, we present a preliminary analysis of the dataset to show the applicability and importance of the thermal conditions in facial thermography. The full dataset, including annotations, are freely available for research purpose at https://github.com/TeCSAR-UNCC/UNCC-ThermalFace.

Fast face detection using a unified architecture for unconstrained and infrared face images

Human face detection along with its localization is a difficult task when the face is presented in the cluttered scene in an unconstrained scenario that might be with arbitrary pose variations, occlusions, random backgrounds and infrared (IR) environment. This paper proposes a novel face detection method which can address some of these issues and challenges quite successfully during face detection in unconstrained as well as infrared environments. It makes use of Fast Successive Mean Quantization Transform (FastSMQT) features for image enhancement and feature representation to deal with illumination and sensor insensitiveness. A split up Sparse Network of Winnows (SNoW) with Winnow updating rule is then exploited to speed up the original SNoW classifier. Finally, the features and classifiers are combined together with skin detection algorithm for face detection in crowd image and head orientation correction of near infrared faces. The proposed face detector is robust in handling pose, occlusion, illumination, blur and low image resolution. The experiment is performed on six challenging and publicly available databases, viz. BIOID, LFW, FDDB, UFI, WIDER FACE and IIT Delhi near infrared. The experimental results depict that the proposed method outperforms traditional as well as some advanced methods in detecting unconstrained and infrared faces under challenging situations.

Smart Door Camera with Facial Recognition Feature for Thermal Screening

Abstract: In these times of COVID-19, it is essential to go through thermal screening for checking one’s body temperature before entering any premises. However, it is a tiring process as it involves measuring body temperature of all people, one at atime. At the same time, those who carry out thermal screening are required to stand for more than 8 hours a day and check each and every person. This takes a lot of time and effort. So to come up with a solution that can do this job effortlessly, we have built a Facial Recognition Thermal Screening System. The device works by recognizing the face of each person and doing thermal screening to detect the body temperature. If a person is found to have a very high temperature, then the systemwill not allow entry and instead will automatically notify that person to take a COVID-19 test. If the body temperature falls between the required normal temperature range and is found to be okay, then entry is allowed after proper sanitization Keywords: Face Recognition, Covid-19, Thermal Scanning, Raspberry-pi, Thermal Sensor .The temporal variations present in thermal face images are mainly due to different environmental conditions, physiological changes of the subjects, and differences of the infrared detectors’ responsivity at the time of the capture, which affect the performance of infrared face recognition systems. To perform this paper, we created two thermal face databases that include capture sessions with real and variable conditions. We also propose two criteria to quantify the temporal variations between data sets. The thermal face recognition systems have been developed using the following five methods: local binary pattern (LBP), Weber linear descriptor (WLD), Gabor jet descriptors, scale invariant feature transform, and speeded up robust features. The results indicate that the local matching-based methods (WLD and LBP) are mostly immune to temporal variations, which is noticeable when the face images have been acquired with a time lapse, while the rest of the methods are clearly affected and are not suitable for practical infrared face recognition.

Thermal Infrared Face Recognition.

The technology for deep learning in the field of thermal infrared face recognition has recently become more available for use in research, therefore allowing for the many groups working on this subject to achieve many novel findings. Thermal infrared face recognition helps recognize faces that are not able to be recognized in visible light and can additionally recognize facial blood vessel structure. Previous research regarding temperature variations, mathematical formulas, wave types, and methods in thermal infrared face recognition is reviewed.

Convolutional Neural Networks for Semantic Segmentation as a Tool for Multiclass Face Analysis in Thermal Infrared

Convolutional neural networks were used for multiclass segmentation in thermal infrared face analysis. The principle is based on existing image-to-image translation approaches, where each pixel in an image is assigned to a class label. We show that established networks architectures can be trained for the task of multiclass face analysis in thermal infrared. Created class annotations consisted of pixel-accurate locations of different face classes. Subsequently, the trained network can segment an acquired unknown infrared face image into the defined classes. Furthermore, face classification in live image acquisition is shown, in order to be able to display the relative temperature in real-time from the learned areas. This allows a pixel-accurate temperature face analysis e.g. for infection detection like Covid-19. At the same time our approach offers the advantage of concentrating on the relevant areas of the face. Areas of the face irrelevant for the relative temperature calculation or accessories such as glasses, masks and jewelry are not considered. A custom database was created to train the network. The results were quantitatively evaluated with the intersection over union (IoU) metric. The methodology shown can be transferred to similar problems for more quantitative thermography tasks like in materials characterization or quality control in production.

An efficient deep learning model for classification of thermal face images

PurposeThe objective of this paper is to perform infrared (IR) face recognition efficiently with convolutional neural networks (CNNs). The proposed model in this paper has several advantages such as the automatic feature extraction using convolutional and pooling layers and the ability to distinguish between faces without visual details.Design/methodology/approachA model which comprises five convolutional layers in addition to five max-pooling layers is introduced for the recognition of IR faces.FindingsThe experimental results and analysis reveal high recognition rates of IR faces with the proposed model.Originality/valueA designed CNN model is presented for IR face recognition. Both the feature extraction and classification tasks are incorporated into this model. The problems of low contrast and absence of details in IR images are overcome with the proposed model. The recognition accuracy reaches 100% in experiments on the Terravic Facial IR Database (TFIRDB).

Disentangled Variational Representation for Heterogeneous Face Recognition

Visible (VIS) to near infrared (NIR) face matching is a challenging problem due to the significant domain discrepancy between the domains and a lack of sufficient data for training cross-modal matching algorithms. Existing approaches attempt to tackle this problem by either synthesizing visible faces from NIR faces, extracting domain-invariant features from these modalities, or projecting heterogeneous data onto a common latent space for cross-modal matching. In this paper, we take a different approach in which we make use of the Disentangled Variational Representation (DVR) for crossmodal matching. First, we model a face representation with an intrinsic identity information and its within-person variations. By exploring the disentangled latent variable space, a variational lower bound is employed to optimize the approximate posterior for NIR and VIS representations. Second, aiming at obtaining more compact and discriminative disentangled latent space, we impose a minimization of the identity information for the same subject and a relaxed correlation alignment constraint between the NIR and VIS modality variations. An alternative optimization scheme is proposed for the disentangled variational representation part and the heterogeneous face recognition network part. The mutual promotion between these two parts effectively reduces the NIR and VIS domain discrepancy and alleviates over-fitting. Extensive experiments on three challenging NIR-VIS heterogeneous face recognition databases demonstrate that the proposed method achieves significant improvements over the state-of-the-art methods.

Human authentication based on fusion of thermal and visible face images

In recent past, considerable amount of research has been done to increase the performance of a face authentication system in uncontrolled environment such as illumination. However, the performance has not been improved significantly since visible face images are dependent of illumination. To overcome the limitation of visible face images, researchers are using infrared (IR) face images. However, it also does not completely independent of illumination. Image fusion of visible and thermal face images is an alternative in research community nowadays. In this work, a fusion method is introduced to fuse visible and IR images for face authentication. The proposed fusion method relies on translation invariant A-trous wavelet transform and fractal dimension using differential box counting method. Five popular fusion metrics namely, ratio of spatial frequency error, normalized mutual information, edge information, universal image quality index, extended frequency comparison index are considered to measure the effectiveness of the proposed fusion algorithm quantitatively over four state-of-the-art methods. A new similarity measure is also proposed to check how close a fused face image from others are. All the experiments are performed on three databases namely, IRIS benchmark face database, UGC-JU face database and SCface face database. All the results depict that the proposed fusion method along with similarity measure for face authentication outperforms all the four state-of-the-art methods in terms of accuracy, precision and recall.

Thermal face segmentation based on circular shortest path

Recently, research on thermal infrared human face analysis grows rapidly. However, due to the low contrast and limited information, human face in thermal infrared images is difficult to segment precisely. To overcome these shortcomings of thermal images, we propose an improved circular shortest path method in this paper. In our method, the proposed gradient based cost function enhances the gradient information and extracts detail of the original image. In addition, we propose a shape constraint by using ellipse derivative in the cost function. The constraint helps the contour to conform to the real human face. Moreover, the proposed certainty penalty term and straight path penalty term restrain the effect of the local minima regions and improve the robustness of our method. Our method could effectively extract the precise human face contour and thus segment the complete human face. Experimental results show that our method performs well for thermal infrared face segmentation, in both visual and quantitative ways.

R-theta local neighborhood pattern for unconstrained facial image recognition and retrieval

In this paper R-Theta Local Neighborhood Pattern (RTLNP) is proposed for facial image retrieval. RTLNP exploits relationships amongst the pixels in local neighborhood of the reference pixel at different angular and radial widths. The proposed encoding scheme divides the local neighborhood into sectors of equal angular width. These sectors are again divided into subsectors of two radial widths. Average grayscales values of these two subsectors are encoded to generate the micropatterns. Performance of the proposed descriptor has been evaluated and results are compared with the state of the art descriptors e.g. LBP, LTP, CSLBP, CSLTP, Sobel-LBP, LTCoP, LMeP, LDP, LTrP, MBLBP, BRINT and SLBP. The most challenging facial constrained and unconstrained databases, namely; AT&T, CARIA-Face-V5-Cropped, LFW, and Color FERET have been used for showing the efficiency of the proposed descriptor. Proposed descriptor is also tested on near infrared (NIR) face databases; CASIA NIR-VIS 2.0 and PolyU-NIRFD to explore its potential with respect to NIR facial images. Better retrieval rates of RTLNP as compared to the existing state of the art descriptors show the effectiveness of the descriptor

Regional parallel structure based CNN for thermal infrared face identification

Regional parallel structure based CNN for thermal infrared face identification

Learning Invariant Deep Representation for NIR-VIS Face Recognition

Visual versus near infrared (VIS-NIR) face recognition is still a challenging heterogeneous task due to large appearance difference between VIS and NIR modalities. This paper presents a deep convolutional network approach that uses only one network to map both NIR and VIS images to a compact Euclidean space. The low-level layers of this network are trained only on large-scale VIS data. Each convolutional layer is implemented by the simplest case of maxout operator. The high-level layer is divided into two orthogonal subspaces that contain modality-invariant identity information and modality-variant spectrum information respectively. Our joint formulation leads to an alternating minimization approach for deep representation at the training time and an efficient computation for heterogeneous data at the testing time. Experimental evaluations show that our method achieves 94% verification rate at FAR=0.1% on the challenging CASIA NIR-VIS 2.0 face recognition dataset. Compared with state-of-the-art methods, it reduces the error rate by 58% only with a compact 64-D representation.

Thermal Face Recognition Under Temporal Variation Conditions

In this paper, we analyze the problems produced by temporal variations of infrared face images when used in face recognition systems. The temporal variations present in thermal face images are mainly due to different environmental conditions, physiological changes of the subjects, and differences of the infrared detectors’ responsivity at the time of the capture, which affect the performance of infrared face recognition systems. To perform this paper, we created two thermal face databases that include capture sessions with real and variable conditions. We also propose two criteria to quantify the temporal variations between data sets. The thermal face recognition systems have been developed using the following five methods: local binary pattern (LBP), Weber linear descriptor (WLD), Gabor jet descriptors, scale invariant feature transform, and speeded up robust features. The results indicate that the local matching-based methods (WLD and LBP) are mostly immune to temporal variations, which is noticeable when the face images have been acquired with a time lapse, while the rest of the methods are clearly affected and are not suitable for practical infrared face recognition.

Composite multilobe descriptors for cross-spectral recognition of full and partial face

Cross-spectral image matching is a challenging research problem motivated by various applications, including surveillance, security, and identity management in general. An example of this problem includes cross-spectral matching of active infrared (IR) or thermal IR face images against a dataset of visible light images. A summary of recent developments in the field of cross-spectral face recognition by the authors is presented. In particular, it describes the original form and two variants of a local operator named composite multilobe descriptor (CMLD) for facial feature extraction with the purpose of cross-spectral matching of near-IR, short-wave IR, mid-wave IR, and long-wave IR to a gallery of visible light images. The experiments demonstrate that the variants of CMLD outperform the original CMLD and other recently developed composite operators used for comparison. In addition to different IR spectra, various standoff distances from close-up (1.5 m) to intermediate (50 m) and long (106 m) are also investigated. Performance of CMLD I to III is evaluated for each of the three cases of distances. The newly developed operators, CMLD I to III, are further utilized to conduct a study on cross-spectral partial face recognition where different facial regions are compared in terms of the amount of useful information they contain for the purpose of conducting cross-spectral face recognition. The experimental results show that among three facial regions considered in the experiments the eye region is the most informative for all IR spectra at all standoff distances.

Study of Local Matching-Based Facial Recognition Methods Using Thermal Infrared Imagery

This paper shows a comparative study among different local matching-based methods for thermal infrared face recognition. The principal assumption of this work is that the thermal face corresponds to the diffuse energy emission captured by an infrared camera, where the thermal signature is unique for each subject and it can be addressed as a texture descriptor with thermal images. Local matching-based methods find inter-class differences that improve the face recognition rate in thermal spectrum. Specifically, this work considers four methods: Local Binary Pattern (LBP), Local Derivative Pattern (LDP), Weber Linear Descriptor (WLD) and Histograms of Oriented Gradients Descriptors (HOG). The methods are evaluated and compared using the UCHThermalFace database, that considers real-world conditions and unconstrained environments, such as indoor and outdoor setups, natural variations in illumination, facial expression, pose, accessories, occlusions, and background. Results indicate that HOG variants followed by LBP method achieved the best recognition rates for face recognition systems.