Visible-to-Thermal Transfer Learning for Facial Landmark Detection

Domenick D Poster,Shuowen Hu,Nasser M Nasrabadi,Benjamin S Riggan,Nathan J Short

doi:10.1109/access.2021.3070233

Abstract

There has been increasing interest in face recognition in the thermal infrared spectrum. A critical step in this process is face landmark detection. However, landmark detection in the thermal spectrum presents a unique set of challenges compared to in the visible spectrum: inherently lower spatial resolution due to longer wavelength, differences in phenomenology, and limited availability of labeled thermal face imagery for algorithm development and training. Thermal infrared imaging does have the advantage of being able to passively acquire facial heat signatures without the need for active or ambient illumination in low light and nighttime environments. In such scenarios, thermal imaging must operate by itself without corresponding/paired visible imagery. Mindful of this constraint, we propose visible-to-thermal parameter transfer learning using a coupled convolutional network architecture as a means to leverage visible face data when training a model for thermal-only face landmark detection. This differentiates our approach from models trained either solely on thermal images or models which require a fusion of visible and thermal images at test time. In this work, we implement and analyze four types of parameter transfer learning methods in the context of thermal face landmark detection: Siamese (shared) layers, Linear Layer Regularization (LLR), Linear Kernel Regularization (LKR), and Residual Parameter Transformations (RPT). These transfer learning approaches are compared against a baseline version of the network and an Active Appearance Model (AAM), both of which are trained only on thermal data. We achieve a 6.5% - 9.5% improvement on the DEVCOM ARL Multi-modal Thermal Face Dataset and a 4% improvement on the RWTH Aachen University Thermal Face Dataset over the baseline model. We show that LLR, LKR, and RPT all result in improved thermal face landmark detection performance compared to the baseline and AAM, demonstrating that transfer learning leveraging visible spectrum data improves thermal face landmarking.

Highlights

Landmark detection is a critical component for facial analysis applications, including face recognition, 3D modeling, and expression classification
We provide the Cumulative Error Distribution (CED) curve, displaying the proportion of images with Normalized Root Mean Square Error (NME) falling below threshold values ranging from 0% to 10%, as well as the Area Under the CED Curve (AUC10%) and the Failure Rate at 10% (FR10%), defined as the proportion of test images with a NME (%) greater than 10%
This work demonstrates that thermal face landmark detection can be improved via transfer learning with visible data

Summary

Introduction

Landmark detection is a critical component for facial analysis applications, including face recognition, 3D modeling, and expression classification. Precise and accurate detection of facial landmarks enable faces to be registered (or aligned) to a common frame of reference, often referred to as canonical coordinates. A significant amount of landmark detection research has been performed on visible spectrum imagery under a wide array of conditions, such as variable pose, illumination, expression, and occlusion, driven by applications in the commercial and government sectors. There has been substantially less landmark detection research for thermal infrared imagery. The primary advantage of thermal imagery is it can be captured by a passive system requiring no illumination. It has been shown that the fusion of thermal and visible face images can lead to increased facial recognition performance [5], [6]. Landmark-annotated face datasets in the thermal spectrum contain substantially

Objectives

Results

Conclusion