Implementation of selected ISO/IEC 29794-5 measures and proposing alternatives

Considerable research efforts have been made for face recognition in various real-world applications. However, degraded face images, acquired in the real-world, make face recognition difficult. In this paper, we propose a new face image quality assessment that aims to realize a robust and reliable face recognition system. The proposed method considers two factors for face image quality, i.e., visual quality and mismatch between training and test face images. A face image quality assessor is learned based on the two factors to discriminate useful faces from unuseful ones. The proposed face image quality assessment model is robust and adaptive to face recognition systems by employing a learned assessment. Our experimental results on a challenging database show significant improvement in face recognition accuracy by the proposed method.

Face image quality assessment (FIQA) has been an area of interest to researchers as a way to improve the face recognition accuracy. By filtering out the low quality images we can reduce various difficulties faced in unconstrained face recognition, such as, failure in face or facial landmark detection or low presence of useful facial information. In last decade or so, researchers have proposed different methods to assess the face image quality, spanning from fusion of quality measures to using learning based methods. Different approaches have their own strength and weaknesses. But, it is hard to perform a comparative assessment of these methods without a database containing wide variety of face quality, a suitable training protocol that can efficiently utilize this large-scale dataset. In this thesis we focus on developing an evaluation platfrom using a large scale face database containing wide ranging face image quality and try to deconstruct the reason behind the predicted scores of learning based face image quality assessment methods. Contributions of this thesis is two-fold. Firstly, (i) a carefully crafted large scale database dedicated entirely to face image quality assessment has been proposed; (ii) a learning to rank based large-scale training protocol is devel- oped. Finally, (iii) a comprehensive study of 15 face image quality assessment methods using 12 different feature types, and relative ranking based label generation schemes, is performed. Evalua- tion results show various insights about the assessment methods which indicate the significance of the proposed database and the training protocol. Secondly, we have seen that in last few years, researchers have tried various learning based approaches to assess the face image quality. Most of these methods offer either a quality bin or a score summary as a measure of the biometric quality of the face image. But, to the best of our knowledge, so far there has not been any investigation on what are the explainable reasons behind the predicted scores. In this thesis, we propose a method to provide a clear and concise understanding of the predicted quality score of a learning based face image quality assessment. It is believed that this approach can be integrated into the FBI’s understandable template and can help in improving the image acquisition process by providing information on what quality factors need to be addressed.

In recent years, the use of Closed-Circuit Television (CCTV) for crime prevention and detection has attracted significant attention and focus, especially after the 11 September 2001 attack on New York’s World Trade Center and London bombings in July 2005. Existing face recognition systems require passport-quality photos to achieve satisfying results, but using CCTV images is much more problematic due to the large variations in lighting conditions, facial expression and pose angle. Recently, Chen and Lovell (2004) developed a new face recognition algorithm, known as Adaptive Principal Component Analysis (APCA), which performs well in the presence of variations in expression and lighting conditions. But like other PCA-derived face recognition algorithms, APCA only performs well with frontal face images. Following the approach of Cootes et al (2001), in this thesis we develop a face model and a rotation model which can be used to interpret facial features and synthesize realistic frontal face images when given a single novel face image. A Viola-Jones based face detector is used to detect faces in real-time and thus solves the initialization problem for our Active Appearance Model search. It can handle pose changes of ± 25 degrees from the frontal view. The work is then extended to accommodate large variations in head pose. A left face model and a right face model are used to represent larger face variation information; a correlation model is developed which can be used to transform the face images from the left or right rotation models to the frontal rotation model. Experiments show that our approach can achieve good recognition rates on face images across a wide range of head poses. Indeed recognition rates are improved by up to a factor of 5 compared to standard PCA. Based on the fact that an individual’s facial features and pose determine the appearance of a person at specific pose, a new face recognition method without synthesis of images is proposed. It uses statistical face models to interpret face information and a correlation model to remove the pose effect. The resultant pose-independent features are measured by Mahalanobis distance and cosine measure for classification. Finally a real time face recognition system, which integrates three major components, is proposed: 1) a Viola-Jones face detection module based on cascaded simple binary features to rapidly detect and locate multiple faces from the input still image or video sequences, 2) a normalization module based on the eye locations detected by a Viola- Jones eye detector. 3) Adaptive Principal Component Analysis to recognize the detected faces. This system demonstrates that it is indeed viable to use face recognition as a part of an intelligent CCTV system.

It is challenging to derive explainability for unsupervised or statistical-based face image quality assessment (FIQA) methods. In this work, we propose a novel set of explainability tools to derive reasoning for different FIQA decisions and their face recognition (FR) performance implications. We avoid limiting the deployment of our tools to certain FIQA methods by basing our analyses on the behavior of FR models when processing samples with different FIQA decisions. This leads to explainability tools that can be applied for any FIQA method with any CNN-based FR solution using activation mapping to exhibit the network’s activation derived from the face embedding. To avoid the low discrimination between the general spatial activation mapping of low and high-quality images in FR models, we build our explainability tools in a higher derivative space by analyzing the variation of the FR activation maps of image sets with different quality decisions. We demonstrate our tools and analyze the findings on four FIQA methods, by presenting inter and intra-FIQA method analyses. Our proposed tools and the analyses based on them point out, among other conclusions, that high-quality images typically cause consistent low activation on the areas outside of the central face region, while low-quality images, despite general low activation, have high variations of activation in such areas. Our explainability tools also extend to analyzing single images where we show that low-quality images tend to have an FR model spatial activation that strongly differs from what is expected from a high-quality image where this difference also tends to appear more in areas outside of the central face region and does correspond to issues like extreme poses and facial occlusions. The implementation of the proposed tools is accessible here <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/fbiying87/Explainable_FIQA_WITH_AMVA

With the development of the face recognition technology, the face recognition techniques are more and more applied in the scenario of the forensic science. Forensic identification of human images is a forensic activity for verifying whether the questioned and the known face images are the same ones. The one to one face verification technique can be well applied in the above application. Researching on the effect of face image quality on the performance of face verification systems in the application of the forensic identification of human images leads to the problem of face image quality assessment. Firstly, we discuss and analyze factors that affect the assessment of face image quality in forensic identification of human images. The factors consist of the age, expression, imaging angle, image quality and others, which will influence the performance of the face verification system. Then we propose a quantitative analysis method for the assessment of face image quality, which is relied on the verification performance of face verification systems. The effect of face images under specific conditions is studied. The face image quality under the specific factor condition is quantitatively scored according to the similarity quantification value between face images calculated by the face verification system. For the implement of the face verification system, the deep learning based face recognition method is used for objective evaluation of the face image quality. The results in the paper have shown the important significance of our proposed method for the objective evaluation of face image quality, and for the reasonable selection of face images in videos in the practical cases of the forensic identification of human images.

Quality of face images may be degraded as they are captured under varying capturing conditions such as illumination and speed of moving subject in videos. Performance of a typical face recognition systems is sensitive to the quality of input face images. Face image quality assessment is necessary for accurate face recognition systems both in the enrollment and recognition stages. Face image quality assessment is considered as a complex task as some of quality factors are in contrast to each other in different environmental conditions. In this study, a face image quality assessment based on photometric quality factors using classification techniques is proposed to justify applicability of used quality factors. The proposed method has three main phases namely, quality factor measurement, feature normalization, and classification. Evaluation of the proposed method on modified NLPR face dataset demonstrates all of the used classifiers have almost equal performance but, MLP classifier outperforms other classifiers in terms of f-score and accuracy measures slightly. Experimental results revealed that brightness, contrast, focus, and illumination are effective factors for purpose of still face image quality assessment. The proposed method also has better performance with comparison with some of the existing methods based on the mentioned dataset.

Quality of facial images significantly impacts the performance of face recognition algorithms. Being able to predict "which facial image is good for recognition" is of great importance for real application scenarios, where a sequence of facial images are always presented and one can select "the best quality" image frame for the subsequent matching and recognition task. To this end, we introduce a novel facial image quality automatic assessment framework directly targeting on "selecting better face image for better face recognition". For such as purpose, a deep convolutional neural network (DCNN) is trained to output a general facial quality metric which comprehensively considers various quality factors including brightness, contrast, blurriness, occlusion, pose etc. Based on this trained facial quality metric network, we are able to sort the input face images accordingly and "select" good face images for recognition. Our method is evaluated on the Color FERET and KinectFace face datasets. Results show that the proposed facial image quality metric network well distinguish "good" images from "bad" ones during face recognition.

Recognition oriented facial image quality assessment via deep convolutional neural network

When storing face biometric samples in accordance with ISO/IEC 19794 as JPEG2000 encoded images, it is necessary to encrypt them for the sake of users’ privacy. Literature suggests selective encryption of JPEG2000 images as fast and efficient method for encryption, the trade-off is that some information is left in plaintext. This could be used by an attacker, in case the encrypted biometric samples are leaked. In this work, we will attempt to utilize a convolutional neural network to perform cryptanalysis of the encryption scheme. That is, we want to assess if there is any information left in plaintext in the selectively encrypted face images which can be used to identify the person. The chosen approach is to train CNNs for biometric face recognition not only with plaintext face samples but additionally conduct a refinement training with partially encrypted data. If this system can successfully utilize encrypted face samples for biometric matching, we can show that the information left in encrypted biometric face samples is information actually usable for biometric recognition.The method works and we can show that a supposedly secure biometric sample still contains identifying information on average over the whole database.

Lossy face image compression can degrade the image quality and the utility for the purpose of face recognition. This work investigates the effect of lossy image compression on a state-of-the-art face recognition model, and on multiple face image quality assessment models. The analysis is conducted over a range of specific image target sizes. Four compression types are considered, namely JPEG, JPEG 2000, downscaled PNG, and notably the new JPEG XL format. Frontal color images from the ColorFERET database were used in a Region Of Interest (ROI) variant and a portrait variant. We primarily conclude that JPEG XL allows for superior mean and worst case face recognition performance especially at lower target sizes, below approximately 5kB for the ROI variant, while there appears to be no critical advantage among the compression types at higher target sizes. Quality assessments from modern models correlate well overall with the compression effect on face recognition performance.

Two-stage unsupervised facial image quality measurement

Face Recognition (FR) plays a crucial role in many critical (high-stakes) applications, where errors in the recognition process can lead to serious consequences. Face Image Quality Assessment (FIQA) techniques enhance FR systems by providing quality estimates of face samples, enabling the systems to discard samples that are unsuitable for reliable recognition or lead to low-confidence recognition decisions. Most state-of-the-art FIQA techniques rely on extensive supervised training to achieve accurate quality estimation. In contrast, unsupervised techniques eliminate the need for additional training but tend to be slower and typically exhibit lower performance. In this paper, we introduce FROQ<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> (Face Recognition Observer of Quality), a semi-supervised, training-free approach that leverages specific intermediate representations within a given FR model to estimate face-image quality, and combines the efficiency of supervised FIQA models with the training-free approach of unsupervised methods. A simple calibration step based on pseudo-quality labels allows FROQ to uncover specific representations, useful for quality assessment, in any modern FR model. To generate these pseudo-labels, we propose a novel unsupervised FIQA technique based on sample perturbations. Comprehensive experiments with four state-of-the-art FR models and eight benchmark datasets show that FROQ leads to highly competitive results compared to the state-of-the-art, achieving both strong performance and efficient runtime, without requiring explicit training. The code for FROQ is available from: https://github.com/LSIbabnikz/FROQ

In recent years, Face Image Quality Assessment (FIQA) plays an important role in the face recognition system. However, how to define face image quality is still an open question. In this work, we argue that a high-quality face image should have more identity-related information than a low-quality face image. Thus, we propose a novel unsupervised Face Image Quality Assessment with the variance of local contribution (VLC-FIQA). In our approach, we alternately mask partial pixels of the face image, then quantify the importance of these pixels and compute the variation of the importance of different parts as the quality of the image. Extensive experiments show that our VLC-FIQA outperforms state-of-the-art approaches on LFW. Our approach can be easily used for any recognition system and be extended to other recognition tasks such as person re-identification.

The automatic recognition of human faces presents a significant challenge to the pattern recognition research community. Typically, human faces are very similar in structure with minor differences from person to person. They are actually within one class of "human face". Furthermore, lighting conditions change, while facial expressions and pose variations further complicate the face recognition task as one of the difficult problems in pattern analysis. This paper proposes a novel concept: namely, that faces can be recognized using a line edge map (LEM). The LEM, a compact face feature, is generated for face coding and recognition. A thorough investigation of the proposed concept is conducted which covers all aspects of human face recognition, i.e. face recognition under (1) controlled/ideal conditions and size variations, (2) varying lighting conditions, (3) varying facial expressions, and (4) varying pose. The system performance is also compared with the eigenface method, one of the best face recognition techniques, and with reported experimental results of other methods. A face pre-filtering technique is proposed to speed up the search process. It is a very encouraging to find that the proposed face recognition technique has performed better than the eigenface method in most of the comparison experiments. This research demonstrates that the LEM, together with the proposed generic line-segment Hausdorff distance measure, provides a new method for face coding and recognition.

This article presents the issues related to applying computer vision techniques to identify facial expressions and recognize the mood of Traumatic Brain Injured (TBI) patients in real life scenarios. Many TBI patients face serious problems in communication and activities of daily living. These are due to restricted movement of muscles or paralysis with lesser facial expression along with non-cooperative behaviour, and inappropriate reasoning and reactions. All these aforementioned attributes contribute towards the complexity of the system for the automatic understanding of their emotional expressions. Existing systems for facial expression recognition are highly accurate when tested on healthy people in controlled conditions. However, their performance is not yet verified on the TBI patients in the real environment. In order to test this, we devised a special arrangement to collect data from these patients. Unlike the controlled environment, it was very challenging because these patients have large pose variations, poor attention and concentration with impulsive behaviours. In order to acquire high-quality facial images from videos for facial expression analysis, effective techniques of data preprocessing are applied. The extracted images are then fed to a deep learning architecture based on Convolution Neural Network (CNN) and Long Short-Term Memory (LSTM) network to exploit the spatiotemporal information with 3D face frontalization. RGB and thermal imaging modalities are used and the experimental results show that better quality of facial images and larger database enhance the system performance in facial expressions and mood recognition of TBI patients under natural challenging conditions. The proposed approach hopefully facilitates the physiotherapists, trainers and caregivers to deploy fast rehabilitation activities by knowing the positive mood of the patients.