Face Verification Task Research Articles

Sim2Word: Explaining Similarity with Representative Attribute Words via Counterfactual Explanations

Recently, we have witnessed substantial success using the deep neural network in many tasks. Although there still exist concerns about the explainability of decision making, it is beneficial for users to discern the defects in the deployed deep models. Existing explainable models either provide the image-level visualization of attention weights or generate textual descriptions as post hoc justifications. Different from existing models, in this article we propose a new interpretation method that explains the image similarity models by salience maps and attribute words. Our interpretation model contains visual salience maps generation and the counterfactual explanation generation. The former has two branches: global identity relevant region discovery and multi-attribute semantic region discovery. The first branch aims to capture the visual evidence supporting the similarity score, which is achieved by computing counterfactual feature maps. The second branch aims to discover semantic regions supporting different attributes, which helps to understand which attributes in an image might change the similarity score. Then, by fusing visual evidence from two branches, we can obtain the salience maps indicating important response evidence. The latter will generate the attribute words that best explain the similarity using the proposed erasing model. The effectiveness of our model is evaluated on the classical face verification task. Experiments conducted on two benchmarks—VGGFace2 and Celeb-A—demonstrate that our model can provide convincing interpretable explanations for the similarity. Moreover, our algorithm can be applied to evidential learning cases, such as finding the most characteristic attributes in a set of face images, and we verify its effectiveness on the VGGFace2 dataset.

LCAM: Low-Complexity Attention Module for Lightweight Face Recognition Networks

Inspired by the human visual system to concentrate on the important region of a scene, attention modules recalibrate the weights of either the channel features alone or along with spatial features to prioritize informative regions while suppressing unimportant information. However, the floating-point operations (FLOPs) and parameter counts are considerably high when one is incorporating these modules, especially for those with both channel and spatial attentions in a baseline model. Despite the success of attention modules in general ImageNet classification tasks, emphasis should be given to incorporating these modules in face recognition tasks. Hence, a novel attention mechanism with three parallel branches known as the Low-Complexity Attention Module (LCAM) is proposed. Note that there is only one convolution operation for each branch. Therefore, the LCAM is lightweight, yet it is still able to achieve a better performance. Experiments from face verification tasks indicate that LCAM achieves similar or even better results compared with those of previous modules that incorporate both channel and spatial attentions. Moreover, compared to the baseline model with no attention modules, LCAM achieves performance values of 0.84% on ConvFaceNeXt, 1.15% on MobileFaceNet, and 0.86% on ProxylessFaceNAS with respect to the average accuracy of seven image-based face recognition datasets.

GhostFaceNets: Lightweight Face Recognition Model From Cheap Operations

The development of deep learning-based biometric models that can be deployed on devices with constrained memory and computational resources has proven to be a significant challenge. Previous approaches to this problem have not prioritized the reduction of feature map redundancy, but the introduction of Ghost modules represents a major innovation in this area. Ghost modules use a series of inexpensive linear transformations to extract additional feature maps from a set of intrinsic features, allowing for a more comprehensive representation of the underlying information. GhostNetV1 and GhostNetV2, both of which are based on Ghost modules, serve as the foundation for a group of lightweight face recognition models called GhostFaceNets. GhostNetV2 expands upon the original GhostNetV1 by adding an attention mechanism to capture long-range dependencies. Evaluation of GhostFaceNets using various benchmarks reveals that these models offer superior performance while requiring a computational complexity of approximately 60-275 MFLOPs. This is significantly lower than that of State-Of-The-Art (SOTA) big convolutional neural network (CNN) models, which can require hundreds of millions of FLOPs. GhostFaceNets trained with the ArcFace loss on the refined MS-Celeb-1M dataset demonstrate SOTA performance on all benchmarks. In comparison to previous SOTA mobile CNNs, GhostFaceNets greatly improve efficiency for face verification tasks. The GhostFaceNets entire code will be made available after publication.

SO-softmax loss for discriminable embedding learning in CNNs

Convolutional neural networks (CNNs)-based classifiers, trained with the softmax cross-entropy loss, have achieved remarkable success in learning embeddings for pattern recognition. The cosine similarity-based softmax variants further improve the performance by focusing on optimizing the angles between embeddings and class weights. However, embeddings learned by these variants still have significant intra-class variances since these methods only optimize the relative differences between intra- and inter-class cosine similarities. To simultaneously optimize intra- and inter-class cosine similarities, this paper proposes a cosine Similarity Optimization-based softmax (SO-softmax) loss, which is based on a generalized softmax loss formulation that combines both similarities. The proposed loss constrains the intra-class (positive) and inter-class (negative) cosine similarity by quadratic transformations, thus making the embedding representation more compact within classes and more distinguishable between classes. It is verified theoretically that SO-softmax loss can optimize both the similarities simultaneously. Thorough experiments are conducted on typical audio classification, image classification, face verification, image retrieval, and person re-identification tasks, and the results show that SO-softmax loss outperforms the state-of-the-art loss functions in CNNs-based frameworks.

Multi-View Cosine Similarity Learning with Application to Face Verification

An instance can be easily depicted from different views in pattern recognition, and it is desirable to exploit the information of these views to complement each other. However, most of the metric learning or similarity learning methods are developed for single-view feature representation over the past two decades, which is not suitable for dealing with multi-view data directly. In this paper, we propose a multi-view cosine similarity learning (MVCSL) approach to efficiently utilize multi-view data and apply it for face verification. The proposed MVCSL method is able to leverage both the common information of multi-view data and the private information of each view, which jointly learns a cosine similarity for each view in the transformed subspace and integrates the cosine similarities of all the views in a unified framework. Specifically, MVCSL employs the constraints that the joint cosine similarity of positive pairs is greater than that of negative pairs. Experiments on fine-grained face verification and kinship verification tasks demonstrate the superiority of our MVCSL approach.

SELM: Siamese extreme learning machine with application to face biometrics.

Extreme learning machine (ELM) is a powerful classification method and is very competitive among existing classification methods. It is speedy at training. Nevertheless, it cannot perform face verification tasks properly because face verification tasks require the comparison of facial images of two individuals simultaneously and decide whether the two faces identify the same person. The ELM structure was not designed to feed two input data streams simultaneously. Thus, in 2-input scenarios, ELM methods are typically applied using concatenated inputs. However, this setup consumes two times more computational resources, and it is not optimized for recognition tasks where learning a separable distance metric is critical. For these reasons, we propose and develop a Siamese extreme learning machine (SELM). SELM was designed to be fed with two data streams in parallel simultaneously. It utilizes a dual-stream Siamese condition in the extra Siamese layer to transform the data before passing it to the hidden layer. Moreover, we propose a Gender-Ethnicity-dependent triplet feature exclusively trained on various specific demographic groups. This feature enables learning and extracting useful facial features of each group. Experiments were conducted to evaluate and compare the performances of SELM, ELM, and deep convolutional neural network (DCNN). The experimental results showed that the proposed feature could perform correct classification at 97.87% accuracy and 99.45% area under the curve (AUC). They also showed that using SELM in conjunction with the proposed feature provided 98.31% accuracy and 99.72% AUC. SELM outperformed the robust performances over the well-known DCNN and ELM methods.

Robust Character Labeling in Movie Videos: Data Resources and Self-Supervised Feature Adaptation

Robust face clustering is a vital step in enabling computational understanding of visual character portrayal in media. Face clustering for long-form content is challenging because of variations in appearance and lack of supporting large-scale labeled data. Our work in this paper focuses on two key aspects of this problem: the lack of domain-specific training or benchmark datasets, and adapting face embeddings learned on web images to long-form content, specifically movies. First, we present a dataset of over 169,000 face tracks curated from 240 Hollywood movies with weak labels on whether a pair of face tracks belong to the same or a different character. We propose an offline algorithm based on nearest-neighbor search in the embedding space to mine hard-examples from these tracks. We then investigate triplet-loss and multiview correlation-based methods for adapting face embeddings to hard-examples. Our experimental results highlight the usefulness of weakly labeled data for domain-specific feature adaptation. Overall, we find that multiview correlation-based adaptation yields more discriminative and robust face embeddings. Its performance on downstream face verification and clustering tasks is comparable to that of the state-of-the-art results in this domain. We also present the SAIL-Movie Character Benchmark corpus developed to augment existing benchmarks. It consists of racially diverse actors and provides face-quality labels for subsequent error analysis. We hope that the large-scale datasets developed in this work can further advance automatic character labeling in videos. All resources are available freely at https://sail.usc.edu/~ccmi/multiface.

XCos: An Explainable Cosine Metric for Face Verification Task

We study the XAI (explainable AI) on the face recognition task, particularly the face verification. Face verification has become a crucial task in recent days and it has been deployed to plenty of applications, such as access control, surveillance, and automatic personal log-on for mobile devices. With the increasing amount of data, deep convolutional neural networks can achieve very high accuracy for the face verification task. Beyond exceptional performances, deep face verification models need more interpretability so that we can trust the results they generate. In this article, we propose a novel similarity metric, called explainable cosine ( xCos ), that comes with a learnable module that can be plugged into most of the verification models to provide meaningful explanations. With the help of xCos , we can see which parts of the two input faces are similar, where the model pays its attention to, and how the local similarities are weighted to form the output xCos score. We demonstrate the effectiveness of our proposed method on LFW and various competitive benchmarks, not only resulting in providing novel and desirable model interpretability for face verification but also ensuring the accuracy as plugging into existing face recognition models.

The Episodic Prototypes Model (EPM): On the nature and genesis of facial representations.

Faces undergo massive changes over time and life events. We need a mental representation which is flexible enough to cope with the existing visual varieties, but which is also stable enough to be the basis for valid recognition. Two main theoretical frameworks exist to describe facial representations: prototype models assuming one central item comprising all visual experiences of a face, and exemplar models assuming single representations of each visual experience of a face. We introduce a much more ecological valid model dealing with episodic prototypes (the Episodic Prototypes Model—EPM), where faces are represented by a low number of prototypes that refer to specific Episodes of Life (EoL, e.g., early adulthood, mature age) during which the facial appearance shows only moderate variation. Such an episodic view of mental representation allows for efficient storage, as the number of needed prototypes is relatively low, and it allows for the needed variation within a prototype that keeps the everyday and steadily ongoing changes across a certain period of time. Studies 1–3 provide evidence that facial representations are highly dependent on temporal aspects which is in accord with EoL, and that individual learning history generates the structure and content of respective prototypes. In Study 4, we used implicit measures (RT) in a face verification task to investigate the postulated power of the EPM. We could demonstrate that episodic prototypes clearly outperformed visual depictions of exhaustive prototypes, supporting the general idea of our approach.

Development of a module for identifying a person by photography to improve the quality of the machine-engineering plant in remote mode

A machine vision performs a variety of face verification tasks in different infrastructure environments. A facial recognition system is technically implemented as an image recognition program. It serves to automatically identify a face in an image, and then identify the person by comparing and analyzing the biometric data of the human face. Cameras are able to capture an image at a distance, which is perfect for building monitoring systems and contactless biometrics. Such software systems are particularly important in the context of remote work of employees of industrial enterprises. The purpose of research is to develop and implement a facial recognition web service for the staff department of a machine-engineering enterprise. The web service should work in conjunction with the existing data bank. Tasks of research are: to consider the fundamental algorithms of facial recognition, to formulate the basic requirements for the development and implementation of the corresponding automated web service. The significant result of research is that the developed module uses library of computer vision algorithms, image processing and general-purpose numerical algorithms with open-source code. Identification system was checked by testing on the company’s website.

Audio-Visual Deep Neural Network for Robust Person Verification

Voice and face are two most popular biometrics for person verification, usually used in speaker verification and face verification tasks. It has already been observed that simply combining the information from these two modalities can lead to a more powerful and robust person verification system. In this article, to fully explore the multi-modal learning strategies for person verification, we proposed three types of audio-visual deep neural network (AVN), including feature level AVN (AVN-F), embedding level AVN (AVN-E), and embedding level combination with joint learning AVN (AVN-J). To further enhance the system robustness in real noisy conditions where not both modalities can be accessed with high-quality, we proposed several data augmentation strategies for each proposed AVN: A feature-level multi-modal data augmentation is proposed for AVN-F and an embedding-level data augmentation with novel noise distribution matching is designed for AVN-E. For AVN-J, both the feature and embedding level multi-modal data augmentation methods can be applied. All the proposed models are trained on the VoxCeleb2 dev dataset and evaluated on the standard VoxCeleb1 dataset, and the best system achieves 0.558, 0.441% and 0.793% EER on the three official trial lists of VoxCeleb1, which is to our knowledge the best published single system results on this corpus for person verification. To validate the robustness of the proposed approaches, a noisy evaluation set based on the VoxCeleb1 is constructed, and experimental results show that the proposed system can significantly boost the system robustness and still show promising performance under this noisy scenario.

A kernel-based weight decorrelation for regularizing CNNs

Recent years has witnessed the success of convolutional neural networks (CNNs) in many machine learning and pattern recognition applications, especially in image recognition. However, due to the increasing model complexity, the parameter redundancy problem arises, and greatly degrades the performance of CNNs. To alleviate this problem, various regularization techniques, such as Dropout, have been proposed and proved their effectiveness. In this paper, we propose a novel adaptive kernel-based weight decorrelation (AKWD) framework, in order to regularize CNNs for better generalization. Different from existing works, the correlation between paring weights is measured by the cosine distance defined in RKHS associated with a specific kernel. The case with the well-known Gaussian kernel is investigated in detail, where the bandwidth parameter is adaptively estimated. By regularizing CNN models of different capacities using AKWD, better performance is achieved on several benchmark databases for both object classification and face verification tasks. In particular, when Dropout or BatchNorm is present, even higher improvements are obtained using the proposed AKWD, that demonstrates a good compatibility of the proposed regularizer with other regularization techniques.

Discriminative metric learning for face verification using enhanced Siamese neural network

Although face verification algorithms have made great success under controlled conditions in recent years, there’s plenty of room at its performance under uncontrolled real-world due to lack of discriminative feature representation ability. From the perspective of metric learning, we proposed a context-aware based Siamese neural network (CASNN) to learn a simple yet powerful network for face verification task to enhance its discriminative feature representation ability. Firstly, a context-aware module is used to automatically focus on the key area of the input facial images without irrelevant background area. Then we design a Siamese network equipped with center-classification loss to compress intra-class features and enlarge between-class ones for discriminative metric learning. Finally, we propose a quantitative indicator named “D-score” to show the discriminative representation ability of the learnt features from different methods. The extensive experiments are conducted on LFW dataset, YouTube Face dataset (YTF) and real-world dataset. The results confirm that CASNN outperforms some state-of-the-art deep learning-based face verification methods.

Face recognition in unconstrained environment with CNN

In recent years, convolutional neural networks have proven to be a highly efficient approach for face recognition. In this paper, we develop such a framework to learn a robust face verification in an unconstrained environment using aggressive data augmentation. Our objective is to learn a deep face representation from large-scale data with massive noisy and occluded face. Besides, we add an adaptive fusion of softmax loss and center loss as supervision signals, which are helpful to improve the performance and to conduct the final classification. The experiment results show that the suggested system achieves comparable performances with other state-of-the-art methods on the Labeled Faces in the Wild and YouTube face verification tasks.

ID Preserving Face Super-Resolution Generative Adversarial Networks

We propose an ID Preserving Face Super-Resolution Generative Adversarial Networks (IP-FSRGAN) to reconstruct realistic super-resolution face images from low-resolution ones. Inspired by the success of generative adversarial networks (GAN), we introduce a novel ID preserving module to help the generator learn to infer the facial details and synthesize more realistic super-resolution faces. Our method produces satisfactory visual results and also quantitatively outperforms state-of-the-art super-resolution methods on the face datasets including CASIA-Webface, CelebA, and LFW datasets under the metrics of PSNR, SSIM, and cosine similarity. In addition, we propose a framework to apply IP-FSRGAN model to address the face verification task on low-resolution face images. The synthesized $4\times $ super-resolution faces achieve a verification accuracy of 97.6%, improved from 92.8% of low resolution faces. We also prove by experiments that the proposed IP-FSRGAN model demonstrates excellent robustness under different downsample scaling factors and extensibility to various face verification models.

Realization of a Hybrid Locally Connected Extreme Learning Machine With DeepID for Face Verification

Most existing state-of-the-art deep learning algorithms discover sophisticated representations in huge datasets using convolutional neural networks (CNNs) that mainly adopt backpropagation (BP) algorithm as the backbone for training the face recognition problems. However, since decades ago, BP has been debated for causing trivial issues such as iterative gradient-descent operation, slow convergence rate, local minima, intensive human intervention, exhaustive computation, time-consuming, and so on. On the other hand, a competitive machine learning algorithm called extreme learning machine (ELM) emerged with extreme fast implementation and simple in theory has overcome the challenges faced by BP. The ELM advocates the convergence of machine learning and biological learning for pervasive learning and intelligence and has been extensively researched in widespread applications. Nonetheless, till date, none of the work of ELM has proved its competency in tackling face verification problem. Hence, in this paper, we are going to probe for the first time the feasibility of ELM-based network in handling the face verification task. We devise and propose a novel and distinguished hybrid local receptive field-based extreme learning machine with DeepID (hereinafter denoted as H-ELM-LRF-DeepID), to discriminate face pairs. The experimental results on the YouTube face database, labeled faces in the wild (LFW), and CelebFaces datasets have shed light upon the feasibility and usefulness of the H-ELM-LRF-DeepID in the face verification task.

A Novel Approach to Face Verification Based on Second-Order Face-Pair Representation

Face representation and matching are two essential issues in face verification task. Various approaches have been proposed focusing on these two issues. However, few of them addressed the joint optimal solutions of these two issues in a unified framework. In this paper, we present a second-order face representation method for face pair and a unified face verification framework, in which the feature extractors and the subsequent binary classification model design can be selected flexibly. Our contributions can be summarized in the following aspects. First, a novel face-pair representation method that employs the second-order statistical property of the face pairs is proposed, which retains more information compared to the existing methods. Second, a flexible binary classification model, which differs from the conventionally used metric learning, is constructed based on the new face-pair representation. Finally, we verify that our proposed face-pair representation can benefit from large training datasets. All the experiments are carried out on Labeled Face in the Wild (LFW) to verify the algorithm’s effectiveness against challenging uncontrolled conditions.

Additive Margin Softmax for Face Verification

In this paper, we propose a conceptually simple and geometrically interpretable objective function, i.e. additive margin Softmax (AM-Softmax), for deep face verification. In general, the face verification task can be viewed as a metric learning problem, so learning large-margin face features whose intra-class variation is small and inter-class difference is large is of great importance in order to achieve good performance. Recently, Large-margin Softmax and Angular Softmax have been proposed to incorporate the angular margin in a multiplicative manner. In this work, we introduce a novel additive angular margin for the Softmax loss, which is intuitively appealing and more interpretable than the existing works. We also emphasize and discuss the importance of feature normalization in the paper. Most importantly, our experiments on LFW BLUFR and MegaFace show that our additive margin softmax loss consistently performs better than the current state-of-the-art methods using the same network architecture and training dataset. Our code has also been made available at https://github.com/happynear/AMSoftmax

Patch strategy for deep face recognition

Convolutional neural network (CNN) has proven to be a highly efficient approach to face recognition. In this study, the authors introduce a new layer to embed the patch strategy in convolutional architectures to improve the effectiveness of face representation. Meanwhile, a multi-branch CNN is constructed to learn features of each cropped patch by the patch strategy and then fuses all the patch features together to form the entire face representation. Compared with the traditional patch methods, their approach has the advantage that no extra space is needed to store the facial patches since the images are cropped online. Moreover, due to the end-to-end training, this approach makes a better use of the interactions between global and local features in the model. Two baseline CNNs (i.e. AlexNet and ResNet) are used to analyse the effectiveness of their method. Experiments show that the proposed system achieves comparable performance with other state-of-the-art methods on the labelled faces in the wild and YouTube face verification tasks. To ensure the reproducibility, the publicly available training set CASIA-WebFace is used.

Entropy and orthogonality based deep discriminative feature learning for object recognition

Inspired by the class-selectivity of the neurons in the inferior temporal (IT) area of the human visual cortex, we propose a novel discriminative feature learning method to improve the object recognition performance of convolutional neural network (CNN) without increasing the network complexity. Specifically, we apply the proposed entropy–orthogonality loss (EOL) to the penultimate layer of the CNN models in the training phase. The EOL explicitly enables the feature vectors learned by a CNN model have the following properties: (1) each dimension of the feature vectors only responds strongly to as few classes as possible, and (2) the feature vectors from different classes are as orthogonal as possible. When combined with the softmax loss, the EOL not only can enlarge the differences in the between-class feature vectors, but also can reduce the variations in the within-class feature vectors. Therefore, the discriminative ability of the learned feature vectors is highly improved. The EOL is general and independent of the CNN structure. Comprehensive experimental comparisons with both the image classification and face verification task on several benchmark datasets demonstrate that utilizing the proposed EOL during training can remarkably improve performance of CNN models compared to the corresponding baseline models trained without utilizing the EOL.