Facial Image Information Research Articles

Using 3D facial information to predict malnutrition and consequent complications.

Phase angle (PhA) correlates with body composition and could predict the nutrition status of patients and disease prognosis. We aimed to explore the feasibility of predicting PhA-diagnosed malnutrition using facial image information based on deep learning (DL). From August 2021 to April 2022, inpatients were enrolled from surgery, gastroenterology, and oncology departments in a tertiary hospital. Subjective global assessment was used as the gold standard of malnutrition diagnosis. The highest Youden index value was selected as the PhA cutoff point. We developed a multimodal DL framework to automatically analyze the three-dimensional (3D) facial data and accurately determine patients' PhA categories. The framework was trained and validated using a cross-validation approach and tested on an independent dataset. Four hundred eighty-two patients were included in the final dataset, including 176 with malnourishment. In male patients, the PhA value with the highest Youden index was 5.55°, and the area under the receiver operating characteristic curve (AUC) = 0.68; in female patients, the PhA value with the highest Youden index was 4.88°, and AUC = 0.69. Inpatients with low PhA had higher incidence of infectious complications during the hospital stay (P = 0.003). The DL model trained with 4096 points extracted from 3D facial data had the best performance. The algorithm showed fair performance in predicting PhA, with an AUC of 0.77 and an accuracy of 0.74. Predicting the PhA of inpatients from facial images is feasible and can be used for malnutrition assessment and prognostic prediction.

Worst-Case Morphs Using Wasserstein ALI and Improved MIPGAN

A morph is a combination of two separate facial images and contains the identity information of two different people. When used in an identity document, both people can be authenticated by a biometric face recognition (FR) system. Morphs can be generated using either a landmark-based approach or approaches based on deep learning, such as generative adversarial networks (GANs). In a recent paper, we introduced a worst-case upper bound on how challenging morphing attacks can be for an FR system. The closer morphs are to this upper bound, the bigger the challenge they pose to FR. We introduced an approach with which it was possible to generate morphs that approximate this upper bound for a known FR system (white box) but not for unknown (black box) FR systems. In this paper, we introduce a morph generation method that can approximate worst-case morphs even when the FR system is not known. A key contribution is that we include the goal of generating difficult morphs during training. Our method is based on adversarially learned inference (ALI) and uses concepts from Wasserstein GANs trained with gradient penalty, which were introduced to stabilise the training of GANs. We include these concepts to achieve a similar improvement in training stability and call the resulting method Wasserstein ALI (WALI). We finetune WALI using loss functions designed specifically to improve the ability to manipulate identity information in facial images and show how it can generate morphs that are more challenging for FR systems than landmark- or GAN-based morphs. We also show how our findings can be used to improve MIPGAN, an existing StyleGAN-based morph generator.

Discovering the hidden personality of lambs: Harnessing the power of Deep Convolutional Neural Networks (DCNNs) to predict temperament from facial images

The objective of this study was to define a more practical and reliable alternative to manual temperament classification methods that rely on the behavioral responses of animals individually subjected to various tests. Specifically, this study evaluated the correlation between facial image information and temperament based on deep convolutional neural networks (DCNNs) to predict the temperament of lambs based on their facial images. In the first phase, the lambs were categorized as to their temperament based on data acquired from a behavioral test to establish a ground truth for the temperament of the lambs. This enabled us to train (70%), validate (20%), and test (10%) deep-learning models in the second phase based on facial images and the corresponding temperament labels derived from the behavioral test. The performance of a custom deep convolutional neural network (C-DCNN) was compared to that of pre-trained VGG19 and Xception models for image classification. The Xception model achieved a training accuracy of 81%, which indicated that it learned well the underlying patterns in the data; however, lower validation (0.75) and test (0.58) accuracies indicate that it overfit the training data and did not generalize well to new samples. The VGG19 model, produced lower training (0.59), validation (0.46), and test (0.34) accuracies, which indicated that it did not learn the underlying patterns in the data as well as the Xception model. Furthermore, its precision (0.47), recall (0.42), and F1 score (0.41) indicated that the model performed poorly in identifying the classes correctly. The C-DCNN produced a moderate accuracy of 60%, which indicated that the model was able to predict the temperament traits of lambs with an accuracy of 60%, which was better than random guessing (33% accuracy), and demonstrated the potential of this approach in assessing temperament. The C-DCNN precision (0.69), recall (0.61) and F1 score (0.63) indicated that it had a moderate ability to correctly identify positive cases; however, the small size of the original dataset remains a limitation of the study because it might have caused the suboptimal performance of the models. To validate this approach, further research is needed based on a larger and more diverse dataset. We will continue to investigate the potential of deep learning and computer vision to predict animal personality traits from facial images based on large, diverse datasets, which might lead to more efficient and objective methods for assessing animal temperament and improving animal welfare.

A Collaborative Learning-based Urban Low-light Small-target Face Image Enhancement Method

Face recognition is an essential technology in intelligent transportation and security within smart cities. Nevertheless, face images taken in nighttime urban environments often suffer from low brightness, small sizes, and low resolution, which pose significant challenges for accurate face feature recognition. To address this issue, we propose the Low-light Small-target Face Enhancement (LSFE) method, a collaborative learning-based image brightness enhancement approach specifically designed for small-target faces in low-light environments. LSFE employs a multilevel feature stratification module to acquire detailed face image features at different levels, revealing hidden facial image information within the dark. In addition, we design a network combining collaborative learning and self-attention mechanisms, which effectively captures long-distance pixel dependencies in low-brightness face images and enhances their brightness in a stepwise manner. The enhanced feature maps are then fused through a branch fusion module. Experimental results demonstrate that LSFE can more effectively enhance the luminance of small-target face images in low-light scenes while retaining more visual information, compared to other existing methods.

A Method for Eliminating Pig Face Recognition Errors Caused by Too Short Pig Growth Cycle

In the process of modern large-scale pig breeding, it is necessary to distinguish the identity of each pig and real-time detect its health status, weight change, dietary status, and other parameters. Traditional methods waste a lot of resources, while the high quality of pork cannot be effectively guaranteed. This project is based on convolutional neural networks to design and develop a pig face recognition system. This system uses an overhead camera suspended above the pig house to monitor the pig house for 24 hours and identify and track each pig. Due to the rapid growth cycle of the pig, the facial image information changes rapidly, which has a significant impact on the pig face recognition model. The acquisition camera module is designed to correct monitoring and tracking data. The acquisition camera is installed in the necessary place of the pig every day to collect real-time information and upload the collected information to the server. By comparing the data of individual pigs at different developmental stages with the server, the identification information of individual pigs is determined, and tracking data is corrected in a timely manner. At the same time, the monitoring and identification screen is displayed on the screen, and behavioral information parameters are recorded to facilitate the information management and breeding of the farm.

Quaternion Capsule Neural Network With Region Attention for Facial Expression Recognition in Color Images

Facial expression recognition (FER) is an essential subject of computer vision and human-computer interaction. It has been reported that many factors are closely related to the FER performance such as the pose, facial muscle variations, and the ignored color information in facial images. In this study, we propose a quaternion capsule neural network (Q-CapsNet) with region attention mechanism for FER in color images. The proposed Q-CapsNet is an end-to-end deep learning framework, which adopts the concept of quaternion theory to the Capsule Neural Network and further uses an attention mechanism to focus on the facial region of interests (ROIs). The Q-CapsNet addresses the FER problems in the following aspects. First, the internal dependencies between color channels are captured by the quaternion technique, which is of great importance in color image processing. Second, the Q-CapsNet could disentangle latent geometry information from facial images by quaternion capsules and quaternion routing algorithm. Third, inspired by the fact that facial expressions are mainly determined by several vital facial regions, the region attention mechanism is introduced to extract emotional features from the ROIs of the face. The proposed Q-CapsNet is evaluated on four public color FER datasets including MMI, Oulu-CASIA, RAF-DB, and SFEW. The comprehensive experiments and visualization results demonstrate that the proposed network outperforms comparative models and the state-of-the-art FER methods.

Multi-feature sparse similar representation for person identification

Person identification with a single feature (e.g., face recognition, speaker verification, person re-identification, etc.) has been studied extensively for many years, while few works focus on multi-feature person identification. Though promising performance has been achieved by only using the information of facial images, voice, or pedestrian appearance, it is still challenging to recognize a person with only a single feature in some situations (e.g., a person at a distance or occluded by other objects, and a partial person out of view). In this paper, we present a multi-feature sparse similar representation (MFSSR) method to effectively fuse face features, body features, and global image features for the task of person identification. In MFSSR, we designed a reconstructed deep spatial feature for representing the appearance of human body by using the spatial correlation coding of partial deep spatial features. Then we presented a multi-feature sparse similar representation model for jointly using different features, e.g., face, body, and the global image. Besides, considering that the coding coefficients associated with good samples but not outliers should be more similar among different features, we jointly represent different features by imposing a weighted ℓ1-norm distance regularization, instead of the conventional ℓ2-norm regularization, on the coefficients. Experimental results on several multi-feature person identification databases have clearly shown the superior performance of the proposed model.

Facial Makeup Detection Using Multi-Scale Local Binary Patterns and Convolutional Neural Network Fusion

This study presents a novel facial makeup detection scheme using fusion of multi-scale Local Binary Patterns (ms-LBP) texture methods and convolutional neural network (CNN) deep learning extractor. Facial makeups affect the accuracy of face recognition systems due to appearance alteration of individuals. In order to fuse the facial features, the proposed scheme first considers concatenation of extracted global histogram of a whole image using three different LBP operators. The LBP scales are used to extract and then concatenate the local histogram of overlapping and nonoverlapping blocks of images. This way utilizes the advantages of microtexton information of local primitives besides the extracted global textures. Finally, the extracted features using CNN feature extractor are combined with global and local multi-scale textures. In general, CNN deep learning extractor learns high-level discriminative characteristics of images and therefore the proposed system improves the facial makeup detection rate by involving both microtexton and discriminative information of facial images. In addition, the proposed method attempts to select the optimized subset of facial features to increase the detection performance of system by applying Particle Swarm Optimization (PSO) technique after feature level fusion. The paper uses Support Vector Machine (SVM) classifier for classifying the facial vectors into makeup or no-makeup classes. The proposed scheme is then evaluated using YMU, VMU and MIW facial makeup databases with consideration of light, medium and heavy makeups on several datasets. Experimental results analysis clarifies the effectiveness of proposed facial makeup detection framework of this study.

Multi-scale deep relational reasoning for facial kinship verification

In this paper, we propose a deep relational network which exploits multi-scale information of facial images for kinship verification. Unlike most existing deep learning based facial kinship verification methods which employ convolutional neural networks to extract holistic features, we present a deep model to exploit facial kinship relationship from local regions. For each given pair of face images, our method uses two convolutional neural networks which share parameters to extract different scales of features, which are expected to provide global contextual information of face images. We split a set of features at the same scale into multiple groups, where different groups capture information of different local regions. For each pair of local feature groups which are extracted from the same scale and position, we propose a relation network to reason their relationship, and use a verification network to infer the kin relation based on the results of local relations from different facial regions. We conduct experiments on two widely used facial kinship datasets: KinFaceW-I and KinFaceW-II, and our experimental results are presented to demonstrate the effectiveness of our approach.

Multiple Network Fusion with Low-Rank Representation for Image-Based Age Estimation

Image-based age estimation is a challenging task since there are ambiguities between the apparent age of face images and the actual ages of people. Therefore, data-driven methods are popular. To improve data utilization and estimation performance, we propose an image-based age estimation method. Theoretically speaking, the key idea of the proposed method is to integrate multi-modal features of face images. In order to achieve it, we propose a multi-modal learning framework, which is called Multiple Network Fusion with Low-Rank Representation (MNF-LRR). In this process, different deep neural network (DNN) structures, such as autoencoders, Convolutional Neural Networks (CNNs), Recursive Neural Networks (RNNs), and so on, can be used to extract semantic information of facial images. The outputs of these neural networks are then represented in a low-rank feature space. In this way, feature fusion is obtained in this space, and robust multi-modal image features can be computed. An experimental evaluation is conducted on two challenging face datasets for image-based age estimation extracted from the Internet Move Database (IMDB) and Wikipedia (WIKI). The results show the effectiveness of the proposed MNF-LRR.

Investigating the Effects of Facial Regions to Age Estimation

Aging process causes evident alterations on human facial appearance. Real world age progression on human face is personalized and related with many factors such as, genetics, living style, eating habits, facial expressions, climate etc. The wide degree of variations on facial appearance of different individuals affects the age estimation performance. In accordance with these facts discovering the aging information contained in facial regions is an important issue in automatic age estimation. Thus the facial regions emphasizing the aging information can be used for more accurate age estimation. In this context, age estimation performances of facial regions (eye, nose, mouth and chin, cheeks and sides of mouth) are investigated in this paper. For this purpose, an age estimation method is designed to produce an estimate of the age of a subject by using the texture features extracted from facial regions. In this method the facial images are warped into the mean shape thus variations of head pose and scale are eliminated and the texture information of facial images are aligned. Then the holistic and spatial texture features are extracted from facial regions using Local Phase Quantization (LPQ) texture descriptor, robust to blur, illumination and expression variations. After the low dimensional representation of these features, a linear aging function is learned using multiple linear regression. In the experiments FGNET and PAL databases are used to evaluate the age estimation accuracies of facial regions i.e. eye, nose, mouth and chin, cheek and sides of mouth, separately. The results have shown that the eye region carries the most significant information for age estimation. Also the mouth and chin, cheek regions are effective in the prediction of age. The results also have shown that, using the spatial texture features enhances the discriminative power of the texture descriptor and thus increases the estimation accuracy.

Hallucinating faces based on adaptive neighborhood selection and dual tree complex wavelet transform

In this paper, a novel face hallucination method is proposed. Instead of utilizing the gray values of pixels directly, the feature vector extracted using dual tree complex wavelet transform (DTCWT) is exploited to describe the directional detailed information of facial image. Then, the adaptive neighborhood selection (ANS) algorithm is used to preserve neighborhood relationship between the low-resolution and high-resolution face by selecting the optimal neighbors of each facial patch from the feature space. The selected neighborhood can well reflect the local geometrical structure of face manifold, so that the linear subspace determined by the optimal linear fitting can approximate the local facial geometry with a higher accuracy. In order to reduce the effect of misalignment, a face alignment method based on three facial keypoints is also used. Experimental results show that the proposed face hallucination method outperforms other state-of-the-art methods in terms of visual inspection and objective evaluations, especially when the facial images are compressed and have low-resolution.

Sequential three-way decision and granulation for cost-sensitive face recognition

Many previous studies on face recognition attempted to seek a precise classifier to achieve a low misclassification error, which is based on an assumption that all misclassification costs are the same. In many real-world scenarios, however, this assumption is not reasonable due to the imbalanced misclassification cost and insufficient high-quality facial image information. To address this issue, we propose a sequential three-way decision method for cost-sensitive face recognition. The proposed method is based on a formal description of granular computing. It develops a sequential strategy in a decision process. In each decision step, it seeks a decision which minimizes the misclassification cost rather than misclassification error, and it incorporates the boundary decision into the decision set such that a delayed decision can be made if available high-quality facial image information is insufficient for a precise decision. To describe the granular information of the facial image in three-way decision steps, we develop a series of image granulation methods based on two-dimensional subspace projection methods including 2DPCA, 2DLDA and 2DLPP. The sequential three-way decisions and granulation methods present an applicable simulation on human decisions in face recognition, which simulate a sequential decision strategy from rough granule to precise granule. The experiments were conducted on two popular facial image database, which validated the effectiveness of the proposed methods.

Face recognition via optimized features fusion

Face recognition has received considerable attention in the field of computer vision and pattern recognition. The important applications of face recognition include but are not limited to Airport security, card security in ATM's, visa processing and the passport verification. Although there has been rigorous research in this area for almost a decade, the scientists have not been able to provide and agree to a standard for obtaining the salient information in facial images utilizing feature's categories. In this article, we have presented an approach where features are collected containing local and global face information i.e. geometric and appearance-based features. These features are fused, resulting in an increase in the face-recognition accuracy. First, the global features are obtained by utilizing Discrete Cosine Transform and Local facial features via Local Binary Pattern. In the next stage, both local and global features are combined using the concatenation method resulting in an increase in features. To reduce the data dimensions, Particle Swarm Optimization PSO along with Genetic Algorithm GA is applied to eliminate the redundant features that provide the optimized feature sets. We also provide empirical results of our proposed system. The system has been evaluated using ORL and Labeled Faces in the Wild LFW face databases. We have been able to obtain a promising 98% accuracy rate by using PSO-GA based optimized features albeit the reduced number of features. Features' fusion enables proposed system to be robust to variations like facial expression change, illumination effects and occlusions.

Age Group Classification of Facial Images Using Rank Based Edge Texture Unit (RETU)

Human beings can easily categorize a person's age group from a facial image where as this ability has not been promising in the computer vision community. To address this problem very important area of research, the present paper proposes a novel scheme of age classification system using features derived from co-occurrence parameters using Rank based Edge Texture Unit (RETU). The Co-occurrence Matrix (CM) on RETU characterizes the relationship between the neighbouring edge values, while preserving local information. The novelty of the proposed RETU is it classifies the age of human into seven categories i.e. in the age groups of 1-10, 11-20, 21-30, 31-40, 41-50, 51-60, and greater than 60. The TU of the proposed RETU ranges from 0 to 17 and thus reduces overall complexity in evaluating features from CM. The co occurrence features extracted from the RETU provide complete facial image information for age classification purpose. The RETU reduces each 3×3 sub image into 2×2 sub image while preserving the texture features and thus reduces the overall dimensionality of the image.

Cost-sensitive ordinal regression for fully automatic facial beauty assessment

In this paper, we propose a new cost-sensitive ordinal regression (CSOR) approach for fully automatic facial beauty assessment. While there have been several facial beauty assessment methods in the literature, most of them require an accurate set of manual landmarks and are not fully automatic. In many real-world applications, face images are usually captured in unconstrained environments and hence it is desirable to develop fully automatic facial beauty assessment systems for these practical applications. To this end, we develop a fully automatic facial beauty assessment system that behaves like human beings in assessing the concept of facial beauty. To achieve this goal, we propose a new CSOR method to predict the beauty information of face images captured in unconstrained environments. Our method is motivated by the fact that the relative order information of the beauty label information in a face dataset and the costs (losses) of different mis-assessments are usually different. To make better use of these information, we learn a regression model by uncovering the nonlinear structure and preserving the ordinal information of facial images simultaneously. Experimental results on our dataset are presented to demonstrate the efficacy of the proposed approach.

Link Mining: A Computer Vision and Pattern Mining Approach

This work addresses the important problem of discovery and analysis of social networks and link between frequent people observed from surveillance video footage where large amount of video data is collected routinely. A computer vision approach has been applied for detecting and extracting people within a group using HAAR classifier. This technique allows us to recognizing people by doing similarity matching between training faces and unknown detected face image. Therefore it is required to obtain high resolution face images of people in order to extract intrinsic feature information of facial images used in detection of person’s faces. We present a novel frequent pattern mining based approach in the domain of frequent person detection i.e. apriori to solve frequent association problem between social networks obtained from low level task of face recognition. Our approach is illustrated with promising results from a fully integrated camera system. General Terms Data Mining, Frequent Pattern Mining, Social Network Analysis, Dynamic Social Network Analysis

Geometric Distribution Weight Information Modeled Using Radial Basis Function with Fractional Order for Linear Discriminant Analysis Method

Fisher linear discriminant analysis (FLDA) is a classic linear feature extraction and dimensionality reduction approach for face recognition. It is known that geometric distribution weight information of image data plays an important role in machine learning approaches. However, FLDA does not employ the geometric distribution weight information of facial images in the training stage. Hence, its recognition accuracy will be affected. In order to enhance the classification power of FLDA method, this paper utilizes radial basis function (RBF) with fractional order to model the geometric distribution weight information of the training samples and proposes a novel geometric distribution weight information based Fisher discriminant criterion. Subsequently, a geometric distribution weight information based LDA (GLDA) algorithm is developed and successfully applied to face recognition. Two publicly available face databases, namely, ORL and FERET databases, are selected for evaluation. Compared with some LDA-based algorithms, experimental results exhibit that our GLDA approach gives superior performance.

A mutual information based face clustering algorithm for movie content analysis

This paper investigates facial image clustering, primarily for movie video content analysis with respect to actor appearance. Our aim is to use novel formulation of the mutual information as a facial image similarity criterion and, by using spectral graph analysis, to cluster a similarity matrix containing the mutual information of facial images. To this end, we use the HSV color space of a facial image (more precisely, only the hue and saturation channels) in order to calculate the mutual information similarity matrix of a set of facial images. We make full use of the similarity matrix symmetries, so as to lower the computational complexity of the new mutual information calculation. We assign each row of this matrix as feature vector describing a facial image for producing a global similarity criterion for face clustering. In order to test our proposed method, we conducted two sets of experiments that have produced clustering accuracy of more than 80%. We also compared our algorithm with other clustering approaches, such as the k-means and fuzzy c-means (FCM) algorithms. Finally, in order to provide a baseline comparison for our approach, we compared the proposed global similarity measure with another one recently reported in the literature.

A novel face recognition method based on sub-pattern and tensor

This paper aims to address one of the many problems existing in current facial recognition techniques using tensor (TensorFace Algorithm and its extensions). Current methods rasterize facial images as vectors, which result in a loss of spatial structure information of facial images. In this paper, we propose a method called Sp-Tensor to extend TensorFace by applying the sub-pattern technique. Advantages of the proposed method include: (1) a portion of spatial structure and local information of facial images is preserved; (2) dramatically reduce the computation complexity than other existing methods when building the model. The experimental results demonstrate that Sp-Tensor has better performance than the original TensorFace and Sp-PCA1, especially for facial images with un-modeled views and light conditions.