Face Bounding Box Research Articles

Advanced techniques for automated emotion recognition in dogs from video data through deep learning

Inter-species emotional relationships, particularly the symbiotic interaction between humans and dogs, are complex and intriguing. Humans and dogs share fundamental mammalian neural mechanisms including mirror neurons, crucial to empathy and social behavior. Mirror neurons are activated during the execution and observation of actions, indicating inherent connections in social dynamics across species despite variations in emotional expression. This study explores the feasibility of using deep-learning Artificial Intelligence systems to accurately recognize canine emotions in general environments, to assist individuals without specialized knowledge or skills in discerning dog behavior, particularly related to aggression or friendliness. Starting with identifying key challenges in classifying pleasant and unpleasant emotions in dogs, we tested advanced deep-learning techniques and aggregated results to distinguish potentially dangerous human--dog interactions. Knowledge transfer is used to fine-tune different networks, and results are compared on original and transformed sets of frames from the Dog Clips dataset to investigate whether DogFACS action codes detailing relevant dog movements can aid the emotion recognition task. Elaborating on challenges and biases, we emphasize the need for bias mitigation to optimize performance, including different image preprocessing strategies for noise mitigation in dog recognition (i.e., face bounding boxes, segmentation of the face or body, isolating the dog on a white background, blurring the original background). Systematic experimental results demonstrate the system’s capability to accurately detect emotions and effectively identify dangerous situations or signs of discomfort in the presence of humans.

Quantitative analysis of facial soft tissue using weighted cascade regression model applicable for facial plastic surgery

Localization of facial landmarks plays an important role in the measurement of facial metrics applicable for beauty analysis and facial plastic surgery. The first step in detecting facial landmarks is to estimate the face bounding box. Clinical images of patients' faces usually show intensity non-uniformity. These conditions cause common face detection algorithms do not perform well in face detection under varying illumination. To solve this problem, a modified fuzzy c-means (MFCM) algorithm is used under varying illumination modeling. The cascade regression method (CRM) has an appropriate performance in face alignment. This algorithm has two main drawbacks. (1) In the training phase, increasing the real data without considering normal data can lead to over-fitting. To solve this problem, a weighted CRM (WCRM) is presented. (2) In the test phase, using a mean shape causes the initial shape to be either near to or far from the face shape. To overcome this problem, a Procrustes-based analysis is presented. One of the most important steps in facial landmark localization is feature extraction. In this study, to increase detection accuracy of the cephalometric landmarks, local phase quantization (LPQ) is used for feature extraction in all three channels of RGB color space. Finally, the proposed algorithm is used to measure facial anthropometric metrics. Experimental results show that the proposed algorithm has a better performance in facial landmark localization than other compared algorithms.

Robust Human Face Emotion Classification Using Triplet-Loss-Based Deep CNN Features and SVM

Human facial emotion detection is one of the challenging tasks in computer vision. Owing to high inter-class variance, it is hard for machine learning models to predict facial emotions accurately. Moreover, a person with several facial emotions increases the diversity and complexity of classification problems. In this paper, we have proposed a novel and intelligent approach for the classification of human facial emotions. The proposed approach comprises customized ResNet18 by employing transfer learning with the integration of triplet loss function (TLF), followed by SVM classification model. Using deep features from a customized ResNet18 trained with triplet loss, the proposed pipeline consists of a face detector used to locate and refine the face bounding box and a classifier to identify the facial expression class of discovered faces. RetinaFace is used to extract the identified face areas from the source image, and a ResNet18 model is trained on cropped face images with triplet loss to retrieve those features. An SVM classifier is used to categorize the facial expression based on the acquired deep characteristics. In this paper, we have proposed a method that can achieve better performance than state-of-the-art (SoTA) methods on JAFFE and MMI datasets. The technique is based on the triplet loss function to generate deep input image features. The proposed method performed well on the JAFFE and MMI datasets with an accuracy of 98.44% and 99.02%, respectively, on seven emotions; meanwhile, the performance of the method needs to be fine-tuned for the FER2013 and AFFECTNET datasets.

TFW: Annotated Thermal Faces in the Wild Dataset

Face detection and subsequent localization of facial landmarks are the primary steps in many face applications. Numerous algorithms and benchmark datasets have been introduced to develop robust models for the visible domain. However, varying conditions of illumination still pose challenging problems. In this regard, thermal cameras are employed to address this problem, because they operate on longer wavelengths. However, thermal face and facial landmark detection in the wild is an open research problem because most of the existing thermal datasets were collected in controlled environments. In addition, many of them were not annotated with face bounding boxes and facial landmarks. In this work, we present a thermal face dataset with manually labeled bounding boxes and facial landmarks to address these problems. The dataset contains 9,982 images of 147 subjects collected under controlled and uncontrolled conditions. As a baseline, we trained the YOLOv5 object detection model and its adaptation for face detection, YOLO5Face, on our dataset. In addition to our test set, we evaluated the models on the external RWTH-Aachen thermal face dataset to show the efficacy of our dataset. We have made the dataset, source code, and pre-trained models publicly available at https://github.com/IS2AI/TFW to bolster research in thermal face analysis.

Research and Application of Face Mask Detection Algorithm Based on YOLOV4-Tiny

This paper mainly studies the target detection of face masks. Based on yOLOV4-TINY algorithm, multi-scale feature fusion of FPN structure and feature enhancement network are used to optimize the algorithm structure. First of all, this paper uses the xml.etree.ElementTree (ET) module in python to read the xml file to obtain the key information such as < object > face parameters, < name > face mask wearing status, < bndbox > face bounding box diagonal two vertex coordinates and so on. According to the vertex coordinates (bndbox) in the xml file, the target box is marked on the original image, and the color state of the target box is distinguished according to the wearing state of < name > face mask. Finally, two pictures of "250.png" and "477.png" are selected to be displayed in the body of this text. Then, a face mask detection algorithm based on YOLOv4-Tiny is established. when training the model, the multi-task joint loss of mask detection task is optimized by combining CIOU loss function and label smoothing strategy, and the Mosaic data enhancement method and learning rate cosine annealing attenuation strategy are used to improve the convergence speed and robustness of the model. After many times of detection and verification, the experimental results show that the detection accuracy of the proposed algorithm model is 94.84% (with_mask), 96.04% (without_mask) and 94.86% (mask_weared_incorrect) respectively in the three categories of human face mask detection: with_mask, without_mask and mask_weared_incorrect. Its quantitative indicator < number of faces correctly classified > / < number of all faces included in the tag file >, mAP is as high as 95.25%, and the accuracy is greatly improved.

Multistage Model for Robust Face Alignment Using Deep Neural Networks

The ability to generalize unconstrained conditions such as severe occlusions and large pose variations remains a challenging goal to achieve in face alignment. In this paper, a multistage model based on deep neural networks is proposed which takes advantage of spatial transformer networks, hourglass networks and exemplar-based shape constraints. First, a spatial transformer-generative adversarial network which consists of convolutional layers and residual units is utilized to solve the initialization issues caused by face detectors, such as rotation and scale variations, to obtain improved face bounding boxes for face alignment. Then, stacked hourglass network is employed to obtain preliminary locations of landmarks as well as their corresponding scores. In addition, an exemplar-based shape dictionary is designed to determine landmarks with low scores based on those with high scores. By incorporating face shape constraints, misaligned landmarks caused by occlusions or cluttered backgrounds can be considerably improved. Extensive experiments based on challenging benchmark datasets are performed to demonstrate the superior performance of the proposed method over other state-of-the-art methods.

Benchmarking Low-Light Image Enhancement and Beyond

In this paper, we present a systematic review and evaluation of existing single-image low-light enhancement algorithms. Besides the commonly used low-level vision oriented evaluations, we additionally consider measuring machine vision performance in the low-light condition via face detection task to explore the potential of joint optimization of high-level and low-level vision enhancement. To this end, we first propose a large-scale low-light image dataset serving both low/high-level vision with diversified scenes and contents as well as complex degradation in real scenarios, called Vision Enhancement in the LOw-Light condition (VE-LOL). Beyond paired low/normal-light images without annotations, we additionally include the analysis resource related to human, i.e. face images in the low-light condition with annotated face bounding boxes. Then, efforts are made on benchmarking from the perspective of both human and machine visions. A rich variety of criteria is used for the low-level vision evaluation, including full-reference, no-reference, and semantic similarity metrics. We also measure the effects of the low-light enhancement on face detection in the low-light condition. State-of-the-art face detection methods are used in the evaluation. Furthermore, with the rich material of VE-LOL, we explore the novel problem of joint low-light enhancement and face detection. We develop an enhanced face detector to apply low-light enhancement and face detection jointly. The features extracted by the enhancement module are fed to the successive layer with the same resolution of the detection module. Thus, these features are intertwined together to unitedly learn useful information across two phases, i.e. enhancement and detection. Experiments on VE-LOL provide a comparison of state-of-the-art low-light enhancement algorithms, point out their limitations, and suggest promising future directions. Our dataset has supported the Track “Face Detection in Low Light Conditions” of CVPR UG2+ Challenge (2019–2020) ( http://cvpr2020.ug2challenge.org/ ).

Face anti-spoofing algorithm combined with CNN and brightness equalization

Face anti-spoofing is a relatively important part of the face recognition system, which has great significance for financial payment and access control systems. Aiming at the problems of unstable face alignment, complex lighting, and complex structure of face anti-spoofing detection network, a novel method is presented using a combination of convolutional neural network and brightness equalization. Firstly, multi-task convolutional neural network (MTCNN) based on the cascade of three convolutional neural networks (CNNs), P-net, R-net, and O-net are used to achieve accurate positioning of the face, and the detected face bounding box is cropped by a specified multiple, then brightness equalization is adopted to perform brightness compensation on different brightness areas of the face image. Finally, data features are extracted and classification is given by utilizing a 12-layer convolution neural network. Experiments of the proposed algorithm were carried out on CASIA-FASD. The results show that the classification accuracy is relatively high, and the half total error rate (HTER) reaches 1.02%.

SAT-087 Identifying Distinct Facial Dysmorphology in Youth with Congenital Adrenal Hyperplasia Using Deep Learning Techniques

Purpose: Classical Congenital Adrenal Hyperplasia (CAH) due to 21-hydroxylase deficiency affects 1:15,000 newborns and involves adrenal insufficiency and androgen excess. These hormone abnormalities are evident as early as 7 weeks’ gestation and persist throughout pregnancy. Structural brain abnormalities are also known to occur in CAH, with abnormalities of brain and facial structure occurring together in conditions such as fetal alcohol spectrum disorder and holoprosencephaly. As well, sex differences in facial morphology are well described in healthy individuals. Thus, we aimed to study facial features using artificial intelligence in CAH youth. Methods: We studied frontal images of the face in 57 youth with severe salt-wasting CAH (60% female; 9.4±5.5y), and 38 controls (47% female, 9.7±5.1y), acquired with an iPad v12.1. We included 32 additional controls (43% female; 4-19y) from a publicly available face image dataset (1). Applying deep learning techniques, we converted 2-D facial photos to mathematical descriptors in order to differentiate features between groups. For a given test image, our pipeline output was a predicted “CAH score” between [0,1]. Due to our small dataset, we employed K-fold cross validation to train and test our deep neural network. At each of the K-9 folds, 88% of data (468 control and 531 CAH images) were used to train the network, with the remaining data (55 control and 63 CAH images) used to test the trained network. Test results were validated in terms of area under the curve (AUC) of receiver operating characteristic curves (generated from predicted CAH scores of test subjects), to analyze true and false positive rates. Our pipeline automatically detected face-bounding boxes and 68 facial landmarks (dlib toolkit) which were then used to compute 27 Euclidean (linear) facial features (2,3). We performed between group analyses of features with t-tests. Results: The averaged AUC of nine folds was 0.83±0.14, representing strong predictive power as a proxy to correlating facial dysmorphology with CAH. Predicted CAH scores were different between control (0.24±0.33) and CAH (0.69±0.37; p<0.0001) youth. Thirteen of 27 facial features were different between controls and CAH (p<0.05 for all) including 3 of 6 features related to sexual dimorphism. We also produced heat (i.e., saliency) maps showing the effect of CAH on facial features, and 2D t-SNE plot visualization of features showing well-defined separation between CAH and control group clusters. Conclusions: Utilizing deep learning, we have shown that CAH youth have facial features that can reliably distinguish them from controls. Further study is merited in regard to the etiology of affected facial morphology in CAH, and associations with disease severity, and/or brain and behavior abnormalities. (1) Masi I et al. Int J Comput Vis 2019. (2) Whitehouse AJ et al. Proc Biol Sci 2015. (3) Lefevre CE et al. Evol Hum Behav 2013

KPNet: Towards Minimal Face Detector

The small receptive field and capacity of minimal neural networks limit their performance when using them to be the backbone of detectors. In this work, we find that the appearance feature of a generic face is discriminative enough for a tiny and shallow neural network to verify from the background. And the essential barriers behind us are 1) the vague definition of the face bounding box and 2) tricky design of anchor-boxes or receptive field. Unlike most top-down methods for joint face detection and alignment, the proposed KPNet detects small facial keypoints instead of the whole face by in the bottom-up manner. It first predicts the facial landmarks from a low-resolution image via the well-designed fine-grained scale approximation and scale adaptive soft-argmax operator. Finally, the precise face bounding boxes, no matter how we define it, can be inferred from the keypoints. Without any complex head architecture or meticulous network designing, the KPNet achieves state-of-the-art accuracy on generic face detection and alignment benchmarks with only ∼ 1M parameters, which runs at 1000fps on GPU and is easy to perform real-time on most modern front-end chips.

End-to-End Spatial Transform Face Detection and Recognition

BackgroundSeveral face detection and recognition methods have been proposed in the past decades that have excellent performance. The conventional face recognition pipeline comprises the following: (1) face detection, (2) face alignment, (3) feature extraction, and (4) similarity, which are independent of each other. The separate facial analysis stages lead to redundant model calculations, and are difficult for use in end-to-end training. MethodsIn this paper, we propose a novel end-to-end trainable convolutional network framework for face detection and recognition, in which a geometric transformation matrix is directly learned to align the faces rather than predicting the facial landmarks. In the training stage, our single CNN model is supervised only by face bounding boxes and personal identities, which are publicly available from WIDER FACE and CASIA-WebFace datasets. Our model is tested on Face Detection Dataset and Benchmark (FDDB) and Labeled Face in the Wild (LFW) datasets. ResultsThe results show 89.24% recall for face detection tasks and 98.63% accuracy for face recognition tasks.

Crowd Detection by Video Processing

The video may consist of multiple shots. The target in the probe video is only annotated once with a face bounding box in a frame. Most video face identification techniques assume that the video is of single shot, and thus the bounding boxes of the target face can be extracted by tracking a face across the video frames. Nevertheless, such automatic annotation is vulnerable to the drifting of the face tracker, and the face tracking algorithm is inadequate to associate the face images of the target across multiple shots. A target face association (TFA) technique retrieves a set of representative face images in a given video that are likely to have the same identity as the target face. These face images are then utilized to construct a robust face representation of the target face for searching the corresponding subject in the gallery. Since two faces that appear in the same video frame cannot belong to the same person, such cannot-link constraints are utilized for learning a target-specific linear classifier for establishing the intra/ inter-shot face association of the target. Experimental results on the newly released JANUS challenge set 3 (JANUS CS3) dataset show that TFA method generates robust representations from target-annotated videos and demonstrates good performance for the task of video-based face identification problem.

Task-Oriented Feature-Fused Network With Multivariate Dataset for Joint Face Analysis.

Deep multitask learning for face analysis has received increasing attentions. From literature, most existing methods focus on optimizing a main task by jointly learning several auxiliary tasks. It is challenging to consider the performance of each task in a multitask framework due to the following reasons: 1) different face tasks usually rely on different levels of semantic features; 2) each task has different learning convergence rate, which could affect the whole performance when joint training; and 3) multitask model needs rich label information for efficient training, but existing facial datasets provide limited annotations. To address these issues, we propose a task-oriented feature-fused network (TFN) for simultaneously solving face detection, landmark localization, and attribute analysis. In this network, a task-oriented feature-fused block is designed to learn task-specific feature combinations; then, an alternative multitask training scheme is presented to optimize each task with considering of their different learning capacities. We also present a large-scale face dataset called JFA in support of proposed method, which provides multivariate labels, including face bounding box, 68 facial landmarks, and 3 attribute labels (i.e., apparent age, gender, and ethnicity). The experimental results suggest that the TFN outperforms several multitask models on the JFA dataset. Furthermore, our approach achieves competitive performances on WIDER FACE and 300W dataset, and obtains state-of-the-art results for gender recognition on the MORPH II dataset.

Eye state recognition method for drivers with glasses

Eye state recognition is a key step in fatigue detection method. However, factors such as occlusion of different types of glasses and changes in lighting conditions may have some impact on eye state recognition. In order to solve these problems, a driver’s eye state recognition method based on deep learning is proposed. Firstly, the driver’s face images are acquired using an infrared acquisition device. Secondly the multi-task cascaded convolution neural networks are used to detect the face bounding box and feature points of the driver’s face image, and then the eye regions are extracted. Finally the Convolution Neural Network (CNN) is adopted to identify the open and closed state of the eyes. Experimental result shows that the proposed method can accurately identify the state of eyes and help to calculate the fatigue parameters of drivers.

Integrated Deep Model for Face Detection and Landmark Localization From “In The Wild” Images

The tasks of face detection and landmark localisation are a key foundation for many facial analysis applications, while great advancements have been achieved in recent years there are still challenges to increase the precision of face detection. Within this paper, we present our novel method the Integrated Deep Model (IDM), fusing two state-of-the-art deep learning architectures, namely, Faster R-CNN and a stacked hourglass for improved face detection precision and accurate landmark localisation. Integration is achieved through the application of a novel optimisation function and is shown in experimental evaluation to increase accuracy of face detection specifically precision by reducing false positive detection’s by an average of 62%. Our proposed IDM method is evaluated on the Annotated Faces In-The-Wild, Annotated Facial Landmarks In The Wild and the Face Detection Dataset and Benchmark face detection test sets and shows a high level of recall and precision when compared with previously proposed methods. Landmark localisation is evaluated on the Annotated Faces In-The-Wild and 300-W test sets, this specifically focuses on localisation accuracy from detected face bounding boxes when compared with baseline evaluations using ground truth bounding boxes. Our findings highlight only a small 0.005% maximum increase in error which is more profound for the subset of facial landmarks which border the face.