Facial Keypoints Research Articles

Distance Estimation in Thermal Cameras Using Multi-Task Cascaded Convolutional Neural Network

The rapid growth of the current pandemic (COVID-19) requires the use of thermal cameras that can perform fast and automatic body temperature measurement. The accuracy of the temperature measurement is affected by its distance from a person. Conventional distance estimation methods utilize the coordinates of the bounding box provided by several face detection algorithms such as YOLOv3 and SSD. The bounding box output of these methods varies which causes inaccurate distance estimation results. In this study, we propose a distance estimation method for thermal camera applications based on the coordinates of the facial key points extracted using multi-task cascaded convolutional neural network. The result obtained in this study proves that the proposed method exhibits higher accuracy (root mean square error of 2.9695 cm in comparison with an RMSE of 25.26 cm using other methods) and the least CPU and memory consumption in comparison with conventional methods.

Driving-signal aware full-body avatars

We present a learning-based method for building driving-signal aware full-body avatars. Our model is a conditional variational autoencoder that can be animated with incomplete driving signals, such as human pose and facial keypoints, and produces a high-quality representation of human geometry and view-dependent appearance. The core intuition behind our method is that better drivability and generalization can be achieved by disentangling the driving signals and remaining generative factors, which are not available during animation. To this end, we explicitly account for information deficiency in the driving signal by introducing a latent space that exclusively captures the remaining information, thus enabling the imputation of the missing factors required during full-body animation, while remaining faithful to the driving signal. We also propose a learnable localized compression for the driving signal which promotes better generalization, and helps minimize the influence of global chance-correlations often found in real datasets. For a given driving signal, the resulting variational model produces a compact space of uncertainty for missing factors that allows for an imputation strategy best suited to a particular application. We demonstrate the efficacy of our approach on the challenging problem of full-body animation for virtual telepresence with driving signals acquired from minimal sensors placed in the environment and mounted on a VR-headset.

Driver Fatigue Detection Based on Facial Key Points and LSTM

In recent years, fatigue driving has been a serious threat to the traffic safety, which makes the research of fatigue detection a hotspot field. Research on fatigue recognition has a great significance to improve the traffic safety. However, the existing fatigue detection methods still have room for improvement in detection accuracy and efficiency. In order to detect whether the driver has fatigue driving, this paper proposes a fatigue state recognition algorithm. The method first uses MTCNN (multitask convolutional neural network) to detect human face, and then DLIB (an open-source software library) is used to locate facial key points to extract the fatigue feature vector of each frame. The fatigue feature vectors of multiple frames are spliced into a temporal feature sequence and sent to the LSTM (long short-term memory) network to obtain a final fatigue feature value. Experiments show that compared with other methods, the fatigue state recognition algorithm proposed in this paper has achieved better results in accuracy. The average accuracy of the proposed method in detecting key points of the face is as high as 93%, and the running time is less than half of the ordinary DLIB method.

Towards Machine Recognition of Facial Expressions of Pain in Horses.

Simple SummaryFacial activity can convey valid information about the experience of pain in a horse. However, scoring of pain in horses based on facial activity is still in its infancy and accurate scoring can only be performed by trained assessors. Pain in humans can now be recognized reliably from video footage of faces, using computer vision and machine learning. We examine the hurdles in applying these technologies to horses and suggest two general approaches to automatic horse pain recognition. The first approach involves automatically detecting objectively defined facial expression aspects that do not involve any human judgment of what the expression “means”. Automated classification of pain expressions can then be done according to a rule-based system since the facial expression aspects are defined with this information in mind. The other involves training very flexible machine learning methods with raw videos of horses with known true pain status. The upside of this approach is that the system has access to all the information in the video without engineered intermediate methods that have filtered out most of the variation. However, a large challenge is that large datasets with reliable pain annotation are required. We have obtained promising results from both approaches.Automated recognition of human facial expressions of pain and emotions is to a certain degree a solved problem, using approaches based on computer vision and machine learning. However, the application of such methods to horses has proven difficult. Major barriers are the lack of sufficiently large, annotated databases for horses and difficulties in obtaining correct classifications of pain because horses are non-verbal. This review describes our work to overcome these barriers, using two different approaches. One involves the use of a manual, but relatively objective, classification system for facial activity (Facial Action Coding System), where data are analyzed for pain expressions after coding using machine learning principles. We have devised tools that can aid manual labeling by identifying the faces and facial keypoints of horses. This approach provides promising results in the automated recognition of facial action units from images. The second approach, recurrent neural network end-to-end learning, requires less extraction of features and representations from the video but instead depends on large volumes of video data with ground truth. Our preliminary results suggest clearly that dynamics are important for pain recognition and show that combinations of recurrent neural networks can classify experimental pain in a small number of horses better than human raters.

Face Alignment Algorithm Based on an Improved Cascaded Convolutional Neural Network

Aiming at the problem of a large number of parameters and high time complexity caused by the current deep convolutional neural network models, an improved face alignment algorithm of a cascaded convolutional neural network (CCNN) is proposed from the network structure, random perturbation factor (shake), and data scale. The algorithm steps are as follows: 3 groups of lightweight CNNs are designed; the first group takes facial images with face frame as input, trains 3 CNNs in parallel, and weighted outputs the facial images with 5 facial key points (anchor points). Then, the anchor points and 2 different windows with a shake mechanism are used to crop out 10 partial images of human faces. The networks in the second group train 10 CNNs in parallel and every 2 networks’ weighted average and colocated a key point. Based on the second group of networks, the third group designed a smaller shake mechanism and windows, to achieve more fine-tuning. When training the network, the idea of parallel within groups and serial between groups is adopted. Experiments show that, on the LFPW face dataset, the improved CCNN in this paper is superior to any other algorithm of the same type in positioning speed, algorithm parameter amount, and test error.

Adaptive Feature Selection-Based AdaBoost-KNN With Direct Optimization for Dynamic Emotion Recognition in Human–Robot Interaction

AdaBoost-KNN using adaptive feature selection with direct optimization is proposed for dynamic emotion recognition in human-robot interaction, where the real-time dynamic emotion is recognized based on facial expression. It can make robots capable of understanding human dynamic emotions, in such a way that human-robot interaction is realized in a smooth manner. Based on the facial key points extracted by Candide-3 model, adaptive feature selection is adopted, namely Plus-L Minus-R Selection is completed. It can determine the features that contribute the most to emotion recognition, thereby forming the basis of emotion classification. Emotion classification is based on AdaBoost-KNN, which builds a series of KNN classifiers. AdaBoost-KNN adjusts the weights of the data in an iterative manner. Moreover, global optimal parameters are approximated with direct optimization until the recognition rate reaches its maximal value. The experimental performance of the proposal is verified by a k-fold cross-validation. Results show that the recognition rate of the proposed approach is higher than that of the AdaBoost-KNN, adaptive feature selection-based AdaBoost-KNN, and AdaBoost-KNN with direct optimization. It is also higher than the rate achieved by other traditional recognition methods, such as AdaBoost, KNN, and SVM. In addition, preliminary application experiments are developed in our emotional social robot system, composed of two mobile robots. The experiments demonstrate the dynamic emotion understanding ability of robots in human-robot interaction.

Face Swapping Using Convolutional Neural Network and Tiny Facet Primitive

Objectives Face swapping technology has important application value in entertainment, virtual reality, film and so on. However, existing methods are limited by face pose consistency and can not overcome the influence of occlusion. Methods We proposes a method of face swapping using convolutional neural network and tiny facet primitive. Firstly, detect the face using the cascade convolutional neural network and segment the face to determine the replacement region using fully convolutional network.Then, the Wallis transform is applied to adjust the skin color of the source image to make it consistent with the skin color of the face in the target image.After that, using facial key points detection method based on an ensemble of regression trees and Delaunay triangulation to construct the face triangulation network, then replacing faces based on tiny facet primitive. Finally, applying Poisson fusion to eliminate splicing traces between different images. Results We evaluate the performance of the proposed method compared with existing method through qualitative and quantitative experiments. Experimental results show that face segment can well solve the problem of occlusion such as cap, glasses, and hair.Moreover, when source image and target image have different face poses, replacing face area using tiny facet primitive separately performs better than using the whole face area. Conclusions Our method can well solve the problem of face pose consistency limitation and occlusion, which has certain practical application value.

Modeling social interaction and intention for pedestrian trajectory prediction

Future pedestrian trajectory prediction offers great prospects for many practical applications. Most existing methods focus on social interaction among pedestrians but ignore the fact that in addition to pedestrians there are other kinds of objects (cars, dogs, bicycles, motorcycles, etc.) with a great influence on the subject pedestrian’s future trajectory. Most existing methods neglect the intentions of the pedestrian, which can be obtained by the key points of the subject pedestrian’s face. Therefore, rich category information about the subject pedestrian’s surroundings and face key points plays a great role in promoting the modeling of pedestrian movement. Motivated by this idea, this paper tries to predict a pedestrian’s future trajectory by jointly using various categories and the relative positions of the subject pedestrian’s surroundings and the key points in his face. We propose a data modeling method to effectively unify rich visual features about categories, interaction and face key points into a multi-channel tensor and build an end-to-end fully convolutional encoder–decoder attention model based on convolutional long–short-term memory utilizing this tensor. We evaluate and compare our method with several existing methods on 5 crowded video sequences from the public dataset multi-object tracking (MOT) -16. Experimental results show that our method outperforms state-of-the-art approaches, with less prediction error.

3D Virtual Animation Instant Network Communication System Design

This project uses Openfire to implement a virtual 3D animation instant messaging system, which is easier to use and more expandable. The main work of the client is to implement the Extensible Messaging and Presence Protocol (XMPP) and use XMPP to transmit data to the server side and receive data from the server side, while Openfire is built by the server side to use. To address the problem that the current mainstream face key point localization model is less robust to complex environments, this project adopts a deep learning‐based approach to design and implement the face key point localization model, through data preprocessing, model design, and model training, to achieve a robust model that can locate 68 face key points and complete the migration of the model to mobile. The current video communication often suffers from delay and lag, so this project uses face key point data instead of video stream data transmission to reduce the pressure on the network. This topic also uses voice coding and decoding, noise reduction, echo cancellation, and other processing to solve the problems of noise interference and echo interference in voice transmission. This paper also introduces the creation, import, and loading of 3D virtual models, and explains how to use face key point association to drive 3D animation models, how to make the drive smoother and more natural, and using individual face key points as an example.

Fatigue Driving Detection Based on Deep Learning and Multi-Index Fusion

In order to reduce traffic accidents caused by fatigue driving, a fatigue driving detection algorithm is proposed based on deep learning and facial multi-index fusion from the driver $'\text{s}$ facial features. Because the scene in the actual driving process is very complex and changeable, this algorithm first improves the multi-task cascaded convolutional neural network (MTCNN) so that it can quickly and accurately locate the face and detect the facial key points. According to the facial key points, the driver’s eyes and mouth regions are determined. Second, these regions are input into the eyes and mouth state recognition network (E-MSR Net) for state recognition. The E-MSR Net is a depth separable convolution neural network that is improved and optimized based on MobilenetV2. Finally, the three facial features of eye closure rate (ECR), mouth opening rate (MOR), and head non-positive face rate (HNFR) are fused to judge the driver’s fatigue state. This algorithm can quickly and accurately make judgments in the face of complex and changeable scenes. At the same time, it can avoid the failure of the algorithm caused by the occlusion of the eyes or mouth due to wearing sunglasses or masks during driving. The accuracy of the proposed algorithm on the self-made data set achieved 97.5%, which proved the feasibility of the algorithm.

基于人脸关键点检测的相机标定方法

基于人脸关键点检测的相机标定方法

CASIA-Face-Africa: A Large-Scale African Face Image Database

Face recognition is a popular and well-studied area with wide applications in our society. However, racial bias had been proven to be inherent in most State Of The Art (SOTA) face recognition systems. Many investigative studies on face recognition algorithms have reported higher false positive rates of African subjects cohorts than the other cohorts. Lack of large-scale African face image databases in public domain is one of the main restrictions in studying the racial bias problem of face recognition. To this end, we collect a face image database namely CASIA-Face-Africa which contains 38,546 images of 1,183 African subjects. Multi-spectral cameras are utilized to capture the face images under various illumination settings. Demographic attributes and facial expressions of the subjects are also carefully recorded. For landmark detection, each face image in the database is manually labeled with 68 facial keypoints. A group of evaluation protocols are constructed according to different applications, tasks, partitions and scenarios. The performances of SOTA face recognition algorithms without re-training are reported as baselines. The proposed database along with its face landmark annotations, evaluation protocols and preliminary results form a good benchmark to study the essential aspects of face biometrics for African subjects, especially face image preprocessing, face feature analysis and matching, facial expression recognition, sex/age estimation, ethnic classification, face image generation, etc. The database can be downloaded from our website.

Feature recognition method for similar key points of human face based on adaptive median filter

In order to overcome the low efficiency and poor accuracy of current face feature recognition methods, this paper proposes a feature recognition method for similar key points of human face based on adaptive median filter. The preprocessing of face image including rotation, scaling and clipping was carried out to remove the influence of image background on feature recognition of similar key points of face. The grey level of the input image is mapped to the output image pixel by function mapping to achieve histogram homogenisation of the grey face image. Based on adaptive median filtering, a face image denoising model is constructed and trained. The experimental results show that the recognition time of the proposed method is less than 0.7 s, the SNR is higher than 24 dB, the recognition accuracy is more than 90%, and the recognition effect is better.

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields.

Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime approach to detect the 2D pose of multiple people in an image. The proposed method uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. This bottom-up system achieves high accuracy and realtime performance, regardless of the number of people in the image. In previous work, PAFs and body part location estimation were refined simultaneously across training stages. We demonstrate that a PAF-only refinement rather than both PAF and body part location refinement results in a substantial increase in both runtime performance and accuracy. We also present the first combined body and foot keypoint detector, based on an internal annotated foot dataset that we have publicly released. We show that the combined detector not only reduces the inference time compared to running them sequentially, but also maintains the accuracy of each component individually. This work has culminated in the release of OpenPose, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints.

RGB-D-based gaze point estimation via multi-column CNNs and facial landmarks global optimization

In this work, we utilize a multi-column CNNs framework to estimate the gaze point of a person sitting in front of a display from an RGB-D image of the person. Given that gaze points are determined by head poses, eyeball poses, and 3D eye positions, we propose to infer the three components separately and then integrate them for gaze point estimation. The captured depth images, however, usually contain noises and black holes which prevent us from acquiring reliable head pose and 3D eye position estimation. Therefore, we propose to refine the raw depth for 68 facial keypoints by first estimating their relative depths from RGB face images, which along with the captured raw depths are then used to solve the absolute depth for all facial keypoints through global optimization. The refined depths will provide us reliable estimation for both head pose and 3D eye position. Given that existing publicly available RGB-D gaze tracking datasets are small, we also build a new dataset for training and validating our method. To the best of our knowledge, it is the largest RGB-D gaze tracking dataset in terms of the number of participants. Comprehensive experiments demonstrate that our method outperforms existing methods by a large margin on both our dataset and the Eyediap dataset.

The Effect of Using Histogram Equalization and Discrete Cosine Transform on Facial Keypoint Detection

This study aims to figure out the effect of using Histogram Equalization and Discrete Cosine Transform (DCT) in detecting facial keypoints, which can be applied for 3D facial reconstruction in face recognition. Four combinations of methods comprising of Histogram Equalization, removing low-frequency coefficients using Discrete Cosine Transform (DCT) and using five feature detectors, namely: SURF, Minimum Eigenvalue, Harris-Stephens, FAST, and BRISK were used for test. Data that were used for test were obtained from Head Pose Image and ORL Databases. The result from the test were evaluated using F-score. The highest F-score for Head Pose Image Dataset is 0.140 and achieved through the combination of DCT & Histogram Equalization with feature detector SURF. The highest F-score for ORL Database is 0.33 and achieved through the combination of DCT & Histogram Equalization with feature detector BRISK.

Image Restoration Effect on DCT High Frequency Removal and Wiener Algorithm for Detecting Facial Key Points

This study aims to figure out the effect of using Histogram Equalization and Discrete Cosine Transform (DCT) in detecting facial keypoints, which can be applied for 3D facial reconstruction in face recognition. Four combinations of methods comprising of Histogram Equalization, removing low-frequency coefficients using Discrete Cosine Transform (DCT) and using five feature detectors, namely: SURF, Minimum Eigenvalue, Harris-Stephens, FAST, and BRISK were used for test. Data that were used for test were obtained from Head Pose Image and ORL Databases. The result from the test were evaluated using F-score. The highest F-score for Head Pose Image Dataset is 0.140 and achieved through the combination of DCT & Histogram Equalization with feature detector SURF. The highest F-score for ORL Database is 0.33 and achieved through the combination of DCT & Histogram Equalization with feature detector BRISK.

End-to-End Training for Compound Expression Recognition

For a long time, expressions have been something that human beings are proud of. That is an essential difference between us and machines. With the development of computers, we are more eager to develop communication between humans and machines, especially communication with emotions. The emotional growth of computers is similar to the growth process of each of us, starting with a natural, intimate, and vivid interaction by observing and discerning emotions. Since the basic emotions, angry, disgusted, fearful, happy, neutral, sad and surprised are put forward, there are many researches based on basic emotions at present, but few on compound emotions. However, in real life, people’s emotions are complex. Single expressions cannot fully and accurately show people’s inner emotional changes, thus, exploration of compound expression recognition is very essential to daily life. In this paper, we recommend a scheme of combining spatial and frequency domain transform to implement end-to-end joint training based on model ensembling between models for appearance and geometric representations learning for the recognition of compound expressions in the wild. We are mainly devoted to digging the appearance and geometric information based on deep learning models. For appearance feature acquisition, we adopt the idea of transfer learning, introducing the ResNet50 model pretrained on VGGFace2 for face recognition to implement the fine-tuning process. Here, we try and compare two minds, one is that we utilize two static expression databases FER2013 and RAF Basic for basic emotion recognition to fine tune, the other is that we fine tune the model on the input three channels composed of images generated by DWT2 and WAVEDEC2 wavelet transforms based on rbio3.1 and sym1 wavelet bases respectively. For geometric feature acquisition, we firstly introduce a densesift operator to extract facial key points and their histogram descriptions. After that, we introduce deep SAE with a softmax function, stacked LSTM and Sequence-to-Sequence with stacked LSTM and define their structures by ourselves. Then, we feed the salient key points and their descriptions into three models to train respectively and compare their performances. When the model training for appearance and geometric features learning is completed, we combine the two models with category labels to achieve further end-to-end joint training, considering that ensembling models, which describe different information, can further improve recognition results. Finally, we validate the performance of our proposed framework on an RAF Compound database and achieve a recognition rate of 66.97%. Experiments show that integrating different models, which express different information, and achieving end-to-end training can quickly and effectively improve the performance of the recognition.

Single-shot face and landmarks detector

Facial landmark detection is an important sub-task in solving a number of biometric facial recognition tasks. In face recognition systems, the construction of a biometric template occurs according to a previously aligned (normalized) face image and the normalization stage includes the task of finding facial keypoints. A balance between quality and speed of the facial keypoints detector is important in such a problem. This article proposes a CNN-based one-stage detector of faces and keypoints operating in real time and achieving high quality on a number of well-known test datasets (such as AFLW2000, COFW, Menpo2D). The proposed face and facial landmarks detector is based on the idea of a one-stage SSD object detector, which has established itself as an algorithm that provides high speed and high quality in object detection task. As a basic CNN architecture, we used the ShuffleNet V2 network. An important feature of the proposed algorithm is that the face and facial keypoint detection is done in one CNN forward pass, which can significantly save time at the implementation stage. Also, such multitasking allows one to reduce the percentage of errors in the facial keypoints detection task, which positively affects the final face recognition algorithm quality.

Automatic Identification of Individual Primates with Deep Learning Techniques.

SummaryThe difficulty of obtaining reliable individual identification of animals has limited researcher's ability to obtain quantitative data to address important ecological, behavioral, and conservation questions. Traditional marking methods placed animals at undue risk. Machine learning approaches for identifying species through analysis of animal images has been proved to be successful. But for many questions, there needs a tool to identify not only species but also individuals. Here, we introduce a system developed specifically for automated face detection and individual identification with deep learning methods using both videos and still-framed images that can be reliably used for multiple species. The system was trained and tested with a dataset containing 102,399 images of 1,040 individuals across 41 primate species whose individual identity was known and 6,562 images of 91 individuals across four carnivore species. For primates, the system correctly identified individuals 94.1% of the time and could process 31 facial images per second.