Face Pixels Research Articles

Face Recognition Based Attendance Software and Reporting using OpenCV

The face is the identity of a person. The methods to exploit this physical feature have seen a great change since the advent of image processing techniques. The attendance is taken in every school, colleges and libraries. Traditional approach for attendance is professor calls student name and record attendance. It takes some time to record attendance. For each lecture this is a waste of time. To avoid these losses, we're going to use automatic process which is based on image processing. Identification of any people in any organization or colleges for the purpose of attendance marking is one of such software. Authentication is a significant issue in computer-based communication. Human face recognition is an important branch of biometric verification and has been widely used in many applications. The use of Attendance Management System is to perform the regular activities of attendance marking and analysis with less human intervention. In this new approach, we are using face detection and face recognition system. This face detection distinguishes faces from non-faces and is essential for accurate attendance. The other strategy is facing recognition for marking the student’s attendance. We have used OpenCV for this system. The camera will be connected using OpenCV module that converts the image into the RGB format which in turn is mapped to the pre-trained neural networks using HOG algorithm to recognize the face pixels. This system offers effective way to manage the attendance system which is time efficient and offers massive scaling for future purposes. The image is processed as follows: first, faces are identified using a Haarcascade classifier, then faces are recognized using the LBPH (Local Binary Pattern Histogram) Algorithm, histogram data is checked against an established dataset, and the device automatically labels attendance. An Excel sheet is developed, and it is updated every hour with the information from the respective class instructor.

Shot classification for human behavioural analysis in video surveillance applications

Human behavior analysis plays a vital role in ensuring security and safety of people in crowded public places against diverse contexts like theft detection, violence prevention, explosion anticipation etc. Analysing human behaviour by classifying of videos in to different shot types helps in extracting appropriate behavioural cues. Shots indicates the subject size within the frame and the basic camera shots include: the close-up, medium shot, and the long shot. If the video is categorised as Close-up shot type, investigating emotional displays helps in identifying criminal suspects by analysing the signs of aggressiveness and nervousness to prevent illegal acts. Mid shot can be used for analysing nonverbal communication like clothing, facial expressions, gestures and personal space. For long shot type, behavioural analysis is by extracting the cues from gait and atomic action displayed by the person. Here, the framework for shot scale analysis for video surveillance applications is by using Face pixel percentage and deep learning based method. Face Pixel ratio corresponds to the percentage of region occupied by the face region in a frame. The Face pixel Ratio is thresholded with predefined threshold values and grouped into Close-up shot, mid shot and long shot categories. Shot scale analysis based on transfer learning utilizes effective pre-trained models that includes AlexNet, VGG Net, GoogLeNet and ResNet. From experimentation, it is observed that, among the pre-trained models used for experimentation GoogLeNet tops with the accuracy of 94.61%.

Contactless Heart Rate Measurement System Based RGB Camera, Cascade Object Detector and Independent Component Analysis

This paper focuses on the development of non-contact heart rate measurement based on video analysis. This project implements an RGB camera (OV2640 with ESP32 Cam) and an FTDI Programmer which enables the sensor to record videos from a subject at a distance of 20 cm. The Region of Interest (ROI) of the facial skin tissue was detected by using Cascade Object Detector (COD) which can detect object categories whose aspect ratio does not vary significantly. Then, the Independent Component Analysis (ICA) was applied by extracting distinct source signals (red, blue and green channels) from the observed mixed colour signals. Next, the average pixel of ROI was calculated based on red, green and blue facial pixel at given time. Finally, the heart rate was extracted from the frequency spectrum of the average pixel. The performance of the developed contactless system was compared with the fingertip pulse oximeter (Oxitech) by means of percentage of error (PE). The lowest range of PE is between 0 to 5.62 % whereas the highest PE is 22.54 %. The highest error was due to incorrectly captured ROI of facial skin area as well as due to the movement made during the recording session. Meanwhile, the average time for the developed system to process the heart rate of the subject is 4.51 minutes. In future, a robust ROI algorithm can be adopted to reduce error contributed by incorrectly ROI selection. Other than that, this project has a potential to contribute to the area of contactless measurement of oxygen saturation.

Deep-HR: Fast heart rate estimation from face video under realistic conditions

This paper presents a novel method for remote heart rate (HR) estimation. Recent studies have proved that blood pumping by the heart is highly correlated to the intense color of face pixels, and surprisingly can be utilized for remote HR estimation. Researchers successfully proposed several methods for this task, but making it work in realistic situations is still a challenging problem in computer vision community. Furthermore, learning to solve such a complex task on a dataset with very limited annotated samples is not reasonable. Consequently, researchers do not prefer to use the deep learning approaches for this problem. In this paper, we propose a simple yet efficient approach to benefit the advantages of the Deep Neural Network (DNN) by simplifying HR estimation from a complex task to learning from very correlated representation to HR. Inspired by previous work, we learn a component called Front-End (FE) to provide a discriminative representation of face videos, afterward a light deep regression auto-encoder as Back-End (BE) is learned to map the FE representation to HR. Regression task on the informative representation is simple and could be learned efficiently on limited training samples. Beside of this, to be more accurate and work well on low-quality videos, two deep encoder–decoder networks are trained to refine the output of FE. We also introduce a challenging dataset (HR-D) to show that our method can efficiently work in realistic conditions. Experimental results on HR-D and MAHNOB datasets confirm that our method could run as a real-time method and estimate the average HR better than state-of-the-art ones.

LARNet: Real-Time Detection of Facial Micro Expression Using Lossless Attention Residual Network.

Facial micro expressions are brief, spontaneous, and crucial emotions deep inside the mind, reflecting the actual thoughts for that moment. Humans can cover their emotions on a large scale, but their actual intentions and emotions can be extracted at a micro-level. Micro expressions are organic when compared with macro expressions, posing a challenge to both humans, as well as machines, to identify. In recent years, detection of facial expressions are widely used in commercial complexes, hotels, restaurants, psychology, security, offices, and education institutes. The aim and motivation of this paper are to provide an end-to-end architecture that accurately detects the actual expressions at the micro-scale features. However, the main research is to provide an analysis of the specific parts that are crucial for detecting the micro expressions from a face. Many states of the art approaches have been trained on the micro facial expressions and compared with our proposed Lossless Attention Residual Network (LARNet) approach. However, the main research on this is to provide analysis on the specific parts that are crucial for detecting the micro expressions from a face. Many CNN-based approaches extracts the features at local level which digs much deeper into the face pixels. However, the spatial and temporal information extracted from the face is encoded in LARNet for a feature fusion extraction on specific crucial locations, such as nose, cheeks, mouth, and eyes regions. LARNet outperforms the state-of-the-art methods with a slight margin by accurately detecting facial micro expressions in real-time. Lastly, the proposed LARNet becomes accurate and better by training with more annotated data.

Facial Expression Recognition Using Hybrid Features of Pixel and Geometry

Facial Expression Recognition (FER) has long been a challenging task in the field of computer vision. Most of the existing FER methods extract facial features on the basis of face pixels, ignoring the relative geometric position dependencies of facial landmark points. This article presents a hybrid feature extraction network to enhance the discriminative power of emotional features. The proposed network consists of a Spatial Attention Convolutional Neural Network (SACNN) and a series of Long Short-term Memory networks with Attention mechanism (ALSTMs). The SACNN is employed to extract the expressional features from static face images and the ALSTMs is designed to explore the potentials of facial landmarks for expression recognition. A deep geometric feature descriptor is proposed to characterize the relative geometric position correlation of facial landmarks. The landmarks are divided into seven groups to extract deep geometric features, and the attention module in ALSTMs can adaptively estimate the importance of different landmark regions. By jointly combining SACNN and ALSTMs, the hybrid features are obtained for expression recognition. Experiments conducted on three public databases, FER2013, CK+, and JAFFE, demonstrate that the proposed method outperforms the previous methods, with the accuracies of 74.31%, 95.15%, and 98.57%, respectively. The preliminary results of Emotion Understanding Robot System (EURS) indicate that the proposed method has the potential to improve the performance of human-robot interaction.

A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition

Nowadays, human emotion recognition is a mandatory task for many human machine interaction fields. This paper proposes a novel multi-modal human emotion recognition framework. The proposed scheme utilizes first the 3D-Convolutional Neural Network (3D-CNN) deep learning architecture for extracting the spatio-temporal features from the electroencephalogram (EEG) signals, and the video data of human faces. Then, a combination of data augmentation, ensemble learning techniques is proposed to get the final fusion predictions. The fusion of the multi-modalities in the proposed scheme is carried out using data, and score fusion methods. Hence, three human recognition approaches are built to achieve the proposed goal. They are namely EEG-based emotion recognition approach, face-based emotion recognition approach, and fusion-based emotion recognition approach. For the EEG approach, the 3D-CNN is used to get the final predictions of the EEG signal. For the face approach, the Mask-RCNN object detection technique in combination with OpenCV libraries are first utilized to extract the exact face pixels with emotional content. Then, the Support Vector Machine (SVM) classifier is utilized to classify the 3D-CNN output features of the face chunks. For the fusion-based emotion recognition approach, two fusion techniques are experimented; bagging, and stacking. It is found that the stacking technique gives the best accuracy, and achieves recognition accuracies 96.13%, and 96.79% for valence, and arousal classes respectively using the grid search ensemble learning technique due to transferring the weights from the EEG, and the face approaches to the fusion-based emotion recognition approach. The proposed approach outperforms recent works in multi-modal emotion recognition field.

Hybrid Siamese Network for Unconstrained Face Verification and Clustering under Limited Resources

In this paper, we propose an unconstrained face verification approach that is dependent on Hybrid Siamese architecture under limited resources. The general face verification trend suggests that larger training datasets and/or complex architectures lead to higher accuracy. The proposed approach tends to achieve high accuracy while using a small dataset and a simple architecture by directly learn face’s similarity/dissimilarity from raw face pixels, which is critical for various applications. The proposed architecture has two branches; the first part of these branches is trained independently, while the other parts shared their parameters. A multi-batch algorithm is utilized for training. The training process takes a few hours on a single GPU. The proposed approach achieves near-human accuracy (98.9%) on the Labeled Faces in the Wild (LFW) benchmark, which is competitive with other techniques that are presented in the literature. It reaches 99.1% on the Arabian faces dataset. Moreover, features learned by the proposed architecture are used in building a face clustering system that is based on an updated version of the Density-Based Spatial Clustering of Applications with Noise (DBSCAN). To handle the cluster quality challenge, a novel post-clustering optimization procedure is proposed. It outperforms popular clustering approaches, like K-Means and spectral by 0.098 and up to 0.344 according to F1-measure.

Face Frontalization Using an Appearance-Flow-Based Convolutional Neural Network.

Facial pose variation is one of the major factors making face recognition (FR) a challenging task. One popular solution is to convert non-frontal faces to frontal ones on which FR is performed. Rotating faces causes facial pixel value changes. Therefore, existing CNN-based methods learn to synthesize frontal faces in color space. However, this learning problem in a color space is highly non-linear, causing the synthetic frontal faces to lose fine facial textures. In this paper, we take the view that the nonfrontal-frontal pixel changes are essentially caused by geometric transformations (rotation, translation, and so on) in space. Therefore, we aim to learn the nonfrontal-frontal facial conversion in the spatial domain rather than the color domain to ease the learning task. To this end, we propose an appearance-flow-based face frontalization convolutional neural network (A3F-CNN). Specifically, A3F-CNN learns to establish the dense correspondence between the non-frontal and frontal faces. Once the correspondence is built, frontal faces are synthesized by explicitly "moving" pixels from the non-frontal one. In this way, the synthetic frontal faces can preserve fine facial textures. To improve the convergence of training, an appearance-flow-guided learning strategy is proposed. In addition, generative adversarial network loss is applied to achieve a more photorealistic face, and a face mirroring method is introduced to handle the self-occlusion problem. Extensive experiments are conducted on face synthesis and pose invariant FR. Results show that our method can synthesize more photorealistic faces than the existing methods in both the controlled and uncontrolled lighting environments. Moreover, we achieve a very competitive FR performance on the Multi-PIE, LFW and IJB-A databases.

Face spoofing detection based on color texture Markov feature and support vector machine recursive feature elimination

Aiming to counterstrike face spoofing attacks such as photo attacks and video attacks, a face spoofing detection scheme based on color texture Markov feature (CTMF) and support vector machine recursive feature elimination (SVM-RFE) is proposed. In this paper, the adjacent facial pixels discrepancy between the real and the fake face is analyzed, and texture information between the color channels is fully considered. Firstly, the directional difference filter is used to capture the facial texture difference between the real and the fake face, which can be regarded as low-level features of CTMF. Then, the facial texture difference is modeled by the Markov process to form a high-level representation of the low-level features. Meanwhile, the mutual information of facial texture between the color channels, which is ignored in the previous literature, is investigated. In addition, SVM-RFE is utilized to reduce the feature dimension and makes it suitable for real-time detection. Experiments on four public benchmark databases indicate that the proposed scheme can effectively resist photo and video spoofing attacks in face recognition.

Face Alignment in Full Pose Range: A 3D Total Solution.

Face alignment, which fits a face model to an image and extracts the semantic meanings of facial pixels, has been an important topic in the computer vision community. However, most algorithms are designed for faces in small to medium poses (yaw angle is smaller than 45 degree), which lack the ability to align faces in large poses up to 90 degree. The challenges are three-fold. First, the commonly used landmark face model assumes that all the landmarks are visible and is therefore not suitable for large poses. Second, the face appearance varies more drastically across large poses, from the frontal view to the profile view. Third, labelling landmarks in large poses is extremely challenging since the invisible landmarks have to be guessed. In this paper, we propose to tackle these three challenges in an new alignment framework termed 3D Dense Face Alignment (3DDFA), in which a dense 3D Morphable Model (3DMM) is fitted to the image via Cascaded Convolutional Neural Networks. We also utilize 3D information to synthesize face images in profile views to provide abundant samples for training. Experiments on the challenging AFLW database show that the proposed approach achieves significant improvements over the state-of-the-art methods.

Simultaneous Feature and Dictionary Learning for Image Set Based Face Recognition.

In this paper, we propose a simultaneous feature and dictionary learning (SFDL) method for image set-based face recognition, where each training and testing example contains a set of face images, which were captured from different variations of pose, illumination, expression, resolution, and motion. While a variety of feature learning and dictionary learning methods have been proposed in recent years and some of them have been successfully applied to image set-based face recognition, most of them learn features and dictionaries for facial image sets individually, which may not be powerful enough because some discriminative information for dictionary learning may be compromised in the feature learning stage if they are applied sequentially, and vice versa. To address this, we propose a SFDL method to learn discriminative features and dictionaries simultaneously from raw face pixels so that discriminative information from facial image sets can be jointly exploited by a one-stage learning procedure. To better exploit the nonlinearity of face samples from different image sets, we propose a deep SFDL (D-SFDL) method by jointly learning hierarchical non-linear transformations and class-specific dictionaries to further improve the recognition performance. Extensive experimental results on five widely used face data sets clearly shows that our SFDL and D-SFDL achieve very competitive or even better performance with the state-of-the-arts.

Face relighting using discriminative 2D spherical spaces for face recognition

As part of the face recognition task in a robust security system, we propose a novel approach for the illumination recovery of faces with cast shadows and specularities. Given a single 2D face image, we relight the face object by extracting the nine spherical harmonic bases and the face spherical illumination coefficients by using the face spherical spaces properties. First, an illumination training database is generated by computing the properties of the spherical spaces out of face albedo and normal values estimated from 2D training images. The training database is then discriminately divided into two directions in terms of the illumination quality and light direction of each image. Based on the generated multi-level illumination discriminative training space, we analyze the target face pixels and compare them with the appropriate training subspace using pre-generated tiles. When designing the framework, practical real-time processing speed and small image size were considered. In contrast to other approaches, our technique requires neither 3D face models nor restricted illumination conditions for the training process. Furthermore, the proposed approach uses one single face image to estimate the face albedo and face spherical spaces. In this work, we also provide the results of a series of experiments performed on publicly available databases to show the significant improvements in the face recognition rates.

Spherical Spaces for Illumination Invariant Face Relighting

In this paper, we propose a robust face relighting technique by using spherical space properties. The proposed method is done for reducing the illumination effects on face recognition. One single image is used as an input for recovering the face illumination and the recognition process. First, an internal training illumination database is generated by computing face albedo and face normal from 2D images under different lighting conditions. Based on the generated database, we analyze the target face pixels and compare them with the database by using pre-generated tiles. In this work, practical real time processing speed and small image size were considered when designing the framework. In contrast to other works, our technique requires no 3D face models for the training process and takes a single 2D image as an input. Experimental results on publicly available databases show that the proposed technique works well under severe illumination conditions and the method significantly improves the face recognition rates.

Reconstruction based face occlusion elimination for recognition

SRC is a recent breakthrough on occluded face recognition, but raw SRC is slow and inaccurate because both occlusion elimination and face recognition are confused and finished once in ℓ1-optimization. In this paper, a reconstruction based occlusion elimination and then recognition framework is put forward. The occlusion elimination procedure is consisted of two consecutive parts, occlusion detection and face reconstruction, where SRC is only used during occlusion detection. Specifically, downsampled SRC is utilized first of all to locate possible face occlusion at low computing complexity, and then the discovered unoccluded face pixels are imported into an overdetermined equation system to reconstruct an intact face. In this approach, since occlusion detection is independent with recognition, it could be carried out on downsampled images to effectively reduce computing complexity at ignorable accuracy loss. After occlusion elimination, all state of the art general recognition approaches, such as CRC_RLS, LDA, and LPP, could be directly utilized to improve classification accuracy. The verification experiments are conducted on both simulated and genuine occlusion.

A color-based face tracking algorithm for enhancing interaction with mobile devices

A color-based face tracking algorithm is proposed to be used as a human-computer interaction tool on mobile devices. The solution provides a natural means of interaction enabling a motion parallax effect in applications. The algorithm considers the characteristics of mobile use-constrained computational resources and varying environmental conditions. The solution is based on color comparisons and works on images gathered from the front camera of a device. In addition to color comparisons, the coherency of the facial pixels is considered in the algorithm. Several applications are also demonstrated in this work, which use the face position to determine the viewpoint in a virtual scene, or for browsing large images. The accuracy of the system is tested under different environmental conditions such as lighting and background, and the performance of the system is measured in different types of mobile devices. According to these measurements the system allows for accurate (7% RMS error) face tracking in real time (20–100 fps).

Live facial feature extraction

Precise facial feature extraction is essential to the high-level face recognition and expression analysis. This paper presents a novel method for the real-time geometric facial feature extraction from live video. In this paper, the input image is viewed as a weighted graph. The segmentation of the pixels corresponding to the edges of facial components of the mouth, eyes, brows, and nose is implemented by means of random walks on the weighted graph. The graph has an 8-connected lattice structure and the weight value associated with each edge reflects the likelihood that a random walker will cross that edge. The random walks simulate an anisotropic diffusion process that filters out the noise while preserving the facial expression pixels. The seeds for the segmentation are obtained from a color and motion detector. The segmented facial pixels are represented with linked lists in the original geometric form and grouped into different parts corresponding to facial components. For the convenience of implementing high-level vision, the geometric description of facial component pixels is further decomposed into shape and registration information. Shape is defined as the geometric information that is invariant under the registration transformation, such as translation, rotation, and isotropic scale. Statistical shape analysis is carried out to capture global facial features where the Procrustes shape distance measure is adopted. A Bayesian approach is used to incorporate high-level prior knowledge of face structure. Experimental results show that the proposed method is capable of real-time extraction of precise geometric facial features from live video. The feature extraction is robust against the illumination changes, scale variation, head rotations, and hand interference.

Efficient illumination normalization of facial images

This paper presents a simple method for normalizing non-linear illumination variations in facial images. Facial pixels are modelled by an empirical probability distribution, which is estimated by kernel density estimation. The illumination is normalized by transforming the empirical distribution function.