Local Binary Patterns On Three Orthogonal Planes Research Articles

Facial micro-expression recognition: A machine learning approach

Micro-expression recognition is a growing research area owing to its application in revealing subtle intention of humans, especially while under high stake conditions. With the rapid increase in security issues all over the world, the use of micro-expressions to understand one's state of mind has received major interest. Micro-expressions are characterized by short duration and low intensity, hence, efforts to train humans in recognizing them have resulted in very low performances. Automatic recognition of micro-expressions using machine learning techniques thus promises a more effective result and saves time and resources. In this study, we explore the use of Extreme Learning Machine (ELM) for micro-expression recognition because of its fast learning ability and higher performance when compared with other models. Support Vector Machine (SVM) is used as a baseline model and its recognition performance and its training time compared with ELM training time. Feature extraction is performed on apex micro-expression frames using Local Binary Pattern (LBP) and on micro-expression videos divided into image sequences using a spatiotemporal feature extraction technique called Local Binary Pattern on Three Orthogonal Planes (LBP-TOP). Evaluation of the two models is performed on spontaneous facial micro-expression samples acquired from Chinese Academy of Sciences (CASME II). Results obtained from the experiments show that ELM produces a higher recognition performance than SVM in terms of accuracy, precision, recall and F-score when temporal features are used. Comparison between SVM and ELM training time also shows that ELM learns faster than SVM. An average training time of 0.3405 seconds is achieved for SVM while an average training time of 0.0409 seconds is achieved for ELM for the five selected micro-expression classes. This study shows that automatic recognition of micro-expressions is produces a better result when temporal features and a machine learning algorithm with fast learning speed are used.

Spontaneous smile intensity estimation by fusing saliency maps and convolutional neural networks

Smile intensity estimation plays important roles in applications such as affective disorder prediction, life satisfaction prediction, camera technique improvement, etc. In recent studies, many researchers applied only traditional features, such as local binary pattern and local phase quantization (LPQ) to represent smile intensity. To improve the performance of spontaneous smile intensity estimation, we introduce a feature set that combines the saliency map (SM)-based handcrafted feature and non-low-level convolutional neural network (CNN) features. We took advantage of the opponent-color characteristic of SMs and the multiple convolutional level features, which were assumed to be mutually complementary. Experiments were made on the Binghamton-Pittsburgh 4D (BP4D) database and Denver Intensity of Spontaneous Facial Action (DISFA) database. We set the local binary patterns on three orthogonal planes (LBPTOP) method as a baseline, and the experimental results show that the CNN features can better estimate smile intensity. Finally, through the proposed SM-LBPTOP feature fusion with the median- and high-level CNN features, we obtained the best result (52.08% on BP4D, 70.55% on DISFA), demonstrating our hypothesis is reasonable: the SM-based handcrafted feature is a good supplement to CNNs in spontaneous smile intensity estimation.

Coupled source domain targetized with updating tag vectors for micro-expression recognition

Micro-expression has raised increasing attention for analyzing human inner emotions. However, most micro-expression recognition methods are developed with specific feature representations and extraction methods, such as local binary pattern on three orthogonal planes (LBP-TOP) and optical flow. The performance in such micro-expression recognition models is not high due to the limited training samples and the unequal size of the sample category. To improve the performance, we present a novel algorithm, named coupled source domain targetized with updating tag vectors, and we apply it to the micro-expression recognition. This method leverages rich speech data to enhance micro-expression recognition by transferring learning from the speech to the micro-expression recognition. The method highlights are: it simultaneously projects micro-expression samples and speech samples into a common space, then minimizes the reconstruction error between the speech and micro-expression samples, with an updating tag vectors added in the reconstruction process. It performs recognition by using dictionary learning together with support vector machine (SVM). Experimental results on the CASIA Chinese emotional corpus and CASME II micro-expression database demonstrate the effectiveness of our method.

Not all frames are equal: aggregating salient features for dynamic texture classification

Many recent studies have proposed methods for the classification of dynamic textures (DT). A method involving local binary patterns on three orthogonal planes (LBP-TOP) has shown promising results and generated considerable interest. However, LBP-TOP and most of its variants suffer from drawbacks caused by the accumulation process in the TOP technique. This process uses features from all frames in the DT sequence, including irrelevant frames, and thus disregards the distinct characteristics of each frame. To overcome this problem, we propose a codebook-based DT descriptor that aggregates salient features on three orthogonal planes. Given a DT sequence, only those frame features that are highly correlated with each cluster are selected and aggregated from the perspective of visual words. The proposed DT descriptor removes the feature from outlier frames that suddenly or rarely appear in a particular context, thus enhancing the emphasis of the salient features. Experimental results using public DT and dynamic scene datasets demonstrate the superiority of the proposed method over comparative approaches. The proposed method also yields outstanding results compared to the state-of-the-art DT representation.

Thermal spatio-temporal data for stress recognition

Stress is a serious concern facing our world today, motivating the development of a better objective understanding through the use of non-intrusive means for stress recognition by reducing restrictions to natural human behavior. As an initial step in computer vision-based stress detection, this paper proposes a temporal thermal spectrum (TS) and visible spectrum (VS) video database ANUStressDB - a major contribution to stress research. The database contains videos of 35 subjects watching stressed and not-stressed film clips validated by the subjects. We present the experiment and the process conducted to acquire videos of subjects' faces while they watched the films for the ANUStressDB. Further, a baseline model based on computing local binary patterns on three orthogonal planes (LBP-TOP) descriptor on VS and TS videos for stress detection is presented. A LBP-TOP-inspired descriptor was used to capture dynamic thermal patterns in histograms (HDTP) which exploited spatio-temporal characteristics in TS videos. Support vector machines were used for our stress detection model. A genetic algorithm was used to select salient facial block divisions for stress classification and to determine whether certain regions of the face of subjects showed better stress patterns. Results showed that a fusion of facial patterns from VS and TS videos produced statistically significantly better stress recognition rates than patterns from VS or TS videos used in isolation. Moreover, the genetic algorithm selection method led to statistically significantly better stress detection rates than classifiers that used all the facial block divisions. In addition, the best stress recognition rate was obtained from HDTP features fused with LBP-TOP features for TS and VS videos using a hybrid of a genetic algorithm and a support vector machine stress detection model. The model produced an accuracy of 86%.

ADHD classification by a texture analysis of anatomical brain MRI data

The ADHD-200 Global Competition provides an excellent opportunity for building diagnostic classifiers of Attention-Deficit/Hyperactivity Disorder (ADHD) based on resting-state functional MRI (rs-fMRI) and structural MRI data. Here, we introduce a simple method to classify ADHD based on morphological information without using functional data. Our test results show that the accuracy of this approach is competitive with methods based on rs-fMRI data. We used isotropic local binary patterns on three orthogonal planes (LBP-TOP) to extract features from MR brain images. Subsequently, support vector machines (SVM) were used to develop classification models based on the extracted features. In this study, a total of 436 male subjects (210 with ADHD and 226 controls) were analyzed to show the discriminative power of the method. To analyze the properties of this approach, we tested disparate LBP-TOP features from various parcellations and different image resolutions. Additionally, morphological information using a single brain tissue type (i.e., gray matter (GM), white matter (WM), and CSF) was tested. The highest accuracy we achieved was 0.6995. The LBP-TOP was found to provide better discriminative power using whole-brain data as the input. Datasets with higher resolution can train models with increased accuracy. The information from GM plays a more important role than that of other tissue types. These results and the properties of LBP-TOP suggest that most of the disparate feature distribution comes from different patterns of cortical folding. Using LBP-TOP, we provide an ADHD classification model based only on anatomical information, which is easier to obtain in the clinical environment and which is simpler to preprocess compared with rs-fMRI data.

Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions

Dynamic texture (DT) is an extension of texture to the temporal domain. Description and recognition of DTs have attracted growing attention. In this paper, a novel approach for recognizing DTs is proposed and its simplifications and extensions to facial image analysis are also considered. First, the textures are modeled with volume local binary patterns (VLBP), which are an extension of the LBP operator widely used in ordinary texture analysis, combining motion and appearance. To make the approach computationally simple and easy to extend, only the co-occurrences of the local binary patterns on three orthogonal planes (LBP-TOP) are then considered. A block-based method is also proposed to deal with specific dynamic events such as facial expressions in which local information and its spatial locations should also be taken into account. In experiments with two DT databases, DynTex and Massachusetts Institute of Technology (MIT), both the VLBP and LBP-TOP clearly outperformed the earlier approaches. The proposed block-based method was evaluated with the Cohn-Kanade facial expression database with excellent results. The advantages of our approach include local processing, robustness to monotonic gray-scale changes, and simple computation.