Local Binary Patterns From Three Orthogonal Planes Research Articles

Blind Quality Evaluator of Light Field Images by Group-Based Representations and Multiple Plane-Oriented Perceptual Characteristics

Due to the emergency of multi-view cameras and commercial Light Field (LF) cameras, the demand of high-performance LF quality evaluator is of great significance for guiding LF acquisition, processing and application and further promoting the visual perceived quality of LF visualizations. However, LF Images (LFIs), as high-dimensional data, suffer from various quality degradations not only in the spatial domain but also in the angular domain. Therefore, it is of great challenge to predict LF quality accurately. An effective LF evaluator should be able to represent these heterogeneous artifacts. In this paper, we provide a novel No-Reference LF Quality Assessment Evaluator (NR LF-QAE) to tackle this problem. Firstly, to measure angular consistency among viewports, we utilize group-based representations to character information similarity of aligned view stacks. Secondly, to better describe the texture information of LFIs, unifying spatial-angular texture statistic measurement is performed via Local Binary Patterns from Three Orthogonal Planes (LBP-TOP). Thirdly, we design 3D Log-Gabor filters to extract LF global structure information in Sub-Aperture Images (SAIs) as spatial feature characterizations and 2D Log-Gabor filters are adopted to characterize ray direction/depth information in Epipolar Plane Images (EPIs) as angular feature characterizations. By comprehensive LF information analyses in angular consistency and spatial-angular feature extraction with texture and structure descriptors, experimental results demonstrate the superiority of the proposed NR LF-QAE over the state-of-the-art comparative models in predicting the quality of LFIs on three available benchmark databases. The code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zerosola/NR-LF-QAE</uri> .

Research on Multi-Scale Feature Extraction and Working Condition Classification Algorithm of Lead-Zinc Ore Flotation Foam

To address the problems of difficult online monitoring, low recognition efficiency and the subjectivity of work condition identification in mineral flotation processes, a foam flotation performance state recognition method is developed to improve the issues mentioned above. This method combines multi-dimensional CNN (convolutional neural networks) characteristics and improved LBP (local binary patterns) characteristics. We have divided the foam flotation conditions into six categories. First, the multi-directional and multi-scale selectivity and anisotropy of nonsubsampled shearlet transform (NSST) are used to decompose the flotation foam images at multiple frequency scales, and a multi-channel CNN network is designed to extract static features from the images at different frequencies. Then, the flotation video image sequences are rotated and dynamic features are extracted by the LBP-TOP (local binary patterns from three orthogonal planes), and the CNN-extracted static picture features are fused with the LBP dynamic video features. Finally, classification decisions are made by a PSO-RVFLNs (particle swarm optimization-random vector functional link networks) algorithm to accurately identify the foam flotation performance states. Experimental results show that the detection accuracy of the new method is significantly improved by 4.97% and 6.55%, respectively, compared to the single CNN algorithm and the traditional LBP algorithm, respectively. The accuracy of flotation performance state classification was as high as 95.17%, and the method reduced manual intervention, thus improving production efficiency.

MFED: A Database for Masked Facial Expression

Humans sometimes intentionally altered facial expression to mask the genuine emotions for certain purposes. Such masked facial expressions have seldom been recorded. In this study, we constructed a facial expression database in which participants were required to watch emotional video clips and make specific facial expressions which were consistent or inconsistent with the emotional context, and named it Masked Facial Expression Database (MFED). Twenty-four participants were recruited and trained to be familiar with six basic facial expressions. They were instructed to display six expressions under six basic emotional context which resulted in 36 combinations. Seven hundred and eighty three emotional video clips including 36 (6 by 6) categories of facial expressions were collected. We also utilized Facial Action Coding System (FACS) to encode these facial expressions. In addition, local binary patterns from three orthogonal planes (LBP-TOP) and support vector machine (SVM) were employed for feature extraction and subsequent leave-one-subject-out cross validation, respectively. Best performance of classification is 78.80% for masked and non-masked, 26.83% for experienced emotion evoked by video clips, and 47.77% for required (presented by instructions) expressions. For improving the performance of experienced emotion and required expression recognition, a well-designed convolutional neural network (CNN) was also utilized to capture high-level representations of facial expression images. Our clear-cut results demonstrated that masked and non-masked expressions could be moderately discriminated in this database. Experienced emotion was difficult to identify partly because it was definitely covered up by the masked expression.

No-Reference Quality Evaluation of Stereoscopic Video Based on Spatio-Temporal Texture

Due to the wide application of stereoscopic display technology, stereoscopic video quality assessment (SVQA) is facing great challenges, but worthwhile. Stereoscopic videos contain a great deal of information, which involves not only the spatial domain but also the spatio-temporal domain. Motion in stereoscopic video plays a critical role in quality perception, while the existing SVQA methods rarely refer to motion factors, and the performance of these methods is restrained. In this article, a novel SVQA based on motion perception is introduced and its performance is superior to that of existing excellent methods. Particularly, to appropriately reduce the amount of data processing, we extract the key-frame sequences according to the influence of movement intensity on binocular visual quality perception. The binocular summation and difference operations are implemented on extracted sequences, and then spatial texture and spatio-temporal texture statistic measurement are extracted simultaneously with local binary patterns from three orthogonal planes (LBP-TOP). Experiments are implemented on two publicly available databases and the results demonstrate the effectiveness and robustness of our algorithm for various categories of distortion stereoscopic video pairs.

Evaluating dynamic texture descriptors to recognize human iris in video image sequence

In the last decades, iris features have been widely used in biometric systems. Because iris features are virtually unique for each person, their usage is highly reliable. However, biometric systems based on iris features are not completely fraud-resistant, as most systems use static images and do not distinguish between a live iris and a photograph. The iris structure and texture change with light variations, and traditional techniques for iris recognition always identify the iris texture in a controlled environment. However, in uncontrolled environments, live irises are recognized by their dynamic response to light: If the light changes, the pupils dilate or contract, and their texture dynamically changes. If a biometric system can identify people during the constriction or dilation time interval, that system will be more fraud-resistant. This paper proposes a new methodology to evaluate the “dynamic texture” from iris image sequences (motion analysis) and measure the discriminant power of these features for biometric system applications. We propose two new dynamic descriptors—dynamic local mapped pattern and dynamic sampled local mapped pattern—which are extensions of the local mapped pattern previously published for texture classification. We applied our proposed dynamic texture descriptors in a sequence of iris images segmented from video under light variation. Then, we compared our results with the well-known dynamic texture descriptor local binary pattern from three orthogonal planes (LBP-TOP). We used statistical measures to evaluate the performance of both descriptors and concluded that our methodology performed better than the LBP-TOP. Moreover, our descriptors can extract dynamic textures faster than the LBP-TOP.

An Improved Micro-Expression Recognition Method Based on Necessary Morphological Patches

Micro-expression is a spontaneous emotional representation that is not controlled by logic. A micro-expression is both transitory (short duration) and subtle (small intensity), so it is difficult to detect in people. Micro-expression detection is widely used in the fields of psychological analysis, criminal justice and human-computer interaction. Additionally, like traditional facial expressions, micro-expressions also have local muscle movement. Psychologists have shown micro-expressions have necessary morphological patches (NMPs), which are triggered by emotion. Furthermore, the objective of this paper is to sort and filter these NMPs and extract features from NMPs to train classifiers to recognize micro-expressions. Firstly, we use the optical flow method to compare the on-set frame and the apex frame of the micro-expression sequences. By doing this, we could find facial active patches. Secondly, to find the NMPs of micro-expressions, this study calculates the local binary pattern from three orthogonal planes (LBP-TOP) operators and cascades them with optical flow histograms to form the fusion features of the active patches. Finally, a random forest feature selection (RFFS) algorithm is used to identify the NMPs and to characterize them via support vector machine (SVM) classifier. We evaluated the proposed method on two popular publicly available databases: CASME II and SMIC. Results show that NMPs are statistically determined and contribute to significant discriminant ability instead of holistic utilization of all facial regions.

A distortion-agnostic video quality metric based on multi-scale spatio-temporal structural information

Considering the high sensitivity of human visual perception towards the structural information, a general purpose no-reference video quality assessment (NR-VQA) metric based on local binary patterns (LBP) is proposed. In the proposed metric, 2D LBP method is extended to third dimension using LBP-TOP (LBP from Three Orthogonal Planes) for extracting the spatio-temporal structural information. Further, spatio-temporal structural information is evaluated for multi-resolution local pixel windows to incorporate the human vision system (HVS) characteristic of multi-scale perception. Since LBP only encodes the sign information of local pixel differences, and ignores the degree of variations, a second set of feature based on the magnitude information of local pixel differences is extracted. The magnitude information based LBP-TOP feature set is accumulated on the sign information based LBP-TOP feature set such that the resultant feature set has more spatio-temporal structural information with same feature vector length. To emulate the HVS mechanism of quality feature mapping to a scalar quantity, a simple two-layer feed-forward artificial neural network (ANN) of single hidden layer with four neurons is proposed. The performance of the proposed metric is validated on LIVE, CSIQ, and IVP video quality datasets and consistent performance is observed. Further, the performance of proposed metric is compared with the existing state-of-the-art NR-VQA and other IQA/VQA metrics and have exhibited better performance than the others. The proposed metric is highly consistent in performance with low computational complexity and thus suitable for real-time applications.

Objective Classes for Micro-Facial Expression Recognition

Micro-expressions are brief spontaneous facial expressions that appear on a face when a person conceals an emotion, making them different to normal facial expressions in subtlety and duration. Currently, emotion classes within the CASME II dataset (Chinese Academy of Sciences Micro-expression II) are based on Action Units and self-reports, creating conflicts during machine learning training. We will show that classifying expressions using Action Units, instead of predicted emotion, removes the potential bias of human reporting. The proposed classes are tested using LBP-TOP (Local Binary Patterns from Three Orthogonal Planes), HOOF (Histograms of Oriented Optical Flow) and HOG 3D (3D Histogram of Oriented Gradient) feature descriptors. The experiments are evaluated on two benchmark FACS (Facial Action Coding System) coded datasets: CASME II and SAMM (A Spontaneous Micro-Facial Movement). The best result achieves 86.35% accuracy when classifying the proposed 5 classes on CASME II using HOG 3D, outperforming the result of the state-of-the-art 5-class emotional-based classification in CASME II. Results indicate that classification based on Action Units provides an objective method to improve micro-expression recognition.

NIRExpNet: Three-Stream 3D Convolutional Neural Network for Near Infrared Facial Expression Recognition

Facial expression recognition (FER) under active near-infrared (NIR) illumination has the advantages of illumination invariance. In this paper, we propose a three-stream 3D convolutional neural network, named as NIRExpNet for NIR FER. The 3D structure of NIRExpNet makes it possible to extract automatically, not just spatial features, but also, temporal features. The design of multiple streams of the NIRExpNet enables it to fuse local and global facial expression features. To avoid over-fitting, the NIRExpNet has a moderate size to suit the Oulu-CASIA NIR facial expression database that is a medium-size database. Experimental results show that the proposed NIRExpNet outperforms some previous state-of-art methods, such as Histogram of Oriented Gradient to 3D (HOG 3D), Local binary patterns from three orthogonal planes (LBP-TOP), deep temporal appearance-geometry network (DTAGN), and adapt 3D Convolutional Neural Networks (3D CNN DAP).

Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns

Spontaneous facial micro-expression analysis has become an active task for recognizing suppressed and involuntary facial expressions shown on the face of humans. Recently, Local Binary Pattern from Three Orthogonal Planes (LBP-TOP) has been employed for micro-expression analysis. However, LBP-TOP suffers from two critical problems, causing a decrease in the performance of micro-expression analysis. It generally extracts appearance and motion features from the sign-based difference between two pixels but not yet considers other useful information. As well, LBP-TOP commonly uses classical pattern types which may be not optimal for local structure in some applications. This paper proposes SpatioTemporal Completed Local Quantization Patterns (STCLQP) for facial micro-expression analysis. Firstly, STCLQP extracts three interesting information containing sign, magnitude and orientation components. Secondly, an efficient vector quantization and codebook selection are developed for each component in appearance and temporal domains to learn compact and discriminative codebooks for generalizing classical pattern types. Finally, based on discriminative codebooks, spatiotemporal features of sign, magnitude and orientation components are extracted and fused. Experiments are conducted on three publicly available facial micro-expression databases. Some interesting findings about the neighboring patterns and the component analysis are concluded. Comparing with the state of the art, experimental results demonstrate that STCLQP achieves a substantial improvement for analyzing facial micro-expressions.

Rotation-Invariant Image and Video Description With Local Binary Pattern Features

In this paper, we propose a novel approach to compute rotation-invariant features from histograms of local noninvariant patterns. We apply this approach to both static and dynamic local binary pattern (LBP) descriptors. For static-texture description, we present LBP histogram Fourier (LBP-HF) features, and for dynamic-texture recognition, we present two rotation-invariant descriptors computed from the LBPs from three orthogonal planes (LBP-TOP) features in the spatiotemporal domain. LBP-HF is a novel rotation-invariant image descriptor computed from discrete Fourier transforms of LBP histograms. The approach can be also generalized to embed any uniform features into this framework, and combining the supplementary information, e.g., sign and magnitude components of the LBP, together can improve the description ability. Moreover, two variants of rotation-invariant descriptors are proposed to the LBP-TOP, which is an effective descriptor for dynamic-texture recognition, as shown by its recent success in different application problems, but it is not rotation invariant. In the experiments, it is shown that the LBP-HF and its extensions outperform noninvariant and earlier versions of the rotation-invariant LBP in the rotation-invariant texture classification. In experiments on two dynamic-texture databases with rotations or view variations, the proposed video features can effectively deal with rotation variations of dynamic textures (DTs). They also are robust with respect to changes in viewpoint, outperforming recent methods proposed for view-invariant recognition of DTs.

Facial expression recognition from near-infrared videos

Facial expression recognition is to determine the emotional state of the face regardless of its identity. Most of the existing datasets for facial expressions are captured in a visible light spectrum. However, the visible light (VIS) can change with time and location, causing significant variations in appearance and texture. In this paper, we present a novel research on a dynamic facial expression recognition, using near-infrared (NIR) video sequences and LBP-TOP (Local binary patterns from three orthogonal planes) feature descriptors. NIR imaging combined with LBP-TOP features provide an illumination invariant description of face video sequences. Appearance and motion features in slices are used for expression classification, and for this, discriminative weights are learned from training examples. Furthermore, component-based facial features are presented to combine geometric and appearance information, providing an effective way for representing the facial expressions. Experimental results of facial expression recognition using a novel Oulu-CASIA NIR&VIS facial expression database, a support vector machine and sparse representation classifiers show good and robust results against illumination variations. This provides a baseline for future research on NIR-based facial expression recognition.

Local Ternary Patterns from Three Orthogonal Planes for human action classification

Human action classification is a new field of study with applications ranging from automatically labeling video segments to recognition of suspicious behavior in video surveillance cameras. In this paper we present some variants of Local Binary Patterns from Three Orthogonal Planes (LBP-TOP), which is considered one of the state of the art texture descriptors for human action classification. The standard LBP-TOP operator is defined as a gray-scale invariant texture measure, derived from the standard Local Binary Patterns (LBP). It is obtained by calculating the LBP features from the xt and yt planes of a space-time volume. Our LBP-TOP variants combine the idea of LBP-TOP with Local Ternary Patterns (LTP). The encoding of LTP is used for the evaluation of the local gray-scale difference in the different planes of the space-time volume. Different histograms are concatenated to form the feature vector and a random subspace of linear support vector machines is used for classifying action using the Weizmann database of video images. To the best of our knowledge, our method offers the first set of classification experiments to obtain 100% accuracy using the 10-class Weizmann dataset.