Graph embedding and extensions: a general framework for dimensionality reduction.

In the last decades, a large family of algorithms - supervised or unsupervised; stemming from statistic or geometry theory - have been proposed to provide different solutions to the problem of dimensionality reduction. In this paper, beyond the different motivations of these algorithms, we propose a general framework, graph embedding along with its linearization and kernelization, which in theory reveals the underlying objective shared by most previous algorithms. It presents a unified perspective to understand these algorithms; that is, each algorithm can be considered as the direct graph embedding or its linear/kernel extension of some specific graph characterizing certain statistic or geometry property of a data set. Furthermore, this framework is a general platform to develop new algorithm for dimensionality reduction. To this end, we propose a new supervised algorithm, Marginal Fisher Analysis (MFA), for dimensionality reduction by designing two graphs that characterize the intra-class compactness and inter-class separability, respectively. MFA measures the intra-class compactness with the distance between each data point and its neighboring points of the same class, and measures the inter-class separability with the class margins; thus it overcomes the limitations of traditional Linear Discriminant Analysis algorithm in terms of data distribution assumptions and available projection directions. The toy problem on artificial data and the real face recognition experiments both show the superiority of our proposed MFA in comparison to LDA.

Collaborative representation (CR), known as an effective way to address the signal representation (regression) problem, has achieved remarkable success in visual classification. According to our theoretical analysis, the subspace learning issue can also be deemed as a signal representation problem. Therefore, we extend the graph embedding (GE) framework as a CR model to improve the discriminating power of the subspace learning algorithm. The new GE framework, which is named collaborative GE (CGE) framework, enjoys many desirable properties of CR. From theoretical analysis, CGE is robust to the noise and has the same computational complexity as GE. From experimental analysis, CGE can generally enhance the subspace learning algorithms and a reasonable regularization parameter can be inferred from its intrinsic graph. Several state-of-the-art subspace learning algorithms are plugged into our framework to produce their collaborative versions. Meanwhile, by exploring the intrinsic relation among GE methods, we present a new collaborative method named collaborative class-scattering locality preserving projections (CCSLPPs). The results of extensive experiments on ORL, AR, Scene15, Caltech256, LFW-A, and OU-ISIR-A databases demonstrate that the collaborative versions consistently outperform their original algorithms with a remarkable improvement and CCSLPP gets the best performance compared with all used methods.

Domain adaptation aims to deal with learning problems in which the labeled training data and unlabeled testing data are differently distributed. Maximum mean discrepancy (MMD), as a distribution distance measure, is minimized in various domain adaptation algorithms for eliminating domain divergence. We analyze empirical MMD from the point of view of graph embedding. It is discovered from the MMD intrinsic graph that, when the empirical MMD is minimized, the compactness within each domain and each class is simultaneously reduced. Therefore, points from different classes may mutually overlap, leading to unsatisfactory classification results. To deal with this issue, we present a graph embedding framework with intrinsic and penalty graphs for MMD-based domain adaptation algorithms. In the framework, we revise the intrinsic graph of MMD-based algorithms such that the within-class scatter is minimized, and thus, the new features are discriminative. Two strategies are proposed. Based on the strategies, we instantiate the framework by exploiting four models. Each model has a penalty graph characterizing certain similarity property that should be avoided. Comprehensive experiments on visual cross-domain benchmark datasets demonstrate that the proposed models can greatly enhance the classification performance compared with the state-of-the-art methods.

Nonnegative matrix factorization (NMF) has received considerable attention in image processing, computer vision, and patter recognition. An important variant of NMF is nonnegative graph embedding (NGE), which encodes the statistical or geometric information of data in the process of matrix factorization. The NGE offers a general framework for unsupervised/supervised settings. However, NGE-like algorithms often suffer from noisy data, unreliable graphs, and noisy labels, which are commonly encountered in real-world applications. To address these issues, in this paper, we first propose a robust nonnegative graph embedding (RNGE) framework, where the joint sparsity in both graph embedding and data reconstruction endues robustness to undesirable noises. Next, we present a robust seminonnegative graph embedding (RsNGE) framework, which only constrains the coefficient matrix to be nonnegative while places no constraint on the base matrix. This extends the applicable range of RNGE to data which are not nonnegative and endows more discriminative power of the learnt base matrix. The RNGE/RsNGE provides a general formulation such that all the algorithms unified within the graph embedding framework can be easily extended to obtain their robust nonnegative/seminonnegative solutions. Further, we develop elegant multiplicative updating solutions that can solve RNGE/RsNGE efficiently and offer a rigorous convergence analysis. We conduct extensive experiments on four real-world data sets and compare the proposed RNGE/RsNGE to other representative NMF variants and data factorization methods. The experimental results demonstrate the robustness and effectiveness of the proposed approaches.

Locality preserving triplet discriminative projections for dimensionality reduction

In this paper, we propose a novel and general framework for dimensionality reduction, called Relational Fisher Analysis (RFA). Unlike traditional dimensionality reduction methods, such as linear discriminant analysis (LDA) and marginal Fisher analysis (MFA), RFA seamlessly integrates relational information among data into the representation learning framework, which in general provides strong evidence for related data to belong to the same class. To address nonlinear dimensionality reduction problems, we extend RFA to its kernel version. Furthermore, the convergence of RFA is also proved in this paper. Extensive experiments on documents understanding and recognition, face recognition and other applications from the UCI machine learning repository demonstrate the effectiveness and efficiency of RFA.

As a general framework, Laplacian embedding, based on a pairwise similarity matrix, infers low dimensional representations from high dimensional data. However, it generally suffers from three issues: 1) algorithmic performance is sensitive to the size of neighbors; 2) the algorithm encounters the well known small sample size (SSS) problem; and 3) the algorithm de-emphasizes small distance pairs. To address these issues, here we propose exponential embedding using matrix exponential and provide a general framework for dimensionality reduction. In the framework, the matrix exponential can be roughly interpreted by the random walk over the feature similarity matrix, and thus is more robust. The positive definite property of matrix exponential deals with the SSS problem. The behavior of the decay function of exponential embedding is more significant in emphasizing small distance pairs. Under this framework, we apply matrix exponential to extend many popular Laplacian embedding algorithms, e.g., locality preserving projections, unsupervised discriminant projections, and marginal fisher analysis. Experiments conducted on the synthesized data, UCI, and the Georgia Tech face database show that the proposed new framework can well address the issues mentioned above.

On connections between Rényi entropy Principal Component Analysis, kernel learning and graph embedding

Over the past few decades, a large family of algorithms-supervised or unsupervised; stemming from statistics or geometry theory-has been designed to provide different solutions to the problem of dimensionality reduction. Despite the different motivations of these algorithms, we present in this paper a general formulation known as graph embedding to unify them within a common framework. In graph embedding, each algorithm can be considered as the direct graph embedding or its linear/kernel/tensor extension of a specific intrinsic graph that describes certain desired statistical or geometric properties of a data set, with constraints from scale normalization or a penalty graph that characterizes a statistical or geometric property that should be avoided. Furthermore, the graph embedding framework can be used as a general platform for developing new dimensionality reduction algorithms. By utilizing this framework as a tool, we propose a new supervised dimensionality reduction algorithm called Marginal Fisher Analysis in which the intrinsic graph characterizes the intraclass compactness and connects each data point with its neighboring points of the same class, while the penalty graph connects the marginal points and characterizes the interclass separability. We show that MFA effectively overcomes the limitations of the traditional Linear Discriminant Analysis algorithm due to data distribution assumptions and available projection directions. Real face recognition experiments show the superiority of our proposed MFA in comparison to LDA, also for corresponding kernel and tensor extensions.

Graph learning framework has become a popular method of dimensional reduction. However, the traditional graph construction heavily relies on the selection of parameters, resulting in unstable performance in real-world face recognition applications. To address this, a label information-based weighted regularized sparsity preserving embedding for face recognition is proposed in this paper. Different from the existing L1-graph, we adaptively construct both intrinsic graph and penalty graph with label information-based L1-graph in the graph embedding framework. In order to preserve the local structure, Gaussian kernel distances between the samples are used as weight matrix to weight graph. In addition, the problem of irreversible matrix is alleviated by regularization instead of PCA that loses some discrimination information. At last, an objective function combining globality and locality is created to reduce dimensionality. Meanwhile, Schmidt orthogonalization is used to obtain the orthogonal basis vectors. The experimental results on public face database illustrate that the proposed algorithm has high recognition rate.

In this paper, we propose a novel graph embedding method for the problem of lipreading. To characterize the temporal connections among video frames of the same utterance, a new distance metric is defined on a pair of frames and graphs are constructed to represent the video dynamics based on the distances between frames. Audio information is used to assist in calculating such distances. For each utterance, a subspace of the visual feature space is learned from a well-defined intrinsic and penalty graph within a graph-embedding framework. Video dynamics are found to be well preserved along some dimensions of the subspace. Discriminatory cues are then decoded from curves of the projected visual features to classify different utterances.

Non-negative data factorization has been widely used recently. However, existing techniques, such as Non-negative Graph Embedding (NGE), often suffer from noisy data, unreliable graphs, and noisy labels, which are commonly encountered in real-world applications. To address these issues, in this paper, we propose a Robust Non-negative Graph Embedding (RNGE) framework. The joint sparsity in both graph embedding and reconstruction endues the robustness of RNGE. We develop an elegant multiplicative updating solution that can solve RNGE efficiently and prove the convergence rigourously. RNGE is robust to unreliable graphs, as well as both sample and label noises in training data. Moreover, RNGE provides a general formulation such that all the algorithms unified with the graph embedding framework can be easily extended to obtain their robust non-negative solutions. We conduct extensive experiments on four real-world datasets and compared the proposed RNGE to NGE and other representative non-negative data factorization and subspace learning methods. The experimental results demonstrate the effectiveness and robustness of RNGE.

Extreme learning machine is characterized by less training parameters, fast training speed, and strong generalization ability. It has been applied to obtain feature representations from the complex data in the tasks of data clustering or classification. In this paper, a graph embedding-based denoising extreme learning machine autoencoder (GDELM-AE) is proposed for capturing the structure of the inputs. Specifically, in GDELM-AE, a graph embedding framework that contains an intrinsic graph and a penalty graph constructed by local Fisher discrimination analysis is integrated into the autoencoder. So, it can exploit both local structure and global structure information in extreme learning machine (ELM) spaces. Further, we propose a stacked graph embedded denoising (SGD)-ELM by stacking several GDELM-AEs. The experimental results on several benchmarks validate that GDELM-AE can obtain efficient and robust feature representation of original data; moreover, the stacked GDELM-AE can obtain high-level and noise-robust representations. The comparative results with the state-of-the-art algorithms indicate that the proposed algorithm can obtain better accuracy as well as faster training speed.

Subclass Graph Embedding and a Marginal Fisher Analysis paradigm

Dimensionality reduction plays an important role in many machine learning and pattern recognition applications. Linear discriminant analysis (LDA) is the most popular supervised dimensionality reduction technique which searches for the projection matrix that makes the data points of different classes to be far from each other while requiring data points of the same class to be close to each other. In this paper, trace ratio LDA is combined with K-means clustering into a unified framework, in which K-means clustering is employed to generate class labels for unlabeled data and LDA is used to investigate low-dimensional representation of data. Therefore, by combining the subspace clustering with dimensionality reduction together, the optimal subspace can be obtained. Differing from other existing dimensionality reduction methods, our novel framework is suitable for different scenarios: supervised, semi-supervised, and unsupervised dimensionality reduction cases. Experimental results on benchmark datasets validate the effectiveness and superiority of our algorithm compared with other relevant techniques.