Penalty Graph Research Articles

Locality preserving triplet discriminative projections for dimensionality reduction

The graph embedding framework is widely used for developing dimensionality-reduction algorithms. Many such supervised algorithms construct an intrinsic graph to compact intra-class samples or describe their local structures and build a penalty graph to increase the separability of inter-class samples. However, in the available intrinsic graph construction, manually selecting a set of appropriate neighbors associated with each sample is challenging. In addition, the construction of a penalty graph may seriously impair the intrinsic structures of the samples. Thus, in an attempt to address these issues, this study proposes an algorithm referred to as locality-preserving triplet discriminative projections. The proposed algorithm comprises locality-preserving and discriminative graphs. First, a weighted least-square function is used to calculate the edge weights of the locality-preserving graph. An improvement of the locality-preserving graph constructed in this study is that suitable neighbors are soft-selected. Meanwhile, the triplets of the samples are exploited for discriminative graph construction. Through the separation of the marginal samples and using less focus on samples with good discriminability, this graph enhances the discriminability with less damage to the intrinsic structure. In addition, to further improve the performance of the discriminative graph, a class-relevant margin is added to separate similar classes. The experimental results on four public datasets show that the proposed algorithm outperforms several other dimensionality reduction methods.

Manifold-Based Multi-Deep Belief Network for Feature Extraction of Hyperspectral Image

Deep belief networks (DBNs) have been widely applied in hyperspectral imagery (HSI) processing. However, the original DBN model fails to explore the prior knowledge of training samples which limits the discriminant capability of extracted features for classification. In this paper, we proposed a new deep learning method, termed manifold-based multi-DBN (MMDBN), to obtain deep manifold features of HSI. MMDBN designed a hierarchical initialization method that initializes the network by local geometric structure hidden in data. On this basis, a multi-DBN structure is built to learn deep features in each land-cover class, and it was used as the front-end of the whole model. Then, a discrimination manifold layer is developed to improve the discriminability of extracted deep features. To discover the manifold structure contained in HSI, an intrinsic graph and a penalty graph are constructed in this layer by using label information of training samples. After that, the deep manifold features can be obtained for classification. MMDBN not only effectively extracts the deep features from each class in HSI, but also maximizes the margins between different manifolds in low-dimensional embedding space. Experimental results on Indian Pines, Salinas, and Botswana datasets reach 78.25%, 90.48%, and 97.35% indicating that MMDBN possesses better classification performance by comparing with some state-of-the-art methods.

Large Margin Graph Embedding-Based Discriminant Dimensionality Reduction

Discriminant graph embedding-based dimensionality reduction methods have attracted more and more attention over the past few decades. These methods construct an intrinsic graph and penalty graph to preserve the intrinsic geometry structures of intraclass samples and separate the interclass samples. However, the marginal samples cannot be accurately characterized only by penalty graphs since they treat every sample equally. In practice, these marginal samples often influence the classification performance, which needs to be specially tackled. In this study, the near neighbors’ hypothesis margin of marginal samples has been further maximized to separate the interclass samples and improve the discriminant ability by integrating intrinsic graph and penalty graph. A novel discriminant dimensionality reduction named LMGE-DDR has been proposed. Several experiments on public datasets have been conducted to verify the effectiveness of the proposed LMGE-DDR such as ORL, Yale, UMIST, FERET, CMIU-PIE09, and AR. LMGE-DDR performs better than other compared methods, and the corresponding standard deviation of LMGE-DDR is smaller than others. This demonstrates that the evaluation method verifies the effectiveness of the introduced method.

Discriminant analysis based on reliability of local neighborhood

To obtain a compact and effective low-dimensional representation, recently, most existing discriminant manifold learning methods have integrated manifold learning into discriminant analysis (DA) for extracting the intrinsic structure of data. These methods learn two kinds of adjacency graphs, such as intrinsic graph and penalty graph, to characterize the similarity between samples from intraclass and the pseudo similarity of interclass. However, they treat every sample equally, which results in the following defects: (1) These methods cannot accurately characterize the marginal region among different classes only through penalty graphs. (2) They can not identify the noisy and outlier samples which reduce the robustness of these methods. To address these problems, we introduce an adaptive adjacency factor to perform the discriminative based reliability analysis for each sample. By integrating the adjacency factor into discriminant manifold learning methods, we propose a novel method for DA namely discriminant analysis based on reliability of local neighborhood (DA-RoLN). We mainly have three contributions in this paper: (1) By the introduction of adjacency factor, sample points can be divided into three parts: intraclass samples, marginal samples, and outliers. Therefore, DA-RoLN emphasizes the effect of valid samples and filters the influence of outliers. (2) We adaptively calculate the adjacency factor in low-dimensional space, thus, the margin between different classes in low-dimensional space is emphasized. (3) An iterative algorithm is developed to solve the objective function of DA-RoLN, and it is easy to solve with a low computational cost. Extensive experimental results show the effectiveness of DA-RoLN.

A graph-based approach to detect unexplained sequences in a log

In this paper we challenge the issue of detecting anomalous events in computer systems log files, through a novel graph mining approach. The basic idea is to model log temporal sequences as a particular graph and event detection as a particular path finding problem. Thus, anomalous sequences correspond to log parts that can not be “explained” by any path in the graph. We propose a novel Iterative Partitioning Log Mining technique to parse any kind of logs and to model their temporal sequence as a probabilistic penalty graph. The approach has been implemented in a framework supporting both real time and batch processing realized on the top of the Apache Spark analytics engine for large-scale data processing. Experimental results show the advantages of the proposed framework in terms of effectiveness for different system configurations.

Generalized Embedding Regression: A Framework for Supervised Feature Extraction.

Sparse discriminative projection learning has attracted much attention due to its good performance in recognition tasks. In this article, a framework called generalized embedding regression (GER) is proposed, which can simultaneously perform low-dimensional embedding and sparse projection learning in a joint objective function with a generalized orthogonal constraint. Moreover, the label information is integrated into the model to preserve the global structure of data, and a rank constraint is imposed on the regression matrix to explore the underlying correlation structure of classes. Theoretical analysis shows that GER can obtain the same or approximate solution as some related methods with special settings. By utilizing this framework as a general platform, we design a novel supervised feature extraction approach called jointly sparse embedding regression (JSER). In JSER, we construct an intrinsic graph to characterize the intraclass similarity and a penalty graph to indicate the interclass separability. Then, the penalty graph Laplacian is used as the constraint matrix in the generalized orthogonal constraint to deal with interclass marginal points. Moreover, the L2,1 -norm is imposed on the regression terms for robustness to outliers and data's variations and the regularization term for jointly sparse projection learning, leading to interesting semantic interpretability. An effective iterative algorithm is elaborately designed to solve the optimization problem of JSER. Theoretically, we prove that the subproblem of JSER is essentially an unbalanced Procrustes problem and can be solved iteratively. The convergence of the designed algorithm is also proved. Experimental results on six well-known data sets indicate the competitive performance and latent properties of JSER.

A deep feature manifold embedding method for hyperspectral image classification

ABSTRACT In this letter, we proposed a novel deep feature manifold embedding method to improve feature extraction ability of traditional deep learning methods. This method first obtains deep features of hyperspectral image (HSI) from a trained autoencoder. Then, an intrinsic graph and a penalty graph are constructed to discover the discriminant manifold structure of deep features. Finally, the deep features are mapped into a low-dimensional embedding space, in which samples in intraclass manifold are compacted and samples from interclass manifolds are separated. Experiments on Pavia University, Indian Pines and Urban datasets demonstrate that the proposed method effectively improves the classification performance of HSI compared with other state-of-the-art approaches.

Discriminative and Uncorrelated Feature Selection With Constrained Spectral Analysis in Unsupervised Learning.

The existing unsupervised feature extraction methods frequently explore low-redundant features by an uncorrelated constraint. However, the constrained models might incur trivial solutions, due to the singularity of scatter matrix triggered by high-dimensional data. In this paper, we propose a regularized regression model with a generalized uncorrelated constraint for feature selection, which leads to three merits: 1) exploring the low-redundant and discriminative features; 2) avoiding the trivial solutions and 3) simplifying the optimization. Besides that, the local cluster structure is achieved via a novel constrained spectral analysis for the unsupervised learning, where Must-Links and Cannot-Links are transformed into a intrinsic graph and a penalty graph respectively, rather than incorporated into a mixed affinity graph. Accordingly, a discriminative and uncorrelated feature selection with constrained spectral analysis (DUCFS) is proposed with adopting σ -norm regularization for interpolating between F-norm and l2,1 -norm. Due to the flexible gradient and global differentiability, our model converges fast. Extensive experiments on benchmark datasets among several state-of-the-art approaches verify the effectiveness of the proposed method.

A Graph Embedding Framework for Maximum Mean Discrepancy-Based Domain Adaptation Algorithms.

Domain adaptation aims to deal with learning problems in which the labeled training data and unlabeled testing data are differently distributed. Maximum mean discrepancy (MMD), as a distribution distance measure, is minimized in various domain adaptation algorithms for eliminating domain divergence. We analyze empirical MMD from the point of view of graph embedding. It is discovered from the MMD intrinsic graph that, when the empirical MMD is minimized, the compactness within each domain and each class is simultaneously reduced. Therefore, points from different classes may mutually overlap, leading to unsatisfactory classification results. To deal with this issue, we present a graph embedding framework with intrinsic and penalty graphs for MMD-based domain adaptation algorithms. In the framework, we revise the intrinsic graph of MMD-based algorithms such that the within-class scatter is minimized, and thus, the new features are discriminative. Two strategies are proposed. Based on the strategies, we instantiate the framework by exploiting four models. Each model has a penalty graph characterizing certain similarity property that should be avoided. Comprehensive experiments on visual cross-domain benchmark datasets demonstrate that the proposed models can greatly enhance the classification performance compared with the state-of-the-art methods.

Stacked Denoising Extreme Learning Machine Autoencoder Based on Graph Embedding for Feature Representation

Extreme learning machine is characterized by less training parameters, fast training speed, and strong generalization ability. It has been applied to obtain feature representations from the complex data in the tasks of data clustering or classification. In this paper, a graph embedding-based denoising extreme learning machine autoencoder (GDELM-AE) is proposed for capturing the structure of the inputs. Specifically, in GDELM-AE, a graph embedding framework that contains an intrinsic graph and a penalty graph constructed by local Fisher discrimination analysis is integrated into the autoencoder. So, it can exploit both local structure and global structure information in extreme learning machine (ELM) spaces. Further, we propose a stacked graph embedded denoising (SGD)-ELM by stacking several GDELM-AEs. The experimental results on several benchmarks validate that GDELM-AE can obtain efficient and robust feature representation of original data; moreover, the stacked GDELM-AE can obtain high-level and noise-robust representations. The comparative results with the state-of-the-art algorithms indicate that the proposed algorithm can obtain better accuracy as well as faster training speed.

Discriminative and Orthogonal Subspace Constraints-Based Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) is one widely used feature extraction technology in the tasks of image clustering and image classification. For the former task, various unsupervised NMF methods based on the data distribution structure information have been proposed. While for the latter task, the label information of the dataset is one very important guiding. However, most previous proposed supervised NMF methods emphasis on imposing the discriminant constraints on the coefficient matrix. When dealing with new coming samples, the transpose or the pseudoinverse of the basis matrix is used to project these samples to the low dimension space. In this way, the label influence to the basis matrix is indirect. Although, there are also some methods trying to constrain the basis matrix in NMF framework, either they only restrict within-class samples or impose improper constraint on the basis matrix. To address these problems, in this article a novel NMF framework named discriminative and orthogonal subspace constraints-based nonnegative matrix factorization (DOSNMF) is proposed. In DOSNMF, the discriminative constraints are imposed on the projected subspace instead of the directly learned representation. In this manner, the discriminative information is directly connected with the projected subspace. At the same time, an orthogonal term is incorporated in DOSNMF to adjust the orthogonality of the learned basis matrix, which can ensure the orthogonality of the learned subspace and improve the sparseness of the basis matrix at the same time. This framework can be implemented in two ways. The first way is based on the manifold learning theory. In this way, two graphs, i.e., the intrinsic graph and the penalty graph, are constructed to capture the intra-class structure and the inter-class distinctness. With this design, both the manifold structure information and the discriminative information of the dataset are utilized. For convenience, we name this method as the name of the framework, i.e., DOSNMF. The second way is based on the Fisher’s criterion, we name it Fisher’s criterion-based DOSNMF (FDOSNMF). The objective functions of DOSNMF and FDOSNMF can be easily optimized using multiplicative update (MU) rules. The new methods are tested on five datasets and compared with several supervised and unsupervised variants of NMF. The experimental results reveal the effectiveness of the proposed methods.

Semisupervised Learning With Parameter-Free Similarity of Label and Side Information.

As for semisupervised learning, both label information and side information serve as pivotal indicators for the classification. Nonetheless, most of related research works utilize either label information or side information instead of exploiting both of them simultaneously. To address the referred defect, we propose a graph-based semisupervised learning (GSL) problem according to both given label information and side information. To solve the GSL problem efficiently, two novel self-weighted strategies are proposed based on solving associated equivalent counterparts of a GSL problem, which can be widely applied to a spectrum of biobjective optimizations. Different from a conventional technique to amalgamate must-link and cannot-link into a single similarity for convenient optimization, we derive a new parameter-free similarity, upon which intrinsic graph and penalty graph can be separately developed. Consequently, a novel semisupervised classification algorithm can be summarized correspondingly with a theoretical analysis.

Discriminant sparse and collaborative preserving embedding for bearing fault diagnosis

Abstract Background noise and small sample size would usually add to the difficulty of extracting most effective information for bearing fault diagnosis in rotating electrical machines. To address this issue, a novel supervised dimensionality reduction algorithm referred to as discriminant sparse and collaborative preserving embedding (DSCPE) is proposed in this paper for bearing defect classification, which utilizes collaborative representation (CR) for an intrinsic graph and sparse representation (SR) for a penalty graph. In the intrinsic graph, CR helps to involve more bases of the dictionary composed by the same labeled samples and generate less sparse solutions; meanwhile in the penalty graph, SR would avoid the mix-class interference when using the dictionary constituted by different labeled samples. DSCPE aims to seek the optimal projection directions that could minimize the intraclass compactness and maximize the interclass separability. After dimension reduction and data projection by DSCPE, the 1-Nearest Neighbor method is applied to classify the bearing defects. The experimental results demonstrate that DSCPE possesses more effective and robust classification performance than other compared algorithms when dealing with small-sample-sized problem and interfered vibration signals.

Graph-regularized multi-view semantic subspace learning

Many real-world datasets are represented by multiple features or modalities which often provide compatible and complementary information to each other. In order to obtain a good data representation that synthesizes multiple features, researchers have proposed different multi-view subspace learning algorithms. Although label information has been exploited for guiding multi-view subspace learning, previous approaches did not well capture the underlying semantic structure in data. In this paper, we propose a new multi-view subspace learning algorithm called multi-view semantic learning (MvSL). MvSL learns a nonnegative latent space and tries to capture the semantic structure of data by a novel graph embedding framework, where an affinity graph characterizing intra-class compactness and a penalty graph characterizing inter-class separability are generally defined. The intuition is to let intra-class items be near each other while keeping inter-class items away from each other in the learned common subspace across multiple views. We explore three specific definitions of the graphs and compare them analytically and empirically. To properly assess nearest neighbors in the multi-view context, we develop a multiple kernel learning method for obtaining an optimal kernel combination from multiple features. In addition, we encourage each latent dimension to be associated with a subset of views via sparseness constraints. In this way, MvSL is able to capture flexible conceptual patterns hidden in multi-view features. Experiments on three real-world datasets demonstrate the effectiveness of MvSL.

Graph embedding and extensions: a general framework for dimensionality reduction.

Over the past few decades, a large family of algorithms - supervised or unsupervised; stemming from statistics or geometry theory - has been designed to provide different solutions to the problem of dimensionality reduction. Despite the different motivations of these algorithms, we present in this paper a general formulation known as graph embedding to unify them within a common framework. In graph embedding, each algorithm can be considered as the direct graph embedding or its linear/kernel/tensor extension of a specific intrinsic graph that describes certain desired statistical or geometric properties of a data set, with constraints from scale normalization or a penalty graph that characterizes a statistical or geometric property that should be avoided. Furthermore, the graph embedding framework can be used as a general platform for developing new dimensionality reduction algorithms. By utilizing this framework as a tool, we propose a new supervised dimensionality reduction algorithm called Marginal Fisher Analysis in which the intrinsic graph characterizes the intraclass compactness and connects each data point with its neighboring points of the same class, while the penalty graph connects the marginal points and characterizes the interclass separability. We show that MFA effectively overcomes the limitations of the traditional Linear Discriminant Analysis algorithm due to data distribution assumptions and available projection directions. Real face recognition experiments show the superiority of our proposed MFA in comparison to LDA, also for corresponding kernel and tensor extensions.

Local Geometric Structure Feature for Dimensionality Reduction of Hyperspectral Imagery

Marginal Fisher analysis (MFA) exploits the margin criterion to compact the intraclass data and separate the interclass data, and it is very useful to analyze the high-dimensional data. However, MFA just considers the structure relationships of neighbor points, and it cannot effectively represent the intrinsic structure of hyperspectral imagery (HSI) that possesses many homogenous areas. In this paper, we propose a new dimensionality reduction (DR) method, termed local geometric structure Fisher analysis (LGSFA), for HSI classification. Firstly, LGSFA uses the intraclass neighbor points of each point to compute its reconstruction point. Then, an intrinsic graph and a penalty graph are constructed to reveal the intraclass and interclass properties of hyperspectral data. Finally, the neighbor points and corresponding intraclass reconstruction points are used to enhance the intraclass-manifold compactness and the interclass-manifold separability. LGSFA can effectively reveal the intrinsic manifold structure and obtain the discriminating features of HSI data for classification. Experiments on the Salinas, Indian Pines, and Urban data sets show that the proposed LGSFA algorithm achieves the best classification results than other state-of-the-art methods.

Self-weighted spectral clustering with parameter-free constraint

The constrained spectral clustering (or known as the semi-supervised spectral clustering) focuses on enhancing the clustering capability by utilizing the side information. In this paper, a novel constrained spectral clustering method is proposed based on deriving a sparse parameter-free similarity. Different from other works, the proposed method transforms the given pairwise constraints into the intrinsic graph similarity and the penalty graph similarity respectively instead of incorporating them into one single similarity. Besides, the optimal weight can be automatically achieved to balance the graph optimization problems between the intrinsic graph and the penalty graph. Equipped with a general framework of efficiently unraveling the bi-objective optimization, the proposed method could obtain both ratio cut and normalized cut clusterings via updating the weighted Laplacian matrix until convergence. Moreover, the proposed method is equivalent to the spectral clustering, when no side information is provided. Consequently, the effectiveness and the superiority of the proposed method are further verified both analytically and empirically.

Local graph embedding based on maximum margin criterion via fuzzy set

Recently, many algorithms based on locally graph embedding are proposed for dimensional reduction in nonlinear data. However, these algorithms are not effective when dealing with face images affected by variations in illumination conditions, poses or perspectives and different facial expressions. So, distant data points are not deemphasized efficiently by locally graph embedding algorithms and it may degrade the performance of classification. In order to solve the aforementioned problem, this paper proposes a new efficient dimension reduction method—local graph embedding method based on maximum margin criterion via Fuzzy Set for face recognition. Firstly, the goal of this algorithm is preserved under nearest neighbor premise by constructing the fuzzy intrinsic graph and the fuzzy penalty graph. Secondly, two novel fuzzy Laplacian scatter matrices are calculated using Fuzzy K-Nearest Neighbor (FKNN) in the proposed method. Finally, Maximum Margin Criterion (MMC) is used to avoid the “small size sample” problem. The results of face recognition experiments on the ORL, YALE and AR face databases demonstrate the effectiveness of our proposed method.

Local uncorrelated local discriminant embedding for face recognition

The feature extraction algorithm plays an important role in face recognition. However, the extracted features also have overlapping discriminant information. A property of the statistical uncorrelated criterion is that it eliminates the redundancy among the extracted discriminant features, while many algorithms generally ignore this property. In this paper, we introduce a novel feature extraction method called local uncorrelated local discriminant embedding (LULDE). The proposed approach can be seen as an extension of a local discriminant embedding (LDE) framework in three ways. First, a new local statistical uncorrelated criterion is proposed, which effectively captures the local information of interclass and intraclass. Second, we reconstruct the affinity matrices of an intrinsic graph and a penalty graph, which are mentioned in LDE to enhance the discriminant property. Finally, it overcomes the small-sample-size problem without using principal component analysis to preprocess the original data, which avoids losing some discriminant information. Experimental results on Yale, ORL, Extended Yale B, and FERET databases demonstrate that LULDE outperforms LDE and other representative uncorrelated feature extraction methods.

A sparse neighborhood preserving non-negative tensor factorization algorithm for facial expression recognition

In this paper, a novel sparse neighborhood preserving non-negative tensor factorization (SNPNTF) algorithm is proposed for facial expression recognition. It is derived from non-negative tensor factorization (NTF), and it works in the rank-one tensor space. A sparse constraint is adopted into the objective function, which takes the optimization step in the direction of the negative gradient, and then projects onto the sparse constrained space. To consider the spatial neighborhood structure and the class-based discriminant information, a neighborhood preserving constraint is adopted based on the manifold learning and graph preserving theory. The Laplacian graph which encodes the spatial information in the face samples and the penalty graph which considers the pre-defined class information are considered in this constraint. By using it, the obtained parts-based representations of SNPNTF vary smoothly along the geodesics of the data manifold and they are more discriminant for recognition. SNPNTF is a quadratic convex function in the tensor space, and it could converge to the optimal solution. The gradient descent method is used for the optimization of SNPNTF to ensure the convergence property. Experiments are conducted on the JAFFE database, the Cohn---Kanade database and the AR database. The results demonstrate that SNPNTF provides effective facial representations and achieves better recognition performance, compared with non-negative matrix factorization, NTF and some variant algorithms. Also, the convergence property of SNPNTF is well guaranteed.