Patch Alignment Framework Research Articles

Learning latent local manifold embeddings for hyperspectral feature extraction

Hyperspectral images with an immense number of spectral bands provide abundant discriminant information for accurate land-cover classification in the remote sensing field. However, these narrow and adjacent bands contain a large amount of redundant information. Analyzing these images always requires a huge storage space with expensive computational costs. Furthermore, their high correlation coefficient would lead to the Hughes phenomenon, hindering the improvement of classification performance. We propose a linear semi-supervised hyperspectral feature extraction method L3ME to learn latent local manifold embeddings. Although labeled samples are beneficial to construct learning models, their number is always limited in real-world tasks. The motivation of this paper is to jointly enhance the contributions of labeled and unlabeled samples for learning local manifold structures of hyperspectral images. Features in labeled samples are extracted by two procedures, the adaptive patch alignment framework and integrated intraclass-interclass relationships, from different perspectives. The former aims to solve the problem of the uneven distribution of classes by introducing spectral angle based adaptive parameters. The latter aims to solve the problem of the uneven distribution of samples by constructing several adjacency graphs. The locality preserving projection is capable of preserving the local neighborhood structure of samples. A penalty for sparse regularization is cleverly integrated into the proposed linear discriminant objective function, which is optimized using a novel updating strategy. The convergence of L3ME is proved in detail and analyzed in this paper. Experiments on three typical hyperspectral datasets illustrate the effectiveness of the proposed method over some state-of-the-art techniques. The implementation of L3ME is available at https://github.com/biowby/L3ME.

Multidomain Subspace Classification for Hyperspectral Images

Hyperspectral imaging offers new opportunities for pattern recognition tasks in the remote sensing community through its improved discrimination in the spectral domain. However, such advanced image processing also brings new challenges due to the high data dimensionality in both the spatial and spectral domains. To relieve this issue, in this paper, we present a novel multidomain subspace (MDS) feature representation and classification method for hyperspectral images. The proposed method is based on a patch alignment framework. In order to optimally combine the feature representations from the various domains and simultaneously enhance the subspace discriminability, we incorporate the supervised label information into each domain and further generalize the framework to a multidomain version. Furthermore, we develop an iterative approach to alternately optimize the MDS objective function by considering it as two subconvex optimizations. The classification performance on three standard hyperspectral remote sensing images confirms the superiority of the proposed MDS algorithm over the state-of-the-art subspace learning methods.

Hessian regularization by patch alignment framework

In recent years, semi-supervised learning has played a key part in large-scale image management, where usually only a few images are labeled. To address this problem, many representative works have been reported, including transductive SVM, universum SVM, co-training and graph-based methods. The prominent method is the patch alignment framework, which unifies the traditional spectral analysis methods. In this paper, we propose Hessian regression based on the patch alignment framework. In particular, we construct a Hessian using the patch alignment framework and apply it to regression problems. To the best of our knowledge, there is no report on Hessian construction from the patch alignment viewpoint. Compared with the traditional Laplacian regularization, Hessian can better match the data and then leverage the performance. To validate the effectiveness of the proposed method, we conduct human face recognition experiments on a celebrity face dataset. The experimental results demonstrate the superiority of the proposed solution in human face classification.

Multi-view non-negative matrix factorization by patch alignment framework with view consistency

Multi-view non-negative matrix factorization (NMF) has been developed to learn the latent representation from multi-view non-negative data in recent years. To make the representation more meaningful, previous works mainly exploit either the consensus information or the complementary information from different views. However, the latent local geometric structure of each view is always ignored. In this paper, we develop a novel multi-view NMF by patch alignment framework with view consistency. Different from previous works, we take the local geometric structure of each view into consideration, and penalize the disagreement of different views at the same time. More specifically, given a data in each view, we construct a local patch utilizing locally linear embedding to preserve its local geometrical structure, and obtain the global representation under the whole alignment strategy. Meanwhile, for different views, we make the representations of views to approximate the latent representation shared by different views via considering the view consistency. We adopt the correntropy-induced metric to measure the reconstruction error and employ the half-quadratic technique to solve the optimization problem. The experimental results demonstrate the proposed method can achieve satisfactory performance compared with single-view methods and other existing multi-view NMF methods.

Cauchy estimator discriminant analysis for face recognition

With the rapid development of computer vision and pattern recognition, face recognition, one of the basic research topics in computer vision and pattern recognition, has received intensive attention in recent years. Usually, traditional face recognition algorithms have considerable discriminant ability; however, when there are some samples that are easy to confuse in the face database, the discriminant ability of traditional face recognition algorithms will inevitably decrease. In this paper, based on the patch alignment framework (PAF) and Cauchy estimator theory, we proposed a novelty subspace learning algorithm for face recognition named Cauchy estimator discriminant analysis (CEDA). Under the framework of PAF, both local and global geometries of the input samples are preserved; by using the Cauchy estimator, large errors caused by samples that are easy to confuse could be overcome. We conducted the experiments on three face databases and strongly illustrated the effectiveness of CEDA for face recognition.

Data-driven facial animation via semi-supervised local patch alignment

This paper reports a novel data-driven facial animation technique which drives a neutral source face to get the expressive target face using a semi-supervised local patch alignment framework. We define the local patch and assume that there exists a linear transformation between a patch of the target face and the intrinsic embedding of the corresponding patch of the source face. Based on this assumption, we compute the intrinsic embeddings of source patches and align these embeddings to form the result. During the course of alignment, we use a set of motion data as shape regularizer to impel the result to approach the unknown target face. The intrinsic embedding can be computed through both locally linear embedding and local tangent space alignment. Experimental results indicate that the proposed framework can obtain decent face driving results. Quantitative and qualitative evaluations of the proposed framework demonstrate its superiority to existing methods.

Three-dimensional image-based human pose recovery with hypergraph regularized autoencoders

Three-Dimensional image-based human pose recovery tries to retrieves 3D poses with 2D image. Therefore, one of the key problem is how to represent 2D images. However, semantic gap exists for current feature extractors, which limits recovery performance. In this paper, we propose a novel feature extractor with deep neural network. It is based on denoising autoencoders and improves previous autoencoders by adopting locality preserved restriction. To impose this restriction, we introduce manifold regularization with hypergraph learning. Hypergraph Laplacian matrix is constructed with patch alignment framework. In this way, an automatic feature extractor for images is achieved. Experimental results on three datasets show that the recovery error can be reduced by 10 % to 20 %, which demonstrates the effectiveness of the proposed method.

Marginal patch alignment for dimensionality reduction

Patch alignment (PA) framework provides us a useful way to obtain the explicit mapping for dimensionality reduction. Under the PA framework, we propose the marginal patch alignment (MPA) for dimensionality reduction. MPA performs the optimization from the part to the whole. In the phase of the patch optimization, the marginal between-class and within-class local neighborhoods of each training sample are selected to build the local marginal patches. By performing the patch optimization, on the one hand, the contributions of each sample for optimal subspace selection are distinguished. On the other hand, the marginal structure information is exploited to extract discriminative features such that the marginal distance between the two different categories is enlarged in the low transformed subspace. In the phase of the whole alignment, a trick is performed to unify all of the local patches into a globally linear system and make MPA obtain the whole optimization. The experimental results on the Yale face database, the UCI Wine dataset, the Yale-B face database, and the AR face database, show the effectiveness and efficiency of MPA.

Hypergraph regularized autoencoder for image-based 3D human pose recovery

Image-based human pose recovery is usually conducted by retrieving relevant poses with image features. However, semantic gap exists for current feature extractors, which limits recovery performance. In this paper, we propose a novel feature extractor with deep learning. It is based on denoising autoencoder and improves traditional methods by adopting locality preserved restriction. To impose this restriction, we introduce manifold regularization with hypergraph Laplacian. Hypergraph Laplacian matrix is constructed with patch alignment framework. In this way, an automatic feature extractor for silhouettes is achieved. Experimental results on two datasets show that the recovery error has been reduced by 10% to 20%, which demonstrates the effectiveness of the proposed method.

Dimensionality Reduction for Hyperspectral Data Based on Class-Aware Tensor Neighborhood Graph and Patch Alignment.

To take full advantage of hyperspectral information, to avoid data redundancy and to address the curse of dimensionality concern, dimensionality reduction (DR) becomes particularly important to analyze hyperspectral data. Exploring the tensor characteristic of hyperspectral data, a DR algorithm based on class-aware tensor neighborhood graph and patch alignment is proposed here. First, hyperspectral data are represented in the tensor form through a window field to keep the spatial information of each pixel. Second, using a tensor distance criterion, a class-aware tensor neighborhood graph containing discriminating information is obtained. In the third step, employing the patch alignment framework extended to the tensor space, we can obtain global optimal spectral-spatial information. Finally, the solution of the tensor subspace is calculated using an iterative method and low-dimensional projection matrixes for hyperspectral data are obtained accordingly. The proposed method effectively explores the spectral and spatial information in hyperspectral data simultaneously. Experimental results on 3 real hyperspectral datasets show that, compared with some popular vector- and tensor-based DR algorithms, the proposed method can yield better performance with less tensor training samples required.

Group sparse multiview patch alignment framework with view consistency for image classification.

No single feature can satisfactorily characterize the semantic concepts of an image. Multiview learning aims to unify different kinds of features to produce a consensual and efficient representation. This paper redefines part optimization in the patch alignment framework (PAF) and develops a group sparse multiview patch alignment framework (GSM-PAF). The new part optimization considers not only the complementary properties of different views, but also view consistency. In particular, view consistency models the correlations between all possible combinations of any two kinds of view. In contrast to conventional dimensionality reduction algorithms that perform feature extraction and feature selection independently, GSM-PAF enjoys joint feature extraction and feature selection by exploiting l(2,1)-norm on the projection matrix to achieve row sparsity, which leads to the simultaneous selection of relevant features and learning transformation, and thus makes the algorithm more discriminative. Experiments on two real-world image data sets demonstrate the effectiveness of GSM-PAF for image classification.

Image clustering based on sparse patch alignment framework

Image clustering methods are efficient tools for applications such as content-based image retrieval and image annotation. Recently, graph based manifold learning methods have shown promising performance in extracting features for image clustering. Typical manifold learning methods adopt appropriate neighborhood size to construct the neighborhood graph, which captures local geometry of data distribution. Because the density of data points’ distribution may be different in different regions of the manifold, a fixed neighborhood size may be inappropriate in building the manifold. In this paper, we propose a novel algorithm, named sparse patch alignment framework, for the embedding of data lying in multiple manifolds. Specifically, we assume that for each data point there exists a small neighborhood in which only the points that come from the same manifold lie approximately in a low-dimensional affine subspace. Based on the patch alignment framework, we propose an optimization strategy for constructing local patches, which adopt sparse representation to select a few neighbors of each data point that span a low-dimensional affine subspace passing near that point. After that, the whole alignment strategy is utilized to build the manifold. Experiments are conducted on four real-world datasets, and the results demonstrate the effectiveness of the proposed method.

Rank Preserving Discriminant Analysis for Human Behavior Recognition on Wireless Sensor Networks

With the rapid development of the intelligent sensing and the prompt growing industrial safety demands, human behavior recognition has received a great deal of attentions in industrial informatics. To deploy an utmost scalable, flexible, and robust human behavior recognition system, we need both innovative sensing electronics and suitable intelligence algorithms. Wireless sensor networks (WSNs) open a novel way for human behavior recognition, because the heavy computation can be immediately transferred to a network server. In this paper, a new scheme for human behavior recognition on WSNs is proposed, which transmits activities' signals compressed by Hamming compressed sensing to the network server and conducts behavior recognition through a collaboration between a new dimension reduction algorithm termed rank preserving discriminant analysis (RPDA) and a nearest neighbor classifier. RPDA encodes local rank information of within-class samples and discriminative information of the between-class under the framework of Patch Alignment Framework. Experiments are conducted on the SCUT Naturalistic 3D Acceleration-based Activity (SCUT NAA) dataset and demonstrate the effectiveness of RPDA for human behavior recognition.

Multiview Locally Linear Embedding for Effective Medical Image Retrieval

Content-based medical image retrieval continues to gain attention for its potential to assist radiological image interpretation and decision making. Many approaches have been proposed to improve the performance of medical image retrieval system, among which visual features such as SIFT, LBP, and intensity histogram play a critical role. Typically, these features are concatenated into a long vector to represent medical images, and thus traditional dimension reduction techniques such as locally linear embedding (LLE), principal component analysis (PCA), or laplacian eigenmaps (LE) can be employed to reduce the “curse of dimensionality”. Though these approaches show promising performance for medical image retrieval, the feature-concatenating method ignores the fact that different features have distinct physical meanings. In this paper, we propose a new method called multiview locally linear embedding (MLLE) for medical image retrieval. Following the patch alignment framework, MLLE preserves the geometric structure of the local patch in each feature space according to the LLE criterion. To explore complementary properties among a range of features, MLLE assigns different weights to local patches from different feature spaces. Finally, MLLE employs global coordinate alignment and alternating optimization techniques to learn a smooth low-dimensional embedding from different features. To justify the effectiveness of MLLE for medical image retrieval, we compare it with conventional spectral embedding methods. We conduct experiments on a subset of the IRMA medical image data set. Evaluation results show that MLLE outperforms state-of-the-art dimension reduction methods.

Multi-view hypergraph learning by patch alignment framework

Graph-based methods are currently popular for dimensionality reduction. However, most of them suffer from over-simplified assumption of pairwise relationships among data. Especially for multi-view data, different relationships from different views are hard to be integrated into a single graph. In this paper, we propose a novel semi-supervised dimensionality reduction method for multi-view data. First, we assume the hyperedges in hypergraph as patches and apply hypergraph to the patch alignment framework. Second, the weights of the hyperedges are computed with statistics of distances between neighboring pairs and the patches from different views are integrated. In this way, we construct Multi-view Hypergraph Laplacian matrix and we get the dimensionality-reduced data by solving the standard eigen-decomposition to obtain the projection matrix. The experimental results demonstrate the effectiveness of the proposed method on retrieval performance.

On Combining Multiple Features for Cartoon Character Retrieval and Clip Synthesis

How do we retrieve cartoon characters accurately? Or how to synthesize new cartoon clips smoothly and efficiently from the cartoon library? Both questions are important for animators and cartoon enthusiasts to design and create new cartoons by utilizing existing cartoon materials. The first key issue to answer those questions is to find a proper representation that describes the cartoon character effectively. In this paper, we consider multiple features from different views, i.e., color histogram, Hausdorff edge feature, and skeleton feature, to represent cartoon characters with different colors, shapes, and gestures. Each visual feature reflects a unique characteristic of a cartoon character, and they are complementary to each other for retrieval and synthesis. However, how to combine the three visual features is the second key issue of our application. By simply concatenating them into a long vector, it will end up with the so-called "curse of dimensionality," let alone their heterogeneity embedded in different visual feature spaces. Here, we introduce a semisupervised multiview subspace learning (semi-MSL) algorithm, to encode different features in a unified space. Specifically, under the patch alignment framework, semi-MSL uses the discriminative information from labeled cartoon characters in the construction of local patches where the manifold structure revealed by unlabeled cartoon characters is utilized to capture the geometric distribution. The experimental evaluations based on both cartoon character retrieval and clip synthesis demonstrate the effectiveness of the proposed method for cartoon application. Moreover, additional results of content-based image retrieval on benchmark data suggest the generality of semi-MSL for other applications.

Discriminative information preservation for face recognition

It is usually difficult to find the optimal low dimensional subspace for face recognition. Patch alignment framework (PAF) is an important systematic framework that can be applied to understand the common thought and essential differences of a numerous dimensionality reduction algorithms, e.g., principal component analysis, linear discriminant analysis and locally linear embedding and ISOMAP. These algorithms do not consider the intra-class local geometry and the inter-class discrimination simultaneously. In this paper, we present a new dimensionality reduction algorithm based on PAF, termed the discriminative information preservation based dimensionality reduction or DIP for short. First, DIP models the local geometry of intra-class samples by using Locality preserving projection (LPP) rebuilt upon PAF. Second, it models the discriminative information of inter-class samples by maximizing the margin. Thoroughly experimental evidence on several public face datasets suggests the effectiveness of DIP compared with the popular algorithms.

On Combining Multiple Features for Hyperspectral Remote Sensing Image Classification

In hyperspectral remote sensing image classification, multiple features, e.g., spectral, texture, and shape features, are employed to represent pixels from different perspectives. It has been widely acknowledged that properly combining multiple features always results in good classification performance. In this paper, we introduce the patch alignment framework to linearly combine multiple features in the optimal way and obtain a unified low-dimensional representation of these multiple features for subsequent classification. Each feature has its particular contribution to the unified representation determined by simultaneously optimizing the weights in the objective function. This scheme considers the specific statistical properties of each feature to achieve a physically meaningful unified low-dimensional representation of multiple features. Experiments on the classification of the hyperspectral digital imagery collection experiment and reflective optics system imaging spectrometer hyperspectral data sets suggest that this scheme is effective.

Non-Negative Patch Alignment Framework

In this paper, we present a non-negative patch alignment framework (NPAF) to unify popular non-negative matrix factorization (NMF) related dimension reduction algorithms. It offers a new viewpoint to better understand the common property of different NMF algorithms. Although multiplicative update rule (MUR) can solve NPAF and is easy to implement, it converges slowly. Thus, we propose a fast gradient descent (FGD) to overcome the aforementioned problem. FGD uses the Newton method to search the optimal step size, and thus converges faster than MUR. Experiments on synthetic and real-world datasets confirm the efficiency of FGD compared with MUR for optimizing NPAF. Based on NPAF, we develop non-negative discriminative locality alignment (NDLA). Experiments on face image and handwritten datasets suggest the effectiveness of NDLA in classification tasks and its robustness to image occlusions, compared with representative NMF-related dimension reduction algorithms.

Complex Object Correspondence Construction in Two-Dimensional Animation

Correspondence construction of objects in key frames is the precondition for inbetweening and coloring in 2-D computer-assisted animation production. Since each frame of an animation consists of multiple layers, objects are complex in terms of shape and structure. Therefore, existing shape-matching algorithms specifically designed for simple structures such as a single closed contour cannot perform well on objects constructed by multiple contours with an open shape. This paper introduces a semisupervised patch alignment framework for complex object correspondence construction. In particular, the new framework constructs local patches for each point on an object and aligns these patches in a new feature space, in which correspondences between objects can be detected by the subsequent clustering. For local patch construction, pairwise constraints, which indicate the corresponding points (must link) or unfitting points (cannot link), are introduced by users to improve the performance of correspondence construction. This kind of input is convenient for animation software users via user-friendly interfaces. A dozen of experimental results on our cartoon data set that is built on industrial production suggest the effectiveness of the proposed framework for constructing correspondences of complex objects. As an extension of our framework, additional shape retrieval experiments on MPEG-7 data set show that its performance is comparable with that of a prominent algorithm published in T-PAMI 2009.