Unsupervised Dimensionality Reduction Techniques Research Articles

Principal component analysis - a tutorial

Dimensionality reduction is one of the preprocessing steps in many machine learning applications and it is used to transform the features into a lower dimension space. Principal component analysis (PCA) technique is one of the most famous unsupervised dimensionality reduction techniques. The goal of the technique is to find the PCA space, which represents the direction of the maximum variance of the given data. This paper highlights the basic background needed to understand and implement the PCA technique. This paper starts with basic definitions of the PCA technique and the algorithms of two methods of calculating PCA, namely, the covariance matrix and singular value decomposition (SVD) methods. Moreover, a number of numerical examples are illustrated to show how the PCA space is calculated in easy steps. Three experiments are conducted to show how to apply PCA in the real applications including biometrics, image compression, and visualisation of high-dimensional datasets.

Nonlocal structure constrained neighborhood preserving embedding model and its application for fault detection

In this paper, an unsupervised dimensionality reduction technique named nonlocal structure constrained neighborhood preserving embedding (NSC-NPE) is developed and applied for fault detection. To exploit the underlying geometrical structure, NSC-NPE constructs a global information-based dual-objective optimization function for modeling the process data. Besides the local variance information refined by the neighborhood preserving embedding algorithm, NSC-NPE also considers to utilize the meaningful nonlocal variance information via maximizing the Euclidean distance between the points outside the neighbors. The objective that having the local and nonlocal characteristics preserved in the low-dimensional space is achieved by minimizing the local scatter and maximizing the nonlocal scatter simultaneously. The proposed method is applied to fault detection based on the Hotelling’s T2 and squared prediction error (SPE) statistics. Three case studies are provided to demonstrate the efficiency of the proposed method.

Assessing the performance of two unsupervised dimensionality reduction techniques on hyperspectral APEX data for high resolution urban land-cover mapping

Despite the high richness of information content provided by airborne hyperspectral data, detailed urban land-cover mapping is still a challenging task. An important topic in hyperspectral remote sensing is the issue of high dimensionality, which is commonly addressed by dimensionality reduction techniques. While many studies focus on methodological developments in data reduction, less attention is paid to the assessment of the proposed methods in detailed urban hyperspectral land-cover mapping, using state-of-the-art image classification approaches. In this study we evaluate the potential of two unsupervised data reduction techniques, the Autoassociative Neural Network (AANN) and the BandClust method – the first a transformation based approach, the second a feature-selection based approach – for mapping of urban land cover at a high level of thematic detail, using an APEX 288-band hyperspectral dataset. Both methods were tested in combination with four state-of-the-art machine learning classifiers: Random Forest (RF), AdaBoost (ADB), the multiple layer perceptron (MLP), and support vector machines (SVM). When used in combination with a strong learner (MLP, SVM) BandClust produces classification accuracies similar to or higher than obtained with the full dataset, demonstrating the method’s capability of preserving critical spectral information, required for the classifier to successfully distinguish between the 22 urban land-cover classes defined in this study. In the AANN data reduction process, on the other hand, important spectral information seems to be compromised or lost, resulting in lower accuracies for three of the four classifiers tested. Detailed analysis of accuracies at class level confirms the superiority of the SVM/Bandclust combination for accurate urban land-cover mapping using a reduced hyperspectral dataset. This study also demonstrates the potential of the new APEX sensor data for detailed mapping of land cover in spatially and spectrally complex urban areas.

Pairwise Sparsity Preserving Embedding for Unsupervised Subspace Learning and Classification

Two novel unsupervised dimensionality reduction techniques, termed sparse distance preserving embedding (SDPE) and sparse proximity preserving embedding (SPPE), are proposed for feature extraction and classification. SDPE and SPPE perform in the clean data space recovered by sparse representation and enhanced Euclidean distances over noise removed data are employed to measure pairwise similarities of points. In extracting informative features, SDPE and SPPE aim at preserving pairwise similarities between data points in addition to preserving the sparse characteristics. This paper calculates the sparsest representation of all vectors jointly by a convex optimization. The sparsest codes enable certain local information of data to be preserved, and can endow SDPE and SPPE a natural discriminating power, adaptive neighborhood and robust characteristic against noise and errors in delivering low-dimensional embeddings. We also mathematically show SDPE and SPPE can be effectively extended for discriminant learning in a supervised manner. The validity of SDPE and SPPE is examined by extensive simulations. Comparison with other related state-of-the-art unsupervised algorithms show that promising results are delivered by our techniques.

Enhancement of Satellite Precipitation Estimation via Unsupervised Dimensionality Reduction

A methodology to enhance satellite precipitation estimation using unsupervised dimensionality reduction (UDR) techniques is developed. This enhanced technique is an extension to the precipitation estimation from remotely sensed imagery using an artificial neural network (PERSIANN) and cloud classification system (CCS) method (PERSIANN-CCS) enriched using wavelet features combined with dimensionality reduction. Cloud-top brightness temperature measurements from the Geostationary Operational Environmental Satellite (GOES)-12 are used for precipitation estimation at 4 km × 4 km spatial resolutions every 30 min. The study area in the continental U.S. covers parts of Louisiana, Arkansas, Kansas, Tennessee, Mississippi, and Alabama. Based on quantitative measures, root mean square error and Heidke skill score (HSS), the results show that the UDR techniques can improve the precipitation estimation accuracy. In addition, the independent component analysis is shown to have better performance than other UDR techniques; and in some cases, it achieves 10% improvement in the HSS.

Improving document clustering in a learned concept space

Most document clustering algorithms operate in a high dimensional bag-of-words space. The inherent presence of noise in such representation obviously degrades the performance of most of these approaches. In this paper we investigate an unsupervised dimensionality reduction technique for document clustering. This technique is based upon the assumption that terms co-occurring in the same context with the same frequencies are semantically related. On the basis of this assumption we first find term clusters using a classification version of the EM algorithm. Documents are then represented in the space of these term clusters and a multinomial mixture model ( MM) is used to build document clusters. We empirically show on four document collections, Reuters-21578, Reuters RCV2-French, 20Newsgroups and WebKB, that this new text representation noticeably increases the performance of the MM model. By relating the proposed approach to the Probabilistic Latent Semantic Analysis ( PLSA) model we further propose an extension of the latter in which an extra latent variable allows the model to co-cluster documents and terms simultaneously. We show on these four datasets that the proposed extended version of the PLSA model produces statistically significant improvements with respect to two clustering measures over all variants of the original PLSA and the MM models.

Analysis of unsupervised dimensionality reduction techniques

Domains such as text, images etc contain large amounts of redundancies and ambiguities among the attributes which result in considerable noise effects (i.e. the data is high dimension). Retrieving the data from high dimensional datasets is a big challenge. Dimensionality reduction techniques have been a successful avenue for automatically extracting the latent concepts by removing the noise and reducing the complexity in processing the high dimensional data. In this paper we conduct a systematic study on comparing the unsupervised dimensionality reduction techniques for text retrieval task. We analyze these techniques from the view of complexity, approximation error and retrieval quality with experiments on four testing document collections.