Intrinsic Dimension Estimation Research Articles

Low Bias Local Intrinsic Dimension Estimation from Expected Simplex Skewness.

In exploratory high-dimensional data analysis, local intrinsic dimension estimation can sometimes be used in order to discriminate between data sets sampled from different low-dimensional structures. Global intrinsic dimension estimators can in many cases be adapted to local estimation, but this leads to problems with high negative bias or high variance. We introduce a method that exploits the curse/blessing of dimensionality and produces local intrinsic dimension estimators that have very low bias, even in cases where the intrinsic dimension is higher than the number of data points, in combination with relatively low variance. We show that our estimators have a very good ability to classify local data sets by their dimension compared to other local intrinsic dimension estimators; furthermore we provide examples showing the usefulness of local intrinsic dimension estimation in general and our method in particular for stratification of real data sets.

Intrinsic Dimension Estimation: Relevant Techniques and a Benchmark Framework

When dealing with datasets comprising high-dimensional points, it is usually advantageous to discover some data structure. A fundamental information needed to this aim is the minimum number of parameters required to describe the data while minimizing the information loss. This number, usually called intrinsic dimension, can be interpreted as the dimension of the manifold from which the input data are supposed to be drawn. Due to its usefulness in many theoretical and practical problems, in the last decades the concept of intrinsic dimension has gained considerable attention in the scientific community, motivating the large number of intrinsic dimensionality estimators proposed in the literature. However, the problem is still open since most techniques cannot efficiently deal with datasets drawn from manifolds of high intrinsic dimension and nonlinearly embedded in higher dimensional spaces. This paper surveys some of the most interesting, widespread used, and advanced state-of-the-art methodologies. Unfortunately, since no benchmark database exists in this research field, an objective comparison among different techniques is not possible. Consequently, we suggest a benchmark framework and apply it to comparatively evaluate relevant state-of-the-art estimators.

Geodesic distances in the intrinsic dimensionality estimation using packing numbers

Dimensionality reduction is a very important tool in data mining. An intrinsic dimensionality of a data set is a key parameter in many dimensionality reduction algorithms. When the intrinsic dimensionality of a data set is known, it is possible to reduce the dimensionality of the data without losing much information. To this end, it is reasonable to find out the intrinsic dimensionality of the data. In this paper, one of the global estimators of intrinsic dimensionality, the packing numbers estimator (PNE), is explored experimentally. We propose the modification of the PNE method that uses geodesic distances in order to improve the estimates of the intrinsic dimensionality by the PNE method.

DANCo: An intrinsic dimensionality estimator exploiting angle and norm concentration

In the past decade the development of automatic intrinsic dimensionality estimators has gained considerable attention due to its relevance in several application fields. However, most of the proposed solutions prove to be not robust on noisy datasets, and provide unreliable results when the intrinsic dimensionality of the input dataset is high and the manifold where the points are assumed to lie is nonlinearly embedded in a higher dimensional space. In this paper we propose a novel intrinsic dimensionality estimator (DANCo) and its faster variant (FastDANCo), which exploit the information conveyed both by the normalized nearest neighbor distances and by the angles computed on couples of neighboring points. The effectiveness and robustness of the proposed algorithms are assessed by experiments on synthetic and real datasets, by the comparative evaluation with state-of-the-art methodologies, and by significance tests.

Intrinsic dimensionality estimation based on manifold assumption

Dimensionality reduction is an important tool and has been widely used in many fields of data mining and machine learning. Intrinsic dimension of data sets is a key parameter for dimensionality reduction. In this paper, a new intrinsic dimension estimation method based on geometrical relationship between manifold intrinsic dimension and data neighborhood geodesic distances is presented. The estimator is derived by manifold sampling assumption. On a densely sampled manifold, the number of samples that fall into a ball is equal to the volume times the density of the ball. The radius of the ball is calculated by graph distance which is approximation of geodesic distance on manifold. Then the intrinsic dimension is estimated on each sample. Experiments conducted on synthetic and real world data set show that the performance of our new method is robust and comparable to other works.

Intrinsic dimension estimation via nearest constrained subspace classifier

We consider the problems of classification and intrinsic dimension estimation on image data. A new subspace based classifier is proposed for supervised classification or intrinsic dimension estimation. The distribution of the data in each class is modeled by a union of a finite number of affine subspaces of the feature space. The affine subspaces have a common dimension, which is assumed to be much less than the dimension of the feature space. The subspaces are found using regression based on the ℓ0-norm. The proposed method is a generalisation of classical NN (Nearest Neighbor), NFL (Nearest Feature Line) classifiers and has a close relationship to NS (Nearest Subspace) classifier. The proposed classifier with an accurately estimated dimension parameter generally outperforms its competitors in terms of classification accuracy. We also propose a fast version of the classifier using a neighborhood representation to reduce its computational complexity. Experiments on publicly available datasets corroborate these claims.

Ensemble estimators for multivariate entropy estimation.

The problem of estimation of density functionals like entropy and mutual information has received much attention in the statistics and information theory communities. A large class of estimators of functionals of the probability density suffer from the curse of dimensionality, wherein the mean squared error (MSE) decays increasingly slowly as a function of the sample size T as the dimension d of the samples increases. In particular, the rate is often glacially slow of order O(T-γ/d ), where γ > 0 is a rate parameter. Examples of such estimators include kernel density estimators, k-nearest neighbor (k-NN) density estimators, k-NN entropy estimators, intrinsic dimension estimators and other examples. In this paper, we propose a weighted affine combination of an ensemble of such estimators, where optimal weights can be chosen such that the weighted estimator converges at a much faster dimension invariant rate of O(T-1). Furthermore, we show that these optimal weights can be determined by solving a convex optimization problem which can be performed offline and does not require training data. We illustrate the superior performance of our weighted estimator for two important applications: (i) estimating the Panter-Dite distortion-rate factor and (ii) estimating the Shannon entropy for testing the probability distribution of a random sample.

Intrinsic Dimensionality Estimation for High-dimensional Data Sets: New Approaches for the Computation of Correlation Dimension

The analysis of high–dimensional data is usually challenging since many standard modelling approaches tend to break down due to the so–called “curse of dimensionality”. Dimension reduction techniques, which reduce the data set (explicitly or implicitly) to a smaller number of variables, make the data analysis more efficient and are furthermore useful for visualization purposes. However, most dimension reduction techniques require fixing the intrinsic dimension of the low-dimensional subspace in advance. The intrinsic dimension can be estimated by fractal dimension estimation methods, which exploit the intrinsic geometry of a data set. The most popular concept from this family of methods is the correlation dimension, which requires estimation of the correlation integral for a ball of radius tending to 0. In this paper we propose approaches to approximate the correlation integral in this limit. Experimental results on real world and simulated data are used to demonstrate the algorithms and compare to other methodology. A simulation study which verifies the effectiveness of the proposed methods is also provided.

Hyperspectral Intrinsic Dimensionality Estimation With Nearest-Neighbor Distance Ratios

The first task to be performed in most hyperspectral unmixing chains is the estimation of the number of endmembers. Several techniques for this problem have already been proposed, but the class of fractal techniques for intrinsic dimensionality estimation is often overlooked. In this paper, we study an intrinsic dimensionality estimation technique based on the known scaling behavior of nearest-neighbor distance ratios, and its performance on the spectral unmixing problem. We present the relation between intrinsic manifold dimensionality and the number of endmembers in a mixing model, and investigate the effects of denoising and the statistics on the algorithm. The algorithm is compared with several alternative methods, such as Hysime, virtual dimensionality, and several fractal-dimension based techniques, on both artificial and real data sets. Robust behavior in the presence of noise, and independence of the spectral dimensionality, is demonstrated. Furthermore, due to its construction, the algorithm can be used for non-linear mixing models as well.

Improved Locally Linear Embedding Based Method for Nonlinear System Fault Detection

In order to detect faults of nonlinear systems, an approach based on improved Locally Linear Embedding (LLE) was proposed. Firstly, the raw data was projected to lower dimensional space by LLE. In this step, tangent space distance was introduced to LLE and certain enhancement had also been made to intrinsic dimension estimation to make the approach more efficient and robust. Secondly, the inner class distance of data was calculated as an index of fault detection. To demonstrate the effectiveness of the improved LLE method, it is applied to Tennessee Eastman (TE) process and compared with kernel principle component analysis (KPCA) method. By simulation analysis, the false negative rate of the proposed approach achieves 4.498% in average, which is much better than 77.53% of KPCA, certifying the effectiveness of the approach to nonlinear fault detection.

Comparative Study of Intrinsic Dimensionality Estimation and Dimension Reduction Techniques on Hyperspectral Images Using K-NN Classifier

Nowadays, hyperspectral remote sensors are readily available for monitoring the Earth's surface with high spectral resolution. The high-dimensional nature of the data collected by such sensors not only increases computational complexity but also can degrade classification accuracy. To address this issue, dimensionality reduction (DR) has become an important aid to improving classifier efficiency on these images. The common approach to decreasing dimensionality is feature extraction by considering the intrinsic dimensionality (ID) of the data. A wide range of techniques for ID estimation (IDE) and DR for hyperspectral images have been presented in the literature. However, the most effective and optimum methods for IDE and DR have not been determined for hyperspectral sensors, and this causes ambiguity in selecting the appropriate techniques for processing hyperspectral images. In this letter, we discuss and compare ten IDE and six DR methods in order to investigate and compare their performance for the purpose of supervised hyperspectral image classification by using K-nearest neighbor (K-NN). Due to the nature of K-NN classifier that uses different distance metrics, a variety of distance metrics were used and compared in this procedure. This letter presents a review and comparative study of techniques used for IDE and DR and identifies the best methods for IDE and DR in the context of hyperspectral image analysis. The results clearly show the superiority of the hyperspectral signal subspace identification by minimum, second moment linear, and noise-whitened Harsanyi-Farrand-Chang estimators, also the principal component analysis and independent component analysis as DR techniques, and the norm L1 and Euclidean distance metrics to process hyperspectral imagery by using the K-NN classifier.

Dimension estimation of image manifolds by minimal cover approximation

Estimating intrinsic dimension of data is an important problem in feature extraction and feature selection. It provides an estimation of the number of desired features. Principal Components Analysis (PCA) is a powerful tool in discovering the dimension of data sets with a linear structure; it, however, becomes ineffective when data have a nonlinear structure. In this paper, we propose a new PCA-based method to estimate the embedding dimension of data with nonlinear structures. Our method works by first finding a minimal cover of the data set, then performing PCA locally on each subset in the cover to obtain local intrinsic dimension estimations and finally giving the estimation result as the average of the local estimations. There are two main innovations in our method. (1) A novel noise filtering procedure is applied in the PCA procedure for local intrinsic dimension estimation. (2) A minimal cover is constructed over the whole data set. Because of these two innovations, our method is fast, robust to noise and outliers, converges to a stable estimation with a wide range of sub-region sizes and can be used in the incremental sense, where the subregion refers to the local approximation of the distributed manifold. Experiments on synthetic and image data sets show effectiveness of the proposed method.

Effect of Enculturation on the Semantic and Acoustic Correlates of Polyphonic Timbre

polyphonic timbre perception was investigated in a cross-cultural context wherein Indian and Western nonmusicians rated short Indian and Western popular music excerpts (1.5 s, n = 200) on eight bipolar scales. Intrinsic dimensionality estimation revealed a higher number of perceptual dimensions in the timbre space for music from one's own culture. Factor analyses of Indian and Western participants' ratings resulted in highly similar factor solutions. The acoustic features that predicted the perceptual dimensions were similar across the two participant groups. Furthermore, both the perceptual dimensions and their acoustic correlates matched closely with the results of a previous study performed using Western musicians as participants. Regression analyses revealed relatively well performing models for the perceptual dimensions. The models displayed relatively high cross-validation performance. The findings suggest the presence of universal patterns in polyphonic timbre perception while demonstrating the increase of dimensionality of timbre space as a result of enculturation.

Fractal-Based Intrinsic Dimension Estimation and Its Application in Dimensionality Reduction

Dimensionality reduction is an important step in knowledge discovery in databases. Intrinsic dimension indicates the number of variables necessary to describe a data set. Two methods, box-counting dimension and correlation dimension, are commonly used for intrinsic dimension estimation. However, the robustness of these two methods has not been rigorously studied. This paper demonstrates that correlation dimension is more robust with respect to data sample size. In addition, instead of using a user selected distance d, we propose a new approach to capture all log-log pairs of a data set to more precisely estimate the correlation dimension. Systematic experiments are conducted to study factors that influence the computation of correlation dimension, including sample size, the number of redundant variables, and the portion of log-log plot used for calculation. Experiments on real-world data sets confirm the effectiveness of intrinsic dimension estimation with our improved method. Furthermore, a new supervised dimensionality reduction method based on intrinsic dimension estimation was introduced and validated.

Intrinsic dimension estimation by maximum likelihood in isotropic probabilistic PCA

A central issue in dimension reduction is choosing a sensible number of dimensions to be retained. This work demonstrates the surprising result of the asymptotic consistency of the maximum likelihood criterion for determining the intrinsic dimension of a dataset in an isotropic version of probabilistic principal component analysis (PPCA). Numerical experiments on simulated and real datasets show that the maximum likelihood criterion can actually be used in practice and outperforms existing intrinsic dimension selection criteria in various situations. This paper exhibits and outlines the limits of the maximum likelihood criterion. It leads to recommend the use of the AIC criterion in specific situations. A useful application of this work would be the automatic selection of intrinsic dimensions in mixtures of isotropic PPCA for classification.

Image Dimensionality Reduction Based on the Intrinsic Dimension and Parallel Genetic Algorithm

In the research of Web content-based image retrieval, how to reduce more of the image dimensions without losing the main features of the image is highlighted. Many features of dimensional reduction schemes are determined by the breaking of higher dimensional general covariance associated with the selection of a particular subset of coordinates. This paper starts with analysis of commonly used methods for the dimension reduction of Web images, followed by a new algorithm for nonlinear dimensionality reduction based on the HSV image features. The approach obtains intrinsic dimension estimation by similarity calculation of two images. Finally, some improvements were made on the Parallel Genetic Algorithm (APGA) by use of the image similarity function as the self-adaptive judgment function to improve the genetic operators, thus achieving a Web image dimensionality reduction and similarity retrieval. Experimental results illustrate the validity of the algorithm.

Intrinsic Dimensionality Estimation of High-Dimension, Low Sample Size Data with D-Asymptotics

High-dimension, low sample size (HDLSS) data are becoming common in various fields such as genetic microarrays, medical imaging, text recognition, finance, chemometrics, and so on. Such data have surprising and often counterintuitive geometric structures because of the high-dimensional noise that dominates and corrupts the local neighborhoods. In this article, we estimate the intrinsic dimension (ID) that allows one to distinguish between deterministic chaos and random noise of HDLSS data. A new ID estimating methodology is given and its properties are studied by using a d-asymptotic approach.

Estimating intrinsic dimensionality using the multi-criteria decision weighted model and the average standard estimator

Information retrieval today is much more challenging than traditional small document retrieval. The main difference is the importance of correlations between related concepts in complex data structures. As collections of data grow and contain more entries, they require more complex relationships, links, and groupings between individual entries. This paper introduces two novel methods for estimating data intrinsic dimensionality based on the singular value decomposition (SVD). The average standard estimator (ASE) and the multi-criteria decision weighted model are used to estimate matrix intrinsic dimensionality for large document collections. The multi-criteria weighted model calculates the sum of weighted values of matrix dimensions which demonstrated best performance using all possible dimensions [1]. ASE estimates the level of significance for singular values that resulted from the singular value decomposition. ASE assumes that those variables with deep relations have sufficient correlation and that only those relationships with high singular values are significant and should be maintained [1]. Experimental results indicate that ASE improves precision and relative relevance for MEDLINE document collection by 10.2% and 12.9% respectively compared to the percentage of variance dimensionality estimation. Results based on testing three document collections over all possible dimensions using selected performance measures indicate that ASE improved matrix intrinsic dimensionality estimation by including the effect of both singular values magnitude of decrease and random noise distracters. The multi-criteria weighted model with dimensionality reduction provides a more efficient implementation for information retrieval than using a full rank model.

On Local Intrinsic Dimension Estimation and Its Applications

In this paper, we present multiple novel applications for local intrinsic dimension estimation. There has been much work done on estimating the global dimension of a data set, typically for the purposes of dimensionality reduction. We show that by estimating dimension locally, we are able to extend the uses of dimension estimation to many applications, which are not possible with global dimension estimation. Additionally, we show that local dimension estimation can be used to obtain a better global dimension estimate, alleviating the negative bias that is common to all known dimension estimation algorithms. We illustrate local dimension estimation's uses towards additional applications, such as learning on statistical manifolds, network anomaly detection, clustering, and image segmentation.

Intrinsic dimension estimation of manifolds by incising balls

Dimensionality reduction is a very important tool in data mining. Intrinsic dimension of data sets is a key parameter for dimensionality reduction. However, finding the correct intrinsic dimension is a challenging task. In this paper, a new intrinsic dimension estimation method is presented. The estimator is derived by finding the exponential relationship between the radius of an incising ball and the number of samples included in the ball. The method is compared with the previous dimension estimation methods. Experiments have been conducted on synthetic and high dimensional image data sets and on data sets of the Santa Fe time series competition, and the results show that the new method is accurate and robust.