Intrinsic Dimension Estimation Research Articles

Unsupervised learning of quantum many-body scars using intrinsic dimension

Quantum many-body scarred systems contain both thermal and non-thermal scar eigenstates in their spectra. When these systems are quenched from special initial states which share high overlap with scar eigenstates, the system undergoes dynamics with atypically slow relaxation and periodic revival. This scarring phenomenon poses a potential avenue for circumventing decoherence in various quantum engineering applications. Given access to an unknown scar system, current approaches for identification of special states leading to non-thermal dynamics rely on costly measures such as entanglement entropy. In this work, we show how two dimensionality reduction techniques, multidimensional scaling and intrinsic dimension estimation, can be used to learn structural properties of dynamics in the PXP model and distinguish between thermal and scar initial states. The latter method is shown to be robust against limited sample sizes and experimental measurement errors.

A coincidence detection perspective for the maximum mean discrepancy

An alternative perspective is proposed for the Maximum Mean Discrepancy (MMD), in which coincidence detectors replace Gaussian kernels. It is conjectured that coincidence-based statistics may be a relevant factor behind MMD, for it may explain why MMD works even for small high-dimensional sets of observations. It is further shown how this proposed perspective can be used to simplify interpretations in MMD-based tests, including a straightforward computation of thresholds for hypothesis tests, which is done through the Grassberger–Procaccia method, originally proposed for intrinsic dimensionality estimation. Experimental results corroborate the conjecture that an MMD based on coincidence detection would perform almost equivalently to the MMD based on (frequently used) Gaussian kernels, with advantages in terms of interpretability and computational load.

A global perspective on the intrinsic dimensionality of COVID-19 data

We develop a novel global perspective of the complexity of the relationships between three COVID-19 datasets, the standardised per-capita growth rate of COVID-19 cases and deaths, and the Oxford Coronavirus Government Response Tracker COVID-19 Stringency Index (CSI) which is a measure describing a country’s stringency of lockdown policies. We use a state-of-the-art heterogeneous intrinsic dimension estimator implemented as a Bayesian mixture model, called Hidalgo. Our findings suggest that these highly popular COVID-19 statistics may project onto two low-dimensional manifolds without significant information loss, suggesting that COVID-19 data dynamics are generated from a latent mechanism characterised by a few important variables. The low dimensionality imply a strong dependency among the standardised growth rates of cases and deaths per capita and the CSI for countries over 2020–2021. Importantly, we identify spatial autocorrelation in the intrinsic dimension distribution worldwide. The results show how high-income countries are more prone to lie on low-dimensional manifolds, likely arising from aging populations, comorbidities, and increased per capita mortality burden from COVID-19. Finally, the temporal stratification of the dataset allows the examination of the intrinsic dimension at a more granular level throughout the pandemic.

Underestimation modification for intrinsic dimension estimation

The intrinsic dimension is the dimension of the low-dimensional manifold where the high-dimensional data is located. Accurately estimating the intrinsic dimension of the data set is helpful for data-dimensionality reduction and preprocessing. Due to the unknown spatial distribution of data and the limited sample size of a dataset, estimation methods which only use distance information tend to underestimate the intrinsic dimension of dataset. To reduce the estimation complexity and improve the accuracy, two estimation algorithms based on ID(κ) are proposed, where κ is the scaling ratio of the neighborhood radius of the sample point. First, according to the selection criteria of parameter κ, an improved algorithm for selecting the optimal scaling ratio κ is proposed, which reduces the computational complexity and improves the stability of estimation. Second, using simulation datasets with the same sample size and known intrinsic dimensions, the relationship between the estimated dimension and the true intrinsic dimension is obtained, and an underestimation modification method for intrinsic dimension estimation is proposed. Results of comparative experiments on simulation and real datasets indicate that the underestimation modification algorithm has high estimation accuracy and robustness.

IntRinsic: An R Package for Model-Based Estimation of the Intrinsic Dimension of a Dataset

This article illustrates intRinsic, an R package that implements novel state-of-the-art likelihood-based estimators of the intrinsic dimension of a dataset, an essential quantity for most dimensionality reduction techniques. In order to make these novel estimators easily accessible, the package contains a small number of high-level functions that rely on a broader set of efficient, low-level routines. Generally speaking, intRinsic encompasses models that fall into two categories: homogeneous and heterogeneous intrinsic dimension estimators. The first category contains the two nearest neighbors estimator, a method derived from the distributional properties of the ratios of the distances between each data point and its first two closest neighbors. The functions dedicated to this method carry out inference under both the frequentist and Bayesian frameworks. In the second category, we find the heterogeneous intrinsic dimension algorithm, a Bayesian mixture model for which an efficient Gibbs sampler is implemented. After presenting the theoretical background, we demonstrate the performance of the models on simulated datasets. This way, we can facilitate the exposition by immediately assessing the validity of the results. Then, we employ the package to study the intrinsic dimension of the Alon dataset, obtained from a famous microarray experiment. Finally, we show how the estimation of homogeneous and heterogeneous intrinsic dimensions allows us to gain valuable insights into the topological structure of a dataset.

Measurement-induced criticality as a data-structure transition

We employ unsupervised learning tools to identify different phases and their transition in quantum systems subject to the combined action of unitary evolution and stochastic measurements. Specifically, we consider principal component analysis and intrinsic dimension estimation to reveal a measurement-induced structural transition in the data space. We test our approach on a 1+1D stabilizer circuit and find the quantities of interest furnish novel order parameters defined directly in the raw data space. Our results provide a first use of unsupervised tools in dynamical quantum phase transitions.

Correlation Integral-Based Intrinsic Dimension: A Deep-Learning-Assisted Empirical Metric to Estimate the Robustness of Physically Unclonable Functions to Modeling Attacks

Physically unclonable functions (PUFs) which are robust to modeling attacks, usually have a complex, high-dimensional, nonlinear relationship between challenges and responses. Often, it is difficult to derive closed-form analytical expressions for these relationships. Consequently, it becomes difficult to compare PUF variants with regard to their robustness to modeling attacks. In this article, we apply a data-driven empirical metric termed the intrinsic dimension (ID), to estimate the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">inherent complexity</i> of the relationship between the challenges and responses of a given PUF variant. This metric is computed from the linear projection layer of a deep neural network (DNN) aimed at modeling the PUF and has a unique advantage that it is independent of the architectural details and chosen hyperparameters of the DNN. It also does not require the knowledge of the structural and functional details of the PUF. The proposed approach is evaluated using two well-known ID estimation methods based on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nearest neighbor method</i> and the full correlation integral (FCI). Through detailed experimental results, we demonstrate that the numerical values of the FCI-based ID metric for different types of PUFs have consistently high positive correlation with the perceived difficulty of modeling several common PUF variants. We also show that the ID metric provides deep insight about various subtleties that affect the robustness of PUFs to modeling attack and provides a convenient mechanism to perform a systematic comparison between different PUF compositions.

Anderson relaxation test for intrinsic dimension selection in model-based clustering

Parsimonious finite mixture models often require the a priori selection of desired model dimensionality. For example, projection-based parsimonious models demand the dimension of the subspace for projection. Other models ask for their own structural restrictions on parameters. The subspace clustering framework is a projection-based parsimonious model for various finite mixtures, including the Gaussian variant. The existing dimension selection methods for subspace clustering are ad-hoc or potentially computationally prohibitive, creating a need for a principled, yet computationally lightweight, approach. In light of this problem, a hypothesis test-based intrinsic dimension estimation method called the Anderson Relaxation Test (ART) is introduced, and its performance is examined in both simulated and real data settings.

Intrinsic Dimension Estimation-Based Feature Selection and Multinomial Logistic Regression for Classification of Bearing Faults Using Compressively Sampled Vibration Signals.

As failures of rolling bearings lead to major failures in rotating machines, recent vibration-based rolling bearing fault diagnosis techniques are focused on obtaining useful fault features from the huge collection of raw data. However, too many features reduce the classification accuracy and increase the computation time. This paper proposes an effective feature selection technique based on intrinsic dimension estimation of compressively sampled vibration signals. First, compressive sampling (CS) is used to get compressed measurements from the collected raw vibration signals. Then, a global dimension estimator, the geodesic minimal spanning tree (GMST), is employed to compute the minimal number of features needed to represent efficiently the compressively sampled signals. Finally, a feature selection process, combining the stochastic proximity embedding (SPE) and the neighbourhood component analysis (NCA), is used to select fewer features for bearing fault diagnosis. With regression analysis-based predictive modelling technique and the multinomial logistic regression (MLR) classifier, the selected features are assessed in two case studies of rolling bearings vibration signals under different working loads. The experimental results demonstrate that the proposed method can successfully select fewer features, with which the MLR-based trained model achieves high classification accuracy and significantly reduced computation times compared to published research.

The role of intrinsic dimension in high-resolution player tracking data—Insights in basketball

Following the introduction of high-resolution player tracking technology, a new range of statistical analysis has emerged in sports, specifically in basketball. However, such high-dimensional data are often challenging for statistical inference and decision making. In this article we employ a state-of-the-art Bayesian mixture model that allows the estimation of heterogeneous intrinsic dimension (ID) within a dataset, and we propose some theoretical enhancements. Informally, the ID can be seen as an indicator of complexity and dependence of the data at hand, and it is usually assumed unique. Our method provides the capacity to reveal valuable insights about the hidden dynamics of sports interactions in space and time which helps to translate complex patterns into more coherent statistics. The application of this technique is illustrated using NBA basketball players’ tracking data, allowing effective classification and clustering. In movement data the analysis identified key stages of offensive actions, such as creating space for passing, preparation/shooting, and following through which are relevant for invasion sports. We found that the ID value spikes, reaching a peak between four and eight seconds in the offensive part of the court, after which it declines. In shot charts we obtained groups of shots that produce substantially higher and lower successes. Overall, game-winners tend to have a larger intrinsic dimension, indicative of greater unpredictability and unique shot placements. Similarly, we found higher ID values in plays when the score margin is smaller rather than larger. The exploitation of these results can bring clear strategic advantages in sports games.

Intrinsic dimension estimation method based on correlation dimension and kNN method

In practical problems, high-dimensional data usually has a low-dimensional structure, or the data is located on a low-dimensional manifold. The dimension of this manifold is called the intrinsic dimension of the data. There are many intrinsic dimension estimation methods, among which methods based on the correlation dimension have received extensive attention. However, correlation dimension-based estimation methods often provide a dimension lower than the true intrinsic dimension of the dataset. To explore the reasons behind underestimation, the probabilities of underestimation, overestimation and proper estimation are analyzed using order statistics. The analysis results show that the probability of underestimation is much higher than that of the other two cases, and is verified by simulation experiments. Based on the above analysis, a new method for the estimation of the intrinsic dimension is proposed based on the correlation dimension and k-nearest neighbor method (kNN), which effectively reduces the underestimation. This method is implemented using two algorithms, namely a search algorithm and a matching algorithm. Comprehensive experimental studies on simulation datasets and real datasets show that the proposed algorithms are more effective than the comparison methods.

Scikit-Dimension: A Python Package for Intrinsic Dimension Estimation.

Dealing with uncertainty in applications of machine learning to real-life data critically depends on the knowledge of intrinsic dimensionality (ID). A number of methods have been suggested for the purpose of estimating ID, but no standard package to easily apply them one by one or all at once has been implemented in Python. This technical note introduces scikit-dimension, an open-source Python package for intrinsic dimension estimation. The scikit-dimension package provides a uniform implementation of most of the known ID estimators based on the scikit-learn application programming interface to evaluate the global and local intrinsic dimension, as well as generators of synthetic toy and benchmark datasets widespread in the literature. The package is developed with tools assessing the code quality, coverage, unit testing and continuous integration. We briefly describe the package and demonstrate its use in a large-scale (more than 500 datasets) benchmarking of methods for ID estimation for real-life and synthetic data.

Intrinsic dimension estimation based on local adjacency information

The intrinsic dimension (ID) of a data set is crucial for data processing, especially for high-dimensional data sets. In order to obtain an accurate ID estimate, two neighborhoods of sample points with a radius ratio of k are considered. The ratio of the number of sample points contained in the two neighborhoods is denoted as q∊. When the data set is located on a d-dimensional submanifold in RD, the expected value of q∊ is kd. Based on this consideration, we redefine the adjacency matrix by using the local adjacency information of sample points and propose a new ID estimation method known as ID(k). The ID(k) algorithm contains only one parameter, the scaling ratio k, and we outline the criterion through which the user can select an appropriate k value. We demonstrate the convergence of the new method both theoretically and experimentally. Experimental results from artificial and real data sets show that the estimates obtained by this new ID(k) method are closer to the true intrinsic dimension than those derived using similar methods.

Locality adaptive preserving projections for linear dimensionality reduction

Dimensionality reduction techniques aim to transform the high-dimensional data into a meaningful reduced representation and have been consistently playing a fundamental role in the study of intrinsic dimensionality estimation and the design of an intelligent expert system towards real-world applications. From the perspective of manifold learning, locality preserving projections is a classical and commonly used dimensionality reduction method and it essentially learns the low-dimensional embedding under the constraint of preserving the local geometry of data. However, since it determines the neighborhood relationships in the original feature space that probably contains noisy and irrelevant features, the derived similarity between the neighbors are unreliable and the corresponding local data manifold tends to be error-prone, which inevitably leads to degraded performance for subsequent data analyses. Hence, how to accurately identify the true neighbor relationships for each sample remains crucial to the robustness improvement. In this work, we propose a novel approach, termed locality adaptive preserving projections (LAPP), to adaptively determine the neighbors and their relationships in the optimal subspace rather than in the original space. Specifically, due to the absence of prior knowledge of local properties of the underlying manifold, LAPP adopts a coarse-to-fine strategy to iteratively update the projected low-dimensional subspace and optimize the identification of the local structure of the data. Moreover, an iterative algorithm with fast convergence is utilized to solve the transformation matrix for explicit out-of-sample extension. Besides, LAPP is easy to implement and its key idea can be potentially extended to other methods for neighbor-finding and similarity measurement. To evaluate the performance of LAPP, we conduct comparative experiments on numerous synthetic and real-world datasets. Experimental results show that seeking the local structure in the original feature space misleads the selection of neighbors and the calculation of similarity and that the proposed method helps alleviate the negative effect of noisy and irrelevant features, which demonstrates its effectiveness. Besides, this study has the potential to enlighten relevant studies to consider the problem of optimizing the neighborhood relationships.

Spectral Variability Aware Blind Hyperspectral Image Unmixing Based on Convex Geometry.

Hyperspectral image unmixing has proven to be a useful technique to interpret hyperspectral data, and is a prolific research topic in the community. Most of the approaches used to perform linear unmixing are based on convex geometry concepts, because of the strong geometrical structure of the linear mixing model. However, many algorithms based on convex geometry are still used in spite of the underlying model not considering the intra-class variability of the materials. A natural question is to wonder to what extent these concepts and tools (Intrinsic Dimensionality estimation, endmember extraction algorithms, pixel purity) are still relevant when spectral variability comes into play. In this paper, we first analyze their robustness in a case where the linear mixing model holds in each pixel, but the endmembers vary in each pixel according to a prescribed variability model. In the light of this analysis, we propose an integrated unmixing chain which tries to adress the shortcomings of the classical tools used in the linear case, based on our previously proposed extended linear mixing model. We show the interest of the proposed approach on simulated and real datasets.

Bias-Compensated Estimator for Intrinsic Dimension and Differential Entropy: A Visual Multiscale Approach

Intrinsic dimension and differential entropy estimators are studied in this paper, including their systematic bias. A pragmatic approach for joint estimation and bias correction of these two fundamental measures is proposed. Shared steps on both estimators are highlighted, along with their usefulconsequences to data analysis. It is shown that both estimators can be complementary parts of a single approach, and that the simultaneous estimation of differential entropy and intrinsic dimension give meaning to each other, where estimates at different observation scales convey different perspectives of underlyingmanifolds. Experiments with synthetic and real datasets are presented to illustrate how to extract meaning from visual inspections, and how to compensate for biases.

Intrinsic dimension estimation for locally undersampled data.

Identifying the minimal number of parameters needed to describe a dataset is a challenging problem known in the literature as intrinsic dimension estimation. All the existing intrinsic dimension estimators are not reliable whenever the dataset is locally undersampled, and this is at the core of the so called curse of dimensionality. Here we introduce a new intrinsic dimension estimator that leverages on simple properties of the tangent space of a manifold and extends the usual correlation integral estimator to alleviate the extreme undersampling problem. Based on this insight, we explore a multiscale generalization of the algorithm that is capable of (i) identifying multiple dimensionalities in a dataset, and (ii) providing accurate estimates of the intrinsic dimension of extremely curved manifolds. We test the method on manifolds generated from global transformations of high-contrast images, relevant for invariant object recognition and considered a challenge for state-of-the-art intrinsic dimension estimators.

Improved Estimation of the Intrinsic Dimension of a Hyperspectral Image Using Random Matrix Theory

Many methods have been proposed in the literature for estimating the number of materials/endmembers in a hyperspectral image. This is sometimes called the “intrinsic” dimension (ID) of the image. A number of recent papers have proposed ID estimation methods based on various aspects of random matrix theory (RMT), under the assumption that the errors are uncorrelated, but with possibly unequal variances. A recent paper, which reviewed a number of the better known methods (including one RMT-based method), has shown that they are all biased, especially when the true ID is greater than about 20 or 30, even when the error structure is known. I introduce two RMT-based estimators ( R M T G , which is new, and R M T K N , which is a modification of an existing estimator), which are approximately unbiased when the error variances are known. However, they are biased when the error variance is unknown and needs to be estimated. This bias increases as ID increases. I show how this bias can be reduced. The results use semi-realistic simulations based on three real hyperspectral scenes. Despite this, when applied to the real scenes, R M T G and R M T K N are larger than expected. Possible reasons for this are discussed, including the presence of errors which are either deterministic, spectrally and/or spatially correlated, or signal-dependent. Possible future research into ID estimation in the presence of such errors is outlined.

Process Monitoring Based on Orthogonal Locality Preserving Projection with Maximum Likelihood Estimation

By integrating two powerful methods of density reduction and intrinsic dimensionality estimation, a new data-driven method, referred to as OLPP-MLE (orthogonal locality preserving projection-maximum likelihood estimation), is introduced for process monitoring. OLPP is utilized for dimensionality reduction, which provides better locality preserving power than locality preserving projection. Then, the MLE is adopted to estimate intrinsic dimensionality of OLPP. Within the proposed OLPP-MLE, two new static measures for fault detection $T_{\scriptscriptstyle {OLPP}}^2$ and ${\rm SPE}_{\scriptscriptstyle {OLPP}}$ are defined. In order to reduce algorithm complexity and ignore data distribution, kernel density estimation is employed to compute thresholds for fault diagnosis. The effectiveness of the proposed method is demonstrated by three case studies.

Basic Units of Inter-Individual Variation in Resting State Connectomes

Resting state functional connectomes are massive and complex. It is an open question, however, whether connectomes differ across individuals in a correspondingly massive number of ways, or whether most differences take a small number of characteristic forms. We systematically investigated this question and found clear evidence of low-rank structure in which a modest number of connectomic components, around 50–150, account for a sizable portion of inter-individual connectomic variation. This number was convergently arrived at with multiple methods including estimation of intrinsic dimensionality and assessment of reconstruction of out-of-sample data. In addition, we show that these connectomic components enable prediction of a broad array of neurocognitive and clinical symptom variables at levels comparable to a leading method that is trained on the whole connectome. Qualitative observation reveals that these connectomic components exhibit extensive community structure reflecting interrelationships between intrinsic connectivity networks. We provide quantitative validation of this observation using novel stochastic block model-based methods. We propose that these connectivity components form an effective basis set for quantifying and interpreting inter-individual connectomic differences, and for predicting behavioral/clinical phenotypes.