Local Intrinsic Dimension Research Articles

NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension

One-shot neural architecture search (NAS) substantially improves the search efficiency by training one supernet to estimate the performance of every possible child architecture (i.e., subnet). However, the inconsistency of characteristics among subnets incurs serious interference in the optimization, resulting in poor performance ranking correlation of subnets. Subsequent explorations decompose supernet weights via a particular criterion, e.g., gradient matching, to reduce the interference; yet they suffer from huge computational cost and low space separability. In this work, we propose a lightweight and effective local intrinsic dimension (LID)-based method NAS-LID. NAS-LID evaluates the geometrical properties of architectures by calculating the low-cost LID features layer-by-layer, and the similarity characterized by LID enjoys better separability compared with gradients, which thus effectively reduces the interference among subnets. Extensive experiments on NASBench-201 indicate that NAS-LID achieves superior performance with better efficiency. Specifically, compared to the gradient-driven method, NAS-LID can save up to 86% of GPU memory overhead when searching on NASBench-201. We also demonstrate the effectiveness of NAS-LID on ProxylessNAS and OFA spaces. Source code:https://github.com/marsggbo/NAS-LID.

Local Intrinsic Dimensional Entropy

Most entropy measures depend on the spread of the probability distribution over the sample space |X|, and the maximum entropy achievable scales proportionately with the sample space cardinality |X|. For a finite |X|, this yields robust entropy measures which satisfy many important properties, such as invariance to bijections, while the same is not true for continuous spaces (where |X|=infinity). Furthermore, since R and R^d (d in Z+) have the same cardinality (from Cantor's correspondence argument), cardinality-dependent entropy measures cannot encode the data dimensionality. In this work, we question the role of cardinality and distribution spread in defining entropy measures for continuous spaces, which can undergo multiple rounds of transformations and distortions, e.g., in neural networks. We find that the average value of the local intrinsic dimension of a distribution, denoted as ID-Entropy, can serve as a robust entropy measure for continuous spaces, while capturing the data dimensionality. We find that ID-Entropy satisfies many desirable properties and can be extended to conditional entropy, joint entropy and mutual-information variants. ID-Entropy also yields new information bottleneck principles and also links to causality. In the context of deep learning, for feedforward architectures, we show, theoretically and empirically, that the ID-Entropy of a hidden layer directly controls the generalization gap for both classifiers and auto-encoders, when the target function is Lipschitz continuous. Our work primarily shows that, for continuous spaces, taking a structural rather than a statistical approach yields entropy measures which preserve intrinsic data dimensionality, while being relevant for studying various architectures.

Scikit-Dimension: A Python Package for Intrinsic Dimension Estimation.

Dealing with uncertainty in applications of machine learning to real-life data critically depends on the knowledge of intrinsic dimensionality (ID). A number of methods have been suggested for the purpose of estimating ID, but no standard package to easily apply them one by one or all at once has been implemented in Python. This technical note introduces scikit-dimension, an open-source Python package for intrinsic dimension estimation. The scikit-dimension package provides a uniform implementation of most of the known ID estimators based on the scikit-learn application programming interface to evaluate the global and local intrinsic dimension, as well as generators of synthetic toy and benchmark datasets widespread in the literature. The package is developed with tools assessing the code quality, coverage, unit testing and continuous integration. We briefly describe the package and demonstrate its use in a large-scale (more than 500 datasets) benchmarking of methods for ID estimation for real-life and synthetic data.

Data segmentation based on the local intrinsic dimension

One of the founding paradigms of machine learning is that a small number of variables is often sufficient to describe high-dimensional data. The minimum number of variables required is called the intrinsic dimension (ID) of the data. Contrary to common intuition, there are cases where the ID varies within the same data set. This fact has been highlighted in technical discussions, but seldom exploited to analyze large data sets and obtain insight into their structure. Here we develop a robust approach to discriminate regions with different local IDs and segment the points accordingly. Our approach is computationally efficient and can be proficiently used even on large data sets. We find that many real-world data sets contain regions with widely heterogeneous dimensions. These regions host points differing in core properties: folded versus unfolded configurations in a protein molecular dynamics trajectory, active versus non-active regions in brain imaging data, and firms with different financial risk in company balance sheets. A simple topological feature, the local ID, is thus sufficient to achieve an unsupervised segmentation of high-dimensional data, complementary to the one given by clustering algorithms.

Application of Local Intrinsic Dimension for Acoustical Analysis of Voice Signal Components.

The overall aim of this study was to apply local intrinsic dimension ( Di) estimation to quantify high-dimensional, disordered voice and discriminate between the 4 types of voice signals. It was predicted that continuous Di analysis throughout the entire time-series would generate comprehensive descriptions of voice signal components, called voice type component profiles (VTCP), that effectively distinguish between the 4 voice types. One hundred thirty-five voice recording samples of the sustained vowel /a/ were obtained from the Disordered Voice Database Model 4337 and spectrographically classified into the voice type paradigm. The Di and correlation dimension ( D2) were then used to objectively analyze the voice samples and compared based on voice type differentiation efficacy. The D2 exhibited limited effectiveness in distinguishing between the 4 voice type signals. For Di analysis, significant differences were primarily observed when comparing voice type component 1 (VTC1) and 4 (VTC4) across the 4 voice type signals ( P < .001). The 4 voice type components (VTCs) significantly differentiated between low-dimensional, type 3 and high-dimensional, type 4 signals ( P < .001). The Di demonstrated improvements over D2 in 2 distinct manners: enhanced resolution at high data dimensions and comprehensive description of voice signal elements.

Local Intrinsic Dimension Estimation by Generalized Linear Modeling.

We propose a method for intrinsic dimension estimation. By fitting the power of distance from an inspection point and the number of samples included inside a ball with a radius equal to the distance, to a regression model, we estimate the goodness of fit. Then, by using the maximum likelihood method, we estimate the local intrinsic dimension around the inspection point. The proposed method is shown to be comparable to conventional methods in global intrinsic dimension estimation experiments. Furthermore, we experimentally show that the proposed method outperforms a conventional local dimension estimation method.

Low Bias Local Intrinsic Dimension Estimation from Expected Simplex Skewness.

In exploratory high-dimensional data analysis, local intrinsic dimension estimation can sometimes be used in order to discriminate between data sets sampled from different low-dimensional structures. Global intrinsic dimension estimators can in many cases be adapted to local estimation, but this leads to problems with high negative bias or high variance. We introduce a method that exploits the curse/blessing of dimensionality and produces local intrinsic dimension estimators that have very low bias, even in cases where the intrinsic dimension is higher than the number of data points, in combination with relatively low variance. We show that our estimators have a very good ability to classify local data sets by their dimension compared to other local intrinsic dimension estimators; furthermore we provide examples showing the usefulness of local intrinsic dimension estimation in general and our method in particular for stratification of real data sets.

Dimension estimation of image manifolds by minimal cover approximation

Estimating intrinsic dimension of data is an important problem in feature extraction and feature selection. It provides an estimation of the number of desired features. Principal Components Analysis (PCA) is a powerful tool in discovering the dimension of data sets with a linear structure; it, however, becomes ineffective when data have a nonlinear structure. In this paper, we propose a new PCA-based method to estimate the embedding dimension of data with nonlinear structures. Our method works by first finding a minimal cover of the data set, then performing PCA locally on each subset in the cover to obtain local intrinsic dimension estimations and finally giving the estimation result as the average of the local estimations. There are two main innovations in our method. (1) A novel noise filtering procedure is applied in the PCA procedure for local intrinsic dimension estimation. (2) A minimal cover is constructed over the whole data set. Because of these two innovations, our method is fast, robust to noise and outliers, converges to a stable estimation with a wide range of sub-region sizes and can be used in the incremental sense, where the subregion refers to the local approximation of the distributed manifold. Experiments on synthetic and image data sets show effectiveness of the proposed method.

On Local Intrinsic Dimension Estimation and Its Applications

In this paper, we present multiple novel applications for local intrinsic dimension estimation. There has been much work done on estimating the global dimension of a data set, typically for the purposes of dimensionality reduction. We show that by estimating dimension locally, we are able to extend the uses of dimension estimation to many applications, which are not possible with global dimension estimation. Additionally, we show that local dimension estimation can be used to obtain a better global dimension estimate, alleviating the negative bias that is common to all known dimension estimation algorithms. We illustrate local dimension estimation's uses towards additional applications, such as learning on statistical manifolds, network anomaly detection, clustering, and image segmentation.

Efficient visual coding and the predictability of eye movements on natural movies

We deal with the analysis of eye movements made on natural movies in free-viewing conditions. Saccades are detected and used to label two classes of movie patches as attended and non-attended. Machine learning techniques are then used to determine how well the two classes can be separated, i.e., how predictable saccade targets are. Although very simple saliency measures are used and then averaged to obtain just one average value per scale, the two classes can be separated with an ROC score of around 0.7, which is higher than previously reported results. Moreover, predictability is analysed for different representations to obtain indirect evidence for the likelihood of a particular representation. It is shown that the predictability correlates with the local intrinsic dimension in a movie.

Application of methods based on higher-order statistics for chaotic time series analysis

The aim of this paper is to illustrate some applications of HOS within the context of chaotic time series analysis. After reviewing briefly some of the most popular methods and approaches used in chaotic signal analysis, we show how HOS may lead to some significant improvement. First, an HOS expansion of the mutual information is shown to provide an easy way to estimate the reconstruction delay that must be used in the embedding reconstruction method. Then, a fourth-order extension of the local intrinsic dimension analysis (LID) is proposed. The ability of this HOS extension to separate between chaotic and stochastic behaviour is illustrated by examples on simulated data and experimental time series.

Estimation of attractor dimension of EEG using singular value decomposition

This paper describes a novel application of singular value decomposition (SVD) of subsets of the phase-space trajectory for calculation of the attractor dimension of a small data set. A certain number of local centres (M) are chosen randomly on the attractor and an adequate number of nearest neighbours (q=50) are ordered around each centre. The local intrinsic dimension of a local centre is determined by the number of significant singular values and the attractor dimension (D 2) by the average of the local intrinsic dimensions of the local centres. The SVD method has been evaluated for model data and EEG. The results indicate that the SVD method is a reliable approach for estimation of attractor dimension at moderate signal to noise ratios. The paper emphasises the importance of SVD approach to EEG analysis.

Characterising experimental time series using local intrinsic dimension

Experimental strange attractors are analysed with the averaged local intrinsic dimension proposed by A. Passamante et al. [Phys. Rev. A 39 (1989) 3640] which is based on singular value decomposition of local trajectory matrices. The results are compared to the values of Kaplan-Yorke and the correlation dimension. The attractors, reconstructed with Takens' delay time coordinates from scalar velocity time series, are measured in the hydrodynamic Taylor-Couette system. A period doubling route towards chaos obtained from a very short Taylor-Couette cylinder yields a sequence of experimental time series where the local intrinsic dimension is applied.

Characterizing attractors using local intrinsic dimension via higher-order statistics.

Characterizing attractors using local intrinsic dimension via higher-order statistics.

Chaos in a bioacoustic signal

Zebra finches (Poephila guttata) emit sexually dimorphic, multiharmonic distance calls. These calls exhibit deterministic chaos. This was demonstrated by first digitizing the time series of the calls of five adult birds of each sex, then embedding a stationary portion of each, selecting an appropriate time delay (tau) value, and generating three‐dimensional phase plots. From these plots, strange attractors were reconstructed. The morphology of these attractors is different for each bird and highly repeatable across calls for a given individual. Correlation dimension and local intrinsic dimension values were also calculated and support the conclusion that the signals are chaotic. The avian vocal tract has been modeled as a nonlinear system [J. H. Brackenbury, J. Theor. Biol. 81, 341–349 (1979); N. H. Fletcher, J. Theor. Biol. 135, 455–481 (1988)]. Its nonlinear dynamics are the probable cause of the chaotic properties of the signals. It is suggested that chaos may be a common property of bioacoustic signals produced by systems which utilize fluid forces to generate sound. It has not yet been determined if the birds can distinguish between sounds based on their chaotic properties.

Characterizing attractors using local intrinsic dimensions calculated by singular-value decomposition and information-theoretic criteria.

An algorithm to estimate the average local intrinsic dimension (〈${\mathit{d}}_{\mathrm{LID}}$〉) of an attractor using signal versus noise separation methods based on information-theoretic criteria is explored in this work. Using noisy sample data the 〈${\mathit{d}}_{\mathrm{LID}}$〉 is computed from an eigenanalysis of local attractor regions, indicating the local orthogonal directions along which the data are clustered. The 〈${\mathit{d}}_{\mathrm{LID}}$〉 algorithm requires the separation of signal eigenvalues, i.e., the dominant eigenvalues, from the noise eigenvalues for which thresholding mechanisms based on two information-theoretic criteria are used. Singular-value decomposition is used to calculate the eigenvalues as well as to determine the rank of the local-phase-space data matrix. The two specific information-theoretic criteria which we consider to separate signal and noise are the Akaike information criterion (AIC) [H. Akaike, IEEE Trans. Auto. Control 19, 716 (1974)] and the minimum description length (MDL) of Rissanen [Automatica 14, 465 (1978)] and Schwarz [Ann. Stat. 6, 461 (1978)]. Several test cases are presented demonstrating the use of the AIC and MDL to calculate 〈${\mathit{d}}_{\mathrm{LID}}$〉. Results are then compared to the correlation dimension as computed by the Grassberger-Procaccia method [Physica D9, 189 (1983)]. The AIC and MDL separation techniques are found to be inappropriate at higher signal-to-noise ratios (SNR), above \ensuremath{\sim}12 dB, and the MDL produces results in good overall agreement with the accepted correlation dimension in the range of SNR between 5 and 12 dB. Two of the primary benefits of the 〈${\mathit{d}}_{\mathrm{LID}}$〉 technique are that it will yield reasonable and fairly accurate estimates of the dimension of attractors and that it can probe small length scales on the attractor even for moderate SNR. Other thresholding schemes are being pursued for SNR's above 12 dB.

Fractal dimension and local intrinsic dimension.

An algorithm to estimate the average local intrinsic dimension of an attractor is described. Of particular concern is the ability to estimate local attractor dimensionality at moderate values of signal-to-noise ratio (SNR). The algorithm is based on the estimation of the local intrinsic dimension (LID) of various randomly chosen regions around the attractor. The LID is determined by the number of significant singular values, or rank, of the data matrix for data points within the local region. The rank of this matrix is an integer and, in general, varies with position on the attractor. To obtain an overall estimate of the local attractor dimension a weighted average of all the local regions is calculated. Results are then compared to conventional estimates of fractal dimension. For high SNR results are in good agreement with the fractal dimension as calculated by the Grassberger-Procaccia algorithm. Limited test results indicate that the LID technique may be more robust when trying to estimate, or bound, the fractal dimension in the presence of moderate amounts of noise.