Structure Of High-dimensional Data Research Articles

Tertiary hash tree-based index structure for high dimensional multimedia data

Dominant features for the content-based image retrieval usually have high-dimensionality. So far, many researches have been done to index such values to support fast retrieval. Still, many existing indexing schemes are suffering from performance degradation due to the curse of dimensionality problem. As an alternative, heuristic algorithms have been proposed to calculate the answer with ‘high probability’ at the cost of accuracy. In this paper, we propose a new hash tree-based indexing structure called tertiary hash tree for indexing high-dimensional feature data. Tertiary hash tree provides several advantages compared to the traditional extendible hash structure in terms of resource usage and search performance. Through extensive experiments, we show that our proposed index structure achieves outstanding performance.

Dependence of locally linear embedding on the regularization parameter

This paper deals with a method, called locally linear embedding (LLE). It is a nonlinear dimensionality reduction technique that computes low-dimensional, neighborhood-preserving embeddings of high-dimensional data and attempts to discover a nonlinear structure (including manifolds) in high-dimensional data. In practice, the nonlinear manifold learning methods are applied in image processing, text mining, etc. The implementation of the LLE algorithm is fairly straightforward, because the algorithm has only two control parameters: the number of neighbors of each data point and the regularization parameter. The mapping quality is quite sensitive to these parameters. In this paper, we propose a new way of selecting a regularization parameter of a local Gram matrix.

A Fully Automated Method for Discovering Community Structures in High Dimensional Data.

Identifying modules, or natural communities, in large complex networks is fundamental in many fields, including social sciences, biological sciences and engineering. Recently several methods have been developed to automatically identify communities from complex networks by optimizing the modularity function. The advantage of this type of approaches is that the algorithm does not require any parameter to be tuned. However, the modularity-based methods for community discovery assume that the network structure is given explicitly and is correct. In addition, these methods work best if the network is unweighted and/or sparse. In reality, networks are often not directly defined, or may be given as an affinity matrix. In the first case, each node of the network is defined as a point in a high dimensional space and different networks can be obtained with different network construction methods, resulting in different community structures. In the second case, an affinity matrix may define a dense weighted graph, for which modularity-based methods do not perform well. In this work, we propose a very simple algorithm to automatically identify community structures from these two types of data. Our approach utilizes a k-nearest-neighbor network construction method to capture the topology embedded in high dimensional data, and applies a modularity-based algorithm to identify the optimal community structure. A key to our approach is that the network construction is incorporated with the community identification process and is totally parameter-free. Furthermore, our method can suggest appropriate preprocessing/normalization of the data to improve the results of community identification. We tested our methods on several synthetic and real data sets, and evaluated its performance by internal or external accuracy indices. Compared with several existing approaches, our method is not only fully automatic, but also has the best accuracy overall.

Finding large average submatrices in high dimensional data

The search for sample-variable associations is an important problem in the exploratory analysis of high dimensional data. Biclustering methods search for sample-variable associations in the form of distinguished submatrices of the data matrix. (The rows and columns of a submatrix need not be contiguous.) In this paper we propose and evaluate a statistically motivated biclustering procedure (LAS) that finds large average submatrices within a given real-valued data matrix. The procedure operates in an iterative-residual fashion, and is driven by a Bonferroni-based significance score that effectively trades off between submatrix size and average value. We examine the performance and potential utility of LAS, and compare it with a number of existing methods, through an extensive three-part validation study using two gene expression datasets. The validation study examines quantitative properties of biclusters, biological and clinical assessments using auxiliary information, and classification of disease subtypes using bicluster membership. In addition, we carry out a simulation study to assess the effectiveness and noise sensitivity of the LAS search procedure. These results suggest that LAS is an effective exploratory tool for the discovery of biologically relevant structures in high dimensional data. Software is available at this https URL

The Mathematics and Statistics of Quantitative Risk Management

The Mathematics and Statistics of Quantitative Risk Management Workshop , organized by Thomas Mikosch (Copenhagen), Richard A. Davis (New York), and Paul Embrechts (Zürich), was held March 16th–March 22nd, 2008. This meeting was well attended with over 40 participants from four continents. This workshop was a blend of researchers with various backgrounds in mathematical finance, statistics, econometrics, extreme value theory, applied probability, and insurance. Modern quantitative risk management integrates a wide range of sophisticated mathematical techniques and tools. An overview from the statistical side is given in the recent monograph by McNeil, Frey, Embrechts. Relevant areas of research include the theory of high-dimensional data structures; rare event simulation; theory of risk measures; (multivariate) time series analysis; extreme event modeling and extreme value statistics; optimization; and linear, quadratic, and convex programming. Recent questions related to multi-period risk measures involve deep results from a variety of fields. Functional data analysis is instrumental for designing and analyzing risk measures, a geometric theory of extremes is useful for the analysis of generalized risk scenarios, Malliavin calculus has become important for the calculation of risk measure sensitivities, functional regular variation is a relevant concept for analyzing stochastic processes exhibiting extreme behavior, advanced rare event simulation techniques, numerical and optimization methods, Lévy processes and more general diffusions are the building blocks for constructing dynamic stochastic models in finance and econometrics. As evidenced by the recent upheavals in the markets and financial institutions, there is a pressing and critical need to develop and refine tools and methods in quantitative risk management. Expanding on the theory in quantitative risk management should have immediate impact for the financial and insurance industries as well as for supervisory authorities. The objective is to design mathematically tractable, practically relevant and statistically estimable risk measures. An advanced theory also allows one to critically study the present use of tools and methods in quantitative risk management. Risks in insurance and finance are described by mathematical and probabilistic models such as partial differential equations and stochastic differential equations describing the evolution of prices of risky assets – price of stock, composite stock indices, interest rates, foreign exchange rates, commodity prices – or difference equations describing the evolution of financial returns. The 2003 Nobel prize winning ARCH model is an outstanding example. Applications of these models require advanced simulation and numerical methods and statistics plays a vital role in the estimation of unknown parameters (possibly infinite dimensional) from historical data. Due to their complexity, problems of quantitative risk management require multidisciplinary solutions. They involve functional analysts who design and analyze risk measures, probabilists who model with stochastic differential equations and time series, applied probabilists who solve the simulation problems, numerical analysts who deal with high-dimensional integration and optimization problems, and statisticians who fit stochastic models to the data and predict future values of risky assets. Among the challenging problems which were discussed at the meeting are: Some of the main objectives of the workshop are summarized here:

Application of Penalized Splines in Analyzing Neuronal Data

Neuron experiments produce high-dimensional data structures. Therefore, application of smoothing techniques in the analysis of neuronal data from electrophysiological experiments has received considerable attention of late. We investigate the use of penalized splines in the analysis of neuronal data. This is first illustrated when interested in the temporal trend of a single neuron. An approach to investigate the maximal firing rate, based on the penalizedspline model is proposed. Determination of the time of maximal firing rate is based on non-linear optimization of the objective function with the corresponding confidence intervals constructed based on the first-order derivative function. To distinguish between the curves from different experimental conditions in a moment-by-moment sense, bias adjusted simulation-based simultaneous confidence bands leading to global inference in the time domain are constructed. The bands are an extension of the approach proposed by Ruppert et al. (2003). These methods are in a second step extended towards the analysis of a population of neurons via a marginal or population-averaged model.

Parallelized prediction error estimation for evaluation of high-dimensional models

There is a multitude of new techniques that promise to extract predictive information in bioinformatics applications. It has been recognized that a first step for validation of the resulting model fits should rely on proper use of resampling techniques. However, this advice is frequently not followed, potential reasons being difficulty of correct implementation and computational demand. This is addressed by the R package peperr, which is designed for reliable prediction error estimation through resampling, potentially accelerated by parallel execution on a compute cluster. Its interface allows easy connection to newly developed model fitting routines. Performance evaluation of the latter is furthermore guided by diagnostic plots, which helps to detect specific problems due to high-dimensional data structures. http://cran.r-project.org, http://www.imbi.uni-freiburg.de/parallel. Supplementary data are available at Bioinformatics online.

Speaker recognition using local models

A system and method for voice recognition is disclosed. The system enrolls speakers using an enrollment voice samples and identification information. An extraction module (220) characterizes enrollment voice samples with high-dimensional feature vectors or speaker data points. A data structuring module (230) organizes data points into a high-dimensional data structure, such as a kd-tree, in which similarity between data points dictates a distance, such as a Euclidean distance, a Minkowski distance, or a Manhattan distance. The system recognizes a speaker using an unidentified voice sample. A data querying module (242) searches the data structure to generate a subset of approximate nearest neighbors based on an extracted high-dimensional feature vector. A data modeling module (244) uses Parzen windows to estimate a probability density function representing how closely characteristics of the unidentified speaker match enrolled speakers, in real-time, without extensive training data or parametric assumptions about data distribution. A smoothing parameter controls the relative contributions of close and far speaker data points to the estimated density.

Perception as Modeling: A neural network that extracts and models predictable elements from input.

Event Abstract Back to Event Perception as Modeling: A neural network that extracts and models predictable elements from input. Perception is not simply the passive analysis of input data, it involves modeling the external world, making inferences about predictable events and noting when something unexpected happens. In most models of sensory processing, activity generated by external stimuli in peripheral sensors is used to drive networks that perform computations such as object recognition. We consider a different scheme in which stimulus driven activity acts as a training signal for a spontaneously active network. This 'predictor' network continually attempts to model the training data in the sense of being able to generate it spontaneously, even if the sensory input is cut off. Of course, only certain aspects of the sensory data may be predictable. To address this, an 'extractor' circuit that is guided by feedback from the predictor produces the training signal for the predictor network. The extractor pulls out from the sensory stream aspects of the data that the predictor network can reproduce. This automatically divides the input data into three categories: 1) noise, which is the part of the input stream ignored by the extractor, 2) a predictable signal that is isolated by the extractor circuit and internally reproduced by the predictor, and 3) surprising events, which are elements of the input stream generated by the extractor that do not match the output of the predictor. The predictor is a recurrent network of firing-rate model neurons with a linear readout that provides both the output of and feedback to the network. As in the work of Jaeger and Haas (Science, 2004), the learning algorithm only modifies the weights connecting the network to the output unit, but unlike their approach, we leave all feedback intact during learning. The key element is a novel learning rule we call FORCE learning that restricts errors in the output to small values throughout training. This was originally a supervised learning scheme, but in the extractor-predictor approach, the extractor circuit acts as the supervisor for the predictor network, so the scheme is unsupervised. Due to the nature of FORCE learning, the modeling network always generates a close match to the target and thus does not generate 'hallucinations' unrelated to external reality. The rate of change of the readout weights provides a measure of whether the target function can be generated autonomously by the predictor. We use this measure as a supervisory signal for the extractor, which is modified if the target signal it is extracting from the input data cannot be autonomously modeled. The combined recurrent predictor network and linear-filter extractor is successful at finding and modeling predictable structure (if it exists) in high-dimensional time series data even when polluted by extremely complex noise. This network can be viewed as a general method for extracting patterns from complicated time series or, from a systems neuroscience perspective, as a model of perception where the way to understand the world is through internal prediction, with further processing used only when sensory input fails to match expectations. Conference: Computational and systems neuroscience 2009, Salt Lake City, UT, United States, 26 Feb - 3 Mar, 2009. Presentation Type: Poster Presentation Topic: Poster Presentations Citation: (2009). Perception as Modeling: A neural network that extracts and models predictable elements from input.. Front. Syst. Neurosci. Conference Abstract: Computational and systems neuroscience 2009. doi: 10.3389/conf.neuro.06.2009.03.253 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 03 Feb 2009; Published Online: 03 Feb 2009. Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract Supplemental Data The Authors in Frontiers Google Google Scholar PubMed Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Kernel Sliced Inverse Regression with Applications to Classification

Sliced inverse regression (SIR) was introduced by Li to find the effective dimension reduction directions for exploring the intrinsic structure of high-dimensional data. In this study, we propose a hybrid SIR method using a kernel machine which we call kernel SIR. The kernel mixtures result in the transformed data distribution being more Gaussian like and symmetric; providing more suitable conditions for performing SIR analysis. The proposed method can be regarded as a nonlinear extension of the SIR algorithm. We provide a theoretical description of the kernel SIR algorithm within the framework of reproducing kernel Hilbert space (RKHS). We also illustrate that kernel SIR performs better than several standard methods for discriminative, visualization, and regression purposes. We show how the features found with kernel SIR can be used for classification of microarray data and several other classification problems and compare the results with those obtained with several existing dimension reduction techniques. The results show that kernel SIR is a powerful nonlinear feature extractor for classification problems.

Noisy manifold learning using neighborhood smoothing embedding

Manifold learning is an important dimensionality reduction tool that discovers the structure of high dimensional data and provides understanding of multidimensional patterns in data mining, pattern recognition, and machine learning. Several manifold learning algorithms are applied to extract the intrinsic features of different prototypes in high dimensional space by preserving the local geometric characteristics. However, due to the locality geometry preservation, these manifold learning methods, including locally linear embedding (LLE), are sensitive to noise. To solve the noisy manifold learning problem, this paper proposes a Neighbor Smoothing Embedding (NSE) for noisy points sampled from a nonlinear manifold. Based on LLE and local linear surface estimator, the NSE smoothes the neighbors of each manifold data and then computes the reconstruction matrix of the projections on the principal surface. Experiments on synthetic data as well as real world patterns demonstrate that the suggested algorithm can efficiently maintain an accurate low-dimensional representation of the noisy manifold data with less distortion, and give higher average classification rates compared to others.

SELECTION OF THE NUMBER OF NEIGHBOURS OF EACH DATA POINT FOR THE LOCALLY LINEAR EMBEDDING ALGORITHM

This paper deals with a method, called locally linear embedding. It is a nonlinear dimensionality reduc- tion technique that computes low-dimensional, neighbourhood preserving embeddings of high dimensional data and attempts to discover nonlinear structure in high dimensional data. The implementation of the algorithm is fairly straightforward, as the algorithm has only two control parameters: the number of neighbours of each data point and the regularisation parameter. The mapping quality is quite sensitive to these parameters. In this paper, we propose a new way for selecting the number of the nearest neighbours of each data point. Our approach is experimentally verified on two data sets: artificial data and real world pictures.

High-Dimensional Unsupervised Selection and Estimation of a Finite Generalized Dirichlet Mixture Model Based on Minimum Message Length

We consider the problem of determining the structure of high-dimensional data, without prior knowledge of the number of clusters. Data are represented by a finite mixture model based on the generalized Dirichlet distribution. The generalized Dirichlet distribution has a more general covariance structure than the Dirichlet distribution and offers high flexibility and ease of use for the approximation of both symmetric and asymmetric distributions. This makes the generalized Dirichlet distribution more practical and useful. An important problem in mixture modeling is the determination of the number of clusters. Indeed, a mixture with too many or too few components may not be appropriate to approximate the true model. Here, we consider the application of the minimum message length (MML) principle to determine the number of clusters. The MML is derived so as to choose the number of clusters in the mixture model which best describes the data. A comparison with other selection criteria is performed. The validation involves synthetic data, real data clustering, and two interesting real applications: classification of web pages, and texture database summarization for efficient retrieval.

From correlation to causation networks: a simple approximate learning algorithm and its application to high-dimensional plant gene expression data

BackgroundThe use of correlation networks is widespread in the analysis of gene expression and proteomics data, even though it is known that correlations not only confound direct and indirect associations but also provide no means to distinguish between cause and effect. For "causal" analysis typically the inference of a directed graphical model is required. However, this is rather difficult due to the curse of dimensionality.ResultsWe propose a simple heuristic for the statistical learning of a high-dimensional "causal" network. The method first converts a correlation network into a partial correlation graph. Subsequently, a partial ordering of the nodes is established by multiple testing of the log-ratio of standardized partial variances. This allows identifying a directed acyclic causal network as a subgraph of the partial correlation network. We illustrate the approach by analyzing a large Arabidopsis thaliana expression data set.ConclusionThe proposed approach is a heuristic algorithm that is based on a number of approximations, such as substituting lower order partial correlations by full order partial correlations. Nevertheless, for small samples and for sparse networks the algorithm not only yield sensible first order approximations of the causal structure in high-dimensional genomic data but is also computationally highly efficient.Availability and RequirementsThe method is implemented in the "GeneNet" R package (version 1.2.0), available from CRAN and from . The software includes an R script for reproducing the network analysis of the Arabidopsis thaliana data.

Hierarchical Approximate Matching for Retrieval of Chinese Historical Calligraphy Character

As historical Chinese calligraphy works are being digitized, the problem of retrieval becomes a new challenge. But, currently no OCR technique can convert calligraphy character images into text, nor can the existing Handwriting Character Recognition approach does not work for it. This paper proposes a novel approach to efficiently retrieving Chinese calligraphy characters on the basis of similarity: calligraphy character image is represented by a collection of discriminative features, and high retrieval speed with reasonable effectiveness is achieved. First, calligraphy characters that have no possibility similar to the query are filtered out step by step by comparing the character complexity, stroke density and stroke protrusion. Then, similar calligraphy characters are retrieved and ranked according to their matching cost produced by approximate shape match. In order to speed up the retrieval, we employed high dimensional data structure -- PK-tree. Finally, the efficiency of the algorithm is demonstrated by a preliminary experiment with 3012 calligraphy character images.

A circuit framework for robust manifold learning

Manifold learning and nonlinear dimensionality reduction addresses the problem of detecting possibly nonlinear structure in high-dimensional data and constructing lower-dimensional configurations representative of this structure. A popular example is the Isomap algorithm which uses local information to approximate geodesic distances and adopts multidimensional scaling to produce lower-dimensional representations. Isomap is accurate on a global scale in contrast to many competing methods which approximate locally. However, a drawback of the Isomap algorithm is that it is topologically instable, that is, incorrectly chosen algorithm parameters or perturbations of data may drastically change the resulting configurations. We propose new methods for more robust approximation of the geodesic distances using a viewpoint of electric circuits. In this way, we achieve both the stability of local methods and the global approximation property of global methods, while compromising with local accuracy. This is demonstrated by a study of the performance of the proposed and competing methods on different data sets.

Incremental nonlinear dimensionality reduction by manifold learning

Understanding the structure of multidimensional patterns, especially in unsupervised cases, is of fundamental importance in data mining, pattern recognition, and machine learning. Several algorithms have been proposed to analyze the structure of high-dimensional data based on the notion of manifold learning. These algorithms have been used to extract the intrinsic characteristics of different types of high-dimensional data by performing nonlinear dimensionality reduction. Most of these algorithms operate in a "batch" mode and cannot be efficiently applied when data are collected sequentially. In this paper, we describe an incremental version of ISOMAP, one of the key manifold learning algorithms. Our experiments on synthetic data as well as real world images demonstrate that our modified algorithm can maintain an accurate low-dimensional representation of the data in an efficient manner.

Probability binning and testing agreement between multivariate immunofluorescence histograms: extending the chi-squared test.

A key problem in immunohistochemistry is assessing when two sample histograms are significantly different. One test that is commonly used for this purpose in the univariate case is the chi-squared test. Comparing multivariate distributions is qualitatively harder, as the "curse of dimensionality" means that the number of bins can grow exponentially. For the chi-squared test to be useful, data-dependent binning methods must be employed. An example of how this can be done is provided by the "probability binning" method of Roederer et al. (1,2,3). We derive the theoretical distribution of the probability binning statistic, giving it a more rigorous foundation. We show that the null distribution is a scaled chi-square, and show how it can be related to the standard chi-squared statistic. A small simulation shows how the theoretical results can be used to (a) modify the probability binning statistic to make it more sensitive and (b) suggest variant statistics which, while still exploiting the data-dependent strengths of the probability binning procedure, may be easier to work with. The probability binning procedure effectively uses adaptive binning to locate structure in high-dimensional data. The derivation of a theoretical basis provides a more detailed interpretation of its behavior and renders the probability binning method more flexible.

Sequential projection pursuit using genetic algorithms for data mining of analytical data.

Sequential projection pursuit (SPP) is proposed to detect inhomogeneities (clusters) in high-dimensional analytical data. Such inhomogeneities indicate that there are groups of objects (samples) with different chemical characteristics. The method is compared with principal component analysis (PCA). PCA is generally applied to visually explore structure in high-dimensional data, but is not specifically used to find clustering tendency. Projection pursuit (PP) is specifically designed to find inhomogeneities, but the original method is computationally very intensive. SPP combines the advantages of both methods and overcomes most of their weak points. In this method, latent variables are obtained sequentially according to their importance measured by the entropy index. This involves an optimization step, which is achieved by using a genetic algorithm. The performance of the method is demonstrated and evaluated, first on simulated data sets, and then on near-infrared and gas chromatography data sets. It is shown that SPP indeed reveals more easily information about inhomogeneities than PCA.

Maximum likelihood and minimum classification error factor analysis for automatic speech recognition

Hidden Markov models (HMMs) for automatic speech recognition rely on high dimensional feature vectors to summarize the short-time properties of speech. Correlations between features can arise when the speech signal is nonstationary or corrupted by noise. We investigate how to model these correlations using factor analysis, a statistical method for dimensionality reduction. Factor analysis uses a small number of parameters to model the covariance structure of high dimensional data. These parameters can be chosen in two ways: (1) to maximize the likelihood of observed speech signals, or (2) to minimize the number of classification errors. We derive an expectation-maximization (EM) algorithm for maximum likelihood estimation and a gradient descent algorithm for improved class discrimination. Speech recognizers are evaluated on two tasks, one small-sized vocabulary (connected alpha-digits) and one medium-sized vocabulary (New Jersey town names). We find that modeling feature correlations by factor analysis leads to significantly increased likelihoods and word accuracies. Moreover, the rate of improvement with model size often exceeds that observed in conventional HMM's.