Choice Of Distance Measure Research Articles

CID: an efficient complexity-invariant distance for time series

The ubiquity of time series data across almost all human endeavors has produced a great interest in time series data mining in the last decade. While dozens of classification algorithms have been applied to time series, recent empirical evidence strongly suggests that simple nearest neighbor classification is exceptionally difficult to beat. The choice of distance measure used by the nearest neighbor algorithm is important, and depends on the invariances required by the domain. For example, motion capture data typically requires invariance to warping, and cardiology data requires invariance to the baseline (the mean value). Similarly, recent work suggests that for time series clustering, the choice of clustering algorithm is much less important than the choice of distance measure used.In this work we make a somewhat surprising claim. There is an invariance that the community seems to have missed, complexity invariance. Intuitively, the problem is that in many domains the different classes may have different complexities, and pairs of complex objects, even those which subjectively may seem very similar to the human eye, tend to be further apart under current distance measures than pairs of simple objects. This fact introduces errors in nearest neighbor classification, where some complex objects may be incorrectly assigned to a simpler class. Similarly, for clustering this effect can introduce errors by "suggesting" to the clustering algorithm that subjectively similar, but complex objects belong in a sparser and larger diameter cluster than is truly warranted.We introduce the first complexity-invariant distance measure for time series, and show that it generally produces significant improvements in classification and clustering accuracy. We further show that this improvement does not compromise efficiency, since we can lower bound the measure and use a modification of triangular inequality, thus making use of most existing indexing and data mining algorithms. We evaluate our ideas with the largest and most comprehensive set of time series mining experiments ever attempted in a single work, and show that complexity-invariant distance measures can produce improvements in classification and clustering in the vast majority of cases.

Supervised Distance Matrices

We introduce a novel statistical concept, called a supervised distance matrix, which quantifies pairwise similarity between variables in terms of their association with an outcome. Supervised distance matrices are derived in two stages. First, the observed data is transformed based on particular working models for association. Examples of transformations include residuals or influence curves from regression models. In the second stage, a choice of distance measure is used to compute all pairwise distances between variables in the transformed data. We present consistent estimators of the resulting distance matrix, including an inverse probability of censoring weighted estimator for use with right-censored outcomes. Supervised distance matrices can be used with standard (unsupervised) clustering algorithms to identify groups of similarly predictive variables and to discover subpopulations of related samples. This approach is illustrated using simulations and an analysis of gene expression data with a censored survival outcome. The proposed methods are widely applicable in genomics and other fields where high-dimensional data is collected on each subject.

INFLUENCE OF SPATIOTEMPORAL SCALE ON THE INTERPRETATION OF PALEOCOMMUNITY STRUCTURE: LATERAL VARIATION IN THE IMPERIAL FORMATION OF CALIFORNIA

Abstract Numerous paleocommunity studies of marine ecosystems have demonstrated that water depth was the primary factor structuring paleocommunities. In contrast, many ecological studies find other factors play a greater role in delineating communities; this difference in results may be owing to the spatiotemporal scale at which the study is performed. To explore this hypothesis, the present study examines a set of Imperial Formation (Pliocene, California) paleocommunities at a scale potentially fine enough to exclude depth as a control over the communities, thus facilitating recognition of fine-scale ecological and environmental processes operating at this scale. Twenty-six taxa from 21 samples were collected in situ from an 8.5-m-thick interval within a 0.32 km2 area. Cluster, Bray-Curtis (polar) ordination, and detrended correspondence analyses were used to infer community structure. Cluster and ordination analyses produced similar results independent of choice of distance measure. To test whether dept...

Reconstructing community relationships: the impact of sampling error, ordination approach, and gradient length

ABSTRACTEffectively summarizing complex community relationships is an important feature in studies such as biodiversity, global change, and invasion ecology. The reliability of such community summaries depends on the degree of sampling variability that is present in the data, the structure of the data, and the choice of ordination method, but the relative importance of these factors is not understood. We compared the validity of results from different ordination methods by applying five levels of sampling error to a simulated coenoplane model at two gradient lengths using two types of data (abundance and presence–absence). The multivariate methods we compared were correspondence analysis (CA), detrended correspondence analysis (DCA), non‐metric multidimensional scaling (NMDS), principal component analysis (PCA) and principal coordinates analysis (PCoA). Our results showed CA and PCA using presence–absence data were the most successful methods regardless of sampling error and gradient length, closely followed by the other methods using presence–absence data. With abundance data, PCA and CA were the most successful approaches with the short and long gradients, respectively. Approaches based on PCoA and NMDS using abundance data did not perform well regardless of the choice of distance measure used in the analysis. Both of these methods, along with the PCA using abundance data, were strongly affected by the longer gradient, leading to more distorted results.

Preliminary comparison between microsatellite and AFLP multilocus genotypes for bovine breed assignment

RiassuntoConfronto tra marcatori microsatelliti ed AFLP per l’analisi di assegnazione nella specie bovina – Marcatori microsatelliti ed AFLP sono stati confrontati per la loro capacità di assegnare 42 bovini alle loro razze di origine. L’analisi è stata condotta con 25 microsatelliti ed 89 AFLP utilizzando un metodo di analisi Bayesiano implementato nel software structure. I microsatelliti hanno assegnato correttamente il 93% degli animali, mentre gli AFLP l’88%. Entrambe le tecniche hanno correttamente assegnato tutti gli individui di razza Chianina analizzati.

Data structures and data transformations for clustering chemical data

The quality of a clustering of chemical data is determined by a proper choice of distance measures and data transformations. The latter aspect is often neglected and its importance is shown here. It is also shown that the V-shaped data structure that is often obtained in a principal component analysis of chemical data may indicate that the clustering of the raw data can lead to classifications that are not relevant from a chemical point of view and that the log double centering transform should be considered as a possible alternative.

Data structures and data transformations for clustering chemical data

The quality of a clustering of chemical data is determined by a proper choice of distance measures and data transformations. The latter aspect is often neglected and its importance is shown here. It is also shown that the V-shaped data structure that is often obtained in a principal component analysis of chemical data may indicate that the clustering of the raw data can lead to classifications that are not relevant from a chemical point of view and that the log double centering transform should be considered as a possible alternative.

Biased Bootstrap Methods for Reducing the Effects of Contamination

Summary Contamination of a sampled distribution, for example by a heavy-tailed distribution, can degrade the performance of a statistical estimator. We suggest a general approach to alleviating this problem, using a version of the weighted bootstrap. The idea is to ‘tilt’ away from the contaminated distribution by a given (but arbitrary) amount, in a direction that minimizes a measure of the new distribution's dispersion. This theoretical proposal has a simple empirical version, which results in each data value being assigned a weight according to an assessment of its influence on dispersion. Importantly, distance can be measured directly in terms of the likely level of contamination, without reference to an empirical measure of scale. This makes the procedure particularly attractive for use in multivariate problems. It has several forms, depending on the definitions taken for dispersion and for distance between distributions. Examples of dispersion measures include variance and generalizations based on high order moments. Practicable measures of the distance between distributions may be based on power divergence, which includes Hellinger and Kullback–Leibler distances. The resulting location estimator has a smooth, redescending influence curve and appears to avoid computational difficulties that are typically associated with redescending estimators. Its breakdown point can be located at any desired value ε∈ (0, ½) simply by ‘trimming’ to a known distance (depending only on ε and the choice of distance measure) from the empirical distribution. The estimator has an affine equivariant multivariate form. Further, the general method is applicable to a range of statistical problems, including regression.