Exploration Of High-dimensional Data Research Articles

Pushing diversity into higher dimensions: The LID effect on diversified similarity searching

Diversified similarity queries extend similarity searches by retrieving data elements simultaneously similar to a given object (reference) but dissimilar to other elements in the result set. Although diversity-related querying routines are expected to be costlier than similarity searches, e.g., k-NN vs. diversified k-NN, experimental evaluations of their quality and performance for exploring datasets embedded in high-dimensional spaces are still open issues. In this manuscript, we apply the concept of Local Intrinsic Dimensionality (LID) to examine the behavior of the method diversity browsing in the exploration of high-dimensional data. Diversity browsing extends the k-NN incremental search algorithm distance browsing towards diversified similarity queries by using both proximity and inner dissimilarity criteria (called Influence rule) to dismiss result set candidates. We empirically investigate the effect of real-world data LIDs over diversity browsing and found shreds of evidence that indicate: (i) the amount of diversity (measured by the total of “retrievable” diversified neighbors) is constrained by both the data fold LID and the Influence-based partitioning of the search space. Counter-intuitively, this number increases with the LID because inner dissimilarities between diversified neighbors grow slower in intrinsically high-dimensional spaces, which generates an inverse correlation between Influence-based pruning and data LID; (ii) exploratory diversity searches (the retrieval of all diversified neighbors) over varying LID sets produce query-based manifolds as the recovered elements are also data samples with enhanced measures of Relative Contrast, Relative Variance, and Intrinsic Dimensionality in comparison to the original dataset. Since exploratory diversity searches preserve the viewpoint of the query object, their outputs also provide an alternative for data visualization with meaningful distance relationships. We illustrate such a data visualization potential with bubble-clustered scatter plots by using Influence rules to determine the size of each cluster; (iii) the tuning of metric indexes is still a relevant factor for speeding up diversified similarity queries over low and medium LID data folds. In particular, our evaluation showed that metric pivots chosen with Maximal Variance criteria outperform Random pivots regarding diversity browsing searches implemented over VP-Tree indexes for most LID ranges; and (iv) the performance difference from diversity browsing to distance browsing reduces with the LID, which suggests that the querying of high-dimensional datasets by diversity may provide richer results at proportional costs to those of similarity searches.

GRay: Ray Casting for Visualization and Interactive Data Exploration of Gaussian Mixture Models.

The Gaussian mixture model (GMM) describes the distribution of random variables from several different populations. GMMs have widespread applications in probability theory, statistics, machine learning for unsupervised cluster analysis and topic modeling, as well as in deep learning pipelines. So far, few efforts have been made to explore the underlying point distribution in combination with the GMMs, in particular when the data becomes high-dimensional and when the GMMs are composed of many Gaussians. We present an analysis tool comprising various GPU-based visualization techniques to explore such complex GMMs. To facilitate the exploration of high-dimensional data, we provide a novel navigation system to analyze the underlying data. Instead of projecting the data to 2D, we utilize interactive 3D views to better support users in understanding the spatial arrangements of the Gaussian distributions. The interactive system is composed of two parts: (1) raycasting-based views that visualize cluster memberships, spatial arrangements, and support the discovery of new modes. (2) overview visualizations that enable the comparison of Gaussians with each other, as well as small multiples of different choices of basis vectors. Users are supported in their exploration with customization tools and smooth camera navigations. Our tool was developed and assessed by five domain experts, and its usefulness was evaluated with 23 participants. To demonstrate the effectiveness, we identify interesting features in several data sets.

Quantitative and Qualitative Comparison of 2D and 3D Projection Techniques for High-Dimensional Data

Projections are well-known techniques that help the visual exploration of high-dimensional data by creating depictions thereof in a low-dimensional space. While projections that target the 2D space have been studied in detail both quantitatively and qualitatively, 3D projections are far less well understood, with authors arguing both for and against the added-value of a third visual dimension. We fill this gap by first presenting a quantitative study that compares 2D and 3D projections along a rich selection of datasets, projection techniques, and quality metrics. To refine these insights, we conduct a qualitative study that compares the preference of users in exploring high-dimensional data using 2D vs. 3D projections, both without and with visual explanations. Our quantitative and qualitative findings indicate that, in general, 3D projections bring only limited added-value atop of the one provided by their 2D counterparts. However, certain 3D projection techniques can show more structure than their 2D counterparts, and can stimulate users to further exploration. All our datasets, source code, and measurements are made public for ease of replication and extension.

An Examination of Grouping and Spatial Organization Tasks for High-Dimensional Data Exploration

How do analysts think about grouping and spatial operations? This overarching research question incorporates a number of points for investigation, including understanding how analysts begin to explore a dataset, the types of grouping/spatial structures created and the operations performed on them, the relationship between grouping and spatial structures, the decisions analysts make when exploring individual observations, and the role of external information. This work contributes the design and results of such a study, in which a group of participants are asked to organize the data contained within an unfamiliar quantitative dataset. We identify several overarching approaches taken by participants to design their organizational space, discuss the interactions performed by the participants, and propose design recommendations to improve the usability of future high-dimensional data exploration tools that make use of grouping (clustering) and spatial (dimension reduction) operations.

Glyphboard: Visual Exploration of High-Dimensional Data Combining Glyphs with Dimensionality Reduction.

Rigorous data science is interdisciplinary at its core. In order to make sense of high-dimensional data, data scientists need to enter into a dialogue with domain experts. We present Glyphboard, a visualization tool that aims to support this dialogue. Glyphboard is a zoomable user interface that combines well-known methods such as dimensionality reduction and glyph-based visualizations in a novel, seamless, and integrated tool. While the dimensionality reduction affords a quick overview over the data, glyph-based visualizations are able to show the most relevant dimensions in the data set at one glance. We contribute an open-source prototype of Glyphboard, a general exchange format for high-dimensional data, and a case study with nine data scientists and domain experts from four exemplary domains in order to evaluate how the different visualization and interaction features of Glyphboard are used.

Double-Arc Parallel Coordinates and its Axes re-Ordering Methods

The Parallel Coordinates Plot (PCP) is a popular technique for the exploration of high-dimensional data. In many cases, researchers apply it as an effective method to analyze and mine data. However, when today’s data volume is getting larger, visual clutter and data clarity become two of the main challenges in parallel coordinates plot. Although Arc Coordinates Plot (ACP) is a popular approach to address these challenges, few optimization and improvement have been made on it. In this paper, we do three main contributions on the state-of-the-art PCP methods. One approach is the improvement of visual method itself. The other two approaches are mainly on the improvement of perceptual scalability when the scale or the dimensions of the data turn to be large in some mobile and wireless practical applications. 1) We present an improved visualization method based on ACP, termed as double arc coordinates plot (DACP). It not only reduces the visual clutter in ACP, but use a dimension-based bundling method with further optimization to deals with the issues of the conventional parallel coordinates plot (PCP). 2)To reduce the clutter caused by the order of the axes and reveal patterns that hidden in the data sets, we propose our first dimensional reordering method, a contribution-based method in DACP, which is based on the singular value decomposition (SVD) algorithm. The approach computes the importance score of attributes (dimensions) of the data using SVD and visualize the dimensions from left to right in DACP according the score in SVD. 3) Moreover, a similarity-based method, which is based on the combination of nonlinear correlation coefficient and SVD algorithm, is proposed as well in the paper. To measure the correlation between two dimensions and explains how the two dimensions interact with each other, we propose a reordering method based on non-linear correlation information measurements. We mainly use mutual information to calculate the partial similarity of dimensions in high-dimensional data visualization, and SVD is used to measure global data. Lastly, we use five case scenarios to evaluate the effectiveness of DACP, and the results show that our approaches not only do well in visualizing multivariate dataset, but also effectively alleviate the visual clutter in the conventional PCP, which bring users a better visual experience.

RadViz++: Improvements on Radial-Based Visualizations

RadViz is one of the few methods in Visual Analytics able to project high-dimensional data and explain formed structures in terms of data variables. However, RadViz methods have several limitations in terms of scalability in the number of variables, ambiguities created in the projection by the placement of variables along the circular design space, and ability to segregate similar instances into visual clusters. To address these limitations, we propose RadViz++, a set of techniques for interactive exploration of high-dimensional data using a RadViz-type metaphor. We demonstrate the added value of our method by comparing it with existing high-dimensional visualization methods, and also by analyzing a complex real-world dataset having over a hundred variables.

Exploring patterns enriched in a dataset with contrastive principal component analysis

Visualization and exploration of high-dimensional data is a ubiquitous challenge across disciplines. Widely used techniques such as principal component analysis (PCA) aim to identify dominant trends in one dataset. However, in many settings we have datasets collected under different conditions, e.g., a treatment and a control experiment, and we are interested in visualizing and exploring patterns that are specific to one dataset. This paper proposes a method, contrastive principal component analysis (cPCA), which identifies low-dimensional structures that are enriched in a dataset relative to comparison data. In a wide variety of experiments, we demonstrate that cPCA with a background dataset enables us to visualize dataset-specific patterns missed by PCA and other standard methods. We further provide a geometric interpretation of cPCA and strong mathematical guarantees. An implementation of cPCA is publicly available, and can be used for exploratory data analysis in many applications where PCA is currently used.

High-dimensional data analysis with subspace comparison using matrix visualization

Due to the intricate relationship between different dimensions of high-dimensional data, subspace analysis is often conducted to decompose dimensions and give prominence to certain subsets of dimensions, i.e. subspaces. Exploring and comparing subspaces are important to reveal the underlying features of subspaces, as well as to portray the characteristics of individual dimensions. To date, most of the existing high-dimensional data exploration and analysis approaches rely on dimensionality reduction algorithms (e.g. principal component analysis and multi-dimensional scaling) to project high-dimensional data, or their subspaces, to two-dimensional space and employ scatterplots for visualization. However, the dimensionality reduction algorithms are sometimes difficult to fine-tune and scatterplots are not effective for comparative visualization, making subspace comparison hard to perform. In this article, we aggregate high-dimensional data or their subspaces by computing pair-wise distances between all data items and showing the distances with matrix visualizations to present the original high-dimensional data or subspaces. Our approach enables effective visual comparisons among subspaces, which allows users to further investigate the characteristics of individual dimensions by studying their behaviors in similar subspaces. Through subspace comparisons, we identify dominant, similar, and conforming dimensions in different subspace contexts of synthetic and real-world high-dimensional data sets. Additionally, we present a prototype that integrates parallel coordinates plot and matrix visualization for high-dimensional data exploration and incremental dimensionality analysis, which also allows users to further validate the dimension characterization results derived from the subspace comparisons.

Reproduction of secondary data in projection pursuit transformation

A challenge when working with multivariate data in a geostatistical context is that the data are rarely Gaussian. Multivariate distributions may include nonlinear features, clustering, long tails, functional boundaries, spikes, and heteroskedasticity. Multivariate transformations account for such features so that they are reproduced in geostatistical models. Projection pursuit as developed for high dimensional data exploration can also be used to transform a multivariate distribution into a multivariate Gaussian distribution with an identity covariance matrix. Its application within a geostatistical modeling context is called the projection pursuit multivariate transform (PPMT). An approach to incorporate exhaustive secondary variables in the PPMT is introduced. With this approach the PPMT can incorporate any number of secondary variables with any number of primary variables. A necessary alteration to the approach to make this numerically practical was the implementation of a continuous probability estimator that relies on Bernstein polynomials for the transformation that takes place in the projections. Stopping criteria were updated to incorporate a bootstrap t test that compares data sampled from a multivariate Gaussian distribution with the data undergoing transformation.

Beyond the Third Dimension: Visualizing High-Dimensional Data with Projections

Multidimensional projections are an increasingly popular technique for visualizing large datasets containing observations having tens or even hundreds of dimensions. Compared to other techniques such as parallel coordinates, tables, and scatterplot matrices, they support tasks such as finding groups of related observations and outliers in simpler, more effective, ways. The authors discuss here the advantages of multidimensional projections, how to compute them, and recent advances that enhance them by visual explanatory techniques, so as to make them efficient and effective instruments that should be part of the toolkit of any scientist interested in high-dimensional data exploration.

An improved diversity visualization system for multivariate data

Exploring and analyzing data is becoming increasingly difficult due to the growth of data. Visual analytics tools can be an attractive solution to support the process to derive insights from data. Currently, there are many visual representation methods to visualize the diversity in multivariate data sets. However, most of these applications focus on visual representation problems, and these solutions support limited interactive components for users to effectively explore and analyze data on screen. In this paper, the adaptive diversity table (ADT) is proposed to solve the visual representation problems (occlusion and technique interference). Furthermore, it integrates the mantra techniques to support users to accomplish seven important tasks (i.e. overview, zoom, filter, details-on-demand, relate, history, and extract) that are useful for high dimensional data exploration and data analysis. Experimental results show that the proposed ADT is a better visual representation as compared to other prior techniques. Majority of the respondents prefers to use the proposed ADT over the other visual representation methods. User studies also show that the proposed ADT is more useful as it enables the respondents to be more efficient in analyzing the data sets provided.

R/qtlcharts: interactive graphics for quantitative trait locus mapping.

Every data visualization can be improved with some level of interactivity. Interactive graphics hold particular promise for the exploration of high-dimensional data. R/qtlcharts is an R package to create interactive graphics for experiments to map quantitative trait loci (QTL) (genetic loci that influence quantitative traits). R/qtlcharts serves as a companion to the R/qtl package, providing interactive versions of R/qtl’s static graphs, as well as additional interactive graphs for the exploration of high-dimensional genotype and phenotype data.

Automated Analytical Methods to Support Visual Exploration of High-Dimensional Data

Visual exploration of multivariate data typically requires projection onto lower dimensional representations. The number of possible representations grows rapidly with the number of dimensions, and manual exploration quickly becomes ineffective or even unfeasible. This paper proposes automatic analysis methods to extract potentially relevant visual structures from a set of candidate visualizations. Based on features, the visualizations are ranked in accordance with a specified user task. The user is provided with a manageable number of potentially useful candidate visualizations, which can be used as a starting point for interactive data analysis. This can effectively ease the task of finding truly useful visualizations and potentially speed up the data exploration task. In this paper, we present ranking measures for class-based as well as non-class-based scatterplots and parallel coordinates visualizations. The proposed analysis methods are evaluated on different data sets.

An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests.

Recursive partitioning methods have become popular and widely used tools for nonparametric regression and classification in many scientific fields. Especially random forests, which can deal with large numbers of predictor variables even in the presence of complex interactions, have been applied successfully in genetics, clinical medicine, and bioinformatics within the past few years. High-dimensional problems are common not only in genetics, but also in some areas of psychological research, where only a few subjects can be measured because of time or cost constraints, yet a large amount of data is generated for each subject. Random forests have been shown to achieve a high prediction accuracy in such applications and to provide descriptive variable importance measures reflecting the impact of each variable in both main effects and interactions. The aim of this work is to introduce the principles of the standard recursive partitioning methods as well as recent methodological improvements, to illustrate their usage for low and high-dimensional data exploration, but also to point out limitations of the methods and potential pitfalls in their practical application. Application of the methods is illustrated with freely available implementations in the R system for statistical computing.

ClusterSculptor: Software for expert-steered classification of single particle mass spectra

To take full advantage of the vast amount of highly detailed data acquired by single particle mass spectrometers requires that the data be organized according to some rules that have the potential to be insightful. Most commonly cluster analysis methods are used to classify the individual particle mass spectra on the basis of their similarity. Cluster analysis is a powerful strategy for the exploration of high-dimensional data in the absence of a-priori hypotheses or data classification models. However, more often than not, the examination of the data clustering results reveals that many clusters contain particles of different types and that many particles of one type end up in a number of separate clusters. Our experience with cluster analysis shows that we have a vast amount of non-compiled knowledge and intuition that if brought to bear in this effort has the potential to greatly improve it. ClusterSculptor is software package designed to provide a comprehensive and intuitive visual framework to aid scientists introduce their vast knowledge into the data classification process. ClusterSculptor offers a wide variety of tools that are necessary for a high-dimensional, expert-driven activity we call cluster sculpting. ClusterSculptor is designed to be coupled to SpectraMiner, our data mining and visualization software package. The data are first visualized with SpectraMiner and identified problems are exported to ClusterSculptor, where the user steers the reclassification and recombination of clusters of tens of thousands of particle mass spectra in real-time. The resulting sculpted clusters can be then imported back into SpectraMiner. Here we present the results of a study, in which ClusterSculptor is used to classify a complex dataset that includes single particle mass spectra of a variety of particle types. The compositions of these laboratory generated particles were carefully chosen to test some of the more difficult aspects of single particle mass spectroscopy. We demonstrate the use of ClusterSculptor to greatly improve chemical speciation of single particles by introducing expert input into data classification process.

DD-HDS: A Method for Visualization and Exploration of High-Dimensional Data

Mapping high-dimensional data in a low-dimensional space, for example, for visualization, is a problem of increasingly major concern in data analysis. This paper presents data-driven high-dimensional scaling (DD-HDS), a nonlinear mapping method that follows the line of multidimensional scaling (MDS) approach, based on the preservation of distances between pairs of data. It improves the performance of existing competitors with respect to the representation of high-dimensional data, in two ways. It introduces (1) a specific weighting of distances between data taking into account the concentration of measure phenomenon and (2) a symmetric handling of short distances in the original and output spaces, avoiding false neighbor representations while still allowing some necessary tears in the original distribution. More precisely, the weighting is set according to the effective distribution of distances in the data set, with the exception of a single user-defined parameter setting the tradeoff between local neighborhood preservation and global mapping. The optimization of the stress criterion designed for the mapping is realized by "force-directed placement" (FDP). The mappings of low- and high-dimensional data sets are presented as illustrations of the features and advantages of the proposed algorithm. The weighting function specific to high-dimensional data and the symmetric handling of short distances can be easily incorporated in most distance preservation-based nonlinear dimensionality reduction methods.

Image Fusion Based on Topographic Mappings Using the Hyperbolic Space

The analysis of multivariate image data is a field of research that is becoming increasingly important in a broad range of applications from remote sensing to medical imaging. While traditional scientific visualization techniques are often not suitable for the analysis of this kind of data, methods of image fusion have evolved as a promising approach for synergistic data integration. In this paper, a new approach for the analysis of multivariate image data by means of image fusion is presented, which employs topographic mapping techniques based on non-Euclidean geometry. The hyperbolic self-organizing map (HSOM) facilitates the exploration of high-dimensional data and provides an interface in the tradition of distortion-oriented presentation techniques. For the analysis of hidden patterns and spatial relationships, the HSOM gives rise to an intuitive and efficient framework for the dynamic visualization of multivariate image data by means of color. In an application, the hyperbolic data explorer (HyDE) is employed for the visualization of image data from dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI). Using 12 image sequences from breast cancer research, the method is introduced by different visual representations of the data and is also quantitatively analyzed. The HSOM is compared to different standard classifiers and evaluated with respect to topology preservation.