Data Exploration Methods Research Articles

A MATLAB toolbox for Principal Component Analysis and unsupervised exploration of data structure

Principal Component Analysis is a multivariate method to project data in a reduced hyperspace, defined by orthogonal principal components, which are linear combinations of the original variables. In this way, data dimension can be reduced, noise can be excluded from the subsequent analysis, and therefore, data interpretation is extremely facilitated. For these reasons, Principal Component Analysis is nowadays the most common chemometric strategy for unsupervised exploratory data analysis.In this paper, the PCA toolbox for MATLAB is described. This is a collection of modules for calculating Principal Component Analysis, as well as Cluster Analysis and Multidimensional Scaling, which are two other well-known multivariate methods for unsupervised data exploration. The toolbox is freely available via Internet and comprises a graphical user interface (GUI), which allows the calculation in an easy-to-use graphical environment. It aims to be useful for both beginners and advanced users. The use of the toolbox is discussed here with an appropriate practical example.

Interactive image-based information visualization for aircraft trajectory analysis

Objectives: The objective of the presented work is to present novel methods for big data exploration in the Air Traffic Control (ATC) domain. Data is formed by sets of airplane trajectories, or trails, which in turn records the positions of an aircraft in a given airspace at several time instants, and additional information such as flight height, speed, fuel consumption, and metadata (e.g. flight ID). Analyzing and understanding this time-dependent data poses several non-trivial challenges to information visualization.Materials and methods: To address this Big Data challenge, we present a set of novel methods to analyze aircraft trajectories with interactive image-based information visualization techniques.As a result, we address the scalability challenges in terms of data manipulation and open questions by presenting a set of related visual analysis methods that focus on decision-support in the ATC domain. All methods use image-based techniques, in order to outline the advantages of such techniques in our application context, and illustrated by means of use-cases from the ATC domain.Results: For each considered use-case, we outline the type of questions posed by domain experts, data involved in addressing these questions, and describe the specific image-based techniques we used to address these questions. Further, for each of the proposed techniques, we describe the visual representation and interaction mechanisms that have been used to address the above-mentioned goals. We illustrate these use-cases with real-life datasets from the ATC domain, and show how our techniques can help end-users in the ATC domain discover new insights, and solve problems, involving the presented datasets.

TOPS: a versatile software tool for statistical analysis and visualization of combinatorial gene-gene and gene-drug interaction screens

BackgroundMeasuring the impact of combinations of genetic or chemical perturbations on cellular fitness, sometimes referred to as synthetic lethal screening, is a powerful method for obtaining novel insights into gene function and drug action. Especially when performed at large scales, gene-gene or gene-drug interaction screens can reveal complex genetic interactions or drug mechanism of action or even identify novel therapeutics for the treatment of diseases.The result of such large-scale screen results can be represented as a matrix with a numeric score indicating the cellular fitness (e.g. viability or doubling time) for each double perturbation. In a typical screen, the majority of combinations do not impact the cellular fitness. Thus, it is critical to first discern true "hits" from noise. Subsequent data exploration and visualization methods can assist to extract meaningful biological information from the data. However, despite the increasing interest in combination perturbation screens, no user friendly open-source program exists that combines statistical analysis, data exploration tools and visualization.ResultsWe developed TOPS (Tool for Combination Perturbation Screen Analysis), a Java and R-based software tool with a simple graphical user interface that allows the user to import, analyze, filter and plot data from double perturbation screens as well as other compatible data. TOPS was designed in a modular fashion to allow the user to add alternative importers for data formats or custom analysis scripts not covered by the original release.We demonstrate the utility of TOPS on two datasets derived from functional genetic screens using different methods. Dataset 1 is a gene-drug interaction screen and is based on Luminex xMAP technology. Dataset 2 is a gene-gene short hairpin (sh)RNAi screen exploring the interactions between deubiquitinating enzymes and a number of prominent oncogenes using massive parallel sequencing (MPS).ConclusionsTOPS provides the benchtop scientist with a free toolset to analyze, filter and visualize data from functional genomic gene-gene and gene-drug interaction screens with a flexible interface to accommodate different technologies and analysis algorithms in addition to those already provided here. TOPS is freely available for academic and non-academic users and is released as open source.

Methods for full resolution data exploration and visualization for large 2D and 3D mass spectrometry imaging datasets

Mass spectrometry imaging (MSI) produces such large amounts of high-resolution data that fast visualization of full data sets in both high spatial and spectral resolution is often problematic. Instrument specific software tools are available, but often struggle with the size and the complexity of the MSI data sets. We describe new methods to improve the handling of these large MSI data sets by means of innovative data structures and visualization strategies developed specifically for MSI. Two new software instruments implement these new methods for rapid data exploration and visualization of both 2D and 3D data sets in full spatial and spectral resolution, and can handle existing MSI data formats, including imzML and BioMap.

On establishing ceramic chemical groups: exploring the influence of data analysis methods and the role of the elements chosen in analysis

Since the 1970s, archaeologists have increasingly depended on archaeometric rather than strictly stylistic data to explore questions of ceramic provenance and technol- ogy, and, by extension, trade, exchange, social networks and even identity. It is accepted as obvious by some archaeometrists and statisti- cians that the results of the analyses of compo- sitional data may be dependent on the format of the data used, on the data exploration method employed and, in the case of multivari- ate analyses, even on the number of elements considered. However, this is rarely articulated clearly in publications, making it less obvious to archaeologists. In this short paper, we re- examine compositional data from a collection of bricks, tiles and ceramics from Hill Hall, near Epping in Essex, England, as a case study to show how the method of data exploration used and the number of elements considered in multivariate analyses of compositional data can affect the sorting of ceramic samples into chemical groups. We compare bivariate data splitting (BDS) with principal component analysis (PCA) and centered log ratio-principal component analysis (CLR-PCA) of different unstandardized data formats [original concen- tration data and logarithmically transformed (i.e. log10 data)], using different numbers of elements. We confirm that PCA, in its various forms, is quite sensitive to the numbers and types of elements used in data analysis.

Small Multiples, Large Singles: A New Approach for Visual Data Exploration

AbstractWe present a novel visual exploration method based on small multiples and large singles for effective and efficient data analysis. Users are enabled to explore the state space by offering multiple alternatives from the current state. Users can then select the alternative of choice and continue the analysis. Furthermore, the intermediate steps in the exploration process are preserved and can be revisited and adapted using an intuitive navigation mechanism based on the well‐known undo‐redo stack and filmstrip metaphor. As proof of concept the exploration method is implemented in a prototype. The effectiveness of the exploration method is tested using a formal user study comparing four different interaction methods. By using Small Multiples as data exploration method users need fewer steps in answering questions and also explore a significantly larger part of the state space in the same amount of time, providing them with a broader perspective on the data, hence lowering the chance of missing important features. Also, users prefer visual exploration with small multiples over non‐small multiple variants.

Curators of the Future

Is there such a thing as too much data? If not,who is going to be responsible for selecting what we keep? There is only starting to be a profession of data curation.Data curation will need at least three skills: expertise from library,archive and museum studies about choosing,preserving and explaining to users;expertise from computer science and engineering about data processing,data exploration and data storage methods;and expertise from the subject area of the material,so as to know what the data means,where it came from,and what its significance is.Will we do this work with a committee,or train one person to do everything;and if the latter,is that person likely to start from the library,computing,or subject domain?

Dynamic Transcriptomic Profiles between Tomato and a Wild Relative Reflect Distinct Developmental Architectures

Developmental differences between species commonly result from changes in the tissue-specific expression of genes. Clustering algorithms are a powerful means to detect coexpression across tissues in single species but are not often applied to multidimensional data sets, such as gene expression across tissues in multiple species. As next-generation sequencing approaches enable interspecific analyses, methods to visualize and explore such data sets will be required. Here, we analyze a data set comprising gene expression profiles across six different tissue types in domesticated tomato (Solanum lycopersicum) and a wild relative (Solanum pennellii). We find that self-organizing maps are a useful means to analyze interspecies data, as orthologs can be assigned to independent levels of a "super self-organizing map." We compare various clustering approaches using a principal component analysis in which the expression of orthologous pairs is indicated by two points. We leverage the expression profile differences between orthologs to look at tissue-specific changes in gene expression between species. Clustering based on expression differences between species (rather than absolute expression profiles) yields groups of genes with large tissue-by-species interactions. The changes in expression profiles of genes we observe reflect differences in developmental architecture, such as changes in meristematic activity between S. lycopersicum and S. pennellii. Together, our results offer a suite of data-exploration methods that will be important to visualize and make biological sense of next-generation sequencing experiments designed explicitly to discover tissue-by-species interactions in gene expression data.

Enriched topological learning for cluster detection and visualization

The exponential growth of data generates terabytes of very large databases. The growing number of data dimensions and data objects presents tremendous challenges for effective data analysis and data exploration methods and tools. Thus, it becomes crucial to have methods able to construct a condensed description of the properties and structure of data, as well as visualization tools capable of representing the data structure from these condensed descriptions. The purpose of our work described in this paper is to develop a method of describing data from enriched and segmented prototypes using a topological clustering algorithm. We then introduce a visualization tool that can enhance the structure within and between groups in data. We show, using some artificial and real databases, the relevance of the proposed approach.

Visual analysis of transcriptome data in the context of anatomical structures and biological networks.

The complexity and temporal as well as spatial resolution of transcriptome datasets is constantly increasing due to extensive technological developments. Here we present methods for advanced visualization and intuitive exploration of transcriptomics data as necessary prerequisites in order to facilitate the gain of biological knowledge. Color-coding of structural images based on the expression level enables a fast visual data analysis in the background of the examined biological system. The network-based exploration of these visualizations allows for comparative analysis of genes with specific transcript patterns and supports the extraction of functional relationships even from large datasets. In order to illustrate the presented methods, the tool HIVE was applied for visualization and exploration of database-retrieved expression data for master regulators of Arabidopsis thaliana flower and seed development in the context of corresponding tissue-specific regulatory networks.

Análisis estadistico multivariado como herramienta Estratégica para el Control de procesos de calidad en la industria agroalimentaria

<p>El uso de técnicas de estadística multivariada para el control de procesos y calidad en la industria agroalimentaria ha venido en crecimiento significativo desde la mitad de los años setenta, como una consecuencia de la revolución informática que facilitó el análisis de matrices de datos de gran tamaño. A diferencia de los métodos univariados de exploración de datos, la estadística multivariada utiliza como gran pilar el análisis de información descrita por tres o más variables que pueden ser estudiadas simultáneamente y comprendidas de una manera rápida, eficiente y sencilla. Debido al extraordinario avance en las máquinas informáticas, hoy es posible aplicar estas metodologías para resolver problemas extremadamente complejos. Este artículo presenta las técnicas de estadística multivariada más reconocidas, así como la compilación de algunos trabajos que sirven como demostración de su aplicabilidad en el campo de los alimentos.</p>

Coupling visualization and data analysis for knowledge discovery from multi-dimensional scientific data

Knowledge discovery from large and complex scientific data is a challenging task. With the ability to measure and simulate more processes at increasingly finer spatial and temporal scales, the growing number of data dimensions and data objects presents tremendous challenges for effective data analysis and data exploration methods and tools. The combination and close integration of methods from scientific visualization, information visualization, automated data analysis, and other enabling technologies -such as efficient data management- supports knowledge discovery from multi-dimensional scientific data. This paper surveys two distinct applications in developmental biology and accelerator physics, illustrating the effectiveness of the described approach.

From Corynebacterium glutamicum to Mycobacterium tuberculosis--towards transfers of gene regulatory networks and integrated data analyses with MycoRegNet

Year by year, approximately two million people die from tuberculosis, a disease caused by the bacterium Mycobacterium tuberculosis. There is a tremendous need for new anti-tuberculosis therapies (antituberculotica) and drugs to cope with the spread of tuberculosis. Despite many efforts to obtain a better understanding of M. tuberculosis' pathogenicity and its survival strategy in humans, many questions are still unresolved. Among other cellular processes in bacteria, pathogenicity is controlled by transcriptional regulation. Thus, various studies on M. tuberculosis concentrate on the analysis of transcriptional regulation in order to gain new insights on pathogenicity and other essential processes ensuring mycobacterial survival. We designed a bioinformatics pipeline for the reliable transfer of gene regulations between taxonomically closely related organisms that incorporates (i) a prediction of orthologous genes and (ii) the prediction of transcription factor binding sites. In total, 460 regulatory interactions were identified for M. tuberculosis using our comparative approach. Based on that, we designed a publicly available platform that aims to data integration, analysis, visualization and finally the reconstruction of mycobacterial transcriptional gene regulatory networks: MycoRegNet. It is a comprehensive database system and analysis platform that offers several methods for data exploration and the generation of novel hypotheses. MycoRegNet is publicly available at http://mycoregnet.cebitec.uni-bielefeld.de.

Resolution Control for Balancing Overview and Detail in Multivariate Spatial Analysis

Parallel coordinates, re-orderable matrices, and dendrograms, widely used methods for visual exploration of multivariate data, are systematically integrated in a complementary manner for supporting multi-resolution visual data analysis with an enhanced overview + detail exploratory strategy. There are three main topics: (1) dynamic control across resolutions at which data are explored; (2) coordination and color mapping among the views; and (3) enhanced features of each view designed for the overview + detail exploratory tasks. A case study analysis is used to demonstrate the potential for boosting productivity for exploration tasks by coordinating the views through user-controlled resolutions within a highly interactive analysis environment. The case study is focused on a complex, geographically referenced dataset including public health, demographic and environmental components.

Size-based Transfer Functions: A New Volume Exploration Technique

The visualization of complex 3D images remains a challenge, a fact that is magnified by the difficulty to classify or segment volume data. In this paper, we introduce size-based transfer functions, which map the local scale of features to color and opacity. Features in a data set with similar or identical scalar values can be classified based on their relative size. We achieve this with the use of scale fields, which are 3D fields that represent the relative size of the local feature at each voxel. We present a mechanism for obtaining these scale fields at interactive rates, through a continuous scale-space analysis and a set of detection filters. Through a number of examples, we show that size-based transfer functions can improve classification and enhance volume rendering techniques, such as maximum intensity projection. The ability to classify objects based on local size at interactive rates proves to be a powerful method for complex data exploration.

Evaluating online data of water quality changes in a pilot drinking water distribution system with multivariate data exploration methods

The distribution of drinking water generates soft deposits and biofilms in the pipelines of distribution systems. Disturbances in water distribution can detach these deposits and biofilms and thus deteriorate the water quality. We studied the effects of simulated pressure shocks on the water quality with online analysers. The study was conducted with copper and composite plastic pipelines in a pilot distribution system. The online data gathered during the study was evaluated with Self-Organising Map (SOM) and Sammon's mapping, which are useful methods in exploring large amounts of multivariate data. The objective was to test the usefulness of these methods in pinpointing the abnormal water quality changes in the online data. The pressure shocks increased temporarily the number of particles, turbidity and electrical conductivity. SOM and Sammon's mapping were able to separate these situations from the normal data and thus make those visible. Therefore these methods make it possible to detect abrupt changes in water quality and thus to react rapidly to any disturbances in the system. These methods are useful in developing alert systems and predictive applications connected to online monitoring.

Heat Maps: An Aid for Data Analysis and Understanding of ROC CAD Experiments

To demonstrate the value of a new data visualization and exploration method for mutlireader-multicase receiver operating characteristic (MRMC-ROC) experiments of computer-aided detection (CAD) algorithms that uses three-dimensional (3D) heat maps tool adapted from gene expression array analysis. We are using data from a clinical trial of a commercial CAD system for lung cancer detection (RapidScreen RS-2000, Riverain Medical Group, Miamisburg, OH, and Rockville, MD). 3D heat maps, originally developed for displaying changes in gene expression after cancer chemotherapy in MATLAB, were modified to display the radiologists confidence levels as they interpreted chest radiographs and used to visualize the radiologists confidence levels before and after the provision of a CAD system. Heat maps demonstrated the variation among radiologists in their interpretation, and the degree of variation in interpretation when a single radiologist reinterpreted the same case without and with CAD modality. They demonstrated the variability in the identification of each cancer/cancer-free case and the variability of change seen when CAD prompts were provided. CAD increases the consistency of interpretation of a single radiologist and of a group of radiologists. Heat maps provide a method for data visualization that clarifies the effects of reader variability in ROC CAD experiments. We demonstrated how heat maps can be used to document the complexity of reader variability and suggested how clustering can reveal both nonintuitive and intuitive groupings of cases, readers, and the interaction of both with CAD.

Assessment of protein spot components applying correspondence analysis for peptide mass fingerprint data

Proteins separated by two-dimensional gel electrophoresis (2-DE) may be distributed over several spots. Otherwise, one spot may contain more than one component. The same protein occurring in several spots supposedly represents differently modified protein species that might be of biological relevance. Identification of spots with peptide mass fingerprinting and database searching leads only to the detection of the major spot components. If a spot also contains additional minor protein components, quantitation of spots with protein staining techniques or antibody detection becomes misleading. In order to find spots containing minor components we applied correspondence analysis, a multivariate data exploration method, to peptide mass fingerprint data. Correspondence analysis using peak lists revealed groups of spots containing the same protein with their characteristic mass-to-charge ratio (m/z) values. In order to detect different protein spot components an interactive threshold setting and removal of m/z values with subsequent recalculation of the correspondence analysis using our software tool CorrAn are performed. The usefulness of this methodical approach was shown by a data set of peptide mass fingerprints of 284 spots of Helicobacter pylori 26695 separated by 2-DE.

Data exploration in phylogenetic inference: scientific, heuristic, or neither.

The methods of data exploration have become the centerpiece of phylogenetic inference, but without the scientific importance of those methods having been identified. We examine in some detail the procedures and justifications of Wheeler's sensitivity analysis and relative rate comparison (saturation analysis). In addition, we review methods designed to explore evidential decisiveness, clade stability, transformation series additivity, methodological concordance, sensitivity to prior probabilities (Bayesian analysis), skewness, computer-intensive tests, long-branch attraction, model assumptions (likelihood ratio test), sensitivity to amount of data, polymorphism, clade concordance index, character compatibility, partitioned analysis, spectral analysis, relative apparent synapomorphy analysis, and congruence with a "known" phylogeny. In our review, we consider a method to be scientific if it performs empirical tests, i.e., if it applies empirical data that could potentially refute the hypothesis of interest. Methods that do not perform tests, and therefore are not scientific, may nonetheless be heuristic in the scientific enterprise if they point to more weakly or ambiguously corroborated hypotheses, such propositions being more easily refuted than those that have been more severely tested and are more strongly corroborated. Based on common usage, data exploration in phylogenetics is accomplished by any method that performs sensitivity or quality analysis. Sensitivity analysis evaluates the responsiveness of results to variation or errors in parameter values and assumptions. Sensitivity analysis is generally interpreted as providing a measure of support, where conclusions that are insensitive (robust, stable) to perturbations are judged to be accurate, probable, or reliable. As an alternative to that verificationist concept, we define support objectively as the degree to which critical evidence refutes competing hypotheses. As such, degree of support is secondary to the scientific optimality criterion of maximizing explanatory power. Quality analyses purport to distinguish good, reliable, accurate data from bad, misleading, erroneous data, thereby assessing the ability of data to indicate the true phylogeny. Only the quality analysis of character compatibility can be judged scientific-and a weak test at that compared to character congruence. Methods judged to be heuristic include Bremer support, long-branch extraction, and safe taxonomic reduction, and we underscore the great heuristic potential of a posteriori analysis of patterns of transformations on the total-evidence cladogram. However, of the more than 20 kinds of data exploration methods evaluated, the vast majority is neither scientific nor heuristic. Given so little demonstrated cognitive worth, we conclude that undue emphasis has been placed on data exploration in phylogenetic inference, and we urge phylogeneticists to consider more carefully the relevance of the methods that they employ. [T]he cult of impressive technicalities or the cult of precision may get the better of us, and interfere with our search for clarity, simplicity, and truth [Popper, 1983, p. 60. Empirical papers chosen for publication are judged to be of interest to a broad systematics audience because they represent exemplary case studies involving some important contemporary issue or issues. These may be unusually thorough explorations of data, applications of new methodology, illustrations of fundamental principles, and/or investigations of interesting evolutionary questions. [Systematic Biology: Instructions for authors, 2002; italics added].

Data exploration in phylogenetic inference: scientific, heuristic, or neither

The methods of data exploration have become the centerpiece of phylogenetic inference, but without the scientific importance of those methods having been identified. We examine in some detail the procedures and justifications of Wheeler’s sensitivity analysis and relative rate comparison (saturation analysis). In addition, we review methods designed to explore evidential decisiveness, clade stability, transformation series additivity, methodological concordance, sensitivity to prior probabilities (Bayesian analysis), skewness, computer-intensive tests, long-branch attraction, model assumptions (likelihood ratio test), sensitivity to amount of data, polymorphism, clade concordance index, character compatibility, partitioned analysis, spectral analysis, relative apparent synapomorphy analysis, and congruence with a “known” phylogeny. In our review, we consider a method to be scientific if it performs empirical tests, i.e., if it applies empirical data that could potentially refute the hypothesis of interest. Methods that do not perform tests, and therefore are not scientific, may nonetheless be heuristic in the scientific enterprise if they point to more weakly or ambiguously corroborated hypotheses, such propositions being more easily refuted than those that have been more severely tested and are more strongly corroborated. Based on common usage, data exploration in phylogenetics is accomplished by any method that performs sensitivity or quality analysis. Sensitivity analysis evaluates the responsiveness of results to variation or errors in parameter values and assumptions. Sensitivity analysis is generally interpreted as providing a measure of support, where conclusions that are insensitive (robust, stable) to perturbations are judged to be accurate, probable, or reliable. As an alternative to that verificationist concept, we define support objectively as the degree to which critical evidence refutes competing hypotheses. As such, degree of support is secondary to the scientific optimality criterion of maximizing explanatory power. Quality analyses purport to distinguish good, reliable, accurate data from bad, misleading, erroneous data, thereby assessing the ability of data to indicate the true phylogeny. Only the quality analysis of character compatibility can be judged scientific—and a weak test at that compared to character congruence. Methods judged to be heuristic include Bremer support, long-branch extraction, and safe taxonomic reduction, and we underscore the great heuristic potential of a posteriori analysis of patterns of transformations on the total-evidence cladogram. However, of the more than 20 kinds of data exploration methods evaluated, the vast majority is neither scientific nor heuristic. Given so little demonstrated cognitive worth, we conclude that undue emphasis has been placed on data exploration in phylogenetic inference, and we urge phylogeneticists to consider more carefully the relevance of the methods that they employ.