Multidimensional Data Space Research Articles

Estimating equations for spatially correlated data in multi-dimensional space

SUMMARY We use the quasilikelihood concept to propose an estimating equation for spatial data with correlation across the study region in a multi-dimensional space. With appropriate mixing con ditions, we develop a central limit theorem for a random field under various Lp metrics. The consistency and asymptotic normality of quasilikelihood estimators can then be derived. We also conduct simulations to evaluate the performance of the proposed estimating equation, and a dataset from East Lansing Woods is used to illustrate the method.

A user-centered design for a spatial data warehouse for data exploration in environmental research

The integration of data from diverse fields of ecological research is paramount in the discovery of new ecological patterns and processes. The spatial exploration of an integrated dataset that spans multiple studies and disciplines can allow researchers to gain unforeseen insight into their data, spawn new research questions and hypotheses and identify data gaps. A user-centered approach was taken to design a spatial data warehouse and online analytical processing (OLAP) tools for data exploration in ecological research. The users in this study had diverse needs and current methods of data management do not easily allow for integration and exploration of data in multidimensional space. A generalizable data warehouse design methodology was created based on the results of a user study. This methodology was then demonstrated in the design of a data warehouse for data exploration in stream ecology resulting in a multidimensional data model with a fact table representing biological stream survey measurements and dimension tables representing spatial and categorical site and landscape variables. A generalizable extraction transformation and loading (ETL) workflow was created to integrate data across spatial dimensions before it was loaded into the data warehouse. A prototype data warehouse was implemented using biological stream survey, hydrologic, and vegetation data to observe spatial patterns in biological community distributions. Based on the exploration requirements identified in the user study, prototype OLAP queries were designed to facilitate spatial data cube exploration. Finally, a web-based interface was implemented to allow for multidimensional spatial visualization of biological stream survey data. The data warehouse and interface will allow researchers to explore biological assessment data at multiple spatial scales across many dimensions.

Range queries on structured overlay networks

The efficient handling of range queries in peer-to-peer systems is still an open issue. Several approaches exist, but their lookup schemes are either too expensive (space-filling curves) or their queries lack expressiveness (topology-driven data distribution). We present two structured overlay networks that support arbitrary range queries. The first one, named Chord #, has been derived from Chord by substituting Chord’s hashing function by a key-order preserving function. It has a logarithmic routing performance and it supports range queries, which is not possible with Chord. Its O(1) pointer update algorithm can be applied to any peer-to-peer routing protocol with exponentially increasing pointers. We present a formal proof of the logarithmic routing performance and show empirical results that demonstrate the superiority of Chord # over Chord in systems with high churn rates. We then extend our routing scheme to multiple dimensions, resulting in SONAR, a Structured Overlay Network with Arbitrary Range queries. SONAR covers multi-dimensional data spaces and, in contrast to other approaches, SONAR’s range queries are not restricted to rectangular shapes but may have arbitrary shapes. Empirical results with a data set of two million objects show the logarithmic routing performance in a geospatial domain.

Cell trees: An adaptive synopsis structure for clustering multi-dimensional on-line data streams

To effectively trace the clusters of recently generated data elements in an on-line data stream, a sibling list and a cell tree are proposed in this paper. Initially, the multi-dimensional data space of a data stream is partitioned into mutually exclusive equal-sized grid-cells. Each grid-cell monitors the recent distribution statistics of data elements within its range. The old distribution statistics of each grid-cell are diminished by a predefined decay rate as time goes by, so that the effect of the obsolete information on the current result of clustering can be eliminated without maintaining any data element physically. Given a partitioning factor h, a dense grid-cell is partitioned into h equal-size smaller grid-cells. Such partitioning is continued until a grid-cell becomes the smallest one called a unit grid-cell. Conversely, a set of consecutive sparse grid-cells can be merged into a single grid-cell. A sibling list is a structure to manage the set of all grid-cells in a one-dimensional data space and it acts as an index for locating a specific grid-cell. Upon creating a dense unit grid-cell on a one-dimensional data space, a new sibling list for another dimension is created as a child of the grid-cell. In such a way, a cell tree is created. By repeating this process, a multi-dimensional dense unit grid-cell is identified by a path of a cell tree. Furthermore, in order to confine the usage of memory space, the size of a unit grid-cell is adaptively minimized such that the result of clustering becomes as accurate as possible at all times. The proposed method is comparatively analyzed by a series of experiments to identify its various characteristics.

Multi-Dimensional Range Search in Unstructured Peer-to-Peer Systems

It is an important problem to efficiently support similarity search for multi-dimensional data spaces in peer-to-peer (P2P) data management environment. Current unstructured P2P data sharing systems provide only a very rudimentary facility in query processing, i.e., matching-based query processing. This paper therefore presents a simple, yet effective index structure called EVARI (extended vector approximation routing index) to address the problem of multi-dimensional range search in unstructured P2P systems, by means of both data approximation and routing index techniques. With the aid of the EVARI, each peer can not only process range queries with its local dataset, but also route queries to promising peers with the desired data objects. In the proposed scheme, each peer summarizes its local content using space-partitioning technique, and exchanges the summarized information with neighboring peers to construct the EVARI. Furthermore, each peer can reconfigure its neighboring peers to keep the relevant peers nearby so as to optimize system resource configuration and improve system performance. Extensive experiments show the good performance of the proposed approach.

A visualization method for operating optimization

This paper proposes a visualization method for finding the optimal operating region and a near-optimal operating point based on process data. The basic principle behind the method is to map data in multi-dimensional space to a two-dimensional plane, and to generate simultaneously the contours of the objective function or functions with a mapping model. The optimal operating region can be located intuitively according to the contour distributions of the objective functions and a point found which is in this region and although not strictly optimal is near-optimal. The optimal point found can be mapped back to the original multidimensional space with a reverse mapping method proposed previously. The visualization method is tested on four operating optimization problems, including problems with multiple objective functions. Results show that the proposed method is able to reveal fully the features of the operating space and to find effectively the optimal operation region and a good operating point.

Dynamic indexing for multidimensional non-ordered discrete data spaces using a data-partitioning approach

Similarity searches in multidimensional Non-ordered Discrete Data Spaces (NDDS) are becoming increasingly important for application areas such as bioinformatics, biometrics, data mining and E-commerce. Efficient similarity searches require robust indexing techniques. Unfortunately, existing indexing methods developed for multidimensional (ordered) Continuous Data Spaces (CDS) such as the R-tree cannot be directly applied to an NDDS. This is because some essential geometric concepts/properties such as the minimum bounding region and the area of a region in a CDS are no longer valid in an NDDS. Other indexing methods based on metric spaces such as the M-tree and the Slim-trees are too general to effectively utilize the special characteristics of NDDSs, resulting in nonoptimized performance. In this article, we propose a new dynamic data-partitioning-based indexing technique, called the ND-tree, to support efficient similarity searches in an NDDS. The key idea is to extend the relevant geometric concepts as well as some indexing strategies used in CDSs to NDDSs. Efficient algorithms for ND-tree construction and techniques to solve relevant issues such as handling dimensions with different alphabets in an NDDS are presented. Our experimental results on synthetic data and real genome sequence data demonstrate that the ND-tree outperforms the linear scan, the M-tree and the Slim-trees for similarity searches in multidimensional NDDSs. A theoretical model is also developed to predict the performance of the ND-tree for random data.

MapQuant: Open‐source software for large‐scale protein quantification

Whole-cell protein quantification using MS has proven to be a challenging task. Detection efficiency varies significantly from peptide to peptide, molecular identities are not evident a priori, and peptides are dispersed unevenly throughout the multidimensional data space. To overcome these challenges we developed an open-source software package, MapQuant, to quantify comprehensively organic species detected in large MS datasets. MapQuant treats an LC/MS experiment as an image and utilizes standard image processing techniques to perform noise filtering, watershed segmentation, peak finding, peak fitting, peak clustering, charge-state determination and carbon-content estimation. MapQuant reports abundance values that respond linearly with the amount of sample analyzed on both low- and high-resolution instruments (over a 1000-fold dynamic range). Background noise added to a sample, either as a medium-complexity peptide mixture or as a high-complexity trypsinized proteome, exerts negligible effects on the abundance values reported by MapQuant and with coefficients of variance comparable to other methods. Finally, MapQuant's ability to define accurate mass and retention time features of isotopic clusters on a high-resolution mass spectrometer can increase protein sequence coverage by assigning sequence identities to observed isotopic clusters without corresponding MS/MS data.

A clustering method based on multidimensional texture analysis

Considering the analogy between image segmentation and cluster analysis, the aim of this paper is to adapt statistical texture measures to describe the spatial distribution of multidimensional observations. The main idea is to consider the cluster cores as domains characterized by their specific textures in the data space. The distribution of the data points is first described as a multidimensional histogram defined on a multidimensional regular array of sampling points. In order to evaluate locally a multidimensional texture, a co-occurrence matrix is introduced, which characterizes the local distribution of the data points in the multidimensional data space. Several local texture features can be computed from this co-occurrence matrix, which accumulates spatial and statistical information on the data distribution in the neighborhoods of the sampling points. Texture features are selected according to their ability to discriminate different distributions of data points. The sampling points where the local underlying texture is evaluated are categorized into different texture classes. The points assigned to these classes tend to form connected components in the data space, which are considered as the cores of the clusters.

A space-partitioning-based indexing method for multidimensional non-ordered discrete data spaces

There is an increasing demand for similarity searches in a multidimensional non-ordered discrete data space (NDDS) from application areas such as bioinformatics and data mining. The non-ordered and discrete nature of an NDDS raises new challenges for developing efficient indexing methods for similarity searches. In this article, we propose a new indexing technique, called the NSP-tree , to support efficient similarity searches in an NDDS. As we know, overlap causes a performance degradation for indexing methods (e.g., the R-tree) for a continuous data space. In an NDDS, this problem is even worse due to the limited number of elements available on each dimension of an NDDS. The key idea of the NSP-tree is to use a novel discrete space-partitioning (SP) scheme to ensure no overlap at each level in the tree. A number of heuristics and strategies are incorporated into the tree construction algorithms to deal with the challenges for developing an SP-based index tree for an NDDS. Our experiments demonstrate that the NSP-tree is quite promising in supporting efficient similarity searches in NDDSs. We have compared the NSP-tree with the ND-tree, a data-partitioning-based indexing technique for NDDSs that was proposed recently, and the linear scan using different NDDSs. It was found that the search performance of the NSP-tree was better than those of both methods.

Wild mouse open field behavior is embedded within the multidimensional data space spanned by laboratory inbred strains

The vast majority of studies on mouse behavior are performed on laboratory mouse strains (Mus laboratorius), while studies of wild-mouse behavior are relatively rare. An interesting question is the relationship between the phenotypes of M. laboratorius and the phenotypes of their wild ancestors. It is commonly believed, often in the absence of hard evidence, that the behavior of wild mice exceeds by far, in terms of repertoire richness, magnitude of variables and variability of behavioral measures, the behavior of the classical inbred strains. Having phenotyped the open field behavior (OF) of eight of the commonly used laboratory inbred strains, two wild-derived strains and a group of first-generation-in-captivity local wild mice (Mus musculus domesticus), we show that contrary to common belief, wild-mouse OF behavior is moderate, both in terms of end-point values and in terms of their variability, being embedded within the multidimensional data space spanned by laboratory inbred strains. The implication could be that whereas natural selection favors moderate locomotor behavior in wild mice, the inbreeding process tends to generate in mice, in some of the features, extreme and more variable behavior.

Visualisation of gait data with Kohonen self-organising neural maps

Self-organising artificial neural networks were used to reduce the complexity of joint kinematic and kinetic data, which form part of a typical instrumented gait assessment. Three-dimensional joint angles, moments and powers during the gait cycle were projected from the multi-dimensional data space onto a topological neural map, which thereby identified gait stem-patterns. Patients were positioned on the map in relation to each other and this enabled them to be compared from their gait patterns. The visualisation of large amounts of complex data in a two-dimensional map labelled with gait patterns is a step towards more objective analysis protocols which may enhance decision making.

On the Low-Rank Approximation of Data on the Unit Sphere

In various applications, data in multidimensional space are normalized to unit length. This paper considers the problem of best fitting given points on the m-dimensional unit sphere Sm-1 by k-dimensional great circles with k much less than m. The task is cast as an algebraically constrained low-rank matrix approximation problem. Using the fidelity of the low-rank approximation to the original data as the cost function, this paper offers an analytic expression of the projected gradient which, on one hand, furnishes the first order optimality condition and, on the other hand, can be used as a numerical means for solving this problem.

Mining constrained gradients in large databases

Many data analysis tasks can be viewed as search or mining in a multidimensional space (MDS). In such MDSs, dimensions capture potentially important factors for given applications, and cells represent combinations of values for the factors. To systematically analyze data in MDS, an interesting notion, called cubegrade was recently introduced by Imielinski et al. [2002], which focuses on the notable changes in measures in MDS by comparing a cell (which we refer to as probe cell) with its gradient cells, namely, its ancestors, descendants, and siblings. We call such queries gradient analysis queries (GQs). Since an MDS can contain billions of cells, it is important to answer GQs efficiently. We focus on developing efficient methods for mining GQs constrained by certain (weakly) antimonotone constraints. Instead of conducting an independent gradient-cell search once per probe cell, which is inefficient due to much repeated work, we propose an efficient algorithm, LiveSet-Driven. This algorithm finds all good gradient-probe cell pairs in one search pass. It utilizes measure-value analysis and dimension-match analysis in a set-oriented manner, to achieve bidirectional pruning between the sets of hopeful probe cells and of hopeful gradient cells. Moreover, it adopts a hypertree structure and an H-cubing method to compress data and to maximize sharing of computation. Our performance study shows that this algorithm is efficient and scalable. In addition to data cubes, we extend our study to another important scenario: mining constrained gradients in transactional databases where each item is associated with some measures such as price. Such transactional databases can be viewed as sparse MDSs where items represent dimensions, although they have significantly different characteristics than data cubes. We outline efficient mining methods for this problem.

비디오 데이터 세트의 하이퍼 사각형 표현에 기초한 비디오 유사성 검색 알고리즘

이 연구에서는 대용량 비디오 데이터 스트림에 대한 유사성 검색 알고리즘을 제시한다. 수많은 프레임으로 이루어진 비디오 스트림은 각 프레임을 다차원 벡터(multidimensional vector)로 나타냄으로써 다차원 데이터 공간 상에서 시퀸스로 나타낼 수 있다. 이 시퀸스의 특성을 분석 함으로써 각 시퀸스를 비디오 세그먼트(video segment)와 이 세그먼트의 집합인 비디오 클러스터(video cluster)로 표현한다. 본 연구에서는 이러한 비디오 세그먼트와 클러스터를 사용하여 두 비디오 스트림 사이의 유사성 함수(similarity function)를 제시하고, 이 함수에 근거하여 비디오 세그먼트의 하이퍼 사각형과 대표 프레임에 기초한 두 가지의 유사성 검색 알고리즘을 제안한다. 전자는 정해성(correctness)을 보장하는 알고리즘이며, 후자는 정해성을 약간 희생하는 대신 상당한 효율성을 얻을 수 있는 알고리즘이다. 다양한 유형의 비디오 스트림 및 가상으로 생성된 스트림 데이터에 대한 실험을 통하여 제시한 알고리즘의 성능을 분석한다. In this research, the similarity search algorithms are provided for large video data streams. A video stream that consists of a number of frames can be expressed by a sequence in the multidimensional data space, by representing each frame with a multidimensional vector By analyzing various characteristics of the sequence, it is partitioned into multiple video segments and clusters which are represented by hyper-rectangles. Using the hyper-rectangles of video segments and clusters, similarity functions between two video streams are defined, and two similarity search algorithms are proposed based on the similarity functions algorithms by hyper-rectangles and by representative frames. The former is an algorithm that guarantees the correctness while the latter focuses on the efficiency with a slight sacrifice of the correctness Experiments on different types of video streams and synthetically generated stream data show the strength of our proposed algorithms.

A neural network approach to selecting indicators for a sustainable ecosystem

The determination of indicators for monitoring the natural environment to ensure the ecological sustainability of areas under stress is problematic and prone to bias. Further, because of limitations in the resources available for data collection, such indicators should be selected carefully and their number confined to the minimum required to effectively monitor the system under study. Statistical techniques have traditionally been used to help select indicators, but often data lack the requirements of valid statistical analyses: minimal noise, variables linearly separable, variables able to be assigned numerical values, and high dimensionality. In this study, an alternate robust technique, artificial neural networks, is used to examine ecosystem data in multidimensional space and to select the minimum number of measured indicators that have the greatest weight in maintaining a sustainable ecosystem. The case study involves selecting indicators for ensuring the completion of ecologically sustainable army training conducted in a mixed grass prairie ecosystem. A neural network model was created that reduced the number of required measured indicators from 62 to 12, while minimizing researcher bias. Key words: ecological sustainability indicators, neural networks, military training.

Statistical grid-based clustering over data streams

A data stream is a massive unbounded sequence of data elements continuously generated at a rapid rate. Due to this reason, most algorithms for data streams sacrifice the correctness of their results for fast processing time. The processing time is greatly influenced by the amount of information that should be maintained. This paper proposes a statistical grid-based approach to clustering data elements of a data stream. Initially, the multidimensional data space of a data stream is partitioned into a set of mutually exclusive equal-size initial cells. When the support of a cell becomes high enough, the cell is dynamically divided into two mutually exclusive intermediate cells based on its distribution statistics. Three different ways of partitioning a dense cell are introduced. Eventually, a dense region of each initial cell is recursively partitioned until it becomes the smallest cell called a unit cell. A cluster of a data stream is a group of adjacent dense unit cells. In order to minimize the number of cells, a sparse intermediate or unit cell is pruned if its support becomes much less than a minimum support. Furthermore, in order to confine the usage of memory space, the size of a unit cell is dynamically minimized such that the result of clustering becomes as accurate as possible. The proposed algorithm is analyzed by a series of experiments to identify its various characteristics.

An integrated sensory-intelligent system for underwater acoustic signal-processing applications

A generic integrated sensory-intelligent system (ISIS) is developed for underwater acoustic signal-processing applications. ISIS constantly monitors the current acoustic channel conditions and smoothly integrates the outputs of the most appropriate signal-processing procedures or algorithms available to it for those conditions. The system is based on a generalization of a tuneable approximate piecewise linear (TAPL) model derived from the modified probabilistic neural network (MPNN). This model was designed to seamlessly integrate a set of local linear signal-processing algorithms within a given multidimensional data space. Depending on the input signal distortions, which are determined by environmental effects, ISIS automatically weighs and adds the outputs from a set of processing algorithms working in parallel. The weighting is related to the closeness of each algorithm to the sensed input signal characteristics or some other measured environmental state. A single tuning parameter is used to smoothly and seamlessly select appropriately among the parallel processing algorithm outputs. A very small tuning-parameter value selects the closest most appropriate algorithm output. At the other extreme, a fixed weighted average of all the algorithm outputs is produced with a very large value. Otherwise, a dynamic weighed average of all algorithm outputs is achieved with values in between. Some features and benefits of ISIS are demonstrated with an illustrative linear sweep chirp signal-detector estimation problem characterized by extremely variable Doppler conditions.

Pre‐whitening of data by covariance‐weighted pre‐processing

AbstractA data pre‐processing method is presented for multichannel ‘spectra’ from process spectrophotometers and other multichannel instruments. It may be seen as a ‘pre‐whitening’ of the spectra, and serves to make the instrument ‘blind’ to certain interferants while retaining its analyte sensitivity. Thereby the instrument selectivity may be improved already prior to multivariate calibration. The result is a reduced need for process perturbation or sample spiking just to generate calibration samples that span the unwanted interferants. The method consists of shrinking the multidimensional data space of the spectra in the off‐axis dimensions corresponding to the spectra of these interferants. A ‘nuisance’ covariance matrix Σ is first constructed, based on prior knowledge or estimates of the major interferants' spectra, and the scaling matrix G = Σ−1/2 is defined. The pre‐processing then consists of multiplying each input spectrum by G. When these scaled spectra are analysed in conventional chemometrics software by PCA, PCR, PLSR, curve resolution, etc., the modelling becomes simpler, because it does not have to account for variations in the unwanted interferants. The obtained model parameter may finally be descaled by G−1 for graphical interpretation. The pre‐processing method is illustrated by the use of prior spectroscopic knowledge to simplify the multivariate calibration of a fibre optical vis/NIR process analyser. The 48‐dimensional spectral space, corresponding to the 48 instrument wavelength channels used, is shrunk in two of its dimensions, defined by the known spectra of two major interferants. Successful multivariate calibration could then be obtained, based on a very small calibration sample set. Then the paper shows the pre‐whitening used for reducing the number of bilinear PLSR components in multivariate calibration models. Nuisance covariance Σ is either based on the prior knowledge of interferants' spectra or based on estimating the interferants' spectral subspace from the calibration data at hand. The relationship of the pre‐processing to weighted and generalized least squares from classical statistics is outlined. Copyright © 2003 John Wiley & Sons, Ltd.

Detection of single unit activity from the rat vagus using cluster analysis of principal components

In vivo recordings from subdiaphragmatic vagal afferent nerves generally lack the resolution to distinguish single unit activity. Several methods for data acquisition and analysis were combined to produce a high degree of reliability in recording electrophysiological signals from gastrointestinal and hepatic afferent fibers in the rat. Recordings with low noise were achieved by paralysis of the respiratory muscles and by pinning the nerve to a recording platform. Single unit activity was isolated using principal component (PC) analysis and cluster cutting of data in multi-dimensional space (1–3 PCs). Cluster assignments were determined by a semi-automated approach using the k-means algorithm. The accuracy of single unit classification was assessed by checking inter-spike intervals (ISIs) to determine the length of the refractory period, and by cross-correlation analysis to assess whether single units were mistakenly split into more than one cluster. These analyses produced up to four isolated single units from each nerve filament (a bundle of nerve fibers), and typically it was possible to further increase yield by recording from several nerve filaments simultaneously using an array of electrodes.