Data Partitioning Method Research Articles

Influence of climate factors on Vibrio cholerae dynamics in the Pearl River estuary, South China

Current research has seldom focused on the quantitative relationships between Vibrio cholerae (V. cholerae) and climate factors owing to the complexities and high cost of field observation in the aquatic environment. This study has focused on the relationships between V. cholerae and climate factors based on linear regression method and data partition method. Data gathered from 2008 to 2009 in the Pearl River estuary, South China, were adopted. Positive rate of V. cholerae was correlated closely with monthly climate factors of water temperature and air temperature, respectively in 2009. Quarterly data analysis from 2008 to 2009 showed that there existed seasonal characteristic for V. cholerae. Positive rate of V. cholerae was correlated positively with quarterly climate factors of land surface temperature, pH, water temperature, air temperature and rainfall, respectively and negatively with quarterly air pressure. Partition data analysis in 2009 showed that there existed geography region characteristic for V. cholerae. V. cholerae dynamics was closely correlated to climate factors in the downstream area. However, it was more greatly affected by human geography factors in the urban area. Positive annual rate of V. cholerae was higher in the downstream area than in the urban area both in 2008 and 2009. At last, a cellular automaton model was used to simulate V. cholerae diffusion downstream, and the distribution of V. cholerae obtained from this model was similar to that obtained from the field observations.

Multivariate analysis of metals content in urban snow near traffic lanes in Novi Sad, Serbia

During December 2009 snow was collected at twenty two locations across the urban area of Novi Sad, directly from roads and from traffic islands near crossroads. The total metal concentration was determined for each of ten metals (Al, Ca, Cu, Fe, K, Mn, Na, Ni, Pb, Zn) using the ICP-OES analytical technique. The metal found to have the lowest concentration was Ni (0.0265 mg dm-3). The metal with the highest concentration was Na (10786 mg dm-3), which is a consequence of sodium chloride being used as de-icing salt on the roads. The metal with the second highest concentration at all locations was Ca; this was most likely the result of soil dust. The Spearman's rank correlation coefficients between analyzed metals were calculated to determine how concentrations of metals were related. A cluster analysis was carried out on the obtained data sets, using both hierarchical and partitioning methods in order to identify associations among metals and/or locations. It was shown that traffic density was not the most important factor that caused differences between the concentrations of the metals in samples.

MR-DBSCAN: a scalable MapReduce-based DBSCAN algorithm for heavily skewed data

DBSCAN (density-based spatial clustering of applications with noise) is an important spatial clustering technique that is widely adopted in numerous applications. As the size of datasets is extremely large nowadays, parallel processing of complex data analysis such as DBSCAN becomes indispensable. However, there are three major drawbacks in the existing parallel DBSCAN algorithms. First, they fail to properly balance the load among parallel tasks, especially when data are heavily skewed. Second, the scalability of these algorithms is limited because not all the critical sub-procedures are parallelized. Third, most of them are not primarily designed for shared-nothing environments, which makes them less portable to emerging parallel processing paradigms. In this paper, we present MR-DBSCAN, a scalable DBSCAN algorithm using MapReduce. In our algorithm, all the critical sub-procedures are fully parallelized. As such, there is no performance bottleneck caused by sequential processing. Most importantly, we propose a novel data partitioning method based on computation cost estimation. The objective is to achieve desirable load balancing even in the context of heavily skewed data. Besides, We conduct our evaluation using real large datasets with up to 1.2 billion points. The experiment results well confirm the efficiency and scalability of MR-DBSCAN.

Based on Scene Classification Method of Wind-PV Output Coupling Mechanism

Load tracking and output forecast error are less researched in wind and photovoltaic (PV) systems synthetic of output coupling power in present time. This paper using the technology of data mining of according to the wind and PV output characteristics with different scenario data partitioning method to the Wind and PV resource rich region of the actual wind power and PV output data divides the scene under the typical scenario data of division of Wind and PV complementary output coupling characteristics are analyzed and studied the mechanism of synthesis of output tracking system load characteristics and improve the output prediction error problem. Studies have shown that Wind and PV complementary synthetic efforts to a certain extent, reduced the coupling characteristics of output volatility, the system load tracking degree reached 14.1%.

A revised phylogeny of Antilopini (Bovidae, Artiodactyla) using combined mitochondrial and nuclear genes

Antilopini (gazelles and their allies) are one of the most diverse but phylogenetically controversial groups of bovids. Here we provide a molecular phylogeny of this poorly understood taxon using combined analyses of mitochondrial (CYTB, COIII, 12S, 16S) and nuclear (KCAS, SPTBN1, PRKCI, MC1R, THYR) genes. We explore the influence of data partitioning and different analytical methods, including Bayesian inference, maximum likelihood and maximum parsimony, on the inferred relationships within Antilopini. We achieve increased resolution and support compared to previous analyses especially in the two most problematic parts of their tree. First, taxa commonly referred to as “gazelles” are recovered as paraphyletic, as the genus Gazella appears more closely related to the Indian blackbuck (Antilope cervicapra) than to the other two gazelle genera (Nanger and Eudorcas). Second, we recovered a strongly supported sister relationship between one of the dwarf antelopes (Ourebia) and the Antilopini subgroup Antilopina (Saiga, Gerenuk, Springbok, Blackbuck and gazelles). The assessment of the influence of taxon sampling, outgroup rooting, and data partitioning in Bayesian analyses helps explain the contradictory results of previous studies.

Data quality evaluation and improvement for prognostic modeling using visual assessment based data partitioning method

When developing Prognostic and Health Management (PHM) applications for manufacturing systems, data acquired frequently comes with issues which hinder further data analysis. However, there is neither a clear definition of the data quality nor evaluation methods to quantify if acquired data is suitable for these prognostic modeling tasks such as failures detection, diagnosis and prediction. Especially, during health diagnosis modeling of engineering systems, based on data-driven method, acquired data is expected to contain clusters that can be used to differentiate multiple system health conditions. So in most cases, once data is acquired, people would like to intuitively believe that data is able to cluster into subgroups. However, this bias could lead to acceptance of false information in data. Furthermore, most of the existing metrics, such as clustering tendency in statistics and cluster-ability in data mining, only individually evaluate data characteristics without considering prognostic modeling. This paper proposes a new method to evaluate and improve data quality for system health diagnosis modeling. The clusters, as critical data characteristics for modeling multiple system conditions, are first estimated by “visualization” on the dissimilarity spectrum from spectral analysis and then evaluated in terms of their fitness and separation with each others. A visual assessment based outlier detection method is also proposed to recognize outliers from the data, which utilizes the graphic intermediate results from previous evaluation. Finally one group of bearing testing dataset acquired from real industrial applications is used to demonstrate how proposed methods are used to evaluate and improve the data quality.

RT4Win: A Windows-Based Program for Randomization Tests

A randomization test (RndT) is a statistical significance test for which the validity is based on the random assignment of experimental units in a designed experiment. In a random sampling setting, it can also be applied in a very general way because its validity does not rely on distributional assumptions, homogeneity of variances, or independence of errors. However, the use of RndTs did not receive much attention in applied research because RndTs rely on computationally intensive algorithms and most popular and common statistical software packages do not provide facilities to easily perform randomization tests. In order to fill this gap, we present a software package, RT4Win. Unlike most stand-alone software and programs for RndTs, RT4Win is a fast Windows-based program with a user-friendly interface. It provides a facility to carry out RndTs in a series of experimental designs, for both systematic and Monte Carlo data partition methods. The program is free of charge and available upon request from the authors.

A novel approximation method for multivariate data partitioning

PurposeThe plain High Dimensional Model Representation (HDMR) method needs Dirac delta type weights to partition the given multivariate data set for modelling an interpolation problem. Dirac delta type weight imposes a different importance level to each node of this set during the partitioning procedure which directly effects the performance of HDMR. The purpose of this paper is to develop a new method by using fluctuation free integration and HDMR methods to obtain optimized weight factors needed for identifying these importance levels for the multivariate data partitioning and modelling procedure.Design/methodology/approachA common problem in multivariate interpolation problems where the sought function values are given at the nodes of a rectangular prismatic grid is to determine an analytical structure for the function under consideration. As the multivariance of an interpolation problem increases, incompletenesses appear in standard numerical methods and memory limitations in computer‐based applications. To overcome the multivariance problems, it is better to deal with less‐variate structures. HDMR methods which are based on divide‐and‐conquer philosophy can be used for this purpose. This corresponds to multivariate data partitioning in which at most univariate components of the Plain HDMR are taken into consideration. To obtain these components there exist a number of integrals to be evaluated and the Fluctuation Free Integration method is used to obtain the results of these integrals. This new form of HDMR integrated with Fluctuation Free Integration also allows the Dirac delta type weight usage in multivariate data partitioning to be discarded and to optimize the weight factors corresponding to the importance level of each node of the given set.FindingsThe method developed in this study is applied to the six numerical examples in which there exist different structures and very encouraging results were obtained. In addition, the new method is compared with the other methods which include Dirac delta type weight function and the obtained results are given in the numerical implementations section.Originality/valueThe authors' new method allows an optimized weight structure in modelling to be determined in the given problem, instead of imposing the use of a certain weight function such as Dirac delta type weight. This allows the HDMR philosophy to have the chance of a flexible weight utilization in multivariate data modelling problems.

A VOMR-TREE BASED PARALLEL RANGE QUERY METHOD ON DISTRIBUTED SPATIAL DATABASE

Abstract. Spatial index impacts upon the efficiency of spatial query seriously in distributed spatial database. In this paper, we introduce a parallel spatial range query algorithm, based on VoMR-tree index, which incorporates Voronoi diagrams into MR-tree, benefiting from the nearest neighbors. We first augments MR-tree to store the nearest neighbors and constructs the VoMR-tree index by Voronoi diagram. We then propose a novel range query algorithm based on VoMR-tree index. In processing a range query, we discuss the data partition method so that we can improve the efficiency by parallelization in distributed database. Just then a verification strategy is promoted. We show the superiority of the proposed method by extensive experiments using data sets of various sizes. The experimental results reveal that the proposed method improves the performance of range query processing up to three times in comparison with the widely-used R-tree variants.

Dosage planning for type 2 diabetes mellitus patients using Indexing HDMR

Diabetes mellitus type 2 (T2DM), is also named as non-insulin dependent diabetes mellitus (NIDDM) or adult-onset diabetes, is a metabolic disease characterized by high blood glucose. Deficient insulin production or insulin resistance of the body are the causes of T2DM. Drug treatment has a very important role as well as exercise and diet. To keep the body sugar level stable within the accepted range values, drug dosage planning is a part of combinational treatment. In this study, Indexing HDMR method which is a multivariate data partitioning method is used to produce a polynomial based rule structure to manage the drug dosage planning process. For this purpose, 142 diabetic assays, 96 of which as training data and 46 of which as testing data, were used in this study. The Indexing HDMR method worked well in modeling drug dosages and it is obvious that the method is reliable for the purpose.

A data clustering algorithm for stratified data partitioning in artificial neural network

The statistical properties of training, validation and test data play an important role in assuring optimal performance in artificial neural networks (ANNs). Researchers have proposed optimized data partitioning (ODP) and stratified data partitioning (SDP) methods to partition of input data into training, validation and test datasets. ODP methods based on genetic algorithm (GA) are computationally expensive as the random search space can be in the power of twenty or more for an average sized dataset. For SDP methods, clustering algorithms such as self organizing map (SOM) and fuzzy clustering (FC) are used to form strata. It is assumed that data points in any individual stratum are in close statistical agreement. Reported clustering algorithms are designed to form natural clusters. In the case of large multivariate datasets, some of these natural clusters can be big enough such that the furthest data vectors are statistically far away from the mean. Further, these algorithms are computationally expensive as well. We propose a custom design clustering algorithm (CDCA) to overcome these shortcomings. Comparisons are made using three benchmark case studies, one each from classification, function approximation and prediction domains. The proposed CDCA data partitioning method is evaluated in comparison with SOM, FC and GA based data partitioning methods. It is found that the CDCA data partitioning method not only perform well but also reduces the average CPU time.

A New Approach to Improve the Cluster-based Parallel Processing Efficiency of High-Resolution Remotely Sensed Image

Object-oriented remotely sensed images processing method has been accepted by more and more experts of remote sensing. To advance the efficiency of data processing, parallel image computing is a good choice since large volumes of data need be analyzed efficiently and rapidly. This paper presents the information extraction method based on per-parcel extraction of high-resolution remotely sensed image; to extract efficiently different information from remotely sensed image, this paper gives the research idea of image rough-classification based on large-scale and subtle-segmentation based on small-scale; to improve the efficiency of image processing, we adapt parallel computing method to solve this problem by presenting an new data-partition method. At last this paper gives the implementation of the research idea based on Message Passing Interface (MPI) and analyzes our experimental system efficiency, and the results show that the new methods can improve the efficiency of high-resolution remotely sensed image data processing efficiently and have a good application.

Optimization of symbolic feature extraction for pattern classification

The concept of symbolic dynamics has been used in recent literature for feature extraction from time series data for pattern classification. The two primary steps of this technique are partitioning of time series to optimally generate symbol sequences and subsequently modeling of state machines from such symbol sequences. The latter step has been widely investigated and reported in the literature. However, for optimal feature extraction, the first step needs to be further explored. The paper addresses this issue and proposes a data partitioning procedure to extract low-dimensional features from time series while optimizing the class separability. The proposed procedure has been validated on two examples: (i) parameter identification in a Duffing system and (ii) classification of fatigue damage in mechanical structures, made of polycrystalline alloys. In each case, the classification performance of the proposed data partitioning method is compared with those of two other classical data partitioning methods, namely uniform partitioning (UP) and maximum entropy partitioning (MEP).

An approximation method to model multivariate interpolation problems: Indexing HDMR

Dealing with univariate or bivariate data sets instead of a multivariate data set is an important concern in interpolation problems and computer-based applications. This paper presents a new data partitioning method that partitions the given multivariate data set into univariate and bivariate data sets and constructs an approximate analytical structure that interpolates function values at arbitrarily distributed points of the given grid. A number of numerical implementations are also given to show the performance of this new method.

A GPU Oriented Data Partitioning Method to Overlap Communication and Computation

A GPU Oriented Data Partitioning Method to Overlap Communication and Computation

Filter-Based Data Partitioning for Training Multiple Classifier Systems

Data partitioning methods such as bagging and boosting have been extensively used in multiple classifier systems. These methods have shown a great potential for improving classification accuracy. This study is concerned with the analysis of training data distribution and its impact on the performance of multiple classifier systems. In this study, several feature-based and class-based measures are proposed. These measures can be used to estimate statistical characteristics of the training partitions. To assess the effectiveness of different types of training partitions, we generated a large number of disjoint training partitions with distinctive distributions. Then, we empirically assessed these training partitions and their impact on the performance of the system by utilizing the proposed feature-based and class-based measures. We applied the findings of this analysis and developed a new partitioning method called "Clustering, Declustering, and Selection" (CDS). This study presents a comparative analysis of several existing data partitioning methods including our proposed CDS approach.

Information extraction for search engines using fast heuristic techniques

We study the structured records of web pages and the relevant problems associated with the extraction and alignment of these structured records. Current automatic wrappers are complicated because they take into consideration the problems of locating relevant data region using visual cues and the use of complicated algorithms to check the similarity of data records. In this paper, we develop a non-visual automatic wrapper which questions the need for complex visual based wrappers in data extraction. The novel techniques for our wrapper are (1) filtering rules to detect and filter out irrelevant data records, (2) a tree matching algorithm using frequency measures to increase the speed of data extraction, (3) an algorithm to calculate the number and size of the components of data records to detect the correct data region, (4) a data alignment algorithm which is able to align iterative (repetitive HTML command tags) and disjunctive (optional) data items and (5) a data merging and partitioning method to solve the imperfect segmentation problem (the problem of correctly identifying the atomic entities in data items). Results show that our wrapper is as robust and in many cases outperforms the state of the art wrappers such as ViNT and DEPTA. This wrapper could have significant speed advantages when processing large volumes of web sites data, which could be helpful in meta search engine development.

XML data partitioning schemes for parallel holistic twig joins

PurposeThe purpose of this paper is to propose Extensible Markup Language (XML) data partitioning schemes that can cope with static and dynamic allocation for parallel holistic twig joins: grid metadata model for XML (GMX) and streams‐based partitioning method for XML (SPX).Design/methodology/approachGMX exploits the relationships between XML documents and query patterns to perform workload‐aware partitioning of XML data. Specifically, the paper constructs a two‐dimensional model with a document dimension and a query dimension in which each object in a dimension is composed from XML metadata related to the dimension. GMX provides a set of XML data partitioning methods that include document clustering, query clustering, document‐based refinement, query‐based refinement, and query‐path refinement, thereby enabling XML data partitioning based on the static information of XML metadata. In contrast, SPX explores the structural relationships of query elements and a range‐containment property of XML streams to generate partitions and allocate them to cluster nodes on‐the‐fly.FindingsGMX provides several salient features: a set of partition granularities that balance workloads of query processing costs among cluster nodes statically; inter‐query parallelism as well as intra‐query parallelism at multiple extents; and better parallel query performance when all estimated queries are executed simultaneously to meet their probability of query occurrences in the system. SPX also offers the following features: minimal computation time to generate partitions; balancing skewed workloads dynamically on the system; producing higher intra‐query parallelism; and gaining better parallel query performance.Research limitations/implicationsThe current status of the proposed XML data partitioning schemes does not take into account XML data updates, e.g. new XML documents and query pattern changes submitted by users on the system.Practical implicationsNote that effectiveness of the XML data partitioning schemes mainly relies on the accuracy of the cost model to estimate query processing costs. The cost model must be adjusted to reflect characteristics of a system platform used in the implementation.Originality/valueThis paper proposes novel schemes of conducting XML data partitioning to achieve both static and dynamic workload balance.

Improved model identification for non-linear systems using a random subsampling and multifold modelling (RSMM) approach

In non-linear system identification, the available observed data are conventionally partitioned into two parts: the training data that are used for model identification and the test data that are used for model performance testing. This sort of ‘hold-out’ or ‘split-sample’ data partitioning method is convenient and the associated model identification procedure is in general easy to implement. The resultant model obtained from such a once-partitioned single training dataset, however, may occasionally lack robustness and generalisation to represent future unseen data, because the performance of the identified model may be highly dependent on how the data partition is made. To overcome the drawback of the hold-out data partitioning method, this study presents a new random subsampling and multifold modelling (RSMM) approach to produce less biased or preferably unbiased models. The basic idea and the associated procedure are as follows. First, generate K training datasets (and also K validation datasets), using a K-fold random subsampling method. Secondly, detect significant model terms and identify a common model structure that fits all the K datasets using a new proposed common model selection approach, called the multiple orthogonal search algorithm. Finally, estimate and refine the model parameters for the identified common-structured model using a multifold parameter estimation method. The proposed method can produce robust models with better generalisation performance.

Morphology and phase diagram ofABClinear triblock copolymers: Parallel real-space self-consistent-field-theory simulation

A parallel algorithm designed for widely used distributed computer clusters is developed for the real-space self-consistent-field theory for polymers. We adopt an efficient data partition method to ensure high performance of this parallel algorithm on clusters, which enables us to explore unknown phase structures of complex block polymers with high accuracy and efficiency. As a benchmark test, the algorithm is applied to an ABC linear triblock copolymer, in which the volume fractions of the two end blocks of the polymer chain are equal (f_(A)=f_(C)) , to simplify the discussion. The three-dimensional microphases and phase diagram of this triblock copolymer are investigated in detail.