Data Partitioning Scheme Research Articles

Package queries: efficient and scalable computation of high-order constraints

Traditional database queries follow a simple model: they define constraints that each tuple in the result must satisfy. This model is computationally efficient, as the database system can evaluate the query conditions on each tuple individually. However, many practical, real-world problems require a collection of result tuples to satisfy constraints collectively, rather than individually. In this paper, we present package queries, a new query model that extends traditional database queries to handle complex constraints and preferences over answer sets. We develop a full-fledged package query system, implemented on top of a traditional database engine. Our work makes several contributions. (1) We design PaQL, a SQL-based query language that supports the declarative specification of package queries. We prove that PaQL is at least as expressive as integer linear programming, and therefore, evaluation of package queries is NP-hard. (2) We present a fundamental evaluation strategy that combines the capabilities of databases and constraint optimization solvers to derive solutions to package queries. The core of our approach is a set of translation rules that transform a package query to an integer linear program. (3) We introduce an offline data partitioning strategy allowing query evaluation to scale to large data sizes. (4) We introduce SketchRefine, a scalable algorithm for package evaluation, with strong approximation guarantees [ $$(1 \pm \varepsilon )$$ -factor approximation]. (5) We present a method for parallelizing the Refine phase of SketchRefine. (6) We present an empirical study of the efficiency gains of providing integer solvers with starting solutions. (7) We present extensive experiments over real-world and benchmark data. The results demonstrate that our methods are effective at deriving high-quality package results and achieve runtime performance that is an order of magnitude faster than directly using ILP solvers over large datasets.

A Survey of Traditional and MapReduceBased Spatial Query Processing Approaches

Various indexing methods of spatial data have come out after rigorous efforts put by many researchers for fast processing of spatial queries. Parallelizing spatial index building and query processing have become very popular for improving efficiency. The MapReduce framework provides a modern way of parallel processing. A MapReduce-based works for spatial queries consider the existing traditional spatial indexing for building spatial indexes in parallel. The majority of the spatial indexes implemented in MapReduce use R-Tree and its variants. Therefore, R-Tree and its variantbased traditional spatial indexes are thoroughly surveyed in the paper. The objective is to search for still less explored spatial indexing approaches, having the potential for parallelism in MapReduce. The review work also provides a detailed survey of MapReduce-based spatial query processing approaches - hierarchical indexed and packed key-value storage based spatial dataset. Both approaches use different data partitioning strategies for distributing data among cluster nodes and managing the partitioned dataset through different indexing. Finally, a number of parameters are selected for comparison and analysis of all the existing approaches in the literature.

A performance comparison of data and memory allocation strategies for sequence aligners on NUMA architectures

Over the last several years, many sequence alignment tools have appeared and become popular for the fast evolution of next generation sequencing technologies. Obviously, researchers that use such tools are interested in getting maximum performance when they execute them in modern infrastructures. Today’s NUMA (Non-uniform memory access) architectures present major challenges in getting such applications to achieve good scalability as more processors/cores are used. The memory system in NUMA systems shows a high complexity and may be the main cause for the loss of an application’s performance. The existence of several memory banks in NUMA systems implies a logical increase in latency associated with the accesses of a given processor to a remote bank. This phenomenon is usually attenuated by the application of strategies that tend to increase the locality of memory accesses. However, NUMA systems may also suffer from contention problems that can occur when concurrent accesses are concentrated on a reduced number of banks. Sequence alignment tools use large data structures to contain reference genomes to which all reads are aligned. Therefore, these tools are very sensitive to performance problems related to the memory system. The main goal of this study is to explore the trade-offs between data locality and data dispersion in NUMA systems. We have performed experiments with several popular sequence alignment tools on two widely available NUMA systems to assess the performance of different memory allocation policies and data partitioning strategies. We find that there is not one method that is best in all cases. However, we conclude that memory interleaving is the memory allocation strategy that provides the best performance when a large number of processors and memory banks are used. In the case of data partitioning, the best results are usually obtained when the number of partitions used is greater, sometimes combined with an interleave policy.

Synchrophasor Sensor Networks for Grid Communication and Protection.

This paper focuses primarily on leveraging synchronized current/voltage amplitudes and phase angle measurements to foster new categories of applications, such as improving the effectiveness of grid protection and minimizing outage duration for distributed grid systems. The motivation for such an application arises from the fact that with the support of communication, synchronized measurements from multiple sites in a grid network can greatly enhance the accuracy and timeliness of identifying the source of instabilities. The paper first provides an overview of synchrophasor networks and then presents techniques for power quality assessment, including fault detection and protection. To achieve this we present a new synchrophasor data partitioning scheme that is based on the formation of a joint space and time observation vector. Since communication is an integral part of synchrophasor networks, the newly adopted wireless standard for machine-to-machine (M2M) communication, known as IEEE 802.11ah, has been investigated. The paper also presents a novel implementation of a hardware in the loop testbed for real-time performance evaluation. The purpose is to illustrate the use of both hardware and software tools to verify the performance of synchrophasor networks under more realistic environments. The testbed is a combination of grid network modeling, and an Emulab-based communication network. The combined grid and communication network is then used to assess power quality for fault detection and location using the IEEE 39-bus and 390-bus systems.

A Scalable Execution Engine for Package Queries

Many modern applications and real-world problems involve the design of item collections, or packages: from planning your daily meals all the way to mapping the universe. Despite the pervasive need for packages, traditional data management does not offer support for their definition and computation. This is because traditional database queries follow a powerful, but very simple model: a query defines constraints that each tuple in the result must satisfy. However, a system tasked with the design of packages cannot consider items independently; rather, the system needs to determine if a set of items collectively satisfy given criteria. In this paper, we present package queries, a new query model that extends traditional database queries to handle complex constraints and preferences over answer sets. We develop a full-fledged package query system, implemented on top of a traditional database engine. Our work makes several contributions. First, we design PaQL, a SQL-based query language that supports the declarative specification of package queries. Second, we present a fundamental strategy for evaluating package queries that combines the capabilities of databases and constraint optimization solvers. The core of our approach is a set of translation rules that transform a package query to an integer linear program. Third, we introduce an offline data partitioning strategy allowing query evaluation to scale to large data sizes. Fourth, we introduce SKETCHREFINE, an efficient and scalable algorithm for package evaluation, which offers strong approximation guarantees. Finally, we present extensive experiments over real-world data. Our results demonstrate that SKETCHREFINE is effective at deriving high-quality package results, and achieves runtime performance that is an order of magnitude faster than directly using ILP solvers over large datasets.

Translating on pairwise entity space for knowledge graph embedding

In addition to feature-based representations that characterize objects with feature vectors, relation-based representations constitute another type of data representation strategies. They typically store patterns as a knowledge graph (KG), consisting of nodes (objects) and edges (relationships between objects). Given that most KGs are noisy and far from being complete, KG analysis and completion is required to establish the likely truth of new facts and correct unlikely ones based on the existing data within the KG. An effective way for tackling this, is through translation techniques which encode entities and links with hidden representations in embedding spaces. In this paper, we aim at improving the state-of-the-art translation techniques by taking into account the multiple facets of the different patterns and behaviors of each relation type. To the best of our knowledge, this is the first latent representation model which considers relational representations to be dependent on the entities they relate. The multi-modality of the relation type over different entities is effectively formulated as a projection matrix over the space spanned by the entity vectors. We develop an economic computation of the projection matrix by directly providing an analytic formulation other than relying on a more consuming iterative optimization procedure. Two large benchmark knowledge bases are used to evaluate the performance with respect to the link prediction task. A new test data partition scheme is proposed to offer better understanding of the behavior of a link prediction model. Experimental results show that the performance of the proposed algorithm is consistently among the top under different evaluation schemes.

Two-dimensional Phase Unwrapping Method Using Cost Function of L0 Norm

Considering cost model and convergence speed of the minimum norm unwrapping, a highly efficient two-dimensional global phase unwrapping method optimized with L0 norm is proposed. As analysing features of cost model in phase unwrapping with minimum norm, a cost function definition is provided in line with the L0 norm, which impose a stronger constraint in the tangent direction of phase discontinuity boundary than that in normal direction, in order to preserve integrity of discontinuity during iterative unwrapping processing for continuous phase. For the sake of slow speed of low-frequency error convergence during linear solving, a data partitioning strategy is introduced into unwrapping processing. Due to independence of minimum norm method in blocks, linear solving only focus on high-frequency information and improve efficiency of iterative work, and the low-frequency processing part is transferred to offsetting-aligning between blocks. With experiments and analysis, reliability and efficiency of the novel phase unwrapping method are certified comparing to existing methods.

The Data Aggregation Privacy Protection Algorithm of Body Area Network Based on Data Partitioning

As a new network, the body area network is widely used in the field of health care, emergency rescue and so on. In the practical application, a large number of data collected by the body area network are closely related to user's personal privacy. However the body area network is an open environment. When transmitted in the network, the user's privacy data is vulnerable to the attacker's eavesdropping, tampering and other security threats. At the present stage, the research of the body area network mainly focuses on the system architecture and node energy consumption, the studies of security aspect of user privacy data is relatively few. This paper puts forward The data aggregation privacy protection algorithm of body area network based on data partitioning based on the above problem, this algorithm uses data partitioning strategy to deal with privacy data, and combines encryption theory to achieve security aggregation of user privacy data of the body area network security aggregation. The result not only improves effectively the privacy protection of the user data of body area network, but also reduces data redundancy and the traffic of the nodes.

The phylogenetic position of eriophyoid mites (superfamily Eriophyoidea) in Acariformes inferred from the sequences of mitochondrial genomes and nuclear small subunit (18S) rRNA gene

Eriophyoid mites (superfamily Eriophyoidea) comprise >4400 species worldwide. Despite over a century of study, the phylogenetic position of these mites within Acariformes is still poorly resolved. Currently, Eriophyoidea is placed in the order Trombidiformes. We inferred the high-level phylogeny of Acari with the mitochondrial (mt) genome sequences of 110 species including four eriophyoid species, and the nuclear small subunit (18S) rRNA gene sequences of 226 species including 25 eriophyoid species. Maximum likelihood (ML), Bayesian inference (BI) and Maximum parsimony (MP) methods were used to analyze the sequence data. Divergence times were estimated for major lineages of Acari using Bayesian approaches. Our analyses consistently recovered the monophyly of Eriophyoidea but rejected the monophyly of Trombidiformes. The eriophyoid mites were grouped with the sarcoptiform mites, or were the sister group of sarcoptiform mites+non-eriophyoid trombidiform mites, depending on data partition strategies. Eriophyoid mites diverged from other mites in the Devonian (384Mya, 95% HPD, 352–410Mya). The origin of eriophyoid mites was dated to the Permian (262Mya, 95% HPD 230–307Mya), mostly prior to the radiation of gymnosperms (Triassic-Jurassic) and angiosperms (early Cretaceous). We propose that the placement of Eriophyoidea in the order Trombidiformes under the current classification system should be reviewed.

FiDoop-DP: Data Partitioning in Frequent Itemset Mining on Hadoop Clusters

Traditional parallel algorithms for mining frequent itemsets aim to balance load by equally partitioning data among a group of computing nodes. We start this study by discovering a serious performance problem of the existing parallel Frequent Itemset Mining algorithms. Given a large dataset, data partitioning strategies in the existing solutions suffer high communication and mining overhead induced by redundant transactions transmitted among computing nodes. We address this problem by developing a data partitioning approach called FiDoop-DP using the MapReduce programming model. The overarching goal of FiDoop-DP is to boost the performance of parallel Frequent Itemset Mining on Hadoop clusters. At the heart of FiDoop-DP is the Voronoi diagram-based data partitioning technique, which exploits correlations among transactions. Incorporating the similarity metric and the Locality-Sensitive Hashing technique, FiDoop-DP places highly similar transactions into a data partition to improve locality without creating an excessive number of redundant transactions. We implement FiDoop-DP on a 24-node Hadoop cluster, driven by a wide range of datasets created by IBM Quest Market-Basket Synthetic Data Generator. Experimental results reveal that FiDoop-DP is conducive to reducing network and computing loads by the virtue of eliminating redundant transactions on Hadoop nodes. FiDoop-DP significantly improves the performance of the existing parallel frequent-pattern scheme by up to 31 percent with an average of 18 percent.

DRSO-EGSM: data replication strategy oriented to automatic energy gear-shifting mechanism

Recently, more and more attention has been paid on research of cloud storage system. However, with the rapid growth of data volume, the problem of high energy consumption and low efficiency has become the key factor for restricting its development. In order to provide better support for the energy gear-shifting mechanism, we designed an energy-effective data replicating strategy (DRSO-EGSM). Firstly, data is classified into two categories, based on which the number of replicas is determined. And then the replica placement strategy oriented to energy efficiency is designed. Corresponding performance model and energy consumption model are established for data partitioning strategy. Meanwhile, the energy effectiveness of the DSRO-EGSM is evaluated in GridSim simulator. Compared with current existing similar mechanisms TDCS and the default replication strategy of HDFS, substantive results obtained from simulation experiments show that DSRO-EGSM can have more advantages in reducing energy consumption while guaranteeing the system performance.

Semiautomated Alignment of High-Throughput Metabolite Profiles with Chemometric Tools.

The rapid increase in the use of metabolite profiling/fingerprinting techniques to resolve complicated issues in metabolomics has stimulated demand for data processing techniques, such as alignment, to extract detailed information. In this study, a new and automated method was developed to correct the retention time shift of high-dimensional and high-throughput data sets. Information from the target chromatographic profiles was used to determine the standard profile as a reference for alignment. A novel, piecewise data partition strategy was applied for the determination of the target components in the standard profile as markers for alignment. An automated target search (ATS) method was proposed to find the exact retention times of the selected targets in other profiles for alignment. The linear interpolation technique (LIT) was employed to align the profiles prior to pattern recognition, comprehensive comparison analysis, and other data processing steps. In total, 94 metabolite profiles of ginseng were studied, including the most volatile secondary metabolites. The method used in this article could be an essential step in the extraction of information from high-throughput data acquired in the study of systems biology, metabolomics, and biomarker discovery.

ParaView visualization of Abaqus output on the mechanical deformation of complex microstructures

Abaqus® is a popular software suite for finite element analysis. It delivers linear and nonlinear analyses of mechanical and fluid dynamics, includes multi-body system and multi-physics coupling. However, the visualization capability of Abaqus using its CAE module is limited. Models from microtomography have extremely complicated structures, and datasets of Abaqus output are huge, requiring a visualization tool more powerful than Abaqus/CAE. We convert Abaqus output into the XML-based VTK format by developing a Python script and then using ParaView to visualize the results. Such capabilities as volume rendering, tensor glyphs, superior animation and other filters allow ParaView to offer excellent visualizing manifestations. ParaView's parallel visualization makes it possible to visualize very big data. To support full parallel visualization, the Python script achieves data partitioning by reorganizing all nodes, elements and the corresponding results on those nodes and elements. The data partition scheme minimizes data redundancy and works efficiently. Given its good readability and extendibility, the script can be extended to the processing of more different problems in Abaqus. We share the script with Abaqus users on GitHub.

Improved pan-specific prediction of MHC class I peptide binding using a novel receptor clustering data partitioning strategy.

Pan-specific prediction of receptor-ligand interaction is conventionally done using machine-learning methods that integrates information about both receptor and ligand primary sequences. To achieve optimal performance using machine learning, dealing with overfitting and data redundancy is critical. Most often so-called ligand clustering methods have been used to deal with these issues in the context of pan-specific receptor-ligand predictions, and the MHC system the approach has proven highly effective for extrapolating information from a limited set of receptors with well characterized binding motifs, to others with no or very limited experimental characterization. The success of this approach has however proven to depend strongly on the similarity of the query molecule to the molecules with characterized specificity using in the machine-learning process. Here, we outline an alternative strategy with the aim of altering this and construct data sets optimal for training of pan-specific receptor-ligand predictions focusing on receptor similarity rather than ligand similarity. We show that this receptor clustering method consistently in benchmarks covering affinity predictions, MHC ligand and MHC epitope identification perform better than the conventional ligand clustering method on the alleles with remote similarity to the training set.

Data Partitioning Strategy of GPU Heterogeneous Clusters Based on Learning

With the rapid progress of computational science and computer simulation ability, a lot of properties can be predicted by the powerful ability of parallel computation before the actual research and development. With the development of high performance computer architecture, GPU is more and more widely used in high performance computation field as an emerging architecture, and a growing number of computations use GPU heterogeneous cluster architecture. However, how to partition workload and map to computing resource has always been the focus and difficult point. In the current study of GPU, according to the problems of the computing power provided by each node and the cluster hardware architecture which the application programmers don't understand, some partitioning strategies will result in serious load imbalance problem. Aimed at the complexity brought by the different computing ability of the nodes of GPU clusters, this paper proposes a GPU data partitioning strategy of heterogeneous clusters based on learning. It collects the states of each node in the process of running a program, and then estimates the calculation ability of each node dynamically, so as to guide the data partitioning. Actual testing results show that, this strategy allocates different tasks to nodes based on computing ability to ensure load balancing among nodes, so as to improve the execution performance of CUDA programs on heterogeneous GPU clusters and it laid a solid foundation for efficient computing on heterogeneous GPU clusters.

Multiresolution and fast decompression for optimal web-based rendering

Limited bandwidth is a strong constraint when efficient transmission of 3D data to Web clients and mobile applications is needed. In this paper we present a novel multi-resolution WebGL based rendering algorithm which combines progressive loading, view-dependent resolution and mesh compression, providing high frame rates and a decoding speed of million of triangles per second in JavaScript. The method is parallelizable and scalable to very large models.The algorithm is based on the local multi-resolution approaches provided by the community, but ad-hoc solutions had to be studied and implemented to provide adequate performances. In particular, a compression mechanism that reached very high compression rate without impact on rendering performance was implemented. Moreover, the data partition strategy was modified in order to be able to load different types of data (i.e. point clouds) and better adapt to the potentials and limitations of web-based rendering.

Scalable 3D hybrid parallel Delaunay image-to-mesh conversion algorithm for distributed shared memory architectures

In this paper, we present a scalable three-dimensional hybrid parallel Delaunay image-to-mesh conversion algorithm (PDR.PODM) for distributed shared memory architectures. PDR.PODM is able to explore parallelism early in the mesh generation process thanks to the aggressive speculative approach employed by the Parallel Optimistic Delaunay Mesh generation algorithm (PODM). In addition, it decreases the communication overhead and improves data locality by making use of a data partitioning scheme offered by the Parallel Delaunay Refinement algorithm (PDR). PDR.PODM supports fully functional volume grading by creating elements with varying size. Small elements are created near boundary or inside the critical regions in order to capture the fine features while big elements are created in the rest of the mesh. We tested PDR.PODM on Blacklight, a distributed shared memory (DSM) machine in Pittsburgh Supercomputing Center. For the uniform mesh generation, we observed a weak scaling speedup of 163.8 and above for up to 256 cores as opposed to PODM whose weak scaling speedup is only 44.7 on 256 cores. PDR.PODM scales well on uniform refinement cases running on DSM supercomputers. The end result is that PDR.PODM can generate 18 million elements per second as opposed to 14 million per second in our earlier work. The varying size version sharply reduces the number of elements compared to the uniform version and thus reduces the time to generate the mesh while keeping the same fidelity.

Resource bricolage and resource selection for parallel database systems

Running parallel database systems in an environment with heterogeneous resources has become increasingly common, due to cluster evolution and increasing interest in moving applications into public clouds. Performance differences among machines in the same cluster pose new challenges for parallel database systems. First, for database systems running in a heterogeneous cluster, the default uniform data partitioning strategy may overload some of the slow machines, while at the same time it may underutilize the more powerful machines. Since the processing time of a parallel query is determined by the slowest machine, such an allocation strategy may result in a significant query performance degradation. Second, since machines might have varying resources or performance, different choices of machines may lead to different costs or performance for executing the same workload. By carefully selecting the most suitable machines for running a workload, we may achieve better performance with the same budget, or we may meet the same performance requirements with a lower cost. We address these challenges by introducing techniques we call resource bricolage and resource selection that improve database performance in heterogeneous environments. Our approaches quantify the performance differences among machines with various resources as they process workloads with diverse resource requirements. For the purpose of better resource utilization, we formalize the problem of minimizing workload execution time and view it as an optimization problem, and then, we employ linear programming to obtain a recommended data partitioning scheme. For the purpose of better resource selection, we formalize two problems: One minimizes the total workload execution time with a given budget, and the other minimizes the total budget with a given performance target. We then employ different mixed-integer programs to search for the optimal resource selection decisions. We verify the effectiveness of both resource bricolage and resource selection techniques with an extensive experimental study.

Resource Bricolage for Parallel DBMSs on Heterogeneous Clusters

Running parallel database systems in an environment with heterogeneous resources has become increasingly common, due to cluster evolution and increasing interest in moving applications into public clouds or shared infrastructures. For database systems running in a heterogeneous cluster, the default uniform data partitioning strategy may overload some of the slow machines while at the same time it may underutilize the more powerful machines. Since the processing time of a parallel query is determined by the slowest machine, such an allocation strategy may result in a significant query performance degradation. We take a first step to address this problem by introducing a technique we call resource bricolage that improves database performance in heterogeneous environments. Our approach quantifies the performance differences among machines with various resources as they process workloads with diverse resource requirements. We formalize the problem of minimizing workload execution time and view it as an optimization problem, and then we employ linear programming to obtain a recommended data partitioning scheme. We verify the effectiveness of our technique with an extensive experimental study on a commercial database system.

Know your customer: computing k-most promising products for targeted marketing

The advancement of World Wide Web has revolutionized the way the manufacturers can do business. The manufacturers can collect customer preferences for products and product features from their sales and other product-related Web sites to enter and sustain in the global market. For example, the manufactures can make intelligent use of these customer preference data to decide on which products should be selected for targeted marketing. However, the selected products must attract as many customers as possible to increase the possibility of selling more than their respective competitors. This paper addresses this kind of product selection problem. That is, given a database of existing products P from the competitors, a set of company's own products Q, a dataset C of customer preferences and a positive integer k, we want to find k-most promising products (k-MPP) from Q with maximum expected number of total customers for targeted marketing. We model k-MPP query and propose an algorithmic framework for processing such query and its variants. Our framework utilizes grid-based data partitioning scheme and parallel computing techniques to realize k-MPP query. The effectiveness and efficiency of the framework are demonstrated by conducting extensive experiments with real and synthetic datasets.