High Quality Partitions Research Articles

Evolutionary Multiobjective Clustering Over Multiple Conflicting Data Views

Multi-view data analysis provides an effective means to integrate the distinct information sources which are inherent to many applications. Data clustering in a multi-view setting specifically aims to identify the most appropriate grouping for a collection of entities, where those entities (or their relationships) can be described from multiple perspectives. Leveraging recent advances in multi-objective clustering, we propose a new evolutionary method to tackle this challenge. Designed around a flexible and unbiased solution representation, together with strategies based on the minimum spanning tree and neighborhood relations, our algorithm optimizes multiple objectives simultaneously to effectively explore the space of candidate trade-offs between the data views. Through a series of experiments, we investigate the suitability of our proposal in the context of a bioinformatics application, clustering of plausible protein structures, and a diverse set of synthetic problems. The specific case of two data views is considered in this paper. The evaluation with respect to a variety of reference approaches demonstrates the effectiveness of our method in discovering high-quality partitions in a multiview setting. Robustness against unreliable data sources and the ability to automatically determine the number of clusters, are additional advantages evidenced by the results obtained.

HSLS: An efficient local search algorithm for the hardware and software partitioning problem

Hardware and software partitioning (HW/SW) is a crucial step in the co-design of embedded systems, which is a typical combinatorial optimisation problem. In practice, HW/SW instances would be of large scale, so it is of significant importance to design effective heuristic algorithms to obtain a high-quality partitioning scheme. Different from previous heuristics, the HW/SW problem is treated as the minimum weight dominating set (MWDS) problem in this letter, and the mature local search framework for solving the MWDS problem is introduced to solve the HW/SW problem. In addition, two suitable vertex selection rules based on scoring function and tabu are proposed. Combined with the introduced local search framework, an efficient local search algorithm named HSLS is obtained. 96 classic benchmark instances are used to evaluate the performance of the HSLS algorithm, and the four latest evolutionary algorithms designed for the HW/SW problem are used as comparison algorithms, which are genetic algorithm, binary particle swarm optimisation, differential evolution, and group theory-based optimisation algorithm. The experimental results show that the HSLS algorithm is far superior to the four comparison algorithms in terms of the optimal solution and obtains optimal solution on 78 benchmark instances, the four comparison algorithms obtain 43, 29, 42, and 41 optimal solutions respectively.

Practical and high-quality partitioning algorithm for large-scale and time-evolving graphs

With the appearance of widespread large-scale graph data, distributed graph processing grows in popularity. In order to store and analyze a large-scale graph in a distributed manner, the graph should be properly partitioned in advance. Although the partitioning of static graphs has been sufficiently investigated, the emerging graphs nowadays are typically dynamic or time-evolving. State-of-the-art partitioning algorithms for such large-scale and time-evolving graphs either are impractical because of the complexity caused by the global optimization or sacrifice partitioning quality due to a huge number of cross partition edges. This paper investigates the problem and proposes a new practical and high-quality graph partitioning algorithm. First, graph updates accumulated in a time interval are expressed as a delta graph. Second, vertices in the delta graph are classified into several subsets based on a non-overlapping community detection method. Third, vertices in the same subset are assigned to a properly chosen partition as a whole. The chosen partition is lightly loaded as well as closely related with vertices in the subset. Experimental results show that compared with the widely used hash-based and heuristic-based partitioning algorithms, our proposed algorithm gains significant decrease in terms of the number of cross partition edges while maintaining a similar level of load balance.

Multilevel Graph Partitioning for Three-Dimensional Discrete Fracture Network Flow Simulations

We present a topology-based method for mesh-partitioning in three-dimensional discrete fracture network (DFN) simulations that takes advantage of the intrinsic multi-level nature of a DFN. DFN models are used to simulate flow and transport through low-permeability fractured media in the subsurface by explicitly representing fractures as discrete entities. The governing equations for flow and transport are numerically integrated on computational meshes generated on the interconnected fracture networks. Modern high-fidelity DFN simulations require high-performance computing on multiple processors where performance and scalability depends partially on obtaining a high-quality partition of the mesh to balance work-loads and minimize communication across all processors. The discrete structure of a DFN naturally lends itself to various graph representations, which can be thought of as coarse-scale representations of the computational mesh. Using this concept, we develop two applications of the multilevel graph partitioning algorithm to partition the mesh of a DFN. In the first, we project a partition of the graph based on the DFN topology onto the mesh of the DFN and in the second, this DFN-based projection is used as the initial condition for further partitioning refinement of the mesh. We compare the performance of these methods with standard multi-level graph partitioning using graph-based metrics (cut, imbalance, partitioning time), computational-based metrics (FLOPS, iterations, solver time), and total run time. The DFN-based and the mesh-based partitioning methods are comparable in terms of the graph-based metrics, but the time required to obtain the partition is several orders of magnitude faster using the DFN-based partitions. The computation-based metrics show comparable performance between both methods so, in combination, the DFN-based partitions are several orders of magnitude faster than the mesh-based partition. Moreover, the method which uses the DFN-partition solution as the initial condition of the mesh partition provided cut and imbalance values that were close to the mesh-based partition but in a fraction of the time. In turn, this hybrid method outperformed both of the other methods in terms of the total run time.

Signal, bias, and the role of transcriptome assembly quality in phylogenomic inference

BackgroundPhylogenomic approaches have great power to reconstruct evolutionary histories, however they rely on multi-step processes in which each stage has the potential to affect the accuracy of the final result. Many studies have empirically tested and established methodology for resolving robust phylogenies, including selecting appropriate evolutionary models, identifying orthologs, or isolating partitions with strong phylogenetic signal. However, few have investigated errors that may be initiated at earlier stages of the analysis. Biases introduced during the generation of the phylogenomic dataset itself could produce downstream effects on analyses of evolutionary history. Transcriptomes are widely used in phylogenomics studies, though there is little understanding of how a poor-quality assembly of these datasets could impact the accuracy of phylogenomic hypotheses. Here we examined how transcriptome assembly quality affects phylogenomic inferences by creating independent datasets from the same input data representing high-quality and low-quality transcriptome assembly outcomes.ResultsBy studying the performance of phylogenomic datasets derived from alternative high- and low-quality assembly inputs in a controlled experiment, we show that high-quality transcriptomes produce richer phylogenomic datasets with a greater number of unique partitions than low-quality assemblies. High-quality assemblies also give rise to partitions that have lower alignment ambiguity and less compositional bias. In addition, high-quality partitions hold stronger phylogenetic signal than their low-quality transcriptome assembly counterparts in both concatenation- and coalescent-based analyses.ConclusionsOur findings demonstrate the importance of transcriptome assembly quality in phylogenomic analyses and suggest that a portion of the uncertainty observed in such studies could be alleviated at the assembly stage.

Genetic algorithm of optimizing the qualification of programmer teams

The partitioning a set of professional programmers into a set of teams when a programming project specifies requirements to the competency in various programming technologies and tools is a hard combinatorial problem. The paper proposes a genetic algorithm, which is capable of finding competitive and high-quality partitioning solutions in acceptable runtime. The algorithm introduces chromosomes in such a way as to assign each programmer to a team, define the team staff and easily reconstruct the teams during optimization process. A fitness function characterizes each chromosome with respect to the quality of the programmers partitioning. It accounts for the average qualification of teams and the qualification of team best representatives on each of the technologies. The function recognizes the teams that meet all constraints on the project and are workable from this point of view. It is also capable of recognizing the teams that do not meet the constraints and are unworkable. The algorithm defines the genetic operations of selection, crossing and mutation in such a way as to move programmers from unworkable to workable teams, to increase the number of workable teams, to ex-change programmers among workable teams, to increase the competency of every workable team, and thus to maximize the teams overall qualification. Experimental results obtained on a set of programmers graduated from Belarus universities show the capability of the genetic algorithm to find good partitioning solutions, maximize the teams’ competency and minimize the number of unemployed programmers.

Scalable, Multi-Constraint, Complex-Objective Graph Partitioning

We introduce XtraPuLP , a distributed-memory graph partitioner designed to process irregular trillion-edge graphs. XtraPuLP is based on the scalable label propagation community detection technique, which has been demonstrated in various prior works as a viable means to produce high quality partitions of skewed and small-world graphs with minimal computation time. Our XtraPuLP implementation can also be generalized to compute partitions with an arbitrary number of constraints, and it can compute partitions with balanced communication load across all parts. On a collection of large sparse graphs, we show that XtraPuLP partitioning is considerably faster than state-of-the-art partitioning methods, while also demonstrating that XtraPuLP can produce partitions of real-world graphs with billion+ vertices and over a hundred billion edges in minutes. Additionally, we demonstrate XtraPuLP on a variety of applications, including large-scale graph analytics and sparse matrix-vector multiplication.

Data clustering via cooperative games: A novel approach and comparative study

Arguably, the main purpose of cluster analysis is to develop algorithms to reveal natural groupings (clusterings) over a set of data points based on their similarity. On the other hand, the focus of cooperative game theory (CGT) is to study the formation of groups (coalitions) of decision makers (players) and ways to split the resulting income among them. Due to the conceptual similitude between these fields, algorithms rooted in CGT have recently emerged for tackling the data clustering problem. In this work, we revisit two such algorithms, one based on cluster prototypes (Biobjective Game Clustering – BiGC) and the other based on dense regions of data points (Density-Restricted Agglomerative Clustering – DRAC). We also present a novel partitional clustering algorithm, referred to as HGC (after Hedonic Game based Clustering), which is grounded on theoretical results stemming from the subclass of hedonic games. Two HGC versions are investigated, which differ in the order of the players in the game, and a detailed factorial simulation study is reported to analyze how sensitive they are to three relevant factors, namely number of clusters, number of features, and noise level. Besides, a heuristic to calibrate the value of HGC’s single control parameter (viz., the number of nearest neighbors of each point) is provided, so as to yield high-quality partitions. To compare the performance of the CGT algorithms, a series of experiments were conducted on UCI and gene-expression data sets, the majority of which being high dimensional. Overall, the results measured by 10 external validation indices evidence that HGC is usually more stable and effective than DRAC and BiGC. They also show that HGC is very competitive (sometimes, considerably better) to well-known clustering algorithms/variants (specifically, k-means, k-means++, affinity propagation, two variants of hierarchical clustering, and the density peak clustering algorithm). Remarkably, HGC could fully recover the true clustering structures for two gene-expression data sets.

R*-Grove: Balanced Spatial Partitioning for Large-Scale Datasets

The rapid growth of big spatial data urged the research community to develop several big spatial data systems. Regardless of their architecture, one of the fundamental requirements of all these systems is to spatially partition the data efficiently across machines. The core challenges of big spatial partitioning are building high spatial quality partitions while simultaneously taking advantages of distributed processing models by providing load balanced partitions. Previous works on big spatial partitioning are to reuse existing index search trees as-is, e.g., the R-tree family, STR, Kd-tree, and Quad-tree, by building a temporary tree for a sample of the input and use its leaf nodes as partition boundaries. However, we show in this paper that none of those techniques has addressed the mentioned challenges completely. This paper proposes a novel partitioning method, termed R*-Grove, which can partition very large spatial datasets into high quality partitions with excellent load balance and block utilization. This appealing property allows R*-Grove to outperform existing techniques in spatial query processing. R*-Grove can be easily integrated into any big data platforms such as Apache Spark or Apache Hadoop. Our experiments show that R*-Grove outperforms the existing partitioning techniques for big spatial data systems. With all the proposed work publicly available as open source, we envision that R*-Grove will be adopted by the community to better serve big spatial data research.

Interpretable clustering: an optimization approach

State-of-the-art clustering algorithms provide little insight into the rationale for cluster membership, limiting their interpretability. In complex real-world applications, the latter poses a barrier to machine learning adoption when experts are asked to provide detailed explanations of their algorithms’ recommendations. We present a new unsupervised learning method that leverages Mixed Integer Optimization techniques to generate interpretable tree-based clustering models. Utilizing a flexible optimization-driven framework, our algorithm approximates the globally optimal solution leading to high quality partitions of the feature space. We propose a novel method which can optimize for various clustering internal validation metrics and naturally determines the optimal number of clusters. It successfully addresses the challenge of mixed numerical and categorical data and achieves comparable or superior performance to other clustering methods on both synthetic and real-world datasets while offering significantly higher interpretability.

Indoor mapping and modeling by parsing floor plan images

ABSTRACT A large proportion of indoor spatial data is generated by parsing floor plans. However, a mature and automatic solution for generating high-quality building elements (e.g., walls and doors) and space partitions (e.g., rooms) is still lacking. In this study, we present a two-stage approach to indoor mapping and modeling (IMM) from floor plan images. The first stage vectorizes the building elements on the floor plan images and the second stage repairs the topological inconsistencies between the building elements, separates indoor spaces, and generates indoor maps and models. To reduce the shape complexity of indoor boundary elements, i.e., walls and openings, we harness the regularity of the boundary elements and extract them as rectangles in the first stage. Furthermore, to resolve the overlaps and gaps of the vectorized results, we propose an optimization model that adjusts the rectangle vertex coordinates to conform to the topological constraints. Experiments demonstrate that our approach achieves a considerable improvement in room detection without conforming to Manhattan World Assumption. Our approach also outputs instance-separate walls with consistent topology, which enables direct modeling into Industry Foundation Classes (IFC) or City Geography Markup Language (CityGML).

H2Pregel : A partition-based hybrid hierarchical graph computation approach

A partition-based hybrid hierarchical graph computation approach, called H2Pregel is proposed to address the redundant supersteps and inefficient computation problems due to low access locality. The H2Pregel preprocesses the input graph through a distributed recode algorithm to ensure the continuity and sequence of vertex ids, then employs a hybrid approach to combine the advantages of both synchronous and asynchronous models, and hierarchically computes the high proportion of interior messages generated by high quality partition algorithms. Moreover, H2Pregel leverages configurable parallel threads to accelerate local computation by “sub-supersteps”, and employs an exterior messages stealing optimization to avoid extra communication overheads between tasks. We implemented H2Pregel on Giraph, a classic open source system based on Pregel. The evaluation results on large-scale graphs show that, compared with Pregel in three partition algorithms, H2Pregel can achieve average speedups by 1.12–4.52 times and decrease average communication messages by 23.5%-55.5%, and average supersteps by 15.8%-82.0%.

A fuzzy clustering based method for attributed graph partitioning

Graph partitioning methods in data mining have been widely used to discover protein complexes in protein–protein interaction (PPI) network. However, PPI networks with attributes need more effective attribute graph partitioning methods. Attribute graph partitioning aims to obtain high quality partitions satisfying the requirement: nodes in the same partition not only connect to each other more densely but also share more similar attribute values. In this paper, we propose a novel method for attributed graph partitioning based on fuzzy clustering. This method firstly devises a unified similarity measure using SimRank to construct the fuzzy similarity matrix of the attributed graph and can integrate structural and attribute similarities of nodes into a flexible weighted framework. Then it deduces the corresponding fuzzy equivalent matrix using fuzzy set theory. Finally, the result of partitioning can be obtained using fuzzy clustering algorithm. We conduct some experiments on several typical attributed graphs, which can also simulate PPI networks with attributes. The results show that our method is very effective to identify high quality partitions of attributed graphs and even performs better than some representative methods.

A K-way spectral partitioning of an ontology for ontology matching

Ontology matching, the process of resolving heterogeneity between two ontologies consumes a lot of computing memory and time. This problem is exacerbated in large ontology matching tasks. To address the problem of time and space complexity in the matching process, ontology partitioning has been adopted as one of the methods, however, most ontology partitioning algorithms either produce incomplete partitions or are slow in the partitioning process hence eroding the benefits of the partitioning. In this paper, we demonstrate that spectral partitioning of an ontology can generate high quality partitions geared towards ontology matching.

Revisiting graph construction for fast image segmentation

In this paper, we propose a simple but effective method for fast image segmentation. We re-examine the locality-preserving character of spectral clustering by constructing a graph over image regions with both global and local connections. Our novel approach to build graph connections relies on two key observations: 1) local region pairs that co-occur frequently will have a high probability to reside on a common object; 2) spatially distant regions in a common object often exhibit similar visual saliency, which implies their neighborship in a manifold. We present a novel energy function to efficiently conduct graph partitioning. Based on multiple high quality partitions, we show that the generated eigenvector histogram based representation can automatically drive effective unary potentials for a hierarchical random field model to produce multi-class segmentation. Sufficient experiments, on the BSDS500 benchmark, large-scale PASCAL VOC and COCO datasets, demonstrate the competitive segmentation accuracy and significantly improved efficiency of our proposed method compared with other state of the arts.

Parallel Graph Partitioning for Complex Networks

Processing large complex networks like social networks or web graphs has attracted considerable interest. To do this in parallel, we need to partition them into pieces of roughly equal size. Unfortunately, previous parallel graph partitioners originally developed for more regular mesh-like networks do not work well for complex networks. Here we address this problem by parallelizing and adapting the label propagation technique originally developed for graph clustering. By introducing size constraints, label propagation becomes applicable for both the coarsening and the refinement phase of multilevel graph partitioning. This way we exploit the hierarchical cluster structure present in many complex networks. We obtain very high quality by applying a highly parallel evolutionary algorithm to the coarsest graph. The resulting system is both more scalable and achieves higher quality than state-of-the-art systems like ParMetis or PT-Scotch. For large complex networks the performance differences are very big. As an example, our algorithm partitions a web graph with 3.3 G edges in 16 seconds using 512 cores of a high-performance cluster while producing a high quality partition—none of the competing systems can handle this graph on our system.

A physics-motivated Centroidal Voronoi Particle domain decomposition method

In this paper, we propose a novel domain decomposition method for large-scale simulations in continuum mechanics by merging the concepts of Centroidal Voronoi Tessellation (CVT) and Voronoi Particle dynamics (VP). The CVT is introduced to achieve a high-level compactness of the partitioning subdomains by the Lloyd algorithm which monotonically decreases the CVT energy. The number of computational elements between neighboring partitioning subdomains, which scales the communication effort for parallel simulations, is optimized implicitly as the generated partitioning subdomains are convex and simply connected with small aspect-ratios. Moreover, Voronoi Particle dynamics employing physical analogy with a tailored equation of state is developed, which relaxes the particle system towards the target partition with good load balance. Since the equilibrium is computed by an iterative approach, the partitioning subdomains exhibit locality and the incremental property. Numerical experiments reveal that the proposed Centroidal Voronoi Particle (CVP) based algorithm produces high-quality partitioning with high efficiency, independently of computational-element types. Thus it can be used for a wide range of applications in computational science and engineering.

Hypergraph partitioning for social networks based on information entropy modularity

A social network is a typical scale-free network with power-law degree distribution characteristics. It demonstrates several natural imbalanced clusters when it is abstracted as a graph, and expands quickly under its generative mechanism. Hypergraph is superior for modeling multi-user operations in social networks, and partitioning the hypergraph modeled social networks could ease the scaling problems. However, today's popular hypergraph partitioning tools are not sufficiently scalable; thus, unable to achieve high partitioning quality for naturally imbalanced datasets. Recently proposed hypergraph partitioner, hyperpart, replaces the balance constraint with an entropy constraint to achieve high-fidelity partitioning solutions, but it is not tailored for scale-free networks, like social networks. In order to achieve scalable and high quality partitioning results for hypergraph modeled social networks, we propose a partitioning method, EQHyperpart, which utilizes information-Entropy-based modularity Q value (EQ) to direct the hypergraph partitioning process. This EQ considers power-law degree distribution while describing the “natural” structure of scale-free networks. We then apply simulated annealing and introduce a new definition of hyperedge cut, micro cut, to avoid the local minima in convergence of partitioning, developing EQHyperpart into two specific partitioners, namely: EQHyperpart-SA and EQHyperpart-MC. Finally, we evaluate the utility of our proposed method using classical social network datasets, including Facebook dataset. Findings show that EQHyperpart partitioners are more scalable than competing approaches, achieving a tradeoff between modularity retaining and cut size minimizing under balance constraints, and an auto-tradeoff without balance constraints for hypergraph modeled social networks.

Partitioning (hierarchically clustered) complex networks via size-constrained graph clustering

The most commonly used method to tackle the graph partitioning problem in practice is the multilevel metaheuristic. In this paper we introduce size-constrained label propagation (SCLaP) and show how it can be used to instantiate both the coarsening phase and the refinement phase of multilevel graph partitioning. We mainly target networks with highly irregular and hierarchically clustered structure (but other network types can be partitioned as well). Additionally, we augment the basic algorithm with several extensions to further improve its speed and/or solution quality. Depending on the configuration of the resulting partitioner using SCLaP, we are able to compute high-quality partitions outperforming all competitors, or instead, to compute similarly good partitions as the best competitor in terms of quality, hMetis, while being an order of magnitude faster. Our fastest configuration partitions the largest real-world graph in our study (it has 3.3 billion edges) with sequential code in about ten minutes while cutting less than half of the edges than the fastest competitor, kMetis.

MTP: discovering high quality partitions in real world graphs

Given a real world graph, how can we find a large subgraph whose partition quality is much better than the original? How can we use a partition of that subgraph to discover a high quality global partition? Although graph partitioning especially with balanced sizes has received attentions in various applications, it is known NP-hard, and also known that there is no good cut at a large scale for real graphs. In this paper, we propose a novel approach for graph partitioning. Our first focus is on finding a large subgraph with high quality partitions, in terms of conductance. Despite the difficulty of the task for the whole graph, we observe that there is a large connected subgraph whose partition quality is much better than the original. Our proposed method MTP finds such a subgraph by removing "hub" nodes with large degrees, and taking the remaining giant connected component. Further, we extend MTP to gb MTP (Global Balanced MTP) for discovering a global balanced partition. gb MTP attaches the excluded nodes in MTP to the partition found by MTP in a greedy way. In experiments, we demonstrate that MTP finds a subgraph of a large size with low conductance graph partitions, compared with competing methods. We also show that the competitors cannot find connected subgraphs while our method does, by construction. This improvement in partition quality for the subgraph is especially noticeable for large scale cuts--for a balanced partition, down to 14 % of the original conductance with the subgraph size 70 % of the total. As a result, the found subgraph has clear partitions at almost all scales compared with the original. Moreover, gb MTP generally discovers global balanced partitions whose conductance are lower than those found by METIS, the state-of-the-art graph partitioning method.