Quality Of Partitions Research Articles

Proposal for Measuring Quality of Decision Trees Partition

To compute a partition quality for a decision tree, we propose a new measure called NIM “New Information Measure”. The measure is simpler, provides similar performance, and sometimes outperforms the existing measures used with tree-based methods. The experimental results using the MONITDIAB application (Taleb & Atmani, 2013) and datasets from the UCI repository (Asuncion & Newman, 2007) confirm the classification capabilities of our proposal in comparison to the Shannon measure used with ID3 and C4.5 decision tree methods.

Impacts of Subperiod Partitioning on Optimization of Multiperiod Hydrogen Networks

Multiperiod hydrogen network optimization problems may arise from the fluctuations of operating parameters in practical refineries. However, multiperiod operating conditions are always assumed to be known and their effects on the optimization of multiperiod hydrogen network optimization are neglected in current research. To address these problems, a subperiod partitioning method based on clustering of uncertain operating parameters is proposed. A reasonable division of the subperiod and corresponding values of parameters in each subperiod can then be determined by taking a trade-off between the number of subperiods and the quality of subperiod partitioning. The procedure of the proposed method is illustrated via a hydrogen system in a practical refinery. Finally, the effects of subperiod partitioning on the total annual cost and the flexibility of the multiperiod hydrogen network are analyzed and discussed to further verify the application of the proposed method.

A Framework for Community Detection in Large Networks Using Game-Theoretic Modeling

Community detection is a fundamental component of large network analysis. In both academia and industry, progressive research has been made on problems related to community network analysis. Community detection is gaining significant attention and importance in the area of network science. Regular and synthetic complex networks have motivated intense interest in studying the fundamental unifying principles of various complex networks. This paper presents a new game-theoretic approach towards community detection in large-scale complex networks based on modified modularity; this method was developed based on modified adjacency, modified Laplacian matrices and neighborhood similarity. This approach was used to partition a given network into dense communities. It is based on determining a Nash stable partition, which is a pure strategy Nash equilibrium of an appropriately defined strategic game in which the nodes of the network were the players and the strategy of a node was to decide to which community it ought to belong. Players chose to belong to a community according to a maximized fitness/payoff. Quality of the community networks was assessed using modified modularity along with a new fitness function. Community partitioning was performed using Normalized Mutual Information and a ‘modularity measure’, which involved comparing the new game-theoretic community detection algorithm (NGTCDA) with well-studied and well-known algorithms, such as Fast Newman, Fast Modularity Detection, and Louvain Community. The quality of a network partition in communities was evaluated by looking at the contribution of each node and its neighbors against the strength of its community.

A Simple Yet Effective Balanced Edge Partition Model for Parallel Computing

Graph edge partition models have recently become an appealing alternative to graph vertex partition models for distributed computing due to their flexibility in balancing loads and their performance in reducing communication cost [6, 16]. In this paper, we propose a simple yet effective graph edge partitioning algorithm. In practice, our algorithm provides good partition quality (and better than similar state-of-the-art edge partition approaches, at least for power-law graphs) while maintaining low partition overhead. In theory, previous work [6] showed that an approximation guarantee of O ( d max √ log n log k ) apply to the graphs with m =Ω( k 2 ) edges ( k is the number of partitions). We further rigorously proved that this approximation guarantee hold for all graphs. We show how our edge partition model can be applied to parallel computing. We draw our example from GPU program locality enhancement and demonstrate that the graph edge partition model does not only apply to distributed computing with many computer nodes, but also to parallel computing in a single computer node with a many-core processor.

A Simple Yet Effective Balanced Edge Partition Model for Parallel Computing

Graph edge partition models have recently become an appealing alternative to graph vertex partition models for distributed computing due to both their flexibility in balancing loads and their performance in reducing communication cost. In this paper, we propose a simple yet effective graph edge partitioning algorithm. In practice, our algorithm provides good partition quality while maintaining low partition overhead. It also outperforms similar state-of-the-art edge partition approaches, especially for power-law graphs. In theory, previous work showed that an approximation guarantee of O(d max √(log n log k)) apply to the graphs with m=Ω(k 2 ) edges (n is the number of vertices, and k is the number of partitions). We further rigorously proved that this approximation guarantee hold for all graphs. We also demonstrate the applicability of the proposed edge partition algorithm in real parallel computing systems. We draw our example from GPU program locality enhancement and demonstrate that the graph edge partition model does not only apply to distributed computing with many computer nodes, but also to parallel computing in a single computer node with a many-core processor.

Improved spectral clustering for multi-objective controlled islanding of power grid

We propose a two-step algorithm for optimal controlled islanding that partitions a power grid into islands of limited volume while optimizing several criteria: maximizing generator coherency inside islands, minimizing power flow disruption due to teared lines, and minimizing load shedding. Several spectral clusterings strategies are used in the first step to lower the problem dimension (taking into account coherency and disruption only), and CPLEX tools for the mixed-integer quadratic problem are employed in the second step to choose a balanced partition of the aggregated grid that minimizes a combination of coherency, disruption and load shedding. A greedy heuristics efficiently limits search space by generating the starting solution for the exact algorithm. Dimension of the second-step problem depends only on the desired number of islands K instead of the dimension of the original grid. The algorithm is tested on the standard systems with 118, 2383, and 9241 nodes showing high quality of partitions and competitive computation time.

An Information-Theoretic-Cluster Visualization for Self-Organizing Maps.

Improved data visualization will be a significant tool to enhance cluster analysis. In this paper, an information-theoretic-based method for cluster visualization using self-organizing maps (SOMs) is presented. The information-theoretic visualization (IT-vis) has the same structure as the unified distance matrix, but instead of depicting Euclidean distances between adjacent neurons, it displays the similarity between the distributions associated with adjacent neurons. Each SOM neuron has an associated subset of the data set whose cardinality controls the granularity of the IT-vis and with which the first- and second-order statistics are computed and used to estimate their probability density functions. These are used to calculate the similarity measure, based on Renyi's quadratic cross entropy and cross information potential (CIP). The introduced visualizations combine the low computational cost and kernel estimation properties of the representative CIP and the data structure representation of a single-linkage-based grouping algorithm to generate an enhanced SOM-based visualization. The visual quality of the IT-vis is assessed by comparing it with other visualization methods for several real-world and synthetic benchmark data sets. Thus, this paper also contains a significant literature survey. The experiments demonstrate the IT-vis cluster revealing capabilities, in which cluster boundaries are sharply captured. Additionally, the information-theoretic visualizations are used to perform clustering of the SOM. Compared with other methods, IT-vis of large SOMs yielded the best results in this paper, for which the quality of the final partitions was evaluated using external validity indices.

Graph Partitioning for Distributed Graph Processing

There is a large demand for distributed engines that efficiently process large-scale graph data, such as social graph and web graph. The distributed graph engines execute analysis process after partitioning input graph data and assign them to distributed computers, so the quality of graph partitioning largely affects the communication cost and load balance among computers during the analysis process. We propose an effective graph partitioning technique that achieves low communication cost and good load balance among computers at the same time. We first generate more clusters than the number of computers by extending the modularity-based clustering, and then merge those clusters into balanced-size clusters until the number of clusters becomes the number of computers by using techniques designed for graph packing problem. We implemented our technique on top of distributed graph engine, PowerGraph, and made intensive experiments. The results show that our partitioning technique reduces the communication cost so it improves the response time of graph analysis patterns. In particular, PageRank computation is 3.2 times faster at most than HDRF, the state-of-the art of streaming-based partitioning approach.

Partitioning dynamic graph asynchronously with distributed FENNEL

Graph partitioning is important in distributed graph processing. Classical method such as METIS works well on relatively small graphs, but hard to scale for huge, dynamic graphs. Streaming graph partitioning algorithms overcome this issue by processing those graphs as streams. Among these algorithms, FENNEL achieves better edge cut ratio, even close to METIS, but consumes less memory and is significantly faster. On the other hand, graph partitioning may also benefit from distributed graph processing. However, to deploy FENNEL on a cluster, it is important to avoid quality loss and keep efficiency high. The direct implementation of this idea yields a synchronous model and a star-shaped network, which limits both scalability and efficiency. Targeting these two problems, we propose an asynchronous model, combined with a dedicated tree-shaped map-reduce network which is prevail in systems such as Apache Hadoop and Spark, to form AsyncFENNEL (asynchronous FENNEL). We theoretically prove that, the impact on partition quality brought by asynchronous model can be kept as minimal. We test AsyncFENNEL with various synthetic and real-world graphs, the comparison between synchronous and asynchronous models shows that, for streamed natural graphs, AsyncFENNEL can improve performance significantly (above 300% with 8 workers/partitions) with negligible loss on edge cut ratio. However, more worker nodes will introduce a heavier network traffic and reduce efficiency. The proposed tree-shaped map-reduce network can mitigate that impact and increase the performance in that case.

QUALITY ASSESSMENT OF FUZZY CLASSIFICATION: AN APPLICATION TO SOLVENCY ANALYSIS

The aim of this paper is to apply and establish the advantages of using fuzzy classification systems in evaluating the quality of a fuzzy partition. The system is applied in a methodological proposal for the analysis of the business financial solvency in a specific sector. The sectorial analysis of the solvency is considered as an unsupervised fuzzy classification problem and it is sought through the application of the system, to evaluate both the classification and the quality of the established partition, which allows determine whether modify, to eliminate or to add classes to the partition of the sample space.

Efficient Vector Influence Clustering Coefficient Based Directed Community Detection Method

Community detection algorithms are important for determining the character statistics of complex networks. Compared with the conventional community detection algorithms, which always focus on undirected networks, our algorithm is concentrated on directed networks such as the WeChat moments relationship network and the Sina Micro-Blog follower relationship network. To address disadvantages such as lower execution efficiency and higher deviation of precision that current directed community detection algorithms always have, we propose a new approach that is based on the triangle structure of community basis and modeled on the local information transfer process to precisely detect communities in directed networks. Based on the directed vector theory in probability graphs and the dynamic information transfer gain (ITG) of vertices in directed networks, we propose the novel ITG method and the corresponding target optimal function for evaluating the partition quality in a community detection algorithm. Then, we combine ITG and the target function to create the new community detection algorithm ITG-directed weighted community clustering for directed networks. With extensive experiments using artificial network data sets and large, real-world network data sets derived from online social media, our algorithm proved to be more accurate and faster in directed networks than several traditional, well-known community detection methods, such as FastGN, order statistics local optimization method, and Infomap.

SCOUT: simultaneous time segmentation and community detection in dynamic networks.

Many evolving complex real-world systems can be modeled via dynamic networks. An important problem in dynamic network research is community detection, which finds groups of topologically related nodes. Typically, this problem is approached by assuming either that each time point has a distinct community organization or that all time points share a single community organization. The reality likely lies between these two extremes. To find the compromise, we consider community detection in the context of the problem of segment detection, which identifies contiguous time periods with consistent network structure. Consequently, we formulate a combined problem of segment community detection (SCD), which simultaneously partitions the network into contiguous time segments with consistent community organization and finds this community organization for each segment. To solve SCD, we introduce SCOUT, an optimization framework that explicitly considers both segmentation quality and partition quality. SCOUT addresses limitations of existing methods that can be adapted to solve SCD, which consider only one of segmentation quality or partition quality. In a thorough evaluation, SCOUT outperforms the existing methods in terms of both accuracy and computational complexity. We apply SCOUT to biological network data to study human aging.

A layer reduction based community detection algorithm on multiplex networks

Detecting hidden communities is important for the analysis of complex networks. However, many algorithms have been designed for single layer networks (SLNs) while just a few approaches have been designed for multiplex networks (MNs). In this paper, we propose an algorithm based on layer reduction for detecting communities on MNs, which is termed as LRCD-MNs. First, we improve a layer reduction algorithm termed as neighaggre to combine similar layers and keep others separated. Then, we use neighaggre to find the community structure hidden in MNs. Experiments on real-life networks show that neighaggre can obtain higher relative entropy than the other algorithm. Moreover, we apply LRCD-MNs on some real-life and synthetic multiplex networks and the results demonstrate that, although LRCD-MNs does not have the advantage in terms of modularity, it can obtain higher values of surprise, which is used to evaluate the quality of partitions of a network.

Incompatibility Boundaries for Properties of Community Partitions

We prove the incompatibility of certain desirable properties of community partition quality functions. Our results generalize the impossibility result of [Kleinberg 2003] by considering sets of weaker properties. In particular, we use an alternative notion to solve the central issue of the consistency property. (The latter means that modifying the graph in a way consistent with a partition should not have counterintuitive effects). Our results clearly show that community partition methods should not be expected to perfectly satisfy all ideally desired properties. We then proceed to show that this incompatibility no longer holds when slightly relaxed versions of the properties are considered, and we provide examples of simple quality functions satisfying these relaxed properties. An experimental study of these quality functions shows a behavior comparable to established methods in some situations, but more debatable results in others. This suggests that defining a notion of good partition in communities probably requires imposing additional properties.

Improving performance of sparse matrix dense matrix multiplication on large-scale parallel systems

A generic model to scale sparse matrix dense matrix multiplication (SpMM).SpMM suffers from high communication volume overhead.Different volume-based metrics such as maximum volume besides total volume.Simultaneous minimization of volume-based communication cost metrics.Portable models based on graph and hypergraph partitioning. We propose a comprehensive and generic framework to minimize multiple and different volume-based communication cost metrics for sparse matrix dense matrix multiplication (SpMM). SpMM is an important kernel that finds application in computational linear algebra and big data analytics. On distributed memory systems, this kernel is usually characterized with its high communication volume requirements. Our approach targets irregularly sparse matrices and is based on both graph and hypergraph partitioning models that rely on the widely adopted recursive bipartitioning paradigm. The proposed models are lightweight, portable (can be realized using any graph and hypergraph partitioning tool) and can simultaneously optimize different cost metrics besides total volume, such as maximum send/receive volume, maximum sum of send and receive volumes, etc., in a single partitioning phase. They allow one to define and optimize as many custom volume-based metrics as desired through a flexible formulation. The experiments on a wide range of about thousand matrices show that the proposed models drastically reduce the maximum communication volume compared to the standard partitioning models that only address the minimization of total volume. The improvements obtained on volume-based partition quality metrics using our models are validated with parallel SpMM as well as parallel multi-source BFS experiments on two large-scale systems. For parallel SpMM, compared to the standard partitioning models, our graph and hypergraph partitioning models respectively achieve reductions of 14% and 22% in runtime, on average. Compared to the state-of-the-art partitioner UMPa, our graph model is overall 14.5 faster and achieves an average improvement of 19% in the partition quality on instances that are bounded by maximum volume. For parallel BFS, we show on graphs with more than a billion edges that the scalability can significantly be improved with our models compared to a recently proposed two-dimensional partitioning model.

Hypergraph partitioning for social networks based on information entropy modularity

A social network is a typical scale-free network with power-law degree distribution characteristics. It demonstrates several natural imbalanced clusters when it is abstracted as a graph, and expands quickly under its generative mechanism. Hypergraph is superior for modeling multi-user operations in social networks, and partitioning the hypergraph modeled social networks could ease the scaling problems. However, today's popular hypergraph partitioning tools are not sufficiently scalable; thus, unable to achieve high partitioning quality for naturally imbalanced datasets. Recently proposed hypergraph partitioner, hyperpart, replaces the balance constraint with an entropy constraint to achieve high-fidelity partitioning solutions, but it is not tailored for scale-free networks, like social networks. In order to achieve scalable and high quality partitioning results for hypergraph modeled social networks, we propose a partitioning method, EQHyperpart, which utilizes information-Entropy-based modularity Q value (EQ) to direct the hypergraph partitioning process. This EQ considers power-law degree distribution while describing the “natural” structure of scale-free networks. We then apply simulated annealing and introduce a new definition of hyperedge cut, micro cut, to avoid the local minima in convergence of partitioning, developing EQHyperpart into two specific partitioners, namely: EQHyperpart-SA and EQHyperpart-MC. Finally, we evaluate the utility of our proposed method using classical social network datasets, including Facebook dataset. Findings show that EQHyperpart partitioners are more scalable than competing approaches, achieving a tradeoff between modularity retaining and cut size minimizing under balance constraints, and an auto-tradeoff without balance constraints for hypergraph modeled social networks.

Hypergraph+: An Improved Hypergraph-Based Task-Scheduling Algorithm for Massive Spatial Data Processing on Master-Slave Platforms

Spatial data processing often requires massive datasets, and the task/data scheduling efficiency of these applications has an impact on the overall processing performance. Among the existing scheduling strategies, hypergraph-based algorithms capture the data sharing pattern in a global way and significantly reduce total communication volume. Due to heterogeneous processing platforms, however, single hypergraph partitioning for later scheduling may be not optimal. Moreover, these scheduling algorithms neglect the overlap between task execution and data transfer that could further decrease execution time. In order to address these problems, an extended hypergraph-based task-scheduling algorithm, named Hypergraph+, is proposed for massive spatial data processing. Hypergraph+ improves upon current hypergraph scheduling algorithms in two ways: (1) It takes platform heterogeneity into consideration offering a metric function to evaluate the partitioning quality in order to derive the best task/file schedule; and (2) It can maximize the overlap between communication and computation. The GridSim toolkit was used to evaluate Hypergraph+ in an IDW spatial interpolation application on heterogeneous master-slave platforms. Experiments illustrate that the proposed Hypergraph+ algorithm achieves on average a 43% smaller makespan than the original hypergraph scheduling algorithm but still preserves high scheduling efficiency.

A modified two-stage Markov clustering algorithm for large and sparse networks

BackgroundGraph-based hierarchical clustering algorithms become prohibitively costly in both execution time and storage space, as the number of nodes approaches the order of millions. ObjectiveA fast and highly memory efficient Markov clustering algorithm is proposed to perform the classification of huge sparse networks using an ordinary personal computer. MethodsImprovements compared to previous versions are achieved through adequately chosen data structures that facilitate the efficient handling of symmetric sparse matrices. Clustering is performed in two stages: the initial connected network is processed in a sparse matrix until it breaks into isolated, small, and relatively dense subgraphs, which are then processed separately until convergence is obtained. An intelligent stopping criterion is also proposed to quit further processing of a subgraph that tends toward completeness with equal edge weights. The main advantage of this algorithm is that the necessary number of iterations is separately decided for each graph node. ResultsThe proposed algorithm was tested using the SCOP95 and large synthetic protein sequence data sets. The validation process revealed that the proposed method can reduce 3–6 times the processing time of huge sequence networks compared to previous Markov clustering solutions, without losing anything from the partition quality. ConclusionsA one-million-node and one-billion-edge protein sequence network defined by a BLAST similarity matrix can be processed with an upper-class personal computer in 100 minutes. Further improvement in speed is possible via parallel data processing, while the extension toward several million nodes needs intermediary data storage, for example on solid state drives.

Balanced, non-contiguous partitioning of power systems considering operational constraints

This paper presents an integer programming based partitioning of dynamic graphs that arise in power systems. The proposed approach allows for explicitly expressing power systems operational constraints in the partitioning algorithm. Balanced, non-contiguous graphs appear in several power systems applications such as network partitioning for high performance computing based parallel transient simulators and in wide area control. Quality of partition obtained using the developed algorithm compares favorably with a well known multi-level graph partitioning approach – METIS. Several test systems, ranging from a 9 bus test system to the 2383 bus western polish test system is used to demonstrate the applicability of proposed approach for power system partitioning problems. In addition, applicability of the proposed approach for a non-stationary system such as power networks is demonstrated by partitioning in real-time. Real-time in this context is defined as the interval between two dispatches as the edge weights of the dynamic graph is expected to change at every dispatch.

MTP: discovering high quality partitions in real world graphs

Given a real world graph, how can we find a large subgraph whose partition quality is much better than the original? How can we use a partition of that subgraph to discover a high quality global partition? Although graph partitioning especially with balanced sizes has received attentions in various applications, it is known NP-hard, and also known that there is no good cut at a large scale for real graphs. In this paper, we propose a novel approach for graph partitioning. Our first focus is on finding a large subgraph with high quality partitions, in terms of conductance. Despite the difficulty of the task for the whole graph, we observe that there is a large connected subgraph whose partition quality is much better than the original. Our proposed method MTP finds such a subgraph by removing "hub" nodes with large degrees, and taking the remaining giant connected component. Further, we extend MTP to gb MTP (Global Balanced MTP) for discovering a global balanced partition. gb MTP attaches the excluded nodes in MTP to the partition found by MTP in a greedy way. In experiments, we demonstrate that MTP finds a subgraph of a large size with low conductance graph partitions, compared with competing methods. We also show that the competitors cannot find connected subgraphs while our method does, by construction. This improvement in partition quality for the subgraph is especially noticeable for large scale cuts--for a balanced partition, down to 14 % of the original conductance with the subgraph size 70 % of the total. As a result, the found subgraph has clear partitions at almost all scales compared with the original. Moreover, gb MTP generally discovers global balanced partitions whose conductance are lower than those found by METIS, the state-of-the-art graph partitioning method.