Quality Of Partitions Research Articles

Robust Multisensor MeMBer Filter for Multiple Extended-Target Tracking

This paper develops a robust extended-target multisensor multitarget multi-Bernoulli (ET-MS-MeMBer) filter for enhancing the unsatisfactory quality of measurement partitions arising in the classical ET-MS-MeMBer filter due to increased clutter intensities. Specifically, the proposed method considers the influence of the clutter measurement set by introducing the ratio of the target likelihood to the clutter likelihood. With the constraint of the clutter measurement set, it can obtain better multisensor measurement partitioning results under the original two-step greedy partitioning mechanism. Subsequently, the single-target multisensor likelihood function for the clutter case is derived. Simulation results reveal a favorable comparison to the ET-MS-MeMBer filter in terms of accuracy in estimating the target cardinality and target state under conditions with increased clutter intensities.

Enhancing Balanced Graph Edge Partition with Effective Local Search

Graph partition is a key component to achieve workload balance and reduce job completion time in parallel graph processing systems. Among the various partition strategies, edge partition has demonstrated more promising performance in power-law graphs than vertex partition and thereby has been more widely adopted as the default partition strategy by existing graph systems. The graph edge partition problem, which is to split the edge set into multiple balanced parts with the objective of minimizing the total number of copied vertices, has been widely studied from the view of optimization and algorithms. In this paper, we study local search algorithms for this problem to further improve the partition results from existing methods. More specifically, we propose two novel concepts, namely adjustable edges and blocks. Based on these, we develop a greedy heuristic as well as an improved search algorithm utilizing the property of max-flow model. To evaluate the performance of our algorithms, we first provide adequate theoretical analysis in terms of approximation quality. We significantly improve the previous known approximation ratio for this problem. Then we conduct extensive experiments on a large number of benchmark datasets and state-of-the-art edge partition strategies. The results show that our proposed local search framework can further improve the quality of graph partition by a wide margin.

A stable community detection approach for complex network based on density peak clustering and label propagation

Dividing a network into communities has great benefits in understanding the characteristics of the network. The label propagation algorithm (LPA) is a fast and convenient community detection algorithm. However, the community initialization of LPA does not take advantage of topological information of networks, and its robustness is poor. In this paper, we propose a stable community detection algorithm based on density peak clustering and label propagation (DS-LPA). First, the local density calculation method in density peak clustering algorithm is improved in finding the community center of the network, so as to build a suitable initial community, which can improve the quality of community partition. Then, the label update order is determined reasonably by computing the information transmission power of nodes, and the solutions for multiple candidate labels are provided, which greatly improved the robustness of the algorithm. DS-LPA is compared with other seven algorithms on the synthetic network and real-world networks. NMI, ARI, and modularity are used to evaluate these algorithms. It can be concluded that DS-LPA has a higher performance than most comparison algorithms on synthetic network with ten different mixed parameters by statistical testing. And DS-LPA can quickly calculate the best community partition on different sizes of real-world networks.

Power grid partition with improved biogeography-based optimization algorithm

With the continuous expansion of the scale of the interconnected power grid and the large amount of new energy access, the power grid suffers from a wider range of faults. Unlocking the electromagnetic ring network and performing layered and partitioned operation of the power grid is the future development trend of the power grid. Grid partition operation can effectively enhance the controllability within each sub-partition of the power grid, the decoupling between the partitions and the robustness of cascade failure. Depending on the power grid topology and electrical characteristics, this paper proposes a new power grid partition algorithm based on the improved biogeography-based optimization (BBO) algorithm, which improves the algorithm's calculation speed and partition quality. The algorithm, first, analyzes the physical and operational characteristics of the power network, and determines the zoning principle based on the actual situation; next combined with the definition of relevant parameters of the complex network theory, constructs a topology model that conforms to the characteristics of the power grid; at the same time, considering the small-world characteristics of the power grid topology, the original biogeography is improved the update method of algorithm node information exchange reduces the time complexity of the algorithm and makes it more suitable for modern power grids. Then, according to the value of the improved modularity function and the grid zoning principle, the optimal zoning plan is output. Finally, the IEEE-39 node system and the IEEE-118 node system are designated as calculation examples. The extensive calculation example results demonstrate that the method of community network analysis considering the physical characteristics of the power grid is reasonable to a certain extent. The calculation method is simple and fast, which meets the needs of complex power grid analysis and engineering calculation.

Sphynx: A parallel multi-GPU graph partitioner for distributed-memory systems

Graph partitioning has been an important tool to partition the work among several processors to minimize the communication cost and balance the workload. While accelerator-based supercomputers are emerging to be the standard, the use of graph partitioning becomes even more important as applications are rapidly moving to these architectures. However, there is no distributed-memory-parallel, multi-GPU graph partitioner available for applications. We developed a spectral graph partitioner, Sphynx, using the portable, accelerator-friendly stack of the Trilinos framework. In Sphynx, we allow using different preconditioners and exploit their unique advantages. We use Sphynx to systematically evaluate the various algorithmic choices in spectral partitioning with a focus on the GPU performance. We perform those evaluations on two distinct classes of graphs: regular (such as meshes, matrices from finite element methods) and irregular (such as social networks and web graphs), and show that different settings and preconditioners are needed for these graph classes. The experimental results on the Summit supercomputer show that Sphynx is the fastest alternative on irregular graphs in an application-friendly setting and obtains a partitioning quality close to ParMETIS on regular graphs. When compared to nvGRAPH on a single GPU, Sphynx is faster and obtains better balance and better quality partitions. Sphynx provides a good and robust partitioning method across a wide range of graphs for applications looking for a GPU-based partitioner.

A Workload-Adaptive Streaming Partitioner for Distributed Graph Stores

Streaming graph partitioning methods have recently gained attention due to their ability to scale to very large graphs with limited resources. However, many such methods do not consider workload and graph characteristics. This may degrade the performance of queries by increasing inter-node communication and computational load imbalance. Moreover, existing workload-aware methods cannot consistently provide good performance as they do not consider dynamic workloads that keep emerging in graph applications. We address these issues by proposing a novel workload-adaptive streaming partitioner named WASP, that aims to achieve low-latency and high-throughput online graph queries. As each workload typically contains frequent query patterns, WASP exploits the existing workload to capture active vertices and edges which are frequently visited and traversed, respectively. This information is used to heuristically improve the quality of partitions either by avoiding the concentration of active vertices in a few partitions proportional to their visit frequencies or by reducing the probability of the cut of active edges proportional to their traversal frequencies. In order to assess the impact of WASP on a graph store and to show how easily the approach can be plugged on top of the system, we exploit it in a distributed graph-based RDF store. Our experiments over three synthetic and real-world graph datasets and the corresponding static and dynamic query workloads show that WASP achieves a better query performance against state-of-the-art graph partitioners, especially in dynamic query workloads.

Effective partitioning mechanisms for time-evolving graphs in the Flink system

Graphs are suitable data structures for expressing the relationship between different types of data. With a continuous increase in the graph size, using suitable methods to divide graphs and parallelize the processing load becomes crucial. Balanced graph partitioning has been extensively studied for static and streaming graphs. However, for a time-evolving graph (TEG), whose size and structure are periodically updated, related partitioning methods are lacking. A straightforward approach is to capture snapshots of a TEG and adopt the partitioning methods designed for static or streaming graphs. Although feasible partitioning quality can be expected, the time overhead is high due to frequent repartitioning. This paper proposes two TEG partitioning methods, namely seed and similarity, to decrease the partitioning time. According to the experimental results, on average, seed and similarity require 29–39% of the partitioning time required by snapshot. Moreover, the proposed methods maintain reasonable partitioning quality.

Evaluating Evolving Structure in Streaming Data With Modified Dunn's Indices

Dunn's internal cluster validity index is used to assess partition quality and identify a “best” crisp c-partition of n objects built from static data sets. This index is quite sensitive to inliers and outliers in the input data, so a subsequent study developed a family of 17 generalized Dunn's indices that extend and improve the original measure in various ways. This paper presents online versions of two modified generalized Dunn's indices that can be used for the dynamic evaluation of an evolving (cluster) structure in streaming data. We argue that this method is a good way to monitor the ongoing performance of streaming clustering algorithms, and we illustrate several types of inferences that can be drawn from such indices. Streaming clustering algorithms are incremental, process incoming data points only once and then discard them, adapt as the data stream evolves, flag outliers, and most importantly, spawn new emerging structures. We compare the two new indices to the incremental Xie-Beni and Davies-Boudin indices, which to our knowledge offer the only comparable approach, with numerical examples on a variety of synthetic and real datasets.

Group Reassignment for Dynamic Edge Partitioning

Graph partitioning is a mandatory step in large-scale distributed graph processing. When partitioning real-world power-law graphs, the edge partitioning algorithm performs better than the traditional vertex partitioning algorithm, because it can cut a single vertex into multiple replicas to apportion the computation. Many advanced edge partitioning methods are designed for partitioning a static graph from scratch. However, the real-world graph structure changes continuously, which leads to a decrease in partition quality and affects the performance of the graph applications. Some studies are devoted to offline repartitioning or batch incremental partitioning, but how to deal with dynamics in real-time is still worthy of in-depth study. In this article, we discuss the impact of dynamic change on partition and discover that both insertion and deletion will lead to local suboptimal partitioning, which is the reason for the degradation of partition quality. As a solution, a dynamic edge partitioning algorithm is proposed to partition dynamics in real-time. Specifically, we deal with dynamics by a distributed stream and improve partition quality by reassigning some closely connected edges. Experiments show that it is robust to initial partition quality, dynamic scale and type, and distributed scale. Compared with the state-of-the-art dynamic partitioner, it can reduce vertex-cuts by 29.5 percent. Compared with the repartitioning algorithms, it can save the partitioning time by 91.0 percent. Applied on the graph task, it can reduce the increase of communication cost and the increase of the total time of task by 41.5 and 71.4 percent.

On using the modularity of recurrence network communities to detect change-point behaviour

The behaviour of a dynamical system can be examined from time series using the technique of recurrence plots through visualization and quantification methods. One such method is the quadrant scan in which a recurrence plot of a time series is converted into a second time series whose local maxima can be identified and interpreted as possible transitions in dynamic behaviour. A recurrence plot can also be represented as a complex network. A quadrant scan can similarly be thought of as a partition of the network vertices into two communities. Here, we argue that different dynamic behavioural regimes can be realized as network communities and the quality of such partitions can be assessed using modularity. Thereby, community modularity can be used as an alternative to the quadrant scan. We investigate this correspondence with respect to two promising arenas for quadrant scan uptake, namely, concept drift detection from machine learning and tipping point or failure and damage monitoring in system maintenance. We also examine two additional data sets to highlight the potential of the methods. These are a geophysical time series of earth tremor data and a physiological time series of an electrocardiogram.

Genetic algorithm of optimizing the qualification of programmer teams

The partitioning a set of professional programmers into a set of teams when a programming project specifies requirements to the competency in various programming technologies and tools is a hard combinatorial problem. The paper proposes a genetic algorithm, which is capable of finding competitive and high-quality partitioning solutions in acceptable runtime. The algorithm introduces chromosomes in such a way as to assign each programmer to a team, define the team staff and easily reconstruct the teams during optimization process. A fitness function characterizes each chromosome with respect to the quality of the programmers partitioning. It accounts for the average qualification of teams and the qualification of team best representatives on each of the technologies. The function recognizes the teams that meet all constraints on the project and are workable from this point of view. It is also capable of recognizing the teams that do not meet the constraints and are unworkable. The algorithm defines the genetic operations of selection, crossing and mutation in such a way as to move programmers from unworkable to workable teams, to increase the number of workable teams, to ex-change programmers among workable teams, to increase the competency of every workable team, and thus to maximize the teams overall qualification. Experimental results obtained on a set of programmers graduated from Belarus universities show the capability of the genetic algorithm to find good partitioning solutions, maximize the teams’ competency and minimize the number of unemployed programmers.

Social Network Community Detection by Combining Self‐Organizing Maps and Genetic Algorithms

Social networks have become an important source of information from which we can extract valuable indicators that can be used in many fields such as marketing, statistics, and advertising among others. To this end, many research works in the literature offer users some tools that can help them take advantage of this mine of information. Community detection is one of these tools and aims to detect a set of entities that share some features within a social network. We have taken part in this effort, and we proposed an approach mainly based on pattern recognition techniques. The novelty of this approach is that we do not directly tackle the social networks to find these communities. We rather proceeded in two stages; first, we detected community cores through a special type of self‐organizing map called the Growing Hierarchical Self‐Organizing Map (GHSOM). In the second stage, the agglomerations resulting from GHSOM were grouped to retrieve the final communities. The quality of the final partition would be under the control of an evaluation function that is maximized through genetic algorithms. Our system was tested on real and artificial databases, and the obtained results are really encouraging.

Scanning Strategy to Improve the Overlapping Quality of Partition in Selective Laser Melting

Scanning Strategy to Improve the Overlapping Quality of Partition in Selective Laser Melting

Ensemble Clustering Algorithm Development for Tabular Data by a Given Partition Quality Measure

Over the years, ensemble algorithms became a common solution for supervised learning tasks as they show more robustness and usually are more accurate. But for unsupervised learning and clustering in particular ensemble methods remain poorly researched. In this work we propose a group of ensemble clustering algorithms by exploiting existing ideas and improving them with modern hyperparameter tuning algorithms and quality measure choosing heuristics. Our proposed approach uses MASSCAH to choose base clustering algorithm and optimize its hyperparameters using SMAC. The proposed solution was experimentally compared to existing solutions and k-means algorithm, tuned by an expert. Experiments were conducted on 64 synthetic datasets from Gaussian distribution and on 20 real-world datasets. We used 4 cluster validity indices to evaluate the quality of resulting partitions. From the results of experiments we can conclude that the proposed solution performs comparable to k-means tuned by an expert and outperforms existing ensemble clustering solutions. Apart from that, the proposed solution can be used as an automatic solution for clustering.

Compression and swelling of hydrogels in polymer solutions: A dominant-mode model.

The swelling and compression of hydrogels in polymer solutions can be understood by considering hydrogel-osmolyte-solvent interactions which determine the osmotic pressure difference between the inside and the outside of a hydrogel particle and the changes in effective solvent quality for the hydrogel network. Using the theory of poroelasticity, we find the exact solution to hydrogel dynamics in a dilute polymer solution, which quantifies the effect of diffusion and partitioning of osmolyte and the related solvent quality change to the volumetric changes of the hydrogel network. By making a dominant-mode assumption, we propose a model for the swelling and compression dynamics of (spherical) hydrogels in concentrated polymer solutions. Osmolyte diffusion induces a biexponential response in the size of the hydrogel radius, whereas osmolyte partitioning and solvent quality effects induce monoexponential responses. Comparison of the dominant-mode model to experiments provides reasonable values for the compressive bulk modulus of a hydrogel particle, the permeability of the hydrogel network, and the diffusion constant of osmolyte molecules inside the hydrogel network. Our model shows that hydrogel-osmolyte interactions can be described in a conceptually simple manner, while still capturing the rich (de)swelling behaviors observed in experiments. We expect our approach to provide a roadmap for further research into and applications of hydrogel dynamics induced by, for example, changes in the temperature and the pH.

A review on suppressed fuzzy c-means clustering models

Abstract Suppressed fuzzy c-means clustering was proposed as an attempt to combine the better properties of hard and fuzzy c-means clustering, namely the quicker convergence of the former and the finer partition quality of the latter. In the meantime, it became much more than that. Its competitive behavior was revealed, based on which it received two generalization schemes. It was found a close relative of the so-called fuzzy c-means algorithm with generalized improved partition, which could improve its popularity due to the existence of an objective function it optimizes. Using certain suppression rules, it was found more accurate and efficient than the conventional fuzzy c-means in several, mostly image processing applications. This paper reviews the most relevant extensions and generalizations added to the theory of fuzzy c-means clustering models with suppressed partitions, and summarizes the practical advances these algorithms can offer.

Distributed Application Global States Monitoring in PEGASUS DA Applied to Parallel Graph Partitioning

SummaryThis paper presents how parallel advanced graph partitioning algorithms can be designed and improved with the use of global application states monitoring of distributed programs. The proposed algorithms have been implemented inside a novel distributed program design framework Program Execution Governed by Asynchronous Supervision of States in Distributed Applications (PEGASUS DA). This framework provides system support for the design of execution control in distributed applications based on automated global state monitoring. Two strategies for the control design of advanced parallel/distributed graph partitioning algorithms are presented and discussed. In the first one, the parallel algorithm control runs on top of the popular graph partitioning METIS tool. The second control strategy is based on genetic programming so that partitioning primitives and the overall algorithmic control can be freely designed by the user. Advanced partitioning methods have been conveniently embedded inside the global state monitoring driven PEGASUS DA framework which controls partitioning distributed actions at the level of processes and threads. The use of the framework allowed easy design and testing of different graph optimization strategies. The presented graph partitioning methods are illustrated by experiments with benchmark graphs. The experiments have comparatively assessed the obtained graph partitioning quality (visible improvement has been observed) and have identified benefits of the proposed approach for programmers.

Evaluation of the quality and relevance of a fuzzy partition

This paper proposes a couple of criteria for evaluating the quality and relevance of a fuzzy partition. These criteria are established from a fuzzy classification system and its recursive De Morgan triplet. We propose a comparison process between the classes of a fuzzy partition, based on a translation invariant similarity relation. Therefore a classification process is carried out with the equivalence relations determined by the similarity relation. Such a relation is built on the commutative group structure formed by the elements of the fuzzy classification system. Our approach is illustrated through an example on image analysis by the fuzzy c-means algorithm.

Evaluating the land-surface energy partitioning in ERA5

Abstract. Climate reanalyses provide a plethora of global atmospheric and surface parameters in a consistent manner over multi-decadal timescales. Hence, they are widely used in many fields, and an in-depth evaluation of the different variables provided by reanalyses is a necessary means to provide feedback on the quality to their users and the operational centres producing these data sets, and to help guide their development. Recently, the European Centre for Medium-Range Weather Forecasts (ECMWF) released the new state-of-the-art climate reanalysis ERA5, following up on its popular predecessor ERA-Interim. Different sets of variables from ERA5 were already evaluated in a handful of studies, but so far, the quality of land-surface energy partitioning has not been assessed. Here, we evaluate the surface energy partitioning over land in ERA5 and concentrate on the appraisal of the surface latent heat flux, surface sensible heat flux, and Bowen ratio against different reference data sets and using different modelling tools. Most of our analyses point towards a better quality of surface energy partitioning in ERA5 than in ERA-Interim, which may be attributed to a better representation of land-surface processes in ERA5 and certainly to the better quality of near-surface meteorological variables. One of the key shortcomings of the reanalyses identified in our study is the overestimation of the surface latent heat flux over land, which – although substantially lower than in ERA-Interim – still remains in ERA5. Overall, our results indicate the high quality of the surface turbulent fluxes from ERA5 and the general improvement upon ERA-Interim, thereby endorsing the efforts of ECMWF to improve their climate reanalysis and to provide useful data to many scientific and operational fields.

Scalable Recommendation Using Large Scale Graph Partitioning With Pregel and Giraph

Social Big Data is generated by interactions of connected users on social network. Sharing of opinions and contents amongst users, reviews of users for products, result in social Big Data. If any user intends to select products such as movies, books, etc., from e-commerce sites or view any topic or opinion on social networking sites, there are a lot of options and these options result in information overload. Social recommendation systems assist users to make better selection as per their likings. Recent research works have improved recommendation systems by using matrix factorization, social regularization or social trust inference. Furthermore, these improved systems are able to alleviate cold start and sparsity, but not efficient for scalability. The main focus of this article is to improve scalability in terms of locality and throughput and provides better recommendations to users with large-scale data in less response time. In this article, the social big graph is partitioned and distributed on different nodes based on Pregel and Giraph. In the proposed approach ScaleRec, partitioning is based on direct as well as indirect trust between users and comparison with state-of-the-art approaches proves that statistically better partitioning quality is achieved using proposed approach. In ScaleRec, hyperedge and transitive closure are used to enhance social trust amongst users. Experiment analysis on standard datasets such as Epinions and LiveJournal proves that better locality and recommendation accuracy is achieved by using ScaleRec.