Quality Of Partitions Research Articles

Applying Wide & Deep Learning Model for Android Malware Classification

Android malware has exploded in popularity in recent years, due to the platform’s dominance of the mobile market. With the advancement of deep learning technology, numerous deep learning-based works have been proposed for the classification of Android malware. Deep learning technology is designed to handle a large amount of raw and continuous data, such as image content data. However, it is incompatible with discrete features, i.e., features gathered from multiple sources. Furthermore, if the feature set is already well-extracted and sparsely distributed, this technology is less effective than traditional machine learning. On the other hand, a wide learning model can expand the feature set to enhance the classification accuracy. To maximize the benefits of both methods, this study proposes combining the components of deep learning based on multi-branch CNNs (Convolutional Network Neural) with wide learning method. The feature set is evaluated and dynamically partitioned according to its meaning and generalizability to subsets when used as input to the model’s wide or deep component. The proposed model, partition, and feature set quality are all evaluated using the K-fold cross validation method on a composite dataset with three types of features: API, permission, and raw image. The accuracy with Wide and Deep CNN (WDCNN) model is 98.64%, improved by 1.38% compared to RNN (Recurrent Neural Network) model.

Limiting the Failure Impact of Interdependent Power-Communication Networks via Optimal Partitioning

The mutual dependency between the power grid and communication network increases the risk of large-scale cascading failure, making the mitigation of outages a difficult task. In this paper, we use the concept of joint partitioning of the power and communication subsystems by identifying the most vulnerable typed-graphlets. This results in higher-order partitions, which prevent cascading failures from propagating globally in the system. For this purpose, we first formulate the problem of partitioning the interdependent system as a large-scale optimization problem with the objective of minimizing the overall load shedding subject to: i) power flow convergence, ii) power stability, iii) partition connectivity, iv) communication congestion/delay control, and v) a high partition quality. Then, using Benders’ method, we solve the problem by decomposing it into a relaxed master problem that consists of constrained spectral clustering, and a linear subproblem that consists of minimizing the load shedding subject to power flow convergence, stability, and congestion/delay control. Simulations are conducted on an IEEE 118-bus supported by a 118-node communication network. Our investigations reveal an average of 62% decrease in damage when the system is optimally partitioned.

DP-Kmeans and Beyond: Optimal Clustering with a new Clustering Validity Index

<p>The K-means clustering algorithm is widely used in many areas for its high efficiency. However, the performance of the traditional K-means algorithm is very sensitive to the selection of initial clustering centers. Furthermore, except the convex distributed datasets, the traditional K-means algorithm still cannot optimally process many non-convex distributed datasets and datasets with outliers. To this end, this paper proposes the DP-Kmeans, an improved K-means algorithm based on the Density Parameter and center replacement, which can be more accurate than the traditional K-means by dropping the random selection of the initial clustering centers and continuous updating of the new centers. Due to the unsupervised learning feature, the number of clusters and the quality of data partitions generated by the clustering algorithm cannot be guaranteed. In order to evaluate the results of the DP-Kmeans algorithm, this paper proposes the SII, a new clustering validity index based on the Sum of the Inner-cluster compactness and the Inter-cluster separateness. Based on the DP-Kmeans algorithm and the SII index, a new method is proposed to determine the optimal clustering numbers for different datasets. Experimental results on ten datasets with different distributions demonstrate that the proposed clustering method is more effective the existing ones. </p> <p> </p>

Graph neural network inspired algorithm for unsupervised network community detection

Network community detection often relies on optimizing partition quality functions, like modularity. This optimization appears to be a complex problem traditionally relying on discrete heuristics. And although the problem could be reformulated as continuous optimization, direct application of the standard optimization methods has limited efficiency in overcoming the numerous local extrema. However, the rise of deep learning and its applications to graphs offers new opportunities. And while graph neural networks have been used for supervised and unsupervised learning on networks, their application to modularity optimization has not been explored yet. This paper proposes a new variant of the recurrent graph neural network algorithm for unsupervised network community detection through modularity optimization. The new algorithm’s performance is compared against the state-of-the-art methods. The approach also serves as a proof-of-concept for the broader application of recurrent graph neural networks to unsupervised network optimization.

Fuzzy C-Means Clustering Algorithms with Weighted Membership and Distance

Fuzzy C-means (FCM) clustering algorithm is an important and popular clustering algorithm which is utilized in various application domains such as pattern recognition, machine learning, and data mining. Although this algorithm has shown acceptable performance in diverse problems, the current literature does not have studies about how they can improve the clustering quality of partitions with overlapping classes. The better the clustering quality of a partition, the better is the interpretation of the data, which is essential to understand real problems. This work proposes two robust FCM algorithms to prevent ambiguous membership into clusters. For this, we compute two types of weights: an weight to avoid the problem of overlapping clusters; and other weight to enable the algorithm to identify clusters of different shapes. We perform a study with synthetic datasets, where each one contains classes of different shapes and different degrees of overlapping. Moreover, the study considered real application datasets. Our results indicate such weights are effective to reduce the ambiguity of membership assignments thus generating a better data interpretation.

Efficient game theoretic approach to dynamic graph partitioning

As a building block in many graph-based applications, graph partitioning aims to divide a graph into smaller parts of roughly equal size, and meanwhile, minimize the number of cutting edges. Existing solutions for graph partitioning are mainly designed for static graphs and are not appropriate for many dynamic graphs in real-world scenarios, including social networks, knowledge graphs, and web graphs. Although there is an incremental method, called IncKGGGP, proposed to efficiently deal with dynamic graphs, it can only be deployed on top of a specific batch partitioning algorithm, called KGGGP, which inherently impairs the final partitioning quality.To alleviate these issues, in this paper, we propose a novel Edge-Cut Partitioning approach based on Game theory for dynamic graphs (ECPG). Generally, ECPG is equipped with the following two nice properties. (1) High effectiveness. It can cope with dynamic graphs on top of any initial partitioning result, and then, achieve higher-quality results by choosing more effective static algorithms. (2) High efficiency. By exerting the advantage of game theory, ECPG can not only assign updated vertices into desirable partitions efficiently based on a low time complexity function but also can reduce redundant computations by reusing existing partitioning results. We also prove that there exists a Nash equilibrium in ECPG. From experimental results over several real-world graphs, it demonstrates that ECPG significantly outperforms the existing algorithms by up to one order of magnitude with comparable partitioning quality.

Edge Repartitioning via Structure-Aware Group Migration

Graph partitioning is a mandatory step in distributed graph computing systems. Some existing systems use edge partitioning methods to partition static graphs. However, the structure of the real-world graphs changes dynamically, which leads to unnecessary vertex replicas and load imbalance, reducing the performance of graph computation. In this article, we focus on improving the lower partitioning quality caused by the dynamics of the graph structure. We propose an edge repartitioning algorithm via structure-aware group migration (SAGM-ER). We define a special structure edge group (EG) consisting of multiple edges, which can reduce vertex replicas by migrating to other partitions. In repartitioning, we search for EGs in parallel by a method based on a structure-aware priority and then migrate EGs to reduce vertex replicas. Compared to the state of the art, SAGM-ER can reduce more vertex replicas. We implement SAGM-ER on Powergraph, which reduces the redundant replicas by 63.33%, thus reducing executing time and communication costs in graph computation by 33.72% and 37.51%, respectively.

Robustness modularity in complex networks.

A basic question in network community detection is how modular a given network is. This is usually addressed by evaluating the quality of partitions detected in the network. The Girvan-Newman (GN) modularity function is the standard way to make this assessment, but it has a number of drawbacks. Most importantly, it is not clearly interpretable, given that the measure can take relatively large values on partitions of random networks without communities. Here we propose a measure based on the concept of robustness: modularity is the probability to find trivial partitions when the structure of the network is randomly perturbed. This concept can be implemented for any clustering algorithm capable of telling when a group structure is absent. Tests on artificial and real graphs reveal that robustness modularity can be used to assess and compare the strength of the community structure of different networks. We also introduce two other quality functions: modularity difference, a suitably normalized version of the GN modularity, and information modularity, a measure of distance based on information compression. Both measures are strongly correlated with robustness modularity, but have lower time complexity, so they could be used on networks whose size makes the calculation of robustness modularity too costly.

Effects of boron on nutrient partitioning, Ca movement, and fruit quality of tomatoes

This study was aimed to investigate the effects of boron on Ca movement, yield, and quality of tomatoes. A soilless culture experiment with three levels of B (0, 25, and 50 µM) and three concentrations of Ca (2, 4, and 8 mM) was carried out to grow tomatoes. The results showed that although the application of boron to the sample was more effective in the vegetative growth and the fruit induction, resulting in higher numbers of fruit, the addition of Ca could more effectively enhance the weight and quality of the fruit. With B application, Ca-symplast increased, while pectin methylesterase activity decreased, resulting in less Blossom End Rot indices and more fruit firmness. The Ca-transfer index from apoplast to symplast slightly increased with the B application. Overall, in terms of fruit quality and quantity, the best result was seen in higher Ca application as well as higher B concentrations. Although exogenous boron could help Ca absorption in the root and the shoot, it lowered the Ca transfer from xylem to apoplast and symplast, from shoot to fruit, and from the proximal to distal end of the fruit. Nevertheless, loading 4 and 8 mM of Ca seems to provide enough calcium reserves in fruits to have better quality and yield.

Net Clusting Based Low Complexity Coarsening Algorithm In k-way Hypergraph Partitioning

With the increasing scale of integrated circuits, hypergraph partitioning is usually applied to Very Large Scale Integration (VLSI) circuit layout and other applications to reduce the computational complexity. However, if without properly coarsening, the hypergraph partitioning problem will become intractable along with the increase of the number of vertices. In this paper, we propose a coarsening algorithm in k-way hypergraph partitioning based on net clustering, where net clustering is used to obtain the initial set of vertices with higher internal similarity. Due to the property of nets connecting through vertices, the proposal can cluster the net by local search and discard unimportant vertices from net clusters to achieve high-quality solutions. The experimental results show that our proposal achieves a high quality of coarsening with lower complexity. Even as the number of partitions rises, the computation time reduction will obviously increase. In the case of setting the number of partitions as 8, our algorithm can achieve almost the same partitioning quality as traditional hMetis, but with a time consumption reduction of 50%.

Fast Power Grid Partition for Voltage Control With Balanced-Depth-Based Community Detection Algorithm

Network partition in complex power networks is essential for the var-voltage control. Traditional partition methods such as Ward method are applied in practical power networks, but they are unable to evaluate the quality of partition results. Moreover, they lack efficiency when dealing with large-scale networks. Since complex operation characteristics and topology have emerged in recent power systems, the power grid partition requires higher efficiency and quality. Therefore, this paper proposes a fast network partition method with a balanced-depth-based community detection algorithm. Its aim is to significantly improve the efficiency of partition while maintaining high quality of partition, with which the inter-zone coupling is minimized while the intra-zone coupling is maximized. In the meantime, a surrogate-optimization-based selection algorithm is proposed to select the zonal pilot bus, based on which the secondary voltage control method is used to evaluate the quality of partition. Results from four case studies conducted in various power networks with different sizes, as compared to other partition methods, validate the high efficiency and high quality of the proposed power grid partition approach.

Assessment of the Possibility of Using Various Types of Renewable Energy Sources Installations in Single-Family Buildings as Part of Saving Final Energy Consumption in Polish Conditions

The aim of this work is to develop a model of heat supply to buildings with almost zero energy consumption, indicating the significant importance of heat losses and gains in heating installations. The prepared model is to indicate the need for changes in the structure and topology of heating installations, resulting from the changing heat demand of buildings. The need to create a new model is heightened by changes that relate to tightening legal regulations related to energy consumption and demand, which must meet the standards of buildings in Poland from 2021. The article presents the assumptions and results of analyses of the use of energy installations in residential buildings that use renewable energy sources to balance energy consumption in various areas of its use. To achieve this goal, calculations were made using simulations of the impact of the use of installations using renewable energy sources on the energy performance of a building with different quality of partitions and improvement of energy efficiency in accordance with the Polish standard PN-EN 12831. The test results allow to choose the most advantageous, from the point of view of economic profitability, option of replacing installations in residential buildings, and they also allow to determine the possibilities of meeting national obligations in the field of final energy reduction and increasing the share of renewable energy sources in meeting its demand in accordance with the EU obligations imposed on Poland. Thermomodernization of buildings in the temperate climate zone allows for a reduction of 38% of energy demand over the entire life cycle of a building and a reduction of CO2 emissions by 99%.

An ensemble model to optimize modularity in dynamic bipartite networks

Distinct non-random quantitative interactions at diverse timestamps formulate real-world dynamic complex networks. The most frequently used class of methods for discovering communities in dynamic networks is modularity optimization that evaluates the quality of the partition of network nodes into distinct communities. The bipartite networks have bipartite modularity and bipartite modularity optimization respectively. Newman's modularity is a consistently used algorithm to evaluate modules of unipartite networks yet it is ineffective for assessing the division of bipartite networks with two types of vertices. Many community detection methods suggest bipartite modularity to accommodate this issue. They usually employ information about the existence or lack of interactions between nodes. In quantitative networks, weighted modularity is a potential approach for measuring the quality of community partitions (Lu et al. IEEE, 179–184, 2013). This study offers an ensemble model for detecting one-mode communities and optimizing modularity in dynamic bipartite weighted networks. By using collaborative weighted projection, bipartite networks get projected into two weighted one-mode networks. The results of experiments both on real-world dynamic network data and synthetic data demonstrate that the modularity of the method is significantly greater than that of current techniques and the communities discovered contain vertices of comparable kinds exhibiting the suggested algorithm's performance is ample.

Robust Regularized Locality Preserving Indexing for Fiedler Vector Estimation

The Fiedler vector of a connected graph is the eigenvector associated with the algebraic connectivity of the graph Laplacian and it provides substantial information to learn the latent structure of a graph. In real-world applications, however, the data may be subject to heavy-tailed noise and outliers which results in deteriorations in the structure of the Fiedler vector estimate. We design a Robust Regularized Locality Preserving Indexing (RRLPI) method for Fiedler vector estimation that aims to approximate the nonlinear manifold structure of the Laplace Beltrami operator while minimizing the negative impact of outliers. First, an analysis of the effects of two fundamental outlier types on the eigen-decomposition for block affinity matrices which are essential in cluster analysis is conducted. Then, an error model is formulated and a robust Fiedler vector estimation algorithm is developed. An unsupervised penalty parameter selection algorithm is proposed that leverages the geometric structure of the projection space to perform robust regularized Fiedler estimation. The performance of RRLPI is benchmarked against existing competitors in terms of detection probability, partitioning quality, image segmentation capability, robustness and computation time using a large variety of synthetic and real data experiments.

An Overview of Metrics for Evaluating the Quality of Graph Partitioners

Nowadays, many applications that involve big data can be modelled as graphs. In many cases these graphs may be too large to be loaded and processed on a single commodity computer. This necessitated the development of frameworks that allow processing large graphs by distributing the graph among nodes in a cluster. To process a graph using these frameworks, first, the graph is partitioned into smaller components called subgraphs or partitions, and then assign these smaller subgraphs to different nodes for parallel processing. Depending on the type of processing (example, computing pagerank, counting number of triangles etc.), there will be some communication between nodes during the execution, this communication affects execution time. Therefore, graph partitioning is an important step in distributed graph processing. Being able to determine the quality of a partition prior to processing is important as this will allow us to predict the execution time before the actual processing. A number of metrics for evaluating the quality of a graph partitions exist, but studies show that these metrics may not serve as accurate predictors in many cases. In this work, we reviewed published papers about graph partitioning and we were able to identify and defined more metrics in order to have a catalogue of these metrics.

A New Clustering Approach for Automatic Oscillographic Records Segmentation

The analysis of waveforms related to transient events is an important task in power system maintenance. Currently, electric power systems are monitored by several event recorders called phasor measurement units (PMUs) which generate a large amount of data. The number of records is so high that it makes human analysis infeasible. An alternative way of solving this problem is to group events in similar classes so that it is no longer necessary to analyze all the events, but only the most representative of each class. Several automatic clustering algorithms have been proposed in the literature. Most of these algorithms use validation indexes to rank the partitioning quality and, consequently, find the optimal number of clusters. However, this issue remains open, as each index has its own performance highly dependent on the data spatial distribution. The main contribution of this paper is the development of a methodology that optimizes the results of any clustering algorithm, regardless of data spatial distribution. The proposal is to evaluate the internal correlation of each cluster to proceed or not in a new partitioning round. In summary, the traditional validation indexes will continue to be used in the cluster’s partition process, but it is the internal correlation measure of each one that will define the stopping splitting criteria. This approach was tested in a real waveforms database using the K-means algorithm with the Silhouette and also the Davies–Bouldin validation indexes. The results were compared with a specific methodology for that database and were shown to be totally consistent.

Local Graph Edge Partitioning

Graph edge partitioning, which is essential for the efficiency of distributed graph computation systems, divides a graph into several balanced partitions within a given size to minimize the number of vertices to be cut. Existing graph partitioning models can be classified into two categories: offline and streaming graph partitioning models. The former requires global graph information during the partitioning, which is expensive in terms of time and memory for large-scale graphs. The latter creates partitions based solely on the received graph information. However, the streaming model may result in a lower partitioning quality compared with the offline model. Therefore, this study introduces a Local Graph Edge Partitioning model, which considers only the local information (i.e., a portion of a graph instead of the entire graph) during the partitioning. Considering only the local graph information is meaningful because acquiring complete information for large-scale graphs is expensive. Based on the Local Graph Edge Partitioning model, two local graph edge partitioning algorithms—Two-stage Local Partitioning and Adaptive Local Partitioning—are given. Experimental results obtained on 14 real-world graphs demonstrate that the proposed algorithms outperform rival algorithms in most tested cases. Furthermore, the proposed algorithms are proven to significantly improve the efficiency of the real graph computation system GraphX.

On the impact of heterogeneity-aware mesh partitioning and non-contributing computation removal on parallel reservoir simulations

Parallel computations have become standard practice for simulating the complicated multi-phase flow in a petroleum reservoir. Increasingly sophisticated numerical techniques have been developed in this context. During the chase of algorithmic superiority, however, there is a risk of forgetting the ultimate goal, namely, to efficiently simulate real-world reservoirs on realistic parallel hardware platforms. In this paper, we quantitatively analyse the negative performance impact caused by non-contributing computations that are associated with the “ghost computational cells” per subdomain, which is an insufficiently studied subject in parallel reservoir simulation. We also show how these non-contributing computations can be avoided by reordering the computational cells of each subdomain, such that the ghost cells are grouped together. Moreover, we propose a new graph-edge weighting scheme that can improve the mesh partitioning quality, aiming at a balance between handling the heterogeneity of geological properties and restricting the communication overhead. To put the study in a realistic setting, we enhance the open-source Flow simulator from the OPM framework, and provide comparisons with industrial-standard simulators for real-world reservoir models.

Efficient parallel edge-centric approach for relaxed graph pattern matching

Prior algorithms on graph simulation for distributed graphs are not scalable enough as they exhibit heavy message passing. Moreover, they are dependent on the graph partitioning quality that can be a bottleneck due to the natural skew present in real-world data. As a result, their degree of parallelism becomes limited. In this paper, we propose an efficient parallel edge-centric approach for distributed graph pattern matching. We design a novel distributed data structure called ST that allows a fine-grain parallelism, and hence guarantees linear scalability. Based on ST, we develop a parallel graph simulation algorithm called PGSim. Furthermore, we propose PDSim, an edge-centric algorithm that efficiently evaluates dual simulation in parallel. PDSim combines ST and PGSim in a Split-and-Combine approach to accelerate the computation stages. We prove the effectiveness and efficiency of these propositions through theoretical guarantees and extensive experiments on massive graphs. The achieved results confirm that our approach outperforms existing algorithms by more than an order of magnitude.

Practical and high-quality partitioning algorithm for large-scale and time-evolving graphs

With the appearance of widespread large-scale graph data, distributed graph processing grows in popularity. In order to store and analyze a large-scale graph in a distributed manner, the graph should be properly partitioned in advance. Although the partitioning of static graphs has been sufficiently investigated, the emerging graphs nowadays are typically dynamic or time-evolving. State-of-the-art partitioning algorithms for such large-scale and time-evolving graphs either are impractical because of the complexity caused by the global optimization or sacrifice partitioning quality due to a huge number of cross partition edges. This paper investigates the problem and proposes a new practical and high-quality graph partitioning algorithm. First, graph updates accumulated in a time interval are expressed as a delta graph. Second, vertices in the delta graph are classified into several subsets based on a non-overlapping community detection method. Third, vertices in the same subset are assigned to a properly chosen partition as a whole. The chosen partition is lightly loaded as well as closely related with vertices in the subset. Experimental results show that compared with the widely used hash-based and heuristic-based partitioning algorithms, our proposed algorithm gains significant decrease in terms of the number of cross partition edges while maintaining a similar level of load balance.