Optimal Partitioning Strategy Research Articles

NOCAP: Near-Optimal Correlation-Aware Partitioning Joins

Storage-based joins are still commonly used today because the memory budget does not always scale with the data size. One of the many join algorithms developed that has been widely deployed and proven to be efficient is the Hybrid Hash Join (HHJ), which is designed to exploit any available memory to maximize the data that is joined directly in memory. However, HHJ cannot fully exploit detailed knowledge of the join attribute correlation distribution. In this paper, we show that given a correlation skew in the join attributes, HHJ partitions data in a suboptimal way. To do that, we derive the optimal partitioning using a new cost-based analysis of partitioning-based joins that is tailored for primary key - foreign key (PK-FK) joins, one of the most common join types. This optimal partitioning strategy has a high memory cost, thus, we further derive an approximate algorithm that has tunable memory cost and leads to near-optimal results. Our algorithm, termed NOCAP (Near-Optimal Correlation-Aware Partitioning) join, outperforms the state of the art for skewed correlations by up to 30%, and the textbook Grace Hash Join by up to 4×. Further, for a limited memory budget, NOCAP outperforms HHJ by up to 10%, even for uniform correlation. Overall, NOCAP dominates state-of-the-art algorithms and mimics the best algorithm for a memory budget varying from below √||relation|| to more than ||relation||.

Time-division based integrated sensing, communication, and computing in integrated satellite-terrestrial networks

In this paper, we investigate time-division based framework for integrated sensing, communication, and computing in integrated satellite-terrestrial networks. We consider a scenario, where Internet-of-Things devices on the ground operate with sensing and communication in a time-division manner, and can process the sensing results locally, at the edge, or in the cloud via the satellite communication link. Based on the proposed framework, we formulate a multi-dimensional optimization problem to maximize the utility performance of sensing, communication, and computing abilities. After decomposing the original optimization problem into two subproblems, we first derive the closed-form solution of the optimal task partitioning strategy for terrestrial users and satellite users. Then, we develop the joint subframe allocation and task partitioning strategy to optimize the overall performance, by means of which the Pareto optimal solutions can be obtained along the Pareto frontier. Extensive simulations are provided to demonstrate the effectiveness of the proposed strategy, which is 10% to 60% superior compared with the benchmarks. Also, the trade-off between the multidimensional resource and multi-functional performance is analyzed from the perspective of network design.

Does partition matter? A new approach to modeling land use change

Cellular automata (CA) is a classical method for studying land use change. However, homogeneous transformation rules have commonly been used to conduct simulations in the past, and these rules seldom consider the spatial heterogeneity of geographic elements. Therefore, in this study, we incorporated spatial-temporal heterogeneity transformation rules into the CA framework based on spatial data mining. A model that couples self-organizing maps (SOM), hierarchical clustering (HC), and patch generation land use simulation (PLUS) was proposed; it is called the SOM-HC-PLUS model. This model considers the difference in local-area driving factors, and therefor it not only determines the optimal partition scheme automatically, but it also measures the contribution of partition driving factors. The Jinpu New Area in Dalian, China, was used to test the validity of the model by comparing the traditional PLUS model and the administrative division (AD)-PLUS model based on AD partition. The results showed that the partition scheme of the SOM-HC-PLUS model was reasonable and credible. Further, compared with other models, this model showed higher simulation accuracy and a more realistic land use distribution pattern. The driving factors showed significant differences in the overall and regional intensity. Moreover, the importance of natural environmental conditions, represented by elevation factors, in the expansion of artificial surfaces increased significantly. By 2030, artificial surfaces were projected to increase significantly through the conversion of cultivated lands. The sustainable development scenario showed a more compact patch layout and exhibited better protection of grasslands and forests than the historical development scenario. In summary, this study proposed a mixture CA model based on the idea of geographic partition, one that proved the reliability of the SOM-HC-PLUS model to conduct spatial-temporal heterogeneity studies on land use partition. It provides the possibility to explore patterns of regional land use changes over multiple periods, and can assist in urban planning and management and promoting sustainable development.

Maximizing Heterogeneous Server Utilization with Limited Availability Times for Divisible Loads Scheduling on Networked Systems

Most of the available divisible-load scheduling models assume that all servers in networked systems are idle before workloads arrive and that they can remain available online during workload computation. In fact, this assumption is not always valid. Different servers on networked systems may have heterogenous available times. If we ignore the availability constraints when dividing and distributing workloads among servers, some servers may not be able to start processing their assigned load fractions or deliver them on time. In view of this, we propose a new multi-installment scheduling model based on server availability time constraints. To solve this problem, we design an efficient heuristic algorithm consisting of a repair strategy and a local search strategy, by which an optimal load partitioning scheme is derived. The repair strategy guarantees time constraints, while the local search strategy achieves optimality. We evaluate the performance via rigorous simulation experiments and our results show that the proposed algorithm is suitable for solving large-scale scheduling problems employing heterogeneous servers with arbitrary available times. The proposed algorithm is shown to be superior to the existing algorithm in terms of achieving a shorter makespan of workloads.

GPartition: An Efficient Alignment Partitioning Program for Genome Datasets

Phylogenomics, or evolutionary inference based on genome alignment, is becoming prominent thanks to next-generation sequencing technologies. In model-based phylogenomics, the partition scheme has a significant impact on inference performance, both in terms of log-likelihoods and computation time. Therefore, finding an optimal partition scheme, or partitioning, is critical in a phylogenomic inference pipeline. To accomplish this, one needs to divide the alignment sites into disjoint partitions so that the sites of similar evolutionary models are in the same partition. Computational partitioning is a recent approach of increasing interest due to its capability of modeling the site-rate heterogeneity within a single gene. State-of-the-art computational partitioning methods, such as mPartition or RatePartition, are, however, ineffective on long alignments of millions of sites. In this paper, we introduce gPartition, a new computational partitioning method leveraging both the site rate and the best-fit substitution model. We conducted experiments on recently published alignments to compare gPartition with mPartition and RatePartition. gPartition was orders of magnitude faster than other methods. The AIC score demonstrated that gPartition produced partition schemes that were better or comparable to mPartition. gPartition outperformed RatePartition on all examined alignments. We implemented our proposed method in the gPartition program to help researchers partition genome alignments with millions of sites more efficiently.

AccTFM: An Effective Intra-Layer Model Parallelization Strategy for Training Large-Scale Transformer-Based Models

Transformer-based deep neural networks have recently swept the field of natural language processing due to their outstanding performance, and are gradually spreading to more applications such as image/video processing. However, compared with general DNNs, training a sizeable transformer-based model is further time-consuming and memory-hungry. The existing distributed training strategies for general DNNs are not appropriate or can not efficiently handle transformer-based networks. In view of this, we propose an intra-layer model parallelization optimization strategy, AccTFM, which introduces a novel fine-grained pipeline execution and hybrid communication compression strategy to overcome the synchronization bottleneck. Specifically, on one hand, it first decouples the inter-layer computation and communication dependencies, and then searches for the optimal partitioning strategy to maximize the overlap of computation and communication. On the other hand, the hybrid communication compression module consists of token-level top- <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> sparsification and piecewise quantization methods aiming at minimizing communication traffic. Experimental results show that AccTFM accelerates transformer-based DNNs training by up to 2.08x compared to state-of-the-art distributed training techniques.

LES 3

Set similarity search is a problem of central interest to a wide variety of applications such as data cleaning and web search. Past approaches on set similarity search utilize either heavy indexing structures, incurring large search costs or indexes that produce large candidate sets. In this paper, we design a learning-based exact set similarity search approach, LES 3 . Our approach first partitions sets into groups, and then utilizes a light-weight bitmap-like indexing structure, called token-group matrix (TGM), to organize groups and prune out candidates given a query set. In order to optimize pruning using the TGM, we analytically investigate the optimal partitioning strategy under certain distributional assumptions. Using these results, we then design a learning-based partitioning approach called L2P and an associated data representation encoding, PTR, to identify the partitions. We conduct extensive experiments on real and synthetic datasets to fully study LES 3 , establishing the effectiveness and superiority over other applicable approaches.

Application of Service Modular Design Based on a Fuzzy Design Structure Matrix: A Case Study from the Mining Industry

The development of customized service is an important way to transform and upgrade China’s mining industry. However, in practice, there remain problems, such as the slow market response speed of service providers and the contradiction between the large-scale development of service providers and the personalized service needs of service demanders. This paper uses the theory and method of service modular design to solve these problems and explores the process-based service modular design method. Service modular design depends largely on the determination of the relationship between service activities and the reasonable division of modules. However, previous research has rarely made use of modular design methods and modeling tools in the mining service context. At the same time, evaluations of the relationship between service activities relying on knowledge and those relying on experience have been inconclusive. Therefore, this paper proposes a service modularization design method based on the fuzzy relation analysis of a design structure matrix (DSM) that solves the optimal module partition scheme. Triangular fuzzy number and fuzzy evidence theory are used to evaluate and fuse the multidimensional and heterogeneous relationship between service activities, and the quantitative processing of the comprehensive relationship between service activities is carried out. On this basis, the service module structure is divided, followed by the construction of the mathematical programming model with the maximum sum of the average cohesion degree in the module and the average coupling degree between modules as the driving goal. The genetic algorithm is used to solve the problem, and the optimal module division result is obtained. Finally, taking the service modular design of SHD coal production enterprises in China as an example, the feasibility of the proposed method is verified.

Optimized Partition Scheme for Prefabricated Underpass with Large Cross Section

With the acceleration of urbanization in China, more underpasses will be constructed in big cities to alleviate the great traffic pressure. The prefabricated and assembly construction method has been introduced to replace the traditional cast‐in‐place method to achieve quick construction. However, for a fully prefabricated and assembled underground structure (PAUS) with large cross section, the structure must be cut into segments in transverse direction to reduce the size and weight for easy transportation and assembly. Therefore, how to develop an optimal partition scheme is a new problem to be studied. Firstly, three preliminary partition schemes were proposed based on the internal force distribution and completed engineering practices. Then, the three schemes were compared in terms of bending moment, shear force, and axial force. The construction efficiencies were also compared with special emphasis on difference of the build period. Finally, an optimal partition scheme was determined and successfully applied in the real project. Furthermore, the construction period of this partition scheme was 1/3 of the traditional cast‐in‐place method. The results of the current paper can provide some design guidance to large cross‐sectional underpasses and other underground structures in the partition stage.

Impact of grid partitioning algorithms on combined distributed AC optimal power flow and parallel dynamic power grid simulation

The complexity of most power grid simulation algorithms scales with the network size, which corresponds to the number of buses and branches in the grid. Parallel and distributed computing is one approach that can be used to achieve improved scalability. However, the efficiency of these algorithms requires an optimal grid partitioning strategy. To obtain the requisite power grid partitionings, the authors first apply several graph theory based partitioning algorithms, such as the Karlsruhe fast flow partitioner (KaFFPa), spectral clustering, and METIS. The goal of this study is an examination and evaluation of the impact of grid partitioning on power system problems. To this end, the computational performance of AC optimal power flow (OPF) and dynamic power grid simulation are tested. The partitioned OPF-problem is solved using the augmented Lagrangian based alternating direction inexact Newton method, whose solution is the basis for the initialisation step in the partitioned dynamic simulation problem. The computational performance of the partitioned systems in the implemented parallel and distributed algorithms is tested using various IEEE standard benchmark test networks. KaFFPa not only outperforms other partitioning algorithms for the AC OPF problem, but also for dynamic power grid simulation with respect to computational speed and scalability.

A New Service Module Partition Approach for Product Service System Based on Fuzzy Graph and Dempster-Shafer Theory of Evidence

Due to the personalized and diverse service needs, service scheme configuration should be more quick and flexible in the process of product service system (PSS) scheme design. Service modularization can effectively improve the service configuration efficiency and modules’ reusability. However, compared with the modularity of tangible products, the partition of service modules in the practical context is still a problem to be discussed. In this paper, a service partition approach for PSS based on the fuzzy graph and Dempster-Shafer theory of evidence is presented. Firstly, service activities correlation analysis is carried out, according to which the fuzzy graph is drawn. By setting different thresholds, the fuzzy graph is cut, and different partition results are obtained. Secondly, the evaluation indexes of customization, generalization, and technological evolution are proposed and used as evidence sources of the Dempster-Shafer theory of evidence. Through the synthesis of the evidence sources, the optimal partition scheme is got. Finally, to verify the method, a case study is illustrated through the NC machine tools module partition. And results show that the proposed method can provide specific ideas and concrete guidance of the service module partition.

Functional and ecophysiological traits of Halimione portulacoides and Sarcocornia perennis ecotypes in Mediterranean salt marshes under different tidal exposures

AbstractThe present paper aims to discuss the functional and ecophysiological traits of two abundant succulent halophytes in Mediterranean salt marsh systems, considering two ecotypes differing on their tidal exposure. In Halimione portulacoides the higher root/shoot (R/S) biomass ratio verified in low inundation frequency (LF) marshes, as compared with that in high inundation frequency (HF), indicates an optimal partitioning strategy, inducing differential allocation patterns along with varying environmental factors. Conversely, Sarcocornia perennis appear to have an isometric allocation strategy, based on intrinsic biomass allocation mechanisms independent from the surrounding environmental conditions. Similarly, the photosynthetic traits appear to reflect the intrinsic ecology of the species. Sarcocornia perennis habitat adaptation to HF conditions is clearly revealed by its high photosynthetic efficiency under HF conditions. On the other hand, the cosmopolitan H. portulacoides shows no differences between ecotypes, in which concerns this trait, mostly due to the high carotenoid content of the HF ecotype, which allows this specie to counteract the stress generated during submersion. Additionally, anti‐oxidant enzymatic activities revealed that S. perennis presents a high degree of stress tolerance under HF conditions, while H. portulacoides is highly adapted to LF conditions. These functional (biomass‐related) and physiological traits determine the optimal habitats for H. portulacoides and S. perennis. While H. portulacoides optimal biomass partitioning, along with its efficient anti‐oxidant defence system, makes this species fitter to LF conditions, S. perennis isometric allocation strategy, along with a higher photosynthetic efficiency under frequent flooding conditions, provides this species the mechanisms to efficiently colonize the intertidal habitat.

Information Criteria for Comparing Partition Schemes.

When inferring phylogenies, one important decision is whether and how nucleotide substitution parameters should be shared across different subsets or partitions of the data. One sort of partitioning error occurs when heterogeneous subsets are mistakenly lumped together and treated as if they share parameter values. The opposite kind of error is mistakenly treating homogeneous subsets as if they result from distinct sets of parameters. Lumping and splitting errors are not equally bad. Lumping errors can yield parameter estimates that do not accurately reflect any of the subsets that were combined whereas splitting errors yield estimates that did not benefit from sharing information across partitions. Phylogenetic partitioning decisions are often made by applying information criteria such as the Akaike information criterion (AIC). As with other information criteria, the AIC evaluates a model or partition scheme by combining the maximum log-likelihood value with a penalty that depends on the number of parameters being estimated. For the purpose of selecting an optimal partitioning scheme, we derive an adjustment to the AIC that we refer to as the AIC$^{(p)}$ and that is motivated by the idea that splitting errors are less serious than lumping errors. We also introduce a similar adjustment to the Bayesian information criterion (BIC) that we refer to as the BIC$^{(p)}$. Via simulation and empirical data analysis, we contrast AIC and BIC behavior to our suggested adjustments. We discuss these results and also emphasize why we expect the probability of lumping errors with the AIC$^{(p)}$ and the BIC$^{(p)}$ to be relatively robust to model parameterization.

Ultraconserved elements (UCEs) resolve the phylogeny of Australasian smurf-weevils.

Weevils (Curculionoidea) comprise one of the most diverse groups of organisms on earth. There is hardly a vascular plant or plant part without its own species of weevil feeding on it and weevil species diversity is greater than the number of fishes, birds, reptiles, amphibians and mammals combined. Here, we employ ultraconserved elements (UCEs) designed for beetles and a novel partitioning strategy of loci to help resolve phylogenetic relationships within the radiation of Australasian smurf-weevils (Eupholini). Despite being emblematic of the New Guinea fauna, no previous phylogenetic studies have been conducted on the Eupholini. In addition to a comprehensive collection of fresh specimens, we supplement our taxon sampling with museum specimens, and this study is the first target enrichment phylogenomic dataset incorporating beetle specimens from museum collections. We use both concatenated and species tree analyses to examine the relationships and taxonomy of this group. For species tree analyses we present a novel partitioning strategy to better model the molecular evolutionary process in UCEs. We found that the current taxonomy is problematic, largely grouping species on the basis of similar color patterns. Finally, our results show that most loci required multiple partitions for nucleotide rate substitution, suggesting that single partitions may not be the optimal partitioning strategy to accommodate rate heterogeneity for UCE loci.

Optimal island partitioning of smart distribution systems to improve system restoration under emergency conditions

In the case of emergency operation, intentional islanding of distributed generations is an interesting solution to maintain the reliable power supply in the smart distribution grid. This paper deals with an optimal systematic strategy to restore the post-contingency distribution grid after the occurrence of severe disturbances causing an outage. Hence, a two-stage solution procedure is developed in this paper. In the first step, the optimal partitioning scheme is addressed through a number of graph-theory related algorithms. By applying these algorithms, the loads are optimally assigned to the distributed generations and optimal load shedding programs are determined and performed if there is a shortage of generation. The second step deals with the adjustments issues. In fact, the preliminarily partitions achieved from the first stage should be checked for feasibility. Also, these partitions should be adjusted if it is necessary in order to satisfy the system related constraints. To this purpose OPF calculation is incorporated and optimal load management measures are taken to mitigate violations considering the voltage constraints, line capacities and controllable loads. Since the proposed method considers the key factors of load priorities, load controllability, power balance, voltage and line capacity constraints, it covers the requirements of practical applications. Numerical results from the PG&E-69 test system are used to verify the effectiveness of the proposed method. The comparison with previous methods demonstrates that the proposed method outperforms other techniques.

A parallel computing method using blocked format with optimal partitioning for SpMV on GPU

For large-scale sparse matrices, SpMV cannot be processed on GPU using the common storage formats because of the memory limitation. In addition, the parallel effect is poor using general formats for the sparse matrices with extremely uneven distribution of non-zero elements, which leads to performance deterioration. This paper presents an optimal partitioning strategy based on the distribution of non-zero elements in a sparse matrix to improve the performance of SpMV, and uses a hybrid format, which mixes CSR and ELL formats, to store the blocks partitioned from the sparse matrix. The hybrid blocked format has better compression effect and more uniform distribution of non-zero elements, which can be suitable for more types of sparse matrices. Our partitioning strategy is proven to be optimal, which can yield the minimum parallel execution time on GPU.

Partitioning large-scale artificial society on distributed cluster with statistical movement graph

ABSTRACTDistributed agent-based simulation is a popular method to realize computational experiment on large-scale artificial society. Meanwhile, the partitioning strategy of the artificial society models among hosts is playing an essential role for simulation engine to offer high execution efficiency as it has great impact on the communication overheads and computational load-balancing during simulation. Aiming at the problem, we firstly analyze the execution and scheduling process of agents during simulation and model it as wide-sense cyclostationary random process. Then, a static statistical partitioning model is proposed to obtain the optimal partitioning strategy with minimum average communication cost and load imbalance factor. To solve the static statistical partitioning model, this paper turns it into a graph-partitioning problem. A statistical movement graph-based partitioning algorithm is then devised which generates task graph model by mining the statistical movement information from initialization data of simulation model. In the experiments, two other popular partitioning methods are used to evaluate the performance of proposed graph-partitioning algorithm. Furthermore, this paper compares the graph-partitioning performance under different task graph model. The results indicate that our proposed statistical movement graph-based static partitioning method outperforms all other methods in reducing the communication overhead while satisfying the load balance constraint.

AppBooster: Boosting the Performance of Interactive Mobile Applications with Computation Offloading and Parameter Tuning

Interactive mobile applications attract lots of attentions recently. They utilize complex algorithms (e.g., machine learning) to provide advanced functions (e.g., object recognition), thus lead to long response time while running on mobile devices. To reduce the response time, researchers propose offloading some compute-intensive parts of mobile applications onto cloud. Existing works aim to optimize general performance (e.g., response time), but ignore the enhancement of application quality (e.g., recognition accuracy), which is also critical to user experience. In this paper, we develop AppBooster, a mobile cloud platform which boosts both general performance and application quality for interactive mobile applications. AppBooster jointly leverages the quality adaptation, computation offloading and parallel speedup to boost the comprehensive performance, which is defined by developers based on the metrics of application quality and general performance. Through combining history-based platform-learned knowledge, developer-provided information and the platform-monitored environment conditions (e.g., workload, network), AppBooster manages applications with optimal computation partitioning scheme and tunable parameter setting thus obtain high comprehensive performance. We evaluate AppBooster with an object recognition application in various network conditions and show AppBooster can significantly boost application performance and obtain 1.3 to 3.5 times better performance than existing strategies.

Intentional Islanding Algorithm for Distribution Network Based on Layered Directed Tree Model

In this study, a novel intentional island model of a distribution system with distributed generations (DGs) is presented and the improved Dijkstra algorithm is used to solve this model. This paper abstracts the distribution network with DGs to the layered directed tree according to its radial structure and power restoration process. In consideration of grade, controllability, capacity, level and electrical betweenness of load, the model weights load and maximizes total load weight in the island. The proposed model considers power balance, node voltage, phase angle and transmission capability of the branch, and network connectivity to meet practical engineering requirements. The improved Dijkstra algorithm formulates a search rule to select the load that can be divided into an island in descending order of the shortest path between the load node and DG node. An optimal island partition scheme is achieved through three stages: origin island, baby island and mature island. Meanwhile, scheme adjustment and constraint checking are used alternately to balance objective functions and constraints. The improved IEEE 43-bus distribution network is applied to verify the validity of the algorithm. A comparison of two island methods shows that the proposed algorithm can generate a reasonable scheme for island partitioning.

Automated Data Partitioning for Highly Scalable and Strongly Consistent Transactions

Modern transactional processing systems need to be fast and scalable, but this means many such systems settled for weak consistency models. It is however possible to achieve all of strong consistency, high scalability and high performance, by using fine-grained partitions and light-weight concurrency control that avoids superfluous synchronization and other overheads such as lock management. Independent transactions are one such mechanism, that rely on good partitions and appropriately defined transactions. On the downside, it is not usually straightforward to determine optimal partitioning schemes, especially when dealing with non-trivial amounts of data. Our work attempts to solve this problem by automating the partitioning process, choosing the correct transactional primitive, and routing transactions appropriately.