Dynamic Partitioning Research Articles

SPEAR: Dynamic Spatio-Temporal Query Processing over High Velocity Data Streams.

With the advent of IoT and emerging 5G technology, real-time streaming data are being generated at unprecedented speed and volume, and coming with both temporal and spatial dimensions. Effective analysis at such scale and speed requires support for dynamically adjusting querying capabilities in real-time. In spatio-temporal domain, this warrants for data as well as query optimization strategies especially for objects with changing motion states. Contemporary spatio-temporal data stream management systems in distributed domain are mostly dominated by specified-once-applied-continuously query model. Any modification in query state requires query restart limiting system responsiveness and producing outdated or in worst case erroneous results. In this paper, we propose adaptations of principles from streaming databases, spatial data management and distributed computing to support dynamic spatio-temporal query processing over high velocity big data streams. We first formulate a set of spatio-temporal data types and functions to seamlessly handle changes in distributed query states. We develop a comprehensive set of streaming spatio-temporal querying methods, and propose geohash based dynamic spatial partitioning for effective parallel processing. We implement a prototype on top of Apache Flink, where the in-memory stream processing fits nicely with our spatio-temporal models. Comparative evaluation of our prototype demonstrates the effectiveness our strategy by maintaining high consistent processing rates for both stationary as well as moving queries over high velocity spatio-temporal big data streams.

SwMR: A Framework for Accelerating MapReduce Applications on Sunway Taihulight

MapReduce has already been an indispensable framework for the programmers to develop big data applications at scale without concerning the underlying complex system details. However, such an important framework is missing on the Sunway many-core processor that powers the world-leading supercomputer Sunway Taihulight. This paper fills the gap by implementing an efficient MapReduce framework on Sunway many-core processor which takes full advantage of the architecture features such as local device memory and register communication. Specifically, we propose three different schemes that adopt different methods to partition the computation of map and reduce across the MPE and CPEs of Sunway processor. Especially in CPE-accelerated Dynamic Partitioning Scheme (CDPS), the processing role of the CPEs within each CPE pair can be transformed dynamically between map and reduce, which is effective to improve the load balance during runtime. In addition, we adopt the local device memory as well as register communication on each CPE to improve the efficiency of data access and communication. The experiment results demonstrate our MapReduce framework based on CDPS achieves 36.9× performance speedup on average across representative benchmarks.

Development and Evaluation of Chemistry-Aerosol-Climate Model CAM5-Chem-MAM7-MOSAIC: Global Atmospheric Distribution and Radiative Effects of Nitrate Aerosol.

An advanced aerosol treatment, with a focus on semivolatile nitrate formation, is introduced into the Community Atmosphere Model version 5 with interactive chemistry (CAM5‐chem) by coupling the Model for Simulating Aerosol Interactions and Chemistry (MOSAIC) with the 7‐mode Modal Aerosol Module (MAM7). An important feature of MOSAIC is dynamic partitioning of all condensable gases to the different fine and coarse mode aerosols, as governed by mode‐resolved thermodynamics and heterogeneous chemical reactions. Applied in the free‐running mode from 1995 to 2005 with prescribed historical climatological conditions, the model simulates global distributions of sulfate, nitrate, and ammonium in good agreement with observations and previous studies. Inclusion of nitrate resulted in ∼10% higher global average accumulation mode number concentrations, indicating enhanced growth of Aitken mode aerosols from nitrate formation. While the simulated accumulation mode nitrate burdens are high over the anthropogenic source regions, the sea‐salt and dust modes respectively constitute about 74% and 17% of the annual global average nitrate burden. Regional clear‐sky shortwave radiative cooling of up to −5 W m−2 due to nitrate is seen, with a much smaller global average cooling of −0.05 W m−2. Significant enhancements in regional cloud condensation nuclei (at 0.1% supersaturation) and cloud droplet number concentrations are also attributed to nitrate, causing an additional global average shortwave cooling of −0.8 W m−2. Taking into consideration of changes in both longwave and shortwave radiation under all‐sky conditions, the net change in the top of the atmosphere radiative fluxes induced by including nitrate aerosol is −0.7 W m−2.

Group Reassignment for Dynamic Edge Partitioning

Graph partitioning is a mandatory step in large-scale distributed graph processing. When partitioning real-world power-law graphs, the edge partitioning algorithm performs better than the traditional vertex partitioning algorithm, because it can cut a single vertex into multiple replicas to apportion the computation. Many advanced edge partitioning methods are designed for partitioning a static graph from scratch. However, the real-world graph structure changes continuously, which leads to a decrease in partition quality and affects the performance of the graph applications. Some studies are devoted to offline repartitioning or batch incremental partitioning, but how to deal with dynamics in real-time is still worthy of in-depth study. In this article, we discuss the impact of dynamic change on partition and discover that both insertion and deletion will lead to local suboptimal partitioning, which is the reason for the degradation of partition quality. As a solution, a dynamic edge partitioning algorithm is proposed to partition dynamics in real-time. Specifically, we deal with dynamics by a distributed stream and improve partition quality by reassigning some closely connected edges. Experiments show that it is robust to initial partition quality, dynamic scale and type, and distributed scale. Compared with the state-of-the-art dynamic partitioner, it can reduce vertex-cuts by 29.5 percent. Compared with the repartitioning algorithms, it can save the partitioning time by 91.0 percent. Applied on the graph task, it can reduce the increase of communication cost and the increase of the total time of task by 41.5 and 71.4 percent.

Coupled whole-tree optimality and xylem hydraulics explain dynamic biomass partitioning.

Trees partition biomass in response to resource limitation and physiological activity. It is presumed that these strategies evolved to optimize some measure of fitness. If the optimization criterion can be specified, then allometry can be modeled from first principles without prescribed parameterization. We present the Tree Hydraulics and Optimal Resource Partitioning (THORP) model, which optimizes allometry by estimating allocation fractions to organs as proportional to their ratio of marginal gain to marginal cost, where gain is net canopy photosynthesis rate, and costs are senescence rates. Root total biomass and profile shape are predicted simultaneously by a unified optimization. Optimal partitioning is solved by a numerically efficient analytical solution. THORP's predictions agree with reported tree biomass partitioning in response to size, water limitations, elevated CO2 and pruning. Roots were sensitive to soil moisture profiles and grew down to the groundwater table when present. Groundwater buffered against water stress regardless of meteorology, stabilizing allometry and root profiles as deep asc.30m. Much of plant allometry can be explained by hydraulic considerations. However, nutrient limitations cannot be fully ignored. Rooting mass and profiles were synchronized with hydrological conditions and groundwater even at considerable depths, illustrating that the below ground shapes whole-tree allometry.

Dynamic Preamble-Resource Partitioning for Critical MTC in Massive MIMO Systems

Preamble resources are scarce and precious in random access (RA), which need to be efficiently utilized to support critical machine-type communication (MTC) with stringent access requirements. In this article, we study a grant-free RA scenario for critical MTC in the context of massive multiple-input–multiple-output (MIMO). To enhance the access reliability of delay-sensitive devices within a predefined latency budget, two dynamic preamble-resource partitioning (DPP) schemes are proposed. Particularly, by leveraging massive MIMO, we first analytically investigate the feasibility of DPP in the considered RA scenario and demonstrate its performance superiority to the conventional scheme. Based on the analytical results, we then propose a greedy DPP scheme that performs locally optimal preamble-resource partitioning in each RA slot. To find the globally optimal DPP solution, we further propose a reinforcement learning (RL)-based scheme by modeling the considered RA scenario as a Markov decision process. Simulation results show the practicality and effectiveness of the proposed DPP schemes. In particular, to achieve a target access failure rate of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1\times 10^{-2}$ </tex-math></inline-formula> , the proposed RL-based scheme is able to enhance the RA traffic load by 42% and improve the preamble resource utilization by over 25% compared to the conventional baseline scheme.

Large-scale flow simulations using lattice Boltzmann method with AMR following free-surface on multiple GPUs

Free-surface flow simulations require high-resolution grids to capture phenomena at the interface as well as a long computational time. In this paper, we propose a numerical method for realizing large-scale free-surface flow simulations using the lattice Boltzmann method and multiple GPUs. By introducing the adaptive mesh refinement (AMR) method, which adapts high-resolution grids to free surfaces, to the lattice Boltzmann method, the number of lattice points can be greatly reduced. In the calculation of the AMR method, the spatial distribution of a computational load changes with time; therefore, the number of lattice points assigned to each GPU is kept equal by dynamic domain partitioning using a space-filling curve. We measured the weak scalability of our AMR code on the TSUBAME3.0 supercomputer at the Tokyo Institute of Technology. By hiding GPU–GPU communication overheads by the overlapping method, the performance increased 1.29 times that of the naïve implementation, and we achieved the fairly high performance of 14,570 MLUPS using 256 GPUs. We demonstrate large-scale simulations for the dam breaking problem and show a reduction in computational cost with the AMR method.

A novel energy partition model for belt grinding of Inconel 718

Analyzing and modelling energy partition is of great significance to provide an in-depth understanding of the grinding process that is essentially dynamic. However, research on dynamic energy partition in belt grinding is rather scarce, especially from the perspective of a comprehensive energy evaluation. Here, we propose a novel energy partition model for belt grinding of Inconel 718, on basis of heat transfer analysis, finite element method and optimization algorithm. The model takes into account varying grinding force, grinding temperature, material removal rate and the heat accumulation in grains. Through an iterative approach, the dynamic energy entering the workpiece, belt, chips and ambient during grinding process can be obtained. Grinding temperature and dynamic energy partition calculation are validated. Results show that the maximum error of the calculated temperature is within 8.4 %. Further, effects of grinding parameters, grinding temperature and abrasive grains on belt grinding energy partition are analyzed and discussed in detail.

Evaluating community detection algorithms for progressively evolving graphs

Abstract Many algorithms have been proposed in the last 10 years for the discovery of dynamic communities. However, these methods are seldom compared between themselves. In this article, we propose a generator of dynamic graphs with planted evolving community structure, as a benchmark to compare and evaluate such algorithms. Unlike previously proposed benchmarks, it is able to specify any desired evolving community structure through a descriptive language, and then to generate the corresponding progressively evolving network. We empirically evaluate six existing algorithms for dynamic community detection in terms of instantaneous and longitudinal similarity with the planted ground truth, smoothness of dynamic partitions and scalability. We notably observe different types of weaknesses depending on their approach to ensure smoothness, namely Glitches, Oversimplification and Identity loss. Although no method arises as a clear winner, we observe clear differences between methods, and we identified the fastest, those yielding the most smoothed or the most accurate solutions at each step.

Improved simulation of plant-animal interactions in African savannas with the extended land use change model LUCIA

Process-based modelling is needed to assess potential ecological impacts of large-scale intensification of African savanna rangelands and to investigate major processes involved, that cannot be tested on appropriate temporal and spatial scales at field level. To date, modelling conversion or integration of crop- and pasture-based systems in savanna ecosystems is not possible at the necessary level of detail. The Land Use Change Impact Assessment tool (LUCIA) dynamically simulates spatially explicit impacts of land-use change on ecosystem functions within a grid-based landscape. We added a physiologically detailed source-sink plant growth module with growth reserves, dynamic biomass partitioning, and dormancy to LUCIA to simulate a grazed rangeland vegetation in Borana, Ethiopia. Grass and tree growth, validated against measured above-ground biomass time series were most sensitive to dormancy break or induction parameters that control the growing period under strong rainfall seasonality. A moderate stocking rate of cattle allowed pasture persistence and continuous fodder production. Overgrazing caused depletion of root and tree stem growth reserves that were needed for seasonal regrowth. Consequently, pasture degraded due to reduced regrowth capacity and reserve replenishment. Physiological processes are well represented, laying the foundation for integration of further ecological processes, such as tree recruitment, bush encroachment, and plant community shifts.

Online reconfiguration of regularity-based resource partitions in cyber-physical systems

We consider the problem of resource provisioning for real-time cyber-physical applications in an open system environment where there does not exist a global resource scheduler that has complete knowledge of the real-time performance requirements of each individual application that shares the resources with the other applications. Regularity-based Resource Partition (RRP) model is an effective strategy to hierarchically partition and assign various resource slices among the applications. However, RRP model does not consider changes in resource requests from the applications at run time. To allow for the run time adaptation to resource requirement changes, we consider in this paper the issues in online resource partition reconfiguration, including semantics issues that arise in configuration transitions that may cause application failures. Based on the reconfiguration semantics, we study the online resource reconfigurability problem under the RRP model where the availability factors of resource partitions may be reconfigured at run time. We formalize and solve the Dynamic Partition Reconfiguration (DPR) problem for uniform environment where the minimal intervals assigned to each task for execution on each resource are the same. Extensive experiments have been conducted to evaluate the performance of the proposed approaches in different scenarios. We also present a case study using the autonomous F1/10 model car; the controller of the F1/10 car requires resource adaptation to satisfy the computing needs of its PID controller and vision system under different operating conditions. Our implementation demonstrates the effectiveness and benefit of online resource partition reconfiguration using the proposed approach in a real-world cyber-physical system.

A study of dynamic energy partition in belt grinding based on grinding effects and temperature dependent mechanical properties

• A dynamic energy partition calculating model in abrasive belt grinding is proposed. • Cutting and ploughing dominate material removal for soft material while ploughing is more dominant for hard material. • Heat accumulation of grains reduces cooling capacity of abrasive belt during grinding. • High grinding temperature softens workpiece and makes cutting action occur easily. Analyzing and modelling the dynamic energy partition is of great significance to revealing the mechanism of belt grinding and improving grinding quality. However, there is lack of comprehensive studies on energy partition in belt grinding, especially when the dynamic characteristics are under consideration. To fill this gap, this paper analyzed the grinding energy partition from the perspective of grinding effects and thermal aspects. Grinding effects are distinguished by combination of single grain scratch tests and force balance of one grain in view of dynamic and elastic contact conditions. Thermal aspects are obtained by a fusion method of finite element method (FEM) and optimization algorithm. Then, by utilizing the iterative approach, heat accumulating effect and temperature dependent mechanical properties of workpieces are taken into account either and the dynamic energy partition is calculated in a continuous grinding process. Validations on two workpieces (made by SUS304 and AA6061-T6) prove that the proposed dynamic energy partition calculating method is effective and the maximum error of q w is 17.2 %. The proposed method not only enhances the understanding of dynamic energy partition in robotic belt grinding, but also offers a new venue for studying abrasive belt grinding.

Optimizing Vertex Pressure Dynamic Graph Partitioning in Many-Core Systems

With the rise of graph-based algorithms in many applications, dynamic graphs have become critical for applications that work with real-time or time-series relationship data. Due to their large storage footprint, these applications require high parallelism and locality. Many-core architectures offer high parallelism for both static and dynamic graphs. However, systems operating on dynamic graphs must continuously repartition graph data across storage units to achieve good locality, load balancing, and performance. In this work, we examine the effectiveness and efficiency of different vertex-pressure repartitioning schemes, which move vertices so to co-locate them near their most relevant neighbors. We describe key repartitioning design choices and provide a thorough evaluation of the impact of a range of design features with different datasets. Our evaluation indicates that optimized dynamic repartitioning techniques can often provide over 2x performance speedup over state-of-the-art static solutions.

Dynamic Reserve and Transmission Capacity Allocation in Wind-Dominated Power Systems

The large shares of wind power generation in electricity markets motivate higher levels of operating reserves. However, current reserve sizing practices fail to account for important topological aspects that might hinder their deployment, thus resulting in high operating costs. Zonal reserve procurement mitigates such inefficiencies, however, the way the zones are defined is still open to interpretation. This paper challenges the efficiency of predetermined zonal setups that neglect the location of stochastic power production in the system, as well as the availability, cost and accessibility of flexible generating units. To this end, we propose a novel reserve procurement approach, formulated as a two-stage stochastic bilevel model, in which the upper level identifies a number of contiguous reserve zones using dynamic grid partitioning and sets zonal requirements based on the total expected operating costs. Using two standard IEEE reliability test cases, we show how the efficient partitioning of reserve zones can reduce expected system cost and promote the integration of stochastic renewables.

Data-driven Distributionally Robust Optimization For Vehicle Balancing of Mobility-on-Demand Systems

With the transformation to smarter cities and the development of technologies, a large amount of data is collected from sensors in real time. Services provided by ride-sharing systems such as taxis, mobility-on-demand autonomous vehicles, and bike sharing systems are popular. This paradigm provides opportunities for improving transportation systems’ performance by allocating ride-sharing vehicles toward predicted demand proactively. However, how to deal with uncertainties in the predicted demand probability distribution for improving the average system performance is still a challenging and unsolved task. Considering this problem, in this work, we develop a data-driven distributionally robust vehicle balancing method to minimize the worst-case expected cost. We design efficient algorithms for constructing uncertainty sets of demand probability distributions for different prediction methods and leverage a quad-tree dynamic region partition method for better capturing the dynamic spatial-temporal properties of the uncertain demand. We then derive an equivalent computationally tractable form for numerically solving the distributionally robust problem. We evaluate the performance of the data-driven vehicle balancing algorithm under different demand prediction and region partition methods based on four years of taxi trip data for New York City (NYC). We show that the average total idle driving distance is reduced by 30% with the distributionally robust vehicle balancing method using quad-tree dynamic region partitions, compared with vehicle balancing methods based on static region partitions without considering demand uncertainties. This is about a 60-million-mile or a 8-million-dollar cost reduction annually in NYC.

Dynamic Graph Partitioning Scheme for Supporting Load Balancing in Distributed Graph Environments

As dynamic graph data have been actively used, incremental graph partition schemes have been studied to efficiently store and manage large graphs. In this paper, we propose a vertex-cut based novel incremental graph partitioning scheme that supports load balancing in a distributed environment. The proposed scheme chooses the load of each node that considers its storage utilization and throughput as the partitioning criterion. The proposed scheme defines hot data that means a particular vertex frequently searched among graphs requested by queries. We manage and utilize hot data for graph partitioning. Finally, we perform vertex-cut based dynamic graph partitioning by using a vertex replication index, the load each node, and hot data to distribute the load evenly in a distributed environment. In order to verify the superiority of the proposed partitioning scheme, we compare it with the existing partitioning schemes through a variety of performance evaluations.

Metaheuristics-based routing optimisation, balanced workload distribution and security strategy in IoT environment

The advancement of wireless sensor networks (WSNs) made the society ever so slightly gain new degrees of freedom, intrinsically with respect to connectivity. The emergence of internet of things (IoT) together with the onset and development of evolutionary computing, address the various needs of the growing urbanisation with a significant allusion altogether. Applications built on cloud infrastructure are deployed for job segregation, rapid processing and utilisation but with limitation of proper computation in high density IoT environment. This paper strives on resolving some of the issues so accosted by the IoT paradigm, with the help of a new metaheuristics-based data routing hybrid algorithm based on directed artificial bat algorithm (DABA) and particle swarm optimisation (PSO) to optimise connection issues such as real time delay and network congestion. This approach also introduces the clustering concept together with fog computing to distribute the network stress and to optimise the usage of bandwidth using the dynamic graph partitioning algorithm. The paper further proposes advance metaheuristics constricted PSO (C-PSO) in the hybrid algorithm and effective proposal for security strategy to enhance the efficiency of the IoT environment.

Secure-Stor: A Novel Hybrid Storage System Architecture to Enhance Security and Performance in Edge Computing

The Internet of Things (IoTs) is attracting the attention of scientists worldwide. With its explosive growth along with applications that require real-time computing power, a new technology called edge computing has emerged. As a result, edge computing has changed the way data are processed and handled back and forth for millions of devices worldwide, such as autonomous vehicles and electric cars. The confinement of cloud computing technology, such as a content delivery network (CDN), contributed significantly to edge computing development. Currently, this technology can meet the demands of ever-growing mobile devices and the IoTs. This paper describes a novel secure framework consisting of a hybrid storage architecture consisting of CDN, edge computing, and centralized storage. Centralized storage consists of multilevel storage systems that comprise solid-state drivers (SSDs) and hard disk drivers (HDDs), which provide optimal data storage solution for a wide variety of real-time data processing applications. Transforming the data back and forth between SSDs and HDDs is crucial for achieving high performance while meeting the edge device request deadline. Additionally, a new dynamic solid-state disk partitioning mechanism is introduced to optimize security for the proposed framework among hard disk drives. A partition from the solid-state disks to hard disk drives is assigned based on hard disk drive workloads.

Metaheuristics-based routing optimisation, balanced workload distribution and security strategy in IoT environment

The advancement of wireless sensor networks (WSNs) made the society ever so slightly gain new degrees of freedom, intrinsically with respect to connectivity. The emergence of internet of things (IoT) together with the onset and development of evolutionary computing, address the various needs of the growing urbanisation with a significant allusion altogether. Applications built on cloud infrastructure are deployed for job segregation, rapid processing and utilisation but with limitation of proper computation in high density IoT environment. This paper strives on resolving some of the issues so accosted by the IoT paradigm, with the help of a new metaheuristics-based data routing hybrid algorithm based on directed artificial bat algorithm (DABA) and particle swarm optimisation (PSO) to optimise connection issues such as real time delay and network congestion. This approach also introduces the clustering concept together with fog computing to distribute the network stress and to optimise the usage of bandwidth using the dynamic graph partitioning algorithm. The paper further proposes advance metaheuristics constricted PSO (C-PSO) in the hybrid algorithm and effective proposal for security strategy to enhance the efficiency of the IoT environment.

Hardware Pipelining of Repetitive Patterns in Processor Instruction Traces

Dynamic partitioning is a promising technique where computations are transparently moved from a Gene-&#x0D; ral Purpose Processor (GPP) to a coprocessor during application execution. To be effective, the mapping&#x0D; of computations to the coprocessor needs to consider aggressive optimizations. One of the mapping opti-&#x0D; mizations is loop pipelining, a technique extensively studied and known to allow substantial performance&#x0D; improvements. This paper describes a technique for pipelining Megablocks, a type of runtime loop deve-&#x0D; loped for dynamic partitioning. The technique transforms the body of Mega-blocks into an acyclic dataflow&#x0D; graph which can be fully pipe-lined and is based on the atomic execution of loop iterations. For a set of 9 ben-&#x0D; chmarks without memory operations, we generated pipelined hardware versions of the loops and esti-mate&#x0D; that the presented loop pipelining technique increases the average speedup of non-pipelined coprocessor&#x0D; accelerated designs from 1.6× to 2.2×. For a larger set of 61 benchmarks which include memory operations,&#x0D; we estimate through simulation a speedup increase from 2.5× to 5.6× with this technique.