Spatial Partitioning Techniques Research Articles

Efficient Large-scale Nonstationary Spatial Covariance Function Estimation Using Convolutional Neural Networks

Spatial processes observed in various fields, such as climate and environmental science, often occur at large-scale and demonstrate spatial nonstationarity. However, fitting a Gaussian process with a nonstationary Matérn covariance is challenging, as it requires handling the complexity and computational demands associated with modeling the varying spatial dependencies over large and heterogeneous domains. Previous studies in the literature have tackled this challenge by employing spatial partitioning techniques to estimate the parameters that vary spatially in the covariance function. The selection of partitions is an important consideration, but it is often subjective and lacks a data-driven approach. To address this issue, in this study, we utilize the power of Convolutional Neural Networks (ConvNets) to derive subregions from the nonstationary data by employing a selection mechanism to identify subregions that exhibit similar behavior to stationary fields. We rely on the ExaGeoStat software for large-scale geospatial modeling to implement the nonstationary Matérn covariance for large-scale exact computation of nonstationary Gaussian likelihood. We also assess the performance of the proposed method with synthetic and real datasets at large-scale. The results revealed enhanced accuracy in parameter estimations when relying on ConvNet-based partition compared to traditional user-defined approaches.

SPAW-SMOTE: Space Partitioning Adaptive Weighted Synthetic Minority Oversampling Technique For Imbalanced Data Set Learning

Abstract The problem of data imbalance is common in reality, which greatly affects the performance of classifiers. Most of the solutions are to balance the data set by generating new minority class samples, which are faced with the problems of selecting the appropriate area for generating samples, fuzzy classification boundary and uneven distribution of samples. To solve these problems, we propose a novel oversampling algorithm named space partitioning adaptive weighted synthetic minority oversampling technique (SPAW-SMOTE). We first divide the data space into boundary space and non-boundary space based on spatial partitioning techniques. The number of samples to be generated is assigned to different spaces by the designed adaptive weighting algorithm, which is used to solve the problems of uneven distribution of samples and easy to blur the classification boundary. Finally, we also endeavor to develop a new generation algorithm to reduce the probability of overlapping samples generated when synthesizing new samples and to ensure the diversity of new samples. Experimental results on 18 real-world data sets show that the average performance (G-mean, F1-measure and Area Under Curve) of SPAW-SMOTE is significantly better than other existing oversampling techniques.

PAGE: Parallel Scalable Regionalization Framework

Regionalization techniques group spatial areas into a set of homogeneous regions to analyze and draw conclusions about spatial phenomena. A recent regionalization problem, called MP-regions, groups spatial areas to produce a maximum number of regions by enforcing a user-defined constraint at the regional level. The MP-regions problem is NP-hard. Existing approximate algorithms for MP-regions do not scale for large datasets due to their high computational cost and inherently centralized approaches to process data. This article introduces a parallel scalable regionalization framework ( PAGE ) to support MP-regions on large datasets. The proposed framework works in two stages. The first stage finds an initial solution through randomized search, and the second stage improves this solution through efficient heuristic search. To build an initial solution efficiently, we extend traditional spatial partitioning techniques to enable parallelized region building without violating the spatial constraints. Furthermore, we optimize the region building efficiency and quality by tuning the randomized area selection to trade off runtime with region homogeneity. The experimental evaluation shows the superiority of our framework to support an order of magnitude larger datasets efficiently compared to the state-of-the-art techniques while producing high-quality solutions.

Efficient distributed algorithms for distance join queries in spark-based spatial analytics systems

ABSTRACT Apache Sedona (formerly GeoSpark) is a new in-memory cluster computing system for processing large-scale spatial data, which extends the core of Apache Spark to support spatial datatypes, partitioning techniques, spatial indexes, and spatial operations (e.g. spatial range, nearest neighbor, and spatial join queries). Distance-based Join Queries (DJQs), like nearest neighbor join (kNNJQ) or closest pairs queries (kCPQ), are not supported by it. Therefore, in this paper, we investigate how to design and implement efficient DJQ distributed algorithms in Apache Sedona, using the most appropriate spatial partitioning and other optimization techniques. The results of an extensive set of experiments with real-world datasets are presented, demonstrating that the proposed kNNJQ and kCPQ distributed algorithms are efficient, scalable, and robust in Apache Sedona. Finally, Sedona is also compared to other similar cluster computing systems, showing the best performance for kCPQ and competitive results for kNNJQ.

Revalidation Technique on Landslide Susceptibility Modelling: An Approach to Local Level Disaster Risk Management in Kuala Lumpur, Malaysia

Landslide susceptibility modelling in tropical climates is hindered by incomplete inventory due to rapid development and natural processes that obliterate field evidence, making validation a challenge. Susceptibility modelling was conducted in Kuala Lumpur, Malaysia using a new spatial partitioning technique for cross-validation. This involved a series of two alternating east-west linear zones, where the first zone served as the training dataset and the second zone was the test dataset, and vice versa. The results show that the susceptibility models have good compatibility with the selected landslide conditioning factors and high predictive accuracy. The model with the highest area under curve (AUC) values (SRC = 0.92, PRC = 0.90) was submitted to the City Council of Kuala Lumpur for land use planning and development control. Rainfall-induced landslides are prominent within the study area, especially during the monsoon period. An extreme rainfall event in December 2021 that triggered 122 landslides provided an opportunity to conduct retrospective validation of the model; the high predictive capability (AUC of PRC = 0.93) was reaffirmed. The findings proved that retrospective validation is vital for landslide susceptibility modelling, especially where the inventory is not of the best quality. This is to encourage wider usage and acceptance among end users, especially decision-makers in cities, to support disaster risk management in a changing climate.

Efficient spatial data partitioning for distributed kNN joins

Parallel processing of large spatial datasets over distributed systems has become a core part of modern data analytic systems like Apache Hadoop and Apache Spark. The general-purpose design of these systems does not natively account for the data’s spatial attributes and results in poor scalability, accuracy, or prolonged runtimes. Spatial extensions remedy the problem and introduce spatial data recognition and operations. At the core of a spatial extension, a locality-preserving spatial partitioner determines how to spatially group the dataset’s objects into smaller chunks using the distributed system’s available resources. Existing spatial extensions rely on data sampling and often mismanage non-spatial data by either overlooking their memory requirements or excluding them entirely. This work discusses the various challenges that face spatial data partitioning and proposes a novel spatial partitioner for effectively processing spatial queries over large spatial datasets. For evaluation, the proposed partitioner is integrated with the well-known k-Nearest Neighbor (kNN) spatial join query. Several experiments evaluate the proposal using real-world datasets. Our approach differs from existing proposals by (1) accounting for the dataset’s unique spatial traits without sampling, (2) considering the computational overhead required to handle non-spatial data, (3) minimizing partition shuffles, (4) computing the optimal utilization of the available resources, and (5) achieving accurate results. This contributes to the problem of spatial data partitioning through (1) providing a comprehensive discussion of the problems facing spatial data partitioning and processing, (2) the development of a novel spatial partitioning technique for in-memory distributed processing, (3) an effective, built-in, load-balancing methodology that reduces spatial query skews, and (4) a Spark-based implementation of the proposed work with an accurate kNN spatial join query. Experimental tests show up to 1.48 times improvement in runtime as well as the accuracy of results.

DISSEC: A distributed deep neural network inference scheduling strategy for edge clusters

New applications such as intelligent manufacturing, autonomous vehicles and smart cities drive large-scale deep learning models deployed in the Internet of Things (IoT) edge environments. However, deep learning models require substantial computations, storage and communication resources to run. It is generally difficult to deploy and execute a complete deep neural network (DNN) on a resource-constrained edge device. One possible solution is to slice the DNN into multiple tiles distributed to different edge devices, which can reduce the number of computations and quantity of data on each edge device. In this paper, we propose DISSEC, a distributed scheduling strategy for DNN inference on IoT edge clusters. DISSEC leverages spatial partitioning techniques through fusing the convolutional layers and dividing them into multiple partitions that can be executed independently, and proposes a method to express the dependencies between partitions. It further proposes a search algorithm based on heuristics to produce a distributed parallel strategy with the best overall inference execution latency. The evaluation shows that our strategy can fully utilize the edge device resources by cooperating with multiple edge devices to perform partitioning tasks in parallel. Furthermore, compared to the existing work scheduling strategy, our strategy reduces communication overhead by 20% and overall execution latency by 9% under different partitioning granularities and numbers of edge devices.

Grids Versus Graphs: Partitioning Space for Improved Taxi Demand-Supply Forecasts

Accurate taxi demand-supply forecasting is a challenging application of ITS (Intelligent Transportation Systems), due to the complex spatial and temporal patterns involved. We investigate the impact of different spatial partitioning techniques on the prediction performance of an LSTM (Long Short-Term Memory) network, in the context of taxi demand-supply forecasting. We consider two tessellation schemes: (i) the variable-sized Voronoi tessellation, and (ii) the fixed-sized Geohash tessellation. While the widely employed ConvLSTM (Convolutional LSTM) method can model fixed-sized Geohash partitions, the standard convolutional filters cannot be applied on variable-sized Voronoi partitions. To explore the impact of the Voronoi strategy, we propose the use of a GraphLSTM (Graph-based LSTM) model, by representing the Voronoi spatial partitions as nodes on an arbitrarily structured graph. The GraphLSTM model offers competitive performance against the ConvLSTM model, at a lower computational complexity, across three real-world large-scale taxi demand-supply data sets, with different performance metrics. To ensure superior performance across diverse settings, a HEDGE based ensemble learning algorithm is applied over the ConvLSTM and the GraphLSTM networks.

Using Deep Learning for Big Spatial Data Partitioning

This article explores the use of deep learning to choose an appropriate spatial partitioning technique for big data. The exponential increase in the volumes of spatial datasets resulted in the development of big spatial data frameworks. These systems need to partition the data across machines to be able to scale out the computation. Unfortunately, there is no current method to automatically choose an appropriate partitioning technique based on the input data distribution. This article addresses this problem by using deep learning to train a model that captures the relationship between the data distribution and the quality of the partitioning techniques. We propose a solution that runs in two phases, training and application. The offline training phase generates synthetic data based on diverse distributions, partitions them using six different partitioning techniques, and measures their quality using four quality metrics. At the same time, it summarizes the datasets using a histogram and well-designed skewness measures. The data summaries and the quality metrics are then use to train a deep learning model. The second phase uses this model to predict the best partitioning technique given a new dataset that needs to be partitioned. We run an extensive experimental evaluation on big spatial data, and we experimentally show the applicability of the proposed technique. We show that the proposed model outperforms the baseline method in terms of accuracy for choosing the best partitioning technique by only analyzing the summary of the datasets.

Improving Distance-Join Query processing with Voronoi-Diagram based partitioning in SpatialHadoop

SpatialHadoop is an extended MapReduce framework supporting global indexing techniques that partition spatial datasets across several machines and improve spatial query processing performance compared to traditional Hadoop systems. SpatialHadoop supports several spatial operations (e.g., K Nearest Neighbor search, range query, spatial intersection join, etc.) and seven spatial partitioning techniques (Grid, Quadtree, STR, STR+, k-d tree, Z-curve and Hilbert-curve). Distance-Join Queries (DJQs), like the K Nearest Neighbors Join Query (KNNJQ) and K Closest Pairs Query (KCPQ), are common operations used in numerous spatial applications. DJQs are costly operations, since they combine spatial joins with distance-based search. Data partitioning improves the management of large datasets and speeds up query performance. Therefore, performing DJQs efficiently with new partitioning methods in SpatialHadoop is a challenging task. In this paper, a new data partitioning technique based on Voronoi-Diagrams is designed and implemented in SpatialHadoop. Moreover, improved KNNJQ and KCPQ MapReduce algorithms, using the new partitioning mechanism, are also designed and developed for SpatialHadoop. Finally, the results of an extensive set of experiments with real-world datasets are presented, demonstrating that the new partitioning technique and the improved DJQ MapReduce algorithms are efficient, scalable and robust in SpatialHadoop.

Spatial data management in apache spark: the GeoSpark perspective and beyond

The paper presents the details of designing and developing GeoSpark, which extends the core engine of Apache Spark and SparkSQL to support spatial data types, indexes, and geometrical operations at scale. The paper also gives a detailed analysis of the technical challenges and opportunities of extending Apache Spark to support state-of-the-art spatial data partitioning techniques: uniform grid, R-tree, Quad-Tree, and KDB-Tree. The paper also shows how building local spatial indexes, e.g., R-Tree or Quad-Tree, on each Spark data partition can speed up the local computation and hence decrease the overall runtime of the spatial analytics program. Furthermore, the paper introduces a comprehensive experiment analysis that surveys and experimentally evaluates the performance of running de-facto spatial operations like spatial range, spatial K-Nearest Neighbors (KNN), and spatial join queries in the Apache Spark ecosystem. Extensive experiments on real spatial datasets show that GeoSpark achieves up to two orders of magnitude faster run time performance than existing Hadoop-based systems and up to an order of magnitude faster performance than Spark-based systems.

Spatial partitioning techniques in SpatialHadoop

SpatialHadoop is an extended MapReduce framework that supports global indexing that spatial partitions the data across machines providing orders of magnitude speedup, compared to traditional Hadoop. In this paper, we describe seven alternative partitioning techniques and experimentally study their effect on the quality of the generated index and the performance of range and spatial join queries. We found that using a 1% sample is enough to produce high quality partitions. Also, we found that the total area of partitions is a reasonable measure of the quality of indexes when running spatial join. This study will assist researchers in choosing a good spatial partitioning technique in distributed environments.

Self-triggered coordination of robotic networks for optimal deployment

This paper studies a deployment problem for a group of robots where individual agents operate with outdated information about each other’s locations. Our objective is to understand to what extent outdated information is still useful and at which point it becomes essential to obtain new, up-to-date information. We propose a self-triggered coordination algorithm based on spatial partitioning techniques with uncertain information. We analyze its correctness in synchronous and asynchronous scenarios, and establish the same convergence guarantees that a synchronous algorithm with perfect information at all times would achieve. The technical approach combines computational geometry, set-valued stability analysis, and event-based systems.

The recovery of acid‐damaged zooplankton communities in Canadian Lakes: the relative importance of abiotic, biotic and spatial variables

Summary1. Acidification has damaged biota in thousands of lakes and streams throughout eastern North America. Fortunately, reduced emissions of sulphur dioxide and nitrogen oxides beginning in the 1960s have allowed pH levels in many affected systems to increase. Determining the extent of biological and pH recovery in these systems is necessary to assess the success of emissions reductions programmes.2. Although there have been promising signs of biological recovery in many systems, recovery has occurred more slowly than expected for some taxa. Past studies with crustacean zooplankton indicate that a mixture of local abiotic variables, biotic variables and dispersal processes may influence the structure of recovering communities. However, most studies have been unable to determine the relative importance of these three groups of variables.3. We assessed chemical and biological recovery of acid‐damaged lakes in Killarney Park, Ontario. In addition, we assessed the relative importance of local abiotic variables, biotic variables and dispersal processes for structuring recovering communities. We collected zooplankton community data, abiotic and biotic data from 45 Killarney Park lakes. To assess the recovery of zooplankton communities, we compared zooplankton data collected in 2005 to a survey conducted for the same lakes in 1972–73 using several univariate measures of community structure, as well as multivariate methods based on relative species abundances. To determine the factors influencing the structure of recovering zooplankton communities, we used hierarchical partitioning for univariate measures and spatial modelling and variation partitioning techniques for multivariate analyses.4. Our survey revealed significant pH increases for the majority of sampled lakes but univariate measures of community structure, such as species richness and diversity, indicated that only minor changes have occurred in many acid‐damaged lakes. Hierarchical partitioning identified several variables that may influence our univariate measures of recovery, including pH, dissolved organic carbon (DOC) levels, fish presence/absence, lake surface area and lake elevation.5. Multivariate methods revealed a shift in communities through time towards a structure more typical of neutral lakes, providing some evidence for recovery. Variation partitioning suggested that the structure of recovering copepod communities was influenced most by dispersal processes and abiotic variables, while biotic (Chaoborus densities, fish presence/absence) and abiotic variables were more important for cladoceran zooplankton.6. Our results indicate that the recovery of zooplankton communities in Killarney Park is not yet complete, despite decades of emission reductions. The importance of variables related to acidification, such as pH and DOC, indicates that further chemical recovery may be necessary. The differing importance of abiotic, biotic and dispersal processes for structuring copepod versus cladoceran zooplankton might indicate that different management approaches and expectations for recovery are needed for these groups.

Does dispersal limitation impact the recovery of zooplankton communities damaged by a regional stressor?

The acidification and ongoing pH recovery of lakes in Killarney Provincial Park, Canada, provide a unique opportunity to increase our understanding of the role of dispersal as communities respond to environmental change. Time lags in community recovery following pH increases in acidified lakes have typically been attributed to local factors; however, no studies have been conducted to determine if colonist availability could also play a role. Moreover, the rates and mechanisms of dispersal to recovering lakes are poorly understood. In this study, we sought to determine if dispersal limitation could impede the recovery of zooplankton communities affected by a regional stressor. To achieve this objective, we used a combination of empirical data collection along with spatial modeling and variation partitioning techniques. Data were collected by measuring dispersal to four recovering lakes in Killarney Park. Dispersal traps were placed next to lakes to measure immigration overland, drift nets were used to measure immigration via streams, and in situ emergence traps were used to quantify immigration from historically deposited resting eggs. Documented dispersal levels were then compared with the theoretical critical density required for reproduction (N(c)) to determine if adequate numbers were dispersing to establish populations of acid-sensitive species in recovering lakes. Spatial modeling and variation partitioning were conducted using community and physical/chemical data for 45 park lakes that were collected in 1972-1973, 1990, and 2005. Field data demonstrated that a variety of zooplankton species were dispersing to recovering lakes through streams and the egg bank, but few individuals were collected dispersing overland. Although we identified 24 species of zooplankton dispersing, only six species absent from the communities of our study lakes were identified from our traps, and two of these species did not disperse in high enough numbers to surpass N(c). Local environmental variables explained the largest proportion of the variation in zooplankton communities (18-37%); however, spatial variables were also important (7-18%). The significant spatial patterns we found in the park's zooplankton communities, combined with the low overland dispersal levels we documented, suggest that dispersal limitation may be a more important impediment to recovery than was previously thought.

Binary spatial partitioning of the central‐difference time integration scheme for explicit fast transient dynamics

AbstractThis paper proposes a generalization of the explicit central‐difference time integration scheme, using a time step variable not only in time but also in space. The solution at each element/node is advanced in time following local rather than global stability limitations. This allows substantial saving of computer time in realistic applications with non‐uniform meshes, especially in multi‐field problems like fluid–structure interactions. A binary scheme in space is used: time steps are not completely arbitrary, but stay in a constant ratio of two when passing from one partition level to the next one. This choice greatly facilitates implementation (via an integer‐based logic), ensures inherent synchronization and avoids any interpolations, necessary in other partitioning schemes in the literature, but which may reduce numerical stability. The mesh partition is automatically built up and continuously updated by simple spatial adjacency considerations. The resulting algorithm deals automatically with large variations in time of stability limits. The paper introduces the core spatial partitioning technique in the Lagrangian formulation. Some academic numerical examples allow a detailed comparison with the standard, spatially uniform algorithm. A final more realistic example shows the application of partitioning in simulations with arbitrary Lagrangian Eulerian formulation and fully‐coupled boundary conditions (fluid–structure interaction). Copyright © 2008 John Wiley & Sons, Ltd.

Design of vehicle routing zones for large-scale distribution systems

This paper proposes algorithms to automatically discretize vehicle routing zones (VRZ) from continuum approximation guidelines. VRZs that satisfy these guidelines (i.e., shape and size requirements) are useful to the planning and operation of distribution systems in several practical contexts. In the literature, however, VRZ design normally requires human intervention, and this is inconvenient and time-consuming especially for large-scale and complex systems. This study fills this gap by utilizing a combination of spatial partitioning techniques to systematically obtain optimum zone designs. Numerical examples show that the proposed algorithm yields satisfying outcomes.

Elements of a representation framework for performance-based design

We describe selected aspects of a computer-based representation framework aimed at supporting performance analysis during schematic and detailed design. The framework is based on two distinct building representations that include sufficient information for a range of performance analysis tasks. Automated, bi-directional mappings maintain consistency between the two representations. Schematic building configurations are defined in a sheet-type representation, and may be expanded into a solid-type representation, where outlines of building and space enclosures may be decomposed into construction layers or components. Rapid model manipulation is supported by spatial partitioning techniques. A case study illustrates how the framework could be used in performance-based design practice.

공간 질의 최적화를 위한 힐버트 공간 순서화에 따른 공간 분할

공간 질의 크기에 대한 근사치를 구하기 위해서는 입력 데이터 공간을 분할한 후 분할된 영역에 대하여 질의 결과 크기를 추정한다. 본 논문에서는 데이터 편재가 심한 공간 데이터에 대한 질의 크기 추정의 문제를 논의한다. 공간을 분할하는 기법으로 관계 데이터베이스에서 많이 사용되는 너비 균등, 높이 균등 히스토그램에 해당되는 면적 균등, 개수 균등 분할에 대한 방법을 검토하고 공간 인덱싱에 기초한 공간 분할방법에 대해서 알아본다. 본 논문에서는 공간 순서화 기법인 힐버트 공간 채움 곡선을 이용한 공간 분할을 제안한다. 제안한 방법과 기존의 방법을 실제 데이터와 인위 데이터를 사용하여 편재된 공간 데이터에 대한 질의 결과 크기의 추정에 대한 정확도를 비교한다. 본 실험에서 힐버트 채움 곡선에 의한 공간 분할이 공간 질의 크기 버켓 수의 변화, 데이터 위치 편재도의 변화, 데이터 크기의 변화에 대해서 기존의 분할 방법보다 질의 결과 크기 추정에 대해서 우수한 성능을 보였다. In order to approximate the spatial query result size we partition the input rectangles into subsets and estimate the query result size based on the partitioned spatial area. In this paper we examine query result size estimation in skewed data. We examine the existing spatial partitioning techniques such as equi-area and equi-count partitioning, which are analogous to the equi-width and equi-height histograms used in relational databases, and examine the other partitioning techniques based on spatial indexing. In this paper we propose a new spatial partitioning technique based on the Hilbert space filling curve. We present a detailed experimental evaluation comparing the proposed technique and the existing techniques using synthetic as well as real-life datasets. The experiments showed that the proposed partitioning technique based on the Hilbert space filling curve achieves better query result size estimation than the existing techniques for space query size, bucket numbers, skewed data, and spatial data size.

Multi-feature speech/music discrimination system

A speech/music discriminator employs data from multiple features of an audio signal (10) as input to a classifier (16). Some of the feature data is determined from individual frames of the audio signal, and other input data is based upon variations of a feature over several frames, to distinguish the changes in voiced and unvoiced components of speech from the more constant characteristics of music. Several different types of classifiers for labeling test points on the basis of the feature data are disclosed. A preferred set of classifiers is based upon variations of a nearest-neighbor approach, including a K-d tree spatial partitioning technique.