Morton Order Research Articles

Coin-Flipping, Ball-Dropping, and Grass-Hopping for Generating Random Graphs from Matrices of Edge Probabilities

Common models for random graphs, such as Erdös--Rényi and Kronecker graphs, correspond to generating random adjacency matrices where each entry is nonzero based on a large matrix of probabilities. Generating an instance of a random graph based on these models is easy, although inefficient, by flipping biased coins (i.e., sampling binomial random variables) for each possible edge. This process is inefficient because most large graph models correspond to sparse graphs where the vast majority of coin flips will result in no edges. We describe some not entirely well-known, but not entirely unknown, techniques that will enable us to sample a graph by finding only the coin flips that will produce edges. Our analogies for these procedures are ball-dropping, which is easier to implement, but may need extra work due to duplicate edges, and grass-hopping, which results in no duplicated work or extra edges. Grass-hopping does this using geometric random variables. In order to use this idea for complex probability matrices such as those in Kronecker graphs, we decompose the problem into three steps, each of which is an independently useful computational primitive: (i) enumerating nondecreasing sequences; (ii) unranking multiset permutations; and (iii) decoding and encoding z-curve and Morton codes and permutations. The third step is the result of a new connection between repeated Kronecker product operations and Morton codes. Throughout, we draw connections to ideas underlying applied math and computer science, including coupon collector problems.

Parallel BVH Construction Using Locally Density Clustering

A novel bounding volume hierarchy (BVH) construction method based on locally dense clustering (LDC) was proposed for the low quality of BVH constructed in a complex scene with uneven distribution of primitives. The quality of the BVH was effectively improved by defining primitive density, analyzing the relation between density and traversal efficiency, selecting primitives with high density as the clustering center at the early stages of construction, and combining with top-down iterative clustering and bottom-up agglomerative clustering (AC). In order to effectively calculate primitive density, a local search strategy based on Morton coding was adopted. This strategy can quickly get approximate primitive density information through GPU. We evaluate the method and show that, in the complex building scene with uneven distribution of primitives, our method is 13% higher in traversal speed and 11% lower in surface area heuristic (SAH) cost. That means our method can improve the rendering speed of ray tracing effectively.

Constructing a High-Dimensional k NN-Graph Using a Z-Order Curve

Although many fast methods exist for constructing a k NN-graph for low-dimensional data, it is still an open question how to do it efficiently for high-dimensional data. We present a new method to construct an approximate k NN-graph for medium- to high-dimensional data. Our method uses one-dimensional mapping with a Z-order curve to construct an initial graph and then continues to improve this using neighborhood propagation. Experiments show that the method is faster than the compared methods with five different benchmark datasets, the dimensionality of which ranges from 14 to 784. Compared to a brute-force approach, the method provides a speedup between 12.7:1 and 414.2:1 depending on the dataset. We also show that errors in the approximate k NN-graph originate more likely from outlier points; and, it can be detected during runtime, which points are likely to have errors in their neighbors.

Efficient Parallel Skyline Query Processing for High-Dimensional Data

Given a set of multidimensional data points, skyline queries retrieve those points that are not dominated by any other points in the set. Due to the ubiquitous se of skyline queries, such as in preference-based query answering and decision making, and the large amount of data that these queries have to deal with, enabling their scalable processing is of critical importance. However, there are several outstanding challenges that have not been well addressed. More specifically, in this paper, we are tackling the data straggler and data skew challenges introduced by distributed skyline query processing, as well as the ensuing high computation cost of merging skyline candidates. We thus introduce a new efficient three-phase approach for large scale processing of skyline queries. In the first preprocessing phase, the data is partitioned along the Z-order curve. We utilize a novel data partitioning approach that formulates data partitioning as an optimization problem to minimize the size of intermediate data. In the second phase, each compute node partitions the input data points into disjoint subsets, and then performs the skyline computation on each subset to produce skyline candidates in parallel. In the final phase, we build an index and employ an efficient algorithm to merge the generated skyline candidates. Extensive experiments demonstrate that the proposed skyline algorithm achieves more than one order of magnitude enhancement in performance compared to existing state-of-the-art approaches.

VECTOR AND RASTER DATA STORAGE BASED ON MORTON CODE

Abstract. Even though geomatique is so developed nowadays, the integration of spatial data in vector and raster formats is still a very tricky problem in geographic information system environment. And there is still not a proper way to solve the problem. This article proposes a method to interpret vector data and raster data. In this paper, we saved the image data and building vector data of Guilin University of Technology to Oracle database. Then we use ADO interface to connect database to Visual C++ and convert row and column numbers of raster data and X Y of vector data to Morton code in Visual C++ environment. This method stores vector and raster data to Oracle Database and uses Morton code instead of row and column and X Y to mark the position information of vector and raster data. Using Morton code to mark geographic information enables storage of data make full use of storage space, simultaneous analysis of vector and raster data more efficient and visualization of vector and raster more intuitive. This method is very helpful for some situations that need to analyse or display vector data and raster data at the same time.

Binarized-octree generation for Cartesian adaptive mesh refinement around immersed geometries

We revisit the generation of balanced octrees for adaptive mesh refinement (AMR) of Cartesian domains with immersed complex geometries. In a recent short note (Hasbestan and Senocak, 2017) [42], we showed that the data locality of the Z-order curve in a hashed linear-octree generation method may not be perfect because of potential collisions in the hash table. Building on that observation, we propose a binarized-octree generation method that complies with the Z-order curve exactly. Similar to a hashed linear-octree generation method, we use Morton encoding to index the nodes of an octree, but use a red-black tree in place of the hash table. Red-black tree is a special kind of a binary tree, which we use for insertion and deletion of elements during mesh adaptation. By strictly working with the bitwise representation of an octree, we remove computer hardware limitations on the depth of adaptation on a single processor. Additionally, we introduce a geometry encoding technique for rapidly tagging a solid geometry for mesh refinement. Our results for several geometries with different levels of adaptations show that the binarized-octree generation method outperforms the linear-octree generation method in terms of runtime performance at the expense of only a slight increase in memory usage. The current AMR capability, rebl-AMR, is available as open-source software.

Cache-Oblivious MPI All-to-All Communications Based on Morton Order

Many-core systems with a rapidly increasing number of cores pose a significant challenge to parallel applications to use their complex memory hierarchies efficiently. Many such applications rely on collective communications in performance-critical phases, which become a bottleneck if they are not optimized. We address this issue by proposing cache-oblivious algorithms for MPI_Alltoall, MPI_Allgather, and the MPI neighborhood collectives to exploit the data locality. To implement the cache-oblivious algorithms, we allocate the send and receive buffers on a shared heap and use Morton order to guide the memory copies. Our analysis shows that our algorithm for MPI_Alltoall is asymptotically optimal. We show an extension to our algorithms to minimize the communication distance on NUMA systems while maintaining optimality within each socket. We further demonstrate how the cache-oblivious algorithms can be applied to multi-node machines. Experiments are conducted on different many-core architectures. For MPI_Alltoall, our implementation achieves on average 1.40X speedup over the naive implementation based on shared heap for small and medium block sizes (less than 16 KB) on a Xeon Phi KNC, achieves on average 3.03X speedup over MVAPICH2 on a Xeon E7-8890, and achieves on average 2.23X speedup over MVAPICH2 on a 256-node Xeon E5-2680 cluster for block sizes less than 1 KB.

Coconut

Many modern applications produce massive amounts of data series that need to be analyzed, requiring efficient similarity search operations. However, the state-of-the-art data series indexes that are used for this purpose do not scale well for massive datasets in terms of performance, or storage costs. We pinpoint the problem to the fact that existing summarizations of data series used for indexing cannot be sorted while keeping similar data series close to each other in the sorted order. This leads to two design problems. First, traditional bulk-loading algorithms based on sorting cannot be used. Instead, index construction takes place through slow top-down insertions, which create a non-contiguous index that results in many random I/Os. Second, data series cannot be sorted and split across nodes evenly based on their median value; thus, most leaf nodes are in practice nearly empty. This further slows down query speed and amplifies storage costs. To address these problems, we present Coconut. The first innovation in Coconut is an inverted, sortable data series summarization that organizes data series based on a z-order curve, keeping similar series close to each other in the sorted order. As a result, Coconut is able to use bulk-loading techniques that rely on sorting to quickly build a contiguous index using large sequential disk I/Os. We then explore prefix-based and median-based splitting policies for bottom-up bulk-loading, showing that median-based splitting outperforms the state of the art, ensuring that all nodes are densely populated. Overall, we show analytically and empirically that Coconut dominates the state-of-the-art data series indexes in terms of construction speed, query speed, and storage costs.

Coconut

Many modern applications produce massive amounts of data series that need to be analyzed, requiring efficient similarity search operations. However, the state-of-the-art data series indexes that are used for this purpose do not scale well for massive datasets in terms of performance, or storage costs. We pinpoint the problem to the fact that existing summarizations of data series used for indexing cannot be sorted while keeping similar data series close to each other in the sorted order. This leads to two design problems. First, traditional bulk-loading algorithms based on sorting cannot be used. Instead, index construction takes place through slow top-down insertions, which create a non-contiguous index that results in many random I/Os. Second, data series cannot be sorted and split across nodes evenly based on their median value; thus, most leaf nodes are in practice nearly empty. This further slows down query speed and amplifies storage costs. To address these problems, we present Coconut. The first innovation in Coconut is an inverted, sortable data series summarization that organizes data series based on a z-order curve, keeping similar series close to each other in the sorted order. As a result, Coconut is able to use bulk-loading techniques that rely on sorting to quickly build a contiguous index using large sequential disk I/Os. We then explore prefix-based and median-based splitting policies for bottom-up bulk-loading, showing that median-based splitting outperforms the state of the art, ensuring that all nodes are densely populated. Overall, we show analytically and empirically that Coconut dominates the state-of-the-art data series indexes in terms of construction speed, query speed, and storage costs.

Morton ordering of 2D arrays for efficient access to hierarchical memory

This article investigates the recursive Morton ordering of two-dimensional arrays as an efficient way to access hierarchical memory across a range of heterogeneous computer platforms, ranging from manycore devices, multicore processors, clusters and distributed environments. A brief overview of previous research in this area is given, and algorithms that make use of Morton ordering are described. These are then used to investigate the efficiency of the Morton ordering approach by performance experiments on different processors. In particular, timing results are presented for matrix multiplication, Cholesky factorization and fast Fourier transform algorithms. The use of the Morton ordering approach leads naturally to algorithms that are recursive and exposes parallelism at each level of recursion. Thus, the approach advocated in this talk not only provides convenient and efficient access to hierarchical memory but also provides a basis for exploiting parallelism.

High-efficiency texture coding and synthesis on point-based pear surface

© 2017 IOS Press and the authors. The fruit images on points cloud acquired by the current 3D scanner from field will appear a visible seams, inconvenient data acquisition or taking large space due to unorganized background. We give a SAOW method to cope with the space efficiency and realistic effects of texture synthesis on pear point model. At first, a point-quadtree is proposed to simplify the pear image division. Then, an adaptive multi-granularity morton coding scheme are presented to optimizing the memory space of pear image. At last, weighted oversampling mixing method is mainly focused on texture quality of pear surface. As shown in the experiment results, our adaptive division makes the memory space decline dramatically about 90.7% than non-division and 92.9% than general division respectively; adaptive code scheme helps to reduce the memory to 72.1% of ordinary morton code; weighted oversampling keeps the mixed texture more real and smoothly than current methods.

Parallel BVH Construction using Progressive Hierarchical Refinement

We propose a novel algorithm for construction of bounding volume hierarchies BVHs for multi-core CPU architectures. The algorithm constructs the BVH by a divisive top-down approach using a progressively refined cut of an existing auxiliary BVH. We propose a new strategy for refining the cut that significantly reduces the workload of individual steps of BVH construction. Additionally, we propose a new method for integrating spatial splits into the BVH construction algorithm. The auxiliary BVH is constructed using a very fast method such as LBVH based on Morton codes. We show that the method provides a very good trade-off between the build time and ray tracing performance. We evaluated the method within the Embree ray tracing framework and show that it compares favorably with the Embree BVH builders regarding build time while maintaining comparable ray tracing speed.

VLSI Architecture of Full‐Search Variable‐Block‐Size Motion Estimation for HEVC Video Encoding

Motion estimation (ME) is the most computationally intensive task in video encoding. This study proposes a full‐search variable‐block‐size ME for the high‐efficiency video coding or H.265 specification. The proposed method reduces memory requirements to a large extent by following a Morton order for data reading and a sum of absolute differences reuse strategy. The data bandwidth demand is also diminished by broadcasting data into multiple processing elements. This ME accelerator supports variable‐block‐size prediction blocks ranging from to , and is reconfigurable in various search ranges for a trade‐off between performance and area. The proposed method for very‐large‐scale integration (VLSI) architecture is synthesized with 32 nm technology, and is capable of real‐time encoding of ultra‐high‐definition (4K‐UHD, at 30 Hz) video with a search range of 64 pixels in both horizontal and vertical directions, operating at a frequency of 282 MHz.

POSTER

In the many-core era, the performance of MPI collectives is more dependent on the intra-node communication component. However, the communication algorithms generally inherit from the inter-node version and ignore the cache complexity. We propose cache-oblivious algorithms for MPI all-to-all operations, in which data blocks are copied into the receive buffers in Morton order to exploit data locality. Experimental results on different many-core architectures show that our cache-oblivious implementations significantly outperform the naive implementations based on shared heap and the highly optimized MPI libraries.

Texture organisation and mapping on Citrus sinensis point cloud

In light of the current problems including coarseness, visible cracks, difficult data organisation, and the expensive memory requirements of the current texture methods, this paper mainly focuses on efficient organisation, linearised memory compression and seamless texture mapping between scanned Citrus sinensis images and point cloud information. Position and colour gradient based top-down splitting is proposed to simplify and organise the texture as texel descriptors to avoid both over-simplification and under-simplification. A Quadtree Morton and Z-order based linearised coding strategy is presented to compress the memory space of our texel descriptor based texture. A Gaussian Markov random field scheme was designed to smooth the `cracks' between neighbouring texels. The simulated results on eight Citrus sinensises show that our simplification method reduces the texture memory requirements by 81.3 % over the original image, and 50 % over conventional simplification. The compression scheme also showed a 61.7 % improvement over the ordinary Morton code. Finally, the Gaussian Markov random field scheme makes the texture mapping smoother in comparison with other methods.

A hierarchical factorization method for efficient radiosity calculations

The radiosity problem can be expressed as a linear system, where light interactions between patches of the scene are considered. Its resolution has been one of the main subjects in Computer Graphics, which has led to the development of methods focused on different goals. For instance, in inverse lighting problems, it is convenient to solve the radiosity equation thousands of times for static geometries. Also, this calculation needs to consider many (or infinite) light bounces to achieve accurate global illumination results. Several methods have been developed to solve the linear system by finding approximations or other representations of the radiosity matrix, because the full storage of this matrix is memory demanding. Some examples are hierarchical radiosity, progressive refinement approaches, or wavelet radiosity, which may become slow for many bounces. Recently, new direct methods have been developed based on matrix factorization.This paper introduces a novel and efficient error-bounded factorization method based on the use of multiple singular value decompositions and the Z-order curve to sort the patches of the model. This technique accelerates the factorization of in-core matrices, and allows to work with out-of-core matrices passing only one time over them. Using this method, the inverse of the radiosity matrix can be efficiently approximated, reducing the memory and time resources needed to compute radiosity with infinite bounces. In the experimental analysis, the presented method is applied to scenes up to 163K patches. After a precomputation stage, it is used to solve the radiosity problem for fixed geometries at interactive times.

IEEE 754 부동 소수점 32비트 float 변수의 Morton Code 변환 분석

GPU 기반 병렬처리에서 대규모 데이터의 인접 정보 검색(nearest neighbor search)에서 Morton code의 역할이 점점 더 중요하게 부각되고 있으며 그 응용 사례도 점차 증가하고 있다. 본 논문에서는 Tero Karras가 제안한 float 형 변수에 기반한 <TEX>$[0,1</TEX><TEX>]</TEX><TEX>^3$</TEX> 공간 내의 3차원 기하 정보를 32비트 unsigned int형 Morton code로 변경하는 기존의 방법을 논의하고 그 기하학적인 의미를 분석함으로써, 보다 높은 해상도를 구현할 수 있는 64비트 unsigned long long형의 Morton code 변환 알고리듬을 제안한다. 제안하는 알고리듬은 GPU에서 구현되었을 때 CPU에서 실행하는 것보다 약 1000배 수준의 성능 향상을 달성한다. Morton codes play important roles in many parallel GPU applications for the nearest neighbor (NN) search in huge data and queries with its applications growing. This paper discusses and analyzes the meaning of Tero Karras's 32-bit 'unsigned int' Morton code algorithm for three-dimensional spatial information in <TEX>$[0,1</TEX><TEX>]</TEX><TEX>^3$</TEX> and its geometric implications. Based on this, this paper proposes 64-bit 'unsigned long long' version of Morton code and compares the results in both CPU vs. GPU and 32-bit vs. 64-bit versions. The proposed GPU algorithm runs around 1000 times faster than the CPU version.

PARALLEL PROCESSING OF BIG POINT CLOUDS USING Z-ORDER-BASED PARTITIONING

As laser scanning technology improves and costs are coming down, the amount of point cloud data being generated can be prohibitively difficult and expensive to process on a single machine. This data explosion is not only limited to point cloud data. Voluminous amounts of high-dimensionality and quickly accumulating data, collectively known as Big Data, such as those generated by social media, Internet of Things devices and commercial transactions, are becoming more prevalent as well. New computing paradigms and frameworks are being developed to efficiently handle the processing of Big Data, many of which utilize a compute cluster composed of several commodity grade machines to process chunks of data in parallel. &amp;lt;br&amp;gt;&amp;lt;br&amp;gt; A central concept in many of these frameworks is data locality. By its nature, Big Data is large enough that the entire dataset would not fit on the memory and hard drives of a single node hence replicating the entire dataset to each worker node is impractical. The data must then be partitioned across worker nodes in a manner that minimises data transfer across the network. This is a challenge for point cloud data because there exist different ways to partition data and they may require data transfer. &amp;lt;br&amp;gt;&amp;lt;br&amp;gt; We propose a partitioning based on &amp;lt;i&amp;gt;Z&amp;lt;/i&amp;gt;-order which is a form of locality-sensitive hashing. The &amp;lt;i&amp;gt;Z&amp;lt;/i&amp;gt;-order or Morton code is computed by dividing each dimension to form a grid then interleaving the binary representation of each dimension. For example, the &amp;lt;i&amp;gt;Z&amp;lt;/i&amp;gt;-order code for the grid square with coordinates (&amp;lt;i&amp;gt;x&amp;lt;/i&amp;gt; = 1 = 01&amp;lt;sub&amp;gt;2&amp;lt;/sub&amp;gt;, &amp;lt;i&amp;gt;y&amp;lt;/i&amp;gt; = 3 = 11&amp;lt;sub&amp;gt;2&amp;lt;/sub&amp;gt;) is 1011&amp;lt;sub&amp;gt;2&amp;lt;/sub&amp;gt; = 11. The number of points in each partition is controlled by the number of bits per dimension: the more bits, the fewer the points. The number of bits per dimension also controls the level of detail with more bits yielding finer partitioning. We present this partitioning method by implementing it on Apache Spark and investigating how different parameters affect the accuracy and running time of the &amp;lt;i&amp;gt;k&amp;lt;/i&amp;gt; nearest neighbour algorithm for a hemispherical and a triangular wave point cloud.

PARALLEL PROCESSING OF BIG POINT CLOUDS USING Z-ORDER-BASED PARTITIONING

Abstract. As laser scanning technology improves and costs are coming down, the amount of point cloud data being generated can be prohibitively difficult and expensive to process on a single machine. This data explosion is not only limited to point cloud data. Voluminous amounts of high-dimensionality and quickly accumulating data, collectively known as Big Data, such as those generated by social media, Internet of Things devices and commercial transactions, are becoming more prevalent as well. New computing paradigms and frameworks are being developed to efficiently handle the processing of Big Data, many of which utilize a compute cluster composed of several commodity grade machines to process chunks of data in parallel. A central concept in many of these frameworks is data locality. By its nature, Big Data is large enough that the entire dataset would not fit on the memory and hard drives of a single node hence replicating the entire dataset to each worker node is impractical. The data must then be partitioned across worker nodes in a manner that minimises data transfer across the network. This is a challenge for point cloud data because there exist different ways to partition data and they may require data transfer. We propose a partitioning based on Z-order which is a form of locality-sensitive hashing. The Z-order or Morton code is computed by dividing each dimension to form a grid then interleaving the binary representation of each dimension. For example, the Z-order code for the grid square with coordinates (x = 1 = 012, y = 3 = 112) is 10112 = 11. The number of points in each partition is controlled by the number of bits per dimension: the more bits, the fewer the points. The number of bits per dimension also controls the level of detail with more bits yielding finer partitioning. We present this partitioning method by implementing it on Apache Spark and investigating how different parameters affect the accuracy and running time of the k nearest neighbour algorithm for a hemispherical and a triangular wave point cloud.

Efficient Implementation of Smoothed Particle Hydrodynamics (SPH) with Plane Sweep Algorithm

AbstractNeighbour search (NS) is the core of any implementations of smoothed particle hydrodynamics (SPH). In this paper,we present an efficientneighbour search method based on the plane sweep (PW) algorithm withNbeing the number of SPH particles. The resulting method, dubbed the PWNS method, is totally independent of grids (i.e., purely meshfree) and capable of treating variable smoothing length, arbitrary particle distribution and heterogenous kernels. Several state-of-the-art data structures and algorithms, e.g., the segment tree and the Morton code, are optimized and implemented. By simply allowingmultiple lines to sweep the SPH particles simultaneously from different initial positions, a parallelization of the PWNS method with satisfactory speedup and load-balancing can be easily achieved. That is, the PWNS SPH solver has a great potential for large scale fluid dynamics simulations.