Two-level Partitioning Research Articles

HLC2: a highly efficient cross-matching framework for large astronomical catalogues on heterogeneous computing environments

ABSTRACT Cross-matching operation, which is to find corresponding data for the same celestial object or region from multiple catalogues, is indispensable to astronomical data analysis and research. Due to the large amount of astronomical catalogues generated by the ongoing and next-generation large-scale sky surveys, the time complexity of the cross-matching is increasing dramatically. Heterogeneous computing environments provide a theoretical possibility to accelerate the cross-matching, but the performance advantages of heterogeneous computing resources have not been fully utilized. To meet the challenge of cross-matching for substantial increasing amount of astronomical observation data, this paper proposes Heterogeneous-computing-enabled Large Catalogue Cross-matcher (HLC2), a high-performance cross-matching framework based on spherical position deviation on CPU-GPU heterogeneous computing platforms. It supports scalable and flexible cross-matching and can be directly applied to the fusion of large astronomical catalogues from survey missions and astronomical data centres. A performance estimation model is proposed to locate the performance bottlenecks and guide the optimizations. A two-level partitioning strategy is designed to generate an optimized data placement according to the positions of celestial objects to increase throughput. To make HLC2 a more adaptive solution, the architecture-aware task splitting, thread parallelization, and concurrent scheduling strategies are designed and integrated. Moreover, a novel quad-direction strategy is proposed for the boundary problem to effectively balance performance and completeness. We have experimentally evaluated HLC2 using public released catalogue data. Experiments demonstrate that HLC2 scales well on different sizes of catalogues and the cross-matching speed is significantly improved compared to the state-of-the-art cross-matchers.

Answering why-not questions on top-[formula omitted] augmented spatial keyword queries

The top-k augmented spatial keyword query (TkASKQ) retrieves k objects with the highest scores based on a scoring function, which considers spatial proximity, textual similarity and attribute matching simultaneously. As far as we know, no work has been conducted on answering why-not questions on TkASKQ queries (WTkASKQ). This paper takes the first step to address WTkASKQ queries by adopting a Query Refinement model. Specifically, we propose a hybrid indexing structure, AkC, which adopts a two-level partitioning scheme, to efficiently organize the textual, attribute, and spatial information of objects. Based on AkC, several filtering strategies are proposed to prune unqualified objects for query processing. To limit the number of refined queries to be explored, we construct new refined queries by sequentially extracting new keywords and attribute–value pairs from missing objects and adding them to the original keyword and attribute–value sets, respectively, so as to efficiently obtain the best refined query with minimal modification cost. In addition, we discuss the applicability of the methods in handling why-not questions on augmented regional queries, ordinary top-k SKQ queries and complex scoring queries. Experimental result shows that our AkC-based method has higher query efficiency compared with other baseline methods.

Service-Aware Two-Level Partitioning for Machine Learning-Based Network Intrusion Detection With High Performance and High Scalability

A network intrusion detection system (NIDS) is an important technology for cyber security. Recently, machine learning based NIDSs are being actively researched as various machine learning techniques are proposed. However, existing NIDSs have limitation in terms of generality because they have been designed based on specific characteristics obtained from analyzing some partial datasets. Moreover, in reality, the NIDS datasets have a significantly imbalanced ratio between normal and abnormal data. It causes the minority class problem, which needs to be addressed for developing robust and reliable NIDSs working in various environments. This paper proposes a novel technique using service-aware dataset partitioning, which provides high scalability to handle huge and rapidly growing network data flexibly, and helps the classifier to improve the classification performance in terms of accuracy and speed. We evaluated our approach with the Kyoto2016 dataset, which is a well-known dataset for highly imbalanced data, using various classification algorithms and parameters for achieving the best performance and compared it with existing state-of-the-art approaches. Our experimental results indicated that our approach can classify network traffics rapidly and accurately with huge imbalanced datasets. We conclude that it can relieve serious existing issues of imbalanced datasets for modern machine learning based NIDS solutions.

Algorithms for processing the group K nearest-neighbor query on distributed frameworks

Given two datasets of points (called Query and Training), the Group (K) Nearest-Neighbor (GKNN) query retrieves (K) points of the Training with the smallest sum of distances to every point of the Query. This spatial query has been studied during the recent years and several performance improving techniques and pruning heuristics have been proposed. In previous work, we presented the first MapReduce algorithm, consisting of alternating local and parallel phases, which can be used to effectively process the GKNN query when the Query fits in memory, while the Training one belongs to the Big Data category. In this paper, we present a significantly improved algorithm that incorporates a new high-performance refining method, a fast way to calculate distance sums for pruning purposes and several other minor coding and algorithmic improvements. Moreover, we transform this algorithm (which has been implemented in the Hadoop framework) to SpatialHadoop (a popular distributed framework that is dedicated to spatial processing), using a novel two-level partitioning method. Using real world and synthetic datasets, we also present a thorough experimental study of the Hadoop and SpatialHadoop versions of the algorithm, including a backstage analysis of the algorithm’s performance, using metrics that highlight its internal functioning. Finally, we present an experimental comparison of the Hadoop, the SpatialHadoop versions and the version of our previous work, showing that the improved versions are the big winners, with the SpatialHadoop one being faster than its Hadoop counterpart.

Multilevel parallelization for simulating compressible turbulent flows on most kinds of hybrid supercomputers

The paper describes a multilevel MPI+OpenMP+OpenCL parallelization approach that provides complete portability across a wide range of hybrid supercomputer architectures. A parallel CFD algorithm for heterogeneous computing of turbulent flows is presented. It simulates the compressible Navier–Stokes equations using a cell-centered finite-volume method with polynomial reconstruction on unstructured hybrid meshes. A two-level partitioning is used for the workload distribution among computing devices of hybrid nodes. The overlap of communications and computations hides the data transfer expenses. The scalability is tested on various HPC systems including a fat node with 8 GPUs and supercomputers using up to 320 GPUs. Comparison of performance is presented for multicore CPUs, Intel Xeon Phi, various GPUs of AMD and NVIDIA. The heterogeneous execution using CPUs and GPUs is studied in detail.

Augmented keyword search on spatial entity databases

In this paper, we propose a new type of query that augments the spatial keyword search with an additional boolean expression constraint. The query is issued against a corpus of structured or semi-structured spatial entities and is very useful in applications like mobile search and targeted location-aware advertising. We devise three types of indexing and filtering strategies. First, we utilize the hybrid $$\hbox {IR}^2$$ -tree and propose a novel hashing scheme for efficient pruning. Second, we propose an inverted index-based solution, named BE-Inv, that is more cache concious and exhibits great pruning power for boolean expression matching. Our third method, named SKB-Inv, adopts a novel two-level partitioning scheme to organize the spatial entities into inverted lists and effectively facilitate the pruning in the spatial, textual, and boolean expression dimensions. In addition, we propose an adaptive query processing strategy that takes into account the selectivity of query keywords and predicates for early termination. We conduct our experiments using two real datasets with 3.5 million Foursquare venues and 50 million Twitter geo-profiles. The results show that the methods based on inverted index are superior to the hybrid $$\hbox {IR}^2$$ -tree; and SKB-Inv achieves the best performance.

Processing Long Queries Against Short Text

Many real applications in real-time news stream advertising call for efficient processing of long queries against short text. In such applications, dynamic news feeds are regarded as queries to match against an advertisement (ad) database for retrieving the k most relevant ads. The existing approaches to keyword retrieval cannot work well in this search scenario when queries are triggered at a very high frequency. To address the problem, we introduce new techniques to significantly improve search performance. First, we devise a two-level partitioning for tight upper bound estimation and a lazy evaluation scheme to delay full evaluation of unpromising candidates, which can bring three to four times performance boosting in a database with 7 million ads. Second, we propose a novel rank-aware block-oriented inverted index to further improve performance. In this index scheme, each entry in an inverted list is assigned a rank according to its importance in the ad. Then, we introduce a block-at-a-time search strategy based on the index scheme to support a much tighter upper bound estimation and a very early termination. We have conducted experiments with real datasets, and the results show that the rank-aware method can further improve performance by an order of magnitude.

Power System Dynamic Simulations Using a Parallel Two-Level Schur-Complement Decomposition

As the need for faster power system dynamic simulations increases, it is essential to develop new algorithms that exploit parallel computing to accelerate those simulations. This paper proposes a parallel algorithm based on a two-level, Schur-complement-based, domain decomposition method. The two-level partitioning provides high parallelization potential (coarse- and fine-grained). In addition, due to the Schur-complement approach used to update the sub-domain interface variables, the algorithm exhibits high global convergence rate. Finally, it provides significant numerical and computational acceleration. The algorithm is implemented using the shared-memory parallel programming model, targeting inexpensive multi-core machines. Its performance is reported on a real system as well as on a large test system combining transmission and distribution networks.

On scalable RDFS reasoning using a hybrid approach

Reasoning is a vital ability for semantic web applications since they aim to understand and interpret the data on the World Wide Web. However, reasoning of large data sets is one of the challenges facing semantic web applications. In this paper, we present new approaches for scalable Resource Description Framework Schema (RDFS) reasoning. Our RDFS specific term-based partitioning algorithm determines required schema elements for each data partition while eliminating the data partitions that will not produce any inferences. With the two-level partitioning approach, we are able to carry out reasoning with limited resources. In our hybrid approach, we integrate two previously mentioned methods to benefit from the advantages of both. In the experimental tests we achieve linear speedups for reasoning times with the proposed hybrid approach. These algorithms and methods presented in the paper enable RDFS-level reasoning of large data sets with limited resources, and they together build up a scalable distributed reasoning approach.

Improving the parallel efficiency of large-scale structural dynamic analysis using a hierarchical approach

In order to improve the parallel efficiency of large-scale structural dynamic analysis, a hierarchical approach adapted to the hardware topology of multi-core clusters is proposed. The hierarchical approach is constructed based on the strategies of two-level partitioning and two-level condensation. The data for parallel computing is first prepared through two-level partitioning to guarantee the load balancing within and across nodes. Then during the analysis of each time step, the convergence rate of interface problem is significantly improved by further reducing its size with two-level condensation. Furthermore, the communication overheads are considerably reduced by separating the intra-node and inter-node communications and minimizing the inter-node communication. Numerical experiments conducted on Dawning-5000A supercomputer indicate that the hierarchical approach was superior in performance compared with the conventional Newmark algorithm based on the domain decomposition method.

Optimal loop scheduling for hiding memory latency based on two-level partitioning and prefetching

The large latency of memory accesses in modern computers is a key obstacle in achieving high processor utilization. As a result, a variety of techniques have been devised to hide this latency. These techniques range from cache hierarchies to various prefetching and memory management techniques for manipulating the data present in the caches. In DSP applications, the existence of large numbers of uniform nested loops makes the issue of loop scheduling very important. In this paper, we propose a new memory management technique that can be applied to computer architectures with three levels of memory, which is the scheme generally adopted in contemporary computer architectures. This technique takes advantage of access pattern information that is available at compile time by prefetching certain data elements from the higher level memory before they are explicitly requested by the lower level memory or CPU. It also maintains certain data for a period of time to prevent unnecessary data swapping. In order to take better advantage of the locality of references present in these loop structures, our technique introduces a new approach to memory management by partitioning it and reducing execution to each partition so that data locality is much improved compared with the usual pattern. These combined approaches-using a new set of memory instructions as well as partitioning the memory-lead to improvements in average execution times of approximately 35% over the one-level partition algorithm and more than 80% over list scheduling and hardware prefetching.

Improving classification performance using fuzzy MLP and two-level selective partitioning of the feature space

A fuzzy MLP model, developed by one of the authors, is used for obtaining selective two-level partitioning of the feature space in order to improve its classification performance. The model can handle uncertainty and/or impreciseness in the input as well as the output. The input to the network is modelled in terms of linguistic pi-sets whose centres and radii along the feature axes in each partition are generated automatically from the distribution of the training data. The performance of the model at the end of the first stage is used as a criterion for guiding the selection of the appropriate partition to be subdivided at the second stage, in order to improve the effectiveness of the model. A comparative study of the performance of the two-level technique with other methods, viz., the conventional MLP, linear discriminant analysis and the k-nearest neighbours algorithms, is also provided to demonstrate its superiority.

Optimization by iterative improvement: an experimental evaluation on two-way partitioning

Recently, Johnson et al. [1989] presented an excellent comparison of simulated annealing and Kernighan-Lin algorithms. However, their test beds were limited to random and geometric graphs. We present a complete evaluation by adding real circuitry into the test beds. A two-level partitioning algorithm called the primal-dual algorithm is also incorporated for comparison. We show that at least 500 runs are necessary to demonstrate the performance of the Fiduccia-Mattheyses algorithm, whereas traditional way of evaluation tends to underestimate. Nevertheless, our new results show that for two-way partitioning on real circuits, the primal-dual algorithm is, in general, a better choice than both the Fiduccia-Mattheyses algorithm and the simulated annealing algorithm. This conclusion is more likely to hold when the primal-dual algorithm is switched to a simpler mode.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Two-level partitioning algorithm with stable performance

By using the concept of 'divide-and-conquer' the complexity of a large problem can be reduced efficiently. Therefore, logic partitioning plays an important role in many aspects of VLSI design. The ratio cut objective function has attracted much attention because it coordinates the two traditional goals of logic partitioning, mincut and equipartition, automatically. How to reduce the total trials needed to run and maintain the quality of solutions is very important to the iterative improvement-based partitioning algorithms. In this paper we present a ratio cut partitioning algorithm which provides stable results without executing the algorithm over many trials. The proposed algorithm also employs a preprocessing circuit-clustering procedure to improve the performance. Our partitioning heuristics were tested on some industry benchmark suites. Good experimental results, which are better than those obtained by RCut1.0 and EIG1-IG algorithms, have been observed.