Comparison-based Algorithms Research Articles

A new approach to Mergesort algorithm: Divide smart and conquer

It is well known that comparison-based algorithms cannot run faster than O(nlogn). Therefore the running times of Mergesort, Heapsort, and Quicksort algorithms are asymptotically optimal. As well as asymptotic running times, how fast sorting algorithms work in practice has gained importance lately. One algorithm is preferred, if it is slightly more efficient than another, due to the rapid increase in data generation. In this study, we consider the Mergesort algorithm with a different approach. Unlike the Mergesort algorithm, we do not perform unconditional division. Instead, we divide the input array into ascending and descending sub-arrays. Then, we merge the sub-arrays obtained as a result of the division by slightly modifying the classical Merge function. As a result of these two operations, we obtain three new sorting algorithms depending on how we get ascending and descending sub-arrays. Among these three algorithms, the third one, which is our main proposed algorithm, is more important both theoretically and practically. In this algorithm, we divide the input array into alternately ascending and descending sub-arrays. Moreover, the terms of the sub-arrays do not have to be consecutive terms of the input array. The asymptotic running time of our proposed algorithm is O(nlogn) in the worst case and O(n) in the best case. In practice, the proposed algorithm performs better than the classical Mergesort algorithm in arrays with Gaussian and Uniform distributions of various sizes. In an array of four million, it provided a 17% improvement in the Gaussian distribution compared to the classical Mergesort algorithm. In addition, it provided a 29% improvement in the Uniform distribution of the same size. The proposed algorithm performs also much better than traditional sorting algorithms in real data collected from various sources.

A sorting algorithm based on ordered block insertions

Among the comparison-based algorithms, insertionsort is recognized as one of the fastest methods to sort relatively small data sets, or when the elements are relatively ordered. However, due to not offering good asymptotic complexity in its runtime, it performs very poorly both in the worst case and in the average case for most large data collections. In this article we offer a new sorting algorithm based on orderer block insertions with worst case optimal time. At the cost of an additional memory space of nk words, for any constant k, our algorithm is able to easily transform it into an in-place algorithm with a time of o(n2) in the worst case. Empirically, our method outperforms insertionsort in all the different cases tested, even for small input collections. Furthermore, our experiments show that, for small or large datasets from either of the two main probability distributions—Uniform and Normal; our algorithms also outperform any traditional method like quicksort, mergesort or heapsort, and even better than the efficient hybrid algorithm introsort —std::sort() method provided by the GNU C++ Standard Library.

Engineering In-place (Shared-memory) Sorting Algorithms

We present new sequential and parallel sorting algorithms that now represent the fastest known techniques for a wide range of input sizes, input distributions, data types, and machines. Somewhat surprisingly, part of the speed advantage is due to the additional feature of the algorithms to work in-place, i.e., they do not need a significant amount of space beyond the input array. Previously, the in-place feature often implied performance penalties. Our main algorithmic contribution is a blockwise approach to in-place data distribution that is provably cache-efficient. We also parallelize this approach taking dynamic load balancing and memory locality into account. Our new comparison-based algorithm In-place Parallel Super Scalar Samplesort ( IPS 4 o ) , combines this technique with branchless decision trees. By taking cases with many equal elements into account and by adapting the distribution degree dynamically, we obtain a highly robust algorithm that outperforms the best previous in-place parallel comparison-based sorting algorithms by almost a factor of three. That algorithm also outperforms the best comparison-based competitors regardless of whether we consider in-place or not in-place, parallel or sequential settings. Another surprising result is that IPS 4 o even outperforms the best (in-place or not in-place) integer sorting algorithms in a wide range of situations. In many of the remaining cases (often involving near-uniform input distributions, small keys, or a sequential setting), our new In-place Parallel Super Scalar Radix Sort ( IPS 2 Ra ) turns out to be the best algorithm. Claims to have the – in some sense – “best” sorting algorithm can be found in many papers which cannot all be true. Therefore, we base our conclusions on an extensive experimental study involving a large part of the cross product of 21 state-of-the-art sorting codes, 6 data types, 10 input distributions, 4 machines, 4 memory allocation strategies, and input sizes varying over 7 orders of magnitude. This confirms the claims made about the robust performance of our algorithms while revealing major performance problems in many competitors outside the concrete set of measurements reported in the associated publications. This is particularly true for integer sorting algorithms giving one reason to prefer comparison-based algorithms for robust general-purpose sorting.

An ODE method to prove the geometric convergence of adaptive stochastic algorithms

We consider stochastic algorithms derived from methods for solving deterministic optimization problems, especially comparison-based algorithms derived from stochastic approximation algorithms with a constant step-size. We develop a methodology for proving geometric convergence of the parameter sequence {θn}n⩾0 of such algorithms. We employ the ordinary differential equation (ODE) method, which relates a stochastic algorithm to its mean ODE, along with a Lyapunov-like function Ψ such that the geometric convergence of Ψ(θn) implies – in the case of an optimization algorithm – the geometric convergence of the expected distance between the optimum and the search point generated by the algorithm. We provide two sufficient conditions for Ψ(θn) to decrease at a geometric rate: Ψ should decrease “exponentially” along the solution to the mean ODE, and the deviation between the stochastic algorithm and the ODE solution (measured by Ψ) should be bounded by Ψ(θn) times a constant. We also provide practical conditions under which the two sufficient conditions may be verified easily without knowing the solution of the mean ODE. Our results are any-time bounds on Ψ(θn), so we can deduce not only the asymptotic upper bound on the convergence rate, but also the first hitting time of the algorithm. The main results are applied to a comparison-based stochastic algorithm with a constant step-size for optimization on continuous domains.

Entropy conservation for comparison-based algorithms

Comparison-based algorithms are algorithms for which the execution of each operation is solely based on the outcome of a series of comparisons between elements [5]. Examples include most sorting algorithms, search algorithms, and more general algorithms such as Heapify which constructs a heap data structure from an input list [5]. Comparison-based computations can be naturally represented via the following computational model[12]: (a) model data structures as partially-ordered finite sets; (b) model data on these by topological sorts (aka linear extensions); (c) considering computation states as finite multisets of such data; (d) represent computations by their induced transformations on states.In this view, an abstract specification of a sorting algorithm has input state given by any possible permutation of a finite set of elements (represented, according to (a) and (b), by a discrete partially-ordered set together with its topological sorts given by all permutations) and output state a sorted list of elements (represented, again according to (a) and (b), by a linearly-ordered finite set with its unique topological sort).Entropy is a measure of “randomness” or “disorder.” Based on the computational model we introduce an entropy conservation result for comparison-based sorting algorithms: “quantitative order gained is proportional to positional order lost.” Intuitively, the result bears some relation to the messy office argument advocating a chaotic office where nothing is in the right place yet each item's place is known to the owner, over the case where each item is stored in the right order and yet the owner can no longer locate the items. Formally, we generalise the result to the class of data structures representable via series-parallel partial orders–a well-known computationally tractable class [6,7]. The resulting “denotational” version of entropy conservation will be extended in follow-up work to an “operational” version for a core part of our computational model.

Comparison-Based Algorithms for One-Dimensional Stochastic Convex Optimization

Stochastic optimization finds a wide range of applications in operations research and management science. However, existing stochastic optimization techniques usually require the information of random samples (e.g., demands in the newsvendor problem) or the objective values at the sampled points (e.g., the lost sales cost), which might not be available in practice. In this paper, we consider a new setup for stochastic optimization, in which the decision maker has access to only comparative information between a random sample and two chosen decision points in each iteration. We propose a comparison-based algorithm (CBA) to solve such problems in one dimension with convex objective functions. Particularly, the CBA properly chooses the two points in each iteration and constructs an unbiased gradient estimate for the original problem. We show that the CBA achieves the same convergence rate as the optimal stochastic gradient methods (with the samples observed). We also consider extensions of our approach to multidimensional quadratic problems as well as problems with nonconvex objective functions. Numerical experiments show that the CBA performs well in test problems.

On Constant Factors in Comparison-Based Geometric Algorithms and Data Structures

Many standard problems in computational geometry have been solved asymptotically optimally as far as comparison-based algorithms are concerned, but there has been little work focusing on improving the constant factors hidden in big-Oh bounds on the number of comparisons needed. In this paper, we consider orthogonal-type problems and present a number of results that achieve optimality in the constant factors of the leading terms, includingan algorithm for the 2D maxima problem that uses $$n\lg h + O(n\sqrt{\lg h})$$nlgh+O(nlgh) comparisons, where $$h$$h denotes the output size; a randomized algorithm for the 3D maxima problem that uses $$n\lg h + O(n\lg ^{2/3} h)$$nlgh+O(nlg2/3h) expected number of comparisons; a randomized algorithm for detecting intersections among a set of orthogonal line segments that uses $$n\lg n + O(n\sqrt{\lg n})$$nlgn+O(nlgn) expected number of comparisons; a data structure for point location among 3D disjoint axis-parallel boxes that can answer queries in $$(3/2)\lg n + O(\lg \lg n)$$(3/2)lgn+O(lglgn) comparisons; a data structure for point location in a 3D box subdivision that can answer queries in $$(4/3)\lg n + O(\sqrt{\lg n})$$(4/3)lgn+O(lgn) comparisons. Some of the results can be adapted to solve nonorthogonal problems, such as 2D convex hulls and general line segment intersection. Our algorithms and data structures use a variety of techniques, including Seidel and Adamy's planar point location method, weighted binary search, and height-optimal BSP trees.

GPUMemSort: A High Performance Graphics Co-processors Sorting Algorithm for Large Scale In-Memory Data

In this paper, we present a GPU-based sorting algorithm, GPUMemSort, which achieves high performance in sorting large-scale in-memory data by take advantage of GPU processors. It consists of two algorithms: an in-core algorithm, which is responsible for sorting data in GPU global memory efficiently, and an out-of-core algorithm, which is responsible for dividing large-scale data into multiple chunks that fit GPU global memory. GPUMemSort is implemented based on NVIDIA’s CUDA framework and some critical and detailed optimization methods are also presented. The tests of different algorithms have been run on multiple data sets. The experimental results show that our in-core sorting can outperform other comparison-based algorithms and GPUMemSort is highly effective in sorting large-scale inmemory data.

Randomized minimum spanning tree algorithms using exponentially fewer random bits

For many fundamental problems there exist randomized algorithms that are asymptotically optimal and are superior to the best-known deterministic algorithm. Among these are the minimum spanning tree (MST) problem, the MST sensitivity analysis problem, the parallel connected components and parallel minimum spanning tree problems, and the local sorting and set maxima problems. (For the first two problems there are provably optimal deterministic algorithms with unknown, and possibly superlinear, running times.) One downside of the randomized methods for solving these problems is that they use a number of random bits linear in the size of input. In this article we develop some general methods for reducing exponentially the consumption of random bits in comparison-based algorithms. In some cases we are able to reduce the number of random bits from linear to nearly constant, without affecting the expected running time. Most of our results are obtained by adjusting or reorganizing existing randomized algorithms to work well with a pairwise or O (1)-wise independent sampler. The prominent exception, and the main focus of this article, is a linear-time randomized minimum spanning tree algorithm that is not derived from the well-known Karger-Klein-Tarjan algorithm. In many ways it resembles more closely the deterministic minimum spanning tree algorithms based on soft heaps. Further, using our algorithm as a guide, we present a unified view of the existing “nongreedy” minimum spanning tree algorithms. Concepts from the Karger-Klein-Tarjan algorithm, such as F -lightness, MST verification, and sampled graphs, are related to the concepts of edge corruption, subgraph contractibility, and soft heaps, which are the basis of the deterministic MST algorithms of Chazelle and Pettie-Ramachandran.

Comparison-Based Algorithms Are Robust and Randomized Algorithms Are Anytime

Randomized search heuristics (e.g., evolutionary algorithms, simulated annealing etc.) are very appealing to practitioners, they are easy to implement and usually provide good performance. The theoretical analysis of these algorithms usually focuses on convergence rates. This paper presents a mathematical study of randomized search heuristics which use comparison based selection mechanism. The two main results are that comparison-based algorithms are the best algorithms for some robustness criteria and that introducing randomness in the choice of offspring improves the anytime behavior of the algorithm. An original Estimation of Distribution Algorithm combining both results is proposed and successfully experimented.

A simplified derivation of timing complexity lower bounds for sorting by comparisons

We present a simplified derivation of the fact that the complexity-theoretic lower bound of comparison-based sorting algorithms, both for the worst-case and for the average-case time measure, is Ω(...

Lower and upper bounds on obtaining history independence

History independent data structures, presented by Micciancio, are data structures that possess a strong security property: even if an intruder manages to get a copy of the data structure, the memory layout of the structure yields no additional information on the history of operations applied on the structure beyond the information obtainable from the content itself. Naor and Teague proposed a stronger notion of history independence in which the intruder may break into the system several times without being noticed and still obtain no additional information from reading the memory layout of the data structure. An open question posed by Naor and Teague is whether these two notions are equally hard to obtain. In this paper we provide a separation between the two requirements for comparison-based algorithms. We show very strong lower bounds for obtaining the stronger notion of history independence for a large class of data structures, including, for example, the heap and the queue abstract data structures. We also provide complementary upper bounds showing that the heap abstract data structure may be made weakly history independent in the comparison based model without incurring any additional (asymptotic) cost on any of its operations. (A similar result is easy for the queue.) Thus, we obtain the first separation between the two notions of history independence. The gap we obtain is exponential: some operations may be executed in logarithmic time (or even in constant time) with the weaker definition, but require linear time with the stronger definition.

Optimal Time-Space Trade-Offs for Non-Comparison-Based Sorting

<p>We study the fundamental problem of sorting n integers of w bits on a unit-cost RAM with word size w, and in particular consider the time-space trade-off (product of time and space in bits) for this problem. For comparison-based algorithms, the time-space complexity is known to be Theta(n^2). A result of Beame shows that the lower bound also holds for non-comparison-based algorithms, but no algorithm has met this for time below the comparison-based <br />Omega(n lg n) lower bound. </p><p>We show that if sorting within some time bound T~ is possible, then time T = O(T~ + n lg* n) can be achieved with high probability using space S = O(n^2/T + w), which is optimal. Given a deterministic priority queue using amortized<br />time t(n) per operation and space n^O(1), we provide a deterministic<br />algorithm sorting in time T = O(n (t(n) + lg* n)) with S = O(n^2/T+w). Both results require that w <= n^(1-Omega(1)).</p><p>Using existing priority queues and sorting algorithms, this implies<br />that we can deterministically sort time-space optimally in time Theta(T) for T >= n(lg lg n)^2, and with high probability for T >= n lg lg n.</p><p>Our results imply that recent lower bounds for deciding element distinctness in o(n lg n) time are nearly tight.</p>

A Wait-Free Sorting Algorithm

Sorting is one of a set of fundamental problems in computer science. In this paper we present the first wait-free algorithm for sorting an input array of size N using P ≤ N processors to achieve optimal running time. We show two variants of the algorithm, one deterministic and one randomized, and prove that, with high probability, the latter suffers no more than $O(\sqrt{P})$ contention when run synchronously. Known sorting algorithms, when made wait-free through previously established transformation techniques, have complexity O(log 3 N) . The algorithm we present here, when run in the CRCW PRAM model, executes with high probability in O(log N) time when P=N , and O((Nlog N)/P) otherwise, which is optimal amongst comparison-based sorting algorithms. The wait-free property guarantees that the sort will complete despite any delays or failures incurred by the processors. This is a very desirable property from an operating systems point of view, since it allows oblivious thread scheduling as well as thread creation and deletion, without fear of losing the algorithm's correctness.

Analysing cache effects in distribution sorting

We study cache effects in distribution sorting algorithms for sorting keys drawn independently at random from a uniform distribution ('uniform keys'). We note that the performance of a recently-published distribution sorting algorithm, Flashsort1, which sorts <i>n</i> uniform floating-point keys in <i>O(n)</i> expected time, does not scale well with the input size due to poor cache utilisation. We present an approximate analysis for distribution sorting uniform keys which, as validated by simulation results, predicts the expected cache misses of Flashsort1 quite well. Using this analysis, we design a multiple-pass variant of Flashsort1 which outperforms Flashsort1 and comparison-based algorithms on uniform floating-point keys for moderate to large values of <i>n</i>. Using experimental results we also show that the integer distribution sorting algorithm MSB radix sort performs well on both uniform integer and uniform floating-point keys.

Optimal algorithms for inserting a random element into a random heap

Two algorithms for inserting a random element into a random heap are shown to be optimal (in the sense that they use the least number of comparisons on the average among all comparison-based algorithms) for different values of n under a uniform model.

Splaysort: Fast, Versatile, Practical

Adaptivity in sorting algorithms is sometimes gained at the expense of practicality. We give experimental results showing that Splaysort - sorting by repeated insertion into a Splay tree - is a surprisingly efficient method for in-memory sorting. Splaysort appears to be adaptive with respect to all accepted measures of presortedness, and it outperforms Quicksort for sequences with modest amounts of existing order. Although Splaysort has a linear space overhead, there are many applications for which this is reasonable. In these situations Splaysort is an attractive alternative to traditional comparison-based sorting algorithms such as Heapsort, Mergesort, and Quicksort.

Trans-dichotomous algorithms for minimum spanning trees and shortest paths

Two algorithms are presented: a linear time algorithm for the minimum spanning tree problem and an O ( m + n log n /log log n ) implementation of Dijkstra's shortest-path algorithm for a graph with n vertices and m edges. The second algorithm surpasses information theoretic limitations applicable to comparison-based algorithms. Both algorithms utilize new data structures that extend the fusion tree method.

A General Lower Bound on the I/O-Complexity of Comparison-based Algorithms

We show a relationship between the number of comparisons and the number of I/O operations needed to solve a given problem. We work in a model, where the permitted operations are l/O-operations and comparisons of two records in internal memory. An I/O- operation swaps <em>B</em> records between external memory and the internal memory (capable of holding <em>M</em> records). An algorithm for this model is called an I/O-algorithm. The main result of this paper is the following: Given an I/O-algorithm that solves an n-record problem P_n using I/O(bar{x}) I/O's on the input bar{x}, there exists an ordinary comparison algorithm that uses no more than <em>n</em> logB + I/O(bar{x}) € T_{merge}(M-B, B) comparisons on input bar{x}. T_{merge}(n, m) denotes the number of comparisons needed to merge two sorted lists of size n and m, respectively. We use the result to show lower bounds on the number of I/O-operations needed to solve the problems of sorting, removing duplicates from a multiset and determining the mode (the most frequently occurring element in a multiset). Aggarwal and Vitter have shown that the sorting bound is tight. We show the same result for the two other problems, by providing optimal algorithms.

Time and Message Bounds for Election in Synchronous and Asynchronous Complete Networks

This paper addresses the problem of distributively electing a leader in both synchronous and asynchronous complete networks. $O(n\log n)$ messages synchronous and asynchronous algorithms are presented. The time complexity of the synchronous algorithm is $O(\log n)$, while that of the asynchronous algorithm is $O(n)$. In the synchronous case, a lower bound of $\Omega (n\log n)$ on the message complexity is proven. It is also proven that any message-optimal synchronous algorithm requires $\Omega (\log n)$ time. In proving these bounds, the type of operations performed by nodes are not restricted. The bounds thus apply to general algorithms and not just to comparison-based algorithms.