We analyze the Shell Sort algorithm under the usual random permutation model. Using empirical distribution functions, we recover Louchard's result that the running time of the 1-stage of.2; 1/-Shell Sort has a limiting distribution given by the area under the absolute Brownian bridge. The analysis extends to .h; 1/- Shell Sort where we find a limiting distribution given by the sum of areas under correlated absolute Brownian bridges. A variation of.h; 1/-Shell Sort which is slightly more efficient is presented and its asymptotic behavior analyzed. 1. Introduction. Shell Sort is an algorithm that generalizes the method of sorting by insertion. It is essentially several stages of insertion sort. The algorithm was proposed in Shell (1959). The method received considerable attention over the past quarter of a century after Knuth's 1973 book popularized it (Knuth, 1973). From a practical point of view, the interest in Shell Sort stems from the fact that it is a rather practicable method of in situ sorting with little overhead and can be implemented with ease. From a theoretical standpoint, the interest is that insertion sort has an average of 2.n 2 / running time to sort n random keys, whereas the appropriate choice of the parameters of the stages of Shell Sort can bring down the order of magnitude. For instance, by a certain choice of the structure of the stages, a 2-stage Shell Sort can sort in O.n 5=3 / average running time. Ultimately, an optimized choice of the parameter can come close to the information-theoretic lower bound of O.n ln n/ average case. The analysis of Shell Sort has stood as a formidable challenge. Most research on Shell Sort has gone in the direction of making good choices for the parameters of the stages to obtain good worst-case behavior (see, for example, the review paper of Sedgewick (1996)). We propose here to take the research along a different axis and to analyze the stochastic structure of the algorithm. We rederive the limit result of Louchard (1986) for .2; 1/-Shell Sort, which he proved by essentially combinatoric arguments, and we generalize the approach to the analysis of .h; 1/-Shell Sort. The integrated alsolute value of the Brownian bridge appears in the limiting distributions; the moments of the distribution of this random variable were found by Shepp (1982), and Johnson and Killeen (1983) gave an explicit characterization of the distribution function.
Read full abstract