External Sorting Research Articles

Sorting on Byte-Addressable Storage: The Resurgence of Tree Structure

The tree structure is notably popular for storage and indexing; however, tree-based sorting such as tree sort is rarely used in practice. Nevertheless, with the advent of byte-addressable storage (BAS), the tree structure captures our attention with its write-once property. This property aligns well with BAS's asymmetric read-write characteristics. In this paper, we seek to answer the question: Can tree-based sorting algorithms outperform existing algorithms in the hybrid DRAM-BAS system? To address this, first, we conduct a comprehensive study to assess the compatibility of existing sorting algorithms with such hybrid memory systems and explore the challenges. We then delve into various design dimensions of tree-sort algorithms which leads to an optimized variant, TSort. Finally, a comparative analysis is conducted among three different sets of sorting algorithms, including in-place sorts, external sorts, and tree-based sorts. The results indicate that TSort not only challenges the traditional negative perceptions of the tree structure in sorting but also exhibits excellent performance. It outperforms all other counterparts across diverse datasets, whether uniformly distributed or skewed, in most cases.

Seeking an optimal design of Search and Merge: its consequences and challenges

Abstract We propose that Merge, both External Merge and Internal Merge, is totally free from Minimal Search, and more specifically, Search Σ to determine the input of Merge only obeys Binarity and the Phase Impenetrability Condition but not Minimal Search (the Minimal Search-free Merge Hypothesis). We argue that our proposal provides a unified account of various movement restrictions, such as the freezing effect, the that-trace effect, the anti-locality effect, the vacuous movement hypothesis, and the economy of derivation. We also argue that our proposal derives the insights/consequences of Minimal Yield such as ruling out so-called extensions of Merge by limiting the search space at a later stage of a derivation in terms of Binarity and the PIC. We further expand the empirical and theoretical scope of our proposal by considering exceptions to the freezing effect. We suggest that the exceptions can be dealt with by adopting Form Copy.

Learning mechanism for a collective classifier based on competition driven by training examples

The purpose of this work is to modify the learning mechanism of a collective classifier in order to provide learning by population dynamics alone, without requiring an external sorting device. A collective classifier is an ensemble of non-identical simple elements, which do not have any intrinsic dynamics neither variable parameters; the classifier admits learning by adjusting the composition of the ensemble, which was provided in the preceding literature by selecting the ensemble elements using a sorting device. Methods. The population dynamics model of a collective classifier is extended by adding a “learning subsystem”, which is controlled by a sequence of training examples and, in turn, controls the strength of intraspecific competition in the population dynamics. The learning subsystem dynamics is reduced to a linear mapping with random parameters expressed via training examples. The solution to the mapping is an asymptotically stationary Markovian random process, for which we analytically find asymptotic expectation and show its variance to vanish in the limit under the specified assumptions, thus allowing an approximate deterministic description of the coupled population dynamics based on available results from the preceding literature. Results. We show analytically and illustrate it by numerical simulation that the decision rule of our classifier in the course of learning converges to the Bayesian rule under assumptions which are essentially in line with available literature on collective classifiers. The implementation of the required competitive dynamics does not require an external sorting device. Conclusion. We propose a conceptual model for a collective classifier, whose learning is fully provided by its own population dynamics. We expect that our classifier, similarly to the approaches taken in the preceding literature, can be implemented as an ensemble of living cells equipped with synthetic genetic circuits, when a mechanism of population dynamics with synthetically controlled intraspecific competition becomes available.

Budding the tree. Towards a theory of structure removal

This paper presents a theory of structure removal based on the operation of Bud. Bud is defined as a variant of External Merge. While External Merge always targets the root of a syntactic object, Bud targets a lower projection. The operation is constrained by minimality and is used to provide a novel account to analytic passive, causative constructions in Italian and infinitival constructions like raising predicates. It also offers an elegant way to account for structure pruning in cases of language disorder, root infinitives in child speech and preference for passive plus subject relativization over object relative clauses.

Halstead’s Complexity Measure of a Merge sort and Modified Merge sort Algorithms

Complexity measuring tools in computer science are deployed to measure and compare different characteristics of algorithms in order to find the best one in solving a particular problem or that suits a particular situation. In view of this, Halstead’s complexity metrics is deployed to compare the efficiency of two external sorting methods; the Merge sort and the Modified Merge sort algorithms. The methodology used in achieving this, lies in the extraction of operators and operands from the C_sharp (C#) implemented program of the two algorithms. Six Halstead metrics are evaluated using these operators and operands as parameters. The results shows that the Modified Merge sort algorithm is much more efficient than the conventional Merge sort as it has a lower Program Volume, lower Program difficulty and a lower Program Effort even though the advantage of a higher Intelligence content goes to the Merge sort algorithm.

Partial Wh-Movement in North Italo-Romance and the FormCopy Analysis of Wh-Doubling

This paper analyzes the syntactic derivation of so-called ‘wh-doubling’, a single-constituent question-formation strategy that features the overt occurrence of two wh-phrases (e.g. kuza fa la ku’zε (lit.) ‘what does she do what?’) in North Italo-Romance. We argue that the wh-phrases involved in the construction are best treated as being generated independently by External Merge (EM), rather than being bona fide syntactic copies constructed by Internal Merge (IM). This raises the theoretical issue why IM to scope position cannot take place in wh-doubling, despite IM being more economical than EM. We propose that the wh-element merged in argumental position undergoes partial movement to the edge of the v-phase, where it becomes ‘frozen’ upon entering into a Focus configuration. This makes the derivation of wh-doubling identical to the derivation of wh-in situ up to the v-phase as recently proposed for different wh-in situ languages. The derivation of wh-doubling then proceeds by EM of an additional wh-phrase that gives phonological content to the scope Q position in the left periphery. The single-constituent reading is obtained at Conceptual-Intentional (C-I) via the operation FormCopy (FC), which connects the independently generated wh-elements in the syntactic workspace. We moreover discuss issues pertaining to parametric variation.

Wh-complement Clauses and (non-)Local Selection

This paper revisits c- and s-selection in the context of wh-clausal complements. The standard treatment of predicate-complement selection can be traced back to generative approaches of 70’s and says that the selection of a complement by a predicate is evaluated both in syntax and semantics. As regards syntax, the grammatical category of the complement must belong to the subcategorization frame of the predicate, and vis-à-vis semantics, the semantic type of the complement must fall in the set of types selected by the predicate. The present paper examines several licit instances of wh-clausal selection that should have been ungrammatical under the standard treatment, but they are not. The analysis offered here says that c-selection reduces to argument selection and is computed derivationally (at the point of External Merge), while s-selection reduces to an interpretation function that spans a larger grammatical domain and is evaluated representationally (at the syntax-semantics interface).

The structure of wh-exclamatives in Jordanian Arabic and beyond

ABSTRACT This paper presents an analysis of the characteristics of wh-exclamatives in Jordanian Arabic. I argue that exclamatives are not complete sentences because they are temporally deictic to the moment of utterance. In other words, they lack Tense specification and are considered tenseless expressions (i.e., root small clauses). Two notable complexities arise in wh-exclamatives in Jordanian Arabic, namely the obligatory presence of the demonstrative-like element ha- and fixed word order. These intricacies are attributed to the exclamative head’s requirement for a specifier. This requirement can be fulfilled either through the combination of a wh-form (external merge) or the movement of the degree phrase to the specifier position (internal merge). The study proposes that the demonstrative-like ha- functions as the actual manifestation of the Relator head which connects and establishes the relationship between the two constituents in exclamatives. Furthermore, based on the distinct semantic and distributional properties of the auxiliary kān ‘was’, the paper argues for the differentiation of the auxiliary in exclamatives from other typical auxiliaries that occupy verb or Tense nodes in sentences. Therefore, it suggests that exclamative auxiliaries encode and express the grammatical features of the functional head, labeled as Excl.

LazySort: A customized sorting algorithm for non-volatile memory

Due to the development of new non-volatile storage technologies, bridging the access delay gap between storage and memory has been promoted. Therefore, since the volume of data continues to increase, the need for new types of non-volatile storage for high-performance computing and database management systems significantly increases. As a basic algorithm of the database management system, external sorting is widely used but suffers from numerous I/O operations caused by the frequent data exchange between the memory and external memory. Non-volatile memory (NVM) is a promising solution. However, the traditional external sorting algorithm is only designed for disk storage and cannot be used for NVM. To solve this issue, we propose LazySort, which is a novel external sorting algorithm that exploits the NVM byte addressing mechanism and the orderly distribution of data to improve the sorting efficiency. Specifically, LazySort first detects the ordered data segment in the memory and then records the storage location of the data segment as well as its maximum/minimum values in an index table named ITable. Next, to reduce memory usage and accelerate the sorting procedure, LazySort performs an optimization strategy RunMerge to merge non-intersecting data blocks according to the range of ITable records. To verify the performance of LazySort, we built an experimental platform with the NVM-DRAM storage architecture combining NVM and dynamic random access memory (DRAM) and conducted a series of experiments. The experimental results show that LazySort is more efficient than the traditional external sorting algorithm, since the sorting time of LazySort is shortened by 93.08% and the number of NVM write operations is reduced by 49.50%. Moreover, the advantage of LazySort over traditional algorithms is more significant when the data amount increases.

Efficient Sorting, Duplicate Removal, Grouping, and Aggregation

Database query processing requires algorithms for duplicate removal, grouping, and aggregation. Three algorithms exist: in-stream aggregation is most efficient by far but requires sorted input; sort-based aggregation relies on external merge sort; and hash aggregation relies on an in-memory hash table plus hash partitioning to temporary storage. Cost-based query optimization chooses which algorithm to use based on several factors, including the sort order of the input, input and output sizes, and the need for sorted output. For example, hash-based aggregation is ideal for output smaller than the available memory (e.g., Query 1 of TPC-H), whereas sorting the entire input and aggregating after sorting are preferable when both aggregation input and output are large and the output needs to be sorted for a subsequent operation such as a merge join. Unfortunately, the size information required for a sound choice is often inaccurate or unavailable during query optimization, leading to sub-optimal algorithm choices. In response, this article introduces a new algorithm for sort-based duplicate removal, grouping, and aggregation. The new algorithm always performs at least as well as both traditional hash-based and traditional sort-based algorithms. It can serve as a system’s only aggregation algorithm for unsorted inputs, thus preventing erroneous algorithm choices. Furthermore, the new algorithm produces sorted output that can speed up subsequent operations. Google’s F1 Query uses the new algorithm in production workloads that aggregate petabytes of data every day.

External merge in spec,CP: Complementizers projecting an argument

AbstractThe standard assumption that spec,CP is always an A′ position has been questioned for several languages where embedded C heads are involved in agreement and case assignment; however, the idea that no XP can be introduced in spec,CP by external merge has remained unchallenged. The article presents novel data from control phenomena in Mari (Uralic; nominative, SOV) and argues that, in this language, a particular type of C head is capable of thematically licensing an overt argument externally merged in spec,CP: the complementizer manən used in infinitival complement clauses projects a dative goal of communication. This behavior of manən follows from its dual nature: it is a semigrammaticalized verb ‘say’ that retains some lexical characteristics. I further suggest that the dative goal can be considered an overt realization of the addressee discourse variable, in line with the recent work on the presence of speaker, addressee, and logophoric center in syntax.

ISort: SSD Internal Sorting Algorithm for Big Data

As a basic algorithm for big data processing, external sorting suffers from massive read and write operations in the external memory. Recent works separate part of the data processing work from the host side to the solid state drive (SSD) to reduce data transmission. However, the internal memory of the SSD is limited, and undesirable data retention could occur during the merge phase. Therefore, to improve the efficiency of memory, we propose an algorithm named ISort. Specifically, we build an index table between the memory and the address. The index table determines the order of pages being read in the merge phase according to their minimum values, which are read into memory sequentially to reduce the data residing in memory and improve memory efficiency. Since the merge phase is performed inside the SSD, ISort can take advantage of the high IO bandwidth within the SSD to speed up the execution of the merge phase. We search for the optimal ratio of read and write channels by comparing the “specialized channel” and the “hybrid channel” for data of read and write performance because the utilization of the channel will directly influence performance. Experimental results show that ISort can maintain better data processing speed when SSD memory is limited, outperforming other robust algorithms. In addition, the algorithm’s performance using the crossover strategy is better than that using the specialization strategy.

IN-PLACE MERGING AND EXTERNAL SORTING ALGORITHM WITH NO ADDITIONAL DISK SPACE

The performance of external sorting mainly depends on its I/O and time complexities. In this paper we represent a more efficient external sorting algorithm with no additional disk space. This algorithm uses Quick sort to produce runs in the first phase. It uses special in-place merging technique in the second phase. The algorithm excels in sorting a huge file, which is larger than the available internal memory of the computer. The algorithm creates no backup file for manipulating huge records. Usually the external sorting algorithm needs additional disk space in merging phase. This algorithm saves disk space, since it does not use any backup file or additional disk space. Here we have briefly reviewed the external sorting algorithms, which do not use additional disk space. The I/O and time complexities of the proposed algorithm are analyzed and compared with that of other similar algorithms. We have found that the I/O complexity of the proposed algorithm is less than that of other similar algorithms (algorithms which do not need additional disk space) with the same time complexity.

Argument ellipsis as external merge after transfer

Argument ellipsis as external merge after transfer

Movement and cyclic Agree

Recent work has argued for derivational expansion of the search space of ɸ-probes as a result of the cyclic interaction of Merge and Agree (cyclic Agree). Empirical evidence for such effects has come from the interaction of Agree with external Merge, but these accounts make the additional predictions that such interactions should also arise from the interaction of Agree with movement. In this paper, we argue based on evidence from Hindi-Urdu that movement may feed cyclic Agree in the same way as external Merge, bearing out this key prediction. The central empirical generalizations that we argue for are that (i) A-scrambling of an object over a subject may feed agreement in certain configurations, (ii) there is nonetheless a preference for agreement with the structurally lower subject, and (iii) overline {text{A}}-scrambling of the object never feeds agreement. We develop a cyclic-Agree analysis that provides a principled explanation of these generalizations. Crucial to this explanation are that (i) search space is dynamic and interacts with movement, (ii) there is a preference for agreement with elements c-commanded by the head that hosts the probe (first-cycle Agree), and (iii) ɸ-probes project as part of the label, but not past the immediate projection line of the head.

Fast continuous streaming sort in big streaming data environment under fixed-size single storage.

Big streaming data environment concerns a complicated scenario where data to be processed continuously flow into a processing unit and certainly cause a memory overflow problem. This obstructs the adaptation of deploying all existing classic sorting algorithms because the data to be sorted must be entirely stored inside the fixed-size storage including the space in internal and external storage devices. Generally, it is always assumed that the size of each data chunk is not larger than the size of storage (M) but in fact the size of the entire stream (n) is usually much larger than M. In this paper, a new fast continuous streaming sorting is proposed to cope with the constraint of storage overflow. The algorithm was tested with various real data sets consisting of 10,000 to 17,000,000 numbers and different storage sizes ranging from 0.01n to 0.50n. It was found that the feasible lower bound of storage size is 0.35n with 100% sorting accuracy. The sorting time outperforms bubble sort, quick sort, insertion sort, and merge sort when data size is greater than 1,000,000 numbers. Remarkably, the sorting time of the proposed algorithm is 1,452 times less than the sorting time of external merge sort and 28.1767 times less than the sorting time of streaming data sort. The time complexity of proposed algorithm is O(n) while the space complexity is O(M).

An Efficient Sorting Algorithm for Non-Volatile Memory

Non-volatile memory (NVM) has emerged as an alternative of the next-generation memory due to its non-volatility, byte addressability, high storage-density, and low-energy consumption. However, NVM also has some limitations, e.g. asymmetric read and write latency. Therefore, at present, it is not realistic to completely replace DRAM with NVM in computer systems. A more feasible scheme is to adopt the hybrid memory architecture composed of NVM and DRAM. Following the assumption of hybrid memory architecture, in this paper, we propose an NVM-friendly sorting algorithm called NVMSorting. Particularly, we introduce a new concept called Natural Run to improve the existing MONTRES algorithm. Further, we apply the proposed NVMSorting to database join algorithms to improve the performance of the existing sort-merge join. To verify the performance of our proposal, we implement six existing sorting algorithms as baselines, including the MONTRES algorithm, and conduct comparative experiments on real Intel Optane DC persistent memory. The results show that NVMSorting outperforms other sorting algorithms in terms of execution time and NVM writes. In addition, the results of the join experiment show that the NVMSorting algorithm achieves the highest performance among all schemes. Especially, in the partially ordered data, the execution time of NVMSorting is 2.9%, 2.7%, and 4.2% less than MONTRES, external sort, and quick sort, respectively. Also, the amount of NVM writes of the NVMSorting is 26.1%, 43.6%, 96.2% less than MONTRES, external sort, and quick sort, respectively.

On the argument structure of complex denominal verbs in Latin: a syntactic approach

Abstract In this paper I offer a syntactic approach to the formation of complex denominal verbs in Latin. Two basic types of prefixed locative denominal verbs can be distinguished in this language: location ones “agglutinate” a PP expressing location, whereas locatum ones contain a noun expressing the locatum object. Assuming a syntactic distinction between Incorporation and Conflation in denominal verb formation, I claim that prefixed location verbs are formed via Incorporation (i.e. Internal Merge), whereas prefixed locatum verbs are formed via Conflation (i.e. External Merge). Unprefixed locative verbs can only be interpreted as locatum predicates, but unlike prefixed locatum verbs, they are analyzed as involving a possessive relation and as being formed via incorporation. The present approach also provides an explanation of why Romance locatum verbs, unlike location ones, are not necessarily prefixed. It is also claimed that unprefixed and prefixed locatum verbs in Romance are formed via incorporation rather than via conflation, its reason being related to the typological shift from the presence of a typical conflation pattern in satellite-framed Latin to a lack of it in verb-framed Romance languages. Finally, I show that Latin prefixed denominal verbs and prefixed deadjectival ones are all telic and project a ResultP in syntax. In contrast, this projection can be argued to be absent from unprefixed denominal and deadjectival verbs.

Finding the optimal execution scheme of external mergesort on solid state drives

As the flash-based solid-state drives(SSDs) gradually replace the mechanical hard disk drives(HDDs) as the mainstream storage, unlike the HDDs, SSDs have rich internal parallelism, which makes it have the excellent characteristics that HDDs do not have. External mergesort, as the classical algorithm of external sorting adopted in many systems and algorithms, has an important impact on the overall performance. Therefore, it is of great significance to optimize and improve the efficiency of external mergesort algorithm. The research work on optimizing raw external mergesort algorithm on SSDs is relatively few. Thus, aiming at the external mergesort problem, based on the characteristics of SSDs, this paper proposes the SortDecision algorithm which can calculate its optimal execution scheme, including merging way, read buffer size, and write buffer size which determine the execution process of external mergesort. Exploiting the above optimal execution scheme, external mergesort can obtain better efficiency. In the SortDecision algorithm, external mergesort problem on SSDs is formalized and transformed into a piecewise convex optimization problem. Then, the optimal external mergesort scheme is obtained by enumerating the solutions of each subconvex problem. The experimental results show that the external mergesort proceeds guided by SortDecision algorithm can achieve a speedup of 1 $\sim $ 6.7 compared to the traditional external mergesort algorithm in the case of limited memory provided. The richer the internal parallelism resources inside SSDs, the better the effect of SortDecision’s acceleration.

Efficient External Sorting for Memory-Constrained Embedded Devices with Flash Memory

Embedded devices are ubiquitous in areas of industrial and environmental monitoring, health and safety, and consumer appliances. A common use case is data collection, processing, and performing actions based on data analysis. Although many Internet of Things (IoT) applications use the embedded device simply for data collection, there are benefits to having more data processing done closer to data collection to reduce network transmissions and power usage and provide faster response. This work implements and evaluates algorithms for sorting data on embedded devices with specific focus on the smallest memory devices. In devices with less than 4 KB of available RAM, the standard external merge sort algorithm has limited application as it requires a minimum of three memory buffers and is not flash-aware. The contribution is a memory-optimized external sorting algorithm called no output buffer sort (NOBsort) that reduces the minimum memory required for sorting, has excellent performance for sorted or near-sorted data, and sorts on external memory such as SD cards or raw flash chips. When sorting large datasets, no output buffer sort reduces I/O and execution time by between 20% to 35% compared to standard external merge sort.