Semi-external Algorithm Research Articles

Efficient construction of the BWT for repetitive text using string compression

We present a new semi-external algorithm that builds the Burrows–Wheeler transform variant of Bauer et al. (a.k.a., BCR BWT) in linear expected time. Our method uses compression techniques to reduce computational costs when the input is massive and repetitive. Concretely, we build on induced suffix sorting (ISS) and resort to run-length and grammar compression to maintain our intermediate results in compact form. Our compression format not only saves space but also speeds up the required computations. Our experiments show important space and computation time savings when the text is repetitive. In moderate-size collections of real human genome assemblies (14.2 GB - 75.05 GB), our memory peak is, on average, 1.7x smaller than the peak of the state-of-the-art BCR BWT construction algorithm (ropebwt2), while running 5x faster. Our current implementation was also able to compute the BCR BWT of 400 real human genome assemblies (1.2 TB) in 41.21 hours using 118.83 GB of working memory (around 10% of the input size). Interestingly, the results we report in the 1.2 TB file are dominated by the difficulties of scanning huge files under memory constraints (specifically, I/O operations). This fact indicates we can perform much better with a more careful implementation of our method, thus scaling to even bigger sizes efficiently.

A linear algorithm for semi-external cutnode computation

In the literature, many algorithms have been proposed for finding cutnodes on undirected graphs, since cutnodes are crucial to graph connectivity. Here, a cutnode of an undirected graph G is a node of G, whose deletion will cause a reachable pair of the other nodes in G to be unreachable. Currently, the difficulty of maintaining the entire G in the main memory makes researchers pay attention to compute cutnodes on semi-external memory model. This paper shows that traditional semi-external algorithms are limited by their unbounded time and I/O consumption, making them impractical when G is relatively large or complex. Thus, we propose a linear semi-external cutnode computation algorithm, named SECN. Assuming that G has n nodes and m edges, and B is the disk block size. SECN is the first that can find all the cutnodes of G in O(m+n) time and with O(mB) I/O cost on the semi-external memory model, as far as we know. SECN also has a smaller minimum memory space requirement than traditional algorithms. Our experimental evaluation conducted on both synthetic and real graphs confirms that SECN significantly outperforms existing algorithms (up to 103 times faster).

Efficient semi-external depth-first search

As graphs grow in size, many real-world graphs are difficult to load into the primary memory of a computer. Thus, computing depth-first search (DFS) results (i.e., depth-first order or DFS-Tree) on the semi-external memory model is important to investigate. Semi-external algorithms assume that the primary memory can at least hold a spanning tree T of a graph G and gradually restructure T into a DFS-Tree, which is nontrivial. In this paper, we present a comprehensive study for the semi-external DFS problem. Based on a theoretical analysis of this problem, we introduce a new semi-external DFS algorithm called EP-DFS with a lightweight index N+-index. Unlike traditional algorithms, we focus on addressing such a complex problem efficiently with fewer I/Os, simpler CPU calculations (implementation-friendly), and less random I/O accesses (key-to-efficiency). Extensive experimental evaluations are performed on both synthetic and real graphs, and experimental results confirm that the proposed EP-DFS algorithm markedly outperforms existing algorithms.

I/O Efficient Core Graph Decomposition: Application to Degeneracy Ordering

Core decomposition is a fundamental graph problem with a large number of applications. Most existing approaches for core decomposition assume that the graph is kept in memory of a machine. Nevertheless, many real-world graphs are too big to reside in memory. In this paper, we study I/O efficient core decomposition following a semi-external model, which only allows node information to be loaded in memory. We propose a semi-external algorithm and an optimized algorithm for I/O efficient core decomposition. To handle dynamic graph updates, we firstly show that our algorithm can be naturally extended to handle edge deletion. Then, we propose an I/O efficient core maintenance algorithm to handle edge insertion, and an improved algorithm to further reduce I/O and CPU cost. In addition, based on our core decomposition algorithms, we further propose an I/O efficient semi-external algorithm for degeneracy ordering, which is an important graph problem that is highly related to core decomposition. We also consider how to maintain the degeneracy order. We conduct extensive experiments on 12 real large graphs. Our optimal core decomposition algorithm significantly outperforms the existing I/O efficient algorithm in terms of both processing time and memory consumption. They are very scalable to handle web-scale graphs. As an example, we are the first to handle a web graph with 978.5 million nodes and 42.6 billion edges using less than 4.2 GB memory. We also show that our proposed algorithms for degeneracy order computation and maintenance can handle big graphs efficiently with small memory overhead.

I/O efficient: computing SCCs in massive graphs

A strongly connected component ($$\mathsf {SCC}$$SCC) is a maximal subgraph of a directed graph $$G$$G in which every pair of nodes is reachable from each other in the $$\mathsf {SCC}$$SCC. With such a property, a general directed graph can be represented by a directed acyclic graph (DAG) by contracting every $$\mathsf {SCC}$$SCC of $$G$$G to a node in DAG. In many real applications that need graph pattern matching, topological sorting, or reachability query processing, the best way to deal with a general directed graph is to deal with its DAG representation. Therefore, finding all $$\mathsf {SCC}$$SCCs in a directed graph $$G$$G is a critical operation. The existing in-memory algorithms based on depth first search (DFS) can find all $$\mathsf {SCC}$$SCCs in linear time with respect to the size of a graph. However, when a graph cannot reside entirely in the main memory, the existing external or semi-external algorithms to find all $$\mathsf {SCC}$$SCCs have limitation to achieve high I/O efficiency. In this paper, we study new I/O-efficient semi-external algorithms to find all $$\mathsf {SCC}$$SCCs for a massive directed graph $$G$$G that cannot reside in main memory entirely. To overcome the deficiency of the existing DFS-based semi-external algorithm that heavily relies on a total order, we explore a weak order based on which we investigate new algorithms. We propose a new two-phase algorithm, namely, tree construction and tree search. In the tree construction phase, a spanning tree of $$G$$G can be constructed in bounded number of sequential scans of $$G$$G. In the tree search phase, it needs to sequentially scan the graph once to find all $$\mathsf {SCC}$$SCCs. In addition, we propose a new single-phase algorithm, which combines the tree construction and tree search phases into a single phase, with three new optimization techniques. They are early acceptance, early rejection, and batch processing. By the single-phase algorithm with the new optimization techniques, we can significantly reduce the number of I/Os and the CPU cost. We prove the correctness of the algorithms. We conduct extensive experimental studies using 4 real datasets including a massive real dataset and several synthetic datasets to confirm the I/O efficiency of our approaches.