In-memory Algorithm Research Articles

PgRC: pseudogenome-based read compressor.

The amount of sequencing data from high-throughput sequencing technologies grows at a pace exceeding the one predicted by Moore's law. One of the basic requirements is to efficiently store and transmit such huge collections of data. Despite significant interest in designing FASTQ compressors, they are still imperfect in terms of compression ratio or decompression resources. We present Pseudogenome-based Read Compressor (PgRC), an in-memory algorithm for compressing the DNA stream, based on the idea of building an approximation of the shortest common superstring over high-quality reads. Experiments show that PgRC wins in compression ratio over its main competitors, SPRING and Minicom, by up to 15 and 20% on average, respectively, while being comparably fast in decompression. PgRC can be downloaded from https://github.com/kowallus/PgRC. Supplementary data are available at Bioinformatics online.

FreshJoin: An Efficient and Adaptive Algorithm for Set Containment Join

This paper revisits set containment join (SCJ) problem, which uses the subset relationship (i.e., subseteq) as condition to join set-valued attributes of two relations and has many fundamental applications in commercial and scientific fields. Existing in-memory algorithms for SCJ are either signature-based or prefix-tree-based. The former incurs high CPU cost because of the enumeration of signatures, while the latter incurs high space cost because of the storage of prefix trees. This paper proposes a new adaptive parameter-free in-memory algorithm, named as frequency-hashjoin or {mathsf {FreshJoin}} in short, to evaluate SCJ efficiently. {mathsf {FreshJoin}} builds a flat index on-the-fly to record three kinds of signatures (i.e., two least frequent elements and a hash signature whose length is determined adaptively by the frequencies of elements in the universe set). The index consists of two sparse inverted indices and two arrays which record hash signatures of all sets in each relation. The index is well organized such that {mathsf {FreshJoin}} can avoid enumerating hash signatures. The rationality of this design is explained. And, the time and space cost of the proposed algorithm, which provide a rule to choose {mathsf {FreshJoin}} from existing algorithms, are analyzed. Experiments on 16 real-life datasets show that {mathsf {FreshJoin}} usually reduces more than 50% of space cost while remains as competitive as the state-of-the-art algorithms in running time.

Truss decomposition of uncertain graphs

The k-truss of a graph is the largest edge-induced subgraph such that every edge is contained in at least k triangles within the subgraph, where a triangle is a cycle consisting of three vertices. As a new notion of cohesive subgraphs, truss has recently attracted a lot of research attentions in the database and data mining fields. At the same time, uncertainty is an intrinsic property of massive graph data, and truss decomposition (i.e., finding all k-trusses of a graph) has become a key primitive on uncertain graphs. In this paper, we study the truss decomposition problem on uncertain graphs, that is, finding all highly probable k-trusses of an uncertain graph. We first give an formal statement of the truss decomposition problem on uncertain graphs. Then, we prove that the truss decomposition of an uncertain graph attains two elegant properties, namely uniqueness and hierarchy. We show that the truss decomposition of an uncertain graph can be found in $$O(m^{1.5}Q)$$O(m1.5Q) time by proposing an in-memory algorithm called $$\mathtt {TD_{mem}}$$TDmem, where m is the number of edges of the uncertain graph, and Q is at most the maximum number of common neighbors of the endpoints of an edge. When an uncertain graph is too large to fit into main memory, we propose an external-memory algorithm $$\mathtt {TD_{I/O}}$$TDI/O to find the truss decomposition of the uncertain graph. Extensive experiments have been carried out to evaluate the practical performance of the proposed algorithms. The experimental results verify that both $$\mathtt {TD_{mem}}$$TDmem and $$\mathtt {TD_{I/O}}$$TDI/O are efficient when an uncertain graph is small enough to fit into main memory, and that $$\mathtt {TD_{I/O}}$$TDI/O is much faster than $$\mathtt {TD_{mem}}$$TDmem when the graph is too large to fit into main memory.

Efficient Distributed RNN Query Processing with Caching

Reverse Nearest Neighbour (RNN) queries play an important role in applications such as internet of vehicles, decision support systems, profile based marketing and so on. Recently, more attention has been paid to the problem of efficient distributed RNN computation in mobile cloud computing environment. A major downside of the existing RNN is its inherent sequential nature and using in-memory algorithm, which limits its applicability to massive data. In this paper, we propose a novel distributed caching based method to efficiently improve the performance of the RNN calculation in a distributed environment. Extensive experiments using both real and synthetic datasets demonstrated that our proposed methods are the state-of-the-art algorithms in scalable RNN queries.

Truss decomposition in massive networks

The k -truss is a type of cohesive subgraphs proposed recently for the study of networks. While the problem of computing most cohesive subgraphs is NP-hard, there exists a polynomial time algorithm for computing k -truss. Compared with k -core which is also efficient to compute, k -truss represents the "core" of a k -core that keeps the key information of, while filtering out less important information from, the k -core. However, existing algorithms for computing k -truss are inefficient for handling today's massive networks. We first improve the existing in-memory algorithm for computing k -truss in networks of moderate size. Then, we propose two I/O-efficient algorithms to handle massive networks that cannot fit in main memory. Our experiments on real datasets verify the efficiency of our algorithms and the value of k -truss.

Practical algorithms for unsatisfiability proof and core generation in SAT solvers

Since Zhang and Malik's work in 2003 [19], it is well-known that modern DPLL-based SAT solvers with learning can be instrumented to write a trace on disk from which, if the input is unsatisfiable, a resolution proof can be extracted (and checked), and hence also an unsatisfiable core: a (frequently small) unsatisfiable subset of the input clauses. In this article we first give a new algorithmic approach for processing these (frequently huge) traces. It achieves the efficiency of a depth-first traversal, while preserving the property that memory usage remains upper bounded by that of the SAT solver that generated the trace. The second part of this article is about in-memory algorithms for generating SAT proofs and cores, without writing traces to disk. We discuss advantages and disadvantages of this approach and investigate why the current SAT solvers with this feature still run out of memory on long SAT runs. We analyze several of these in-memory algorithms, based on carefully designed experiments with our implementation of each one of them, as well as with (our implementation of) a trace-based one. Then we describe a new in-memory algorithm which saves space by doing more bookkeeping to discard unnecessary information, and show that it can handle significantly more instances than the previously existing algorithms, at a negligible expense in time.