Additional Data Structures Research Articles

Approximate Distance Oracle for Fault-Tolerant Geometric Spanners

In this paper, we present approximate distance and shortest-path oracles for fault-tolerant Euclidean spanners motivated by the routing problem in real-world road networks. A fault-tolerant Euclidean spanner for a set of points in Euclidean space is a graph in which, despite the deletion of small number of any points, the distance between any two points in the damaged graph is an approximation of their Euclidean distance. Given a fault-tolerant Euclidean spanner and a small approximation factor, our data structure allows us to compute an approximate distance between two points in the damaged spanner in constant time when a query involves any two points and a small set of failed points. Additionally, by incorporating additional data structures, we can return a path itself in time almost linear in the length of the returned path. Both data structures require near-linear space.

A vector method for finding sequences in big data

A technological software solution is proposed for metric search and identification of logical-temporal patterns of a business data flow by creating additional vector data structures and a parallel method for their processing. The subject of research is the methods of searching and identifying logical-temporal patterns in big data. The purpose of the study is to increase the efficiency of searching and recognizing logical-temporal patterns that semantically form business functionality in an 8-hour frame of screenshots with "garbage" data. Applied methods: apparatus of set theory and Boolean algebra, metric models for determining parameters for sets of binary vectors, elements of probability theory, theory of algorithms, software modeling. The results obtained: a method for searching and recognizing patterns based on a vector problem of character sequences that identify patterns in big data streams using unitary coding of information primitives and data; vector models are unitary-encoded data structures for describing a big data flow as Cartesian products of a set of primitive-string-markers and a discrete sequence of implementation of a given time frame. The practical significance of the work: the implementation of the vector method, which made it possible to create a pattern recognition program in a big data stream with a probability of 0.77%.

The two‐echelon routing problem with truck and drones

AbstractIn this paper, we study novel variants of the well‐known two‐echelon vehicle routing problem in which a truck works on the first echelon to transport parcels and a fleet of drones to intermediate depots while in the second echelon, the drones are used to deliver parcels from intermediate depots to customers. The objective is to minimize the completion time instead of the transportation cost as in classical two‐echelon vehicle routing problems. Depending on the context, a drone can be launched from the truck at an intermediate depot once (single‐trip drone) or several times (multiple‐trip drone). Mixed‐integer linear programming models are first proposed to formulate mathematically the problems and solve to optimality small‐sized instances. To handle larger instances, a metaheuristic based on the idea of greedy randomized adaptive search procedure is introduced. The main novel feature of our metaheuristic lies in the design of initial solution construction and local search operators, which can cover all the decision layers of the problems and run in using additional data structures. Experimental results obtained on instances of different contexts are reported and analyzed.

A carbuncle cure for the Harten-Lax-van Leer contact (HLLC) scheme using a novel velocity-based sensor

A hybrid numerical flux scheme is proposed by adapting the carbuncle-free modified Harten-Lax-van Leer contact (HLLCM) scheme to smoothly revert to the Harten-Lax-van Leer contact (HLLC) scheme in regions of shear. This hybrid scheme, referred to as the HLLCT scheme, employs a novel, velocity-based shear sensor. In contrast to the non-local pressure-based shock sensors often used in carbuncle cures, the proposed shear sensor can be computed in a localized manner meaning that the HLLCT scheme can be easily introduced into existing codes without having to implement additional data structures. Through numerical experiments, it is shown that the HLLCT scheme is able to resolve shear layers accurately without succumbing to the shock instability.

D-PBWT: dynamic positional Burrows-Wheeler transform.

Durbin's positional Burrows-Wheeler transform (PBWT) is a scalable data structure for haplotype matching. It has been successfully applied to identical by descent (IBD) segment identification and genotype imputation. Once the PBWT of a haplotype panel is constructed, it supports efficient retrieval of all shared long segments among all individuals (long matches) and efficient query between an external haplotype and the panel. However, the standard PBWT is an array-based static data structure and does not support dynamic updates of the panel. Here, we generalize the static PBWT to a dynamic data structure, d-PBWT, where the reverse prefix sorting at each position is stored with linked lists. We also developed efficient algorithms for insertion and deletion of individual haplotypes. In addition, we verified that d-PBWT can support all algorithms of PBWT. In doing so, we systematically investigated variations of set maximal match and long match query algorithms: while they all have average case time complexity independent of database size, they have different worst case complexities and dependencies on additional data structures. The benchmarking code is available at genome.ucf.edu/d-PBWT. Supplementary data are available at Bioinformatics online.

A numerical method for multi-body separation with collisions

Multi-body separation with collisions is ubiquitous in aerospace engineering. In this paper, a numerical method for this problem is developed. Coupling with the equations for six-degree-of-freedom rigid-body motion, the unsteady compressible Reynolds-Averaged Navier-Stokes equations are solved. The dynamic overset unstructured grid based on the criteria of wall distance is developed to handle the relative motion of multiple bodies. The collision among rigid bodies is modeled by using an impulse-based method. Efficient algorithms for collision detection and contact determination utilizing hole cutting information of overset unstructured grid are proposed. In the simulation of six-degree-of-freedom rigid-body motion, a collision event can be robustly detected and the collision time can be accurately determined, without needing additional data structures. Test cases of store separation and ball collision are used to validate the accuracy of the dynamic solvers and the collision algorithms. An application to the release of a large number of projectiles from a carrier assesses the capability of the present method in simulating the problems of complex multi-body separation with collisions.

Gsufsort: constructing suffix arrays, LCP arrays and BWTs for string collections

BackgroundThe construction of a suffix array for a collection of strings is a fundamental task in Bioinformatics and in many other applications that process strings. Related data structures, as the Longest Common Prefix array, the Burrows–Wheeler transform, and the document array, are often needed to accompany the suffix array to efficiently solve a wide variety of problems. While several algorithms have been proposed to construct the suffix array for a single string, less emphasis has been put on algorithms to construct suffix arrays for string collections.ResultIn this paper we introduce gsufsort, an open source software for constructing the suffix array and related data indexing structures for a string collection with N symbols in O(N) time. Our tool is written in ANSI/C and is based on the algorithm gSACA-K (Louza et al. in Theor Comput Sci 678:22–39, 2017), the fastest algorithm to construct suffix arrays for string collections. The tool supports large fasta, fastq and text files with multiple strings as input. Experiments have shown very good performance on different types of strings.Conclusionsgsufsort is a fast, portable, and lightweight tool for constructing the suffix array and additional data structures for string collections.

Variable-order reference-free variant discovery with the Burrows-Wheeler Transform

BackgroundIn [Prezza et al., AMB 2019], a new reference-free and alignment-free framework for the detection of SNPs was suggested and tested. The framework, based on the Burrows-Wheeler Transform (BWT), significantly improves sensitivity and precision of previous de Bruijn graphs based tools by overcoming several of their limitations, namely: (i) the need to establish a fixed value, usually small, for the order k, (ii) the loss of important information such as k-mer coverage and adjacency of k-mers within the same read, and (iii) bad performance in repeated regions longer than k bases. The preliminary tool, however, was able to identify only SNPs and it was too slow and memory consuming due to the use of additional heavy data structures (namely, the Suffix and LCP arrays), besides the BWT.ResultsIn this paper, we introduce a new algorithm and the corresponding tool ebwt2InDel that (i) extend the framework of [Prezza et al., AMB 2019] to detect also INDELs, and (ii) implements recent algorithmic findings that allow to perform the whole analysis using just the BWT, thus reducing the working space by one order of magnitude and allowing the analysis of full genomes. Finally, we describe a simple strategy for effectively parallelizing our tool for SNP detection only. On a 24-cores machine, the parallel version of our tool is one order of magnitude faster than the sequential one. The tool ebwt2InDel is available at github.com/nicolaprezza/ebwt2InDel.ConclusionsResults on a synthetic dataset covered at 30x (Human chromosome 1) show that our tool is indeed able to find up to 83% of the SNPs and 72% of the existing INDELs. These percentages considerably improve the 71% of SNPs and 51% of INDELs found by the state-of-the art tool based on de Bruijn graphs. We furthermore report results on larger (real) Human whole-genome sequencing experiments. Also in these cases, our tool exhibits a much higher sensitivity than the state-of-the art tool.

Efficient haplotype matching between a query and a panel for genealogical search.

MotivationWith the wide availability of whole-genome genotype data, there is an increasing need for conducting genetic genealogical searches efficiently. Computationally, this task amounts to identifying shared DNA segments between a query individual and a very large panel containing millions of haplotypes. The celebrated Positional Burrows-Wheeler Transform (PBWT) data structure is a pre-computed index of the panel that enables constant time matching at each position between one haplotype and an arbitrarily large panel. However, the existing algorithm (Durbin’s Algorithm 5) can only identify set-maximal matches, the longest matches ending at any location in a panel, while in real genealogical search scenarios, multiple ‘good enough’ matches are desired.ResultsIn this work, we developed two algorithmic extensions of Durbin’s Algorithm 5, that can find all L-long matches, matches longer than or equal to a given length L, between a query and a panel. In the first algorithm, PBWT-Query, we introduce ‘virtual insertion’ of the query into the PBWT matrix of the panel, and then scanning up and down for the PBWT match blocks with length greater than L. In our second algorithm, L-PBWT-Query, we further speed up PBWT-Query by introducing additional data structures that allow us to avoid iterating through blocks of incomplete matches. The efficiency of PBWT-Query and L-PBWT-Query is demonstrated using the simulated data and the UK Biobank data. Our results show that our proposed algorithms can detect related individuals for a given query efficiently in very large cohorts which enables a fast on-line query search.Availability and implementationgenome.ucf.edu/pbwt-querySupplementary information Supplementary data are available at Bioinformatics online.

Ray Classification for Accelerated BVH Traversal

AbstractFor ray tracing based methods, traversing a hierarchical acceleration data structure takes up a substantial portion of the total rendering time. We propose an additional data structure which cuts off large parts of the hierarchical traversal. We use the idea of ray classification combined with the hierarchical scene representation provided by a bounding volume hierarchy. We precompute short arrays of indices to subtrees inside the hierarchy and use them to initiate the traversal for a given ray class. This arrangement is compact enough to be cache‐friendly, preventing the method from negating its traversal gains by excessive memory traffic. The method is easy to use with existing renderers which we demonstrate by integrating it to the PBRT renderer. The proposed technique reduces the number of traversal steps by 42% on average, saving around 15% of time of finding ray‐scene intersection on average.

Dynamic structural operational semantics

We introduce Dynamic Structural Operational Semantics (DSOS or Dynamic SOS) as a framework for describing semantics of programming languages that include dynamic software upgrades, i.e., for upgrading software code during run-time. DSOS is built on top of the Modular SOS of P. Mosses, with an underlying category theory formalization. The idea of Dynamic SOS is to bring out the essential differences between dynamic upgrade constructs and program execution constructs. The important feature of Modular SOS (MSOS) that we exploit in DSOS is the sharp separation of the program execution code from the additional (data) structures needed at run-time. In DSOS we aim to achieve the same modularity and decoupling for dynamic software upgrades. This is partly motivated by the long term goal of having machine-checkable proofs for general results like type safety.We exemplify Dynamic SOS on two languages supporting dynamic software upgrades, namely the C-like Proteus, which supports updating of variables, functions, records, or types at specific program points, and Creol, which supports dynamic class upgrades in the setting of concurrent objects. Existing type analyses for software upgrades can be done on top of DSOS too, as we illustrate for Proteus.As a side contribution we define a general encapsulating construction on Modular SOS useful in situations where a form of encapsulation of the execution is needed. We use encapsulation to give modular semantics to the concurrent object-oriented programming language Creol with active objects and asynchronous method invocations.

Maintenance of datalog materialisations revisited

Datalog is a rule-based formalism that can axiomatise recursive properties such as reachability and transitive closure. Datalog implementations often materialise (i.e., precompute and store) all facts entailed by a datalog program and a set of explicit facts. Queries can thus be answered directly in the materialised facts, which is beneficial to the performance of query answering, but the materialised facts must be updated whenever the explicit facts change. Rematerialising all facts ‘from scratch’ can be very inefficient, so numerous materialisation maintenance algorithms have been developed that aim to efficiently identify the facts that require updating and thus reduce the overall work. Most such approaches are variants of the counting or Delete/Rederive (DRed) algorithms. Algorithms in the former group maintain additional data structures and are usually applicable only if datalog rules are not recursive, which limits their applicability in practice. Algorithms in the latter group do not require additional data structures and can handle recursive rules, but they can be inefficient when facts have multiple derivations. Finally, to the best of our knowledge, these approaches have not been compared and their practical applicability has not been investigated. Datalog is becoming increasingly important in practice, so a more comprehensive understanding of the tradeoffs between different approaches to materialisation maintenance is needed. In this paper we present three such algorithms for datalog with stratified negation: a new counting algorithm that can handle recursive rules, an optimised variant of the DRed algorithm that does not repeat derivations, and a new Forward/Backward/Forward (FBF) algorithm that extends DRed to better handle facts with multiple derivations. Furthermore, we study the worst-case performance of these algorithms and compare the algorithms' behaviour on several examples. Finally, we present the results of an extensive, first-of-a-kind empirical evaluation in which we investigate the robustness and the scaling behaviour of our algorithms. We thus provide important theoretical and practical insights into all three algorithms that will provide invaluable guidance to future implementors of datalog systems.

Bootstrap clustering approaches for organization of data: Application in improving grade separability in cervical neoplasia

Abstract This study introduces a novel technique for the self-organization of large datasets, without prior knowledge on the statistical distribution of data. The particular application of interest concerns the self-organization of diffuse reflectance time-lapse curves, expressing the biomarker uptake-wash out kinetics in cervical epithelium. Dynamic spectral imaging generates one curve per pixel resulting to about 700.000 curves per person examined. It comprises an established technology for the non-invasive diagnosis of precancerous lesions, since various curve profiles represent distinct neoplasia grades. The methodology developed in this study constitutes an effective automatic clustering approach improving the discrimination ability between a number of precancerous and non-precancerous abnormalities. The automatic clustering of such a large number of curves is expected to facilitate early diagnosis and prognosis, as well as to assist the management of cervical neoplasia. The effectiveness of the proposed approach stems from the incorporation of data bootstrapping within the clustering approach and the adoption of appropriate distance metrics for assessing class coherence. Each bootstrap step derives a set of cluster centroids, which are then regrouped into active class centers based on a meta-data clustering step. The proposed methodology searches for hidden characteristics within the processed dataset and reveals additional data structures or subclasses that can be utilized for identifying irregular groups, which are of particular importance in disease modeling and management. More specifically, a hidden class was revealed in cervical neoplasia with significant confidence indicated by the metrics of Silhouette, Calonski Harabasz and Dunn´s indices, standard deviation of minimum distance metrics. The results of this study show that appropriate bootstrap extensions of simple clustering schemes can effectively organize large time-series data, giving rise to exploratory approaches for subclass identification that facilitate accurate and early disease diagnosis.

A generic approach for integrity verification of big data

Data integrity verification is a primary issue for the prominent and unchallenged success of Big Data. The large scale of data includes extra security overheads on customers that used Big Data services. In Big Data environment, many security challenges have pinched out such as data security, malicious insider attack, cyber attack, and abuse of data. In this paper, we have analyzed and identified a novel data integrity verification technique that provides better performance, efficiency, and quick review of Big Data. The milestone of our work are algebraic signature, homomorphic tag, and Combinatorial Batch Codes (CBC). Homomorphic tag delegates a special verifiable value to each data blocks. CBC allocates integral data to store on Big Data servers. This research work used multiple third-party auditors for batch auditing which ensure data integrity in a Big Data environment. Our approach handles single point failure error of single Third Party Auditor (TPA) efficiently. Without using any additional data structure, it supports dynamic data operations. The security and performance analysis with comparative results show the genuine application of our approach within the latest Big Data environment.

New database compression assists in the IBM z14 processor

In this paper, we describe how the IBM z14 processor, together with Db2 for z/OS Version 12, can improve data compression rates and thus reduce data storage requirements and cost for large databases. The new processor improves on the compression hardware accelerator available in earlier IBM Z generations by adding new hardware algorithms that increase the compression ratio and extend the applicability to additional data structures in Db2 for z/OS databases. A new entropy coding step employed after Ziv–Lempel compression reduces the size of data compressed with the prior algorithms by 30% on average. Moreover, the new order-preserving compression algorithm enables database index compression, reducing index sizes by roughly 30%. This results in an overall improvement of 30% of the database size for many applications, with associated benefits in storage requirements, input/output bandwidth, and buffer pool efficiency.

Generalized enhanced suffix array construction in external memory

BackgroundSuffix arrays, augmented by additional data structures, allow solving efficiently many string processing problems. The external memory construction of the generalized suffix array for a string collection is a fundamental task when the size of the input collection or the data structure exceeds the available internal memory.ResultsIn this article we present and analyze mathsf {eGSA} [introduced in CPM (External memory generalized suffix and mathsf {LCP} arrays construction. In: Proceedings of CPM. pp 201–10, 2013)], the first external memory algorithm to construct generalized suffix arrays augmented with the longest common prefix array for a string collection. Our algorithm relies on a combination of buffers, induced sorting and a heap to avoid direct string comparisons. We performed experiments that covered different aspects of our algorithm, including running time, efficiency, external memory access, internal phases and the influence of different optimization strategies. On real datasets of size up to 24 GB and using 2 GB of internal memory, mathsf {eGSA} showed a competitive performance when compared to mathsf {eSAIS} and mathsf {SAscan}, which are efficient algorithms for a single string according to the related literature. We also show the effect of disk caching managed by the operating system on our algorithm.ConclusionsThe proposed algorithm was validated through performance tests using real datasets from different domains, in various combinations, and showed a competitive performance. Our algorithm can also construct the generalized Burrows-Wheeler transform of a string collection with no additional cost except by the output time.

Joint models for mixed categorical outcomes: a study of HIV risk perception and disease status in Mozambique

ABSTRACTTwo types of bivariate models for categorical response variables are introduced to deal with special categories such as ‘unsure’ or ‘unknown’ in combination with other ordinal categories, while taking additional hierarchical data structures into account. The latter is achieved by the use of different covariance structures for a trivariate random effect. The models are applied to data from the INSIDA survey, where interest goes to the effect of covariates on the association between HIV risk perception (quadrinomial with an ‘unknown risk’ category) and HIV infection status (binary). The final model combines continuation-ratio with cumulative link logits for the risk perception, together with partly correlated and partly shared trivariate random effects for the household level. The results indicate that only age has a significant effect on the association between HIV risk perception and infection status. The proposed models may be useful in various fields of application such as social and biomedical sciences, epidemiology and public health.

On Frequency Estimation and Detection of Frequent Items in Time Faded Streams

We deal with the problem of detecting frequent items in a stream under the constraint that items are weighted, and recent items must be weighted more than older ones. This kind of problem naturally arises in a wide class of applications in which recent data is considered more useful and valuable with regard to older, stale data. The weight assigned to an item is, therefore, a function of its arrival timestamp. As a consequence, whilst in traditional frequent item mining applications we need to estimate frequency counts, we are instead required to estimate decayed counts . These applications are said to work in the time fading model. Two sketch-based algorithms for processing time-decayed streams have been recently published independently near the end of 2016. The Filtered Space Saving with Quasi-Heap (FSSQ) algorithm, besides a sketch, also uses an additional data structure called quasi-heap to maintain frequent items. Forward Decay Count-Min Space Saving (FDCMSS), our algorithm, cleverly combines key ideas borrowed from forward decay, the Count-Min sketch and the Space Saving algorithm. Therefore, it makes sense to compare and contrast the two algorithms in order to fully understand their strengths and weaknesses. We show, through extensive experimental results, that FSSQ is better for detecting frequent items than for frequency estimation. The use of the quasi-heap data structure slows down the algorithm owing to the huge number of maintenance operations. Therefore, FSSQ may not be able to cope with high-speed data streams. FDCMSS is better suitable for frequency estimation; moreover, it is extremely fast and can be used in the context of high-speed data streams and for the detection of frequent items as well, since its recall is always greater than 99%, even when using an extremely tiny amount of space. Therefore, FDCMSS proves to be an overall good choice when considering jointly the recall, precision, average relative error and the speed.

Narrowing Support Searching Range in Maintaining Arc Consistency for Solving Constraint Satisfaction Problems

Arc consistency is the most popular filtering technique for solving constraint satisfaction problems. Constraint check plays a central role in establishing arc consistency. In this paper, we propose a method to save constraint checks in maintaining coarse-grained arc consistency during backtracking search for solving the constraint satisfaction problems. We reduce the support searching range by utilizing the information generated by an AC3.1 algorithm at preprocessing step. Compared with the existing maintaining arc consistency (MAC) algorithms, the proposed MAC3 <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">be</sub> algorithm saves constraint checks without maintaining additional data structures at each search tree node. Our experimental results show that MAC3 <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">be</sub> saves both constraint checks and CPU time while solving some benchmark problems.

A Memory Efficient Pattern Matching Scheme for Regular Expressions

Many malicious packets are common and spread over the Internet in recent years when the scale of Internet traffic grows at a rapid speed. Thus, the regular expression matching in Network Intrusion Detection System (NIDS) that supervises network activities needs to be very fast and consume small memory. In this paper, we propose a memory efficient regular expression matching algorithm called Failureless Segmented Finite Automata (FSFA) with an acceptable searching speed. In FSFA, We eliminate Kleene closures by using additional data structures to reduce a large amount of states. We further reduce the transitions by using default state compression technique. Our performance results implemented on a PC software environment show that our scheme only needs 1% of states needed in DFA and 2 to 22% of states needed in JFA.