Edit Distance Computation Research Articles

BWBEV: A Bitwise Query Processing Algorithm for Approximate Prefix Search

We tackle the challenge of conducting an approximate prefix search within datasets of strings. We explore using a bit-parallelism technique to compute the edit distance between distinct strings and illustrate its adaptation for an approximate prefix search procedure referred to as BWBEV. This technique employs a unary representation of edit vectors alongside bitwise operations to efficiently update these vectors during the edit distance computation. We also show how to apply our new bit-parallelism technique strategy to online edit distance computation between strings without index structure. Our experiments with BWBEV applied to approximate prefix search for a query autocompletion task revealed a substantial acceleration of over 36% when contrasted against state-of-the-art methods.

1426. Integrating artificial intelligence and dynamic programming to identify microbial clusters through antimicrobial susceptibility sequence analysis

Abstract Background The spread of infectious diseases and antimicrobial resistance poses a significant threat, especially in developing countries where traditional DNA fingerprinting techniques are often not available. We propose a novel approach using dynamic programming algorithms to compare antimicrobial susceptibility sequences and identify clusters of related microbes. Methods We selected pathogens from specific infections and tested them with a range of antimicrobials, recording the test results as either sensitive (S), resistant (R), or unknown (U) to each drug. These results were combined into a sequence, or ATB string, for each pathogen (Fig. 1). To compare the sequences, we used dynamic programming (Fig. 2) to calculate the minimum edit distance between each pair of sequences. The edit distance reflects the number of atomic changes needed to transform one sequence into the other. In this case, atomic changes include insertion, deletion, substitution, and matching of individual test results (Fig. 3). We created a matrix of pairwise similarities by comparing all ATB strings. A cluster of similar strains was identified based on a cutoff in the edit distance matrix, such as a distance of less than 1. The method, which is available in the SACIH+ system (https://plus.sacihweb.com), was used to analyze the same strain of specific healthcare infections from a public hospital in Belo Horizonte, Brazil, diagnosed between Jan 2022 and Mar 2023.Figure 1.Antimicrobial testing and ATB string generation for selected pathogens. Antimicrobial testing and ATB string generation for selected pathogens.Figure 2.Sequence comparison using dynamic programming. Sequence comparison using dynamic programming. Results We analyzed S. aureus strains from surgical site infection - SSI (23 cases), pneumonia - PNEU (29 cases), and bloodstream infection - BSI (13 cases). Identified 4 SSI, 10 PNEU clusters (Fig. 4 and 5), and 4 BSI clusters using edit distance matrix. For K. pneumoniae strains from 21 BSI, 22 PNEU, and 34 urinary tract infection - UTI cases, found 5 clusters for PNEU and UTI, and 7 clusters for BSI (Fig. 6).Figure 3.Minimum edit distance calculation for sequence comparison. Minimum edit distance calculation for sequence comparison.Figure 4.Clustering of Staphylococcus aureus strains from 29 nosocomial pneumonia cases. Clustering of Staphylococcus aureus strains from 29 nosocomial pneumonia cases.Figure 5.Identification of a large cluster with 16 identical Staphylococcus aureus strains from 29 nosocomial pneumonia cases.Identification of a large cluster with 16 identical Staphylococcus aureus strains from 29 nosocomial pneumonia cases. Conclusion Our novel approach using dynamic programming to compare antimicrobial susceptibility sequences has shown promise in identifying clusters of related microorganisms, even without traditional DNA fingerprinting methods. The method represents a valuable tool for tracking the spread of infectious diseases, especially in settings where traditional methods are not available.Figure 6.Clustering of Klebsiella pneumoniae strains from 21 bloodstream infection cases.Clustering of Klebsiella pneumoniae strains from 21 bloodstream infection cases. Disclosures All Authors: No reported disclosures

RAPIDx: High-Performance ReRAM Processing In-Memory Accelerator for Sequence Alignment

Genome sequence alignment is the core of many biological applications. The advancement of sequencing technologies produces a tremendous amount of data, making sequence alignment a critical bottleneck in bioinformatics analysis. The existing hardware accelerators for alignment suffer from limited on-chip memory, costly data movement, and poorly optimized alignment algorithms. They cannot afford to concurrently process the massive amount of data generated by sequencing machines. In this paper, we propose a ReRAM-based accelerator, RAPIDx, using processing in-memory (PIM) for sequence alignment. RAPIDx achieves superior efficiency and performance via software-hardware co-design. First, we propose an adaptive banded parallelism alignment algorithm suitable for PIM architecture. Compared to the original dynamic programming-based alignment, the proposed algorithm significantly reduces the required complexity, data bit width, and memory footprint at the cost of negligible accuracy degradation. Then we propose the efficient PIM architecture that implements the proposed algorithm. The data flow in RAPIDx achieves four-level parallelism and we design an in-situ alignment computation flow in ReRAM, delivering $5.5$-$9.7\times$ efficiency and throughput improvements compared to our previous PIM design, RAPID. The proposed RAPIDx is reconfigurable to serve as a co-processor integrated into existing genome analysis pipeline to boost sequence alignment or edit distance calculation. On short-read alignment, RAPIDx delivers $131.1\times$ and $46.8\times$ throughput improvements over state-of-the-art CPU and GPU libraries, respectively. As compared to ASIC accelerators for long-read alignment, the performance of RAPIDx is $1.8$-$2.9\times$ higher.

Integrating FPGA Acceleration in the DNAssim Framework for Faster DNA-Based Data Storage Simulations

DNA-based data storage emerged in this decade as a promising solution for long data durability, low power consumption, and high density. However, such technology has not yet reached a good maturity level, requiring many investigations to improve the information encoding and decoding processes. Simulations can be key to overcoming the time and the cost burdens of the many experiments imposed by thorough design space explorations. In response to this, we have developed a DNA storage simulator (DNAssim) that allows simulating the different steps in the DNA storage pipeline using a proprietary software infrastructure written in Python/C language. Among the many operations performed by the tool, the edit distance calculation used during clustering operations has been identified as the most computationally intensive task in software, thus calling for hardware acceleration. In this work, we demonstrate the integration in the DNAssim framework of a dedicated FPGA hardware accelerator based on the Xilinx VC707 evaluation kit to boost edit distance calculations by up to 11 times with respect to a pure software approach. This materializes in a clustering simulation latency reduction of up to 5.5 times and paves the way for future scale-out DNA storage simulation platforms.

GPU acceleration of Levenshtein distance computation between long strings

Computing edit distance for very long strings has been hampered by quadratic time complexity with respect to string length. The WFA algorithm reduces the time complexity to a quadratic factor with respect to the edit distance between the strings. This work presents a GPU implementation of the WFA algorithm and a new optimization that can halve the elements to be computed, providing additional performance gains. The implementation allows to address the computation of the edit distance between strings having hundreds of millions of characters. The performance of the algorithm depends on the similarity between the strings. For strings longer than million characters, the performance is the best ever reported, which is above TCUPS for strings with similarities greater than 70% and above one hundred TCUPS for 99.9% similarity.

Efficient graph edit distance computation using isomorphic vertices

In this paper, we study the problem of graph edit distance (GED) computation. We empirically observe that real graph data contain many isomorphic substructures, which incur redundant computation. Based on this observation, we aim at reducing the cost of GED computation by avoiding redundant computation caused by isomorphic substructures. To detect isomorphic substructures, we precisely define a notion of vertex isomorphism and propose a dynamic programming algorithm that identifies isomorphic vertices in a graph. By taking advantage of isomorphic vertices, we develop an efficient GED computation algorithm. In experiments, we show that isomorphic vertices are effectively used in reducing the search space of GED computation, and as a result our approach improves the performance of GED computation.

Toward Efficient Similarity Search under Edit Distance on Hybrid Architectures

Edit distance is the most widely used method to quantify similarity between two strings. We investigate the problem of similarity search under edit distance. Given a collection of sequences, the goal of similarity search under edit distance is to find sequences in the collection that are similar to a given query sequence where the similarity score is computed using edit distance. The canonical method of computing edit distance between two strings uses a dynamic programming-based approach that runs in quadratic time and space, which may not provide results in a reasonable amount of time for large sequences. It advocates for parallel algorithms to reduce the time taken by edit distance computation. To this end, we present scalable parallel algorithms to support efficient similarity search under edit distance. The efficiency and scalability of the proposed algorithms is demonstrated through an extensive set of experiments on real datasets. Moreover, to address the problem of uneven workload across different processing units, which is mainly caused due to the significant variance in the size of the sequences, different data distribution schemes are discussed and empirically analyzed. Experimental results have shown that the speedup achieved by the hybrid approach over inter-task and intra-task parallelism is 18 and 13, respectively.

Enumerating dissimilar minimum cost perfect and error-correcting bipartite matchings for robust data matching

Matchings between objects from two datasets, domains, or ontologies have to be computed in various application scenarios. One often used meta-approach—which we call bipartite data matching—is to leverage domain knowledge for defining costs between the objects that should be matched, and to then use the classical Hungarian algorithm to compute a minimum cost bipartite matching. In this paper, we introduce and study the problem of enumerating K dissimilar minimum cost bipartite matchings. We formalize this problem, prove that it is NP-hard, and present heuristics based on greedy dynamic programming. The presented enumeration techniques are not only interesting in themselves, but also mitigate an often overlooked shortcoming of bipartite data matching, namely, that it is sensitive w.r.t. the storage order of the input data. Extensive experiments show that our enumeration heuristics clearly outperform existing algorithms in terms of dissimilarity of the obtained matchings, that they are effective at rendering bipartite data matching approaches more robust w.r.t. random storage order, and that they significantly improve the upper bounds of state-of-the art algorithms for graph edit distance computation that are based on bipartite data matching.

Equipment Quality Data Integration and Cleaning Based on Multiterminal Collaboration

With the advancement of digital manufacturing technology, data-driven quality management is getting more and more attention, and it is developing rapidly under the impetus of technology and management. Quality data are growing exponentially with the help of increasingly interconnected devices and IoT (Internet of Things technologies). Aiming at the problems of insufficient quality data acquisition and poor data quality of complex equipment, the research on quality data integration and cleaning based on digital total quality management is carried out. The data integration architecture of complex equipment quality based on multiterminal collaboration is constructed. The architecture integrates a variety of integration methods and standards, such as XML, OPC-UA, and QIF protocol. Then, to unify the data view, a cleaning method of complex equipment quality data based on the combination of edit distance and longest common subsequence similarity calculation is proposed, and its effectiveness is verified. It provides the basis for the design of the digital total quality management system of complex equipment.

Similar Supergraph Search Based on Graph Edit Distance

Subgraph and supergraph search methods are promising techniques for the development of new drugs. For example, the chemical structure of favipiravir—an antiviral treatment for influenza—resembles the structure of some components of RNA. Represented as graphs, such compounds are similar to a subgraph of favipiravir. However, the existing supergraph search methods can only discover compounds that match exactly. We propose a novel problem, called similar supergraph search, and design an efficient algorithm to solve it. The problem is to identify all graphs in a database that are similar to any subgraph of a query graph, where similarity is defined as edit distance. Our algorithm represents the set of candidate subgraphs by a code tree, which it uses to efficiently compute edit distance. With a distance threshold of zero, our algorithm is equivalent to an existing efficient algorithm for exact supergraph search. Our experiments show that the computation time increased exponentially as the distance threshold increased, but increased sublinearly with the number of graphs in the database.

Accel-Align: a fast sequence mapper and aligner based on the seed\u2013embed\u2013extend method

BackgroundImprovements in sequencing technology continue to drive sequencing cost towards $100 per genome. However, mapping sequenced data to a reference genome remains a computationally-intensive task due to the dependence on edit distance for dealing with INDELs and mismatches introduced by sequencing. All modern aligners use seed–filter–extend methodology and rely on filtration heuristics to reduce the overhead of edit distance computation. However, filtering has inherent performance–accuracy trade-offs that limits its effectiveness.ResultsMotivated by algorithmic advances in randomized low-distortion embedding, we introduce SEE, a new methodology for developing sequence mappers and aligners. While SFE focuses on eliminating sub-optimal candidates, SEE focuses instead on identifying optimal candidates. To do so, SEE transforms the read and reference strings from edit distance regime to the Hamming regime by embedding them using a randomized algorithm, and uses Hamming distance over the embedded set to identify optimal candidates. To show that SEE performs well in practice, we present Accel-Align an SEE-based short-read sequence mapper and aligner that is 3–12times faster than state-of-the-art aligners on commodity CPUs, without any special-purpose hardware, while providing comparable accuracy.ConclusionsAs sequencing technologies continue to increase read length while improving throughput and accuracy, we believe that randomized embeddings open up new avenues for optimization that cannot be achieved by using edit distance. Thus, the techniques presented in this paper have a much broader scope as they can be used for other applications like graph alignment, multiple sequence alignment, and sequence assembly.

Clinical Term Normalization Using Learned Edit Patterns and Subconcept Matching: System Development and Evaluation.

BackgroundClinical terms mentioned in clinical text are often not in their standardized forms as listed in clinical terminologies because of linguistic and stylistic variations. However, many automated downstream applications require clinical terms mapped to their corresponding concepts in clinical terminologies, thus necessitating the task of clinical term normalization.ObjectiveIn this paper, a system for clinical term normalization is presented that utilizes edit patterns to convert clinical terms into their normalized forms.MethodsThe edit patterns are automatically learned from the Unified Medical Language System (UMLS) Metathesaurus as well as from the given training data. The edit patterns are generalized sequences of edits that are derived from edit distance computations. The edit patterns are both character based as well as word based and are learned separately for different semantic types. In addition to these edit patterns, the system also normalizes clinical terms through the subconcepts mentioned within them.ResultsThe system was evaluated as part of the 2019 n2c2 Track 3 shared task of clinical term normalization. It obtained 80.79% accuracy on the standard test data. This paper includes ablation studies to evaluate the contributions of different components of the system. A challenging part of the task was disambiguation when a clinical term could be normalized to multiple concepts.ConclusionsThe learned edit patterns led the system to perform well on the normalization task. Given that the system is based on patterns, it is human interpretable and is also capable of giving insights about common variations of clinical terms mentioned in clinical text that are different from their standardized forms.

Weighted automata computation of edit distances with consolidations and fragmentations

We study edit distances between strings, based on weighted operations such as character substitutions, insertions, deletions, and consolidations and fragmentations. The two latter operations transform a sequence of characters into one character and vice-versa. They correspond to the compression and expansion in Dynamic Time-Warping algorithms for speech recognition and are used for the formal analysis of written music.We show that such edit distances are not computable in general, and propose weighted automata constructions to compute restricted cases of edit distances, taking into account both consolidations and deletions, or both fragmentations and insertions. Assuming that the operation ruleset has a constant size, these constructions are polynomial into the lengths of the involved strings. We finally show that the optimal weight of sequences made of consolidations chained with fragmentations, in that order, is computable for arbitrary rulesets, and not computable for some rulesets when reversing the order of fragmentations and consolidations.

Learning-Based Efficient Graph Similarity Computation via Multi-Scale Convolutional Set Matching

Graph similarity computation is one of the core operations in many graph-based applications, such as graph similarity search, graph database analysis, graph clustering, etc. Since computing the exact distance/similarity between two graphs is typically NP-hard, a series of approximate methods have been proposed with a trade-off between accuracy and speed. Recently, several data-driven approaches based on neural networks have been proposed, most of which model the graph-graph similarity as the inner product of their graph-level representations, with different techniques proposed for generating one embedding per graph. However, using one fixed-dimensional embedding per graph may fail to fully capture graphs in varying sizes and link structures—a limitation that is especially problematic for the task of graph similarity computation, where the goal is to find the fine-grained difference between two graphs. In this paper, we address the problem of graph similarity computation from another perspective, by directly matching two sets of node embeddings without the need to use fixed-dimensional vectors to represent whole graphs for their similarity computation. The model, Graph-Sim, achieves the state-of-the-art performance on four real-world graph datasets under six out of eight settings (here we count a specific dataset and metric combination as one setting), compared to existing popular methods for approximate Graph Edit Distance (GED) and Maximum Common Subgraph (MCS) computation.

Approximate Graph Edit Distance in Quadratic Time.

Graph edit distance is one of the most flexible and general graph matching models available. The major drawback of graph edit distance, however, is its computational complexity that restricts its applicability to graphs of rather small size. Recently, the authors of the present paper introduced a general approximation framework for the graph edit distance problem. The basic idea of this specific algorithm is to first compute an optimal assignment of independent local graph structures (including substitutions, deletions, and insertions of nodes and edges). This optimal assignment is complete and consistent with respect to the involved nodes of both graphs and can thus be used to instantly derive an admissible (yet suboptimal) solution for the original graph edit distance problem in O(n3) time. For large scale graphs or graph sets, however, the cubic time complexity may still be too high. Therefore, we propose to use suboptimal algorithms with quadratic rather than cubic time for solving the basic assignment problem. In particular, the present paper introduces five different greedy assignment algorithms in the context of graph edit distance approximation. In an experimental evaluation, we show that these methods have great potential for further speeding up the computation of graph edit distance while the approximated distances remain sufficiently accurate for graph based pattern classification.

Distributed Algorithm for Parallel Edit Distance Computation

The edit distance is the measure that quantifies the difference between two strings. It is an important concept because it has its usage in many domains such as natural language processing, spell checking, genome matching, and pattern recognition. Edit distance is also known as Levenshtein distance. Sequentially, the edit distance is computed by using dynamic programming based strategy that may not provide results in reasonable time when input strings are large. In this work, a distributed algorithm is presented for parallel edit distance computation. The proposed algorithm is both time and space efficient. It is evaluated on a hybrid setup of distributed and shared memory systems. Results suggest that the proposed algorithm achieves significant performance gain over the existing parallel approach.

Edit distance computation with minimum number of edit operations in database management system and information retrieval

Edit distance computation with minimum number of edit operations in database management system and information retrieval

NvPD: novel parallel edit distance algorithm, correctness, and performance evaluation

Edit distance has applications in many domains such as bioinformatics, spell checking, plagiarism checking, query optimization, speech recognition, and data mining. Traditionally, edit distance is computed by dynamic programming based sequential solution which becomes infeasible for large problems. In this paper, we introduce NvPD, a novel algorithm for parallel edit distance computation by resolving dependencies in the conventional dynamic programming based solution. We also establish the correctness of modified dependencies. NvPD exhibits certain characteristics such as balanced workload among processors, less synchronization overhead, maximum utilization of resources and it can exploit spatial locality. It requires $$\min (m,n)$$ steps to complete as compared to diagonal based approach that completes in $$\max (m,n)$$ . Experimental evaluation using variety of random and real life data sets over shared memory multi-core systems and graphic processing units (GPUs) show that NvPD outperforms state-of-the-art parallel edit distance algorithms.

Scalable Hardware Mechanism for Partitioned Circuits Operation

For designing hardware with a high-level synthesis tool using a programming language such as C or Java, its large size of logic circuit makes it difficult to implement the design in a single FPGA. In such a case, partitioning the logic circuit and implementing in multiple FPGAs is a commonly used approach.
 We propose the Scalable Hardware Mechanism, which enables the operation of a partitioned circuit to prevent the degradation of clock frequency by minimizing its dependence on the usage and the type of FPGA. Our mechanism provides a reduced delay by the collective signal transmission with the partitioned AES code generation circuit and the character string edit distance calculation circuit as partitioned circuits. The collective signal transmission has attained 1.27 times improvement in the speed for the AES code generation circuit and 3.16 times improvement for the character string edit distance calculation circuit compared with the circuit by the conventional method.

Where No Universal Health Care Identifier Exists: Comparison and Determination of the Utility of Score-Based Persons Matching Algorithms Using Demographic Data.

BackgroundA universal health care identifier (UHID) facilitates the development of longitudinal medical records in health care settings where follow up and tracking of persons across health care sectors are needed. HIV case-based surveillance (CBS) entails longitudinal follow up of HIV cases from diagnosis, linkage to care and treatment, and is recommended for second generation HIV surveillance. In the absence of a UHID, records matching, linking, and deduplication may be done using score-based persons matching algorithms. We present a stepwise process of score-based persons matching algorithms based on demographic data to improve HIV CBS and other longitudinal data systems.ObjectiveThe aim of this study is to compare deterministic and score-based persons matching algorithms in records linkage and matching using demographic data in settings without a UHID.MethodsWe used HIV CBS pilot data from 124 facilities in 2 high HIV-burden counties (Siaya and Kisumu) in western Kenya. For efficient processing, data were grouped into 3 scenarios within (1) HIV testing services (HTS), (2) HTS-care, and (3) within care. In deterministic matching, we directly compared identifiers and pseudo-identifiers from medical records to determine matches. We used R stringdist package for Jaro, Jaro-Winkler score-based matching and Levenshtein, and Damerau-Levenshtein string edit distance calculation methods. For the Jaro-Winkler method, we used a penalty (р)=0.1 and applied 4 weights (ω) to Levenshtein and Damerau-Levenshtein: deletion ω=0.8, insertion ω=0.8, substitutions ω=1, and transposition ω=0.5.ResultsWe abstracted 12,157 cases of which 4073/12,157 (33.5%) were from HTS, 1091/12,157 (9.0%) from HTS-care, and 6993/12,157 (57.5%) within care. Using the deterministic process 435/12,157 (3.6%) duplicate records were identified, yielding 96.4% (11,722/12,157) unique cases. Overall, of the score-based methods, Jaro-Winkler yielded the most duplicate records (686/12,157, 5.6%) while Jaro yielded the least duplicates (546/12,157, 4.5%), and Levenshtein and Damerau-Levenshtein yielded 4.6% (563/12,157) duplicates. Specifically, duplicate records yielded by method were: (1) Jaro 5.7% (234/4073) within HTS, 0.4% (4/1091) in HTS-care, and 4.4% (308/6993) within care, (2) Jaro-Winkler 7.4% (302/4073) within HTS, 0.5% (6/1091) in HTS-care, and 5.4% (378/6993) within care, (3) Levenshtein 6.4% (262/4073) within HTS, 0.4% (4/1091) in HTS-care, and 4.2% (297/6993) within care, and (4) Damerau-Levenshtein 6.4% (262/4073) within HTS, 0.4% (4/1091) in HTS-care, and 4.2% (297/6993) within care.ConclusionsWithout deduplication, over reporting occurs across the care and treatment cascade. Jaro-Winkler score-based matching performed the best in identifying matches. A pragmatic estimate of duplicates in health care settings can provide a corrective factor for modeled estimates, for targeting and program planning. We propose that even without a UHID, standard national deduplication and persons-matching algorithm that utilizes demographic data would improve accuracy in monitoring HIV care clinical cascades.