Edit Distance Research Articles

High throughput edit distance computation on FPGA-based accelerators using HLS

Edit distance is a computational grand challenge problem to quantify the minimum number of editing operations required to modify one string of characters to the other, finding many applications of natural language processing. In recent years, relevant and increasing interest has also emerged from deoxyribonucleic acid (DNA) applications, like Next Generation Sequencing and DNA storage technologies. Both applications share two crucial features: i) the information is coded into the four bases of DNA and ii) the level of operational noise is still high causing errors in the data, requiring inclusion in the workflow of the computation of algorithms such as the edit distance for finding similarities between sequences. To boost this computation many solutions are available in the literature. Among them, the FPGAs are largely used since the data domain of those applications is strings of 4 characters represented as two-bit values, inconveniently fitting the basic data types of ordinary CPUs and GPUs, with additional benefits of providing a high level of parallelism and low processing latency. This contribution presents a computing- and energy-efficient design implementing the edit distance algorithm combining metaprogramming and High-Level Synthesis. We also assess the performance of our design targeting recent FPGA-based accelerators. Our solution uses nearly 90% of FPGA basic-block hardware resources achieving about 90% of computing efficiency delivering a maximum throughput of 16.8 TCUPS and an energy efficiency of 46 Mpair/Joule, enabling the use of FPGAs as a new class of accelerators for High Performance Computing in DNA applications.

Pre-trained models for linking process in data washing machine

Entity Resolution (ER) has been investigated for decades in various domains as a fundamental task in data integration and data quality. The emerging volume of heterogeneously structured data and even unstructured data challenges traditional ER methods. This research mainly focuses on the Data Washing Machine (DWM). The DWM was developed in the NSF DART Data Life Cycle and Curation research theme, which helps to detect and correct certain types of data quality errors automatically. It also performs unsupervised entity resolution to identify duplicate records. However, it uses traditional methods that are driven by algorithmic pattern rules such as Levenshtein Edit Distances and Matrix comparators. The goal of this research is to assess the replacement of rule-based methods with machine learning and deep learning methods to improve the effectiveness of the processes using 18 sample datasets. The DWM has different processes to improve data quality, and we are currently focusing on working with the scoring and linking processes. To integrate the machine model into the DWM, different pre-trained models were tested to find the one that helps to produce accurate vectors that can be used to calculate the similarity between the records. After trying different pre-trained models, distilroberta was chosen to get the embeddings, and cosine similarity metrics were later used to get the similarity scores, which helped us assess the machine learning model into DWM and gave us closer results to what the scoring matrix is giving. The model performed well and gave closer results overall, and the reason can be that it helped to pick up the important features and helped at the entity matching process.

Train & Constrain: Phonologically Informed Tongue Twister Generation from Topics and Paraphrases

Abstract Previous work in phonologically and phonetically grounded language generation has mainly focused on domains such as puns and poetry. In this article, we present new work on the generation of English tongue twisters - a form of language that is required to be conditioned on a phoneme level to maximize sound overlap, while maintaining semantic consistency with an input topic or phrase and still being grammatically correct. We present TwisterLister, a pipeline for generating phonologically informed tongue twisters from large language models (LLMs) that we use to generate TwistList 2.0, the largest annotated dataset of tongue twisters to date, consisting of 17K+ examples from a combination of human and LLM authors. Our generation pipeline involves the use of a phonologically constrained vocabulary alongside LLM prompting to generate novel, non-derivative tongue twister examples.We additionally present the results of automatic and human evaluation of smaller models trained on our generated dataset to demonstrate the extent to which phonologically motivated language types can be generated without explicit injection of phonological knowledge. Additionally, we introduce a phoneme-aware constrained decoding module (PACD) that can be integrated into an autoregressive language model and demonstrate that this method generates good quality tongue twisters both with and without fine-tuning the underlying language model. We also design and implement a range of automatic metrics for the task of tongue twister generation that is phonologically motivated and captures the unique essence of tongue twisters, primarily based on phonemic edit distance (PED).

Evaluating Sequence Alignment Tools for Antimicrobial Resistance Gene Detection in Assembly Graphs

Antimicrobial resistance (AMR) is an escalating global health threat, often driven by the horizontal gene transfer (HGT) of resistance genes. Detecting AMR genes and understanding their genomic context within bacterial populations is crucial for mitigating the spread of resistance. In this study, we evaluate the performance of three sequence alignment tools—Bandage, SPAligner, and GraphAligner—in identifying AMR gene sequences from assembly and de Bruijn graphs, which are commonly used in microbial genome assembly. Efficiently identifying these genes allows for the detection of neighboring genetic elements and possible HGT events, contributing to a deeper understanding of AMR dissemination. We compare the performance of the tools both qualitatively and quantitatively, analyzing the precision, computational efficiency, and accuracy in detecting AMR-related sequences. Our analysis reveals that Bandage offers the most precise and efficient identification of AMR gene sequences, followed by GraphAligner and SPAligner. The comparison includes evaluating the similarity of paths returned by each tool and measuring output accuracy using a modified edit distance metric. These results highlight Bandage’s potential for contributing to the accurate identification and study of AMR genes in bacterial populations, offering important insights into resistance mechanisms and potential targets for mitigating AMR spread.

BWBEV: A Bitwise Query Processing Algorithm for Approximate Prefix Search

We tackle the challenge of conducting an approximate prefix search within datasets of strings. We explore using a bit-parallelism technique to compute the edit distance between distinct strings and illustrate its adaptation for an approximate prefix search procedure referred to as BWBEV. This technique employs a unary representation of edit vectors alongside bitwise operations to efficiently update these vectors during the edit distance computation. We also show how to apply our new bit-parallelism technique strategy to online edit distance computation between strings without index structure. Our experiments with BWBEV applied to approximate prefix search for a query autocompletion task revealed a substantial acceleration of over 36% when contrasted against state-of-the-art methods.

Entropy formulae on Feldman–Katok metric of random dynamical systems

Feldman–Katok metric is the weakest metric in some sense that makes entropy formulae valid, which can also be considered as the topological counterpart of the edit distance. In this paper, we study Feldman–Katok metric of random dynamical systems and establish topological entropy formula as well as Brin–Katok local entropy formula and Katok entropy formula on Feldman–Katok metric.

Research on 3D geographic entity recognition method based on the double matching degree

Abstract In the process of real-view 3D construction, the description of the same geospatial object by multiple departments is not the same, and how to link the city’s multi-source datasets to the same entity and construct the mapping relationship between multi-source data has become the current difficulty of multi-source data convergence. A double-matching degree entity recognition method was proposed in this paper for the problem of unified geographic entities in data convergence. Firstly, the address information is extracted from the original address data according to the address elements, and the hierarchical address tree is established by using the organization of a dictionary tree for hierarchical matching. The BERT model and twin neural network are introduced to divide the address information on the remaining paths at the byte and word level to calculate the similarity of address information. Then, the similarity of entity names is calculated by the method based on editing distance, and by combining the string-based method and the neural network-based method with each other, the hierarchical step-by-step matching is performed to narrow the matching space, determine the matching entities, and establish the entity links. Experiments on the self-built dataset show that this method has performed better in entity recognition compared to other methods.

Lossless Approximate Pattern Matching: Automated Design of Efficient Search Schemes.

This study introduces a pioneering approach to automate the creation of search schemes for lossless approximate pattern matching. Search schemes are combinatorial structures that define a series of searches over a partitioned pattern. Each search specifies the processing order of these parts and the cumulative lower and upper bounds on the number of errors in each part of the pattern. Together, these searches ensure the identification of all approximate occurrences of a search pattern within a predefined limit of k errors. While existing literature offers designed schemes for up to k = 4 errors, designing search schemes for larger k values incurs escalating computational costs. Our method integrates a greedy algorithm and a novel Integer Linear Programming (ILP) formulation to design efficient search schemes for up to k = 7 errors. Comparative analyses demonstrate the superiority of our ILP-optimal schemes over alternative strategies in both theoretical and practical contexts. Additionally, we propose a dynamic scheme selection technique tailored to specific search patterns, further enhancing efficiency. Combined, this yields runtime reductions of up to 53% for higher k values. To facilitate search scheme generation, we present Hato, an open-source software tool (AGPL-3.0 license) employing the greedy algorithm and utilizing CPLEX for ILP solving. Furthermore, we introduce Columba 1.2, an open-source lossless read-mapper (AGPL-3.0 license) implemented in C++. Columba surpasses existing state-of-the-art tools by identifying all approximate occurrences of 100,000 Illumina reads (150 bp) in the human reference genome within 24 seconds (maximum edit distance of 4) and 75 seconds (maximum edit distance of 6) using a single CPU core. Notably, our study showcases Columba's capability to align 100,000 reads of length 50, with high error rates and up to an edit distance of 7, in a mere 2 hours and 15 minutes. This achievement is unmatched by other lossless aligners, which require over 3 hours for edit distance 5 alignments. Moreover, Columba exhibits a mapping rate four times higher than that of a lossy tool for this dataset.

An open-source fine-tuned large language model for radiological impression generation: a multi-reader performance study

BackgroundThe impression section integrates key findings of a radiology report but can be subjective and variable. We sought to fine-tune and evaluate an open-source Large Language Model (LLM) in automatically generating impressions from the remainder of a radiology report across different imaging modalities and hospitals.MethodsIn this institutional review board-approved retrospective study, we collated a dataset of CT, US, and MRI radiology reports from the University of California San Francisco Medical Center (UCSFMC) (n = 372,716) and the Zuckerberg San Francisco General (ZSFG) Hospital and Trauma Center (n = 60,049), both under a single institution. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) score, an automatic natural language evaluation metric that measures word overlap, was used for automatic natural language evaluation. A reader study with five cardiothoracic radiologists was performed to more strictly evaluate the model’s performance on a specific modality (CT chest exams) with a radiologist subspecialist baseline. We stratified the results of the reader performance study based on the diagnosis category and the original impression length to gauge case complexity.ResultsThe LLM achieved ROUGE-L scores of 46.51, 44.2, and 50.96 on UCSFMC and upon external validation, ROUGE-L scores of 40.74, 37.89, and 24.61 on ZSFG across the CT, US, and MRI modalities respectively, implying a substantial degree of overlap between the model-generated impressions and impressions written by the subspecialist attending radiologists, but with a degree of degradation upon external validation. In our reader study, the model-generated impressions achieved overall mean scores of 3.56/4, 3.92/4, 3.37/4, 18.29 s,12.32 words, and 84 while the original impression written by a subspecialist radiologist achieved overall mean scores of 3.75/4, 3.87/4, 3.54/4, 12.2 s, 5.74 words, and 89 for clinical accuracy, grammatical accuracy, stylistic quality, edit time, edit distance, and ROUGE-L score respectively. The LLM achieved the highest clinical accuracy ratings for acute/emergent findings and on shorter impressions.ConclusionsAn open-source fine-tuned LLM can generate impressions to a satisfactory level of clinical accuracy, grammatical accuracy, and stylistic quality. Our reader performance study demonstrates the potential of large language models in drafting radiology report impressions that can aid in streamlining radiologists’ workflows.

Elastic-Degenerate String Matching with 1 Error or Mismatch

An elastic-degenerate (ED) string is a sequence of n finite sets of strings of total length N, introduced to represent a set of related DNA sequences, also known as a pangenome. The ED string matching (EDSM) problem consists in reporting all occurrences of a pattern of length m in an ED text. The EDSM problem has recently received some attention by the combinatorial pattern matching community, culminating in an O~(nmω-1)+O(N)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {\ ilde{O}}(nm^{\\omega -1})+\\mathcal {O}(N)$$\\end{document}-time algorithm [Bernardini et al., SIAM J. Comput. 2022], where ω\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\omega $$\\end{document} denotes the matrix multiplication exponent and the O~(·)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {\ ilde{O}}(\\cdot )$$\\end{document} notation suppresses polylog factors. In the k-EDSM problem, the approximate version of EDSM, we are asked to report all pattern occurrences with at most k errors. k-EDSM can be solved in O(k2mG+kN)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O}(k^2mG+kN)$$\\end{document} time, under edit distance, or O(kmG+kN)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O}(kmG+kN)$$\\end{document} time, under Hamming distance, where G denotes the total number of strings in the ED text [Bernardini et al., Theor. Comput. Sci. 2020]. Unfortunately, G is only bounded by N, and so even for k=1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$k=1$$\\end{document}, the existing algorithms run in Ω(mN)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varOmega (mN)$$\\end{document} time in the worst case. In this paper we make progress in this direction. We show that 1-EDSM can be solved in O((nm2+N)logm)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O}((nm^2 + N)\\log m)$$\\end{document} or O(nm3+N)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O}(nm^3 + N)$$\\end{document} time under edit distance. For the decision version of the problem, we present a faster O(nm2logm+Nloglogm)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O}(nm^2\\sqrt{\\log m} + N\\log \\log m)$$\\end{document}-time algorithm. We also show that 1-EDSM can be solved in O(nm2+Nlogm)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mathcal {O}(nm^2 + N\\log m)$$\\end{document} time under Hamming distance. Our algorithms for edit distance rely on non-trivial reductions from 1-EDSM to special instances of classic computational geometry problems (2d rectangle stabbing or 2d range emptiness), which we show how to solve efficiently. In order to obtain an even faster algorithm for Hamming distance, we rely on employing and adapting the k-errata trees for indexing with errors [Cole et al., STOC 2004]. This is an extended version of a paper presented at LATIN 2022.

Understanding the effectiveness of automated feedback: Using process data to uncover the role of behavioral engagement

In the last couple of years, feedback research has shifted towards a feedback-as-process approach, taking a learner-centered perspective and focusing on the proactive role of the learner in feedback effectiveness. Process measures can provide new insights into the role of the learner by making learners’ actual behavioral engagement visible. We conducted an experimental study, comparing two groups (feedback vs. no feedback) of English-as-a-foreign-language learners in lower secondary schools (N = 189). The learners completed a writing task and revised it with or without feedback. A second writing task served as a transfer task. Performance was automatically assessed using a scoring algorithm. To determine the level of learners’ behavioral engagement during the text revision, we used the revision time and the edit distance (i.e., a similarity measure) as behavioral measures. Our analyses showed a positive effect of feedback on text revision. We found a full mediation of the effect of feedback on text revision through revision time with an estimated portion of mediation (POM) of .63∗∗∗ and a partial mediation of the feedback effect on text revision through the edit distance with a POM of .30∗∗. We did not find significant mediation effects of either engagement variable regarding performance in a transfer task. Our findings contribute to the understanding of feedback effectiveness, highlighting the central role of learner engagement in the feedback process.

Scene Chinese Recognition with Local and Global Attention

Recently, scene Chinese recognition has attracted increasing attention. While mainstream scene text recognition methods exhibit outstanding performance in English recognition, they are considerably limited in Chinese recognition, due to inter-class similarity, intra-class variability, and complex combination of components in scene Chinese text. In this paper, we design Adaptive Position Encoding(APE) to enhance the model’s ability to perceive spatial information. Based on APE, we have innovatively designed Local Attention Module (LAM) and Global Attention Module (GAM). Specifically, LAM captures local features to identify common characteristics among characters of the same category, addressing the issue of intra-class variability. Meanwhile, LAM captures global features to identify the subordination relationships of Chinese character components. By integrating LAM and GAM, combining both local and global features, it is possible to find differences in the details among features that are fundamentally similar, thus solving the problem of inter-class similarity. Further, we contrive the transformer encoder–decoder structure to identify the vast variety of Chinese characters. Based on the Local/Global Attention Module and transformer encoder–decoder framework, we devise the novel sequence-to-sequence Local and Global Attention Network(LGANet), where both the backbone and the encoder/decoder are composed of attention mechanisms. Subsequent experiments on the Chinese scene dataset show that the recognition accuracy of our proposed LGANet is 77.3% and the normalized editing distance is 88.6%, both of which achieve the SOTA results in Fig. 1.

Video text tracking with transformer-based local search

Video text tracking is a highly significant branch in the field of multi-object tracking (MOT), aiming to detect all text instances in video frames and construct trajectories for each text. Conventional text tracking methods typically follow the Tracking-By-Detection (TbD) paradigm which involves two separate steps of detection and association. In complex scenarios, abundant detection omission of text instances is caused by exposure, occlusion, motion blur, and the like, leading to text trajectory breakpoints in the TbD methods. To this end, video text tracking with transformer-based local search (LSTrack) is proposed in this paper. We utilize historical trajectory information to estimate the approximate search areas where omitted texts are located. In this case, our local search tracker leverages the text images in historical trajectories as references to directly recall text instances in the search areas. In addition, our tracking framework can combine the explicit semantic information output from an OCR model to obtain the semantic version (SV) of LSTrack, which for the first time uses the text edit distance as the distance measurement in matching stages to achieve better text tracking results. Ultimately, our method achieves an advanced performance on several public benchmarks. In particular, LSTrack (SV) achieves state-of-the-art performance on the Minetto benchmark.

On Computing the k -Shortcut Fréchet Distance

The Fréchet distance is a popular measure of dissimilarity for polygonal curves. It is defined as a min–max formulation that considers all orientation-preserving bijective mappings between the two curves. Because of its susceptibility to noise, Driemel and Har-Peled introduced the shortcut Fréchet distance in 2012, where one is allowed to take shortcuts along one of the curves, similar to the edit distance for sequences. We analyse the parameterised version of this problem, where the number of shortcuts is bounded by a parameter \(k\) . The corresponding decision problem can be stated as follows: Given two polygonal curves \(T\) and \(B\) of at most \(n\) vertices, a parameter \(k\) and a distance threshold \(\delta\) , is it possible to introduce \(k\) shortcuts along \(B\) such that the Fréchet distance of the resulting curve and the curve \(T\) is at most \(\delta\) ? We study this problem for polygonal curves in the plane. We provide a complexity analysis for this problem with the following results: (1) there exists a decision algorithm with running time in \(\mathcal{O}(kn^{2k+2}\log n)\) ; (2) assuming the exponential-time hypothesis (ETH), there exists no algorithm with running time bounded by \(n^{o(k)}\) . In contrast, we also show that efficient approximate decider algorithms are possible, even when \(k\) is large. We present a \((3+\varepsilon)\) -approximate decider algorithm with running time in \(\mathcal{O}(kn^{2}\log^{2}n)\) for fixed \(\varepsilon\) . In addition, we can show that, if \(k\) is a constant and the two curves are \(c\) -packed for some constant \(c\) , then the approximate decider algorithm runs in near-linear time.

Attention-Based Deep Spiking Neural Networks for Temporal Credit Assignment Problems.

The temporal credit assignment (TCA) problem, which aims to detect predictive features hidden in distracting background streams, remains a core challenge in biological and machine learning. Aggregate-label (AL) learning is proposed by researchers to resolve this problem by matching spikes with delayed feedback. However, the existing AL learning algorithms only consider the information of a single timestep, which is inconsistent with the real situation. Meanwhile, there is no quantitative evaluation method for TCA problems. To address these limitations, we propose a novel attention-based TCA (ATCA) algorithm and a minimum editing distance (MED)-based quantitative evaluation method. Specifically, we define a loss function based on the attention mechanism to deal with the information contained within the spike clusters and use MED to evaluate the similarity between the spike train and the target clue flow. Experimental results on musical instrument recognition (MedleyDB), speech recognition (TIDIGITS), and gesture recognition (DVS128-Gesture) show that the ATCA algorithm can reach the state-of-the-art (SOTA) level compared with other AL learning algorithms.

Stability from graph symmetrization arguments in generalized Turán problems

AbstractGiven graphs and , denotes the largest number of copies of in ‐free ‐vertex graphs. Let . We say that is F‐Turán‐stable if the following holds. For any there exists such that if an ‐vertex ‐free graph contains at least copies of , then the edit distance of and the ‐partite Turán graph is at most . We say that is weakly F‐Turán‐stable if the same holds with the Turán graph replaced by any complete ‐partite graph . It is known that such stability implies exact results in several cases. We show that complete multipartite graphs with chromatic number at most are weakly ‐Turán‐stable. Partly answering a question of Morrison, Nir, Norin, Rzażewski, and Wesolek positively, we show that for every graph , if is large enough, then is ‐Turán‐stable. Finally, we prove that if is bipartite, then it is weakly ‐Turán‐stable for large enough.

GraphSlimmer: Preserving Read Mappability with the Minimum Number of Variants.

Modern genomic datasets, like those generated under the 1000 Genome Project, contain millions of variants belonging to known haplotypes. Although these datasets are more representative than a single reference sequence and can alleviate issues like reference bias, they are significantly more computationally burdensome to work with, often involving large-indexed genome graph data structures for tasks such as read mapping. The construction, preprocessing, and mapping algorithms can require substantial computational resources depending on the size of these variant sets. Moreover, the accuracy of mapping algorithms has been shown to decrease when working with complete variant sets. Therefore, a drastically reduced set of variants that preserves important properties of the original set is desirable. This work provides a technique for finding a minimal subset of variants such that for given parameters α and δ, all substrings up to length α in the haplotypes are guaranteed to be still alignable to the appropriate locations with either Hamming or edit distance at most δ, using only . Our contributions include showing the NP-hardness and inapproximability of these optimization problems and providing Integer Linear Programming (ILP) formulations. Our edit distance ILP formulation carefully decomposes the problem according to variant locations, which allows it to scale to support all of chromosome 22's variants from the 1000 Genome Project. Our experiments also demonstrate a significant reduction in the number of variants. For example, for moderately long reads, e.g., α = 1000, over 75% of the variants can be removed while preserving read mappability with edit distance at most one.

Learning locality-sensitive bucketing functions.

Many tasks in sequence analysis ask to identify biologically related sequences in a large set. The edit distance, being a sensible model for both evolution and sequencing error, is widely used in these tasks as a measure. The resulting computational problem-to recognize all pairs of sequences within a small edit distance-turns out to be exceedingly difficult, since the edit distance is known to be notoriously expensive to compute and that all-versus-all comparison is simply not acceptable with millions or billions of sequences. Among many attempts, we recently proposed the locality-sensitive bucketing (LSB) functions to meet this challenge. Formally, a (d1,d2)-LSB function sends sequences into multiple buckets with the guarantee that pairs of sequences of edit distance at most d1 can be found within a same bucket while those of edit distance at least d2 do not share any. LSB functions generalize the locality-sensitive hashing (LSH) functions and admit favorable properties, with a notable highlight being that optimal LSB functions for certain (d1,d2) exist. LSB functions hold the potential of solving above problems optimally, but the existence of LSB functions for more general (d1,d2) remains unclear, let alone constructing them for practical use. In this work, we aim to utilize machine learning techniques to train LSB functions. With the development of a novel loss function and insights in the neural network structures that can potentially extend beyond this specific task, we obtained LSB functions that exhibit nearly perfect accuracy for certain (d1,d2), matching our theoretical results, and high accuracy for many others. Comparing to the state-of-the-art LSH method Order Min Hash, the trained LSB functions achieve a 2- to 5-fold improvement on the sensitivity of recognizing similar sequences. An experiment on analyzing erroneous cell barcode data is also included to demonstrate the application of the trained LSB functions. The code for the training process and the structure of trained models are freely available at https://github.com/Shao-Group/lsb-learn.

Median and small parsimony problems on RNA trees.

Noncoding RNAs (ncRNAs) express their functions by adopting molecular structures. Specifically, RNA secondary structures serve as a relatively stable intermediate step before tertiary structures, offering a reliable signature of molecular function. Consequently, within an RNA functional family, secondary structures are generally more evolutionarily conserved than sequences. Conversely, homologous RNA families grouped within an RNA clan share ancestors but typically exhibit structural differences. Inferring the evolution of RNA structures within RNA families and clans is crucial for gaining insights into functional adaptations over time and providing clues about the Ancient RNA World Hypothesis. We introduce the median problem and the small parsimony problem for ncRNA families, where secondary structures are represented as leaf-labeled trees. We utilize the Robinson-Foulds (RF) tree distance, which corresponds to a specific edit distance between RNA trees, and a new metric called the Internal-Leafset (IL) distance. While the RF tree distance compares sets of leaves descending from internal nodes of two RNA trees, the IL distance compares the collection of leaf-children of internal nodes. The latter is better at capturing differences in structural elements of RNAs than the RF distance, which is more focused on base pairs. We also consider a more general tree edit distance that allows the mapping of base pairs that are not perfectly aligned. We study the theoretical complexity of the median problem and the small parsimony problem under the three distance metrics and various biologically relevant constraints, and we present polynomial-time maximum parsimony algorithms for solving some versions of the problems. Our algorithms are applied to ncRNA families from the RFAM database, illustrating their practical utility. https://github.com/bmarchand/rna\_small\_parsimony.

Predicting redox potentials by graph-based machine learning methods.

The evaluation of oxidation and reduction potentials is a pivotal task in various chemical fields. However, their accurate prediction by theoretical computations, which is a complementary task and sometimes the only alternative to experimental measurement, may be often resource-intensive and time-consuming. This paper addresses this challenge through the application of machine learning techniques, with a particular focus on graph-based methods (such as graph edit distances, graph kernels, and graph neural networks) that are reviewed to enlighten their deep links with theoretical chemistry. To this aim, we establish the ORedOx159 database, a comprehensive, homogeneous (with reference values stemming from density functional theory calculations), and reliable resource containing 318 one-electron reduction and oxidation reactions and featuring 159 large organic compounds. Subsequently, we provide an instructive overview of the good practice in machine learning and of commonly utilized machine learning models. We then assess their predictive performances on the ORedOx159 dataset through extensive analyses. Our simulations using descriptors that are computed in an almost instantaneous way result in a notable improvement in prediction accuracy, with mean absolute error (MAE) values equal to 5.6 kcal mol for reduction and 7.2 kcal mol for oxidation potentials, which paves a way toward efficient in silico design of new electrochemical systems.