Common Substring Research Articles

Common substring with shifts in b-ary expansions

Common substring with shifts in b-ary expansions

Quantum Speed-Ups for String Synchronizing Sets, Longest Common Substring, and k -mismatch Matching

Longest common substring (LCS) is an important text processing problem, which has recently been investigated in the quantum query model. The decision version of this problem, LCS with threshold \(d\) , asks whether two length- \(n\) input strings have a common substring of length \(d\) . The two extreme cases, \(d=1\) and \(d=n\) , correspond, respectively to Element Distinctness and Unstructured Search, two fundamental problems in quantum query complexity. However, the intermediate case \(1\ll d\ll n\) was not fully understood. We show that the complexity of LCS with threshold \(d\) smoothly interpolates between the two extreme cases up to \(n^{o(1)}\) factors: — LCS with threshold \(d\) has a quantum algorithm in \(n^{2/3+o(1)}/d^{1/6}\) query complexity and time complexity, and requires at least \(\Omega(n^{2/3}/d^{1/6})\) quantum query complexity. Our result improves upon previous upper bounds \(\widetilde{O}(\min\{n/d^{1/2},n^{2/3}\})\) (Le Gall and Seddighin ITCS 2022, Akmal and Jin SODA 2022), and answers an open question of Akmal and Jin. Our main technical contribution is a quantum speed-up of the powerful String Synchronizing Set technique introduced by Kempa and Kociumaka (STOC 2019). It consistently samples \(n/\tau^{1-o(1)}\) synchronizing positions in the string depending on their length- \(\Theta(\tau)\) contexts, and each synchronizing position can be reported by a quantum algorithm in \(\widetilde{O}(\tau^{1/2+o(1)})\) time. Our quantum string synchronizing set also yields a near-optimal LCE data structure in the quantum setting. As another application of our quantum string synchronizing set, we study the \(k\) -mismatch Matching problem, which asks if the pattern has an occurrence in the text with at most \(k\) Hamming mismatches. Using a structural result of Charalampopoulos et al. (FOCS 2020), we obtain: — \(k\) -mismatch matching has a quantum algorithm with \(k^{3/4}n^{1/2+o(1)}\) query complexity and \(\widetilde{O}(kn^{1/2})\) time complexity. We also observe a non-matching quantum query lower bound of \(\Omega(\sqrt{kn})\) .

An average-case efficient two-stage algorithm for enumerating all longest common substrings of minimum length between genome pairs.

A problem extension of the longest common substring (LCS) between two texts is the enumeration of all LCSs given a minimum length (ALCS- ), along with their positions in each text. In bioinformatics, an efficient solution to the ALCS- for very long texts -genomes or metagenomes- can provide useful insights to discover genetic signatures responsible for biological mechanisms. The ALCS- problem has two additional requirements compared to the LCS problem: one is the minimum length , and the other is that all common strings longer than must be reported. We present an efficient, two-stage ALCS- algorithm exploiting the spectrum of text substrings of length ( -mers). Our approach yields a worst-case time complexity loglinear in the number of -mers for the first stage, and an average-case loglinear in the number of common -mers for the second stage (several orders of magnitudes smaller than the total -mer spectrum). The space complexity is linear in the first phase (disk-based), and on average linear in the second phase (disk- and memory-based). Tests performed on genomes for different organisms (including viruses, bacteria and animal chromosomes) show that run times are consistent with our theoretical estimates; further, comparisons with MUMmer4 show an asymptotic advantage with divergent genomes.

WAS IT A MATch I SAW? Approximate palindromes lead to overstated false match rates in benchmarks using reversed sequences.

Software for labeling biological sequences typically produces a theory-based statistic for each match (the E-value) that indicates the likelihood of seeing that match's score by chance. E-values accurately predict false match rate for comparisons of random (shuffled) sequences, and thus provide a reasoned mechanism for setting score thresholds that enable high sensitivity with low expected false match rate. This threshold-setting strategy is challenged by real biological sequences, which contain regions of local repetition and low sequence complexity that cause excess matches between non-homologous sequences. Knowing this, tool developers often develop benchmarks that use realistic-seeming decoy sequences to explore empirical tradeoffs between sensitivity and false match rate. A recent trend has been to employ reversed biological sequences as realistic decoys, because these preserve the distribution of letters and the existence of local repeats, while disrupting the original sequence's functional properties. However, we and others have observed that sequences appear to produce high scoring alignments to their reversals with surprising frequency, leading to overstatement of false match risk that may negatively affect downstream analysis. We demonstrate that an alignment between a sequence S and its (possibly mutated) reversal tends to produce higher scores than alignment between truly unrelated sequences, even when S is a shuffled string with no notable repetitive or low-complexity regions. This phenomenon is due to the unintuitive fact that (even randomly shuffled) sequences contain palindromes that are on average longer than the longest common substrings (LCS) shared between permuted variants of the same sequence. Though the expected palindrome length is only slightly larger than the expected LCS, the distribution of alignment scores involving reversed sequences is strongly right-shifted, leading to greatly increased frequency of high-scoring alignments to reversed sequences. Overestimates of false match risk can motivate unnecessarily high score thresholds, leading to potentially reduced true match sensitivity. Also, when tool sensitivity is only reported up to the score of the first matched decoy sequence, a large decoy set consisting of reversed sequences can obscure sensitivity differences between tools. As a result of these observations, we advise that reversed biological sequences be used as decoys only when care is taken to remove positive matches in the original (un-reversed) sequences, or when overstatement of false labeling is not a concern. Though the primary focus of the analysis is on sequence annotation, we also demonstrate that the prevalence of internal palindromes may lead to an overstatement of the rate of false labels in protein identification with mass spectrometry.

Logical Synonymy Construction for Translated Texts Based on Edge Technology

Abstract This paper combines edge technology and communication transmission control technology to characterize the translation logical synonymity of machine-automatically converted translated text, the mapping results of logical synonymity of machine-automatically converted translated text, and construct the logical synonymity model of machine-automatically translated text. Starting from the classification of synonym identification, the similarity algorithm is used to calculate the editing distance and the longest common substring of the synonymous terms of the translated text, and according to the translated text, the synonym matching items are extracted from an arbitrary text collection, and the pairing determines whether there is any synonymity in it. The analysis of data analysis involves analyzing the logical synonymity features of translated texts that use edge technology and Chinese-English and English-Chinese metaphor translation. The results show that the lexical distribution of the translated corpus based on edge technology is basically the same as that of the original Chinese language, except for the gap in the use of nouns (3.35%), verbs (5.19%), adjectives (3.21%), and pronouns (0.30%), the rest of the lexical distributions have very small gaps, and thus show an obvious trend of paradigm, which further confirms that the study of logical synonymy of text based on edge technology is Feasibility. Both explicit metaphors and borrowed metaphors are more than 20% higher in the proportion of direct translations from English to Chinese than from Chinese to English, indicating that the direct translation method is more frequently used in English to Chinese translation. This study contributes to the creation of a linguistic and culturally diverse situation and positively influences the translation process.

A cross-guidance cross-lingual model on generated parallel corpus for classical Chinese machine reading comprehension

Chinese diachronic gap is a key issue in classical Chinese machine reading comprehension (CCMRC). Preceding work on bridging this gap has been mostly restricted to limited monolingual classical Chinese corpora pre-training and lexical knowledge integration, which require a great deal of human resources. In this paper, we propose a cross-guidance cross-lingual model (CGCLM), pre-trained on a classical and modern Chinese parallel corpus generated from a large language model, to bridge the Chinese diachronic gap and reduce the manual effort. The CGCLM facilitates accurate translation by providing in-context examples and feedback based on the longest common substring between source and target sentences, thereby avoiding untranslated Chinese words. Specifically, we consider three pre-training tasks, i.e., cross-masked language modeling, linguistic label cross-prediction, and semantic cross-aware translation language modeling. The knowledge acquired from masked tokens uncovering and linguistic label predicting can lead to the implicit semantic alignment between two language styles. Taking advantage of the semantic similarity between the same syntactic levels of parallel pairs, cross-aware modeling integrates and transmits contextualized semantic information. We utilize an 18.6G monolingual corpus to create a 37.2G parallel corpus. Manual evaluation has resulted in only acceptable discrepancies between our generated and human-edited parallel corpora. Extensive experimental results show that our proposed model outperforms the state-of-the-art by an average accuracy of 3.13%, 2.44%, and 2.17% on CCMRC, classical Chinese language understanding evaluation (CCLUE), and modern Chinese language understanding evaluation (MCLUE) tasks.

Orbits closeness for slowly mixing dynamical systems

AbstractGiven a dynamical system, we prove that the shortest distance between two n-orbits scales like n to a power even when the system has slow mixing properties, thus building and improving on results of Barros, Liao and the first author [On the shortest distance between orbits and the longest common substring problem. Adv. Math.344 (2019), 311–339]. We also extend these results to flows. Finally, we give an example for which the shortest distance between two orbits has no scaling limit.

Longest common substring in Longest Common Subsequence’s solution service: A novel hyper-heuristic

The Longest Common Subsequence (LCS) is the problem of finding a subsequence among a set of strings that has two properties of being common to all and the longest. The LCS has applications in computational biology and text editing, among many others. Due to the NP-hardness of the general longest common subsequence, numerous heuristic algorithms and solvers have been proposed to give the best possible solution for different sets of strings. None of them has the best performance for all types of sets. In addition, there is no method to specify the type of a given set of strings. Besides that, the available hyper-heuristic is not efficient and fast enough to solve this problem in real-world applications. This paper proposes a novel hyper-heuristic to solve the longest common subsequence problem using a new criterion to classify a set of strings based on their similarity. To do this, we offer a general stochastic framework to identify the type of a given set of strings. Following that, we introduce the set similarity dichotomizer (S2D) algorithm based on the framework that divides the type of sets into two. This algorithm is introduced for the first time in this paper and opens a new way to go beyond the current LCS solvers. Then, we present our proposed hyper-heuristic that exploits the S2D and one of the internal properties of the given strings to choose the best matching heuristic among a set of heuristics. We compare the results on benchmark datasets with the best heuristics and hyper-heuristics. The results show that our proposed dichotomizer (i.e., S2D) can classify datasets with 98% of accuracy. Also, our proposed hyper-heuristic obtains competitive performance in comparison with the best methods and outperforms best hyper-heuristics for uncorrelated datasets in terms of both quality of solutions and run time factors. All supplementary files, including the source codes and datasets, are publicly available on GitHub.11https://github.com/BioinformaticsIASBS/LCS-DSclassification.

Publisher Correction: Longest Common Substring with Approximately k Mismatches

Publisher Correction: Longest Common Substring with Approximately k Mismatches

Diffusion Alignment Coefficient (DAC): A Novel Similarity Metric for Protein-Protein Interaction Network.

Interaction networks can be used to predict the functions of unknown proteins using known interactions and proteins with known functions. Many graph theory or diffusion-based methods have been proposed, using the assumption that the topological properties of a protein in a network are related to its biological function. Here we seek to improve function prediction by finding more similar neighbors with a new diffusion-based alignment technique to overcome the topological information loss of the node. In this study, we introduce the Diffusion Alignment Coefficient (DAC) algorithm, which combines diffusion, longest common subsequence, and longest common substring techniques to measure the similarity of two nodes in protein interaction networks. As a proof of concept, our experiments, conducted on a real PPI networks S.cerevisiae and Homo Sapiens, demonstrated that our method obtained better results than competitors for MIPS and MSigDB Collections hallmark gene set functional categories. This is the first study to develop a measure of node function similarity using alignment to consider the positions of nodes in protein-protein interaction networks. According to the experimental results, the use of spatial information belonging to the nodes in the network has a positive effect on the detection of more functionally similar neighboring nodes.

Near-Optimal Quantum Algorithms for String Problems

AbstractWe study quantum algorithms for several fundamental string problems, including Longest Common Substring, Lexicographically Minimal String Rotation, and Longest Square Substring. These problems have been widely studied in the stringology literature since the 1970s, and are known to be solvable by near-linear time classical algorithms. In this work, we give quantum algorithms for these problems with near-optimal query complexities and time complexities. Specifically, we show that: Longest Common Substring can be solved by a quantum algorithm in $$\tilde{O}(n^{2/3})$$ O ~ ( n 2 / 3 ) time, improving upon the recent $$\tilde{O}(n^{5/6})$$ O ~ ( n 5 / 6 ) -time algorithm by Le Gall and Seddighin (in: Proceedings of the 13th innovations in theoretical computer science conference (ITCS 2022), pp 97:1–97:23, 2022. https://doi.org/10.4230/LIPIcs.ITCS.2022.97). Our algorithm uses the MNRS quantum walk framework, together with a careful combination of string synchronizing sets (Kempa and Kociumaka, in: Proceedings of the 51st annual ACM SIGACT symposium on theory of computing (STOC 2019), ACM, pp 756–767, 2019. https://doi.org/10.1145/3313276.3316368) and generalized difference covers. Lexicographically Minimal String Rotation can be solved by a quantum algorithm in $$n^{1/2 + o(1)}$$ n 1 / 2 + o ( 1 ) time, improving upon the recent $$\tilde{O}(n^{3/4})$$ O ~ ( n 3 / 4 ) -time algorithm by Wang and Ying (in: Quantum algorithm for lexicographically minimal string rotation. CoRR, 2020. arXiv:2012.09376). We design our algorithm by first giving a new classical divide-and-conquer algorithm in near-linear time based on exclusion rules, and then speeding it up quadratically using nested Grover search and quantum minimum finding. Longest Square Substring can be solved by a quantum algorithm in $$\tilde{O}(\sqrt{n})$$ O ~ ( n ) time. Our algorithm is an adaptation of the algorithm by Le Gall and Seddighin (2022) for the Longest Palindromic Substring problem, but uses additional techniques to overcome the difficulty that binary search no longer applies. Our techniques naturally extend to other related string problems, such as Longest Repeated Substring, Longest Lyndon Substring, and Minimal Suffix.

Optimization of designing multiple genes encoding the same protein based on NSGA-II for efficient execution on GPUs

<abstract><p>In synthetic biology, it is a challenge to increase the production of target proteins by maximizing their expression levels. In order to augment expression levels, we need to focus on both homologous recombination and codon adaptation, which are estimated by three objective functions, namely HD (Hamming distance), LRCS (length of repeated or common substring) and CAI (codon adaptation index). Optimizing these objective functions simultaneously becomes a multi-objective optimization problem. The aim is to find satisfying solutions that have high codon adaptation and a low incidence of homologous recombination. However, obtaining satisfactory solutions requires calculating the objective functions multiple times with many cycles and solutions. In this paper, we propose an approach to accelerate the method of designing a set of CDSs (CoDing sequences) based on NSGA-II (non-dominated sorting genetic algorithm II) on NVIDIA GPUs. The implementation accelerated by GPUs improves overall performance by 187.5$ \times $ using $ 100 $ cycles and $ 128 $ solutions. Our implementation allows us to use larger solutions and more cycles, leading to outstanding solution quality. The improved implementation provides much better solutions in a similar amount of time compared to other available methods by 1.22$ \times $ improvements in hypervolume. Furthermore, our approach on GPUs also suggests how to efficiently utilize the latest computational resources in bioinformatics. Finally, we discuss the impacts of the number of cycles and the number of solutions on designing a set of CDSs.</p></abstract>

An Algorithm for the Longest Common Subsequence and Substring Problem

In this note, we first introduce a new problem called the longest common subsequence and substring problem. Let X and Y be two strings over an alphabet ∑. The longest common subsequence and substring problem for X and Y is to find the longest string, which is a subsequence of X and a substring of Y. We propose an algorithm to solve the problem.

Quantum Meets Fine-Grained Complexity: Sublinear Time Quantum Algorithms for String Problems

Longest common substring (LCS), longest palindrome substring (LPS), and Ulam distance (UL) are three fundamental string problems that can be classically solved in near linear time. In this work, we present sublinear time quantum algorithms for these problems along with quantum lower bounds. Our results shed light on a very surprising fact: Although the classic solutions for LCS and LPS are almost identical (via suffix trees), their quantum computational complexities are different. While we give an exact {{tilde{O}}}(sqrt{n}) time algorithm for LPS, we prove that LCS needs at least time tilde{Omega }(n^{2/3}) even for 0/1 strings.

Terminal Waveform Similarity Measurement Method Based on the Improved Dynamic Time Warping Algorithm

The dynamic time warping algorithm (DTW) has problems such as high computational complexity and “ill-conditioned matching.” Aiming at the above two main problems, this paper proposes an improved DTW algorithm for the final wave recording of the primary and secondary deep fusion equipment detection platform. The terminal recorded waveform and the waveform with non-Gaussian noise added as the research object, the two sets of waveforms are divided into frames and windowed, and the short-term energy entropy ratio of the two sets of waveforms is input into the DTW as the test vector. Using the optimal matching paths and distances of the two input vectors, the common substring lengths of the two sets of short-term energy entropy ratio sequences are calculated. Then, we define the optimal matching coefficient and correct the waveform similarity. The experimental data show that the improved DTW algorithm can accurately quantify the similarity between terminal waveforms, which can provide effective data support for the health status assessment of power distribution terminals.

Solving String Problems on Graphs Using the Labeled Direct Product

Suffix trees are an important data structure at the core of optimal solutions to many fundamental string problems, such as exact pattern matching, longest common substring, matching statistics, and longest repeated substring. Recent lines of research focused on extending some of these problems to vertex-labeled graphs, either by using efficient ad-hoc approaches which do not generalize to all input graphs, or by indexing difficult graphs and having worst-case exponential complexities. In the absence of an ubiquitous and polynomial tool like the suffix tree for labeled graphs, we introduce the labeled direct product of two graphs as a general tool for obtaining optimal algorithms in the worst case: we obtain conceptually simpler algorithms for the quadratic problems of string matching (SMLG) and longest common substring (LCSP) in labeled graphs. Our algorithms run in time linear in the size of the labeled product graph, which may be smaller than quadratic for some inputs, and their run-time is predictable, because the size of the labeled direct product graph can be precomputed efficiently. We also solve LCSP on graphs containing cycles, which was left as an open problem by Shimohira et al. in 2011. To show the power of the labeled product graph, we also apply it to solve the matching statistics (MSP) and the longest repeated string (LRSP) problems in labeled graphs. Moreover, we show that our (worst-case quadratic) algorithms are also optimal, conditioned on the Orthogonal Vectors Hypothesis. Finally, we complete the complexity picture around LRSP by studying it on undirected graphs.

Accurate and efficient privacy-preserving string matching

The task of calculating similarities between strings held by different organisations without revealing these strings is an increasingly important problem in areas such as health informatics, national censuses, genomics, and fraud detection. Most existing privacy-preserving string matching approaches are either based on comparing sets of encoded characters allowing only exact matching of encoded strings, or they are aimed at long genomics sequences that have a small alphabet. The set-based privacy-preserving similarity functions that are commonly used to compare name and address strings in the context of privacy-preserving record linkage do not take the positions of sub-strings into account. As a result, two very different strings can potentially be considered as a match leading to wrongly linked records. Furthermore, existing set-based techniques cannot identify the length of the longest common sub-string across two strings. In this paper, we propose two new approaches for accurate and efficient privacy-preserving string matching that provide privacy against various attacks. In the first approach we apply hashing-based encoding on sub-strings (q-grams) to compare sensitive strings, while in the second approach we generate one-bit array from the sub-strings of a string to identify the longest common bit sequences. We evaluate our approaches on several data sets with different types of strings, and validate their privacy, accuracy, and complexity compared to three baseline techniques, showing that they outperform all baselines.

On the Longest Common Cartesian Substring Problem

AbstractA Cartesian tree is associated with a string of numbers and is structured as a heap from which the original string can be recovered. Although Cartesian trees have been introduced 40 years ago, the Cartesian tree matching problem appeared very recently. It consists in finding all substrings of given text, which have the same Cartesian tree as that of a given pattern. In this paper, we address the problem of computing the longest common Cartesian substrings of two strings and present three methods for such problem. Our first method is based on a classical suffix tree construction and solves the problem in randomized linear time and linear space, although the space overhead is quite prohibitive in the case of large strings. Our second solution is based on classical dynamic programming, and our third solution is based on a constructive approach. Both of them run in quadratic worst case time but are more space economical in practice. From our experimental results, it turns out that our second solution runs faster than the standard suffix tree solution for short strings, whereas our third solution is more suitable for large strings, when storing a full suffix tree becomes prohibitive.

SaAlign: Multiple DNA/RNA sequence alignment and phylogenetic tree construction tool for ultra-large datasets and ultra-long sequences based on suffix array

Multiple DNA/RNA sequence alignment is an important fundamental tool in bioinformatics, especially for phylogenetic tree construction. With DNA-sequencing improvements, the amount of bioinformatics data is constantly increasing, and various tools need to be iterated constantly. Mitochondrial genome analyses of multiple individuals and species require bioinformatics software; therefore, their performances need to be optimized. To improve the alignment of ultra-large datasets and ultra-long sequences, we optimized a dynamic programming algorithm using longest common substring methods. Ultra-large test DNA datasets, containing sequences of different lengths, some over 300 kb (kilobase), revealed that the Multiple DNA/RNA Sequence Alignment Tool Based on Suffix Tree (SaAlign) saved time and computational space. It outperformed the existing technical tools, including MAFFT and HAlign-II. For mitochondrial genome datasets having limited numbers of sequences, MAFFT performed the required tasks, but it could not handle ultra-large mitochondrial genome datasets for core dump error. We implement a multiple DNA/RNA sequence alignment tool based on Center Star strategy and use suffix array algorithm to optimize the spatial and time efficiency. Nowadays, whole-genome research and NGS technology are becoming more popular, and it is necessary to save computational resources for laboratories. That software is of great significance in these aspects, especially in the study of the whole-mitochondrial genome of plants.

Identification of missing concepts in biomedical terminologies using sequence-based formal concept analysis

BackgroundAs biomedical knowledge is rapidly evolving, concept enrichment of biomedical terminologies is an active research area involving automatic identification of missing or new concepts. Previously, we prototyped a lexical-based formal concept analysis (FCA) approach in which concepts were derived by intersecting bags of words, to identify potentially missing concepts in the National Cancer Institute (NCI) Thesaurus. However, this prototype did not handle concept naming and positioning. In this paper, we introduce a sequenced-based FCA approach to identify potentially missing concepts, supporting concept naming and positioning.MethodsWe consider the concept name sequences as FCA attributes to construct the formal context. The concept-forming process is performed by computing the longest common substrings of concept name sequences. After new concepts are formalized, we further predict their potential positions in the original hierarchy by identifying their supertypes and subtypes from original concepts. Automated validation via external terminologies in the Unified Medical Language System (UMLS) and biomedical literature in PubMed is performed to evaluate the effectiveness of our approach.ResultsWe applied our sequenced-based FCA approach to all the sub-hierarchies under Disease or Disorder in the NCI Thesaurus (19.08d version) and five sub-hierarchies under Clinical Finding and Procedure in the SNOMED CT (US Edition, March 2020 release). In total, 1397 potentially missing concepts were identified in the NCI Thesaurus and 7223 in the SNOMED CT. For NCI Thesaurus, 85 potentially missing concepts were found in external terminologies and 315 of the remaining 1312 appeared in biomedical literature. For SNOMED CT, 576 were found in external terminologies and 1159 out of the remaining 6647 were found in biomedical literature.ConclusionOur sequence-based FCA approach has shown the promise for identifying potentially missing concepts in biomedical terminologies.