Common Substring Research Articles

The longest common substring problem

Given a set$\mathcal{D}$ofqdocuments, the Longest Common Substring (LCS) problem asks, for any integer 2 ⩽k⩽q, the longest substring that appears inkdocuments. LCS is a well-studied problem having a wide range of applications in Bioinformatics: from microarrays to DNA sequences alignments and analysis. This problem has been solved by Hui (2000International Journal of Computer Science and Engineering1573–76) by using a famous constant-time solution to the Lowest Common Ancestor (LCA) problem in trees coupled with the use of suffix trees.In this article, we present a simple method for solving the LCS problem by using suffix trees (STs) and classical union-find data structures. In turn, we show how this simple algorithm can be adapted in order to work with other space efficient data structures such as the enhanced suffix arrays (ESA) and the compressed suffix tree.

네트워크 트래픽 분석을 위한 Snort Content 규칙 자동 생성

효과적인 네트워크 관리를 위해 응용 트래픽 분석의 중요성이 강조되고 있다. Snort는 트래픽 탐지를 위해 사용되는 보편적인 엔진으로써 기 정의된 규칙을 기반으로 트래픽을 차단하거나 로그를 기록한다. 하지만 Snort 규칙을 생성하기 위해서는 탐지 대상 트래픽을 전수 조사해야하기 때문에 많은 한계점이 존재할 뿐만 아니라 생성된 규칙의 정확성을 보장하기 어렵다. 본 논문에서는 순차 패턴 알고리즘을 활용하여 입력된 트래픽에서 최소 지지도를 만족하는 문자열을 찾는 방법을 제안한다. 또한, 추출된 문자열을 사용한 규칙을 입력 트래픽에 적용하여 트래픽에서 해당 문자열이 존재하는 위치 정보 및 헤더 정보를 추출한다. 이렇게 추출된 문자열과 위치정보, 그리고 헤더 정보를 조합하여 Snort 규칙을 자동 생성하는 방법을 제안한다. 생성된 규칙을 이용하여 다시 트래픽 분석을 실시했을 때 대부분의 응용이 97%이상 탐지되는 것을 확인하였다. The importance of application traffic analysis for efficient network management has been emphasized continuously. Snort is a popular traffic analysis system which detects traffic matched to pre-defined signatures and perform various actions based on the rules. However, it is very difficult to get highly accurate signatures to meet various analysis purpose because it is very tedious and time-consuming work to search the entire traffic data manually or semi-automatically. In this paper, we propose a novel method to generate signatures in a fully automatic manner in the form of sort rule from raw packet data captured from network link or end-host. We use a sequence pattern algorithm to generate common substring satisfying the minimum support from traffic flow data. Also, we extract the location and header information of the signature which are the components of snort content rule. When we analyzed the proposed method to several application traffic data, the generated rule could detect more than 97 percentage of the traffic data.

Longest common substrings with k mismatches

The longest common substring with k-mismatches problem is to find, given two strings S1 and S2, a longest substring A1 of S1 and A2 of S2 such that the Hamming distance between A1 and A2 is ≤k. We introduce a practical O(nm) time and O(1) space solution for this problem, where n and m are the lengths of S1 and S2, respectively. This algorithm can also be used to compute the matching statistics with k-mismatches of S1 and S2 in O(nm) time and O(m) space. Moreover, we also present a theoretical solution for the k=1 case which runs in O(nlog⁡m) time, assuming m≤n, and uses O(m) space, improving over the existing O(nm) time and O(m) space bound of Babenko and Starikovskaya [1].

A note on the longest common substring with k-mismatches problem

The recently introduced longest common substring with k-mismatches (k-LCF) problem is to find, given two sequences S1 and S2 of length n each, a longest substring A1 of S1 and A2 of S2 such that the Hamming distance between A1 and A2 is at most k. So far, the only subquadratic time result for this problem was known for k=1[5]. We present two output-dependent algorithms solving the k-LCF problem and show that for k=O(log1−ε⁡n), where ε>0, at least one of them works in subquadratic time, using O(n) words of space. The choice of one of these two algorithms to be applied for a given input can be done after linear time and space preprocessing.

Модель и алгоритм выравнивания последовательностей

The change detection problem is aimed at identifying common and different strings and usually has non-unique solutions. The identification of the best alignment is canonically based on finding a longest common subsequence (LCS) and is widely used for various purposes. However, many recent version control systems prefer alternative heuristic algorithms which not only are faster but also usually produce better alignment than finding an LCS. Two basic shortcomings of known alignment algorithms are outlined in the paper: (1) even when the length of the longest common substring is close to that of the LCS, the latter may consist of a great number of short uninformative substrings; (2) known alternative algorithms start with identifying the most informative common string, which sometimes omits from consideration common subse- quence containing arbitrarily many aligned substrings of similar quality. The sequence alignment problem is considered to be an abstract model for change detection in collaborative text editing designed to minimize the probability of merge conflict. A new cost function is defined as the probability of overlap between detected changes and a random string. This optimization avoids both shortcomings mentioned above. The simple cubic algorithm is proposed.

A Similarity Detection Method Based on Distance Matrix Model with Row-Column Order penalty Factor

Paper detection involves multiple disciplines, and making a comprehensive and correct evaluation of academic misconduct is quite a complex and sensitive issue. There are some problems in the existing main detection models, such as incomplete segmentation preprocessing specification, impact of the semantic orders on detection, near-synonym evaluation, slow paper backtrack and so on. This paper presents a sentence-level paper similarity comparison model with segmentation preprocessing based on special identifier. This model integrates the characteristics of vector detection, hamming distance and the longest common substring and carries out detection specific to near-synonyms, word deletion and changes in word order by redefining distance matrix and adding ordinal measures, making sentence similarity detection in terms of semantics and backbone word segmentation more effective. Compared with the traditional paper similarity retrieval, the present method adopts modular-2 arithmetic with low computation. Paper detection method with reliability and high efficiency is of great academic significance in word segmentation, similarity detection and document summarization.

Malware clustering using suffix trees

Clustering is an important problem in malware research, as the number of malicious samples that appear every day makes manual analysis impractical. Although these samples belong to a limited number of malware families, it is difficult to categorize them automatically as obfuscation is involved. By extracting relevant features we can apply clustering algorithms, then only analyze a couple of representatives from each cluster. However, classic clustering algorithms that compute the similarity between each pair of samples are slow when a large collection is involved. In this paper, the features will be strings of operation codes extracted from the binary code of each sample. With a modified suffix tree data structure we can find long enough substrings that correspond to portions of a program’s code. These substrings must be filtered against a database of known substrings so that common library code will be ignored. The items that have common substrings above a certain threshold will be grouped into the same cluster. Our algorithm was tested with data extracted from real-world malware and constructed quality clusters.

Merging of multi-string BWTs with applications.

The throughput of genomic sequencing has increased to the point that is overrunning the rate of downstream analysis. This, along with the desire to revisit old data, has led to a situation where large quantities of raw, and nearly impenetrable, sequence data are rapidly filling the hard drives of modern biology labs. These datasets can be compressed via a multi-string variant of the Burrows-Wheeler Transform (BWT), which provides the side benefit of searches for arbitrary k-mers within the raw data as well as the ability to reconstitute arbitrary reads as needed. We propose a method for merging such datasets for both increased compression and downstream analysis. We present a novel algorithm that merges multi-string BWTs in [Formula: see text] time where LCS is the length of their longest common substring between any of the inputs, and N is the total length of all inputs combined (number of symbols) using [Formula: see text] bits where F is the number of multi-string BWTs merged. This merged multi-string BWT is also shown to have a higher compressibility compared with the input multi-string BWTs separately. Additionally, we explore some uses of a merged multi-string BWT for bioinformatics applications.

Paper Similarity Detection Method Based on Distance Matrix Model with Row-Column Order Penalty Factor

Paper similarity detection depends on grammatical and semantic analysis, word segmentation, similarity detection, document summarization and other technologies, involving multiple disciplines. However, there are some problems in the existing main detection models, such as incomplete segmentation preprocessing specification, impact of the semantic orders on detection, near-synonym evaluation, difficulties in paper backtrack and etc. Therefore, this paper presents a two-step segmentation model of special identifier and Sharpley value specific to above problems, which can improve segmentation accuracy. In the aspect of similarity comparison, a distance matrix model with row-column order penalty factor is proposed, which recognizes new words through search engine exponent. This model integrates the characteristics of vector detection, hamming distance and the longest common substring and carries out detection specific to near-synonyms, word deletion and changes in word order by redefining distance matrix and adding ordinal measures, making sentence similarity detection in terms of semantics and backbone word segmentation more effective. Compared with the traditional paper similarity retrieval, the present method has advantages in accuracy of word segmentation, low computation, reliability and high efficiency, which is of great academic significance in word segmentation, similarity detection and document summarization.

Suffix tree searcher: exploration of common substrings in large DNA sequence sets.

BackgroundLarge DNA sequence data sets require special bioinformatics tools to search and compare them. Such tools should be easy to use so that the data can be easily accessed by a wide array of researchers. In the past, the use of suffix trees for searching DNA sequences has been limited by a practical need to keep the trees in RAM. Newer algorithms solve this problem by using disk-based approaches. However, none of the fastest suffix tree algorithms have been implemented with a graphical user interface, preventing their incorporation into a feasible laboratory workflow.ResultsSuffix Tree Searcher (STS) is designed as an easy-to-use tool to index, search, and analyze very large DNA sequence datasets. The program accommodates very large numbers of very large sequences, with aggregate size reaching tens of billions of nucleotides. The program makes use of pre-sorted persistent "building blocks" to reduce the time required to construct new trees. STS is comprised of a graphical user interface written in Java, and four C modules. All components are automatically downloaded when a web link is clicked. The underlying suffix tree data structure permits extremely fast searching for specific nucleotide strings, with wild cards or mismatches allowed. Complete tree traversals for detecting common substrings are also very fast. The graphical user interface allows the user to transition seamlessly between building, traversing, and searching the dataset.ConclusionsThus, STS provides a new resource for the detection of substrings common to multiple DNA sequences or within a single sequence, for truly huge data sets. The re-searching of sequence hits, allowing wild card positions or mismatched nucleotides, together with the ability to rapidly retrieve large numbers of sequence hits from the DNA sequence files, provides the user with an efficient method of evaluating the similarity between nucleotide sequences by multiple alignment or use of Logos. The ability to re-use existing suffix tree pieces considerably shortens index generation time. The graphical user interface enables quick mastery of the analysis functions, easy access to the generated data, and seamless workflow integration.

Quick greedy computation for minimum common string partition

In the minimum common string partition problem one is given two strings S and T with the same character statistics and one seeks for the smallest partition of S into substrings so that T can also be partitioned into the same substring multiset. The problem is fundamental in several variants of edit distance with block operations, e.g. signed reversal distance with duplicates and edit distance with moves.The minimum common string partition problem is known to be NP-complete and the best approximation algorithm known has an approximation factor of O(lognlog⁎n). Since the minimum common string partition problem is of utmost practical importance one seeks a heuristic that will (1) usually have a low approximation factor and (2) will run fast.A simple greedy algorithm is known, which iteratively choose non-overlapping longest common substrings of the input strings. This algorithm has been well-studied from an approximation point of view and it has been shown to have a bad worst case approximation factor. However, all the bad approximation factors presented so far stem from complicated recursive construction. In practice the greedy algorithm seems to have small approximation factors. However, the best current implementation of greedy runs in quadratic time.We propose a novel method to implement greedy in linear time.

Kmacs: the k-mismatch average common substring approach to alignment-free sequence comparison.

Motivation: Alignment-based methods for sequence analysis have various limitations if large datasets are to be analysed. Therefore, alignment-free approaches have become popular in recent years. One of the best known alignment-free methods is the average common substring approach that defines a distance measure on sequences based on the average length of longest common words between them. Herein, we generalize this approach by considering longest common substrings with k mismatches. We present a greedy heuristic to approximate the length of such k-mismatch substrings, and we describe kmacs, an efficient implementation of this idea based on generalized enhanced suffix arrays.Results: To evaluate the performance of our approach, we applied it to phylogeny reconstruction using a large number of DNA and protein sequence sets. In most cases, phylogenetic trees calculated with kmacs were more accurate than trees produced with established alignment-free methods that are based on exact word matches. Especially on protein sequences, our method seems to be superior. On simulated protein families, kmacs even outperformed a classical approach to phylogeny reconstruction using multiple alignment and maximum likelihood.Availability and implementation: kmacs is implemented in C++, and the source code is freely available at http://kmacs.gobics.de/Contact: chris.leimeister@stud.uni-goettingen.deSupplementary information: Supplementary data are available at Bioinformatics online.

Detection and reconstruction of freeform sweeps

AbstractWe study the difficult problem of deciding if parts of a freeform surface can be generated, or approximately generated, by the motion of a planar profile through space. While this task is basic for understanding the geometry of shapes as well as highly relevant for manufacturing and building construction, previous approaches were confined to special cases like kinematic surfaces or “moulding” surfaces. The general case remained unsolved so far. We approach this problem by a combination of local and global methods: curve analysis with regard to “movability”, curve comparison by common substring search in curvature plots, an exhaustive search through all planar cuts enhanced by quick rejection procedures, the ordering of candidate profiles and finally, global optimization. The main applications of our method are digital reconstruction of CAD models exhibiting sweep patches, and aiding in manufacturing freeform surfaces by pointing out those parts which can be approximated by sweeps.

Automated evaluation of approximate matching algorithms on real data

Bytewise approximate matching is a relatively new area within digital forensics, but its importance is growing quickly as practitioners are looking for fast methods to screen and analyze the increasing amounts of data in forensic investigations. The essential idea is to complement the use of cryptographic hash functions to detect data objects with bytewise identical representation with the capability to find objects with bytewise similar representations.Unlike cryptographic hash functions, which have been studied and tested for a long time, approximate matching ones are still in their early development stages and evaluation methodology is still evolving. Broadly, prior approaches have used either a human in the loop to manually evaluate the goodness of similarity matches on real world data, or controlled (pseudo-random) data to perform automated evaluation.This work's contribution is to introduce automated approximate matching evaluation on real data by relating approximate matching results to the longest common substring (LCS). Specifically, we introduce a computationally efficient LCS approximation and use it to obtain ground truth on the t5 set. Using the results, we evaluate three existing approximate matching schemes relative to LCS and analyze their performance.

On Bidirectional Transitions between Localist and Distributed Representations: The Case of Common Substrings Search Using Vector Symbolic Architecture

The contribution of this article is twofold. First, it presents an encoding approach for seamless bidirectional transitions between localist and distributed representation domains. Second, the approach is demonstrated on the example of using Vector Symbolic Architecture for solving a problem of finding common substrings. The proposed algorithm uses elementary operations on long binary vectors. For the case of two patterns with respective lengths L1 and L2 it requires Θ(L1 + L2 – 1) operations on binary vectors, which is equal to the suffix trees approach – the fastest algorithm for this problem. The simulation results show that in order to be robustly detected by the proposed approach the length of a common substring should be more than 4% of the longest pattern.

Improving Honeyd for Automatic Generation of Attack Signatures

In this paper, we design and implement a new Plugin to Honeyd which generates attack signature, automatically. Current network intrusion detection systems work on misuse detectors, where the packets in the monitored network are compared against a repository of signatures. But, we focus on automatic signature generation from malicious network traffic. Our proposed system inspects honeypot traffic and generates intrusion signatures for unknown traffic.The signature is based on traffic patterns, using Longest Common Substring (LCS) algorithm. It is noteworthy that our system is a plugin to honeyd - a low interaction honeypot. The system's output is a file containing honeypot intrusion signatures in pseudo-snort format. Signature generation system has been implemented for Linux Operating System (OS) but due to the common use of Windows OS, we implement for Windows OS, using C programming language.

A semantics-based method for clustering of Chinese web search results

Information explosion is a critical challenge to the development of modern information systems. In particular, when the application of an information system is over the Internet, the amount of information over the web has been increasing exponentially and rapidly. Search engines, such as Google and Baidu, are essential tools for people to find the information from the Internet. Valuable information, however, is still likely submerged in the ocean of search results from those tools. By clustering the results into different groups based on subjects automatically, a search engine with the clustering feature allows users to select most relevant results quickly. In this paper, we propose an online semantics-based method to cluster Chinese web search results. First, we employ the generalised suffix tree to extract the longest common substrings (LCSs) from search snippets. Second, we use the HowNet to calculate the similarities of the words derived from the LCSs, and extract the most representative features by constructing the vocabulary chain. Third, we construct a vector of text features and calculate snippets’ semantic similarities. Finally, we improve the Chameleon algorithm to cluster snippets. Extensive experimental results have shown that the proposed algorithm has outperformed over the suffix tree clustering method and other traditional clustering methods.

ZipHMMlib: a highly optimised HMM library exploiting repetitions in the input to speed up the forward algorithm

BackgroundHidden Markov models are widely used for genome analysis as they combine ease of modelling with efficient analysis algorithms. Calculating the likelihood of a model using the forward algorithm has worst case time complexity linear in the length of the sequence and quadratic in the number of states in the model. For genome analysis, however, the length runs to millions or billions of observations, and when maximising the likelihood hundreds of evaluations are often needed. A time efficient forward algorithm is therefore a key ingredient in an efficient hidden Markov model library.ResultsWe have built a software library for efficiently computing the likelihood of a hidden Markov model. The library exploits commonly occurring substrings in the input to reuse computations in the forward algorithm. In a pre-processing step our library identifies common substrings and builds a structure over the computations in the forward algorithm which can be reused. This analysis can be saved between uses of the library and is independent of concrete hidden Markov models so one preprocessing can be used to run a number of different models.Using this library, we achieve up to 78 times shorter wall-clock time for realistic whole-genome analyses with a real and reasonably complex hidden Markov model. In one particular case the analysis was performed in less than 8 minutes compared to 9.6 hours for the previously fastest library.ConclusionsWe have implemented the preprocessing procedure and forward algorithm as a C++ library, zipHMM, with Python bindings for use in scripts. The library is available at http://birc.au.dk/software/ziphmm/.

A Multitask Parallel Algorithm Based on Multi-Core Processor

Multi-core processor has been a hot topic since it improves operation speed. It is not easy to get efficient parallel processing data algorithms because of waste of hardware resources. In this paper, a novel multitask parallel algorithm based on getting common substring of two strings is described in order to improve the data-handling capacity of the multi-processor. Firstly, this algorithm performs Task Parallel Library (TPL) in VS.NET, and then schedule the algorithm proposed in this paper to process data. This algorithm is tested by actual parallel data. The results demonstrate that this algorithm overcomes the problem of waste of hardware resource, can take full advantage of the features of multi-core parallel processing data thereby enhancing the parallel speedup, greatly improving the efficiency of data processing.

Constraint Based Periodic Pattern Mining in Multiple Longest Common Subsequences

The periodicity search in longest common subsequences in multiple strings has a number of application, is an interesting data mining problem. Periodicity is very common practice in longest common subsequence mining algorithm. This work introduces a new parallel algorithm for finding periodicity in multiple strings. Few existing algorithms lacks in poor scalability, lacks in finding all longest pattern, and for finding symbol, partial and full periodicity. We designed the algorithm using FP-tree for finding periodicity for most common longest substring in multiple sources. We introduce a parallel algorithm for Constraint Based Periodic Pattern Mining (CBPPM) algorithm, which takes O( kN ) for finding periodicity and O((N×L×h)/p) time for MLCS pattern. We tested parallel algorithm on a coarse-grained multi-computer (BSP/CGM) model with p<√m processors that takes O(N×L√p) space per processor, with O(log p) communication rounds. We derive a practical implementation that works better for arbitrary length of input sequence. The algorithm is noise resilient, and shown its performance in presence of replacement, insertion, deletion, or mixture of these types of noise. We experimented with synthetic and real data reveals a near linear speedup with scalable performance. The comparative study shows algorithm's applicability and effectiveness, generally more noise resilient.