Damerau-Levenshtein Distance Research Articles

Development of fuzzy search method for creating an efficient information search system in text data

The object of research is the processes of effective search for information in a set of textual data. The subject of the research is the fuzzy search method, which will allow to effectively solve the problem of searching for information in a set of textual data. The paper considers the process of developing a fuzzy search method, which consists of 9 consecutive steps and is required for a quick search for matches in a large set of text data. Based on this method, it is proposed to create a fuzzy search system that will solve the problem of finding the most relevant documents from a set of such documents. The proposed fuzzy search method combines the advantages of algorithms based on deterministic finite automata and algorithms based on dynamic programming for calculating the Damerau-Levenshtein distance. Such a combination allows to implement the symbol similarity table in an optimal way. As part of the work, an approach for creating a symbol similarity table was proposed and an example of such a table was created for symbols from the English alphabet, which allows to find the degree of similarity between two symbols with constant asymptotics and to convert the current symbol into its basic counterpart. For document filtering, a metric was developed to evaluate the correspondence of text data to a search phrase, which simultaneously takes into account the number of found and not found characters and the number of found and not found words. The Damerau-Levenstein algorithm allows to find the edit distance between two words, taking into account the following types of errors: substitution, addition, deletion, and transposition of characters. The work proposed a modification of this algorithm by using a similarity table to more accurately estimate the editing distance between two words. The developed method makes it possible to create a fuzzy search system that will help find the desired results faster and increase the relevance of the obtained results by sorting them according to the values of the proposed test data similarity metric.

Damerau-Levenshtein Distance Algorithm Based on Abstract Syntax Tree to Detect Code Plagiarism

Purpose: This research aimed to detect source code plagiarism based on Abstract Syntax Tree using Damerau-Levenshtein Distance algorithm, which is expected to streamline the inaccuracies and time-consumption associated with the manual process.Methods: Damerau-Levenshtein Distance algorithm was used to determine the similarity between source code files and calculate F-Measure. The dataset, which consisted of 178 source code files from 20 coursework assignments, was obtained from GitHub by Lawton Nichols in 2019. Damerau-Levenshtein Distance algorithm was used to compute the minimum cost required to transform one line of code into another. Furthermore, ANTLR detected AST, which was processed through preprocessing, including node pruning, function and variable sorting, and log output removal. Result: The result showed that the two methods took 5.704 seconds and 0.996 seconds to complete. The lowest and highest values obtained using F-Measure were 0.16 and 0.8, respectively. Therefore, the system performed detection processes quickly and effectively detected common forms of code plagiarism with difficulty in the more complex forms. Novelty: In conclusion, this research used AST and Damerau-Levenshtein Distance algorithm to calculate the 5 levels of similarity in Java programming language source code. For further development, preprocessing steps were needed to prune unnecessary nodes and detect equivalent but differently syntaxed code.

Methodology for Analyzing Bitstreams Based on the Use of the Damerau–Levenshtein Distance and Other Metrics

Methodology for Analyzing Bitstreams Based on the Use of the Damerau–Levenshtein Distance and Other Metrics

Comparison of fuzzy search algorithms based on Damerau-Levenshtein automata on large data

The object of research is fuzzy search algorithms based on Damerau-Levenshtein automata and Levenshtein automata. The paper examines and compares solutions based on finite state machines for efficient and fast finding of words and lines with a given editing distance in large text data using the concept of fuzzy search. Fuzzy search algorithms allow finding significantly more relevant results than standard explicit search algorithms. However, such algorithms usually have a higher asymptotic complexity and, accordingly, work much longer. Fuzzy text search using Damerau-Levenshtein distance allows taking into account common errors that the user may have made in the search term, namely: character substitution, extra character, missing character, and reordering of characters. To use a finite automaton, it is necessary to first construct it for a specific input word and edit distance, and then perform a search on that automaton, discarding words that the automaton will not accept. Therefore, when choosing an algorithm, both phases should be taken into account. This is because building a machine can take a long time. To speed up one of the machines, SIMD instructions were used, which gave a speedup of 1-10% depending on the number of search words, the length of the search word and the editing distance. The obtained results can be useful for use in various industries where it is necessary to quickly and efficiently perform fuzzy search in large volumes of data, for example, in search engines or in autocorrection of errors.

Early warning method of unsafe behavior accidents for offshore drilling operators based on eye-tracking trajectory

Despite the implementation of highly automated and intelligent equipment and facilities at the offshore drilling platform in recent years, drilling operators remain indispensable. During offshore drilling operations, the behavior of operators is susceptible to numerous factors including personal attributes, environmental interferences, and equipment feedback. When the factors that affect the operation of drilling change adversely, unsafe behavior of drilling operators can easily occur, resulting in offshore drilling accidents. As such, the early warning of unsafe behavior accidents of drilling operators is essential to reduce the occurrence of offshore drilling accidents. Firstly, attempts were made to use eye-tracking technology to record the eye movement data of unsafe behavior of drilling operators in real-time. In total, 42 drilling operation trainers had their eye-tracking trajectories recorded in real-time during normal tripping operations throughout the drilling process. A total of 252 eye-tracking trajectory data points were collected for six drilling behaviors, including normal operation, unskilled operation, fatigue operation, action errors, omitted actions and wrong order. Secondly, based on the IETTSM (Improved Eye-tracking Trajectory Sequence Method) algorithm, the eye-tracking trajectory of drilling operators was serialized. Thirdly, an evaluation model of drilling operator operation behavior was constructed based on the DLD (Damerau-Levenshtein Distance) algorithm. Fourthly, the IETTSM-DLD model was used to prompt abnormal behavior among drilling operators, which was accomplished by referencing the appropriate trajectory nodes and issuing early warnings of unsafe behavior based on the coupling of said trajectory nodes. The results reveal that the proposed IETTSM-DLD model can realize real-time prompting of abnormal operation of offshore drilling operators and early warning of accidents caused by unsafe behavior. The average early warning accuracy rate was found to be 91.9%, with an average early warning time of 120.97 s.

The development of a mobile app-focused deduplication strategy for the Apple Heart Study that informs recommendations for future digital trials.

An app-based clinical trial enrolment process can contribute to duplicated records, carrying data management implications. Our objective was to identify duplicated records in real time in the Apple Heart Study (AHS). We leveraged personal identifiable information (PII) to develop a dissimilarity score (DS) using the Damerau-Levenshtein distance. For computational efficiency, we focused on four types of records at the highest risk of duplication. We used the receiver operating curve (ROC) and resampling methods to derive and validate a decision rule to classify duplicated records. We identified 16,398 (4%) duplicated participants, resulting in 419,297 unique participants out of a total of 438,435 possible. Our decision rule yielded a high positive predictive value (96%) with negligible impact on the trial's original findings. Our findings provide principled solutions for future digital trials. When establishing deduplication procedures for digital trials, we recommend collecting device identifiers in addition to participant identifiers; collecting and ensuring secure access to PII; conducting a pilot study to identify reasons for duplicated records; establishing an initial deduplication algorithm that can be refined; creating a data quality plan that informs refinement; and embedding the initial deduplication algorithm in the enrolment platform to ensure unique enrolment and linkage to previous records.

Linking Biomedical Data Warehouse Records With the National Mortality Database in France: Large-scale Matching Algorithm.

Often missing from or uncertain in a biomedical data warehouse (BDW), vital status after discharge is central to the value of a BDW in medical research. The French National Mortality Database (FNMD) offers open-source nominative records of every death. Matching large-scale BDWs records with the FNMD combines multiple challenges: absence of unique common identifiers between the 2 databases, names changing over life, clerical errors, and the exponential growth of the number of comparisons to compute. We aimed to develop a new algorithm for matching BDW records to the FNMD and evaluated its performance. We developed a deterministic algorithm based on advanced data cleaning and knowledge of the naming system and the Damerau-Levenshtein distance (DLD). The algorithm's performance was independently assessed using BDW data of 3 university hospitals: Lille, Nantes, and Rennes. Specificity was evaluated with living patients on January 1, 2016 (ie, patients with at least 1 hospital encounter before and after this date). Sensitivity was evaluated with patients recorded as deceased between January 1, 2001, and December 31, 2020. The DLD-based algorithm was compared to a direct matching algorithm with minimal data cleaning as a reference. All centers combined, sensitivity was 11% higher for the DLD-based algorithm (93.3%, 95% CI 92.8-93.9) than for the direct algorithm (82.7%, 95% CI 81.8-83.6; P<.001). Sensitivity was superior for men at 2 centers (Nantes: 87%, 95% CI 85.1-89 vs 83.6%, 95% CI 81.4-85.8; P=.006; Rennes: 98.6%, 95% CI 98.1-99.2 vs 96%, 95% CI 94.9-97.1; P<.001) and for patients born in France at all centers (Nantes: 85.8%, 95% CI 84.3-87.3 vs 74.9%, 95% CI 72.8-77.0; P<.001). The DLD-based algorithm revealed significant differences in sensitivity among centers (Nantes, 85.3% vs Lille and Rennes, 97.3%, P<.001). Specificity was >98% in all subgroups. Our algorithm matched tens of millions of death records from BDWs, with parallel computing capabilities and low RAM requirements. We used the Inseehop open-source R script for this measurement. Overall, sensitivity/recall was 11% higher using the DLD-based algorithm than that using the direct algorithm. This shows the importance of advanced data cleaning and knowledge of a naming system through DLD use. Statistically significant differences in sensitivity between groups could be found and must be considered when performing an analysis to avoid differential biases. Our algorithm, originally conceived for linking a BDW with the FNMD, can be used to match any large-scale databases. While matching operations using names are considered sensitive computational operations, the Inseehop package released here is easy to run on premises, thereby facilitating compliance with cybersecurity local framework. The use of an advanced deterministic matching algorithm such as the DLD-based algorithm is an insightful example of combining open-source external data to improve the usage value of BDWs.

Quantifying graphemic variation via large text corpora

Abstract The use of some basic computer science concepts could expand the possibilities of (manual) graphematic text corpus analysis. With these it can be shown that graphematic variation decreases constantly in printed German texts from 1600 to 1900. While the variability is continuously lesser on a text-internal level, it decreases faster for the whole available writing system of individual decades. But which changes took place exactly? Which types of variation went away more quickly, which ones persisted? How do we deal with large amounts of data which cannot be processed manually anymore? Which aspects are of special importance or go missing while working with a large textual base? The use of a measurement called entropy quantifies the variability of the spellings of a given word form, lemma, text or subcorpus, with few restrictions but also less details in the results. The difference between two spellings can be measured via Damerau-Levenshtein distance. To a certain degree, automated data handling can also determine the exact changes that took place. Afterwards, these differences can be counted and ranked. As data source the German Text Archive of the Berlin-Brandenburg Academy of Sciences and Humanities is used. It offers for example orthographic normalization – which is extremely useful –, preprocessing of parts of speech and lemmatization. As opposed to many other approaches the establishment of today’s normed spellings is not seen as the aim of the developments and is therefore not the focus of the research. Instead, the differences between individual spellings are of interest. Afterwards intra- and extralinguistic factors which caused these developments should be determined. These methodological findings could subsequently be used for improving research methods in other graphematic fields of interest, e. g. computer-mediated communication.

STEMMING BAHASA JAWA MENGGUNAKAN DAMERAU LEVENSHTEIN DISTANCE (DLD)

Stemming is one of the essential stages of text mining. This process removes prefixes and suffixes to produce root words in a text. This study uses a string matching algorithm, namely Damerau Levenshtein Distance (DLD), to find the basic word forms of Javanese. Test data of 300 words that have a prefix, insertion, suffix, a combination of prefix and suffix, and word repetition. The results of this study indicate that the Damerau Levenshtein Distance (DLD) algorithm can be used for Stemming Javanese text with an accuracy value of 49.6%.

Design and implementation of a novel two-phase spectrum handoff scheme for QoS aware mobile users in cognitive radio networks

Quality of Service (QoS) aware spectrum handoff operation is critical for Secondary Users (SUs) executing real-time applications like Voice over IP (VoIP) in Cognitive Radio Networks (CRNs). The problem is aggravated by the inclusion of occasionally mobile SUs where pre-selected target channels are often rendered insignificant during handoff under dynamic conditions. Conventional proactive and reactive schemes may underperform due to channel obsolescence issues and unacceptable handoff latency respectively while a combined approach is not yet explored under the constraints of practical applicability. In view of these challenges and enormous significance of VoIP based CRN in 5G networks, this paper proposes a practically feasible two-phase spectrum handoff methodology with I_Phase and S_Phase operating under alternate conditions. I_Phase deploys a three-level dropping decision scheme followed by a two-tier spectrum handoff policy and is motivated by an integrated approach spanning every CR operation. S_Phase incorporates user mobility and application-oriented aspects where channel drop is governed by a joint two-order Hidden Markov Model — custom Damerau–Levenshtein distance metric based policy and spectrum handoff is monitored by a novel non-cooperative TOPSIS method. These phases are assisted by two proposed prerequisite techniques namely ACT (Adaptive Call Transmission) involving CR timing parameters and VER (VoIP Early Resumption) dealing with early call resumption. Comparative analysis of QoS metrics including per-instance handoff latency, transmission duration per target channel, channel drop and call drop probabilities, etc. in analytical and simulation platforms record significant performance improvement. Novelty of this work further relies on design and execution of SU prototype with the proposed scheme in test-bed that successfully provides QoS guarantees during spectrum handoff.

Guest Editorial Special Issue: “From Deletion-Correction to Graph Reconstruction: In Memory of Vladimir I. Levenshtein”

There are few mathematicians whose contributions go beyond named conjectures and theorems: Vladimir Iosifovich Levenshtein ( , 1935–2017) is one such true exception. During the five decades of his active research career, he enriched combinatorics, coding, and information theory with elegant problem formulations, ingenious algorithmic solutions, and highly original proof techniques. However, his work accomplished much more—it paved the way for the creation and advancement of new scientific disciplines, such as natural language processing, metagenomics, sequence alignment, and reference-based genome assembly, as well as DNA-based data storage, to name a few. A crucial concept behind sequence alignment algorithms used in phylogeny, comparative, and cancer genomics, as well as in natural language processing is the Levenshtein (edit) distance and its extension, termed the Damerau–Levenshtein distance between strings. The Levenshtein distance equals the smallest number of insertions, deletions, or substitutions required to convert one string into another. Levenshtein introduced this metric in 1965 [item 1) in the Appendix], followed by the notion of deletion and insertion error-correcting codes that have since been used in a myriad of systems presented with synchronization errors [items 1) and 2) in the Appendix]. Levenshtein’s work also inspired the introduction of the trace reconstruction problem [items 3) and 4) in the Appendix] which has since sparked substantial interest in the field of DNA-based data storage.

MeDict: Health Dictionary Application Using Damerau-Levenshtein Distance Algorithm

Typographical error often happens. It can occur due to mechanical errors or missed hands or fingers when typing. Someone's ignorance of how to spell correctly also can cause typographical errors. Dictionary application development has been carried out by various parties so that the searching process in the dictionary becomes more efficient. However, there is no word search optimization when the typographical error happens. Typographical errors in the searching process can result in the information sought cannot be found. The Damerau-Levenshtein Distance algorithm implemented to provide search suggestions when a typographical error occurs. This research aims to design and build a health dictionary application, MeDict, using the Damerau-Levenshtein Distance algorithm. Technology Acceptance Model (TAM) used to evaluate the application. The result is 86.2% stating strongly agree that the application can be useful and 86.9% stating strongly agree that the application can be used easily.

Linear space string correction algorithm using the Damerau-Levenshtein distance

BackgroundThe Damerau-Levenshtein (DL) distance metric has been widely used in the biological science. It tries to identify the similar region of DNA,RNA and protein sequences by transforming one sequence to the another using the substitution, insertion, deletion and transposition operations. Lowrance and Wagner have developed an O(mn) time O(mn) space algorithm to find the minimum cost edit sequence between strings of length m and n, respectively. In our previous research, we have developed algorithms that run in O(mn) time using only O(s∗min{m,n}+m+n) space, where s is the size of the alphabet comprising the strings, to compute the DL distance as well as the corresponding edit sequence. These are so far the fastest and most space efficient algorithms. In this paper, we focus on the development of algorithms whose asymptotic space complexity is linear.ResultsWe develop linear space algorithms to compute the Damerau-Levenshtein (DL) distance between two strings and determine the optimal trace (corresponding edit operations.)Extensive experiments conducted on three computational platforms–Xeon E5 2603, I7-x980 and Xeon E5 2695–show that, our algorithms, in addition to using less space, are much faster than earlier algorithms.ConclusionBesides using less space than the previously known algorithms,significant run-time improvement was seen for our new algorithms on all three of our experimental platforms. On all platforms, our linear-space cache-efficient algorithms reduced run time by as much as 56.4% and 57.4% in respect to compute the DL distance and an optimal edit sequences compared to previous algorithms. Our multi-core algorithms reduced the run time by up to 59.3% compared to the best previously known multi-core algorithms.

A Tool for Automatic Scoring of Spelling Performance.

PurposeThe evaluation of spelling performance in aphasia reveals deficits in written language and can facilitate the design of targeted writing treatments. Nevertheless, manual scoring of spelling performance is time-consuming, laborious, and error prone. We propose a novel method based on the use of distance metrics to automatically score spelling. This study compares six automatic distance metrics to identify the metric that best corresponds to the gold standard—manual scoring—using data from manually obtained spelling scores from individuals with primary progressive aphasia.MethodThree thousand five hundred forty word and nonword spelling productions from 42 individuals with primary progressive aphasia were scored manually. The gold standard—the manual scores—were compared to scores from six automated distance metrics: sequence matcher ratio, Damerau–Levenshtein distance, normalized Damerau–Levenshtein distance, Jaccard distance, Masi distance, and Jaro–Winkler similarity distance. We evaluated each distance metric based on its correlation with the manual spelling score.ResultsAll automatic distance scores had high correlation with the manual method for both words and nonwords. The normalized Damerau–Levenshtein distance provided the highest correlation with the manual scoring for both words (rs = .99) and nonwords (rs = .95).ConclusionsThe high correlation between the automated and manual methods suggests that automatic spelling scoring constitutes a quick and objective approach that can reliably substitute the existing manual and time-consuming spelling scoring process, an important asset for both researchers and clinicians.

Typo handling in searching of Quran verse based on phonetic similarities

The Quran search system is a search system that was built to make it easier for Indonesians to find a verse with text by Indonesian pronunciation, this is a solution for users who have difficulty writing or typing Arabic characters. Quran search system with phonetic similarity can make it easier for Indonesian Muslims to find a particular verse. Lafzi was one of the systems that developed the search, then Lafzi was further developed under the name Lafzi+. The Lafzi+ system can handle searches with typo queries but there are still fewer variations regarding typing error types. In this research Lafzi++, an improvement from previous development to handle typographical error types was carried out by applying typo correction using the autocomplete method to correct incorrect queries and Damerau Levenshtein distance to calculate the edit distance, so that the system can provide query suggestions when a user mistypes a search, either in the form of substitution, insertion, deletion, or transposition. Users can also search easily because they use Latin characters according to pronunciation in Indonesian. Based on the evaluation results it is known that the system can be better developed, this can be seen from the accuracy value in each query that is tested can surpass the accuracy of the previous system, by getting the highest recall of 96.20% and the highest Mean Average Precision (MAP) reaching 90.69%. The Lafzi++ system can improve the previous system.

Kombinasi Damerau Levenshtein dan Jaro-Winkler Distance Untuk Koreksi Kata Bahasa Inggris

Writing is one of the efforts made by the writer to express ideas and ideas to others. But sometimes when writing, there are many errors in typing spelling, especially English spelling, resulting in errors in capturing the meaning and meaning of the writing. To overcome this problem, we need a system that can detect word spelling errors. Damerau Levenshtein and Jaro Winkler Distance Algorithms are algorithms that can be used as a solution to detect English typing errors. From the test results, it can be concluded that the Damerau Levenshtein and Jaro-Winkler Distance are able to optimally detect word mismatches and look for similarities of words compared. The Damerau Levenshtein Distance works by finding the smallest distance value, while the Jaro-Winkler Distance works by finding the greatest proximity value of the string being compared. Using this algorithm, errors in writing the spelling of words can be minimized.&#x0D; &#x0D; Keywords— Algorithm; Damerau Levenshtein; Jaro Winkler; Spelling Cheker; String Matching.

Spelling Checker using Algorithm Damerau Levenshtein Distance and Cosine Similarity

Writing is an embodiment of the author's ideas that are to be conveyed to others. A writer often experiences typos in typing the script, so that it can influence the meaning of the text. Therefore, a system is needed to detect word errors. In this study, checking is done by using the Dictionary Lookup method and giving the candidate words using the Damerau Levenshtein Distance algorithm. Candidates will then determine the ranking by breaking the word into Bigram form and calculating the similarity value using the Cosine Similarity algorithm. The test results based on the data used yield different Mean Reciprocal Rank (MRR) values for each type of error. The type of error deletion produces an MRR value of 88.89%, the type of insertion error produces an MRR value of 97.78%, the type of substitution error produces an MRR value of 88.89%, the type of transposition error produces an MRR value of 89%

Spelling Correction Application with Damerau-Levenshtein Distance to Help Teachers Examine Typographical Error in Exam Test Scripts

This research was intended to create Spelling Correction Application to help teachers examine questions scripts with the capability to found typographical error and give suggestion for non-real word error. This application is built with simple Damerau-Levenshtein Distance method to detect errors and give word suggestions from the typo word. This application can be used by the teacher to examine documents in the form of short answer, essay and multiple choices then save them back in the form of original documents. This application is built using a dictionary lookup consist of 41 312 words in Indonesian. The first test result is the application can detect non-real word errors from 50 sentences that have non-real word error in each sentence and produce an accuracy of 88 %. The second test is try to detect typographical error in exam test script that consist of 15 sample questions, consisting of five essay questions, five short answer, and five multiple choices.

VoIP-HDK – A novel channel allocation technique for QoS aware VoIP communication over heterogeneous networks

Abstract Channel allocation for VoIP (Voice over IP) is a critical problem in modern-day heterogeneous networks having diverse application requirements. Conventional stochastic and machine-learning based models that predict channel suitability for VoIP underperform under dynamic conditions. In contrast, this paper deploys VoIP-HDK2 – a novel technique that uses an efficient combination of K-Nearest Neighbors algorithm, 2-order Hidden Markov Model, Damerau-Levenshtein distance based model and K-Means Clustering technique to accurately predict and allocate channels with sufficient QoS (Quality of Service) guarantees to VoIP calls. The primary contribution of this paper lays both in the formulation of the proposed methodology involving tailor-made algorithms as well as complete test-bed based dataset creation, execution of VoIP-HDK2 and validation of its performance in terms of maintaining high quality VoIP calls. Comparative performance evaluation records significant reduction in call drops and packet-loss and enhancement in call quality and throughput for VoIP-HDK2 with respect to existing works in literature.

Keywords Search Correction Using Damerau Levenshtein Distance Algorithm

Searching is one of the important features on the website, but it is not uncommon for users to make typos when typing keywords. Typing errors of these keywords is usually referred to as typo. This study aims to build a system by providing suggestions for correcting typos in the search feature. Keywords search correction are obtained using the Damerau-Levenshtein Distance Approximate String Matching algorithm by to calculate the editing distance of each word in a keywords with each word in the Indonesian word dictionary. Testing was carried out as many as 40 experiments, with 10 keywords and 250 articles taken randomly. The test results show the Damerau-Levenshtein Distance algorithm is able to provide precision and recall values of 91.24% and 89.58% in providing keyword improvement suggestions. With the improvement of the system, each trial increases with precision value of 0.80 and recall value of 0.98