Levenshtein Edit Distance Research Articles

Optimization of Image Processing Algorithms for Character Recognition in Cultural Typewritten Documents

Linked data is used in various fields as a new way of structuring and connecting data. Cultural heritage institutions have been using linked data to improve archival descriptions and facilitate the discovery of information. Most archival records have digital representations of physical artifacts in the form of scanned images that are non-machine-readable. Optical Character Recognition (OCR) recognizes text in images and translates it into machine-encoded text. This article evaluates the impact of image processing methods and parameter tuning in OCR applied to typewritten cultural heritage documents. The approach uses a multi-objective problem formulation to minimize Levenshtein edit distance and maximize the number of words correctly identified with a non-dominated sorting genetic algorithm (NSGA-II) to tune the methods’ parameters. Evaluation results show that parameterization by digital representation typology benefits the performance of image pre-processing algorithms in OCR. Furthermore, our findings suggest that employing image pre-processing algorithms in OCR might be more suitable for typologies where the text recognition task without pre-processing does not produce good results. In particular, Adaptive Thresholding, Bilateral Filter, and Opening are the best-performing algorithms for the theater plays’ covers, letters, and overall dataset, respectively, and should be applied before OCR to improve its performance.

Comparisons of machine learning techniques for detecting fraudulent criminal identities

This paper focuses on applications of various machine learning techniques on an anonymized policing dataset used in EU SPIRIT Horizon 2020 project to identify fraudulent identities and help Law Enforcement Agencies (LEAs) in their investigation in finding potential criminals and identity resolution. Lack of qualitative data and appropriate methodology to carry out research on criminal fraudulent identities is a common reason for fewer research in this area. Additionally, it is a very sensitive data to work with and minor inaccuracy in prediction of result causes massive impact in the society as genuine people could be questioned whereas criminals could be sent free. Both of these issues are addressed in this paper by application of 39 million records from policing dataset and working towards higher accuracy while building the model. Various machine learning approaches are applied to train the dataset to make predictions and the research focus on being able to predict the 5 suspected fraudulent identities out of 39 million records in the policing dataset. One of the applied machine learning techniques include TensorFlow along with Keras model which has seldomly been applied by researchers in detection of criminal data. To compare the results and test accuracy of TensorFlow model, other machine learning techniques such as Support Vector Machine, Naïve Bayes and K-nearest Neighbours are also applied to have a comparative study on the obtained outcomes from each model. The goal of this research is to find fraudulent IDs amongst all the anonymized IDs in the criminal dataset using TensorFlow and three other machine learning models and select the most optimal model out of them. Since the model is comparing two names so string-matching techniques such as Levenshtein edit distance, Hamming Distance, Jaro-Winkler and Soundex were applied to select an effective approach first before building the model and analysing the results. TensorFlow model demonstrated highest accuracy with relatively least execution time and the only model to successfully predict all the 5 suspects from the policing dataset.

Using optimized clustering to identify students' science learning paths to knowledge integration

Background: This study captured students' repertoire of science ideas and determined the varied paths students take to integrate their disconnected ideas as they studied a web-based Genetic Inheritance unit. Method: We analyzed 6th graders' responses to embedded items and activities to establish progress in knowledge integration in two different learning conditions: revisiting and critiquing. Learning paths were established by measuring students' idea dissimilarities using Levenshtein edit distance, clustering using silhouette coefficient and K-means, and determining the most representative path via generalized median method. Results: Four learning paths emerged from the revisit condition (isolated links, partial links, valid links, integrated links) and three learning paths emerged from the critique condition (isolated links, partial links, and integrated links). Conclusion: We found that by providing opportunities for students to revisit or critique ideas, the curriculum supported them to follow multiple paths in building their repertoire of ideas and integrating initial and new information.

Using language models and ontology topology to perform semantic mapping of traits between biomedical datasets.

Human traits are typically represented in both the biomedical literature and large population studies as descriptive text strings. Whilst a number of ontologies exist, none of these perfectly represent the entire human phenome and exposome. Mapping trait names across large datasets is therefore time-consuming and challenging. Recent developments in language modelling have created new methods for semantic representation of words and phrases, and these methods offer new opportunities to map human trait names in the form of words and short phrases, both to ontologies and to each other. Here, we present a comparison between a range of established and more recent language modelling approaches for the task of mapping trait names from UK Biobank to the Experimental Factor Ontology (EFO), and also explore how they compare to each other in direct trait-to-trait mapping. In our analyses of 1191 traits from UK Biobank with manual EFO mappings, the BioSentVec model performed best at predicting these, matching 40.3% of the manual mappings correctly. The BlueBERT-EFO model (finetuned on EFO) performed nearly as well (38.8% of traits matching the manual mapping). In contrast, Levenshtein edit distance only mapped 22% of traits correctly. Pairwise mapping of traits to each other demonstrated that many of the models can accurately group similar traits based on their semantic similarity. Our code is available at https://github.com/MRCIEU/vectology.

Disclaimer effect of key audit matters in China: negative press coverage and boilerplate

PurposeUsing China’s key audit matters (KAMs) data, this study aims to examine whether negative press coverage alleviates boilerplate KAMs.Design/methodology/approachThis study uses Levenshtein edit distance (LVD) to calculate the horizontal boilerplate of KAMs and investigates how boilerplate changes under different levels of the perceived legal risk.FindingsThe findings indicate that auditors of firms exposed to substantial negative press coverage will reduce the boilerplate of KAMs. This association is more significant for auditing firms with lower market share and client firms with higher financial distress. Additionally, the authors find that negative press coverage is more likely to alleviate the boilerplate disclosure of KAMs related to managers’ subjective estimation and material transactions and events. Furthermore, the association between negative press coverage and boilerplate KAMs varies with the source of negative news.Originality/valueThe findings suggest that upon exposure to negative press coverage, reducing the boilerplate of KAMs has a disclaimer effect for auditors.

Hybrid Tamil spell checker with combined character splitting

SummarySpell checker is the application, which helps in finding the spelling errors in a given text. Applications like word processors, mails, search engines, speech recognition and social media forums need these kinds of spell checking tools to increase the correctness of the system. Spell checking is completely implemented in languages such as English, French, and Chinese. But as far as Indian regional languages is concerned, very few works have been carried out, that too partially. Tamil is one such Indian regional language, which requires a fully implemented spell checking application as many people started using this language in social media platforms like Facebook and Twitter. Spelling errors fall on different categories in Tamil language, which involves Sandhi errors, Homophone errors (Mayangoli), and misspelt words error. To tackle all these errors, a new ensemble approach is proposed in this paper. The proposed approach consists of Levenshtein's edit distance algorithm, rule‐based algorithm, Soundex algorithm along with LSTM (Long Short Term Memory) model. We have used a special feature called combine character splitting of Tamil alphabets for feeding the LSTM model to improve the performance of the system. Proposed system produced an accuracy of 95.67%, which is approved by the Tamil scholar.

Analysis of End User Access of Warn-on-Forecast Guidance Products during an Experimental Forecasting Task

AbstractConvection-allowing model ensemble guidance, such as that provided by the Warn-on-Forecast System (WoFS), is designed to provide predictions of individual thunderstorm hazards within the next 0–6 h. The WoFS web viewer provides a large suite of storm and environmental attribute products, but the applicability of these products to the National Weather Service forecast process has not been objectively documented. Therefore, this study describes an experimental forecasting task designed to investigate what WoFS products forecasters accessed and how they accessed them for a total of 26 cases (comprised of 13 weather events, each worked by two forecasters). Analysis of web access log data revealed that in all 26 cases, product accesses were dominated in the reflectivity, rotation, hail, and surface wind categories. However, the number of different product types viewed and the number of transitions between products varied in each case. Therefore, the Levenshtein (Edit Distance) method was used to compute similarity scores across all 26 cases, which helped identify what it meant for relatively similar vs. dissimilar navigation of WoFS products. The Spearman’s Rank correlation coefficient (R) results found that forecasters working the same weather event had higher similarity scores for events that produced more tornado reports and for events in which forecasters had higher performance scores. The findings from this study will influence subsequent efforts for further improving WoFS products and developing an efficient and effective user interface for operational applications.

Guest Editorial Special Issue: “From Deletion-Correction to Graph Reconstruction: In Memory of Vladimir I. Levenshtein”

There are few mathematicians whose contributions go beyond named conjectures and theorems: Vladimir Iosifovich Levenshtein ( , 1935–2017) is one such true exception. During the five decades of his active research career, he enriched combinatorics, coding, and information theory with elegant problem formulations, ingenious algorithmic solutions, and highly original proof techniques. However, his work accomplished much more—it paved the way for the creation and advancement of new scientific disciplines, such as natural language processing, metagenomics, sequence alignment, and reference-based genome assembly, as well as DNA-based data storage, to name a few. A crucial concept behind sequence alignment algorithms used in phylogeny, comparative, and cancer genomics, as well as in natural language processing is the Levenshtein (edit) distance and its extension, termed the Damerau–Levenshtein distance between strings. The Levenshtein distance equals the smallest number of insertions, deletions, or substitutions required to convert one string into another. Levenshtein introduced this metric in 1965 [item 1) in the Appendix], followed by the notion of deletion and insertion error-correcting codes that have since been used in a myriad of systems presented with synchronization errors [items 1) and 2) in the Appendix]. Levenshtein’s work also inspired the introduction of the trace reconstruction problem [items 3) and 4) in the Appendix] which has since sparked substantial interest in the field of DNA-based data storage.

Unconstrained online handwritten Uyghur word recognition based on recurrent neural networks and connectionist temporal classification

This paper conducts the first experiments applying recurrent neural networks-RNN accompanied with connectionist temporal classification (CTC) to build end-to-end online Uyghur handwriting word recognition system. The traced pen-tip trajectory is fed to network without conducting segmentation and feature extraction. The network is trained to transcribe handwritten word trajectory to a string of characters in alphabet which has total 128 character forms. In order to avoid overfitting during training and improve generalisation of the model, dropout technique is implemented. An online handwritten word dataset has been established and used for model training and evaluation in writer independent manner. Recognition results are evaluated by calculating the Levenshtein-edit distance and 14.73% character error rate CER on test set of 3,600 samples for 900 word classes has been observed without help of any lexicon search and language model.

Analysis and safety engineering of fuzzy string matching algorithms

In this paper we explore fuzzy string matching in an automatic ticket classification and processing system. We compare performance of the following string similarity algorithms: Longest Common Subsequence (LCS), Dice coefficient, Cosine Similarity, Levenshtein (edit) distance and Damerau distance. Through optimisation, we accomplished a 15% improvement in the ratio of false positives to true positive classifications over the existing approach used by a customer support system for free customers. To introduce greater safety; we compliment fuzzy string matching algorithms with a second layer Convolutional Neural Network (CNN) binary classifier, achieving an improved keyword classification ratio for two ticket categories by a relative 69% and 78%. Such an approach allows for classification to only be applied where a desired level of safety achieved, such as in instances where automated answers.

Optimized SAT encoding of conformance checking artefacts

Conformance checking is a growing discipline that aims at assisting organizations in monitoring their processes. On its core, conformance checking relies on the computation of particular artefacts which enable reasoning on the relation between observed and modeled behavior. It is widely acknowledge that the computation of these artifacts is the lion’s share of conformance checking techniques. This paper shows how important conformance artefacts like alignments, anti-alignments or multi-alignments, defined over the Levenshtein edit distance, can be efficiently computed by encoding the problem as an optimized SAT instance. From a general perspective, the work advocates for a unified family of techniques that can compute conformance artefacts in the same way. The implementation of the techniques presented in this paper show capabilities for dealing with both synthetic and real-life instances, which may open the door for a fresh way of applying conformance checking in the near future.

The bird's-eye view: A data-driven approach to understanding patient journeys from claims data.

In preference-sensitive conditions such as back pain, there can be high levels of variability in the trajectory of patient care. We sought to develop a methodology that extracts a realistic and comprehensive understanding of the patient journey using medical and pharmaceutical insurance claims data. We processed a sample of 10000 patient episodes (comprised of 113215 back pain-related claims) into strings of characters, where each letter corresponds to a distinct encounter with the healthcare system. We customized the Levenshtein edit distance algorithm to evaluate the level of similarity between each pair of episodes based on both their content (types of events) and ordering (sequence of events). We then used clustering to extract the main variations of the patient journey. The algorithm resulted in 12 comprehensive and clinically distinct patterns (clusters) of patient journeys that represent the main ways patients are diagnosed and treated for back pain. We further characterized demographic and utilization metrics for each cluster and observed clear differentiation between the clusters in terms of both clinical content and patient characteristics. Despite being a complex and often noisy data source, administrative claims provide a unique longitudinal overview of patient care across multiple service providers and locations. This methodology leverages claims to capture a data-driven understanding of how patients traverse the healthcare system. When tailored to various conditions and patient settings, this methodology can provide accurate overviews of patient journeys and facilitate a shift toward high-quality practice patterns.

Development of Density Functional Tight-Binding Parameters Using Relative Energy Fitting and Particle Swarm Optimization.

We provide a strategy to optimize density functional tight-binding (DFTB) parameterization for the calculation of the structures and properties of organic molecules consisting of hydrogen, carbon, nitrogen, and oxygen. We utilize an objective function based on similarity measurements and the Particle Swarm Optimization (PSO) method to find an optimal set of parameters. This objective function considers not only the common DFTB descriptors of binding energies and atomic forces but also incorporates relative energies of isomers into the fitting procedure for more chemistry-driven results. The quality in the description of the binding energies and atomic forces is measured based on the Ballester similarity index and relative energies through a similarity index induced by the Levenshtein edit distance to quantify the correct energetic order of isomers. Training and testing datasets were created to include all relevant chemical functional groups. The accuracy of this strategy is assessed, and its range of applicability is discussed by comparison against our previous parameterization [A. Krishnapriyan, et al., J. Chem. Theory Comput. 13, 6191 (2017)]. The improved performance of the new DFTB parameterization is validated with respect to the density functional theory large datasets QM-9 [R. Ramakrishnan, et al., Sci. Data 1, 140022 (2014)] and ANI-1 [J. S. Smith, et al., Sci. Data 4, 170193 (2017)], where excellent agreement is found between the structures and properties available in these datasets, and the ones obtained with DFTB.

Protein function prediction from dynamic protein interaction network using gene expression data.

Computational prediction of functional annotation of proteins is an uphill task. There is an ever increasing gap between functional characterization of protein sequences and deluge of protein sequences generated by large-scale sequencing projects. The dynamic nature of protein interactions is frequently observed which is mostly influenced by any new change of state or change in stimuli. Functional characterization of proteins can be inferred from their interactions with each other, which is dynamic in nature. In this work, we have used a dynamic protein-protein interaction network (PPIN), time course gene expression data and protein sequence information for prediction of functional annotation of proteins. During progression of a particular function, it has also been observed that not all the proteins are active at all time points. For unannotated active proteins, our proposed methodology explores the dynamic PPIN consisting of level-1 and level-2 neighboring proteins at different time points, filtered by Damerau-Levenshtein edit distance to estimate the similarity between two protein sequences and coefficient variation methods to assess the strength of an edge in a network. Finally, from the filtered dynamic PPIN, at each time point, functional annotations of the level-2 proteins are assigned to the unknown and unannotated active proteins through the level-1 neighbor, following a bottom-up strategy. Our proposed methodology achieves an average precision, recall and F-Score of 0.59, 0.76 and 0.61 respectively, which is significantly higher than the reported state-of-the-art methods.

Automating Error Frequency Analysis via the Phonemic Edit Distance Ratio.

Purpose Many communication disorders result in speech sound errors that listeners perceive as phonemic errors. Unfortunately, manual methods for calculating phonemic error frequency are prohibitively time consuming to use in large-scale research and busy clinical settings. The purpose of this study was to validate an automated analysis based on a string metric-the unweighted Levenshtein edit distance-to express phonemic error frequency after left hemisphere stroke. Method Audio-recorded speech samples from 65 speakers who repeated single words after a clinician were transcribed phonetically. By comparing these transcriptions to the target, we calculated the percent segments with a combination of phonemic substitutions, additions, and omissions and derived the phonemic edit distance ratio, which theoretically corresponds to percent segments with these phonemic errors. Results Convergent validity between the manually calculated error frequency and the automated edit distance ratio was excellent, as demonstrated by nearly perfect correlations and negligible mean differences. The results were replicated across 2 speech samples and 2 computation applications. Conclusions The phonemic edit distance ratio is well suited to estimate phonemic error rate and proves itself for immediate application to research and clinical practice. It can be calculated from any paired strings of transcription symbols and requires no additional steps, algorithms, or judgment concerning alignment between target and production. We recommend it as a valid, simple, and efficient substitute for manual calculation of error frequency.

Spatial Analysis of New Testament Textual Emendations Utilizing Confusion Distances

Abstract Before the interpretation of any text can start, the original wording of the text itself must be critically established. Conventionally, this is done following qualitative criteria. This article, however, explores the application of spatial analyses to New Testament textual criticism by demonstrating how the Levenshtein edit distance could be adapted to calculate confusion distances for variant readings in New Testament manuscripts, i.e. the possibility that a (combination of) letter(s) is confused by another (combination of) letter(s). Subsequently the outcomes are translated to Euclidian space using classical Multi-Dimensional Scaling, which enables visualisation and spatial analyses (in this case not related to geographical space). The article focuses on the data preparation and algorithm to make the data suitable for spatial analyses, thus providing the New Testament textual critic with new analytical tools.

Exploring Edit Distance for Normalising Out-of-Vocabulary Malay Words on Social Media

Users interact using short-formed words and abbreviations and this results in a message full of noisy words that are not recognized by the system's knowledge. The aim of this research is to overcome the limitations that still bar the progression of normalizing Malay noisy words from social media platforms. The testing data gathered is 25,000; 15,000 Tweets from Twitter and 10,000 comments from Facebook respectively. Pre-processing steps were carried out to clean the entire dataset which consists of unique 179,786 words. 36,587 out-of-vocabulary (OOV) Malay terms were then extracted and checked against an in- vocabulary (IV) Malay corpus using the Levenshtein edit distance formula and character manipulation rules. The resultant output is 3,964 unique IV Malay words. Based on the results, the usage of edit distance and rules can be further improved to elevate the normalisation of the ever changing colloquial terms of the Malay language.

Классификация последовательностей на основе коротких мотивов

Задачи, связанные с классификацией последовательностей символов некоторого алфавита, часто возникают в таких областях, как биоинформатика и обработка естественного языка. Методы глубокого обучения, в особенности модели на основе рекуррентных нейронных сетей, в последние несколько лет зарекомендовали себя как наиболее эффективный способ решения подобных задач. Однако существующие подходы имеют серьезный недостаток — низкую интерпретируемость получаемых результатов. Крайне сложно установить какие именно свойства входной последовательности ответственны за её принадлежность к тому или иному классу. Упрощение же таких моделей с целью повышения их интерпретируемости, в свою очередь, приводит к снижению качества классификации. Такие недостатки ограничивают применение современных методов машинного обучения во многих предметных областях. В настоящей работе мы представляем принципиально новую, интерпретируемую архитектуру нейронных сетей, основанную на поиске набора коротких подпоследовательностей — мотивов, наличие которых влияет на принадлежность последовательности к определенному классу. Ключевой составляющей предлагаемого решения является разработанный нами алгоритм дифференцируемого выравнивания, являющийся дифференцируемым аналогом таких классических способов сравнения строк, как редакционное расстояние Левенштейна и алгоритм Смита–Ватермана. В отличие от предыдущих работ, посвященных классификации последовательностей на основе мотивов, новый метод позволяет не только выполнять поиск в произвольной части строки, но и учитывать возможные вставки.

Interactive feature selection for efficient customer recognition in contact centers: Dealing with common names

We propose an interactive decision-making framework to assist a Customer Service Representative (CSR) in the efficient and effective recognition of customer records in a database with many ambiguous entries. Our proposed framework consists of three integrated modules. The first module focuses on the detection and resolution of duplicate records to improve effectiveness and efficiency in customer recognition. The second module determines the level of ambiguity in recognizing an individual customer when there are multiple records with the same name. The third module recommends the series of feature-related questions that the CSR should ask the customer to enable rapid recognition, based on that level of ambiguity. In the first module, the F-Swoosh approach for duplicate detection is used, and in the second module a dynamic programming-based technique is used to determine the level of ambiguity within the customer database for a given name. In the third module, Levenshtein edit distance is used for feature selection in combination with weights based on the Inverse Document Frequency (IDF) of terms. The algorithm that requires the minimum number of questions to be put to the customer to achieve recognition is the algorithm that is chosen. We evaluate the proposed framework on a synthetic dataset and demonstrate how it assists the CSR to rapidly recognize the correct customer.

IMPLEMENTASI FUZZY SEARCH UNTUK PENDETEKSI KATA ASING PADA DOKUMEN MICROSOFT WORD

Detection of foreign words is intended to help in reducing errors in the writing of scientific papers and thesis statements as of workmanship. There are rules in writing scientific papers that document must meet the rules of Indonesian rule of good writing and really one of them is use of italics in the word foreign languages (English). Therefore, made an application to aid in the detection of foreign words so as to tilt the letters that do not exist in the governance of the Indonesian word. In the making of the application will use the fuzzy search method for detecting foreign word using levenshtein edit distance matrix to calculate the distance semblance of a word. Fuzzy search conducted with fuzzy matching which returns a list of results based on the variables that have been determined. A fuzzy search using a matrix with levenshtein direpresentaasikan distance to calculate the distance of the similarity of the word. so will produce a word that has italic style and not if there is in Indonesian grammar. From the test results, the use of fuzzy search algorithms to detect foreign words in a word document with a different number of words resulted in an average accuracy value of 89.6%. the results of this process makes this application can help foreign grammar in writing scientific papers.