Levenshtein Distance Algorithm Research Articles

Deduplication Methods Using Levenshtein Distance Algorithm

The study aimed to propose methods to improve the data integrity of the Relational databases such as MS SQL, MySQL and PostgreSQL via record duplication detection. The FODORS and ZAGAT Restaurant database benchmark datasets have been utilized to facilitate the processes involved in preparing and delivering high-quality data. Furthermore, the Levenshtein distance algorithm was used to propose three (3) methods namely: default, eliminating equal string, and knowledge-based libraries to cut duplicate records in the database. In the 70% selected threshold, the average detected duplicate records of 88 out of 112 records in the restaurant dataset. Finally, to efficiently detect duplicate records in the database, depend on the data being analyzed and threshold selected.

Implementation of Levenshtein Distance Algorithm for Product Search Query Suggestions on Koro Pedang Edutourism E-Commerce

Users sometimes write queries that are inaccurate or typos in the product search contained in the Koro Pedang Educational Tourism e-commerce, so the system is not find product search results because the query entered in the system is incorrect. This can frustrate users because they cannot find the product they are looking for, so the users leave the website. According to these problems, it is necessary to suggest a query on the product search function. This is expected to assist users in finding the product they are looking for if there is an error in typing the query. This research purposes were to implement the Levenshtein Distance Algorithm for product search query suggestions on Koro Pedang Educational Tourism e-commerce. The stages of this research, namely the development of the search module, implementation of the Levenshtein Distance Algorithm and testing. The implementation of the Levenshtein Distance Algorithm in the search function for Koro Pedang Educational Tourism e-commerce products, a Suggestion Query is generated for Query typos in the search function with an accuracy value of 90%, Precision 95% and Recall 90.9%. This shows that the performance of the algorithm that has been applied to the search function for query suggestion is very good. The application of the Levenshtein Distance Algorithm gives a positive value to the usability of searching for e-commerce products for Koro Pedang Educational Tourism.

Retracted: Mitigation Impact of Energy and Time Delay for Computation Offloading in an Industrial IoT Environment Using Levenshtein Distance Algorithm

Retracted: Mitigation Impact of Energy and Time Delay for Computation Offloading in an Industrial IoT Environment Using Levenshtein Distance Algorithm

Search Optimization of PIP Scholarship Recipients In Web-Based Student Data Application Using The Levenshtein Distance Algorithm

Realizing that education is very important, the government supports every citizen to get education. One of the government programs is the Smart Indonesia Program. PlP is a scholarship designed to help school-age children from poor/vulnerable families to continue to receive education services, both through formal elementary to high school/vocational schools and non-formal pathways from package a to package c and special education. SDN II Babakanloa has not been touched by technology for processing student data. So that the student section has difficulties in recording and updating student data. Student names have unique identities and errors often occur in typing the keywords to be searched. This results in an information that is desired or sought can not be found. Therefore we need a web-based data application that can provide keyword corrections in searching for student names. This study aims to create a web-based student data application by optimizing corrections to typing keywords searched by implementing the Levenshtein Distance Algorithm and also making it easier to process and search student data. The development method used is the Rational Unified Process (RUP) with the stages of Inception, Elaboration, Construction, and Transition. Designed using the CodeIgniter Framework with the PHP and JavaScript programming languages. The application of the Levenshtein Distance Algorithm can optimize the search for student data and reduce the occurrence of search errors by School Operators. The application of the Levenshtein Distance Algorithm produces a very good accuracy rate of 94% of the results of student data correction. accordance with the expectations of the School Operator. So it shows that the application of the Levenshtein Distance Algorithm is appropriate to use in optimizing the search.

Rancang Bangun Chatbot Pengaduan Kekerasan Perempuan Anak Dengan Metode Fuzzy String Matching Dan Enhanced Confix Stripping Stemmer

Violence against women is a serious issue affecting many individuals in society. In Indonesia, including Kota Probolinggo, the number of cases of violence against women remains high. This research aims to design a chatbot application as a means for reporting violence against women and children (VAWC) in Kota Probolinggo. The application provides an anonymous and safe space for victims to report their complaints without revealing their identities. Additionally, the chatbot delivers information and education about different types of violence against women and preventive measures. The development of the chatbot application utilizes Fuzzy String Matching, Enhanced Confix Stripping Stemmer, and preprocessing methods to enhance data search accuracy and provide appropriate responses to users. The testing results show that the chatbot offers responses with satisfactory accuracy. Leveraging AI technology in the form of a chatbot is expected to provide an effective solution in addressing and preventing violence against women, offering a secure and anonymous platform for victims to report incidents. The application is envisioned to offer better support and assistance to victims of violence.
 The chatbot application implements fuzzy string matching, the Levenshtein distance algorithm, and Enhanced Confix Stripping Stemmer to process user input before matching it with the database. During the Enhanced Confix Stripping Stemmer process, sentences are transformed into an array of basic words, enhancing the accuracy of data matching when using fuzzy string matching and the Levenshtein distance algorithm. Testing results indicate a 4.77% difference between manual calculations and the chatbot system. This difference may be attributed to variations in text processing and other factors.

Similarity Detection in Two or More Submitted Documents

Abstract: Educational Institutions & Organizations of the modern era are rapidly adopting the hybrid work culture, in the aftermath of the worldwide pandemic. Institutions & Organizations frequently require authentic, original content for a wide variety of purposes. Though students & employees strive towards that objective, plagiarized content often slips through scrutinization more often than the acceptable count. This paper aims to analyze and record the process & outcomes of building a web application that enables similarity detection between two or more files by uploading them on the application, or in other words, determine the similarity between the files. The web application will accept two or more files as input, and return similarity percentage which will be calculated using the Levenshtein distance Algorithm, as an automated report that will be sent to the user as an email. In our case, the web application will be majorly utilized to curb similarity in assignments. Along with this, the application will have a feature that allows any teacher to compare any two or more assignments submitted in a particular class, with just one click.

Iliash Implementation of the password complexity diagnostic system

Password diagnostic systems are becoming increasingly relevant in the modern world due to the growth of internet services and the need to protect personal data. Evaluating password complexity allows for assessing the strength of passwords against different types of attacks, including brute force, dictionary attacks, and other methods. In this work, a system for checking password complexity was developed, which takes into account new approaches to forming the uniqueness of the selected sequence of characters. As opposed to standard methods used in the most common password generating systems, such as length, the presence of special characters and numbers, and the use of upper and lower case letters, this work proposes to check for repetitions and matches with the most common passwords. For this purpose, a database of 10,000 passwords was connected to the developed system. To speed up the checking for matches, the Levenshtein distance algorithm was used. The Levenshtein algorithm works using dynamic programming. It considers two strings that need to be compared and creates a matrix that shows the minimum number of editing operations required to transform one string into the other. This implementation allowed for faster password generation compared to existing systems. The work details the algorithms for all the main stages of password complexity checking and proposes code implementation in Javascript. The Levenshtein algorithm can be useful for determining text similarity or searching for possible correct variants of misspelled words. It can also be used for implementing autocomplete in programs that use user input. The implemented system can be integrated into many different programs and web services for automatic password complexity determination, allowing for its use in various areas such as banking, medical services, and others. As a result of work on this project, it has been confirmed that using a password complexity diagnostic system can significantly reduce the risk of unauthorized access to systems containing important confidential information.

Implementation of the Levenshtein Distance Algorithm and the Regular Search Expression Method for Detecting Typors in Javascript

Implementation of the Levenshtein Distance Algorithm and the Regular Search Expression Method for Detecting Typors in Javascript

Leveraging Text-Mining Techniques On Electronic Medical Records to Analyze National Drug-insured Medication Use

Processing electronic medical record (EMR) data has become a common practice among scientists for extracting valuable insights and studying diseases. Given the large volumes of text data in EMRs, efficient computerized text-mining techniques are necessary. As academics, we recognize that drug-used analysis from EMR data in Indonesia is currently limited. This study focuses on obtaining meaningful insights from EMR data to make positive recommendations for hospitals. The proposed method uses pattern-based Regular Expressions (regex) to extract drug names and a Levenshtein distance algorithm to check their compatibility. We developed the pattern based on analyzing Indonesia EMR data. The extracted drug names were compared to a list of selected drugs (National Drug-Insured/Fornas) that are required and must be provided at healthcare facilities in Indonesia. The Levenshtein distance threshold was set to two to decide whether the extracted drug names belonged to nationally drug-insured or not. Only about 11.09 – 16.11% of medications given by doctors are listed in the Fornas drug list. Between 2019 and 2021, there was an inaccuracy in the writing of prescriptions for Fornas drugs, with as many as 57.53% to 63.21% of drug names being written incorrectly. The results of this study indicate that the Levenshtein distance algorithm has promising potential for implementation in the Ministry of Health of Indonesia, with a precision rate of 97.07%.

Investigating the Impact of Utilizing the K-Nearest Neighbor and Levenshtein Distance Algorithms for Arabic Sentiment Analysis on Mobile Applications

The field of sentiment analysis of Arabic scripts remains a major challenge due to the features, characteristics, and complexity of the Arabic language. Few studies on Arabic Sentiment Analysis (ASA) have been conducted to the best of our knowledge when compared to English or other Latin languages. In addition, for ASA, very few studies have been conducted regarding the comments of the users (customers) of the apps available on the Google Play Store within various mobile application reviews. Most of the current studies have been conducted on datasets collected from Twitter user comments. In this paper, we propose a new approach to the analysis of sentiment in Arabic script based on the comments dataset of users of some mobile applications available on the Google Play Store. The proposed approach involves improving algorithms such as the Levenshtein distance (LD) algorithm for data preprocessing and then combining it with the K-Nearest Neighbor (K-NN) algorithm. Through the proposed approach, the results of the experiment were shown, investigating the impact of utilizing the K-NN and LD algorithms for ASA on mobile applications effectively. The experiments were carried out. The K-NN with LD algorithm has gained a better level of evaluation compared to the K-NN algorithm. K-NN with LD algorithm has achieved the highest accuracy, recall, precision, and F-score, which were 83.11% in accuracy, 66.30% in recall, 85.10% in precision, and 74.53% in f-score evaluation measure when =3.

Adapting Off-the-Shelf Speech Recognition Systems for Novel Words

Current speech recognition systems with fixed vocabularies have difficulties recognizing Out-of-Vocabulary words (OOVs) such as proper nouns and new words. This leads to misunderstandings or even failures in dialog systems. Ensuring effective speech recognition is crucial for the proper functioning of robot assistants. Non-native accents, new vocabulary, and aging voices can cause malfunctions in a speech recognition system. If this task is not executed correctly, the assistant robot will inevitably produce false or random responses. In this paper, we used a statistical approach based on distance algorithms to improve OOV correction. We developed a post-processing algorithm to be combined with a speech recognition model. In this sense, we compared two distance algorithms: Damerau–Levenshtein and Levenshtein distance. We validated the performance of the two distance algorithms in conjunction with five off-the-shelf speech recognition models. Damerau–Levenshtein, as compared to the Levenshtein distance algorithm, succeeded in minimizing the Word Error Rate (WER) when using the MoroccanFrench test set with five speech recognition systems, namely VOSK API, Google API, Wav2vec2.0, SpeechBrain, and Quartznet pre-trained models. Our post-processing method works regardless of the architecture of the speech recognizer, and its results on our MoroccanFrench test set outperformed the five chosen off-the-shelf speech recognizer systems.

OPTICAL CHARACTER RECOGNITION MENGGUNAKAN UIPATH DAN PENCOCOKAN DATA SERTIFIKAT DENGAN ALGORITMA LEVENSHTEIN DISTANCE

The collection of the certificates was one of the requirements for graduating from Tarumanagara University that the certificate became an important point of attention to improving the competency of Tarumanagara University students. Certificates can be collected from seminars, workshops, courses, and so forth. The Certificate Information Extraction design were created using Uipath Studio application that uses vb.Net programming language and Levenshtein Distance Algorithm. The design aims to assist the study program in validation on student certificates file by obtaining better accuracy using the Levenshtein Distance Algorithm.. The design uses input data as student certificates that’s next to be processed by text preprocessing consisting of text deductions (parsing), case folding, lexical analysis (tokenizing), and text removal (stopword removal). After processing, the Levenshtein Distance Algorithm will be used to calculate the minimum distance between one text and the other with a two-dimensional matrix operation, thus determining the validity of student certificates. The results of this design represent that using the Levenshtein Distance Algorithm, obtaining the best word accuracy result of 83.52% and RPA running time of 94.7 ms.

An efficient regular expression inference approach for relevant image extraction

Traditional approaches for extracting relevant images automatically from web pages are error-prone and time-consuming. To improve this task, operations such as preparing a larger dataset and finding new features are used in the web data extraction approaches. However, these operations are difficult and laborious. In this study, we propose a fully-automated approach based on alignment of regular expressions to automatically extract the relevant images from web pages. The automatically constructed regular expressions has been applied to a classification task for the first time. In this respect, a multi-stage inference approach is developed for generating regular expressions from the attribute values of relevant and irrelevant image elements in web pages. The proposed approach reduces the complexity of the alignment of two regular expressions by applying a constraint on a version of the Levenshtein distance algorithm. The classification accuracy of regular expression approaches is compared with the naive Bayes, logistic regression, J48, and multilayer perceptron classifiers on a balanced relevant image retrieval dataset consisting of 360 image element samples for 10 shopping websites. According to the cross-validation results, the regular expression inference-based classification achieved a 0.98 f-measure with only 5 frequent n-grams, and it outperformed other classifiers on the same set of features. The classification efficiency of the proposed approach is measured at 0.108 ms, which is very competitive with other classifiers.

BERT Model-based Natural Language to NoSQL Query Conversion using Deep Learning Approach

Databases are commonly used to store complex and distinct information. With the advancement of the database system, non-relational databases have been used to store a vast amount of data as traditional databases are not sufficient for making queries on a wide range of massive data. However, storing data in a database is always challenging for non-expert users. We propose a conversion technique that enables non-expert users to access and filter data as close to human language as possible from the NoSQL database. Researchers have already explored a variety of technologies in order to develop more precise conversion procedures. This paper proposed a generic NoSQL query conversion learning method to generate a Non-Structured Query Language from natural language. The proposed system includes natural language processing-based text preprocessing and the Levenshtein distance algorithm to extract the collection and attributes if there were any spelling errors. The analysis of the result shows that our suggested approach is more efficient and accurate than other state-of-the-art methods in terms of bilingual understudy scoring with the WikiSQL dataset. Additionally, the proposed method outperforms the existing approaches because our method utilizes a bidirectional encoder representation from a transformer multi-text classifier. The classifier process extracts database operations that might increase the accuracy. The model achieves state-of-the-art performance on WikiSQL, obtaining 88.76% average accuracy.

Variable indexing method in rule documents for ship design using extraction of portable document format elements

Abstract Design rules for ships have become more extensive and detailed due to an increase in the sizes of ships. Several variables and equations used in the rules are complex, thereby impeding their review by reviewers due to their voluminosity. In addition, because these rules are constantly revised, professional investigators may miss these changes. To prevent such confusion, a shipping register, which approves ship drawings, constantly automates the search and review processes of the rules. Consequently, this study proposes a method for recognizing variables in documents to review the rules and build relationships between variables. Each component of a document must be accurately identified. The document containing these rules includes different components such as equations, figures, and strings. Because these rules are mainly converted to a portable document format (PDF) for compatibility, it is challenging to extract each component as raw data. This study used a public library to extract elements from the PDF and utilized the positional relationship between the elements to identify the variables. By applying the Levenshtein distance algorithm, which compares the differences between two strings, the document was partitioned following to the table of contents. Hence, the identified variables were indexed into sections of the table of content. Additionally, based on the indexed information, a data structure was proposed to show the equations, definition of variables, and relationships. This study applied it to common structural rules, which are widely used in the shipbuilding industry. The effectiveness of the proposed method was confirmed by achieving the F1 score = 0.93 in variable recognition and intuitively visualizing the relationship between the variables.

Web-Based Mail Search Using the Levenshtein Distance Algorithm

The use of information technology in filing letters has been widely applied, especially in government institutions. In the Administrative Affairs of the Police Information Technology Bureau, the archiving of incoming and outgoing letters is still done manually, by utilizing records in a large agenda book and stored in a large folder. The absence of a system capable of assisting mail processing causes work to be inefficient, especially in terms of searching for letters. The purpose of this research is to produce a website that can help search letters by applying the Levenshtein Distance Algorithm. The design begins with needs analysis, design, implementation, verification (system testing), operation and maintenance. A website that can search letters by applying the Levenshtein Distance Algorithm was generated. This website is very helpful in the process of searching for letters to the Administrative Affairs of the Police Information Technology Bureau.

Accurate and Language Agnostic Code Clone Detection by Measuring Edit Distance of ANTLR Parse Tree

In spite of significant research done in the past 3 decades introducing more than 250 clone detection tools/ techniques for finding the same language clones, there exists no single framework to detect and classify all 4 basic types of clones with great accuracy (precision and recall). In this paper, we propose an accurate and language agnostic technique to classify 4 types of clones. The method first generates an ANTLR parse tree for the input program file using freely available ANTLR grammar files then finds the edit distance between the two parse trees using the Levenshtein distance algorithm and converts the edit distance into similarity using. We obtained 100% precision and recall in detecting type 1 & 2 clone types and achieve 98.50 and 98.12 respectively for type 3 and 4 clone types for our datasets containing microprograms of C, CPP, and Java. This paper provides evidence that the Levenshtein distance on ANTLR parse tree is the good choice to build a complete and accurate software clone detector and act as proper validation tools to detect code plagiarism.

PENERAPAN ALGORITMA LEVENSHTEIN DISTANCE PADA SISTEM PENCARIAN DATA BUKU BERBASIS WEB

In the world of education today really need a search engine that makes it easier for students to search for information related to book data. One of them is in the library of SMP N 2 SOOKO, the process of finding the number and stock of books at SMP N 2 SOOKO can take a lot of money and time because it needs to be recorded manually. With this web-based search system, it can save costs and time because the process is carried out using a computer. This study aims to create a web-based book data search system that makes it easier for the SMP N 2 SOOKO library in the process of searching for the number and stock of books as well as in the archiving process. This search system uses the Levenshtein distance algorithm which is used to find the smallest string distance or distance, so that the book data entered into the search column displays results that are close to the keywords. For the data collection method using the method of observation and interviews. The tools used to describe this system model are in the form of flowcharts, context diagrams and level 1 DFD. The software used for design and implementation is XAMPP as a webserver, phpMySQL as a database and notepad++ as an editor for writing PHP and HTML files. The research shows that the search for book data using the single target formula is 75% and multi target is 87%.

Perceptual effects of interpolated Austrian and German standard varieties

This article focuses on the perception of standard varieties produced by Austrian and German TV newscasters from the perspective of listeners from both countries, Germany and Austria. Thus, the paper's sociolinguistic scope is located in the pluricentric realm. It assesses (the perception of) standard language variation on fine phonetic levels. In addition to naturally produced stimuli (read sentences), “intermediate” samples were generated by means of a two-step interpolation procedure: First, the symbolic phone sequences were aligned with a Levenshtein-distance algorithm. Second, a phone-level Dynamic-Time-Warping (DTW) algorithm was applied to align the two utterances on a frame level, taking phoneme boundaries into account. Additionally, the spectrum and the fundamental frequency (F0) contour of the utterances was manipulated to be either interpolated or fixed to a given speaker. This procedure allowed for assessing these features’ contribution to listeners’ judgments of whether a given utterance sounds as if spoken by a speaker from (rather) Germany or (rather) Austria. Results of the judgment task showed that the interpolated samples were perceived in a continuous fashion, similarly by both groups, and with overall greater reliance on spectral than F0 information. These findings suggest that even fine phonetic differences between Standard German from Germany and Austria are recognized and evaluated by listeners from both countries. An overall bias towards perceiving all speech samples as closer to the listeners’ own national standard points to a factor of familiarity with one's own and uncertainty towards the ‘foreign’ standard. The similar degree of this bias indicates that both groups have similar levels of familiarity with their own and uncertainty about the other standard variety.

3GOLD: optimized Levenshtein distance for clustering third-generation sequencing data

BackgroundThird-generation sequencing offers some advantages over next-generation sequencing predecessors, but with the caveat of harboring a much higher error rate. Clustering-related sequences is an essential task in modern biology. To accurately cluster sequences rich in errors, error type and frequency need to be accounted for. Levenshtein distance is a well-established mathematical algorithm for measuring the edit distance between words and can specifically weight insertions, deletions and substitutions. However, there are drawbacks to using Levenshtein distance in a biological context and hence has rarely been used for this purpose. We present novel modifications to the Levenshtein distance algorithm to optimize it for clustering error-rich biological sequencing data.ResultsWe successfully introduced a bidirectional frameshift allowance with end-user determined accommodation caps combined with weighted error discrimination. Furthermore, our modifications dramatically improved the computational speed of Levenstein distance. For simulated ONT MinION and PacBio Sequel datasets, the average clustering sensitivity for 3GOLD was 41.45% (S.D. 10.39) higher than Sequence-Levenstein distance, 52.14% (S.D. 9.43) higher than Levenshtein distance, 55.93% (S.D. 8.67) higher than Starcode, 42.68% (S.D. 8.09) higher than CD-HIT-EST and 61.49% (S.D. 7.81) higher than DNACLUST. For biological ONT MinION data, 3GOLD clustering sensitivity was 27.99% higher than Sequence-Levenstein distance, 52.76% higher than Levenshtein distance, 56.39% higher than Starcode, 48% higher than CD-HIT-EST and 70.4% higher than DNACLUST.ConclusionOur modifications to Levenshtein distance have improved its speed and accuracy compared to the classic Levenshtein distance, Sequence-Levenshtein distance and other commonly used clustering approaches on simulated and biological third-generation sequenced datasets. Our clustering approach is appropriate for datasets of unknown cluster centroids, such as those generated with unique molecular identifiers as well as known centroids such as barcoded datasets. A strength of our approach is high accuracy in resolving small clusters and mitigating the number of singletons.