Jaro-Winkler Distance Research Articles

Using Polars to Improve String Similarity Performance in Python

IntroductionString similarity is central to textual record linkage, and is often calculated with Levenshtein distance or Jaro-Winkler distance. In polars-strsim, we leverage the Polars DataFrame interface to surpass all existing Python libraries in computing these distances. Polars-strsim is especially useful for larger-than-memory datasets. Objectives and ApproachPolars is a library built to analyse and manipulate tabular data. It is written in Rust, but can be called from Python for ease of use. Polars-strsim implements the Levenshtein and Jaro-Winkler algorithms as a Polars extension. We compare performance against nine other Python libraries on three million pairs of first names. ResultsWhen computing Levenshtein/Jaro-Winkler distance for this benchmark, polars-strsim is respectively 5x/4x faster than the current fastest Python alternative and 47x/35x faster than the median. The Polars streaming engine can be seamlessly used to manipulate larger than memory datasets. One practical application of this in record linkage is to find all pairs of records in a dataset where the Levenshtein distance computed across one (or more) fields is below a certain threshold (e.g., during blocking). With polars-strsim, this calculation can be performed in Python with CPU parallelism even if the input and/or output doesn’t fit in memory. Conclusions/ImplicationsThis work introduces a powerful library for record linkage practitioners. Polars-strsim provides the convenience of Python, the performance of a strongly/statically typed compiled language, and the full power of the Polars streaming engine when computing string similarity, and can be updated to implement additional similarity algorithms.

Framework for Measuring Relevancy in Discovery Environments

Institutional discovery environments now serve as central resource databases for researchers in the academic environment. Over the last several decades, there have been numerous discovery layer research inquiries centering primarily on user satisfaction measures of discovery system effectiveness. This study focuses on the creation of a largely automated method for evaluating discovery layer quality, utilizing the bibliographic sources from student research projects. Building on past research, the current study replaces a semiautomated Excel Fuzzy Lookup Add-In process witha fully scripted R-based approach, which employs the stringdist R package and applies the Jaro-Winkler distance metric as the matching evaluator. The researchers consider the error rate incurred by relying solely on an automated matching metric. They also use Open Refine for normalization processes and package the tools together on an OSF site for other institutions to use. Since the R-based approach does not require special processing or time and can be reproduced with minimal effort, it will allow future studies and users of our method to capture larger sample sizes, boosting validity. While the assessment process has been streamlined and shows promise, there remain issues in establishing solid connections between research paper bibliographies and discovery layer use. Subsequent research will focus on creating alternatives to paper titles as search proxies that better resemble genuine information-seeking behavior and comparing undergraduate and graduate student interactions within discovery environments.

Generate fuzzy string-matching to build self attention on Indonesian medical-chatbot

Chatbot is a form of interactive conversation that requires quick and precise answers. The process of identifying answers to users’ questions involves string matching and handling incorrect spelling. Therefore, a system that can independently predict and correct letters is highly necessary. The approach used to address this issue is to enhance the fuzzy string-matching method by incorporating several features for self-attention. The combination of fuzzy string-matching methods employed includes Jaro Winkler distance + Levenshtein Damerau distance and Damerau Levenshtein + Rabin Carp. The reason for using this combination is their ability not only to match strings but also to correct word typing errors. This research contributes by developing a self-attention mechanism through a modified fuzzy string-matching model with enhanced word feature structures. The goal is to utilize this self-attention mechanism in constructing the Indonesian medical bidirectional encoder representations from transformers (IM-BERT). This will serve as a foundation for additional features to provide accurate answers in the Indonesian medical question and answer system, achieving an exact match of 85.7% and an F1-score of 87.6%.

Arabic spellchecking: a depth-filtered composition metric to achieve fully automatic correction

<p>Digital environments for human learning have evolved a lot in recent years thanks to incredible advances in information technologies. Computer assistance for text creation and editing tools represent a future market in which natural language processing (NLP) concepts will be used. This is particularly the case of the automatic correction of spelling mistakes used daily by data operators. Unfortunately, these spellcheckers are considered writing aids tools, they are unable to perform this task automatically without user’s assistance. In this paper, we suggest a filtered composition metric based on the weighting of two lexical similarity distances in order to reach the auto-correction. The approach developed in this article requires the use of two phases: the first phase of correction involves combining two well-known distances: the edit distance weighted by relative weights of the proximity of the Arabic keyboard and the calligraphical similarity between Arabic alphabet, and combine this measure with the JaroWinkler distance to better weight, filter solutions having the same metric. The second phase is considered as a booster of the first phase, this use the probabilistic bigram language model after the recognition of the solutions of error, which may have the same lexical similarity measure in the first correction phase. The evaluation of the experimental results obtained from the test performed by our filtered composition measure on a dataset of errors allowed us to achieve a 96% of auto-correction rate.</p>

Intelligent Edge Based Efficient Disease Diagnosis using Optimization Based Deep Maxout Network

The healthcare model is considered an imperative part of remote sensing of health. Finding the disease requires constant monitoring of patients’ health and the detection of diseases. In order to diagnose the disease utilizing an edge computing platform, this study develops a method called grey wolf invasive weed optimization-deep maxout network (GWIWO-DMN). The proposed GWIWO, which is developed by integrating invasive weed optimization (IWO) and grey wolf optimization (GWO), is used here to train the DMN. The distributed edge computing platform consists of four units, namely monitoring devices, first layer edge server, second layer edge server, and cloud server. The monitoring devices are used for accumulating patient information. The preprocessing and feature selection are performed in the first layer edge server. Here, the preprocessing is carried out using the exponential kernel function. The selection of features is done using Jaro–Winkler distance in the first layer edge server. Then, at the second layer edge server, clustering and classification are carried out using deep fuzzy clustering and DMN, respectively. The proposed GWIWO algorithm is used to do the DMN training. Finally, the cloud server processes the decision fusion. The proposed GWIWO-DMN outperformed with the highest true positive rate (TPR) of 89.2%, highest true negative rate (TNR) of 93.7%, and highest accuracy of 90.9%.

The use of binary digit mapping on ASCII characters to create a high-capacity, undetectable text steganography.

Plain text is commonly utilized for online news and social media transmission due to its lightweight and cross-platform nature. Unfortunately, it may easily be exploited by attackers (For instance, in cases when the language is altered for specific goals, illegal access to the material or its abuse is possible.) To solve this problem, we embed secret texts (ST) within seemingly innocuous cover texts (CT) so that we can track CT for modifications. Based on the findings of this study, a method is provided for ST embedding in CT in which binary ST digits are converted to CT binary digits through Characters in ASCII (containing whitespace, punctuation, and special characters). Before commencing the embedding procedure, Using a One-Time Pad (OTP), the ST text was translated into ciphertext, and the value of each character was changed into a binary integer with a length of seven bits. As opposed to the ST text, which required additional time to encode, the CT text could be converted into a binary integer using just the first seven bits. During the embedding process, It was decided to swap the first bit of the ST character for the first bit of the CT character, which both had the same amount of bits (putting the first bit of the ST character, for example, on the same line as the first bit of the CT character). Any piece of ST that was recorded on any bit of CT served as a stage key. Extracting ST from CT requires the stage key. The embedded information served effectively as text steganography or watermark, and the CT's appearance was unaffected by the embedding procedure. In other words, the 1 character of ST might be concealed by using 7 CT characters. Also, a Jaro-Winkler Distance of 1 meant that the stage texts made couldn't be told apart from CT by looking at them.

Investigation of optimized ELM using Invasive Weed-optimization and Cuckoo-Search optimization

Abstract In order to classify data and improve extreme learning machine (ELM), this study explains how a hybrid optimization-driven ELM technique was devised. Input data are pre-processed in order to compute missing values and convert data to numerical values using the exponential kernel transform. The Jaro–Winkler distance is used to identify the relevant features. The feed-forward neural network classifier is used to categorize the data, and it is trained using a hybrid optimization technique called the modified enhanced Invasive Weed, a meta heuristic algorithm, and Cuckoo Search, a non-linear optimization algorithm ELM. The enhanced Invasive Weed optimization (IWO) algorithm and the enhanced Cuckoo Search (CS) algorithm are combined to create the modified CSIWO. The experimental findings presented in this work demonstrate the viability and efficacy of the created ELM method based on CSIWO, with good experimental result as compared to other ELM techniques.

Feature Extraction Based Deep Indexing by Deep Fuzzy Clustering for Image Retrieval Using Jaro Winkler Distance

Abstract Image retrieval aims to locate similar image based on given query from large-sized image dataset. In the recent decades, different methods are developed to retrieve similar images, but increasing the retrieval performance is a challenging task. Hence, an effective method named Lion Henry Gas Solubility Optimization-based Deep Fuzzy Clustering (LHGSO-based DFC) is developed to increase the retrieval efficiency of image retrieval. Based on the features extraction, images are indexed efficiently to the corresponding cluster using clustering model. The proposed method retrieves the relevant images with respect to query image based on the similarity. The similarity measure is computed using Jaro Winkler distance, in which the images corresponding to the cluster that have higher similarity measure is retrieved more effectively. The proposed method achieved higher performance in terms of the metrics, like F-measure, precision and recall with the values of 0.887, 0.891 and 0.882.

Performance Comparison of Similarity Measure Algorithm as Data Preprocessing Stage: Text Normalization in Bahasa

Purpose: More and more data are stored in text form due to technological developments, making text data processing more difficult. It also causes problems in the text preprocessing algorithm, one of which is when two texts are identical, but are considered distinct by the algorithm. Therefore, it is necessary to normalize the text to get the standard form of words in a particular language. Spelling correction is often used to normalize text, but for Bahasa Indonesia, there has not been much research on the spell correction algorithm. Thus, there needs to be a comparison of the most appropriate spelling correction algorithms for the normalization process to be effective.Methods: In this study, we compared three algorithms, namely Levenshtein Distance, Jaro-Winkler Distance, and Smith-Waterman. These algorithms were evaluated using questionnaire data and tweet data, which both are in Bahasa Indonesia.Result: The fastest normalization time is obtained by the Jaro-Winkler, taking an average of 31.01 seconds for questionnaire data and 59.27 seconds for tweet data. The best accuracy is obtained by the Levenshtein Distance with a value of 44.90% for the questionnaire data and 60.04% for the tweet data. Novelty: The novelty of this research is to compare the similarity measure algorithm in Bahasa Indonesia. Therefore, the most suitable similarity measure algorithm for Bahasa Indonesia will be obtained.

Quality Analysis of Extreme Learning Machine based on Cuckoo Search and Invasive Weed Optimization

This paper explicates hybrid optimization driven Extreme Machine Learning (ELM) strategy is developed with feed forward neural network (FFNN) for the classification of data and improving ELM. The pre-processing of input data is carried for the missing value imputation and transformation of data into numerical value using exponential kernel transform. The significant feature is determined using the Jaro–Winkler distance. The classification of data is done using the FFNN classifier, which is trained with the help of the hybrid optimization algorithm, namely developed modified Cuckoo Search and Invasive Weed Optimization (CSIWO) ELM. The modified CSIWO is devised by integrating the modified Cuckoo search (CS) algorithm and Invasive Weed Optimization (IWO) algorithm. The experimental results proposed in this paper show the feasibility and effectiveness of the developed CSIWO ELM method with encouraging performance compared with other ELM methods.

Active Verb Spell Checking Mem- + P in Indonesian Language Using the Jaro-Winkler Distance Algorithm

The Indonesian language is used as a means of communication, including written communication. Unfortunately, many mistakes are found in Indonesian language writing, such as the writing of active verbs with the prefix mem- followed by the letter P. This problem can be addressed with the spell-checking method. Spell checking is a process in computer programs to check the spelling of each word in electronic text or documents in the correct order. To better the active verb this study used the Jaro-Winkler Distance algorithm. Meanwhile, for system development, the Iterative Waterfall method was used. The system output is active verbs of mem- + P which is standardized according to the Great Dictionary of the Indonesian Language of the Language Center (KBBI) and the morphological rules of the Indonesian Language Spelling System. The accuracy value of the algorithm was 80% for common mistakes, 95% for specific mistakes, and 80% for correctable mistakes with word suggestions.

Software product line regression testing based on fuzzy clustering approach using distance method

Testing is a process that takes much time and effort in software companies. This becomes even more difficult and boring when it comes to testing a software product line (SPL). The SPL is a model in which multiple products from the same family are made simultaneously. Testing of all products is not possible. Hence a lot of testing methods have been given from time to time to test the product line, given by researchers based on contemporary conception. In the direction of testing product lines, this article has proposed a method, which used fuzzy C-means clustering with the Jaro-Winkler distance method. Variable features of the product form the basis for cluster development. The proposed method is compared with other distance methodologies. After comparison, it is concluded that the proposed method provides better results than other methods. This article has resorted to some product lines to compare with the proposed methods.

A new framework for Arabic recitation using speech recognition and the Jaro Winkler algorithm

Automated recitation plays an important role in improving self-learning. It is based on Speech/Text recognition. The research in Arabic speech recognition is very limited. The few existing applications are only based on the Holy Qur’an. This article proposed a new system (Samee’a - ) to facilitate memorizing any kind of text such that poems, speeches and the Holy Qur’an. Samee’a system is based on Google Cloud Speech Recognition API to convert the Arabic speech to text and Jaro Winkler Distance algorithm to determine the similarity between the original and converted texts. The system has been tested using 70 collected files ranging between 12 to 400 words and some chapters from the Holy Qur’an. The average similarity achieved 83.33% for the 70 files and 69% for the selected chapters of the Holy Qur’an. These results were enhanced to 91.33 % and 95.66% after applying preprocessing operations on the text files and the Holly Qur’an respectively. To validate the obtained results, two comparison studies were performed. The Jaro Winker distance was successfully compared to the cosine and the Euclidean distance. In addition, the proposed system outperformed the related work with an improvement of the similarity reaching 5% when using section 30 of the Holy Qur’an. Finally, the user experience testing was carried out by 10 users of different ages (between 5 and 50-year-old) using small texts and some small chapters of the Holy Qur’an. The proposed system proved its efficiency.

Food Ingredients Similarity Based on Conceptual and Textual Similarity

Open Food Facts provides a database of food products such as product names, compositions, and additives, where everyone can contribute to add the data or reuse the existing data. The open food facts data are dirty and needs to be processed before storing the data to our system. To reduce redundancy in food ingredients data, we measure the similarity of ingredient food using two similarities: the conceptual similarity and textual similarity. The conceptual similarity measures the similarity between the two datasets by its word meaning (synonym), while the textual similarity is based on fuzzy string matching, namely Levenshtein distance, Jaro-Winkler distance, and Jaccard distance. Based on our evaluation, the combination of similarity measurements using textual and Wordnet similarity (conceptual) was the most optimal similarity method in food ingredients.

Efficient fuzzy keyword search scheme over encrypted data in cloud computing based on Bed-tree index structure

In cloud computing, enabling search directly over encrypted data is an important technique to effectively utilize encrypted data. Most of the existing techniques are focusing on fuzzy keyword search as it helps achieve more robust search performance by tolerating misspelling or typos of data users. Existing works always build index without classifying keywords in advance. They suffer from efficiency issue. Furthermore, Euclidean distance or Hamming distance is often chosen to evaluate strings’ similarity, ignoring prefixes matching and the influence of strings’ length on the accuracy. We propose an efficient fuzzy keyword search scheme with lower computation cost and higher accuracy to address the aforementioned problems. We employ the sub-dictionaries technique and the Bed-tree structure to build an index with three layers for achieving better search efficiency. With this index structure, the server could locate the keyword and could narrow the search scope quickly. The Jaro-Winkler distance is introduced to qualify the strings’ similarity by considering the prefixes matching and string length. The secure privacy mechanism is incorporated into the design of our work. Security analysis and performance evaluation demonstrate our scheme is more efficient compared to the existing one while guaranteeing security.

Spell corrector for Bangla language using Norvig’s algorithm and Jaro-Winkler distance

In the online world, especially in the social media platform most of us write without much regard to correct spelling and grammar. The spelling mistakes are much larger in proportion when it comes to Bangla language. In our paper, we presented a method for error detection and correction in Bangla words' spellings. Our system could detect a misspelled Bangla word and provide two following services-suggesting correct spellings for the word and correcting the word. We had used Norvig's algorithm for the purpose but instead of using probabilities of the words to prepare the suggestions and corrections, we had used Jaro-Winkler distance. The previous works done in this field for Bangla language are either very slow or offers less accuracy. Our system successfully achieved a 97% accuracy when evaluated with 1000 Bangla words.

Tunicate Swarm Magnetic Optimization With Deep Convolution Neural Network For Collaborative Filter Recommendation

Abstract Collaborative filtering (CF) is a well-known and eminent recommendation technique to predict the preference of new users by revealing the structures of historical records of the examined users. Even though CF is effectively adapted in several commercial areas, many limitations still exist, particularly in the sparsity of rating data that raises many issues. This paper devises a novel deep learning strategy for CF to recognize user preferences. Here, black hole entropic fuzzy clustering (BHEFC) is devised for clustering item sequences to form groups with similar item sequences. Moreover, cluster centroids are optimized using the tunicate swarm magnetic optimization algorithm (TSMOA), which is devised by combining tunicate swarm algorithm and magnetic optimization algorithm. After grouping similar items together, the group matching is performed based on a deep convolutional neural network (Deep CNN). Subsequently, the visitor sequence and query sequence are compared using Jaro–Winkler distance, which contributes to the best visitor sequence. From this best visitor sequence, the recommended product is acquired. The proposed TSMOA–BHEFC and Deep CNN outperformed other methods with minimal mean absolute error of 0.200, mean absolute percentage error of 0.198 and root mean square error of 0.447, respectively.

Product Recommendation System Using Tunicate Swarm Magnetic Optimization Algorithm-Based Black Hole Renyi Entropy Fuzzy Clustering and K-Nearest Neighbour

The recommendation system takes the information related to the user profile or interest to suggest the user with convenient materials that the user is interested in. Most of the existing implicit methods find the user preferences and automatically recommend the desired products in the interface, but failed to generate user-oriented results. Hence, an effective product recommendation method is developed in this research using the proposed Tunicate Swarm Magnetic Optimisation Algorithm-based Black Hole Renyi Entropy Fuzzy Clustering+K-Nearest Neighbour (TSMOA-based BHrEFC+KNN) to generate more user convenient result by grouping relevant products and recommends the similar products to users with great interest. The proposed TSMOA is designed by integrating the Tunicate Swarm Algorithm (TSA) and Magnetic Optimisation Algorithm (MOA), respectively. With the entropy measure and Jaro–Winkler distance, the process of group matching and the matching sequence of visitor and query are performed more effectively that enable to achieve the sentiment classification based on the binary visitor sequence. The performance obtained by the proposed TSMOA-based BHrEFC+KNN is evaluated in terms of accuracy, True Positive Rate (TPR), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE). The performance of the proposed system is compared with the existing methods, such as Deep learning+Naïve–Bayes, K-medoids clustering+Long Short-Term Memory (LSTM), BHEFC+Support Vector Machine (SVM) and TSMOA-Black Hole Entropy Fuzzy Clustering (BHEFC)[Formula: see text]Neural Network (NN), in which the TSMOA-BHEFC+NN obtained better results. The proposed TSMOA-based BHrEFC+KNN is 0.32%, 0.29%, 50%, and 6.44% is better than the existing TSMOA-BHEFC+NN in terms of accuracy, TPR, MAPE, and RMSE, respectively.

The Hybrid of Jaro-Winkler and Rabin-Karp Algorithm in Detecting Indonesian Text Similarity

The String-matching technique is part of the similarity technique. This technique can detect the similarity level of the text. The Rabin-Karp is an algorithm of string-matching type. The Rabin-Karp is capable of multiple patterns searching but does not match a single pattern. The Jaro-Winkler Distance algorithm can find strings within approximate string matching. This algorithm is very suitable and gives the best results on the matching of two short strings. This study aims to overcome the shortcomings of the Rabin-Karp algorithm in the single pattern search process by combining the Jaro-Winkler and Rabin-Karp algorithm methods. The merging process started from pre-processing and forming the K-Gram data. Then, it was followed by the calculation of the hash value for each K-Gram by the Rabin-Karp algorithm. The process of finding the same hash score and calculating the percentage level of data similarity used the Jaro-Winkler algorithm. The test was done by comparing words, sentences, and journal abstracts that have been rearranged. The average percentage of the test results for the similarity level of words in the combination algorithm has increased. In contrast, the results of the percentage test for the level of similarity of sentences and journal abstracts have decreased. The experimental results showed that the combination of the Jaro-Winkler algorithm on the Rabin-Karp algorithm can improve the similarity of text accuracy.

Web-based E-Commerce Products Grouping

E-Commerce is a group of technologies, applications, and businesses that connect companies or individuals as consumers to conduct electronic transactions, exchange of goods or information through the internet or television, www, or other computer networks. The rapid growth of E-Commerce is caused by the many conveniences for buyers and sellers. This can be seen by the increasing number of E-Commerce sites such as Bukalapak.com, Tokopedia.com, and so on. The buyers want to get products with the lowest prices, the best quality, and from trusted sellers. This encourages them to search for and compare the same products from one store to another, even between different sites. Variables that they compare vary. Some are based on price, number of reviews, and so on. But it is troublesome when they have to look for the same product from another site. This study used the product name as a reference in grouping the same product from 5 different sites. The algorithm used to group online shop products was Jaro-Winkler Distance. The study was to create a system that can group products from 5 different sites with the highest level of similarity product as the output. From the 20 times of the search tests, the accuracy of the online shop product grouping with Jaro-Winkler Distance was 81.27%.