String Similarity Research Articles

Miary podobieństw łańcuchów znakowych a deduplikacja rekordów w bibliograficznych bazach danych

Teza/cel artykułu – Celem artykułu jest przedstawienie metody deduplikacji/ łączenia (ang. deduplication/linkage) rekordów opisujących jednostki bibliograficzne w bazach danych opartej na miarach podobieństw łańcuchów znakowych. Algorytm opracowano na podstawie własnych doświadczeń nabytych podczas tworzenia bibliograficznej bazy danych oraz podczas realizacji badań bibliometrycznych, na podstawie publicznie dostępnych bibliograficznych baz danych. Formalny opis metody zilustrowano przykładami zaczerpniętymi z krajowej bibliograficznej bazy CYTBIN. Metody badawcze – Opracowanie metody wymagało przeglądu architektur informacyjnych wybranych krajowych bibliograficznych baz danych, określenia typologii problemów ich dotykających, wynikających nie tylko z przyjętych modeli składowania danych, ale i budowy graficznych interfejsów użytkownika, którymi są zasilane, analizy i wyboru miar podobieństw łańcuchów znakowych oraz ostatecznie zaproponowania miary złożonej umożliwiającej ewaluację podobieństwa rekordów bibliograficznych w oparciu o wartości ich atrybutów składowych. Wyniki – Przedstawione na przykładzie danych pochodzących z wybranej bazy bibliograficznej wyniki pozwoliły empirycznie zweryfikować użyteczność zaproponowanej metody. Dodatkowo dokonano analizy rozkładu podobieństwa rekordów bibliograficznych bazy CYTBIN określanego na podstawie zaproponowanej metody złożonej i metody opartej na mierze Jaro-Winkler wyliczanej dla tytułów jednostek bibliograficznych. Wnioski – Zaproponowana metoda, po dostrojeniu jej parametrów do specyfiki (występujących anomalii) konkretnych baz bibliograficznych, może być wprost zastosowana do poprawy jakości opisów bibliograficznych w nich gromadzonych, zarówno w proaktywnym modelu pracy (przed zatwierdzeniem opisu przez operatora), jak i modelu reaktywnym (weryfikacja wszystkich lub nowo zgromadzonych rekordów wykonywana np. w czasie mniejszego obciążenia systemu w dobowych odstępach czasu).

De indeling van de dialecten in Noord-Limburg en het aangrenzende Duitse gebied

Abstract Mapping dialects in North-Limburg and the adjacent German region. How relevant is the Uerdinger line as dividing line? The Uerdingen isogloss is claimed to define the border between the Kleverlandish and South Lower Franconian dialect areas, both in the Netherlands and Germany. In earlier research we concluded that the border between these two dialect areas had to be reduced to other isoglosses. In this study we aim to give a definite answer on the basis of the Wenker survey data. As in our earlier studies, we applied the strategy of selecting the dialect variants of the most common words on both sides of the state border. We again applied two research methods in computing linguistic distances, one based on carefully classifying linguistic features, the other one based on string distances. The results turned out to converge. We found two dialect areas in Germany and three in the Netherlands. The Uerdinger line was not a main dialect border, on either side of the state border. We found breaks in the old dialect continuum crossing the state border. On the Dutch side a middle zone has appeared that could emerge because of the state border of 1815. The result is that the border of the Kleverlandish and South Franconian dialect areas runs more to the north in the Netherlands.

Estudo para integração entre a Plataforma Lattes a Biblioteca Digital Brasileira de Teses e Dissertações (BDTD) e o Banco de Teses e Dissertações da Capes

Este trabalho exibe uma estudo para integração entre a Biblioteca Digital Brasileira de Teses e Dissertações (BDTD), o Banco Teses e Dissertações da CAPES e a base nacional de currículos de pesquisadores (Plataforma Lattes). Ao todo foram analisados mais de 2 milhões de registros e foram adotados alguns procedimentos computacionais para coletar, normalizar e transformar os dados, além da aplicação de algoritmos de similaridade de strings para identificação de registros comuns entre as bases. Como resultado, observou-se que mais de 240 mil registros são estão na intersecção de BDTD e BTD CAPES, e que a Plataforma Lattes contém quase que a totalidade dos registros destas duas bases.

Toponym matching through deep neural networks

ABSTRACTToponym matching, i.e. pairing strings that represent the same real-world location, is a fundamental problemfor several practical applications. The current state-of-the-art relies on string similarity metrics, either specifically developed for matching place names or integrated within methods that combine multiple metrics. However, these methods all rely on common sub-strings in order to establish similarity, and they do not effectively capture the character replacements involved in toponym changes due to transliterations or to changes in language and culture over time. In this article, we present a novel matching approach, leveraging a deep neural network to classify pairs of toponyms as either matching or nonmatching. The proposed network architecture uses recurrent nodes to build representations from the sequences of bytes that correspond to the strings that are to be matched. These representations are then combined and passed to feed-forward nodes, finally leading to a classification decision. We present the results of a wide-ranging evaluation on the performance of the proposed method, using a large dataset collected from the GeoNames gazetteer. These results show that the proposed method can significantly outperform individual similarity metrics from previous studies, as well as previous methods based on supervised machine learning for combining multiple metrics.

FrepJoin: an efficient partition-based algorithm for edit similarity join

String similarity join (SSJ) is essential for many applications where near-duplicate objects need to be found. This paper targets SSJ with edit distance constraints. The existing algorithms usually adopt the filter-andrefine framework. They cannot catch the dissimilarity between string subsets, and do not fully exploit the statistics such as the frequencies of characters. We investigate to develop a partition-based algorithm by using such statistics. The frequency vectors are used to partition datasets into data chunks with dissimilarity between them being caught easily. A novel algorithm is designed to accelerate SSJ via the partitioned data. A new filter is proposed to leverage the statistics to avoid computing edit distances for a noticeable proportion of candidate pairs which survive the existing filters. Our algorithm outperforms alternative methods notably on real datasets.

Learning to combine multiple string similarity metrics for effective toponym matching

ABSTRACTSeveral tasks related to geographical information retrieval and to the geographical information sciences involve toponym matching, that is, the problem of matching place names that share a common referent. In this article, we present the results of a wide-ranging evaluation on the performance of different string similarity metrics over the toponym matching task. We also report on experiments involving the usage of supervised machine learning for combining multiple similarity metrics, which has the natural advantage of avoiding the manual tuning of similarity thresholds. Experiments with a very large dataset show that the performance differences for the individual similarity metrics are relatively small, and that carefully tuning the similarity threshold is important for achieving good results. The methods based on supervised machine learning, particularly when considering ensembles of decision trees, can achieve good results on this task, significantly outperforming the individual similarity metrics.

Crowd-Guided Entity Matching with Consolidated Textual Data

Entity matching (EM) identifies records referring to the same entity within or across databases. Existing methods using structured attribute values (such as digital, date or short string values) may fail when the structured information is not enough to reflect the matching relationships between records. Nowadays more and more databases may have some unstructured textual attribute containing extra consolidated textual information (CText) of the record, but seldom work has been done on using the CText for EM. Conventional string similarity metrics such as edit distance or bag-of-words are unsuitable for measuring the similarities between CText since there are hundreds or thousands of words with each piece of CText, while existing topic models either cannot work well since there are no obvious gaps between topics in CText. In this paper, we propose a novel cooccurrence-based topic model to identify various sub-topics from each piece of CText, and then measure the similarity between CText on the multiple sub-topic dimensions. To avoid ignoring some hidden important sub-topics, we let the crowd help us decide weights of different sub-topics in doing EM. Our empirical study on two real-world datasets based on Amzon Mechanical Turk Crowdsourcing Platform shows that our method outperforms the state-of-the-art EM methods and Text Understanding models.

LS-Join: Local Similarity Join on String Collections

String similarity join, as an essential operation in applications including data integration and data cleaning, has attracted significant attention in the research community. Previous studies focus on global similarity join. In this paper, we study local similarity join with edit distance constraints, which finds string pairs from two string collections that have similar substrings. We study two kinds of local similarity join problems: checking local similar pairs and locating local similar pairs. We first consider the case where if two strings are locally similar to each other, they must share a common gram of a certain length. We show how to do efficient local similarity verification based on a matching gram pair. We propose two pruning techniques and an incremental method to further improve the efficiency of finding matching gram pairs. Then, we devise a method to locate the longest similar substring pair for two local similar strings. We conducted a comprehensive experimental study to evaluate the efficiency of these techniques.

A Novel Technique for Detecting Plagiarism in Documents Exploiting Information Sources

Plagiarism takes place when we use any person’s work without giving due acknowledgment. There are several fields where the text similarity is involved like web document retrieval, information mining, and searching related articles. Several approaches have been introduced for detecting plagiarism in the text documents based on the syntactic structure of the text, string similarity, fingerprinting, semantic meaning underlying the text, etc. The basic limitation of plagiarism detection systems these days is that they fail to detect tough cases of plagiarism. The proposed plagiarism detection approach is the hybrid of semantic and syntactic similarity between the text documents. This novel approach exploits linguistic information sources non-linearly using the lexical database for finding the relatedness between text documents. The proposed approach uses semantic knowledge to perform cognitive-inspired computing. The framework is capable of detecting intelligent plagiarism cases like a verbatim copy, paraphrasing, rewording in a sentence, and sentence transformation. The approach has been evaluated on the standard PAN-PC-11 dataset. The experiments show that our technique has outperformed other strong baseline techniques in terms of precision, recall, F-measure, and plagiarism detection (PlagDet) score.

Learning abstract visual concepts via probabilistic program induction in a Language of Thought

The ability to learn abstract concepts is a powerful component of human cognition. It has been argued that variable binding is the key element enabling this ability, but the computational aspects of variable binding remain poorly understood. Here, we address this shortcoming by formalizing the Hierarchical Language of Thought (HLOT) model of rule learning. Given a set of data items, the model uses Bayesian inference to infer a probability distribution over stochastic programs that implement variable binding. Because the model makes use of symbolic variables as well as Bayesian inference and programs with stochastic primitives, it combines many of the advantages of both symbolic and statistical approaches to cognitive modeling. To evaluate the model, we conducted an experiment in which human subjects viewed training items and then judged which test items belong to the same concept as the training items. We found that the HLOT model provides a close match to human generalization patterns, significantly outperforming two variants of the Generalized Context Model, one variant based on string similarity and the other based on visual similarity using features from a deep convolutional neural network. Additional results suggest that variable binding happens automatically, implying that binding operations do not add complexity to peoples’ hypothesized rules. Overall, this work demonstrates that a cognitive model combining symbolic variables with Bayesian inference and stochastic program primitives provides a new perspective for understanding people’s patterns of generalization.

BIOSSES: a semantic sentence similarity estimation system for the biomedical domain.

MotivationThe amount of information available in textual format is rapidly increasing in the biomedical domain. Therefore, natural language processing (NLP) applications are becoming increasingly important to facilitate the retrieval and analysis of these data. Computing the semantic similarity between sentences is an important component in many NLP tasks including text retrieval and summarization. A number of approaches have been proposed for semantic sentence similarity estimation for generic English. However, our experiments showed that such approaches do not effectively cover biomedical knowledge and produce poor results for biomedical text.MethodsWe propose several approaches for sentence-level semantic similarity computation in the biomedical domain, including string similarity measures and measures based on the distributed vector representations of sentences learned in an unsupervised manner from a large biomedical corpus. In addition, ontology-based approaches are presented that utilize general and domain-specific ontologies. Finally, a supervised regression based model is developed that effectively combines the different similarity computation metrics. A benchmark data set consisting of 100 sentence pairs from the biomedical literature is manually annotated by five human experts and used for evaluating the proposed methods.ResultsThe experiments showed that the supervised semantic sentence similarity computation approach obtained the best performance (0.836 correlation with gold standard human annotations) and improved over the state-of-the-art domain-independent systems up to 42.6% in terms of the Pearson correlation metric.Availability and implementationA web-based system for biomedical semantic sentence similarity computation, the source code, and the annotated benchmark data set are available at: http://tabilab.cmpe.boun.edu.tr/BIOSSES/.

An Elusive Method to Identify Isomorphism and Inversions of Kinematic Chains and Mechanisms

Objectives: To detect structural similarity i.e. isomorphism between the kinematic chains and the number of inversions i.e. distinct mechanisms possessed by a kinematic chain. Methods/Statistical Analysis: A shortest distance matrix and the corresponding relative shortest distance matrix are generated in a completely different manner. Thereafter a relative shortest distance string for each link is obtained whose similarity indicates number of inversions i.e. distinct mechanisms in a kinematic chain. Then relative shortest distance link value and the corresponding relative shortest distance chain value are obtained whose similarity indicates isomorphism between kinematic chains. Findings: The proposed method is applied to a family of 40 kinematic chains with 9 links and 2 degree of freedom. Relative shortest distance string table reveals that the total numbers of inversions i.e. distinct mechanisms are 254 and relative shortest distance chain value table indicates that all the chains of the family are not isomorphic. Many other methods have been applied, for the same family, in the past by many other researchers of repute and the results obtained by present method are in full consensus with the results of the previous researchers. The proposed method provides two unique structural invariants i.e. shortest distance string and relative shortest distance chain value as strongest and simplest tools for detection of isomorphism and number of inversions of kinematic chains. Application/Improvements: Like many other methods applied by our parent researchers, with due respect, the proposed method does not involve long and complicated calculations, therefore it is easy, efficient and reliable.

A Novel Cost-Based Model for Data Repairing

Integrity constraint based data repairing is an iterative process consisting of two parts: detect and group errors that violate given integrity constraints (ICs); and modify values inside each group such that the modified database satisfies those ICs. However, most existing automatic solutions treat the process of detecting and grouping errors straightforwardly (e.g., violations of functional dependencies using string equality), while putting more attention on heuristics of modifying values within each group. In this paper, we propose a revised semantics of violations and data consistency w.r.t. a set of ICs. The revised semantics relies on string similarities, in contrast to traditional methods that use syntactic error detection using string equality. Along with the revised semantics, we also propose a new cost model to quantify the cost of data repair by considering distances between strings. We show that the revised semantics provides a significant change for better detecting and grouping errors, which in turn improves both precision and recall of the following data repairing step. We prove that finding minimum-cost repairs in the new model is NP-hard, even for a single FD. We devise efficient algorithms to find approximate repairs. In addition, we develop indices and optimization techniques to improve the efficiency. Experiments show that our approach significantly outperforms existing automatic repair algorithms in both precision and recall.

Efficient string similarity join in multi-core and distributed systems

In big data area a significant challenge about string similarity join is to find all similar pairs more efficiently. In this paper, we propose a parallel processing framework for efficient string similarity join. First, the input is split into some disjoint small subsets according to the joint frequency distribution and the interval distribution of strings. Then the filter-verification strategy is adopted in the computation of string similarity for each subset so that the number of candidate pairs is reduced before an effective pruning strategy is used to improve the performance. Finally, the operation of string join is executed in parallel. Para-Join algorithm based on the multi-threading technique is proposed to implement the framework in a multi-core system while Pada-Join algorithm based on Spark platform is proposed to implement the framework in a cluster system. We prove that Para-Join and Pada-Join cannot only avoid reduplicate computation but also ensure the completeness of the result. Experimental results show that Para-Join can achieve high efficiency and significantly outperform than state-of-the-art approaches, meanwhile, Pada-Join can work on large datasets.

B-BabelNet: Business-Specific Lexical Database for Improving Semantic Analysis of Business Process Models

Similarity calculation between business process models has an important role in managing repository of business process model. One of its uses is to facilitate the searching process of models in the repository. Business process similarity is closely related to semantic string similarity. Semantic string similarity is usually performed by utilizing a lexical database such as WordNet to find the semantic meaning of the word. The activity name of the business process uses terms that specifically related to the business field. However, most of the terms in business domain are not available in WordNet. This case would decrease the semantic analysis quality of business process model. Therefore, this study would try to improve semantic analysis of business process model. We present a new lexical database called B-BabelNet. B-BabelNet is a lexical database built by using the same method in BabelNet. We attempt to map the Wikipedia page to WordNet database but only focus on the word related to the domain of business. Also, to enrich the vocabulary in the business domain, we also use terms in the business-specific online dictionary (businessdictionary.com). We utilize this database to do word sense disambiguation process on business process model activity’s terms. The result from this study shows that the database can increase the accuracy of the word sense disambiguation process especially in particular terms related to the business and industrial domains.

Efficient String Similarity Join in Multi-core and Distributed Systems

Efficient String Similarity Join in Multi-core and Distributed Systems

A Novel SSPS Framework for String Similarity Join

A Novel SSPS Framework for String Similarity Join

A Comparative Study for String Metrics and the Feasibility of Joining them as Combined Text Similarity Measures

This paper aims to introduce an optimized Damerau–Levenshtein and dice-coefficients using enumeration operations (ODADNEN) for providing fast string similarity measure with maintaining the results accuracy; searching to find specific words within a large text is a hard job which takes a lot of time and efforts. The string similarity measure plays a critical role in many searching problems. In this paper, different experiments were conducted to handle some spelling mistakes. An enhanced algorithm for string similarity assessment was proposed. This algorithm is a combined set of well-known algorithms with some improvements (e.g. the dice-coefficient was modified to deal with numbers instead of characters using certain conditions). These algorithms were adopted after conducting on a number of experimental tests to check its suitability. The ODADNN algorithm was tested using real data; its performance was compared with the original similarity measure. The results indicated that the most convincing measure is the proposed hybrid measure, which uses the Damerau–Levenshtein and dicedistance based on n-gram of each word to handle; also, it requires less processing time in comparison with the standard algorithms. Furthermore, it provides efficient results to assess the similarity between two words without the need to restrict the word length.

Efficient String Edit Similarity Join Algorithm

String similarity join is a basic and essential operation in many applications. In this paper, we investigate the problem of string similarity join with edit distance constraints. A trie-based edit similarity join framework has been proposed recently. The main advantage of existing trie-based algorithms is support for similarity join on short strings. The main problem is when joining long and distant strings. These methods generate and maintain lots of similar prefixes called active nodes which need to be further removed in a subsequent pruning phase. With large edit distance, the number of active nodes becomes quite large. In this paper, we propose a new trie-based join algorithm called PreJoin, which improves upon current trie-based join methods. It efficiently finds all similar string pairs using a novel active-node generation method, which minimizes the number of generated active nodes by applying the pruning heuristics early in the generation process. The performance of PreJoin is scaled in two different ways: First, a dynamic reordering of the trie index is used to accelerate the search for similar string pairs. Second, a partitioning method of string space is used to improve performance on large edit distance thresholds. Experiments show that our approach is highly efficient for processing short as well as long strings, and outperforms the state-of-the-art trie-based join approaches by a factor five.

Automatic Schema-Independent Linked Data Instance Matching System

The goal of linked data instance matching is to detect all instances that co-refer to the same objects in two linked data repositories, the source and the target. Since the amount of linked data is rapidly growing, it is important to automate this task. However, the difference between the schemata of source and target repositories remains a challenging barrier. This barrier reduces the portability, accuracy, and scalability of many proposed approaches. The authors present automatic schema-independent interlinking (ASL), which is a schema-independent system that performs instance matching on repositories with different schemata, without prior knowledge about the schemata. The key improvements of ASL compared to previous systems are the detection of useful attribute pairs for comparing instances, an attribute-driven token-based blocking scheme, and an effective modification of existing string similarities. To verify the performance of ASL, the authors conducted experiments on a large dataset containing 246 subsets with different schemata. The results show that ASL obtains high accuracy and significantly improves the quality of discovered coreferences against recently proposed complex systems.